CXL.io Question

27.1 ended by naming three protocols. This chapter is the follow-up question about the first of them, and it is the one most likely to be asked because it is the one with an easy wrong answer available.

How does CXL.io relate to PCIe? — "it is PCIe". That is true. It is also the whole of what most candidates say, and it leaves five things unsaid that the question was actually about.

1. The Engineering Problem — It Is PCIe, And That Is The First Of Six Things

The link trains as PCIe and is asked to become CXL. A negotiation that does not complete leaves sixteen lanes up, a working device and no CXL at all — and every link-status instrument reports a healthy link. Section 5.

CXL.io is mandatory, not one of three options. A device whose io path is gone is not a degraded CXL device; no configuration read is answered, no capability is found, and the device is absent. Section 6.

One wire, three protocols. Sixteen, thirty-two and forty-eight gigabytes a second of demand on a sixty-four gigabyte link is ninety-six of demand and a thirty-two gigabyte shortfall — the bandwidth is shared, not summed. Section 7.

The semantics are PCIe's and the framing is not. Sixty-four payload bytes cost eighty-four in a TLP and sixty-eight in a flit, and costing CXL.io at PCIe framing is wrong in the same direction every time. Section 8.

The memory windows are programmed over io. CXL.mem traffic never touches the io path and cannot begin until the io path has told the device which addresses it owns. Section 9.

This chapter against 27.1, stated precisely. That one owns the shape of an answer. This one owns one answer's content — which is why every model here is about the io-to-PCIe relationship rather than about how to say it, and why section 14's weak definition is the sentence this chapter opened with.

2. The One-Sentence Model

CXL.io is PCIe in the sense that matters when it is present and mandatory, when the link negotiated up to CXL rather than staying PCIe, when its bandwidth is understood as shared with two other protocols, when its framing is understood to be flits rather than TLPs, when the PCIe ordering rules are understood to cover it and not the other two, and when the PCIe error path is understood the same way — and "it is PCIe" is one of those six.

3. What This Chapter Owns

The content first, because the models are about the relationship and the relationship needs stating.

CXL.io carries the discovery and management plane, with PCIe's transaction semantics. Configuration reads and writes, enumeration, capability structures, interrupts, and non-coherent DMA. A driver that knows how to talk to a PCIe device knows how to talk to this, because the transactions mean what the same PCIe transactions mean.

It is mandatory on every CXL device. There is no CXL device without CXL.io, because without it there is nothing to find the device with. Section 6 is about the consequence.

It shares the link with CXL.cache and CXL.mem. The three are multiplexed over one physical link — one set of lanes, one link budget — and the multiplexing is what the flit format exists to do. Section 7 is about the bandwidth consequence and section 8 about the framing one.

The link becomes CXL by negotiation, not by being CXL. Training starts as PCIe. An alternate-protocol negotiation during training decides whether the link comes up speaking CXL or stays speaking PCIe, and it can fail while leaving a perfectly serviceable PCIe link behind. Section 5 owns this and it is the single fact most likely to distinguish a good answer from a memorised one.

It is the management plane for the other two. The host address decoders that tell a device which physical addresses it owns are programmed through CXL.io configuration writes. CXL.mem traffic then flows without touching io again — but it could not have started without it. Section 9.

Its rules do not extend to the other two. PCIe's producer-consumer ordering guarantees cover CXL.io transactions. PCIe's advanced error reporting covers CXL.io errors. Neither covers CXL.cache or CXL.mem, which have their own ordering semantics and their own error paths. Sections 10 and 11 are the two halves of this, and it is the part of the relationship most likely to be assumed rather than checked.

4. Teaching-Model Boundary

Every model in this chapter is a teaching model, not a protocol implementation. It computes the one relationship the section is about and nothing else. There is no TLP, no flit encoder, no configuration space and no link trainer anywhere in this file.

Each model is built twice from one source. A parameter selects between the measured build, which computes what the relationship actually is, and the just-PCIe build, which computes what follows from treating CXL.io as nothing but PCIe. The two are instantiated side by side against identical stimulus, and every section's headline number is the gap between them.

5. RTL 1 — The Link Trains As PCIe And Is Asked To Become CXL

Start with the fact that makes the PCIe relationship concrete rather than asserted, because it is the one that turns "it is PCIe" from a slogan into a mechanism.

A CXL link does not come up as CXL. It comes up as PCIe — the same training, the same link-training-and-status state machine, the same electrical negotiation — and during that training an alternate protocol is offered and either accepted or not. If it is accepted, the link runs CXL. If it is not, the link runs PCIe, and everything above it is a PCIe device.

The failure this creates is quiet. The link is up. It is at full width, at full speed, with no errors. Every instrument that reports on links reports a healthy one, and the device behind it is not the device anybody expected.

// RTL 1 - the link trains as PCIe and is asked to become CXL. A negotiation
// that does not complete leaves a working link carrying a device that is not
// the device anybody ordered.
module alt_negotiation #(parameter int A_TRAINED_LINK_IS_A_CXL_LINK = 0) (
  input  logic clk, rst_n,
  input  logic        train_it,
  input  logic [15:0] lanes, lanes_trained, alt_offered, alt_accepted,
  output logic [15:0] width_up, alt_ok, cxl_lanes, pcie_lanes,
  output logic        came_up_cxl,
  output logic [7:0]  n_trains, n_fellback,
  output logic        negotiation_err
);
  logic        truly_pcie_only;
  // A training report cannot bring up more lanes than the slot has.
  assign width_up  = (lanes_trained > lanes) ? lanes : lanes_trained;
  // The alternate protocol is accepted only if it was also offered.
  assign alt_ok    = ((alt_offered != 16'd0) && (alt_accepted != 16'd0))
                     ? 16'd1 : 16'd0;
  assign cxl_lanes  = (alt_ok != 16'd0) ? width_up : 16'd0;
  assign pcie_lanes = width_up - cxl_lanes;
  assign truly_pcie_only = (width_up != 16'd0) && (alt_ok == 16'd0);
  // A link that trained is treated as a link that came up CXL.
  assign came_up_cxl = (A_TRAINED_LINK_IS_A_CXL_LINK != 0)
                       ? (width_up != 16'd0) : (alt_ok != 16'd0);
  assign negotiation_err = train_it && truly_pcie_only && came_up_cxl;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_trains <= 8'd0; n_fellback <= 8'd0;
    end else if (train_it) begin
      n_trains <= n_trains + 8'd1;
      if (truly_pcie_only) n_fellback <= n_fellback + 8'd1;
    end
  end
endmodule

Sixteen lanes trained with the alternate protocol offered and not accepted is sixteen lanes of ordinary PCIe: no CXL lanes at all, on a link that trained perfectly. The trained-is-CXL view reports that the link came up CXL, because from where it is standing the link came up.

Figure 1 — the same link read two ways. The upper path is not a broken instrument: the link genuinely is up, at full width, with no errors, and every layer below the protocol negotiation is working exactly as designed. The lower path asks the one extra question. The gap between them is a single bit in a capability structure, and it is the difference between a memory expander and a PCIe card.

The third case separates the two halves of the negotiation. Acceptance without an offer is not a negotiation — the model requires both, because a report that one end accepted something the other never proposed is an instrument artefact, and treating it as success would manufacture a CXL link out of a log inconsistency.

The last case is worth distinguishing carefully because it is the one that gets confused with this. A CXL link that trained narrow — four of sixteen lanes, alternate protocol accepted — is a width problem. It is CXL, it is degraded, and it is 26.2's subject rather than this one. A candidate who conflates "did not come up CXL" with "came up narrow" has two different failures with two different fixes collapsed into one.

The clamp is worth one line because of what it says about the data rather than the model. A training report naming more lanes than the slot has is what a stale configuration file or a hot-plug event caught mid-flight produces, and it is common enough in bring-up logs that a model which believed it would compute a negative PCIe lane count and report a fabric that does not exist.

The answer this section supplies is one sentence long and it is the sentence that makes the PCIe relationship mechanical: CXL and PCIe share the physical layer and the training sequence, and which protocol runs on top is decided during training by an alternate-protocol negotiation.

Two consequences follow from that sentence and both are worth having ready, because they are the natural follow-ups.

The first is that a CXL device is a valid PCIe device by construction. It is not a compatibility mode bolted on afterwards; the PCIe behaviour is what the device does before the negotiation and what it continues doing if the negotiation does not complete. That is why the fallback in section 12 is graceful rather than a failure — the device was never pretending.

The second is that the negotiation needs both ends. A device that supports CXL in a root port that does not gets PCIe, and a root port that supports CXL with a device that does not gets PCIe, and in both cases the link is healthy and neither end is at fault. There is no error to report because nothing went wrong; two components agreed on the highest protocol they both spoke. Every mechanism in the machine for reporting problems is looking for something that went wrong, and nothing did.

6. RTL 2 — CXL.io Is Not Optional

The second fact, and the one that explains why CXL.io is listed first rather than alphabetically.

Every CXL device is found the way every PCIe device is found: by configuration reads walking a bus. The CXL capability itself — the structure that says this is a CXL device and here is what it supports — is a capability in configuration space, read over CXL.io like any other. Without the io path there is no read, no capability, no enumeration and no device.

This is why CXL.io is mandatory on all three device types while the other two are optional. It is not a ranking of importance; it is a statement about what discovery requires.

// RTL 2 - CXL.io is not optional. Every CXL device is discovered, configured
// and interrupted over it, so a device whose io path is broken is not a
// degraded CXL device; it is an absent one.
module io_mandatory #(parameter int THE_OTHER_TWO_ARE_THE_DEVICE = 0) (
  input  logic clk, rst_n,
  input  logic        enumerate,
  input  logic [15:0] io_present, cfg_reads, cfg_ok, dvsec_ok,
  output logic [15:0] cfg_served, cfg_failed, cfg_pct, caps_found,
  output logic        device_visible,
  output logic [7:0]  n_enums, n_invisible,
  output logic        no_io_err
);
  logic [31:0] c_q;
  logic        truly_invisible;
  // Configuration reads are answered only while the io path is present.
  assign cfg_served = (io_present == 16'd0) ? 16'd0
                    : ((cfg_ok > cfg_reads) ? cfg_reads : cfg_ok);
  assign cfg_failed = cfg_reads - cfg_served;
  assign c_q = (cfg_reads == 16'd0) ? 32'd100
             : (({16'd0, cfg_served} * 32'd100) / {16'd0, cfg_reads});
  assign cfg_pct = (c_q > 32'd100) ? 16'd100 : c_q[15:0];
  // The CXL capability structure is read over io like any other capability.
  assign caps_found = ((io_present != 16'd0) && (dvsec_ok != 16'd0))
                      ? 16'd1 : 16'd0;
  assign truly_invisible = (io_present == 16'd0) && (cfg_reads != 16'd0);
  // Treating the memory and cache paths as the device makes io look optional.
  assign device_visible = (THE_OTHER_TWO_ARE_THE_DEVICE != 0)
                          ? 1'b1 : (caps_found != 16'd0);
  assign no_io_err = enumerate && truly_invisible && device_visible;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_enums <= 8'd0; n_invisible <= 8'd0;
    end else if (enumerate) begin
      n_enums <= n_enums + 8'd1;
      if (truly_invisible) n_invisible <= n_invisible + 8'd1;
    end
  end
endmodule

A device with its io path gone answers none of sixteen configuration reads, finds no capability and is not visible — regardless of what its memory and cache paths can do. The other-two-are-the-device view reports a present device, because the paths it is looking at are present.

The fifth case is the one that keeps this from being a slogan. An io path that works perfectly on a device with no CXL capability is a working PCIe device — all sixteen reads answered, nothing found, and the model correctly reports it as not a visible CXL device rather than as a broken one. That distinction is the whole of section 12.

The sixth case is the other boundary. An io path answering a quarter of its reads is a flaky device and a different failure entirely; this model owns the absent case and declines to claim the degraded one. A model that reported both would be less useful, because the two have different causes and different fixes.

The degenerate case matters because it is the state a machine is in before anything happens. An enumeration that was never attempted proves nothing — no reads, no failures, no conclusion — and a model that reported an absent device from an absent enumeration would fire on every cold boot.

There is a framing of this section that lands better in an interview than the mandatory-versus-optional phrasing. CXL.io is not the first of three protocols; it is the protocol the other two are configured through and discovered by. Calling it mandatory is accurate and sounds like a specification detail. Saying that a device without it cannot be found says the same thing and explains why.

The same framing covers the device types cleanly. Type 1 is io plus cache, Type 2 is all three, Type 3 is io plus mem — and io appears in all three lists not because it was thought important but because removing it would remove the device from the bus. A question about why io is in every device type is really a question about whether you understand discovery, and the answer is one sentence.

7. RTL 3 — One Wire, Three Protocols

The third fact, and the first one that is genuinely counter-intuitive rather than merely unstated.

CXL.io, CXL.cache and CXL.mem are three protocols. They are not three links. They are multiplexed over one physical link with one lane count and one link budget, and the bandwidth available to all three together is what that one link carries.

The mistake this invites is a specific arithmetic one: adding up what the three protocols want and comparing it to what one of them could have had. It produces a number that is larger than the link and reads as a capability.

// RTL 3 - one wire, three protocols. The bandwidth of a CXL link is not the
// sum of what the three protocols want; it is one link's worth, divided among
// whichever of them is asking.
module shared_bandwidth #(parameter int EACH_PROTOCOL_GETS_THE_LINK = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] link_bw, io_demand, cache_demand, mem_demand,
  output logic [15:0] total_demand, served_bw, shortfall, served_pct,
  output logic        shared_ok,
  output logic [7:0]  n_measures, n_oversubscribed,
  output logic        added_err
);
  logic [15:0] true_served;
  logic [31:0] s_q;
  logic        truly_over;
  assign total_demand = io_demand + cache_demand + mem_demand;
  // One link carries what one link carries.
  assign true_served = (total_demand > link_bw) ? link_bw : total_demand;
  assign served_bw = (EACH_PROTOCOL_GETS_THE_LINK != 0) ? total_demand : true_served;
  assign shortfall = total_demand - true_served;
  assign s_q = (total_demand == 16'd0) ? 32'd100
             : (({16'd0, true_served} * 32'd100) / {16'd0, total_demand});
  assign served_pct = (s_q > 32'd100) ? 16'd100 : s_q[15:0];
  assign shared_ok = (served_bw <= link_bw);
  assign truly_over = (total_demand > link_bw) && (link_bw != 16'd0);
  assign added_err = measure && truly_over && (served_bw == total_demand);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_oversubscribed <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (truly_over) n_oversubscribed <= n_oversubscribed + 8'd1;
    end
  end
endmodule

Sixteen, thirty-two and forty-eight gigabytes a second of demand on a sixty-four gigabyte link is ninety-six of demand, sixty-four served and a thirty-two gigabyte shortfall. The each-gets-the-link view serves all ninety-six on a wire that carries sixty-four.

Figure 2 — three protocols and one set of lanes. The upper path is the arithmetic that follows from thinking of CXL.io, CXL.cache and CXL.mem as three things rather than three views of one thing. It produces a capability number larger than the link, which is the shape to watch for: any bandwidth figure that exceeds the link rate has summed the protocols somewhere.

The third case is driven deliberately at the boundary. Demand exactly at the link rate is not oversubscribed — sixty-four of demand on sixty-four of link is fully served, with no shortfall, and a model that alarmed at the boundary would alarm on every well-sized system. The mutation campaign has a mutation that moves that boundary by one and it is killed by this case.

The last case is the one that connects to 26.6. One protocol can oversubscribe the link on its own — a hundred gigabytes a second of CXL.mem demand on a sixty-four gigabyte link is thirty-six short before the other two have asked for anything. The sharing is not a fairness question; it is a capacity one, and adding the three of them together makes it harder to see rather than easier.

The degenerate case is the honest limit. A link with no bandwidth serves nothing in both builds, and the model declines to call it oversubscribed — an absent link is a different fault with a different owner.

The practical form of this section is a habit rather than a calculation, and it is worth stating as one. Any CXL bandwidth figure larger than the link rate has summed the protocols somewhere. That is a check anybody can apply to a number in a slide without knowing where it came from, and it catches the mistake in the form it usually arrives in — not as an explicit sum, but as a per-protocol figure quoted without saying that the protocols share.

It is also the reason section 7 sits before section 8 rather than after. The sharing is why the framing is what it is: fixed-size flits exist so that three protocols can interleave on one link with predictable arbitration, and a variable-length packet format cannot do that. The two facts are one design decision seen from two angles, and an answer that gives them in that order is telling a story rather than listing properties.

8. RTL 4 — The Semantics Are PCIe's And The Framing Is Not

This is the sentence that separates a good answer from a correct one, and it is the part of "CXL.io is PCIe" that is most precisely half-true.

The semantics are PCIe's. A CXL.io configuration read means what a PCIe configuration read means. The address spaces, the completion rules, the transaction types — all of it is PCIe, which is exactly why existing drivers work.

The framing is not. PCIe moves variable-length transaction-layer packets with per-packet overhead. CXL moves fixed-size flits, because fixed-size flits are what let three protocols share one link with predictable interleaving. A CXL.io transaction travels inside flits, and the cost of moving it is a flit cost rather than a TLP cost.

// RTL 4 - the semantics are PCIe's and the framing is not. A CXL.io
// transaction means what the same PCIe transaction means and does not travel
// the way it travels, and confusing the two makes every bandwidth estimate
// wrong in the same direction.
module framing_difference #(parameter int SAME_SEMANTICS_SAME_WIRE = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] payload_bytes, tlp_overhead, flit_bytes, flit_payload,
  output logic [15:0] tlp_frame, flit_frame, frame_used, efficiency_pct,
  output logic        framing_same,
  output logic [7:0]  n_measures, n_mismatched,
  output logic        framing_err
);
  logic [15:0] slots_needed, carried;
  logic [31:0] n_q, f_q, e_q;
  logic        truly_differs;
  assign tlp_frame = payload_bytes + tlp_overhead;
  // A flit is a fixed size and carries a fixed payload, whatever is in it.
  assign n_q = (flit_payload == 16'd0) ? 32'd0
             : (({16'd0, payload_bytes} + {16'd0, flit_payload} - 32'd1)
                / {16'd0, flit_payload});
  assign slots_needed = (n_q > 32'hFFFF) ? 16'hFFFF : n_q[15:0];
  // Taken in 32 bits: the 16-bit product wraps silently and reports a frame
  // smaller than the payload it carries.
  assign f_q = {16'd0, slots_needed} * {16'd0, flit_bytes};
  assign flit_frame = (f_q > 32'hFFFF) ? 16'hFFFF : f_q[15:0];
  assign frame_used = (SAME_SEMANTICS_SAME_WIRE != 0) ? tlp_frame : flit_frame;
  assign carried = (payload_bytes > frame_used) ? frame_used : payload_bytes;
  assign e_q = (frame_used == 16'd0) ? 32'd0
             : (({16'd0, carried} * 32'd100) / {16'd0, frame_used});
  assign efficiency_pct = (e_q > 32'd100) ? 16'd100 : e_q[15:0];
  assign framing_same = (flit_frame == tlp_frame);
  assign truly_differs = (flit_frame != tlp_frame) && (payload_bytes != 16'd0);
  assign framing_err = measure && truly_differs && (frame_used == tlp_frame);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_mismatched <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (truly_differs) n_mismatched <= n_mismatched + 8'd1;
    end
  end
endmodule

Sixty-four payload bytes with twenty of TLP overhead is an eighty-four byte packet; the same payload in sixty-eight-byte flits carrying sixty-four is one flit of sixty-eight — ninety-four percent efficient against the TLP's seventy-six. The same-wire view charges the TLP framing, which is wrong in the same direction every time.

The second case is the one that makes the mistake survivable in practice.A note on what this section is not claiming first, because the arithmetic invites an overreach. Flit framing is not simply more efficient than TLP framing. It is fixed-size, which means a small transaction pays for a whole flit whether it needs it or not, and a payload of four bytes costs sixty-eight the same as a payload of sixty-four does. The example in the model happens to favour flits because the payload fits one exactly; shift the payload down and the comparison inverts. The claim is that the two are different and that using the wrong one is wrong by a fixed fraction, not that either is universally better — and a candidate who says flits are more efficient has made a claim a follow-up will find.

A TLP that happens to be the same size as its flit produces identical numbers either way, and a back-of-envelope estimate on a payload that happens to land near that size will be right for the wrong reason. The multi-flit case shows the divergence growing: two hundred and fifty-six payload bytes is four flits of sixty-eight against a two-hundred-and-seventy-six byte packet.

The degenerate case says something worth knowing about the model rather than the protocol. A flit format with no payload configured carries nothing, and the same-wire view happily charges the TLP anyway — which is the shape of every framing estimate made without asking what the flit layout actually is.

The clamp is the one that bit this chapter during authoring and is worth recording. The flit total is taken in thirty-two bits, because a sixteen-bit product of slot count and flit size wraps silently and reports a frame smaller than the payload it carries. 26.7 section 16 has the same defect found the same way.

One more thing belongs with this section because it is the most common follow-up. The flit format is not the same in every CXL generation, and a bandwidth estimate that uses the wrong one is wrong by a different fixed fraction rather than being approximately right. An answer that says "flits rather than TLPs, and the flit layout depends on the generation and the link width" has said something checkable and has invited a question it can answer. An answer that quotes a specific flit size as though it were universal has made a claim that a follow-up will find.

The general shape is the one 27.1 section 12 priced: the supported version of a claim is the version that survives the next question, and here the support is a dependency rather than a justification.

9. RTL 5 — The Memory Windows Are Programmed Over io

The fifth fact, and the one that turns "io is the management plane" from a phrase into a dependency with consequences.

A CXL.mem device does not know which physical addresses it owns. The host decides, and tells it — through configuration writes over CXL.io, programming the host-managed device memory decoders that map a range of the host's physical address space onto the device. Once those are set, CXL.mem traffic flows without touching the io path again.

The dependency is therefore asymmetric and one-directional in time: io before mem, always, and never again afterwards. A memory path that is perfectly healthy cannot serve a single request until an io path it will never use has finished talking to it.

// RTL 5 - the memory windows are programmed over io. CXL.mem traffic never
// touches the io path and cannot begin until the io path has told the device
// which addresses it owns.
module mgmt_plane #(parameter int MEM_IS_INDEPENDENT_OF_IO = 0) (
  input  logic clk, rst_n,
  input  logic        bring_up,
  input  logic [15:0] windows_needed, cfg_writes_ok, decoders_set, mem_requests,
  output logic [15:0] windows_live, windows_dark, live_pct, mem_answered,
  output logic        mem_usable,
  output logic [7:0]  n_bringups, n_dark,
  output logic        dependency_err
);
  logic [15:0] true_live, programmed;
  logic [31:0] l_q;
  logic        truly_dark;
  // A decoder is live only if it was programmed and the write that programmed
  // it was answered.
  assign programmed = (decoders_set > windows_needed) ? windows_needed : decoders_set;
  assign true_live  = (cfg_writes_ok == 16'd0) ? 16'd0 : programmed;
  assign windows_live = (MEM_IS_INDEPENDENT_OF_IO != 0) ? windows_needed : true_live;
  assign windows_dark = windows_needed - true_live;
  assign l_q = (windows_needed == 16'd0) ? 32'd100
             : (({16'd0, true_live} * 32'd100) / {16'd0, windows_needed});
  assign live_pct = (l_q > 32'd100) ? 16'd100 : l_q[15:0];
  assign mem_answered = (windows_live == 16'd0) ? 16'd0 : mem_requests;
  assign mem_usable = (windows_live >= windows_needed);
  assign truly_dark = (windows_dark != 16'd0) && (mem_requests != 16'd0);
  assign dependency_err = bring_up && truly_dark && mem_usable;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_bringups <= 8'd0; n_dark <= 8'd0;
    end else if (bring_up) begin
      n_bringups <= n_bringups + 8'd1;
      if (truly_dark) n_dark <= n_dark + 8'd1;
    end
  end
endmodule

Four windows with the decoders set and the configuration writes never landing is four dark windows, none of the memory reachable and no request answered — on a device whose memory subsystem is entirely functional. The mem-is-independent view sees four live windows and a hundred answered requests.

The fourth case is the one that produces the worst debug sessions. Half the decoders programmed is two live windows and two dark ones — a memory map that works for some addresses and not others, with nothing in the memory path to explain why. 26.4 is where that failure lands, and this is where it comes from.

The sixth case is why it ships. Dark windows that nothing addresses cost nothing — no memory traffic, no failures, no symptom — and the model reports the dark windows while declining to call it a fault. That is the state a machine is in between bring-up and first use.

The last case separates two things that look identical from the memory side. A working io path that was never asked to program anything produces exactly the same dark windows as a broken one, and distinguishing them needs evidence from the io path rather than the memory path. That is the practical form of the dependency: when memory does not work, the first question is about configuration, not about memory.

There is a second dependency worth mentioning because it catches people out in the opposite direction. Once the decoders are programmed, CXL.mem traffic genuinely does not touch CXL.io — the memory path is independent at run time, and an io path that degrades after bring-up does not slow memory down. Both halves of that matter: io is required to start and is irrelevant afterwards, and an answer that gives only one half is half wrong in whichever direction it chose.

The failure mode this creates in debugging is a specific one. A memory problem that appeared after a long period of working correctly is not a decoder problem, because the decoders were programmed once and have not changed. A memory problem present from first use very likely is. The time of first occurrence separates the two, and it is the cheapest piece of evidence available.

10. RTL 6 — PCIe Ordering Applies To CXL.io And Nothing Else

Now the two halves of the relationship most likely to be assumed, and the first of them is the one that produces races nobody can reproduce.

PCIe has ordering rules. A posted write followed by a read to the same target completes in order; a producer-consumer pattern built on that guarantee works, and a great deal of driver code is built on it without ever naming it.

Those rules are CXL.io's rules. They are not CXL.cache's and not CXL.mem's, which have their own ordering semantics rooted in the coherency model rather than in the PCIe transaction ordering rules. A pattern that is guaranteed on io is a race on the other two, and the guarantee is invisible — nothing fails when it is assumed wrongly except occasionally, under load, unreproducibly.

// RTL 6 - PCIe ordering applies to CXL.io and to nothing else. A producer-
// consumer pattern that is guaranteed on io is a race on cache and mem, and
// the guarantee is the thing most likely to be assumed rather than checked.
module ordering_scope #(parameter int ONE_LINK_ONE_ORDERING = 0) (
  input  logic clk, rst_n,
  input  logic        check_it,
  input  logic [15:0] io_pairs, other_pairs, io_ordered, other_ordered,
  output logic [15:0] guaranteed, unguaranteed, races, io_in_order,
  output logic [15:0] guarantee_pct,
  output logic        ordering_scoped,
  output logic [7:0]  n_checks, n_racy,
  output logic        ordering_err
);
  logic [15:0] io_ok, other_ok, true_unguaranteed;
  logic [31:0] g_q;
  logic        truly_racy;
  // Ordering holds on io pairs; on the other two it is a property of the
  // traffic, not of the link.
  assign io_ok    = (io_ordered > io_pairs) ? io_pairs : io_ordered;
  assign other_ok = (other_ordered > other_pairs) ? other_pairs : other_ordered;
  assign true_unguaranteed = other_pairs - other_ok;
  // PCIe ordering covers every io pair by rule. The other two protocols are
  // ordered only where the traffic itself ordered them.
  assign guaranteed   = (ONE_LINK_ONE_ORDERING != 0)
                        ? (io_pairs + other_pairs) : (io_pairs + other_ok);
  assign unguaranteed = (io_pairs + other_pairs) - guaranteed;
  assign races = true_unguaranteed;
  assign io_in_order = io_ok;
  assign g_q = ((io_pairs + other_pairs) == 16'd0) ? 32'd100
             : (({16'd0, guaranteed} * 32'd100)
                / {16'd0, (io_pairs + other_pairs)});
  assign guarantee_pct = (g_q > 32'd100) ? 16'd100 : g_q[15:0];
  assign ordering_scoped = (unguaranteed == true_unguaranteed);
  assign truly_racy = (true_unguaranteed != 16'd0);
  assign ordering_err = check_it && truly_racy && (unguaranteed == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_checks <= 8'd0; n_racy <= 8'd0;
    end else if (check_it) begin
      n_checks <= n_checks + 8'd1;
      if (truly_racy) n_racy <= n_racy + 8'd1;
    end
  end
endmodule

Ten io pairs and ten pairs on the other two protocols, with four of the others ordered by the traffic itself, is fourteen pairs guaranteed and six races — seventy percent of the traffic covered by a rule. The one-ordering view guarantees all twenty.

The fifth case is why a PCIe background hides this completely. On a pure io link the two views cannot differ — every pair is an io pair, PCIe ordering covers all of them, and both builds report the same thing. An engineer whose experience is PCIe has never been in a situation where the assumption was wrong, which is exactly the experience that makes it feel safe.

The last case is subtle and was a genuine testbench gap during authoring. Three io pairs observed out of order on an otherwise ordered link is an io-layer bug, not a scoping question: the PCIe rule still guarantees those pairs, and the discrepancy shows up in a separate output rather than in the guarantee count. A model that reduced the guarantee to what was observed would be reporting a link fault as an ordering-scope issue, and until a case existed where the two differed, nothing distinguished the two behaviours.

The practical advice this section supports is narrower than "be careful about ordering", which is advice nobody can act on. It is this: any code path that writes over one protocol and reads over another needs its ordering argument written down. A driver that programs a control register over CXL.io and then expects a CXL.mem read to observe the effect has crossed the boundary, and the fact that it works in testing is not evidence — it is the absence of evidence, which is exactly what makes this class of bug reach production.

The honest caveat is that the coherency model provides its own ordering guarantees, and they are real and are often sufficient. The mistake is not assuming ordering; it is assuming the PCIe rules are the ones providing it, because those rules say nothing about a transaction on another protocol and the guarantee actually in force may be weaker, stronger or differently scoped.

11. RTL 7 — The Error Paths Are Scoped The Same Way

The second half, and the one with the most direct operational consequence.

PCIe advanced error reporting is a mature, well-supported, widely-monitored mechanism. It is also, on a CXL device, a report about CXL.io errors and nothing else. Errors on the cache and memory protocols have their own reporting paths, and a monitor watching only the familiar one sees a device in excellent health.

// RTL 7 - the error paths are scoped the same way the ordering is. CXL.io
// errors are reported through PCIe's advanced error reporting; the other two
// protocols have their own paths, and a monitor watching only the first sees a
// healthy device.
module error_scope #(parameter int AER_SEES_EVERYTHING = 0) (
  input  logic clk, rst_n,
  input  logic        monitor,
  input  logic [15:0] io_errs, cache_errs, mem_errs, aer_enabled,
  output logic [15:0] all_errs, aer_reports, unreported, reported_pct,
  output logic        errors_scoped,
  output logic [7:0]  n_monitors, n_unreported,
  output logic        aer_err
);
  logic [15:0] other_errs, true_reports;
  logic [31:0] r_q;
  logic        truly_unreported;
  assign other_errs = cache_errs + mem_errs;
  assign all_errs   = io_errs + other_errs;
  // Advanced error reporting carries the io errors and only those.
  assign true_reports = (aer_enabled == 16'd0) ? 16'd0 : io_errs;
  assign aer_reports  = (AER_SEES_EVERYTHING != 0)
                        ? ((aer_enabled == 16'd0) ? 16'd0 : all_errs)
                        : true_reports;
  assign unreported = all_errs - true_reports;
  assign r_q = (all_errs == 16'd0) ? 32'd100
             : (({16'd0, true_reports} * 32'd100) / {16'd0, all_errs});
  assign reported_pct = (r_q > 32'd100) ? 16'd100 : r_q[15:0];
  assign errors_scoped = (aer_reports == true_reports);
  assign truly_unreported = (unreported != 16'd0) && (aer_enabled != 16'd0);
  assign aer_err = monitor && truly_unreported && (aer_reports == all_errs);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_monitors <= 8'd0; n_unreported <= 8'd0;
    end else if (monitor) begin
      n_monitors <= n_monitors + 8'd1;
      if (truly_unreported) n_unreported <= n_unreported + 8'd1;
    end
  end
endmodule

Four io errors and eight on the other two protocols is twelve errors on the device and four in the reporting path: a third of them visible. The sees-everything view reports all twelve, which is what an engineer reading a clean error log concludes.

The fifth case is the operational one. Errors only on the other two protocols is eight errors and a completely empty log — not a partial picture, an empty one. A monitoring system built on the familiar mechanism reports a healthy device while the memory protocol is failing, and the first indication is an application complaint.

Worth separating two things that are easy to merge here. The PCIe error path is not deficient and the other protocols' paths are not hidden. Both exist, both are specified, and both work. What is missing is that one of them is the path a decade of tooling already watches and the others are paths somebody has to be told about. The failure is entirely in what is being monitored, which makes it a deployment problem rather than a silicon one — and deployment problems are the kind that persist across generations of hardware because nothing in a new device fixes them.

The last case is the hardest to doubt and therefore the most dangerous. Ten io errors and one elsewhere is ninety percent visible, which means the log is populated, plausible, detailed and slightly incomplete. A log that shows nothing invites suspicion. A log that shows ninety percent of the truth does not.

This section and the previous one are the same fact twice, and saying so in an interview is worth more than either of them alone. PCIe supplies CXL.io's transaction semantics, its ordering rules and its error reporting, and all three are scoped to CXL.io. The relationship is complete and it is bounded, and the boundary is in the same place for all three. A candidate who says that has compressed two follow-ups into one sentence and demonstrated that they are reasoning about the relationship rather than recalling facts about it.

The operational corollary is a single question to ask of any CXL monitoring setup: what would it show if the memory protocol were failing and nothing else was? For a great many deployments the answer is nothing, and the deployment does not know it.

The degenerate case draws the boundary. Reporting turned off entirely is a different fault — no log at all, both views agree, and the model declines to claim it. Scoping is about what a working log covers, not about whether one exists.

12. RTL 8 — A CXL Device In A PCIe Slot

The sixth fact, and the one an interviewer is most likely to reach for as a practical follow-up.

A CXL device in a PCIe slot works. The alternate-protocol negotiation does not complete, the link comes up PCIe, and the device presents as whatever it can be as a PCIe device. For a Type 3 memory expander that is usually very little; for an accelerator with a PCIe personality it may be most of the product.

It works, and it is a different product from the one that was ordered. The question is not whether it functions but whether anybody knows.

// RTL 8 - a CXL device in a PCIe slot. It works, it is useful, and it is a
// different product from the one the purchase order described - which is worth
// knowing before the machine is racked rather than after.
module pcie_fallback #(parameter int IT_WORKS_SO_ITS_FINE = 0) (
  input  logic clk, rst_n,
  input  logic        check_it,
  input  logic [15:0] caps_as_cxl, slot_is_cxl, caps_as_pcie, workload_needs,
  output logic [15:0] caps_live, caps_lost, lost_pct, needs_met,
  output logic        fallback_known,
  output logic [7:0]  n_checks, n_degraded,
  output logic        fallback_err
);
  logic [15:0] pcie_caps, true_lost;
  logic [31:0] p_q;
  logic        truly_degraded;
  // In a PCIe slot the device keeps whatever it can do as a PCIe device.
  assign pcie_caps = (caps_as_pcie > caps_as_cxl) ? caps_as_cxl : caps_as_pcie;
  assign caps_live = (slot_is_cxl != 16'd0) ? caps_as_cxl : pcie_caps;
  assign true_lost = caps_as_cxl - caps_live;
  assign caps_lost = (IT_WORKS_SO_ITS_FINE != 0) ? 16'd0 : true_lost;
  assign p_q = (caps_as_cxl == 16'd0) ? 32'd0
             : (({16'd0, true_lost} * 32'd100) / {16'd0, caps_as_cxl});
  assign lost_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign needs_met = (caps_live > workload_needs) ? workload_needs : caps_live;
  // Knowing the fallback happened is the whole of the requirement.
  assign fallback_known = (IT_WORKS_SO_ITS_FINE != 0)
                          ? (caps_live != 16'd0) : (caps_lost == 16'd0);
  assign truly_degraded = (true_lost != 16'd0) && (workload_needs > caps_live);
  assign fallback_err = check_it && truly_degraded && fallback_known;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_checks <= 8'd0; n_degraded <= 8'd0;
    end else if (check_it) begin
      n_checks <= n_checks + 8'd1;
      if (truly_degraded) n_degraded <= n_degraded + 8'd1;
    end
  end
endmodule

Ten capabilities as CXL and four as PCIe, in a PCIe slot, is six capabilities lost — sixty percent of the device — with four of the eight the workload needed. The it-works view loses nothing, because from its point of view the device came up.

The fifth case is where the it-works view happens to be right, and it is worth conceding cleanly. A fallback the workload survives — eight of ten capabilities live, six needed — is not a fault today. The model reports the loss and does not report an error, because the requirement is knowing rather than never falling back.

The last case is the one both views catch. A device that falls back to nothing is noticed by everything, which is why the partial fallback is the shape that reaches production: enough works that nothing alarms, and not enough works that the workload runs as designed.

Two practical notes belong with this, because the fallback is usually discussed as a risk and it is also a feature.

The fallback is what makes a CXL device safe to buy early. A memory expander in a machine whose root port turns out not to support CXL is not a brick; it is a PCIe device that the machine can still enumerate, power-manage and report on. That property is deliberate and it is a consequence of section 5's design: the device was speaking PCIe before the negotiation and simply keeps doing it.

What is missing is not the fallback but the notification. Nothing in the machine treats a successful negotiation-to-PCIe as an event worth reporting, because from the link's point of view it is a normal outcome. The gap is between the protocol layer, where this is routine, and the operator, for whom it is the difference between the machine they provisioned and the machine they have. Closing that gap is one read at provisioning and it is nobody's job by default.

The degenerate case is the state a fleet is in on day one. A fallback on a machine with no workload yet costs nothing and is invisible, and it is exactly when it is cheapest to detect. The practical answer this section supplies is a single check at provisioning: confirm the link came up CXL, not merely up.

The cost asymmetry here is the largest in the chapter and worth quantifying. The check is one configuration read at provisioning time. The alternative is discovering the fallback from a workload, which means a machine in service, an application complaint, a triage that starts at the application and works down, and — in the common case — a device team spending days proving a working device is working. One read against several days, and the read can be scripted once for a fleet.

The same asymmetry applies to the decoder read-back in section 9 and to the per-protocol error registers in section 11. All three are single reads, all three are cheap to automate once, and all three replace a debugging session that starts from a symptom several layers above the cause. The pattern across the chapter is that the io relationship is almost entirely observable and almost never observed, because the observations are configuration reads rather than traffic, and nothing generates them unless somebody decides to.

13. RTL 9 — Which Fact About io Answers The Question

Nine sections of content. This one is about which of it to say, because the interview question has a time budget that 27.1 already priced.

There are several true statements available about CXL.io and PCIe, and they are not equally useful. It is PCIe is true and says the least. The link trains as PCIe and negotiates up is true and says the most, because it explains the relationship mechanically and invites the follow-up the candidate wants.

// RTL 9 - which fact about io answers the question. The relationship to PCIe
// has several true statements available and they are not equally useful, so
// the cheapest decision in the answer is which one to lead with.
module io_fact_choice #(parameter int SAY_IT_IS_JUST_PCIE = 0) (
  input  logic clk, rst_n,
  input  logic        choose,
  input  logic [15:0] fact_a_value, fact_b_value, fact_c_value, seconds_each,
  output logic [15:0] best_value, chosen_value, value_lost, value_per_sec,
  output logic        best_chosen,
  output logic [7:0]  n_choices, n_wasteful,
  output logic        fact_blind_err
);
  logic [15:0] max_ab;
  logic [31:0] s_q;
  logic        truly_wasteful;
  assign max_ab     = (fact_a_value > fact_b_value) ? fact_a_value : fact_b_value;
  assign best_value = (max_ab > fact_c_value) ? max_ab : fact_c_value;
  // The it-is-just-PCIe answer always leads with the first fact.
  assign chosen_value = (SAY_IT_IS_JUST_PCIE != 0) ? fact_a_value : best_value;
  assign value_lost = best_value - chosen_value;
  // There is time for one fact, so the only thing to optimise is which.
  assign s_q = (seconds_each == 16'd0) ? 32'd0
             : ({16'd0, chosen_value} / {16'd0, seconds_each});
  assign value_per_sec = (s_q > 32'hFFFF) ? 16'hFFFF : s_q[15:0];
  assign best_chosen = (chosen_value >= best_value);
  assign truly_wasteful = (best_value > fact_a_value) && (seconds_each != 16'd0);
  assign fact_blind_err = choose && truly_wasteful && (chosen_value == fact_a_value);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_choices <= 8'd0; n_wasteful <= 8'd0;
    end else if (choose) begin
      n_choices <= n_choices + 8'd1;
      if (truly_wasteful) n_wasteful <= n_wasteful + 8'd1;
    end
  end
endmodule

Three facts worth two, four and nine, with time for one, is nine of value if the best one leads and two if the first one does — seven lost, on a decision that costs nothing to make.

The second case is the honest one and it is why the habit survives. Sometimes the first fact really is the best — for a listener who has just asked whether a CXL device needs a new driver, it is PCIe is precisely the right answer and nothing beats it. The habit is not wrong; it is unconditional, and unconditional is what makes it cost something.

The last case shows the scaling. The better the unsaid fact, the more the habit costs, and the habit does not scale with the stakes because it is not reading them. A question about driver compatibility and a question about link bring-up both get the same four words.

The three facts worth having ranked, for this question specifically: the negotiation is the highest-value one because it is mechanical, checkable and explains the most common failure; the scoping of ordering and errors is second because it changes what somebody writes; and it is PCIe semantically is third — not because it is unimportant, but because it is the one the listener has probably already assumed. Leading with what they already believe spends the opening on confirmation.

That ranking is not universal and the model is explicit about it. For a listener asking whether existing drivers work, the third fact is the first one, and the model's second case is exactly that situation. The decision is cheap, it is available, and the habit is what skips it.

There is a broader point here that applies past this question. The facts about CXL.io are ordered differently for different listeners and the ordering is almost never decided. For a software engineer, the semantics-carry-over fact leads. For someone bringing a board up, the negotiation leads. For someone sizing a system, the shared-bandwidth fact leads. All three are the same body of knowledge with three different entry points, and choosing the entry point takes about a second once it has occurred to you that there is a choice.

27.1 section 13 makes the same argument about ordering within one answer. This is the same argument about which answer to give, and the two compound: the wrong lead in the wrong order spends a ninety-second budget confirming what the listener already knew.

14. RTL 10 — The CXL.io Answer Assembled

Nine models, nine facts about one relationship. This one puts them in one place and makes the weak claim visible as what it is: one bit of six.

"CXL.io is PCIe" is the first thing to say and is not the answer. It is one of six conditions, and the only one that can be said without knowing anything else.

// RTL 10 - the CXL.io answer assembled. Everything that must hold before
// "CXL.io is PCIe" is a complete account of the relationship, with that
// sentence as one of the six rather than the whole claim.
module io_answer_signoff #(parameter int ITS_JUST_PCIE = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       io_present,        // CXL.io is there, and is mandatory
  input  logic       negotiated_up,     // the link came up CXL, not PCIe
  input  logic       bandwidth_shared,  // one link's bandwidth, three protocols
  input  logic       framing_differs,   // flits on the wire, not TLPs
  input  logic       ordering_scoped,   // PCIe ordering covers io and not the rest
  input  logic       errors_scoped,     // AER covers io and not the rest
  output logic       answer_sound,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_sound,
  output logic       false_io_err
);
  assign fail_mask[0] = ~io_present;
  assign fail_mask[1] = ~negotiated_up;
  assign fail_mask[2] = ~bandwidth_shared;
  assign fail_mask[3] = ~framing_differs;
  assign fail_mask[4] = ~ordering_scoped;
  assign fail_mask[5] = ~errors_scoped;
  // The just-PCIe build stops at the first bit.
  assign answer_sound = (ITS_JUST_PCIE != 0) ? io_present : (fail_mask == 6'd0);
  assign false_io_err = evaluate && answer_sound && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_sound <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (answer_sound) n_sound <= n_sound + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When io is present and any one of the other five is misunderstood, the assembled model reports an incomplete answer and the just-PCIe build reports a complete one.

Across the eight evaluations the stimulus drives, the assembled model calls one answer complete and the just-PCIe build calls six of them complete. The five it gets wrong are the five single-bit failures with the io bit set.

The bit order is by how much each one changes what somebody would do. Bit 0 changes whether you look for the device at all. Bit 1 changes whether you believe a link-status register. Bits 2 and 3 change a bandwidth estimate. Bits 4 and 5 change what you can assume in code and what you can trust in a log — and those two are last because they are the ones that cause damage silently rather than immediately.

Figure 4 — the mask read as a follow-up order. Each decision is a question an interviewer can ask after "it is PCIe", and the ordering is by how much the answer changes what somebody would do — the negotiation changes whether you believe a register, and the last two change what you can assume in code and trust in a log.

15. Quantitative Reasoning

Numbers from the models, stated so they can be argued with rather than admired.

Sixteen lanes trained and zero lanes of CXL. Not a degraded link, not a narrow one — a fully healthy PCIe link where a CXL link was expected, reported as up by everything that reports on links.

Zero of sixteen configuration reads answered. With the io path gone, the memory and cache paths are irrelevant: nothing finds the device, so nothing uses them.

Ninety-six of demand on sixty-four of link is a thirty-two gigabyte shortfall. The three protocols share; adding them produces a number larger than the wire.

Sixty-four payload bytes cost eighty-four as a TLP and sixty-eight as a flit. Ninety-four percent efficient against seventy-six, and the error runs the same direction at every size.

Four dark windows and a hundred unanswered requests, on a healthy memory subsystem. The configuration writes never landed, and nothing in the memory path says so.

Fourteen of twenty pairs guaranteed, six races. PCIe ordering covers ten; the other ten are ordered only where the traffic ordered them.

Four of twelve errors in the reporting path is thirty-three percent visible. With no io errors at all it is zero percent, on a device with eight real failures.

Six of ten capabilities lost to a PCIe slot is sixty percent of the device, with four of the eight the workload needed.

Nine of value against two, for the same four seconds. The fact chosen by habit and the fact chosen by the question.

One of eight answers complete; the just-PCIe view counts six. The assembled model's summary number, and the chapter's.

16. Assertions

The testbenches carry 543 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported twenty-three on the first run and none of them was a real gap — every broken build in this chapter was internally consistent from the start, which is the first time in the batch that has happened. The reason is a habit acquired in 26.7, where a build that reported zero dwell and then a full total had to be repaired mid-authoring: a build that is wrong has to be wrong consistently, or a mutation to either half produces a mixture no assertion pins down.

Both builds are asserted on every degenerate case. A dead link, an enumeration never attempted, a link with no bandwidth, a transaction with no payload, a device with no memory map, an idle link with no pairs to order, a healthy device with no errors, a device with no capabilities, nothing worth saying.

Every clamp is driven past its limit exactly once. More lanes trained than the slot has, more configuration successes than reads, more decoders than windows, more ordered pairs than pairs, more PCIe capabilities than CXL ones, a payload large enough to saturate the flit count.

Every error output is checked in both directions in every case. Section 6's fifth case, section 10's fifth case and section 12's fifth case exist entirely to assert the quiet half: a working PCIe device that is not a broken CXL one, a pure io link where the two views cannot differ, and a fallback the workload survives. In all three the measured build must stay silent.

17. Mutation Testing

100 mutations, 100 killed. Forty-eight against the first testbench, fifty-two against the second.

One mutation survived the first run, and it was a real testbench gap rather than an equivalent mutant. Narrowing the ordering guarantee from every io pair by rule to every io pair observed in order changed nothing, because no stimulus case had io pairs arriving out of order — every case either had them all ordered or had the count clamped up to the total.

That is a distinction the model exists to make. PCIe guarantees the ordering of io pairs; whether they were observed in order is a separate fact about the link, and collapsing the two would report a link fault as a scoping question. The fix is a case where they differ: three io pairs out of order on a link with nothing else racing, where the guarantee stays at twenty and the discrepancy shows up in its own output. The mutation died on the next run and the testbench gained the case that distinguishes the chapter's two most easily confused quantities.

Every counter in this chapter was checked against the even-split rule from 26.7 section 17 before the campaign rather than after, and none needed repair — the first chapter in the batch where that check came back clean on the first pass.

18. Verification Strategy

A verification plan for the io relationship, which in practice means a bring-up checklist.

Check the link came up CXL, not merely up. Section 5. This is one register read and it distinguishes the chapter's most common silent failure from a working system.

Check the CXL capability is readable before anything else. Section 6. If the capability structure cannot be read, nothing downstream is worth testing.

Size the bandwidth against one link, not three. Section 7. Any estimate that sums the three protocols' demands and compares to a single-protocol figure is wrong before measurement begins.

Cost transactions at flit granularity. Section 8. A TLP-based estimate is wrong in the same direction at every size, and knowing the flit layout makes it arithmetic.

Verify the decoders are programmed before testing memory. Section 9. A memory test against unprogrammed windows tests nothing and looks like a memory failure.

Do not assume PCIe ordering across protocols. Section 10. A producer-consumer pattern that spans io and mem needs its own ordering argument, and the absence of failures is not evidence.

Monitor the other two protocols' error paths separately. Section 11. The familiar mechanism reports a third of the errors on a mixed-traffic device and none on a memory-only one.

Audit provisioning for slot type. Section 12. A CXL device in a PCIe slot is a configuration fact, checkable once, and expensive to discover from a workload.

19. Synthesis and Implementation Reality

What the relationship costs in a real design, since "it is PCIe" is also a claim about effort.

The PCIe controller is reused and that is most of the value. Configuration space, the transaction layer's semantics, the driver model — all of it carries over, and this is the genuine engineering content behind the one-liner. A team that has shipped PCIe has shipped most of CXL.io.

The flit layer is new and is shared by all three protocols. It is where the multiplexing lives, it is what makes the framing differ, and it is the part that does not come free from a PCIe background. Section 8's arithmetic is a consequence of a design decision made here.

The alternate-protocol negotiation is small and is a hard requirement on both ends. A device that supports it and a root port that does not produces section 5's failure, and neither end is at fault.

The host decoders are a host-side feature with a device-side contract. Section 9's dependency lives in that contract, and the common implementation bug is a device that accepts the configuration writes and does not act on them, which produces dark windows that report as programmed.

Ordering and error scope are documentation problems as much as design ones. Both are correctly implemented far more often than they are correctly understood, and the cost lands on whoever writes the driver or the monitor rather than on whoever built the silicon.

20. Silicon Observability

What can be read from a real device, ordered by cost.

Free, already there. Link status, trained width and speed. These answer nothing in this chapter except that the link is up, which is section 5's entire point.

Cheap. The CXL capability structure in configuration space, and whatever it reports about which protocols are active. This is the direct answer to sections 5 and 6 and is one configuration read.

Cheap, if the registers exist. The host decoder registers, read back after programming. Section 9's dark windows are visible here and nowhere else, and reading back what was written is the single most valuable habit in CXL bring-up.

Moderate. Per-protocol traffic counters, which give section 7's sharing directly. These are less common than they should be, and their absence turns a measurement into an inference.

Moderate. The other two protocols' error registers, which are section 11's subject. They exist; they are simply not where anybody is looking.

Expensive. A protocol analyser, which shows the flit stream and therefore answers section 8 by direct observation rather than arithmetic. 26.7 is about using one well.

21. Debug Lab

A CXL memory expander in a new machine. The operating system sees a PCIe device and no additional memory.

Step 1 — read the link status. It is up, at full width, at full speed. This takes seconds and, on its own, means very little.

Step 2 — read the CXL capability structure. Section 6. If it is absent, the device is presenting as PCIe and step 3 explains why. If it is present, the device is CXL and the problem is downstream.

Step 3 — check whether the alternate-protocol negotiation completed. Section 5. A link that trained as PCIe and stayed there is the most common cause of exactly this symptom, and the most common cause of that is the slot.

Step 4 — confirm the slot is CXL-capable. Section 12. One provisioning fact, and it ends a large fraction of these sessions.

Step 5 — if the device is CXL and the memory is still absent, read back the host decoders. Section 9. Written and not acted on is the failure to look for, and reading back is the only way to see it.

Step 6 — if some memory works and some does not, count the live windows. Section 9's fourth case. A partial map is the shape that produces 26.4's symptoms.

Step 7 — if throughput is disappointing rather than absent, size it against one link. Section 7, then 26.6.

Step 8 — if the error log is clean and the behaviour is not, look at the other two protocols' error registers. Section 11.

The order is by cost and by how much each step eliminates. Steps 1 through 4 are all configuration reads and together they resolve most occurrences of this symptom.

22. Design Review

Questions worth asking about the io relationship before a board exists.

Does the root port support the alternate-protocol negotiation, and is that verified rather than assumed?

Is the CXL capability structure readable at the earliest point in enumeration?

Are there per-protocol traffic counters? Without them, section 7's sharing is an inference.

What is the flit layout, and is it documented where a bandwidth estimate would find it?

Are the host decoder registers readable back after programming? This is the single highest-value observability item in the chapter.

Where are the cache and memory protocol error registers, and is anything monitoring them?

Is slot type checked at provisioning, or discovered by a workload?

Does the driver assume PCIe ordering for anything that crosses protocols?

23. How This Appears In Real Engineering

The machine is racked, the device is installed, and the operating system reports a PCIe device with no additional memory. The device team is called, and the device is fine.

What happened is usually section 5 and usually because of section 12: the slot was not CXL-capable, the negotiation did not complete, and the link came up PCIe. Everything reported healthy at every layer, because at every layer everything was healthy. The mismatch was between what the link negotiated and what the purchase order assumed, and nothing in the machine is responsible for noticing that.

The second shape is section 9. The device is CXL, the capability is present, the decoders were programmed and the memory does not appear. The configuration writes were accepted and not acted on, and from the memory side there is no evidence at all — no failed request, no error, no traffic. Reading the decoders back is a five-second check that is skipped because the writes returned successfully.

The third is section 11, and it is slower and more expensive than both. A device is monitored through the familiar error path for weeks. The path is correct, the monitoring is correct, and it covers a third of the device's errors — or none of them, on a memory-only workload. The first indication is an application complaint, and the elapsed time between the first error and the first indication is the cost.

A fourth shape deserves a mention because it is the one that damages a team rather than a machine. Sections 10 and 11 both produce failures that are statistically rare, load-dependent and unreproducible on demand — a race that fires under contention, an error that is real and invisible. Neither has a clean reproduction, and the absence of one has a predictable organisational consequence: the report is treated as unreliable, closed for lack of evidence, and reopened three times over six months by three different people who each rediscover the same absence of evidence.

The evidence exists; it is simply not in the place anybody is looking. The ordering argument is in a specification rather than in a log, and the memory-protocol errors are in a register nothing polls. Both are cheap to check once somebody knows to check them, which is the entire reason these two sections are in a chapter about an interview question rather than only in 26.1 onward.

The pattern under all three is that the PCIe relationship is real enough to be trusted past where it holds. The controller is reused, the semantics carry over, the drivers work, the error path is familiar — and each of those true things makes the next assumption slightly more comfortable. The six bits are the places where the comfort runs out.

24. Common Misconceptions

"CXL.io is just PCIe." It is PCIe, and that is one of six things worth saying. This is the chapter.

"The link is up, so it's a CXL link." Training is PCIe. Becoming CXL is a negotiation that can fail with the link still up. Section 5.

"The device has memory and cache paths, so io doesn't matter much." Without io there is no enumeration, so there is no device. Section 6.

"Three protocols, so three times the bandwidth." One wire. Section 7.

"Same transactions, so same overhead." Same semantics, different framing. Flits, not TLPs. Section 8.

"CXL.mem doesn't use io, so io can't break it." io programs the decoders that mem depends on. Section 9.

"PCIe ordering guarantees this." On CXL.io it does. On the other two it does not. Section 10.

"The error log is clean." It covers CXL.io. Section 11.

"It came up, so the slot is fine." It came up as PCIe. Section 12.

25. Interview Reasoning

"How does CXL.io relate to PCIe?" It is PCIe semantically — same transactions, same configuration space, same driver model — carried in CXL flits rather than TLPs, sharing one link with two other protocols. The link trains as PCIe and negotiates up to CXL. That is the answer; everything below is a follow-up.

"Why is CXL.io mandatory when the other two are optional?" Because discovery goes through it. The CXL capability is a configuration-space capability read over io, so without io there is nothing to enumerate and no device. The reasoning being tested is whether you distinguish important from load-bearing.

"A CXL device is in a machine and the OS sees a PCIe device. What happened?" The alternate-protocol negotiation did not complete — most often because the slot is not CXL-capable. The link is up and healthy; it is simply PCIe. A strong answer notes that nothing in the machine will report this as an error.

"CXL.mem traffic doesn't touch CXL.io. Can an io problem break memory?" Yes, once. The host decoders that tell the device which addresses it owns are programmed over io. After that, mem runs independently — but it cannot start.

"Your driver relies on PCIe producer-consumer ordering. Does that hold on a CXL device?" For CXL.io transactions, yes. Across protocols, no — CXL.cache and CXL.mem have their own ordering semantics. The dangerous part is that assuming otherwise produces a rare, load-dependent, unreproducible race rather than a failure.

"Your monitoring uses PCIe AER. What are you missing?" Everything that is not a CXL.io error. On a memory-only workload that is potentially all of it, with a completely clean log.

26. Exercises

1. A link has 8 lanes, trains all 8, and the alternate protocol is offered but not accepted. How many CXL lanes are there? What does a link-status register report, and what single read would have told you the truth?

2. A device's io path answers 3 of 20 configuration reads and the CXL capability is found. Is the device visible? Is it healthy? Which of the two failures does section 6's model own?

3. CXL.io wants 10, CXL.cache wants 40 and CXL.mem wants 40 GB/s on a 64 GB/s link. Compute the total demand, served bandwidth and shortfall. Now remove CXL.cache entirely and recompute.

4. A 128-byte payload has 20 bytes of TLP overhead; flits are 68 bytes carrying 64. Compute both frame sizes and both efficiencies. At what payload size do the two frame costs coincide?

5. A device needs 8 host decoder windows. 8 are written and the configuration writes are not acknowledged. How many windows are live, and what does a memory test see? Now acknowledge the writes for 5 of them.

6. A workload has 20 CXL.io ordered pairs and 30 pairs spanning cache and mem, of which 12 are ordered by the traffic. How many pairs are guaranteed, and how many are races? What fraction of the traffic is covered by a rule?

7. A device has 2 io errors, 9 cache errors and 4 mem errors, with AER enabled. What percentage of its errors appear in the PCIe error log? At what mix does the log become actively misleading rather than merely incomplete?

8. Extend the assembled model with a seventh bit for a property of the io relationship this chapter does not cover. Justify its position using the rule that the ordering is by how much each bit changes what somebody would do.

27. Summary

CXL.io is PCIe, and that is the first of six things to say.

The link trains as PCIe and negotiates up. A negotiation that fails leaves a healthy link and no CXL, and nothing reports it.

CXL.io is mandatory because discovery goes through it. No io, no enumeration, no device — whatever the other two paths can do.

One wire, three protocols. The bandwidth is shared, and adding the three demands produces a number larger than the link.

The semantics are PCIe's and the framing is not. Flits, not TLPs, and a TLP-based estimate is wrong the same way at every size.

The memory windows are programmed over io. CXL.mem never touches the io path and cannot start without it.

PCIe ordering covers CXL.io and not the other two — and assuming otherwise produces a race rather than a failure.

PCIe error reporting covers CXL.io and not the other two. A clean log on a memory-only device means nothing at all.

A CXL device in a PCIe slot works, and is a different product. The requirement is knowing, not never falling back.

Six bits, and "it is PCIe" is one of them. One answer of eight is complete; the just-PCIe view counts six.

27.3 takes the second protocol, where the device caches the host's memory and the coherency protocol crosses the link.