Why CXL Uses PCIe

Module 4 took the CXL stack apart layer by layer and finished by classifying each layer's relationship to PCIe: inherited, extended, added.

That answered what is shared. Module 5 answers the harder question underneath it — why the reuse exists at all, and what the machinery that exploits it actually has to do. This chapter is the argument; 5.2 is the inventory; 5.3, 5.4 and 5.5 are the mechanisms.

1. The Engineering Problem — A Faster Interconnect Nobody Can Buy

Suppose you have designed a genuinely better interconnect. Lower latency than PCIe, native coherency semantics, memory expansion that works. On paper you have solved Chapter 1.1's memory wall.

Now ship it.

You need a host CPU with your root complex in it — a silicon design decision taken three to four years before the part is on sale. You need a connector, and a mechanical standard, and a card form factor. You need signal-integrity engineering at 32 GT/s or faster, retimers, and a compliance programme so that a card from one vendor works in a board from another. You need every OS to enumerate your devices. You need every server OEM to route your traces. And you need all of that to happen before the first customer can evaluate whether your protocol is actually better, because until then there is nothing to plug in.

This is the real problem CXL faced, and it is not a protocol problem. A protocol's technical merit does not determine whether it gets deployed. Its ability to reach hardware does. That constraint, not coherency theory, is why CXL is built on PCIe.

2. The One-Sentence Model

CXL rides on PCIe because the expensive, slow, ecosystem-wide parts of an interconnect — the electricals, the connector, the deployed slots, the enumeration model — are the parts CXL had no reason to reinvent; so CXL spends its novelty budget entirely on the semantics PCIe cannot express, and inherits everything else.

Call it the novelty budget. Every new standard has one, it is small, and spending it on a connector is spending it on nothing.

3. What This Chapter Owns

Deliberately not repeated here: the per-layer classification from 4.5 and the memory-wall economics from 1.1. This chapter assumes both.

4. The Deployed Base, In Numbers

The usual phrasing — "CXL reuses PCIe for ecosystem reasons" — is true and carries no engineering weight until you count.

Consider a market with H host platforms and D device designs. Each host-device pair either works or does not.

A new independent interconnect. A device only works with a host that implemented the new root complex. If Hn hosts and Dn devices adopt it, the number of working pairs is Hn x Dn. Every pair where either end lacks the new interconnect is dead — not degraded, dead, because there is no shared lower layer to fall back to.

Riding on PCIe. Every pair links, because every pair shares PCIe. Pairs where both ends are CXL-capable additionally reach CXL.

The CXL-capable count is identical. The reuse does not buy you more coherent links. It buys you zero dead pairs, and that is the whole argument.

Put numbers on a plausible first generation: H = 10 server platforms of which Hn = 3 shipped CXL support, D = 40 device designs of which Dn = 12 are CXL devices.

• New interconnect: 3 x 12 = 36 working pairs out of 400. 91% of the matrix is dead.
• On PCIe: 400 working pairs, 36 of them coherent.

Now ask the question a device vendor actually asks: if I build this device, how much of the market can use it at all? Under the new interconnect, a CXL device reaches 3 of 10 platforms. Under reuse it reaches 10 of 10 — three of them coherently, seven as ordinary PCIe devices.

5. The Combinatorial Consequence — What You Must Validate

The same H x D grid that saves the ecosystem creates the validation problem, and it is worth being precise about which cells exist.

Each end is independently either capable or not. That is a 2x2:

Three of four cells must operate as plain PCIe. Only one cell is the feature. A controller that is validated only in the bottom-right cell has tested 25% of its combinatorial obligation, and the three untested cells are the ones a customer hits first, because early in a generation most partners are not capable.

This asymmetry is worth stating plainly and is easy to get backwards in an interview:

The compatibility path is not the fallback case. Statistically, early in a generation, it is the common case.

6. What Building a New PHY Would Have Cost

The ecosystem argument is the strong one, but there is a schedule argument beside it, and it is concrete.

Exactly one row is new under reuse. Every other row is a multi-year, multi-company effort with no coherency benefit whatsoever — you would be spending four years to arrive at the starting line PCIe was already standing on.

There is a risk dimension too, and it is separate from schedule. Signal integrity at 32 and 64 GT/s is empirical work: it is proven by measurement across thousands of real boards, connectors, and cable assemblies. A new PHY does not merely take time, it takes deployment to become trustworthy, and there is no way to compress that. Inheriting PCIe's PHY means inheriting the field evidence with it.

7. Backwards Compatibility Is a Design Constraint, Not a Courtesy

It is tempting to file compatibility under goodwill. It is not; it is load-bearing, and three properties follow from it directly.

1. The link must come up before anyone knows what the partner is. Capability cannot be a precondition for linking, because discovering capability requires a link. This forces the ordering that 5.3 and 5.5 both depend on.
2. Reset must land in the compatible mode. A link whose reset state is CXL has no safety net when CXL is what is broken. RTL 3 below makes this concrete.
3. Errors fall back, never forward. A degraded CXL link becomes a PCIe link. It does not become a faster CXL link, and it does not stay CXL and hope.

These are not three policies. They are one property — the compatible mode is the floor — stated at reset, at link-up, and at error.

8. Teaching-model boundary

9. RTL 1 — The Two-Key Rule

Everything in Module 5 rests on one invariant: CXL operation requires both ends, and neither end may decide alone. Isolated at its simplest:

module two_key_gate #(
  parameter bit LOCAL_ONLY = 1'b0     // 1 = decide from local capability alone
) (
  input  logic clk,
  input  logic rst_n,
  input  logic local_cxl_capable,
  input  logic peer_cxl_capable,
  input  logic peer_info_valid,   // we have actually HEARD from the peer
  input  logic base_link_up,
  output logic cxl_permitted,
  output logic pcie_mode,
  output logic decided,
  output logic assumed_peer_err,  // decided without hearing the peer
  output logic unilateral_err     // CXL permitted without peer support
);
  logic both;
  assign both = local_cxl_capable && peer_cxl_capable;
 
  // Three terms, all required. Dropping any one is a real defect shape.
  // LOCAL_ONLY is the broken shape: it turns CXL on from its own capability
  // alone, without waiting to hear whether the partner can speak it.
  assign cxl_permitted = base_link_up &&
                         (LOCAL_ONLY ? local_cxl_capable
                                     : (peer_info_valid && both));
  assign pcie_mode     = base_link_up && !cxl_permitted;
  assign decided       = base_link_up && (peer_info_valid || LOCAL_ONLY);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      assumed_peer_err <= 1'b0; unilateral_err <= 1'b0;
    end else begin
      // Stated about the OUTPUT, not about the expression that produced it:
      // a checker rewritten from the design equation can never fail.
      if (cxl_permitted && !peer_info_valid)  assumed_peer_err <= 1'b1;
      if (cxl_permitted && !peer_cxl_capable) unilateral_err   <= 1'b1;
    end
  end
endmodule

Note the three terms in cxl_permitted. They are not decoration — each is a distinct failure if omitted:

Simulated, with both parameterisations side by side:

  local YES peer YES heard : both-keys=1 local-only=1
  local YES peer NO  heard : both-keys=0 local-only=1  <-- unilateral
  local NO  peer YES heard : both-keys=0 local-only=0
  local YES peer YES UNHEARD: both-keys=0 local-only=1  <-- assumed
  local-only gate: assumed_peer_err=1 unilateral_err=1

Row 3 is the one people find surprising: a capable peer is not sufficient. Capability is a property of the pair, and AND is the only operator that expresses that.

10. RTL 2 — One Shared Lower Service

The reuse argument says the expensive layer underneath is built once and both modes ride it. A design that duplicates the lower service per mode has paid for exactly the thing reuse was supposed to avoid — so the invariant is that both paths are gated by the same readiness.

module shared_path_select #(
  parameter bit DUPLICATE_LOWER = 1'b0
) (
  input  logic        clk, rst_n, req_valid,
  input  logic        req_is_cxl_class, mode_is_cxl, lower_ready,
  output logic        to_pcie_path, to_cxl_path, refuse,
  output logic [15:0] n_pcie_q, n_cxl_q, n_refused_q,
  output logic        cxl_class_in_pcie_mode_err,
  output logic        lower_bypassed_err
);
  logic gated;
  // Everything crosses the SAME lower service. That is the reuse.
  assign gated = req_valid && (DUPLICATE_LOWER ? 1'b1 : lower_ready);
 
  assign to_cxl_path  = gated &&  req_is_cxl_class && mode_is_cxl;
  assign to_pcie_path = gated && !req_is_cxl_class;
  assign refuse       = req_valid && !(to_pcie_path || to_cxl_path);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_pcie_q <= '0; n_cxl_q <= '0; n_refused_q <= '0;
      cxl_class_in_pcie_mode_err <= 1'b0; lower_bypassed_err <= 1'b0;
    end else begin
      if (to_pcie_path) n_pcie_q    <= n_pcie_q + 16'd1;
      if (to_cxl_path)  n_cxl_q     <= n_cxl_q + 16'd1;
      if (refuse)       n_refused_q <= n_refused_q + 16'd1;
      if (req_is_cxl_class && !mode_is_cxl && (to_pcie_path || to_cxl_path))
        cxl_class_in_pcie_mode_err <= 1'b1;
      if ((to_pcie_path || to_cxl_path) && !lower_ready)
        lower_bypassed_err <= 1'b1;
    end
  end
endmodule

  PCIe-class, CXL mode, lower ready : pcie=1 cxl=0 refuse=0
  CXL-class,  CXL mode, lower ready : pcie=0 cxl=1 refuse=0
  CXL-class,  PCIe mode             : pcie=0 cxl=0 refuse=1  <-- refused
  CXL-class,  lower NOT ready       : correct refuse=1 | duplicated-lower cxl=1
  duplicated-lower lower_bypassed_err=1

Row 1 is the compatibility guarantee in one line: PCIe-class traffic is unaffected by the mode. Row 3 is its complement — a CXL-class request on a PCIe-mode link is refused, not silently downgraded, because there is no correct PCIe rendering of a coherent operation.

11. RTL 3 — Reset Lands in the Mode That Always Works

Section 7's second property, isolated. RESET_TO_CXL = 1 is the unsafe shape, kept as a parameterisation so the testbench can prove the diagnostic fires.

module safe_reset_mode #(
  parameter bit RESET_TO_CXL = 1'b0     // 1 = the unsafe reset shape
) (
  input  logic       clk, rst_n, promote_to_cxl, link_error,
  output logic [1:0] mode_q,
  output logic       is_compat_mode, unsafe_reset_err, stale_after_error_err
);
  localparam logic [1:0] COMPAT = 2'd0, CXLM = 2'd1;
  logic err_q;                      // the error, one cycle old
  assign is_compat_mode = (mode_q == COMPAT);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      // The whole point: reset enters the mode that is always legal.
      mode_q <= RESET_TO_CXL ? CXLM : COMPAT;
      err_q <= 1'b0;
      unsafe_reset_err <= 1'b0; stale_after_error_err <= 1'b0;
    end else begin
      if (link_error)          mode_q <= COMPAT;   // fall back, never forward
      else if (promote_to_cxl) mode_q <= CXLM;
 
      if (RESET_TO_CXL && (mode_q == CXLM) && !promote_to_cxl)
        unsafe_reset_err <= 1'b1;
      // The invariant is about the cycle AFTER the error: on the error cycle
      // itself the mode has not had an edge to change on yet. Checking the
      // state rather than the outcome flags the normal fallback path.
      err_q <= link_error;
      if (err_q && (mode_q == CXLM)) stale_after_error_err <= 1'b1;
    end
  end
endmodule

  safe-reset  : mode=0 compat=1  <-- always-legal mode
  unsafe-reset: mode=1 compat=0  <-- no safe mode to fall back to
  promoted   : safe mode=1 unsafe mode=1
  after error: safe mode=0 (compat) unsafe mode=0

Note the priority: link_error is checked before promote_to_cxl, so an error and a promotion arriving together resolve to COMPAT. Reverse the two branches and you have a link that promotes itself out of its own recovery path.

12. RTL 4 — Did the Link Actually Reach CXL Mode?

The reuse argument is only falsifiable if a platform can say what fraction of its links reached CXL. A link that silently stayed in compatibility mode is a real and quiet failure — everything works, nothing is coherent, and the memory expansion you paid for is not there.

module mode_usage_counters (
  input  logic        clk, rst_n, link_up, in_cxl_mode, fell_back,
  output logic [15:0] cycles_up_q, cycles_cxl_q, cycles_compat_q, n_fallback_q,
  output logic        accounting_err
);
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      cycles_up_q <= '0; cycles_cxl_q <= '0; cycles_compat_q <= '0;
      n_fallback_q <= '0; accounting_err <= 1'b0;
    end else begin
      if (link_up) begin
        cycles_up_q <= cycles_up_q + 16'd1;
        if (in_cxl_mode) cycles_cxl_q    <= cycles_cxl_q + 16'd1;
        else             cycles_compat_q <= cycles_compat_q + 16'd1;
      end
      if (fell_back) n_fallback_q <= n_fallback_q + 16'd1;
      // Conservation: every up-cycle is in exactly one mode.
      if (cycles_up_q != cycles_cxl_q + cycles_compat_q) accounting_err <= 1'b1;
    end
  end
endmodule

  up=60 cxl=40 compat=20 fallbacks=1
  CXL-mode fraction = 66%

Counters are not checkers. These four would happily report nonsense; the conservation line is what makes them a verification structure. cycles_up_q == cycles_cxl_q + cycles_compat_q says every up-cycle is in exactly one mode, and it is independent of how any individual counter is written — which is precisely why mutation M6 could not survive it.

13. Assertions

Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog. Not executed; each maps to the procedural stand-in in the testbench and to a mutation below.

// SAFETY -------------------------------------------------------------------
// V1 — CXL is never permitted without the peer having been heard.
a_no_assumption: assert property (@(posedge clk) disable iff (!rst_n)
  cxl_permitted |-> peer_info_valid);
 
// V2 — CXL is never permitted unless BOTH ends are capable.
a_two_keys: assert property (@(posedge clk) disable iff (!rst_n)
  cxl_permitted |-> (local_cxl_capable && peer_cxl_capable));
 
// V3 — the link is in exactly one mode whenever it is up.
a_one_mode: assert property (@(posedge clk) disable iff (!rst_n)
  base_link_up |-> (cxl_permitted ^ pcie_mode));
 
// V4 — CXL-class traffic never moves on a PCIe-mode link.
a_class_respects_mode: assert property (@(posedge clk) disable iff (!rst_n)
  (req_is_cxl_class && !mode_is_cxl) |-> !(to_pcie_path || to_cxl_path));
 
// V5 — no path bypasses the shared lower service.
a_lower_shared: assert property (@(posedge clk) disable iff (!rst_n)
  (to_pcie_path || to_cxl_path) |-> lower_ready);
 
// V6 — CONSERVATION: every up-cycle is in exactly one mode.
a_mode_conserved: assert property (@(posedge clk) disable iff (!rst_n)
  cycles_up_q == cycles_cxl_q + cycles_compat_q);
 
// V7 — an error returns the link to the compatible mode by the next cycle.
a_error_falls_back: assert property (@(posedge clk) disable iff (!rst_n)
  link_error |=> is_compat_mode);
 
// LIVENESS -----------------------------------------------------------------
// V8 — a link that is up with both ends capable eventually decides.
//      ENVIRONMENT ASSUMPTION: peer_info_valid is eventually asserted. Without
//      an unreachable partner this cannot be proven, and it is not a bug.
a_eventually_decides: assert property (@(posedge clk) disable iff (!rst_n)
  base_link_up |-> s_eventually decided);

V1 through V7 are safety: nothing bad happens. V8 is liveness and is only meaningful under a stated environment assumption — a partner that never responds makes it unprovable, and that is a property of the environment, not a defect in the gate. Chapter 5.3 turns that assumption into a timeout, which is how a real design discharges it.

14. Mutation Testing

Six mutations, each a defect shape a real controller could plausibly contain. Run against the clean testbench; clean code restored after each.

6/6 killed, 0 escaped

Two of these six only got killed after the testbench was fixed, and the two failures are more instructive than the four successes.

M2 escaped initially because the assertion had been derived from the design equation (§9's callout). The lesson generalises: an assertion you can obtain from the RTL by substitution is not independent evidence.

M5 escaped initially for a different reason. On the correct instance lower_bypassed_err is unreachable — gated already requires lower_ready, so the condition cannot hold. The diagnostic's only meaningful exercise is on the DUPLICATE_LOWER = 1 instance, and the testbench merely printed that flag rather than requiring it. Deleting the check therefore changed no verdict.

The fix was a positive test on the diagnostic: at end of simulation, each deliberately-broken variant must have raised its flag.

  duplicated-lower variant flagged lower_bypassed_err   : ok
  local-only gate flagged assumed_peer_err              : ok
  local-only gate flagged unilateral_err                : ok
  unsafe-reset variant flagged unsafe_reset_err         : ok

A checker that never fires proves nothing. It passes whether or not it works, and the only way to tell the difference is to make it fail on purpose.

15. Debug Lab

// The link is up and traffic flows. Report health.
assign link_healthy = base_link_up;

  up=60 cxl=0 compat=60 fallbacks=0
  CXL-mode fraction = 0%
  link_healthy=1   <-- "healthy" all the way down

// Health has two questions, not one.
assign link_healthy    = base_link_up;
assign link_in_mode    = base_link_up && in_cxl_mode;
assign silent_downgrade = base_link_up && cxl_expected && !in_cxl_mode;

// Both ends advertise CXL. Enable it.
assign cxl_permitted = base_link_up && local_cxl_capable && peer_cxl_capable;

  local YES peer YES UNHEARD: both-keys=0
  assumed_peer_err=0   unilateral_err=0

assign cxl_permitted = base_link_up && peer_info_valid &&
                       local_cxl_capable && peer_cxl_capable;

// Optimistic: come up in the fast mode.
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) mode_q <= CXLM;
  else if (promote_to_cxl) mode_q <= CXLM;
  else if (link_error)     mode_q <= COMPAT;

  unsafe-reset: mode=1 compat=0  <-- no safe mode to fall back to
  unsafe_reset_err=1

always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) mode_q <= COMPAT;              // the floor is the default
  else if (link_error)     mode_q <= COMPAT; // errors are checked FIRST
  else if (promote_to_cxl) mode_q <= CXLM;

// Route by class. Mode is handled elsewhere.
assign to_cxl_path  = gated && req_is_cxl_class;
assign to_pcie_path = gated && !req_is_cxl_class;

  CXL-class, PCIe mode : pcie=0 cxl=1 refuse=0
  cxl_class_in_pcie_mode_err=1

assign to_cxl_path  = gated &&  req_is_cxl_class && mode_is_cxl;
assign to_pcie_path = gated && !req_is_cxl_class;
assign refuse       = req_valid && !(to_pcie_path || to_cxl_path);

16. Verification Plan

The last row is not optional bookkeeping. §14 showed a mutation surviving purely because a diagnostic was never required to fire, and a plan that treats diagnostics as untested infrastructure will repeat that.

17. Design Review

Questions worth asking of any controller that claims to implement this chapter.

• What is the reset mode, and can the recovery path reach it? If reset enters the new mode, the fallback is decorative.
• Is the peer's capability read behind a validity term, or sampled whenever convenient?
• Which branch wins when an error and a promotion arrive together?
• Can the platform report the fraction of links that reached CXL mode? If not, silent downgrade is undetectable in the field.
• Is a CXL-class request on a PCIe-mode link refused, or routed anyway?
• Are the diagnostics exercised by any test, or only by inspection?

18. How This Appears in Real Engineering

The reuse decision shows up in places that look unrelated to protocol design.

Validation matrices are dominated by the compatibility cells. Section 5's 2x2 says three of four combinations are PCIe. A test plan weighted toward the CXL cell — which is where the interesting engineering is — inverts the field distribution, and early in a generation most of your partners are not capable.

PCIe verification IP finds a large fraction of your bugs. Because the lower layers are inherited, existing PCIe VIP is directly applicable, and that is a real schedule saving. It is also a trap: the bugs PCIe VIP cannot find are exactly the ones in the added layers, so a green PCIe regression is not evidence about coherency.

Silicon bring-up gets a working link on day one. The link comes up as PCIe at 2.5 GT/s before any CXL logic is exercised, which means bring-up has a functioning channel to debug from. Teams building a new interconnect have no such starting point — their first debug session has no link.

Field failures degrade rather than fail. A CXL link with a marginal channel becomes a PCIe link and the system keeps running. Operationally this is excellent and diagnostically it is awful, which is Debug Lab 1's entire subject.

19. Common Misconceptions

20. Interview Reasoning

21. Exercises

1. Calculate. A market has 12 host platforms (4 CXL-capable) and 60 device designs (20 CXL-capable). Compute working pairs, CXL-capable pairs and dead pairs for both the new-interconnect and the reuse model. Then recompute at 10 of 12 and 50 of 60, and state at what adoption level the reuse argument stops being decisive.

2. Explain. Section 7 gives three properties — reset lands compatible, link before capability, errors fall back never forward. Argue that these are one property stated at three moments, then give a design that satisfies two and violates the third, and describe its field failure mode.

3. DV task. Write the coverage cross that would have caught mutation M2, and separately the one that would have caught M5. Explain why the second is not a stimulus problem, and what a coverage model that treats diagnostics as untested infrastructure will miss.

4. Debug task. A fleet reports 100% link health and a CXL-mode fraction of 40%. The 60% are all one server model. List the checks you would run in order, and state which ones distinguish "host not capable" from "device not capable" from "capable pair that failed to negotiate".

5. Design. Extend two_key_gate with a timeout that converts a never-answering peer into a decision. State the new property this creates, whether it is safety or liveness, and which assertion in §13 it discharges.

6. Critique. Argue that CXL should have defined its own PHY, using the strongest case you can. Then identify the single assumption your argument depends on most, and what evidence would settle it.

22. Summary

CXL rides on PCIe because a protocol's technical merit does not determine whether it gets deployed — its ability to reach hardware does.

• The reuse buys zero dead pairs, not more coherent links. The CXL-capable pair count is identical either way.
• It is a launch argument. At full adoption the two models converge.
• Compatibility is a design constraint expressed at three moments: reset lands compatible, the link comes up before capability is known, and errors fall back rather than forward.
• Three of four host-by-device capability combinations run as PCIe. That is the coverage obligation, and it is inverted by test plans that focus on the interesting cell.
• A fallback that reports success is indistinguishable from the feature working, so mode residency has to be measured, not assumed.
• Verification lessons from this chapter's own failures: an assertion derived from the design equation cannot fail; a diagnostic never required to fire is untested; and a transition property checked against present state flags correct behaviour.

Chapter 5.2 turns the argument into an inventory — precisely which PCIe structures CXL keeps, which it extends, and which are revision-dependent.