Relationship to PCIe

The most commonly mis-stated fact about CXL is its relationship to PCIe. It is called a replacement for PCIe, a layer on top of PCIe, a competitor to PCIe, and a tunnel over PCIe. Each of those is wrong in a way that leads somewhere unproductive.

This chapter states the relationship precisely and then shows what it costs in RTL, because the reuse is not free — it is the source of the arbitration, budgeting and mode-selection problems every CXL implementation has to solve.

1. The One-Sentence Model

CXL is not layered on PCIe and does not replace it: a CXL link is a PCIe link whose physical layer, connector and enumeration are shared, carrying a set of protocol classes that PCIe does not define — so every CXL slot is a working PCIe slot, and CXL is what a compatible pair negotiates up to.

Two consequences follow immediately, and they are the whole chapter:

shared below   electricals · connector · form factor · enumeration
added above    coherent protocol classes PCIe never defined

2. What This Chapter Owns

Chapter 1.6 argued that PCIe's semantics could not express coherent participation. This chapter is the constructive half: given that argument, what did CXL keep?

3. The Four Wrong Statements

Each of these is worth stating and killing, because each leads to a different wrong prediction.

The last row is the one that matters most and the one this track has been building toward. At a given generation the raw rate is the same. CXL 3.x and PCIe 6.x both run 64 GT/s. What CXL adds is not throughput; it is the ability to say things PCIe has no encoding for — a device holding a coherent cached copy, and host memory that physically lives on a device.

4. What Is Actually Shared

The reuse is deep and specific.

Electricals and signalling. The same differential pairs, the same equalisation and training machinery, the same PAM4 signalling from PCIe 6.0 onward. This is the part that took the industry two decades to get right at these rates.

Connector and form factor. The same physical slot and the same add-in-card mechanicals. A CXL device goes into a PCIe slot, which is why the ecosystem did not need new chassis, new risers or new cabling.

Enumeration and configuration. A CXL device is discovered by the existing mechanism. That is not a convenience — it is the reason unmodified system software can bring a machine up with a CXL device present, before anything knows what CXL is.

Link-layer structure at the newer generations. The flit-based framing with FEC and CRC introduced in CXL 3.0 tracks PCIe 6.x, and CXL 4.0's 128 GT/s is built on PCIe 7.0.

5. What Is Added

Above that boundary sits everything PCIe does not define: the coherent protocol classes from Chapter 2.1, and — the part that is easy to overlook — the machinery that lets three classes share one physical link.

That machinery is where this chapter's engineering content lives. One link, three classes with different traffic characters:

Sharing a link between traffic with different characters is a scheduling problem, and scheduling problems have famous failure modes. The next four sections build and measure them.

6. Why Reuse Was the Decisive Choice

Worth being explicit about the counterfactual, because it explains why the reuse boundary sits where it does.

A new coherent interconnect with its own physical layer would have needed its own electricals, its own connectors, its own retimers and cables, its own enumeration, and its own place in every server chassis and BIOS. Chapter 2.2 showed that a standard's value is the pairwise validation it eliminates; a new physical layer would have re-introduced most of that cost at the hardest layer to get right.

Reuse converted a platform problem into a controller problem. A CXL device is a PCIe card mechanically and electrically; the new work is inside the controller. That is why CXL could appear in servers on a normal product cycle rather than a chassis-generation one.

7. The Cost of Reuse

Reuse is a trade, and the payments are real.

Everything must degrade to PCIe. A CXL controller contains a working PCIe controller, and every mode transition must land somewhere legal. That is validation surface that exists purely because of compatibility.

Three classes contend for one link. Section 9's arbiter exists only because the classes share physical resources. A dedicated interconnect per class would not need it — and would need three sets of everything else.

The physical layer's constraints are inherited whole. Reach, retimer count, lane widths and training behaviour come from PCIe, so a CXL design cannot solve a channel problem by changing the channel.

Bandwidth is inherited too. Sharing the physical layer means sharing its rate. That is exactly why CXL is not a bandwidth story.

8. RTL 1 — Classifying Traffic

Purpose

Before anything can be scheduled, every inbound request must land in exactly one class.

// Route an inbound request to one of the three CXL protocol classes.
// GENERIC teaching model: the classification inputs here are an abstraction,
// not CXL header decode.
module traffic_classifier (
  input  logic clk,
  input  logic rst_n,
  input  logic valid,
  input  logic is_config,      // enumeration / register access
  input  logic is_coherent,    // device caching host memory
  input  logic is_hostmem,     // host reading device-attached memory
  output logic sel_io,
  output logic sel_cache,
  output logic sel_mem,
  output logic unclassified,
  output logic multi_class_err
);
  // Priority is fixed and total: every valid request lands in exactly one
  // class. io first because config traffic must never be starved by data.
  assign sel_io    = valid &&  is_config;
  assign sel_cache = valid && !is_config &&  is_coherent;
  assign sel_mem   = valid && !is_config && !is_coherent && is_hostmem;
 
  assign unclassified = valid && !(sel_io || sel_cache || sel_mem);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) multi_class_err <= 1'b0;
    // One-hot or zero. Never two.
    else if (({2'b0,sel_io} + {2'b0,sel_cache} + {2'b0,sel_mem}) > 3'd1)
      multi_class_err <= 1'b1;
  end
endmodule

The !is_config terms are load-bearing. Without them, a request with two attributes set would select two classes, and downstream a transaction would be counted twice, arbitrated twice, and completed once. Writing the priority as a chain of exclusions makes one-hot structural rather than a property you hope holds.

Simulation evidence

=== EXP1: classifying inbound traffic into the three classes ===
  config write (enumeration)           io=1 cache=0 mem=0
  device caching host memory           io=0 cache=1 mem=0
  host reading device memory           io=0 cache=0 mem=1
  config + coherent + hostmem all set  io=1 cache=0 mem=0
  -> priority is total: exactly one class, never two

The fourth row is the one worth driving. All three attributes set is not a realistic request, and that is exactly why it should be in the regression: it is the input that distinguishes a priority chain from three independent decoders, and the two are indistinguishable on every realistic stimulus.

9. RTL 2 — Sharing One Link

Purpose

Three classes, one link, and the scheduling policy decides whether the design works or merely appears to.

// Share one physical link between the three protocol classes.
//
// MODE 0 -- strict priority: io > cache > mem. Simple, and it starves.
// MODE 1 -- weighted round-robin with a starvation guard: each class holds a
//           credit budget; a class that has waited too long is promoted.
//
// GENERIC teaching model. NOT the CXL arbitration or link-layer scheduling
// mechanism; the weights and the age threshold are teaching values.
module class_arbiter #(
  parameter int unsigned W_IO    = 1,
  parameter int unsigned W_CACHE = 4,
  parameter int unsigned W_MEM   = 3,
  parameter int unsigned AGE_MAX = 8
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       mode_wrr,      // 0 = strict priority, 1 = WRR + guard
  input  logic       req_io,
  input  logic       req_cache,
  input  logic       req_mem,
  output logic       gnt_io,
  output logic       gnt_cache,
  output logic       gnt_mem,
  output logic [7:0] age_io_q,
  output logic [7:0] age_cache_q,
  output logic [7:0] age_mem_q,
  output logic       starved_err,
  output logic       multi_grant_err
);
  logic [7:0] cr_io_q, cr_cache_q, cr_mem_q;
  logic promote_io, promote_cache, promote_mem;
  logic budget_empty;
 
  assign promote_io    = mode_wrr && (age_io_q    >= AGE_MAX[7:0]);
  assign promote_cache = mode_wrr && (age_cache_q >= AGE_MAX[7:0]);
  assign promote_mem   = mode_wrr && (age_mem_q   >= AGE_MAX[7:0]);
 
  // A round ends when nobody who is asking still has budget.
  assign budget_empty = !((req_io && cr_io_q != 0) || (req_cache && cr_cache_q != 0)
                                                   || (req_mem && cr_mem_q != 0));
 
  always_comb begin
    gnt_io = 1'b0; gnt_cache = 1'b0; gnt_mem = 1'b0;
    if (!mode_wrr) begin
      // Strict priority. io always wins, then cache, then mem.
      if      (req_io)    gnt_io    = 1'b1;
      else if (req_cache) gnt_cache = 1'b1;
      else if (req_mem)   gnt_mem   = 1'b1;
    end else begin
      // Starvation guard FIRST: an aged requester jumps the queue.
      if      (req_io    && promote_io)    gnt_io    = 1'b1;
      else if (req_cache && promote_cache) gnt_cache = 1'b1;
      else if (req_mem   && promote_mem)   gnt_mem   = 1'b1;
      // Otherwise serve whoever still has credit this round.
      else if (req_io    && cr_io_q    != 0) gnt_io    = 1'b1;
      else if (req_cache && cr_cache_q != 0) gnt_cache = 1'b1;
      else if (req_mem   && cr_mem_q   != 0) gnt_mem   = 1'b1;
    end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      cr_io_q <= W_IO[7:0]; cr_cache_q <= W_CACHE[7:0]; cr_mem_q <= W_MEM[7:0];
      age_io_q <= '0; age_cache_q <= '0; age_mem_q <= '0;
      starved_err <= 1'b0; multi_grant_err <= 1'b0;
    end else begin
      if (({2'b0,gnt_io} + {2'b0,gnt_cache} + {2'b0,gnt_mem}) > 3'd1)
        multi_grant_err <= 1'b1;
 
      // Age: reset on grant, increment while asking and not granted.
      age_io_q    <= gnt_io    ? 8'd0 : (req_io    ? age_io_q    + 8'd1 : age_io_q);
      age_cache_q <= gnt_cache ? 8'd0 : (req_cache ? age_cache_q + 8'd1 : age_cache_q);
      age_mem_q   <= gnt_mem   ? 8'd0 : (req_mem   ? age_mem_q   + 8'd1 : age_mem_q);
 
      if (mode_wrr) begin
        if (budget_empty) begin
          cr_io_q <= W_IO[7:0]; cr_cache_q <= W_CACHE[7:0]; cr_mem_q <= W_MEM[7:0];
        end else begin
          if (gnt_io    && cr_io_q    != 0) cr_io_q    <= cr_io_q    - 8'd1;
          if (gnt_cache && cr_cache_q != 0) cr_cache_q <= cr_cache_q - 8'd1;
          if (gnt_mem   && cr_mem_q   != 0) cr_mem_q   <= cr_mem_q   - 8'd1;
        end
      end
 
      if ((age_io_q > AGE_MAX[7:0]*8'd4) || (age_cache_q > AGE_MAX[7:0]*8'd4)
                                         || (age_mem_q > AGE_MAX[7:0]*8'd4))
        starved_err <= 1'b1;
    end
  end
endmodule

Backpressure and timing. Grant is combinational from request, which is the standard arbiter shape and the standard timing risk: the request-to-grant path plus whatever the grant feeds is often the critical path in a link controller. The credit and age state is registered, so only the selection logic is in the path.

Simulation evidence — strict priority starves

Both modes instantiated on identical stimulus, 200 cycles, all three classes asking every cycle:

=== EXP2: one link, three classes, 200 cycles all asking ===
  strict priority : io=200 cache=0 mem=0  max mem age=199
  WRR + guard     : io=23  cache=89 mem=66  max mem age=6
  -> strict priority STARVED CXL.mem completely
  -> the guarded arbiter served every class

Zero grants in 200 cycles for two of three classes. Under strict priority, the highest-priority class asking continuously means nothing else ever runs — not a slow class, a stopped one.

That result is easy to dismiss as an unrealistic stimulus, so the more damaging case was measured too. Here config traffic asks only one cycle in ten:

=== EXP2b: io only asks 1 cycle in 10 (a realistic config mix) ===
  strict priority : io=20 cache=180 mem=0  max mem age=255
  WRR + guard     : io=19 cache=91  mem=68  max mem age=6

This is the important row. The top-priority class is now almost idle, and CXL.mem still receives zero grants across 200 cycles — because the middle class absorbs everything the top one leaves. Its age counter saturates at 255, meaning host memory access is not slow, it is dead, and no amount of reducing the top class's traffic fixes it.

The guarded arbiter serves all three with a maximum wait of 6 cycles. That number is the design's actual latency guarantee, and it exists because the age counter exists.

Simulation evidence — what the guarded arbiter does cycle by cycle

=== EXP2c: 12-cycle grant trace, all three asking, WRR + guard ===
    cyc 0: gnt=mem    ages 0/3/1
    cyc 1: gnt=none   ages 1/4/0
    cyc 2: gnt=io     ages 2/5/1
    cyc 3: gnt=cache  ages 0/6/2
    cyc 4: gnt=cache  ages 1/0/3
    cyc 5: gnt=cache  ages 2/0/4
    cyc 6: gnt=cache  ages 3/0/5
    cyc 7: gnt=mem    ages 4/0/6
    cyc 8: gnt=mem    ages 5/1/0
    cyc 9: gnt=mem    ages 6/2/0
    cyc 10: gnt=none  ages 7/3/0
    cyc 11: gnt=io    ages 8/4/1

Two things are worth reading honestly here.

The weights work: cache and mem receive runs proportional to their credit budgets, and io — weight 1 — is served once per round. And cycles 1 and 10 grant nothing while all three classes are asking. That is a real inefficiency in this implementation, not a measurement artefact: at a round boundary the credits are exhausted and the refill takes a cycle, so the link idles. Two wasted cycles in twelve is roughly 17% of link bandwidth.

A production arbiter would refill combinationally or overlap the refill with the last grant of the round. It is left visible here because credit-refill bubbles are a real and frequently-shipped defect, and they are invisible in a throughput average taken over a long run.

10. RTL 3 — Per-Class Outstanding Budgets

Purpose

Arbitration decides who goes next. Budgets decide who is allowed to ask.

// Independent outstanding-transaction budgets, one per protocol class.
//
// The point of this module is what it PREVENTS: a burst of bulk traffic in one
// class exhausting a shared pool and blocking a latency-critical class.
// SHARED=1 collapses the three budgets into one pool to show the failure.
module outstanding_per_class #(
  parameter int unsigned LIM_IO    = 4,
  parameter int unsigned LIM_CACHE = 6,
  parameter int unsigned LIM_MEM   = 6,
  parameter bit          SHARED    = 1'b0
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       iss_io,   input logic iss_cache, input logic iss_mem,
  input  logic       ret_io,   input logic ret_cache, input logic ret_mem,
  output logic       can_io,   output logic can_cache, output logic can_mem,
  output logic [7:0] out_io_q, output logic [7:0] out_cache_q,
  output logic [7:0] out_mem_q, output logic [7:0] pool_q,
  output logic       overflow_err,
  output logic       blocked_by_other_class
);
  localparam logic [7:0] POOL = LIM_IO[7:0] + LIM_CACHE[7:0] + LIM_MEM[7:0];
 
  // With a shared pool every class sees the SAME gate; per class, each has
  // its own headroom and one class cannot consume another's.
  assign can_io    = SHARED ? (pool_q < POOL) : (out_io_q    < LIM_IO[7:0]);
  assign can_cache = SHARED ? (pool_q < POOL) : (out_cache_q < LIM_CACHE[7:0]);
  assign can_mem   = SHARED ? (pool_q < POOL) : (out_mem_q   < LIM_MEM[7:0]);
 
  // A latency-critical class refused while its OWN budget is free is the
  // symptom that only a shared pool can produce.
  assign blocked_by_other_class = SHARED && !can_io && (out_io_q < LIM_IO[7:0]);
 
  // ... counter maintenance elided; see the invariant note below ...
endmodule

The invariant differs by configuration, and that matters. Per-class budgets bound each counter individually; a shared pool bounds only the total. An overflow check written for one configuration is not merely useless in the other — it fires spuriously, which is how a correct design gets "fixed" into a broken one. The simulated model checks the applicable invariant in each mode.

Simulation evidence

Both configurations instantiated on identical stimulus: a 12-cycle flood of cache and mem issues with no returns, then a config request.

=== EXP3: bulk traffic vs a latency-critical class ===
  after a 12-cycle bulk flood:
    per-class : io headroom=4 can_io=1
    shared    : io headroom=4 can_io=0 blocked_by_other_class=1
  -> the shared pool refused io while its own budget was untouched

Read blocked_by_other_class=1 carefully. The config class has used none of its four slots and is refused anyway, because bulk traffic in two other classes drained the common pool. The class that needs a slot least often is the one that cannot get one when it finally asks — during bring-up, error handling, or recovery, which is precisely when config traffic matters.

This is the same shape as Chapter 1.5's shared-resource argument and it recurs at every level of a memory system: sharing a budget couples classes that have nothing to do with each other, and the coupling only appears under load.

11. RTL 4 — PCIe First, CXL as an Upgrade

Purpose

The mode selection that makes the whole compatibility story true.

// A slot that trains as PCIe and only becomes CXL when both ends agree.
//
//   00 DOWN     link not up
//   01 PCIE     operating as plain PCIe -- the fallback, and it always works
//   10 CXL      both ends advertised CXL and negotiation succeeded
//
// GENERIC teaching model. NOT PCIe link training, alternate-protocol
// negotiation, or any specification-defined procedure.
module mode_select (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       link_up,
  input  logic       host_cxl_capable,
  input  logic       dev_cxl_capable,
  input  logic       cxl_nego_ok,
  output logic [1:0] mode_q,
  output logic       is_cxl,
  output logic       is_pcie,
  output logic       degraded,        // could have been CXL, is not
  output logic       illegal_cxl_err  // CXL entered without both ends capable
);
  localparam logic [1:0] DOWN = 2'b00, PCIE = 2'b01, CXLM = 2'b10;
 
  assign is_cxl  = (mode_q == CXLM);
  assign is_pcie = (mode_q == PCIE);
  // Both ends could do CXL and we are running PCIe anyway. Works, but slower
  // and without coherence -- the silent-degradation case.
  assign degraded = is_pcie && host_cxl_capable && dev_cxl_capable;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      mode_q <= DOWN; illegal_cxl_err <= 1'b0;
    end else begin
      if (!link_up) begin
        mode_q <= DOWN;
      end else begin
        case (mode_q)
          // Come up as PCIe FIRST, always. CXL is an upgrade from a working
          // link, never a precondition for having one.
          DOWN: mode_q <= PCIE;
          PCIE: if (host_cxl_capable && dev_cxl_capable && cxl_nego_ok) mode_q <= CXLM;
          default: ;
        endcase
      end
      if (is_cxl && !(host_cxl_capable && dev_cxl_capable)) illegal_cxl_err <= 1'b1;
    end
  end
endmodule

The DOWN → PCIE edge is unconditional, and that is the design. The link does not evaluate CXL capability before coming up; it comes up as PCIe and then considers upgrading. A design that gates link-up on CXL negotiation converts every negotiation failure into a dead slot — which is the difference between a device that runs slower than expected and a machine that does not boot.

Simulation evidence

=== EXP4: a slot that trains as PCIe first ===
  link up, neither end CXL      : mode=01 pcie=1 cxl=0 degraded=0
  both capable, nego not done   : mode=01 pcie=1 cxl=0 degraded=1
  negotiation succeeds          : mode=10 pcie=0 cxl=1 degraded=0
  link drops                    : mode=00 pcie=0 cxl=0
  host CXL, device PCIe-only    : mode=01 pcie=1 cxl=0 illegal=0

Row two is the operationally interesting one. The link works, and it is not what it should be. Both ends are capable, negotiation has not completed, and the system runs as PCIe — no error, no failure, just missing coherence and a workload quietly performing worse than the hardware allows. degraded exists so that state is reportable; without it, this is a support case that starts with "the card seems slower than the datasheet".

Row five is the compatibility promise doing its job: a PCIe-only device in a CXL-capable host works as a PCIe device, with illegal_cxl_err never asserting.

12. Where the Boundary Is Visible in Practice

Three concrete places the shared-below / added-above split shows up.

In bring-up. The machine enumerates the device before anything knows it is a CXL device, because enumeration is inherited. A CXL bring-up failure therefore usually presents as a working PCIe device rather than as a missing one — which sends the investigation in the wrong direction unless the degraded signal from Section 11 is exposed.

In the physical channel. Signal-integrity, reach and retimer questions are PCIe questions with PCIe answers. A CXL design cannot solve them by changing the protocol, because the protocol is above the boundary and the problem is below it.

In performance debugging. If the symptom is throughput, the boundary says look below — lanes, generation, channel. If it is latency under mixed traffic, look above — arbitration and budgets, which is Sections 9 and 10. Getting this backwards costs weeks, and the boundary tells you which half to instrument first.

13. Assertions

Icarus does not execute concurrent SVA, so these were not run; the table gives the procedural check.

// P1 — classification is one-hot or zero. Never two classes.
a_class_onehot: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({sel_io, sel_cache, sel_mem}));
 
// P2 — every valid request is classified.
a_no_unclassified: assert property (@(posedge clk) disable iff (!rst_n)
  valid |-> (sel_io || sel_cache || sel_mem));
 
// P3 — one grant per cycle.
a_grant_onehot: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({gnt_io, gnt_cache, gnt_mem}));
 
// P4 — LIVENESS: no class waits forever. This is the property strict
// priority violates, and no safety property can express it.
a_no_starvation: assert property (@(posedge clk) disable iff (!rst_n)
  req_mem |-> ##[1:BOUND] gnt_mem);
 
// P5 — a class never exceeds its own outstanding budget.
a_budget_respected: assert property (@(posedge clk) disable iff (!rst_n)
  (out_io_q <= LIM_IO) && (out_cache_q <= LIM_CACHE) && (out_mem_q <= LIM_MEM));
 
// P6 — CXL mode is never entered unless both ends are capable.
a_cxl_requires_both: assert property (@(posedge clk) disable iff (!rst_n)
  is_cxl |-> (host_cxl_capable && dev_cxl_capable));
 
// P7 — a link that is up is always in a legal operating mode.
a_up_means_legal: assert property (@(posedge clk) disable iff (!rst_n)
  link_up |-> ##[0:2] (is_pcie || is_cxl));

P4 is the one that matters and the one that cannot be checked by inspection. Strict priority satisfies P1, P2, P3, P5, P6 and P7 perfectly — every safety property passes on a design in which two of three classes never run. Only the liveness property distinguishes them, and only under stimulus that keeps the higher-priority class busy.

14. Debug Lab

// Simple and obviously correct: serve the most important class first.
if      (req_io)    gnt_io    = 1'b1;
else if (req_cache) gnt_cache = 1'b1;
else if (req_mem)   gnt_mem   = 1'b1;

  strict priority : io=20 cache=180 mem=0  max mem age=255
  WRR + guard     : io=19 cache=91  mem=68  max mem age=6

assign promote_mem = (age_mem_q >= AGE_MAX);
// Guard first, credits second.
if      (req_io  && promote_io)  gnt_io  = 1'b1;
else if (req_mem && promote_mem) gnt_mem = 1'b1;
// ... then serve whoever still has budget this round

// One pool is simpler and uses the buffer better.
assign can_io = (pool_q < POOL);
assign can_cache = (pool_q < POOL);
assign can_mem = (pool_q < POOL);

    per-class : io headroom=4 can_io=1
    shared    : io headroom=4 can_io=0 blocked_by_other_class=1

assign can_io    = (out_io_q    < LIM_IO);
assign can_cache = (out_cache_q < LIM_CACHE);
assign can_mem   = (out_mem_q   < LIM_MEM);

// Bring the link up in the mode we intend to use.
case (mode_q)
  DOWN: if (host_cxl_capable && dev_cxl_capable && cxl_nego_ok) mode_q <= CXLM;
  // ... no unconditional path to PCIE
endcase

  link up, neither end CXL      : mode=01 pcie=1 cxl=0
  host CXL, device PCIe-only    : mode=01 pcie=1 cxl=0 illegal=0

DOWN: mode_q <= PCIE;                       // always, no conditions
PCIE: if (host_cxl_capable && dev_cxl_capable && cxl_nego_ok) mode_q <= CXLM;

// The link is up and traffic flows. Nothing to report.
assign link_ok = is_pcie || is_cxl;
// no signal distinguishes "PCIe by design" from "PCIe by failure"

  both capable, nego not done   : mode=01 pcie=1 cxl=0 degraded=1

// Both ends could have been CXL and we are running PCIe anyway.
assign degraded = is_pcie && host_cxl_capable && dev_cxl_capable;

15. How This Appears in Real Engineering

Architecture and platform

The reuse boundary is the reason a CXL device fits an existing platform, and it constrains what a design can fix. Reach, lane width and generation are PCIe properties; a channel problem cannot be solved above the boundary. Conversely, latency under mixed traffic is entirely an above-the-boundary problem and no amount of PCIe generation buys it back.

RTL engineer

Four mechanisms, all in Sections 8 to 11. Make classification total and one-hot structurally. Give the arbiter a bounded wait, because a priority chain is the default thing you write and it starves. Give each class its own outstanding budget. And make the path to PCIe unconditional, so CXL is an upgrade rather than a precondition.

Verification engineer

Three requirements the measured runs justify. Saturating stimulus, because both the starvation and the shared-pool defects are invisible below full occupancy. Liveness properties, because P4 is the only property that separates a fair arbiter from a starving one. And a PCIe-only device model in the regression, because a bench where every device is CXL-capable cannot find Debug Lab 3.

Firmware and system software

Two behaviours to implement. Report the degraded state rather than treating "link up" as success — Debug Lab 4 is otherwise a permanent silent regression. And do not assume CXL because the slot is CXL-capable: read the achieved mode.

Performance engineering

Use the boundary as a triage rule. Throughput symptom → below the boundary, look at lanes and generation. Latency-under-mixed-traffic symptom → above it, look at arbitration and per-class budgets. The measured arbiter run gives the shape to look for: one class with a saturated age counter while others run normally.

16. Common Misconceptions

17. Interview Reasoning

18. Summary

CXL and PCIe share the physical layer, connector, form factor and enumeration, and — at the newer generations — the flit-based link structure. Above that boundary CXL adds coherent protocol classes PCIe does not define, plus the arbitration and budgeting machinery that lets three classes share one link. A CXL link is a PCIe link carrying more kinds of traffic, not a replacement, not a tunnel, and not a faster PCIe.

At a given generation the raw rate is identical. What reuse bought was adoption: the difficulty moved into the controller instead of into every chassis, connector and BIOS in the industry.

What it costs shows up in RTL, and all four costs were measured. Everything must degrade to PCIe, which is why the mode FSM comes up as PCIe unconditionally — and why it must also report degraded, because a link that is up, working and silently not CXL produces no error anywhere. Three classes share one link, which is why strict priority is not an option: it gave CXL.mem zero grants in 200 cycles, and reducing the top class to 10% duty did not help. Each class needs its own outstanding budget, because a shared pool refused a config request with all four of its own slots free. And the guarded arbiter that fixes the starvation costs about 17% of link bandwidth in refill bubbles in this implementation — a cost worth naming rather than averaging away.

The through-line: every one of those defects passes every safety property, and every one of them is invisible until the link is saturated.

19. What Comes Next

Modules 1 and 2 have established the problem, the definition, the standards process, the chronology, and the relationship to PCIe. Chapter 2.5 closes the module with the architectural goals — what CXL is for, stated precisely enough to be tested against, including the placement, pooling and coherent-mode decisions those goals imply.

For adjacent material: Why PCIe Was Not Enough is the semantic argument this chapter's reuse boundary answers, What Is CXL? has the protocol classes, and the PCIe track covers the shared foundation in depth. The path is on the CXL tutorials index.