Relationship to the PCIe Stack

Chapter 2.4 answered "what is shared with PCIe" at the level of a platform: connector, electricals, enumeration, and the mode FSM that comes up as PCIe unconditionally.

Module 4 has since taken the stack apart. This chapter closes the module by asking the same question with the resolution that work bought: not what is shared, but what is inherited, what is extended and what is new — layer by layer.

1. The Engineering Problem — "Built on PCIe" Is Not Precise Enough

"CXL is built on PCIe" is true and useless for engineering.

It does not tell you whether a signal-integrity problem is yours or PCIe's. It does not tell you whether a PCIe verification IP will find your bugs. It does not tell you which of your controller's modes a PCIe-only device will exercise, or how much of your validation matrix exists purely because of compatibility.

Those are the questions that consume schedule, and each has a different answer at a different layer. A single boundary cannot answer them, because there is no single boundary — the relationship changes as you go up, and the practical value is in knowing where it changes.

2. The One-Sentence Model

CXL's relationship to PCIe is three relationships stacked: the physical layer is inherited unchanged, the logical PHY and link layer are extended — same structure, CXL-aware behaviour — and the transaction layer and protocol classes are added outright; so the correct question is never "is this PCIe?" but "at which layer am I asking?"

Call it the three-relationship stack: inherited, extended, added.

3. What This Chapter Owns

Deliberately not repeated here: Chapter 2.4's four wrong statements, its arbitration measurements, and its mode-FSM analysis. Those stand; this chapter advances the model rather than restating it.

4. The Three Relationships

The distinction between extended and added is the one that carries engineering weight.

Extended means the layer exists in both and does more in CXL. Consortium material describes CXL 3.0 as using the PCIe 6.0 PHY at 64 GT/s with PAM-4 and high BER mitigated by PCIe 6.0 FEC and CRC — so the link layer's integrity machinery tracks PCIe's generation while Chapter 4.1 showed the logical PHY doing CXL-specific work that PCIe never needs, because PCIe has nothing to multiplex.

Added means PCIe has no counterpart. .cache and .mem are not extensions of PCIe transactions; they are semantics PCIe never expressed, which is Chapter 1.6's whole argument.

5. Why the Boundary Sits Where It Does

One layer lower and CXL would define its own framing on PCIe's electricals — rebuilding a solved problem for no semantic gain.

One layer higher and CXL would inherit PCIe's transaction semantics, which is exactly what Chapter 1.6 showed to be insufficient: PCIe can move data to and from a device but cannot express a device holding a coherent cached copy or device memory being system memory.

So the boundary is at the highest layer reusable without constraining semantics — and the extended band exists because that boundary is not clean. The logical PHY and link layer are where CXL had to keep PCIe's structure while changing its behaviour, and that is precisely why those two layers are where compatibility bugs live.

6. What CXL.io Costs

.io is the one class with a PCIe counterpart, and its budget makes the extended relationship concrete.

Published analysis of CXL flit modes gives the payload available to a carried transaction as 236 bytes in 256-byte flit mode — the same as a PCIe 6.0 flit — and 232 bytes in the Latency-Optimized flit mode.

Two observations follow.

236 matching PCIe 6.0 is the extended relationship in a number. CXL.io in that mode gives a carried transaction exactly what PCIe would, so the encapsulation costs nothing relative to running PCIe natively — which is why .io traffic on a CXL link is not penalised for being on a CXL link.

232 versus 236 is what the Latency-Optimized flit costs .io. Four bytes of every 256, about 1.7%, on top of the FIT and efficiency trade Chapter 4.2 quoted. A design choosing that flit for its coherent traffic's benefit is charging .io for it — a cross-class cost that is easy to miss because the decision is usually made on coherence grounds.

Section 10 measures both configurations.

7. The Cost of Being Both

A CXL controller contains a working PCIe controller, and that is a validation statement before it is an area one.

The mode space is a product. Link state × capability × negotiation outcome × flit mode, and every reachable combination must be exercised. Section 11 measures four distinct reachable modes in a deliberately small model — a PCIe-only controller reaches two of them.

The asymmetry is what makes it expensive. A PCIe-only device never exercises the CXL modes, so it cannot help validate them; but a CXL device must exercise all the PCIe modes, because Chapter 2.4's mode FSM comes up as PCIe unconditionally. Compatibility is one-directional in cost: the newer part carries the whole matrix.

8. Teaching-model boundary

9. RTL 1 — The Relationship, Per Layer

Purpose

Make "inherited", "extended" and "added" checkable rather than descriptive.

// Where in the stack CXL stops being PCIe.
//
// The boundary is not one line -- it moves by layer. This model makes the
// per-layer relationship explicit and checks the one rule that must hold:
// anything CXL claims to INHERIT must behave identically to PCIe, and anything
// it claims to ADD must not be required for a PCIe-only device to work.
//
// ARCHITECTURAL TEACHING MODEL. The layer indices and relationship encodings
// are teaching values; no CXL or PCIe layer definition is modelled.
module layer_divergence #(
  parameter bit STRICT = 1'b1
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       op_valid,
  input  logic [2:0] layer,        // 0 phy, 1 logphy, 2 link, 3 txn, 4 protocol
  input  logic       is_cxl_mode,
  input  logic       behaviour_differs_from_pcie,
  output logic [1:0] relationship,  // 0 inherited, 1 extended, 2 added
  output logic       legal,
  output logic       inherited_diverged_err,
  output logic       added_required_in_pcie_err
);
  always_comb begin
    unique case (layer)
      3'd0: relationship = 2'd0;   // physical      -- inherited whole
      3'd1: relationship = 2'd1;   // logical PHY   -- shared, CXL-aware
      3'd2: relationship = 2'd1;   // link          -- structure tracks PCIe gen
      3'd3: relationship = 2'd2;   // transaction   -- CXL adds its own
      default: relationship = 2'd2; // protocol classes -- entirely new
    endcase
  end
 
  // Legal if a PCIe-only link is not being asked to do something CXL added.
  assign legal = op_valid && (STRICT ? (is_cxl_mode || (relationship != 2'd2)) : 1'b1);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      // A layer declared inherited must behave the same as PCIe. If it does
      // not, the reuse claim is false and every compatibility argument built
      // on it is unsound.
      if (op_valid && (relationship == 2'd0) && behaviour_differs_from_pcie)
        inherited_diverged_err <= 1'b1;
      // A PCIe-only link was asked to do something only CXL defines.
      if (legal && !is_cxl_mode && (relationship == 2'd2))
        added_required_in_pcie_err <= 1'b1;
    end
  end
endmodule

The two error outputs encode the two ways a classification can be wrong, and they point in opposite directions.

inherited_diverged_err catches a layer claimed as inherited that behaves differently. That is the more dangerous of the two, because every compatibility argument — the mode FSM, the fallback path, the reuse of PCIe verification IP — is built on the claim being true.

added_required_in_pcie_err catches the reverse: a PCIe-only link being asked to perform something only CXL defines. That is the mode-FSM failure of Chapter 2.4 expressed as a layer-classification rule.

Simulation evidence

Each layer queried against a link operating as PCIe only:

=== EXP1: the relationship, layer by layer ===
  physical     : INHERITED | PCIe-only link legal=1 | unchecked legal=1
  logical PHY  : EXTENDED  | PCIe-only link legal=1 | unchecked legal=1
  link         : EXTENDED  | PCIe-only link legal=1 | unchecked legal=1
  transaction  : ADDED     | PCIe-only link legal=0 | unchecked legal=1
  protocol     : ADDED     | PCIe-only link legal=0 | unchecked legal=1
 
  strict added_required_err=0 | unchecked=1

The first three rows are legal on a PCIe-only link and the last two are not. That is the boundary, located: everything up to and including the link layer works without CXL; everything above requires it.

And the injected divergence:

=== EXP2: an 'inherited' layer that does not actually match ===
  physical layer declared INHERITED but behaving differently:
    inherited_diverged_err=1  <-- the reuse claim is false

A layer that is claimed as inherited and is not invalidates more than itself. If the physical layer diverges, the PCIe fallback may not work, PCIe verification IP may not apply, and the platform's compatibility guarantee is unsupported — from one layer's behaviour.

10. RTL 2 — What CXL.io Gets to Carry

Purpose

Make the encapsulation budget explicit and conserved.

// CXL.io carries PCIe-style transactions inside CXL framing, so its payload
// budget is the flit minus what the framing costs.
//
// ARCHITECTURAL TEACHING MODEL. Byte budgets are supplied as parameters by the
// instantiator; nothing here asserts a CXL or PCIe flit layout.
module io_encapsulation #(
  parameter int unsigned FLIT_BYTES = 256,
  parameter int unsigned TXN_BYTES  = 236     // available to the carried txn
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        flit_valid,
  input  logic [15:0] txn_bytes_offered,
  output logic        fits,
  output logic [15:0] txn_bytes_carried,
  output logic [15:0] framing_bytes,
  output logic [31:0] total_wire_q,
  output logic [31:0] total_txn_q,
  output logic [15:0] n_flit_q,
  output logic        overbook_err
);
  assign fits              = (txn_bytes_offered <= TXN_BYTES[15:0]);
  assign txn_bytes_carried = fits ? txn_bytes_offered : TXN_BYTES[15:0];
  assign framing_bytes     = FLIT_BYTES[15:0] - TXN_BYTES[15:0];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else if (flit_valid) begin
      total_wire_q <= total_wire_q + FLIT_BYTES;
      total_txn_q  <= total_txn_q + {16'b0, txn_bytes_carried};
      n_flit_q     <= n_flit_q + 16'd1;
      // Carried more than the budget allows.
      if (txn_bytes_carried > TXN_BYTES[15:0]) overbook_err <= 1'b1;
    end
  end
endmodule

The budget is a parameter, not a constant in the logic, and that is deliberate for a reason beyond flexibility: it keeps the module free of any claim about CXL's flit layout. The testbench supplies 236 and 232 from Section 6's published figures; the module only enforces that whatever budget it is given is not exceeded.

Simulation evidence

Fifty flits carrying 200-byte transactions, in both budget configurations:

=== EXP3: what CXL.io actually gets to carry ===
  50 flits carrying 200-byte transactions:
    256B flit, 236B budget : wire=12800 carried=10000 framing/flit=20  ratio=78.12%
    256B flit, 232B budget : wire=12800 carried=10000 framing/flit=24  ratio=78.12%
 
  a 250-byte transaction offered: fits(236)=0 carried=236 | fits(232)=0 carried=232

Both ratios are identical at 78.12%, and that is the honest result: with 200-byte transactions, neither budget is the binding constraint — the transaction size is. The framing difference (20 versus 24 bytes) is real and invisible at this offered size.

The second line is where the budgets diverge. A 250-byte transaction does not fit either, and each carries only what its budget allows — 236 versus 232. The 4-byte difference matters only for transactions that approach the budget, which is exactly the case where the Latency-Optimized flit's cost to .io becomes visible.

That is worth stating plainly rather than dramatising: the Latency-Optimized flit's cost to .io is small and size-dependent, and the reason to notice it is that the decision to use that flit is normally made for coherent traffic's benefit, by people not looking at .io at all.

11. RTL 3 — The Mode Space a Dual Stack Must Validate

Purpose

Count what compatibility actually costs in validation surface.

// A CXL controller contains a working PCIe controller, so its mode space is
// the product of what it supports -- and every reachable combination is
// validation surface that must be exercised.
//
// ARCHITECTURAL TEACHING MODEL. Mode encodings are teaching values; no CXL or
// PCIe negotiation, training or mode mechanism is modelled.
module dual_stack_modes (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       link_up,
  input  logic       both_cxl_capable,
  input  logic       nego_ok,
  input  logic [1:0] flit_mode,
  output logic [2:0] mode_id,
  output logic       is_pcie,
  output logic       is_cxl,
  output logic [7:0] modes_seen_mask_q,
  output logic [7:0] n_modes_seen_q,
  output logic       illegal_mode_err
);
  assign is_cxl  = link_up && both_cxl_capable && nego_ok;
  assign is_pcie = link_up && !is_cxl;
 
  // Mode identity: down, PCIe, or CXL crossed with flit mode.
  always_comb begin
    if      (!link_up) mode_id = 3'd0;
    else if (is_pcie)  mode_id = 3'd1;
    else               mode_id = 3'd2 + {1'b0, flit_mode};
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      // Coverage, in hardware: which modes has this link actually reached?
      if ((modes_seen_mask_q & mask) == 8'd0 && mask != 8'd0)
        n_modes_seen_q <= n_modes_seen_q + 8'd1;
      modes_seen_mask_q <= modes_seen_mask_q | mask;
      // CXL mode without both ends capable is not a reachable state.
      if (is_cxl && !both_cxl_capable) illegal_mode_err <= 1'b1;
    end
  end
endmodule

modes_seen_mask_q is functional coverage implemented in hardware, and that is unusual enough to justify. A mode this link has never reached is a mode nobody has validated on this silicon — so exposing the mask lets a validation team read, from a running system, which parts of the compatibility matrix have actually been exercised. It costs a few flops and answers a question no test log can.

Simulation evidence

=== EXP4: the mode space a dual stack must validate ===
  distinct modes reached: 4  mask=00001111
  -> a PCIe-only controller has 2 of these; the CXL controller has all of them

Four modes from three inputs and one two-valued flit selector — link down, PCIe, CXL with the smaller flit, CXL with the larger. A PCIe-only controller reaches the first two.

Two of the four exist purely because of compatibility, and the asymmetry is the point: the PCIe device never exercises the CXL modes and the CXL device must exercise all the PCIe ones. In a real controller the multiplier is larger — link widths, rates, revisions, device types — and the shape is identical.

12. Assertions

Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog. Not executed; each maps to its procedural stand-in and mutation.

// SAFETY -------------------------------------------------------------------
// V1 — a layer classified INHERITED behaves identically to PCIe.
a_inherited_matches: assert property (@(posedge clk) disable iff (!rst_n)
  (relationship == INHERITED) |-> !behaviour_differs_from_pcie);
 
// V2 — a PCIe-only link is never asked to perform a CXL-added operation.
a_pcie_only_safe: assert property (@(posedge clk) disable iff (!rst_n)
  (legal && !is_cxl_mode) |-> (relationship != ADDED));
 
// V3 — every layer has exactly one classification.
a_one_relationship: assert property (@(posedge clk) disable iff (!rst_n)
  relationship inside {INHERITED, EXTENDED, ADDED});
 
// V4 — the encapsulation never carries more than its budget.
a_budget_respected: assert property (@(posedge clk) disable iff (!rst_n)
  flit_valid |-> (txn_bytes_carried <= TXN_BYTES));
 
// V5 — CONSERVATION: wire bytes equal flits times flit size.
a_wire_conserved: assert property (@(posedge clk) disable iff (!rst_n)
  total_wire_q == n_flit_q * FLIT_BYTES);
 
// V6 — CXL mode is never entered without both ends capable.
a_cxl_needs_both: assert property (@(posedge clk) disable iff (!rst_n)
  is_cxl |-> both_cxl_capable);
 
// V7 — the link is always in exactly one mode.
a_mode_exclusive: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({is_pcie, is_cxl}));
 
// GOAL / COVERAGE ----------------------------------------------------------
// V8 — every reachable mode is eventually exercised. This is a coverage
// obligation stated as a property; failing it means validation is incomplete,
// not that the design is wrong.
a_all_modes_seen: assert property (@(posedge clk) disable iff (!rst_n)
  (validation_complete) |-> (n_modes_seen_q == EXPECTED_MODES));

V8 is a coverage obligation written as a property, which is worth distinguishing from the rest of the table. Failing it does not mean the design is broken — it means the validation is incomplete, and for a dual stack that is the more likely problem.

13. Mutation Testing

14. Debug Lab

// The operation is well-formed; perform it.
assign legal = op_valid;

  physical     : INHERITED | legal=1 | unchecked legal=1
  link         : EXTENDED  | legal=1 | unchecked legal=1
  transaction  : ADDED     | legal=0 | unchecked legal=1
  protocol     : ADDED     | legal=0 | unchecked legal=1
 
  strict added_required_err=0 | unchecked=1

assign legal = op_valid && (is_cxl_mode || (relationship != ADDED));

// The physical layer is PCIe's; nothing to check.
// (no divergence check at all)

  physical layer declared INHERITED but behaving differently:
    inherited_diverged_err=1  <-- the reuse claim is false

if (op_valid && (relationship == INHERITED) && behaviour_differs_from_pcie)
  inherited_diverged_err <= 1'b1;

// Carry what was offered.
assign txn_bytes_carried = txn_bytes_offered;

  a 250-byte transaction offered: fits(236)=0 carried=236 | fits(232)=0 carried=232

assign fits              = (txn_bytes_offered <= TXN_BYTES);
assign txn_bytes_carried = fits ? txn_bytes_offered : TXN_BYTES;

assign is_cxl = link_up && nego_ok;      // capability term dropped

assign is_cxl = link_up && both_cxl_capable && nego_ok;

// in the testbench:
lu=1; bc=0; nok=1;   // one end not capable, negotiation reports success

15. Verification Plan

Reference model. A layer-classification table and a mode-transition model. The first is small and static; the second is the interesting one, because it enumerates the reachable mode space and lets the scoreboard confirm each was reached rather than assumed.

Directed tests. Every layer on both a PCIe-only and a CXL link. Divergence injected at each inherited layer. Transaction sizes at, just below and just above each budget, in both flit configurations. Every capability × negotiation combination including the off-diagonal ones.

Constrained-random dimensions. Transaction size distribution around the budget boundary, mode-transition sequences, and capability/negotiation pairings weighted toward the unusual.

Functional coverage:

The bolded row is Debug Lab 4's lesson. A cross whose off-diagonal bins are never hit is a cross that has tested nothing, and it is invisible in a pass/fail log.

Error injection. Divergence at an inherited layer; a transaction exceeding the budget; negotiation succeeding with one end not capable; a CXL-added operation requested on a PCIe-only link.

16. Design Review

On classification. Is each layer's relationship to PCIe written down and enforced, or only documented? Is there a check that an inherited layer actually matches? Is there a gate preventing added-layer operations on a PCIe-only link?

On encapsulation. Is the transaction budget a parameter or hard-coded? Is it enforced, and is there wire conservation alongside it? Does the design know what the Latency-Optimized flit costs .io, and was that cost part of the decision?

On the mode space. How many reachable modes does this controller have? Is that number written down? Is there hardware coverage of which modes a running system has actually reached? And which of those modes exist purely for compatibility, since those are the ones no PCIe-only partner will help you validate?

And the question this chapter adds. For each layer, if PCIe changes in the next generation, what in this design changes with it? Inherited layers track PCIe automatically; extended layers require work; added layers do not care. A design that cannot answer per layer has not understood its own relationship to PCIe.

17. How This Appears in Real Engineering

CXL / protocol architect

The classification is a planning tool. Inherited layers track PCIe's roadmap for free, extended layers cost engineering at every PCIe generation, and added layers are entirely yours. Knowing which is which per layer is what makes a multi-generation plan credible.

RTL engineer

Three disciplines: gate added-layer operations on the achieved mode; enforce the encapsulation budget with conservation alongside it; and keep the capability term in the mode condition — mutation R4 showed how easily it is dropped and how rarely it is exercised.

DV engineer

Two findings. A diagnostic needs a positive test (R2, and the same finding as Chapter 4.3's P5). And a checker that is correct can still never fire if the stimulus never produces the distinguishing combination (R4) — which is a stimulus gap, not a checker gap, and only mutation testing distinguishes them.

Performance engineer

The .io encapsulation budget is the number that matters here, and it is size-dependent: at 200-byte transactions both flit configurations delivered identical ratios, and the 4-byte difference only appears near the budget. Do not attribute a throughput difference to flit choice without checking the transaction-size distribution.

Firmware and system software

The reachable mode space is what a platform must configure and report. A controller that exposes which modes it has actually entered turns a validation question into a readable fact.

Silicon debug / validation

The mode-coverage mask is the highest-value observable in this chapter. It answers "has this silicon ever been in this mode" directly, which no test log can — a test that was run is not the same as a mode that was reached.

18. Common Misconceptions

19. Interview Reasoning

20. Exercises

1. Explain. The physical layer is inherited and the link layer is extended. State what changes for each when PCIe advances a generation, and which of the two appears on your schedule.

2. Calculate. A workload's CXL.io transactions are uniformly distributed between 64 and 256 bytes. What fraction is affected by the 236-versus-232 budget difference? Now for transactions uniformly distributed between 200 and 240 bytes.

3. Calculate. A controller supports 2 link widths, 3 rates, PCIe-or-CXL, and 2 flit modes (CXL only). How many reachable mode combinations? How many exist only because of CXL, and how many would a PCIe-only partner help you validate?

4. DV task. Write the capability × negotiation cross that would have caught mutation R4. Which bins are off-diagonal, and what should happen if one is unreachable in your design?

5. Debug task. A CXL link passes all PCIe-mode tests and fails intermittently in CXL mode at the same signal-integrity margin. Using the three-band classification, name the band you would investigate first and the reasoning that eliminates the other two.

21. Summary

CXL's relationship to PCIe is three relationships stacked, and the question "is this PCIe?" has no answer until you name the layer.

Physical is inherited. Consortium material describes CXL 3.0 on the PCIe 6.0 PHY at 64 GT/s with PAM-4 and PCIe 6.0 FEC and CRC. A problem there is a PCIe problem with PCIe answers, and it is the one place PCIe verification IP cleanly applies — which is also the test of whether the inheritance claim is true.

Logical PHY and link are extended. Same position, CXL-aware behaviour: Chapter 4.1 showed the logical PHY doing multiplexing work PCIe never needs, and the link layer's structure tracks the PCIe generation. This is the band where compatibility bugs concentrate, because structure is shared and behaviour is not.

Transaction and protocol classes are added. Measured, operations at those layers are illegal on a PCIe-only link while everything up to the link layer is legal — which is the boundary, located rather than described.

And compatibility is one-directional in cost. Measured, four reachable modes in a small model where a PCIe-only controller reaches two. The PCIe device never exercises the CXL modes; the CXL device must exercise all the PCIe ones.

Two verification results close the module. A diagnostic needs a positive test — the same finding as Chapter 4.3, reproduced independently here. And a correct checker can still never fire: mutation R4 survived the entire suite not because the check was wrong but because no stimulus ever produced the combination that distinguishes the mutated expression from the correct one. Those are different problems with different fixes, and only mutation testing separates them.

22. Module 4 Complete

Module 4 has taken the CXL stack apart layer by layer.

4.1 — the inherited PCIe PHY, and the sorting office CXL adds above it: labelling flit kinds, scheduling protocol stacks, and coordinating link state across every stack that shares the wire.

4.2 — correction before detection, the replay buffer whose depth bounds how far a sender may run ahead, credits as permission before commitment, and the wire time recovery consumes.

4.3 — where meaning enters, and where the three classes stop being interchangeable: different ordering, different resources, different latency budgets.

4.4 — the three protocols side by side, distinguished by direction before anything else.

4.5 — the whole relationship to PCIe, resolved per layer.

A reader who has followed the module should now be able to say, for any CXL question: which layer owns this, is that layer inherited, extended or added, and what does the answer imply for who fixes it. Module 5 goes deeper into the link and physical layers with that framing in place.

For adjacent material: Relationship to PCIe has the platform-level reuse argument this chapter refines, CXL Layered Architecture has layering as an idea, and Evolution of CXL has the rate and flit progression by revision. The path is on the CXL tutorials index.