CXL · Module 5
CXL Link Initialisation
How a CXL link decides what it is: the two-phase alternate protocol negotiation carried in modified TS ordered sets, why timeouts are mandatory, what retimers do to an advertisement, and the three outcomes. Six RTL models simulated, twelve mutations, twelve killed.
Chapter 5.1 established the two-key rule: CXL operation requires both ends, and neither may decide alone. Chapter 5.2 established that the mechanism is PCIe's own, not a CXL invention.
This chapter is the mechanism. How do two devices that have never met agree on what protocol to speak — over a link that must already be working before the conversation can start?
1. The Engineering Problem — A Conversation That Must Precede Itself
To find out whether your link partner supports CXL, you have to ask it. To ask it, you need a working link. To bring up a working link, you need to know what protocol to run.
That is circular, and every multi-protocol interconnect has to break the circle the same way: bring the link up in the mode that always works, hold the conversation over that link, then decide. Nothing else is possible, because there is no side channel.
Three consequences follow immediately, and they shape everything below.
- The link is a PCIe link before it is anything else. Not conceptually — literally. It trains, reaches a usable state, and carries ordered sets, all as PCIe.
- The negotiation is in-band. It travels in the same ordered sets the link uses to train, which is why its fields live in specific symbol positions of a specific ordered-set variant.
- A partner that never answers must not hang the link. There is no way to distinguish "slow" from "absent" except by deciding that enough time has passed.
2. The One-Sentence Model
CXL link initialisation is a two-phase capability exchange carried inside PCIe's own training traffic, guarded by a timeout, whose result is the intersection of what both ends offered and what the path between them can actually carry — and whose only three outcomes are CXL, PCIe, or error.
Call it decide, then use. Every failure in this chapter is some version of using before deciding, or never deciding at all.
3. What This Chapter Owns
| Question | Owned by |
|---|---|
| Why the reuse exists | 5.1 |
| Which structures are reused, and their class | 5.2 |
| How the two ends agree — the negotiation itself | this chapter |
| Widths, rates, recovery, the operating point | 5.4 |
| How software learns what was agreed | 5.5 |
4. The Ordering, and Why It Cannot Be Rearranged
Each arrow depends on the one above it, and none of them commute.
| Step | Depends on | If rearranged |
|---|---|---|
| Train as PCIe | nothing | no link to talk over |
| Advertise | a working link | shouting into a dead channel |
| Respond | hearing the advert | answering an unasked question |
| Decide | both adverts | deciding on one end's opinion |
| Use | the decision | unilateral turn-on (5.1) |
5. Teaching-model boundary
6. RTL 1 — The Bring-Up Decision as a State Machine
Three properties of this graph carry the chapter.
DECIDE has no self-loop. It resolves in one step, to exactly one of two states. A decision state that can wait is a decision state that can hang.
EXCH has two exits. One on hearing the partner, one on the clock running out. Remove the second and the machine has no path out of waiting — which is mutation M1.
Every arrow into CXL passes through DECIDE. There is no shortcut from BASE, which is mutation M2. Once that edge exists, a locally-capable device enters CXL without the exchange having happened.
module init_fsm #(
parameter int unsigned TIMEOUT = 8,
parameter bit NO_TIMEOUT = 1'b0 // 1 = the hanging shape
) (
input logic clk, rst_n,
input logic base_link_up, // link reached the base operating state
input logic local_cxl,
input logic peer_advert_seen, // partner's advertisement observed
input logic peer_cxl, link_error,
output logic [2:0] state_q,
output logic cxl_ready, pcie_fallback, in_error, decided,
output logic [7:0] wait_q,
output logic used_before_decided_err, both_outcomes_err, hung_err
);
localparam logic [2:0] S_RESET = 3'd0, S_BASE = 3'd1, S_EXCH = 3'd2,
S_DECIDE = 3'd3, S_CXL = 3'd4, S_PCIE = 3'd5, S_ERR = 3'd6;
logic [2:0] nxt;
logic timed_out;
assign timed_out = !NO_TIMEOUT && (wait_q >= TIMEOUT[7:0]);
assign cxl_ready = (state_q == S_CXL);
assign pcie_fallback = (state_q == S_PCIE);
assign in_error = (state_q == S_ERR);
assign decided = cxl_ready || pcie_fallback;
always_comb begin
nxt = state_q;
case (state_q)
S_RESET : if (base_link_up) nxt = S_BASE;
// The link is up. It is a PCIe link. Nothing CXL has happened yet.
S_BASE : nxt = S_EXCH;
// Wait to HEAR the partner. A timeout converts waiting into a decision.
S_EXCH : if (peer_advert_seen) nxt = S_DECIDE;
else if (timed_out) nxt = S_PCIE;
// Both keys, or fall back. This state cannot loop.
S_DECIDE: nxt = (local_cxl && peer_cxl) ? S_CXL : S_PCIE;
S_CXL : if (link_error) nxt = S_PCIE; // fall back, never forward
S_PCIE : ; // terminal for this bring-up
S_ERR : ;
default : nxt = S_ERR;
endcase
if (state_q != S_RESET && !base_link_up) nxt = S_ERR;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
state_q <= S_RESET; wait_q <= 8'd0;
used_before_decided_err <= 1'b0; both_outcomes_err <= 1'b0; hung_err <= 1'b0;
end else begin
state_q <= nxt;
wait_q <= (nxt == S_EXCH) ? (wait_q + 8'd1) : 8'd0;
if (cxl_ready && !decided) used_before_decided_err <= 1'b1;
if (cxl_ready && pcie_fallback) both_outcomes_err <= 1'b1;
// Liveness made checkable: waiting past the bound is a detected fault.
if ((state_q == S_EXCH) && (wait_q > TIMEOUT[7:0] + 8'd2)) hung_err <= 1'b1;
end
end
endmodule7. Waveform — The Same Bring-Up, Two Partners
The stimulus below is identical in both cases except for one bit: whether the partner is CXL-capable. Everything through cycle 5 is the same link doing the same thing.
Success and incompatible: identical until the decision resolves
10 cyclesRead the two state rows together. Six of ten cycles are identical, and the divergence is a single input bit resolved in a single state. That is what "the compatibility path is not a separate design" means in practice — it is the same machine reaching a different terminal state.
8. Waveform — The Partner That Never Answers
Timeout: waiting is converted into a decision
14 cyclesThe bottom row is the point. The design without a timeout is not slower — it never finishes. In simulation it sat in EXCH for the remainder of the run, and the testbench's own liveness check reported wait_q = 111 at the end.
9. RTL 2 — Two Phases, In Order
Public material describes the first phase as the root complex advertising its capabilities, and the endpoint then responding to indicate which it wishes to enable. The ordering is the whole content.
module two_phase_negotiate #(
parameter bit RESPOND_EARLY = 1'b0 // 1 = the broken shape
) (
input logic clk, rst_n, start,
input logic [3:0] host_offer,
input logic host_offer_valid,
input logic [3:0] dev_wants,
input logic dev_reply_valid,
output logic [1:0] phase_q,
output logic [3:0] enabled_q,
output logic complete,
output logic replied_before_offer_err, enabled_not_offered_err
);
localparam logic [1:0] P_IDLE = 2'd0, P_OFFER = 2'd1, P_REPLY = 2'd2, P_DONE = 2'd3;
logic [3:0] offer_q;
logic may_reply, reply_now;
// The guard must be evaluated OUTSIDE the P_REPLY branch, or it is
// tautological -- inside that branch phase_q == P_REPLY by construction, so
// the term can never be false and mutating it away changes nothing.
assign may_reply = RESPOND_EARLY ? (phase_q != P_DONE) : (phase_q == P_REPLY);
assign reply_now = dev_reply_valid && may_reply;
assign complete = (phase_q == P_DONE);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
phase_q <= P_IDLE; offer_q <= 4'h0; enabled_q <= 4'h0;
replied_before_offer_err <= 1'b0; enabled_not_offered_err <= 1'b0;
end else begin
if (reply_now) begin
// What is enabled is the INTERSECTION, never the request.
enabled_q <= offer_q & dev_wants;
phase_q <= P_DONE;
end else begin
case (phase_q)
P_IDLE : if (start) phase_q <= P_OFFER;
P_OFFER : if (host_offer_valid) begin
offer_q <= host_offer; phase_q <= P_REPLY;
end
default : ;
endcase
end
if (reply_now && (phase_q != P_REPLY)) replied_before_offer_err <= 1'b1;
// Nothing may be enabled that the host did not offer.
if (complete && ((enabled_q & ~offer_q) != 4'h0)) enabled_not_offered_err <= 1'b1;
end
end
endmodule after start : correct phase=1 | respond-early phase=1
device replies EARLY: correct enabled=0x0 complete=0 | early enabled=0x0 complete=0
host offers 0xf : correct phase=2
device wants 0x6 : correct enabled=0x6 complete=1
respond-early replied_before_offer_err=1
partial offer 0x3, wants 0x6 : enabled=0x2 (expected 0x2)10. RTL 3 — The Result Is an Intersection
module capability_intersect #(
parameter bit USE_UNION = 1'b0 // 1 = the broken shape
) (
input logic clk, rst_n,
input logic [3:0] host_caps, dev_caps,
input logic valid,
output logic [3:0] enabled,
output logic any_enabled,
output logic not_subset_host_err, not_subset_dev_err, empty_but_claimed_err
);
assign enabled = valid ? (USE_UNION ? (host_caps | dev_caps)
: (host_caps & dev_caps)) : 4'h0;
assign any_enabled = |enabled;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
not_subset_host_err <= 1'b0; not_subset_dev_err <= 1'b0;
empty_but_claimed_err <= 1'b0;
end else if (valid) begin
// Subset of each side, stated separately: a union breaks both, and a
// one-sided bug breaks exactly one.
if ((enabled & ~host_caps) != 4'h0) not_subset_host_err <= 1'b1;
if ((enabled & ~dev_caps) != 4'h0) not_subset_dev_err <= 1'b1;
if (any_enabled && ((host_caps & dev_caps) == 4'h0)) empty_but_claimed_err <= 1'b1;
end
end
endmodule host=0xF dev=0x6 : correct=0x6 | union-bug=0xf
host=0x9 dev=0x6 : correct=0x0 any=0 | union-bug=0xf any=1 <-- disjoint
host=0x3 dev=0x7 : correct=0x3 | union-bug=0x7
union-bug not_subset_host_err=1 not_subset_dev_err=1Row 2 is the case worth dwelling on: host and device advertise disjoint capability sets, so the correct intersection is empty and nothing CXL is enabled. The union bug enables everything, on a pair that agrees on nothing.
The two subset checks are stated separately on purpose. A union breaks both; a one-sided bug — using only the host's set, say — breaks exactly one. A single combined check would conflate two distinct failures into one report.
11. RTL 4 — The Negotiation Has a Window
Public material places the negotiation in modified TS1/TS2 ordered sets during specific Configuration states. Two conditions, both required: the right ordered-set variant, and the right point in bring-up.
module negotiation_window #(
parameter bit IGNORE_WINDOW = 1'b0 // 1 = the broken shape
) (
input logic clk, rst_n, os_valid, os_modified,
input logic in_window, // one of the permitted config states
input logic [3:0] adv_field,
output logic adv_accepted,
output logic [3:0] adv_q,
output logic accepted_outside_window_err, accepted_unmodified_err
);
logic gate;
assign gate = os_valid && os_modified && (IGNORE_WINDOW ? 1'b1 : in_window);
assign adv_accepted = gate;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
adv_q <= 4'h0;
accepted_outside_window_err <= 1'b0; accepted_unmodified_err <= 1'b0;
end else begin
if (adv_accepted) adv_q <= adv_field;
if (adv_accepted && !in_window) accepted_outside_window_err <= 1'b1;
if (adv_accepted && !os_modified) accepted_unmodified_err <= 1'b1;
end
end
endmodule modified OS, in window : accepted=1 adv=0x5
modified OS, OUTSIDE window: accepted=0 adv=0x5 | ignore-window accepted=1
standard OS, in window : accepted=0
ignore-window accepted_outside_window_err=1Two mutations target this module separately — M8 drops the window, M9 drops the variant check — and each is caught by its own diagnostic. That separation is deliberate: two conditions guarding one action need two error signals, or a report tells you something was accepted wrongly without telling you which rule was broken.
12. RTL 5 — What Arrives Is Not What Was Sent
Public material states plainly that retimers may modify the relevant bits during alternate protocol negotiation. This is not a corruption case — it is the mechanism working correctly. A retimer that cannot support a capability removes it, because a path is only as capable as its narrowest element.
module retimer_path #(
parameter bit ASSUME_UNMODIFIED = 1'b0 // 1 = the broken shape
) (
input logic clk, rst_n, valid,
input logic [3:0] far_end_caps, rt1_caps, rt2_caps,
input logic rt1_present, rt2_present,
output logic [3:0] received_caps, path_caps,
output logic path_overclaim_err, silently_narrowed_err
);
logic [3:0] m1, m2;
assign m1 = rt1_present ? rt1_caps : 4'hF;
assign m2 = rt2_present ? rt2_caps : 4'hF;
// What actually arrives has already been narrowed by the path.
assign received_caps = valid ? (far_end_caps & m1 & m2) : 4'h0;
// The broken shape ignores the path and trusts the far end's own claim.
assign path_caps = valid ? (ASSUME_UNMODIFIED ? far_end_caps : received_caps) : 4'h0;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
path_overclaim_err <= 1'b0; silently_narrowed_err <= 1'b0;
end else if (valid) begin
if ((path_caps & ~received_caps) != 4'h0) path_overclaim_err <= 1'b1;
// Diagnostic, not a fault: the path removed something the far end offered.
if ((far_end_caps & ~received_caps) != 4'h0) silently_narrowed_err <= 1'b1;
end
end
endmodule no retimers : received=0xf correct path=0xf | assume-unmodified=0xf
one retimer (0x7) : received=0x7 correct path=0x7 | assume-unmodified=0xf
two retimers : received=0x3 correct path=0x3 | assume-unmodified=0xf <-- overclaim
far end offered 0xf, path carries 0x3
assume-unmodified path_overclaim_err=1 | correct silently_narrowed_err=1The far end offered 0xF and the path can carry 0x3. Both statements are true, and a design that reads the first as "the device's capability" has read the wrong thing. Capability, at negotiation time, is a property of the path — the far end intersected with every retimer on it.
silently_narrowed_err is deliberately not a fault. It fires on the correct design and reports something an operator needs to know: the link is less capable than the device, and the reason is on the board rather than in either endpoint. Distinguishing "diagnostic" from "error" in the same module is worth doing explicitly, because a naming convention that calls everything _err guarantees someone will treat the informational one as a bug.
13. RTL 6 — Three Outcomes, and Timeouts Are Not One of Them
module init_outcome_counters (
input logic clk, rst_n, attempt,
input logic got_cxl, got_pcie, got_timeout, got_error,
output logic [15:0] n_attempt_q, n_cxl_q, n_pcie_q, n_timeout_q, n_error_q,
output logic accounting_err, multi_outcome_err
);
logic [2:0] hot;
assign hot = {1'b0, 1'b0, 1'b0} + got_cxl + got_pcie + got_error;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
n_attempt_q <= '0; n_cxl_q <= '0; n_pcie_q <= '0;
n_timeout_q <= '0; n_error_q <= '0;
accounting_err <= 1'b0; multi_outcome_err <= 1'b0;
end else begin
if (attempt) n_attempt_q <= n_attempt_q + 16'd1;
if (got_cxl) n_cxl_q <= n_cxl_q + 16'd1;
if (got_pcie) n_pcie_q <= n_pcie_q + 16'd1;
if (got_timeout) n_timeout_q <= n_timeout_q + 16'd1;
if (got_error) n_error_q <= n_error_q + 16'd1;
// Every attempt reaches exactly one terminal outcome. Timeouts are NOT a
// fourth outcome -- a timeout resolves to PCIe, so it is counted twice
// on purpose and must not appear in the conservation sum.
if (n_attempt_q != n_cxl_q + n_pcie_q + n_error_q) accounting_err <= 1'b1;
if (hot > 3'd1) multi_outcome_err <= 1'b1;
end
end
endmodule attempts=40 cxl=8 pcie=29 timeout=12 error=3
conservation attempts == cxl+pcie+error : 40 == 40
timeouts are a REASON, not an outcome: 12 of 29 PCIe resultsA timeout is a reason, not an outcome. It resolves to PCIe like any other non-CXL result, so it is counted alongside the outcome rather than within the sum — which is why n_timeout_q is deliberately absent from the conservation identity, and why mutation M12 (adding it) is caught immediately.
This distinction has an operational point. "40% of links fell back to PCIe" and "40% of links fell back because the partner never answered" are very different reports: the first is mostly non-capable partners, the second is a bug or a signal-integrity problem.
14. Assertions
Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog. Not executed; each maps to a procedural stand-in and a mutation.
// SAFETY -------------------------------------------------------------------
// V1 — CXL is never in use before a decision was reached.
a_decide_then_use: assert property (@(posedge clk) disable iff (!rst_n)
cxl_ready |-> decided);
// V2 — the two outcomes are mutually exclusive.
a_one_outcome: assert property (@(posedge clk) disable iff (!rst_n)
!(cxl_ready && pcie_fallback));
// V3 — CXL is entered only from the decision state, never from BASE.
a_no_shortcut: assert property (@(posedge clk) disable iff (!rst_n)
(state_q == S_CXL) |-> $past(state_q) inside {S_DECIDE, S_CXL});
// V4 — an error in CXL falls back by the next cycle.
a_error_falls_back: assert property (@(posedge clk) disable iff (!rst_n)
(cxl_ready && link_error) |=> pcie_fallback);
// V5 — nothing is enabled that the host did not offer.
a_subset_of_offer: assert property (@(posedge clk) disable iff (!rst_n)
complete |-> ((enabled_q & ~offer_q) == '0));
// V6 — the enabled set is a subset of BOTH advertisements.
a_subset_both: assert property (@(posedge clk) disable iff (!rst_n)
valid |-> (((enabled & ~host_caps) == '0) && ((enabled & ~dev_caps) == '0)));
// V7 — an advertisement is accepted only in the window, and only modified.
a_window_and_variant: assert property (@(posedge clk) disable iff (!rst_n)
adv_accepted |-> (in_window && os_modified));
// V8 — nothing is claimed that the path cannot carry.
a_path_bounds_claim: assert property (@(posedge clk) disable iff (!rst_n)
valid |-> ((path_caps & ~received_caps) == '0));
// V9 — CONSERVATION: every attempt reaches exactly one terminal outcome.
a_outcomes_conserved: assert property (@(posedge clk) disable iff (!rst_n)
n_attempt_q == n_cxl_q + n_pcie_q + n_error_q);
// V10 — a reply is never acted on before the offer has been recorded.
a_reply_after_offer: assert property (@(posedge clk) disable iff (!rst_n)
reply_now |-> (phase_q == P_REPLY));
// SAFETY, formerly LIVENESS -------------------------------------------------
// V11 — the machine leaves EXCH within the bound, UNCONDITIONALLY.
// This was a liveness property in Chapter 5.1 with an environment
// assumption about the peer. The timeout discharges the assumption and
// makes it a safety property of the design alone.
a_bounded_wait: assert property (@(posedge clk) disable iff (!rst_n)
(state_q == S_EXCH) |-> (wait_q <= TIMEOUT + 1));
// LIVENESS ------------------------------------------------------------------
// V12 — a link that comes up eventually decides.
// With the timeout present this is IMPLIED BY V11 and needs no
// environment assumption. Stated separately because it is the property
// anyone actually cares about, and V11 is how it is discharged.
a_eventually_decides: assert property (@(posedge clk) disable iff (!rst_n)
base_link_up |-> s_eventually decided);V11 and V12 are the pair worth studying. V12 is what you want; V11 is what you can prove. A good design turns the property you want into one you can prove by adding a bound, and the assertion file should show both so the relationship is visible to whoever reads it next.
15. Mutation Testing
Twelve mutations. Clean code restored after each.
| ID | Mutation | Result |
|---|---|---|
| M1 | the wait has no bound | KILLED — timeout scenario |
| M2 | skip the exchange when locally capable | KILLED — incompatible scenario |
| M3 | one capable end is enough | KILLED — incompatible scenario |
| M4 | an error in CXL does not fall back | KILLED — post-error check |
| M5 | the responder may answer in any phase | KILLED — early-reply check |
| M6 | enable the request, not the intersection | KILLED — partial-offer stimulus |
| M7 | union instead of intersection | KILLED — intersection check |
| M8 | advertisement accepted in any state | KILLED — accepted_outside_window_err |
| M9 | the modified-variant check dropped | KILLED — accepted_unmodified_err |
| M10 | only the first retimer narrows | KILLED — path check |
| M11 | an absent retimer still narrows | KILLED — path check |
| M12 | timeouts counted as a fourth outcome | KILLED — conservation |
12/12 killed, 0 escapedSix of twelve escaped on the first run, and diagnosing them separated into three distinct causes — which is the most useful thing in this chapter for anyone who runs mutation testing.
| Escaped | Cause | Fix |
|---|---|---|
| M3, M4, M10, M11 | missing check — the trace was printed, never asserted | assert the scenario outcome |
| M6 | missing stimulus — host_offer = 0xF makes intersection equal the request | a partial offer |
| M5 | equivalent mutant — the guard was dead logic | restructure so the guard can be false |
M6 is the subtle one. With host_offer = 0xF the intersection 0xF & dev_wants equals dev_wants, so "enable the intersection" and "enable what the device asked for" produce identical results. The testbench was correct, the assertion was correct, and the stimulus made two different designs indistinguishable. Adding one case with host_offer = 0x3 and dev_wants = 0x6 — expecting 0x2 — killed it immediately.
Choose stimulus values that make wrong designs produce wrong answers. An all-ones input is the identity for AND, and identities hide bugs.
M5 is the one that is not a testbench problem at all. The guard may_reply was evaluated inside the P_REPLY case branch, where it is true by construction — deleting it changed nothing because it did nothing. The fix belonged in the RTL, not the tests.
16. Debug Lab
Bring-up hangs and the link never comes up at all
UNBOUNDED-WAIT// Wait for the partner's advertisement.
S_EXCH : if (peer_advert_seen) nxt = S_DECIDE;A device works with every partner in the lab and hangs in one customer system. Not a slow link, not a degraded link — no link. The port never reaches a usable state and there is no error to report.
with timeout : decided=1 pcie_fallback=1
without timeout: decided=0 still waiting, wait=111There is no exit from EXCH other than hearing the partner. If the partner never advertises — because it is not CXL-capable and does not participate, or because it is broken, or because a retimer dropped the traffic — the machine waits forever.
The circularity in §1 makes this unavoidable in principle: there is no way to distinguish a slow partner from an absent one except by deciding that enough time has passed. A design without that decision has no way to conclude anything.
S_EXCH : if (peer_advert_seen) nxt = S_DECIDE;
else if (timed_out) nxt = S_PCIE; // waiting becomes a decisionAny state that waits on a remote party needs a bound, and the bound must lead somewhere legal. The timeout is not error handling — it resolves to the perfectly good PCIe outcome, which is the right answer for the most common cause. It also converts an unprovable liveness property into a provable safety one, which is why V11 and V12 in §14 are a pair.
A device enters CXL mode against a PCIe-only partner
EXCHANGE-SKIPPED// We support CXL, so go straight there and save the round trip.
S_BASE : nxt = local_cxl ? S_CXL : S_EXCH;A CXL device in a PCIe-only slot brings the link up and immediately starts transmitting CXL traffic. The partner sees malformed PCIe, the link retrains, and the cycle repeats — presenting as a signal-integrity problem because retraining is what marginal channels do.
incompatible scenario did not fall back (cxl=1 pcie=0)An edge into S_CXL that does not pass through S_DECIDE. The optimisation reasoning is superficially sound — the round trip costs time, and the device does support CXL — but it answers the wrong question. Local capability is not the decision. Chapter 5.1's two-key rule is exactly this: capability is a property of the pair.
The failure looks like a physical problem because the symptom is retraining, which sends debugging to the channel rather than to the FSM.
S_BASE : nxt = S_EXCH; // no shortcut
S_DECIDE: nxt = (local_cxl && peer_cxl) ? S_CXL : S_PCIE; // the only way inAudit the in-edges of your terminal states, not just the out-edges. A state machine is correct about how it leaves a state and quietly wrong about how it got there. Every arrow into an operating state should be traceable to the decision that authorised it — and an assertion of the form "state X is only entered from state Y" catches this class cheaply.
Both ends agree, and the link enables a protocol neither supports
UNION-NOT-INTERSECTION// Combine both ends' capabilities.
assign enabled = host_caps | dev_caps;Negotiation completes and reports success. The link then fails on the first transaction of a protocol class one end does not implement — and the failure is at the protocol layer, far from bring-up.
host=0x9 dev=0x6 : correct=0x0 any=0 | union-bug=0xf any=1 <-- disjoint
not_subset_host_err=1 not_subset_dev_err=1"Combine" was implemented as union. The correct operation is intersection, and the difference is invisible whenever one side is a superset of the other — which is the common case in a lab, where the host typically supports everything.
The disjoint case makes it obvious: host 0x9 and device 0x6 share nothing, so the correct answer is that no CXL capability is enabled at all. The union enables all four.
assign enabled = host_caps & dev_caps;
// and check it from both sides, separately:
if ((enabled & ~host_caps) != 4'h0) not_subset_host_err <= 1'b1;
if ((enabled & ~dev_caps) != 4'h0) not_subset_dev_err <= 1'b1;Test agreement logic with disjoint and partial inputs, never with a superset on one side. This is the same trap as mutation M6: an all-ones operand is the identity for AND, so it makes intersection and "just take the other side" indistinguishable. The two subset assertions are stated separately because a union breaks both while a one-sided bug breaks exactly one, and that difference is the diagnosis.
Capability bits are read from traffic that was not carrying them
OUTSIDE-THE-WINDOW// A modified ordered set arrived; take the advertisement.
assign gate = os_valid && os_modified;Negotiation results vary between otherwise identical bring-ups on the same hardware. Occasionally a capability is enabled that the partner does not have, and occasionally one that both have is missed.
modified OS, OUTSIDE window: accepted=0 adv=0x5 | ignore-window accepted=1
accepted_outside_window_err=1The negotiation is carried during specific Configuration states. Outside that window the same ordered-set positions are not carrying negotiation information, so accepting them samples whatever is there.
Two conditions guard this action — the right ordered-set variant and the right point in bring-up — and only one was checked. Because the variant check passed, the read looked legitimate.
assign gate = os_valid && os_modified && in_window;
// two conditions, two diagnostics:
if (adv_accepted && !in_window) accepted_outside_window_err <= 1'b1;
if (adv_accepted && !os_modified) accepted_unmodified_err <= 1'b1;When two conditions guard one action, give them two error signals. A single combined diagnostic reports that something was accepted wrongly without saying which rule was broken, and the two have different fixes in different parts of the design. Mutations M8 and M9 exist as a pair for the same reason — each drops one condition, and each must be attributable.
A link claims capabilities the board cannot carry
RETIMER-NARROWING-IGNORED// The far end told us what it supports.
assign path_caps = far_end_caps;Two systems with the same host and the same device negotiate different results, and the one with the longer trace run — two retimers rather than none — fails after negotiation reports success. Swapping devices does not help; swapping boards does.
two retimers : received=0x3 correct path=0x3 | assume-unmodified=0xf <-- overclaim
far end offered 0xf, path carries 0x3
path_overclaim_err=1Retimers on the path may modify the relevant bits during negotiation — that is the mechanism working as designed, not corruption. A retimer that cannot support a capability removes it, because a path is only as capable as its narrowest element.
Reading far_end_caps as "the device's capability" reads a true statement that answers the wrong question. What matters at negotiation time is what the path can carry: the far end intersected with every retimer on it.
assign received_caps = far_end_caps & m1 & m2; // what actually arrived
assign path_caps = received_caps; // claim only what the path carries
// and report the narrowing, which is information rather than a fault:
if ((far_end_caps & ~received_caps) != 4'h0) silently_narrowed_err <= 1'b1;Capability at negotiation time is a property of the path, not of the endpoints. Note also that the narrowing report is a diagnostic, not an error — it fires on the correct design and tells an operator something they need: the link is less capable than the device, and the cause is on the board. A convention that suffixes everything _err guarantees this gets triaged as a bug.
Fleet statistics say 30% PCIe fallback and the cause is unknowable
REASON-CONFLATED-WITH-OUTCOME// Count what happened.
if (got_pcie || got_timeout) n_pcie_q <= n_pcie_q + 16'd1;A fleet reports 30% of links falling back to PCIe. That number is consistent with a healthy deployment of mixed-capability partners and with a serious negotiation bug, and nothing in the data distinguishes them.
attempts=40 cxl=8 pcie=29 timeout=12 error=3
conservation attempts == cxl+pcie+error : 40 == 40
timeouts are a REASON, not an outcome: 12 of 29 PCIe resultsTimeout was modelled as a kind of outcome and merged into the PCIe count. It is not an outcome — it is a reason for one. A link that times out lands in PCIe exactly like a link whose partner cheerfully advertised no CXL support, and those two populations need very different responses.
Merging them also breaks the conservation law, because the same attempt gets counted twice in a sum that is supposed to partition.
if (got_pcie) n_pcie_q <= n_pcie_q + 16'd1; // the outcome
if (got_timeout) n_timeout_q <= n_timeout_q + 16'd1; // the reason, alongside
// conservation partitions the OUTCOMES only:
if (n_attempt_q != n_cxl_q + n_pcie_q + n_error_q) accounting_err <= 1'b1;Outcomes partition; reasons annotate. Keep them in separate counters and let only the outcomes appear in the conservation sum. Of this run's 29 PCIe results, 12 were timeouts — a proportion that is a red flag on a fleet where most partners are known CXL-capable, and completely unremarkable on one where they are not. Neither reading is available from a merged count.
17. Verification Plan
| Item | Approach and goal |
|---|---|
| Bring-up outcomes | assert the terminal state per scenario — all four asserted |
| Terminal in-edges | CXL entered only from DECIDE — no shortcut reachable |
| Bounded wait | sweep partner latency across the bound — exits every time |
| Phase ordering | reply before, at and after the offer — only in-phase completes |
| Intersection | cross host x device caps incl. disjoint and partial |
| Window and variant | cross window x modified — accepted in one of four cells |
| Retimer path | 0, 1, 2 retimers, distinct masks — received = full path AND |
| Outcome accounting | mixed population, exclusive by construction — conserves |
| Diagnostic liveness | broken variants, latched _ever — every diagnostic fires |
The fifth row carries this chapter's sharpest lesson: a superset operand is an identity, and identities make wrong designs look right. Any cross that only ever offers 0xF on one side has not tested the operation.
18. Design Review
- Does every waiting state have a bound, and does the bound lead to a legal outcome rather than an error?
- Can any operating state be entered without passing through the decision state?
- Is the combining function an intersection, and is it checked from both sides separately?
- Are the variant and the window checked as two conditions with two diagnostics?
- Is capability computed over the path or read from the far end's claim?
- Are reasons kept out of the outcome partition?
- Does any assertion depend on a stimulus value that happens to be an identity for its operation?
- Has every diagnostic been observed to fire, in a way that survives the test's own resets?
19. How This Appears in Real Engineering
Bring-up hangs are triaged as physical problems. Debug Lab 2's symptom is repeated retraining, which is exactly what a marginal channel looks like, so effort goes to the board while the fault is in the FSM. Asking "does the state sequence match what we expect" early is cheap and rarely done first.
Retimer count is a system-level variable that endpoint teams do not control. Lab 5's failure appears when a device moves from a short lab trace to a long production one. Any negotiation test plan that does not include retimer configurations is testing one point of a space the customer will sweep.
Timeouts are tuned late and badly. Too short and capable partners get dropped to PCIe under load; too long and a dead partner delays boot. Because the failure mode of a slightly-too-short timeout is a working PCIe link, it is easy to ship and hard to notice — which is Debug Lab 6's reporting problem again.
The lab always has a superset host. Validation hosts support everything, so intersection bugs are systematically invisible until a customer with a partial-capability host finds them. Deliberately configuring a reduced-capability host is one of the cheapest high-yield tests available.
20. Common Misconceptions
| Claim | Why it is wrong |
|---|---|
| "CXL has its own link training state machine" | It does not. Public material describes CXL using the alternate protocol negotiation mechanism defined in the PCIe specifications, carried in modified TS ordered sets during existing PCIe Configuration states. The PCIe LTSSM is not replaced. |
| "Negotiation happens after the link is at full speed" | It happens before entering L0 at Gen 1 speed, during Configuration states, with the link at 2.5 GT/s. The link reaches 8.0 GT/s or higher afterwards. |
| "The device tells the host what it supports" | Two phases, and the host goes first: the root complex advertises, then the endpoint responds indicating which it wishes to enable. A responder that speaks first is answering an unasked question. |
| "What the device advertised is what the host receives" | Retimers may modify the relevant bits. What arrives is the far end narrowed by every retimer on the path. |
| "A timeout is an error" | A timeout is a reason, and it resolves to the entirely correct PCIe outcome. Treating it as an error inverts the common case, where the partner is simply not CXL-capable. |
| "Negotiation enables the union of both ends' capabilities" | The intersection. A union enables protocols one end cannot speak, and the bug is invisible whenever one side is a superset. |
| "If the link is up, the negotiation succeeded" | The link being up is the precondition for negotiation, not its result. A PCIe link is a fully successful bring-up outcome. |
21. Interview Reasoning
22. Exercises
-
Trace. Using §6's FSM, write the state sequence for: a capable partner that advertises at cycle 3; a capable partner that advertises at cycle 12 with
TIMEOUT = 8; and a partner that advertises at cycle 3 withpeer_cxl = 0. State which two produce the same terminal state for different reasons, and what report would distinguish them. -
Calculate. A host advertises
0xDand a device0xB, with two retimers masking0xEand0x7. Compute what arrives, what should be enabled, and what a design that trusts the far end would claim. Then find a retimer pair that makes the correct and incorrect designs agree, and explain why that configuration is a bad test. -
DV task. Write the coverage cross for the intersection logic that would have caught mutation M6. Explain why a cross over
host_capsanddev_capswith an all-ones bin is not sufficient, and state the bin you would add. -
Debug task. A port hangs during bring-up in one system. You can capture one signal group. Choose it, justify the choice, and give the decision tree from what you would see to the subsystem you would investigate.
-
Design. Add a fourth outcome to §13's counters — "negotiated CXL but the path narrowed it below what the device offered". State whether it belongs in the conservation sum and why, and which existing diagnostic it makes redundant.
-
Critique. Argue that the timeout in §6 should transition to
ERRORrather thanPCIE. Give the strongest case, then identify what it costs in the most common real-world scenario and which fleet statistic would reveal the mistake.
23. Summary
CXL link initialisation breaks a circular dependency — you must ask the partner what it supports, and asking requires a link — by bringing the link up as PCIe and holding the conversation in-band.
- The mechanism is PCIe's own alternate protocol negotiation, carried in modified TS1/TS2 ordered sets during Configuration states at 2.5 GT/s. CXL does not define a separate training state machine and does not replace the LTSSM.
- It is two-phase and ordered: the root complex advertises, the endpoint responds with what it wishes to enable. Neither step commutes.
- The result is the intersection, and capability is a property of the path — the far end narrowed by every retimer, which may modify the relevant bits.
- Waiting must be bounded. The timeout converts an unprovable liveness property into a provable safety one, and it resolves to PCIe rather than to an error, because a non-capable partner is the common case.
- Three outcomes: CXL, PCIe, error. A timeout is a reason, not an outcome — outcomes partition, reasons annotate.
- Verification lessons: a surviving mutation has three possible causes — missing check, missing stimulus, equivalent mutant — with three different fixes; identity-valued stimulus hides bugs; and a conservation law is strong enough to contradict the testbench that feeds it.
Chapter 5.4 takes up what happens next: the widths, rates and recovery behaviour that determine the operating point the link actually settles on.
Standards & specifications
- Governing standard
- CXL Specification (CXL Consortium)(opens CXL Consortium in a new tab)
Defines CXL.io, CXL.cache and CXL.mem, and the coherence and memory-pooling behaviour built on them. System design and deployment topology are not mandated.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the CXL curriculum.
