Skip to content
VLSI Mentor

CXL · Module 5

CXL Link Initialisation

How a CXL link decides what it is: the two-phase alternate protocol negotiation carried in modified TS ordered sets, why timeouts are mandatory, what retimers do to an advertisement, and the three outcomes. Six RTL models simulated, twelve mutations, twelve killed.

Chapter 5.1 established the two-key rule: CXL operation requires both ends, and neither may decide alone. Chapter 5.2 established that the mechanism is PCIe's own, not a CXL invention.

This chapter is the mechanism. How do two devices that have never met agree on what protocol to speak — over a link that must already be working before the conversation can start?

1. The Engineering Problem — A Conversation That Must Precede Itself

To find out whether your link partner supports CXL, you have to ask it. To ask it, you need a working link. To bring up a working link, you need to know what protocol to run.

That is circular, and every multi-protocol interconnect has to break the circle the same way: bring the link up in the mode that always works, hold the conversation over that link, then decide. Nothing else is possible, because there is no side channel.

Three consequences follow immediately, and they shape everything below.

  1. The link is a PCIe link before it is anything else. Not conceptually — literally. It trains, reaches a usable state, and carries ordered sets, all as PCIe.
  2. The negotiation is in-band. It travels in the same ordered sets the link uses to train, which is why its fields live in specific symbol positions of a specific ordered-set variant.
  3. A partner that never answers must not hang the link. There is no way to distinguish "slow" from "absent" except by deciding that enough time has passed.

2. The One-Sentence Model

CXL link initialisation is a two-phase capability exchange carried inside PCIe's own training traffic, guarded by a timeout, whose result is the intersection of what both ends offered and what the path between them can actually carry — and whose only three outcomes are CXL, PCIe, or error.

Call it decide, then use. Every failure in this chapter is some version of using before deciding, or never deciding at all.

3. What This Chapter Owns

QuestionOwned by
Why the reuse exists5.1
Which structures are reused, and their class5.2
How the two ends agree — the negotiation itselfthis chapter
Widths, rates, recovery, the operating point5.4
How software learns what was agreed5.5

4. The Ordering, and Why It Cannot Be Rearranged

Link comes up as PCIe at Gen 1, root complex advertises Flex Bus capabilities in modified TS ordered sets, retimers may narrow them in transit, endpoint responds with what it wishes to enable, and only then is the operating protocol decidedroot complexretimer(s)endpointlink trains as PCIeat Gen 1 (2.5 GT/s)phase 1 — advertisecapabilities(modified TS)may modify therelevant bits intransitphase 2 — respond:which to enabledecision: enabled =intersectionlink proceeds to 8.0GT/s or higher

Each arrow depends on the one above it, and none of them commute.

StepDepends onIf rearranged
Train as PCIenothingno link to talk over
Advertisea working linkshouting into a dead channel
Respondhearing the advertanswering an unasked question
Decideboth advertsdeciding on one end's opinion
Usethe decisionunilateral turn-on (5.1)

5. Teaching-model boundary

6. RTL 1 — The Bring-Up Decision as a State Machine

Architectural teaching FSM for CXL bring-up: reset waits for the base link, the base state moves to an exchange state which either sees the partner advertisement and decides, or times out to PCIe fallback; decide enters CXL only when both ends are capable, and an error in CXL falls back to PCIeRESETBASEEXCHDECIDECXLPCIEERRORbase link upbase link upalwaysalwaysadvert seenadvert seentimeouttimeoutboth capableboth capablenot bothnot bothlink errorlink errorbase link lostbase linklost

Three properties of this graph carry the chapter.

DECIDE has no self-loop. It resolves in one step, to exactly one of two states. A decision state that can wait is a decision state that can hang.

EXCH has two exits. One on hearing the partner, one on the clock running out. Remove the second and the machine has no path out of waiting — which is mutation M1.

Every arrow into CXL passes through DECIDE. There is no shortcut from BASE, which is mutation M2. Once that edge exists, a locally-capable device enters CXL without the exchange having happened.

init_fsm.sv — the ordering, as a machine
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module init_fsm #(
  parameter int unsigned TIMEOUT    = 8,
  parameter bit          NO_TIMEOUT = 1'b0   // 1 = the hanging shape
) (
  input  logic       clk, rst_n,
  input  logic       base_link_up,     // link reached the base operating state
  input  logic       local_cxl,
  input  logic       peer_advert_seen, // partner's advertisement observed
  input  logic       peer_cxl, link_error,
  output logic [2:0] state_q,
  output logic       cxl_ready, pcie_fallback, in_error, decided,
  output logic [7:0] wait_q,
  output logic       used_before_decided_err, both_outcomes_err, hung_err
);
  localparam logic [2:0] S_RESET = 3'd0, S_BASE = 3'd1, S_EXCH = 3'd2,
                         S_DECIDE = 3'd3, S_CXL = 3'd4, S_PCIE = 3'd5, S_ERR = 3'd6;
  logic [2:0] nxt;
  logic       timed_out;
 
  assign timed_out     = !NO_TIMEOUT && (wait_q >= TIMEOUT[7:0]);
  assign cxl_ready     = (state_q == S_CXL);
  assign pcie_fallback = (state_q == S_PCIE);
  assign in_error      = (state_q == S_ERR);
  assign decided       = cxl_ready || pcie_fallback;
 
  always_comb begin
    nxt = state_q;
    case (state_q)
      S_RESET : if (base_link_up) nxt = S_BASE;
      // The link is up. It is a PCIe link. Nothing CXL has happened yet.
      S_BASE  : nxt = S_EXCH;
      // Wait to HEAR the partner. A timeout converts waiting into a decision.
      S_EXCH  : if (peer_advert_seen) nxt = S_DECIDE;
                else if (timed_out)   nxt = S_PCIE;
      // Both keys, or fall back. This state cannot loop.
      S_DECIDE: nxt = (local_cxl && peer_cxl) ? S_CXL : S_PCIE;
      S_CXL   : if (link_error) nxt = S_PCIE;   // fall back, never forward
      S_PCIE  : ;                               // terminal for this bring-up
      S_ERR   : ;
      default : nxt = S_ERR;
    endcase
    if (state_q != S_RESET && !base_link_up) nxt = S_ERR;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      state_q <= S_RESET; wait_q <= 8'd0;
      used_before_decided_err <= 1'b0; both_outcomes_err <= 1'b0; hung_err <= 1'b0;
    end else begin
      state_q <= nxt;
      wait_q  <= (nxt == S_EXCH) ? (wait_q + 8'd1) : 8'd0;
 
      if (cxl_ready && !decided)      used_before_decided_err <= 1'b1;
      if (cxl_ready && pcie_fallback) both_outcomes_err       <= 1'b1;
      // Liveness made checkable: waiting past the bound is a detected fault.
      if ((state_q == S_EXCH) && (wait_q > TIMEOUT[7:0] + 8'd2)) hung_err <= 1'b1;
    end
  end
endmodule

7. Waveform — The Same Bring-Up, Two Partners

The stimulus below is identical in both cases except for one bit: whether the partner is CXL-capable. Everything through cycle 5 is the same link doing the same thing.

Success and incompatible: identical until the decision resolves

10 cycles
Both bring-ups train as PCIe, reach the exchange state, observe the partner advertisement at cycle 4 and decide at cycle 5; the capable partner enters the CXL state at cycle 6 while the non-capable partner falls back to the PCIe stateno link yetno link yetPCIe link up — exchangePCIe link up — exchangedecidedecideoutcomeoutcomepartner advertisement observedpartner advertisementobservedoutcomes diverge — one bit of differenceoutcomes diverge — one bitof differenceclkbase_link_upadvert_seenwait_q0001200000peer_cxl (A)state (A)RESETRESETBASEEXCHEXCHDECIDECXLCXLCXLCXLpeer_cxl (B)state (B)RESETRESETBASEEXCHEXCHDECIDEPCIEPCIEPCIEPCIEt0t1t2t3t4t5t6t7t8t9
Icarus Verilog 13.0, EXP1 and EXP2. Cycle-by-cycle values taken from the simulation transcript.

Read the two state rows together. Six of ten cycles are identical, and the divergence is a single input bit resolved in a single state. That is what "the compatibility path is not a separate design" means in practice — it is the same machine reaching a different terminal state.

8. Waveform — The Partner That Never Answers

Timeout: waiting is converted into a decision

14 cycles
With no partner advertisement the exchange state counts up to the timeout bound of eight and then falls back to PCIe at cycle eleven; the variant built without a timeout stays in the exchange state indefinitelywaiting — boundedwaiting — boundeddecided: PCIedecided: PCIetimeout reached — waiting becomes a decisiontimeout reached — waitingbecomes a decisionclkbase_link_upadvert_seenwait_q00012345678000stateRESETRESETBASEEXCHEXCHEXCHEXCHEXCHEXCHEXCHEXCHPCIEPCIEPCIEno-timeoutRESETRESETBASEEXCHEXCHEXCHEXCHEXCHEXCHEXCHEXCHEXCHEXCHEXCHt0t1t2t3t4t5t6t7t8t9t10t11t12t13
Icarus Verilog 13.0, EXP3, TIMEOUT = 8. The no-timeout row is the same design with the bound removed.

The bottom row is the point. The design without a timeout is not slower — it never finishes. In simulation it sat in EXCH for the remainder of the run, and the testbench's own liveness check reported wait_q = 111 at the end.

9. RTL 2 — Two Phases, In Order

Public material describes the first phase as the root complex advertising its capabilities, and the endpoint then responding to indicate which it wishes to enable. The ordering is the whole content.

two_phase_negotiate.sv — advertise, then respond
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module two_phase_negotiate #(
  parameter bit RESPOND_EARLY = 1'b0    // 1 = the broken shape
) (
  input  logic       clk, rst_n, start,
  input  logic [3:0] host_offer,
  input  logic       host_offer_valid,
  input  logic [3:0] dev_wants,
  input  logic       dev_reply_valid,
  output logic [1:0] phase_q,
  output logic [3:0] enabled_q,
  output logic       complete,
  output logic       replied_before_offer_err, enabled_not_offered_err
);
  localparam logic [1:0] P_IDLE = 2'd0, P_OFFER = 2'd1, P_REPLY = 2'd2, P_DONE = 2'd3;
  logic [3:0] offer_q;
  logic       may_reply, reply_now;
 
  // The guard must be evaluated OUTSIDE the P_REPLY branch, or it is
  // tautological -- inside that branch phase_q == P_REPLY by construction, so
  // the term can never be false and mutating it away changes nothing.
  assign may_reply = RESPOND_EARLY ? (phase_q != P_DONE) : (phase_q == P_REPLY);
  assign reply_now = dev_reply_valid && may_reply;
  assign complete  = (phase_q == P_DONE);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      phase_q <= P_IDLE; offer_q <= 4'h0; enabled_q <= 4'h0;
      replied_before_offer_err <= 1'b0; enabled_not_offered_err <= 1'b0;
    end else begin
      if (reply_now) begin
        // What is enabled is the INTERSECTION, never the request.
        enabled_q <= offer_q & dev_wants;
        phase_q   <= P_DONE;
      end else begin
        case (phase_q)
          P_IDLE  : if (start) phase_q <= P_OFFER;
          P_OFFER : if (host_offer_valid) begin
                      offer_q <= host_offer; phase_q <= P_REPLY;
                    end
          default : ;
        endcase
      end
 
      if (reply_now && (phase_q != P_REPLY)) replied_before_offer_err <= 1'b1;
      // Nothing may be enabled that the host did not offer.
      if (complete && ((enabled_q & ~offer_q) != 4'h0)) enabled_not_offered_err <= 1'b1;
    end
  end
endmodule
Icarus Verilog 13.0 — EXP4
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  after start        : correct phase=1 | respond-early phase=1
  device replies EARLY: correct enabled=0x0 complete=0 | early enabled=0x0 complete=0
  host offers 0xf    : correct phase=2
  device wants 0x6   : correct enabled=0x6 complete=1
  respond-early replied_before_offer_err=1
  partial offer 0x3, wants 0x6 : enabled=0x2 (expected 0x2)

10. RTL 3 — The Result Is an Intersection

capability_intersect.sv — subset of both, stated separately
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module capability_intersect #(
  parameter bit USE_UNION = 1'b0        // 1 = the broken shape
) (
  input  logic       clk, rst_n,
  input  logic [3:0] host_caps, dev_caps,
  input  logic       valid,
  output logic [3:0] enabled,
  output logic       any_enabled,
  output logic       not_subset_host_err, not_subset_dev_err, empty_but_claimed_err
);
  assign enabled     = valid ? (USE_UNION ? (host_caps | dev_caps)
                                          : (host_caps & dev_caps)) : 4'h0;
  assign any_enabled = |enabled;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      not_subset_host_err <= 1'b0; not_subset_dev_err <= 1'b0;
      empty_but_claimed_err <= 1'b0;
    end else if (valid) begin
      // Subset of each side, stated separately: a union breaks both, and a
      // one-sided bug breaks exactly one.
      if ((enabled & ~host_caps) != 4'h0) not_subset_host_err <= 1'b1;
      if ((enabled & ~dev_caps)  != 4'h0) not_subset_dev_err  <= 1'b1;
      if (any_enabled && ((host_caps & dev_caps) == 4'h0)) empty_but_claimed_err <= 1'b1;
    end
  end
endmodule
Icarus Verilog 13.0 — EXP5
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  host=0xF dev=0x6 : correct=0x6 | union-bug=0xf
  host=0x9 dev=0x6 : correct=0x0 any=0 | union-bug=0xf any=1  <-- disjoint
  host=0x3 dev=0x7 : correct=0x3 | union-bug=0x7
  union-bug not_subset_host_err=1 not_subset_dev_err=1

Row 2 is the case worth dwelling on: host and device advertise disjoint capability sets, so the correct intersection is empty and nothing CXL is enabled. The union bug enables everything, on a pair that agrees on nothing.

The two subset checks are stated separately on purpose. A union breaks both; a one-sided bug — using only the host's set, say — breaks exactly one. A single combined check would conflate two distinct failures into one report.

11. RTL 4 — The Negotiation Has a Window

Public material places the negotiation in modified TS1/TS2 ordered sets during specific Configuration states. Two conditions, both required: the right ordered-set variant, and the right point in bring-up.

negotiation_window.sv — right variant, right moment
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module negotiation_window #(
  parameter bit IGNORE_WINDOW = 1'b0    // 1 = the broken shape
) (
  input  logic       clk, rst_n, os_valid, os_modified,
  input  logic       in_window,        // one of the permitted config states
  input  logic [3:0] adv_field,
  output logic       adv_accepted,
  output logic [3:0] adv_q,
  output logic       accepted_outside_window_err, accepted_unmodified_err
);
  logic gate;
  assign gate         = os_valid && os_modified && (IGNORE_WINDOW ? 1'b1 : in_window);
  assign adv_accepted = gate;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      adv_q <= 4'h0;
      accepted_outside_window_err <= 1'b0; accepted_unmodified_err <= 1'b0;
    end else begin
      if (adv_accepted) adv_q <= adv_field;
      if (adv_accepted && !in_window)   accepted_outside_window_err <= 1'b1;
      if (adv_accepted && !os_modified) accepted_unmodified_err     <= 1'b1;
    end
  end
endmodule
Icarus Verilog 13.0 — EXP6
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  modified OS, in window     : accepted=1 adv=0x5
  modified OS, OUTSIDE window: accepted=0 adv=0x5 | ignore-window accepted=1
  standard OS, in window     : accepted=0
  ignore-window accepted_outside_window_err=1

Two mutations target this module separately — M8 drops the window, M9 drops the variant check — and each is caught by its own diagnostic. That separation is deliberate: two conditions guarding one action need two error signals, or a report tells you something was accepted wrongly without telling you which rule was broken.

12. RTL 5 — What Arrives Is Not What Was Sent

Public material states plainly that retimers may modify the relevant bits during alternate protocol negotiation. This is not a corruption case — it is the mechanism working correctly. A retimer that cannot support a capability removes it, because a path is only as capable as its narrowest element.

retimer_path.sv — capability is a property of the path
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module retimer_path #(
  parameter bit ASSUME_UNMODIFIED = 1'b0   // 1 = the broken shape
) (
  input  logic       clk, rst_n, valid,
  input  logic [3:0] far_end_caps, rt1_caps, rt2_caps,
  input  logic       rt1_present, rt2_present,
  output logic [3:0] received_caps, path_caps,
  output logic       path_overclaim_err, silently_narrowed_err
);
  logic [3:0] m1, m2;
  assign m1 = rt1_present ? rt1_caps : 4'hF;
  assign m2 = rt2_present ? rt2_caps : 4'hF;
 
  // What actually arrives has already been narrowed by the path.
  assign received_caps = valid ? (far_end_caps & m1 & m2) : 4'h0;
  // The broken shape ignores the path and trusts the far end's own claim.
  assign path_caps     = valid ? (ASSUME_UNMODIFIED ? far_end_caps : received_caps) : 4'h0;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      path_overclaim_err <= 1'b0; silently_narrowed_err <= 1'b0;
    end else if (valid) begin
      if ((path_caps & ~received_caps) != 4'h0) path_overclaim_err <= 1'b1;
      // Diagnostic, not a fault: the path removed something the far end offered.
      if ((far_end_caps & ~received_caps) != 4'h0) silently_narrowed_err <= 1'b1;
    end
  end
endmodule
Icarus Verilog 13.0 — EXP7
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  no retimers        : received=0xf correct path=0xf | assume-unmodified=0xf
  one retimer (0x7)  : received=0x7 correct path=0x7 | assume-unmodified=0xf
  two retimers       : received=0x3 correct path=0x3 | assume-unmodified=0xf  <-- overclaim
  far end offered 0xf, path carries 0x3
  assume-unmodified path_overclaim_err=1 | correct silently_narrowed_err=1

The far end offered 0xF and the path can carry 0x3. Both statements are true, and a design that reads the first as "the device's capability" has read the wrong thing. Capability, at negotiation time, is a property of the path — the far end intersected with every retimer on it.

silently_narrowed_err is deliberately not a fault. It fires on the correct design and reports something an operator needs to know: the link is less capable than the device, and the reason is on the board rather than in either endpoint. Distinguishing "diagnostic" from "error" in the same module is worth doing explicitly, because a naming convention that calls everything _err guarantees someone will treat the informational one as a bug.

13. RTL 6 — Three Outcomes, and Timeouts Are Not One of Them

init_outcome_counters.sv — with a conservation law
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module init_outcome_counters (
  input  logic        clk, rst_n, attempt,
  input  logic        got_cxl, got_pcie, got_timeout, got_error,
  output logic [15:0] n_attempt_q, n_cxl_q, n_pcie_q, n_timeout_q, n_error_q,
  output logic        accounting_err, multi_outcome_err
);
  logic [2:0] hot;
  assign hot = {1'b0, 1'b0, 1'b0} + got_cxl + got_pcie + got_error;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_attempt_q <= '0; n_cxl_q <= '0; n_pcie_q <= '0;
      n_timeout_q <= '0; n_error_q <= '0;
      accounting_err <= 1'b0; multi_outcome_err <= 1'b0;
    end else begin
      if (attempt)     n_attempt_q <= n_attempt_q + 16'd1;
      if (got_cxl)     n_cxl_q     <= n_cxl_q     + 16'd1;
      if (got_pcie)    n_pcie_q    <= n_pcie_q    + 16'd1;
      if (got_timeout) n_timeout_q <= n_timeout_q + 16'd1;
      if (got_error)   n_error_q   <= n_error_q   + 16'd1;
 
      // Every attempt reaches exactly one terminal outcome. Timeouts are NOT a
      // fourth outcome -- a timeout resolves to PCIe, so it is counted twice
      // on purpose and must not appear in the conservation sum.
      if (n_attempt_q != n_cxl_q + n_pcie_q + n_error_q) accounting_err <= 1'b1;
      if (hot > 3'd1) multi_outcome_err <= 1'b1;
    end
  end
endmodule
Icarus Verilog 13.0 — EXP8, 40 attempts
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  attempts=40 cxl=8 pcie=29 timeout=12 error=3
  conservation attempts == cxl+pcie+error : 40 == 40
  timeouts are a REASON, not an outcome: 12 of 29 PCIe results

A timeout is a reason, not an outcome. It resolves to PCIe like any other non-CXL result, so it is counted alongside the outcome rather than within the sum — which is why n_timeout_q is deliberately absent from the conservation identity, and why mutation M12 (adding it) is caught immediately.

This distinction has an operational point. "40% of links fell back to PCIe" and "40% of links fell back because the partner never answered" are very different reports: the first is mostly non-capable partners, the second is a bug or a signal-integrity problem.

14. Assertions

Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog. Not executed; each maps to a procedural stand-in and a mutation.

cxl_link_init_sva.sv — bind-ready properties
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SAFETY -------------------------------------------------------------------
// V1 — CXL is never in use before a decision was reached.
a_decide_then_use: assert property (@(posedge clk) disable iff (!rst_n)
  cxl_ready |-> decided);
 
// V2 — the two outcomes are mutually exclusive.
a_one_outcome: assert property (@(posedge clk) disable iff (!rst_n)
  !(cxl_ready && pcie_fallback));
 
// V3 — CXL is entered only from the decision state, never from BASE.
a_no_shortcut: assert property (@(posedge clk) disable iff (!rst_n)
  (state_q == S_CXL) |-> $past(state_q) inside {S_DECIDE, S_CXL});
 
// V4 — an error in CXL falls back by the next cycle.
a_error_falls_back: assert property (@(posedge clk) disable iff (!rst_n)
  (cxl_ready && link_error) |=> pcie_fallback);
 
// V5 — nothing is enabled that the host did not offer.
a_subset_of_offer: assert property (@(posedge clk) disable iff (!rst_n)
  complete |-> ((enabled_q & ~offer_q) == '0));
 
// V6 — the enabled set is a subset of BOTH advertisements.
a_subset_both: assert property (@(posedge clk) disable iff (!rst_n)
  valid |-> (((enabled & ~host_caps) == '0) && ((enabled & ~dev_caps) == '0)));
 
// V7 — an advertisement is accepted only in the window, and only modified.
a_window_and_variant: assert property (@(posedge clk) disable iff (!rst_n)
  adv_accepted |-> (in_window && os_modified));
 
// V8 — nothing is claimed that the path cannot carry.
a_path_bounds_claim: assert property (@(posedge clk) disable iff (!rst_n)
  valid |-> ((path_caps & ~received_caps) == '0));
 
// V9 — CONSERVATION: every attempt reaches exactly one terminal outcome.
a_outcomes_conserved: assert property (@(posedge clk) disable iff (!rst_n)
  n_attempt_q == n_cxl_q + n_pcie_q + n_error_q);
 
// V10 — a reply is never acted on before the offer has been recorded.
a_reply_after_offer: assert property (@(posedge clk) disable iff (!rst_n)
  reply_now |-> (phase_q == P_REPLY));
 
// SAFETY, formerly LIVENESS -------------------------------------------------
// V11 — the machine leaves EXCH within the bound, UNCONDITIONALLY.
//       This was a liveness property in Chapter 5.1 with an environment
//       assumption about the peer. The timeout discharges the assumption and
//       makes it a safety property of the design alone.
a_bounded_wait: assert property (@(posedge clk) disable iff (!rst_n)
  (state_q == S_EXCH) |-> (wait_q <= TIMEOUT + 1));
 
// LIVENESS ------------------------------------------------------------------
// V12 — a link that comes up eventually decides.
//       With the timeout present this is IMPLIED BY V11 and needs no
//       environment assumption. Stated separately because it is the property
//       anyone actually cares about, and V11 is how it is discharged.
a_eventually_decides: assert property (@(posedge clk) disable iff (!rst_n)
  base_link_up |-> s_eventually decided);

V11 and V12 are the pair worth studying. V12 is what you want; V11 is what you can prove. A good design turns the property you want into one you can prove by adding a bound, and the assertion file should show both so the relationship is visible to whoever reads it next.

15. Mutation Testing

Twelve mutations. Clean code restored after each.

IDMutationResult
M1the wait has no boundKILLED — timeout scenario
M2skip the exchange when locally capableKILLED — incompatible scenario
M3one capable end is enoughKILLED — incompatible scenario
M4an error in CXL does not fall backKILLED — post-error check
M5the responder may answer in any phaseKILLED — early-reply check
M6enable the request, not the intersectionKILLED — partial-offer stimulus
M7union instead of intersectionKILLED — intersection check
M8advertisement accepted in any stateKILLED — accepted_outside_window_err
M9the modified-variant check droppedKILLED — accepted_unmodified_err
M10only the first retimer narrowsKILLED — path check
M11an absent retimer still narrowsKILLED — path check
M12timeouts counted as a fourth outcomeKILLED — conservation
Mutation run — final
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
12/12 killed, 0 escaped

Six of twelve escaped on the first run, and diagnosing them separated into three distinct causes — which is the most useful thing in this chapter for anyone who runs mutation testing.

EscapedCauseFix
M3, M4, M10, M11missing check — the trace was printed, never assertedassert the scenario outcome
M6missing stimulushost_offer = 0xF makes intersection equal the requesta partial offer
M5equivalent mutant — the guard was dead logicrestructure so the guard can be false

M6 is the subtle one. With host_offer = 0xF the intersection 0xF & dev_wants equals dev_wants, so "enable the intersection" and "enable what the device asked for" produce identical results. The testbench was correct, the assertion was correct, and the stimulus made two different designs indistinguishable. Adding one case with host_offer = 0x3 and dev_wants = 0x6 — expecting 0x2 — killed it immediately.

Choose stimulus values that make wrong designs produce wrong answers. An all-ones input is the identity for AND, and identities hide bugs.

M5 is the one that is not a testbench problem at all. The guard may_reply was evaluated inside the P_REPLY case branch, where it is true by construction — deleting it changed nothing because it did nothing. The fix belonged in the RTL, not the tests.

16. Debug Lab

1

Bring-up hangs and the link never comes up at all

UNBOUNDED-WAIT
Buggy Code
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Wait for the partner's advertisement.
S_EXCH : if (peer_advert_seen) nxt = S_DECIDE;
Symptom

A device works with every partner in the lab and hangs in one customer system. Not a slow link, not a degraded link — no link. The port never reaches a usable state and there is no error to report.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  with timeout   : decided=1 pcie_fallback=1
  without timeout: decided=0 still waiting, wait=111
Root Cause

There is no exit from EXCH other than hearing the partner. If the partner never advertises — because it is not CXL-capable and does not participate, or because it is broken, or because a retimer dropped the traffic — the machine waits forever.

The circularity in §1 makes this unavoidable in principle: there is no way to distinguish a slow partner from an absent one except by deciding that enough time has passed. A design without that decision has no way to conclude anything.

Fix
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
S_EXCH : if (peer_advert_seen) nxt = S_DECIDE;
         else if (timed_out)   nxt = S_PCIE;   // waiting becomes a decision
Lesson

Any state that waits on a remote party needs a bound, and the bound must lead somewhere legal. The timeout is not error handling — it resolves to the perfectly good PCIe outcome, which is the right answer for the most common cause. It also converts an unprovable liveness property into a provable safety one, which is why V11 and V12 in §14 are a pair.

2

A device enters CXL mode against a PCIe-only partner

EXCHANGE-SKIPPED
Buggy Code
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// We support CXL, so go straight there and save the round trip.
S_BASE : nxt = local_cxl ? S_CXL : S_EXCH;
Symptom

A CXL device in a PCIe-only slot brings the link up and immediately starts transmitting CXL traffic. The partner sees malformed PCIe, the link retrains, and the cycle repeats — presenting as a signal-integrity problem because retraining is what marginal channels do.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  incompatible scenario did not fall back (cxl=1 pcie=0)
Root Cause

An edge into S_CXL that does not pass through S_DECIDE. The optimisation reasoning is superficially sound — the round trip costs time, and the device does support CXL — but it answers the wrong question. Local capability is not the decision. Chapter 5.1's two-key rule is exactly this: capability is a property of the pair.

The failure looks like a physical problem because the symptom is retraining, which sends debugging to the channel rather than to the FSM.

Fix
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
S_BASE  : nxt = S_EXCH;                                    // no shortcut
S_DECIDE: nxt = (local_cxl && peer_cxl) ? S_CXL : S_PCIE;  // the only way in
Lesson

Audit the in-edges of your terminal states, not just the out-edges. A state machine is correct about how it leaves a state and quietly wrong about how it got there. Every arrow into an operating state should be traceable to the decision that authorised it — and an assertion of the form "state X is only entered from state Y" catches this class cheaply.

3

Both ends agree, and the link enables a protocol neither supports

UNION-NOT-INTERSECTION
Buggy Code
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Combine both ends' capabilities.
assign enabled = host_caps | dev_caps;
Symptom

Negotiation completes and reports success. The link then fails on the first transaction of a protocol class one end does not implement — and the failure is at the protocol layer, far from bring-up.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  host=0x9 dev=0x6 : correct=0x0 any=0 | union-bug=0xf any=1  <-- disjoint
  not_subset_host_err=1  not_subset_dev_err=1
Root Cause

"Combine" was implemented as union. The correct operation is intersection, and the difference is invisible whenever one side is a superset of the other — which is the common case in a lab, where the host typically supports everything.

The disjoint case makes it obvious: host 0x9 and device 0x6 share nothing, so the correct answer is that no CXL capability is enabled at all. The union enables all four.

Fix
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
assign enabled = host_caps & dev_caps;
// and check it from both sides, separately:
if ((enabled & ~host_caps) != 4'h0) not_subset_host_err <= 1'b1;
if ((enabled & ~dev_caps)  != 4'h0) not_subset_dev_err  <= 1'b1;
Lesson

Test agreement logic with disjoint and partial inputs, never with a superset on one side. This is the same trap as mutation M6: an all-ones operand is the identity for AND, so it makes intersection and "just take the other side" indistinguishable. The two subset assertions are stated separately because a union breaks both while a one-sided bug breaks exactly one, and that difference is the diagnosis.

4

Capability bits are read from traffic that was not carrying them

OUTSIDE-THE-WINDOW
Buggy Code
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// A modified ordered set arrived; take the advertisement.
assign gate = os_valid && os_modified;
Symptom

Negotiation results vary between otherwise identical bring-ups on the same hardware. Occasionally a capability is enabled that the partner does not have, and occasionally one that both have is missed.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  modified OS, OUTSIDE window: accepted=0 adv=0x5 | ignore-window accepted=1
  accepted_outside_window_err=1
Root Cause

The negotiation is carried during specific Configuration states. Outside that window the same ordered-set positions are not carrying negotiation information, so accepting them samples whatever is there.

Two conditions guard this action — the right ordered-set variant and the right point in bring-up — and only one was checked. Because the variant check passed, the read looked legitimate.

Fix
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
assign gate = os_valid && os_modified && in_window;
// two conditions, two diagnostics:
if (adv_accepted && !in_window)   accepted_outside_window_err <= 1'b1;
if (adv_accepted && !os_modified) accepted_unmodified_err     <= 1'b1;
Lesson

When two conditions guard one action, give them two error signals. A single combined diagnostic reports that something was accepted wrongly without saying which rule was broken, and the two have different fixes in different parts of the design. Mutations M8 and M9 exist as a pair for the same reason — each drops one condition, and each must be attributable.

5

A link claims capabilities the board cannot carry

RETIMER-NARROWING-IGNORED
Buggy Code
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// The far end told us what it supports.
assign path_caps = far_end_caps;
Symptom

Two systems with the same host and the same device negotiate different results, and the one with the longer trace run — two retimers rather than none — fails after negotiation reports success. Swapping devices does not help; swapping boards does.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  two retimers : received=0x3 correct path=0x3 | assume-unmodified=0xf  <-- overclaim
  far end offered 0xf, path carries 0x3
  path_overclaim_err=1
Root Cause

Retimers on the path may modify the relevant bits during negotiation — that is the mechanism working as designed, not corruption. A retimer that cannot support a capability removes it, because a path is only as capable as its narrowest element.

Reading far_end_caps as "the device's capability" reads a true statement that answers the wrong question. What matters at negotiation time is what the path can carry: the far end intersected with every retimer on it.

Fix
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
assign received_caps = far_end_caps & m1 & m2;   // what actually arrived
assign path_caps     = received_caps;            // claim only what the path carries
// and report the narrowing, which is information rather than a fault:
if ((far_end_caps & ~received_caps) != 4'h0) silently_narrowed_err <= 1'b1;
Lesson

Capability at negotiation time is a property of the path, not of the endpoints. Note also that the narrowing report is a diagnostic, not an error — it fires on the correct design and tells an operator something they need: the link is less capable than the device, and the cause is on the board. A convention that suffixes everything _err guarantees this gets triaged as a bug.

6

Fleet statistics say 30% PCIe fallback and the cause is unknowable

REASON-CONFLATED-WITH-OUTCOME
Buggy Code
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Count what happened.
if (got_pcie || got_timeout) n_pcie_q <= n_pcie_q + 16'd1;
Symptom

A fleet reports 30% of links falling back to PCIe. That number is consistent with a healthy deployment of mixed-capability partners and with a serious negotiation bug, and nothing in the data distinguishes them.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  attempts=40 cxl=8 pcie=29 timeout=12 error=3
  conservation attempts == cxl+pcie+error : 40 == 40
  timeouts are a REASON, not an outcome: 12 of 29 PCIe results
Root Cause

Timeout was modelled as a kind of outcome and merged into the PCIe count. It is not an outcome — it is a reason for one. A link that times out lands in PCIe exactly like a link whose partner cheerfully advertised no CXL support, and those two populations need very different responses.

Merging them also breaks the conservation law, because the same attempt gets counted twice in a sum that is supposed to partition.

Fix
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
if (got_pcie)    n_pcie_q    <= n_pcie_q    + 16'd1;   // the outcome
if (got_timeout) n_timeout_q <= n_timeout_q + 16'd1;   // the reason, alongside
// conservation partitions the OUTCOMES only:
if (n_attempt_q != n_cxl_q + n_pcie_q + n_error_q) accounting_err <= 1'b1;
Lesson

Outcomes partition; reasons annotate. Keep them in separate counters and let only the outcomes appear in the conservation sum. Of this run's 29 PCIe results, 12 were timeouts — a proportion that is a red flag on a fleet where most partners are known CXL-capable, and completely unremarkable on one where they are not. Neither reading is available from a merged count.

17. Verification Plan

ItemApproach and goal
Bring-up outcomesassert the terminal state per scenario — all four asserted
Terminal in-edgesCXL entered only from DECIDE — no shortcut reachable
Bounded waitsweep partner latency across the bound — exits every time
Phase orderingreply before, at and after the offer — only in-phase completes
Intersectioncross host x device caps incl. disjoint and partial
Window and variantcross window x modified — accepted in one of four cells
Retimer path0, 1, 2 retimers, distinct masks — received = full path AND
Outcome accountingmixed population, exclusive by construction — conserves
Diagnostic livenessbroken variants, latched _ever — every diagnostic fires

The fifth row carries this chapter's sharpest lesson: a superset operand is an identity, and identities make wrong designs look right. Any cross that only ever offers 0xF on one side has not tested the operation.

18. Design Review

  • Does every waiting state have a bound, and does the bound lead to a legal outcome rather than an error?
  • Can any operating state be entered without passing through the decision state?
  • Is the combining function an intersection, and is it checked from both sides separately?
  • Are the variant and the window checked as two conditions with two diagnostics?
  • Is capability computed over the path or read from the far end's claim?
  • Are reasons kept out of the outcome partition?
  • Does any assertion depend on a stimulus value that happens to be an identity for its operation?
  • Has every diagnostic been observed to fire, in a way that survives the test's own resets?

19. How This Appears in Real Engineering

Bring-up hangs are triaged as physical problems. Debug Lab 2's symptom is repeated retraining, which is exactly what a marginal channel looks like, so effort goes to the board while the fault is in the FSM. Asking "does the state sequence match what we expect" early is cheap and rarely done first.

Retimer count is a system-level variable that endpoint teams do not control. Lab 5's failure appears when a device moves from a short lab trace to a long production one. Any negotiation test plan that does not include retimer configurations is testing one point of a space the customer will sweep.

Timeouts are tuned late and badly. Too short and capable partners get dropped to PCIe under load; too long and a dead partner delays boot. Because the failure mode of a slightly-too-short timeout is a working PCIe link, it is easy to ship and hard to notice — which is Debug Lab 6's reporting problem again.

The lab always has a superset host. Validation hosts support everything, so intersection bugs are systematically invisible until a customer with a partial-capability host finds them. Deliberately configuring a reduced-capability host is one of the cheapest high-yield tests available.

20. Common Misconceptions

ClaimWhy it is wrong
"CXL has its own link training state machine"It does not. Public material describes CXL using the alternate protocol negotiation mechanism defined in the PCIe specifications, carried in modified TS ordered sets during existing PCIe Configuration states. The PCIe LTSSM is not replaced.
"Negotiation happens after the link is at full speed"It happens before entering L0 at Gen 1 speed, during Configuration states, with the link at 2.5 GT/s. The link reaches 8.0 GT/s or higher afterwards.
"The device tells the host what it supports"Two phases, and the host goes first: the root complex advertises, then the endpoint responds indicating which it wishes to enable. A responder that speaks first is answering an unasked question.
"What the device advertised is what the host receives"Retimers may modify the relevant bits. What arrives is the far end narrowed by every retimer on the path.
"A timeout is an error"A timeout is a reason, and it resolves to the entirely correct PCIe outcome. Treating it as an error inverts the common case, where the partner is simply not CXL-capable.
"Negotiation enables the union of both ends' capabilities"The intersection. A union enables protocols one end cannot speak, and the bug is invisible whenever one side is a superset.
"If the link is up, the negotiation succeeded"The link being up is the precondition for negotiation, not its result. A PCIe link is a fully successful bring-up outcome.

21. Interview Reasoning

22. Exercises

  1. Trace. Using §6's FSM, write the state sequence for: a capable partner that advertises at cycle 3; a capable partner that advertises at cycle 12 with TIMEOUT = 8; and a partner that advertises at cycle 3 with peer_cxl = 0. State which two produce the same terminal state for different reasons, and what report would distinguish them.

  2. Calculate. A host advertises 0xD and a device 0xB, with two retimers masking 0xE and 0x7. Compute what arrives, what should be enabled, and what a design that trusts the far end would claim. Then find a retimer pair that makes the correct and incorrect designs agree, and explain why that configuration is a bad test.

  3. DV task. Write the coverage cross for the intersection logic that would have caught mutation M6. Explain why a cross over host_caps and dev_caps with an all-ones bin is not sufficient, and state the bin you would add.

  4. Debug task. A port hangs during bring-up in one system. You can capture one signal group. Choose it, justify the choice, and give the decision tree from what you would see to the subsystem you would investigate.

  5. Design. Add a fourth outcome to §13's counters — "negotiated CXL but the path narrowed it below what the device offered". State whether it belongs in the conservation sum and why, and which existing diagnostic it makes redundant.

  6. Critique. Argue that the timeout in §6 should transition to ERROR rather than PCIE. Give the strongest case, then identify what it costs in the most common real-world scenario and which fleet statistic would reveal the mistake.

23. Summary

CXL link initialisation breaks a circular dependency — you must ask the partner what it supports, and asking requires a link — by bringing the link up as PCIe and holding the conversation in-band.

  • The mechanism is PCIe's own alternate protocol negotiation, carried in modified TS1/TS2 ordered sets during Configuration states at 2.5 GT/s. CXL does not define a separate training state machine and does not replace the LTSSM.
  • It is two-phase and ordered: the root complex advertises, the endpoint responds with what it wishes to enable. Neither step commutes.
  • The result is the intersection, and capability is a property of the path — the far end narrowed by every retimer, which may modify the relevant bits.
  • Waiting must be bounded. The timeout converts an unprovable liveness property into a provable safety one, and it resolves to PCIe rather than to an error, because a non-capable partner is the common case.
  • Three outcomes: CXL, PCIe, error. A timeout is a reason, not an outcome — outcomes partition, reasons annotate.
  • Verification lessons: a surviving mutation has three possible causes — missing check, missing stimulus, equivalent mutant — with three different fixes; identity-valued stimulus hides bugs; and a conservation law is strong enough to contradict the testbench that feeds it.

Chapter 5.4 takes up what happens next: the widths, rates and recovery behaviour that determine the operating point the link actually settles on.

Standards & specifications

Governing standard
CXL Specification (CXL Consortium)(opens CXL Consortium in a new tab)

Defines CXL.io, CXL.cache and CXL.mem, and the coherence and memory-pooling behaviour built on them. System design and deployment topology are not mandated.

This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.

Where this fits

Part of the CXL curriculum.