Physical Layer

Module 3 finished with a complete architectural picture: host, device, fabric, coherence, layers, and one request travelling through all of them. Module 4 goes down into the stack itself, one layer at a time.

This chapter is the bottom of it — and the bottom is where CXL borrows most and still has real work to do.

1. The Engineering Problem — One Wire, Several Conversations

Chapter 2.4 established that CXL inherits the PCIe physical layer essentially whole. That inheritance is the reason CXL was adoptable, and it creates a problem the inherited layer never had to solve.

A PCIe link carries one kind of traffic. A CXL link carries three protocol classes plus the management traffic needed to run a link shared between them — and they all arrive on the same lanes, interleaved, with no separate channel to distinguish them.

So the receiving side faces a question a PCIe receiver never asks: what kind of thing did I just receive? And the transmitting side faces its mirror: whose turn is it on the wire? Both questions have to be answered before any protocol logic runs, at full line rate, every cycle.

Get the first wrong and traffic lands in the wrong protocol stack. Get the second wrong and one class starves another, or the link cannot manage itself because management never gets a turn.

2. The One-Sentence Model

The CXL physical layer is a PCIe physical layer with a sorting office bolted on top: below the boundary, inherited signalling moves bits; above it, a logical layer tags every unit with what kind it is, decides which protocol stack gets the wire next, and coordinates any change of link state across every stack that shares it — because a shared wire needs a shared answer.

Call it the sorting office. Bits arrive; something has to read the label, route the item, and decide what goes out next.

3. What This Chapter Owns

Chapter 3.5 established that layers have contracts. This chapter is the bottom contract's actual content.

4. What CXL Inherits

Everything in that table is PCIe's. The lanes, the signalling, the equalisation, the training, the error-correction machinery at 64 GT/s — CXL did not build any of it, which is Chapter 2.4's reuse argument expressed as a bill of materials.

The consequence for a CXL engineer is worth stating bluntly: a signal-integrity problem on a CXL link is a PCIe problem. Reach, retimers, lane widths, marginal channels and temperature sensitivity all belong to the inherited layer, and no protocol-level change can address any of them.

5. What CXL Adds — the Logical PHY

Above the inherited electrical layer sits a logical layer that PCIe does not need in this form, because PCIe has nothing to disambiguate.

The Consortium's own description of the CXL stack is compact and precise: CXL is "dynamically multiplexed IO, cache and memory in flit format on PCIe PHY", with a "PCIe/CXL Logical PHY" above the PCIe PHY on both ends of the link.

Unpack "dynamically multiplexed" and three jobs fall out:

Label every unit. Each flit carries an identifier saying what kind it is, so the receiver can route it. Published material describes the physical layer disambiguating between CXL.io, CXL.cache/mem, ALMP and NULL flits.

Decide whose turn. Several stacks want the wire. Something must schedule them, and management traffic cannot be allowed to lose.

Agree on link state. If the link is going to change state — quiescing for power, for example — every stack sharing it must agree, because a stack that goes quiet while its partner still has traffic strands that traffic.

6. Quantitative Reasoning — Signalling Rate Is Not Payload Rate

A link advertised at 64 GT/s does not deliver 64 GT/s of protocol payload, and the gap is large enough to change design decisions.

Three costs, all structural:

Flit overhead. A 68-byte flit carries 64 bytes of payload, so the flit itself costs 64/68 of the wire — about 94.1% before anything else.

Encoding and framing overhead. Published analysis of CXL.cache+mem in 68-byte flit mode gives 128/130 for sync-header overhead (unless sync-header bypass is supported) and 374/375 for bandwidth lost to periodic ordered sets for common clock.

The product. Multiplying those:

0.9412  ×  0.9846  ×  0.9973  ≈  0.924

which matches the published figure of 0.924 link efficiency with sync header on, and 0.939 with it off. That published figure explicitly includes credits and link-reliability mechanisms such as Ack/Nak.

For the larger flits, published analysis gives 15/16 = 0.938 overall link efficiency for 256-byte and 128-byte Latency-Optimized flits for CXL.cache and CXL.mem — 15 usable slots with one slot equivalent spent on FEC, CRC and header.

Two things follow that matter more than the numbers.

Efficiency is similar across flit types, which is a deliberate design outcome rather than a coincidence — a larger flit spends proportionally similar overhead. So the reason to move to 256-byte flits is not efficiency; it is that the format enables capabilities, which is why the Consortium lists enhanced coherency, memory sharing and fabric capabilities as requiring the 256-byte flit.

And efficiency is not the same as utilisation. A link at 0.924 efficiency still delivers nothing on a cycle where no stack had anything to send. Section 11 measures both, and the distinction is the difference between "how good is this link" and "how well are we using it".

7. Teaching-model boundary

8. RTL 1 — Reading the Label

Purpose

Decide what kind of unit arrived, before anything else can act on it.

// The logical PHY's first job on receive: decide what KIND of flit this is.
//
// ARCHITECTURAL TEACHING MODEL. The encodings below are teaching values -- the
// CXL-defined protocol-ID encodings, widths and reserved values are NOT
// modelled and no specification encoding is claimed.
module protocol_id_demux #(
  parameter bit STRICT = 1'b1        // 0 = "unknown falls through as data"
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        flit_valid,
  input  logic [1:0]  pid,           // 0=io, 1=cache+mem, 2=ALMP, 3=NULL
  input  logic        pid_legal,     // the received ID decoded to a known kind
  output logic        to_io,
  output logic        to_cachemem,
  output logic        to_almp,
  output logic        is_null,
  output logic        pid_error,
  output logic [15:0] n_unknown_q,
  output logic        multi_route_err,
  output logic        unknown_routed_err
);
  logic known;
  // STRICT=1 requires the ID to have decoded to something known before the
  // flit is routed anywhere. STRICT=0 is the bug shape: route on validity
  // alone and let an unrecognised ID land in whichever stack it names.
  assign known = STRICT ? (flit_valid && pid_legal) : flit_valid;
 
  // Exactly one destination, and NULL is a destination -- a flit that carries
  // nothing still occupies a slot on the wire and must be accounted for.
  assign to_io      = known && (pid == 2'd0);
  assign to_cachemem= known && (pid == 2'd1);
  assign to_almp    = known && (pid == 2'd2);
  assign is_null    = known && (pid == 2'd3);
 
  assign pid_error  = flit_valid && !pid_legal && STRICT;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_unknown_q <= '0; multi_route_err <= 1'b0; unknown_routed_err <= 1'b0;
    end else begin
      if (flit_valid && !pid_legal) n_unknown_q <= n_unknown_q + 16'd1;
      if (({2'b0,to_io} + {2'b0,to_cachemem} + {2'b0,to_almp} + {2'b0,is_null}) > 3'd1)
        multi_route_err <= 1'b1;
      // An unknown flit that nevertheless reached a protocol stack.
      if (flit_valid && !pid_legal && (to_io || to_cachemem || to_almp))
        unknown_routed_err <= 1'b1;
    end
  end
endmodule

Position. Immediately above the inherited PHY, on receive, before any protocol stack sees anything.

State. Only counters. The decode itself is combinational, which it must be — this is per-flit at line rate and a pipeline stage here costs latency on every unit that crosses the link.

Synthesis. A two-bit decoder, a legality gate and a small adder for the one-hot check. The one-hot check is the only thing that costs anything and it is worth keeping in silicon rather than only in simulation, because a receiver that routes one flit to two stacks corrupts both.

NULL is a first-class output, not an absence. A flit that carries nothing still consumed wire time, and Section 11 shows why counting it as "nothing happened" makes a link look better than it is.

Simulation evidence

=== EXP1: the logical PHY disambiguating flit kinds ===
  protocol ID -> io          strict: io=1 cm=0 almp=0 null=0 err=0 | loose: routed=1
  protocol ID -> cache+mem   strict: io=0 cm=1 almp=0 null=0 err=0 | loose: routed=1
  protocol ID -> ALMP        strict: io=0 cm=0 almp=1 null=0 err=0 | loose: routed=1
  protocol ID -> NULL        strict: io=0 cm=0 almp=0 null=1 err=0 | loose: routed=0
  UNRECOGNISED protocol ID   strict: io=0 cm=0 almp=0 null=0 err=1 | loose: routed=1
 
  unknown flits seen: strict=1 loose=1
  loose demux unknown_routed_err=1  <-- it delivered it anyway

The first four rows are identical between the two designs, which is the recurring shape of this course: the permissive receiver is correct for every well-formed flit it will ever see in a healthy system. Only the fifth row diverges, and it diverges into a protocol stack that is about to interpret a unit nobody could identify.

Both designs counted the unknown flit. Counting it is not the same as refusing it — the loose design knows it received something unrecognisable and delivers it anyway, which is worth noticing because a counter can make a design look instrumented while the datapath ignores what the counter saw.

9. RTL 2 — Whose Turn on the Wire

Purpose

Several stacks want the same lanes, and one of them is the link's own management traffic.

// Two protocol stacks share one logical PHY. Management traffic must not be
// starved by data, and neither data stack may starve the other.
//
// ARCHITECTURAL TEACHING MODEL. This is NOT the CXL Arb/Mux arbitration
// mechanism; no CXL arbitration rule, weight or ordering requirement is
// modelled or claimed.
module stack_arbiter (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       mode_rr,
  input  logic       req_almp,       // link management -- always highest
  input  logic       req_io,
  input  logic       req_cachemem,
  input  logic       phy_ready,
  output logic       gnt_almp,
  output logic       gnt_io,
  output logic       gnt_cachemem,
  output logic [7:0] wait_io_q,
  output logic [7:0] wait_cm_q,
  output logic [7:0] max_wait_q,
  output logic       multi_grant_err
);
  logic data_ptr_q;                  // 0 = io next, 1 = cache+mem next
 
  always_comb begin
    gnt_almp = 1'b0; gnt_io = 1'b0; gnt_cachemem = 1'b0;
    if (phy_ready) begin
      if (req_almp) begin
        // Management always wins. It is rare, and a link that cannot manage
        // itself cannot recover.
        gnt_almp = 1'b1;
      end else if (!mode_rr) begin
        if      (req_io)       gnt_io       = 1'b1;
        else if (req_cachemem) gnt_cachemem = 1'b1;
      end else begin
        // Rotate between the two DATA stacks only.
        if (data_ptr_q == 1'b0) begin
          if      (want_io) gnt_io       = 1'b1;
          else if (want_cm) gnt_cachemem = 1'b1;
        end else begin
          if      (want_cm) gnt_cachemem = 1'b1;
          else if (want_io) gnt_io       = 1'b1;
        end
      end
    end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    // ... one-hot check, pointer rotation, per-stack wait counters ...
    // Track the worst wait across BOTH data stacks, not just one.
    if (req_io && !gnt_io && (wait_io_q + 8'd1 > max_wait_q))
      max_wait_q <= wait_io_q + 8'd1;
    if (req_cachemem && !gnt_cachemem && (wait_cm_q + 8'd1 > max_wait_q))
      max_wait_q <= wait_cm_q + 8'd1;
  end
endmodule

Management is absolutely prioritised and that is deliberate. It is not a fairness violation — it is a recognition that management traffic is rare, small, and the thing that lets a link recover. A link whose management cannot get a turn cannot change power state, cannot renegotiate, and cannot report that it is in trouble. Strict priority is the right answer for a class that is bounded in volume, which is precisely the condition Chapter 2.4 showed does not hold between data classes.

phy_ready gates every grant. Backpressure from the inherited layer reaches every stack simultaneously, which is the coupling a shared wire creates and cannot avoid.

Timing. Grant is combinational from request and phy_ready. This is the per-flit scheduling decision at line rate, so it is a genuine critical path — a real implementation would pipeline it and accept a cycle of latency, which does not change the argument.

Simulation evidence — 200 saturated cycles

Both modes on identical stimulus, with both data stacks requesting continuously and one management request injected mid-run:

=== EXP2: link management must not be starved by data ===
  fixed priority : almp=1 io=200 cache+mem=0   worst data wait=201
  RR between data: almp=1 io=100 cache+mem=100 worst data wait=2
 
  -> fixed priority STARVED cache+mem for 200 cycles
  -> RR served both data stacks, and ALMP still went first

Management got its single turn in both designs. The absolute priority works, and it costs one cycle out of 200.

Under fixed priority, cache+mem received zero. Not less — none, for the entire run, with its wait counter reaching 201. Round robin between the two data stacks gave each exactly 100 and bounded the worst wait at 2 cycles, which is the design's stated service guarantee.

The fairness-checker trap, measured

Chapter 1.2's arbiter carried a fairness checker that watched only requester 0. This run shows exactly what that checker would have reported:

=== EXP2b: a fairness checker that only watches io ===
  under the SAME saturated run above:
    io-only checker sees   : io wait = 0   -> reports FAIR
    both-stack checker sees: worst wait = 201 -> reports STARVATION
 
  -> a fairness property instantiated on the favoured requester cannot fail

Zero versus 201, on the same design, in the same run. A fairness property instantiated on the winning requester is not a weak check — it is a tautology, and it will pass forever on a design that serves nobody else. Section 13's a_no_starvation is therefore written for every stack, and Section 14's mutation suite confirms the checker actually fires.

10. RTL 3 — Agreeing to Change Link State

Purpose

A shared wire needs a shared answer about what state it is in.

// Coordinating a link-state change across the stacks that share one link.
//
// When several protocol stacks share a physical link, none of them may change
// the link's state alone -- a stack that entered a low-power state while its
// partner still had traffic would strand that traffic. So the two ends
// exchange management packets and the change happens only on agreement.
//
// ARCHITECTURAL TEACHING MODEL of that coordination. CXL's Arb/Mux exchanges
// ALMPs (Arb/Mux Link Management Packets) with its link partner for exactly
// this purpose; the state names, encodings, timeout and sequence below are
// TEACHING VALUES and are not the CXL-defined ALMP mechanism.
module almp_handshake #(
  parameter int unsigned NSTACK    = 2,
  parameter int unsigned TIMEOUT   = 12,
  parameter bit          UNANIMOUS = 1'b1   // 0 = act on the first stack only
) (
  input  logic              clk,
  input  logic              rst_n,
  input  logic [NSTACK-1:0] stack_wants_low, // each stack's own vote
  input  logic              peer_ack,
  output logic              send_request,
  output logic              entered_low,
  output logic [1:0]        state_q,         // 0 ACTIVE 1 REQ 2 LOW
  output logic [7:0]        age_q,
  output logic              timeout_err,
  output logic              stranded_err     // went low with a stack still busy
);
  localparam logic [1:0] ACTIVE = 2'd0, REQ = 2'd1, LOW = 2'd2;
 
  assign all_vote = (stack_wants_low == {NSTACK{1'b1}});
  assign any_vote = (stack_wants_low != {NSTACK{1'b0}});
  // The whole point: the link may only go quiet when EVERY sharer agrees.
  assign gate     = UNANIMOUS ? all_vote : any_vote;
 
  always_ff @(posedge clk or negedge rst_n) begin
    // ACTIVE: on gate, request
    // REQ:    a withdrawn vote cancels; peer_ack commits; timeout falls back
    //         to ACTIVE -- never forward to LOW
    // LOW:    return to ACTIVE when no stack wants to stay
    //
    // The invariant is about the TRANSITION into the quiet state, not about
    // being in it -- a stack withdrawing its vote while LOW is the normal
    // wake path, and checking the state rather than the edge flags it.
    if ((state_q == REQ) && gate && peer_ack && !all_vote) stranded_err <= 1'b1;
  end
endmodule

The timeout falls back to ACTIVE, never forward to LOW, and that direction is the entire safety argument. A peer that never answers might not have heard the request — so committing to the quiet state would be a unilateral decision about a shared resource. Falling back costs power and preserves correctness.

A withdrawn vote cancels an in-flight request. Between deciding to quiesce and the peer agreeing, a stack can receive work. Without the cancel path, that work waits for a link that is on its way to sleep.

Simulation evidence

=== EXP5: coordinating a link-state change across stacks ===
  one stack votes : unanimous state=0 low=0 | first-stack state=1 low=0
  peer acks       : unanimous low=0 | first-stack low=1  <-- one stack still busy
  first-stack stranded_err=1
  both stacks vote: unanimous low=1 state=2

One stack's vote was enough for the permissive design and not for the correct one. The permissive variant quiesced the link while a second stack still had traffic, and stranded_err records it. The unanimous design waited, and went quiet only when both agreed.

=== EXP6: the peer never answers ===
  after 16 cycles with no ack: state=1 timeout_err=1 entered_low=0
  -> it fell back to ACTIVE. Falling back to LOW would strand both stacks.

The timeout fired and the link stayed up. entered_low=0 is the important zero.

11. RTL 4 — Where the Wire Time Went

Purpose

Turn "the link is 64 GT/s" into "the link delivered this much payload".

// Where wire time actually goes.
//
// ARCHITECTURAL TEACHING MODEL: the per-flit byte split is parameterised so
// different flit shapes can be compared. Byte counts must be supplied by the
// instantiator; nothing here asserts a CXL-defined flit layout.
module link_efficiency #(
  parameter int unsigned FLIT_BYTES    = 68,
  parameter int unsigned PAYLOAD_BYTES = 64,
  parameter int unsigned SYNC_NUM      = 128,
  parameter int unsigned SYNC_DEN      = 130,
  parameter bit          SYNC_ON       = 1'b1
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        flit_on_wire,      // a flit occupied the wire
  input  logic        flit_carried_data, // ... and it was not NULL/idle
  input  logic        skp_cycle,         // periodic maintenance took the wire
  output logic [31:0] wire_bytes_q,
  output logic [31:0] payload_bytes_q,
  output logic [31:0] framing_bytes_q,
  output logic [15:0] n_data_q,
  output logic [15:0] n_null_q,
  output logic [15:0] n_skp_q,
  output logic        overcount_err
);
  localparam int unsigned OVERHEAD = FLIT_BYTES - PAYLOAD_BYTES;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      // BOTH contributions accumulate in ONE assignment. Two non-blocking
      // writes to the same register in a cycle lose one of them, and a wire
      // counter that under-reports makes efficiency look better than it is.
      wire_bytes_q <= wire_bytes_q
                    + (flit_on_wire ? FLIT_BYTES : 0)
                    + (skp_cycle    ? FLIT_BYTES : 0);
 
      if (flit_on_wire) begin
        framing_bytes_q <= framing_bytes_q + OVERHEAD;
        // A NULL flit costs full wire time and delivers nothing. Counting it
        // as payload is how a link reports throughput it never achieved.
        if (flit_carried_data) begin
          payload_bytes_q <= payload_bytes_q + PAYLOAD_BYTES;
          n_data_q        <= n_data_q + 16'd1;
        end else n_null_q <= n_null_q + 16'd1;
      end
      if (skp_cycle) n_skp_q <= n_skp_q + 16'd1;
 
      if (payload_bytes_q > wire_bytes_q) overcount_err <= 1'b1;
    end
  end
endmodule

The single-assignment accumulation is not style. Two non-blocking writes to wire_bytes_q in one cycle lose one contribution, and the direction of the error matters — under-reporting the wire makes efficiency look better. This is the same hazard Chapter 2.5 met in a latency accumulator, and it survived into this model until a mutation exposed it.

Three counters exist so two conservation laws can be checked, which Section 14 shows is the difference between a checker and a decoration.

Simulation evidence

100 flits of which 10 are NULL, plus one maintenance interval, into a 68-byte configuration and a 256-byte one:

=== EXP3: link efficiency -- wire time vs payload ===
  68B flit  : wire=6868 payload=5760 framing=400 null=10 skp=1
              payload/wire = 83.86%
  256B flit : wire=25856 payload=21600 framing=1600
              payload/wire = 83.53%

Note what this measures and what it does not. The framing cost alone is 64/68 = 94.1% for the 68-byte flit; the measured 83.86% is lower because 10% of the flits carried nothing and one interval went to maintenance. That gap — 94.1% structural versus 83.9% achieved — is the difference between link efficiency and link utilisation, and conflating them is how a design gets blamed for overhead that was actually idleness.

The two flit shapes land within a third of a point of each other, which is the published observation that bandwidth efficiency tends to be similar across flit types reproduced in a model built from the same parameters.

12. RTL 5 — A Link That Came Up Smaller Than Designed

Purpose

"Link up" is not the same as "link up as configured", and the difference is silent.

// A link that trains to fewer lanes or a lower rate still works -- at less
// bandwidth. The system must notice, because "working" and "working as
// designed" are different conditions.
//
// ARCHITECTURAL TEACHING MODEL. Lane widths and rate codes here are teaching
// values; no PCIe or CXL training, negotiation or degradation rule is modelled.
module width_degrade #(
  parameter int unsigned NOM_LANES = 16,
  parameter int unsigned NOM_RATE  = 64      // GT/s, teaching value
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        link_up,
  input  logic [5:0]  trained_lanes,
  input  logic [7:0]  trained_rate,
  output logic        degraded_width,
  output logic        degraded_rate,
  output logic [15:0] rel_bandwidth_pct,
  output logic [15:0] degraded_cycles_q,
  output logic        silent_degrade_err
);
  assign degraded_width = link_up && (trained_lanes < NOM_LANES[5:0]);
  assign degraded_rate  = link_up && (trained_rate  < NOM_RATE[7:0]);
 
  // Bandwidth scales with BOTH, so the losses multiply rather than add.
  assign w_pct = link_up ? (16'(trained_lanes) * 16'd100) / NOM_LANES[15:0] : 16'd0;
  assign r_pct = link_up ? (16'(trained_rate)  * 16'd100) / NOM_RATE[15:0]  : 16'd0;
  assign rel_bandwidth_pct = (w_pct * r_pct) / 16'd100;
  // ... degraded-cycle accumulation, and a check that the condition is real ...
endmodule

Simulation evidence

=== EXP4: a link that trained below its nominal configuration ===
  x16 @ 64 : width_deg=0 rate_deg=0 relative bandwidth=100%
  x8  @ 64 : width_deg=1 rate_deg=0 relative bandwidth=50%
  x8  @ 32 : width_deg=1 rate_deg=1 relative bandwidth=25%  <-- losses multiply

Half the lanes at half the rate is a quarter of the bandwidth, not three quarters. Both degradations are individually survivable and their combination is not, which is why a design that reports each independently — "width nominal? no. rate nominal? no." — understates the problem by a factor of three.

This connects directly to Chapter 3.6's triage rule: when the symptom is throughput, the first question is what the link actually trained to, and the answer must be a single relative-bandwidth number rather than two independent flags.

13. Waveform — Management Preempting Saturated Data

Read four things off this trace.

Cycles 0–3: strict alternation. gnt_io and gnt_cm swap every cycle with both requesting. That is the round-robin pointer working, and it is what makes the worst wait 2 rather than 201.

Cycle 4: one management flit displaces everything. Both data stacks are still requesting and neither is granted. The cost is exactly one cycle, and the benefit is that the link can always manage itself.

Cycles 8–9: phy_ready low, no grant at all. Backpressure from the inherited layer reaches every stack at once — there is no per-stack flow control at this level because there is only one wire. This is the coupling a shared physical layer creates and is the reason per-class isolation has to be built above it, which is Chapter 2.4's per-class-budget argument.

Cycle 10: alternation resumes where it left off. The pointer survived the stall, so the stack that was next before the stall is next after it. A pointer reset on stall would systematically favour whichever stack happened to be first — an unfairness that appears only on links that stall, which is all of them under load.

14. Mutation Testing — Which Checkers Actually Work

A checker that has never caught a fault may be decorative. Six deliberate mutations were injected into the exact tutorial RTL, compiled and simulated; the good RTL was then restored and re-verified.

15. Assertions

Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog. The properties below were not executed; each is mapped to the procedural check that stood in for it, and to the mutation that confirms the check fires.

// SAFETY -------------------------------------------------------------------
// P1 — a received flit is routed to at most one destination.
a_route_onehot: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({to_io, to_cachemem, to_almp, is_null}));
 
// P2 — an unrecognised flit never reaches a protocol stack.
a_no_unknown_route: assert property (@(posedge clk) disable iff (!rst_n)
  (flit_valid && !pid_legal) |-> !(to_io || to_cachemem || to_almp));
 
// P3 — one stack is granted the wire at a time, and never without the PHY.
a_grant_legal: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({gnt_almp, gnt_io, gnt_cachemem}) &&
  ((gnt_almp || gnt_io || gnt_cachemem) |-> phy_ready));
 
// P4 — the link never quiesces unless every stack agreed.
a_no_strand: assert property (@(posedge clk) disable iff (!rst_n)
  $rose(entered_low) |-> $past(stack_wants_low) == '1);
 
// P5 — CONSERVATION: payload credited equals data flits times payload size.
// This is the property counting alone cannot express, and M2 proves it.
a_payload_conserved: assert property (@(posedge clk) disable iff (!rst_n)
  payload_bytes_q == n_data_q * PAYLOAD_BYTES);
 
// P6 — CONSERVATION: every flit that occupied the wire is accounted for.
a_wire_conserved: assert property (@(posedge clk) disable iff (!rst_n)
  wire_bytes_q == (n_data_q + n_null_q + n_skp_q) * FLIT_BYTES);
 
// LIVENESS -----------------------------------------------------------------
// P7 — EVERY data stack is eventually served. Note "every": a property
// instantiated only on the favoured stack passes on a design that starves
// the other one, measured at 0 versus 201 in Section 9.
// ASSUMPTION: phy_ready is not permanently low, and management traffic is
// bounded. Both are constraints on the environment, not on this module.
a_no_starvation: assert property (@(posedge clk) disable iff (!rst_n)
  req_cachemem |-> ##[1:BOUND] gnt_cachemem);
a_no_starvation_io: assert property (@(posedge clk) disable iff (!rst_n)
  req_io |-> ##[1:BOUND] gnt_io);
 
// P8 — a link-state request always resolves; the timeout falls back to
// ACTIVE, never forward to LOW.
a_request_resolves: assert property (@(posedge clk) disable iff (!rst_n)
  (state_q == REQ) |-> ##[1:TIMEOUT+1] (state_q != REQ));
 
// GOAL / PERFORMANCE -------------------------------------------------------
// P9 — the design's stated service guarantee: with both data stacks asking
// and the PHY ready, neither waits longer than one turn.
a_bounded_data_wait: assert property (@(posedge clk) disable iff (!rst_n)
  (mode_rr && phy_ready && !req_almp) |-> (max_wait_q <= 2));
 
// P10 — a degraded link is always reportable as one number.
a_degrade_visible: assert property (@(posedge clk) disable iff (!rst_n)
  (degraded_width || degraded_rate) |-> (rel_bandwidth_pct < 100));

P7 and P9 are different claims and both are needed. P7 says a stack is eventually served — a liveness property that fixed priority violates outright. P9 says it is served within two cycles — a goal property that a technically-fair-but-slow arbiter could violate while satisfying P7. Neither is a safety property, and every safety property in this table passes on the design that gave one stack zero turns in 200 cycles.

16. Verification Plan

Reference model. A flit-level model that, for each cycle, records the kind of flit on the wire and which stack owned it. Two independent totals fall out: per-stack service counts, and the wire/payload conservation pair. The scoreboard compares both against the DUT's counters — which is what turns Section 14's mutations from invisible into immediate.

Directed tests. Every flit kind including illegal IDs; both arbiter modes under saturation; a management request during saturation; a single-stack vote; a withheld peer acknowledgement; every width/rate combination at the nominal boundary and one step below.

Constrained-random dimensions. Flit kind distribution, NULL density, maintenance interval spacing, phy_ready duty cycle, request density per stack, and vote patterns across stacks.

Functional coverage worth writing:

The cross worth taking is grant winner × phy_ready stall length, because that is where the pointer-preservation behaviour in Section 13 lives. Crossing flit kind with trained configuration would be a combinatorial explosion with no defect behind it.

Error injection. Illegal protocol IDs, a peer that never acknowledges, a peer that acknowledges twice, phy_ready held low indefinitely, and a vote withdrawn in the same cycle the acknowledgement arrives.

17. Debug Lab

// A flit arrived; route it to the stack its ID names.
assign known       = flit_valid;
assign to_cachemem = known && (pid == 2'd1);

  protocol ID -> io          strict: io=1 ... | loose: routed=1
  protocol ID -> cache+mem   strict: cm=1 ... | loose: routed=1
  protocol ID -> ALMP        strict: almp=1  | loose: routed=1
  UNRECOGNISED protocol ID   strict: err=1   | loose: routed=1

assign known     = flit_valid && pid_legal;
assign pid_error = flit_valid && !pid_legal;

// Serve in priority order. Simple and obviously correct.
if      (req_almp)     gnt_almp     = 1'b1;
else if (req_io)       gnt_io       = 1'b1;
else if (req_cachemem) gnt_cachemem = 1'b1;

  fixed priority : almp=1 io=200 cache+mem=0   worst data wait=201
  RR between data: almp=1 io=100 cache+mem=100 worst data wait=2

if (req_almp) gnt_almp = 1'b1;            // bounded class, absolute priority
else if (data_ptr_q == 1'b0) begin        // unbounded classes, rotated
  if      (want_io) gnt_io       = 1'b1;
  else if (want_cm) gnt_cachemem = 1'b1;
end else begin
  if      (want_cm) gnt_cachemem = 1'b1;
  else if (want_io) gnt_io       = 1'b1;
end

// Fairness check: the arbiter must not make anyone wait too long.
a_bounded_wait: assert property (req_io |-> ##[1:BOUND] gnt_io);

    io-only checker sees   : io wait = 0     -> reports FAIR
    both-stack checker sees: worst wait = 201 -> reports STARVATION

a_no_starvation_io: assert property (req_io       |-> ##[1:BOUND] gnt_io);
a_no_starvation_cm: assert property (req_cachemem |-> ##[1:BOUND] gnt_cachemem);
// and in RTL, a max that watches BOTH:
if (req_cachemem && !gnt_cachemem && (wait_cm_q + 1 > max_wait_q))
  max_wait_q <= wait_cm_q + 1;

// This stack is idle, so the link can go quiet.
assign gate = (stack_wants_low != '0);      // any vote, not all

  one stack votes : unanimous state=0 low=0 | first-stack state=1 low=0
  peer acks       : unanimous low=0 | first-stack low=1  <-- one stack still busy
  first-stack stranded_err=1

assign gate = (stack_wants_low == '1);      // every sharer agrees
...
REQ: if (!gate) state_q <= ACTIVE;          // a withdrawn vote cancels

// Count payload.
payload_bytes_q <= payload_bytes_q + PAYLOAD_BYTES;   // every flit, including NULL
...
// Count wire time.
if (flit_on_wire) wire_bytes_q <= wire_bytes_q + FLIT_BYTES;
if (skp_cycle)    wire_bytes_q <= wire_bytes_q + FLIT_BYTES;   // second write loses

  68B flit  : wire=6868 payload=5760 framing=400 null=10 skp=1
              payload/wire = 83.86%

wire_bytes_q <= wire_bytes_q + (flit_on_wire ? FLIT_BYTES : 0)
                             + (skp_cycle    ? FLIT_BYTES : 0);
// and the two relations that make the counters checkable:
assert (payload_bytes_q == n_data_q * PAYLOAD_BYTES);
assert (wire_bytes_q    == (n_data_q + n_null_q + n_skp_q) * FLIT_BYTES);

18. Design Review

On the receive path. Is the one-hot routing check in silicon or only in simulation? Is NULL an explicit output, or is a non-data flit simply "not routed"? What happens to a flit whose ID does not decode — refused with a cause, or delivered to the stack it names? Is there a counter for it, and does the datapath act on what the counter saw?

On arbitration. Is management absolutely prioritised, and is its volume genuinely bounded — because absolute priority is only safe for a bounded class? Is the fairness property instantiated for every data stack? What is the stated worst-case wait, and is it a number? Does the round-robin pointer survive a phy_ready stall, or does it reset and quietly favour one stack?

On link state. Can any single stack quiesce the link? Does a withdrawn vote cancel an in-flight request? Does the timeout fall back to ACTIVE or forward to LOW — and is that direction written down as a decision rather than an accident?

On instrumentation. Are there conservation relations between the counters, or only the counters? Is NULL wire time counted as wire time? Is a degraded link reported as one relative-bandwidth number, or as two independent flags that understate the loss by their product?

And the structural question this chapter adds. Which of this design's behaviour exists only because the wire is shared? Everything in that set — arbitration, labelling, unanimous state change — is work PCIe never had to do, and it is where CXL-specific PHY-adjacent bugs live.

19. How This Appears in Real Engineering

CXL / protocol architect

The reuse boundary decides what you can and cannot fix. Rate, reach, retimers and training are inherited and unchangeable from above; multiplexing policy, management priority and link-state coordination are yours. The design question that recurs is which class deserves absolute priority, and the answer depends entirely on whether its volume is bounded.

RTL engineer

Five disciplines from the measured runs: make routing one-hot and check it in silicon; refuse an unrecognised ID rather than delivering it; give data stacks a rotating pointer that survives stalls; never let one stack quiesce a shared link; and accumulate shared counters in a single assignment.

DV engineer

The finding to carry is Section 14's: three of six mutations escaped a testbench full of counters and were caught immediately by two conservation relations. Write the relations, not just the counters. And instantiate every fairness property for every requester — measured, the io-only checker reported 0 where the real worst wait was 201.

Performance engineer

Separate efficiency from utilisation. Structural framing cost is 64/68 = 94.1% for the 68-byte flit and published overall link efficiency is 0.924 with sync header on; the measured 83.9% in Section 11 is lower because 10% of flits were NULL. Those are different problems with different owners — one is the format, the other is the traffic source.

Firmware and system software

Read the achieved link configuration, not the nominal one, and compute the relative bandwidth as a product. A platform reporting "link up, width degraded, rate degraded" as two booleans has told you a quarter-bandwidth link looks like two minor warnings.

Silicon debug

Three counters are worth their area at this layer: unknown-protocol-ID count, per-stack grant count, and degraded-configuration cycles. The first says the receiver saw something it could not identify; the second localises starvation without a trace; the third distinguishes a slow link from a small one.

20. Common Misconceptions

21. Interview Reasoning

22. Exercises

1. Explain. Management traffic gets absolute priority here and data traffic does not. State the property of management traffic that makes absolute priority safe, then name a hypothetical class that looks important but must not get it, and say why.

2. Calculate. Derive the 0.924 figure from its three components — 64/68, 128/130, 374/375. Then recompute assuming sync-header bypass is supported and confirm you land near the published 0.939. Which of the three terms dominates, and what does that imply about where to look for efficiency improvements?

3. Calculate. A link nominally x16 at 64 GT/s trains to x4 at 32 GT/s. What is the relative bandwidth? Now suppose a platform reports only two booleans, width_degraded and rate_degraded. Explain, with the number, why that report understates the problem.

4. RTL modification. The round-robin pointer in stack_arbiter survives a phy_ready stall. Change it to reset on stall, then construct the stimulus that exposes the resulting unfairness. Which of Section 15's properties fails, and does the io-only fairness checker notice?

5. DV task. Write the SVA for the conservation property that caught mutation M6, and then construct a different mutation that satisfies it while still mis-reporting efficiency. What third relation would you need?

6. Debug task. A link reports 97% payload efficiency, above the published structural maximum for its flit mode. Name the two counter defects that produce a figure above the structural ceiling, and say which single check distinguishes them.

23. Summary

The CXL physical layer is PCIe's, and the logical layer above it is CXL's — the sorting office that a shared wire requires.

What is inherited is inherited whole. The Consortium describes CXL 3.0 using the PCIe 6.0 PHY at 64 GT/s with PAM-4 and PCIe 6.0 FEC and CRC. A signal-integrity problem on a CXL link is a PCIe problem with PCIe answers, and no protocol change addresses it.

What is added is what sharing costs. Every unit must be labelled so the receiver can route it — measured, a receiver that routed on validity alone delivered an unrecognisable flit into a protocol stack while behaving identically on all four well-formed kinds. Stacks must be scheduled — measured, strict priority between data stacks gave one of them zero turns in 200 saturated cycles while round robin gave each 100 with a worst wait of 2. And link state must be unanimous — measured, a design acting on one stack's vote quiesced the link with another stack still busy.

Signalling rate is not payload rate. The 68-byte flit spends 64/68 on payload; published overall efficiency for CXL.cache+mem in that mode is 0.924 with sync header on and 0.939 with it off, and 256-byte and 128-byte Latency-Optimized flits land at 15/16 = 0.938. Efficiency is similar across flit types, which is why the 256-byte flit exists for capability rather than for bandwidth. And the Consortium published the price of the Latency-Optimized flit exactly: 2–5 ns for FIT from 5×10⁻⁸ to 0.026 and efficiency from 0.94 to 0.92.

Two methodological results carry forward. A fairness checker watching one requester reported a wait of 0 where the true worst was 201 — the same defect shape Chapter 1.2's arbiter carried, measured here directly. And three of six mutations escaped a testbench full of counters until two conservation relations were added: counters record what happened, checkers assert what must hold between things that happened, and only the second kind can fail.

24. What Comes Next

This chapter labelled the units and decided who transmits. It said nothing about whether what was transmitted arrived intact — the 2-byte CRC in the 68-byte flit was named and then set aside.

Chapter 4.2 is that: framing, integrity, and what a link does when a transfer fails. Chapter 3.5 established that retry belongs to the link layer and that its consequences must not escape it; 4.2 builds the mechanism and measures what it costs.

For adjacent material: Relationship to PCIe has the reuse argument this chapter's inheritance implements, CXL Layered Architecture has the contracts between layers, and Evolution of CXL has the rate and flit progression by revision. The path is on the CXL tutorials index.