Data Link Layer

Chapter 4.1 got units onto the wire and labelled them. It named the 2-byte CRC in the 68-byte flit and then deliberately set it aside.

This chapter picks it up. The question here is narrow and consequential: the channel corrupts things — what does the link do about it, and what does that cost?

1. The Engineering Problem — The Wire Is Not Reliable

At 64 GT/s with PAM-4 signalling, bit errors are not exceptional events to be designed around. They are a designed-for rate. The CXL Consortium says so directly, describing CXL 3.0 as using the PCIe 6.0 PHY at 64 GT/s with "PAM-4 and high BER mitigated by PCIe 6.0 FEC and CRC".

Read that phrasing carefully. The bit error rate is high — high enough to need mitigating — and two different mechanisms mitigate it. That is the whole chapter in one sentence, because the two mechanisms do different jobs at different costs, and every interesting decision at this layer follows from telling them apart.

Above this layer, Chapter 3.4's coherence protocol assumes messages arrive. Chapter 3.1's home agent waits for acknowledgements that it assumes will come. Neither has any notion of a corrupted flit, and neither should. Something has to turn a channel with a high error rate into a delivery service those layers can build on — and it has to do it without telling them, because the moment retry becomes visible upstream, Chapter 3.5's layering falls apart.

2. The One-Sentence Model

The data link layer is an insurance policy with a premium: it corrects what it can in place, detects what it cannot and asks for it again, holds every transmitted unit until the far side confirms receipt — which is what bounds how far ahead a sender may run — and pays for all of it in wire time that carries no new payload.

Call it the insurance model. Correction is the cheap policy that covers small claims silently. Detection plus replay is the expensive policy that covers everything else. The replay buffer is the reserve you must hold to be able to pay a claim at all, and credits are the promise from the other side that they can accept the payout.

3. What This Chapter Owns

Chapter 3.5 established that retry belongs here and that its duplicates must not escape. This chapter builds the mechanism and measures the premium.

4. Two Mechanisms, Two Jobs

The distinction that organises everything below.

Correction is paid on every unit whether or not it is needed. Detection is nearly free until it fires, and then it costs an entire round trip. That asymmetry is why both exist: a channel with a high error rate would spend all its time retrying if detection were the only mechanism, and a code strong enough to correct every possible error would spend an unacceptable fraction of every unit on redundancy.

5. Why the Order Matters

Correction runs first. Detection runs on the result.

That ordering is not a convention — it is the only order that makes sense, and reversing it produces a specific, measurable waste:

correct first:   error arrives → correction fixes it → check passes → deliver
                 cost: correction decode. No round trip.
 
check first:     error arrives → check fails → request replay → ...
                 cost: a full round trip, for an error that was fixable.

Section 11 measures both. The wrong order turns a correctable error — the common case on a high-BER channel — into a retransmission.

The deeper point is Chapter 3.5's responsibility-placement rule applied within a layer: handle a problem at the lowest level that can actually solve it. Escalating from correction to replay is the same mistake as escalating from replay to the transaction layer, one step down.

6. Quantitative Reasoning — What Recovery Costs

Three calculations that a link-layer design cannot avoid.

Retry costs wire time twice

A replayed unit occupies the link exactly as much as an original. So if a fraction r of units are replayed, the effective delivery rate is:

effective_rate  =  nominal_rate  /  (1 + r)

At a 5% replay rate the link delivers about 95.2% of what it otherwise would — and it does so while every counter that measures transmissions looks perfectly healthy. Section 12 measures the distinction, and Chapter 3.5 §13 showed the consequence upstream: a boundary sized against nominal drain rate will backpressure on a link that is merely working harder.

Replay-buffer depth bounds how far you may run ahead

A sender may not transmit a unit it could not replay. So the buffer depth directly caps in-flight units, and by Little's Law:

required_depth  ≈  send_rate  ×  round_trip_to_acknowledgement

A link sending one unit per cycle with a 40-cycle acknowledgement round trip needs 40 entries to avoid stalling. With 8, the achievable rate is capped at 8/40 = 0.2 units per cycle — a 5× shortfall that no amount of signalling rate fixes, because the limit is the sender's own ability to recover.

This is structurally the same calculation as Chapter 3.2's outstanding-table sizing, and it recurs for the same reason: any resource held across a round trip is bounded by rate times latency.

Credits are the same arithmetic from the receiver's side

Credits let a sender commit before checking, which is what makes a long round trip workable:

max_in_flight  =  credits  ×  bytes_per_credit

A ready/valid handshake tells you about this cycle after you have offered. Credits tell you about the future before you commit. On a link whose round trip is long, only the second is usable — which is why credits exist at this layer and not inside a chip.

7. Teaching-model boundary

8. RTL 1 — Detect, and Decide

Purpose

The decision this layer exists to make: deliver, or ask again.

// Integrity checking, and the distinction that decides what happens next.
//
// ARCHITECTURAL TEACHING MODEL. The CXL 68-byte flit carries a 2-byte CRC
// protecting its 64-byte payload, and CXL 3.0 uses PCIe 6.0 FEC and CRC with a
// different CRC for latency-optimized flits -- but NO CXL or PCIe CRC
// polynomial, FEC code, syndrome or check sequence is modelled or claimed here.
// The 8-bit parity-style check below is a teaching stand-in.
module crc_guard #(
  parameter bit ACCEPT_ON_MISMATCH = 1'b0   // 1 = the bug shape
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        flit_valid,
  input  logic [31:0] payload,
  input  logic [7:0]  rx_check,        // the check value that arrived with it
  input  logic        fec_corrected,   // a lower-level code already fixed it
  output logic [7:0]  computed_check,
  output logic        integrity_ok,
  output logic        deliver,
  output logic        request_retry,
  output logic [15:0] n_corrected_q,   // fixed below, no retry needed
  output logic [15:0] n_detected_q,    // not fixable, retry required
  output logic [15:0] n_delivered_q,
  output logic        bad_delivered_err
);
  logic match;
 
  // A deliberately weak teaching check: byte-wise XOR of the payload.
  assign computed_check = payload[31:24] ^ payload[23:16]
                        ^ payload[15:8]  ^ payload[7:0];
  assign match          = (computed_check == rx_check);
  assign integrity_ok   = flit_valid && match;
 
  // The decision the whole layer exists to make.
  assign deliver       = flit_valid && (ACCEPT_ON_MISMATCH ? 1'b1 : match);
  assign request_retry = flit_valid && !match && !ACCEPT_ON_MISMATCH;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      if (flit_valid &&  fec_corrected) n_corrected_q <= n_corrected_q + 16'd1;
      if (flit_valid && !match)         n_detected_q  <= n_detected_q  + 16'd1;
      if (deliver)                      n_delivered_q <= n_delivered_q + 16'd1;
      // Something failed its integrity check and went up the stack anyway.
      if (deliver && !match) bad_delivered_err <= 1'b1;
    end
  end
endmodule

Corrected and detected are counted separately, and that separation is the diagnostic. A rising corrected count with a flat detected count is a channel degrading within the correcting code's power — an early warning. A rising detected count means errors are escaping correction and each one now costs a round trip. Collapsing them into one "error count" discards exactly the information that distinguishes a link to watch from a link to fix.

Synthesis. The check itself is a small XOR tree here; a real CRC is a wider XOR network, still combinational, still in the receive path. The important structural point is that this is combinational and in the critical path of every received unit — which is why the code's complexity is a real design constraint and not just a mathematical choice.

Aliasing is the limit nobody escapes. Any finite check maps many corrupted payloads onto the same value, so some corruptions pass. This is where the published FIT figures come from, and it is why "the CRC passed" means "no detected error" rather than "correct".

Simulation evidence

=== EXP1: integrity check -- what gets delivered ===
  intact flit    : computed=44 received=44 ok=1 deliver=1 retry=0
  corrupted flit : computed=44 received=ff ok=0 deliver=0 retry=1
                   accept-on-mismatch variant: deliver=1
 
  accept-on-mismatch bad_delivered_err=1

Row one is identical between the two designs. Every intact flit — which is almost all of them — behaves the same way, so a receiver that ignores its own check is indistinguishable from a correct one on a healthy channel. The divergence appears exactly when the mechanism was supposed to earn its keep.

9. RTL 2 — The Replay Buffer

Purpose

You cannot resend what you did not keep.

// The replay buffer: hold every transmitted unit until the far side confirms
// it arrived, so a failed transfer can be sent again.
//
// The buffer's DEPTH is what bounds how far ahead the transmitter may run. A
// sender that transmits beyond its ability to replay has no way to recover,
// which is why `can_send` is a hard gate rather than advice.
//
// ARCHITECTURAL TEACHING MODEL. CXL 68-byte flit mode includes link
// reliability mechanisms such as Ack/Nak; the sequence numbering, buffer
// management and replay policy below are TEACHING VALUES and are not the
// CXL-defined retry mechanism.
module retry_buffer #(
  parameter int unsigned DEPTH        = 8,
  parameter int unsigned MAX_RETRY    = 3,
  parameter bit          FREE_ON_SEND = 1'b0   // 1 = the bug shape
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       send,
  input  logic [7:0] send_data,
  input  logic       rx_ack,          // far side got everything up to ack_seq
  input  logic [4:0] ack_seq,
  input  logic       rx_nak,          // far side rejected from nak_seq onward
  input  logic [4:0] nak_seq,
  output logic       can_send,
  output logic [4:0] send_seq,
  output logic       replaying,
  output logic [4:0] replay_seq,
  output logic [7:0] replay_data,
  output logic [7:0] occupancy_q,
  output logic [7:0] max_occupancy_q,
  output logic       give_up,
  output logic       lost_unit_err,     // acknowledged more than was outstanding
  output logic       overflow_err
);
  // A sender may only transmit what it could still replay.
  assign can_send = (occupancy_q < DEPTH[7:0]) && !replaying;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      if (send && can_send) begin
        buf_q[seq_q % DEPTH] <= send_data;
        seq_q <= seq_q + 5'd1;
        // FREE_ON_SEND=1 never grows the buffer: the unit is considered done
        // the moment it leaves, so nothing can be replayed.
        if (!FREE_ON_SEND) occupancy_q <= occupancy_q + 8'd1;
      end
 
      if (rx_ack) begin
        // Everything up to and including ack_seq is confirmed and may go.
        // Acknowledging more than was outstanding means the two ends disagree
        // about what was sent -- reportable, not silently clamped.
        if (occupancy_q >= (ack_seq - oldest_q + 5'd1))
          occupancy_q <= occupancy_q - 8'(ack_seq - oldest_q + 5'd1);
        else begin occupancy_q <= '0; lost_unit_err <= 1'b1; end
        oldest_q      <= ack_seq + 5'd1;
        retry_count_q <= 8'd0;
      end
 
      if (rx_nak && !replaying) begin
        replaying <= 1'b1; replay_ptr_q <= nak_seq;
        retry_count_q <= retry_count_q + 8'd1;
      end else if (replaying) begin
        if (replay_ptr_q + 5'd1 == seq_q) replaying <= 1'b0;   // caught up
        else                              replay_ptr_q <= replay_ptr_q + 5'd1;
      end
    end
  end
endmodule

An acknowledgement is cumulative and that is a deliberate efficiency. Confirming "everything up to sequence N" retires many entries with one message, rather than one acknowledgement per unit. The cost is that a single lost acknowledgement delays retiring everything behind it — which is fine, because the next one covers it.

Replay restarts from the failure point and continues forward, not just the single failed unit. The receiver rejected from that sequence onward, so everything after it must be resent in order — which is why replaying blocks new sends until it catches up, and why a replay costs proportional to how far ahead the sender had run.

give_up bounds it. A link that retries forever converts a permanently failed transfer into a permanently stalled transaction — Chapter 3.5's point, and the reason escalation to the layer above must eventually happen.

Simulation evidence

=== EXP2: replay -- what a sender must keep ===
  5 sent, none acknowledged:
    keep-until-acked : occupancy=5 can_send=1
    free-on-send     : occupancy=0 can_send=1  <-- nothing to replay
 
  nak from seq 2:
    keep-until-acked : replaying=1 seq=2 data=a2
 
  after replay: keep-until-acked replaying=0 replayed=3
  ack up to seq 4: occupancy=0 retry_count=0

The correct buffer held five, replayed three (sequences 2, 3 and 4), and retired everything on a single cumulative acknowledgement. The retry counter reset on the acknowledgement, which matters — retries must count consecutive failures, not lifetime failures, or a long-lived healthy link eventually exceeds any threshold.

=== EXP3: buffer depth bounds how far ahead you may run ===
  12 sends into a depth-8 buffer:
    keep-until-acked : can_send=0 occupancy=8 max=8  <-- STALLED at 8
    free-on-send     : can_send=1 occupancy=0        <-- ran ahead unbounded
 
  -> the correct buffer refused to transmit a 9th unacknowledged unit.
     The free-on-send buffer transmitted all 12 and retained none,
     so it has run 12 units ahead of anything it could replay.

The correct sender stalled at 8, which is the entire mechanism working. The free-on-send variant transmitted all twelve and kept none — so a rejection of any of them is unrecoverable, and the failure surfaces not as a link error but as a transaction that never completes.

10. RTL 3 — Credits

Purpose

Ask permission before committing, because on a long link asking afterwards is too late.

// Credit-based flow control: a transmitter may only send what the receiver has
// already promised it has room for.
//
// The difference from a ready/valid handshake matters. Ready/valid tells you
// about THIS cycle after you have offered; credits tell you about the future
// before you commit, which is what makes them workable across a link whose
// round trip is long.
//
// ARCHITECTURAL TEACHING MODEL. CXL 68-byte flit mode includes credits; the
// credit unit, initialisation and return mechanism below are TEACHING VALUES
// and are not the CXL-defined credit scheme.
module credit_flow #(
  parameter int unsigned INIT_CREDITS   = 8,
  parameter bit          IGNORE_CREDITS = 1'b0   // 1 = the bug shape
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        want_send,
  input  logic        credit_return,     // receiver freed one buffer
  input  logic        rx_buffer_free,    // receiver's actual free space > 0
  output logic        may_send,
  output logic        did_send,
  output logic [7:0]  credits_q,
  output logic [15:0] blocked_cycles_q,
  output logic [15:0] n_sent_q,
  output logic        overrun_err,       // sent with no room at the receiver
  output logic        credit_leak_err    // credits exceeded what was issued
);
  assign may_send = IGNORE_CREDITS ? 1'b1 : (credits_q != 8'd0);
  assign did_send = want_send && may_send;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      if (want_send && !may_send) blocked_cycles_q <= blocked_cycles_q + 16'd1;
 
      // Spend on send, replenish on return -- in ONE assignment, because both
      // can happen in the same cycle and two writes would lose one.
      credits_q <= credits_q
                 - ((did_send && credits_q != 8'd0) ? 8'd1 : 8'd0)
                 + (credit_return ? 8'd1 : 8'd0);
 
      // The invariant the whole scheme exists to guarantee.
      if (did_send && !rx_buffer_free) overrun_err <= 1'b1;
      // More credits than were ever issued means a return was double-counted.
      if (credits_q > INIT_CREDITS[7:0]) credit_leak_err <= 1'b1;
    end
  end
endmodule

Two error outputs guard the two directions. overrun_err catches spending credit that was not there; credit_leak_err catches gaining credit that was never issued. The second is easy to overlook and is the more insidious — a leak inflates the sender's permission over time, so the design works perfectly and then overruns the receiver after hours of operation.

rx_buffer_free is a verification input, not a design input. A real transmitter cannot see the receiver's occupancy — that is precisely why credits exist. It is present here so the testbench can check that the credit scheme actually protected what it claims to protect, which is the difference between checking the mechanism and checking its purpose.

Simulation evidence

=== EXP4: credits -- permission before commitment ===
  12 send attempts, 8 initial credits, no returns:
    honouring credits : sent=8 credits=0 blocked=4
    ignoring credits  : sent=12 overrun_err=0
 
  receiver now genuinely full:
    honouring credits : did_send=0 overrun_err=0
    ignoring credits  : did_send=1 overrun_err=1  <-- data lost
 
  4 credits returned: credits=4

The credit-ignoring sender showed overrun_err=0 for the first twelve sends, and that is the important subtlety: it sent four units it had no permission for, and nothing went wrong because the receiver happened to have room. The scheme was violated and the consequence did not arrive until the receiver was genuinely full.

That gap — between violating a protocol and suffering for it — is why this defect survives testing. A bench where the receiver is fast never punishes it.

11. RTL 4 — Correct First, Then Check

Purpose

Order the two mechanisms so the cheap one runs first.

// Two mechanisms, in order, doing different jobs.
//
// A correcting code runs FIRST and fixes what it can in place -- no retry, no
// latency cost beyond its own. A detecting check runs SECOND on the result and
// catches what correction could not, which is what triggers a replay.
//
// ARCHITECTURAL TEACHING MODEL. CXL 3.0 uses PCIe 6.0 FEC and CRC at 64 GT/s,
// with a different CRC for latency-optimized flits. NO FEC code, CRC
// polynomial, correction capability or syndrome is modelled or claimed.
module fec_then_crc #(
  parameter bit CRC_FIRST = 1'b0        // 1 = the wrong order
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        flit_valid,
  input  logic        err_correctable,   // within the correcting code's power
  input  logic        err_uncorrectable, // beyond it
  output logic        fec_fixed,
  output logic        crc_failed,
  output logic        deliver,
  output logic        trigger_retry,
  output logic [15:0] n_fixed_q,
  output logic [15:0] n_retried_q,
  output logic [15:0] n_clean_q,
  output logic        wasteful_retry_err // retried something FEC could fix
);
  // Correct first: a correctable error is gone by the time integrity is judged.
  assign fec_fixed    = flit_valid && err_correctable && !CRC_FIRST;
  assign post_fec_bad = CRC_FIRST ? (err_correctable || err_uncorrectable)
                                  : err_uncorrectable;
  assign crc_failed   = flit_valid && post_fec_bad;
 
  assign deliver       = flit_valid && !crc_failed;
  assign trigger_retry = crc_failed;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      // A replay was requested for an error the correcting code could have
      // handled: full round-trip cost paid for nothing.
      if (trigger_retry && err_correctable && !err_uncorrectable)
        wasteful_retry_err <= 1'b1;
    end
  end
endmodule

Simulation evidence

=== EXP5: correct first, then check ===
  clean         : correct-order deliver=1 retry=0 | crc-first deliver=1 retry=0
  CORRECTABLE   : correct-order fixed=1 deliver=1 retry=0 | crc-first retry=1
  UNCORRECTABLE : correct-order deliver=0 retry=1 | crc-first retry=1
 
  crc-first wasteful_retry_err=1  (retried an error FEC could fix)
  correct order: fixed=1 retried=2 clean=3

Rows one and three are identical between the orderings. Clean units and unrecoverable units behave the same either way — the entire difference is row two, the correctable error, which the wrong order converts into a full round trip.

That matters more than one row suggests, because on a high-BER channel correctable errors are the common case. That is the whole reason a correcting code is there. So an implementation with the order reversed pays a round trip on the errors that occur most often, and its symptom is not incorrectness — it is a link that mysteriously spends a large fraction of its capacity on retransmission.

12. RTL 5 — What Recovery Cost

Purpose

Separate "how much did we send" from "how much of the wire did we use".

// Link-layer health accounting: how much of the wire went to recovery, and
// how often integrity actually failed.
//
// ARCHITECTURAL TEACHING MODEL. Real link layers expose considerably richer
// error registers and telemetry than this.
module link_health (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        unit_sent,
  input  logic        unit_replayed,
  input  logic        crc_fail,
  input  logic        fec_fix,
  input  logic        blocked,
  output logic [15:0] n_sent_q,
  output logic [15:0] n_replayed_q,
  output logic [15:0] n_crcfail_q,
  output logic [15:0] n_fecfix_q,
  output logic [15:0] blocked_cycles_q,
  output logic [15:0] max_block_run_q,
  output logic [15:0] wire_units_q      // sent + replayed = wire occupancy
);
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      // Wire occupancy counts BOTH first transmissions and replays. A replay
      // consumes the link exactly as much as an original does.
      wire_units_q <= wire_units_q + {15'b0, unit_sent} + {15'b0, unit_replayed};
      // ... per-event counts, and a maximum blocked run alongside the total ...
    end
  end
endmodule

wire_units_q exists as a separate quantity from n_sent_q because they answer different questions. Sent units measure work delivered; wire units measure capacity consumed. Their ratio is the retry overhead, and a link that reports only the first will look healthy while spending a third of its capacity resending.

Simulation evidence

=== link health ===
  sent=13 replayed=3 wire_units=16 crc_fail=5 fec_fix=1 blocked=5 max_block=5

Sixteen units of wire time delivered thirteen units of work — an 18.75% recovery overhead in this run, entirely invisible in the sent count. Applying Section 6's formula in reverse: 13/16 = 0.8125, so the effective rate was about 81% of nominal, and every transmission counter reported success.

13. Waveform — A Rejection and Its Replay

Four readings.

Occupancy rises and does not fall until the acknowledgement. Every one of those five units is being held, and that holding is what makes recovery possible. A design that freed on transmission would show occupancy flat at zero and would have nothing to resend at cycle 6.

can_send drops for the entire replay. New work is blocked while old work is resent — the link is fully occupied and delivering nothing new. This is where the 1/(1+r) rate penalty physically happens, and the trace makes it concrete: three cycles of wire time producing zero new payload.

Replay is a range, not a unit. Sequences 2, 3 and 4 all go again, because the receiver rejected from sequence 2 onward. So the cost of a rejection scales with how far ahead the sender had run — which is the argument for a buffer sized to the round trip and no larger.

One acknowledgement retires five units. Cumulative acknowledgement is why the return path does not need a message per unit, and it is why a single lost acknowledgement is harmless: the next one covers everything behind it.

14. Assertions

Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog. Not executed; each property maps to its procedural stand-in and to the mutation that proves the check fires.

// SAFETY -------------------------------------------------------------------
// D1 — nothing that failed its integrity check is delivered upward.
a_no_bad_delivery: assert property (@(posedge clk) disable iff (!rst_n)
  deliver |-> integrity_ok);
 
// D2 — the replay buffer never exceeds its depth, and never accepts when full.
a_buffer_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (occupancy_q <= DEPTH) && (send && can_send |-> occupancy_q < DEPTH));
 
// D3 — a sender never transmits a unit it could not replay.
a_replayable: assert property (@(posedge clk) disable iff (!rst_n)
  (send && can_send) |-> (occupancy_q < DEPTH));
 
// D4 — credits are never spent below zero, and never exceed what was issued.
a_credit_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (credits_q <= INIT_CREDITS) && (did_send |-> (credits_q != 0)));
 
// D5 — the purpose, not the mechanism: a credited send always had room.
a_no_overrun: assert property (@(posedge clk) disable iff (!rst_n)
  did_send |-> rx_buffer_free);
 
// D6 — CONSERVATION: wire occupancy equals originals plus replays. This is
// the relation that makes recovery cost visible, and M/N5 proves it fires.
a_wire_conserved: assert property (@(posedge clk) disable iff (!rst_n)
  wire_units_q == n_sent_q + n_replayed_q);
 
// D7 — a correctable error never triggers a replay.
a_no_wasteful_retry: assert property (@(posedge clk) disable iff (!rst_n)
  (err_correctable && !err_uncorrectable) |-> !trigger_retry);
 
// LIVENESS -----------------------------------------------------------------
// D8 — a replay always terminates: it catches up or the sender gives up.
// ASSUMPTION: the far side eventually acknowledges or rejects. That is a
// constraint on the environment, not on this module.
a_replay_terminates: assert property (@(posedge clk) disable iff (!rst_n)
  $rose(replaying) |-> ##[1:DEPTH+1] (!replaying || give_up));
 
// D9 — a blocked sender eventually proceeds once credits return.
a_credits_unblock: assert property (@(posedge clk) disable iff (!rst_n)
  (want_send && !may_send && credit_return) |-> ##[1:2] may_send);
 
// GOAL / PERFORMANCE -------------------------------------------------------
// D10 — recovery overhead stays within the design's budget.
a_retry_overhead: assert property (@(posedge clk) disable iff (!rst_n)
  (n_sent_q > 100) |-> (n_replayed_q * 10 < n_sent_q));

D10 is a goal property and it failed the budget in this run — deliberately, because the stimulus injected a rejection into a short run. That is the right behaviour for a performance property: it is not a correctness violation, it is the design telling you it is outside its intended operating envelope. A safety property has no way to express that, and a link at 18.75% recovery overhead is a link worth investigating even though every unit was eventually delivered correctly.

15. Mutation Testing

Six deliberate mutations injected into the exact tutorial RTL, compiled and simulated; the good RTL restored and re-verified.

N3 is the interesting one. A credit returned without a corresponding spend is a slow leak — the sender gains permission it was never granted, a little at a time. It was caught by the overrun check rather than by a credit check, because the leak's consequence is eventually sending when there is no room. That is worth noting as a general pattern: a resource-accounting bug is often easier to catch at its consequence than at its cause, which is an argument for asserting the purpose (did_send |-> rx_buffer_free) alongside the mechanism.

16. Debug Lab

assign integrity_ok = (computed_check == rx_check);   // computed
assign deliver      = flit_valid;                     // ... and not used

  intact flit    : ok=1 deliver=1 retry=0
  corrupted flit : ok=0 deliver=0 retry=1 | accept-on-mismatch deliver=1
  accept-on-mismatch bad_delivered_err=1

assign deliver = flit_valid && integrity_ok;
a_no_bad_delivery: assert property (deliver |-> integrity_ok);

// The unit is on the wire; the buffer entry is done.
if (send && can_send) begin
  buf_q[seq_q % DEPTH] <= send_data;
  seq_q <= seq_q + 1;
  // occupancy never incremented
end

  5 sent, none acknowledged:
    keep-until-acked : occupancy=5 can_send=1
    free-on-send     : occupancy=0 can_send=1  <-- nothing to replay
 
  12 sends into a depth-8 buffer:
    keep-until-acked : can_send=0 occupancy=8  <-- STALLED at 8
    free-on-send     : can_send=1 occupancy=0  <-- ran ahead unbounded

if (send && can_send) occupancy_q <= occupancy_q + 1;
if (rx_ack)           occupancy_q <= occupancy_q - (ack_seq - oldest_q + 1);
assign can_send = (occupancy_q < DEPTH) && !replaying;

// Return credits as the receiver frees buffers.
credits_q <= credits_q + (credit_return ? 1 : 0);   // spend path lost

  receiver now genuinely full:
    honouring credits : did_send=0 overrun_err=0
    ignoring credits  : did_send=1 overrun_err=1  <-- data lost

credits_q <= credits_q
           - ((did_send && credits_q != 0) ? 1 : 0)
           + (credit_return ? 1 : 0);

// Check integrity on what arrived from the wire.
assign crc_failed = flit_valid && (err_correctable || err_uncorrectable);

  clean         : correct-order retry=0 | crc-first retry=0
  CORRECTABLE   : correct-order fixed=1 retry=0 | crc-first retry=1
  UNCORRECTABLE : correct-order retry=1 | crc-first retry=1
 
  crc-first wasteful_retry_err=1

assign post_fec_bad = err_uncorrectable;      // correctable errors are gone
assign crc_failed   = flit_valid && post_fec_bad;

// Count what we transmitted.
if (unit_sent) wire_units_q <= wire_units_q + 1;   // replays not counted

  sent=13 replayed=3 wire_units=16 crc_fail=5 fec_fix=1

wire_units_q <= wire_units_q + {15'b0, unit_sent} + {15'b0, unit_replayed};
a_wire_conserved: assert property (wire_units_q == n_sent_q + n_replayed_q);

17. Design Review

On integrity. Is the check in the decision path or merely computed? Does the property name the check independently, or does it assert deliver against deliver? Are corrected and detected counted separately — because their ratio is the early warning that a channel is degrading?

On replay. When exactly is a buffer entry freed — on transmission, on acknowledgement, or on the data being consumed? Does occupancy actually gate can_send, or is the depth decorative? Is replay a range from the failure point, and does it block new sends while it catches up? Is there a retry limit, and does exceeding it escalate rather than spin? Does the retry counter reset on a successful acknowledgement, or does it accumulate over the link's lifetime?

On credits. Are spend and return in one assignment? Are both directions asserted — never below zero, never above what was issued? Is the purpose asserted alongside the mechanism, so a credited send is checked to have actually had room? Who issues the initial credits, and what happens on reset with credits outstanding?

On ordering. Does correction run before detection? Is there a check that a correctable error never triggers a replay?

On instrumentation. Is wire occupancy counted separately from delivered work? Is there a recovery-overhead budget expressed as a property? Are maxima recorded alongside totals?

And the structural question. Which upstream behaviour would change if this layer's retry became visible? If the answer is "none", the layering holds. If any upstream logic branches on a retry count, Chapter 3.5's boundary has already been breached.

18. Verification Plan

Reference model. A transmitter-side model tracking every sequence number's state — sent, acknowledged, replaying — and a receiver-side model of expected delivery order. Two conservation relations fall out: wire units equal originals plus replays, and delivered units equal sent units minus those still outstanding. The scoreboard is keyed on sequence number, because link-layer defects are relationships between a unit and its acknowledgement.

Directed tests. Intact and corrupted units; each of the three error classes separately; more sends than buffer depth with no acknowledgements; a rejection at the oldest, middle and newest outstanding sequence; a cumulative acknowledgement covering several units; a lost acknowledgement followed by a later one; credit exhaustion; a credit return coinciding with a send; retry limit exceeded; reset with units outstanding.

Constrained-random dimensions. Error injection rate and class mix, acknowledgement latency, rejection position within the outstanding window, credit-return timing relative to sends, and send duty cycle.

Functional coverage:

The cross worth taking is error class × buffer occupancy, because a rejection with a full buffer is where replay and flow control interact. The coincidence row is the one that finds the NBA-hazard class and it is routinely omitted.

Error injection. Bit corruption at a controlled rate; a rejection during an in-progress replay; an acknowledgement for a sequence never sent; duplicate acknowledgements; and a far side that stops responding entirely.

19. How This Appears in Real Engineering

CXL / protocol architect

The Latency-Optimized flit decision lives here, and the Consortium quantified it: 2–5 ns against FIT from 5×10⁻⁸ to 0.026 and efficiency from 0.94 to 0.92. That is a deliberate weakening of integrity for latency, and it should be a decision with a stated rationale rather than a default.

RTL engineer

Five disciplines from the measured runs: put the check in the decision path; free a buffer entry on acknowledgement, never on transmission; keep spend and return in one assignment; run correction before detection; and count replays as wire time.

DV engineer

The channel model is the test. A bench with a clean channel cannot distinguish a receiver that enforces its check from one that ignores it — every intact unit behaves identically. Drive the three error classes separately, and include the coincidence cases where a send and a return land in the same cycle, because that is where the credit leak lives.

Performance engineer

Two ratios. Corrected-to-detected tells you whether the channel is degrading within or beyond the correcting code's power. Sent-to-wire tells you the recovery overhead — measured at 18.75% here, on a link whose transmission counters all reported success. Then apply effective = nominal / (1 + r) to convert it into the rate the layers above will actually see.

Firmware and system software

The link-layer error registers are the earliest warning a platform gets that a channel is degrading. A corrected count climbing with a flat detected count is a link to schedule maintenance on; a climbing detected count is a link already costing throughput.

Silicon debug / validation

The counters worth their area are corrected, detected, replayed, and blocked-cycle maximum. The first two localise the channel's condition, the third quantifies its cost, and the fourth distinguishes steady backpressure from a single long stall.

20. Common Misconceptions

21. Interview Reasoning

22. Exercises

1. Explain. Correction runs before detection. State the general principle that ordering follows, and identify one other place in this course where the same principle decided a design.

2. Calculate. A link sends one unit per cycle with a 60-cycle acknowledgement round trip. What replay-buffer depth avoids stalling? If the design ships with 16, what is the achievable rate as a fraction of nominal?

3. Calculate. A link reports a 12% replay fraction. What is the effective delivery rate? If the layer above is a boundary queue sized against nominal drain rate, what does Chapter 3.5's analysis predict it will do?

4. RTL modification. Change retry_buffer so a rejection replays only the named sequence rather than the range from it forward. Construct the stimulus that shows this is wrong, and say which of Section 14's properties detects it.

5. DV task. Write the SVA for the credit-leak property, then construct a mutation that satisfies it while still overrunning the receiver. What third property closes the gap?

6. Debug task. A link reports corrected=4000, detected=3, replayed=3, over a million units. A second link reports corrected=40, detected=800, replayed=800. Both deliver correct data. Which one would you investigate first, and what is each one telling you?

23. Summary

The data link layer is an insurance policy with a premium. It makes an unreliable channel look reliable to everything above it, and it charges wire time for the service.

Two mechanisms, in order. Correction fixes what it can with no round trip; detection catches what it cannot and asks again. Measured, reversing the order turned every correctable error into a retransmission while clean and uncorrectable units behaved identically — and correctable errors are the common case on a channel the Consortium describes as having a high BER mitigated by FEC and CRC.

The replay buffer is what makes recovery possible, and its depth is a promise. A sender may transmit only what it could resend. Measured, twelve sends into a depth-8 buffer stalled at eight; the free-on-transmission variant kept nothing, and lost its flow control as a side effect because occupancy never gated transmission. Depth must track rate × acknowledgement round trip, which is the same calculation as Chapter 3.2's outstanding table.

Credits are permission before commitment. Measured, a credit-ignoring sender transmitted four units it had no right to and nothing went wrong — until the receiver was genuinely full. The gap between violating the protocol and suffering for it is exactly why the defect survives testing.

Recovery is not free. Sixteen units of wire time delivered thirteen units of work — 18.75% overhead with every transmission counter reporting success. The effective rate is nominal / (1 + r), and only a wire-occupancy counter separate from a delivered-work counter makes it visible.

Two results carry beyond this layer. Assert the purpose, not only the mechanism — a credit leak was caught by "a credited send had room" rather than by any credit bound. And the same non-blocking-write hazard has now appeared three times in this course, in a latency accumulator, a wire counter and a credit register: any register written from two conditions in one cycle needs one assignment, and the failure is always silent and always in the flattering direction.

24. What Comes Next

This chapter delivered units intact. It said nothing about what those units mean.

Chapter 4.3 is the layer that knows: what a request is, what ordering it requires, how the three protocol classes differ in what they need from the layers below, and why the Consortium's stated target of near-CPU-cache latency for .cache and .mem shapes every decision at that level.

For adjacent material: Physical Layer has the flit structure whose CRC this chapter used, CXL Layered Architecture has the contract that keeps retry invisible upstream, and The CXL System View has the end-to-end path a replay silently lengthens. The path is on the CXL tutorials index.