Training Failures

Chapter 21.1's dependency ledger localised the hang and handed over two bitmaps: which lanes have ever measured, and which have ever qualified. Training is not idle — it is measuring, scoring and retrying continuously — and none of that activity is converging.

1. The One-Sentence Model

Training is repeated measurement, followed by qualification, followed by atomic commit. A failure to converge means exactly one of those three contracts is broken — and the three have different owners, different evidence and different fixes.

That decomposition is the chapter's spine, and it is also its triage. Three questions in order (§8) route every training failure to one of three sections, and the routing costs two register reads.

2. What This Chapter Owns

8.3 and 8.4 built the machinery. 21.1 established that training is the blocker. This chapter finds out why, and the four things it owns are:

Three separable contracts (§4, §8), because "training fails" is not a diagnosis and "measurements arrive, no candidate is ever accepted" is.

The instrumentation that makes a lane nameable (§9–§14). An aggregate all_lanes_ok bit is the least informative register a training block can expose, and §11 is what it costs.

Four qualification bugs that all present as "never converges" (§20–§27) — a run counter that counts idle cycles, one that resets on a bubble, one that wraps, and a threshold that cannot be represented.

And three flagship failures with distinct signatures: the candidate pointer that advances only on success (§33), the one silent lane a global watchdog can never detect (§47), and the stale sample from a previous attempt (§38).

3. Sourcing

4. Three Contracts

Training is three contracts in series, and a convergence failure breaks exactly one.

Three properties of this table.

They are strictly ordered, so a failure in one makes the others unobservable. No measurements means qualification has nothing to work with, so a qualification investigation on a measurement failure finds a run counter sitting at zero and concludes the qualification logic is broken.

Each has a different owner. Measurement is the PHY and the sample path; qualification is the training controller's counting logic; commit is the controller's gating. Three teams, and the triage of §8 decides which one gets the bug.

And "training fails" spans all three. That is why it is not a diagnosis — it names the subsystem, not the contract, and the subsystem contains three independent failure surfaces.

5. Debug the Candidate Before the Active Setting

The single most common wrong conclusion in training debug:

"The active setting never changed, so the search never found anything."

8.4 §4's three copies of the setting is why that is wrong. Candidates may have been generated, applied, measured, scored and rejected dozens of times — and the active register is the last thing to move. A design exposing only the active value has hidden the entire search.

Six questions, in order, and each eliminates a stage:

And the instrumentation consequence is one line: expose the candidate, not only the active setting. §31 is the failure of not doing so, and it is the most expensive instrumentation omission in this chapter.

6. The Debug Pipeline

Three things to read.

Three arrows leave MEASURE and QUALIFY between them, and each is a different failure family. MEASURE → RETRY is contract 1; QUALIFY → ISSUE is the normal reject path and its absence is §33's bug; QUALIFY → CAND VALID never firing is contract 2.

QUALIFY → ISSUE is drawn as an ordinary arrow deliberately. A rejected candidate leading to the next candidate is normal, healthy progress — and §33 is the design where that arrow exists in the diagram and not in the RTL, so the controller re-issues the same candidate forever.

And CAND VALID → RETRY is the invalidation path. A candidate that was accepted and then invalidated is §41's peer-restart case and §38's stale-evidence case — which is 21.1 §10's shape 2 at the candidate level.

7. Waveform — Training That Never Converges

Four readings, and the first two are the diagnosis.

sample_valid is active on seven of ten cycles. Measurement works — contract 1 is satisfied, and the entire measurement path is eliminated by one signal.

qual_run is the discriminating signal. It reaches 3 twice and resets once, so the failure is a rate problem: failing samples arrive often enough that the run never reaches threshold. That is a contract-2 observation pointing at signal quality or the threshold, not at the counting logic.

Cycles 3 and 6 show the counter correctly holding across idle cycles, which eliminates §23's bug. A design with that bug would reset at cycle 3 and never exceed 2 — and distinguishing the two is a one-cycle observation.

And commit flat at zero is the symptom the report arrives with, carrying no information (21.1 §4's argument). candidate_valid flat at zero is one step better and still derived — it is blocked behind the run counter.

8. The Three-Question Triage

Two register reads route the failure to one of three sections.

Worked, on the three bitmap distributions 21.1 §34 hands over:

The last two rows differ by one bit and they are completely different bugs. A lane that measures and never qualifies has a signal quality or threshold problem; a lane that never measures has a path problem — and confusing them sends the investigation to the wrong team.

9. The Per-Lane Debug Record

// ILLUSTRATIVE debug state, per lane. Symbolic widths; no UCIe lane-training
// field is claimed (§3). This is the record that makes a lane NAMEABLE.
typedef struct packed {
  logic                 ever_seen;      // has this lane EVER produced a valid sample?
  logic                 ever_qualified; // has it EVER reached the qualified state?
  logic                 qualified_now;
  logic [RUN_W-1:0]     pass_run;       // consecutive passing samples (§20)
  logic [CNT_W-1:0]     samples_valid;  // saturating
  logic [CNT_W-1:0]     samples_failed; // saturating — the RATIO is the finding
  logic [AGE_W-1:0]     age_since_sample;   // saturating (§48)
  logic [ATT_W-1:0]     attempts;       // retries attributable to this lane
  logic [CAND_W-1:0]     last_candidate; // which candidate it last measured
  logic [RSN_W-1:0]     last_fail_reason;   // §13
  logic [EPOCH_W-1:0]   evidence_epoch; // WHICH attempt this evidence belongs to (§37)
} lane_dbg_t;
 
lane_dbg_t lane_dbg_q [NUM_LANES];

Five fields carry most of the diagnostic value, and one is the field nobody includes.

samples_valid and samples_failed as a pair. The ratio distinguishes "marginal" from "dead" from "clean": 1000 valid with 2 failed is a healthy lane; 1000 valid with 400 failed is marginal; 0 valid is a path problem, not a quality problem. A single "error count" cannot express any of that.

age_since_sample, per lane, which is §47's entire diagnostic and the one a global watchdog cannot substitute for.

last_candidate — because a lane that fails only on certain candidates is a scoring or setting problem rather than a lane problem (§14).

evidence_epoch is the field that nobody includes and §38 is what it costs. Evidence is scoped to an attempt (21.1 §6's ledger column 6), and without the epoch a sample from attempt 7 satisfies attempt 8.

And the two ever_ bits are 21.1 §10's split applied per lane — never-measured versus measured-then-regressed, for two flops per lane.

10. The Two Bitmaps

// ILLUSTRATIVE. The two registers 21.1 §34 reads, and the cheapest useful
// training instrumentation there is: two bits per lane.
logic [NUM_LANES-1:0] lane_seen_q;        // ever produced a valid sample
logic [NUM_LANES-1:0] lane_qualified_q;   // ever reached the qualified state
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    lane_seen_q      <= '0;
    lane_qualified_q <= '0;
  end else begin
    for (int l = 0; l < NUM_LANES; l++) begin
      if (lane_sample_valid[l]) lane_seen_q[l]      <= 1'b1;
      if (lane_qualified[l])    lane_qualified_q[l] <= 1'b1;
    end
  end
end

Three properties.

Two bits per lane, and they answer §8's first two questions outright. For a 32-lane link that is 64 flops for a diagnostic that otherwise requires a simulation.

They are sticky across attempts, deliberately. A lane that qualified on attempt 3 and not since has a different problem from one that never qualified — and clearing these on retry would erase exactly that distinction. This is 14.2 §14's capture-before-retrain rule applied per lane.

And they are cleared only by por_n, for the same reason.

11. Wrong Debug State — the Aggregate Bit

// WRONG — the only training status exposed.
assign all_lanes_ok = &lane_qualified;    // one bit for 32 lanes

What the debugger receives: all_lanes_ok == 0.

Four questions it cannot answer, and they are the only questions there are.

Which lane? One lane failing and thirty-two lanes failing produce the same bit — and they are a lane-level problem and a system-level problem respectively.

Measurement or qualification? §8's rows 3 and 4 differ by one bit in a bitmap and are indistinguishable in an AND.

Did anything ever work? A link where all 32 lanes qualified on attempt 1 and one regressed on attempt 2 reads identically to one where nothing ever qualified.

And is it improving? Attempt 1 with 20 lanes qualified and attempt 17 with 31 is convergence in progress; the aggregate bit reads 0 for both.

The fix is 64 flops (§10). The aggregate bit is not merely less useful than the bitmaps — it is the single most expensive instrumentation omission in a training block, because it forces every training investigation into simulation.

12. The First Failing Lane

A sticky record of the first lane to fail or invalidate, for the same reason 14.5 §8 captures the first fault.

// ILLUSTRATIVE. Sticky-first, per attempt AND per power cycle — two records,
// because they answer different questions.
logic                  first_fail_valid_q;      // this power cycle
logic [LANE_W-1:0]     first_fail_lane_q;
logic [CAND_W-1:0]     first_fail_candidate_q;
logic [ATT_W-1:0]      first_fail_attempt_q;
logic [RSN_W-1:0]      first_fail_reason_q;
 
logic                  att_first_fail_valid_q;  // this attempt only
logic [LANE_W-1:0]     att_first_fail_lane_q;
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    first_fail_valid_q     <= 1'b0;
    att_first_fail_valid_q <= 1'b0;
  end else begin
    // Per-attempt record clears at the attempt boundary — that is its scope.
    if (attempt_start) att_first_fail_valid_q <= 1'b0;
    else if (lane_fail_event && !att_first_fail_valid_q) begin
      att_first_fail_valid_q <= 1'b1;
      att_first_fail_lane_q  <= fail_lane;
    end
    // Power-cycle record is STICKY across attempts (§10's argument).
    if (lane_fail_event && !first_fail_valid_q) begin
      first_fail_valid_q     <= 1'b1;
      first_fail_lane_q      <= fail_lane;
      first_fail_candidate_q <= fail_candidate;
      first_fail_attempt_q   <= attempt_q;
      first_fail_reason_q    <= fail_reason;
    end
  end
end

Two records, two questions.

The power-cycle record answers "what started this?" — 21.1 §23's first-versus-last comparison, at lane granularity. first_fail_attempt == 1 means the problem was present from the start.

The per-attempt record answers "is it the same lane every time?" Reading it at each attempt boundary produces a sequence: lane 12, lane 12, lane 12, … is a lane problem; lane 3, lane 19, lane 7, … is a system problem — signal integrity across the link, a shared reference, a supply.

And that sequence is the single most valuable output of training instrumentation, because it separates the two top-level hypotheses in one read per attempt.

13. Per-Lane Failure Reason

// ILLUSTRATIVE. A generic taxonomy — no UCIe failure class is claimed (§3).
// The value is that each reason points at a DIFFERENT owner.
typedef enum logic [RSN_W-1:0] {
  LF_NONE            = 'd0,
  LF_NO_SAMPLES      = 'd1,   // contract 1: nothing arrived    -> sample path
  LF_SAMPLE_INVALID  = 'd2,   // arrived, malformed or X        -> sample path
  LF_UNSTABLE        = 'd3,   // pass/fail alternating          -> signal quality
  LF_RUN_TOO_SHORT   = 'd4,   // never reached threshold        -> quality/threshold
  LF_TIMEOUT         = 'd5,   // no progress within the window  -> §46
  LF_CAND_REJECTED   = 'd6,   // this candidate scored poorly   -> the candidate
  LF_STALE_EVIDENCE  = 'd7,   // wrong epoch                    -> §38 — the guard working
  LF_INVALIDATED     = 'd8    // qualified, then lost           -> §41 peer restart
} lane_fail_reason_e;

Three readings the taxonomy enables.

LF_NO_SAMPLES versus LF_UNSTABLE is the contract-1/contract-2 split at lane granularity, and they go to different teams.

LF_STALE_EVIDENCE counts a correctly rejected sample. It is not an error — the epoch guard is working (§40) — but its rate is diagnostic: frequent stale evidence during steady training means attempts are overlapping in a way the design did not intend.

And LF_INVALIDATED is the reason that distinguishes a lane that never worked from one that stopped, which is §9's two ever_ bits expressed as a cause.

14. Reading the Distribution

14.5 §13 taught reading a per-lane fault distribution. Four shapes matter for training specifically.

Two notes.

Row 4 is the shape most often misdiagnosed as a lane problem. If every lane fails on candidates 5–7 and passes on 1–4, the lanes are fine and the candidate range is bad — and the evidence for that is the last_candidate field, not the fault counts.

And row 2 has a specific trap: it can be a threshold problem rather than a physical one. All lanes marginal at similar rates, with a qualification threshold set too high for the achievable error rate, produces exactly this — and the fix is in the training controller, not the PHY. §25 develops the threshold side.

15. Measurement Validity

Contract 1's first requirement: a sample must describe the thing it claims to describe.

Three ways it fails, and all three present as "training is random":

The second is the worst because it is self-consistent. The training algorithm receives a coherent-looking stream in which lane 7's result is labelled lane 6 — so it "repairs" a healthy lane and leaves the failing one alone, and the resulting behaviour looks like a search that cannot converge.

16. The Bundled Sample

// ILLUSTRATIVE. Everything that describes one measurement travels TOGETHER,
// in one packed struct, through one pipeline. 19.4 §16's payload-and-metadata
// rule applied to the training sample path.
typedef struct packed {
  logic                  valid;
  logic [LANE_W-1:0]     lane;
  logic [CAND_W-1:0]     candidate;
  logic [EPOCH_W-1:0]    epoch;        // §37 — which attempt (§38's guard)
  logic                  pass;
  logic [SCORE_W-1:0]    score;        // where the architecture scores rather than passes
} train_sample_t;
 
train_sample_t sample_pipe_q [SAMPLE_PIPE_DEPTH];
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    for (int i = 0; i < SAMPLE_PIPE_DEPTH; i++) sample_pipe_q[i].valid <= 1'b0;
  end else begin
    sample_pipe_q[0] <= sample_in;                      // ONE assignment
    for (int i = 1; i < SAMPLE_PIPE_DEPTH; i++)
      sample_pipe_q[i] <= sample_pipe_q[i-1];           // the WHOLE bundle moves
  end
end

Two properties.

One struct, one pipeline, one assignment per stage. 19.4 §16's rule: payload and metadata written on one event at one index. A bundle cannot become misaligned with itself.

And epoch is inside the bundle, not alongside it. A sample whose epoch travels on a separate path can be checked against the wrong attempt — which defeats §40's guard in exactly the way §38's bug needs.

17. Wrong RTL — Separate Pipelines

// WRONG — the fields travel separately, with different depths.
logic [LANE_W-1:0] lane_pipe_q [3];      // 3 deep
logic [CAND_W-1:0] cand_pipe_q [2];      // 2 deep
logic              pass_pipe_q [1];      // 1 deep
 
always_ff @(posedge clk) begin
  lane_pipe_q[0] <= lane_in;  lane_pipe_q[1] <= lane_pipe_q[0];
  lane_pipe_q[2] <= lane_pipe_q[1];
  cand_pipe_q[0] <= cand_in;  cand_pipe_q[1] <= cand_pipe_q[0];
  pass_pipe_q[0] <= pass_in;
end
// The consumer reads lane_pipe_q[2], cand_pipe_q[1], pass_pipe_q[0] —
// three fields from three DIFFERENT measurements.

Worked, over four cycles of a stream measuring lanes 5, 6, 7, 8:

Every record is a chimera of three different measurements.

Five properties.

The stream is coherent. No X, no gaps, no protocol violation — so nothing detects it, and every assertion about the sample path passes.

The algorithm behaves plausibly. It accumulates evidence, rejects candidates, retries. It looks like a search that cannot converge, which is the symptom of half this chapter.

It repairs the wrong lane. Lane 7's failure is attributed to lane 5; the training algorithm adjusts lane 5, which was fine, and lane 7 continues to fail — so the failure appears to move, which reads as §14's row 2.

Per-lane counters increase confidence in the wrong answer. samples_failed[5] is genuinely high, and it is high because lane 7's failures were recorded there.

And the fix is structural, not a delay adjustment. Equalising the depths works until somebody adds a stage; bundling the fields makes the misalignment unrepresentable (§16).

18. SVA — the Sample Bundle Is Aligned

// MANDATORY. The bundle's fields describe ONE measurement. Where the sample
// path is white-box, this is checkable directly at each stage boundary.
property p_sample_bundle_moves_together;
  @(posedge clk) disable iff (!por_n)
    1 |=> (sample_pipe_q[1] == $past(sample_pipe_q[0]));
endproperty
a_sample_bundle_moves_together: assert property (p_sample_bundle_moves_together);
 
// MANDATORY. The EFFECT form, and it is the one that catches §17 even when the
// pipelines are separate: a result must be attributed to the lane that was
// MEASURED, established by an independent tag rather than by pipeline depth.
property p_result_matches_measured_lane;
  @(posedge clk) disable iff (!por_n)
    result_valid |-> (result_lane == chk_measured_lane[result_tag]);
endproperty
a_result_matches_measured_lane: assert property (p_result_matches_measured_lane);
 
// MANDATORY. No result for a lane that was never issued a measurement.
property p_no_result_without_issue;
  @(posedge clk) disable iff (!por_n)
    result_valid |-> chk_issue_outstanding[result_lane];
endproperty
a_no_result_without_issue: assert property (p_no_result_without_issue);

Why the second property is the one that matters. The first checks the bundle against itself and cannot detect §17, because §17 has no bundle. The second is written against an independent record of which lane was issued which measurement — 20.3 §27's effect-form discipline — so a misattribution fails it regardless of how the pipeline is built.

19. Wrong Measurement — the Stale Valid

// WRONG — sample_valid is a LEVEL that stays high between measurements.
assign sample_valid = measurement_engine_busy;   // high for many cycles

Worked. One measurement is taken. sample_valid stays high for eight cycles while the engine settles and reports.

A qualification counter that increments on sample_valid && sample_pass counts eight passing samples from one measurement (§21).

Three properties.

It qualifies a lane on a single measurement, with a threshold of four satisfied by one sample held for four cycles. The lane is declared good on one data point — and 8.3 §5's one sample is not evidence argument is defeated not by a design decision but by a level.

The failure direction is dangerous. It makes training succeed too easily, so the link comes up and then misbehaves under real traffic. That is a family-G failure in 21.1 §37's terms — operational reached and lost — reached from a training bug.

And the discriminating observation is the ratio of samples_valid to distinct measurements issued. A design where the two differ by a factor of eight has a level where it needed an edge, and §9's counter pair is what exposes it.

20. The Qualification Run Counter

// ILLUSTRATIVE. Correct consecutive-pass qualification for one lane. Four
// requirements, and each of the next four sections is one of them violated.
localparam int RUN_W = $clog2(QUAL_THRESHOLD + 1);   // §25 — represent the threshold
 
logic [RUN_W-1:0] pass_run_q;
logic             qualified_q;
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    pass_run_q  <= '0;
    qualified_q <= 1'b0;
  end else if (attempt_start) begin
    pass_run_q  <= '0;                 // §37 — evidence is attempt-scoped
    qualified_q <= 1'b0;
  end else if (sample_valid) begin     // (1) EVENT-qualified — §21
    if (sample_pass) begin
      if (pass_run_q != RUN_W'(QUAL_THRESHOLD))          // (3) SATURATE — §26
        pass_run_q <= pass_run_q + RUN_W'(1);
      if (pass_run_q == RUN_W'(QUAL_THRESHOLD - 1))
        qualified_q <= 1'b1;
    end else begin
      pass_run_q  <= '0;               // (2) a FAILING sample resets the run
      qualified_q <= 1'b0;
    end
  end
  // (4) NO else — an idle cycle HOLDS the run. §23.
end

Four requirements, and the comments number them because each has its own failure section.

(1) Event-qualified. The counter moves only on sample_valid. §21.

(2) A failing sample resets the run — not an idle cycle, and not a timeout.

(3) Saturating at the threshold. §26.

(4) No else branch. An idle cycle leaves the counter alone. §23 is the version with an else.

And the attempt_start clear is requirement zero, easy to miss: evidence is scoped to an attempt (21.1 §6), so a new attempt starts from nothing. §38 is what happens without it.

21. Wrong Qualification 1 — Counting Every Cycle

// WRONG — increments whenever pass is high, sample or no sample.
always_ff @(posedge clk)
  if (sample_pass) pass_run_q <= pass_run_q + 1'b1;
  else             pass_run_q <= '0;

Worked, with §19's stale valid and a threshold of 4.

One measurement. Four counts. The lane is qualified.

Four properties.

It fails in the permissive direction — the lane qualifies when it should not, so training succeeds and the link is bad (§19).

It is invisible in a passing regression. Everything comes up; nothing asserts. The failure appears as data errors under traffic, several layers away.

The two bugs compound. §19's level and §21's unqualified counter are independently wrong and each alone is survivable: a level with an event-qualified counter counts once, and an unqualified counter with a proper pulse counts correctly. Together they multiply.

And the discriminating observation is samples_valid versus distinct issues (§19), or §22's assertion, which catches it directly.

22. SVA — the Run Changes Only on a Sample

// MANDATORY. The counter moves only when a sample event occurs. This catches
// §21 directly, and it is one line.
property p_run_changes_only_on_sample;
  @(posedge clk) disable iff (!por_n)
    (pass_run_q != $past(pass_run_q)) |-> ($past(sample_valid) || $past(attempt_start));
endproperty
a_run_changes_only_on_sample: assert property (p_run_changes_only_on_sample);
 
// MANDATORY. And the reverse direction: a valid passing sample below threshold
// MUST advance the run. Catches a counter that is stuck rather than miscounting.
property p_passing_sample_advances_run;
  @(posedge clk) disable iff (!por_n)
    (sample_valid && sample_pass && (pass_run_q < RUN_W'(QUAL_THRESHOLD)))
      |=> (pass_run_q == RUN_W'($past(pass_run_q) + 1));
endproperty
a_passing_sample_advances_run: assert property (p_passing_sample_advances_run);

Why both directions are needed. The first catches counting too much (§21); the second catches counting too little — a lane whose samples arrive and pass and whose counter never advances, which is a distinct bug with the opposite symptom and the same report.

23. Wrong Qualification 2 — the Bubble Resets the Run

// WRONG — the else branch resets the run on an IDLE cycle.
always_ff @(posedge clk)
  if (sample_valid && sample_pass) pass_run_q <= pass_run_q + 1'b1;
  else                             pass_run_q <= '0;   // fires on NO SAMPLE too

Worked, with a sample cadence that has bubbles — which is normal.

Every measurement passed. The lane never qualifies.

Five properties, and this is the mirror of §21 in every respect.

It fails in the restrictive direction — training never converges on a link that is electrically fine. So the symptom is 21.1's hang, not a bad link.

The failure depends on the cadence, not the signal. A sample stream with no bubbles converges; one with a bubble every other sample never does. So it can appear when a measurement engine is retimed, or at a different rate, or on a different product — with no change to the training logic.

§7's waveform is specifically drawn to eliminate it. Cycles 3 and 6 show the counter holding across idle cycles; a design with this bug would reset there, and that is a one-cycle observation.

The two bugs are indistinguishable from the run counter alone and trivially distinguishable from the run counter alongside sample_valid — which is why §7 plots both.

And the fix is the absence of a branch. No else. That is genuinely hard to review, because reviewers look for wrong code rather than for code that should not be there — which is why §24's assertion earns its place.

24. SVA — a Bubble Preserves the Run

// MANDATORY. An idle cycle does not disturb the qualification run. This is the
// property that catches §23, and it cannot be satisfied by the buggy design.
property p_no_sample_preserves_run;
  @(posedge clk) disable iff (!por_n)
    (!sample_valid && !attempt_start) |=> $stable(pass_run_q);
endproperty
a_no_sample_preserves_run: assert property (p_no_sample_preserves_run);
 
// MANDATORY. And the qualified bit is equally stable across a bubble.
property p_no_sample_preserves_qualified;
  @(posedge clk) disable iff (!por_n)
    (!sample_valid && !attempt_start) |=> $stable(qualified_q);
endproperty
a_no_sample_preserves_qualified: assert property (p_no_sample_preserves_qualified);

25. The Threshold Width

// ILLUSTRATIVE. The run counter must represent 0..QUAL_THRESHOLD inclusive,
// which is QUAL_THRESHOLD+1 values. 19.5 §10's argument, in the training path.
localparam int RUN_W = $clog2(QUAL_THRESHOLD + 1);
 
// And the elaboration check, because a truncated threshold produces a LEGAL
// value that no runtime assertion can object to (19.5 §12).
initial begin
  assert (QUAL_THRESHOLD >= 1)
    else $fatal(1, "QUAL_THRESHOLD must be at least 1");
  assert (RUN_W >= $clog2(QUAL_THRESHOLD + 1))
    else $fatal(1, "RUN_W=%0d cannot represent QUAL_THRESHOLD=%0d",
                RUN_W, QUAL_THRESHOLD);
end

The failure this prevents, worked with QUAL_THRESHOLD = 8.

$clog2(8) is 3. A 3-bit counter represents 0 through 7. The comparison pass_run_q == 8 is never true — the value is unrepresentable.

The lane never qualifies, on a perfect link, and no assertion fires because 7 is a legal value and the counter never leaves its range. 19.5 §11's classic parameterisation bug, in a different structure.

And it fails only for power-of-two thresholds, so a threshold of 6 or 10 works and 8 does not — which is exactly the value a product is likely to pick.

26. Wrong Counter — the Wrapping Run

// WRONG — the run counter wraps instead of saturating.
if (sample_valid && sample_pass) pass_run_q <= pass_run_q + 1'b1;   // no guard

Two failure modes, depending on where the threshold sits.

If the threshold is below the maximum, the counter passes it, keeps counting, wraps to zero, and the lane qualifies intermittently — qualified for one cycle every 2^RUN_W passing samples. Training may converge, slowly and unreliably.

If the threshold is the maximum, the counter reaches it, wraps on the next passing sample, and the qualified bit may be missed entirely depending on how the comparison is written.

Three properties.

Both modes are intermittent, which is the worst class. The link comes up sometimes. A test that passes 90% of the time reads as flaky silicon rather than as a counter without a guard.

The guard is one comparison (§20's requirement 3), and it converts an intermittent failure into a deterministic one.

And the diagnostic signature is a pass_run value that decreases without a failing sample — which is exactly what §27 asserts.

27. SVA — the Run Saturates

// MANDATORY. The run never wraps: it saturates at the threshold.
property p_run_saturates;
  @(posedge clk) disable iff (!por_n)
    (pass_run_q == RUN_W'(QUAL_THRESHOLD)) |=>
      ((pass_run_q == RUN_W'(QUAL_THRESHOLD))
        || (sample_valid && !sample_pass) || attempt_start);
endproperty
a_run_saturates: assert property (p_run_saturates);
 
// MANDATORY. The run decreases only on a failing sample or an attempt boundary.
// This catches §26 directly, and it is the cheapest of the three.
property p_run_decreases_only_on_fail;
  @(posedge clk) disable iff (!por_n)
    (pass_run_q < $past(pass_run_q))
      |-> ($past(sample_valid && !sample_pass) || $past(attempt_start));
endproperty
a_run_decreases_only_on_fail: assert property (p_run_decreases_only_on_fail);
 
// MANDATORY. Once qualified, a lane stays qualified until a failing sample or
// an attempt boundary — it does not flicker.
property p_qualified_stable;
  @(posedge clk) disable iff (!por_n)
    (qualified_q && !$past(qualified_q, 0))
      |=> qualified_q until_with ((sample_valid && !sample_pass) || attempt_start);
endproperty

The second property is the workhorse. "The run decreased and no failing sample arrived" is impossible in a correct implementation — which is 21.1 §1's definition of the observation to look for, and it catches both of §26's modes.

28. Candidate Versus Active

8.4 §4's three copies, as a debug table. Six questions, and the first no names the stage.

Two notes.

Rows 4 and 5 differ, and the gap between them is a real bug class. A candidate that scored valid and for which no commit was ever requested means the commit condition has an extra term — a quiesce requirement never satisfied, a validation that never completes, an enable never set. 8.4 §11's commit gating.

And rows 5 and 6 differ too. A commit requested and refused is the commit path; a commit requested and accepted with the active register unchanged is a write that did not land, which is a different and much simpler bug.

29. Candidate History

// ILLUSTRATIVE. A bounded ring of candidate EVALUATIONS. Not every sample —
// one record per candidate evaluated, which is tens of records, not thousands.
typedef struct packed {
  logic [TIME_W-1:0]  timestamp;
  logic [ATT_W-1:0]   attempt;
  logic [CAND_W-1:0]  candidate;
  logic [SCORE_W-1:0] score;            // where the architecture scores
  logic [NUM_LANES-1:0] lanes_passed;   // WHICH lanes passed on this candidate
  logic               accepted;
  logic [RSN_W-1:0]   reject_reason;    // §13's taxonomy
} cand_eval_t;
 
cand_eval_t cand_hist [CAND_HIST_DEPTH];
logic [CH_PTR_W-1:0] cand_hist_wr_q;
logic                cand_hist_wrapped_q;    // 21.1 §12's honesty flag
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    cand_hist_wr_q      <= '0;
    cand_hist_wrapped_q <= 1'b0;
  end else if (cand_eval_complete) begin      // one record per EVALUATION
    cand_hist[cand_hist_wr_q] <= '{ timestamp: time_q, attempt: attempt_q,
                                    candidate: cand_q, score: score_q,
                                    lanes_passed: lane_pass_mask,
                                    accepted: cand_accepted,
                                    reject_reason: reject_reason_q };
    cand_hist_wr_q <= cand_hist_wr_q + CH_PTR_W'(1);
    if (cand_hist_wr_q == CH_PTR_W'(CAND_HIST_DEPTH-1)) cand_hist_wrapped_q <= 1'b1;
  end
end

Three readings the lanes_passed mask enables that a score alone cannot.

One lane clear on every candidate → that lane is the blocker, and no candidate will ever be accepted. §47's shape, visible from the candidate history rather than from per-lane counters.

Different lanes clear on different candidates → the candidates are trading lanes off against each other, which means the search space may genuinely have no point where all lanes pass — a link-level finding, not a lane-level one.

And all lanes passing with accepted == 0 → the rejection is not about lanes at all. Read reject_reason: a score threshold, a validation step, a stability requirement.

30. Wrong Instrumentation — the Active Setting Only

// WRONG — the only calibration/training state exposed.
assign dbg_active_setting = active_setting_q;

What the debugger receives: an active setting that never changes.

The conclusion drawn: the search never ran.

What actually happened, invisibly: candidates 0 through 11 were generated, applied, measured and rejected — eleven of them because lane 12 never passed, and candidate 7 because its score was marginal.

Four properties.

The conclusion is not merely incomplete; it is wrong. The search ran vigorously. A team told "the search never ran" investigates the generator, the enable and the sequencer — three healthy blocks.

The real finding was one lane, available from lanes_passed in the candidate history (§29) or from lane_qualified (§10).

The active register is correctly unchanged. 8.4 §11's commit gating requires a valid result, and there was none. The design is behaving exactly as specified, which is why nothing asserts.

And this is §5's misconception in instrumentation form. Exposing only the last stage of a six-stage pipeline (§28) hides five stages — and the five hidden ones are where the failure is.

31. Search-Space Exhaustion

A candidate search has three possible outcomes, and a design must distinguish them.

The middle row is the one that must be distinguishable from the third, and it is what a visited record buys:

// ILLUSTRATIVE. A visited bitmap where the candidate space is small enough,
// or a bounded evaluation counter where it is not.
logic [NUM_CANDIDATES-1:0] cand_visited_q;   // small spaces
logic [EVAL_W-1:0]         evals_this_attempt_q;
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n)              cand_visited_q <= '0;
  else if (attempt_start)  cand_visited_q <= '0;        // attempt-scoped (§37)
  else if (cand_eval_complete) cand_visited_q[cand_q] <= 1'b1;
end
 
assign space_exhausted = (cand_visited_q == '1);   // all evaluated, none accepted

Two notes.

space_exhausted is a diagnosis, not a failure to hide. A link with no operating point across the whole candidate space is telling you something real — about the channel, the package, the settings range, or the scoring threshold — and reporting it explicitly is far better than retrying forever.

And where the space is too large for a bitmap, the evaluation count plus the visited span is enough: evals_this_attempt far exceeding the space size means candidates are being revisited (§33), which is the discriminator that matters.

32. Wrong Search — the Pointer That Advances Only on Success

// WRONG — the candidate pointer advances only when a candidate is ACCEPTED.
always_ff @(posedge clk)
  if (cand_accepted) cand_q <= cand_q + 1'b1;

Worked. Candidate 0 is evaluated and rejected.

The search evaluates candidate 0 forever.

Five properties, and this is one of the chapter's three flagship failures.

There is abundant activity and zero progress. Samples arrive, measurements complete, evaluations finish, counters increment. Every activity-based watchdog is satisfied — which is §44's whole subject.

The candidate history makes it obvious and nothing else does. Twelve records, all candidate 0 (§29). From active_setting alone (§30) it looks like a search that never started; from the history it is a search that never advances.

The condition is inverted from correct. The pointer should advance when an evaluation completes, regardless of outcome — acceptance should stop the search, not advance it. §34.

It is a plausible thing to write. "Move to the next candidate when this one is accepted" sounds like sequencing and is exactly backwards. A reviewer reading the line in isolation may not catch it, which is why §35's liveness property earns its place.

And it will never time out under an activity watchdog, so the failure presents as 21.1's hang with no timeout — the hardest shape to localise without the history.

33. Candidate Advance, Correctly

// RIGHT — advance on a COMPLETED evaluation; ACCEPTANCE stops the search.
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    cand_q          <= '0;
    search_done_q   <= 1'b0;
  end else if (attempt_start) begin
    cand_q          <= '0;
    search_done_q   <= 1'b0;
  end else if (cand_eval_complete) begin
    if (cand_accepted) begin
      search_done_q <= 1'b1;             // STOP — we have a result
    end else if (cand_q == CAND_W'(NUM_CANDIDATES-1)) begin
      search_done_q <= 1'b1;             // STOP — exhausted (§31)
      space_exhausted_q <= 1'b1;         // and SAY SO
    end else begin
      cand_q <= cand_q + CAND_W'(1);     // advance on REJECTION
    end
  end
end

Three properties.

The advance is on cand_eval_complete && !cand_accepted — the rejection path. That is the arrow §6's figure draws as QUALIFY → ISSUE, and §32 is the design where the figure has it and the RTL does not.

Exhaustion sets an explicit flag rather than wrapping silently. A search that wraps to candidate 0 and starts again is indistinguishable from §32 in the history unless the wrap is recorded.

And both terminations set search_done. 8.4 §16's bounded-search requirement: a search with no termination condition is a liveness bug, and the two conditions here are found and exhausted.

34. SVA — a Rejected Candidate Causes Progress

// MANDATORY, with its assumption named. This is the property that catches §32,
// and it is a LIVENESS property so it needs an assumption (20.3 §42).
//
//   ASSUMPTION: evaluations complete — samples eventually arrive for the
//   candidate under test. Without it the property fails on a measurement
//   failure, which is contract 1's problem and not the search's.
assume property (@(posedge clk) s_eventually cand_eval_complete);
 
property p_rejection_advances_search;
  @(posedge clk) disable iff (!por_n)
    (cand_eval_complete && !cand_accepted && !space_exhausted)
      |=> ($changed(cand_q) || search_done_q || attempt_start);
endproperty
a_rejection_advances_search: assert property (p_rejection_advances_search);
 
// MANDATORY. The safety half: a candidate is not re-evaluated within an attempt
// unless the architecture explicitly permits re-measurement. This catches §32
// without needing the liveness assumption at all.
property p_no_candidate_revisit_within_attempt;
  @(posedge clk) disable iff (!por_n)
    cand_eval_complete |-> !cand_visited_q[cand_q];
endproperty
a_no_candidate_revisit_within_attempt:
  assert property (p_no_candidate_revisit_within_attempt);
 
// MANDATORY. Acceptance stops the search — it does not advance it.
property p_acceptance_stops_search;
  @(posedge clk) disable iff (!por_n)
    (cand_eval_complete && cand_accepted) |=> search_done_q;
endproperty
a_acceptance_stops_search: assert property (p_acceptance_stops_search);

Why the second property is the more useful of the three. It is safety, not liveness — so it needs no assumption, fails at a specific cycle, and catches §32 on the second evaluation of candidate 0 rather than at the end of a run. 20.3 §44's bounded-form argument: a safety property that implies the liveness one is worth more than the liveness property.

And where an architecture does permit re-measurement — averaging several passes over one candidate — the property becomes a bounded one: at most N evaluations per candidate per attempt. The bound must be stated; unbounded re-measurement is §32.

35. Oscillation Detection

A search that alternates between two candidates indefinitely is a distinct failure from §32's stuck pointer, and it has different causes.

// ILLUSTRATIVE. A small repeat detector over the recent candidate sequence.
// Two records are enough to catch an A-B-A-B pattern.
logic [CAND_W-1:0] cand_prev_q, cand_prev2_q;
logic [CNT_W-1:0]  oscillation_count_q;
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    oscillation_count_q <= '0;
  end else if (cand_eval_complete) begin
    cand_prev2_q <= cand_prev_q;
    cand_prev_q  <= cand_q;
    // A-B-A: this candidate equals the one two evaluations ago.
    if ((cand_q == cand_prev2_q) && (cand_q != cand_prev_q)
        && (oscillation_count_q != '1))
      oscillation_count_q <= oscillation_count_q + CNT_W'(1);
  end
end

Four causes, and the count alone does not distinguish them — but it tells you to look:

The last row is the one that requires two-sided evidence. Two devices each re-tuning in response to the other's re-tuning produces an oscillation neither side can diagnose alone — and it is 20.7 §33's two-sided attribution problem in the training domain.

36. The Training Attempt Epoch

Every retry starts a new attempt, and evidence belongs to exactly one. 21.1 §6's ledger records local training evidence as attempt-scoped, and this is that scope made concrete.

// ILLUSTRATIVE. A LOCAL control-and-debug construct. No claim is made that any
// training exchange carries an epoch field (§3).
logic [EPOCH_W-1:0] train_epoch_q;
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n)            train_epoch_q <= '0;
  else if (attempt_start) train_epoch_q <= train_epoch_q + EPOCH_W'(1);
end

Three requirements on the width and the scope.

EPOCH_W must be wide enough that the tag cannot wrap within the maximum in-flight lifetime of a sample. 19.5 §28's argument: two bits is almost certainly too few, and the cost of more is trivial against the failure they prevent.

It advances on attempt_start, atomically with the evidence clear (§20's requirement zero). Advancing it separately from the clear leaves a window in which the epoch has moved and the old evidence has not — which is the window §37 lives in.

And it is not reset by anything narrower than a power cycle, or an epoch could repeat and a stale sample would match again.

37. Flagship Bug — the Stale Sample After Retry

One of the chapter's three flagship failures, and the one that makes training succeed wrongly.

Worked. Attempt 7 issues a measurement for lane 12 against candidate 3. The measurement engine is slow, or the sample path is deep, or a response is delayed. Attempt 7 times out and attempt 8 begins, clearing the run counters.

Lane 12 qualifies with one fewer real measurement than the threshold requires.

Five properties, and the first is why it is worth a flagship section.

It fails in the permissive direction. The lane qualifies too easily, training converges, and the link comes up on evidence that includes a measurement of a different candidate under a different attempt's conditions. The failure appears later, as data errors under traffic — a 21.1 §37 family-G shape.

The stale sample measured a different candidate. Attempt 7 was testing candidate 3; attempt 8 is testing candidate 0. So the evidence is not merely old, it is about something else entirely — and nothing in the sample distinguishes them without the epoch.

The window is small and the consequence is not. The sample must land after the attempt boundary — a window of a few cycles to a few hundred, depending on the measurement pipeline's depth. Miss it and everything is fine; hit it and a lane is falsely qualified.

It requires a retry to reproduce, which is the thing most functional tests are configured not to have. The bug lives at the intersection of the retry path and the sample path, and those are usually verified separately.

And the mirror case is silent. If the stale sample happens to fail, it resets attempt 8's run — so a lane that would have qualified takes longer, with no assertion and no symptom other than slower convergence. That direction has no evidence at all except a higher-than-expected evaluation count.

38. Epoch-Tagged Samples

// ILLUSTRATIVE. The guard, and it is three lines. The epoch travels INSIDE the
// sample bundle (§16) so it cannot be separated from the measurement it tags.
assign sample_accept = sample_pipe_q[SAMPLE_PIPE_DEPTH-1].valid
                    && (sample_pipe_q[SAMPLE_PIPE_DEPTH-1].epoch == train_epoch_q);
 
assign sample_stale  = sample_pipe_q[SAMPLE_PIPE_DEPTH-1].valid
                    && (sample_pipe_q[SAMPLE_PIPE_DEPTH-1].epoch != train_epoch_q);
 
// And the COUNTER, because a rejected sample is evidence that the guard is
// being exercised — silently dropping it discards that (19.5 §30).
always_ff @(posedge clk or negedge por_n)
  if (!por_n)            stale_sample_count_q <= '0;
  else if (sample_stale && (stale_sample_count_q != '1))
                         stale_sample_count_q <= stale_sample_count_q + 1'b1;

Three notes.

The qualification counter uses sample_accept, never sample_valid. §20's requirement (1) becomes epoch-qualified as well as event-qualified — and both qualifications are necessary.

stale_sample_count_q counts correct rejections. Not an error. But its rate is diagnostic: stale samples immediately after a retry are expected; stale samples during steady training mean the epoch is advancing when it should not, which points at something other than attempt_start writing it.

And where the architecture provides no way to tag a sample with an epoch, the implementable fallback is a quarantine window after each attempt boundary — discard samples for longer than the maximum sample-path latency. It is strictly weaker (19.5 §30's argument): it also discards legitimate new-attempt samples that arrive inside the window, and it depends on a latency bound. State the assumed bound and assert it.

39. SVA — Stale Evidence Is Ignored

// MANDATORY. A sample from a dead attempt changes nothing.
property p_stale_sample_inert;
  @(posedge clk) disable iff (!por_n)
    sample_stale |=> ($stable(pass_run_q) && $stable(qualified_q));
endproperty
a_stale_sample_inert: assert property (p_stale_sample_inert);
 
// MANDATORY. The epoch advances atomically with the evidence clear, so no
// window exists in which one has moved and the other has not (§36).
property p_epoch_atomic_with_clear;
  @(posedge clk) disable iff (!por_n)
    $changed(train_epoch_q) |-> attempt_start;
endproperty
a_epoch_atomic_with_clear: assert property (p_epoch_atomic_with_clear);
 
// MANDATORY. And the clear actually happens: after an attempt boundary the
// run and the qualified bit are zero.
property p_attempt_start_clears_evidence;
  @(posedge clk) disable iff (!por_n)
    attempt_start |=> ((pass_run_q == '0) && !qualified_q);
endproperty
a_attempt_start_clears_evidence: assert property (p_attempt_start_clears_evidence);
 
// MANDATORY. The epoch is monotonic — a decrease means a reset reached a scope
// it should not have (19.6 §27), and every attempt annotation becomes unsound.
property p_train_epoch_monotonic;
  @(posedge clk) disable iff (!por_n)
    ($changed(train_epoch_q))
      |-> (train_epoch_q == EPOCH_W'($past(train_epoch_q) + 1));
endproperty
a_train_epoch_monotonic: assert property (p_train_epoch_monotonic);

The third property is the one that fails on §37 directly. A design that advances the epoch and does not clear the run has a window in which stale evidence is epoch-current — so the guard passes and the evidence is still wrong. Clearing and tagging are both required, and only asserting both catches the combination.

40. Peer Restart

8.5 §14 established peer restart as a bring-up hazard. In training it invalidates evidence in a way that is invisible locally.

What a peer restart does to local training state:

And the debug problem: a peer restart may be invisible locally. Unless the design observes and records it, the local side sees measurements that suddenly behave differently with no local cause — which reads as an intermittent electrical problem.

So the instrumentation requirement is one counter and one record:

// ILLUSTRATIVE. A peer restart is a first-class debug event (21.1 §13).
logic [CNT_W-1:0]  peer_restart_count_q;
logic [TIME_W-1:0] last_peer_restart_time_q;
logic [ATT_W-1:0]  last_peer_restart_attempt_q;

Reading it against the candidate history is the diagnosis: evaluations whose scores changed character at a timestamp matching a peer restart are not a local problem at all.

41. Wrong Peer-Restart Handling — Keeping Qualified Lanes

// WRONG — an optimisation that retrains only the failing lanes.
always_ff @(posedge clk)
  if (peer_restart_observed) begin
    // "Lanes 0-30 already qualified; only retrain lane 31."
    for (int l = 0; l < NUM_LANES; l++)
      if (!lane_qualified_q[l]) lane_run_q[l] <= '0;    // clear only failures
  end

Worked. Thirty-one lanes qualified against the peer's previous configuration. The peer restarts and comes back with a different negotiated configuration — a different rate, a different width, different settings.

Lane 31 is retrained against the new configuration. Lanes 0–30 carry evidence from the old one.

Four properties.

The evidence set is now mixed, and that is the specific hazard 8.5 §6 names. Thirty-one lanes validated under one set of conditions and one under another; the aggregate "all lanes qualified" is true and means nothing.

Training converges and the link is wrong. Another permissive-direction failure, and it produces a link that works for lane 31's conditions and not for the rest.

The optimisation is attractive and that is why the bug exists. Retraining thirty-one qualified lanes is expensive, and skipping it is an obvious saving. The saving is only valid if the conditions are unchanged, and a peer restart is precisely the event that changes them.

And the correct behaviour is to advance the epoch (§36), which invalidates all attempt-scoped evidence by construction — so the optimisation becomes unrepresentable rather than merely discouraged.

42. Evidence Lifetime

The chapter's organising state-lifetime table, and every row is a debug question.

Three readings.

Rows 1–3 are attempt-scoped and everything else is not. That split is the whole of §37's bug: a design that scopes the run counter to the attempt and the epoch to something else has two lifetimes where it needed one.

Rows 5–8 are deliberately retry-surviving. They are the debug history, and clearing them on retry destroys the sequence that §12 exists to produce. 14.2 §14's capture-before-retrain rule, per lane.

And the table is the specification for the reset logic. A design whose retry path clears rows 4–10 has an optimisation that removed its own diagnostics — which is the most common instrumentation regression in a training block, because clearing everything on retry looks like hygiene.

43. Progress Metrics for Training

"The phase changed" is not a progress metric for training, because training does not change phase while it works (21.1 §19).

Five candidate progress events, and each has a different blind spot:

Two consequences.

No single metric is sufficient, and rows 1 and 2 are the ones most often chosen because they are the easiest to produce. Both are satisfied by §32's infinite re-evaluation of candidate 0 — which is why that bug never times out.

Row 3 is the strongest single metric and it is monotonic within an attempt: the qualified-lane count only increases until something invalidates it. A watchdog on "the qualified-lane count has not increased" catches §32, catches §23, and catches §47 — and it is one counter.

44. Stuck Versus Slow

The third row is the discriminator, and §43's argument is why. A design that measures progress by sample arrival cannot distinguish these two columns at all — and a watchdog built on it either fires on a converging link or never fires.

45. Per-Lane Age

The instrumentation that makes §47 detectable, and a global watchdog cannot substitute for it.

// ILLUSTRATIVE. Saturating age since the last valid sample, per lane.
logic [AGE_W-1:0] lane_age_q [NUM_LANES];
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    for (int l = 0; l < NUM_LANES; l++) lane_age_q[l] <= '0;
  end else begin
    for (int l = 0; l < NUM_LANES; l++) begin
      if (lane_sample_valid[l])              lane_age_q[l] <= '0;
      else if (lane_age_q[l] != '1)          lane_age_q[l] <= lane_age_q[l] + AGE_W'(1);
      // else HOLD at all-ones — 21.1 §21's saturation argument
    end
  end
end
 
// The aggregate the watchdog should use: the OLDEST lane, not "any activity".
always_comb begin
  max_lane_age = '0;
  for (int l = 0; l < NUM_LANES; l++)
    if (lane_required[l] && (lane_age_q[l] > max_lane_age))
      max_lane_age = lane_age_q[l];
end

Two notes.

The watchdog uses max_lane_age, not an OR of activity. That one substitution is the difference between detecting §47 and not detecting it — and it is a max-reduce over a required-lane mask.

And lane_required matters. A link operating at a degraded width has lanes that are legitimately silent (14.4); including them in the max makes the watchdog fire on a correct degraded link, which is 21.1 §19's false-timeout failure.

46. Flagship Failure — the One Silent Lane

The second flagship, and it is the one a global watchdog is structurally unable to detect.

Worked. 32 lanes. Lane 12's sample path is dead — a bump, a receiver, an enable, a mux select.

Five properties.

Training never finishes and never times out. The permissive watchdog is satisfied by 31 active lanes; the design searches forever. 21.1 sees a hang with no timeout — the hardest shape.

Three independent instruments name lane 12 immediately, and any one is sufficient: lane_seen, lane_age, or the candidate history's mask. The failure is trivially diagnosable and only if the instrumentation is per lane.

The candidate history adds something the bitmaps do not: it proves no candidate would ever help. Bit 12 clear on every evaluated candidate means the search cannot succeed, so retrying is pointless and the correct action is to report the lane — or to degrade the width if the architecture permits (14.4).

A global activity watchdog is not merely weak here — it is anti-diagnostic. It actively confirms that training is alive, which is true and misleading, and it is why §45's max-reduce exists.

And this is the failure that most strongly justifies per-lane instrumentation's cost. Two bits plus an age counter per lane, against a failure mode that is otherwise invisible until somebody reads a waveform of 32 lanes.

47. The Timestamp Alternative

Where a counter per lane is too expensive, a shared timestamp plus a per-lane last-seen value is cheaper — with one caveat.

// ILLUSTRATIVE. One free-running timestamp, plus a last-seen per lane.
// Cheaper when NUM_LANES is large and AGE_W would be wide.
logic [TIME_W-1:0] now_q;
logic [TIME_W-1:0] lane_last_seen_q [NUM_LANES];
logic              lane_ever_seen_q [NUM_LANES];   // §10, and it matters here
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    now_q <= '0;
    for (int l = 0; l < NUM_LANES; l++) lane_ever_seen_q[l] <= 1'b0;
  end else begin
    now_q <= now_q + TIME_W'(1);                    // wraps — see the caveat
    for (int l = 0; l < NUM_LANES; l++)
      if (lane_sample_valid[l]) begin
        lane_last_seen_q[l] <= now_q;
        lane_ever_seen_q[l] <= 1'b1;
      end
  end
end

The caveat, and it is why lane_ever_seen is in the snippet. now_q wraps, so now - last_seen is meaningless once the interval exceeds the timestamp range — and a lane that never produced a sample has a last_seen of zero that is indistinguishable from one that produced a sample exactly one wrap ago.

Two resolutions, and both are needed:

lane_ever_seen disambiguates the never-seen case outright — one bit per lane, and it is the case §46 turns on.

And the age must be clamped rather than computed raw: software reading the pair should treat any interval beyond a defined maximum as "at least maximum", which is saturation moved from hardware to the reader. A design that reports a raw wrapped difference will report a fresh lane during an ancient silence — 21.1 §21's wrapping-age trap, relocated.

48. Training Failure Classification

The chapter's centerpiece table. Seven classes, each with a distinct signature and a distinct owner.

Three properties of the table.

The signature column is what makes it usable. Every row is identified by a register reading, not by a hypothesis — so the table is a lookup rather than a discussion.

Two rows are not defects. Space exhausted is a real finding about the channel, and committed, still not done means training succeeded and something downstream is the blocker. A classification table without non-defect rows sends investigations after non-bugs (21.1 §40's cause-tree argument).

And three rows point at physical causes while the other nine point at logic. The split matters because the two go to different teams, and rows 3 and 5 are the ones that can be either — which is why their owner column names both.

49. Retry Counting, Per Cause

// ILLUSTRATIVE. One counter per retry CAUSE, saturating. A single total is
// §50's failure.
typedef enum logic [RC_W-1:0] {
  RC_MEASURE_TIMEOUT = 'd0,   // no sample within the window
  RC_QUAL_FAILED     = 'd1,   // run never reached threshold
  RC_CAND_EXHAUSTED  = 'd2,   // search space exhausted (§31)
  RC_PEER_RESTART    = 'd3,   // §40
  RC_VALIDATION      = 'd4,   // candidate valid, validation refused
  RC_COMMIT_REFUSED  = 'd5,   // §28 row 6
  RC_EXTERNAL        = 'd6    // a higher layer requested a retrain
} retry_cause_e;
 
logic [CNT_W-1:0] retry_count_q [NUM_RETRY_CAUSES];

Three readings the split enables.

RC_MEASURE_TIMEOUT dominating → contract 1, and §46's silent lane is the likely shape.

RC_QUAL_FAILED dominating → contract 2, and §7's rate problem or §23's bubble bug.

RC_PEER_RESTART dominating → not a local problem at all, and the investigation moves to the peer with a count as evidence.

50. Wrong Retry Metric — the Total

// WRONG — one counter.
if (retry_event) retry_count_q <= retry_count_q + 1'b1;

What the debugger receives: retry_count == 100.

Four questions it cannot answer.

Same cause every time, or a hundred different ones? A hundred measurement timeouts is one bug; a mixture of timeouts, peer restarts and exhaustions is a system with several problems, and the responses differ.

Same lane? §12's per-attempt first-failing-lane sequence answers this and the total does not.

Is it converging? A hundred retries with the qualified-lane count rising is progress; with it flat, it is §32.

And is it even local? A hundred retries all caused by peer restarts is the peer's bug counted a hundred times on the local side — and the total presents it as a local failure.

51. Flagship Trace 1 — the Candidate That Never Advances

Illustrative candidate history, read directly out of §29's ring.

Five readings, and the second is the one that surprises people.

One candidate, sixteen evaluations, no advance. §32's signature, and it is unmistakable in the history.

Every lane passed, every time. lanes_passed is all-ones throughout — so this is not a lane problem and not a measurement problem. The lanes are fine; the candidate is being rejected for some other reason, and rejected consistently.

The reject reason is LF_CAND_REJECTED, so the scoring rejected it. Combined with all lanes passing, that points at a score threshold rather than at pass/fail measurement — a subtly different investigation.

Two bugs are visible at once, and that is worth noticing: the pointer does not advance (§32) and candidate 0's score does not clear the threshold. Fixing the pointer alone would move the search to candidate 1 and may well converge; fixing only the threshold would accept candidate 0, which may be the wrong operating point. Both are real.

And active_setting was unchanged throughout, so §30's instrumentation would have reported "the search never ran" — with sixteen evaluations in the history.

52. Flagship Trace 2 — the Silent Lane

Illustrative. §46's failure as a register snapshot rather than a narrative.

Four readings.

samples_valid[12] == 0 with samples_failed[12] == 0 is the decisive pair. Nothing arrived — so this is a path problem, not a signal-quality problem, and §9's counter pair is what distinguishes them. A design with a single "error count" would read zero here and suggest a healthy lane.

first_fail_lane == 12 at attempt 1 says the failure was present from the start (21.1 §23's discriminator) — static, not degrading.

Bit 12 clear in all forty candidate records proves retrying is futile, which converts "keep trying" into "report the lane or degrade the width."

And global activity asserted alongside a saturated max_lane_age is the contradiction that identifies the watchdog as the wrong one — 21.1 §21's two-counter contradiction pattern.

53. Flagship Trace 3 — the Stale Sample After Retry

Illustrative. §37's failure, with the evidence that catches it.

And the same trace without the guard:

Four readings.

The guarded version qualifies at t=1,150 and the unguarded at t=1,138 — twelve cycles earlier, on one fewer measurement. The difference is invisible without the epoch, and it is the whole bug.

stale_sample_count incrementing at t=1,102 is the guard working (§38). A run where it is zero has not exercised the guard, which is a coverage gap rather than a clean result.

The stale sample measured candidate 3 and was counted toward candidate 0's evidence. Not merely old — about a different setting entirely, which is why an age-based quarantine is a weaker guard than an epoch tag.

And the mirror case has no trace at all. Had the stale sample failed, it would have reset attempt 8's run at t=1,102 and convergence would simply have taken longer — with no assertion, no counter and no symptom beyond a higher evaluation count.

54. The Event Alphabet

Long training traces become readable when reduced to a small closed alphabet — 21.1 §43's milestone technique at training granularity.

And the diagnostic value is that each failure has a distinct string:

Two properties.

The last row of the table is the one that cannot be produced any other way. "Q appeared, then V, and K never did" is a commit-condition failure stated in six characters — and it is 21.1 §41's differential technique with a training alphabet.

And counting symbols is itself diagnostic. P count minus threshold times Q count should be small in a healthy run; a large excess means passes are being discarded — by a bubble reset (§23), a stale rejection (§38), or a candidate change mid-run.

55. Differential Tracing

Align by event, never by cycle (21.1 §41).

KNOWN GOOD                          FAILING
A  attempt 1                        A  attempt 1
C  candidate 0                      C  candidate 0
M  measure                          M  measure
P  pass                             P  pass
P  pass                             P  pass
P  pass                             F  FAIL          <- first divergence
P  pass                             P  pass
Q  lane 0 qualified                 P  pass
...                                 P  pass
Q  lane 31 qualified                (no Q — run reset by the F)
V  candidate valid                  ...
K  commit requested                 R  retry
D  active changed                   A  attempt 2 ...
 
FIRST DIVERGENCE: an F where the good run had a P.
=> measurement path works (P's present, both runs)
=> qualification counting works (P's accumulate in both)
=> the failure is the RATE of failing samples -> signal quality or threshold
=> and NOT §23, because the good run's P's are contiguous too

Three readings.

The matched prefix eliminates two contracts. A C M P P matching means candidate selection, application, measurement issue and the sample path all work — four stages, eliminated by five characters.

The divergence is an F, not a missing P. A failing sample is a measurement that worked and reported a bad result; a missing sample would be an absent M-to-P pair, which is a completely different class (§48's rows 1 and 2).

And ruling out §23 requires the good trace. If the good run's passes were also interrupted by bubbles and still accumulated, the bubble-reset bug is excluded — which is an elimination the failing trace alone cannot make.

56. Reproducing a Training Failure

// ILLUSTRATIVE. Synchronise on EVENTS, never on delays (21.1 §46).
task reproduce_stale_sample();          // §37
  // 1. A measurement latency long enough to outlive an attempt boundary.
  meas_model.set_latency(LATENCY_EXCEEDING_ATTEMPT_WINDOW);
 
  // 2. Wait for a measurement to be ISSUED — not for a delay.
  wait (evt_seen(EVT_MEASURE_ISSUED));
  saved_epoch = dbg.train_epoch;
 
  // 3. Force the attempt to time out while that measurement is in flight.
  meas_model.withhold_response();
  wait (evt_seen(EVT_ATTEMPT_START));   // the boundary crossed
 
  // 4. NOW release the stale response, tagged with the OLD epoch.
  meas_model.release_response(saved_epoch);
 
  // 5. Positive expectation (20.1 §44): the guard must reject it.
  fork
    begin wait (dbg.stale_sample_count > 0);
          `uvm_info("REPRO", "guard rejected the stale sample", UVM_LOW) end
    begin repeat (REPRO_BOUND) @(posedge clk);
          `uvm_error("REPRO", "stale sample was NOT rejected — §37 reproduced") end
  join_any
  disable fork;
endtask

Three notes.

Step 3 is where the reproduction is made deterministic. The stale sample must be in flight across the attempt boundary — and waiting for EVT_ATTEMPT_START guarantees it, where a delay would land the response on either side at random.

Step 5's expectation is that the guard fires. 20.1 §44's rule: the pass criterion is the stale counter incrementing, not the absence of a crash. A run where the counter stays at zero means the stale sample was accepted — which is the bug, reproduced.

And the reproduction doubles as the coverage item. 20.5 §25's fault-moment dimension: this is a fault injected at a specific observed moment, and it is one of the moments no timer-based injection reaches.

57. The Assertion Inventory

// The last two, written out — both are instrumentation-integrity properties.
property p_qualified_implies_seen;
  @(posedge clk) disable iff (!por_n)
    (lane_qualified_q & ~lane_seen_q) == '0;
endproperty
a_qualified_implies_seen: assert property (p_qualified_implies_seen);
 
property p_training_done_implies_evidence;
  @(posedge clk) disable iff (!por_n)
    training_done |-> ((lane_qualified & lane_required) == lane_required);
endproperty
a_training_done_implies_evidence: assert property (p_training_done_implies_evidence);

Three observations about the inventory's shape.

Rows 2, 6, 10, 13 and 17 are the highest value per line, and four of the five are the cheap form of a property whose obvious form is weaker — an effect form, a safety form standing in for a liveness one, and two negative statements about what must not change.

Row 19 asserts the instrumentation, not the design. A lane qualified without ever having been seen means the bitmaps are lying, and every conclusion drawn from them is void (20.4 §42).

And row 8 is an elaboration check — §25's truncated threshold produces a legal value no runtime assertion can object to, so it must be caught where widths are known (19.5 §12).

58. The Reference Model

A verification model of training must model the contract, not the algorithm.

ILLUSTRATIVE reference model. Per lane, per attempt.
 
  PER LANE:
    samples_observed      counted at the observed sample event
    passes_observed
    expected_run          recomputed from the CONTRACT's rules:
                            + 1 on a valid passing sample
                            0  on a valid failing sample
                            unchanged on no sample
                            0  at an attempt boundary
    expected_qualified    = (expected_run >= QUAL_THRESHOLD)
    epoch_of_each_sample  so stale samples are excluded independently
 
  CHECKS:
    C1  design run       == expected_run             -- the counting logic
    C2  design qualified == expected_qualified        -- the threshold logic
    C3  qualified => seen                             -- instrumentation (§57 row 19)
    C4  no qualified lane's evidence includes a stale sample
 
  DELIBERATELY NOT MODELLED:
    the SEARCH ALGORITHM.  The model does not reproduce candidate selection.
    It checks the CONTRACT outcomes instead:
      O1  a rejected candidate is followed by a different candidate, or by
          an explicit exhaustion  (§34's row 13, as a model check)
      O2  no candidate is evaluated twice within an attempt
      O3  an accepted candidate is followed by a commit request

Two properties, and the second is the important one.

C1 and C2 are independent recomputations, so a design whose counter is wrong disagrees with the model. 20.4 §17's independence rule: the model's rules come from the architecture, not from the RTL.

And the search algorithm is deliberately not modelled. 20.6 §56's predictor argument: a model that reproduces the design's search reproduces its bugs, and a search is exactly the kind of logic where two independent implementations legitimately differ. Model the contract — progress, no revisits, commit on acceptance — and the model catches §32 without needing to agree about which candidate should be next.

59. Coverage

// ILLUSTRATIVE. The bins without which this chapter's failures are unverified.
// 20.5 owns the methodology; these are the training-specific items.
covergroup cg_training;
  // Per-lane participation — §46's silent lane needs the per-lane bin.
  cp_lane_sampled:   coverpoint lane_index iff (sample_valid) { bins l[] = {[0:NUM_LANES-1]}; }
  cp_lane_failed:    coverpoint lane_index iff (sample_valid && !sample_pass)
                       { bins l[] = {[0:NUM_LANES-1]}; }
  // Every lane must fail at least once, and the FIRST and LAST lanes matter.
  cp_first_last_lane: coverpoint fail_lane_class {
    bins first = {0}; bins last = {NUM_LANES-1}; bins middle = {[1:NUM_LANES-2]};
  }
  // Qualification boundary — §25's threshold edge.
  cp_run_value: coverpoint pass_run_q {
    bins zero = {0}; bins one = {1};
    bins below = {[2:QUAL_THRESHOLD-1]};
    bins at_threshold = {QUAL_THRESHOLD};      // §25 fails HERE
  }
  // The bubble case — §23 is unverified without it.
  cp_bubble_in_run: coverpoint bubble_during_run { bins yes = {1}; bins no = {0}; }
  // Candidate behaviour.
  cp_cand_outcome: coverpoint cand_outcome {
    bins accepted = {CO_ACCEPT}; bins rejected = {CO_REJECT};
    bins exhausted = {CO_EXHAUST};            // §31
  }
  cp_cand_revisit: coverpoint cand_revisited { bins yes = {1}; bins no = {0}; }
  cp_oscillation:  coverpoint oscillation_seen { bins yes = {1}; }   // §35
  // Retry and epoch.
  cp_retry_cause: coverpoint retry_cause { bins c[] = {[0:NUM_RETRY_CAUSES-1]}; }
  cp_stale_sample: coverpoint stale_seen { bins yes = {1}; bins no = {0}; }  // §37
  cp_peer_restart: coverpoint peer_restart_seen { bins yes = {1}; }          // §40
  cp_qual_then_inval: coverpoint qualified_then_invalidated { bins yes = {1}; }
  // Crosses, each with a named bug (20.5 §31).
  x_lane_x_candidate: cross cp_lane_failed, cp_cand_outcome;   // §14 row 4
  x_stale_x_retry:    cross cp_stale_sample, cp_retry_cause;   // §37
  x_bubble_x_run:     cross cp_bubble_in_run, cp_run_value;    // §23
endgroup

Four bins that must be non-empty or this chapter's failures are unverified.

cp_bubble_in_run.yes — without it, §23's bug is untested however many training runs completed.

cp_run_value.at_threshold — §25's truncated threshold makes this bin unreachable, so an unhittable bin is the coverage-model form of the width assertion (20.5 §32's argument).

cp_stale_sample.yes — requires a retry with a measurement in flight (§56), which no random regression produces.

And cp_lane_failed must cover every lane. A regression where lanes 0–30 have failed and lane 31 never has has not tested the last-lane case, which is where index and boundary bugs live.

60. Debug Taxonomy

Seven training symptoms, each with the fastest resolving observation.

No activity at all. Contract 1, globally. Read lane_seen — all zero means the measurement path, not training.

Abundant activity, no convergence, one candidate in the history. §32. Read the candidate history; sixteen records of candidate 0 settles it in one read.

Abundant activity, candidates advancing, no lane ever qualifies. Contract 2. Read pass_run behaviour across bubbles — if it resets on idle cycles, §23; if it resets on failing samples at a high rate, §7's rate problem; if it reaches N−1 and stops, §25's threshold width.

31 of 32 lanes qualified, forever. §46. Read lane_seen and max_lane_age — a clear bit with a saturated age names the lane.

Training converges and the link misbehaves under traffic. A permissive failure — §19's stale valid, §21's cycle counting, or §37's stale sample. Read stale_sample_count and the ratio of samples_valid to measurements issued.

Failures appear to move between lanes. §17's misattribution, or §14's shared cause. Read the candidate history's lanes_passed — if the same bit is clear across candidates it is one lane; if different bits clear on different candidates it is shared or misattributed.

And candidates alternate indefinitely. §35. Read oscillation_count, then decide between scoring, hysteresis and the peer using the peer-restart count.

61. Debug Checklist

Training runs and never converges. In order:

1. Which lanes have ever produced a valid sample (§10)?
2. Which lanes have ever qualified (§10)?
3. Is any required lane's lane_seen clear (§46)?
4. Is any required lane's lane_age saturated (§45)?
5. Does the watchdog use max_lane_age or global activity (§45)?
6. For a silent lane: is samples_failed zero too — path or quality (§52)?
7. For a failing lane: what is the valid-to-failed ratio (§9)?
8. Is sample_valid an edge or a level (§19)?
9. Does the run counter advance on cycles with no sample (§21)?
10. Does the run counter reset on a bubble (§23)?
11. Does the run counter saturate, or wrap (§26)?
12. Can the run counter represent the threshold (§25)?
13. Do the sample fields travel bundled, or in separate pipelines (§17)?
14. Is a result attributable to the lane that was measured (§18)?
15. Did the candidate register ever change (§28)?
16. How many distinct candidates appear in the history (§29)?
17. Is the same candidate evaluated repeatedly (§32)?
18. Does the pointer advance on rejection, or only on acceptance (§33)?
19. Do candidates alternate (§35)?
20. Was the candidate space exhausted (§31)?
21. On which candidates did each lane pass (§29's mask)?
22. Is the same lane clear across all candidates (§52)?
23. Was any candidate ever scored valid (§28 row 4)?
24. Was a commit ever requested (§28 row 5)?
25. Did the active setting change (§28 row 6)?
26. What is the training epoch, and when did it last advance (§36)?
27. Is stale_sample_count non-zero (§38)?
28. Was it incremented after a retry, or during steady training (§38)?
29. Does an attempt boundary clear the run counters (§39)?
30. Did the peer restart, and how many times (§40)?
31. Was qualified evidence retained across a peer restart (§41)?
32. Which retry cause dominates (§49)?
33. Is the retry count split by cause, or a single total (§50)?
34. What is the first failing lane, this power cycle (§12)?
35. What is the first failing lane sequence, per attempt (§12)?
36. Is the qualified-lane count rising across attempts (§43, §44)?
37. What is the first divergence from a known-good event trace (§55)?
38. Which symbols does the failing trace lack (§54)?
39. Did DV ever exercise a retry with a measurement in flight (§59)?
40. What observation would prove the current hypothesis wrong?