UCIe Silicon Debug

Six chapters of Module 21 assumed unlimited evidence: full waveforms, complete event logs, independent models, and every identity retained forever. On silicon you have a few hundred flops, a small SRAM, and a part that is failing.

1. The One-Sentence Model

Silicon debug is information-budget engineering. You cannot observe everything, so every bit of observability must be spent on distinguishing competing hypotheses — and a bit that cannot change what you would conclude is a bit wasted, however interesting the signal behind it.

The reasoning does not change. Symptom, evidence, competing hypotheses, discriminating observation, first divergence, root cause, structural fix. What changes is that the evidence must be chosen and built years before the failure happens — which makes this chapter as much about design decisions as about lab work.

2. What This Chapter Owns

This chapter is not a lab-equipment tutorial. There is no oscilloscope section, no JTAG walkthrough, no logic-analyser buying guide. Four things are genuinely new here:

Evidence must be chosen at design time (§5–§6). The compression table is the chapter's centre: for each thing simulation gave you free, what is the silicon-feasible equivalent, and what does it cost?

The instrument's shape determines what it can prove (§13–§14). A trace buffer that starts recording at the fault captures only the recovery — and pre-trigger history is worth more than any amount of post-trigger depth.

The experiment is part of the evidence (§23–§31). A run that changed four variables produces no information regardless of how carefully it was instrumented.

And observation is destructive (§35, §40). A reset erases the cause while fixing the symptom; enabling trace can make the failure disappear — and neither is a reason to stop reasoning.

3. Sourcing

4. The Budget

Three resources, and every observability decision spends one of them.

And one resource that is not negotiable: you cannot add observability after tape-out. Every question you will want to ask in the lab must have had a register built for it years earlier, by someone who guessed which questions would matter.

So the design-time question is not "what would be interesting to see?" It is "which hypotheses will I need to distinguish, and what is the cheapest observation that separates them?" §5 is that question answered systematically.

5. Simulation Evidence → Silicon Equivalent

The chapter's centrepiece. For each thing simulation gives you free, the feasible equivalent, its cost, and — critically — what it can no longer prove.

Four readings.

The fourth column is the one to design against. Each compression removes a provable claim, and knowing which claim you gave up is what stops you over-reading the residue later. A rolling signature that matches does not prove agreement (21.4 §26) — it fails to disprove it.

Row 3 is the hardest loss and the cheapest partial recovery. Full identity history is infeasible; the first offending identity is one register (14.5 §8) and answers most of the questions you actually ask.

Row 5 is why 21.4's ledger was built the way it was. Nine counters and three exact identities were chosen precisely because they survive this compression: they need no quiescence, no latency model, and no per-object storage.

And the last row inverts the usual direction. Simulation stops at the first failure, which destroys the aftermath; silicon can freeze the evidence and let the system continue, which is strictly more information.

6. Three Tiers of Observability

Three properties.

Tier 1 must be sufficient to classify a failure, even if not to root-cause it. A field failure that returns only tier-1 state should still tell you which of §42's branches to take — and if it cannot, the tier-1 set is wrong.

Tier 2 must be armed before the event, which is the whole difficulty with intermittent failures (§27): you are asking the system to be recording when something you cannot predict happens. Hence a circular buffer that is always recording and only stops on the trigger (§13).

And tier 3 is where most teams start and should usually finish. 21.1 §51's discipline applies: ask what observation would falsify the hypothesis before booking the bench.

7. Where the Evidence Lives Relative to the Trigger

Four readings, and the first is the chapter's most practical single lesson.

The cause is at cycle 2 and the fault is at cycle 3. A buffer that begins recording at the trigger starts at cycle 3 and never contains the cause — §14's failure, and it is the single most common trace-buffer design error.

trace_wr stays high through the trigger, for two more entries. Some post-trigger depth is genuinely useful: it shows which recovery the design chose, which distinguishes "the design did not notice" from "the design noticed and responded wrongly."

frozen rises and stays. After that the buffer is immutable — which is what makes the dump trustworthy and is checked by an assertion in §17, because a buffer that keeps writing after freeze silently overwrites the evidence with recovery traffic.

And the marker at cycle 2 is the design-time argument. Every additional pre-trigger entry increases the chance the cause is inside the window; every additional post-trigger entry mostly captures consequences (14.5 §9's last-wins problem, in the time domain).

8. First Fault Is Not First Divergence

Module 21's two most important concepts, and silicon separates them further than simulation does.

Three consequences.

The first fault is evidence about the divergence, not the divergence itself. A design only flags what it was built to notice; the divergence may have happened in a subsystem with no checker at all, and then the first fault is a downstream consequence.

So a first-fault register with a timestamp is worth much more than one without — because the gap between the divergence and the flag is the thing you are trying to bound, and without a timestamp you cannot even tell whether the trace window covers it.

And when the first fault is at the very start of the trace window, treat the window as suspect (21.6 §31): a divergence at the edge of a capture is usually a capture that started too late, not a divergence that happened to occur exactly then.

9. The Silicon Snapshot

// ILLUSTRATIVE, DEBUG-ONLY. NOT a UCIe register map (§3). Everything a triage
// engineer needs in ONE atomically captured record — because the alternative
// is §11's torn read.
typedef struct packed {
  // WHEN
  logic [TIME_W-1:0]   timestamp;
  logic [15:0]         snap_seq;        // increments per capture — §11's guard
 
  // WHERE IN THE LIFECYCLE  (21.1's dependency ledger, compressed)
  logic [ST_W-1:0]     link_state;      // RESET/SBINIT/.../ACTIVE (§3)
  logic [PH_W-1:0]     debug_phase;     // coarse bring-up milestone
  logic [ATT_W-1:0]    train_attempt;   // WHICH attempt — 21.1's discriminator
  logic [EPOCH_W-1:0]  link_epoch;      // which agreement — 21.4 §21
 
  // WHAT WAS AGREED  (21.1 §29 — active, NEVER requested)
  logic [CFG_W-1:0]    active_cfg;
  logic [CFG_W-1:0]    requested_cfg;   // both, so the DELTA is visible
 
  // WHAT BROKE  (14.5's first/last, and the disagreement is the signal)
  logic                first_fault_valid;
  logic [FLT_W-1:0]    first_fault_code;
  logic [TIME_W-1:0]   first_fault_time;
  logic [ATT_W-1:0]    first_fault_attempt;
  logic [FLT_W-1:0]    last_fault_code;
 
  // LANES, REDUCED  (§20 — not per-lane history)
  logic [LANE_W-1:0]   worst_lane;
  logic [15:0]         worst_lane_errors;
  logic [NUM_LANES-1:0] lane_ever_ok;   // 21.2's two bitmaps — never-vs-regressed
  logic [NUM_LANES-1:0] lane_ok_now;
 
  // RESOURCES  (21.4's ledger, compressed to what fits — §21)
  logic [31:0]         credit_now;
  logic [31:0]         credit_min_seen;
  logic [63:0]         cnt_consume;
  logic [63:0]         cnt_return_apply;
  logic [31:0]         max_return_age;
  logic [31:0]         outstanding_now;
 
  // BEHAVIOUR  (21.5's minimum viable set — §22)
  logic [63:0]         useful_cycles;
  logic [63:0]         no_credit_cycles;
  logic [63:0]         recovery_cycles;
  logic [63:0]         opportunities_active;   // the DENOMINATOR — 21.5 §40
 
  // INSTRUMENT HEALTH  (§17 — the trace is evidence about itself)
  logic                trace_wrapped;
  logic                trace_frozen;
  logic [15:0]         trace_valid_count;
} silicon_snapshot_t;

Architecture. One record, captured in one cycle, holding the state that decides which branch of §42 to take. It is deliberately a triage instrument rather than a root-cause instrument.

State. One frozen bank. Roughly a few hundred flops — the size of a small FIFO, and the cheapest high-leverage debug structure in the design.

Event behaviour. Written wholly on a trigger (§10). Never partially.

Contract. Three fields exist as pairs because the pair is the evidence, not the value. active_cfg with requested_cfg (21.1 §29) — the delta is §38's whole case and §37's denominator. first_fault_code with last_fault_code (14.5 §9) — their disagreement says a cascade occurred. lane_ever_ok with lane_ok_now (21.2 §12) — never-worked and worked-then-regressed are different failures with different owners.

Failure. Capturing only last_fault loses the cause in every cascade. Capturing only active_cfg makes §38 undiagnosable. Omitting opportunities_active means every performance number in the record has no denominator (21.5 §42) and cannot be interpreted at all.

Debug/DV. This record should be dumpable from a field failure, by software, without a debugger attached — which is the test of whether tier 1 was chosen correctly (§6).

10. Atomic Capture

// ILLUSTRATIVE. The whole record, one cycle, one trigger. This is the ONLY
// correct way to build it (§11).
silicon_snapshot_t snap_q;
logic              snap_valid_q;
 
// Triggers: any condition that means "the interesting thing just happened."
// Deliberately EDGE-qualified — a level would re-capture continuously and the
// bank would hold the most RECENT event rather than the first (14.5 §8).
logic snap_trigger;
assign snap_trigger = first_fault_pulse            // 14.5's first fault
                    | invariant_failed_pulse       // §17's sticky checker trip
                    | credit_zero_unexpected_pulse // 21.3 §42
                    | trace_freeze_pulse           // §13 — capture WITH the trace
                    | sw_forced_capture_pulse;     // software, for A/B runs (§26)
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    snap_valid_q     <= 1'b0;
    snap_q.snap_seq  <= '0;
  end else if (snap_trigger && !snap_valid_q) begin
    // STICKY-FIRST: the bank holds the FIRST capture, not the latest. A later
    // trigger during a cascade must not overwrite the cause (14.5 §8).
    snap_valid_q <= 1'b1;
    snap_q.snap_seq <= snap_q.snap_seq + 16'd1;
 
    snap_q.timestamp            <= time_q;
    snap_q.link_state           <= link_state_q;
    snap_q.debug_phase          <= debug_phase_q;
    snap_q.train_attempt        <= train_attempt_q;
    snap_q.link_epoch           <= link_epoch_q;
    snap_q.active_cfg           <= active_cfg_q;
    snap_q.requested_cfg        <= requested_cfg_q;
    snap_q.first_fault_valid    <= first_fault_valid_q;
    snap_q.first_fault_code     <= first_fault_code_q;
    snap_q.first_fault_time     <= first_fault_time_q;
    snap_q.first_fault_attempt  <= first_fault_attempt_q;
    snap_q.last_fault_code      <= last_fault_code_q;
    snap_q.worst_lane           <= worst_lane_q;
    snap_q.worst_lane_errors    <= worst_lane_errors_q;
    snap_q.lane_ever_ok         <= lane_ever_ok_q;
    snap_q.lane_ok_now          <= lane_ok_now_q;
    snap_q.credit_now           <= credit_q;
    snap_q.credit_min_seen      <= credit_min_q;
    snap_q.cnt_consume          <= cnt_consume_q;
    snap_q.cnt_return_apply     <= cnt_return_apply_q;
    snap_q.max_return_age       <= max_return_age_q;
    snap_q.outstanding_now      <= outstanding_q;
    snap_q.useful_cycles        <= useful_cycles_q;
    snap_q.no_credit_cycles     <= no_credit_cycles_q;
    snap_q.recovery_cycles      <= recovery_cycles_q;
    snap_q.opportunities_active <= opportunities_active_q;
    snap_q.trace_wrapped        <= trace_wrapped_q;
    snap_q.trace_frozen         <= trace_frozen_q;
    snap_q.trace_valid_count    <= trace_valid_count_q;
  end
  // NOTE: cleared ONLY by an explicit diagnostic clear, NEVER by recovery or
  // by a functional reset of the link (§35). That is the whole point.
end

Architecture. One always_ff, one trigger, every field. The parallelism is the correctness argument — there is no ordering between the fields because they all land on the same edge.

State. One bank plus a validity bit and a sequence number.

Event behaviour. Edge-qualified and sticky-first. A level-qualified trigger recaptures every cycle the condition holds, so the bank ends up holding the last moment of a cascade — 14.5 §9's failure, arriving through the trigger rather than through the capture.

Contract. Every source signal must be stable and meaningful in the trigger cycle. A field sourced from a signal that is only valid in a different state captures noise — and because the record looks complete, that noise is indistinguishable from data.

Failure. Two. Non-sticky capture loses the first event. Clearing on reset is §35 — the single most destructive thing software does to a debug flow.

Debug/DV. snap_seq incrementing on capture lets software detect a re-capture during a read (§11), and the freeze relationship with the trace (§13) means the snapshot and the trace describe the same instant — which is what makes them jointly interpretable rather than two separate stories.

11. The Torn Read

WRONG software procedure — read the same information as 20 separate registers:
 
  t=0        read link_state       -> ACTIVE
  t=200      read first_fault      -> 0x00 (none)
  t=400      read credit_now       -> 0
  t=600      read active_cfg       -> CFG_A
  t=800      read useful_cycles    -> 1,204,556
 
  CONCLUSION DRAWN: "operational, no fault, but starved of credit."

What actually happened. Between t=0 and t=800 the link took a fault at t=250, recovered, and renegotiated to CFG_B. The record describes three different link states stitched together.

Four properties.

Every individual read was correct. No register lied. The record as a whole describes a system that never existed — which is the defining property of a torn read and why it is so hard to catch.

It is more likely on a healthy debug interface than a slow one is bad, because the failure mode requires the system to keep running — and a system that keeps running is exactly the case you most want to inspect.

The discriminating check is the sequence number (§10). Software reads snap_seq, reads the bank, reads snap_seq again, and discards the result if it changed. Without it, a re-capture landing mid-read reintroduces the tear one level up.

And the structural fix is that there is nothing to stitch: the bank is written in one cycle, so a consistent read is the only possible read. 21.4 §41 makes the same argument for credit counters, and it generalises — any two pieces of state that will be reasoned about together must be captured together.

12. What a Trace Buffer Must Do

And the two that are most often missing are the self-describing ones — the wrapped flag and the valid count. A buffer that cannot say how much of itself is real produces analyses that confidently reason about uninitialised entries.

13. The Trace Buffer

// ILLUSTRATIVE, DEBUG-ONLY. Circular pre-trigger capture with bounded
// post-trigger depth and a hard freeze. This is the centrepiece instrument of
// silicon debug (§7).
module trace_buffer #(
  parameter int DEPTH      = 256,          // need NOT be a power of two
  parameter int POST_DEPTH = 64,
  parameter int PTR_W      = $clog2(DEPTH),
  parameter int CNT_W      = $clog2(DEPTH + 1)   // must represent DEPTH itself
) (
  input  logic              clk,
  input  logic              por_n,
  input  logic              ev_valid,
  input  trace_event_t      ev,
  input  logic              trigger,           // EDGE-qualified upstream
  input  logic              diag_arm,          // re-arm after a dump
  output logic              frozen,
  output logic              wrapped,
  output logic [CNT_W-1:0]  valid_count,
  output logic [PTR_W-1:0]  wr_ptr_out
);
 
  trace_event_t          mem_q [DEPTH];      // NOT reset — see below
  logic [PTR_W-1:0]      wr_ptr_q;
  logic [CNT_W-1:0]      valid_q;
  logic                  wrapped_q;
  logic                  frozen_q;
  logic                  triggered_q;
  logic [$clog2(POST_DEPTH+1)-1:0] post_q;
 
  // Writing is permitted until the buffer freezes. Note that it continues
  // THROUGH the trigger — that is what gives the post-trigger window (§7).
  logic wr_en;
  assign wr_en = ev_valid && !frozen_q;
 
  always_ff @(posedge clk or negedge por_n) begin
    if (!por_n) begin
      // Pointers and status ONLY. The payload SRAM is deliberately NOT reset:
      // it is DEPTH x width bits, resetting it costs area and routing for no
      // benefit, and `valid_count` already tells the reader what is real (§12).
      wr_ptr_q    <= '0;
      valid_q     <= '0;
      wrapped_q   <= 1'b0;
      frozen_q    <= 1'b0;
      triggered_q <= 1'b0;
      post_q      <= '0;
    end else if (diag_arm) begin
      wr_ptr_q    <= '0;
      valid_q     <= '0;
      wrapped_q   <= 1'b0;
      frozen_q    <= 1'b0;
      triggered_q <= 1'b0;
      post_q      <= '0;
    end else begin
 
      // STICKY TRIGGER — the FIRST trigger owns the capture. A later trigger
      // during the cascade must not extend the window (14.5 §8).
      if (trigger && !triggered_q && !frozen_q) begin
        triggered_q <= 1'b1;
        post_q      <= POST_DEPTH[$clog2(POST_DEPTH+1)-1:0];
      end
 
      if (wr_en) begin
        mem_q[wr_ptr_q] <= ev;
 
        // Non-power-of-two safe wrap. A bare `wr_ptr_q + 1` relies on natural
        // rollover and is WRONG whenever DEPTH is not a power of two — it
        // would skip entries and corrupt the ordering.
        if (wr_ptr_q == PTR_W'(DEPTH-1)) begin
          wr_ptr_q  <= '0;
          wrapped_q <= 1'b1;              // sticky — 21.6 §31
        end else begin
          wr_ptr_q <= wr_ptr_q + PTR_W'(1);
        end
 
        // valid_count SATURATES at DEPTH. CNT_W is one bit wider than PTR_W
        // precisely so DEPTH is representable (21.2 §17's threshold trap).
        if (valid_q != CNT_W'(DEPTH)) valid_q <= valid_q + CNT_W'(1);
 
        // Post-trigger countdown advances on ENTRIES, not cycles — so a quiet
        // link does not burn the window while nothing is happening.
        if (triggered_q) begin
          if (post_q == '0) frozen_q <= 1'b1;
          else              post_q   <= post_q - 1'b1;
        end
      end else if (triggered_q && (post_q == '0)) begin
        // Freeze even if no further events arrive, so a link that goes silent
        // after the fault still ends in a defined, dumpable state.
        frozen_q <= 1'b1;
      end
    end
  end
 
  assign frozen      = frozen_q;
  assign wrapped     = wrapped_q;
  assign valid_count = valid_q;
  assign wr_ptr_out  = wr_ptr_q;
 
endmodule

Architecture. A circular buffer that is always recording, a sticky trigger, an entry-counted post-trigger window, and a permanent freeze.

State. DEPTH event entries plus a pointer, a count, and three status bits. The SRAM dominates the area, which is why §15's event selection matters more than depth.

Event behaviour. Writes on ev_valid. The post-trigger countdown advances per entry, not per cycle — a deliberate choice: on a link that goes quiet after a fault, a cycle-counted window expires with the buffer half-empty, whereas an entry-counted window captures the next POST_DEPTH things that actually happen.

Contract. trigger must be edge-qualified by the caller; a level re-triggers. DEPTH need not be a power of two, and the explicit wrap comparison is what makes that safe — a bare increment relying on natural rollover skips entries for any non-power-of-two depth, silently reordering the dump.

Failure. Four specific ones. CNT_W = PTR_W cannot represent DEPTH, so a full buffer reports zero. Non-sticky trigger extends the window through a cascade until the pre-trigger history is overwritten. Resetting the payload SRAM costs real area for no benefit. And no else freeze leaves a quiet link with an unfrozen, still-armed buffer that will be overwritten by the next unrelated event.

Debug/DV. The reader reconstructs order from wr_ptr and wrapped: not wrapped means entries 0 .. valid_count-1 in order; wrapped means the oldest is at wr_ptr. Getting that backwards produces a dump that is correct and reversed — which is why §17 asserts the invariants rather than trusting the convention.

14. Wrong — Recording Only After the Fault

// WRONG — the buffer arms ON the fault, so it records only the aftermath.
assign wr_en = ev_valid && fault_seen_q;    // fault_seen_q sets at the fault

Worked. The root cause is a stale configuration commit at cycle 4,000. The fault is flagged at cycle 4,500. The buffer holds cycles 4,500 onward.

Four properties.

The dump looks excellent. It is complete, ordered, timestamped and entirely about consequences — 14.5 §9's last-error-wins problem expressed in time rather than in a register.

Every hypothesis it can support is a recovery hypothesis, so the investigation converges on the recovery logic, which is working correctly.

And this design is chosen surprisingly often, because "start recording when something goes wrong" is the intuitive reading of a debug buffer. The correct reading is the opposite: record always, and stop when something goes wrong.

The fix is §13 — wr_en gated on !frozen_q rather than on fault_seen_q, which inverts the polarity of the whole instrument and costs nothing extra.

15. Choosing What to Record

Recording every cycle is infeasible; the buffer would cover microseconds. Record events, and choose them by which hypothesis they separate.

Two design rules.

Every event on the list earns its place by appearing in a discriminator column of some earlier chapter. An event that no hypothesis needs is SRAM spent on nothing.

And rare events are worth more per entry than frequent ones. A state transition happens tens of times; a credit-zero event may happen constantly. Recording a high-rate event unconditionally can flush the entire buffer between the cause and the trigger — so high-rate events need rate limiting or a first-N-only qualifier, or they defeat the pre-trigger window they share.

16. The Trace Event

// ILLUSTRATIVE, DEBUG-ONLY. NOT a UCIe debug format — no such format is
// standardised here or implied (§3). Compact, because DEPTH x this width is
// the area (§4).
typedef struct packed {
  logic [TIME_W-1:0]  timestamp;    // delta or absolute — see below
  logic [EVK_W-1:0]   kind;         // §15's list
  logic [EPOCH_W-1:0] epoch;        // which agreement (21.4 §21)
  logic [ATT_W-1:0]   attempt;      // which training attempt (21.2)
  logic [LANE_W-1:0]  lane;         // valid only for lane-scoped kinds
  logic [CTX_W-1:0]   context;      // kind-specific: fault code, state, reason
} trace_event_t;

Architecture. Six narrow fields. context is deliberately kind-specific — a union in practice — because giving each event kind its own dedicated field would multiply the width by the number of kinds.

State. One instance per buffer entry.

Event behaviour. Constructed at the event and written once.

Contract. epoch and attempt must be present on every event, not only on the ones that seem to need them. Without them a dump spanning a recovery cannot be partitioned, and comparing events from two agreements is 21.6 §11's stale-event error made unavoidable by the format.

Failure. Two width traps. A timestamp narrow enough to wrap inside the buffer's own time span makes ordering ambiguous — and delta encoding, which saves real width, must saturate rather than wrap, with saturation itself visible so a long gap is not read as a short one. A lane field reused as generic context on non-lane events produces dumps where a fault code is decoded as a lane number.

Debug/DV. The struct being packed and fixed-width is what lets software decode a dump without knowing the design's internals — and the format being versioned somewhere in the snapshot is what stops a decoder from silently misreading a different stepping's layout.

17. Verifying the Debug Logic

// MANDATORY. Debug infrastructure is RTL and gets the same treatment. A silent
// bug here produces CONFIDENTLY WRONG evidence, which is worse than none.
 
// (1) A frozen buffer is immutable. English: once frozen, no entry, pointer or
// count changes. Fires if recovery traffic overwrites the captured evidence —
// the failure that makes a dump quietly untrustworthy.
a_frozen_is_immutable: assert property (
  @(posedge clk) disable iff (!por_n)
    (frozen && !diag_arm) |=> ($stable(wr_ptr_out) && $stable(valid_count))
);
 
// (2) The first fault is never overwritten. English: once captured, the code
// and its timestamp hold until an explicit diagnostic clear (14.5 §8).
a_first_fault_sticky: assert property (
  @(posedge clk) disable iff (!por_n)
    (first_fault_valid_q && !diag_clear)
      |=> ($stable(first_fault_code_q) && $stable(first_fault_time_q))
);
 
// (3) The valid count is bounded and monotonic until re-arm. English: a count
// above DEPTH is impossible; a count that decreases means a width bug.
a_valid_count_sane: assert property (
  @(posedge clk) disable iff (!por_n)
    (valid_count <= CNT_W'(DEPTH)) &&
    (diag_arm || (valid_count >= $past(valid_count)))
);
 
// (4) The write pointer is always legal. English: it indexes a real entry.
// Fires on the non-power-of-two rollover bug (§13).
a_wr_ptr_legal: assert property (
  @(posedge clk) disable iff (!por_n) (wr_ptr_out < PTR_W'(DEPTH))
);
 
// (5) A trigger eventually freezes. English: after a trigger, the buffer
// reaches a frozen state within the post-trigger window — otherwise a dump can
// be requested from a still-recording buffer, which is a torn read (§11) in
// the time domain.
a_trigger_eventually_freezes: assert property (
  @(posedge clk) disable iff (!por_n)
    $rose(triggered_q) |-> ##[1:MAX_FREEZE_LATENCY] frozen
);
 
// (6) Wrapped is sticky. English: once history has been lost, that fact is
// never un-said (21.6 §31).
a_wrapped_sticky: assert property (
  @(posedge clk) disable iff (!por_n)
    (wrapped && !diag_arm) |=> wrapped
);
 
// (7) THE MOST IMPORTANT ONE — instrumentation is passive (§18). English: no
// functional output depends on any debug state. Written as a formal
// non-interference check rather than a simulation property.
a_debug_does_not_drive_function: assert property (
  @(posedge clk) disable iff (!por_n)
    $stable({frozen, wrapped, valid_count, snap_valid_q}) ||
    $stable(functional_outputs)
);

Architecture. Six structural invariants plus one non-interference property.

State. None beyond the buffer's own.

Sampled timing. Properties (1), (2) and (6) use |=>, so they compare this cycle's condition against next cycle's stability — the correct phase for checking that a registered value holds. Property (5) uses $rose on the sticky trigger, so it fires once per capture rather than every cycle the trigger is asserted.

Contract. MAX_FREEZE_LATENCY must be derived from POST_DEPTH and the event rate. Because the countdown is entry-based (§13), a silent link takes unbounded cycles to consume the window — which is why the else freeze exists, and why this bound must be written against the design's guaranteed freeze path rather than against traffic.

Failure. Without (1), the dump is contaminated by recovery. Without (3), a full buffer reporting zero valid entries reads as "nothing was captured". Without (7), the design's behaviour depends on whether debug was enabled — which is §40's Heisenbug promoted from a timing accident to a logical one.

Debug/DV. These are illustrative architectural properties, not UCIe requirements (§3). Property (7) is the one that belongs in a formal flow rather than simulation: non-interference is a proof obligation, and sampling it in a testbench only shows it held for the stimulus you ran.

18. Observability Must Be Passive

No functional decision may depend on a debug counter, a trace state, a snapshot bit, or whether debug is enabled.

Three specific violations, all seen in real designs:

A trace buffer that backpressures the functional path when it cannot accept an event. The link now runs differently with tracing on — which is §40, built in deliberately.

A fault counter that gates a recovery decision — "after N errors, retrain" — where the same counter is also cleared by a debug read. Software reading the diagnostic now changes the recovery policy.

And a snapshot trigger wired into a functional enable, so arming debug alters the link's behaviour. 14.5 §5's rule, and the reason §17's property (7) exists: the instrument must be observable from the design and invisible to it.

19. Per-Lane Evidence, Compressed

Simulation keeps every lane's full history. Silicon must choose.

Three notes.

The two bitmaps are the highest value per bit in the entire chapter. 21.2 §12: never worked and worked then regressed are different failures with different owners, and one bit per lane per bitmap separates them.

The worst-lane reduction costs a constant rather than scaling with N (§20), which matters when N is large and area review is real.

And per-lane counters are the expensive middle option. They answer "is one lane dominant or is it spread?" — genuinely valuable (§36) — but if the budget only allows one, take the bitmaps, because a dominant lane usually shows up in worst_lane anyway.

20. Worst-Lane Reduction

// ILLUSTRATIVE. Constant-cost reduction: which lane has the most errors, and
// how many. Replaces N counters with one comparison tree plus two registers.
logic [15:0]        worst_lane_errors_q;
logic [LANE_W-1:0]  worst_lane_q;
logic               worst_lane_tie_q;     // more than one lane at the maximum
 
// Per-lane counters still exist locally if affordable; where they do not, the
// same shape works on a per-lane error PULSE with a small saturating counter.
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    worst_lane_errors_q <= '0;
    worst_lane_q        <= '0;
    worst_lane_tie_q    <= 1'b0;
  end else if (diag_clear) begin
    worst_lane_errors_q <= '0;
    worst_lane_q        <= '0;
    worst_lane_tie_q    <= 1'b0;
  end else begin
    for (int l = 0; l < NUM_LANES; l++) begin
      if (lane_err_count_q[l] > worst_lane_errors_q) begin
        worst_lane_errors_q <= lane_err_count_q[l];
        worst_lane_q        <= LANE_W'(l);
        worst_lane_tie_q    <= 1'b0;       // a new strict maximum clears the tie
      end else if ((lane_err_count_q[l] == worst_lane_errors_q)
                   && (worst_lane_errors_q != '0)
                   && (LANE_W'(l) != worst_lane_q)) begin
        // A TIE is diagnostically important: "lane 7 is worst" and "lanes 3, 7
        // and 11 are equally worst" point at completely different causes (§36).
        worst_lane_tie_q <= 1'b1;
      end
    end
  end
end

Architecture. A maximum-reduction with a tie flag, at constant cost.

State. A count, a lane index, and one bit.

Event behaviour. Evaluated as a reduction. Simultaneous updates resolve to a strict maximum, with the tie recorded rather than silently broken by loop order.

Contract. The tie flag is not defensive padding. "Lane 7 is worst with 40 errors" points at that lane. "Lanes 3, 7 and 11 are tied at 40" points at something common to all of them — a clock, a supply, a shared structure, or a systemic condition — and the two conclusions send the investigation to different teams. A reduction without a tie flag reports the first-indexed winner and hides the distinction entirely.

Failure. Comparing with >= rather than > makes the last-indexed lane win every tie, so the reported worst lane changes with unrelated design edits and looks like a moving failure.

Debug/DV. In a single snapshot this gives the lane to investigate. Across many runs (§27) the distribution of worst_lane is the real evidence: always the same lane is a lane-specific hypothesis; uniformly distributed is not.

21. Credit Evidence in Silicon

21.4's nine-counter ledger compresses well, because its highest-value checks were already register-based.

Two readings.

cnt_return_apply > cnt_consume is arithmetically impossible and checkable from two registers, at any moment, on a running link (21.4 §25). It is the single most valuable credit instrument that survives into silicon — and it fires in the profitable phase of an inflation, not in the corruption phase.

And max_return_age earns its place because absence cannot be captured (21.4 §28). Counters detect events that happened; only an age detects an event that should have happened and did not — which is §37's discriminator.

22. Performance Evidence in Silicon

21.5's nine classes compress to a handful, and the choice is dictated by §42's tree rather than by completeness.

This is deliberately not a hardware analytics dashboard. Six quantities, and between them they resolve §42's performance branch. The full nine-class histogram belongs in simulation, where the classifier can be exhaustive without costing area.

And the first row is the one most often omitted and most often needed: a performance counter set without an active-configuration opportunity count produces numbers that cannot be turned into a percentage of anything — 21.5 §41's 49%-versus-98% trap, in a field failure where you cannot go back and measure again.

Both sets together are one small, entirely passive block:

// ILLUSTRATIVE. The complete always-on silicon counter set implied by §21 and
// §22 — credit evidence and performance evidence in one passive block. This is
// what compresses Module 21's nine-counter ledger and nine-class histogram into
// something a review will actually approve (§4).
module ucie_debug_counters #(
  parameter int CNT_W = 64
) (
  input  logic             clk,
  input  logic             por_n,
  input  logic             diag_clear,        // NEVER a functional reset (§35)
 
  // observed boundary events — all inputs, nothing driven back (§18)
  input  logic             consume_fire,
  input  logic             return_apply_fire,
  input  logic [31:0]      credit_now,
  input  logic             return_pending,
  input  logic             link_operational,
  input  logic             xfer_fire,
  input  logic             is_replay,
  input  logic [15:0]      xfer_payload_bytes,
  input  logic [7:0]       slots_this_cycle,  // from the ACTIVE config (21.5 §40)
  input  logic             work_pending,
  input  logic             credit_available,
  input  logic             src_valid,
  input  logic             src_ready,
 
  output logic [CNT_W-1:0] cnt_consume_q,
  output logic [CNT_W-1:0] cnt_return_apply_q,
  output logic [31:0]      credit_min_q,
  output logic [31:0]      max_return_age_q,
  output logic [CNT_W-1:0] zero_credit_cycles_q,
  output logic [CNT_W-1:0] opportunities_active_q,
  output logic [CNT_W-1:0] useful_cycles_q,
  output logic [CNT_W-1:0] no_credit_cycles_q,
  output logic [CNT_W-1:0] recovery_cycles_q,
  output logic [CNT_W-1:0] bytes_wire_q,
  output logic [CNT_W-1:0] bytes_unique_q,
  output logic [CNT_W-1:0] offered_q,
  output logic [CNT_W-1:0] accepted_q
);
 
  logic [31:0] return_age_q;
 
  always_ff @(posedge clk or negedge por_n) begin
    if (!por_n || diag_clear) begin
      cnt_consume_q <= '0; cnt_return_apply_q <= '0;
      credit_min_q  <= '1;              // starts at MAX so the first sample wins
      return_age_q  <= '0; max_return_age_q <= '0;
      zero_credit_cycles_q <= '0; opportunities_active_q <= '0;
      useful_cycles_q <= '0; no_credit_cycles_q <= '0; recovery_cycles_q <= '0;
      bytes_wire_q <= '0; bytes_unique_q <= '0;
      offered_q <= '0; accepted_q <= '0;
    end else begin
 
      // ---- CREDIT (§21) ----
      if (consume_fire)      cnt_consume_q      <= cnt_consume_q + CNT_W'(1);
      if (return_apply_fire) cnt_return_apply_q <= cnt_return_apply_q + CNT_W'(1);
 
      // MINIMUM, not current — the current value tells you nothing about
      // whether starvation ever happened (21.3 §42).
      if (credit_now < credit_min_q) credit_min_q <= credit_now;
      if (credit_now == 32'd0) zero_credit_cycles_q <= zero_credit_cycles_q + CNT_W'(1);
 
      // Return age resets on EMPTY, not on service — a boundary serviced
      // continuously but never drained has a genuinely old oldest item
      // (21.4 §29). And it SATURATES: a wrapping age reports a fresh boundary
      // during an ancient stall (21.1 §21).
      if (!return_pending)                return_age_q <= '0;
      else if (return_age_q != '1)        return_age_q <= return_age_q + 32'd1;
      if (return_age_q > max_return_age_q) max_return_age_q <= return_age_q;
 
      // ---- PERFORMANCE (§22) ----
      // The DENOMINATOR, derived from the ACTIVE configuration every cycle —
      // not computed offline from an assumed width (21.5 §41).
      opportunities_active_q <= opportunities_active_q + CNT_W'(slots_this_cycle);
 
      if (!link_operational) begin
        recovery_cycles_q <= recovery_cycles_q + CNT_W'(1);
      end else if (xfer_fire) begin
        bytes_wire_q <= bytes_wire_q + CNT_W'(xfer_payload_bytes);
        // A REPLAY occupies the wire and delivers nothing new. Counting it as
        // useful makes the metric RISE as the link degrades (21.5 §16).
        if (!is_replay) begin
          useful_cycles_q <= useful_cycles_q + CNT_W'(1);
          bytes_unique_q  <= bytes_unique_q + CNT_W'(xfer_payload_bytes);
        end
      end else if (work_pending && !credit_available) begin
        // Only the FIRST unmet condition is charged, so the classes stay
        // mutually exclusive and a percentage means something (21.5 §8).
        no_credit_cycles_q <= no_credit_cycles_q + CNT_W'(1);
      end
 
      // Offered vs accepted — the pair that ends a source-limited
      // investigation before it starts (21.5 §21).
      if (src_valid)              offered_q  <= offered_q  + CNT_W'(1);
      if (src_valid && src_ready) accepted_q <= accepted_q + CNT_W'(1);
    end
  end
 
endmodule

Architecture. Thirteen counters — six for credit, seven for performance — in one block with only inputs from the design and no outputs back into it (§18).

State. Thirteen registers plus a working age. Roughly the area of a small FIFO, which is what makes it survivable in review.

Event behaviour and sampled timing. Everything is registered off the same clock, so all thirteen advance on the same edge — which is what lets §10's snapshot capture them as a consistent set. credit_min_q initialises to all-ones so the first real sample wins; initialising it to zero would report permanent starvation from cycle one.

Contract. Three details carry most of the value. slots_this_cycle must come from the active configuration, sampled live — a denominator computed offline from an assumed width is 21.5 §41's trap in a field failure you cannot re-run. return_age_q resets on empty, never on service (21.4 §29). And diag_clear must be separate from any functional reset (§35), or §34's driver erases all of it.

Failure. Four. Counting a replay as useful makes the metric improve as the link degrades. Charging more than one class per cycle breaks the accounting (21.5 §8). A wrapping age hides an ancient stall. And credit_now without credit_min_q cannot answer "did this ever starve?" — which is the actual question in §37.

Debug/DV. These thirteen registers resolve every branch of §42 except the lane branch, which §20 covers. cnt_return_apply_q > cnt_consume_q is arithmetically impossible and checkable from two of them (21.4 §25) — the highest-value single check that survives into silicon.

23. Designing a Lab Experiment

A good experiment changes exactly one meaningful variable and records everything else.

And the rule that makes it an experiment rather than an attempt: state the hypothesis and the prediction before the run. "If the failure is lane-specific, then swapping to CFG_B — which does not use that lane — will pass." A run with no prediction cannot produce a surprise, and a surprise is the only thing that teaches you anything.

24. The Reproducibility Record

A failing run that was not recorded is an anecdote.

Two notes.

Reset history is the field most often missing and most often decisive. A run that was reset twice before the capture may have lost the first fault (§35) — and without the record you cannot tell whether the snapshot you have is the first one or the third.

And no proprietary manufacturing or process data belongs in this record. Stepping and revision identify the design; wafer-level or process data is neither needed for this reasoning nor appropriate to circulate.

25. Bad Experiment, Good Experiment

BAD — four variables, one run:
 
  change configuration class   CFG_A -> CFG_B
  update firmware              v1.2  -> v1.3
  swap peer                    peer P -> peer Q
  raise operating margin
  re-run
 
  RESULT: the failure disappears.
  CONCLUSION: none. Four candidate causes, no way to separate them,
              and the failing setup no longer exists to go back to.

GOOD — one variable, stated prediction:
 
  HYPOTHESIS:  the failure is specific to one lane in the CFG_A mapping.
  PREDICTION:  CFG_B, which maps around that lane, will pass; everything
               else held constant.
  HOLD:        firmware v1.2, peer P, same board, same workload, same seed.
  CHANGE:      configuration class only.
 
  RESULT: CFG_B passes, 20/20 runs. CFG_A fails 6/20.
  CONCLUSION: consistent with a lane-specific hypothesis; NOT yet proof of
              an electrical cause (§32). Next experiment: does the worst_lane
              distribution across the 6 failures concentrate? (§20, §27)

Three properties.

The bad run destroyed its own evidence. The failing configuration is gone; reverting all four changes may not reproduce it, because intermittent failures do not return on demand (§27).

The good run's conclusion is deliberately weak — "consistent with", not "proves". 21.6's discipline: a passing alternative configuration is one discriminating observation, not a root cause.

And the good run states its next experiment, which is what turns a sequence of attempts into an investigation.

26. A/B Comparison

A known-good unit and a failing one, held identical in everything but the suspected factor.

Compare milestones and counts, not absolute timing. Two correct runs differ in cycle counts for a dozen legitimate reasons — clock domains, arbitration, workload jitter. A comparison keyed on timing reports every healthy pair as divergent and produces nothing.

This is 21.6 §40's event-aligned trace comparison, run across silicon units rather than across simulation runs — and the alignment key is the same: semantic event, not cycle.

27. Intermittent Failures

A failure that happens sometimes is a distribution, and the distribution is the evidence.

Two cautions.

Do not claim causation from a handful of runs. Three failures out of five under one condition and one out of five under another is not evidence of a condition dependence — it is four events, and small samples of a random process routinely look structured.

And the distributions are cheap to gather precisely because §9's snapshot is cheap. Twenty runs give twenty first-fault codes and twenty worst-lane values, which is a far stronger artefact than one heavily-instrumented failure — and it is the argument for making tier-1 state dumpable without a bench (§6).

28. Counting Failures Honestly

WEAK:    "it failed three times"
 
USABLE:  "6 failures in 20 runs, CFG_A, peer P, firmware v1.2,
          nominal condition, same board"
 
AFTER A FIX:
  before:  6 / 20
  after:   0 / 20
 
  This is consistent with a fix. It does NOT establish one — a change that
  merely made the failure rarer produces exactly this result at this sample
  size. Establishing the fix requires either enough runs that the prior rate
  would very likely have produced a failure, or a MECHANISM: an assertion
  that fires on the old design and not the new, or a trace showing the
  specific event no longer occurs.

Three properties.

A denominator is mandatory — the same discipline 21.5 §42 demands of a percentage.

"Zero after the fix" is the weakest possible evidence of a fix, and it is the evidence most often accepted. A rate that dropped from 30% to 2% shows zero in 20 runs about two-thirds of the time.

And a mechanism beats any amount of statistics. If the trace showed the specific stale commit at §14's cycle 4,000, and the new design cannot produce that event — and an assertion proves it — the argument is closed without a sample-size discussion.

29. Bisection

When failure depends on configuration, bisect the configuration space rather than guessing.

ILLUSTRATIVE. Configurations written SYMBOLICALLY (§3) — this chapter states
no UCIe width or mode, so CFG_A..CFG_D are the design's own classes.
 
  CFG_A (widest)          FAIL 6/20
  CFG_B                   PASS 20/20
  CFG_C (same lanes as A, lower rate)   PASS 20/20
  CFG_D (same rate as A, fewer lanes)   FAIL 2/20
 
  READING:
    A fails, C passes  -> the failure is RATE-sensitive, not purely lane count
    D fails, B passes  -> a subset of A's lanes reproduces it
    INTERSECTION: the lanes in D, at A's rate.
 
  NEXT: which lanes are in D but not in B? That set is the candidate.

Three properties.

Each row eliminates a dimension, which is what makes this bisection rather than sampling.

Rate-sensitivity is a suspicion pointer, not a conclusion (§32). It is consistent with a physical/electrical cause and also with a rate-dependent digital defect — a synchroniser, a clock-crossing, a timing path. Do not collapse it prematurely.

And the widths are symbolic on purpose. This chapter states no UCIe configuration set (§3), and a bisection written against invented widths would teach the wrong specific and the right general.

30. What Loopback Cannot Prove

Passing loopback proves that the local transmit and receive paths agree with each other. Two ends that share a misinterpretation agree perfectly — which is 21.6 §24's shared-model failure, expressed physically.

And that is exactly why it is useful. A loopback failure is a strong local finding — the problem is on this die, in this path, with no peer involved. A loopback pass is a weak result that eliminates one branch of §42 and nothing more.

31. Peer Swap

OBSERVED:
  A <-> B    FAIL
  A <-> C    PASS
  D <-> B    PASS
 
TEMPTING CONCLUSION: "neither A nor B is broken, so it is an interaction."
 
That is closer to correct than blaming either — but it is still not proof.

Four properties.

The matrix eliminates "A is broken" and "B is broken" and nothing else. Four hypotheses survive.

The fourth row is the one teams miss. A pass is not a clean bill of health: C may tolerate the same defect that B exposes, so A↔C passing tells you about C's tolerance as much as about A.

The cheapest discriminator is active_cfg across all three pairings (§9), because it is one field in a snapshot that already exists — and a pairing that negotiates differently is a completely different experiment from one that negotiates identically and then fails.

And "interaction" is a category, not a root cause. 21.6's work still has to happen: which contract, whose interpretation, under which revision and configuration.

32. Digital Evidence Versus Physical Suspicion

The chapter's most consequential branch, and the one where overreach is most common.

Three properties.

The left column supports a proof; the right column supports a hypothesis. 21.6's standard applies unchanged: a digital violation can be established from a trace, and a physical cause cannot be — not from digital evidence alone.

The right column's most characteristic pattern is the fourth row. Every protocol event is legal, every state transition is legal, and then an integrity check fails. That shape says the logic was correct and something corrupted the data — which is a physical suspicion and points the investigation at a different discipline.

And the two are not mutually exclusive. A marginal channel produces retries; retries expose a retry-classification bug that never ran before (21.4 §35). The physical condition is the trigger; the digital defect is the bug. Fixing the channel hides it; fixing the classification fixes it.

33. The Limit of This Chapter's Authority

34. Software Must Preserve, Not Clear

The most destructive bug in a silicon debug flow is usually in the driver.

WRONG driver initialisation sequence:
 
  1. detect link error
  2. read first_fault register        <- evidence exists here
  3. log a summary string             <- most of it discarded
  4. call block_init()                <- CLEARS first_fault, counters, trace
  5. retrain
  6. link recovers
  7. user reports "intermittent link errors, no further information"

Four properties.

Step 3 is where the evidence dies, not step 4. A summary string keeps the fault code and discards the attempt number, epoch, active configuration, worst lane and trace — the fields that would have chosen a branch of §42.

Step 4 is defensible in isolation. Initialising a block before reuse is correct engineering; the bug is the ordering, not the call.

And the fix is a contract, not a code change: diagnostic state is cleared only by an explicit diagnostic clear (§10, §17's property 2), never by a functional initialisation path — which means the hardware must offer that separation, and a design that ties them together makes this bug unavoidable in software.

The correct sequence is §35's.

35. Capture Before Reset

A reset may fix the symptom and destroy the cause. Capture first, always, in this order.

Three notes.

Step 1 is first because it is smallest and most likely to survive. If the system dies mid-procedure, the snapshot alone still routes the investigation.

Step 2 can wait behind step 1 only because the trace is frozen (§13). An unfrozen buffer is being overwritten while you read step 1 — which is the practical argument for the freeze, beyond the assertion in §17.

And this order must be built into the driver's error path, not into a debug mode, because the failures that matter most are the ones that happen where no debug mode was enabled — which is §6's tier-1 argument arriving from the software side.

36. Case 1 — Sometimes Fails to Become Operational

Five readings.

lane_ever_ok clear means lane 7 never worked in those runs — 21.2 §12's never-versus-regressed distinction, and it rules out a degradation.

first_fault_attempt is 1 in five of six, so this is a static failure rather than something that develops over attempts (21.2).

The CFG_C result is the most informative row and the one that complicates the story. Lane 7 is present at a lower rate and passes — so lane 7 is not simply dead. The failure is rate-sensitive on that lane.

That is consistent with a physical cause and does not establish one (§32). It is also consistent with a rate-dependent digital defect in that lane's own capture or alignment path — and nothing above distinguishes those two.

So the honest output is §33's handoff, plus one more digital experiment that is cheap and could still close it: does the design's own per-lane failure code differ between lane 7 and its neighbours? A distinct code points back at a specific digital mechanism; the same code as a generic integrity failure leaves the physical branch open.

37. Case 2 — Operates, Then Freezes

Five readings, and the first is why this case is in the chapter.

No fault was ever flagged and no checker tripped. Every design-level error detector is silent — the link is simply not making progress, which is 21.3's signature and is invisible to fault-based instrumentation.

The two counters differ by exactly outstanding_now. 64 credits are charged and unreturned, and 64 objects are outstanding — so nothing leaked; 64 obligations are stuck (21.4 §10's in-flight term, in silicon).

max_return_age saturated is the discriminator and the only instrument that could have produced it. A missing event cannot be counted (21.4 §28) — it is found by ageing, and the age says a boundary stopped rather than slowed.

The first divergence is therefore thousands of cycles before the freeze, at whatever stopped that boundary — and the freeze is the consequence the user reported.

And the trace closes it. With credit-zero and return-age-threshold both on §15's event list, the frozen buffer's pre-trigger region contains the last successful return and everything after it — which names the boundary directly (21.4 §18) without any of the nine counters being present.

38. Case 3 — Bandwidth at 55–70% of Expectation

Four readings.

The link is at 96.8% of what it can currently do. There is no throughput bug (21.5 §41).

The real finding is the degradation — why the link negotiated CFG_B when CFG_A was requested — which is 21.2's investigation, not this one, and would have run for weeks as a performance problem.

The requested_cfg / active_cfg pair in §9 is what makes this a two-minute triage rather than a misdirected project. One field, present because 21.1 §29 argued for it.

And the retry ratio at 1.00 matters as an exclusion: it rules out 21.5 §53's high-wire-low-delivery case entirely, from one division, before anyone looks at a scheduler.

39. Case 4 — Only One Peer Fails

Five readings.

The fourth row was the experiment worth running, and it changes the shape of the problem entirely: the failure is not "A and B together", it is "CFG_A" — which only A and B ever negotiate to.

The first three rows alone would have supported an "interaction" conclusion (§31) and sent two teams into a comparison of A's and B's implementations. The fourth row says the interaction is only in which configuration they agree on.

But forcing CFG_B is a workaround, not a fix, and shipping it hides a defect that will return whenever another pair negotiates CFG_A.

The remaining work is 21.6's. A protocol checker tripped — so there is a digital claim to prove: which contract, whose interpretation, under which revision, and at which first divergent event. §31's four hypotheses are still open, now scoped to one configuration.

And first_fault_code being a checker trip rather than an integrity failure is what keeps this in the left-hand column of §32. This is provable from a trace; no bench is needed.

40. Case 5 — It Disappears When Trace Is Enabled

The classic, and it is not a reason to stop.

Five properties.

Check the last row first. If the baseline rate was 6/20, a run of 20 clean passes is much less surprising than it feels — and "enabling trace fixed it" may be a sample-size artefact costing days.

Check the fourth row second, because it is the only one that can be settled by reading the RTL rather than by experiment — and §17's non-interference property is what makes it a proof.

The first row's discriminator is the useful trick: build with the debug logic present but never armed. That separates timing/area perturbation from activity perturbation, because the logic exists either way and only the switching differs.

And the correct response is to reduce instrumentation until the failure returns, then reason from the minimum that still reproduces it — the opposite of the instinct to add more.

Which is the chapter's model restated (§1): more observability is not free and not always better. A debug architecture is a budget, and this case is what overspending looks like.

41. Instrumentation Is Not Free

Two rules, and then this chapter stops.

Budget observability deliberately, at architecture time, using §5's table: each instrument justified by which hypotheses it separates, not by what would be interesting.

And the tiering of §6 is what makes the budget affordable. Tier 1 must be small enough to be always on; tier 2 can be large because it is only armed when someone is looking. Getting that split wrong in either direction — a huge always-on block, or a triggered-only design that captures nothing in the field — is the most common silicon-debug architecture mistake.

Physical implementation of that budget is a different discipline and this chapter does not enter it.

42. The Silicon Debug Decision Tree

1. Did the link ever reach ACTIVE? (link_state, debug_phase) No → bring-up or training. Check lane_ever_ok versus lane_ok_now, first_fault_attempt, and the requested/active delta. 21.1 if it never progressed; 21.2 if it progressed and regressed. → §36.

2. It reached ACTIVE. Is this a correctness or a performance symptom?

3a. Correctness — did any checker trip or fault flag? Yes → a digital claim exists. → 21.6, and §39. No → §3b.

3b. No fault, no progress — is resource accounting closed? cnt_consume versus cnt_return_apply; max_return_age; outstanding_now. Not closed → → 21.4, and §37. Closed → look downstream of the link (21.5 §50).

4. Performance — is active_cfg what the benchmark assumed? No → the denominator is wrong. The real finding is the degradation. → §38. Yes → §5.

5. Is bytes_unique / bytes_wire near 1.0? No → retry overhead → §32's physical branch. Yes → → 21.5's class histogram, reduced to §22's six counters.

6. Is it peer-specific? Build the matrix (§31), and compare active_cfg across pairings — then force a common configuration. → §39.

7. Is it lane- or configuration-condition-specific? worst_lane distribution (§27), tie flag (§20), bisection (§29). Concentrated and rate-sensitive → physical suspicion, stated as a hypothesis and handed off (§33). Never as a conclusion.

43. Debug Checklist

Forty questions.

Trust the evidence before reading it (1–8).

1. Is the snapshot atomic, or assembled from separate register reads (§11)?
2. Did snap_seq change during the read (§10)?
3. Is the trace frozen (§13)?
4. Did the trace wrap (§12)?
5. What is valid_count — how much of the buffer is real?
6. Does the trace contain pre-trigger history, or only post-fault (§14)?
7. Was anything reset before capture (§24, §35)?
8. Could software have cleared the evidence (§34)?

Locate the failure (9–16). 9. What was link_state at capture? 10. What debug_phase milestone was reached? 11. Which training attempt (§9)? 12. Which epoch? 13. active_cfg versus requested_cfg — do they differ (§38)? 14. first_fault_code, and its timestamp? 15. last_fault_code — does it disagree with the first (§9)? 16. Is the first fault at the very start of the trace window (§8)?

Lanes (17–21). 17. lane_ever_ok — did any lane never work (§19)? 18. lane_ok_now — did any lane work and then regress? 19. worst_lane, and is the tie flag set (§20)? 20. Across many runs, is worst_lane concentrated or uniform (§27)? 21. Do lanes fail with the same code or different ones (§36)?

Resources (22–26). 22. cnt_return_apply versus cnt_consume — is either impossible (21.4 §25)? 23. Does the shortfall equal outstanding_now (§37)? 24. credit_min_seen — did it reach zero? 25. max_return_age — is it saturated (§37)? 26. Were any returns rejected, and why (21.4 §46)?

Performance (27–31). 27. What is opportunities_active — is there a denominator at all (§22)? 28. Utilisation against active, not requested (§38)? 29. bytes_unique / bytes_wire — retry overhead (§22)? 30. recovery_cycles — a recovery storm? 31. Offered versus accepted — was the source the limiter?

The experiment (32–40). 32. How many variables changed in this run (§25)? 33. What was the hypothesis and prediction before it (§23)? 34. What is the reproduction rate, with a denominator (§28)? 35. Is the reproducibility record complete (§24)? 36. Was the A/B comparison keyed on milestones, not cycle timing (§26)? 37. Does loopback pass, and what does that actually exclude (§30)? 38. Across a peer matrix, does active_cfg differ (§31, §39)? 39. Does bisection separate rate from lane count (§29)? 40. Is the conclusion digital and provable, or physical and a hypothesis (§32, §33)?

44. Common Misconceptions

"Reset fixed it, so the problem is gone." §35: a reset commonly fixes the symptom and destroys the cause. The problem is now intermittent and uninstrumented.

"More trace signals always help." §40, §41: instrumentation costs timing, power and activity, and can make the failure disappear without fixing anything.

"Capturing after the trigger is enough." §14: the cause precedes the fault. A post-trigger-only buffer records the recovery in perfect detail.

"One failing lane proves an analog defect." §36: a lane that fails at one rate and passes at another is consistent with a physical cause and equally consistent with a rate-dependent digital defect.

"Passing loopback proves the link works." §30: it proves the local paths agree with each other, which two ends sharing a misinterpretation also do.

"If one peer fails, that peer is wrong." §31: a matrix eliminates "A is broken" and "B is broken" and leaves four hypotheses, including that the passing pairs are passing for different reasons.

"Lab counters cannot have bugs." §17: debug RTL is RTL. A wrong valid_count width makes a full buffer report empty.

"Software can read live counters sequentially." §11: the result describes a system that never existed, and every individual read was correct.

"A rare failure has no pattern." §27: the distribution of first-fault code, worst lane and attempt number is the evidence — and it is cheap to gather.

"Performance counters can be added later." §4: there is no later. Every question you will ask needs a register that already exists.

"Instrumentation is physically free." §41: area, timing, congestion, power, and verification effort.

"The final fatal error is the root cause." §8: it is the end of a cascade. The first fault is a lower bound on how early the divergence was, not the divergence.

"Silicon debug is fundamentally different from simulation reasoning." §1: the reasoning is identical. Only the evidence budget changed — which is why Module 21's six chapters transfer intact.

45. Understanding Check

46. Module 21, Complete

Seven chapters, one method, applied to progressively harder evidence.

And the through-line is one sentence. Every chapter looked for the first observation that could not have occurred in a correct implementation — a dependency never satisfied, a run counter that reset on an idle cycle, a conservation equation that broke, a boundary whose two sides disagreed, an opportunity that was avoidably lost, an event no legal continuation explains. Silicon does not change that question. It changes how much you get to look.

Which is why the design decisions in this chapter are debug decisions. The two bitmaps, the requested/active pair, the first-fault register, the two credit counters, the return age, the active-opportunity count and a circular trace buffer are perhaps a few hundred flops and one small SRAM — and between them they carry the whole method onto a part that has already been built.