Link Bring-Up Failures

Module 20 built a verification programme. This module starts from the other end: the hardware is failing, and the question is what evidence to collect and what each observation eliminates.

1. The One-Sentence Model

Debug is not finding the last bad signal. Debug is identifying the first observation that could not have occurred in a correct implementation.

Everything in this chapter follows from that. operational == 0 is the symptom the report arrives with; it is the consequence of an event that happened earlier, in a block the report does not name. The chapter's whole job is to walk backwards from the symptom to that event, eliminating hypotheses in an order that costs the least.

2. What This Chapter Owns

Those chapters built the machinery. This chapter reads it. The distinction is sharp and it is worth stating plainly:

14.5 §8 taught how to build a first-fault register. §23 below teaches what its disagreement with the last fault proves — and that is a different skill.

8.2 §7 taught the wrapping-timer bug as an RTL defect. §21 teaches it as a debug symptom — a hang whose watchdog never fires, and how to recognise that from evidence.

8.5 §7 built the dependency cone. §6 turns it into a ledger you interrogate, with a column for the invalidation event because a dependency that regressed and one that was never true are different bugs.

And this chapter deliberately stops at "training is the blocker." §34 hands off. 21.2 owns lanes, candidates and qualification, and diving into them here would make both chapters worse.

3. Sourcing

4. The Symptom Is Not the Root Cause

Three pairs, and the gap between the columns is the work.

Three properties of the right-hand column.

It names a dependency, not a state. "Waiting for peer agreement" points at one owner; "in the validate phase" points at nobody.

It names the first occurrence. Attempt 3, not attempt 17 (§25).

And it says what is healthy. "31 other lanes are active" eliminates the whole class of hypotheses about the training machinery in general — which is the discriminating half of the observation and the half most reports omit.

A good debug statement eliminates hypotheses. A symptom eliminates none.

5. Bring-Up Is a Dependency Graph

Not one state machine. Operational readiness is a conjunction of several independent proofs, and 8.5 §5 established that the operational status must be derived from them rather than asserted.

The debug consequence: a hang has exactly four shapes, and distinguishing them is the first branch of every investigation.

Two notes.

Shapes 1 and 2 are the ones people conflate, and §10's one-line ever_valid bitmap separates them for free. A dependency that regressed has a producer that stopped; one that was never true has a producer that never started — completely different searches.

Shape 3 is rarer and nastier. The evidence exists and the consumer is not looking at it — a gating term with an extra condition, a missed clock domain crossing, an enable that was never set. It is identified by the contradiction between the dependency bitmap and the phase, which is why §11's tuple carries both.

6. The Dependency Ledger

The chapter's centerpiece. Every dependency answers seven questions, and a ledger row that cannot fill the invalidate event column is a row whose shape-2 failures are undiagnosable.

Five readings, and each is a debug technique.

Column 5 is the one that makes shape-2 diagnosis possible. "Peer restart" appears in four rows — so a single peer restart invalidates four dependencies at once, and a report saying "four prerequisites went false simultaneously" has one cause rather than four.

Column 6 shows why an epoch is a debug tool and not just a design one. "Local training evidence" is scoped to the training attempt — so evidence from attempt 7 must not satisfy attempt 8, which is 21.2's flagship stale-sample bug seen from the ledger.

Column 7 is empty for four rows, and that is a finding. A dependency with no progress signal cannot be distinguished between slow and stuck (§18), and those four rows are where a progress watchdog cannot help.

The last row is derived, not owned. 8.5 §5: traffic permission is a conclusion. A design where something sets it has removed the conjunction that makes every row above it matter.

And the first two rows have no epoch, which is why they are checked first (§28, §30). They are the only rows whose failure makes every other row's evidence meaningless.

7. The Debug Phase Abstraction

Three things to read.

Four phases can reach RETRY, and one of them is OPERATIONAL. That last edge is §37's whole subject: a link that reaches operational and immediately falls back produces a trace that looks like a training hang — and the only thing distinguishing them is phase history.

VALIDATE is a separate phase from TRAINING deliberately. Local training evidence being valid does not mean the link is ready; the bilateral and configuration dependencies are checked after it (§25), and separating them means the ledger can say which of the two is missing.

And FAILED is terminal until a deliberate reset. 14.2 §21's sticky-fatal argument: a self-clearing failed state destroys the evidence for the failure, and a link that quietly retries out of it produces a hang with no record of why.

8. Waveform — the Bring-Up Stall

Four readings, and the fourth is the lesson.

operational is flat at zero for the whole trace and carries no information. It is the signal the bug report names and the only signal in the figure that eliminates nothing.

local_train_ok rising at cycle 4 eliminates a great deal. Training produced valid evidence, so the whole class of "the PHY is bad" and "training is broken" hypotheses is gone — and that is the most expensive class to investigate, eliminated by one signal.

peer_train_ok flat at zero is the finding, and it points at a different owner: the peer, the bilateral indication path, or the local logic that observes it (§5's shape 3).

And cfg_valid flat at zero is a consequence, not a second bug. It is blocked behind the bilateral dependency (§6's blocks column). A debugger who investigates configuration because cfg_valid is low is chasing a derived symptom — which is exactly the mistake the ledger prevents.

9. Do Not Start With "Which State?"

The current phase tells you where the controller is. It does not tell you why.

Worked, on the trace above. The phase is VAL. That is true at cycle 5 and at cycle 9 and would be true at cycle 900,000. The question the phase cannot answer is what VAL is waiting for — and VAL has three inputs, of which one is missing and two are fine.

Three questions that replace it, in order:

1 — What prerequisite is the current phase waiting for? The dependency bitmap, not the phase.

2 — Was that prerequisite ever true? §11 — one bit, and it splits the search in half.

3 — If it is true now, why is the controller still waiting? §5's shape 3 — a contradiction between the bitmap and the phase, which is a consumer bug rather than a producer bug.

And the corollary for instrumentation: a design that exposes its phase and not its dependency bitmap has exposed the least informative register it has. §14 is that failure.

10. Never True Versus Regressed

One bit per dependency, and it is the highest value-per-flop instrumentation in the chapter.

// ILLUSTRATIVE debug instrumentation. One sticky bit per dependency, set the
// first time the dependency is observed true and never cleared except by a
// scope that legitimately invalidates the whole attempt.
logic [NUM_DEPS-1:0] deps_now;         // combinational, the live conjunction inputs
logic [NUM_DEPS-1:0] deps_ever_q;      // sticky: has this EVER been true?
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) deps_ever_q <= '0;
  else        deps_ever_q <= deps_ever_q | deps_now;
end

The two-bit truth table that results, per dependency:

Four properties.

One flop per dependency, and it halves the search space. With eleven ledger rows that is eleven flops for a diagnostic that is otherwise a guess.

Row 3 changes the question entirely. "It was true at some point" means there was a working producer, so the investigation is about what took it away — and §6's invalidate column lists the candidates.

Row 4 must be asserted. now && !ever cannot occur if the update is correct; if it fires, the bitmap is lying and every conclusion drawn from it is void. 20.4 §42's environment-self-check discipline applied to hardware instrumentation.

And the reset scope is por_n, deliberately. A bitmap cleared by a recovery cannot report that a dependency was true before the recovery — which is the single most useful thing it could have told you.

// MANDATORY. The instrumentation must not lie about itself.
property p_ever_implies_consistency;
  @(posedge clk) disable iff (!por_n)
    (deps_now & ~deps_ever_q) == '0;
endproperty
a_ever_implies_consistency: assert property (p_ever_implies_consistency);

11. The Progress Tuple

// ILLUSTRATIVE. The minimum debug state that answers §9's three questions in
// one read. Symbolic widths; no register name or encoding is claimed (§3).
typedef struct packed {
  logic [PHASE_W-1:0] phase;            // where the controller is
  logic [NUM_DEPS-1:0] deps_now;        // what is true right now
  logic [NUM_DEPS-1:0] deps_ever;       // §10 — what was ever true
  logic [EPOCH_W-1:0] attempt;          // which bring-up attempt
  logic [EPOCH_W-1:0] cfg_epoch;        // which configuration
  logic [AGE_W-1:0]   phase_age;        // cycles in this phase (saturating)
  logic [AGE_W-1:0]   progress_age;     // cycles since the last progress event
  logic [CAUSE_W-1:0] wait_reason;      // §15 — the primary blocker
  logic [CAUSE_W-1:0] last_invalidate;  // §24 — why a dependency last regressed
} bringup_debug_t;

Nine fields, and each answers a question no other field answers.

Two notes.

phase_age and progress_age are both present and they are different quantities. A training phase can legitimately sit for a very long time while making steady internal progress; phase_age large with progress_age small is healthy-but-slow, and progress_age large is stuck. §19.

And the whole struct is one read. That matters in silicon: a debugger that must issue nine separate reads sees a torn snapshot, and the fields will disagree about which cycle they describe.

12. The State-Entry Snapshot

Capture on every phase transition, not every cycle.

// ILLUSTRATIVE. A snapshot per semantic transition. TRACE_DEPTH is small —
// bring-up has tens of transitions, not thousands.
typedef struct packed {
  logic [TIME_W-1:0]   timestamp;
  logic [PHASE_W-1:0]  from_phase;
  logic [PHASE_W-1:0]  to_phase;
  logic [NUM_DEPS-1:0] deps_at_entry;    // WHY the transition happened
  logic [EPOCH_W-1:0]  attempt;
  logic [CAUSE_W-1:0]  cause;            // what triggered it
} bringup_trace_t;
 
bringup_trace_t              trace_mem [TRACE_DEPTH];
logic [TRACE_PTR_W-1:0]      trace_wr_q;
logic                        trace_wrapped_q;   // an honesty flag — see below
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    trace_wr_q      <= '0;
    trace_wrapped_q <= 1'b0;
  end else if (trace_event) begin          // §13's event list, NOT every cycle
    trace_mem[trace_wr_q] <= '{ timestamp: time_q,
                                from_phase: phase_q,
                                to_phase:   phase_d,
                                deps_at_entry: deps_now,
                                attempt:    attempt_q,
                                cause:      trace_cause };
    trace_wr_q <= trace_wr_q + TRACE_PTR_W'(1);
    if (trace_wr_q == TRACE_PTR_W'(TRACE_DEPTH-1)) trace_wrapped_q <= 1'b1;
  end
end

Three notes.

deps_at_entry is what makes the trace explanatory rather than descriptive. A transition record saying "TRAINING → RETRY" says nothing; one saying "TRAINING → RETRY, deps = 0b0011, cause = timeout" says which dependencies were still missing when the timeout fired.

trace_wrapped_q is a one-bit honesty flag, and it is the field most often omitted. A ring that has wrapped no longer contains the beginning of the run — so a debugger reading it must know whether the earliest record is the first event or merely the oldest surviving one. Without the flag, a wrapped ring reads as a complete history that starts mysteriously late.

And the reset scope is por_n. A trace cleared by a recovery loses the events leading up to the recovery, which is §37's entire diagnostic.

13. What Counts as a Trace Event

Semantic events only. A trace that records every cycle is a waveform with extra steps, and it will have wrapped before anything interesting happened.

And what does not: every cycle, every sample, every lane measurement, every credit event. Those belong to per-lane counters (21.2) and per-domain counters (21.3), not to a bring-up trace whose whole value is that it is short enough to read.

14. Wrong Instrumentation — the Phase-Only Trace

// WRONG — records the phase, and nothing about why.
always_ff @(posedge clk)
  if (phase_q != phase_d) begin
    trace_mem[trace_wr_q] <= phase_d;    // just the phase
    trace_wr_q <= trace_wr_q + 1'b1;
  end

What the debugger receives:

PREREQ
DISCOVERY
TRAINING
RETRY
DISCOVERY
TRAINING
RETRY
DISCOVERY
TRAINING
RETRY
... × 17
FAILED

Four things this cannot answer, and they are the only questions that matter.

Which dependency was missing at each retry. Every retry looks identical. The seventeen attempts may have had seventeen different blockers or one, and the trace cannot distinguish them — which is the difference between a flaky lane and a dead peer.

Whether anything improved across attempts. Attempt 1 with two dependencies missing and attempt 17 with one is progress; the trace shows neither.

What triggered each retry. A timeout, a regression, or a peer restart are three different causes with three different owners.

And whether the ring wrapped. With seventeen attempts and a depth of sixteen, the first attempt — the one whose cause was original — is gone, and nothing says so (§12).

The fix is deps_at_entry and cause, which cost a few bits per record and convert a list of phases into a causal history.

15. The Waiting-Reason Encoder

One primary reason per cycle, derived combinationally, for classification only.

// ILLUSTRATIVE. A DEBUG CLASSIFIER. The priority order is a debug convention,
// not an architectural claim: several conditions can be false at once, and the
// encoder reports the one that is causally furthest upstream.
typedef enum logic [CAUSE_W-1:0] {
  WR_NONE            = 'd0,
  WR_NO_CLOCK        = 'd1,   // nothing else is meaningful without this (§30)
  WR_IN_RESET        = 'd2,
  WR_NO_PEER         = 'd3,
  WR_PEER_INCOMPAT   = 'd4,
  WR_TRAINING        = 'd5,   // hand off to 21.2 (§34)
  WR_CALIBRATION     = 'd6,
  WR_NO_BILATERAL    = 'd7,   // §25 — local done, peer not
  WR_CFG_NOT_AGREED  = 'd8,
  WR_LINK_INIT       = 'd9,
  WR_RECOVERY        = 'd10,
  WR_FATAL           = 'd11
} wait_reason_e;
 
always_comb begin
  // UPSTREAM-FIRST priority. A missing clock makes every downstream condition
  // uninterpretable, so it wins; a fatal condition is reported last because it
  // is usually a consequence of one of the others (§23).
  if      (!clock_alive)          wait_reason = WR_NO_CLOCK;
  else if (!reset_released)       wait_reason = WR_IN_RESET;
  else if (!peer_observed)        wait_reason = WR_NO_PEER;
  else if (!peer_compatible)      wait_reason = WR_PEER_INCOMPAT;
  else if (!training_evidence)    wait_reason = WR_TRAINING;
  else if (!calib_committed)      wait_reason = WR_CALIBRATION;
  else if (!peer_bilateral_ok)    wait_reason = WR_NO_BILATERAL;
  else if (!cfg_agreed)           wait_reason = WR_CFG_NOT_AGREED;
  else if (!link_init_done)       wait_reason = WR_LINK_INIT;
  else if (recovery_active)       wait_reason = WR_RECOVERY;
  else if (fatal_latched)         wait_reason = WR_FATAL;
  else                            wait_reason = WR_NONE;
end

Four notes, and the third is the one that keeps this honest.

The ordering is causally upstream first, and that is a deliberate debug choice. A missing clock and a missing peer are both false; reporting the clock is more useful because the peer condition cannot be evaluated meaningfully without it.

WR_FATAL is last, not first, which is counter-intuitive and correct. A fatal condition raised during bring-up is usually the result of one of the earlier conditions persisting — reporting it first would name the consequence (§4's second row).

The priority is a classifier and not a claim. Several conditions genuinely are false simultaneously, and §17 is what happens when that is forgotten. The encoder answers "where should I look first?", not "what is the only thing wrong?" — and §16's counters are read alongside deps_now, never instead of it.

And it is always_comb with a single assignment per branch. 19.1 §31's discipline: one writer, no latch, and the output is a pure function of the inputs so it can be sampled at any cycle without a race.

16. Per-Reason Cycle Counters

// ILLUSTRATIVE. One saturating counter per reason. Increment ONLY the primary
// reason, so the counters sum to the elapsed time and can be read as a budget.
logic [63:0] wait_cycles_q [NUM_WAIT_REASONS];
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    for (int r = 0; r < NUM_WAIT_REASONS; r++) wait_cycles_q[r] <= '0;
  end else if (!operational) begin
    // Saturate rather than wrap — a wrapped diagnostic counter destroys its
    // own evidence (§21, and 14.5 §11's saturating-counter discipline).
    if (wait_cycles_q[wait_reason] != '1)
      wait_cycles_q[wait_reason] <= wait_cycles_q[wait_reason] + 64'd1;
  end
end

Why this is the first thing to read on a reported hang, before any waveform.

It is a budget. The counters sum to the time spent not operational, so one reason owning 99.9% of it names the phase to investigate in a single register read — no trace, no waveform, no simulation.

Worked, on three real shapes:

And the third row is the one a single snapshot cannot produce. A snapshot reads the current reason; the counters read the whole history, which is how an alternating failure becomes visible without a trace.

17. Wrong Counters — Counting Every False Dependency

// WRONG — increments every counter whose dependency is false.
always_ff @(posedge clk)
  if (!operational)
    for (int d = 0; d < NUM_DEPS; d++)
      if (!deps_now[d]) dep_false_cycles_q[d] <= dep_false_cycles_q[d] + 1'b1;

Worked. At cycle 1000 of a hang, five dependencies are false: peer_observed, peer_compatible, training_evidence, cfg_agreed, link_init_done.

All five counters read 1000. The distribution is flat and identifies nothing.

Three properties.

Four of the five are derived. training_evidence cannot be true without a peer; cfg_agreed cannot be true without compatibility. So four counters are measuring the consequence of the fifth — and the flat distribution actively conceals which is causal.

The counters no longer sum to anything meaningful. Five counters at 1000 for 1000 cycles of hang; the budget reading of §16 is gone, and with it the ability to say "this reason owns the time."

And it is not useless — it is the wrong level. The raw deps_now bitmap is genuinely valuable (§9) and it is a snapshot question. Cycle counting belongs to the primary reason; bitmap reading belongs to the instant. Keeping both, distinguished, is the correct design — the error is using one where the other was needed.

18. Progress Versus Residence

Two different quantities, and conflating them produces both a false timeout and a missed hang.

The key asymmetry: a phase can make steady internal progress for a very long time without transitioning. 8.3 §14's phase-local timers exist because of this. A watchdog on phase_age fires on a healthy slow phase; a watchdog on progress_age fires only when the phase has actually stopped.

What counts as a progress event is architecture-specific, and the ledger's column 7 lists the candidates (§6): a lane qualifying, a candidate being evaluated, a peer response arriving, a dependency becoming true. A phase with no progress event has no stuck-detector — which is a finding about the instrumentation.

19. Wrong Watchdog — Resetting on Transition Only

// WRONG — one global timer, reset only when the phase changes.
always_ff @(posedge clk)
  if (phase_q != phase_d) global_timer_q <= '0;
  else                    global_timer_q <= global_timer_q + 1'b1;
 
assign timeout = (global_timer_q > GLOBAL_TIMEOUT);

Worked. Training legitimately runs for a long time: 32 lanes, several candidates each, several retries. No phase transition occurs, because all of that happens inside the training phase. The global timer expires.

The design reports a timeout on a link that was converging.

Four consequences.

A false timeout aborts a working bring-up, which is worse than a hang because the link could have come up.

The fix that gets applied is to raise GLOBAL_TIMEOUT, and that makes the watchdog useless for the fast phases — a discovery phase that hangs now takes the training-sized timeout to be detected. 20.2 §16's fixed-bound failure, in hardware.

And after two false timeouts the watchdog is disabled, so the real hang — when it comes — is found by a system-level timeout with no localisation at all.

The correction is two watchdogs with two meanings: a progress watchdog on progress_age, which fires when the phase has stopped doing anything, and a phase watchdog on phase_age with a per-phase bound derived from that phase's work (20.2 §17's derived bound). Neither alone is sufficient and they detect different failures.

20. The Saturating Age Counter

// ILLUSTRATIVE. Saturating, never wrapping. The saturation is the whole point:
// a wrapped age counter reports a fresh phase during an ancient hang (§21).
logic [AGE_W-1:0] progress_age_q;
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n)                     progress_age_q <= '0;
  else if (progress_event)        progress_age_q <= '0;
  else if (progress_age_q != '1)  progress_age_q <= progress_age_q + AGE_W'(1);
  // else: HOLD at all-ones. Do NOT wrap.
end
 
assign progress_stalled = (progress_age_q >= AGE_W'(PROGRESS_BOUND));

Three notes.

The != '1 guard is the saturation, and it is one comparison. Without it the counter wraps (§21).

AGE_W must be wide enough that saturation means "much longer than any legitimate gap". A counter that saturates at a value a healthy phase reaches reports every run as stalled, which is §19's false timeout arriving from the other direction.

And the reset scope is por_n, not the phase. 14.2 §14's capture-before-retrain rule: an age counter cleared by a retry cannot report that seventeen attempts each stalled the same way.

21. The Wrapping-Age Trap

8.2 §7 taught this as an RTL defect in a discovery timer. Here it is as a debug symptom, which is a different skill: recognising it from the evidence rather than from a code review.

// WRONG — the age counter wraps.
always_ff @(posedge clk)
  if (progress_event) progress_age_q <= '0;
  else                progress_age_q <= progress_age_q + 1'b1;   // wraps

The evidence signature, and it is distinctive:

Three properties.

The hang lasts forever and is never reported. Worse than a false timeout: the design has a watchdog and it does not work, so nothing escalates and the failure presents as a system-level hang with no UCIe-level diagnosis.

The discriminating observation is the contradiction between two counters. A small progress_age alongside a huge wait_cycles_q is impossible in a correct implementation — and that impossibility is exactly §1's definition of the observation to look for.

And it is why §16's counters are saturating too. If both counters wrap, the contradiction disappears and there is no evidence left at all. Saturating diagnostics preserve the evidence that something has been true for a very long time, which is the one thing a hang needs to report.

// MANDATORY. A diagnostic age counter must not wrap.
property p_age_saturates;
  @(posedge clk) disable iff (!por_n)
    (progress_age_q == '1) |=> ((progress_age_q == '1) || progress_event);
endproperty
a_age_saturates: assert property (p_age_saturates);
 
// MANDATORY. And it must not decrease except on a progress event.
property p_age_monotonic_between_events;
  @(posedge clk) disable iff (!por_n)
    (progress_age_q < $past(progress_age_q)) |-> $past(progress_event);
endproperty
a_age_monotonic_between_events: assert property (p_age_monotonic_between_events);

22. Reading the First-Fault Record

14.5 §8 built the sticky first-fault register and 14.5 §9 showed why last-error-wins destroys the evidence. This section is about reading it.

Four fields, and the debug value of each:

The single most useful operation is a comparison, and it is available from two registers:

Read the first fault and the last fault. If they differ, the last one is a consequence and the first one is the lead.

23. First Versus Last Fault — the Difference Is the Finding

Worked, on a bring-up that fails after seventeen attempts.

What the pair proves, in order.

The original failure was in discovery, not training. Attempt 1, DISCOVERY — so every attempt after the first was retrying a link whose peer never responded properly.

RETRY_BUDGET_EXHAUSTED is arithmetic, not a cause. It says the design gave up. A debugger who starts there investigates the retry policy — and the retry policy is working correctly.

And the attempt number is the discriminator between two very different bugs:

Three notes.

A first fault at attempt 1 with a last fault at attempt 17 is the common shape, and it is the shape where the last register is most misleading — because seventeen attempts have generated a long tail of derivative faults.

A first fault at a later attempt is rarer and more interesting. Something worked and stopped: thermal, a peer that restarted, a resource that leaked across attempts. That is §10's shape 2 at the attempt level.

And if the two registers are equal, that is also information: the failure happened once, was captured, and nothing derivative followed — which usually means the design failed cleanly rather than looping (§40's absence).

24. Dependency Invalidation History

§10's ever bitmap says a dependency regressed. This says why.

// ILLUSTRATIVE. One record of the most recent invalidation, plus a count.
// The COUNT is what distinguishes a single regression from a flapping one.
logic [NUM_DEPS-1:0]  deps_prev_q;
logic [CAUSE_W-1:0]   last_invalidate_cause_q;
logic [DEP_W-1:0]     last_invalidate_which_q;
logic [TIME_W-1:0]    last_invalidate_time_q;
logic [CNT_W-1:0]     invalidate_count_q [NUM_DEPS];
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    deps_prev_q             <= '0;
    last_invalidate_cause_q <= '0;
    for (int d = 0; d < NUM_DEPS; d++) invalidate_count_q[d] <= '0;
  end else begin
    deps_prev_q <= deps_now;
    for (int d = 0; d < NUM_DEPS; d++) begin
      if (deps_prev_q[d] && !deps_now[d]) begin          // a 1 -> 0 transition
        last_invalidate_cause_q <= invalidate_cause;      // why, from the source
        last_invalidate_which_q <= DEP_W'(d);
        last_invalidate_time_q  <= time_q;
        if (invalidate_count_q[d] != '1)
          invalidate_count_q[d] <= invalidate_count_q[d] + CNT_W'(1);
      end
    end
  end
end

Three readings the count enables:

The third pattern is the valuable one. Four dependencies with identical counts were invalidated by the same event seventeen times — so the investigation is one cause, not four, and §6's invalidate column names the candidates that touch four rows at once.

// MANDATORY, illustrative architectural contract. A dependency does not
// regress silently: an invalidation must produce an observable event.
property p_invalidation_is_observable;
  @(posedge clk) disable iff (!por_n)
    ((deps_prev_q & ~deps_now) != '0) |-> (invalidate_event || recovery_entry);
endproperty
a_invalidation_is_observable: assert property (p_invalidation_is_observable);

25. Bilateral Evidence

8.5 §4 established that local completion is not bilateral completion. §8's waveform is that failure, and it is worth stating what makes it hard to debug.

Three properties of a bilateral failure.

Every local signal is healthy. Training completed, lanes qualified, calibration committed. A debugger examining the local side finds nothing wrong — because nothing local is wrong.

The missing evidence is absence, and absence has no waveform. peer_train_ok flat at zero looks the same whether the peer never sent it, the peer sent it and the local receiver dropped it, or the local logic requires an extra condition nobody documented (§5's shape 3).

And the natural next step is the wrong one. With everything local healthy, the instinct is to conclude the peer is broken — which is a hypothesis, not an observation, and it is wrong a meaningful fraction of the time.

The three sub-hypotheses, and the observation that separates them:

And the first row is why bring-up debug is frequently a two-sided activity. 20.7 §33's attribution procedure: each side reads its own ledger, and the side whose ledger shows it waiting on the other is not the blocker. Two devices each waiting on the other is a fifth shape, and it is a protocol or configuration disagreement rather than a hardware fault.

26. Wrong RTL — Operational From Local Completion

// WRONG — operational asserted from local evidence alone.
assign operational = local_training_done && local_calib_done;

What happens, and the timing is the cruel part.

The link asserts operational. Traffic is admitted. The peer is not ready — it has not finished its own training, or has not agreed a configuration, or is still in a reset domain that has not released.

The first transfer produces a fault, and the link enters recovery. Then bring-up runs again, asserts operational again, admits traffic again, faults again.

Four properties.

The symptom is a recovery loop, not a bring-up hang (§40) — and the two are reported identically by a user.

The fault is attributed to the wrong layer. A transfer into an unready peer produces an integrity error, a timeout, or a protocol violation — so the first fault register names a transport cause for a bug whose root is a missing bring-up dependency.

Local signals are all healthy throughout, so a local-only investigation finds nothing (§25).

And the correct derivation is the conjunction — 8.5 §5's derived-operational rule:

// RIGHT — every modelled dependency, derived, with nothing setting it directly.
assign operational = clock_alive
                  && reset_released
                  && peer_observed
                  && peer_compatible
                  && training_evidence
                  && calib_committed
                  && peer_bilateral_ok      // the term §26 omitted
                  && cfg_agreed
                  && link_init_done
                  && !fatal_latched;

// MANDATORY. Operational implies every dependency. The conjunction is the
// contract; an assign that sets operational any other way violates it.
property p_operational_requires_all_deps;
  @(posedge clk) disable iff (!por_n)
    operational |-> (deps_now == '1);
endproperty
a_operational_requires_all_deps: assert property (p_operational_requires_all_deps);
 
// MANDATORY. And no traffic before it.
property p_no_admission_before_operational;
  @(posedge clk) disable iff (!por_n)
    admit_fire |-> operational;
endproperty
a_no_admission_before_operational: assert property (p_no_admission_before_operational);

27. The Failure Families

Seven families. The waiting-reason distribution (§16) selects one in a single register read, and each family has a different first question.

Two properties of this table.

Family G is identified by a pattern, not a value. Alternating reasons in the counter distribution rather than one dominant reason (§16's third row) — and it is the family most often misreported, because the user's description is identical to a hang.

And families A and C are the cheapest to eliminate and are frequently skipped. A stopped clock and an empty capability intersection are both answerable in minutes and both produce a total failure to come up. Investigating training first, because training is the interesting part, is the most common wasted week in bring-up.

28. Family A — Reset and Prerequisites

8.1 owns reset design. The debug questions are narrower and there are five.

Three notes.

Row 2 is the one that produces a partial bring-up. Multiple reset domains (8.1 §6) mean one domain can be running while another is held, so the design makes progress up to the point where it needs the held domain — and the phase it stops in points at the consumer, not at the reset controller.

Row 5 is the shape-2 case at the very bottom of the ledger. A reset that asserted, released, and asserted again produces a trace that starts over — which reads as a retry loop and is not one.

And row 4 is where an illegal-state check earns its place. 20.3 §36's encoding property: a phase register holding a value outside its enumeration after a bad release will behave in a way no state diagram describes, and every subsequent observation is uninterpretable.

The instrumentation that answers rows 2 and 3 in one read:

// ILLUSTRATIVE. Per-domain release status, plus a sticky record of the release
// ORDER. The order matters because 8.1 §14 sequences the release, and a domain
// released out of order produces a partial bring-up that reads as a hang.
logic [NUM_RST_DOMAINS-1:0] domain_released_q;    // current
logic [NUM_RST_DOMAINS-1:0] domain_ever_rel_q;    // §10's ever, per domain
logic [ORD_W-1:0]           release_order_q [NUM_RST_DOMAINS];
logic [ORD_W-1:0]           release_seq_q;
 
always_ff @(posedge clk or negedge por_n) begin
  if (!por_n) begin
    domain_released_q <= '0;
    domain_ever_rel_q <= '0;
    release_seq_q     <= '0;
  end else begin
    domain_released_q <= domain_release_status;         // synchronised inputs
    domain_ever_rel_q <= domain_ever_rel_q | domain_release_status;
    for (int d = 0; d < NUM_RST_DOMAINS; d++)
      if (domain_release_status[d] && !domain_released_q[d]) begin
        release_order_q[d] <= release_seq_q;             // WHEN, relative to others
        release_seq_q      <= release_seq_q + ORD_W'(1);
      end
  end
end

Two readings this enables that a single "reset done" bit cannot.

A domain with ever_rel == 0 never released, and release_order_q for every other domain says how far the sequence got before it stalled — which names the domain whose release condition is unmet, not merely the fact that something is held.

And a release order that differs from the expected sequence is a finding even when every domain eventually releases. 8.1 §14 sequences the release for a reason; a domain that releases early operates before its prerequisites, and the resulting failure is downstream and unattributable without this record.

29. Flagship Trace 1 — the Domain That Never Released

Illustrative. Four reset domains, one held, and the bring-up stalls three phases later.

Five readings, and the ordering of the discoveries is the lesson.

The stall is reported in VALIDATE, three phases after the cause. Domain 3 was never released at t=0, and the design made progress for four hundred cycles before reaching anything that needed it. A debugger who starts at VALIDATE is four hundred cycles and three phases downstream.

domain_ever_rel == 0111 is the finding, and it is one read. Bit 3 was never set — so the question is not "why is link-init incomplete" but "why did domain 3 never release", which has a completely different owner.

The release order says how far the sequence got: domains 0, 1 and 2 released in order at t=20, 24, 28. The sequence stalled at step 4, which points at domain 3's release condition rather than at the reset controller in general.

WR_LINK_INIT dominating the waiting-reason counters gets you to VALIDATE in one register read (§16) — and it is exactly as far as that counter can take you. The domain bitmap is what crosses the remaining three phases.

And note what the controller did at t=28. It advanced to DISCOVERY because its prerequisite view was satisfied — which is §5's shape 4 in a mild form: the controller advanced on an incomplete prerequisite set, and the design would be more debuggable if PREREQ required all four domains rather than the three the controller happens to check.

30. Clock First — the Question That Costs Nothing

A reset that deasserts synchronously cannot release without a clock. No amount of reset debugging fixes a stopped clock.

Why this is the first question of every bring-up investigation, and not the third.

It is answerable in minutes — any free-running counter in the target domain, incrementing or not.

It invalidates every other observation. A domain with no clock has registers holding their reset values forever. The phase reads PREREQ, every dependency reads false, every age counter reads zero — and all of that is consistent with a dozen other hypotheses, so an investigation that starts elsewhere burns time on a system where no evidence can be produced.

And the failure has several causes that are not "the clock is broken": a PLL that never locked, a clock gate left disabled, a divider held in reset, a clock-mux selecting a dead source, a domain whose enable is derived from a configuration that was never written. Five different owners, and the shared symptom is that the domain is dark.

The diagnostic, and it is one register:

// ILLUSTRATIVE. A free-running counter per clock domain, readable from a
// domain that is known alive. Its ONLY job is to answer "are there edges?"
logic [ALIVE_W-1:0] domain_tick_q;
 
always_ff @(posedge domain_clk or negedge por_n)
  if (!por_n) domain_tick_q <= '0;
  else        domain_tick_q <= domain_tick_q + ALIVE_W'(1);   // wrapping is FINE here

Note that wrapping is deliberately acceptable here, unlike every other counter in this chapter. The question is "did the value change between two reads?", not "how large is it?" — so a narrow wrapping counter answers it with fewer flops, and a saturating one would answer it worse by reading all-ones forever.

31. Family B — Discovery

8.2 owns the design. Five debug questions, in order.

The critical instrumentation point is rows 2 and 3 being separate counters. 8.2 §8's stale-response rejection is correct behaviour — and it makes "responses arrived" and "responses were accepted" different numbers. A design exposing only the accepted count cannot distinguish "the peer never answered" from "the peer answered and every answer was rejected as stale", which are opposite problems with opposite owners.

32. Flagship Trace 2 — the Stale Discovery Response

Illustrative. A peer that restarts mid-exchange, and the local side correctly rejecting its answers.

Four readings, and the first is the whole point.

The raw counter reads 34 and the matched counter reads 0. That pair is the finding: the peer is present and talking, and not one of its answers belongs to a live request. A design exposing only matched == 0 would report "no peer" — the opposite conclusion.

The rejections are correct. 8.2 §8's epoch guard is working exactly as designed. The bug is not in the rejection; it is that the peer is stuck at epoch 4 — so the investigation moves to the peer, with specific evidence.

The first fault (§22) reads PEER_RESPONSE_TIMEOUT, attempt 1, which is accurate and incomplete on its own; the counter pair is what turns it into a diagnosis.

And this is why §31's rows 2 and 3 must be separate counters. One counter cannot express "arrived but not accepted", and that state is the entire content of this failure.

33. Family C — Configuration Mismatch

Both ends present. Training evidence exists. No compatible configuration is ever agreed.

This family is characteristically misdiagnosed as a PHY failure, because the link visibly does electrical work and still fails.

Three readings.

An empty intersection is not a defect. 20.2 §24 and 20.7 §32: two devices with no common configuration should fail to link, explicitly and cleanly. The correct outcome is an explicit failure, and a design that instead picks a default has the much worse bug — two dies operating on configurations neither agreed.

A committed configuration that does not match the independent rule is a negotiation bug, and it is found by computing the expected result offline from both capability sets rather than by reading the design's own computation (20.2 §19's shared-function failure).

And the discriminating observation against family D is that training evidence is valid. deps_now.training_evidence == 1 with cfg_agreed == 0 eliminates every training hypothesis in one bit — which is §8's waveform lesson applied one dependency along.

34. Family D — Training: Localise and Hand Off

This chapter's job with training is to establish that training is the blocker, and to characterise it enough to know which of 21.2's three sub-problems applies. Then stop.

Three observations, and each selects a different section of 21.2:

The minimum instrumentation this chapter needs from training is two bitmaps — which lanes have ever produced a valid measurement, and which have ever qualified (21.2 builds them):

// ILLUSTRATIVE. Two bitmaps, read once, and they select the sub-problem.
logic [NUM_LANES-1:0] lane_seen_q;        // ever produced a valid measurement
logic [NUM_LANES-1:0] lane_qualified_q;   // ever reached the qualified state

Worked, on three distributions:

Two notes.

The third row is the one that stops a wasted investigation. All lanes qualified and training still reports incomplete means the failure is in the commit or validation path, not in the lanes — and a debugger who keeps looking at lanes is looking where the evidence says not to.

And row 2 names a lane. "Lane 3 measures and never qualifies" is a starting point with an owner; "training fails" is not. 21.2 takes it from here.

35. Family E — Calibration

8.4 owns the design. The debug question is narrow and it is the same shape as family D's third row.

Separate "a pattern was recognised" from "a setting was committed."

The characteristic misdiagnosis: the active setting never changed, so the conclusion is that calibration did nothing. 8.4 §4's three copies of the setting is why that is wrong — candidates may have been evaluated and rejected many times, and the active register is the last thing to move. 21.2 §-on-candidate-history develops the same point for training, and the instrumentation lesson is identical: exposing only the active value hides the entire search.

36. Family F — Link Initialisation and Validation

Everything below is true and the link is still not usable. Two sub-cases, and §25's table separates them.

The bilateral sub-case is the one §8's waveform shows, and it is worth repeating the discriminator because it is cheap and decisive: deps_now with the local bits set and the bilateral bit clear eliminates the entire local stack in one read.

37. Family G — Operational Reached, Then Lost

The family whose user-facing description is identical to a hang and whose cause is completely different.

The reported symptom: "the link never comes up."

What actually happened: the link came up, admitted traffic, faulted, and re-ran bring-up. Repeatedly.

Three observations that identify it, and any one is sufficient:

And the reason a snapshot misses it entirely: at any given instant the link is in TRAINING or DISCOVERY, exactly as a hang would be. A debugger who reads the current phase, sees TRAINING, and concludes training is broken has been misled by a correct observation — which is §9's argument in its most expensive form.

38. Flagship Trace 3 — the Hidden Brief ACTIVE

Illustrative event trace. The OPERATIONAL residency is one cycle, and everything else is a consequence.

Five readings.

Row 6 is two cycles long and it is the entire diagnosis. Without §12's trace ring, OPERATIONAL is invisible — any snapshot taken during the run finds TRAINING, DISCOVERY or RETRY.

deps_ever.perm == 1 is a one-bit version of the same finding, available without a trace, which is why §10's bitmap is worth eleven flops.

The first fault points at a resource cause, not a bring-up cause (§26's failure shape): the link became operational before something downstream was ready. So the investigation moves out of Module 21's bring-up family and into 21.3's resource family.

Seventeen identical attempts is itself information. A failure that reproduces identically every attempt is static — a missing initialisation, a wrong constant, an ordering bug — rather than marginal. A flapping failure would show different attempt counts and different faults (§24's count patterns).

And the sticky first fault at attempt 1 is what makes rows 9–17 skippable. 14.5 §8's sticky capture: sixteen derivative attempts generated nothing new, and the register says so by not changing.

39. The Recovery Loop Versus the Hang

Four distinguishing observations, and they cost almost nothing to collect.

Two notes.

The published state names help here. 8.6 §2 records that PHYRETRAIN retrains a link that was working while TRAINERROR handles an error during training — so a design that distinguishes them gives the loop away in the state history alone. A trace passing through the working-link retrain path proves the link had been working.

And the fix targets differ completely. A hang's fix is in whatever dependency never became true; a loop's fix is in whatever faults immediately after admission — a resource that was not initialised, a credit agreement not established (19.5 §35), a peer not actually ready (§26). Investigating the bring-up path on a loop finds nothing wrong, because nothing in the bring-up path is wrong.

40. The Cause Tree

The chapter's second centerpiece. Two top-level branches, decided by one bit.

Was traffic permission EVER derived?   (deps_ever.traffic_permission)
 
NO — the link never became operational
├── A. reset / prerequisite            §28
│      A1 clock absent in a domain          §30
│      A2 one reset domain never released
│      A3 asynchronous release, illegal state
│      A4 reset re-asserted later           (shape 2)
├── B. discovery                       §31
│      B1 no request emitted
│      B2 no response observed at all
│      B3 responses observed, none matched  §32  (stale / epoch)
│      B4 compatibility never evaluated
├── C. configuration                   §33
│      C1 capability sets do not intersect  -> CORRECT refusal, not a bug
│      C2 committed value != independent rule  (negotiation bug)
│      C3 committed but never marked agreed
├── D. training                        §34  -> 21.2
│      D1 no lane produced evidence
│      D2 some lanes never qualified
│      D3 all qualified, never completed
├── E. calibration                     §35
│      E1 no measurements
│      E2 no candidate ever valid
│      E3 candidate valid, never committed
└── F. link init / validation          §36
       F1 local post-training work incomplete
       F2 bilateral agreement absent        §25
 
YES — it became operational and was lost
├── G. first traffic faults immediately §37, §38
│      G1 a resource was not initialised   -> 21.3
│      G2 credit agreement not established -> 19.5 §35, 21.3
│      G3 integrity / replay state invalid -> 19.3
│      G4 peer was not actually ready      §26
├── H. it survives briefly, then faults under load  -> 21.3
├── I. it flaps at a marginal condition   -> 14.2 §19's flap detector
└── J. a configuration change re-entered bring-up   (not a failure at all)

Three properties of this tree.

The root is one bit (§10), so the first branch is free.

C1 and J are not defects. A correct refusal on incompatible capability sets and a deliberate reconfiguration both look like a bring-up failure and are correct behaviour. A tree without those leaves sends investigations after non-bugs.

And four leaves hand off out of this chapter — D to 21.2, G1/G2/H to 21.3, I to 14.2. A cause tree whose every leaf is inside its own chapter is a tree that has claimed too much.

41. Do Not Debug From the Final State

The state at capture time is frequently derivative, and three examples show how far.

The general rule, and it is §1 restated operationally:

Find the first transition onto an unhealthy path, not the state the design ended in. The phase trace gives you the transition; the first-fault register gives you the cause; and the ever bitmap gives you the branch (§40).

And the reason this is hard in practice is that the final state is the only thing a simple status read produces. A design whose observability is one status register forces final-state debugging, which is why §49's budget puts the trace ring and the ever bitmap ahead of almost everything else.

42. Differential Tracing — Align by Milestone

Comparing a failing run against a known-good one is the highest-yield technique in this chapter, and it only works if the alignment is semantic.

Align by milestone, never by cycle number. Two runs diverge in timing constantly and legitimately; what matters is the first milestone one of them is missing.

Worked:

KNOWN GOOD                          FAILING
-----------                         -------
reset_release        t=0            reset_release        t=0
clock_alive          t=0            clock_alive          t=0
peer_seen            t=12           peer_seen            t=14
train_start          t=20           train_start          t=25
first_lane_ok        t=180          first_lane_ok        t=210
all_lanes_ok         t=340          -- ABSENT --
calib_commit         t=360          -- ABSENT --
cfg_commit           t=402          -- ABSENT --
bilateral_ok         t=410          -- ABSENT --
operational          t=411          -- ABSENT --
 
FIRST MISSING MILESTONE: all_lanes_ok
=> the failure is in training convergence -> §34, then 21.2
=> everything before it MATCHES -> reset, clock, discovery, and the
   measurement path are all eliminated

Three readings.

The timing differences are noise. t=12 versus t=14, t=180 versus t=210 — a cycle-aligned comparison would flag every one of these and bury the real finding.

first_lane_ok being present is the discriminating observation. At least one lane measured and qualified, so the measurement path and the qualification logic both work in general — which eliminates 21.2's first sub-problem entirely and points at the per-lane case.

And the milestones before the divergence are the valuable half. Five milestones matched; five subsystems eliminated, in one comparison, with no waveform.

43. The Milestone Alphabet

A small closed set, chosen so every milestone is observable from evidence the design produces — the same constraint 20.2 §7 applies to a reference model's phases.

Two notes.

Seventeen milestones, and every one is a single bit or a counter transition. That is what makes the alphabet usable in silicon — a trace of seventeen events per attempt is small enough for a bounded ring (§12) and readable enough to diff by hand.

And first_admit and first_fault are in the list deliberately. They are what turn §38's family-G trace into a diagnosis: operational followed immediately by first_admit followed immediately by first_fault is the loop's signature, and all three are in the alphabet.

44. Assertions That Localise

Not a property catalogue — 20.3 owns that. These are the properties whose failure message names a bring-up blocker.

Two notes.

Rows 3, 4 and 5 assert the instrumentation, not the design. 20.4 §42's self-check discipline in hardware: a diagnostic that can lie must be checked, because every conclusion drawn from it is downstream of its correctness.

And the last row's bound must be derived per phase (20.2 §17), never a global constant — which is §19's whole argument.

The four properties from that table that are specific to bring-up debug, written out — the rest are 20.3's:

// MANDATORY. The phase register holds a legal encoding. After a bad reset
// release (§28 row 3) it may not, and every observation downstream of an
// illegal phase is uninterpretable.
property p_phase_encoding_legal;
  @(posedge clk) disable iff (!por_n)
    phase_q inside {PH_PREREQ, PH_DISCOVERY, PH_TRAINING,
                    PH_VALIDATE, PH_OPERATIONAL, PH_RETRY, PH_FAILED};
endproperty
a_phase_encoding_legal: assert property (p_phase_encoding_legal);
 
// MANDATORY. Only legal arcs, via an INDEPENDENTLY written arc function —
// not the controller's own next-state logic (20.3 §19).
property p_phase_arc_legal;
  @(posedge clk) disable iff (!por_n)
    $changed(phase_q) |-> chk_legal_arc($past(phase_q), phase_q);
endproperty
a_phase_arc_legal: assert property (p_phase_arc_legal);
 
// MANDATORY. Each phase is bounded by ITS OWN derived bound (§19). One
// property instance per phase, with a per-phase constant — never one global.
property p_phase_bounded(int unsigned ph, int unsigned bound);
  @(posedge clk) disable iff (!por_n)
    (phase_q == ph) |-> (phase_age_q <= AGE_W'(bound));
endproperty
a_prereq_bounded:   assert property (p_phase_bounded(PH_PREREQ,   PREREQ_BOUND));
a_disc_bounded:     assert property (p_phase_bounded(PH_DISCOVERY, DISC_BOUND));
a_train_bounded:    assert property (p_phase_bounded(PH_TRAINING,  TRAIN_BOUND));
a_validate_bounded: assert property (p_phase_bounded(PH_VALIDATE,  VALID_BOUND));
 
// MANDATORY. The attempt counter is monotonic within a power cycle. A decrease
// means a reset reached a scope that should not have touched it (19.6 §27), and
// every attempt-scoped conclusion in the trace becomes unsound.
property p_attempt_monotonic;
  @(posedge clk) disable iff (!por_n)
    (attempt_q != $past(attempt_q))
      |-> (attempt_q == EPOCH_W'($past(attempt_q) + 1));
endproperty
a_attempt_monotonic: assert property (p_attempt_monotonic);
 
// MANDATORY. A retry must invalidate attempt-scoped evidence. Ledger column 6
// (§6): local training evidence is attempt-scoped, so it cannot survive an
// attempt boundary — that is 21.2's stale-evidence bug, asserted here.
property p_retry_invalidates_attempt_evidence;
  @(posedge clk) disable iff (!por_n)
    (attempt_q != $past(attempt_q)) |=> (deps_now[DEP_TRAIN_EVIDENCE] == 1'b0);
endproperty
a_retry_invalidates_attempt_evidence:
  assert property (p_retry_invalidates_attempt_evidence);

Three notes on what these buy in a debug context specifically.

p_phase_arc_legal uses an independently written arc function, not the controller's next-state logic (20.3 §19). A property built from the design's own transition function asserts that the controller agrees with itself — which is precisely the tautology that lets an illegal shortcut through, and §5's shape 4 is that shortcut.

p_attempt_monotonic protects the trace's interpretability rather than the design's function. 19.6 §27's reset-matrix row: an attempt counter cleared by something narrower than a power cycle makes every "attempt N" annotation in the trace ring meaningless, and §23's first-versus-last discriminator depends on it.

And p_retry_invalidates_attempt_evidence is the bring-up-side statement of 21.2's flagship bug. Evidence scoped to attempt 7 must not satisfy attempt 8 (§6, column 6). Asserted here it fails at the attempt boundary; discovered in 21.2 it presents as a lane that qualified on a stale sample — same defect, and the earlier failure is far cheaper.

45. Coverage as a Debug Question

When a bring-up failure appears in silicon, one question is worth asking before any waveform:

Did the DV regression ever exercise this path?

Four checks, and each has a specific consequence:

Two notes.

An uncovered path reframes the investigation. A silicon failure on a path DV never ran is expected rather than mysterious, and the fix has two parts: the bug, and the coverage gap that let it ship. 20.5 §50's classification — this is the stimulus missing case discovered the expensive way.

And coverage is not debug. It tells you whether the path was tested; it does not tell you what is wrong with it. Reading a coverage report instead of the first-fault register is the mirror of §41's error.

46. Reproducing a Failure in DV

A trace becomes a test by synchronising on milestones, never on cycle counts.

// ILLUSTRATIVE. The failing trace of §38, as a deterministic sequence.
// Every wait is on an OBSERVED milestone (§43), not a delay.
task reproduce_family_g();
  // 1. The configuration that failed — read back, not assumed (20.7 §38).
  set_configuration(cfg_from_failing_run);
 
  // 2. The peer behaviour class that was present. Not "a peer" — the class.
  peer_agent.set_latency_class(LATENCY_CLASS_FROM_TRACE);
  peer_agent.set_bilateral_delay(BILATERAL_DELAY_FROM_TRACE);
 
  // 3. Wait for the milestone the fault followed — NOT #500.
  wait (milestone_seen(M_OPERATIONAL));
 
  // 4. Admit exactly one object, as the trace shows.
  protocol_agent.admit_one();
 
  // 5. The expectation, stated positively (20.1 §44).
  //    The trace says a fault followed within a few cycles.
  fork
    begin wait (milestone_seen(M_FIRST_FAULT));
          `uvm_info("REPRO", "reproduced: fault after first admission", UVM_LOW) end
    begin repeat (REPRO_BOUND) @(posedge clk);
          `uvm_error("REPRO", "not reproduced — the trace's conditions differ") end
  join_any
  disable fork;
endtask

Three notes.

Step 3 is the whole technique. The fault followed operational by two cycles in the trace; waiting for operational reproduces that relationship regardless of how long bring-up takes in simulation — which will be a completely different number from silicon.

Step 5's failure message is "the trace's conditions differ", not "the design is fine". A non-reproduction means the reproduction is missing a condition — and that is a finding about the reproduction, which sends you back to the trace for what else was different.

And the peer's class is reproduced, not its exact timing. 20.6 §58's adversarial peer: matching a latency class is reproducible; matching an exact cycle count is not, and a reproduction that depends on the latter is fragile.

47. Wrong Reproduction — Arbitrary Delays

// WRONG — cycle-count synchronisation.
task reproduce_bad();
  set_configuration(cfg);
  #500ns;                       // "about when it comes up"
  protocol_agent.admit_one();
  #100ns;
  if (!fault_seen) `uvm_error("REPRO", "not reproduced")
endtask

Four properties.

It reproduces intermittently. Simulation bring-up takes a different time than silicon and a different time on every seed; #500ns sometimes lands before operational and sometimes after, so the test passes and fails at random.

When it lands early, it tests the wrong thing. An admission before operational exercises §26's admission-gating assertion — a real check, and not the bug being reproduced, so the failure is misattributed.

The natural fix makes it worse. Increasing the delay to #5us makes it land after operational reliably and also after the fault, the recovery and two more attempts — so the observation is of attempt 3, not attempt 1.

And it cannot be tightened into a regression test. 20.2 §16's fixed-bound argument: a delay-based reproduction has no derivation, so nobody can adjust it correctly when the configuration changes.

48. The Instrumentation Budget

World-class bring-up debug state is small and deliberate. 19.6 §43's observability contract, sized for this problem.

Worth its flops, in priority order:

Two properties of that list.

It is roughly a few hundred flops for a subsystem with tens of thousands, and it converts "the link does not come up" into a named blocker without a simulation.

And it is ordered by how much it eliminates per bit. The two bitmaps are first because eleven bits each split the entire hypothesis space; the trace ring is fourth because it is the only item that catches family G, which no snapshot can.

49. What Not to Expose

More observability is not better observability, and four categories are worth refusing.

And two specific anti-patterns for this chapter.

Do not clear a diagnostic to "get a clean run." 14.2 §14 and 14.5 §25: the first fault is the most valuable register in the design, and a debug procedure whose first step is a clear has destroyed it. §1's principle in operational form.

And do not restart repeatedly hoping for a different outcome. Seventeen attempts produced one useful record (§38); an eighteenth produces nothing unless the run is changed deliberately, and each restart risks overwriting a non-sticky diagnostic.

50. Silicon Versus Simulation Observability

The same method, two evidence budgets — and the mapping is what makes a pre-silicon investment pay off later.

Two consequences.

Every row where silicon "needs a register" is a design decision that must be made before tape-out. 20.7 §50's observability argument: a device that cannot produce a first-divergence report is a black box in exactly the situation where attribution matters most — and for a die-to-die link that situation involves another company.

And the method transfers even where the evidence does not. A team that debugs pre-silicon failures with the dependency ledger, the milestone alphabet and the first-versus-last comparison already knows how to read the silicon registers, because the registers were designed to answer the same questions. 21.7 develops the lab side.

51. Debug Taxonomy

Seven symptoms, each with the observation that resolves it fastest.

Nothing happens at all — no phase transitions, ever. Clock first (§30). Then per-domain reset release. A dark domain produces exactly this and nothing downstream is interpretable.

Progress stops at the same phase every attempt. Read deps_now in that phase (§9), then deps_ever (§10). The pair names the blocker and says whether it ever worked.

Progress stops at a different phase each attempt. Something marginal or environmental. Read invalidate_count_q per dependency (§24) — a high count on one dependency is a flapping producer; high counts on several with equal values is one upstream cause.

A timeout fires on a link that was converging. §19 — the watchdog is on residence rather than progress. Check whether progress_age was small when the timeout fired.

The link hangs and no timeout ever fires. §21 — a wrapping age counter. The signature is a small progress_age alongside a huge wait_cycles_q.

TRAINERROR or FAILED at capture. §41 — read the first fault, not the state. If first and last differ, the state is derivative.

And "it never comes up" with an alternating reason distribution. §37 — it comes up and falls. Check deps_ever.traffic_permission; one bit settles it.

52. Debug Checklist

A UCIe endpoint does not become usable. In order:

1. Is the clock running in every relevant domain (§30)?
2. Did every reset domain release (§28)?
3. Was the release synchronous to a live clock?
4. Is the phase register holding a legal value?
5. What is the earliest phase in the trace ring (§12)?
6. Did the ring wrap — is the earliest record actually the first (§12)?
7. What is the last phase entered?
8. What does deps_now read in that phase (§9)?
9. Which dependencies are false?
10. For each false dependency: was it ever true (§10)?
11. For each regressed dependency: what was the invalidation cause (§24)?
12. How many times has each dependency been invalidated (§24)?
13. Do several dependencies share an invalidation count — one upstream cause?
14. Which waiting reason dominates the time (§16)?
15. Is the distribution concentrated or alternating (§37)?
16. Is deps_ever.traffic_permission set — did the link ever become operational (§40)?
17. Does the phase trace contain OPERATIONAL (§38)?
18. Was a request emitted, and were raw responses observed (§31)?
19. Were any responses matched, or all rejected as stale (§32)?
20. Do the two capability sets intersect at all (§33)?
21. Was a configuration committed, and does it match an independent rule (§33)?
22. Which lanes have ever produced a measurement (§34)?
23. Which lanes have ever qualified (§34)?
24. Was a calibration candidate ever valid, and was it committed (§35)?
25. Is the missing dependency local or bilateral (§36)?
26. What is the first fault — cause, phase, attempt, timestamp (§22)?
27. What is the last fault, and does it differ (§23)?
28. Which attempt produced the first fault — 1, or a later one (§23)?
29. Is progress_age large, or is phase_age large (§18)?
30. Did any age counter wrap (§21)?
31. Is the watchdog on progress or on residence (§19)?
32. Which epoch was in force at each milestone?
33. Did the peer restart (§32)?
34. Did recovery ever run, and how many times (§39)?
35. What is the first missing milestone versus a known-good run (§42)?
36. Which milestones matched — what does that eliminate (§42)?
37. Which assertion failed first (§44)?
38. Did DV ever exercise this path (§45)?
39. Can the failure be reproduced by synchronising on a milestone (§46)?
40. What observation would prove the current hypothesis wrong?

Question 40 is the one that separates debugging from guessing. A hypothesis with no disproving observation is not a hypothesis — and asking for it before collecting more evidence is how an investigation stays bounded.

53. Common Misconceptions

"If it never works, start with the PHY." Start with the clock (§30), then the dependency bitmap (§9). The PHY is one of seven families and it is the most expensive to investigate — families A and C are answerable in minutes and produce identical symptoms.

"The current FSM state tells you the root cause." It tells you where the controller is. VALIDATE is true at cycle 5 and at cycle 900,000, and it never says which of its three inputs is missing (§9).

"TRAINERROR means training is the original bug." It may be attempt 17's derivative of a discovery timeout at attempt 1. Read the first fault (§23).

"One global timeout is enough." It fires on a healthy slow phase and misses a stalled fast one. Two watchdogs, two meanings: progress and residence (§18, §19).

"Resetting repeatedly helps debug." It destroys non-sticky diagnostics and produces nothing new — seventeen identical attempts yield one useful record (§38, §49).

"The last error is the most important error." It is usually a consequence. The difference between first and last is the finding (§23).

"If local training is done, the link is done." Local completion is not bilateral completion, and asserting operational from local evidence produces a recovery loop reported as a hang (§26).

"OPERATIONAL was never observed, so it never happened." A snapshot cannot see a two-cycle residency. deps_ever and the trace ring both can (§38).

"Dump every signal — more observability is better." A per-cycle trace wraps before anything interesting happens, and every exposed register becomes a compatibility constraint (§49).

"A timeout identifies the failed block." It identifies where the watchdog was, which may be several layers from the cause — and a wrapping counter means no timeout fires at all (§21).

"Compare failing traces by cycle number." Timing differs legitimately between runs. Align by milestone; the first missing one is the finding (§42).

"A recovery loop and a training hang are the same thing." They have opposite fixes, and one bit distinguishes them (§39).

"Clear the fault registers to get a clean run." The first fault is the most valuable register in the design, and clearing it is the one irreversible step in the whole procedure (§49).

54. Understanding Check

55. Summary and What Comes Next

Debug is identifying the first observation that could not have occurred in a correct implementation — not the last bad signal, and not the state the design ended in.

Bring-up is a dependency graph, not one state machine. A hang has four shapes: never true, regressed, true-but-unobserved, or advanced-then-rolled-back — and two bitmaps of about eleven bits each distinguish them.

The phase never explains why the link is in it. VALIDATE is true forever and says nothing about which of its three inputs is missing.

Progress and residence are different quantities. A watchdog on residence fires on a converging link; one on progress fires when the phase has actually stopped — and a wrapping age counter means no watchdog fires at all, with a contradiction between two counters as its only evidence.

Read the first fault, and compare it with the last. The difference is the finding, and the attempt number separates a static failure from a degradation.

Local completion is not bilateral completion. Deriving operational from local evidence produces a recovery loop that a user reports as a hang, and one bit — was traffic permission ever derived — settles which one you have.

Align differential traces by milestone. The first missing milestone localises; the matching prefix eliminates, and the prefix is the more valuable half.

And keep the instrumentation small and sticky. A few hundred flops, none of it cleared by a recovery, because a debug procedure whose first step is a clear has destroyed the register it needed.

The ledger has localised the hang. When the answer is training, this chapter deliberately stops — it can say which lanes measured and which qualified, and no more. The next problem is harder, because training is not idle: it is measuring, scoring and retrying continuously, and none of that activity converges into a stable usable result. The next chapter debugs why the candidate keeps failing.

• 21.2 — Training Failures — training never converges, and the root-cause path.

Browse the full path on the UCIe tutorials index.