Senior Debugging

Chapter 25.8 built the environment that catches bugs before silicon. This is what happens when one gets through — and the interviewer is listening for a method, not a list of things that can go wrong.

1. What They Ask

"Walk me through how you would debug a UCIe bring-up failure."

Or: "The link is up but traffic doesn't work — what now?" · "You have a hang. Where do you start?" · "What would you have built into the design to make this debuggable?"

Candidates who answer with a list of possible causes lose. Candidates who answer with a narrowing procedure win, because the list is infinite and the procedure is not.

2. The One-Sentence Model

Debugging is finding the first boundary where observed behaviour diverges from the expected contract — not looking at waveforms until something seems wrong.

Two consequences follow immediately. You need to know what the contract at each boundary is, which is what Modules 19–25 were for. And you need evidence at each boundary, which is a design decision made years earlier (§10–§14).

3. What They Are Really Testing

And the last row is a real senior signal. "Before anyone resets it, I want the first-failure register and the trace dumped" is a sentence that comes from having lost a bug that way.

4. What You Can Safely Assert

5. The Answer Ladder

15 SECONDS — the method, not a cause.
 
  "I'd start by finding the first contract that failed, not by guessing at a
   block. Bring-up has a natural order — power, clocks, reset, sideband,
   training, negotiated config, then traffic — and I'd find the earliest
   stage where what I observe doesn't match what that stage promises."

90 SECONDS — adds the sharp part.
 
  "The method is to locate the first divergence. Bring-up narrows in a
   fixed order, so I'd walk it: is power and clock present, did reset
   deassert in the right order, did sideband come up, did training
   complete, what configuration did it actually negotiate — and only then
   traffic.
 
   The trap is that people stop at 'the link is ACTIVE' and treat that as
   success. ACTIVE tells you the link trained. It tells you almost nothing
   about whether traffic is being accepted, whether resources are being
   returned, or whether anything is completing.
 
   So past ACTIVE I switch to counters rather than waveforms. Offered
   versus accepted, allocated versus released, returns generated versus
   returns applied. Those staged counts localise the failure to one
   boundary in a single read, and then I look at the trace around that
   boundary rather than scrolling a waveform."
 
  [STOP — "counters rather than waveforms" is the hook, §21.]

Three properties.

Neither version names a cause, which is deliberate — naming a likely cause early is how a debug session becomes a search for confirmation.

"ACTIVE tells you the link trained" is the sentence that separates levels (§6). It is one clause and it demonstrates you have been past bring-up into traffic debug.

And "counters rather than waveforms" is a methodological claim, not a tool preference: a waveform shows you one window; staged counters show you which window to open (§12).

6. The Bring-Up Hierarchy — and Where It Stops Helping

The ACTIVE row is the one that matters in an interview. Most bring-up debug guides stop there. Everything hard happens below it, and the tools change: above ACTIVE you use waveforms and state, below it you use counters and identity.

7. Reading the First Divergence

Four readings, and this is the trace to be able to describe.

link_active is high for the whole window and is completely uninformative — which is exactly why it is drawn. §6's row in one signal.

release is pulsing, so the receiver is draining normally. That eliminates the consumer, which is the hypothesis most people reach for when credit falls.

return_gen stops at cycle 3 while release continues — an exact boundary with no legitimate transient (21.4 §9), so this is a bug at the moment it appears, not a latency artefact.

And credit is monotonically falling. That is what distinguishes a leak from a window limit (25.6 §19): a window limit sawtooths back to full every round trip; a leak ratchets down and never recovers.

8. Flagship — Throughput Collapses Under Sustained Traffic

Symptom: the link trains, first traffic works, and under sustained load throughput collapses to zero with credit at zero and the receiver apparently idle.

Five readings.

Step 3 is the discriminating observation and it costs one register read. Reaching for "leak" before checking recovery is the most common misdiagnosis in this failure class.

Step 6 is the trap in the counter walk. The alloc-minus-release difference is the receiver's occupancy — a tool that flags it reports a leak equal to the fill level (21.4 §9).

Step 7 is an exact boundary — release and return generation should be the same event (21.4 §38) — so a difference of 848 needs no quiescence argument and no latency model.

Step 10 is half the value of the method. Four subsystems eliminated by four closed comparisons, each of which is a plausible hypothesis someone would otherwise spend a day on.

And the likely mechanism is now narrow: either the three derivations of "release" disagree (21.4 §38), or the per-entry return guard is stuck set so legitimate releases find it already returned (25.6 §15).

9. Flagship — Recovery, Then a Hang and a Stray Completion

Symptom: the link recovers successfully, one transaction never completes, and a later transaction receives a completion it did not expect.

Five readings.

Steps 4–6 are three hypotheses and step 7 eliminates two of them with one comparison — which is §15's whole discipline.

Step 9 is why rejection counters matter. Without them, a correctly rejected stray is indistinguishable from a lost one (21.4 §44) — and the investigation goes to the transport, which is innocent.

Step 10 is the first divergence, and it is earlier than the first observed symptom (21.6 §30). The hang at step 3 is a consequence.

Step 12 is the escalation from hang to wrong data, prevented by generation matching (25.4 §12).

And every transport instrument was clean throughout — which is why a waveform of the link would have shown nothing at all.

10. First Fault Is Not First Divergence

And §9 is the example: the first fault was a rejected completion; the first divergence was the table clear that happened earlier and flagged nothing. A design only flags what it was built to notice.

11. RTL — First-Failure Capture

// ILLUSTRATIVE ONLY. Capture the FIRST failure with enough context to place it.
// Sticky by construction — a cascade must not overwrite the cause.
module first_fault_capture #(
  parameter int ERR_W = 6,
  parameter int SEM_W = 16
) (
  input  logic              clk,
  input  logic              rst_n,
  input  logic              diag_clear,        // EXPLICIT software clear only
  input  logic              err_fire,
  input  logic [ERR_W-1:0]  err_class,
  input  logic [SEM_W-1:0]  err_sem_id,
  input  logic [7:0]        err_generation,
  input  logic [7:0]        cfg_epoch,
  input  logic [3:0]        link_state,
  input  logic [31:0]       credit_snapshot,
  input  logic [31:0]       outstanding_snapshot,
  input  logic [47:0]       timestamp,
 
  output logic              ff_valid,
  output logic [ERR_W-1:0]  ff_class,
  output logic [SEM_W-1:0]  ff_sem_id,
  output logic [7:0]        ff_generation,
  output logic [7:0]        ff_cfg_epoch,
  output logic [3:0]        ff_link_state,
  output logic [31:0]       ff_credit,
  output logic [31:0]       ff_outstanding,
  output logic [47:0]       ff_time,
  output logic [15:0]       ff_count          // how many since — §11's contract
);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      ff_valid <= 1'b0;
      ff_count <= '0;
    end else if (diag_clear) begin
      ff_valid <= 1'b0;
      ff_count <= '0;
    end else if (err_fire) begin
      // STICKY FIRST. Everything below the guard is captured ONCE.
      if (!ff_valid) begin
        ff_valid       <= 1'b1;
        ff_class       <= err_class;
        ff_sem_id      <= err_sem_id;
        ff_generation  <= err_generation;
        ff_cfg_epoch   <= cfg_epoch;
        ff_link_state  <= link_state;
        ff_credit      <= credit_snapshot;
        ff_outstanding <= outstanding_snapshot;
        ff_time        <= timestamp;
      end
      // The COUNT keeps rising regardless — one fault and ten thousand are
      // different findings, and the sticky register alone cannot tell them apart.
      if (ff_count != '1) ff_count <= ff_count + 16'd1;
    end
  end
endmodule

Architecture. One sticky snapshot plus a free-running count of subsequent faults.

State. Nine context registers, a validity bit, and a saturating counter. Roughly 150 flops — the cheapest high-value debug structure in a design.

Event. Captured on the first err_fire after a clear. Everything after increments only the count.

Contract. diag_clear must be separate from any functional reset. If a recovery path or a block-init routine clears this, the evidence disappears exactly when it is needed (21.7 §34) — and a driver that calls block_init() before dumping is the single most destructive thing software does to a debug flow.

Failure. §12. Also: capturing without the context fields gives you an error code and no way to place it — cfg_epoch and sem_id are what let you correlate the fault to a specific operation and configuration.

DV/debug. ff_count == 1 with a named sem_id is a specific event to reproduce; ff_count == 40,000 with the same class is a systematic mechanism. The pair distinguishes them, and neither register alone can.

12. Wrong RTL — Last Error Wins

// WRONG — plausible, common, and it destroys the causal evidence.
always_ff @(posedge clk) begin
  if (err_fire) begin
    err_code_q  <= err_class;      // <-- unconditional
    err_sem_q   <= err_sem_id;
    err_time_q  <= timestamp;
  end
end

The timeline:

Four properties.

The register now holds a symptom, and the causal fault at cycle 10,000 is unrecoverable — there is no other copy.

It is worse than useless because it is confident. An engineer reads TIMEOUT and opens a timeout investigation, which is 900 cycles and four subsystems away from the cause.

The fix is three characters — if (!ff_valid) — plus an explicit clear so the register can be re-armed deliberately.

And 14.5 §9 owns the design argument; what this chapter adds is the debug consequence: last-error-wins guarantees that the field return will contain the least informative value in the entire cascade.

13. RTL — Staged Event Counters

// ILLUSTRATIVE ONLY. Counters exist to localise a boundary in one read (§8),
// not as telemetry. Nine of them answer "which stage first diverges?"
logic [47:0] cnt_offered_q,   cnt_accepted_q;
logic [47:0] cnt_allocated_q, cnt_transmitted_q, cnt_retried_q;
logic [47:0] cnt_delivered_q, cnt_retired_q;
logic [47:0] cnt_return_gen_q, cnt_return_applied_q;
logic [47:0] cnt_recovery_entry_q, cnt_recovery_exit_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n || diag_clear) begin
    cnt_offered_q <= '0; cnt_accepted_q <= '0;
    cnt_allocated_q <= '0; cnt_transmitted_q <= '0; cnt_retried_q <= '0;
    cnt_delivered_q <= '0; cnt_retired_q <= '0;
    cnt_return_gen_q <= '0; cnt_return_applied_q <= '0;
    cnt_recovery_entry_q <= '0; cnt_recovery_exit_q <= '0;
  end else begin
    // OFFERED counts a held valid every cycle it is asserted — deliberately.
    // It measures OFFERED PRESSURE, and the gap to `accepted` is the
    // backpressure evidence (21.5 §20). Do not "fix" it to count objects.
    if (src_valid)                cnt_offered_q     <= cnt_offered_q + 48'd1;
    if (src_valid && src_ready)   cnt_accepted_q    <= cnt_accepted_q + 48'd1;
    if (alloc_fire)               cnt_allocated_q   <= cnt_allocated_q + 48'd1;
    if (tx_fire)                  cnt_transmitted_q <= cnt_transmitted_q + 48'd1;
    if (retx_fire)                cnt_retried_q     <= cnt_retried_q + 48'd1;
    if (deliver_fire)             cnt_delivered_q   <= cnt_delivered_q + 48'd1;
    if (retire_fire)              cnt_retired_q     <= cnt_retired_q + 48'd1;
    if (return_gen_fire)          cnt_return_gen_q  <= cnt_return_gen_q + 48'd1;
    if (return_apply_fire)        cnt_return_applied_q <= cnt_return_applied_q + 48'd1;
    if (recovery_entry_fire)      cnt_recovery_entry_q <= cnt_recovery_entry_q + 48'd1;
    if (recovery_exit_fire)       cnt_recovery_exit_q  <= cnt_recovery_exit_q + 48'd1;
  end
end

Architecture. One counter per boundary event, wide enough not to wrap in a soak (21.4 §34).

State. Eleven 48-bit counters — about 530 flops, and they replace hours of waveform scrolling.

Event. One increment per observed event, observed at the boundary rather than derived from internal state (21.3 §10) — a counter fed from a design-internal signal inherits that signal's bugs.

Contract. cnt_offered_q counting held-valid cycles is intentional and must be documented, because the next engineer will try to "fix" it. The offered-versus-accepted gap is the backpressure measurement, and counting objects instead destroys it.

Failure. Counters that saturate rather than run wide make late differences meaningless. Counters that reset on recovery make cross-recovery reasoning impossible — which is why diag_clear is the only clear.

DV/debug. Read in order and stop at the first unexplained difference (§8). A later mismatch is usually a consequence of an earlier one, so the walk direction matters.

14. RTL — the Trace Ring, and the Trigger Policy

// ILLUSTRATIVE ONLY. The trace is only useful if it contains the cycles BEFORE
// the fault. Recording continuously and freezing on the trigger is the whole
// design; recording FROM the trigger captures only the aftermath (§15).
localparam int DEPTH = 256;
 
typedef struct packed {
  logic [23:0] timestamp;
  logic [3:0]  ev_kind;
  logic [15:0] sem_id;
  logic [7:0]  generation;
  logic [7:0]  cfg_epoch;
  logic [3:0]  link_state;
} trace_ev_t;
 
trace_ev_t              ring_q [DEPTH];
logic [$clog2(DEPTH)-1:0] wr_ptr_q;
logic                   wrapped_q;    // sticky — the oldest entry is NOT first
logic                   frozen_q;
logic [7:0]             post_q;       // post-trigger entries still to take
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n || diag_arm) begin
    wr_ptr_q  <= '0;
    wrapped_q <= 1'b0;
    frozen_q  <= 1'b0;
    post_q    <= POST_DEPTH;
    // NOTE: ring_q is deliberately NOT reset. It is DEPTH x width bits, and
    // `wrapped_q` plus wr_ptr_q already tell the reader what is real.
  end else if (!frozen_q) begin
    if (ev_valid) begin
      ring_q[wr_ptr_q] <= ev;
      if (wr_ptr_q == ($clog2(DEPTH))'(DEPTH-1)) begin
        wr_ptr_q  <= '0;
        wrapped_q <= 1'b1;
      end else begin
        wr_ptr_q <= wr_ptr_q + 1'b1;
      end
 
      // Post-trigger countdown advances on ENTRIES, not cycles — a link that
      // goes quiet after the fault would otherwise freeze half-empty.
      if (trigger_seen_q) begin
        if (post_q == '0) frozen_q <= 1'b1;
        else              post_q   <= post_q - 8'd1;
      end
    end else if (trigger_seen_q && (post_q == '0)) begin
      // Freeze even with no further events, so a silent link ends in a
      // defined, dumpable state rather than staying armed.
      frozen_q <= 1'b1;
    end
  end
end

Architecture. A circular buffer always recording, frozen a bounded number of entries after the trigger.

State. DEPTH entries plus a pointer and three status bits.

Event. Writes on ev_valid; the post-trigger countdown advances per entry, so a link that goes quiet after the fault still captures the next POST_DEPTH things that actually happen.

Contract. wrapped_q is sticky and must be reported with the dump. A wrapped ring's oldest entry looks like the beginning of history and is not — and an analyst reasoning about a "causeless" first event is usually looking at a wrapped ring (21.6 §31).

Failure. §15. Also: resetting the payload array costs real area for no benefit, since wrapped_q and wr_ptr_q already describe what is valid.

DV/debug. Reconstruct order from the pointer: not wrapped → entries 0..wr_ptr-1 in order; wrapped → the oldest is at wr_ptr. Getting that backwards yields a dump that is correct and reversed.

15. Wrong — Recording From the Trigger

// WRONG — the ring arms ON the fault, so it holds only the aftermath.
assign wr_en = ev_valid && fault_seen_q;

Three properties.

The dump looks excellent — complete, ordered, timestamped, and entirely about consequences (14.5 §9's last-wins problem in the time domain).

Every hypothesis it supports is a recovery hypothesis, so the investigation converges on logic that is working correctly.

And the fix inverts one signal: gate on !frozen_q rather than on fault_seen_q. Record always; stop when something goes wrong. That is the whole design decision, and it costs nothing.

16. Designing the Experiment

One hypothesis, one experiment, two predicted outcomes. If you cannot say in advance what each outcome would mean, the experiment is not going to tell you anything.

BAD:      "let me try disabling retry and see what happens"
 
GOOD:     HYPOTHESIS  the failure needs a retry to occur
          EXPERIMENT  reduce the error rate so retries stop, hold everything
                      else constant
          IF PASSES   retry is necessary to the failure -> look at the retry
                      path's interaction with allocation
          IF FAILS    retry is NOT necessary -> eliminate it, and the earlier
                      counter walk (§8) is where to look next
          RESULT      ...

Three properties.

Both outcomes are informative, which is the test of a good experiment. "Let's see what happens" has a failure mode where nothing changes and you have learned nothing.

One variable at a time. Changing the error rate and the traffic mix and the configuration leaves you unable to attribute the result — 21.7 §25's bad experiment.

And record the reproduction rate with a denominator (21.7 §28). "It stopped failing" after a change that merely made it rarer looks identical to a fix, and 6/20 → 0/20 is weak evidence at that sample size.

17. Bisection

Reduce dimensions until it passes, then restore them one at a time.

And the most productive single reduction is outstanding depth to one. It eliminates every same-cycle and identity-reuse bug class at once — so if it still fails at depth one, a large family of hypotheses is gone.

18. Clocks, Reset and Domain Crossings

At bring-up, a surprising fraction of "protocol" failures are none of those things.

Two properties.

The pulse row produces the most confusing symptoms. A lost pulse looks like a missing event; a duplicated one looks like an inflation — and both present as accounting failures far from the crossing.

And the ordering row is the one to raise unprompted, because "did reset deassert in the right order?" is a question most candidates never ask, and a mis-ordered deassertion produces a link that trains and then behaves inexplicably.

19. Assertions Turn Silicon Bugs Into Simulation Failures

And the last row is worth calling out in an interview. The debug infrastructure is RTL and gets asserted too — a first-fault register that can be overwritten produces confidently wrong evidence, which is worse than no evidence at all.

The general framing: "Every one of these turns a symptom I'd otherwise chase across a package into a named failure at a specific simulation cycle."

20. Weak Answers

21. Capture Before You Reset

The reset that fixes the symptom destroys the cause. Before anyone power-cycles, retrains or re-initialises: dump.

And the software failure to name (21.7 §34): a driver that reads a fault code, logs a summary string, then calls block_init() has discarded the sem_id, generation, epoch, configuration and trace — everything that would have chosen the branch. The summary string survives and the evidence does not.

22. Controlling the Next Question

And the first is the strongest hook, because it is a methodological claim rather than a fact, and the follow-up lands on §8 — a ten-step walk that eliminates four subsystems and names an exact boundary.

23. Understanding Check

24. Summary

Five things.

Find the first divergent boundary (§2), not a likely block. The list of causes is infinite; the narrowing procedure is not.

ACTIVE proves the link trained (§6) — and past that point the tools change from waveforms and state to counters and identity.

Staged counters localise in one read (§8, §13), and half their value is the four subsystems they eliminate without investigation.

Capture the first fault, not the last (§11, §12), and make the trace contain the cycles before the fault (§14, §15) — both are one small change from being useless.

And capture before you reset (§21). The action that restores service destroys the evidence, and the driver that logs a summary and calls block_init() has thrown away everything that mattered.