Senior Debugging Question

27.9 built the thing that proves a design correct. This chapter is what happens when it was not — walk me through a real CXL bring-up debug — and it is the question that finds out whether somebody has actually done this or has only read the post-mortems.

We reproduced it. True, necessary, and where most answers begin and end: one observation of the failure, no cost attached to getting another, no layer ruled out, no root cause, and nothing said about why the test suite missed it.

1. The Engineering Problem — A Repro Is Not A Result

Reproducing it once is not reproducing it. One failure in forty runs at twenty minutes a run is eight hundred minutes for every look at it — and six looks is eighty hours of machine time before anybody has learned anything. Section 5.

One symptom belongs to several layers. "It did not enumerate" is consistent with five of them, and ruling out one leaves four still open and seven hundred and twenty minutes of checks to eliminate the rest. Section 9.

A buffer that wraps before the trigger recorded the consequence. A thousand-entry buffer with two hundred cycles kept after the failure captures eight hundred cycles of run-up against the four thousand wanted — a fifth of the cause. Section 10.

A workaround hides more than the bug. Disabling a region with nine known bugs in it while finding one root cause leaves eight bugs now unreachable, and eighteen percent of the performance gone to buy that. Section 11.

Why it escaped is a separate question from what it was. Six escapes with two causes named and one hole closed is four escapes whose route through verification nobody has traced — and the next bug takes the same route. Section 12.

This chapter against 26.7, stated precisely. That one owns the instruments — what a protocol analyser sees, what registers report, what the link layer exposes. This one owns the method: the arithmetic of narrowing a search, ordering hypotheses, and knowing when a debug is finished rather than merely quiet.

2. The One-Sentence Model

A bring-up debug is finished when the failure has been reproduced, when it has been isolated to one layer, when the moment of failure was actually observed, when a root cause has been named, when why it escaped verification is understood, and when the fix has been verified against the original failure — and "we reproduced it" is one of those six.

3. What This Chapter Owns

Some vocabulary, because debugging words are used loosely and each of these has a cost attached.

A repro is a procedure that produces the failure, and its useful property is not that it works but how often. A procedure that works one time in forty is a repro whose price is forty runs.

Bisection is halving a search space and testing which half contains the failure. It is the only technique in this chapter whose cost grows logarithmically rather than linearly, and it requires a space that can be halved and a test that answers reliably.

Observability is what can be seen at the moment of failure, which was decided when the board was laid out and the silicon taped out. It is not negotiable during the debug.

A layer is one of the stacked systems the symptom could be coming from — physical, link, protocol, configuration, software — and isolation means reducing the candidate set to one.

And escape analysis is the question of how the bug got past verification, which is separate from what the bug was and is the only part of a debug that pays forward.

4. Teaching-Model Boundary

Every model in this chapter is a teaching model, not a debug tool. It computes the one relationship the section is about and nothing else. There is no protocol analyser, no register map and no failure signature anywhere in this file.

Each model is built twice from one source. A parameter selects between the measured build, which counts what the debug actually costs and actually knows, and the repro build, which counts having seen the failure as having understood it. Every section's headline number is the gap between them.

5. RTL 1 — Reproducing It Once Is Not Reproducing It

Start with the number that prices every other technique in the chapter.

A repro's value is its rate, not its existence. Every technique below — bisection, hypothesis testing, verifying a fix — needs to see the failure again, and each of those observations costs the whole run divided by the probability that it fails. A procedure that reproduces one time in forty turns a twenty-minute run into eight hundred minutes of machine time per look.

That number is the debug's exchange rate, and it decides which techniques are affordable at all.

// RTL 1 - reproducing it once is not reproducing it. A failure that appears
// on one run in forty has a cost per observation, and every debug technique
// that needs to see the failure again is multiplied by that cost.
module repro_rate #(parameter int WE_REPRODUCED_IT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] attempts, hits, run_minutes, observations_needed,
  output logic [15:0] hits_ok, repro_pct, cost_per_hit, total_minutes,
  output logic        reliably_repro,
  output logic [7:0]  n_evals, n_intermittent,
  output logic        repro_err
);
  logic [31:0] r_q, c_q, t_q;
  logic [15:0] true_cost, true_total;
  logic        truly_intermittent;
  // A run cannot fail more often than it ran.
  assign hits_ok = (hits > attempts) ? attempts : hits;
  assign r_q = (attempts == 16'd0) ? 32'd0
             : (({16'd0, hits_ok} * 32'd100) / {16'd0, attempts});
  assign repro_pct = r_q[15:0];
  // What one observation of the failure costs in machine time.
  assign c_q = (hits_ok == 16'd0) ? 32'd9999
             : (({16'd0, attempts} * {16'd0, run_minutes}) / {16'd0, hits_ok});
  assign true_cost = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign cost_per_hit = (WE_REPRODUCED_IT != 0) ? run_minutes : true_cost;
  assign t_q = {16'd0, cost_per_hit} * {16'd0, observations_needed};
  assign true_total = (t_q > 32'd9999) ? 16'd9999 : t_q[15:0];
  assign total_minutes = true_total;
  assign reliably_repro = (cost_per_hit <= run_minutes) && (hits_ok != 16'd0);
  assign truly_intermittent = (true_cost > run_minutes) && (attempts != 16'd0);
  assign repro_err = evaluate && truly_intermittent && reliably_repro;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_intermittent <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_intermittent) n_intermittent <= n_intermittent + 8'd1;
    end
  end
endmodule

Forty attempts with one failure at twenty minutes a run is eight hundred minutes for each observation, and four thousand eight hundred for the six that the debug needs — where the we-reproduced-it view charges one run apiece and reports two hours.

Figure 1 — the exchange rate, and why it is the first number to establish. Both paths describe the same failure and the same hardware. The upper one is what a status update says; the lower one is what the calendar will say. A factor of forty is not a detail of the estimate — it is the difference between a debug that finishes this week and one that does not, and every technique in the rest of this chapter is priced in these units.

The last case is the distinction the cost figure exists to make. A slow deterministic failure — nine hundred minutes a run, failing every time — costs nine hundred minutes a look and is not intermittent. It is expensive and completely tractable: every technique works, they are simply slow. Conflating "expensive" with "intermittent" leads teams to reach for statistical approaches on a problem that bisection would settle.

The sixth case is the one that is easy to feel good about. One run in four sounds like a solid repro and costs four times a run per observation — eight hours for six looks. It is the rate at which people stop worrying about the rate, and it is already a factor of four on everything downstream.

The fourth case is the one to name precisely. A failure never reproduced is not an intermittent failure; it is an unreproduced one, and the model saturates the cost and declines to call it intermittent. Those are different states with different next actions: an intermittent failure needs more runs, and an unreproduced one needs a different procedure.

The degenerate case bounds it: a setup that has never been run reports no rate at all, which is the honest answer to a question nobody has asked yet.

The levers on the rate are worth listing, because "make it fail more often" is otherwise advice with no handles. Remove margin. Raise the frequency, lower the voltage, raise the temperature, shorten a timeout, tighten a delay — a failure that needs a coincidence happens more often when the window the coincidence has to land in is widened. Increase the traffic that matters and remove the traffic that does not, so that the suspicious transaction type occupies a larger fraction of the run. Reduce the run's setup, so the same wall-clock time contains more attempts. And run more copies at once, which is section 13's subject and improves the rate per hour without improving the rate per run.

The first of those is the one with a caveat worth stating. A failure that only appears outside the specified operating conditions may be a different failure, and a debug that chased a margin-induced symptom to a root cause has sometimes found a real bug and sometimes found the behaviour of a part being used out of spec. The discipline is to reproduce at the margin, find the mechanism, and then confirm the mechanism explains the in-spec failure too.

The third case is worth a line because the clamp says something real. More failures claimed than runs is clamped to the runs that happened rather than reported as a rate above a hundred percent — which sounds like arithmetic hygiene and is actually a common bookkeeping error, where a single run producing several error messages is counted as several failures and the rate is overstated by the multiplicity.

6. RTL 2 — Bisection Is The Only Cheap Technique

The second thing, and the one whose cost structure is different in kind from everything else.

Every other technique in a debug is linear in the search space. Reading code, reasoning about a trace, asking somebody who knows — all of them scale with how much there is to look at. Bisection is logarithmic: a thousand candidates is ten rounds, a million is twenty, and the difference compounds in favour of splitting rather than staring.

It needs two things. A space that can actually be halved — a commit range, a lane set, a device list, a configuration bit vector — and a test that answers reliably, which is section 5's subject arriving as a multiplier on every round.

// RTL 2 - bisection is the only technique whose cost is logarithmic, and it
// needs two things the failure rarely gives you: a search space that can be
// halved, and a test that answers reliably. With an unreliable test each
// round has to be repeated, and the logarithm is multiplied by section 5's
// cost per observation.
module bisect_search #(parameter int JUST_LOOK_AT_IT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] space_size, rounds_done, cost_per_round, repro_repeats,
  output logic [15:0] remaining, rounds_needed, true_cost, bisect_pct,
  output logic        narrowed,
  output logic [7:0]  n_evals, n_wide,
  output logic        bisect_err
);
  logic [31:0] c_q, b_q;
  logic [15:0] shrunk, true_remaining, rounds_ok;
  logic        truly_wide;
  // Each round halves what is left. Sixteen rounds is more than any search
  // space this model represents needs.
  assign rounds_ok = (rounds_done > 16'd16) ? 16'd16 : rounds_done;
  assign shrunk = space_size >> rounds_ok;
  assign true_remaining = (shrunk == 16'd0) ? 16'd1 : shrunk;
  assign remaining = (JUST_LOOK_AT_IT != 0) ? 16'd1 : true_remaining;
  // How many more rounds would finish the job: the number of halvings that
  // takes what is left down to one, which is what makes bisection cheap.
  assign rounds_needed = (true_remaining <= 16'd1)   ? 16'd0
                       : (true_remaining <= 16'd2)   ? 16'd1
                       : (true_remaining <= 16'd4)   ? 16'd2
                       : (true_remaining <= 16'd8)   ? 16'd3
                       : (true_remaining <= 16'd16)  ? 16'd4
                       : (true_remaining <= 16'd32)  ? 16'd5
                       : (true_remaining <= 16'd64)  ? 16'd6
                       : (true_remaining <= 16'd128) ? 16'd7
                       : (true_remaining <= 16'd256) ? 16'd8
                       : (true_remaining <= 16'd512) ? 16'd9 : 16'd10;
  assign c_q = {16'd0, rounds_needed} * ({16'd0, cost_per_round} * {16'd0, repro_repeats});
  assign true_cost = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign b_q = (space_size == 16'd0) ? 32'd100
             : 32'd100 - (({16'd0, true_remaining} * 32'd100) / {16'd0, space_size});
  assign bisect_pct = b_q[15:0];
  assign narrowed = (remaining <= 16'd1) && (space_size != 16'd0);
  // No space_size guard: an empty space shifts to zero and is floored to one,
  // so a search with nothing in it is already narrowed.
  assign truly_wide = (true_remaining > 16'd1);
  assign bisect_err = evaluate && truly_wide && narrowed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_wide <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_wide) n_wide <= n_wide + 8'd1;
    end
  end
endmodule

A space of a thousand and twenty-four with three rounds done leaves a hundred and twenty-eight candidates, seven rounds to finish, and two thousand two hundred and forty minutes to do them — eighty-eight percent of the space already gone, and the just-look-at-it view reports one candidate.

The sixth case is where sections 5 and 6 multiply, and it is the most important arithmetic in the chapter. An unreliable test that needs forty repeats per round drives the cost of finishing past the model's ceiling — the same seven rounds, the same eighty minutes each, and a factor of forty on top. Bisection's logarithm is in the number of rounds, not in the cost of a round, so an intermittent failure does not make bisection slower by a little; it makes it arithmetically unaffordable while leaving the round count unchanged and reassuring.

That gives the practical rule the two sections produce together: spend effort making the repro reliable before spending it narrowing the space. Going from one-in-forty to one-in-two divides every remaining round by twenty, and there are usually a handful of cheap ways to do it — tightening a timing margin, raising a frequency, removing a delay, running the suspicious traffic alone.

The last case is the boundary the model insists on. Two candidates left is not one. A search that has narrowed from a thousand to two has done ninety-nine point eight percent of the work and has not finished, and the difference between two candidates and one is the difference between a hypothesis and a root cause.

The fifth case is the starting state: nothing bisected yet, the whole space open, ten rounds to do — and the just-look-at-it view reports one candidate from the beginning, which is what an intuition about the cause looks like when it is stated as a conclusion.

The degenerate case bounds it: a search space with nothing in it floors at one candidate and is an absent search rather than a narrowed one.

What can be bisected is a longer list than most people reach for, and the value of writing it down is that several of the entries cost nothing. A commit range, which is the one everybody knows. A configuration bit vector — disable half the features, then half of those. A device or slot population — half the cards out. A traffic mix — half the transaction types suppressed. A time window, by starting the workload at different points. A frequency or voltage range. And a revision axis, between two silicon steppings or two firmware versions where one is known good.

Each of those is a different space and they can be searched independently, which matters because a failure that resists bisection on one axis often collapses immediately on another. A bug that survives forty commits of bisection and disappears when one lane is masked was never a software bug.

The last thing worth saying about the technique is its precondition, which is the one that fails silently. Bisection assumes the failure is caused by one thing. A failure that needs two independent conditions will give inconsistent answers as the search proceeds — one half fails, then neither half fails, then both do — and the symptom of that is a bisection that does not converge. A non-converging bisection is evidence, not a failure of method, and reading it that way is faster than repeating the rounds.

7. RTL 3 — What You Can See Was Decided Long Ago

The third thing, and the one nobody can change during the debug.

Observability at the moment of failure is a property of the hardware, the board and the silicon, all of which were finished before bring-up started. A signal that was not brought out to a test point, a state that no register exposes, an internal queue with no occupancy counter — none of those become visible because the failure is important.

There is a real technique that partially recovers from this, and it is worth counting separately: some of what cannot be seen can be inferred from what can. A queue's occupancy can be reconstructed from its input and output counts. A state machine's position can be narrowed from what it did next. That is inference, it is legitimate, and it runs out.

// RTL 3 - what you can see when it fails is decided long before it fails. A
// bring-up failure is debugged with whatever observability the board, the
// device and the host already have, and a signal that is not brought out is
// not brought out today either.
module observability_at_failure #(parameter int WE_CAN_SEE_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] signals_needed, signals_visible, inferable, layers_spanned,
  output logic [15:0] visible_ok, blind_signals, inferred_ok, visible_pct,
  output logic        can_see_it,
  output logic [7:0]  n_evals, n_blind,
  output logic        visibility_err
);
  logic [31:0] v_q;
  logic [15:0] true_blind, uncovered;
  logic        truly_blind;
  assign visible_ok = (signals_visible > signals_needed) ? signals_needed : signals_visible;
  assign uncovered = signals_needed - visible_ok;
  // Some of what cannot be seen can be worked out from what can, and that is
  // a real technique rather than a consolation - but only up to a point.
  assign inferred_ok = (inferable > uncovered) ? uncovered : inferable;
  assign true_blind = uncovered - inferred_ok;
  assign blind_signals = (WE_CAN_SEE_ENOUGH != 0) ? 16'd0 : true_blind;
  assign v_q = (signals_needed == 16'd0) ? 32'd100
             : ((({16'd0, visible_ok} + {16'd0, inferred_ok}) * 32'd100)
                / {16'd0, signals_needed});
  assign visible_pct = v_q[15:0];
  assign can_see_it = (blind_signals == 16'd0) && (signals_needed != 16'd0);
  assign truly_blind = (true_blind != 16'd0);
  assign visibility_err = evaluate && truly_blind && can_see_it;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_blind <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_blind) n_blind <= n_blind + 8'd1;
    end
  end
endmodule

Fourteen signals needed with five visible and three inferable is six that cannot be seen at all — fifty-seven percent of what the failure needs — and the we-can-see-enough view reports an observable failure.

The sixth case is the one that makes inference worth modelling rather than dismissing. Five visible and nine inferable closes the gap completely, and the model calls the failure observed — because a failure fully reconstructed from what can be seen has been seen. Inference is not a consolation prize; it is most of what an experienced debugger does with a limited instrument, and a model that refused to credit it would be describing a discipline nobody practises.

The fifth case is the floor. Nothing visible and nothing inferable is a failure that cannot be debugged from where you are standing, and the correct response is not more hypotheses but a different vantage point — a different board, an emulation run, an instrumented build. Recognising that state early is worth more than any amount of cleverness inside it.

The last case keeps inference honest: more inference claimed than there is gap to close earns no credit, because reconstructing a signal that was already visible has recovered nothing.

The degenerate case is the one that is common and quiet. A failure whose signal list nobody has written reports full observability of an empty set. Writing down what you would need to see, before checking what you can see, is the step that turns this section into a decision.

The order of those two steps is the whole of the discipline and it is easy to get backwards. Writing the list from the instruments produces a list of what is convenient to look at, which is then examined thoroughly, and the failure remains unexplained because the thing that would have explained it was never on the list. Writing the list from the failure produces a list that includes things nobody can see — and that is the useful output, because it names the gap precisely enough to argue about.

Naming the gap precisely is what buys the alternatives. An unobservable signal on this board may be observable on another — a validation board with more test points, an FPGA prototype, an emulation run where everything is visible and nothing runs at speed. Each of those is expensive and each becomes obviously worth it once the list says the current setup cannot answer the question. Without the list, the same decision is made on a feeling that the debug is going badly.

The layers the signals span is the second-order number the model carries and does not judge. A failure whose needed signals live in four different layers requires four different instruments and usually four different people, which is a coordination cost on top of the observability one — and it is a good early indicator that section 9's isolation should come first.

8. RTL 4 — The Order Hypotheses Are Tested In

The fourth thing, and the cheapest decision in the chapter to make correctly.

Hypotheses have a probability and a cost, and those are independent. The most likely explanation can be the most expensive to test; the cheapest test can eliminate half the candidates. Testing in order of likelihood alone is a common and expensive habit, and testing in order of cost alone is worse when the cheap tests settle nothing.

The quantity that matters is how much of the remaining space a test disposes of per unit of cost, and the model prices the two orders against each other.

// RTL 4 - hypotheses have a probability and a cost, and the order they are
// tested in is the whole of the method. Testing the most likely one first is
// wrong when it is also the most expensive, and testing the cheapest first is
// wrong when it explains nothing.
module hypothesis_order #(parameter int START_WITH_THE_LIKELIEST = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] hyps, cheap_cost, dear_cost, cheap_settles_pct,
  output logic [15:0] ordered_cost, naive_cost, saving, settled_by_cheap,
  output logic        nothing_to_gain,
  output logic [7:0]  n_evals, n_misordered,
  output logic        order_err
);
  logic [31:0] s_q, o_q, d_q;
  logic [15:0] settled_ok, unsettled, true_ordered, true_naive, true_saving;
  logic        truly_misordered;
  // How many of the open hypotheses the cheap check disposes of outright.
  assign s_q = ({16'd0, hyps} * {16'd0, cheap_settles_pct}) / 32'd100;
  assign settled_ok = (s_q > {16'd0, hyps}) ? hyps : s_q[15:0];
  assign settled_by_cheap = settled_ok;
  assign unsettled = hyps - settled_ok;
  // Cheap first: every hypothesis pays the cheap check, and only the ones it
  // does not settle go on to pay the expensive one.
  assign o_q = ({16'd0, hyps} * {16'd0, cheap_cost})
             + ({16'd0, unsettled} * {16'd0, dear_cost});
  assign true_ordered = (o_q > 32'd9999) ? 16'd9999 : o_q[15:0];
  // Dearest first: every hypothesis pays the expensive check.
  assign d_q = {16'd0, hyps} * {16'd0, dear_cost};
  assign true_naive = (d_q > 32'd9999) ? 16'd9999 : d_q[15:0];
  assign ordered_cost = (START_WITH_THE_LIKELIEST != 0) ? true_naive : true_ordered;
  assign naive_cost = true_naive;
  assign true_saving = (true_naive > true_ordered) ? (true_naive - true_ordered) : 16'd0;
  assign saving = (START_WITH_THE_LIKELIEST != 0) ? 16'd0 : true_saving;
  // The claim: there is no cheaper order than the one being used.
  assign nothing_to_gain = (saving == 16'd0) && (hyps != 16'd0);
  // No hyps guard: every term of the saving is scaled by the hypothesis
  // count, so a search with none already has nothing to gain.
  assign truly_misordered = (true_saving != 16'd0);
  assign order_err = evaluate && truly_misordered && nothing_to_gain;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_misordered <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_misordered) n_misordered <= n_misordered + 8'd1;
    end
  end
endmodule

Twelve hypotheses with a cheap check that settles two thirds is three hundred and sixty for the cheap-first order against seven hundred and twenty for the other — half the cost, from a decision that costs nothing to make.

The second case is the honest limit and it inverts the rule. A cheap check that settles nothing makes cheap-first the worse order — seven hundred and eighty against seven hundred and twenty — because every hypothesis pays for a question that eliminated none of them. The ordering value is in what a question resolves, not in what it costs, and a cheap question with no resolving power is a question worth skipping entirely.

The sixth case is the other limit. Two checks that cost the same make the order irrelevant, and both views agree. Ordering pays exactly in proportion to the spread between the costs, which is why the technique is worth most when one of the tests involves a lab, a rebuild or a vendor.

The third case is the best outcome available. A cheap check that settles everything — a single register read that distinguishes all twelve candidates — costs sixty units against seven hundred and twenty, and finding one of those is what a good first hour of a debug looks like.

The seventh case is where the ceiling interferes with the comparison and it is worth stating plainly. Two costs that both saturate cannot be told apart, and the model reports no saving between them. That is a limit of the model rather than a fact about debugging, and it is the honest thing for it to report.

The degenerate case bounds it: a debug with no hypotheses left is stuck rather than efficient, and the model declines to call it well ordered.

There is a second ordering principle that the model does not compute and that a good debugger uses constantly. Prefer a test that splits the hypotheses over one that confirms a single hypothesis. A check that tells you which half of the list is wrong is worth more than a check that confirms one entry, even when the entry is the likely one — because confirming the likely hypothesis when it is wrong tells you almost nothing, while a split is informative in both directions.

That is the same property as section 6's bisection, applied to a list of causes rather than to a search space, and it is why the two sections belong next to each other. A hypothesis list with no splitting test available is a list that will be walked linearly, and recognising that early is a reason to look for a different kind of evidence rather than to start walking.

The fifth case is worth reading for the limit it exposes rather than the number. A settle rate above a hundred percent is clamped to the hypotheses that exist, which is the model refusing to credit a check with eliminating candidates that were never on the list. That is not a hypothetical input: a test that "rules out the whole software stack" is often claiming to eliminate more than was ever being considered, and the useful question is which specific entries it removes.

9. RTL 5 — One Symptom Belongs To Several Layers

The fifth thing, and the one that decides whether a debug is a search or a guess.

A symptom is evidence about a set of layers, not a pointer to one. "The device did not enumerate" is consistent with a physical problem, a link-training problem, an alternate-protocol negotiation problem, a configuration-space problem and a driver problem — five layers, each with its own experts, instruments and checks.

The work is ruling layers out, and each elimination has a price: a set of checks in that layer, each costing time on an instrument or a rig.

// RTL 5 - one symptom belongs to several layers. "The device did not
// enumerate" is consistent with a physical problem, a link training problem,
// an alternate-protocol negotiation problem, a configuration problem and a
// software problem, and the debug is the process of ruling layers out rather
// than of guessing which one it is.
module symptom_to_layer #(parameter int THE_SYMPTOM_NAMES_IT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] layers_possible, ruled_out, checks_per_layer, check_cost,
  output logic [15:0] ruled_ok, layers_left, isolation_cost, ruled_pct,
  output logic        isolated,
  output logic [7:0]  n_evals, n_ambiguous,
  output logic        layer_err
);
  logic [31:0] c_q, r_q;
  logic [15:0] true_left, true_cost;
  logic        truly_ambiguous;
  assign ruled_ok = (ruled_out > layers_possible) ? layers_possible : ruled_out;
  assign true_left = layers_possible - ruled_ok;
  assign layers_left = (THE_SYMPTOM_NAMES_IT != 0) ? 16'd1 : true_left;
  // Ruling out the rest costs a set of checks in each remaining layer.
  assign c_q = ({16'd0, true_left} * {16'd0, checks_per_layer})
             * {16'd0, check_cost};
  assign true_cost = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign isolation_cost = (THE_SYMPTOM_NAMES_IT != 0) ? 16'd0 : true_cost;
  assign r_q = (layers_possible == 16'd0) ? 32'd100
             : (({16'd0, ruled_ok} * 32'd100) / {16'd0, layers_possible});
  assign ruled_pct = r_q[15:0];
  assign isolated = (layers_left <= 16'd1) && (layers_possible != 16'd0);
  assign truly_ambiguous = (true_left > 16'd1);
  assign layer_err = evaluate && truly_ambiguous && isolated;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_ambiguous <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_ambiguous) n_ambiguous <= n_ambiguous + 8'd1;
    end
  end
endmodule

Five layers consistent with the symptom and one ruled out leaves four still open and seven hundred and twenty minutes of checks to eliminate the rest — a fifth of the way, and the symptom-names-it view reports a single layer at no cost.

Figure 2 — the shape of an isolation, and the reason it is drawn as a chain rather than a selection. Every arrow is an elimination, not a choice: the debug does not pick the likely layer, it removes the ones that cannot be responsible until one is left. Each node is a question that can be answered with an instrument the team already has, and section 8's ordering decides which of them to ask first — because the five checks cost very different amounts and eliminate very different fractions.

The sixth case is why the weak view exists rather than being a straw man. Some symptoms really do span one layer — a CRC error counter incrementing names the link layer and nothing else — and there the symptom does name the cause. The symptom-names-it view was generalised from cases like that, which is why it survives: it is right whenever the evidence happens to be specific, and it is silent about the difference.

The third case is worth a moment. More layers ruled out than the symptom spans leaves nothing, which the model reports as complete elimination — a state that means the symptom has been explained away entirely and the reasoning needs re-examining rather than celebrating.

The last case is the shape of a hard bring-up. Nine layers open and the isolation cost saturated is a symptom that could be almost anything, and it is where section 8's ordering is worth the most, because the difference between a good and a bad order is multiplied by nine.

The CXL-specific part of this list is worth spelling out, because it is the layer that the PCIe version of this question does not have. Alternate-protocol negotiation sits between link training and enumeration, and its failures look like both of its neighbours: a device that negotiated PCIe instead of CXL trains perfectly, enumerates perfectly, and simply is not a CXL device — which reads as a configuration or software problem and is neither. 27.2 is where that negotiation is built; here it is a layer to eliminate, and the check that eliminates it is a register read that costs nothing.

That makes it the highest-value first elimination on a CXL bring-up, by section 8's criterion: it is nearly free and it removes a whole layer from a list of five. A debug that starts with a protocol analyser capture has spent an hour to learn less than that register read would have told it in a minute.

The fourth case bounds the model at the degenerate end and it is a real state. A symptom nobody has mapped to layers reports every layer eliminated because none was named — which is what "it just does not work" is, before anybody has written the list. The list is the artefact that converts a complaint into a search.

10. RTL 6 — A Buffer That Wraps Before The Trigger

The sixth thing, and the one that produces the most frustrating kind of failed debug: the one where the instrument worked perfectly and captured the wrong thing.

The interesting part of a failure is what led up to it. A trace buffer triggered on the failure has to hold enough history to contain the cause, and buffers are sized in entries rather than in causes. A thousand-entry buffer on a busy link is a fraction of a millisecond, and the transaction that poisoned the state may be several milliseconds back.

There is a real lever, and it costs resolution. Filtering multiplies the reach — recording one cycle in eight extends the window eightfold and loses the seven in between.

// RTL 6 - a trace buffer that wraps before the trigger has recorded the
// wrong thing. The interesting part of a failure is what happened leading up
// to it, and a buffer sized for the failure itself captures the consequence
// and none of the cause.
module trace_window #(parameter int WE_HAVE_A_TRACE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] buffer_depth, prior_cycles_wanted, post_cycles, rate_div,
  output logic [15:0] captured_prior, lost_cycles, post_ok, prior_pct,
  output logic        cause_captured,
  output logic [7:0]  n_evals, n_lost,
  output logic        trace_err
);
  logic [31:0] p_q, d_q;
  logic [15:0] effective_depth, true_prior, true_lost;
  logic        truly_lost;
  // Filtering multiplies the reach of a buffer by recording one cycle in N,
  // at the cost of not seeing the ones in between.
  assign d_q = {16'd0, buffer_depth} * {16'd0, (rate_div == 16'd0) ? 16'd1 : rate_div};
  assign effective_depth = (d_q > 32'd9999) ? 16'd9999 : d_q[15:0];
  // The post-trigger window is spent first; what is left reaches backwards.
  assign post_ok = (post_cycles > effective_depth) ? effective_depth : post_cycles;
  assign true_prior = ((effective_depth - post_ok) > prior_cycles_wanted)
                    ? prior_cycles_wanted : (effective_depth - post_ok);
  assign captured_prior = (WE_HAVE_A_TRACE != 0) ? prior_cycles_wanted : true_prior;
  assign true_lost = prior_cycles_wanted - true_prior;
  assign lost_cycles = (WE_HAVE_A_TRACE != 0) ? 16'd0 : true_lost;
  assign p_q = (prior_cycles_wanted == 16'd0) ? 32'd100
             : (({16'd0, true_prior} * 32'd100) / {16'd0, prior_cycles_wanted});
  assign prior_pct = p_q[15:0];
  assign cause_captured = (lost_cycles == 16'd0) && (prior_cycles_wanted != 16'd0);
  assign truly_lost = (true_lost != 16'd0);
  assign trace_err = evaluate && truly_lost && cause_captured;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_lost <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_lost) n_lost <= n_lost + 8'd1;
    end
  end
endmodule

A thousand-entry buffer with two hundred cycles kept after the trigger captures eight hundred cycles of run-up against the four thousand wanted — a fifth of the cause, and three thousand two hundred cycles lost.

The third case is the lever working. Filtering at one cycle in eight reaches the whole four-thousand-cycle window on the same thousand-entry buffer, and the model calls the cause captured — at a coarser resolution, which is a real cost and usually the right trade. Seeing every eighth cycle of the run-up beats seeing all of the wrong stretch.

The fourth case is the failure mode that looks like success. A post-trigger window larger than the buffer spends the entire capture after the failure — a trace consisting wholly of the consequence, with nothing before the trigger at all. Every debug tool defaults to some post-trigger capture, and on a short buffer that default is the whole thing.

The eighth case is the two effects interacting. A saturated reach spent after the trigger captures no cause either: filtering bought a large window and the post-trigger setting consumed all of it. The lever and the default work against each other, and the arithmetic is the only way to see it before running the capture.

The degenerate case bounds the model: a capture nobody has said what they want from reports the whole run-up captured because none was asked for, which is an unasked question rather than a successful trace.

The estimate the model needs and cannot supply is how far back the cause is, and there is a usable way to bound it. The cause is at most as far back as the oldest state the failure depends on — an outstanding transaction, a queue entry, a credit, a directory line, a negotiated parameter. Each of those has a lifetime, and the longest lifetime among the ones the failure could involve is the window the trace has to cover. That converts an unbounded question into an arithmetic one, and it usually says the window needs to be much longer than anybody has provisioned.

The second lever, after filtering, is the trigger. A buffer that cannot reach back far enough can still capture the cause if it triggers on something earlier than the failure — the transaction type that is suspected, a queue crossing a threshold, a state entry rather than the eventual error. That trades certainty for reach: most captures will contain no failure at all. On a failure that is expensive to reproduce, section 5's arithmetic says that trade is usually wrong; on a cheap one it is usually right.

The seventh case is the filter lever pushed to its limit. A filter rate that saturates the effective depth still covers the window, and the model reports the cause captured — at one cycle in sixty, which is enough to see a transaction arrive and not enough to see what it contained. Reach and resolution trade against each other continuously, and the right point on that curve depends on whether the question is when or what.

11. RTL 7 — A Workaround Hides More Than The Bug

The seventh thing, and the one that ends more debugs than any root cause does.

A workaround removes the conditions the failure needed. Disabling a feature, dropping a link speed, pinning a buffer, turning off an optimisation — each of them makes the symptom stop, and each of them takes a region of the design out of reach. Every other bug in that region is now equally invisible, whether or not anybody knew it was there.

The trade is sometimes right. What makes it dangerous is that it looks identical to a fix from outside: the failure is gone, the regression is green, and the ticket closes.

// RTL 7 - a workaround that makes the symptom go away has not made the bug
// go away. Disabling a feature, slowing a link or pinning a buffer removes
// the conditions the failure needed, and every other failure those conditions
// would have produced is now hidden too.
module workaround_cost #(parameter int IT_STOPPED_FAILING = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] bugs_in_region, root_found, perf_lost_pct, region_pct,
  output logic [15:0] found_ok, hidden_bugs, cost_pct, exposure_pct,
  output logic        resolved,
  output logic [7:0]  n_evals, n_hidden,
  output logic        workaround_err
);
  logic [31:0] e_q;
  logic [15:0] true_hidden;
  logic        truly_hidden;
  assign found_ok = (root_found > bugs_in_region) ? bugs_in_region : root_found;
  // Every bug the workaround's conditions would have exposed is now silent,
  // whether or not it was the one being chased.
  assign true_hidden = bugs_in_region - found_ok;
  assign hidden_bugs = (IT_STOPPED_FAILING != 0) ? 16'd0 : true_hidden;
  assign cost_pct = perf_lost_pct;
  // How much of the design's behaviour the workaround has taken out of reach.
  assign e_q = {16'd0, region_pct};
  assign exposure_pct = (e_q > 32'd100) ? 16'd100 : e_q[15:0];
  assign resolved = (hidden_bugs == 16'd0) && (bugs_in_region != 16'd0);
  assign truly_hidden = (true_hidden != 16'd0);
  assign workaround_err = evaluate && truly_hidden && resolved;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_hidden <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_hidden) n_hidden <= n_hidden + 8'd1;
    end
  end
endmodule

Nine known bugs in a region with one root cause found is eight bugs now unreachable, at eighteen percent of the performance and thirty percent of the design taken out of reach — and the it-stopped-failing view reports a resolved problem.

The fifth case is the shipping configuration and it deserves to be said plainly. A workaround applied with no root cause found at all hides nine bugs and resolves none, and it is an extremely common end state — not through negligence but because the schedule ended and the symptom was gone.

The last case is the one that makes the section a measurement rather than an objection. A workaround that costs nothing and masks a region whose bugs are all understood is a fix. If every bug in the disabled region has been found and the performance cost is zero, disabling it is simply the correct design change, and the model says so.

The fourth case is the honest limit of what this model can see. A region with no known bugs in it hides nothing measurable — and the whole risk of a workaround is the bugs nobody knows about, which by construction are not in the count. The model prices what is known and the real exposure is larger, which is an argument for the workaround being a decision with a review rather than a ticket closure.

The sixth case drives the clamp: an exposure figure past a hundred percent saturates at the whole design, which is what a workaround that disables a fundamental behaviour amounts to.

There is a version of this that is entirely legitimate and worth separating, because otherwise the section reads as an instruction never to ship a workaround. A workaround with a known cause is a design decision. If the mechanism is understood, the region's exposure is understood, and the performance cost is accepted deliberately, then turning a feature off is a scoping choice like any other — and it is frequently the right one for a first stepping.

What the model objects to is the workaround that substitutes for the cause, and the distinguishing question is short: do we know why this made it stop? A workaround whose mechanism is understood predicts what else it masks. One whose mechanism is not understood masks an unknown region and may not even be masking the actual failure — the symptom could have stopped for an unrelated reason, which is the worst outcome available because it looks identical to success.

The second and third cases together give the model's honest boundary. Every bug in the region found makes the workaround a fix, and more root causes claimed than bugs earns no extra credit. Between them they say that the model is counting understanding rather than activity, which is the same distinction sections 9 and 12 are making in their own terms.

12. RTL 8 — Why It Escaped Is A Separate Question

The eighth thing, and the only part of a debug that pays forward.

A bug in bring-up got past a verification environment. That happened for a reason: a coverage hole, a check that could not fail, an unexercised error path, a scenario nobody thought of — which is 27.9's entire chapter, viewed from the far side. Naming that reason is escape analysis, and it is the difference between fixing one bug and closing the route that a class of bugs takes.

And naming it is not doing it. Closing the hole is a separate step, and the one that gets written into a ticket and not scheduled.

// RTL 8 - a bug that reached bring-up escaped verification, and why it
// escaped is a separate question from what it was. A fix with no escape
// analysis behind it leaves the hole that let this one through open for the
// next one.
module escape_analysis #(parameter int A_FIX_IS_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] escapes, causes_named, suite_gaps_closed, similar_bugs,
  output logic [15:0] named_ok, unexplained, gaps_closed_ok, explained_pct,
  output logic        escape_understood,
  output logic [7:0]  n_evals, n_unexplained,
  output logic        escape_err
);
  logic [31:0] x_q;
  logic [15:0] true_unexplained, true_gaps_closed;
  logic        truly_unexplained;
  assign named_ok = (causes_named > escapes) ? escapes : causes_named;
  assign true_unexplained = escapes - named_ok;
  assign unexplained = (A_FIX_IS_ENOUGH != 0) ? 16'd0 : true_unexplained;
  // Naming why a bug escaped is not the same as closing the hole, and only a
  // closed hole stops the next one.
  assign true_gaps_closed = (suite_gaps_closed > named_ok) ? named_ok : suite_gaps_closed;
  // The a-fix-is-enough view assumes the hole is closed wherever a cause was
  // written down, which is the step that never happens on its own.
  assign gaps_closed_ok = (A_FIX_IS_ENOUGH != 0) ? named_ok : true_gaps_closed;
  assign x_q = (escapes == 16'd0) ? 32'd100
             : (({16'd0, named_ok} * 32'd100) / {16'd0, escapes});
  assign explained_pct = x_q[15:0];
  assign escape_understood = (unexplained == 16'd0) && (gaps_closed_ok == named_ok)
                           && (escapes != 16'd0);
  assign truly_unexplained = (true_unexplained != 16'd0) || (true_gaps_closed != named_ok);
  assign escape_err = evaluate && truly_unexplained && escape_understood;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_unexplained <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_unexplained) n_unexplained <= n_unexplained + 8'd1;
    end
  end
endmodule

Six escapes with two causes named and one hole closed is four escapes whose route nobody has traced — a third understood — and the a-fix-is-enough view assumes a hole closed wherever a cause was written down.

The fifth case is the one this section is really about. Every cause named and not one hole closed is an analysis that exists as a document, and the next bug will take the same route as the last one. The measured build declines to call it understood; the a-fix-is-enough view assumes the closing followed the naming, which is precisely the step that does not happen on its own.

The last case is what the arithmetic looks like on a real programme. Forty escapes with three analysed — seven percent — is a project that has fixed forty bugs and learned from three, and the forty-one-th will arrive by a route that was described nowhere.

The degenerate case bounds it, and the boundary is worth respecting. A project with no escapes yet reports every escape explained because none happened, and the model declines to call that understanding. It is an absence of evidence, and treating it as evidence of a good verification environment is how a programme gets surprised late.

The categories an escape falls into are short enough to be a checklist, and each points at a different fix in 27.9. The scenario was never produced — a coverage hole, and the fix is stimulus. It was produced and not checked — a scoreboard gap, and the fix is a check. It was checked by something that could not fail — a derived check, and the fix is an independent expectation. It was on an error path nobody exercised — the fix is injection. Or it needed an agent the environment does not have — the fix is a component.

Five categories, and the useful property is that they have different owners and different costs, so classifying an escape is most of deciding what to do about it. An escape that is "we never produced that combination" costs a constraint; one that is "our reference model agreed with the bug" costs a review of every check derived the same way, which is a much larger piece of work and one that finds more than the original bug.

The similar-bugs input the model carries is the reason any of this is worth doing. An escape route that is open has a population of bugs waiting to use it, and the count is not one. Closing a hole is priced against that population rather than against the single bug that revealed it, which is what makes escape analysis pay and why it is mis-priced when it is treated as paperwork attached to a fix.

13. RTL 9 — How Long Before You Know Anything

The ninth thing, and the one that silently determines how many of the others are affordable.

The loop time is how long it takes to learn one thing. A setup that takes four hours to produce its first signal and forty minutes per run afterwards allows one hypothesis every forty minutes; one that takes seven minutes allows one every two. Over the twelve hypotheses a real debug tests, and over the seven bisection rounds of section 6, that difference is the whole schedule.

It is also the most improvable number in the chapter. Setup is paid once, runs can go in parallel, and both are engineering decisions rather than properties of the bug.

// RTL 9 - the number that decides how a debug goes is how long it takes to
// learn anything. A setup that takes a day to produce its first signal
// allows one hypothesis per day; one that takes a minute allows hundreds, and
// the difference compounds over every round of section 6's bisection.
module time_to_first_signal #(parameter int WE_WILL_GET_THERE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] setup_min, run_min, hyps_to_test, parallel_rigs,
  output logic [15:0] first_signal_min, per_hyp_min, total_min, rigs_ok,
  output logic        fast_loop,
  output logic [7:0]  n_evals, n_slow,
  output logic        loop_err
);
  logic [31:0] t_q, p_q;
  logic [15:0] true_total, true_per;
  logic        truly_slow;
  assign rigs_ok = (parallel_rigs == 16'd0) ? 16'd1 : parallel_rigs;
  assign first_signal_min = setup_min + run_min;
  // Setup is paid once; each hypothesis after the first costs a run, and
  // several rigs test several hypotheses at a time.
  assign p_q = ({16'd0, run_min} + {16'd0, rigs_ok} - 32'd1) / {16'd0, rigs_ok};
  assign true_per = (p_q > 32'd9999) ? 16'd9999 : p_q[15:0];
  assign per_hyp_min = true_per;
  assign t_q = {16'd0, setup_min} + ({16'd0, hyps_to_test} * {16'd0, true_per});
  assign true_total = (t_q > 32'd9999) ? 16'd9999 : t_q[15:0];
  assign total_min = (WE_WILL_GET_THERE != 0) ? first_signal_min : true_total;
  assign fast_loop = (total_min <= first_signal_min) && (hyps_to_test != 16'd0);
  // No hyps_to_test guard: with nothing to test the total is the setup alone,
  // which cannot exceed the setup plus a run.
  assign truly_slow = (true_total > first_signal_min);
  assign loop_err = evaluate && truly_slow && fast_loop;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_slow <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_slow) n_slow <= n_slow + 8'd1;
    end
  end
endmodule

A four-hour setup with forty-minute runs and twelve hypotheses on one rig is twelve hours against four and a half to the first signal — and the we-will-get-there view charges the first signal for the whole debug.

The second case is the lever and it is worth quantifying rather than asserting. Four rigs in parallel takes the twelve hypotheses from twelve hours to six — a halving, not a quartering, because the setup is paid once and does not divide. Parallelism improves the marginal cost and not the fixed one, which is why a slow setup is worth attacking first even though parallel rigs are easier to ask for.

The fifth case is the target state and it makes a point about what "fast" means. A five-minute setup and two-minute runs gives twenty-nine minutes for all twelve hypotheses, and the model still reports the loop as slower than a single look — which is correct and is the point. No loop is ever faster than its first signal; the question is how much slower, and twenty-nine minutes against seven is a debug that can afford to be wrong repeatedly.

The last case is where the ceiling defeats the comparison, and the model says so rather than inventing a verdict. A fifteen-hour run saturates the total while the first signal does not, and two figures on opposite sides of a ceiling cannot be compared. That is a limit of the model, stated.

The degenerate case bounds it: a debug with nothing left to test costs the setup alone, which is a finished debug rather than a fast one.

14. RTL 10 — A Bring-Up Debug Assembled

Nine sections of inputs. This one puts them in one place and makes the confident answer visible as what it is: one bit of six.

// RTL 10 - a bring-up debug assembled. Nine sections of inputs, one summary.
// "We reproduced it" is bit 0: true, necessary, and one sixth of what makes a
// debug session finished.
module debug_signoff #(parameter int A_REPRO_IS_A_RESULT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic        reproduced, isolated, observed,
  input  logic        root_named, escape_known, fix_verified,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_evals, n_sound, n_claimed,
  output logic        signoff_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~reproduced;
  assign fail_mask[1] = ~isolated;
  assign fail_mask[2] = ~observed;
  assign fail_mask[3] = ~root_named;
  assign fail_mask[4] = ~escape_known;
  assign fail_mask[5] = ~fix_verified;
  assign conditions_met = {15'd0, reproduced} + {15'd0, isolated}
                        + {15'd0, observed} + {15'd0, root_named}
                        + {15'd0, escape_known} + {15'd0, fix_verified};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp: conditions_met sums six one-bit values, so the quotient cannot
  // exceed a hundred and a ceiling would be unreachable code.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  // The repro view reads bit 0 and stops.
  assign claimed = (A_REPRO_IS_A_RESULT != 0) ? reproduced : truly_sound;
  assign sound = claimed;
  assign signoff_err = evaluate && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When the failure has been reproduced and any one of the other five fails, the assembled model reports that the debug is not finished and the repro view reports a result.

Across the eight evaluations, the assembled model calls one debug sound and the repro view calls six of them a result.

The bit order is the order the work actually happens in, which is unusual among this module's masks and is the right ordering here because each condition genuinely depends on the one before it. You cannot isolate what you cannot reproduce; you cannot observe what you have not isolated to a layer with instruments; you cannot name a root cause from an unobserved failure; escape analysis needs a root cause to trace; and verifying the fix needs the repro from bit 0, which is why an intermittent failure makes the last step as expensive as the first.

"We reproduced it" is bit 0, and it is genuinely the hardest bit on some debugs. That is what makes it the right weak definition for this chapter: it is not a trivial claim, it is frequently the breakthrough, and the relief of achieving it is exactly what makes it feel like the end. It says nothing about which layer, nothing about whether the failure was observed when it happened, nothing about cause, nothing about why verification missed it, and nothing about whether the fix works.

The five other bits fail independently. A failure can be reproduced reliably and never isolated. It can be isolated to a layer nobody can see into. It can be observed in full and produce a workaround rather than a cause — section 11, and the most common stopping point. A root cause can be named with no escape analysis at all, which is the most common stopping point on a programme that is otherwise doing well. And a fix can be applied and never tested against the original failure, which on a one-in-forty repro is a decision about eighty hours rather than an oversight.

Figure 4 — the mask in the order the work happens, which is also the order in which each step becomes possible. The last box is the one that catches people out: verifying the fix needs the repro from the first box, so a one-in-forty failure charges eighty hours at the end as well as at the beginning. A debug that spent its budget reaching a cause has nothing left for the step that proves the cause was right.

15. Quantitative Reasoning

Eight hundred minutes per observation of a failure that happens one run in forty at twenty minutes a run — and four thousand eight hundred for the six observations a debug needs.

A hundred and twenty-eight candidates after three rounds of a thousand-and-twenty-four space, seven rounds left, and the cost of finishing multiplied by forty when the test is unreliable.

Six signals of fourteen that cannot be seen at all, after five visible and three inferred — fifty-seven percent of what the failure needs.

Three hundred and sixty against seven hundred and twenty, from testing the cheap question first when it settles two thirds — and seven hundred and eighty against seven hundred and twenty when it settles nothing.

Four layers still open and seven hundred and twenty minutes of checks to eliminate them, after one of five was ruled out.

Three thousand two hundred cycles of run-up lost, from a thousand-entry buffer with two hundred cycles kept after the trigger.

Eight bugs made unreachable by a workaround that found one root cause, at eighteen percent of the performance and thirty percent of the design.

Four escapes of six whose route nobody traced, and six causes named with zero holes closed in the case that ships.

Twelve hours against four and a half, from twelve hypotheses on a four-hour setup — halved by four rigs, not quartered, because the setup is paid once.

One debug of eight sound; the repro view counts six. The assembled model's summary, and the chapter's.

16. Assertions

The testbenches carry 593 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported nothing on either testbench, the third chapter running.

Both builds are asserted on every degenerate case. A setup never run, a failure never reproduced, a search space with nothing in it, a failure whose signal list nobody wrote, a debug with no hypotheses left, a symptom nobody has mapped to layers, a capture with no window requested, a buffer with no depth, a region with no known bugs, a project with no escapes, and a debug with nothing left to test.

Every clamp that an input can reach is driven past its limit exactly once. More failures than runs, an observation cost that saturates, a debug budget that saturates, more bisection rounds than the space has, a search cost that saturates, more probes than the failure needs, more inference than there is gap, a settle rate above a hundred percent, two costs that both saturate, more layers ruled out than the symptom spans, an isolation cost that saturates, a filter rate that saturates the reach, an exposure above a hundred percent, more causes named than escapes, and a per-hypothesis cost that saturates.

Every percentage whose numerator is clamped is asserted in the case that over-claims — the rule extracted in 27.9 section 17, applied here while writing rather than after a survivor asked.

Every error output is checked in both directions in every case. Section 5's second, third, fourth and last cases, section 6's second and third, section 7's second, third, sixth and last, section 8's second, sixth and seventh, section 9's second, third and sixth, section 10's second, third and last, section 11's second, third, fourth and last, section 12's second, third and fourth, and section 13's second, fourth, sixth and last exist to assert the quiet half. Each is a case where the repro view is right, and a model that alarmed on them would be unusable.

17. Mutation Testing

123 mutations, 123 killed. Sixty-three against the first testbench, sixty against the second. The first run killed a hundred and nineteen and left four, all of one class.

All four survivors were clamps that the stimulus could not reach, which is the same class 27.9 closed by switching from ceiling-raising to clamp-removal. Removing a clamp only shows up if an input can drive the value past it — and in four models nothing did. Two of them were reachable by widening an input; two were masked by a downstream minimum, so the clamp fired identically either way and only a case that changed what the minimum selected could distinguish them.

That second pair is worth naming as a rule, because it has now appeared in two consecutive chapters: a clamp whose output feeds a minimum is unobservable unless the clamp changes which side of the minimum wins. 27.7's serial and parallel ceilings were hidden behind a clamp on their sum; this chapter's effective trace depth was hidden behind a minimum against the post-trigger window. The fix is the same in both: drive the case where the clamped value crosses the other operand.

splitcheck.py earned its place mid-campaign. Adding a stimulus case to reach one of those clamps moved a counter to exactly four of eight, which makes its inversion an equivalent mutant — and the campaign duly reported that inversion as a survivor. The script named it from the source before any re-run, and a ninth case restored the imbalance. A fix for one gap created another, which is the ordinary way that a batch of checks earns its cost.

No dominated guards and no dead clamps. domcheck.py was clean from the first model onward, because two guards that it cannot see — both comparisons rather than minimums — were removed by hand while writing, on the strength of the same reasoning it automates.

18. Verification Strategy

Measure the reproduction rate before planning anything. Section 5. It is the exchange rate for every technique below.

Spend effort on the repro before spending it on the search. Sections 5 and 6 together. Going from one-in-forty to one-in-two divides every bisection round by twenty.

Write down what you would need to see, then check what you can see. Section 7. In that order, or the list is written from the instruments.

Ask what each hypothesis costs and what fraction it eliminates. Section 8. Order by the second divided by the first.

Enumerate the layers the symptom is consistent with, and eliminate rather than select. Section 9.

Compute the trace window against the distance back to the plausible cause. Section 10. And check the post-trigger setting, which defaults to a value nobody chose.

Before applying a workaround, count what it takes out of reach. Section 11.

Name why it escaped, then close the hole — and treat those as two tasks. Section 12.

Attack the setup time before asking for more rigs. Section 13. Parallelism divides the marginal cost, not the fixed one.

19. Synthesis and Implementation Reality

Bring-up observability is designed in or absent. Debug registers, trace buffers, test points and a way to read them at speed are silicon and board decisions, and section 7's number is fixed before the first power-on.

A CXL link has more layers to eliminate than a PCIe one, because alternate-protocol negotiation sits between training and enumeration and fails in ways that look like both of its neighbours.

Protocol analysers have deep buffers and shallow triggers, which is the opposite of what section 10 wants. Triggering on a condition several transactions before the failure is usually the difference between a useful capture and a faithful record of the consequence.

Most bring-up failures are found by bisection against a known-good configuration rather than by reasoning — a previous silicon revision, a different slot, a lower speed, a smaller topology. Section 6's arithmetic is the justification for building that known-good reference early.

And escape analysis has no natural owner. It sits between the team that found the bug and the team that should have caught it, which is why section 12's second count — holes closed rather than causes named — is the one that stalls.

20. Silicon Observability

Free, and from the logs. The reproduction rate. Section 5's whole argument needs two numbers a regression already records and nobody divides.

Free. The run length and the setup time. Section 13.

Cheap. The list of layers consistent with the symptom. Section 9's denominator is a whiteboard exercise.

Cheap, and fixed. Which signals are visible. Section 7's supply side is a datasheet and a board schematic.

Moderate. The trace buffer's depth in cycles at the link's actual rate, against the distance back to a plausible cause. Section 10 needs the second figure, which is an estimate.

Expensive. The cost and resolving power of each hypothesis test. Section 8's inputs usually come from having done the debug before.

Unobtainable during the debug. What else the workaround is hiding. Section 11 prices the known bugs, and the exposure that matters is the unknown ones.

Unobtainable without a verification review. Why it escaped. Section 12 cannot be answered from the failure; it is answered from 27.9's artefacts.

21. Debug Lab

A CXL device fails to enumerate on one board in a rack of forty, intermittently.

Step 1 — establish the rate. Section 5. One board in forty, or one boot in forty on that board, are different failures with different searches.

Step 2 — make it fail more often before narrowing anything. Sections 5 and 6. Lower the link speed, raise the temperature, shorten the timeout — whatever moves the rate, applied before any bisection round is paid for.

Step 3 — bisect against a known-good configuration. Section 6. Swap the board, the slot, the cable, the device, the host, the firmware revision. Each swap halves a space.

Step 4 — enumerate the layers and pick the cheap eliminations first. Sections 8 and 9. Link status registers are free; a protocol analyser capture is not.

Step 5 — check the trace window before trusting a capture. Section 10. A capture that starts at the failure will show a failure.

Step 6 — if a workaround appears, count what it masks before accepting it. Section 11.

Step 7 — when the cause is named, ask why verification missed it. Section 12, and it is the step with no natural owner.

Steps 1 and 2 are where the schedule is won or lost, and they are the two most often skipped in favour of starting the search.

22. Design Review

What is the reproduction rate, and what does one observation cost?

What has been done to make it fail more often?

What is the search space, and can it be halved?

Which signals would you need to see at the moment of failure, and which of those can you see?

Which layers is this symptom consistent with, and which have been eliminated?

How far back does the trace buffer reach, and how far back is the plausible cause?

If this is a workaround, what else does it take out of reach?

What is the root cause, and how did it get past verification?

Has the fix been tested against the original failure, at the original rate?

23. How This Appears In Real Engineering

The debug ends before it is finished, and it ends at one of four places.

The most common is section 11. A workaround made the symptom stop, the schedule needed the board working, and the ticket closed with a configuration change and no cause. Nobody decided to stop debugging — the reason to continue simply evaporated when the failure did, and the eight other bugs in the disabled region are still there.

The second is section 12, and it happens on well-run programmes. The root cause was found, named, fixed and verified, and the question of why the verification environment missed it was written in the ticket and never scheduled. Six months later a bug of the same shape arrives by the same route.

The third is section 10. The capture was taken, it was clean, and it showed a device responding correctly right up to the moment it did not. The cause was three milliseconds earlier and the buffer held one. The debug loops — capture, inspect, find nothing, capture again — because each attempt is individually reasonable and none of them can work.

The fourth is sections 5 and 13 together, and it is the quiet one. The failure is one in forty and the loop is four hours, so each hypothesis costs a day. The debug is not stuck on anything intellectual; it is running at one idea per day against a problem that needs twelve, and nobody has spent an afternoon on the rate or the setup because both feel like preparation rather than progress.

The pattern is that the two cheapest things to improve — the rate and the loop time — are the two that feel least like debugging, and every hour spent on them is multiplied across everything that follows.

24. Common Misconceptions

"We reproduced it." At what rate, and what does the next observation cost? Section 5.

"It is intermittent, so we cannot bisect." You can; each round costs the repeat count. Fix the rate first. Section 6.

"We took a trace." Starting where? Section 10.

"We can see everything we need." Which signals did you write down before you looked? Section 7.

"It is obviously the link layer." It is consistent with the link layer and four other things. Section 9.

"We tried the most likely cause first." And it was the most expensive to test. Section 8.

"It stopped failing." What else stopped happening? Section 11.

"We fixed it." Do you know why the suite missed it? Section 12.

"We will get there." At one hypothesis per day, or per two minutes? Section 13.

25. Interview Reasoning

"Walk me through a bring-up debug." First establish the reproduction rate, because it prices everything else — a failure at one in forty on a twenty-minute run costs eight hundred minutes a look. Then spend effort making it fail more often, before narrowing anything, because every subsequent round is multiplied by that rate. Then bisect against a known-good configuration. In parallel, enumerate the layers the symptom is consistent with and eliminate the cheap ones first. Check the trace window reaches back to a plausible cause before trusting a capture. When a cause is named, verify the fix at the original rate — and ask why verification missed it.

"It is intermittent. Can you still bisect?" Yes, and each round costs the repeat count needed for a reliable answer. That is why making it fail more often comes first: going from one-in-forty to one-in-two divides every remaining round by twenty, and the round count does not change.

"Your capture shows the device behaving correctly and then failing." Then the capture starts after the cause. The question is how far back the buffer reaches at the link's rate against how far back the state was poisoned, and the answer is usually to filter — one cycle in eight buys eight times the window at a coarser resolution.

"Which hypothesis do you test first?" The one with the highest ratio of candidates eliminated to cost. Not the most likely one, unless it is also cheap — testing in order of likelihood is the common habit and it is expensive when the likely explanation needs a lab.

"The symptom is that it does not enumerate. What is the cause?" That symptom is consistent with at least five layers — physical, training, alternate-protocol negotiation, configuration and driver — so the answer is a list and a plan to eliminate, not a guess. The first eliminations should be the register reads that are free.

"You have a workaround that makes it go away. Is that a fix?" No, and the cost is worth stating: it masks a region of the design, so every other bug in that region is now invisible too. If the region's bugs are all known and the performance cost is nil, the workaround is a design change and I would call it a fix. Otherwise it is a decision that needs a review.

"The bug is fixed. Are you done?" Not until the fix has been verified against the original failure at the original rate — which on a one-in-forty repro is a real cost — and not until we know why the verification environment missed it, because otherwise the next one arrives the same way.

26. Exercises

1. A failure occurs 3 times in 120 runs of 35 minutes. Compute the rate and the cost per observation. What does the cost become at 1 in 5?

2. A search space of 4,096 with 5 rounds done. How many candidates remain and how many rounds are left? At 45 minutes a round and 6 repeats, what does finishing cost?

3. A failure needs 20 signals; 7 are visible and 6 of the remainder are inferable. Compute the blind count and the observable fraction. What is the fraction if 3 more test points had been laid out?

4. 18 hypotheses, a cheap check costing 8 that settles 40%, an expensive one costing 150. Compute both orders. At what settle rate do they cost the same?

5. A symptom spans 7 layers, 3 ruled out, 5 checks per layer at 25 minutes. Compute the remaining cost. Which two layers would you eliminate first and why?

6. A 4,000-entry buffer, 1,500 cycles kept post-trigger, 12,000 cycles of run-up wanted. Compute what is captured and what is lost. What filter rate closes the gap, and what does it cost?

7. A workaround masks a region containing 14 known bugs; 3 root causes are found; 25% of performance is lost. Compute what is hidden. What would make this a fix rather than a workaround?

8. 22 escapes, 9 causes named, 4 holes closed. Compute the unexplained count and the explained fraction. Which of the two counts would you put on a programme review, and which one matters?

9. A 6-hour setup, 25-minute runs, 20 hypotheses. Compute the total on 1 rig and on 5. Then compute it with a 20-minute setup on 1 rig, and say which change you would make.

10. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify its position using the rule that the ordering is the order the work happens in.

27. Summary

A repro's value is its rate. One failure in forty at twenty minutes a run is eight hundred minutes for every look, and that is the exchange rate for every technique below it.

Bisection is the only logarithmic technique, and an unreliable test multiplies every round without changing the round count.

What you can see at the moment of failure was decided when the board was laid out, and inference closes part of the gap and then runs out.

Hypotheses have a probability and a cost, and those are independent — order by resolution per unit cost, and skip a cheap question that settles nothing.

One symptom belongs to several layers, and isolation is elimination rather than selection.

A buffer that wraps before the trigger recorded the consequence, and the post-trigger default is a setting nobody chose.

A workaround hides more than the bug — every other failure in the region it disabled is now invisible too.

Why it escaped is a separate question from what it was, and naming the cause is a separate step from closing the hole.

The loop time decides how many hypotheses are affordable, and the setup is paid once, so parallel rigs halve rather than quarter.

Six bits, and "we reproduced it" is one of them. One debug of eight is sound; the repro view counts six.