Debug Review Checklist

Every review in this module has examined something that exists: an architecture, a block, an environment, a protocol, a measurement, an interface. This chapter examines a process — the one an engineer runs when a link is failing and nobody yet knows why.

The review question this chapter turns on, asked of every piece of evidence:

What does this rule out, and what is still alive?

Debug is elimination, not inspiration. A piece of evidence that rules nothing out has cost time and bought nothing, and a team that cannot say how many hypotheses remain is not debugging — it is guessing with instruments.

1. The Two Failures Of A Debug Session

A debug session can fail in two ways, and they look nothing alike.

The second is the dangerous one, and it is what most of this chapter is about. A masking fix, a correlation mistaken for a cause, a fault that vanished when the analyser went on, a reset that destroyed the evidence — every one of them ends a debug session early and produces a confident wrong answer.

2. How To Use This Chapter

Each of the eleven review dimensions below is a working review item, and every one answers the same eight questions:

3. The One-Sentence Model

A debug conclusion is sound when the failure was reproduced, when the hypothesis set was narrowed by named evidence, when the first fault was kept rather than the last, when the order of the faults is known, when the layer is named by what excluded the others, and when a snapshot was taken before recovery — and "we reproduced it" is bit 0.

4. What This Chapter Owns

The boundary with 30.5 is worth stating. That chapter reviews an instrument that reports a number nobody can contradict. This one reviews a conclusion nobody can contradict — and the two failures rhyme, because both come from evidence that was never asked to eliminate anything.

5. Teaching-Model Boundary And Source Discipline

Every model in this chapter is a teaching model. Each isolates one triage property so it can be examined, mutated and broken on purpose. None is a production CXL controller, a link-training engine, an error-reporting block or an implementation of any specification flow.

Nothing here states a normative CXL detail. No error code, status-register layout, bit position, field width, opcode, timeout constant, retry rule, training state or revision number from the specification appears anywhere in this chapter. The layer signatures used in the triage model — a physical problem tracking with lane and temperature, a link problem recovering on retrain, a protocol problem being deterministic on a transaction, a software problem surviving a reset — are general interconnect reasoning, and they are given in their general form deliberately, so the technique transfers.

One heuristic needs stating up front. The intermittent-rate model uses "roughly three times the mean interval between failures" as a confidence threshold. That is an illustrative engineering heuristic, not a statistical derivation, and it is declared as one in the model header. The review item it supports — state the pre-fix rate and compare the clean runs you have against the clean runs luck alone would produce — does not depend on the constant.

6. Review Item 1 — How Many Hypotheses Are Still Alive?

Under review. Every observation made during triage.

Claim at risk. That progress is being made.

Where it lives. The process, not the design.

The failure. "It is probably the PHY" is a hypothesis with no elimination behind it. The number that matters during triage is how many hypotheses remain, and a team that cannot state it has no way to tell measurement from motion.

// RTL 1 - a symptom is not a diagnosis, and the evidence is what eliminates.
//
// Debug is elimination, not inspiration. Every piece of evidence either rules a
// hypothesis out or it does not, and a piece that rules nothing out has cost
// time and bought nothing. The number that matters during triage is how many
// hypotheses remain, and a team that cannot say that number is guessing.
//
//   BAD  : "it is probably the PHY" - a hypothesis with no elimination behind it
//   GOOD : five hypotheses, evidence ruling out three, two remaining, named
//
// TEACHING MODEL. Isolates one triage property. It is not a CXL link-training
// engine and contains no opcode, layout, encoding, register definition or
// timing from any specification.
module elimination_count #(parameter int GUESS_WITHOUT_RULING_OUT = 0) (
  input  logic clk, rst_n,
  input  logic       observe, commit_to_one,
  input  logic [7:0] hypotheses, ruled_out,
  output logic [7:0] remaining, n_observations, n_guesses,
  output logic [15:0] narrowed_pct,
  output logic       narrowed, safe_to_commit,
  output logic       guess_err
);
  logic [31:0] n_q;
  logic [7:0]  truly_out;

  // Evidence cannot rule out more hypotheses than there were.
  assign truly_out = (ruled_out > hypotheses) ? hypotheses : ruled_out;
  assign remaining = hypotheses - truly_out;
  assign n_q = (hypotheses == 8'd0) ? 32'd0
             : (({24'd0, truly_out} * 32'd100) / {24'd0, hypotheses});
  assign narrowed_pct = n_q[15:0];
  assign narrowed = (remaining < hypotheses) && (hypotheses != 8'd0);
  // The whole review point: committing to one cause is safe only when the
  // evidence has actually left one standing.
  assign safe_to_commit = (GUESS_WITHOUT_RULING_OUT != 0) ? 1'b1
                                                          : (remaining <= 8'd1);
  // SAFETY-OF-EVIDENCE VIOLATION: a cause was committed to while several
  // hypotheses were still alive.
  assign guess_err = commit_to_one && safe_to_commit && (remaining > 8'd1);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_observations <= 8'd0; n_guesses <= 8'd0;
    end else begin
      if (observe)       n_observations <= n_observations + 8'd1;
      if (guess_err)     n_guesses      <= n_guesses + 8'd1;
    end
  end
endmodule

The measurement. Five hypotheses, evidence ruling out two:

5 hypotheses, 2 ruled out : remaining=3 narrowed=40% counted_safe=0 uncounted_safe=1

Three are still alive and the guessing build is ready to commit to one. The counting build refuses, because committing to one of three remaining hypotheses is a coin flip with two extra sides.

The run drives the case that matters most in real triage: an observation that rules nothing out. The remaining count does not move, and the honest build records an observation that bought nothing. That case was missing from the first campaign and a mutation found it, which is recorded in section 21.

Evidence to demand. The hypothesis list as it stood at the start, and for each observation, which entries it removed. An observation that removed none is not a finding.

What escapes. Days of instrumented measurement that never narrowed anything, followed by a commitment made on schedule pressure rather than on evidence.

Telemetry. None — this item is process. Its artefact is a written list that shrinks, and the absence of one is the finding.

Misleading evidence. Volume. A long log of careful measurements is not narrowing, and it reads exactly like narrowing to anybody not counting.

Figure 1 — the only measure of progress in triage is the size of the surviving set. The top path is the session that feels productive and is not; the bottom-right node is how it ends when the schedule runs out.

7. Review Item 2 — Which Fault Is In The Status Register?

Under review. Every error-status register.

Claim at risk. That the recorded fault is the one that started it.

Where it lives. One line of RTL.

The failure. A failing link produces a cascade: one root fault, then a hundred consequences. A status register that records the most recent fault records the hundredth, and the root is gone before anybody reads it.

// RTL 2 - the first fault and the last fault are different registers.
//
// A failing link produces a cascade: one root fault, then a hundred
// consequences. A status register that records the MOST RECENT fault records
// the hundredth, and the root is gone. A first-fault register is sticky: it
// captures once and refuses to be overwritten until it is explicitly cleared.
//
//   BAD  : status <= fault_code            // last one wins
//   GOOD : if (!captured) first <= code    // first one is kept
//
// TEACHING MODEL. Sequential.
//   State remembered : the captured code and whether capture has happened.
//   Safety           : the recorded code is the FIRST fault since the clear.
module first_fault #(parameter int RECORD_LAST = 0) (
  input  logic clk, rst_n,
  input  logic       fault, clear_status, report_now,
  input  logic [7:0] fault_code, root_code,
  output logic [7:0] recorded, n_faults,
  output logic       captured, is_root,
  output logic       fault_err
);
  logic [7:0] rec_q;
  logic       cap_q;

  assign recorded = rec_q;
  assign captured = cap_q;
  // The truth, computed the same way in BOTH builds: whether the code the
  // status register holds is the root cause.
  assign is_root = (rec_q == root_code);
  // SAFETY-OF-EVIDENCE VIOLATION: a fault was captured and the register holds
  // a consequence rather than the root.
  assign fault_err = report_now && cap_q && !is_root;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      rec_q <= 8'd0; cap_q <= 1'b0; n_faults <= 8'd0;
    end else begin
      if (clear_status) begin
        rec_q <= 8'd0; cap_q <= 1'b0;
      end else if (fault) begin
        n_faults <= n_faults + 8'd1;
        cap_q <= 1'b1;
        // The whole review point.
        if (RECORD_LAST != 0)      rec_q <= fault_code;
        else if (!cap_q)           rec_q <= fault_code;
      end
    end
  end
endmodule

The measurement. Root fault 11, then 22, 33, 44:

root 11 then 22,33,44 : sticky=11 last=44

Both builds cost one register. if (!captured) first <= code against status <= code — the same flops, the same width, and one of the two answers the question.

Evidence to demand. For every status register, whether it is sticky, and what clears it. A sticky register with no clear is a register that answers once per power-up.

What escapes. A team debugging fault 44, which is a consequence, for as long as it takes somebody to ask what happened first.

How DV proves it. Drive a cascade and assert the recorded code is the first. A test driving one fault cannot tell the two builds apart.

Telemetry. A first-fault register with an explicit clear, beside the last-fault register. Both are useful; only one of them is a diagnosis.

8. Review Item 3 — Which Of The Two Happened First?

Under review. Every status word with more than one error bit.

Claim at risk. The direction of causation.

The failure. Two sticky bits tell you both faults happened. They do not tell you the order, and the order is the whole diagnosis: a CRC error followed by a retrain is a link problem, and a retrain followed by a CRC error is a consequence of the retrain. The same two bits, and two opposite conclusions.

// RTL 3 - two faults, and no way to say which came first.
//
// Two error bits in one status word tell you both happened. They do not tell
// you the order, and the order is the whole diagnosis: a CRC error followed by
// a retrain is a link problem, and a retrain followed by a CRC error is a
// consequence of the retrain. A shared timestamp or a sequence number turns two
// bits into an ordering.
//
//   BAD  : two sticky bits in a status word
//   GOOD : two bits plus the sequence number each was captured at
//
// TEACHING MODEL. Sequential.
module fault_ordering #(parameter int BITS_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic fault_a, fault_b, report_now,
  output logic [7:0] seq_now, stamp_a, stamp_b,
  output logic       seen_a, seen_b, both_seen,
  output logic       a_first, order_known,
  output logic       ord_err
);
  logic [7:0] seq_q, sa_q, sb_q;
  logic       a_q, b_q;

  assign seq_now = seq_q;
  assign stamp_a = sa_q;
  assign stamp_b = sb_q;
  assign seen_a  = a_q;
  assign seen_b  = b_q;
  assign both_seen = a_q && b_q;
  // The truth, computed the same way in BOTH builds.
  assign a_first = both_seen && (sa_q < sb_q);
  // The whole review point: whether the status can answer the ordering question.
  assign order_known = (BITS_ONLY != 0) ? 1'b0 : both_seen;
  // SAFETY-OF-EVIDENCE VIOLATION: two faults are recorded and the record
  // cannot say which came first, so the cascade cannot be unwound.
  assign ord_err = report_now && both_seen && !order_known;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      seq_q <= 8'd0; sa_q <= 8'd0; sb_q <= 8'd0; a_q <= 1'b0; b_q <= 1'b0;
    end else begin
      if (seq_q != 8'hFF) seq_q <= seq_q + 8'd1;
      if (fault_a && !a_q) begin a_q <= 1'b1; sa_q <= seq_q; end
      if (fault_b && !b_q) begin b_q <= 1'b1; sb_q <= seq_q; end
    end
  end
endmodule

The measurement. Fault A at sequence 0, fault B at sequence 2:

A at seq 0, B at seq 2 : stamped_knows=1 bits_only_knows=0

The bits-only build has both faults and no ordering. The stamped build has the same two bits plus the sequence number each was captured at, and the ordering falls out of a comparison.

The run drives the case where both faults arrive in the same cycle. The stamps are equal, and the ordering is correctly reported as unknown rather than as a tie broken by bit position. An instrument that always produces an answer cannot express "I do not know", and that is the difference between evidence and a guess with a register behind it.

Evidence to demand. For any multi-bit status word, the shared counter or timestamp the bits are captured against. Two bits without one is two facts and no relationship.

What escapes. A causation arrow drawn the wrong way, and a fix applied to a consequence.

Telemetry. One free-running sequence counter, and a capture register per error bit. A counter shared across every error source in the block costs almost nothing and orders all of them.

9. Review Item 4 — Did The Trace Keep The Cause?

Under review. Every circular trace buffer.

Claim at risk. That the captured evidence contains the failure.

The failure. A circular trace keeps the most recent N entries. A cascade produces far more than N consequences, so by the time a human notices and stops the capture, the root entry has been overwritten by the noise it caused. The buffer is full of perfectly accurate evidence about the wrong part of the failure.

// RTL 4 - the trace buffer that recorded the consequences and lost the cause.
//
// A circular trace keeps the most recent N entries. A cascade produces far more
// than N consequences, so by the time a human stops the capture the root entry
// has been overwritten by the noise it caused. The buffer is full of perfectly
// accurate evidence about the wrong part of the failure.
//
//   BAD  : free-running circular trace, stopped by a human
//   GOOD : stop-on-first-fault, or a reserved slot the root cannot be evicted from
//
// TEACHING MODEL. Sequential.
//   Safety : the root entry is still present when the trace is read.
module trace_retention #(parameter int FREE_RUNNING = 0) (
  input  logic clk, rst_n,
  input  logic       ev, is_root, read_trace,
  input  logic [7:0] depth,
  output logic [7:0] entries_written, since_root, n_overwrites,
  output logic       root_seen, root_in_trace, stopped,
  output logic       log_err
);
  logic [7:0] ew_q, sr_q, ov_q;
  logic       rs_q, stop_q;

  assign entries_written = ew_q;
  assign since_root      = sr_q;
  assign n_overwrites    = ov_q;
  assign root_seen       = rs_q;
  assign stopped         = stop_q;
  // The root survives while fewer than `depth` entries have landed on top of it.
  assign root_in_trace = rs_q && (sr_q < depth);
  // SAFETY-OF-EVIDENCE VIOLATION: the trace was read, a root fault had
  // occurred, and the entry recording it is gone.
  assign log_err = read_trace && rs_q && !root_in_trace;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      ew_q <= 8'd0; sr_q <= 8'd0; ov_q <= 8'd0; rs_q <= 1'b0; stop_q <= 1'b0;
    end else begin
      // The whole review point: a stop-on-fault trace freezes at the root; a
      // free-running one keeps recording the cascade over it.
      if (is_root && !rs_q) begin
        rs_q <= 1'b1;
        if (FREE_RUNNING == 0) stop_q <= 1'b1;
      end
      if (ev && !stop_q) begin
        if (ew_q != 8'hFF) ew_q <= ew_q + 8'd1;
        if (rs_q && (sr_q != 8'hFF)) sr_q <= sr_q + 8'd1;
        if (rs_q && (sr_q >= depth) && (ov_q != 8'hFF)) ov_q <= ov_q + 8'd1;
      end
    end
  end
endmodule

The measurement. A four-deep trace, six entries after the root:

depth 4, 6 entries after the root : frozen_since=0 free_since=6

The free-running build has six entries since the root and a four-entry buffer. The root is two evictions gone. The stop-on-first-fault build froze at the root and its since_root count is zero, because nothing was written after it.

Evidence to demand. The trace depth, the expected cascade length, and which of the two is larger. If the second is larger, a free-running trace cannot contain the root.

What escapes. A capture that looks complete, is entirely accurate, and does not contain the failure.

How DV proves it. Drive a cascade longer than the buffer and assert the root is still readable. A cascade shorter than the depth cannot distinguish the two builds.

Telemetry. An overwrite counter, and a stop-on-first-fault mode. The overwrite counter is what tells a reader the capture is incomplete, which is the difference between a trace and a trap.

10. Review Item 5 — How Many Clean Runs Would Luck Have Produced?

Under review. Every claim that an intermittent fault is fixed.

Claim at risk. That the fix worked.

The failure. "It has not failed since the fix" is the commonest debug error there is. The question it skips is how many clean runs the fix would have produced by luck alone — and for a fault that appears once in ten runs, five clean runs is an unremarkable afternoon.

// RTL 5 - one failure in a thousand runs is not a rate.
//
// An intermittent fault produces a failure count and an observation count, and
// the ratio is only a rate once there are enough observations to support it.
// Declaring a fix effective after the failure stops appearing is the commonest
// debug error there is, and the question it skips is: how many clean runs would
// this fix have produced by luck alone?
//
//   BAD  : "it hasn't failed since the fix"
//   GOOD : state the pre-fix rate, derive the runs needed for confidence,
//          and compare that against the runs actually done
//
// TEACHING MODEL. Rates are scaled by 10,000 to stay in integer arithmetic.
// The confidence rule used here - roughly three times the mean interval - is an
// illustrative engineering heuristic, not a statistical derivation.
module intermittent_rate #(parameter int DECLARE_ON_SILENCE = 0) (
  input  logic clk, rst_n,
  input  logic        run, failure, declare_fixed,
  input  logic [15:0] runs_since_fix,
  output logic [15:0] observations, failures, rate_x10k, runs_needed,
  output logic        credible, evidence_enough,
  output logic [7:0]  n_declarations, n_premature,
  output logic        rate_err
);
  logic [15:0] obs_q, fail_q;
  logic [31:0] r_q, need_q;

  assign observations = obs_q;
  assign failures     = fail_q;
  assign r_q = (obs_q == 16'd0) ? 32'd0
             : (({16'd0, fail_q} * 32'd10000) / {16'd0, obs_q});
  assign rate_x10k = r_q[15:0];
  // The mean interval between failures is the reciprocal of the rate; three of
  // those is the illustrative bar this model uses for confidence.
  assign need_q = (fail_q == 16'd0) ? 32'd0
                : ((({16'd0, obs_q} * 32'd3) / {16'd0, fail_q}));
  assign runs_needed = (need_q > 32'd65535) ? 16'd65535 : need_q[15:0];
  // The truth, computed the same way in BOTH builds.
  assign evidence_enough = (runs_since_fix >= runs_needed) && (runs_needed != 16'd0);
  // The whole review point.
  assign credible = (DECLARE_ON_SILENCE != 0) ? 1'b1 : evidence_enough;
  // SAFETY-OF-EVIDENCE VIOLATION: a fix was declared effective on fewer clean
  // runs than the pre-fix rate would have produced by chance.
  assign rate_err = declare_fixed && credible && !evidence_enough;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      obs_q <= 16'd0; fail_q <= 16'd0;
      n_declarations <= 8'd0; n_premature <= 8'd0;
    end else begin
      if (run)     obs_q  <= obs_q + 16'd1;
      if (failure) fail_q <= fail_q + 16'd1;
      if (declare_fixed) begin
        n_declarations <= n_declarations + 8'd1;
        if (rate_err) n_premature <= n_premature + 8'd1;
      end
    end
  end
endmodule

The measurement. A pre-fix rate of one in ten, five clean runs since:

rate 1 in 10, needs 30 clean runs, has 5 : counted_credible=0 silence_credible=1

The silence-is-proof build declares the fix effective. The counting build needs 30 clean runs — roughly three times the mean interval, an illustrative heuristic stated as one — and it has five.

The run drives the case where no failure was ever observed. The pre-fix rate is zero, runs_needed is zero, and a "fix" can be declared for a defect that was never measured. The guard preventing that is the one domcheck flagged as possibly dead, and section 20 records how it was settled.

Evidence to demand. The pre-fix rate as a number, the clean-run count, and the comparison between them. "It stopped happening" is not any of the three.

What escapes. An unfixed intermittent defect, shipped, with the team's confidence in it raised rather than lowered by the silence.

Telemetry. An observation counter and a failure counter, both readable, both cleared explicitly and never implicitly. A rate is two numbers; publishing only the second is what makes silence look like evidence.

11. Review Item 6 — Did The Probe Change One Thing?

Under review. Every experiment run during triage.

Claim at risk. That the result is interpretable.

The failure. Bisection halves the search space per probe, and it only works if each probe changes exactly one variable. Change two and the result is uninterpretable whichever way it comes out: the space does not halve, and the probe has to be repeated.

// RTL 6 - change one thing, or change two and learn nothing.
//
// Bisection halves the search space per probe, and it only works if each probe
// changes exactly one variable. Change two and the result is uninterpretable
// whichever way it comes out: the space does not halve, and the probe has to be
// repeated. The arithmetic is the argument - a space of 64 needs six probes,
// and a team changing two things at a time is not bisecting at all.
//
//   BAD  : "we changed the clock and the terminator and it got better"
//   GOOD : one variable per probe, and the remaining span published
//
// TEACHING MODEL.
module bisect_discipline #(parameter int CHANGE_TWO = 0) (
  input  logic clk, rst_n,
  input  logic       probe, report_now,
  input  logic [7:0] span_lo, span_hi,
  output logic [7:0] span_size, midpoint, remaining, n_probes, n_useless,
  output logic [7:0] probes_needed,
  output logic       halved, interpretable,
  output logic       bisect_err
);
  logic [7:0] rem_q, np_q, nu_q;
  logic [7:0] size;

  assign size      = (span_hi >= span_lo) ? (span_hi - span_lo + 8'd1) : 8'd0;
  assign span_size = size;
  assign midpoint  = span_lo + (size >> 1);
  assign remaining = rem_q;
  assign n_probes  = np_q;
  assign n_useless = nu_q;
  // Illustrative: a span of 64 needs six probes, of 32 five, and so on. This
  // model publishes the count rather than computing a logarithm.
  assign probes_needed = (size > 8'd32) ? 8'd6 : (size > 8'd16) ? 8'd5
                       : (size > 8'd8)  ? 8'd4 : (size > 8'd4)  ? 8'd3
                       : (size > 8'd2)  ? 8'd2 : (size > 8'd1)  ? 8'd1 : 8'd0;
  // The whole review point: a probe that moved two variables cannot be read.
  assign interpretable = (CHANGE_TWO != 0) ? 1'b0 : 1'b1;
  assign halved = probe && interpretable;
  // SAFETY-OF-EVIDENCE VIOLATION: a probe was spent and the search space did
  // not shrink, because the result cannot be attributed.
  assign bisect_err = probe && !interpretable;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      rem_q <= 8'd0; np_q <= 8'd0; nu_q <= 8'd0;
    end else begin
      if (report_now) rem_q <= size;
      if (probe) begin
        np_q <= np_q + 8'd1;
        if (interpretable) rem_q <= (rem_q == 8'd0) ? 8'd0 : (rem_q >> 1);
        else               nu_q <= nu_q + 8'd1;
      end
    end
  end
endmodule

The measurement. One probe on a 64-wide span:

one probe on a 64-wide span : single_var_left=32 two_var_left=64

The arithmetic is the whole argument. A span of 64 needs six probes at one variable each. A team changing two things at a time is not bisecting at all — its remaining span after a probe is the span it started with.

Evidence to demand. For each probe, what changed and what did not, and the remaining span after it. Two changes in one probe is one probe wasted, not two probes saved.

What escapes. "We changed the clock and the terminator and it got better" — a session that has learned nothing and believes it has learned two things.

Telemetry. None — this item is process. Its artefact is a probe log with one variable per row.

Misleading evidence. Speed. Changing two things at once is faster per probe and slower per session, and the first is what gets noticed.

12. Review Item 7 — Did The Cause Move, Or Only The Symptom?

Under review. Every fix.

Claim at risk. That the defect is gone.

The failure. Raising a retry limit, widening a timeout or adding a buffer can make a symptom disappear without touching its cause. The failure rate drops, the dashboard goes green, and the defect is still there — now with less margin between it and the next symptom.

// RTL 7 - the fix that made the symptom go away.
//
// Raising a retry limit, widening a timeout or adding a buffer can make a
// symptom disappear without touching its cause. The failure rate drops, the
// dashboard goes green, and the defect is still there - now with less margin
// between it and the next symptom. The distinguishing question is whether the
// CAUSE indicator moved, not whether the symptom did.
//
//   BAD  : symptom gone, ship it
//   GOOD : symptom gone AND the cause counter went to zero; otherwise it is
//          masking, and the margin it consumed must be stated
//
// TEACHING MODEL. Sequential.
module masking_fix #(parameter int SYMPTOM_IS_PROOF = 0) (
  input  logic clk, rst_n,
  input  logic       run, cause_active, report_now,
  input  logic [7:0] retry_budget, retries_used,
  output logic [7:0] n_runs, n_symptoms, n_cause_seen, margin_left,
  output logic       symptom, cause_present, fixed_claimed, truly_fixed,
  output logic       mask_err
);
  logic [7:0] nr_q, ns_q, nc_q;

  assign n_runs       = nr_q;
  assign n_symptoms   = ns_q;
  assign n_cause_seen = nc_q;
  // A symptom appears only when the cause outlasts the retry budget.
  assign symptom       = cause_active && (retries_used >= retry_budget);
  assign cause_present = cause_active;
  assign margin_left   = (retry_budget > retries_used)
                       ? (retry_budget - retries_used) : 8'd0;
  // The truth, computed the same way in BOTH builds.
  assign truly_fixed = !cause_present;
  // The whole review point: what "fixed" is judged on.
  assign fixed_claimed = (SYMPTOM_IS_PROOF != 0) ? !symptom : truly_fixed;
  // SAFETY-OF-EVIDENCE VIOLATION: a fix was claimed while the cause is still
  // firing and only the budget is absorbing it.
  assign mask_err = report_now && fixed_claimed && cause_present;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      nr_q <= 8'd0; ns_q <= 8'd0; nc_q <= 8'd0;
    end else if (run) begin
      nr_q <= nr_q + 8'd1;
      if (symptom)       ns_q <= ns_q + 8'd1;
      if (cause_present) nc_q <= nc_q + 8'd1;
    end
  end
endmodule

The measurement. A retry budget raised from 2 to 8, with three retries actually used:

budget raised 2 to 8 : symptom=0 cause=1 margin=5 cause_judged=0 symptom_judged=1

The symptom is gone and the cause is still active. Five retries of margin remain — which is the number the review needs and the dashboard does not show. The symptom-is-proof build calls it fixed.

The distinguishing question is whether the cause indicator moved, not whether the symptom did. The run drives the genuinely fixed case too: cause inactive, symptom gone, and both builds agree.

Evidence to demand. The cause counter before and after, and the margin the fix consumed. A fix that consumed margin and left the cause running is a deferral with a date nobody wrote down.

What escapes. A defect that ships, with its early-warning system spent.

How DV proves it. Drive the cause with the budget raised, and require the cause counter to be non-zero while the symptom counter is zero. A test that only watches the symptom cannot distinguish a fix from a mask.

Telemetry. A cause counter separate from a symptom counter, and the remaining margin as a number. Three registers, and they convert a green dashboard into a reviewable claim.

13. Review Item 8 — Correlation, Or A Common Driver?

Under review. Every causal claim made from two counters that move together.

Claim at risk. The direction, and the existence, of causation.

The failure. Two counters that rise together may be cause and effect, or may both be driven by offered load. The test that separates them is available, cheap, and almost never run.

Hold the suspected cause, and keep the load on. If both counters still move, the correlation was the load. If the suspect's counter stops while the other carries on, the suspect was driving it.

// RTL 8 - two counters that move together, and the experiment that separates
// a common driver from a link.
//
// Correlation during triage is genuinely useful and genuinely dangerous. Two
// counters that rise together may be cause and effect, or may both be driven by
// offered load. The test that separates them is available and almost never run:
//
//   HOLD the suspected cause, and keep the load on.
//     - if BOTH counters still move, the correlation was the load
//     - if the suspect's counter STOPS while the other keeps moving, the
//       suspect was driving it
//
//   BAD  : "errors correlate with retries, so retries cause errors"
//   GOOD : run the experiment, and claim causation only from its result
//
// The first version of this model had both counters follow the same condition,
// which made the experiment impossible to express and two mutations equivalent.
// `x_driven_by_suspect` is what lets the decisive case exist at all.
//
// TEACHING MODEL. General triage reasoning, not CXL behaviour.
module correlation_test #(parameter int CORRELATION_IS_CAUSE = 0) (
  input  logic clk, rst_n,
  input  logic       tick, suspect_held, x_driven_by_suspect, conclude,
  input  logic [7:0] load,
  output logic [7:0] counter_x, counter_y, n_conclusions, n_false,
  output logic       x_moved, y_moved, both_move,
  output logic       load_explains, causal_claimed, truly_causal,
  output logic       corr_err
);
  logic [7:0] x_q, y_q, px_q, py_q;

  assign counter_x = x_q;
  assign counter_y = y_q;
  assign x_moved   = (x_q > px_q);
  assign y_moved   = (y_q > py_q);
  assign both_move = x_moved && y_moved;
  // With the suspect held and both counters still moving, the load is what
  // they were both following.
  assign load_explains = suspect_held && both_move;
  // The decisive result: held the suspect, and X stopped while Y carried on.
  assign truly_causal  = suspect_held && y_moved && !x_moved;
  // The whole review point: what a conclusion is drawn from.
  assign causal_claimed = (CORRELATION_IS_CAUSE != 0) ? both_move : truly_causal;
  // SAFETY-OF-EVIDENCE VIOLATION: causation was concluded without the
  // experiment having produced it.
  assign corr_err = conclude && causal_claimed && !truly_causal;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      x_q <= 8'd0; y_q <= 8'd0; px_q <= 8'd0; py_q <= 8'd0;
      n_conclusions <= 8'd0; n_false <= 8'd0;
    end else begin
      if (tick) begin
        px_q <= x_q; py_q <= y_q;
        if (load != 8'd0) begin
          // Y follows the load, always.
          y_q <= y_q + 8'd1;
          // X follows the load too - unless the suspect is genuinely its
          // driver and the suspect is being held.
          if (!(x_driven_by_suspect && suspect_held)) x_q <= x_q + 8'd1;
        end
      end
      if (conclude) begin
        n_conclusions <= n_conclusions + 8'd1;
        if (corr_err) n_false <= n_false + 8'd1;
      end
    end
  end
endmodule

The measurement. Both cases, on the same model:

suspect held, not the driver : x=2 y=2 both_move=1 load_explains=1
suspect held, IS the driver : x_moved=0 y_moved=1 truly_causal=1

The first line is the negative result and it is the more valuable one. The suspect is held and both counters still move — so whatever they were following, it was not the suspect. The second is the decisive positive: X stopped, Y carried on, and that asymmetry is what licenses a causal claim.

The run drives a third case: the experiment was never run. The suspect is not held, both counters move, and the shortcut build claims causation from a bare correlation while the tested build claims nothing. Both builds see the identical counter data — the testbench asserts that explicitly — and only the conclusion differs.

This model had to be rebuilt to express its own experiment

The first version drove both counters from the same condition. Two mutations were then unkillable — not because the checkers were weak, but because the model had no way to represent the decisive case. A held suspect that genuinely drives X and a held suspect that does not were the same state.

An explicit is the suspect actually X's driver input was added, which lets both outcomes of "hold the suspect" exist. Both mutations now die, and the chapter gained its sharpest case. A mutation that cannot be killed is sometimes telling you the model cannot express the experiment.

Evidence to demand. The experiment, or the absence of a causal claim. There is no third option — a correlation with no hold-the-suspect result is an observation, and it belongs in the hypothesis list rather than in the conclusion.

What escapes. A fix applied to something that was never the cause, and a real cause that keeps running.

Telemetry. The two counters, plus a way to hold the suspect in the field — a disable bit, a forced mode, a rate limit. The experiment is worth designing for.

14. Review Item 9 — Did The Probe Supply The Margin?

Under review. Every fault that disappears when it is instrumented.

Claim at risk. That the instrumented path is innocent.

The failure. Attaching an analyser, enabling a trace or lowering a clock changes the timing of the thing being debugged. A marginal fault that depended on that timing stops reproducing, and the natural reading — "the probe is clean, so the problem is elsewhere" — is exactly backwards.

// RTL 9 - the fault that disappears when you look at it.
//
// Attaching an analyser, enabling a trace or lowering a clock changes the
// timing of the thing being debugged. A marginal fault that depended on that
// timing stops reproducing, and the natural reading - "the probe is clean, so
// the problem is elsewhere" - is exactly backwards. The disappearance IS
// evidence: it says the fault is timing-dependent.
//
//   BAD  : probe on, fault gone, conclude the probe's path is innocent
//   GOOD : record that the fault is probe-sensitive, and treat that
//          sensitivity as a narrowing observation in its own right
//
// TEACHING MODEL.
module probe_sensitivity #(parameter int SILENCE_IS_INNOCENCE = 0) (
  input  logic clk, rst_n,
  input  logic       run, probe_on, conclude,
  input  logic [7:0] margin_ns, probe_cost_ns,
  output logic [7:0] effective_margin, n_runs, n_faults, n_wrong_calls,
  output logic       fault_seen, probe_sensitive, cleared, truly_clear,
  output logic       heisen_err
);
  // The three counters ARE the output ports. The first version of this model
  // kept them as internal registers and never drove the ports, so `n_runs`,
  // `n_faults` and `n_wrong_calls` were X for the whole run - undriven outputs
  // that `-Wall` does not warn about and that only the X/Z rejection caught.
  logic sens_q;

  // The probe adds delay, which RESTORES margin a marginal path had lost.
  assign effective_margin = probe_on ? (margin_ns + probe_cost_ns) : margin_ns;
  assign fault_seen       = (effective_margin < 8'd10);
  assign probe_sensitive  = sens_q;
  // The truth: the path is clear only if it passes WITHOUT the probe.
  assign truly_clear = (margin_ns >= 8'd10);
  // The whole review point: what a quiet probe run is taken to mean.
  assign cleared = (SILENCE_IS_INNOCENCE != 0) ? (probe_on && !fault_seen)
                                               : truly_clear;
  // SAFETY-OF-EVIDENCE VIOLATION: a path was declared clear on the strength of
  // a run whose own instrumentation supplied the margin.
  assign heisen_err = conclude && cleared && !truly_clear;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_runs <= 8'd0; n_faults <= 8'd0; n_wrong_calls <= 8'd0; sens_q <= 1'b0;
    end else begin
      if (run) begin
        n_runs <= n_runs + 8'd1;
        if (fault_seen) n_faults <= n_faults + 8'd1;
        // A fault that is absent with the probe and present without it is
        // probe-sensitive, and that is a finding rather than a nuisance.
        if (probe_on && !fault_seen && !truly_clear) sens_q <= 1'b1;
      end
      if (conclude && heisen_err) n_wrong_calls <= n_wrong_calls + 8'd1;
    end
  end
endmodule

The measurement. A margin of 6 units, a probe adding 5:

margin 6, probe adds 5 : effective=11 fault=0 truth_cleared=0 silence_cleared=1

The probe supplied nearly as much margin as the design had. The fault stops appearing, and the silence-is-innocence build concludes the path is clear. The honest build records the disappearance as probe-sensitive — which is a narrowing observation in its own right, and one of the most informative available.

The disappearance IS evidence. It says the fault is timing-dependent.

Evidence to demand. The probe's cost in the same units as the margin. If nobody can state it, the probe's result cannot be interpreted in either direction.

What escapes. A timing-marginal defect declared absent, on the strength of a measurement that removed it.

How DV proves it. Run with the probe on and off, and require the honest build to set probe_sensitive and refuse to clear. A test with the probe always on can never see this.

Telemetry. A probe-enabled bit alongside every fault count, so a field capture can be read as "faults with instrumentation" rather than as "faults".

15. Review Item 10 — Which Layer, And What Excluded The Others?

Under review. Every statement of the form "the link is broken".

Claim at risk. That anything has been isolated at all.

The failure. A failing interconnect looks the same from the top whatever broke. Triage by layer is the fastest narrowing available, and it works because each layer leaves a different signature.

// RTL 10 - which layer is this, and what evidence would say so?
//
// A failing interconnect looks the same from the top whatever broke. Triage by
// layer is the fastest narrowing available, and it works because each layer
// leaves a different signature: a physical problem tracks with temperature,
// voltage and lane, a link problem tracks with traffic and recovers, a protocol
// problem is deterministic on a particular transaction, and a software problem
// survives a link reset.
//
//   BAD  : "the link is broken"
//   GOOD : name the layer, and name the observation that excluded the others
//
// TEACHING MODEL. The layer signatures here are general interconnect reasoning,
// not CXL specification behaviour.
module layer_triage #(parameter int SKIP_THE_LAYERS = 0) (
  input  logic clk, rst_n,
  input  logic       observe, conclude,
  input  logic       lane_specific, recovers_on_retrain, survives_reset,
  output logic [7:0] layer_id, sig_count, n_observations, n_unisolated,
  output logic       isolated, reported_isolated,
  output logic       layer_err
);
  // 1 physical, 2 link, 3 protocol, 4 software, 0 not isolated.
  assign layer_id = lane_specific        ? 8'd1
                  : recovers_on_retrain  ? 8'd2
                  : survives_reset       ? 8'd4
                                         : 8'd3;
  // How many layer signatures the observation carries. This is the quantity
  // that decides whether anything was isolated, and writing it out is what
  // stops `isolated` becoming a tautology - the first version of this line was
  // `a || b || c || (!a && !b && !c)`, which is true for every input.
  assign sig_count = {7'd0, lane_specific} + {7'd0, recovers_on_retrain}
                   + {7'd0, survives_reset};
  // The truth: an observation isolates a layer when it carries exactly one
  // signature, or none - none excludes the other three and leaves protocol by
  // elimination. TWO signatures is contradictory evidence and isolates nothing.
  assign isolated = (sig_count <= 8'd1);
  // The whole review point: whether the triage publishes a layer at all.
  assign reported_isolated = (SKIP_THE_LAYERS != 0) ? 1'b0 : isolated;
  // SAFETY-OF-EVIDENCE VIOLATION: a conclusion was drawn with no layer named,
  // so nothing was excluded and the next step is a guess.
  assign layer_err = conclude && !reported_isolated;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_observations <= 8'd0; n_unisolated <= 8'd0;
    end else begin
      if (observe) n_observations <= n_observations + 8'd1;
      if (layer_err) n_unisolated <= n_unisolated + 8'd1;
    end
  end
endmodule

The measurement. A fault that is lane-specific and nothing else:

lane-specific only : signatures=1 layer=1 layered_names_it=1 unlayered_names_it=0

Both builds identify the same layer internally. The difference is that one of them publishes it — and a conclusion with no layer named is the thing this item exists to catch.

Two signatures is contradictory evidence, and it isolates nothing

lane-specific AND survives a reset : signatures=2 isolates=0

A fault cannot be lane-specific and survive a link reset, and an observation carrying both signatures has isolated nothing at all. The honest model reports that as a non-isolation, and drawing a conclusion from it is a safety violation in both builds.

This case is why the model was rebuilt. Its original isolated output was a || b || c || (!a && !b && !c) — true for every possible input, a tautology dressed as a test. Section 20 records how it was found. The replacement counts signatures: exactly one isolates, none isolates the protocol layer by elimination, and two isolates nothing.

Evidence to demand. The layer, and the observation that excluded each of the other three. The second half is the review item.

What escapes. A team working the wrong layer for as long as the schedule allows.

Telemetry. Per-layer counters — lane-correlated errors, retrain counts, per-transaction failure counts, post-reset persistence. Four counters that narrow four ways.

16. Review Item 11 — Was The Snapshot Taken Before The Reset?

Under review. Every recovery action.

Claim at risk. That the failure can be diagnosed at all.

The failure. The first instinct on a hung link is to reset it, and a reset clears exactly the state a diagnosis needs: the status registers, the outstanding table, the trace. Recovery and diagnosis want opposite things from the same moment.

// RTL 11 - the reset that destroyed the evidence.
//
// The first instinct on a hung link is to reset it, and a reset clears exactly
// the state a diagnosis needs: the status registers, the outstanding table, the
// trace. Recovery and diagnosis want opposite things from the same moment, and
// the only way to have both is to snapshot before recovering.
//
//   BAD  : reset, restore service, then ask what happened
//   GOOD : capture the snapshot, THEN reset - and refuse to reset until the
//          capture is acknowledged
//
// TEACHING MODEL. Sequential.
//   Safety : no recovery reset happens with an uncaptured failure present.
module snapshot_first #(parameter int RESET_IMMEDIATELY = 0) (
  input  logic clk, rst_n,
  input  logic failure, capture, reset_req, report_now,
  output logic [7:0] n_failures, n_captures, n_lost,
  output logic       failure_live, captured, reset_allowed, evidence_lost,
  output logic       esc_err
);
  logic fl_q, cap_q;

  assign failure_live = fl_q;
  assign captured     = cap_q;
  // The whole review point: whether the reset waits for the capture.
  assign reset_allowed = (RESET_IMMEDIATELY != 0) ? reset_req
                                                  : (reset_req && cap_q);
  assign evidence_lost = reset_allowed && fl_q && !cap_q;
  // SAFETY-OF-EVIDENCE VIOLATION: the state that explains the failure was
  // cleared before anybody read it.
  assign esc_err = evidence_lost;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      fl_q <= 1'b0; cap_q <= 1'b0;
      n_failures <= 8'd0; n_captures <= 8'd0; n_lost <= 8'd0;
    end else begin
      if (failure && !fl_q) begin
        fl_q <= 1'b1;
        n_failures <= n_failures + 8'd1;
      end
      if (capture && fl_q && !cap_q) begin
        cap_q <= 1'b1;
        n_captures <= n_captures + 8'd1;
      end
      if (evidence_lost) n_lost <= n_lost + 8'd1;
      // The recovery clears the diagnostic state, which is the whole problem.
      if (reset_allowed) begin
        fl_q <= 1'b0; cap_q <= 1'b0;
      end
    end
  end
endmodule

The measurement. A live failure with no snapshot taken:

failure live, no snapshot : waits=0 resets=1

The snapshot-first build waits and does not reset. The reset-immediately build resets, restores service, and destroys the evidence — and the run asserts that its snapshot captures nothing, because the failure had already been reset away.

That last point was a wrong oracle of mine, and section 20 keeps it: I expected the immediate-reset build to capture something. It captures nothing, and that is the failure mode rather than an artefact of the model.

Evidence to demand. The order: capture, acknowledge, then reset. And what the design does if the capture is never acknowledged, which is the question that decides whether this is a policy or a hang.

What escapes. A failure that recurs monthly and has never once been diagnosed, because every occurrence was recovered before it was read.

How DV proves it. Raise a failure, request a reset without capturing, and require the reset to be refused. A test that always captures first cannot distinguish the two builds.

Telemetry. A lost-evidence counter — failures that were reset without capture. Permanently zero, and a non-zero value explains years of undiagnosable escapes in one read.

17. The Review Assembled

Eleven dimensions, one summary — and the same trap, in its debug form.

// RTL 12 - a debug review assembled. Eleven triage dimensions, one summary.
// "We reproduced it" is bit 0: the failure happened again, and one sixth of a
// triage.
module dbg_review_signoff #(parameter int REPRODUCED_IS_PROOF = 0) (
  input  logic clk, rst_n,
  input  logic        review,
  input  logic        reproduced, hypotheses_narrowed, first_fault_kept,
  input  logic        order_known, layer_named, snapshot_taken,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_reviews, n_sound, n_claimed,
  output logic        dbg_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~reproduced;
  assign fail_mask[1] = ~hypotheses_narrowed;
  assign fail_mask[2] = ~first_fault_kept;
  assign fail_mask[3] = ~order_known;
  assign fail_mask[4] = ~layer_named;
  assign fail_mask[5] = ~snapshot_taken;
  assign conditions_met = {15'd0, reproduced} + {15'd0, hypotheses_narrowed}
                        + {15'd0, first_fault_kept} + {15'd0, order_known}
                        + {15'd0, layer_named} + {15'd0, snapshot_taken};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp: six one-bit values over six cannot exceed a hundred.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  assign claimed = (REPRODUCED_IS_PROOF != 0) ? reproduced : truly_sound;
  assign sound = claimed;
  assign dbg_err = review && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reviews <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (review) begin
      n_reviews <= n_reviews + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The measurement. Two views of the same debug conclusion:

no snapshot taken : mask=100000 met=5 sound=83%
we reproduced it, and nothing else : mask=111110 met=1 sound=16%

The first line is a real session with one finding open — bit 5, no snapshot was taken before recovery. Five of six conditions met, and the conclusion is one capture away from being defensible.

The second line is what this chapter exists to prevent. Bit 0 is clear and everything else is set: the failure was reproduced, and nothing was eliminated, nothing was ordered, no layer was named, and no evidence was kept. Sixteen percent of a triage.

Reproduction is genuinely necessary — it is the precondition for everything else — and it is the one step that feels like a result. That is exactly why it is bit 0.

Figure 3 — reproduction is the precondition, not the result. The five conditions above it are each a statement about what the session eliminated, and each has an artefact: a shrinking list, a sticky register, a sequence number, a named layer, a capture.

18. Quantitative Reasoning

Every figure here is a teaching parameter or a value derived from one and asserted by the testbench. None is a measurement of a real system, and none is a CXL specification number.

Narrowing, derived. Five hypotheses with two ruled out leaves 3 and is 2 x 100 / 5 = 40 percent narrowed. Committing to one of three is a one-in-three guess, which is to say a 67 percent chance of working the wrong problem. An observation that rules out none leaves the narrowing at zero however long it took to make.

Cascade against status depth. A root fault followed by three consequences gives a last-fault register holding the fourth code and a first-fault register holding the first. The general form: a last-fault register holds the root only when the cascade length is exactly one, which is the case a single-fault test drives.

Trace retention, derived. A four-deep trace with six entries written after the root has evicted 6 − 4 = 2 entries, and the root is one of them. The root survives only while entries-since-root is below the depth, so the decision is a comparison between the expected cascade length and the buffer depth — two numbers that are known before tapeout.

Ordering. Fault A at sequence 0 and fault B at sequence 2 gives an ordering from one comparison. Two sticky bits with no stamps give two facts and no relationship — and there are exactly two orderings consistent with them, pointing at opposite causes.

Intermittent confidence, derived. A pre-fix rate of one failure in ten runs has a mean interval of 10 runs. The heuristic used here — three times the mean interval, declared as a heuristic — needs 30 clean runs. Five clean runs is 16.7 percent of that, and for an unfixed defect at that rate, five clean runs in a row is an entirely ordinary outcome. The silence is not evidence; it is a short sample.

Bisection, derived. A span of 64 needs 6 probes at one variable each, because 2^6 = 64. One probe leaves 32. A probe changing two variables leaves 64 — the span it started with — so a session changing two things at a time does not converge at all, however many probes it runs. Six probes against never is the entire argument for the discipline.

Masking, derived. A retry budget raised from 2 to 8 with 3 retries used leaves 8 − 3 = 5 of margin. The symptom threshold moved from 2 to 8, and the cause is unchanged. Before the fix the margin was 2 − 3 = negative, which is what made the symptom visible. The fix bought 5 units of silence and removed the early warning.

Correlation, derived. With the suspect held and the load on, both counters advance to 2 — no asymmetry, and the correlation is the load. With the suspect genuinely driving X, X stops at its previous value and Y advances, and that difference is the only evidence in this chapter that supports a causal claim. Neither result requires more than one extra experiment.

Probe cost, derived. A margin of 6 with a probe adding 5 gives an effective margin of 11, an 83 percent increase. Against a fault threshold of 10, the design's own 6 fails and the instrumented 11 does not. The probe supplied more margin than the gap it was measuring.

Layer signatures. Four layers and four signatures. Exactly one signature isolates one layer. No signature isolates the protocol layer by elimination — it is the layer with no distinguishing signature of its own, so its evidence is the absence of the other three. Two signatures isolate nothing, because no single layer produces both.

The sign-off arithmetic. Six conditions; five met is 5 x 100 / 6 = 83 percent under integer division, and one met is 100 / 6 = 16 percent.

19. Verification Method

Order of work

compile → inspect warnings → legal baseline → reset → boundaries → simultaneous events → abuse and error cases → configuration contrasts → structural gates → PASS → mutation campaign

A mutation campaign on a failing baseline is invalid, and every campaign in this chapter ran against a green one. The baseline was re-run after every testbench modification before the campaign was re-run — twice in this chapter, once for a rebuilt model and once for three output nets, both recorded in section 20.

Independent oracles

chkv prints got against expected, which is what lets an oracle be wrong out loud. In this chapter it caught five, all mine, recorded in section 20.

X and Z rejected explicitly — and it earned its keep here

chk(c, …) tests c !== 1'b1, so an X-valued condition fails rather than passing. chkv(got, exp, …) reduces the result and reports an explicit X/Z failure before comparing.

That rejection found this chapter's most serious RTL defect. Three outputs of the probe-sensitivity model were declared, counted into internal registers, and never connected — X for the entire run, with no compiler warning of any kind. chkv reported X/Z in result rather than a value mismatch, which is what stopped it being "fixed" by adjusting an expected number.

Pulses are latched, never sampled

Every evidence output — guess_err, fault_err, ord_err, log_err, rate_err, bisect_err, mask_err, corr_err, heisen_err, layer_err, esc_err, dbg_err — is caught by a continuous always @(posedge clk) monitor into a sticky bit, asserted outside any conditional, in a window containing a clock edge.

Stimulus never lands on the active edge, and reset is released after it

step_clk is @(posedge clk); #1;, and reset release lands one delta after the edge. One model needed an explicit mid-run reset as well: the ordering model's sequence counter free-runs, and earlier experiments had consumed cycles, so absolute stamps were not what the oracle assumed.

Safety, liveness and performance kept apart

Safety-of-evidence is this chapter's safety class, and it is worth naming as such. The recorded code is the first fault since the clear. The root entry is still present when the trace is read. No recovery reset happens with an uncaptured failure present. No causal claim is made without the experiment that licenses it. None requires an assumption, and each is a property of the instrumentation rather than of the datapath.

Liveness — nothing in this chapter is a liveness claim. The snapshot-first policy deliberately blocks a reset, and whether that is a hang is a design decision the model exposes rather than settles.

Performance — nothing here is a performance claim. A masking fix is a safety-of-evidence failure, not a slow one.

20. Baseline Defects Found Before Mutation

RTL defects — two, and both are the chapter's own subject

Three outputs declared and never driven. n_runs, n_faults and n_wrong_calls in the probe-sensitivity model were counted into internal registers and never connected to their ports. They were X for the whole run.

Found by the X/Z rejection in chkv, and by nothing else. -Wall does not warn on an undriven output. A checker written as a plain equality would have compared X against 1, failed, and been "fixed" by adjusting the expected number — which is how an undriven output survives into published code.

Containment was checked, not assumed. A static sweep for the same shape ran over all 60 models in this batch and all 30 in batch 030: this was the only instance. The sweep is cheap and is now part of the review.

A tautology dressed as a test. The layer-triage model's isolated output was a || b || c || (!a && !b && !c) — true for every possible input. It looked like a disjunction over the layer signatures and was a constant.

domcheck did not catch it: it models domination between a guard and an enclosing condition, not a term that is universally true. It was found by reading the line and asking what input makes it false.

Replaced with a signature count, which is a better model as well as a correct one: exactly one signature isolates, none isolates the protocol layer by elimination, and two signatures is contradictory evidence that isolates nothing — a case the tautology could never have expressed, and now one of the chapter's sharpest teaching points.

A model that could not express its own experiment

The correlation model originally drove both counters from the same condition, so both_move's && could not be distinguished from ||, and the previous-value snapshot was half untestable. Two mutations were unkillable — not because the checkers were weak but because the model had no way to represent the decisive experiment.

Rebuilt with an explicit "is the suspect actually X's driver" input, which lets both outcomes of holding the suspect exist. Both mutations now die.

Testbench defects — two

Wrong oracles — five, all mine

The fourth became prose, in section 16. I expected the wrong build to capture something; it captures nothing, and that is the failure mode rather than an artefact of the model.

Four of the five are the same class as every wrong oracle in this batch — not arithmetic, but when. A registered value is available the cycle after the condition that produced it, and reading it in the same cycle is the single most productive mistake available to an oracle author.

Coverage gaps found by the structural gates

The three reopened by the rebuild are worth a sentence. Nothing had regressed; the gate had new ground to cover. Closing them produced a teaching assertion: both builds see the identical counter data, and the testbench now asserts that explicitly, so "only the conclusion differs" is checked rather than claimed.

The domcheck hit was settled by injection, not by argument

domcheck reported the intermittent-rate model's runs_needed != 0 conjunct as possibly dead. Rather than reason about it in prose, the mutation that removes it was injected — and it was killed, by the case where no failures were ever observed: runs_needed is zero, the comparison against it is trivially true, and the guard is the only thing preventing a fix being declared for a defect that was never measured.

The guard is load-bearing and the tool's hit was a false positive. This is 30.2's precedent applied: a structural tool's finding is settled by injecting the corresponding mutation and watching what happens.

The standing tooling rule held again

The first run of the structural gates in this session was given a file argument where the tools expect a directory. They reported 0 printed references scanned and 0 of 0 mutations — two confident zeros on input they had never read. Re-running them against the directory produced the three real findings above.

A hit is real; a zero is unread until independently confirmed.

Compiler-warning findings

Under -Wall the twelve models produce no truncation, select-width, signedness, cast, multiplication-width, shift-width or loop-width warnings — and that silence included an output that was never driven at all. It is the clearest demonstration in this batch that compiler silence is not a correctness result.

Simulator constraints

Icarus Verilog 13.0 rejects ref task arguments, carried forward from every chapter in this module.

21. Mutation Testing

99 mutations attempted, 99 non-equivalent, 99 killed. Zero unexplained survivors, zero equivalent mutants withdrawn.

Ten survivors across the two first runs, every one classified before anything was changed.

Seven were a boundary the stimulus approached and never landed on

This is the eleventh through seventeenth instance of the family across three chapters in this batch. 30.5 produced eight, 30.6 produced three, and this chapter produces seven. Path coverage is not value coverage, and a mutation campaign is the only instrument in the suite that reliably tells them apart.

For every comparison in a model, drive the value below it, above it, and exactly on it.

That single rule would have killed seven of the ten survivors before either campaign ran. It is the highest-yield stimulus rule this batch has produced.

Three were checkers that did not look

The third is the one to remember. A counter assertion taken at a single point can be satisfied by an inverted counter, and the only defence is a second assertion after a state change that separates them.

The classification rule

Never add an assertion for a survivor before classifying it.

The fifth row is this chapter's addition, and it is the one that produced a better model rather than a better testbench.

22. Synthesis And Implementation Reality

A first-fault register costs the same as a last-fault register. One flop of capture state and an enable term. if (!captured) first <= code against status <= code is a two-input gate, and there has never been an area argument for the second.

Keeping both costs one extra register of the code width. For an 8-bit code that is 8 flops. The last fault tells you where the cascade ended and the first tells you where it began, and a block with both is diagnosable from outside.

A shared sequence counter orders every error source in a block. One counter, plus a capture register per error bit. At 8 bits and six error sources that is 8 + 48 = 56 flops for an ordering across the whole block — the cheapest causal evidence available in hardware.

Stop-on-first-fault is a free-running trace plus one enable term. The buffer already exists. The expensive version of this item is the reserved-slot design, which needs an extra entry and eviction logic that skips it.

An observation counter and a failure counter are two counters. They are what convert "it stopped happening" into a rate, and a rate is the only form in which an intermittent claim can be reviewed.

A cause counter beside a symptom counter is one extra counter, and it is the difference between a dashboard and a review. The margin figure is a subtraction on numbers the design already has.

Designing for the hold-the-suspect experiment costs a disable bit. A rate limit, a forced mode, a way to stop the suspected cause while the load continues. It is the only item in this chapter whose cost is a feature rather than a register, and it is the one that converts a correlation into a conclusion.

A probe-enabled bit stored alongside every fault count costs one flop per count. It is what makes a field capture readable as "faults with instrumentation" rather than as "faults".

Four per-layer counters are four counters. Lane-correlated errors, retrain counts, per-transaction failures, post-reset persistence. Each narrows a different way, and together they are the fastest triage available.

A snapshot-before-reset policy is a handshake: a capture-done bit, a reset-request input, and a gate. Its cost is a design decision about what happens when the capture is never acknowledged, which is a specification question rather than a gate count.

A lost-evidence counter is one counter. It counts failures reset without capture, it must read zero, and a non-zero value explains years of undiagnosable escapes in one register read.

No area, frequency or power figures appear in this chapter, because none was measured.

23. Silicon Observability

Two of these must read permanently zero — lost evidence, and false conclusions. Each costs one counter, and each captures a failure that is otherwise invisible for the life of the product.

The pattern to read on the cause-and-symptom pair: a symptom count falling to zero while the cause count keeps rising is a masking fix, visible from outside the design. Neither number alone shows it.

A first-fault register reading the same code as the last-fault register means one of two things — the cascade was one fault long, or the sticky logic is not sticky. Distinguishing them takes a fault counter, which is why the three are specified together.

The lost-evidence counter is the one to keep if only one survives area review. It is one counter, it must read zero, and it is the only item here that tells you the reason you have no data.

24. DebugLabs

Lab 1 — Three weeks of measurement and the hypothesis list has not moved

Symptom. A link fails intermittently in a customer system. The team has produced traces, scope captures and error logs for three weeks.

Evidence. Every measurement is accurate. Nobody can say how many candidate causes remain.

Hypothesis. No observation has ruled anything out.

Investigation. Write the hypothesis list as it stood at the start, then walk each measurement and mark which entries it removed. Most removed none.

Root cause. Measurement without elimination. Each observation was chosen because it was available, not because it would distinguish two hypotheses.

Fix. Before each measurement, state which hypothesis it will remove if it comes out either way. A measurement that removes nothing under both outcomes is not run.

Prevention. A standing triage artefact: a numbered hypothesis list that only shrinks, with the observation that removed each entry.

Silicon observability. None — this is the one item in the chapter with no register behind it. Its artefact is the list, and the absence of one is the finding.

Lab 2 — The status register always reads the same unhelpful code

Symptom. Every failure in the field reports the same error code, and it is a generic one.

Evidence. The code is real and correctly reported. Lab reproductions show many different faults.

Hypothesis. The register records the last fault, and every cascade ends the same way.

Investigation. Add a fault counter. It reads over a hundred per failure.

Root cause. status <= fault_code — last one wins, and the hundredth is always the generic consequence.

Fix. A sticky first-fault register with an explicit clear, kept beside the last-fault register.

Prevention. A review rule that reads every status register's enable term. if (!captured) is the whole fix.

Silicon observability. First, last, and a count. The three together say whether there was a cascade and where it began.

Lab 3 — Two error bits, and the team fixed the wrong one

Symptom. A status word shows a CRC error and a retrain. The team fixes the retrain path. The failure persists.

Evidence. Both bits are set on every occurrence. There is no ordering anywhere.

Hypothesis. The retrain is a consequence of the CRC error, not its cause.

Investigation. Add a sequence stamp per bit. The CRC error is stamped earlier on every occurrence.

Root cause. Two sticky bits with no shared ordering, and a causal arrow drawn by intuition.

Fix. A free-running sequence counter and a capture register per error bit.

Prevention. For any multi-bit status word, ask what the bits are ordered against. Two bits admit two orderings and two opposite conclusions.

Silicon observability. The stamps. One comparison replaces a guess.

Lab 4 — The trace is complete, accurate, and does not contain the failure

Symptom. A 1024-entry circular trace is captured on every failure. Every entry is correct. The root cause is in none of them.

Evidence. The trace is stopped by a software handler tens of microseconds after the first fault.

Hypothesis. The cascade is longer than the buffer.

Investigation. Add an entries-since-root counter. It reads in the thousands.

Root cause. A free-running trace stopped by a human. The root was evicted by the noise it caused.

Fix. Stop-on-first-fault, or a reserved slot the root cannot be evicted from.

Prevention. Compare the expected cascade length against the buffer depth before tapeout. Both numbers are available.

Silicon observability. An overwrite counter. A non-zero value says the capture is incomplete, which a complete-looking trace does not.

Lab 5 — The fix worked and the failure came back six months later

Symptom. An intermittent failure at roughly one run in ten stops after a change. It returns at scale.

Evidence. Five clean runs after the fix, and a release decision made on them.

Hypothesis. Five clean runs is what luck produces at that rate.

Investigation. Compute the runs needed for confidence from the pre-fix rate. It is thirty, six times what was done.

Root cause. Silence read as evidence, with no pre-fix rate to compare against.

Fix. State the pre-fix rate, derive the clean-run target, and hold the release until it is met — or ship with the uncertainty written down.

Prevention. Observation and failure counters on every intermittent defect, from the first occurrence.

Silicon observability. The two counters, cleared explicitly. A rate is two numbers, and publishing only the failure count is what makes silence look like proof.

Lab 6 — Every probe makes it better and nothing converges

Symptom. A marginal link improves with each change and never becomes reliable. Six weeks, no convergence.

Evidence. The probe log shows two or three changes per experiment.

Hypothesis. The team is not bisecting.

Investigation. Replay the log and compute the remaining span after each probe. It never halves.

Root cause. Multiple variables per experiment. Every result is uninterpretable, so the space never shrinks.

Fix. One variable per probe, and the remaining span published after each.

Prevention. A probe log with one variable per row. A span of 64 needs six probes; a session changing two things at a time needs infinitely many.

Silicon observability. None — process. The artefact is the log.

Lab 7 — The dashboard went green and the margin went with it

Symptom. A retry-related failure disappears after a configuration change. Six months later a different, harder failure appears under load.

Evidence. The retry budget was raised from 2 to 8. The retry counter still shows 3 retries per transaction.

Hypothesis. The symptom was absorbed; the cause is unchanged.

Investigation. Read the cause counter. It never went to zero.

Root cause. A masking fix. The budget rose past the cause, and the early-warning margin was spent.

Fix. Address the cause, and restore the budget. If the budget must stay raised, state the margin consumed as a known risk with an owner.

Prevention. Require the cause counter to move, not the symptom counter. Every fix, every time.

Silicon observability. Symptom and cause as separate counters, plus remaining margin. A symptom falling to zero while the cause keeps rising is visible from outside the design.

Lab 8 — Every occurrence was recovered and none was ever diagnosed

Symptom. A link hangs roughly monthly across a fleet. Recovery is automatic and works. After a year there is no diagnosis.

Evidence. Every occurrence is logged as "link reset, service restored". No status register content survives.

Hypothesis. The recovery destroys the evidence before anything reads it.

Investigation. Add a capture-before-reset path on one system and wait. The first occurrence produces a complete snapshot and a diagnosis in an afternoon.

Root cause. Reset as the first response. Recovery and diagnosis want opposite things from the same moment.

Fix. Capture, acknowledge, then reset — and refuse the reset until the capture is acknowledged, with a stated timeout for the case where nobody acknowledges.

Prevention. A recovery path is a diagnosis path with a snapshot in front of it. Review them together.

Silicon observability. A lost-evidence counter. Permanently zero, and a non-zero value explains a year of undiagnosable escapes in one read.

25. Coverage Reasoning

Functional coverage measures what the stimulus reached. It does not measure whether a conclusion was licensed, and this chapter contains eleven wrong conclusions that a fully covered environment would still draw.

Four coverage models are worth adding to any environment reviewed with this chapter:

Cascade-length coverage. Bins on the number of faults per failure: one, two, and more than the trace depth. The bin a last-fault register can never be distinguished in is "exactly one" — which is the bin a directed single-fault test fills exclusively.

Elimination coverage. A bin per observation on how many hypotheses it removed: none, one, more than one. The "none" bin is the one that matters, it was missing from the first campaign here, and a mutation found it.

Boundary coverage per comparison. For every comparison in the design, three bins: below, above, and exactly on. Section 21 records seven survivors this one model would have caught.

Experiment coverage on every causal claim. A cross of "suspect held" against "counters moved". The cell that licenses a causal claim is held-and-only-one-moved, and a session that never holds the suspect populates none of it.

The bin the weak build cannot hit is the most valuable bin in any model. In section 7 it is "recorded code differs from the last code". In section 12 it is "symptom absent while cause present". In section 15 it is "two signatures present". Each is unreachable in the shortcut build and trivial in the honest one, which makes the coverage report a direct test of the review item.

26. How This Appears In Real Engineering

Triage without elimination is the default state of a debug session under pressure, because every available measurement feels like progress and the hypothesis list is usually in somebody's head rather than on a page.

Last-fault status registers are everywhere because status <= code is the obvious line to write and the cascade is invisible until somebody counts it.

Ordering is missing from most status words because each error bit was added by a different person at a different time, and a shared sequence counter is an architectural decision nobody was asked to make.

Circular traces are stopped by software, tens of microseconds after the fault, which is thousands of entries too late on a fast link.

"It has not failed since the fix" is the most common release criterion for an intermittent defect, because the alternative requires stating a rate and the rate is usually unknown.

Multi-variable probes happen under schedule pressure, and they are genuinely faster per experiment — which is the only thing anybody measures.

Masking fixes are rewarded. The symptom stops, the dashboard turns green, and the margin that was spent does not appear on any chart.

Correlation is presented as causation in almost every triage review, because the correlation is real, the counters are real, and the hold-the-suspect experiment requires a way to hold the suspect that nobody designed in.

Probe-sensitive faults are declared absent because the natural reading of silence is innocence, and the instrument's own timing cost is rarely quantified.

Layer triage is skipped because "the link is broken" is a true statement that feels like a diagnosis.

Reset is the first response to a hang because service restoration has an owner and a metric, and diagnosis has neither until somebody asks why a year of occurrences produced no data.

27. Common Misconceptions

"We are making progress, we measured a lot." Measurement is not elimination. How many hypotheses are still alive?

"The status register says error 44." That may be the hundredth consequence. Ask what the first one was.

"Both bits are set, so both happened." True, and it does not say in which order — and the order is the diagnosis.

"The trace is complete." It is complete about the wrong part of the failure. Ask how many entries were written after the root.

"It has not failed since the fix." How many clean runs would luck have produced? For a one-in-ten fault, five is an ordinary afternoon.

"We changed two things and it got better." Then you have learned nothing, and the search space did not shrink.

"The symptom is gone." Did the cause counter move? If not, the fix spent margin and deferred the failure.

"The two counters track each other, so one causes the other." Or both follow the load. Hold the suspect and look again.

"The fault went away when we put the analyser on, so the analyser's path is clean." The analyser supplied margin. The disappearance is evidence that the fault is timing-dependent.

"The link is broken." Which layer, and what excluded the other three?

"Two signatures, so we have lots of evidence." Two signatures that no single layer produces is contradictory evidence, and it isolates nothing.

"Reset it and see if it comes back." Then it will come back, and you will know exactly as much as you do now.

"We reproduced it." That is bit 0, and it is worth one sixth of a triage.

28. Interview And Design-Review Questions

Elimination and evidence

1. What is the only measure of progress during triage? The size of the surviving hypothesis set. Everything else is activity.

2. Five hypotheses, an observation rules out two. What have you got? Three remaining and 40 percent narrowed — and committing to one of three is a one-in-three guess.

3. What is an observation that rules nothing out? Time spent. It is not a finding, and a triage log full of them reads exactly like a productive session.

4. How do you choose the next measurement? By which hypothesis it eliminates under each outcome. If neither outcome removes anything, do not run it.

5. Why is a long, careful measurement log misleading? Volume reads as narrowing to anybody not counting. The two are unrelated.

Status, ordering and capture

6. What does a last-fault status register record during a cascade? The last consequence, because every subsequent fault overwrote the one before it. With a hundred faults per failure the register holds the hundredth, and the root that caused them all is gone before anybody reads it.

7. What is the fix, and what does it cost? if (!captured) first <= code. One flop of capture state and a two-input gate — the same registers.

8. Why keep the last-fault register as well? It says where the cascade ended, which is a different and also useful question.

9. What does a first-fault register need that a last-fault register does not? An explicit clear. Without one it answers once per power-up.

10. Two error bits are set. What do you know? That both happened. Not the order, and the order decides which one you fix.

11. CRC error and retrain, both set. Give the two readings. CRC then retrain is a link problem. Retrain then CRC is a consequence of the retrain. Opposite fixes.

12. What turns two bits into an ordering? A shared free-running counter and a capture register per bit.

13. What should an ordering report when two faults arrive in the same cycle? Unknown. An instrument that always produces an answer cannot express "I do not know".

14. Why does a 1024-entry trace routinely miss the root cause? The cascade is longer than the buffer, and the capture is stopped by software microseconds later.

15. Name two fixes. Stop-on-first-fault, and a reserved slot the root cannot be evicted from.

16. What single counter tells you the capture is incomplete? An overwrite count, or entries-since-root. Either one converts a trace that looks complete into one that is known to be missing its beginning — which is the difference between a trace and a trap.

Fixes, rates and experiments

17. Why is "it has not failed since the fix" not evidence? Because it does not say how many clean runs the pre-fix rate would have produced anyway.

18. A one-in-ten fault, five clean runs after a fix. Conclusion? None. Five is an ordinary sequence at that rate.

19. What three numbers make an intermittent claim reviewable? The pre-fix rate, the clean-run target derived from it, and the clean runs actually done.

20. What does a fix declared for a defect that was never measured look like? A zero pre-fix rate and a target of zero clean runs — which is why the guard against it is load-bearing.

21. A search space of 64 needs how many probes? Six, because each probe halves the space and two to the sixth is 64. That arithmetic only holds if each probe changes one variable; change two and the space does not halve at all, so the session does not converge however many probes it runs.

22. What happens to the span if a probe changes two variables? Nothing. The result is uninterpretable and the probe must be repeated.

23. Why does multi-variable probing survive? It is faster per experiment, and that is the only thing anyone measures.

24. Distinguish a fix from a mask. A fix moves the cause indicator. A mask moves the symptom and spends margin.

25. A retry budget goes from 2 to 8 and the symptom stops. What do you ask for? The cause counter, and the remaining margin. Three retries against a budget of eight leaves five.

26. Why is a masking fix worse than no fix? The defect remains and the early warning is gone.

Correlation, probes and layers

27. Two counters rise together. What are the two explanations? Cause and effect, or a common driver — usually offered load.

28. State the experiment that separates them. Hold the suspected cause, keep the load on. If both still move, it was the load. If the suspect's counter stops and the other carries on, the suspect was driving it.

29. Which result is more valuable in practice? The negative one. It removes a hypothesis, and most sessions never produce one.

30. What does designing for that experiment cost? A disable bit or a rate limit on the suspect. It is the one item here whose cost is a feature.

31. A fault stops reproducing when you attach an analyser. What have you learned? That the fault is timing-dependent. The disappearance is the evidence.

32. What is the wrong reading? That the instrumented path is clean. It is exactly backwards: the probe added margin, the fault depended on the margin, and the disappearance is a positive result about the fault's nature rather than a negative one about the path.

33. What number makes the probe's result interpretable? Its timing cost, in the same units as the design's margin. A margin of 6 against a probe adding 5 is not a measurement of the design.

34. Name the four layer signatures. Tracks with lane, temperature or voltage — physical. Recovers on retrain, tracks with traffic — link. Deterministic on a transaction — protocol. Survives a link reset — software.

35. Which layer has no signature of its own? The protocol layer, in this model — it is reached by elimination when none of the other three signatures is present.

36. An observation carries two signatures. What has it isolated? Nothing. No single layer produces both, so the evidence is contradictory.

37. What must accompany a named layer in a review? The observation that excluded each of the other three.

Evidence preservation and method

38. Why is reset the wrong first response to a hang? It clears the status registers, the outstanding table and the trace — exactly the state a diagnosis needs.

39. What is the correct order? Capture, acknowledge, then reset — and the reset is refused until the capture is acknowledged. Recovery and diagnosis want opposite things from the same moment, and that ordering is the only way to have both.

40. What question does that policy raise? What happens if the capture is never acknowledged. That is a specification decision, and it must be made rather than discovered.

41. What single counter explains a year of undiagnosable escapes? Failures reset without capture. It costs one flop, it must read permanently zero, and a non-zero value tells you why every occurrence produced a recovery log and no evidence.

42. A mutation cannot be killed and is not equivalent. Name the possibility this chapter found. The model has no way to express the decisive experiment. Two counters driven by the same condition cannot represent holding one of them.

43. What is the right response to that? Rebuild the model so the experiment exists, then re-mutate. Not a stronger checker on a model that cannot represent the case.

44. An output is declared and never driven. What catches it? Explicit X/Z rejection in the checker. Not the compiler — -Wall says nothing.

45. Why does a plain equality checker make that defect worse? It compares X against a number, fails, and invites somebody to adjust the expected value instead of finding the undriven net.

46. Name the single question this chapter's review turns on. What does this evidence rule out, and what is still alive?

29. Exercises

1 — Debug · Intermediate. Builds: converting activity into elimination. You inherit a three-week debug session with a folder of accurate measurements and no hypothesis list. Bounded scope: reconstruct the list, classify each measurement by how many entries it removed, and state the next measurement and what it eliminates under each outcome. Hint: a measurement that eliminates nothing under both outcomes should not be run.

2 — Design · Intermediate. Builds: specifying status registers that survive a cascade. Specify the error-status scheme for a block with six error sources. Bounded scope: name every register, say what clears each, and state how a reader determines whether there was a cascade and which fault began it. Hint: three registers answer "was there a cascade" and only one answers "which was first".

3 — Design review · Advanced. Builds: sizing a trace against the failure it must capture. A design has a 512-entry circular trace stopped by a software handler. Bounded scope: state what you need to know to decide whether it can contain a root cause, do the comparison with a stated cascade length, and give two fixes with their costs. Hint: both numbers are available before tapeout.

4 — Debug · Advanced. Builds: turning silence into a rate. An intermittent failure occurred 4 times in 200 runs. A change is made and 25 clean runs follow. Bounded scope: compute the pre-fix rate and the mean interval, apply a stated confidence heuristic, and say what you would tell a release meeting. Hint: state the heuristic you used as a heuristic.

5 — Debug · Advanced. Builds: designing the experiment that licenses a causal claim. A fabric shows retry counts and CRC error counts rising together under load. Bounded scope: design the experiment that separates cause from common driver, say what hardware feature it requires, and state what each of the two outcomes would let you claim. Hint: one of the two outcomes is the more valuable and the less satisfying.

6 — Debug · Expert. Builds: reading an instrument's effect on its subject. A marginal fault reproduces at 1 in 50 runs and stops entirely when a trace is enabled. Bounded scope: state what you have learned, what you have not, and the two measurements that would separate "the trace path is innocent" from "the trace supplied margin". Hint: the disappearance is a positive result.

7 — Design review · Advanced. Builds: making a recovery path diagnosable. Review an automatic link-recovery path that resets on a hang. Bounded scope: specify the snapshot, the acknowledgement, the timeout policy for an unacknowledged capture, and the counter that proves the policy held. Hint: the timeout is the part that turns a policy into a specification.

8 — Verification · Expert. Builds: coverage that targets an unlicensed conclusion. Define a coverage model that would find a masking fix without anybody suspecting one. Bounded scope: specify the bins, the cross, identify precisely which cell the symptom-only environment can never fill, and say what filling it proves. Hint: the proof is a cell where one counter is zero and the other is not.

30. Summary

Debug is elimination, not inspiration. The only measure of progress is how many hypotheses are still alive, and a team that cannot state that number is guessing with instruments.

An observation that rules nothing out cost time and bought nothing, and a log full of them reads exactly like a productive session.

A last-fault register records the hundredth consequence. if (!captured) costs a two-input gate and records the first.

Two error bits are two facts and no relationship. A shared sequence counter turns them into an ordering, and the ordering is the diagnosis.

A circular trace stopped by a human is full of accurate evidence about the wrong part of the failure. Compare the cascade length against the depth before tapeout.

"It has not failed since the fix" is not evidence. State the pre-fix rate and the clean runs luck alone would have produced.

A probe that changes two variables halves nothing. A span of 64 needs six probes, and a two-variable session needs infinitely many.

A fix that moves the symptom and not the cause spent margin and deferred the failure, and the dashboard turns green either way.

A correlation is not a claim until the suspect has been held. The negative result is the more valuable one, and it requires a feature nobody designs in unless asked.

A fault that disappears when you instrument it has told you something. It is timing-dependent, and the silence is the evidence.

Name the layer, and name the observation that excluded the other three. Two signatures at once is contradictory evidence and isolates nothing.

Recovery and diagnosis want opposite things from the same moment. Capture, acknowledge, then reset.

Six conditions, and "we reproduced it" is one of them. A real session with one finding open is 83 percent. A symptom seen twice is 16.