CXL Silicon Debug

26.6 was about a number measured wrongly. This chapter is about the bench where the measuring happens — and it is last in the module because every instrument it describes is one you reach for after the cheap evidence has run out.

The analyser is on the link. The capture fired. Two gigabytes of trace are sitting on a workstation, and the trace is ground truth about one link of five, for nine hundred cycles after a trigger that fired four hundred cycles too late.

1. The Engineering Problem — A Trace Is Evidence About One Link

An analyser sees the link you put it on. Two probes on an eight-link fabric with four faulty links is a quarter of the fabric visible and three faults nobody is watching — and a clean trace on the two proves nothing about the six. Section 5.

The trigger decides what the capture holds. A thousand-cycle buffer with ten percent of it before the trigger keeps a full buffer and none of the cause, when the cause is five hundred cycles back. Section 6.

The probe changes what it measures. Forty millivolts of loading on a hundred millivolts of margin against a seventy-millivolt threshold is a link that fails only while it is being watched. Section 7.

A rare failure is a schedule, not a difficulty. Ten failures per million runs at a thousand runs an hour is a hundred hours of bench time per capture — thirteen shifts, and the plan that assumed it would turn up budgeted none of them. Section 8.

Two instruments cannot order events closer together than their skew. A hundred nanoseconds of skew against fifty-nanosecond intervals is two hundred orderings from two clocks that never agreed. Section 9.

This chapter against the rest of module 26, stated precisely. 26.1 through 26.6 each own a failure. This one owns the evidence — which is why every model here is about an instrument rather than a fabric, and why section 14's weak definition is a trace.

2. The One-Sentence Model

A capture is evidence when it is on the link that failed, its buffer holds the cause rather than only the symptom, the probe did not take the margin that caused the failure, the instruments can be ordered against each other, one layer rather than three shows the symptom, and the failure comes back inside a shift — and "the analyser shows it" is one of those six.

3. What This Chapter Owns

This is the only chapter in the module whose subject is not a CXL behaviour. That is deliberate and it is where it belongs: every previous chapter ends with an instrument, and the quality of the answer is bounded by the quality of that instrument long before it is bounded by the engineer's understanding of the protocol.

The boundary worth stating is against simulation. A bug reproducible in simulation should never reach a bench, and section 12 puts a number on why. Silicon debug is for the failures simulation could not have produced — a marginal channel, a timing corner across two clock domains, an interaction with a device nobody modelled — and every hour spent at a bench on something a testbench could have caught is an hour spent at twenty times the price.

4. Teaching-Model Boundary

Every model in this chapter is a teaching model, not an instrument. It computes the one relationship the section is about and nothing else. There is no waveform, no capture format, no serial protocol and no trigger engine anywhere in this file.

Each model is built twice from one source. A parameter selects between the measured build, which computes what the evidence actually supports, and the assumed build, which computes what an engineer holding that evidence tends to conclude. The two are instantiated side by side against identical stimulus, and every section's headline number is the gap between them.

The error output in each model separates what is true of the session from what the build concludes about it, and fires only when the second contradicts the first.

5. RTL 1 — An Analyser Sees One Link

Start with the property of a trace that is most often forgotten the moment the trace exists, because a capture is so much more detailed than anything else available that its narrowness stops registering.

A protocol analyser inserted on a CXL link gives complete, cycle-accurate, unambiguous truth about that link. Every flit, every credit, every retry. It is the best evidence in the building, and its scope is one link out of however many the fabric has.

An engineer holding that trace has a strong and entirely natural instinct: the trace is clean, therefore the problem is elsewhere. The first half is a measurement. The second half is a deduction that requires knowing how much of the fabric the trace covers, and that number is almost never computed.

// RTL 1 - an analyser sees one link. A capture on the link you chose is ground
// truth about that link and says nothing whatever about the other four, which
// is the first thing a trace tempts you to forget.
module capture_reach #(parameter int THE_TRACE_IS_THE_SYSTEM = 0) (
  input  logic clk, rst_n,
  input  logic        probe_it,
  input  logic [15:0] links_total, links_probed, fault_links, trace_hours,
  output logic [15:0] links_seen, blind_links, reach_pct, faults_seen,
  output logic        capture_reach_ok,
  output logic [7:0]  n_probes, n_blind,
  output logic        one_link_err
);
  logic [31:0] r_q, o_q;
  logic        truly_blind;
  // A probe list cannot name more links than the fabric has.
  assign links_seen  = (links_probed > links_total) ? links_total : links_probed;
  assign blind_links = links_total - links_seen;
  assign r_q = (links_total == 16'd0) ? 32'd0
             : (({16'd0, links_seen} * 32'd100) / {16'd0, links_total});
  assign reach_pct = (r_q > 32'd100) ? 16'd100 : r_q[15:0];
  // Faults spread evenly over the fabric land on a probed link in proportion.
  assign o_q = (links_total == 16'd0) ? 32'd0
             : (({16'd0, fault_links} * {16'd0, links_seen}) / {16'd0, links_total});
  assign faults_seen = (o_q > {16'd0, fault_links}) ? fault_links : o_q[15:0];
  assign truly_blind = (links_total != 16'd0) && (blind_links != 16'd0)
                       && (fault_links != 16'd0) && (trace_hours != 16'd0);
  // A clean trace on one link is treated as a clean system.
  assign capture_reach_ok = (THE_TRACE_IS_THE_SYSTEM != 0)
                            ? (links_seen != 16'd0) : (blind_links == 16'd0);
  assign one_link_err = probe_it && truly_blind && capture_reach_ok;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_probes <= 8'd0; n_blind <= 8'd0;
    end else if (probe_it) begin
      n_probes <= n_probes + 8'd1;
      if (blind_links != 16'd0) n_blind <= n_blind + 8'd1;
    end
  end
endmodule

Two probes on an eight-link fabric with four faulty links is a quarter of the fabric visible and one of the four faults expected to land on an instrumented link. The trace-is-the-system view signs off on a clean capture; the reach model counts three faults nobody is watching.

Figure 1 — the inference a clean trace supports, next to the one it is used for. The capture on the upper path is perfect evidence: those two links are clean, cycle by cycle, with no ambiguity at all. The conclusion drawn from it covers eight links. The gap between the two is not a measurement problem and cannot be improved by a better analyser — it is a reach problem, and the only instrument that fixes it is a second probe.

Two degenerate cases bracket the model. A blind spot with no fault in it costs nothing — the capture still does not reach the fabric, and neither build reports a problem, because uninstrumented links that are working are not a debug problem. And a blind spot with a fault in it and no trace actually running cannot mislead anybody: the model requires a capture to be in progress before it will claim one is being over-read.

The clamp is the usual shape and the usual reason. A probe list naming nine links on a four-link fabric is a configuration file that outlived its fabric, and a model that believed it would compute negative blind links and report full reach on a fabric with a blind spot.

The number to carry out of this section is not the percentage. It is the sentence: a clean trace is evidence that the link it was taken on is clean. Everything beyond that is an inference whose strength is exactly the fraction of the fabric under instrumentation.

It is worth separating this from a related and much weaker complaint. The objection is not that an analyser is unreliable — it is the most reliable instrument in the room, and nothing in this section suggests distrusting what it records. The objection is that the scope of the record and the scope of the conclusion are different sizes, and only one of them is printed on the screen. The trace window shows a link. The sentence written in the ticket says "the fabric". Nothing between those two points flags the change of subject, which is why the model publishes a reach percentage rather than a warning: a number survives being pasted into a ticket, and a warning does not.

What to do when the reach is low and cannot be raised is not to distrust the trace; it is to state the reach alongside the conclusion. "Links 3 and 7 are clean" is a finding. "The fabric is clean" is not supported. A team that writes the first sentence stays able to reason about the other six links a week later; a team that writes the second has to rediscover that the other six were never examined.

6. RTL 2 — The Trigger Decides The Capture

A capture buffer is a fixed number of cycles. Where those cycles sit relative to the failure is a configuration decision made before the failure happens, and it determines whether the capture contains an explanation or a photograph of the aftermath.

Most triggers fire on the symptom, because the symptom is the thing that is easy to describe: a CRC error, a retrain, a timeout, a completion with a bad status. The cause is earlier — sometimes much earlier — and it is only in the buffer if the pre-trigger fraction was set large enough to reach back to it.

// RTL 2 - the trigger decides the capture. A buffer is a fixed number of
// cycles placed relative to one point in time, and everything outside it is
// gone whether or not it was the cause.
module trigger_placement #(parameter int THE_BUFFER_IS_THE_HISTORY = 0) (
  input  logic clk, rst_n,
  input  logic        capture,
  input  logic [15:0] buf_depth, pre_frac, cause_ahead, symptom_after,
  output logic [15:0] pre_span, post_span, kept_span, holds_cause_pct,
  output logic        trig_useful,
  output logic [7:0]  n_captures, n_missed,
  output logic        trigger_err
);
  logic [15:0] frac, needed_span;
  logic [31:0] p_q, h_q;
  logic        truly_missed;
  // A pre-trigger fraction is a percentage of the buffer, not more than it.
  // The product is taken in 32 bits: a thousand-cycle buffer at eighty
  // percent overflows sixteen and silently reports a hundred and forty-four.
  assign frac      = (pre_frac > 16'd100) ? 16'd100 : pre_frac;
  assign p_q       = ({16'd0, buf_depth} * {16'd0, frac}) / 32'd100;
  assign pre_span  = (p_q > {16'd0, buf_depth}) ? buf_depth : p_q[15:0];
  assign post_span = buf_depth - pre_span;
  assign kept_span = pre_span + post_span;
  // How much history the cause needs against how much the buffer holds.
  assign needed_span = cause_ahead + symptom_after;
  assign h_q = (needed_span == 16'd0) ? 32'd100
             : (({16'd0, kept_span} * 32'd100) / {16'd0, needed_span});
  assign holds_cause_pct = (h_q > 32'd100) ? 16'd100 : h_q[15:0];
  assign truly_missed = (cause_ahead > pre_span) && (cause_ahead != 16'd0);
  // A capture that fired is treated as a capture that holds the cause.
  assign trig_useful = (THE_BUFFER_IS_THE_HISTORY != 0)
                       ? (kept_span != 16'd0) : (cause_ahead <= pre_span);
  assign trigger_err = capture && truly_missed && trig_useful;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_captures <= 8'd0; n_missed <= 8'd0;
    end else if (capture) begin
      n_captures <= n_captures + 8'd1;
      if (truly_missed) n_missed <= n_missed + 8'd1;
    end
  end
endmodule

A thousand-cycle buffer at ten percent pre-trigger holds a hundred cycles of history and nine hundred of aftermath. With the cause five hundred cycles back, that is a full buffer that holds none of the cause — and the buffer-is-the-history view reports a successful capture, because the capture did fire and the buffer is full.

The second case is the fix and it costs nothing: the same buffer at eighty percent pre-trigger holds the cause. No more memory, no better analyser, no new trigger condition — one configuration field, set from an estimate of how far back the cause is likely to be. That estimate is a guess, and a guess is enormously better than the default.

The one-cycle case is worth driving deliberately. A cause one cycle outside the history is exactly as lost as a cause a thousand cycles outside it, and the capture looks identical. There is no partial credit and no warning; a capture that nearly reached the cause presents as a capture that did not.

The last case is the one that changes what you do rather than how you configure it. A buffer too small to hold the story at any placement — three hundred cycles of run-up and fifty of aftermath in a two-hundred-cycle buffer — holds fifty-seven percent of it wherever you put the trigger. No amount of re-triggering helps. The move is a deeper buffer, a narrower capture filter, or a trigger placed on something earlier than the symptom, and knowing which of those three is needed is the entire value of computing the ratio first.

There is a variant of this failure that deserves its own mention because the arithmetic hides it. A trigger placed on the symptom can be correct and still be the wrong trigger, when the symptom is a consequence several steps removed from the cause. A CRC error triggers reliably and is easy to describe; the marginal bit that produced it may be thousands of cycles earlier, and the decision that put the link into a marginal state earlier still. Pushing the pre-trigger fraction to its maximum buys one buffer depth of history and no more. Beyond that, the move is to trigger on something closer to the cause — a credit dropping below a threshold, a retry counter incrementing, a state transition — which requires having some hypothesis about the cause before the capture. The trigger condition is where a hypothesis enters a bench session, and a session with no hypothesis triggers on the symptom by default and learns very little.

7. RTL 3 — The Probe Changes What It Measures

The previous two sections are about evidence that is incomplete. This one is about evidence that is wrong in a specific and infuriating way: the instrument causes the failure it records.

A CXL link runs at multi-gigabit rates over a channel with a finite eye. A physical tap places a load on that channel. The load is small, it is specified, and it is subtracted from margin the link may or may not have had to spare. On a healthy link it is invisible. On a link already near its limit it is the difference between passing and failing.

The signature is unmistakable once you know it and baffling before: the link fails when the analyser is connected and passes when it is not. Half of the people who encounter this conclude the analyser is faulty. The other half conclude the failure is intermittent. Both are wrong.

// RTL 3 - the probe changes what it measures. A physical tap on a multi-
// gigabit serial link costs margin, and a link that only fails when the
// analyser is disconnected is not a link that works.
module probe_perturbation #(parameter int THE_PROBE_IS_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] margin_mv, probe_load_mv, fail_threshold_mv, runs,
  output logic [15:0] margin_left, margin_lost, margin_pct, runs_affected,
  output logic        clean_probe,
  output logic [7:0]  n_measures, n_perturbed,
  output logic        observer_err
);
  logic [15:0] true_left, load_seen;
  logic [31:0] m_q;
  logic        truly_perturbed;
  // A probe cannot take more margin than the link had.
  assign load_seen   = (probe_load_mv > margin_mv) ? margin_mv : probe_load_mv;
  assign margin_lost = (THE_PROBE_IS_FREE != 0) ? 16'd0 : load_seen;
  assign true_left   = margin_mv - load_seen;
  assign margin_left = margin_mv - margin_lost;
  assign m_q = (margin_mv == 16'd0) ? 32'd0
             : (({16'd0, margin_left} * 32'd100) / {16'd0, margin_mv});
  assign margin_pct = (m_q > 32'd100) ? 16'd100 : m_q[15:0];
  // Runs that fail only because the probe is attached.
  assign runs_affected = ((true_left < fail_threshold_mv)
                          && (margin_mv >= fail_threshold_mv)) ? runs : 16'd0;
  assign clean_probe = (margin_left >= fail_threshold_mv);
  assign truly_perturbed = (true_left < fail_threshold_mv)
                           && (margin_mv >= fail_threshold_mv);
  assign observer_err = measure && truly_perturbed && clean_probe;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_perturbed <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (truly_perturbed) n_perturbed <= n_perturbed + 8'd1;
    end
  end
endmodule

Forty millivolts of probe loading on a hundred millivolts of margin, against a seventy-millivolt failure threshold, leaves sixty millivolts — below the threshold, with all fifty runs failing because of the probe. The free-probe view reports the full hundred and a clean link.

Figure 3 — the subtraction that takes one minute and is almost never done. Both numbers are already written down: the loading is in the analyser's datasheet and the margin is in the channel characterisation. They live in different documents owned by different people, which is the entire reason the two are rarely on the same page at the moment somebody attaches a probe to a marginal link.

The fifth case is the one that keeps the model honest and is the harder judgement in practice. A link that was already below its threshold before the probe went on is not an observer effect. Fifty millivolts of margin against a seventy-millivolt threshold fails with or without the analyser, and the model explicitly declines to attribute it — the condition requires the link to have been passing without the probe. Getting this wrong in the other direction is common: an engineer who has been bitten once by probe loading starts blaming the probe for genuine failures.

The clamp says something real too. A probe cannot take more margin than the link had. Eighty millivolts of loading on a fifty-millivolt margin leaves zero, not negative thirty, and the model reports a link with nothing left rather than an impossible number.

The practical procedure this section argues for is one line long and is skipped constantly: run the failing case once with the probe attached and once without, before believing anything the probe shows. If the failure needs the probe, the probe is the subject of the investigation.

The same reasoning extends to instruments that are not physical taps, and it is easy to forget there. Enabling a trace buffer changes power draw. Turning on error logging adds transactions. Polling a register from software changes the traffic pattern the link is carrying. None of these takes millivolts, and all of them are capable of changing whether a marginal condition is reached. The general form of this section is that every instrument has a cost paid by the thing it measures, and the only question is whether that cost is large relative to the margin at stake. On a link with plenty of margin it never is — and it is exactly the links with no margin that get debugged.

8. RTL 4 — A Rare Failure Is A Schedule, Not A Difficulty

A one-in-a-million failure feels like a hard problem. It is usually not a hard problem; it is an arithmetic problem whose answer is a number of shifts, and the number is computable on the first day rather than discovered on the fifteenth.

The inputs are all known early: the observed failure rate, the rate at which the bench can run the test, how many captures are needed to be confident, and how long a shift is. The output is how long this will take, and it decides something important — whether the bench is the right place to look at all.

// RTL 4 - how long a rare failure takes to catch. A one-in-a-million event is
// not a difficult capture; it is a scheduled one, and the schedule is the
// number that decides whether the bench is the right place to look.
module reproduction_time #(parameter int IT_WILL_TURN_UP = 0) (
  input  logic clk, rst_n,
  input  logic        plan_it,
  input  logic [15:0] fail_rate_ppm, runs_per_hour, shift_hours, captures_needed,
  output logic [15:0] runs_to_fail_k, dwell_hours, total_hours, shifts_needed,
  output logic        reproducible,
  output logic [7:0]  n_plans, n_unreachable,
  output logic        repro_err
);
  logic [15:0] true_dwell;
  logic [31:0] r_q, k_q, d_q, t_q, s_q;
  logic        truly_unreachable;
  // Runs predicted per failure at this rate; a rate of zero never fails.
  assign r_q = (fail_rate_ppm == 16'd0) ? 32'hFFFFFFFF
             : (32'd1000000 / {16'd0, fail_rate_ppm});
  assign k_q = r_q / 32'd1000;
  assign runs_to_fail_k = (k_q > 32'hFFFF) ? 16'hFFFF : k_q[15:0];
  // Hours of bench time to see one, at the rate the bench can run.
  assign d_q = (runs_per_hour == 16'd0) ? 32'hFFFF
             : (r_q / {16'd0, runs_per_hour});
  assign true_dwell = (d_q > 32'hFFFF) ? 16'hFFFF : d_q[15:0];
  assign dwell_hours = (IT_WILL_TURN_UP != 0) ? 16'd0 : true_dwell;
  assign t_q = {16'd0, dwell_hours} * {16'd0, captures_needed};
  assign total_hours = (t_q > 32'hFFFF) ? 16'hFFFF : t_q[15:0];
  assign s_q = (shift_hours == 16'd0) ? 32'hFFFF
             : (({16'd0, total_hours} + {16'd0, shift_hours} - 32'd1)
                / {16'd0, shift_hours});
  assign shifts_needed = (s_q > 32'hFFFF) ? 16'hFFFF : s_q[15:0];
  assign truly_unreachable = (fail_rate_ppm != 16'd0)
                             && (true_dwell > shift_hours)
                             && (shift_hours != 16'd0);
  assign reproducible = (dwell_hours <= shift_hours);
  assign repro_err = plan_it && truly_unreachable && reproducible;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_unreachable <= 8'd0;
    end else if (plan_it) begin
      n_plans <= n_plans + 8'd1;
      if (truly_unreachable) n_unreachable <= n_unreachable + 8'd1;
    end
  end
endmodule

Ten failures per million runs at a thousand runs an hour is a hundred thousand runs per failure: a hundred hours of bench time for one capture, thirteen shifts. The it-will-turn-up view budgets zero.

The third case multiplies it. Three captures of the same failure is thirty-eight shifts, and needing three is normal — one to see it, one to confirm the trigger caught the cause, one to confirm a fix. A plan that budgets for one capture has budgeted for a third of the work.

Two degenerate inputs matter more than they look. A failure rate of zero is not a rare failure; it is no failure — a report of "it happened once, months ago, and we have no rate" produces an unbounded schedule, and the honest response is to say so rather than to schedule bench time against a number nobody has. And a bench that cannot run the test at all produces an unreachable plan from an entirely reasonable-looking failure rate, which is the case where the answer is to fix the bench before anything else.

The reason to compute this on day one is that a thirteen-shift capture changes the plan. It justifies building a cheaper reproduction, or improving the trigger so one capture suffices, or going back to simulation — which is section 12. The schedule is not the problem; discovering the schedule three weeks in is the problem.

The model publishes a shift count rather than an hour count for a reason that is organisational rather than technical. Hours are a quantity an engineer negotiates with; shifts are a quantity a schedule negotiates with. A hundred hours reads as a large number and a fortnight reads as a decision, and the decision is the thing that needs making. It is the same information, and only one form of it reliably reaches the person who can authorise building a faster reproduction.

There is one more use for this number that is easy to miss. Once a fix is in, the same arithmetic says how long the bench must run clean before the fix is confirmed. A failure at a hundred hours per occurrence is not confirmed fixed by a twenty-hour clean run — that run would very likely have been clean anyway. Teams declare victory on this basis routinely, and the failure returns from the field, where it is considerably more expensive.

9. RTL 5 — Two Instruments, Two Timebases

Silicon debug almost always needs two instruments: an analyser and a logic capture, a scope and a trace, a host-side log and a device-side counter. The moment there are two, there is a question nobody asks until it is too late: can they be ordered against each other?

Causality is ordering. The retrain happened after the CRC error is a claim about ordering, and it is the claim the whole diagnosis rests on. Two instruments with independent time references can produce beautifully detailed, entirely correct records and still be unable to support that claim — because the skew between their clocks is larger than the interval being ordered.

Nothing about this looks wrong. Both traces are right. Every sample is accurate. The ordering derived from them is a coin flip.

// RTL 5 - two instruments, two timebases. Ordering is the whole of causality,
// and a skew larger than the interval you are trying to order destroys it
// without producing a single wrong sample.
module timebase_skew #(parameter int THE_TIMESTAMPS_ARE_ALIGNED = 0) (
  input  logic clk, rst_n,
  input  logic        co_check,
  input  logic [15:0] skew_ns, span_ns, samples, ordered_pairs,
  output logic [15:0] resolvable_ns, ambiguous, ordered_ok, confidence_pct,
  output logic        causal_ok,
  output logic [7:0]  n_checks, n_ambiguous,
  output logic        timebase_err
);
  logic [15:0] true_ambiguous, pairs_seen;
  logic [31:0] c_q;
  logic        truly_ambiguous;
  // Nothing closer together than the skew can be ordered between instruments.
  assign resolvable_ns  = (skew_ns > span_ns) ? 16'd0 : (span_ns - skew_ns);
  assign pairs_seen     = (ordered_pairs > samples) ? samples : ordered_pairs;
  assign true_ambiguous = (span_ns <= skew_ns) ? pairs_seen : 16'd0;
  assign ambiguous      = (THE_TIMESTAMPS_ARE_ALIGNED != 0) ? 16'd0 : true_ambiguous;
  assign ordered_ok     = pairs_seen - ambiguous;
  assign c_q = (pairs_seen == 16'd0) ? 32'd100
             : (({16'd0, ordered_ok} * 32'd100) / {16'd0, pairs_seen});
  assign confidence_pct = (c_q > 32'd100) ? 16'd100 : c_q[15:0];
  assign causal_ok = (ambiguous == 16'd0);
  assign truly_ambiguous = (true_ambiguous != 16'd0);
  assign timebase_err = co_check && truly_ambiguous && causal_ok;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_checks <= 8'd0; n_ambiguous <= 8'd0;
    end else if (co_check) begin
      n_checks <= n_checks + 8'd1;
      if (truly_ambiguous) n_ambiguous <= n_ambiguous + 8'd1;
    end
  end
endmodule

A hundred nanoseconds of skew against fifty-nanosecond intervals makes every one of two hundred pairs ambiguous: zero percent confidence in any ordering. The aligned-timestamp view orders all two hundred at full confidence, because it takes the timestamps at face value.

The boundary case is driven deliberately, because this is precisely where a correlation quietly stops working. A skew exactly equal to the span is ambiguity, not resolution — at the boundary the two events could have occurred in either order, and a model that resolved the tie would be inventing a causal direction. The stimulus drives the equality explicitly for that reason, and the mutation campaign has a mutation that loosens it.

The last case is the configuration every correlation wants: a skew two orders of magnitude below the interval, which resolves everything and needs no thought. The distance between that and the failing case is a factor of twenty in one number, which is why this is worth checking rather than assuming — a single shared trigger, or one common reference edge captured by both instruments, converts the failing case into the working one.

Worth stating explicitly: the skew that matters is between the instruments, not within either one. An analyser with picosecond resolution and a logic capture with picosecond resolution, started from two free-running references, can be tens of microseconds apart from each other. Each instrument's internal precision is excellent and entirely irrelevant to the question, because the question is about a difference between them. This is the specific reason the failure is so hard to spot from inside the data: every quality metric either instrument reports is excellent.

The practical consequence is that this is the one bit of section 14's mask that must be checked before the capture rather than after. Every other failure in this chapter leaves some trace of itself — a short buffer, a blind link, a probe attached. An unalignable pair of instruments leaves nothing at all: two clean records and a conclusion with no support, which is exactly the shape that survives review.

10. RTL 6 — One Symptom, Three Layers

A CXL link that loses traffic produces the same complaint from above regardless of which layer lost it. The eye closed; a retry never completed; a transaction was never issued. The application sees missing data in all three cases and can distinguish none of them.

The instinct is to start at the layer the complaint arrived at, which is the transaction layer, because that is where the complaint is expressed. That is sometimes right — and the fact that it is sometimes right is exactly why the habit survives long past the point where it should have been replaced by looking at which layers are actually showing symptoms.

// RTL 6 - one symptom, three layers. A CXL link that drops traffic looks the
// same from the application whether the cause is an eye that closed, a retry
// that never completed, or a transaction that was never issued.
module layer_attribution #(parameter int THE_TOP_LAYER_IS_THE_FAULT = 0) (
  input  logic clk, rst_n,
  input  logic        diagnose,
  input  logic [15:0] phy_sym, link_sym, txn_sym, probe_cost,
  output logic [15:0] layers_lit, candidates, isolate_cost, saving_pct,
  output logic        layer_named,
  output logic [7:0]  n_diagnoses, n_unnamed,
  output logic        layer_err
);
  logic [15:0] true_lit, blind_cost;
  logic [31:0] i_q, b_q, s_q;
  logic        truly_nameable;
  assign true_lit = ((phy_sym  != 16'd0) ? 16'd1 : 16'd0)
                  + ((link_sym != 16'd0) ? 16'd1 : 16'd0)
                  + ((txn_sym  != 16'd0) ? 16'd1 : 16'd0);
  assign layers_lit = true_lit;
  // Blaming the layer the complaint arrived at names one layer every time.
  assign candidates = (THE_TOP_LAYER_IS_THE_FAULT != 0) ? 16'd1 : true_lit;
  assign i_q = {16'd0, candidates} * {16'd0, probe_cost};
  assign isolate_cost = (i_q > 32'hFFFF) ? 16'hFFFF : i_q[15:0];
  assign b_q = 32'd3 * {16'd0, probe_cost};
  assign blind_cost = (b_q > 32'hFFFF) ? 16'hFFFF : b_q[15:0];
  assign s_q = (blind_cost == 16'd0) ? 32'd0
             : ((blind_cost > isolate_cost)
                ? ((({16'd0, (blind_cost - isolate_cost)}) * 32'd100)
                   / {16'd0, blind_cost}) : 32'd0);
  assign saving_pct = (s_q > 32'd100) ? 16'd100 : s_q[15:0];
  assign layer_named = (candidates == 16'd1);
  // More than one layer showing symptoms means the layer is not yet named.
  assign truly_nameable = (true_lit == 16'd1);
  assign layer_err = diagnose && (true_lit > 16'd1) && layer_named;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_diagnoses <= 8'd0; n_unnamed <= 8'd0;
    end else if (diagnose) begin
      n_diagnoses <= n_diagnoses + 8'd1;
      if (!truly_nameable) n_unnamed <= n_unnamed + 8'd1;
    end
  end
endmodule

Three layers lit at once is three candidates and twenty-four hours of probing, with nothing saved — the diagnosis has not narrowed. The blame-the-top-layer view names one, claims sixty-six percent saved, and is choosing a layer by where the phone call came from.

The second and sixth cases are the ones that make this a real judgement rather than a rule. One layer lit is one candidate, and when that layer is the transaction layer the top-layer habit gets the right answer for the wrong reason. An engineer whose last four debugs worked that way has four data points and no method, and the fifth one costs a week.

The two-layers case is the useful middle. Two lit layers is a narrowing, not an answer — a third cheaper than probing all three, and the model reports the saving without claiming the attribution. A model that named a winner from two tied candidates would be doing exactly what the habit does.

The degenerate case is quietly the most common one on a real bench. A complaint with no layer showing anything — no physical errors, no link retries, no transaction anomalies — names nothing, and the top-layer view names one anyway. This is the state a session is in when the counters are clean and the application is unhappy, and the correct next move is to question the instrumentation rather than the layer.

The cost figure in this model is an argument for a specific and cheap piece of design work. Error counters at all three layers turn a three-candidate diagnosis into a one-candidate one for free, and the layer missing in practice is almost always the link layer — the physical layer has them because the SerDes team wanted them, the transaction layer has them because software wanted them, and the link layer sits between two teams who each assumed the other was counting. Section 22 asks about this directly for that reason.

11. RTL 7 — One Variable Per Experiment

Bench time is expensive and the temptation that follows is universal: change two things per run and halve the number of runs. It does halve the number of runs. It also removes all of the information, and the halving is what makes it feel like progress.

The arithmetic is unforgiving. A run that changes one variable and produces a different result attributes that difference to that variable. A run that changes two and produces a different result attributes it to neither. The second run is not half as informative; it is not informative.

// RTL 7 - one variable per experiment. Changing two things at once on a bench
// halves the number of runs and destroys all of the information, and the run
// that finally passes tells you nothing about which change did it.
module experiment_design #(parameter int CHANGE_THEM_TOGETHER = 0) (
  input  logic clk, rst_n,
  input  logic        plan_it,
  input  logic [15:0] variables, runs_used, run_hours, changed_per_run,
  output logic [15:0] runs_needed, info_runs, hours_spent, isolated_pct,
  output logic        one_at_a_time,
  output logic [7:0]  n_plans, n_confounded,
  output logic        confound_err
);
  logic [15:0] per_run, true_info;
  logic [31:0] h_q, p_q;
  logic        truly_confounded;
  // A bench log that does not record how many things changed is read as one
  // change per run - the most generous reading available, which is the right
  // default for a model whose job is to show that even that is not enough.
  assign per_run     = (changed_per_run == 16'd0) ? 16'd1 : changed_per_run;
  assign runs_needed = variables;
  assign true_info   = (per_run > 16'd1) ? 16'd0
                     : ((runs_used > variables) ? variables : runs_used);
  assign info_runs   = (CHANGE_THEM_TOGETHER != 0) ? runs_used : true_info;
  assign h_q = {16'd0, runs_used} * {16'd0, run_hours};
  assign hours_spent = (h_q > 32'hFFFF) ? 16'hFFFF : h_q[15:0];
  assign p_q = (variables == 16'd0) ? 32'd100
             : (({16'd0, info_runs} * 32'd100) / {16'd0, variables});
  assign isolated_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign one_at_a_time = (per_run <= 16'd1);
  assign truly_confounded = (per_run > 16'd1) && (runs_used != 16'd0)
                            && (variables != 16'd0);
  // Runs were spent and the result attributes to nothing.
  assign confound_err = plan_it && truly_confounded && (info_runs != 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_confounded <= 8'd0;
    end else if (plan_it) begin
      n_plans <= n_plans + 8'd1;
      if (truly_confounded) n_confounded <= n_confounded + 8'd1;
    end
  end
endmodule

Four variables with two changed per run over two runs is twelve hours of bench time and nothing isolated: zero percent. The change-them-together view counts both runs as useful and claims half the variables isolated.

The last case shows the scaling and it is the wrong direction. Three changes per run over two runs on six variables still isolates nothing, and the together view now claims a third of them. The more variables move per run, the more confident the wrong accounting becomes — which matches the psychology exactly, because a run that changes three things is more likely to produce a visible difference and therefore feels more productive.

Two cases keep the model from being a scold. More runs than variables is waste, not confounding — nine runs on three variables isolates all three and costs fifty-four hours, which is inefficient and interpretable. And a confounded plan that has not been run yet costs nothing, which is the last moment it is cheap to fix and the moment this model is worth reading.

The degenerate case is the one that describes most real bench logs. A log that does not record how many things changed per run is read here as one change per run — the most generous reading available — because the model's job is to show that even under the generous reading the accounting fails. Section 22 argues for recording it.

The honest counter-argument to this section is worth stating, because engineers who run good benches make it. Changing several things at once is a legitimate technique for establishing that a fix works at all, before the more careful work of finding out which part of it mattered. A kitchen-sink run that makes the failure disappear is real information: it bounds the problem to the set of things changed. What it does not do is attribute, and the mistake is not running it — the mistake is stopping there, shipping the whole bundle, and never learning which change was load-bearing. The model's accounting reflects exactly that: the run cost real hours and isolated nothing, which is a true description of a legitimate first move and an indictment of it as a last one.

12. RTL 8 — What The Bench Costs Against What Simulation Costs

This section is the argument that keeps the rest of the chapter from being needed, and it belongs at the end of a debug chapter rather than the start of a verification one because it is most persuasive to somebody who has just spent a week at a bench.

A bug findable in simulation and found on silicon is not a harder bug. It is the same bug, at a multiple of the price, discovered later and fixed under more pressure. The multiple is measurable, it is large, and it is the whole of the argument.

// RTL 8 - what the bench costs against what simulation costs. A bug findable
// in simulation and found on silicon is not a harder bug; it is the same bug
// at a multiple of the price, and the multiple is the argument.
module escape_cost #(parameter int SILICON_IS_THE_TEST = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] sim_hours, lab_hours, findable_pct, bug_count,
  output logic [15:0] findable_bugs, sim_spend, lab_spend, escape_mult,
  output logic        caught_early,
  output logic [7:0]  n_evals, n_escaped,
  output logic        escape_err
);
  logic [15:0] share, true_findable;
  logic [31:0] f_q, s_q, l_q, m_q;
  logic        truly_escaped;
  // A findable share is a percentage of the bugs, not more than all of them.
  assign share = (findable_pct > 16'd100) ? 16'd100 : findable_pct;
  assign f_q = ({16'd0, bug_count} * {16'd0, share}) / 32'd100;
  assign true_findable = (f_q > {16'd0, bug_count}) ? bug_count : f_q[15:0];
  assign findable_bugs = (SILICON_IS_THE_TEST != 0) ? 16'd0 : true_findable;
  assign s_q = {16'd0, findable_bugs} * {16'd0, sim_hours};
  assign sim_spend = (s_q > 32'hFFFF) ? 16'hFFFF : s_q[15:0];
  assign l_q = {16'd0, true_findable} * {16'd0, lab_hours};
  assign lab_spend = (l_q > 32'hFFFF) ? 16'hFFFF : l_q[15:0];
  assign m_q = (sim_hours == 16'd0) ? 32'd0
             : ({16'd0, lab_hours} / {16'd0, sim_hours});
  assign escape_mult = (m_q > 32'hFFFF) ? 16'hFFFF : m_q[15:0];
  assign caught_early = (findable_bugs >= true_findable);
  assign truly_escaped = (true_findable != 16'd0) && (lab_hours > sim_hours);
  assign escape_err = evaluate && truly_escaped && (findable_bugs == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_escaped <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_escaped) n_escaped <= n_escaped + 8'd1;
    end
  end
endmodule

Six of ten bugs findable in simulation, at two hours each against forty on the bench, is twelve hours of simulation against two hundred and forty of bench time: a twentyfold escape multiplier. The silicon-is-the-test view spends nothing in simulation and all of the two hundred and forty.

The second case is the honest limit of the argument and it has to be stated first or nobody believes the rest. A bug simulation cannot find is not an escape. The marginal channel, the cross-domain timing corner, the interaction with a device nobody modelled — these are what a bench is for, the model reports no escape for them, and any argument that pretends otherwise deserves the scepticism it gets.

The fifth case is the other limit. When the bench is as cheap as simulation the multiplier is one and there is no argument. That is rare, it happens on mature platforms with excellent automation, and where it is true the whole of this section is moot.

Between those two limits sits the case that matters, and the model's fourth stimulus makes it as strong as it goes: free simulation still leaves two hundred and forty hours on the bench if it is skipped. The multiplier is not the cost of simulation; it is the cost of not having done it.

The multiplier in this model is deliberately conservative. Twenty times covers engineering hours and nothing else: not board spins, not schedule slip, not the opportunity cost of a bench occupied for a fortnight, and not the difference in how a fix is reviewed when it is made under pressure with a tape-out date approaching. A bug fixed in simulation is fixed properly; a bug fixed at a bench at three in the morning is fixed quickly, and the two are not the same thing even when the code change is identical. Teams that have been through both know this and rarely have a number for it, which is why the conservative version is the one worth quoting: it is defensible, and it is already decisive.

13. RTL 9 — Which Instrument Answers The Question

Nine sections about what evidence is worth. This one is about the choice made before any of it: which instrument to reach for.

The instinct is to reach for the most capable tool. An analyser sees everything, so an analyser will answer the question. This is true and it is expensive — setup, insertion, a maintenance window, and section 7's probe loading — and it is frequently the answer to a question that a register read would have answered in two hours.

// RTL 9 - which instrument answers the question. Reaching for the most capable
// tool is not the same as reaching for the one that answers, and on a bench the
// difference is measured in days of setup.
module tool_choice #(parameter int REACH_FOR_THE_ANALYSER = 0) (
  input  logic clk, rst_n,
  input  logic        choose,
  input  logic [15:0] cost_a, cost_b, cost_c,
  input  logic [2:0]  answers,
  output logic [15:0] chosen_cost, cheapest_cost, wasted_cost, saving_pct,
  output logic        tool_chosen,
  output logic [7:0]  n_choices, n_wasteful,
  output logic        tool_blind_err
);
  logic [15:0] eff_a, eff_b, eff_c, min_ab;
  logic [31:0] s_q;
  logic        truly_wasteful;
  // A tool that cannot answer the question is not a candidate at any price.
  assign eff_a = answers[0] ? cost_a : 16'hFFFF;
  assign eff_b = answers[1] ? cost_b : 16'hFFFF;
  assign eff_c = answers[2] ? cost_c : 16'hFFFF;
  assign min_ab        = (eff_a < eff_b) ? eff_a : eff_b;
  assign cheapest_cost = (min_ab < eff_c) ? min_ab : eff_c;
  assign chosen_cost   = (REACH_FOR_THE_ANALYSER != 0) ? cost_c : cheapest_cost;
  assign wasted_cost = (chosen_cost > cheapest_cost)
                       ? (chosen_cost - cheapest_cost) : 16'd0;
  assign s_q = (cost_c == 16'd0) ? 32'd0
             : ((cost_c > cheapest_cost)
                ? ((({16'd0, (cost_c - cheapest_cost)}) * 32'd100)
                   / {16'd0, cost_c}) : 32'd0);
  assign saving_pct = (s_q > 32'd100) ? 16'd100 : s_q[15:0];
  assign tool_chosen = (REACH_FOR_THE_ANALYSER != 0) ? answers[2]
                                                     : (answers != 3'd0);
  assign truly_wasteful = (answers != 3'd0) && (cheapest_cost < cost_c);
  assign tool_blind_err = choose && truly_wasteful && (chosen_cost == cost_c);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_choices <= 8'd0; n_wasteful <= 8'd0;
    end else if (choose) begin
      n_choices <= n_choices + 8'd1;
      if (truly_wasteful) n_wasteful <= n_wasteful + 8'd1;
    end
  end
endmodule

A two-hour tool that answers the question, against a forty-hour analyser that also answers it, is ninety-five percent cheaper. The reach-for-the-analyser habit pays the forty and wastes thirty-eight of them.

The second and third cases are what make this a model rather than a slogan. When only the analyser can answer, the analyser is the right choice and the model says so with no saving to quote — this is not an argument against analysers. And when nothing answers the question, no tool is chosen, nothing is wasted, and the correct move is to change the question or add instrumentation rather than to capture something and hope.

The last case is the sharpest. When the cheap tool is the only one that answers, the analyser habit pays forty hours for a tool that cannot answer the question at all — the model reports that no answering tool was chosen, which is the worst outcome available: full cost, no information. That is what happens when the tool is selected by capability rather than by fit, and it is common precisely because capability is easy to compare and fit is not.

The sixth case is the happy one and worth knowing: an analyser already inserted and running is the cheapest tool there is. The cost being avoided is setup, not capability, which means the calculus changes completely on a bench where the instrumentation is permanent. Section 19 argues for making it permanent.

One caution about this model, because it is the kind that gets over-applied. Tool cost here is setup cost, not purchase cost, and the two rank differently. An analyser the team owns and has used a hundred times may be cheaper to reach for than a vendor margin register nobody has ever read, documented in an appendix, requiring a driver nobody has built. Familiarity is a real component of cost and it belongs in the estimate. What does not belong is familiarity substituting for the question of whether the tool answers — which is the seventh case, where the habit pays full price for an instrument that returns nothing.

14. RTL 10 — A Silicon-Debug Session Assembled

Nine models, nine independent claims about evidence. This one puts them in one place and makes the weak claim visible as what it is: one bit of six.

"The analyser shows it" is what a trace on a bench reports, and it is not wrong — it is one of six conditions, and the only one visible from the capture itself.

// RTL 10 - a silicon-debug session assembled. Everything that must hold before
// a capture is evidence, with "the analyser shows it" as one of the six rather
// than the whole claim.
module capture_signoff #(parameter int THE_ANALYSER_SHOWS_IT = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       link_captured,  // the trace is on the link that failed
  input  logic       trig_hit,       // the buffer holds the cause, not just the symptom
  input  logic       probe_clean,    // the probe did not take the margin
  input  logic       timebase_ok,    // the instruments can be ordered against each other
  input  logic       layer_named,    // one layer, not three, shows the symptom
  input  logic       reproducible,   // the failure returns inside a shift
  output logic       capture_sound,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_sound,
  output logic       false_evidence_err
);
  assign fail_mask[0] = ~link_captured;
  assign fail_mask[1] = ~trig_hit;
  assign fail_mask[2] = ~probe_clean;
  assign fail_mask[3] = ~timebase_ok;
  assign fail_mask[4] = ~layer_named;
  assign fail_mask[5] = ~reproducible;
  // The analyser-shows-it build is what a trace on the bench reports.
  assign capture_sound = (THE_ANALYSER_SHOWS_IT != 0)
                         ? link_captured : (fail_mask == 6'd0);
  assign false_evidence_err = evaluate && capture_sound && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_sound <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (capture_sound) n_sound <= n_sound + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When the trace is on the right link and any one of the other five fails, the assembled model reports that the capture is not evidence and the trace view reports that the analyser shows it. Only when the trace is on the wrong link do the two agree.

Across the eight evaluations the stimulus drives, the assembled model calls one session evidence and the trace view calls six of them shown. The five it gets wrong are the five single-bit failures with the capture bit set, and every one of them is a real session where a real engineer held a real trace and drew a conclusion it did not support.

The bit order here is different in character from the previous two chapters. Bits 0, 1 and 2 are properties of the capture and are checkable in minutes from the capture configuration. Bit 3 is checkable before the capture and impossible to fix afterwards. Bit 4 needs counters at three layers. Bit 5 needs a rate. The ordering is by when you can still do something about it, which puts the timebase check earlier than its cost would suggest, because a correlation with unaligned instruments cannot be repaired from the data.

Figure 4 — the mask read as a checklist, and unusually, one to run mostly before the capture rather than after. The first three decisions are settled by how the capture was configured; the fourth has to be settled before either instrument starts recording, because no amount of post-processing recovers an ordering the data never supported.

15. Quantitative Reasoning

Numbers from the models, stated so they can be argued with rather than admired.

Two probes on eight links is twenty-five percent of the fabric. With four faulty links, one is expected to be visible. A clean trace on the two instrumented links is a true statement about two links and a probability statement about the other six.

A thousand-cycle buffer at ten percent pre-trigger holds a hundred cycles of history. If the cause is five hundred cycles back, nine hundred of the thousand cycles captured are aftermath. The same buffer at eighty percent holds the whole story, at no cost.

Forty millivolts of probe on a hundred of margin against a seventy threshold fails every run. The link has thirty millivolts of margin over its threshold and the probe takes forty. The arithmetic is a single subtraction and it is almost never done.

Ten failures per million runs at a thousand runs an hour is a hundred hours. Thirteen shifts for one capture, thirty-eight for three. Both numbers are available on day one.

A hundred nanoseconds of skew cannot order fifty-nanosecond events. Not "with reduced confidence" — zero percent, on two hundred pairs, from two instruments that are each individually correct.

Three lit layers is three candidates. The top-layer habit names one and claims a sixty-six percent saving it does not deliver. Two lit layers is a genuine thirty-three percent narrowing and still not an answer.

Two variables per run over two runs isolates nothing. Twelve hours of bench time, zero percent attributed. Three per run over two runs on six variables also isolates nothing, and the wrong accounting claims more.

Six findable bugs at two hours of simulation against forty of bench time is a twentyfold multiplier. Twelve hours against two hundred and forty. Even at zero simulation cost the escape is two hundred and forty hours.

A two-hour tool against a forty-hour analyser is ninety-five percent. And when the cheap tool is the only one that answers, the analyser costs forty hours and returns nothing.

One of eight sessions is evidence; a trace calls six of them shown. The assembled model's summary number, and the chapter's.

16. Assertions

The testbenches carry 523 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported twenty on the first run. Two were real gaps, both in the first testbench: the probe model's margin percentage, which reads a hundred in the free-probe build and sixty in the measured one, and the reproduction model's total hours. The second of those turned out to be a modelling defect rather than a missing assertion — the optimistic build reported zero dwell and then computed a full total from the true dwell, so it was wrong inconsistently. A build that is wrong inconsistently cannot be mutated meaningfully, because a mutation to either half produces a mixture that no assertion pins down. It was corrected before the campaign.

Both builds are asserted on every degenerate case. An empty fabric, a capture with no buffer, a link with no margin, a failure rate of zero, a bench that cannot run, two instruments with nothing to correlate, a complaint with no symptom, an experiment with no variables, a design with no bugs, a question no tool answers.

Every clamp is driven past its limit exactly once. A probe list longer than the fabric, a pre-trigger fraction above a hundred percent, a probe load larger than the margin, a findable share above a hundred percent, more ordered pairs than samples, a bench time that saturates, a capture count that saturates the schedule.

Every error output is checked in both directions in every case. Section 7's fifth case and section 12's fifth case exist entirely to assert the quiet half — a link already failing before the probe, and a bench as cheap as simulation. In both, the measured build must stay silent, and a model that alarmed on them would be one an engineer learns to ignore.

A 16-bit multiply overflowed in the trigger model and the assertions caught it on the first run. A thousand-cycle buffer at eighty percent pre-trigger computed a hundred and forty-four cycles of history instead of eight hundred, because the product was taken in sixteen bits before the division. Ten checks failed at once. The fix is a 32-bit intermediate, and the reason it is worth recording is that the mutation campaign would not have caught it: every mutation to that line would have produced a differently-wrong number, and all of them would have been killed by assertions that were already failing.

17. Mutation Testing

115 mutations, 115 killed. Fifty-four against the first testbench, sixty-one against the second.

One mutation survived the first run, and it is the most useful result in the chapter. Inverting the ambiguity counter in the timebase model changed nothing, because the stimulus happened to split six correlations three-and-three — the inverted counter produced an identical total. That is an equivalent mutant created by the stimulus rather than by the code, and the same coincidence had already appeared twice in 26.6, where two counter inversions had to be rewritten as double-steps.

The fix chosen here is different and better. Rather than weakening the mutation to a double-step, a seventh correlation was added to the stimulus — a small skew against a long interval, which is the configuration every correlation should have — and the split became three-and-four. The mutation was killed on the next run and the testbench gained a case it should have had anyway. When a counter mutation survives because the totals coincide, the stimulus is unbalanced; fix the stimulus, not the mutation.

The rule generalises and the general form is checkable without running anything: a counter inversion is undistinguishable exactly when the interesting cases number half the total, because the inverted counter then counts the other half and reaches the same number. That is a comparison between two numbers the testbench already asserts, so it can be read straight out of the source. Applying it across this batch found three more counters — two in 26.5 and one in 26.6 — where the inversion had been quietly replaced by a double-step during authoring rather than fixed. A mutation that was weakened to make it pass is a testbench gap wearing a passing score, and all three were repaired the same way: one more case, then the real mutation.

Three mutations were designed and discarded as equivalent before injection, all of the loosened-comparison shape, in models where no stimulus produced the equality the mutation depended on.

The zero-guard family is now the second-largest in the batch at thirteen, up from nine in 26.5. That growth is not accidental: every one of those guards exists because a real instrument supplies the degenerate value — a failure rate nobody measured, a bench that cannot run, a fabric with no links, a log that does not record what changed — and each flipped guard produces a model that answers confidently from an input containing nothing.

18. Verification Strategy

A verification plan for a bench is unusual because the thing being verified is the evidence rather than the design, and most of it happens before the first capture.

Compute capture reach before the first trace. The fraction of the fabric under instrumentation bounds every conclusion drawn from a clean trace, and it is one division.

Set the pre-trigger fraction from an estimate of how far back the cause is. Any estimate beats the default. If the estimate is unknown, capture once at fifty percent and measure the distance from the cause to the symptom, then set it properly for the captures that matter.

Run the failing case with and without the probe, first. One extra run, and it eliminates the single most confusing failure mode a bench produces.

Compute the reproduction schedule on day one. If it is more than a shift, the plan changes before the time is spent rather than after.

Align the instruments before capturing, not after. A shared trigger or a common reference edge costs minutes beforehand and is impossible afterwards. This is the only item on the list that cannot be repaired from the data.

Read counters at all three layers before choosing a layer. The number of lit layers is the diagnosis's current state, and it is cheaper than any probe.

Change one thing per run and record what changed. Both halves matter: the discipline is worthless if the log does not support reconstructing it afterwards.

Ask what simulation could have produced before booking bench time. Not as a reproach — as a routing decision, with section 12's multiplier as the reason.

19. Synthesis and Implementation Reality

The models are teaching models, but the bench they describe is built from decisions made years earlier.

Analyser insertion points are a board-design decision. A link with no interposer footprint cannot be probed without rework, which moves section 5's reach from a configuration choice to a hardware one. The footprints are cheap at layout and unobtainable afterwards.

Permanent instrumentation changes section 13's arithmetic completely. An analyser already inserted and running costs nothing to reach for, which makes it the cheapest tool rather than the most expensive one. On platforms that debug continuously, leaving the instrumentation in is usually correct.

A shared trigger output is a few pins and resolves section 9 entirely. One instrument triggering the other, or both triggering from a common source, converts every correlation from ambiguous to exact. This is the highest-value, lowest-cost item in the chapter.

Capture buffer depth is bought, and pre-trigger placement is free. Teams routinely spend on the first and never configure the second. Section 6's failing case and its fix use the same buffer.

Per-layer error counters are the substitute for probing. Section 10's attribution needs a count at the physical, link and transaction layers. These are small, they are almost always present at one layer and almost never at all three, and the missing one is usually the link layer.

Probe loading is specified and is usually ignored. The number is in the analyser's datasheet and the link's margin is in the characterisation report. Comparing them is a subtraction that takes one minute and is worth a week.

20. Silicon Observability

What can be read from real silicon, ordered by what it costs to get — and the ordering is the chapter's own recommendation.

Free, already there. Link status, trained width and speed, error counters at whatever layers implement them. These answer section 10's attribution when all three layers are instrumented, and they cost nothing.

Cheap. Vendor margin and eye-measurement registers, where they exist. These answer section 7 without attaching anything, which is strictly better than measuring with a probe whose loading is the question.

Moderate. A logic analyser on side-band signals, or an embedded trace buffer if the silicon has one. An embedded buffer is exempt from section 7 entirely — nothing is attached — and is bounded by section 6, because it is usually shallow.

Expensive. An inserted protocol analyser. Complete truth about one link, subject to probe loading, requiring a maintenance window. It is the best instrument in the building and it should be the last one reached for, which is section 13.

Unobtainable. Simultaneous cycle-accurate visibility of every link in a fabric. Nothing provides it, which is why section 5's reach calculation is a permanent constraint rather than a budget problem.

21. Debug Lab

A CXL link in a five-link fabric fails once every few hours. There is an analyser available and a maintenance window tonight.

Step 1 — compute the reproduction schedule. Section 8. A failure every few hours against the rate the bench can run decides whether tonight's window is enough. If it is thirteen shifts, tonight is not a capture session; it is a session for building a faster reproduction.

Step 2 — decide which link to probe, and compute what the trace will cover. Section 5. One of five is twenty percent. If the failing link is not known, the trace is a bet, and the size of the bet should be stated before it is placed.

Step 3 — read the error counters at all three layers first. Section 10. This is free, it takes minutes, and if exactly one layer is lit it may make the capture unnecessary.

Step 4 — check the probe loading against the link margin. Section 7. One subtraction, from two numbers already written down. If the margin is thinner than the loading, the capture will produce a failure caused by the capture.

Step 5 — set the pre-trigger fraction from where the cause is likely to be. Section 6. The default is wrong for almost every real failure, and it is one field.

Step 6 — if a second instrument is involved, align them before starting. Section 9. A shared trigger, or a common reference edge in both captures. Afterwards is too late.

Step 7 — capture. Then change one thing. Section 11. The temptation at three in the morning with a window closing is to change two, and that is the moment the whole night's data stops attributing.

Step 8 — before booking the next window, ask what simulation could have produced. Section 12. The multiplier is twenty, and the question is a routing decision rather than a criticism.

The order is by what is cheap and irreversible. Steps 1 through 6 all happen before the capture, and every one of them is impossible to apply afterwards.

22. Design Review

Questions worth asking long before a bench exists.

Which links have analyser insertion points? Section 5's reach is bounded by this and it is a layout decision.

Is there a shared trigger path between instruments? A few pins, and section 9 becomes a non-issue permanently.

How deep is the capture buffer, and is the pre-trigger fraction configurable? Depth is bought; placement is free and is the one that is usually wrong.

Are there error counters at the physical, link and transaction layers? All three, or section 10's attribution cannot be done without probing.

Is there an embedded trace buffer? It is shallow and it is exempt from probe loading, which is a combination nothing else offers.

Are margin or eye-measurement registers exposed? They answer section 7 without attaching anything.

Does the bench log record how many variables changed per run? Section 11's discipline is unenforceable and unreconstructable without it.

What is the expected escape rate from simulation, and has anybody computed the multiplier? Section 12's argument needs a number to be made at all.

23. How This Appears In Real Engineering

The session starts at eight in the evening with a maintenance window until six. The analyser goes on the link that failed last time. The capture fires at two in the morning.

At three, the trace is open and the failure is in it — the symptom, cleanly, unambiguously. The cause is not, because the buffer was at its default pre-trigger fraction and the cause was earlier than anybody guessed. The window is closing, the failure is now understood to be real and is no better understood than it was at eight, and the next window is a week away.

The second recurring shape is the probe. The link fails reliably with the analyser attached and passes without it. Two hours go into deciding whether the analyser is broken. Nobody subtracts the loading from the margin, because the loading is in one document and the margin is in another and no procedure has ever put them on the same page.

The third is the one that damages a team rather than a night. Two instruments, two traces, an ordering derived from their timestamps, a diagnosis built on that ordering, and a fix that does not work. Reconstructing why takes days, and the answer is that the ordering was never supported by the data. The traces were correct. The conclusion was not. Nothing in either trace indicated a problem, which is what makes this the most expensive of the three and the cheapest to prevent.

The pattern underneath all three is the same and it is the reason this chapter is last in the module. The instrument is trusted more than the reasoning about the instrument, because the instrument produces something detailed and concrete while the reasoning produces a caveat. Every one of the six bits is a caveat, and every one of them is cheaper to check than the session it saves.

24. Common Misconceptions

"The analyser shows it." It shows one link, for the cycles the trigger placed, through a probe that changed the link. Six conditions; this is the chapter.

"The trace is clean, so the problem is elsewhere." The trace is clean on the link it is on. What fraction of the fabric is that? Section 5.

"The capture fired, so we have the failure." You have the symptom. The cause is in the buffer only if the pre-trigger fraction reached back to it. Section 6.

"The analyser must be faulty — it fails only when connected." The analyser is working. It is taking margin the link did not have to spare. Section 7.

"It's intermittent, we'll catch it eventually." It is a rate. Divide, and you get a number of shifts. Section 8.

"Both traces are timestamped, so we can order them." Only if the skew is smaller than the interval. Section 9.

"The application sees it, so it's a transaction-layer problem." The application sees every layer's failures identically. Count lit layers first. Section 10.

"We're short on bench time, so let's change two things per run." Then the run attributes to nothing, and the time is spent anyway. Section 11.

"We'll catch it in bring-up." At twenty times the cost, for the bugs simulation could have found. Section 12.

"Get the analyser on it." Only if a cheaper tool cannot answer the question — and check whether the analyser can answer it at all. Section 13.

25. Interview Reasoning

"You have one protocol analyser and a five-link CXL fabric with an intermittent failure. How do you decide where to put it?" By computing the reach and stating the bet. If the failing link is not known, one of five is twenty percent, and the alternative is to read error counters on all five first — free, and it may identify the link. The reasoning being tested is whether you quantify the coverage of your evidence before collecting it.

"Your capture fired and contains the failure, but you cannot explain it. What do you check first?" The pre-trigger fraction against how far back the cause is likely to be. A buffer full of aftermath is the most common wasted capture there is, and the fix costs one configuration field rather than a new window.

"A CXL link fails only when the protocol analyser is attached. What is happening?" Probe loading against a thin margin. The link is marginal and the probe takes the margin. The follow-up is how to confirm it without the probe, and the answer is vendor margin registers, which measure the eye without attaching anything.

"Two instruments, two traces, and a diagnosis that depends on which event came first. What do you need to check?" Skew against the interval being ordered. If the skew is larger, no ordering is supported regardless of how precise each trace is. A strong answer notes that this cannot be fixed after the capture and needs a shared trigger beforehand.

"A failure occurs once per million runs. Is the bench the right place to look?" Compute the schedule first: runs per failure, divided by runs per hour, times captures needed. If it is more than a couple of shifts, the answer is to build a faster reproduction or return to simulation. The point being tested is that rarity is arithmetic, not difficulty.

"You have four hours of bench time left and four hypotheses. What do you do?" One variable per run, and accept that four hours may only cover two of them. Changing two per run covers all four and attributes none. The interesting part of the answer is the willingness to leave hypotheses untested rather than test them uninterpretably.

26. Exercises

1. A fabric has 12 links and 3 analyser insertion points. Six links are suspected. Compute the reach and the expected number of suspected links visible. What reach would be needed for even odds on any one fault?

2. A 4,000-cycle buffer is set to 25 percent pre-trigger. The cause precedes the symptom by 1,500 cycles. Is it captured? What is the minimum pre-trigger fraction that captures it, and what is the minimum buffer depth at 50 percent?

3. A link has 120 mV of margin and fails below 85 mV. An analyser loads 30 mV; an interposer loads 45 mV. Which instruments can be used? What if the margin is 100 mV?

4. A failure occurs 25 times per million runs. The bench runs 400 per hour. Three captures are needed and a shift is 10 hours. Compute the shifts. How much faster must the bench be to fit inside two shifts?

5. Two instruments have 40 ns of skew. What is the shortest interval whose ordering they support? If the events of interest are 25 ns apart, what has to change?

6. All three layers are lit, and probing a layer costs 6 hours. Compute the isolation cost and the saving over probing all three. Now suppose link-layer counters are missing — what does that do to the attribution?

7. Six variables, 3-hour runs, and 12 hours of bench time. How many can be isolated one at a time? If two are changed per run, how many are isolated, and how many hours are spent?

8. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify its position using the rule that the ordering is by when you can still do something about it.

27. Summary

A trace is evidence about one link. Everything beyond that is an inference whose strength is the fraction of the fabric under instrumentation.

The trigger decides what the capture holds. A full buffer of aftermath is the most common wasted capture there is, and the fix is one configuration field.

The probe changes what it measures. A link that fails only while it is being watched is a link whose margin was thinner than the probe.

A rare failure is a schedule. Thirteen shifts for one capture, thirty-eight for three, and both numbers are computable on day one.

Two instruments cannot order events closer together than their skew — and this is the only failure in the chapter that cannot be repaired after the capture.

One symptom, three layers. The habit of blaming the layer the complaint arrived at is right often enough to survive and wrong expensively.

One variable per run. Two changes halve the runs and remove all of the information, and the accounting that says otherwise gets more confident as it gets more wrong.

A bug findable in simulation and found on a bench costs twenty times more. Even free simulation leaves two hundred and forty hours on the table if it is skipped.

The most capable instrument is not the cheapest answer — and sometimes it is not an answer at all.

Six bits, and "the analyser shows it" is one of them. One session of eight is evidence; a trace calls six of them shown.

That closes module 26. Seven chapters, seven families of CXL failure, and the same shape in every one: a confident single-bit claim, and five other bits that had to hold for it to mean anything.