Coherency Bugs

26.2 was about a link carrying fewer bits than it should. This chapter is about a link carrying the right bits and delivering the wrong data — and the instrument everybody reaches for is a capture that shows one observer's view of an ordering that is a property of all of them.

The trace looks correct. It was taken at one point on a fabric with six agents on it, and eighteen of the thirty claims anybody wants to make from it are about agents that were not in the capture.

1. The Engineering Problem — One Observation, One Observer

A staleness window bounds where the bug can live. A read nine hundred nanoseconds after a write, on a three-hundred-nanosecond snoop path, rules out two causes of eight — and the interval is in the trace already. Section 5.

A snoop never sent and a snoop ignored are different bugs in different blocks. Twelve never sent and eight sent-unanswered is twelve on the host side and eight on the device side — and without the trace all twenty are attributed to whoever sent them. Section 6.

The precursor happens long before the symptom. Forty snoop-filter entries evicted while caches still held the lines is twelve hundred reads at risk, created at the eviction and observed at a read much later. Section 7.

"Before" is a property of an observer. Twelve of forty event pairs sit inside the observers' skew, and no trace can order them — while a single-global-order reading orders all forty. Section 8.

This chapter against 26.2, stated precisely. That one owns a link that delivers less. This one owns a link that delivers wrong — which is why every model here is about what an observation supports rather than what it shows, and why section 14's weak definition is a trace that looks correct.

2. The One-Sentence Model

A coherency bug is diagnosed when the trace shows no violation, sent-and-ignored is separated from never-sent, no filter entry was evicted while a cache held it, the event pairs are outside the observer skew, every agent is in the capture, and the blast radius is counted — and "the trace looks correct" is one of those six.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model takes one observation from a coherency failure and computes what it supports. A real investigation has a protocol analyser at one or two points, the device's own counters, a host's error log and a workload that may or may not reproduce it, and none of that is reproduced. What is reproduced is the arithmetic of attribution — which causes an interval rules out, which side an unanswered snoop belongs to, how many lines a condition touched.

Three simplifications are worth stating. Section 5 treats causes as a flat count where a real cause set is structured. Section 8 models observer disagreement as a single skew figure. Section 13 treats the condition rate as uniform across lines. In each case the conclusion is the same and the model is abbreviated.

Each model is built twice — a correct build and a broken build selected by a parameter. Every broken build here is what a single capture supports: staleness is staleness, no response means no snoop, an eviction is harmless, there is one global order, an eviction completes, bias is a hint, one reproduction is enough, one probe is enough, one bad line is one bug. Each is a reasonable reading of the evidence in hand, and each is a claim about evidence that was not gathered.

Figure 1 — The trace on top is accurate about everything it captured. Eighteen of thirty claims are about agents it did not capture, and nothing in the trace says which eighteen.

5. RTL 1 — The Staleness Window Bounds Where The Bug Can Live

// RTL 1 - the staleness window. A read that returns old data was issued some
// time after the write, and that interval bounds where the bug can live.
module stale_window #(parameter int ANY_STALENESS_IS_THE_SAME = 0) (
  input  logic clk, rst_n,
  input  logic        observe,
  input  logic [15:0] write_to_read_ns, snoop_latency_ns, writeback_ns,
  input  logic [15:0] candidates_all,
  output logic [15:0] window_ns, candidates, ruled_out, narrowed_pct,
  output logic        narrowed,
  output logic [7:0]  n_observations, n_wide,
  output logic        window_unused_err
);
  logic [31:0] p_q;
  assign window_ns = write_to_read_ns;
  // A gap longer than the snoop path rules out the snoop being merely late.
  assign ruled_out = (ANY_STALENESS_IS_THE_SAME != 0) ? 16'd0
                   : ((write_to_read_ns > snoop_latency_ns)
                      ? ((write_to_read_ns > writeback_ns) ? 16'd2 : 16'd1)
                      : 16'd0);
  assign candidates = (candidates_all > ruled_out)
                      ? (candidates_all - ruled_out) : 16'd0;
  assign p_q = (candidates_all == 16'd0) ? 32'd0
             : (({16'd0, ruled_out} * 32'd100) / {16'd0, candidates_all});
  assign narrowed_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign narrowed = (ruled_out != 16'd0);
  // A staleness window wide enough to rule something out, and nothing ruled.
  assign window_unused_err = observe && (write_to_read_ns > snoop_latency_ns)
                             && !narrowed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_observations <= 8'd0; n_wide <= 8'd0;
    end else if (observe) begin
      n_observations <= n_observations + 8'd1;
      if (!narrowed) n_wide <= n_wide + 8'd1;
    end
  end
endmodule

Five observations. A 300 ns snoop path, a 600 ns writeback, eight candidates.

Two observations that narrow nothing; all five when the interval is ignored.

The interval between the write and the stale read is in the trace and is almost never used. A read nine hundred nanoseconds after the write is not a snoop still in flight on a three-hundred-nanosecond path, and it is not a writeback still in progress on a six-hundred-nanosecond one. Two causes gone, for the price of subtracting two timestamps.

Rows four and five are the boundary, one nanosecond apart. At exactly the snoop latency nothing can be ruled out — the snoop could be arriving as the read issues — and one nanosecond later it could not. The threshold is the mechanism's own latency, which is a number the design has.

Row two is the case with no leverage and it is the common one. A two-hundred-nanosecond gap is inside every mechanism's window, so every cause remains live and the model says so. An investigation starting here needs a different observation, not a more careful reading of this one.

6. RTL 2 — Never Sent And Ignored Are Different Bugs

// RTL 2 - not sent against ignored. A snoop that was never issued and one that
// was issued and not acted on are different bugs in different blocks.
module snoop_fate #(parameter int NO_DATA_IS_NO_SNOOP = 0) (
  input  logic clk, rst_n,
  input  logic        classify,
  input  logic [15:0] writes, snoops_seen, responses_seen, stale_reads,
  output logic [15:0] never_sent, sent_unanswered, answered, host_side, device_side,
  output logic        fate_known,
  output logic [7:0]  n_classified, n_unknown,
  output logic        fate_guessed_err
);
  assign never_sent = (writes > snoops_seen) ? (writes - snoops_seen) : 16'd0;
  assign answered = (responses_seen > snoops_seen) ? snoops_seen : responses_seen;
  assign sent_unanswered = (snoops_seen > answered) ? (snoops_seen - answered) : 16'd0;
  // A snoop never sent is the sender's bug; one sent and unanswered is the
  // receiver's. Without the trace, both get attributed to one side.
  assign host_side = (NO_DATA_IS_NO_SNOOP != 0) ? (never_sent + sent_unanswered)
                   : never_sent;
  assign device_side = (NO_DATA_IS_NO_SNOOP != 0) ? 16'd0 : sent_unanswered;
  assign fate_known = (NO_DATA_IS_NO_SNOOP == 0);
  // Unanswered snoops attributed to the side that sent them.
  assign fate_guessed_err = classify && (sent_unanswered != 16'd0)
                            && (device_side == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_classified <= 8'd0; n_unknown <= 8'd0;
    end else if (classify) begin
      n_classified <= n_classified + 8'd1;
      if (!fate_known) n_unknown <= n_unknown + 8'd1;
    end
  end
endmodule

Five classifications. Sixty writes.

The trace always knows the fate; the no-data view never does.

A snoop that was never issued is the sending agent's bug, and a snoop that was issued and not acted on is the receiving agent's. Twelve and eight here — two different blocks, two different owners, two different fixes — and a symptom-side investigation attributes all twenty to whichever side is easier to instrument.

Row one is what the trace buys. The distinction requires seeing the snoop on the wire, which is one capture point and no analysis at all. Without it the classification is a guess, and the model reports the fate as unknown rather than guessing.

Row three is the smallest instance. One unanswered snoop of sixty is enough to place a bug on the device side — and it is the observation that a response-counting check, which sees fifty-nine of sixty, reports as a rounding error.

7. RTL 3 — The Precursor Is An Eviction, Not A Read

// RTL 3 - the precursor. A snoop filter that evicts an entry while a cache
// still holds the line creates the fault long before the stale read.
module filter_eviction #(parameter int EVICTION_IS_HARMLESS = 0) (
  input  logic clk, rst_n,
  input  logic        audit,
  input  logic [15:0] evictions, still_held, lines_at_risk, reads_per_line,
  output logic [15:0] unsafe_evictions, exposed_lines, exposed_reads, safe_pct,
  output logic        evictions_safe,
  output logic [7:0]  n_audits, n_unsafe,
  output logic        precursor_ignored_err
);
  logic [31:0] p_q, r_q;
  assign unsafe_evictions = (EVICTION_IS_HARMLESS != 0) ? 16'd0
                          : ((still_held > evictions) ? evictions : still_held);
  assign exposed_lines = (unsafe_evictions > lines_at_risk) ? lines_at_risk
                       : unsafe_evictions;
  assign r_q = {16'd0, exposed_lines} * {16'd0, reads_per_line};
  assign exposed_reads = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  assign p_q = (evictions == 16'd0) ? 32'd100
             : ((({16'd0, evictions} - {16'd0, unsafe_evictions}) * 32'd100)
                / {16'd0, evictions});
  assign safe_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign evictions_safe = (unsafe_evictions == 16'd0);
  // Evictions of lines a cache still holds, reported as safe.
  assign precursor_ignored_err = audit && (still_held != 16'd0)
                                 && (evictions != 16'd0) && evictions_safe;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_audits <= 8'd0; n_unsafe <= 8'd0;
    end else if (audit) begin
      n_audits <= n_audits + 8'd1;
      if (!evictions_safe) n_unsafe <= n_unsafe + 8'd1;
    end
  end
endmodule

Six audits. Two hundred evictions, a hundred lines at risk, thirty reads each.

Three audits with an unsafe eviction; none when evictions are assumed harmless.

A snoop filter that drops an entry while a cache still holds the line creates the fault at the moment of the eviction, and the symptom arrives at whatever read happens next — which may be a minute later and a thousand transactions away. The two events are not adjacent in any trace.

Row one is why this is the counter to build. Forty unsafe evictions is twelve hundred reads at risk, and every one of those reads is a potential stale value with no visible cause near it. 25.2 §20 named this the only precursor counter; this is the arithmetic behind that claim.

Row six is the exemption that keeps the check honest. Forty lines still held and no eviction at all is not a defect — the model reports it safe and raises nothing, because the fault is the coincidence of the two, not either alone.

8. RTL 4 — "Before" Is A Property Of An Observer

// RTL 4 - whose ordering. "Before" is a property of an observer, and two
// agents on one fabric do not share one.
module ordering_point #(parameter int ONE_GLOBAL_ORDER = 0) (
  input  logic clk, rst_n,
  input  logic        judge,
  input  logic [15:0] pairs, agreed_by_both, observer_skew_ns, gap_ns,
  output logic [15:0] orderable, ambiguous, agreement_pct, resolvable,
  output logic        order_decidable,
  output logic [7:0]  n_judgements, n_ambiguous,
  output logic        false_order_err
);
  logic [31:0] p_q;
  // Two events closer together than the observers' skew cannot be ordered
  // from a trace at all.
  assign ambiguous = (ONE_GLOBAL_ORDER != 0) ? 16'd0
                   : ((gap_ns < observer_skew_ns)
                      ? ((pairs > agreed_by_both) ? (pairs - agreed_by_both) : 16'd0)
                      : 16'd0);
  assign orderable = (pairs > ambiguous) ? (pairs - ambiguous) : 16'd0;
  assign resolvable = (gap_ns >= observer_skew_ns) ? pairs : agreed_by_both;
  assign p_q = (pairs == 16'd0) ? 32'd100
             : (({16'd0, orderable} * 32'd100) / {16'd0, pairs});
  assign agreement_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign order_decidable = (ambiguous == 16'd0);
  // Event pairs inside the skew, reported as ordered.
  assign false_order_err = judge && (gap_ns < observer_skew_ns)
                           && (pairs > agreed_by_both) && order_decidable;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_judgements <= 8'd0; n_ambiguous <= 8'd0;
    end else if (judge) begin
      n_judgements <= n_judgements + 8'd1;
      if (!order_decidable) n_ambiguous <= n_ambiguous + 8'd1;
    end
  end
endmodule

Six judgements. Forty pairs, twenty-eight agreed by both observers.

Two judgements undecidable; none under a single global order.

A coherency claim is about the order events happened in, and a trace shows the order they arrived at one probe in. Two events twenty nanoseconds apart on a fabric with fifty nanoseconds of observer skew cannot be ordered from any single capture — and a reading that orders them anyway is inventing the fact it needs.

Rows three and four are the boundary and it is the skew. Exactly at the skew the pairs still resolve; one nanosecond inside and twelve become undecidable. The threshold is a property of the instrumentation, which means the fix is an instrumentation change rather than an analysis one.

Row five is the case that makes the model honest. When both observers agree on the order, the pairs are decidable even inside the skew — the skew bounds what a disagreement can be resolved from, not what agreement can establish.

9. RTL 5 — A Dropped Writeback Is Data Loss Wearing A Stale Read

// RTL 5 - a writeback that went nowhere. A dirty line dropped on eviction is
// data loss that looks exactly like a stale read much later.
module dropped_writeback #(parameter int EVICT_IS_COMPLETE = 0) (
  input  logic clk, rst_n,
  input  logic        audit,
  input  logic [15:0] dirty_evictions, writebacks_seen, bytes_per_line,
  input  logic [15:0] reads_after,
  output logic [15:0] dropped, bytes_lost, stale_reads_expected, delivered_pct,
  output logic        writebacks_complete,
  output logic [7:0]  n_audits, n_lossy,
  output logic        silent_loss_err
);
  logic [31:0] b_q, p_q;
  assign dropped = (EVICT_IS_COMPLETE != 0) ? 16'd0
                 : ((dirty_evictions > writebacks_seen)
                    ? (dirty_evictions - writebacks_seen) : 16'd0);
  assign b_q = {16'd0, dropped} * {16'd0, bytes_per_line};
  assign bytes_lost = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  assign stale_reads_expected = (dropped != 16'd0) ? reads_after : 16'd0;
  assign p_q = (dirty_evictions == 16'd0) ? 32'd100
             : (({16'd0, writebacks_seen} * 32'd100) / {16'd0, dirty_evictions});
  assign delivered_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign writebacks_complete = (dropped == 16'd0);
  // Dirty evictions with no writeback, reported as complete.
  assign silent_loss_err = audit && (dirty_evictions > writebacks_seen)
                           && writebacks_complete;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_audits <= 8'd0; n_lossy <= 8'd0;
    end else if (audit) begin
      n_audits <= n_audits + 8'd1;
      if (!writebacks_complete) n_lossy <= n_lossy + 8'd1;
    end
  end
endmodule

Five audits. Ninety dirty evictions, 64-byte lines, two hundred later reads.

Two audits lossy; none when eviction is assumed complete.

A dirty line evicted with no writeback on the wire is data that no longer exists anywhere. It is not a coherency violation at the moment it happens — nothing is inconsistent yet — and it becomes one at every subsequent read of that line, two hundred of them here.

Row one is the shape of the confusion. Three hundred and eighty-four bytes lost and two hundred stale reads later, and the reads are what gets investigated because they are what somebody noticed. The eviction is six transactions in a trace nobody kept.

Row three is the detector's resolution. One dropped writeback of ninety is sixty-four bytes, which is exactly one cache line of wrong data and is the smallest failure this mechanism can produce.

10. RTL 6 — A Device-Bias Line Read Without A Flush

// RTL 6 - bias confusion. A line in device bias read by the host without a
// flush returns whatever memory holds, which is not what the device wrote.
module bias_confusion #(parameter int BIAS_IS_A_HINT = 0) (
  input  logic clk, rst_n,
  input  logic        classify,
  input  logic [15:0] host_reads, device_bias_lines, flushed_lines, versions_behind,
  output logic [15:0] unsafe_reads, safe_reads, staleness, safe_pct,
  output logic        reads_coherent,
  output logic [7:0]  n_classified, n_unsafe,
  output logic        bias_ignored_err
);
  logic [31:0] p_q;
  // A device-bias line the host reads without a flush is not coherent.
  assign unsafe_reads = (BIAS_IS_A_HINT != 0) ? 16'd0
                      : ((device_bias_lines > flushed_lines)
                         ? (device_bias_lines - flushed_lines) : 16'd0);
  assign safe_reads = (host_reads > unsafe_reads)
                      ? (host_reads - unsafe_reads) : 16'd0;
  assign staleness = (unsafe_reads != 16'd0) ? versions_behind : 16'd0;
  assign p_q = (host_reads == 16'd0) ? 32'd100
             : (({16'd0, safe_reads} * 32'd100) / {16'd0, host_reads});
  assign safe_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign reads_coherent = (unsafe_reads == 16'd0);
  // Device-bias lines read without a flush, reported as coherent.
  assign bias_ignored_err = classify && (device_bias_lines > flushed_lines)
                            && reads_coherent;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_classified <= 8'd0; n_unsafe <= 8'd0;
    end else if (classify) begin
      n_classified <= n_classified + 8'd1;
      if (!reads_coherent) n_unsafe <= n_unsafe + 8'd1;
    end
  end
endmodule

Five classifications. Five hundred host reads, three versions behind.

Two classifications with an unsafe read; none when bias is treated as a hint.

Device bias means the device may write the line without telling the host, and a host read of such a line without a flush returns whatever memory holds. Forty reads here, three versions behind — and the mechanism is working exactly as specified. The bug is the missing flush, not the bias.

Row one is why it is hard to see in a trace. The reads complete normally, return data, and are indistinguishable from correct reads except by knowing the line's bias state at that instant — which is device-internal and is in the trace only if the device exports it.

Row three is the smallest instance, and it is worth stating because bias transitions happen in batches: one line of a hundred and twenty missed by a flush loop is an off-by-one in the flush, not a coherency bug in the protocol.

11. RTL 7 — A Race Needs A Rate Before A Campaign

// RTL 7 - reproducing it. A coherency race that needs two operations to
// overlap appears at a rate, and a campaign has to be sized against that rate.
module race_repro #(parameter int IT_REPRODUCED_ONCE = 0) (
  input  logic clk, rst_n,
  input  logic        plan_it,
  input  logic [15:0] hits_per_million, iterations_k, minutes_per_k,
  output logic [15:0] expected_hits, iters_for_four_k, hours_needed, confidence_pct,
  output logic        campaign_sized,
  output logic [7:0]  n_plans, n_undersized,
  output logic        undersized_err
);
  logic [31:0] e_q, i_q, h_q, c_q;
  assign e_q = ({16'd0, iterations_k} * {16'd0, hits_per_million}) / 32'd1000;
  assign expected_hits = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  assign i_q = (hits_per_million == 16'd0) ? 32'd0
             : (32'd4000 / {16'd0, hits_per_million});
  assign iters_for_four_k = (i_q > 32'd65535) ? 16'hFFFF : i_q[15:0];
  assign h_q = ({16'd0, iters_for_four_k} * {16'd0, minutes_per_k}) / 32'd60;
  assign hours_needed = (h_q > 32'd65535) ? 16'hFFFF : h_q[15:0];
  assign c_q = (expected_hits > 16'd3) ? 32'd95
             : ({16'd0, expected_hits} * 32'd30);
  assign confidence_pct = (c_q > 32'd100) ? 16'd100 : c_q[15:0];
  assign campaign_sized = (IT_REPRODUCED_ONCE != 0) ? 1'b1
                        : (expected_hits > 16'd3);
  assign undersized_err = plan_it && (expected_hits <= 16'd3) && campaign_sized;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_undersized <= 8'd0;
    end else if (plan_it) begin
      n_plans <= n_plans + 8'd1;
      if (!campaign_sized) n_undersized <= n_undersized + 8'd1;
    end
  end
endmodule

Five plans. A race at 40 hits per million, two minutes per thousand iterations.

Three plans undersized; none when one reproduction is taken as enough.

A coherency race reproduces at a rate, and a campaign that does not know the rate cannot be sized. Forty hits per million iterations needs a hundred thousand iterations for four expected hits — three hours here — and fifty thousand gives two, which is a coin toss reported as a result.

Row five is the honest failure and it is the usual starting state. With no measured rate there is no iteration count to aim for, and the first job is to measure it. That is the same finding as 26.1 §13 and 26.2 §13, arriving for the third time in three chapters — which is what makes it a method rather than an observation.

Rows three and four are the threshold. Four expected hits is enough to take an absence seriously and three is not; the number is arguable and having one is not.

12. RTL 8 — A Trace Is One Observer

// RTL 8 - what a trace proves. A capture at one point on the fabric shows one
// observer's order, and a coherency claim is about every observer's.
module trace_scope #(parameter int ONE_PROBE_IS_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        audit,
  input  logic [15:0] agents, probed, claims, claims_needing_all,
  output logic [15:0] visible_agents, blind_agents, provable, unprovable,
  output logic        claim_supported,
  output logic [7:0]  n_audits, n_unsupported,
  output logic        overclaimed_err
);
  assign visible_agents = (probed > agents) ? agents : probed;
  assign blind_agents = (agents > visible_agents)
                        ? (agents - visible_agents) : 16'd0;
  // A claim about every observer needs every observer probed.
  assign unprovable = (ONE_PROBE_IS_ENOUGH != 0) ? 16'd0
                    : ((blind_agents != 16'd0) ? claims_needing_all : 16'd0);
  assign provable = (claims > unprovable) ? (claims - unprovable) : 16'd0;
  assign claim_supported = (unprovable == 16'd0);
  assign overclaimed_err = audit && (blind_agents != 16'd0)
                           && (claims_needing_all != 16'd0) && claim_supported;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_audits <= 8'd0; n_unsupported <= 8'd0;
    end else if (audit) begin
      n_audits <= n_audits + 8'd1;
      if (!claim_supported) n_unsupported <= n_unsupported + 8'd1;
    end
  end
endmodule

Five audits. Six agents, thirty claims, eighteen needing every observer.

Two audits with an unsupported claim; none when one probe is taken as enough.

A capture supports claims about the agents it captured. Two probes on six agents leaves four whose transactions are not in the trace at all — and a claim of the form "no agent held this line modified" is about all six. Eighteen of thirty claims fall, and the trace gives no indication which.

Row three is the sensitivity and it is the point. One unprobed agent is enough to invalidate every global claim — the property is a conjunction over observers, so partial coverage buys nothing for it. Five of six probed supports exactly the same eighteen-claim shortfall as two of six.

Row four is the exemption. A claim set that needs no global scope — "this agent issued this transaction", "this response carried this data" — is fully supported by two probes, and most of what a trace is actually used for is of that kind. The distinction is worth making explicitly rather than discovering it in a review.

13. RTL 9 — One Wrong Line Is Not One Bug

// RTL 9 - the blast radius. One coherency bug corrupts every line that met the
// same condition, and the count decides whether a fix is enough.
module blast_radius #(parameter int ONE_LINE_ONE_BUG = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [15:0] lines_touched, condition_rate_ppm, hours_running,
  input  logic [15:0] lines_per_hour_k,
  output logic [15:0] exposed_lines, condition_lines, detected, undetected,
  output logic        radius_bounded,
  output logic [7:0]  n_assessments, n_unbounded,
  output logic        radius_understated_err
);
  logic [31:0] e_q;
  // lines_per_hour_k is in thousands, so hours x k x ppm / 1000 is lines.
  assign e_q = ({16'd0, hours_running} * {16'd0, lines_per_hour_k}
                * {16'd0, condition_rate_ppm}) / 32'd1000;
  assign condition_lines = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  // One observed failure is one line; the condition applies to all of them.
  assign exposed_lines = (ONE_LINE_ONE_BUG != 0) ? lines_touched : condition_lines;
  assign detected = lines_touched;
  assign undetected = (exposed_lines > detected)
                      ? (exposed_lines - detected) : 16'd0;
  assign radius_bounded = (undetected == 16'd0);
  // A condition that applies to thousands of lines, scoped to the one seen.
  assign radius_understated_err = assess && (condition_lines > lines_touched)
                                  && radius_bounded;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assessments <= 8'd0; n_unbounded <= 8'd0;
    end else if (assess) begin
      n_assessments <= n_assessments + 8'd1;
      if (!radius_bounded) n_unbounded <= n_unbounded + 8'd1;
    end
  end
endmodule

Five assessments. A condition at 50 ppm, 40 thousand lines an hour.

Two assessments unbounded; none when one line is one bug.

A coherency bug has a condition, and the condition applied to every line that met it. One wrong value observed over twelve hours at forty thousand lines an hour with a fifty-parts-per-million condition is twenty-four lines corrupted and twenty-three nobody looked at.

Row one is what the number changes. A fix that repairs one line is a different piece of work from a fix that must find and repair twenty-four — and the scope of the recovery is decided by this arithmetic, not by the count of complaints.

Row three is the boundary where the intuition is right. When the condition rate and the workload produce exactly one line, the one failure is the whole story — and the model reports that rather than inflating it. The error fires only when the condition reaches further than the detection did.

Figure 3 — The complaint is one line and the condition is twenty-four. The difference decides whether the fix ships with a data-recovery step, and it is arithmetic on two numbers a running system already has.

14. RTL 10 — A Coherency Diagnosis Assembled

// RTL 10 - a coherency diagnosis assembled. Everything that must hold before
// "the trace looks correct" is a claim about observers the trace did not have.
module coherency_signoff #(parameter int TRACE_LOOKS_RIGHT = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       trace_clean,      // the captured trace shows no violation
  input  logic       fate_known,       // sent-and-ignored separated from never-sent
  input  logic       evictions_safe,   // no filter entry evicted while a cache holds it
  input  logic       order_decidable,  // event pairs are outside the observer skew
  input  logic       all_agents_probed,// every observer is in the capture
  input  logic       radius_bounded,   // the blast radius is counted, not assumed
  output logic       diagnosed,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_diagnosed,
  output logic       false_diagnosis_err
);
  assign fail_mask[0] = ~trace_clean;
  assign fail_mask[1] = ~fate_known;
  assign fail_mask[2] = ~evictions_safe;
  assign fail_mask[3] = ~order_decidable;
  assign fail_mask[4] = ~all_agents_probed;
  assign fail_mask[5] = ~radius_bounded;
  // The trace-looks-right build is what one capture supports.
  assign diagnosed = (TRACE_LOOKS_RIGHT != 0) ? trace_clean : (fail_mask == 6'd0);
  assign false_diagnosis_err = evaluate && diagnosed && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_diagnosed <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (diagnosed) n_diagnosed <= n_diagnosed + 8'd1;
    end
  end
endmodule

Seven configurations.

One configuration diagnosed under the full model; six under the trace reading.

Row four is the one that makes the point without argument. Twelve event pairs sit inside the observer skew, so the trace looks ordered because it cannot show them as anything else — the ordering in the capture is an artefact of one probe's arrival times, and a reading that takes it as fact is reading its own instrument.

Rows three and six are the two time horizons. The filter eviction happened before the symptom and the twenty-three corrupted lines will be read after it — neither is in the window anybody captured, and both are countable from data the device already has.

These six are not 25.2's six. That chapter's failures are checks that were too narrow to see a violation; these are observations too narrow to attribute one that has already happened. A team can pass that chapter's standard and still spend a month here.

Figure 4 — Snoop fate is asked first because one capture point answers it and it halves the cause set by naming a side. The blast radius is asked last because it does not change the diagnosis — it changes how much of the machine the fix has to touch.

15. Quantitative Reasoning

Staleness window. A read 900 ns after the write on a 300 ns snoop path and 600 ns writeback rules out two causes of eight; at exactly 300 ns it rules out none, and at 301 it rules out one.

Snoop fate. Sixty writes with 48 snoops seen and 40 answered is twelve never sent and eight unanswered — twelve on one side and eight on the other, or all twenty on one.

Filter eviction. Forty evictions of lines still held is twelve hundred reads at risk, created at the eviction and observed much later.

Ordering. Twelve of forty pairs inside a 50 ns observer skew is seventy percent orderable; a single global order reports a hundred.

Dropped writebacks. Six of ninety dirty evictions is 384 bytes lost and two hundred later reads expected stale.

Bias. A hundred and twenty device-bias lines with eighty flushed is forty unsafe reads, three versions behind — 92% safe.

Reproduction. A race at 40 per million needs a hundred thousand iterations for four expected hits — three hours.

Trace scope. Two probes of six agents leaves four blind and eighteen of thirty claims unsupportable; one unprobed agent is enough.

Blast radius. One wrong line at a 50 ppm condition over twelve hours at 40k lines an hour is twenty-four lines, twenty-three of them unexamined.

The assembled diagnosis. Six properties, seven configurations, one diagnosed. The trace reading called six diagnosed.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Three checks ran before the campaign and a fourth found a modelling error.

The scripted mutual-exclusivity check — introduced in 25.7, made a script in 26.2 after a reading missed one — reported clean on all ten models.

The scripted output-listing step reported twenty-six unasserted nets and six were real, all six the broken build's own claim: a safety percentage, an orderable count, a coherent-read count and their percentages.

Every code block is generated from the verified source file, so the chapter and the simulation cannot disagree.

And the first testbench run caught a unit error the compiler could not. Section 13's condition count divided by a million when the input was already in thousands of lines per hour, so every case computed zero and twelve assertions failed at once. A model that compiles and simulates can still be wrong about its own units, and the only detector is asserting a value you computed by hand first.

Alongside those: every inclusive threshold at exactly equal, every clamp driven past its cap, and both builds asserted on every degenerate case.

Staleness window. The gap is driven at exactly the snoop latency and one nanosecond past it — the boundary at which an observation starts to mean something.

chk(wGr == 16'd0, "at exactly the snoop latency nothing is ruled out");
chk(wGr == 16'd1, "one nanosecond past, and a cause falls");

Snoop fate. More responses than snoops is driven, exercising the clamp.

Filter eviction. Lines still held with no eviction at all is driven — the configuration in which the fault's two halves are separated — and it killed one survivor.

Ordering. The gap is driven at exactly the skew and one nanosecond inside, and a case where both observers agree is driven to show the skew bounds disagreement rather than ordering.

Dropped writebacks. A complete-writeback case asserts no later stale reads are expected, which killed a second survivor.

chk(bGs == 16'd0,  "and no later read is expected to be stale");

Bias. More lines flushed than were in device bias is driven, exercising the clamp.

Reproduction. Exactly four expected hits and exactly three are both driven.

Trace scope. A claim set needing no global scope is driven, where two probes support all thirty.

Blast radius. The condition is driven exactly equal to the detection — the boundary where one failure is the whole story — and one line past it.

The assembled diagnosis. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 299 checks across two testbenches, 153 on the front five models and 146 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Seventy-five mutations were injected one at a time. 75 injected, 75 killed, after three survivors.

All three survivors were the same class: a degenerate input at the top of a model rather than at a threshold. Lines held with no eviction, a capture with no pairs, and a complete writeback whose later-read count was never asserted. Each was one case or one assertion.

Not one survivor was a clamp never driven past its cap, which is the class that produced four of five survivors in batch 024's last chapter. Nine clamp mutations appear in the table above and every one was killed on the first run, because the rule is now applied while writing the stimulus rather than after a survivor asks. Three batches on, it costs nothing and removes the class.

18. Verification Strategy

What a testbench for an attribution model must cover.

Assert a value you computed by hand before running. Section 13's unit error passed compilation, passed elaboration and produced zero on every input. The twelve failing assertions were the only thing standing between it and a chapter full of wrong numbers.

Drive the degenerate input at the top of the model, not just at its thresholds. All three survivors here. A guard on an input being non-zero needs that input at zero, and it is the case a realistic stimulus never generates.

Separate the two halves of a coincidence. §7's fault is an eviction and a line still held; driving each without the other is what proves the model detects the conjunction rather than either term.

The cases where the single observation is right. A gap inside the snoop path. Every snoop answered. No line still held. A gap outside the skew. Observers that agree. A claim set needing no global scope. A condition reaching exactly as far as the detection. Seven exemptions across nine models.

Counters as a second signature. Ten models, ten pairs of totals, differing in nine. The tenth is §13's assessment count in the one-line build, which is deliberately equal because the condition rate is a fact about the workload rather than about the reading.

What a real investigation needs that these models do not have. A structured cause set for §5, a per-observer skew matrix for §8, and a non-uniform condition rate for §13. All three are abbreviations that preserve the conclusion, and section 26 exercises 1, 4 and 9 are where they come back.

19. Synthesis and Implementation Reality

The staleness interval is two timestamps in a trace that already exists, and section 5's narrowing costs a subtraction. The reason it is not done is that the snoop latency and writeback time are design numbers that live in a different document from the trace.

Separating a never-sent snoop from an unanswered one needs the snoop on the wire, which is one capture point on the link rather than instrumentation in either agent. It is the cheapest disambiguation in the chapter and it has to be planned before the failure.

Snoop-filter evictions are visible in the device's own counters and almost never exported. Section 7's precursor is a counter, not a capture — and it is the only entry in this chapter that fires before the symptom.

Observer skew is a property of where the probes are, and on a multi-switch fabric it is tens of nanoseconds. Section 8's undecidable pairs are not a limitation of the analyser but of the geometry, and the fix is a second synchronised probe rather than a longer capture.

Bias state is device-internal. Section 10 is invisible in any host-side trace unless the device exports the bias transitions — which makes it a design decision like 26.1 §20's latched negotiation outcome, made long before the debug that needs it.

The blast radius in section 13 is arithmetic on numbers a running system already has — lines touched per hour and the condition's rate — and it is the calculation that decides whether a fix ships with a data-recovery step.

20. Silicon Observability

"Snoop-filter evictions of lines a cache still holds" is the one entry that changes the shape of the investigation rather than its speed. Every other counter helps after a wrong value is observed. This one fires at the moment the fault is created, which is minutes earlier and in a completely different part of the trace.

21. Debug Lab

Symptom. A CXL type-2 accelerator returns a wrong tensor element roughly once every few hours under a customer workload. The protocol analyser is connected, the trace around the failing read looks entirely correct — the read is issued, a completion returns, the data is what memory holds.

Step 1 — when was the line last written? Section 5. The trace buffer reaches back far enough: the device wrote the line 1.4 microseconds earlier. Against a 300 ns snoop path and a 600 ns writeback, that rules out both "the snoop is still in flight" and "the writeback is in progress" — two causes of eight, from two timestamps.

Step 2 — was a snoop sent? Section 6. The capture covers the link, so the answer is in it: no snoop for that line appears at all. That places the bug on the sending side and removes the entire "sent and ignored" family.

Step 3 — why was no snoop sent. The host's snoop filter had no entry for the line. Section 7. The device's filter-eviction counter is added and a soak run shows evictions of lines caches still hold, at a low rate — the entry was evicted under filter pressure while the device still had the line modified.

Step 4 — how far does this reach? Section 13. The condition is filter pressure at a particular set, measured at about fifty parts per million of lines touched. At the customer's forty thousand lines an hour over twelve hours, twenty-four lines met it and one was noticed. The fix needs a data-recovery step.

Step 5 — can the ordering claim be made? Section 8. The team wants to state that no other agent held the line. Two probes on a six-agent fabric support claims about two of them. Eighteen of the thirty claims in the draft report are not supportable from this capture — including that one.

Step 6 — sizing the confirmation. Section 11. At fifty parts per million the reproduction needs about a hundred thousand iterations for four expected hits — three hours on the bench — and the first attempted confirmation ran fifty thousand and came back clean. That clean run meant nothing and was nearly taken as a fix.

The finding. A snoop-filter eviction of a line a cache still held, created minutes before the symptom and in a different part of the trace, reaching twenty-four lines of which one was seen — and a draft report containing eighteen claims the capture could not support.

The fix. In the design, gate filter eviction on the line's cached state. Add the eviction-while-held counter (section 20), because it is the only thing that fires before the corruption. Add a second synchronised probe so ordering claims are supportable. And ship the fix with a recovery step for the other twenty-three lines, which the radius arithmetic named.

What made this hard. The trace around the failure was correct in every detail. Everything that explained it was outside the window, on the other side of an eviction nobody counted, or about agents the capture did not include.

22. Design Review

1. What is the interval between the last write and the stale read? Two timestamps, and it rules out two causes of eight. Sections 5 and 19.

2. Does a snoop for that line appear on the wire at all? It separates the sender's bug from the receiver's in one observation. Section 6.

3. Is there a counter for filter evictions of lines a cache still holds? The only one that fires before the symptom. Sections 7 and 20.

4. What is the observer skew between the capture points? Below it, ordering is not a fact. Sections 8 and 19.

5. How many agents are on the fabric, and how many are in the capture? One unprobed agent invalidates every global claim. Section 12.

6. Which claims in the report need every observer? The distinction is worth making before the report is written. Section 12.

7. Do dirty evictions and writebacks balance? A mismatch is data that no longer exists. Section 9.

8. Are bias transitions and flushes exported? Device-internal, and invisible otherwise. Sections 10 and 19.

9. What is the condition rate, and how many lines did it touch? It decides whether the fix needs a recovery step. Sections 11 and 13.

10. What does "the trace looks correct" establish? Section 14 exists because the answer is the first property only.

23. How This Appears In Real Engineering

A debug engineer gets a capture around the failing transaction because that is what the trigger caught. Everything in this chapter that explains a coherency bug is outside that window, which is a statement about trigger design rather than about skill.

A device architect decides whether bias transitions and filter evictions are exported. Both decisions are made years before the debug that needs them, and neither has a customer asking for it at the time.

A verification engineer meets 25.2 and this chapter as two halves of one problem: could the check have seen it, and can the trace attribute it. Passing the first does not help with the second.

A support engineer writes the report, and section 12 is about what the report may claim. Eighteen of thirty is a large fraction of a draft, and the distinction is invisible without counting agents against probes.

24. Common Misconceptions

"The trace looks correct." From one probe, on a fabric with six agents (section 12). Eighteen of thirty claims are about agents that are not in it, and nothing in the trace indicates which.

"There was no snoop response, so no snoop was sent." Those are different observations (section 6) — twelve never sent and eight sent-unanswered are two bugs in two blocks, and the wire tells you which.

"The eviction was harmless — nothing broke." Nothing broke yet. Forty unsafe evictions is twelve hundred reads at risk (section 7), and the symptom arrives at whichever one comes first.

"The trace shows A before B." It shows A arriving before B at one probe. Twelve of forty pairs sit inside the observer skew (section 8) and cannot be ordered from any single capture.

"We fixed the line that came back wrong." The condition touched twenty-four (section 13). One was noticed, and a fix without a recovery step leaves twenty-three.

"It reproduced once, so we can test the fix." One reproduction is not a rate. A hundred thousand iterations for four expected hits (section 11) — and fifty thousand clean means nothing.

25. Interview Reasoning

"A read returns stale data and the trace around it looks fine. What do you do first?" Find when the line was last written and compare the interval against the snoop path and writeback time. Two timestamps, two causes of eight removed — and it uses a trace you already have.

"No snoop response appears. What does that tell you?" Nothing yet — look for the snoop itself (section 6). Never-sent and sent-unanswered are different blocks, and the wire distinguishes them in one observation. A candidate who starts debugging the receiver has assumed the harder half.

"Your trace shows the write completing before the read issues. Is that an ordering fact?" Only if the gap exceeds the observer skew. Twelve of forty pairs here do not (section 8), and ordering them is reading the instrument rather than the system.

"You found the bug and fixed the line. Are you done?" How many lines met the condition? Twenty-four over twelve hours at fifty parts per million (section 13) — and the fix ships with a recovery step or it does not ship.

"The fix has run for fifty thousand iterations with no failure. Is it fixed?" At forty hits per million that is two expected hits and sixty percent confidence (section 11). The question to ask back is what the measured rate is — without it, no run length is defensible.

26. Exercises

1. Structure the cause set. §5 uses eight flat causes. Write the real ones for a stale CXL.cache read and say which mechanism's latency rules out each.

2. Place the probe. For a six-agent fabric, choose two capture points that maximise the claims supportable, and state which claims still are not.

3. Build the eviction counter. Specify the counter in §7 precisely — what increments it, and what state it needs — and estimate its cost in the filter.

4. Measure the skew. Design the experiment that measures observer skew between two capture points, and state what it costs and how often it must be redone.

5. Balance the evictions. Write the check that compares dirty evictions against writebacks (§9), and say where it lives and what it does when they disagree.

6. Export the bias. Propose the minimum device-side export that makes §10 diagnosable from a host-side trace, and price it in registers.

7. Size the campaign. For rates of 40, 4 and 0.4 hits per million at two minutes per thousand iterations, tabulate the hours for four expected hits. Where does the bench stop being the right instrument?

8. Scope the report. Take a ten-claim draft and classify each claim as local or global, then state the probe count each needs.

9. Model a non-uniform condition. §13 assumes a uniform rate. Make the condition address-dependent and re-derive the blast radius for a workload that touches one region ten times more than the rest.

10. Add the seventh property. Propose one none of §14's six implies, name its section, and construct the configuration where the six hold and it fails. A property that cannot fail alone is not a seventh property.

27. Summary

A stale read is one observation and a trace is one observer, and almost everything that explains a coherency bug is outside the window the trigger caught.

The staleness interval rules out two causes of eight. A read nine hundred nanoseconds after a write on a three-hundred-nanosecond snoop path is not a snoop in flight — and at exactly three hundred nanoseconds it might be, which is what makes the threshold the mechanism's own latency.

A snoop never sent and a snoop ignored are different bugs in different blocks. Twelve and eight, distinguished by one capture point on the link — and without it, all twenty are attributed to whoever sent them.

The precursor is an eviction, not a read. Forty filter entries dropped while caches still held the lines is twelve hundred reads at risk, created minutes before any symptom and visible only in a counter almost nobody exports.

"Before" is a property of an observer. Twelve of forty event pairs sit inside a fifty-nanosecond skew and cannot be ordered from any single capture — while a single-global-order reading orders all forty and calls the trace correct.

A dropped writeback is data loss wearing a stale read. Six of ninety dirty evictions is 384 bytes that no longer exist and two hundred later reads that will return the wrong thing.

A device-bias line read without a flush returns what memory holds. Forty reads, three versions behind, completing normally and indistinguishable from correct ones except by a bias state the host cannot see.

A race needs a rate before a campaign. Forty hits per million is a hundred thousand iterations for four expected hits — and fifty thousand clean is a coin toss reported as a fix. Three chapters running have ended on the same arithmetic, which is what makes it a method.

A trace supports claims about the agents it captured. Two probes of six leaves eighteen of thirty claims unsupportable, and one unprobed agent is enough — the property is a conjunction over observers.

And one wrong line is not one bug. A fifty-parts-per-million condition over twelve hours at forty thousand lines an hour is twenty-four lines, twenty-three unexamined — which decides whether the fix ships with a recovery step.

All three mutation survivors were degenerate inputs at the top of a model rather than at a threshold — lines held with no eviction, a capture with no pairs, a writeback that dropped nothing. Not one was a clamp never driven past its cap, the class that dominated two chapters ago: nine clamp mutations appear in this chapter's table and every one died on the first run, because the rule is now applied while writing the stimulus rather than after a survivor asks.

And the first testbench run caught a unit error the compiler could not. Section 13 divided by a million where the input was already in thousands, so every case computed zero — twelve assertions failed at once, and nothing else in the flow would have noticed a chapter full of zeroes.

"The trace looks correct" is one property of six. The reading called six of seven configurations diagnosed when one was — and §21 is an accelerator whose failing trace was correct in every detail, explained by an eviction minutes earlier, reaching twenty-three lines nobody looked at, in a report with eighteen claims the capture could not support.

26.4 — Memory-Access Issues stays above the link and moves off .cache. This chapter is about a line whose value is wrong because coherency failed; the next is about a read that went to the wrong place entirely.