Debug Review Checklist

Module 21 · Chapter 21.5 · CHI Design-Review Checklist

Project thread — 21.4 audited performance. 21.5 is the triage checklist for when a bug fires; 21.6 is the interview revision list.

1. The Review

The debug review is a triage checklist for the moment a CHI bug fires — in simulation, emulation, or silicon. Unlike the earlier reviews (gates before a stage), this is a response: classify the symptom, capture the evidence, and work backward to the root cause. Its central truth is that coherence bugs are non-local — the place a bug shows is almost never the place it is caused — so a disciplined triage that follows the mechanism beats reacting at the crash site.

2. When To Run It

Run this triage the instant a bug fires — the first minutes decide whether the evidence survives.

• Trigger: a failure — an assertion fire, a hang, a data mismatch, a performance cliff, a RAS event.
• Precondition: tracing / logging is available (or can be re-run) — triage needs evidence, and evidence is perishable.
• Output: a localized root cause with a reproduction, handed to the owner — not a symptom-site patch.
• Cost of skipping: a relocated bug (symptom hidden, cause alive) or a lost repro — the rare bug that never comes back to be fixed.

3. Key Terms

4. Previous Chapter Connection

This review distills Module 18 (CHI debugging) into a first response. The waveform-driven method, the missing-snoop and lost-update patterns, the deadlock analysis, and the coherence-state reconstruction all become triage steps. The protocol knowledge (Chapters 4–14) is what lets you walk backward from effect to cause along the real mechanism.

Where 21.1–21.4 tried to prevent bugs (right, clean, verified, fast), 21.5 accepts that some fire anyway — in a long regression, in emulation, in silicon — and asks: now what? It adds the triage discipline that turns a panic into a procedure, and its output — a root cause and a repro — is what a fix and a new test (feeding back to Chapter 21.3) are built on.

5. How To Use This Checklist

The method — capture first, classify, then walk backward.

1. Preserve the evidence immediately. Save the trace, logs, and final state before anything is overwritten or the run is lost. Evidence is perishable; capture beats cleverness.
2. Classify the symptom (section 7) — the class determines which triage path (sections 9–13) to follow.
3. Walk backward from the manifestation — never assume the crash site is the cause; follow the transaction and the line to their origin.
4. Localize to a root cause — a specific agent, transaction, and cycle — with a reproduction.
5. Hand off cause, not symptom — a fix at the root, plus a new test so it cannot recur (Chapter 21.3).

6. The Triage Areas

The seven areas of CHI bug triage, and what each isolates.

Each area below is a checklist plus the why. A step is done when it points at a cause, not when it describes the symptom.

7. Checklist — Symptom Classification

• The failure is classified — hang, data corruption, ordering violation, performance regression, or RAS event.
• The first observable is pinned down — which assertion, which signal, which mismatch, at what time.
• The class selects the triage path (sections 9–13) — you are not debugging blind.
• Multiple symptoms are checked for a common cause — one root cause can fire several assertions.
• The symptom is not yet assumed to be the cause — classification is orientation, not conclusion.

Why. The class routes the whole triage (Module 18). Misclassifying (calling a coherence bug a performance one) sends you down the wrong path — so classify precisely and from the first observable, not a downstream effect.

8. Checklist — Trace & Evidence Capture

• The full trace (all four channels, all relevant agents) around the failure is captured and saved.
• The final state — caches, directory, trackers, credits — is dumped before it is lost.
• The capture window is wide enough to include the cause, which precedes the symptom (possibly by many cycles).
• A reproduction (seed, trace, config) is recorded — a rare bug you cannot re-run may be unfixable.
• Evidence is captured before any "quick fix" perturbs the scenario.

Why. Evidence is perishable and the cause is upstream (Module 18). Capture too narrow or too late and the cause is off-screen — so widen the window and save state first, before debugging destroys the very thing you need.

9. Checklist — Hang & Deadlock Triage

• The stuck transaction(s) are identified — which agent waits on what, in which state.
• The dependency chain is traced — A waits on B waits on C — looking for a cycle (Chapter 13).
• Credit / flow-control state is checked — a lost or un-returned credit stalls without a cycle (Chapter 14).
• Channel independence is verified — a response blocked behind a request is a classic hang.
• The root of the wait is found — the transaction everything else is (transitively) waiting on.

Why. A hang's cause is a cycle or a starved resource, not the victim you first notice (Chapters 13–14). Fixing the most visible stuck agent does nothing if the real cycle is elsewhere — so trace the dependency chain to its root.

10. Checklist — Data-Corruption Triage

• The corrupted line and the wrong value are pinned — expected vs observed, at the reader.
• The line's full history is reconstructed — every access, snoop, and state change, across all agents.
• The divergence point is found — the transaction that should have updated/invalidated but did not (Module 18).
• The cause is traced to a different agent / earlier cycle than the read — a missed invalidation, a lost update.
• The coherence rule that was violated is named — so the fix targets the rule, not the symptom.

Why. Data corruption is the canonical non-local bug (Module 18): the stale read is at the reader, but the cause — a missed invalidation — is at another agent, earlier. Fix the reader and you fix nothing; trace to the missed operation.

11. Checklist — Ordering & Coherence-State Triage

• The observed order each agent saw is reconstructed — who saw what, when (Chapter 9).
• The ordering point (usually the home) is checked — did it serialize the conflict correctly (Chapter 10)?
• The coherence-state transitions are replayed against the spec — find the illegal or skipped transition (Chapter 4).
• The hazard handling for the conflicting addresses is examined — was an overlap missed?
• The mismatch is traced to its origin transaction — not the agent that later observed it.

Why. Ordering/state bugs are subtle and non-local (Chapters 9–10). The agent that observes the wrong order is downstream of the ordering point that failed — so replay the states and find the transaction where the order actually broke.

12. Checklist — Performance-Regression Triage

• The regression is quantified — what got slower, by how much, on which workload (Chapter 21.4).
• The new bottleneck is located — a hotspot home, a saturated link, an SLC hit-rate drop (Chapters 15.5, 15.7).
• The change that introduced it is bisected — config, RTL, or workload shift.
• The latency distribution is compared before/after — the tail, not just the mean (Chapter 15.1).
• The cause is a specific resource, not a vague "it's slower" — localized like any other bug.

Why. A performance regression is a bug (Chapter 21.4). Its cause is a specific resource that shifted — a hotspot, a cache loss — not the place the slowdown is felt, so triage it quantitatively, to a resource.

13. Checklist — RAS & Error Triage

• The originating fault is identified — where poison or an error response first arose.
• The propagation is traced — how the error/poison flowed through transactions and agents.
• Containment is verified — did the error stay contained, or leak past its boundary (Chapter 20.6)?
• The handling is checked — was the error response correct, was the poison consumed safely?
• The distinction fault vs consequence is made — the first fault, not the downstream errors it caused.

Why. RAS bugs cascade (Chapter 20.6): one fault spawns many downstream errors. Chasing a downstream error wastes time — so find the originating fault and verify containment, the two things that actually matter.

14. A Worked Triage

Triaging a data-corruption bug — non-locality in action.

1. Classify (7). A scoreboard mismatch: a read returned a stale value. Class: data corruption. First observable: the read at agent A, cycle N.
2. Capture (8). Save the trace wide (back to cycle N−2000) and dump all caches + the directory. The cause is upstream.
3. Reconstruct (10). Walk the line's history: agent B wrote it at cycle N−1500, and the home was supposed to invalidate A's copy — but the snoop to A was never sent.
4. Localize. Root cause: a missed invalidation at the home, cycle N−1500 — 1500 cycles and one agent away from the symptom at A.
5. Reproduce & hand off. Record the seed; hand the home's missed-snoop logic to its owner, plus a new directed test (Chapter 21.3).

The symptom was at agent A, cycle N; the cause was at the home, cycle N−1500. Fixing agent A would have hidden the mismatch and left the missed-snoop bug alive — the pitfall in section 16.

15. Sign-off Criteria

What "the bug is triaged" actually means.

• The symptom is classified and the first observable pinned to an agent, signal, and time.
• Evidence — trace, state, and a reproduction — is captured and saved.
• The root cause is localized to a specific agent, transaction, and cycle — not the manifestation site.
• The violated rule (coherence, ordering, flow-control, RAS) is named.
• The cause (not the symptom) is handed off, with a new test so it cannot recur.

16. The Pitfall

17. Common Mistakes & Red Flags

• Fixing the symptom site. Assumption: fix where it fired. Bug: the defect relocates (the Pitfall). Fix: trace to root cause.
• Capturing too late/narrow. Assumption: the symptom window is enough. Bug: cause off-screen. Fix: capture wide and early.
• Losing the repro. Assumption: it'll happen again. Bug: rare bug never returns. Fix: record the seed immediately.
• Misclassifying. Assumption: it's a perf issue. Bug: wrong triage path. Fix: classify from the first observable.
• Chasing downstream errors. Assumption: the loud error is the bug. Bug: it's a consequence. Fix: find the originating fault.
• Red flag — "just add a workaround at the reader." A symptom-site patch; find the missed operation.
• Red flag — no saved state dump. The cause is upstream and now gone; always dump caches/directory/credits.
• Red flag — "can't reproduce it anymore." The repro was lost; capture it before touching anything.

18. Quick Revision

Debug Review Checklist. A triage checklist for the moment a CHI bug fires — a response, not a pre-stage gate. Its central truth: coherence bugs are non-local — the place a bug manifests (the reader with stale data, the stuck transaction) is almost never the place it is caused (a missed invalidation at another agent, a dependency cycle or lost credit elsewhere, an ordering point that failed). Run it the instant a bug fires, because evidence is perishable. Capture first — a wide trace and a state dump (caches, directory, trackers, credits) plus a reproduction — then classify the symptom (hang, data corruption, ordering, performance regression, RAS), which selects the triage path, then walk backward to a specific agent, transaction, and cycle. The seven areas: Symptom Classification (route by class), Trace & Evidence Capture (wide, early, saved), Hang & Deadlock (dependency chain → cycle/starved credit, Chapters 13–14), Data-Corruption (line history → missed invalidate/lost update, Module 18), Ordering & Coherence-State (replay states → the failed ordering point, Chapters 9–10), Performance-Regression (quantify → the shifted resource, Chapter 21.4), and RAS & Error (originating fault, propagation, containment, Chapter 20.6). Sign-off means the root cause is localized (not the symptom site), the violated rule named, and the cause handed off with a new test so it cannot recur. The pitfall is symptom-site fixing — patching where the bug showed instead of where it was caused, so you hide the symptom and relocate the defect (and, worse, lose the repro by capturing too late). Fix the cause, not the crash site. Next, 21.6 is the interview revision list — the last-mile before you walk in.

Coming Next

Chapter 21.6 — Interview Review Checklist. The engineering checklists are done; the last one is for you, the night before. Chapter 21.6 covers the last-mile revision list before a CHI interview — a rapid self-audit of the fundamentals, channels and flow control, coherence states, transactions, ordering and hazards, deadlock and QoS, and system design, so you walk in with the whole track compressed into a checklist you can run in your head.