Coherency Bug Debug Question

Module 19 · Chapter 19.9 · CHI Interview Mastery

Project thread — 19.8 explained credits. 19.9 is the senior debug question; 19.10 is the whiteboard architecture finale.

1. The Question

"How do you debug a coherency bug from waveforms?" — or: "A core reads stale data — walk me through your debug," "How do you approach a coherency failure?" "You've got a waveform and a coherence bug — go." It is a senior question, often the hardest in the interview, and it tests methodology — the interviewer wants a process, not a lucky guess.

2. What the Interviewer Is Probing

This tests whether you have a structured, repeatable debug process for the hardest class of bug. The interviewer wants:

• Do you have a methodology — or do you flail at the symptom?
• Do you classify the failure family — and know each has a different tool?
• Do you work backward — from symptom to cause, not debug where it manifests?
• Do you check invariants — SWMR, one-owner, conservation — to localize?

A candidate who says "I'd look at the waveform and see what's wrong" fails; one who says "I classify the family, work backward from the symptom, and check the invariant that family breaks" demonstrates a real, senior debug discipline.

3. Key Terms

4. Where This Sits in the Interview

This is the ninth rung — the senior debug question that synthesizes everything. It draws on the entire debugging module (Chapter 18): the eight failure families and their detectors. A structured answer here signals you can lead a debug effort, not just execute steps — the mark of a senior engineer. It is often the question that decides a senior-level offer.

It is the practical counterpart to the coherence and flow questions (19.4–19.8): those tested whether you know how the protocol works; this tests whether you can find out why it broke. And it directly reflects Module 18's methodology — symptom, signature, backward trace, root cause — packaged as an interview answer.

5. The Core Answer

The five-step methodology:

"My process is: capture the symptom and classify the failure family; work backward from the symptom, because the cause is far from where it shows; apply the tool that matches the family; find the first divergence from correct behavior; and map that to the root-cause mechanism. Throughout, I check the coherence invariant that family breaks — SWMR, one owner, conservation — to localize the bug in time."

The five steps:

• 1. Classify the family. Is the symptom a stale read (lost ownership / missing snoop), a wrong state, mangled data, a frozen fabric (deadlock), a stalled channel (credit leak), or a forbidden order? Each family has a different cause and tool.
• 2. Work backward. Coherency bugs manifest far from their cause — a stale read appears long after the transaction that dropped the data. Don't debug at the symptom; trace back.
• 3. Apply the family's tool. Trace ownership for a lost dirty line; a shadow FSM for a wrong state; diff snoops vs sharers for a missing snoop; follow the data for corruption; diff directory vs reality for directory corruption; a wait-for graph for deadlock; credit conservation for a leak; an inversion for ordering.
• 4. Find the first divergence. The first cycle the design deviates from correct behavior is the bug — everything after is a consequence.
• 5. Map to the root cause. The divergence points at the mechanism (a data-less snoop, a directory race, a dropped barrier) — fix that.

The synthesis:

Debug a coherency bug by method, not luck: classify the family, work backward from the symptom (the cause is far away), apply the family's tool, find the first divergence, and map it to the root-cause mechanism — checking the invariant each family breaks (SWMR, one-owner, conservation) to localize the bug in time.

6. Structuring Your Answer

The template — lead with the process, then the per-family detail.

• 1. State it's a methodology. "There's no single trick — I have a process." Frame it.
• 2. The five steps. Classify → backward → tool → first divergence → root cause. Deliver the skeleton.
• 3. The key principle. "Coherency bugs manifest far from their cause, so I trace backward, not at the symptom." The insight.
• 4. Ground it with one family. "For a stale read, I'd trace ownership back to whoever last held the line dirty and check the transaction that released it preserved the data." Show the tool in action.

The point to carry:

Lead with "it's a methodology, not a trick," because that alone separates a senior answer from a junior one. Then deliver the five steps as a crisp skeleton and ground one family concretely (a stale read → trace ownership). The backward-tracing principle is the single most important thing to say — it's what makes the debug tractable instead of a hunt at the symptom.

7. The Methodology Skeleton

The five-step process to deliver.

The rule to carry: the process is the answer; the families are the detail. Deliver the five steps as your headline, then descend into per-family tools as the interviewer probes. The backward-tracing step is the crux — it encodes the hard truth that coherency bugs are non-local, so the productive move is always symptom → cause, never staring at the symptom.

8. The Family Tools

The tool for each failure family — have these ready.

The rule to carry: each family has a specific tool, and naming them proves depth. A generic "I'd look at the waveform" is junior; "for a missing snoop I diff the snoops sent against the directory's sharers; for a deadlock I reconstruct the wait-for cycle" is senior — it shows you have a toolkit, not a hunch. You don't need all eight verbatim, but naming three or four with their tools demonstrates the methodology is real.

9. The Model Answer

A full, polished answer:

"There's no single trick — coherency debug is a methodology. My process has five steps.

First, I classify the failure family from the symptom — is it a stale read, a cache in the wrong state, a missing snoop, mangled data, a frozen fabric, a stalled channel, or a forbidden order? Each family has a different cause and a different tool.

Second — and this is the key — I work backward. Coherency bugs manifest far from their cause: a core reads stale data long after and far from the transaction that actually dropped it. So I never debug at the symptom; I trace back to the cause.

Third, I apply the tool for that family. For a stale read, I trace the line's ownership backward to whoever last held it dirty, and check whether the transaction that released them preserved the data — if it went to invalid with an empty data channel, that's lost ownership. For a wrong state, I run a golden shadow FSM and find the first cycle it diverges. For a missing snoop, I diff the snoops sent against the directory's sharers. For a frozen fabric, I reconstruct the wait-for cycle.

Fourth, I find the first divergence from correct behavior — that's the bug; everything after is a consequence. Fifth, I map it to the root-cause mechanism and fix that.

And throughout, I check the invariant the family breaks — single-writer-multiple-reader, one owner, credit conservation — because a violated invariant localizes the bug in time far better than eyeballing signals."

That is the methodology framing, the five steps, the backward-tracing principle, grounded family tools, and the invariant discipline — a complete, senior debug answer.

10. A Concrete Example

The grounding example — one family, traced end to end:

"Concretely, a stale read: core B reads a line and gets the old value, though core A wrote a new one. I don't debug at B's read — that's the symptom. I trace the line's ownership back: A held it dirty, so A was the owner. What took A out of dirty? A SnpUnique from a third core. I inspect that transaction: A went to invalid, but its response was a plain SnpResp with an empty data channel — the dirty value was never captured. That's the first divergence — a dirty owner released without preserving the data. Lost ownership. Root cause: a snoop-response path that didn't attach data for a dirty line. The stale read was thousands of cycles downstream of the real bug."

Tracing one family (a stale read → lost ownership) end to end shows the methodology working — symptom, backward trace, first divergence, root cause — which is far more convincing than reciting the steps abstractly.

11. Mental Map

The recall structure — five steps and the anchor.

• 1. Classify the family. → "which of the eight?"
• 2. Backward from the symptom. → "the cause is far away."
• 3. Family tool. → "the right detector per family."
• 4. First divergence. → "the first deviation is the bug."
• 5. Root cause. → "fix the mechanism."
• Anchor: check the invariant (SWMR / one-owner / conservation) to localize.

The rule to carry: hold the five steps, and anchor on backward-tracing + invariants. The steps are the process; the two principles — trace backward (the cause is non-local) and check the invariant (it localizes in time) — are what make the process work. If you deliver only two things, make them the five-step skeleton and "coherency bugs are non-local, so I trace backward and check invariants."

12. What a Strong Answer Includes

The rubric.

The rule to carry: the backward-tracing principle plus named family tools are the two must-haves. The first proves you understand coherency bugs are non-local (the hardest thing to internalize); the second proves your methodology is concrete (a toolkit, not a hunch). Together they mark a senior debug answer.

13. Weak Answer vs Strong Answer

Same knowledge, two deliveries.

• Weak: "I'd look at the waveform, find where the data is wrong, and work out what happened around there. I'd check the states and the transactions near the failure." — symptom-local ("around there," "near the failure"), no family classification, no backward tracing, no tools, no invariants.
• Strong: "It's a methodology: classify the family, trace backward (the cause is far from the symptom), apply the family's tool — trace ownership, shadow FSM, diff snoops-vs-sharers, wait-for graph — find the first divergence, and check the invariant it breaks." — process, backward, tools, divergence, invariants.

The difference is local vs backward: the weak answer debugs near the symptom, which for a non-local coherency bug is exactly the wrong place; the strong answer traces back to the cause with a family-specific tool.

14. The Interview Trap

15. Common Mistakes

• Debugging the symptom. Assumption: the bug is near the failure. Trap: it's non-local. Fix: trace backward to the cause.
• No family classification. Assumption: one approach fits all. Trap: wrong tool. Fix: classify, then pick the tool.
• No methodology. Assumption: "look and see." Trap: flailing. Fix: five-step process.
• No invariants. Assumption: eyeball the signals. Trap: can't localize. Fix: check SWMR/one-owner/conservation.
• No first-divergence. Assumption: any anomaly is the bug. Trap: consequences look like bugs. Fix: find the first deviation.
• Generic tools. Assumption: "check states and transactions." Trap: not specific. Fix: name the per-family tool.

16. Interview Checklist

• Frame it as a methodology, not a trick.
• Classify the failure family from the symptom.
• Work backward — the cause is far from the symptom.
• Apply the family's tool (name three or four).
• Find the first divergence from correct behavior.
• Check the invariant the family breaks — SWMR, one-owner, conservation.

17. Key Takeaways

• Coherency debug is a methodology, not a lucky guess.
• Classify the family — each has a different cause and tool.
• Work backward — coherency bugs manifest far from their cause.
• Apply the family's tool — ownership trace, shadow FSM, set diff, wait-for graph.
• Find the first divergence; check the invariant (SWMR, one-owner, conservation).
• Fix the root-cause mechanism, not the symptom; the guidance here is representative.

18. Quick Revision

Coherency bug debug. (capsule answer.) "There's no single trick — it's a methodology, five steps. First, classify the failure family from the symptom — stale read, wrong state, missing snoop, mangled data, directory mismatch, frozen fabric, stalled channel, or forbidden order. Second, and most important, work backward: coherency bugs manifest far from their cause — a stale read appears long after and far from the transaction that dropped the data — so I never debug at the symptom, I trace to the cause. Third, apply the tool for that family: trace ownership for a lost dirty line, a golden shadow FSM for a wrong state, diff snoops-vs-sharers for a missing snoop, follow the data for corruption, diff directory-vs-reality for directory corruption, a wait-for graph for deadlock, credit conservation for a leak, an inversion for ordering. Fourth, find the first divergence from correct behavior — that's the bug; everything after is a consequence. Fifth, map it to the root-cause mechanism and fix that. Throughout, I check the invariant the family breaks — single-writer-multiple-reader, one owner, conservation — because a violated invariant localizes the bug in time far better than eyeballing signals." Lead with methodology, deliver the five steps, stress backward tracing and invariants. The trap: debugging at the symptom. Representative interview guidance; 19.10 is the large-SoC architecture finale.

Coming Next

Chapter 19.10 — Large SoC Architecture Question. The interview finishes at the whiteboard: design a large coherent SoC. Chapter 19.10 covers the large-SoC-architecture question — how to structure a 32-core coherent design with distributed home nodes, a mesh, and the scalability levers, and the whiteboarding approach an interviewer expects for the most senior, open-ended question in the set.