Directory Corruption

Module 18 · Chapter 18.5 · CHI Debugging

Project thread — 18.4 debugged data corruption. 18.5 debugs directory corruption; 18.6 debugs deadlock.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Define directory corruption — the directory's record disagreeing with the actual caches.
• Explain why it is a root-cause family behind missing/spurious snoops and wrong sourcing.
• Read the signature — a directory sharer set not matching the real cache holders.
• Diagnose by diffing the directory against an aggregate of real cache states.
• Trace the first divergence back to the corrupting directory update.
• Implement a representative directory-mismatch detector in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

The directory is the home's model of reality — its record of which caches hold each line and in what state — and the home makes every coherence decision from it: whom to snoop, where to source data, what state to grant. If the directory's model diverges from the actual caches, the home makes wrong decisions, and the surface symptoms are varied: a missing snoop (Chapter 18.3), a spurious snoop, stale sourcing. Directory corruption is the root-cause family behind many of them, so recognizing it collapses several symptoms to one underlying failure.

The diagnosis is a comparison you already have the pieces for. The coherency checker of Chapter 17.2 aggregates the real cache states; directory corruption is when the directory's record does not match that aggregate. So the debug is a diff: directory sharer set vs actual holder set, run continuously — and the first cycle they disagree is the corruption, with the directory update at that point being the bug. This turns "coherence is misbehaving in confusing ways" into "the directory disagreed with reality here, because of this update." This chapter is that diff and trace.

3. Key Terms

4. Previous Chapter Connection

This chapter debugs the directory-management RTL of Chapters 16.1 and 16.3. The corrupting updates are ones you have seen: a lost update from a concurrent same-line race (Chapter 16.1) that dropped a sharer, or an eviction that skipped back-invalidation (Chapter 16.3) that orphaned one. This chapter finds the effect — the record diverging from reality — and traces it to that update.

It is the root-cause counterpart to Chapter 18.3. There, a missing snoop was the symptom; here, the corrupt sharer set that caused it is the failure. The diagnostic tool is the aggregate of Chapter 17.2 — the global view of the real cache states — now used not to check coherence but to check the directory against reality. So the three fit together: 17.2 aggregates the caches, 18.3 finds a snoop that was skipped, and 18.5 finds why — the directory's record was wrong. This chapter is the deepest of the coherence-bookkeeping debugs, and the source of many otherwise-baffling symptoms.

5. Core Concept — diff the directory against reality

Directory corruption is the directory's record disagreeing with the actual cache states; the diagnosis is to diff the directory's sharer set against an aggregate of the real cache holders, continuously.

• The directory is the home's model. It records which caches hold each line and in what state; the home decides everything from it (snoops, sourcing, grants).
• Corruption is a divergence from reality. The record lists a cache that no longer holds the line, or omits one that does, or has the wrong state — the model diverges from the caches.
• It causes varied symptoms. A wrong sharer set → a missing snoop (a real sharer omitted) or a spurious snoop (a stale entry); a wrong state → stale sourcing.
• Diff against the aggregate. Compare the directory's sharer set against the actual holder set (aggregated from real cache states, Chapter 17.2). The first cycle they disagree is the corruption; the directory update there is the bug.

The synthesis:

Directory corruption is the directory's record (sharer set / line state) disagreeing with the actual cache states. Because the home decides from the directory, it is a root-cause family behind missing snoops, spurious snoops, and stale sourcing. The diagnosis is a continuous diff of the directory's sharer set against an aggregate of the real holders: the first divergence is the corruption, and the directory update at that point — a lost update (16.1) or a skipped back-invalidation (16.3) — is the bug.

6. Engineering Mental Model — a guest registry that drifts from reality

Think of a hotel front desk whose registry (the directory) records which guests are in which rooms — and every service decision uses it.

• The desk routes everything by the registry: deliveries, wake-up calls, evacuations. If the registry is accurate, service is correct.
• Directory corruption is the registry drifting from reality: it lists a guest who checked out (a stale entry), or omits a guest who checked in (a missing entry).
• The symptoms are varied: a delivery sent to an empty room (spurious), or an evacuation notice that skips a real guest (missing) — different surface problems, one cause: the registry is wrong.
• To find it, you walk the floors (aggregate the real occupancy) and compare against the registry. The first room where the registry and reality disagree is the corruption, and the check-in/out log entry for that room is the mistake that caused it.

The registry is the directory; walking the floors is aggregating the real cache states; the diff finds the divergent room; the log entry is the corrupting update. Varied symptoms, one root — found by comparing the record against reality.

7. Engineering Diagram — the directory vs reality

The directory records RN0 as the sharer; the aggregate of real cache states shows RN0 and RN1 hold the line. The comparator's difference is RN1 — present in reality, missing from the directory. That omission is directory corruption, and it would cause RN1's snoop to be skipped (Chapter 18.3). The debugger sees the diff.

8. Waveform Signature

The signature is dir.sharers dropping to RN0 while real.holders stays RN0,1 — the mismatch goes high right after the dir.update that dropped RN1. The corrupting update is at the mismatch's onset; that write (a lost update or skipped back-invalidation) is the bug. A directory record that disagrees with the aggregate of real holders is directory corruption.

9. Diagnosis Path

The methodical diff-and-trace.

The rule to carry: directory corruption is a divergence between a model and reality, so you debug it by holding the two side by side. The home cannot see its own error — its decisions look consistent with its (wrong) record. Only an external aggregate of the real cache states reveals the divergence. The first cycle the diff is nonzero is the corruption, and the directory update at that instant is the bug — almost always a lost update (a concurrent same-line race, Chapter 16.1) or a skipped back-invalidation (an eviction that orphaned a sharer, Chapter 16.3).

10. Tracing Directory Corruption — a worked trace

A snoop is missing (Chapter 18.3) — RN1 was not snooped though it holds the line.

1. Symptom: a missed snoop. RN1 kept a stale copy because a write did not snoop it. The missing-snoop diff (Chapter 18.3) showed RN1 was not in the directory's sharer set.
2. Aggregate reality. Building the actual holder set from cache states shows RN0 and RN1 both hold the line — RN1 really holds it.
3. Read the record. The directory's sharer set is RN0 only — it omits RN1. Directory corruption confirmed: the record disagrees with reality.
4. Diff and find the onset. The diff (directory XOR reality = RN1) first became nonzero at a specific cycle — the corruption onset.
5. Trace the update. At that cycle, two requests to the line were being processed concurrently, and their read-modify-write updates to the directory entry raced — one overwrote the other, dropping RN1 from the sharer set (the Chapter 16.1 bug).
6. Fix. Serialize same-line requests with the address-hazard check (Chapter 16.1) so directory updates are atomic. The record then tracks reality; RN1 stays in the sharer set and is snooped.

The missing snoop traced through the corrupt sharer set to a racing directory update. The DebugLab formalizes this.

11. Detector View — a directory-mismatch detector

Compare the directory's sharer set against the actual holder set; flag any divergence. Representative.

// Representative directory-mismatch detector (educational).
// The directory's recorded sharer set MUST match the ACTUAL set of caches holding the
// line (aggregated from real cache states). Any difference (XOR nonzero) is directory
// corruption: a cache recorded but not holding (stale entry) or holding but not recorded
// (missing entry). Bind it to each line; the first mismatch is the corruption onset.
module chi_dir_mismatch #(parameter NCACHE = 8) (
  input  logic [NCACHE-1:0] dir_sharers,     // directory's recorded sharers
  input  logic [NCACHE-1:0] real_holders,    // actual caches holding the line (aggregate)
  output logic [NCACHE-1:0] diff,             // caches where record != reality
  output logic              mismatch,          // the directory disagrees with reality
  output logic [NCACHE-1:0] stale_entry,       // recorded but not holding
  output logic [NCACHE-1:0] missing_entry      // holding but not recorded
);
  assign diff          = dir_sharers ^ real_holders;   // symmetric difference
  assign mismatch      = (diff != '0);
  assign stale_entry   = dir_sharers & ~real_holders;  // recorded, not holding
  assign missing_entry = real_holders & ~dir_sharers;  // holding, not recorded
endmodule

The same behavior in Verilog-2001:

// Representative directory-mismatch detector (Verilog-2001).
module chi_dir_mismatch #(parameter NCACHE = 8) (
  input  [NCACHE-1:0] dir_sharers, real_holders,
  output [NCACHE-1:0] diff, stale_entry, missing_entry,
  output              mismatch
);
  assign diff          = dir_sharers ^ real_holders;
  assign mismatch      = (diff != {NCACHE{1'b0}});
  assign stale_entry   = dir_sharers & ~real_holders;
  assign missing_entry = real_holders & ~dir_sharers;
endmodule

And in VHDL:

-- Representative directory-mismatch detector (VHDL).
library ieee;
use ieee.std_logic_1164.all;
 
entity chi_dir_mismatch is
  generic ( NCACHE : integer := 8 );
  port (
    dir_sharers   : in  std_logic_vector(NCACHE-1 downto 0);
    real_holders  : in  std_logic_vector(NCACHE-1 downto 0);
    diff          : out std_logic_vector(NCACHE-1 downto 0);
    mismatch      : out std_logic;
    stale_entry   : out std_logic_vector(NCACHE-1 downto 0);
    missing_entry : out std_logic_vector(NCACHE-1 downto 0)
  );
end entity;
 
architecture rtl of chi_dir_mismatch is
  signal d : std_logic_vector(NCACHE-1 downto 0);
begin
  d             <= dir_sharers xor real_holders;
  diff          <= d;
  mismatch      <= '1' when d /= (d'range => '0') else '0';
  stale_entry   <= dir_sharers and (not real_holders);
  missing_entry <= real_holders and (not dir_sharers);
end architecture;

All three diff the directory against reality, flagging mismatch on any difference and splitting it into stale entries (recorded, not holding) and missing entries (holding, not recorded) — which distinguishes the corruption kind. Bind it per line and the first mismatch is the corruption onset. The DebugLab's corruption is a missing entry (RN1 dropped).

12. Assertion View — the directory matches reality

The properties formalize the invariant: the directory's record equals the actual holders.

// Bind to chi_dir_mismatch (with the aggregate of real cache states).
// 1. The directory's sharer set always matches the actual holder set.
property p_directory_matches_reality;
  @(posedge clk) disable iff (!rst_n)
    (dir_sharers == real_holders);
endproperty
 
// 2. No stale entry: the directory never records a cache that is not holding.
property p_no_stale_entry;
  @(posedge clk) disable iff (!rst_n)
    (dir_sharers & ~real_holders) == '0;
endproperty
 
// 3. No missing entry: every actual holder is recorded (else a snoop would be missed).
property p_no_missing_entry;
  @(posedge clk) disable iff (!rst_n)
    (real_holders & ~dir_sharers) == '0;
endproperty

The system point, beyond the checks:

Directory corruption is a model-reality divergence, and its debug illustrates the single most powerful pattern in coherence debugging: maintain an independent aggregate of ground truth and diff the design's model against it. The home's directory is a derived model — it is supposed to be a faithful summary of the cache states, but it is maintained by fallible update logic that can drift. You cannot catch the drift by examining the directory alone, because the directory is self-consistent with the home's decisions (the home believes its own record). Only an external reconstruction of reality — the aggregate of actual cache states, the same one Chapter 17.2 uses for SWMR — exposes the divergence. This is why the aggregate is such a load-bearing verification/debug artifact: it is ground truth against which every derived model (the directory, the trackers) can be checked. The p_no_missing_entry property is the one that matters most, because a missing entry silently causes missed snoops (the highest-severity coherence bug), whereas a stale entry causes merely spurious snoops (wasteful but safe). Distinguishing the two — which the detector does — tells you not just that the directory is wrong but how, and therefore which class of bug it will cause downstream.

• What it proves: the directory matches the actual holders (no stale/missing entries).
• What it does not prove: the aggregate itself is correctly built — a monitor obligation.
• Bug signature: mismatch asserted — missing_entry (→ missed snoops) or stale_entry (→ spurious).

13. Testbench — a dropped sharer must show as a mismatch

Records a sharer set, drops one from the directory, and checks the detector names the missing entry.

module tb_chi_dir_mismatch;
  localparam NCACHE = 8;
  logic [NCACHE-1:0] dir_sharers, real_holders, diff, stale_entry, missing_entry;
  logic mismatch;
  int errors = 0;
 
  chi_dir_mismatch #(.NCACHE(NCACHE)) dut (.*);
 
  initial begin
    // Directory and reality agree: caches 0 and 2 hold the line.
    dir_sharers = 8'b0000_0101; real_holders = 8'b0000_0101; #1;
    if (mismatch) begin errors++; $display("FAIL false mismatch when agreeing"); end
    else $display("PASS directory matches reality");
 
    // CORRUPTION: directory DROPS cache 2 (a lost update); reality still has it.
    dir_sharers = 8'b0000_0001; #1;  // directory now only cache 0
    if (!mismatch) begin errors++; $display("FAIL dropped sharer NOT detected"); end
    else if (missing_entry !== 8'b0000_0100) begin errors++; $display("FAIL wrong missing entry: %b", missing_entry); end
    else $display("PASS mismatch: missing_entry = %b (cache 2 dropped -> missed snoop)", missing_entry);
 
    // A STALE entry: directory lists cache 5 that reality does not hold.
    dir_sharers = 8'b0010_0101; real_holders = 8'b0000_0101; #1;
    if (stale_entry !== 8'b0010_0000) begin errors++; $display("FAIL stale entry not identified: %b", stale_entry); end
    else $display("PASS mismatch: stale_entry = %b (cache 5 -> spurious snoop)", stale_entry);
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS directory matches reality
PASS mismatch: missing_entry = 00000100 (cache 2 dropped -> missed snoop)
PASS mismatch: stale_entry = 00100000 (cache 5 -> spurious snoop)
ALL TESTS PASSED

14. DebugLab — a sharer dropped from the directory by a race

symptom: RN1 not snooped (missing snoop) -> suspect the directory
step 2 - aggregate: real holders = (RN0, RN1)   (RN1 really holds the line)
step 3 - record: directory sharers = (RN0)      (RN1 OMITTED)
step 4 - diff: missing_entry = RN1  -> directory corruption (missing entry)
step 5/6 - onset: mismatch began at a directory UPDATE where two same-line requests
  raced their read-modify-writes; both read old sharers, second write overwrote first,
  DROPPING RN1 (the Chapter 16.1 lost-update bug)
correct: address-hazard check serializes same-line requests -> atomic updates -> RN1 kept

15. Common Mistakes

• Examining the directory alone. Assumption: the record shows the bug. Bug: it is self-consistent. Prevention: diff against real cache states.
• No reality aggregate. Assumption: the directory is ground truth. Bug: it is a derived model. Prevention: aggregate the actual caches.
• Not distinguishing stale vs missing. Assumption: all mismatches alike. Bug: different severities. Prevention: split the diff.
• Debugging the symptom, not the record. Assumption: fix the missing snoop. Bug: the directory is wrong. Prevention: trace to the record.
• Ignoring the corrupting update. Assumption: the mismatch is the bug. Bug: the update caused it. Prevention: trace to the update.
• Confusing with a missing snoop. Assumption: same family. Bug: symptom vs root cause. Prevention: 18.3 is the snoop; 18.5 is the record.

16. Engineering Checklist

• Build an aggregate of the real cache states (Chapter 17.2).
• Diff the directory's sharer set against the actual holder set.
• Split the diff into stale (spurious snoops) and missing (missed snoops) entries.
• Find the first cycle the diff is nonzero — the corruption onset.
• Trace to the directory update at the onset (race or eviction).
• Fix the update — serialize same-line requests (16.1) or back-invalidate (16.3).

17. Key Takeaways

• Directory corruption is the record disagreeing with the actual caches.
• It is a root-cause family behind missing/spurious snoops and stale sourcing.
• The signature is a directory sharer set not matching the real holders.
• Diagnose by diffing the directory against an aggregate of reality.
• The first divergence localizes the corrupting update (race or eviction).
• The directory must match reality; the model here is representative.

18. Quick Revision

Directory corruption. The directory is the home's record of which caches hold each line and in what state, and the home makes every coherence decision from it. Directory corruption is when that record disagrees with the actual caches — the sharer set lists a cache that no longer holds the line (stale entry), or omits one that does (missing entry), or the recorded state is wrong. It is a root-cause family: a wrong sharer set causes missing snoops (Chapter 18.3, from a missing entry) or spurious snoops (from a stale entry), and a wrong state causes stale sourcing. The home cannot see its own error — its decisions are self-consistent with its wrong record — so the diagnosis needs an external ground truth: an aggregate of the real cache states (Chapter 17.2). Diff the directory's sharer set against that aggregate continuously; the first cycle they diverge is the corruption, and the directory update at that instant is the bug — typically a lost update from a concurrent same-line race (Chapter 16.1) or a skipped back-invalidation on eviction (Chapter 16.3). The detector splits the diff into missing entries (→ missed snoops, severe) and stale entries (→ spurious snoops, wasteful), telling you not just that the directory is wrong but how. Maintain an independent aggregate and diff the derived model against it. Representative model; 18.6 debugs deadlock.

Coming Next

Chapter 18.6 — Deadlock. Directory corruption breaks coherence; deadlock stops the fabric entirely. Chapter 18.6 covers deadlock — the signature of a frozen fabric where a set of transactions all wait on one another, the waveform of channels stalled with valid high and ready low, and the diagnosis of reconstructing the wait-for cycle to find the forbidden dependency that closed it.