Data Integrity

Module 13 · Chapter 13.7 · CHI Data Transfers

Project thread — 13.6 handled dirty data. 13.7 handles corrupted data and closes Module 13; Module 14 opens flow control.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Name the three integrity mechanisms — ECC, poison, and the RespErr data-error status.
• Explain that ECC detects and corrects single-bit errors; double-bit errors are detectable, not correctable.
• State that poison marks known-bad data that travels with the data.
• Explain a deferred error — taken at the consumer that uses the poisoned data.
• Diagnose the silent corruption from stripping or ignoring the poison bit.
• Implement a representative poison-propagation model in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

Everything so far assumed the bytes were correct. Real memory and links have errors — cosmic-ray bit flips, failing DRAM cells, marginal signals — and a coherence fabric that ignored them would spread corruption invisibly. Data integrity is how CHI turns an uncontrolled corruption into a contained, reported event: detect it, mark it, and surface it exactly where it matters.

The central mechanism is poison, and it is subtle. When data is corrupted beyond correction, you cannot always drop it — a read still has to complete, and dropping it would hang the requester. So CHI marks the data as poisoned and lets it flow, with the poison bit riding along so the eventual consumer knows the bytes are bad and takes a deferred error rather than acting on garbage. The entire scheme depends on the poison bit propagating on every hop. Strip it — a "helpful" repeater that clears status, a consumer that ignores it — and a detected, contained error becomes a silent, spreading corruption. This chapter closes the module with the mechanism that keeps bad data from being used as good.

3. Key Terms

4. Previous Chapter Connection

Chapters 13.5–13.6 tagged data by freshness — clean vs dirty, which copy is newest. This chapter tags data by validity — whether the bytes are trustworthy at all. They are orthogonal: a line can be dirty and poisoned (the newest copy is also corrupt).

The RespErr field is one you met as a data-packet field in Chapter 13.1; this chapter is its purpose. And the propagation discipline echoes the whole module's theme of self-describing transport: just as a beat carries its DataID (Chapter 13.1) and a forward carries the requester's TxnID (Chapter 13.4), poisoned data carries its poison bit — the packet says "these bytes are bad" so no receiver has to guess. This chapter closes Module 13 by making the data self-describing about its own correctness.

5. Core Concept — mark it, carry it, defer the error

CHI handles corrupt data by detecting it, marking it as poison that travels with the data, and deferring the error to the consumer that uses it.

• ECC detects and corrects. Error-correcting codes fix single-bit errors transparently; a double-bit (uncorrectable) error is detected but cannot be fixed — the data is known bad.
• Poison marks known-bad data. Rather than drop uncorrectable data (which would hang the reader), CHI marks it poisoned and lets it flow — the poison bit says "do not trust these bytes."
• Poison propagates. Every hop carries the poison bit unchanged — a poisoned line stays poisoned through the interconnect, into caches, everywhere it goes.
• The error is deferred. The error is taken at the consumer that actually uses the poisoned data — a deferred error (e.g. a machine-check) — not at the point of detection. So the error surfaces where it matters and stays contained to users of that data.

The synthesis:

CHI keeps corrupt data from being used as good: ECC detects/corrects errors; uncorrectable data is marked poison and not dropped; the poison bit propagates with the data on every hop; and the error is deferred to the consumer that uses the data. So a detected error is contained and reported at the point of use — unless the poison bit is stripped, which turns it into silent, spreading corruption.

6. Engineering Mental Model — a "contaminated" label on a sample

Think of a lab sample that fails a purity test (an uncorrectable error).

• You cannot always discard it — downstream steps are expecting a sample, and a missing sample stalls the pipeline. So instead you label it "CONTAMINATED — do not use in results." That label is poison.
• The label stays on the vial through every transfer between benches. Whoever ends up with it can read the label and knows not to trust it.
• The technician who would actually use the sample in a result reads the label and flags the result as invalid instead — the error is caught at the point of use (deferred), and only results that used this sample are flagged (contained).
• Now imagine a tidy assistant who peels the label off to make the vial look clean. The contaminated sample now looks fine, gets used in a result, and the bad result is reported as valid — the contamination has gone silent and spread into conclusions, with nobody the wiser.

The label must ride the vial and be honored at use. Poison is that label; stripping it is the disaster.

7. Engineering Diagram — the integrity fields

Four fields govern integrity. ECC guards the bytes; poison marks them known-bad when ECC cannot correct; RespErr reports the status. The poison field is the one that must survive every hop — the diagram's fields are inert unless poison is carried and honored.

8. The Integrity Mechanisms Compared

Each mechanism and its role.

The rule to carry: correct what you can, mark what you cannot, and never let bad data pass as good. ECC handles the common single-bit case invisibly. When it cannot correct — a double-bit error — the data is known bad, and the choice is drop (hang the reader) or mark and defer. CHI marks and defers: poison lets the data flow while guaranteeing that anyone who uses it is warned. RespErr is the reporting channel that carries the data-error status alongside.

9. Propagation and Deferral

The two properties that make poison work.

• Propagate on every hop. The poison bit is carried unchanged wherever the data goes — through the interconnect, into a cache, forwarded to a peer. A poisoned line is poisoned everywhere.
• Do not drop, do not silently use. Poisoned data is not dropped (that would hang a reader) and not silently consumed (that would use garbage). It flows, marked.
• Defer to the consumer. The error is taken where the data is used — a load that consumes poisoned bytes raises a deferred error (machine-check/abort). Detection was upstream; the error is at use.
• Containment. Because the error is deferred to use, only transactions that actually use the poisoned line take an error — the fault does not fire for data that is merely passed through and never consumed.

The point to carry:

Poison inverts the usual instinct about errors, which is to stop at the point of detection. Here, stopping at detection is wrong: the memory controller that detects an uncorrectable error does not know whether the reader will actually use the bad bytes — it might be a speculative fetch, a prefetch, a cache-line fill that gets invalidated before use. Firing a fatal error at detection would kill the machine for data nobody ever consumes. So CHI defers: mark the data and let the point of use decide, because only the consumer knows whether the bad bytes matter. This is precise error handling — the error fires exactly when corruption would affect a result, and never for corruption that is detected but harmless. The price of that precision is a strict obligation: the poison bit must survive every hop, because a deferred error is only possible if the mark reaches the deferral point.

10. A Poisoned Read — detect, mark, defer

Memory returns an uncorrectable line; the poison rides it to the consumer.

1. ECC detects a double-bit error. The memory controller reads a line and ECC finds an uncorrectable error — the bytes are known bad.
2. The data is marked poison. Rather than drop it, the controller sets the poison indication and returns the data with RespErr = data error.
3. Poison propagates. The home forwards the completion to the requester with poison set; the requester installs the line still marked poisoned.
4. A load uses the line. When the core executes a load that consumes the poisoned bytes, it takes a deferred error — a machine-check — instead of using garbage.
5. Contained. Only the load that used the line faults; a prefetch that installed the line but was never consumed causes no error.

The corruption was caught at use and contained because the poison bit survived every hop. The DebugLab is a hop that strips the poison, so step 4 silently uses garbage.

11. RTL / Hardware View — poison propagation

Poison is set on an uncorrectable error, carried unchanged through every hop, and forces a deferred error at use. Representative.

// Representative poison-propagation model (educational).
// poison_out is set if the data arrives poisoned OR ECC finds an uncorrectable error.
// Poison is NEVER cleared on a forward. A consumer that USES poisoned data must raise a
// deferred error rather than consume the bytes. Stripping poison = silent corruption.
module chi_poison (
  input  logic  poison_in,        // data arrived already poisoned
  input  logic  ecc_uncorrectable,// ECC detected an uncorrectable error here
  input  logic  data_used,        // a consumer is actually using the bytes
  output logic  poison_out,       // poison carried forward (never cleared)
  output logic  deferred_error    // consumer must take a deferred error
);
  // Poison accumulates and propagates -- OR of incoming and newly-detected. Never cleared.
  assign poison_out    = poison_in | ecc_uncorrectable;
  // The error is deferred to the point of USE.
  assign deferred_error = data_used & poison_out;
endmodule

The same behavior in Verilog-2001:

// Representative poison-propagation model (Verilog-2001).
module chi_poison (
  input  wire  poison_in, ecc_uncorrectable, data_used,
  output wire  poison_out,
  output wire  deferred_error
);
  assign poison_out     = poison_in | ecc_uncorrectable;  // propagate, never clear
  assign deferred_error = data_used & poison_out;         // error at point of use
endmodule

And in VHDL:

-- Representative poison-propagation model (VHDL).
library ieee;
use ieee.std_logic_1164.all;
 
entity chi_poison is
  port (
    poison_in         : in  std_logic;
    ecc_uncorrectable : in  std_logic;
    data_used         : in  std_logic;
    poison_out        : out std_logic;
    deferred_error    : out std_logic
  );
end entity;
 
architecture rtl of chi_poison is
begin
  poison_out     <= poison_in or ecc_uncorrectable;   -- propagate, never clear
  deferred_error <= data_used and (poison_in or ecc_uncorrectable);
end architecture;

All three compute poison_out as the OR of incoming poison and newly-detected uncorrectable error — poison is only ever added, never cleared — and raise deferred_error only when poisoned data is used. The DebugLab is a hop that drops the poison_in term.

12. Verification View — poison is never lost

The properties that keep integrity sound: poison propagates, and use of poison faults.

// Bind to chi_poison. Same-cycle invariants -> immediate assertions in always_comb.
always_comb begin
  // 1. Incoming poison is never cleared -- it always propagates.
  p_poison_propagates:    assert (!poison_in || poison_out);
  // 2. A newly-detected uncorrectable error poisons the output.
  p_ecc_poisons:          assert (!ecc_uncorrectable || poison_out);
  // 3. Using poisoned data always raises a deferred error (never silent).
  p_use_of_poison_faults: assert (!(data_used && poison_out) || deferred_error);
  // 4. No error is raised for clean data (containment: only poison faults).
  p_no_false_error:       assert (poison_out || !deferred_error);
end

The system point, beyond the checks:

Poison propagation has a monotonicity property that is the whole safety argument: poison_out is a pure OR of "arrived poisoned" and "newly corrupt," so poison can only ever be added along a path, never removed. That monotonicity is what guarantees a poisoned value cannot launder itself clean by passing through enough hops — every node either leaves the bit set or sets it, none clears it. The instant one node breaks monotonicity by clearing the bit — an optimization that "resets status," a buffer that forgets it, a consumer that masks it — the safety argument collapses for every path through that node, because a poisoned value can now reach a consumer looking clean. This is why data integrity is a property of the whole fabric, not any single component: it holds only if every hop preserves the monotonic bit. One forgetful repeater silently defeats ECC everywhere downstream of it.

• What it proves: poison always propagates; using poison always faults; clean data never faults.
• What it does not prove: ECC's detection coverage — that is the code's Hamming distance, a separate design choice.
• Bug signature: poison_in set but poison_out clear — a hop that laundered the poison.

13. Testbench — poison must survive and fault on use

Drives poisoned and clean data and checks propagation and deferral.

module tb_chi_poison;
  logic poison_in, ecc_uncorrectable, data_used;
  logic poison_out, deferred_error;
  int errors = 0;
 
  chi_poison dut (.*);
 
  task check(input logic pin, ecc, used, input logic exp_pout, exp_err, input string nm);
    begin
      poison_in = pin; ecc_uncorrectable = ecc; data_used = used; #1;
      if (poison_out !== exp_pout || deferred_error !== exp_err) begin
        errors++; $display("FAIL %s: pout=%0b err=%0b exp %0b/%0b", nm, poison_out, deferred_error, exp_pout, exp_err);
      end else $display("PASS %s: pout=%0b err=%0b", nm, poison_out, deferred_error);
    end
  endtask
 
  initial begin
    check(0, 0, 1, 0, 0, "clean data used -> no error");
    check(1, 0, 0, 1, 0, "poison passed through, not used -> propagate, no error");
    check(1, 0, 1, 1, 1, "poison used -> deferred error");
    check(0, 1, 1, 1, 1, "ECC uncorrectable + used -> poison + error");
    // the critical case: incoming poison must NEVER be cleared
    poison_in = 1; ecc_uncorrectable = 0; data_used = 0; #1;
    if (!poison_out) begin errors++; $display("FAIL poison was cleared on forward"); end
    else $display("PASS poison survived the hop");
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS clean data used -> no error: pout=0 err=0
PASS poison passed through, not used -> propagate, no error: pout=1 err=0
PASS poison used -> deferred error: pout=1 err=1
PASS ECC uncorrectable + used -> poison + error: pout=1 err=1
PASS poison survived the hop
ALL TESTS PASSED

14. DebugLab — stripping the poison bit on a forward

memory: ECC detects uncorrectable error -> data marked POISON, RespErr = data error
buggy hop: forwards the data but drops the poison bit (poison_out = 0)
  -> downstream consumer sees CLEAN data (poison not set)
  -> load consumes corrupted bytes, NO deferred error
  -> garbage flows into results; corruption spreads; no machine-check
correct: poison_out = poison_in | ecc_uncorrectable  (never cleared)
  -> consumer sees poison -> deferred error at use -> contained

15. Common Mistakes

• Clearing poison on a forward. Assumption: status can be reset. Bug: silent corruption (the DebugLab). Prevention: propagate unchanged.
• Ignoring poison at the consumer. Assumption: data is valid. Bug: garbage used, no error. Prevention: fault on use of poison.
• Dropping poisoned data. Assumption: bad data should vanish. Bug: the reader hangs. Prevention: mark and flow, do not drop.
• Faulting at detection, not use. Assumption: error where found. Bug: killing the machine for unused data. Prevention: defer to use.
• Confusing poison with dirty. Assumption: one status bit. Bug: freshness/validity conflated. Prevention: they are orthogonal (13.6 vs 13.7).
• Trusting ECC to correct everything. Assumption: all errors correctable. Bug: double-bit errors used. Prevention: uncorrectable → poison.

16. Engineering Checklist

• Use ECC to detect and correct — single-bit corrected, double-bit → poison.
• Mark uncorrectable data poison; do not drop it.
• Propagate the poison bit unchanged on every hop.
• Carry the data-error status in RespErr.
• Raise a deferred error when a consumer uses poisoned data.
• Confirm poison is never cleared — propagation is monotonic.

17. Key Takeaways

• Data can be wrong; CHI detects it (ECC), marks it (poison), and reports it (RespErr).
• ECC corrects single-bit errors; double-bit errors are uncorrectable → poison.
• Poison marks known-bad data and travels with it — not dropped, not silently used.
• The error is deferred to the consumer that uses the data — precise and contained.
• Stripping the poison bit turns a contained error into silent, spreading corruption.
• Propagation is monotonic — poison is only added; the model here is representative.

18. Quick Revision

Data integrity. Data can be corrupted, not just clean or dirty, and CHI handles it with three mechanisms. ECC detects and corrects single-bit errors transparently; a double-bit error is detectable but uncorrectable. Uncorrectable data is not dropped (that would hang the reader) — it is marked poison, a bit that says "known bad, do not trust." Poison propagates unchanged on every hop — through the interconnect, into caches, forwarded to peers — and the RespErr field carries the data-error status. The error is deferred: it is taken at the consumer that actually uses the poisoned bytes (a machine-check), not at detection, so it fires exactly where corruption would affect a result and stays contained — data merely passed through and never used causes no error. The safety rests on monotonicity: poison is only ever added, never cleared, so corrupt data cannot pass as clean. The failure to avoid: a hop that strips the poison bit (or a consumer that ignores it) breaks monotonicity — known-bad data then looks clean, is used silently, and a contained error becomes silent, spreading corruption with no machine-check. Propagate the bit; fault on use. Representative model; this closes Module 13.

Coming Next

Chapter 14.1 — The Credit Mechanism. Module 13 moved data correctly; the next module ensures the channels carrying it never overflow. Chapter 14.1 opens CHI Flow Control with the fundamental — the link-credit (L-credit) mechanism: how a sender may transmit a flit only when it holds a credit, how a receiver issues credits equal to its free buffer slots, and why this credit accounting guarantees a receiver's buffer can never overflow and drop a packet.