Snoop Data Transfer

Module 9 · Chapter 9.5 · Snoop Flows

Project thread — 9.4 cleared the other copies via invalidation. This chapter sources the data from a peer cache. 9.6 moves ownership between caches via snoop.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Describe cache-to-cache transfer: sourcing the requested data from a peer cache, not memory.
• Explain the performance win — a holding cache is closer than memory.
• State the correctness rule: a dirty line's data must come from the cache, never memory.
• Trace the home-mediated path — holder → home (SnpRespData) → requester (CompData).
• Diagnose why sourcing a dirty line from memory returns stale data.
• Implement a representative data-source selector in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

Where the data comes from decides both speed and correctness. In a multi-core system, the line a requester wants is very often already sitting in another core's cache — closer, and much faster to reach, than main memory. Sourcing it cache-to-cache is one of the biggest performance levers in a coherent fabric: it cuts latency and takes load off the memory controller. If you care about how a coherent system actually performs, this is where a lot of the win lives.

But the same mechanism is load-bearing for correctness. When the holding cache has the line dirty, memory does not have the latest value — it is stale by definition. So for a dirty line, cache-to-cache is not an optimization, it is the only correct source. Understanding snoop data transfer means understanding both the opportunity and the obligation: source from the cache to go fast, and source from the cache because for dirty data there is nowhere else valid.

3. Key Terms

4. Previous Chapter Connection

Chapter 9.3 introduced SnpRespData — a snoop response that returns the holder's line. Chapter 9.4 made the writer wait for invalidations. This chapter puts SnpRespData to a different purpose: sourcing the data for the requester from the cache that returned it.

You have seen this data path in flows already — a dirty holder's data forwarded through the home to the requester (Chapters 7.4, 8.2). This chapter names it as a first-class choice: for any request, the home decides whether to source the line from a cache (via the snoop response) or from memory (a read). The snoop response is not only a state report; it is a data-delivery channel, and choosing it over memory is both a performance decision and, for dirty lines, a correctness one.

5. Core Concept — source from the cache, and for dirty data you must

When a request needs a line's data, the home has two possible sources — a cache that holds it, or memory — and it chooses by what the directory shows.

• A holder can supply the data. If a cache holds a valid copy, the home can snoop it and take the returned SnpRespData as the requester's data — a cache-to-cache transfer.
• Cache is faster. A peer cache is closer than main memory, so sourcing from it cuts latency and saves memory bandwidth. CHI sources from the cache whenever a holder has valid data.
• For dirty, cache is mandatory. If the holder has the line dirty, memory holds the stale pre-modification value. The latest data exists only in that cache, so it must be the source — memory cannot be.
• Home-mediated path. The data flows holder → home (SnpRespData) → requester (CompData). The home never needs a memory read when a cache supplies the line. (Direct holder-to-requester forwarding is Chapter 9.7.)

The synthesis:

When a cache holds the line, the home sources the requester's data from that cache — a cache-to-cache transfer via SnpRespData, forwarded as CompData. It is faster than memory for any line, and for a dirty line it is the only correct source, because memory is stale. Source from the cache to go fast; source from the cache because, when the data is dirty, there is nowhere else valid.

6. Engineering Mental Model — borrow from a colleague, not the archive

You need a document, and the coordinator (home) can get it two ways.

• The archive (memory) always has a copy — but it is far away, slow to retrieve, and everyone queues for it.
• A colleague nearby (a peer cache) may already have the document on their desk — much faster to grab.
• So the coordinator, seeing a colleague holds it, borrows from the colleague instead of trekking to the archive. Quicker for you, and no queue at the archive.
• Crucially, if the colleague has edited their copy (dirty), the archive's version is out of date — it never saw the edits. Now borrowing from the colleague is not just faster; it is the only way to get the current document. Fetch from the archive and you would hand over a stale version.

Nearby-and-current beats far-and-maybe-stale. For an unedited copy it is a speed choice; for an edited one it is the only correct choice.

7. Engineering Diagram — a cache-to-cache transfer

Read what is not there: no memory read. The line went RN1 → HN → RN0, entirely through caches and the fabric. For a clean line that is a speed win; for a dirty line it is the only way RN0 gets the current value.

8. Cache versus Memory as the Source

The home's source decision, by what the directory shows.

The rule to carry: source from the cache whenever a holder has the line — for clean lines because it is faster, for dirty lines because it is mandatory. Only when no cache holds the line does the home read memory. The dirty row is the sharp one: memory is not merely slower there, it is wrong.

9. Why Dirty Data Must Come From the Cache

The correctness rule deserves its own statement.

• Dirty means memory is behind. A dirty line (UD/SD) has been modified in a cache and not written back. Memory still holds the old value.
• So memory is not a valid source. Reading memory for a dirty line returns the pre-modification data — a stale read, a coherence violation.
• The cache is the only holder of the latest. The current value exists only in the modifying cache until it is written back. Cache-to-cache is the sole path to it.
• PassDirty travels with it. When the cache supplies dirty data, the response sets PassDirty (Chapter 9.3), so the writeback responsibility moves with the data — the value is preserved as well as delivered.

The point to carry:

For a dirty line, cache-to-cache is not optional — it is the definition of correct. The latest value lives in exactly one place, the modifying cache, and memory is by construction behind. So the data source is not a free policy choice: the home must route a dirty line's data from the cache that holds it, or it delivers the past. The performance framing (cache is faster) applies to clean lines; the correctness framing (cache is the only source) applies to dirty ones — and conflating them, by treating memory as always-valid, is exactly the bug.

10. Walkthrough — cache-to-cache on a dirty line

RN0 issues ReadShared; RN1 holds the line in UD (dirty). Memory holds the old value.

1. REQ. RN0 needs the line's data.
2. Directory lookup. HN sees RN1 holds it dirty — so memory is stale. The data must come from RN1.
3. Snoop. HN sends SnpShared to RN1. RN1 downgrades UD → SC and returns the line as SnpRespData, with PassDirty (it is giving up dirty ownership).
4. Source from cache. HN takes RN1's returned data as the source — it does not read memory (which is stale). It forwards CompData to RN0 and writes the dirty data back to memory as required.
5. RN0 installs. RN0 installs the line SC with the current value — the one RN1 had modified.

RN0 got the latest data, sourced from RN1's cache, faster than a memory read and — because RN1 was dirty — the only correct source. Had HN read memory in step 4, RN0 would have received the stale old value.

11. RTL / Hardware View — a data-source selector

The home's decision is combinational: source from a cache if a holder has the line, else from memory — and flag that memory is stale when a holder is dirty. Representative.

// Representative snoop data-source selector (educational).
// Source the requested line from a CACHE whenever a holder has valid data (faster,
// and REQUIRED when dirty because memory is stale). Only when no cache holds the
// line is memory the source. memory_stale flags that a dirty cache copy exists.
module chi_snoop_data_source (
  input  logic holder_present,   // a cache holds a valid copy of the line
  input  logic holder_dirty,     // that cache holds the line dirty
  output logic src_cache,        // source data from the snooped cache
  output logic src_memory,       // source data from memory
  output logic memory_stale      // memory does NOT hold the latest (dirty in cache)
);
  // A dirty cache copy means memory is behind.
  assign memory_stale = holder_dirty;
  // Prefer the cache when a holder has the data (faster; mandatory if dirty).
  assign src_cache    = holder_present;
  // Only read memory when no cache holds the line.
  assign src_memory   = !holder_present;
endmodule

The same behavior in Verilog-2001:

// Representative snoop data-source selector (Verilog-2001).
module chi_snoop_data_source (
  input  holder_present,
  input  holder_dirty,
  output src_cache,
  output src_memory,
  output memory_stale
);
  assign memory_stale = holder_dirty;
  assign src_cache    = holder_present;
  assign src_memory   = !holder_present;
endmodule

And in VHDL:

-- Representative snoop data-source selector (VHDL).
library ieee;
use ieee.std_logic_1164.all;
 
entity chi_snoop_data_source is
  port (
    holder_present : in  std_logic;
    holder_dirty   : in  std_logic;
    src_cache      : out std_logic;
    src_memory     : out std_logic;
    memory_stale   : out std_logic
  );
end entity;
 
architecture rtl of chi_snoop_data_source is
begin
  memory_stale <= holder_dirty;
  src_cache    <= holder_present;
  src_memory   <= not holder_present;
end architecture;

All three source from the cache whenever a holder has the line and from memory only when none does — so a dirty line (which implies a holder) is always sourced from the cache. The DebugLab shows the stale read when memory is used for a dirty line.

12. Verification View — dirty from cache, exactly one source

The properties that keep sourcing correct: a dirty line comes from the cache, and exactly one source is chosen.

// Bind to chi_snoop_data_source.
// 1. A dirty line MUST be sourced from the cache — never memory (memory is stale).
property p_dirty_from_cache;
  @(*) holder_dirty |-> (src_cache && !src_memory);
endproperty
 
// 2. Exactly one source is selected — cache xor memory.
property p_one_source;
  @(*) src_cache ^ src_memory;
endproperty
 
// 3. memory_stale is asserted exactly when a cache holds the line dirty.
property p_memory_stale_iff_dirty;
  @(*) memory_stale == holder_dirty;
endproperty

The system point, beyond the checks:

The data source is where performance and correctness share a single decision. The performance story — a cache is nearer than memory — makes cache-to-cache the preferred source for any line a holder has. The correctness story — memory lags a dirty cache — makes it the required source for a dirty line. They point the same way, so a design that always prefers the cache when a holder exists is both fast and correct for free. The danger is only in the exception: a policy that reaches for memory as a "simpler" or "authoritative" source forgets that memory is authoritative only when no cache holds the line dirty. Authority over a line's value belongs to whoever most recently wrote it — and that is the cache, until writeback.

• What it proves: dirty lines source from the cache, exactly one source, memory_stale tracks dirtiness.
• What it does not prove: the cache actually returned correct data — that is the response (Chapter 9.3).
• Bug signature: src_memory asserted while holder_dirty — a stale value delivered.

13. Testbench — the source decision across cases

Drives the holder presence and dirtiness and checks the source and staleness.

module tb_chi_snoop_data_source;
  logic holder_present, holder_dirty, src_cache, src_memory, memory_stale;
  int errors = 0;
 
  chi_snoop_data_source dut (.*);
 
  task automatic check(input logic hp, hd,
                       input logic exp_cache, exp_mem, exp_stale, input string name);
    holder_present = hp; holder_dirty = hd; #1;
    if (src_cache !== exp_cache || src_memory !== exp_mem || memory_stale !== exp_stale) begin
      errors++; $display("FAIL %s: cache=%0b mem=%0b stale=%0b",
                         name, src_cache, src_memory, memory_stale);
    end else $display("PASS %s: cache=%0b mem=%0b stale=%0b",
                      name, src_cache, src_memory, memory_stale);
  endtask
 
  initial begin
    check(1'b0, 1'b0, 1'b0, 1'b1, 1'b0, "no holder        -> memory");
    check(1'b1, 1'b0, 1'b1, 1'b0, 1'b0, "holder clean     -> cache (faster)");
    check(1'b1, 1'b1, 1'b1, 1'b0, 1'b1, "holder dirty     -> cache (required)");
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS no holder        -> memory: cache=0 mem=1 stale=0
PASS holder clean     -> cache (faster): cache=1 mem=0 stale=0
PASS holder dirty     -> cache (required): cache=1 mem=0 stale=1
ALL TESTS PASSED

14. DebugLab — sourcing dirty data from memory

RN1 = UD (value = NEW),  memory = OLD  (not written back)
RN0 ReadShared -> HN snoops RN1 (gets NEW via SnpRespData)
HN sources the requester's data from MEMORY (= OLD)  <-- wrong for a dirty line
HN forwards CompData(OLD) to RN0
RN0 installs OLD -> read returns OLD, not NEW  -> COHERENCE VIOLATION

15. Common Mistakes

• Sourcing dirty data from memory. Assumption: memory is authoritative. Bug: stale read (the DebugLab). Prevention: dirty lines source from the cache.
• Always reading memory. Assumption: memory is the single source. Bug: slow, and wrong for dirty. Prevention: prefer the holding cache.
• Racing a redundant memory read. Assumption: read memory alongside the snoop. Bug: wasted bandwidth, possible stale use. Prevention: use the cache-sourced data when a holder exists.
• Dropping PassDirty on cache-sourced dirty data. Assumption: forwarding suffices. Bug: ownerless dirty line (Chapter 9.3). Prevention: carry PassDirty with the data.
• Ignoring the performance win for clean lines. Assumption: always go to memory. Bug: needless latency. Prevention: cache-to-cache for clean too.
• Assuming memory is current. Assumption: memory always has the latest. Bug: it lags dirty caches. Prevention: check dirtiness before choosing memory.

16. Engineering Checklist

• Source the line from a cache whenever a holder has valid data.
• For a dirty holder, source from the cache — memory is stale.
• Read memory only when no cache holds the line.
• Route cache-sourced data holder → home → requester (SnpRespData → CompData).
• Carry PassDirty with cache-sourced dirty data (Chapter 9.3).
• Use cache-to-cache for clean lines too — it is faster than memory.

17. Key Takeaways

• Cache-to-cache transfer sources a line's data from a peer cache via the snoop response, not memory.
• It is faster for any line — a holding cache is closer than memory and saves memory bandwidth.
• For a dirty line it is mandatory — memory is stale, so the cache is the only valid source.
• The data flows holder → home (SnpRespData) → requester (CompData); memory need not be read.
• PassDirty travels with cache-sourced dirty data so ownership moves with it.
• Prefer the cache, require it for dirty, read memory only when no holder exists; the model here is representative.

18. Quick Revision

Snoop data transfer. When a cache holds the line a requester needs, the home sources the data from that cache — a cache-to-cache transfer via SnpRespData, forwarded to the requester as CompData — instead of reading memory. Two reasons: performance, because a peer cache is closer than memory and saves memory bandwidth (so CHI prefers the cache for any line a holder has); and correctness, because a dirty holder means memory is stale, so the cache is the only valid source — reading memory would deliver the old pre-modification value. The data flows holder → home → requester, and PassDirty travels with dirty data so ownership moves too. Only when no cache holds the line does the home read memory. The trap: sourcing a dirty line from memory, treating memory as always-authoritative — it is authoritative only when no cache holds the line dirty, so the read returns stale data. Prefer the cache; require it for dirty; memory only when there is no holder. Representative model; 9.6 covers snoop ownership transfer.

Coming Next

Chapter 9.6 — Snoop Ownership Transfer. Data moved cache-to-cache; the next chapter moves ownership the same way. Chapter 9.6 follows how a snoop transfers the dirty-owner role from one Request Node to another — not just copying the data, but handing off the responsibility to write it back — building on the PassDirty bit to show how ownership migrates across caches without ever passing through memory, and the invariant that exactly one owner exists throughout.