Data Sources

Module 13 · Chapter 13.2 · CHI Data Transfers

Project thread — 13.1 framed the data packet. 13.2 asks where the data comes from; 13.3 asks where it goes.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Name the three data sources — a peer RN cache, the home's cache (SLC), and memory (SN).
• State the selection rule — source from wherever the current copy of the line lives.
• Explain why a close source (peer, SLC) is preferred over memory when it holds the line.
• Describe the system-level cache as a distinct source between the peers and memory.
• Diagnose the stale/slow bug from never consulting the SLC and always fetching memory.
• Implement a representative source selector in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

A read has to be satisfied from some copy of the line, and choosing the right one is a correctness and performance decision. Choose a source that holds current data and the read is correct and fast; choose memory when a cache holds a newer copy and the read is stale. The home node makes this choice on every read, so understanding the sources is understanding how CHI turns a read request into the right bytes.

The stakes rise once the home has its own cache — a system-level cache (SLC) sitting between the peer caches and memory. The SLC is a distinct source, and a design that has one but never consults it defeats its purpose: it re-reads memory for lines the SLC already holds (wasting bandwidth) and, when the SLC holds a dirty copy newer than memory, it reads memory and serves stale data. This chapter enumerates the three sources and the rule that picks among them, so you source data from where the current copy actually lives.

3. Key Terms

4. Previous Chapter Connection

Chapter 13.1 framed the data packet; this chapter asks which producer fills it. You have already seen sources individually: a peer cache forwards data (Chapter 9.5, 9.7), and memory supplies clean data when no cache has it (Chapter 9.5). This chapter enumerates them and adds the third — the home's own system-level cache.

The selection rule is the coherence principle from Module 10 applied to sourcing: the current copy of a line may be dirty in a cache (state UD/SD) and therefore newer than memory, so the source must be whichever holds that current copy (Chapter 10.5). This chapter turns that principle into a concrete source choice the home makes on every read, with the SLC as a new source in the middle.

5. Core Concept — three sources, current copy wins

A read can be supplied from three sources, and the home selects the one holding the current copy, closest first.

• Peer RN cache. Another RN may hold the line, possibly dirty (state UD/SD). It is the closest source and, when dirty, the only one with current data. It supplies by forwarding (Chapter 9.7).
• System-level cache (SLC). The home node may have its own cache. If the SLC holds the line, it can supply without touching memory — faster than a memory read, and it may hold a dirty copy newer than memory.
• Memory (SN). The backing store always has a copy, but possibly a stale one if a cache holds the line dirty. It is the source of last resort — used only when no cache holds the current data.
• The rule. Source from wherever the current copy lives, preferring a close source (peer, SLC) over memory. Memory is correct only when it is the current copy — no cache holds it dirty.

The synthesis:

A read is supplied from a peer RN's cache, the home's system-level cache, or memory — and the home selects the source holding the current copy, preferring the closer source. A peer holding the line dirty forwards it; an SLC holding the line supplies without a memory read; memory is used only when no cache holds the current data. Choosing memory while a cache holds a newer copy serves stale data.

6. Engineering Mental Model — asking the nearest person who knows

Think of finding out a fact that might have just changed.

• The most recent version of the fact might be in a colleague's head (a peer cache) — they just updated it and haven't written it down yet. Ask them: they have the truth, and they are right next to you.
• A shared team notebook on your desk (the SLC) might have it — closer than the archive, and possibly more current than the archive if someone wrote to the notebook recently.
• The central archive (memory) always has a version, but maybe an old one — nobody has filed the latest update yet.

You ask the nearest source that has the current answer: the colleague if they just changed it, the desk notebook if it is there, the archive only if neither has it. Going straight to the archive every time is slow, and if the archive is out of date you get a wrong answer. The home node reasons the same way about where a line's current data lives.

7. Engineering Diagram — three sources into the requester

Three sources converge on the home, which selects one and delivers to the requester. The peer and SLC are the close sources; memory is the fallback. The home's job is to pick whichever holds the current copy.

8. The Three Sources Compared

Each source, its data, and when it is chosen.

The rule to carry: current copy, closest first. A dirty peer is both the closest and the only current copy, so it wins. An SLC hit avoids a memory round-trip and may itself be newer than memory. Memory is correct only when it is the current copy — that is, when no cache holds the line dirty. The distance order (peer, SLC, memory) is also the latency order, so preferring close sources is both correct and fast.

9. Selecting the Source

The selection logic, stated as a priority.

• Dirty peer first. If a peer holds the line dirty, it has the only current copy — source from it (forward). Memory and a clean SLC would both be stale.
• SLC next. If no peer is dirty but the SLC holds the line, source from the SLC — no memory read needed, and the SLC's copy is current (it tracks writes).
• Memory last. Only when no cache holds the current line does the home read memory. Memory is then, by definition, the current copy.
• Never skip a current cache. The home must consult the caches (peer via snoop, SLC via lookup) before falling to memory — skipping them risks stale data and wastes bandwidth.

The point to carry:

Source selection is a freshness decision disguised as a routing decision. The three sources are not interchangeable copies — they can disagree, because a cache can hold a line the writer updated but has not written back to memory. So the home cannot pick a source by convenience (e.g. "always memory, it is simplest"); it must pick by which copy is current, and the current copy is wherever the most recent write landed. Coherence (Module 10) is what tells the home where that is: the directory and snoops locate the dirty copy. Source selection is the point where that coherence information becomes an actual data path — and short-circuiting it, by not consulting a cache that might hold the current copy, reintroduces exactly the stale-data bug coherence exists to prevent.

10. Sourcing a Read — three scenarios

The same read, resolved from three different sources depending on where the current copy is.

1. A peer holds it dirty. The home snoops, the peer forwards its dirty data (Chapter 9.7). Source = peer cache. Memory is stale and untouched for the data.
2. No dirty peer, SLC hit. No peer holds the line dirty, but the home's SLC has it. The home supplies from the SLC — no memory read. Fast, and current.
3. No cache has it. Neither a peer nor the SLC holds the line. The home reads memory (SN). Source = memory, which is the current copy because no cache holds it.

In every case the home sourced from the current copy, closest first. The DebugLab is a home that has an SLC but jumps straight to scenario 3's memory read even when scenario 2 (or a dirty peer) applies.

11. RTL / Hardware View — a source selector

The home picks the source holding the current copy: a dirty peer first, then the SLC, then memory. Representative — the priority encoder for the choice.

// Representative data-source selector (educational).
// Priority: a peer holding the line DIRTY has the only current copy -> forward.
// Else if the system-level cache (SLC) holds the line -> supply from SLC (no memory).
// Else read memory (SN). Never skip a cache that holds the current copy.
typedef enum logic [1:0] { SRC_PEER, SRC_SLC, SRC_MEM } src_e;
 
module chi_source_select (
  input  logic  peer_has,     // a peer holds the line
  input  logic  peer_dirty,   // ... and holds it dirty (only current copy)
  input  logic  slc_has,      // the home's SLC holds the line
  output src_e  source,       // chosen source
  output logic  reads_memory  // asserted only when memory is used
);
  always_comb begin
    if (peer_dirty)      source = SRC_PEER;  // dirty peer: only current copy
    else if (slc_has)    source = SRC_SLC;   // SLC hit: no memory read
    else if (peer_has)   source = SRC_PEER;  // clean peer can still forward
    else                 source = SRC_MEM;   // no cache has it: memory
  end
  assign reads_memory = (source == SRC_MEM);
endmodule

The same behavior in Verilog-2001:

// Representative data-source selector (Verilog-2001).
module chi_source_select (
  input  wire       peer_has, peer_dirty, slc_has,
  output reg  [1:0] source,        // 0=PEER, 1=SLC, 2=MEM
  output wire       reads_memory
);
  localparam SRC_PEER = 2'd0, SRC_SLC = 2'd1, SRC_MEM = 2'd2;
  always @* begin
    if (peer_dirty)     source = SRC_PEER;
    else if (slc_has)   source = SRC_SLC;
    else if (peer_has)  source = SRC_PEER;
    else                source = SRC_MEM;
  end
  assign reads_memory = (source == SRC_MEM);
endmodule

And in VHDL:

-- Representative data-source selector (VHDL).
library ieee;
use ieee.std_logic_1164.all;
 
entity chi_source_select is
  port (
    peer_has, peer_dirty, slc_has : in  std_logic;
    source                        : out std_logic_vector(1 downto 0); -- 00=PEER 01=SLC 10=MEM
    reads_memory                  : out std_logic
  );
end entity;
 
architecture rtl of chi_source_select is
  constant SRC_PEER : std_logic_vector(1 downto 0) := "00";
  constant SRC_SLC  : std_logic_vector(1 downto 0) := "01";
  constant SRC_MEM  : std_logic_vector(1 downto 0) := "10";
  signal sel : std_logic_vector(1 downto 0);
begin
  process (peer_has, peer_dirty, slc_has)
  begin
    if peer_dirty = '1' then
      sel <= SRC_PEER;         -- dirty peer: only current copy
    elsif slc_has = '1' then
      sel <= SRC_SLC;          -- SLC hit: no memory read
    elsif peer_has = '1' then
      sel <= SRC_PEER;         -- clean peer forwards
    else
      sel <= SRC_MEM;          -- memory: last resort
    end if;
  end process;
  source       <= sel;
  reads_memory <= '1' when sel = SRC_MEM else '0';
end architecture;

All three consult the caches first and fall to memory only when no cache holds the line — reads_memory is high solely in the final else. The DebugLab is a selector that ignores slc_has and reads memory whenever no peer forwards.

12. Verification View — memory only when no cache has it

The properties that keep sourcing correct: memory is used only as a last resort, and a dirty peer always wins.

// Bind to chi_source_select. Same-cycle invariants -> immediate assertions in always_comb.
always_comb begin
  // 1. Memory is read ONLY when no cache holds the line.
  p_memory_last_resort: assert (!reads_memory || (!peer_has && !slc_has));
  // 2. A dirty peer is always the source (only current copy).
  p_dirty_peer_wins:    assert (!peer_dirty || (source == SRC_PEER));
  // 3. An SLC hit (no dirty peer) sources from the SLC, not memory.
  p_slc_over_memory:    assert (!(slc_has && !peer_dirty && !peer_has) || (source == SRC_SLC));
end

The system point, beyond the checks:

The presence of an SLC changes the shape of the problem: without it there are two sources (a cache or memory) and the choice is "is the line cached dirty?"; with it there are three, and the middle source is one the home owns. That makes the SLC easy to forget — it is not a peer you must snoop, so a naive home treats "no forwarding peer" as "go to memory." But the SLC is a cache like any other: it can hold the current, even dirty, copy, and it exists precisely to keep hot lines off the memory bus. Correct source selection therefore has to treat the SLC as a first-class cache source, checked before memory, or the home both loses the SLC's bandwidth benefit and risks reading stale memory behind a dirty SLC line.

• What it proves: memory is last resort; a dirty peer wins; an SLC hit beats memory.
• What it does not prove: the beat-level data transport — that is 13.1 and the sinks (13.3).
• Bug signature: reads_memory asserted while slc_has (or a dirty peer) is true.

13. Testbench — the SLC must beat memory

Drives the source cases and checks memory is used only when no cache holds the line.

module tb_chi_source_select;
  logic peer_has, peer_dirty, slc_has;
  logic [1:0] source; logic reads_memory;
  localparam SRC_PEER = 2'd0, SRC_SLC = 2'd1, SRC_MEM = 2'd2;
  int errors = 0;
 
  chi_source_select dut (.*);
 
  task check(input logic ph, pd, sh, input logic [1:0] exp, input string nm);
    begin
      peer_has = ph; peer_dirty = pd; slc_has = sh; #1;
      if (source !== exp) begin errors++; $display("FAIL %s: source=%0d exp=%0d", nm, source, exp); end
      else $display("PASS %s: source=%0d reads_mem=%0b", nm, source, reads_memory);
    end
  endtask
 
  initial begin
    check(1, 1, 0, SRC_PEER, "dirty peer wins");
    check(0, 0, 1, SRC_SLC,  "SLC hit beats memory");
    check(1, 0, 1, SRC_SLC,  "clean peer, SLC preferred over peer");
    check(0, 0, 0, SRC_MEM,  "no cache -> memory");
    // memory must never be chosen while a cache holds the line
    check(0, 0, 1, SRC_SLC,  "SLC present -> not memory");
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS dirty peer wins: source=0 reads_mem=0
PASS SLC hit beats memory: source=1 reads_mem=0
PASS clean peer, SLC preferred over peer: source=1 reads_mem=0
PASS no cache -> memory: source=2 reads_mem=1
PASS SLC present -> not memory: source=1 reads_mem=0
ALL TESTS PASSED

14. DebugLab — a home that never consults its SLC

read arrives; no peer holds the line dirty (no forward)
SLC state: HOLDS the line (recently written -> newer than memory)
buggy selector: "no forwarding peer" -> reads_memory = 1  (ignores slc_has)
  -> memory read issued though SLC had the line   (wasted bandwidth)
  -> memory copy is OLD (SLC not yet written back) -> STALE data returned
correct: slc_has = 1 -> source = SLC -> current data, no memory read

15. Common Mistakes

• Ignoring the SLC. Assumption: no peer means memory. Bug: wasted bandwidth and stale reads (the DebugLab). Prevention: consult the SLC first.
• Always sourcing from memory. Assumption: memory is simplest. Bug: stale data behind dirty caches. Prevention: source from the current copy.
• Missing a dirty peer. Assumption: memory is current. Bug: stale — the dirty peer had the only current copy. Prevention: snoop before memory.
• Treating all sources as equal copies. Assumption: any copy will do. Bug: reading a stale one. Prevention: pick the current copy.
• Preferring memory over a close cache. Assumption: distance does not matter. Bug: needless latency. Prevention: closest current source first.
• Not writing SLC updates back. Assumption: SLC and memory agree. Bug: stale memory. Prevention: track the SLC as authoritative when it holds the line.

16. Engineering Checklist

• Enumerate the three sources — peer cache, SLC, memory.
• Source from wherever the current copy of the line lives.
• Prefer a close source (peer, SLC) over memory when it holds the line.
• Source from a dirty peer first — it has the only current copy.
• Consult the SLC before falling to memory.
• Use memory only when no cache holds the current data.

17. Key Takeaways

• A read is supplied from a peer RN cache, the home's SLC, or memory.
• The home sources from wherever the current copy lives, closest first.
• A dirty peer has the only current copy and wins.
• The SLC supplies without a memory read and may be newer than memory.
• Memory is the source of last resort — used only when no cache holds the line.
• Skipping the SLC wastes bandwidth and can serve stale data; the model is representative.

18. Quick Revision

Data sources. A CHI read can be supplied from three sources: a peer RN's cache (closest; may hold the line dirty and forwards it — the only current copy when dirty), the home node's system-level cache (SLC) (a source the home owns, supplying without a memory read and possibly newer than memory), and memory (the SN) (the backing store, the source of last resort). The home selects the source holding the current copy, closest first: a dirty peer wins; else an SLC hit beats memory; else, when no cache holds the line, the home reads memory. The trap is a home that has an SLC but never consults it — treating "no forwarding peer" as "read memory." That wastes memory bandwidth on lines the SLC already holds and, when the SLC holds a copy newer than memory, returns stale data. Consult the caches before memory; source from where the current copy lives. Representative model; 13.3 covers where data ends up.

Coming Next

Chapter 13.3 — Data Sinks. 13.2 asked where data comes from; the mirror question is where it goes. Chapter 13.3 covers the data sinks — the requesting Request Node that receives read data, the home node whose cache absorbs written-back or snooped data, and memory as the sink for write-backs — and why routing a data packet to the wrong sink misdelivers it.