Data Sinks

Module 13 · Chapter 13.3 · CHI Data Transfers

Project thread — 13.2 was where data comes from. 13.3 is where it goes; 13.4 is cache-to-cache forwarding.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Name the three data sinks — the requesting RN, the home (buffer/cache), and memory (SN).
• State that the sink is chosen by the packet's opcode and correlation, not its arrival port.
• Explain that read data sinks to the requester by TgtID, write-back data to memory by DBID.
• Describe how a mis-selected sink misdelivers the packet.
• Diagnose the corruption and lost update from routing by arrival instead of opcode.
• Implement a representative sink selector in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

Data that reaches the right node still has to reach the right place inside it. Read data must land in the requester's fill path; write-back data must land in the home's write buffer and then memory. These are different destinations with different consequences, and the packet itself says which — through its opcode and correlation tag. Read the packet correctly and it lands where it belongs; read it by the wrong signal and it is misdelivered.

Misdelivery is uniquely damaging because it fails two things at once. If write-back data is dispatched to a requester's fill path, the requester installs garbage as a valid line (corruption) and the write-back never reaches memory (a lost update). One mis-selected sink both installs wrong data and drops correct data. This chapter enumerates the sinks and shows why the sink must come from the packet's opcode and correlation — the same self-describing-transport lesson as 13.1, now applied to destination rather than reassembly.

3. Key Terms

4. Previous Chapter Connection

Chapter 13.2 named the sources; this chapter names the sinks — the mirror. The correlation fields are the ones you have used since Module 6: read completions carry a TgtID to reach the requester (Chapter 6.6), and write-backs carry a DBID naming the home's data buffer (Chapter 7.2, 8.6).

The distinction from routing (Chapter 6.7) matters: routing gets a packet to the right node; sink selection dispatches it to the right place inside that node once it arrives. A write-back that correctly reaches the home can still be dispatched to the wrong internal sink if the home reads the wrong field. This chapter is that last hop — from "arrived at the home" to "landed in the write buffer."

5. Core Concept — three sinks, chosen by the packet

A data packet ends at one of three sinks, and the sink is chosen by the packet's opcode and correlation, not where it arrived.

• Requesting RN. Read data — a completion (CompData) — sinks into the requester's fill path, installing the line in its cache. Routed by TgtID.
• Home node. Snooped data and written-back data sink into the home's write buffer / cache, to be merged, held, or forwarded. Matched by DBID.
• Memory (SN). Write-back and clean-eviction data sink into memory, updating the backing store. Reached via the home's buffer, by DBID.
• Chosen by the packet. The sink follows the opcode (read-completion vs write-back) and the correlation tag — never the port the data arrived on. A packet is self-describing: it says where it must land.

The synthesis:

A CHI data packet sinks into the requesting RN (read data, by TgtID), the home (snooped/written-back data, by DBID), or memory (write-backs). The sink is determined by the packet's opcode and correlation tag, not its arrival. Reading the wrong signal misdelivers the packet — installing garbage in a cache and losing a write-back.

6. Engineering Mental Model — a mailroom sorting by the label

Think of a mailroom that has to put each parcel in the right bin.

• Every parcel has a shipping label — a destination and a tracking number. That is the opcode and correlation.
• The clerk sorts by the label, not by which loading-dock door the parcel came through. A parcel for the "outgoing to warehouse" bin goes there even if it arrived on the same truck as the "deliver to office 3" parcels.
• If the clerk instead sorted by arrival door — everything off truck A into bin 1 — parcels would land in the wrong bins: a warehouse-bound parcel dumped on someone's desk, and the desk's real mail sent to the warehouse and lost.

The label is the truth about destination; the arrival door is an accident. A data packet's opcode and correlation are its label — the sink selector must sort by them, never by where the packet came in.

7. Engineering Diagram — one source, three sinks

One arriving packet, three possible sinks. The home reads the packet's opcode and correlation to pick — read data to the requester, snoop/write-back data to its buffer, write-back on to memory. The arrival edge is one; the sink edge depends on the packet.

8. The Three Sinks Compared

Each sink, what it absorbs, and its correlation.

The rule to carry: the sink is a function of the packet, not the port. Read data is a completion and sinks to the requester its TgtID names. Write-back data is a write and sinks to the home's buffer its DBID names, then to memory. The opcode says which kind; the correlation says which instance. Together they fully determine the sink — the arrival port adds nothing and, if used, misleads.

9. Selecting the Sink

The dispatch logic, stated plainly.

• Read completion → requester. A CompData packet sinks into the RN named by its TgtID — the requester's fill path installs the line.
• Write-back → home buffer → memory. A write-back packet sinks into the home's write buffer named by its DBID, and the home writes it to memory.
• Snoop data → home. Snooped data sinks into the home to be merged/forwarded (Chapter 9.5), matched by its correlation.
• Never dispatch by arrival. The sink comes from opcode + correlation. Dispatching by the arrival port (or a static assumption) misroutes packets whose opcode does not match that port's usual traffic.

The point to carry:

Sink selection is the receive-side twin of source selection (13.2), and both obey the same discipline: decide from the packet's own fields, never from context the fabric does not guarantee. On the source side the field was "where is the current copy"; on the sink side it is "what does this packet's opcode and correlation say its destination is." The reason arrival cannot be trusted is that a single port at the home carries mixed traffic — completions passing through, write-backs landing, snoop data returning — distinguished only by opcode. A selector that keys off the port collapses that distinction and sends whole classes of packet to the wrong place. The opcode is the discriminator the protocol provides precisely so the receiver never has to guess from arrival.

10. Dispatching Data — read vs write-back

Two packets arrive at the home on the same path; each sinks differently.

1. Read completion arrives. Its opcode is a read-completion; its TgtID names RN A. The home dispatches it to RN A's fill path — the read data installs in A's cache.
2. Write-back arrives. Its opcode is a write-back; its DBID names the home's buffer slot. The home dispatches it to the write buffer, then writes it to memory.
3. Different sinks, same port. Both arrived on the home's data path, but their opcodes sent them to different sinks — the requester and memory respectively.
4. Correlation confirms the instance. TgtID picked which requester; DBID picked which buffer slot. The opcode picked the kind of sink.

The two packets landed correctly because the home read their opcodes and correlation. The DebugLab is a home that instead dispatches by arrival, sending a write-back into a fill path.

11. RTL / Hardware View — a sink selector

The home selects the sink from the data opcode and routes by the correlation. Representative — read-completion vs write-back.

// Representative data-sink selector (educational).
// The sink is chosen by the data OPCODE, not the arrival port:
//   read completion  -> requesting RN's fill path (by TgtID)
//   write-back       -> home write buffer -> memory (by DBID)
// Routing by arrival misdelivers write-backs into fill paths.
typedef enum logic [1:0] { SINK_REQ, SINK_MEM, SINK_HOME } sink_e;
 
module chi_sink_select (
  input  logic  is_read_comp,   // opcode: read completion (CompData)
  input  logic  is_writeback,   // opcode: write-back (CopyBackWrData)
  input  logic  is_snoop_data,  // opcode: snooped data returning
  output sink_e sink,
  output logic  to_requester    // asserted only for read data
);
  always_comb begin
    if      (is_read_comp)  sink = SINK_REQ;   // read data -> requester
    else if (is_writeback)  sink = SINK_MEM;   // write-back -> memory (via home buffer)
    else if (is_snoop_data) sink = SINK_HOME;  // snoop data -> home
    else                    sink = SINK_HOME;  // default: hold at home
  end
  assign to_requester = (sink == SINK_REQ);
endmodule

The same behavior in Verilog-2001:

// Representative data-sink selector (Verilog-2001).
module chi_sink_select (
  input  wire       is_read_comp, is_writeback, is_snoop_data,
  output reg  [1:0] sink,          // 0=REQ, 1=MEM, 2=HOME
  output wire       to_requester
);
  localparam SINK_REQ = 2'd0, SINK_MEM = 2'd1, SINK_HOME = 2'd2;
  always @* begin
    if      (is_read_comp)  sink = SINK_REQ;
    else if (is_writeback)  sink = SINK_MEM;
    else if (is_snoop_data) sink = SINK_HOME;
    else                    sink = SINK_HOME;
  end
  assign to_requester = (sink == SINK_REQ);
endmodule

And in VHDL:

-- Representative data-sink selector (VHDL).
library ieee;
use ieee.std_logic_1164.all;
 
entity chi_sink_select is
  port (
    is_read_comp, is_writeback, is_snoop_data : in  std_logic;
    sink         : out std_logic_vector(1 downto 0); -- 00=REQ 01=MEM 10=HOME
    to_requester : out std_logic
  );
end entity;
 
architecture rtl of chi_sink_select is
  constant SINK_REQ  : std_logic_vector(1 downto 0) := "00";
  constant SINK_MEM  : std_logic_vector(1 downto 0) := "01";
  constant SINK_HOME : std_logic_vector(1 downto 0) := "10";
  signal s : std_logic_vector(1 downto 0);
begin
  process (is_read_comp, is_writeback, is_snoop_data)
  begin
    if is_read_comp = '1' then
      s <= SINK_REQ;         -- read data -> requester
    elsif is_writeback = '1' then
      s <= SINK_MEM;         -- write-back -> memory
    elsif is_snoop_data = '1' then
      s <= SINK_HOME;        -- snoop data -> home
    else
      s <= SINK_HOME;
    end if;
  end process;
  sink         <= s;
  to_requester <= '1' when s = SINK_REQ else '0';
end architecture;

All three branch on the data opcode and never on an arrival port — to_requester is high only for a read completion. The DebugLab is a selector that instead sends whatever arrives on the fill port to the requester, misdelivering write-backs.

12. Verification View — the sink follows the opcode

The properties that keep dispatch correct: only read data reaches the requester, write-backs reach memory.

// Bind to chi_sink_select. Same-cycle invariants -> immediate assertions in always_comb.
always_comb begin
  // 1. Only a read completion sinks to the requester.
  p_only_read_to_requester: assert (!to_requester || is_read_comp);
  // 2. A write-back sinks to memory, never to the requester.
  p_writeback_to_memory:    assert (!is_writeback || (sink == SINK_MEM));
end
 
// 3. The sink is a function of the opcode alone (no arrival-port term exists).
//    read_comp -> REQ, writeback -> MEM, snoop -> HOME (checked exhaustively).

The system point, beyond the checks:

A misdelivered data packet is a double fault, which is what makes sink selection worth isolating as its own concern. Send a write-back to a fill path and the fill path treats bytes meant for memory as a fresh cache line — installing them as valid (fault one: corruption) — while the write-back they actually were never reaches memory (fault two: a lost update). Most protocol bugs fail in one direction; this one fails in both because a data packet is simultaneously something to install and something that came from somewhere that expects it delivered. That is why the sink must be pinned to the opcode: the opcode is the only field that distinguishes "this is a line to install" from "this is a line to store back," and getting that distinction from anything less reliable than the opcode risks both installing wrong data and dropping right data.

• What it proves: only read data reaches the requester; write-backs reach memory.
• What it does not prove: the correlation matching within a sink — that is the buffer's job.
• Bug signature: to_requester asserted for a non-read opcode — a write-back in a fill path.

13. Testbench — write-backs must not reach the requester

Drives each opcode and checks the sink follows it, not the arrival.

module tb_chi_sink_select;
  logic is_read_comp, is_writeback, is_snoop_data;
  logic [1:0] sink; logic to_requester;
  localparam SINK_REQ = 2'd0, SINK_MEM = 2'd1, SINK_HOME = 2'd2;
  int errors = 0;
 
  chi_sink_select dut (.*);
 
  task check(input logic rc, wb, sd, input logic [1:0] exp, input string nm);
    begin
      is_read_comp = rc; is_writeback = wb; is_snoop_data = sd; #1;
      if (sink !== exp) begin errors++; $display("FAIL %s: sink=%0d exp=%0d", nm, sink, exp); end
      else $display("PASS %s: sink=%0d to_req=%0b", nm, sink, to_requester);
    end
  endtask
 
  initial begin
    check(1, 0, 0, SINK_REQ,  "read comp -> requester");
    check(0, 1, 0, SINK_MEM,  "write-back -> memory");
    check(0, 0, 1, SINK_HOME, "snoop data -> home");
    // a write-back must NOT be delivered to the requester
    is_read_comp = 0; is_writeback = 1; is_snoop_data = 0; #1;
    if (to_requester) begin errors++; $display("FAIL write-back reached requester"); end
    else $display("PASS write-back not delivered to requester");
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS read comp -> requester: sink=0 to_req=1
PASS write-back -> memory: sink=1 to_req=0
PASS snoop data -> home: sink=2 to_req=0
PASS write-back not delivered to requester
ALL TESTS PASSED

14. DebugLab — dispatching by arrival instead of opcode

write-back packet (CopyBackWrData) arrives at home on the shared data path
buggy dispatch: "arrived on the fill path port" -> deliver to requesting RN
  -> RN installs write-back bytes as a VALID line   (fault 1: corruption)
  -> write-back never routed to memory              (fault 2: lost update)
packet opcode = write-back, DBID = home buffer slot 7  (both IGNORED)
correct: opcode=write-back -> SINK_MEM via DBID 7 -> memory

15. Common Mistakes

• Dispatching by arrival port. Assumption: a port carries one opcode. Bug: misdelivery (the DebugLab). Prevention: select by opcode.
• Ignoring the opcode. Assumption: all data is fill data. Bug: write-backs installed in caches. Prevention: branch on opcode.
• Dropping the correlation. Assumption: one outstanding per sink. Bug: wrong requester/slot. Prevention: route by TgtID/DBID.
• Sending write-back data to the requester. Assumption: data means a fill. Bug: corruption + lost update. Prevention: write-backs sink to memory.
• Sending read data to memory. Assumption: symmetric paths. Bug: the requester never fills. Prevention: read data sinks to the requester.
• Assuming node routing is enough. Assumption: right node = done. Bug: wrong internal sink. Prevention: dispatch inside the node by opcode.

16. Engineering Checklist

• Enumerate the three sinks — requester, home buffer, memory.
• Select the sink from the packet's opcode and correlation, never its arrival port.
• Sink read data to the requester by TgtID.
• Sink write-back data to the home buffer and memory by DBID.
• Sink snoop data to the home to be merged/forwarded.
• Confirm a write-back never reaches a requester's fill path.

17. Key Takeaways

• A data packet sinks into the requester, the home buffer, or memory.
• The sink is chosen by the packet's opcode and correlation, not its arrival.
• Read data sinks to the requester by TgtID; write-back data to memory by DBID.
• Dispatching by arrival misdelivers packets that share a path.
• A misdelivered write-back both corrupts a cache and loses an update.
• The sink follows the packet; the model here is representative.

18. Quick Revision

Data sinks. A CHI data packet ends at one of three sinks: the requesting RN (read data / CompData lands in its fill path, routed by TgtID), the home (snooped and written-back data land in its write buffer/cache, matched by DBID), or memory / the SN (write-backs and clean evictions, via the home's buffer). The sink is determined by the packet's opcode (read-completion vs write-back) and its correlation tag — never the port the data arrived on, because a single home data path carries mixed opcodes. Dispatching by arrival misdelivers: a write-back sent to a requester's fill path installs garbage as a valid line (corruption) and never reaches memory (a lost update) — a double fault. Select the sink from the opcode and correlation: read data → the requester by TgtID; write-back → the home buffer and memory by DBID; snoop data → the home. The packet is self-describing; route it by the packet. Representative model; 13.4 covers cache-to-cache forwarded data.

Coming Next

Chapter 13.4 — Forwarded Data. Sources and sinks assumed data flows through the home; forwarding lets a peer cache send data directly to the requester. Chapter 13.4 covers forwarded data — the cache-to-cache data path on the DAT channel, how the forwarded packet carries the requester's target and its correlation so the requester can match it to its outstanding transaction, and why forwarded data the requester cannot correlate is dropped.