The Slave Node (SN)

Module 4 · Chapter 4.4 · CHI Architecture Overview

Project thread — 4.2 and 4.3 covered the Request and Home Nodes. This chapter completes the triad with the Subordinate Node, the storage endpoint. 4.5 then asks why the whole thing is distributed.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Define the Subordinate Node (SN) as the storage endpoint behind a Home Node.
• Explain why the SN is coherency-agnostic — no directory, no snoop, no cache state.
• Distinguish SN-F (coherent memory) from SN-I (I/O / peripheral).
• Trace how the Home Node forwards a miss or writeback to the SN and gets a completion.
• State the architectural rule: the SN completes to the Home Node, never directly to a requester.
• Implement a representative SN memory completer in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

The SN completes CHI's node model — with it, you can read any transaction end to end: RN initiates, HN arbitrates, SN stores. Knowing that the SN does no coherency is what makes CHI's layering click: coherency lives entirely at the Home Node, and the SN is just memory. That separation is why CHI can pair one coherency protocol with many kinds of storage.

It also prevents a common and dangerous misconception — that memory can serve coherent data directly. It cannot: the SN's copy is only authoritative after the Home Node has resolved coherency. Understanding why the SN must stay behind the HN is the core safety lesson of this chapter.

3. Key Terms

4. Previous Chapter Connection

Chapter 4.3 established the Home Node as the Point of Coherence and Serialization: it orders requests, holds the directory, snoops holders, and — on a miss or writeback — forwards to memory. That last step is where the SN enters.

By the time the HN forwards a request to the SN, every coherency decision is already made: the HN has serialized the access, snooped the holders, and knows it needs data from (or must write data to) memory. The SN inherits none of that work. This chapter is the endpoint of every chain the previous two chapters began — and, deliberately, the simplest node.

5. Core Concept — a storage endpoint, nothing more

The SN has one job: complete the reads and writes the Home Node forwards.

• It stores and completes. On a read, the SN returns data on DAT; on a write or writeback, it accepts the data and returns a completion on RSP. It is a completer at the back of the transaction chain RN → HN → SN.
• It does no coherency. The SN has no directory, no snoop logic, and no cache-state tracking. It never asks who holds a line, because that is not its job — the Home Node already did it.
• It talks to the Home Node, not requesters. The SN receives requests from the HN and completes to the HN. It never responds directly to an RN. This is what keeps the Home Node the single point of coherence.
• Two variants by address space. SN-F backs coherent memory (DRAM behind an HN-F); SN-I backs I/O and peripheral space (behind an HN-I). Both are coherency-agnostic — the difference is which space they serve.

The synthesis:

The Subordinate Node is memory with a handshake. The clean separation CHI draws is: coherency at the Home Node, storage at the Subordinate Node. By the time a request reaches the SN, coherency is resolved; the SN just performs the raw access and completes back to the HN. That simplicity is a feature — it lets one coherency protocol sit in front of any storage.

6. Engineering Mental Model — the warehouse behind the front desk

Return to the registrar model from Chapter 4.3. The Home Node is the registrar who owns authority over records and their order. The Subordinate Node is the warehouse behind the registrar's desk.

• The warehouse stores boxes (data) and does exactly what the front desk tells it: fetch box A, store box B. It keeps no index of who is using what and never decides order — that is the registrar's job.
• Crucially, the warehouse never hands a box directly to a customer. A customer who walked into the warehouse and grabbed a box might take a stale one — the current version could be a marked-up copy the registrar is still collecting from someone else. Everything must go through the front desk.
• Swap the warehouse for a bigger one (more memory) or a different kind (I/O) and the front desk is unchanged. Storage and authority are cleanly separated.

The SN is that warehouse: essential, simple, and deliberately kept behind the desk.

7. Engineering Diagram — the SN as endpoint

The SN sits one hop past the Home Node, and only the Home Node. That position — behind the point of coherence — is the whole design: storage is reachable only through the authority that keeps it coherent.

8. SN Variants

Two variants, split by the address space they back.

Two facts to carry: neither SN variant does coherency — that is always the Home Node's job — and the split is purely about which space they serve (coherent memory vs I/O). An SN is chosen by what it stores, not by any coherency capability, because it has none.

9. Why the SN Is Coherency-Agnostic

It is worth being explicit about why the endpoint holds no coherency state.

• The work is already done. When the HN forwards to the SN, it has serialized the access, consulted the directory, and snooped the holders. The data the SN reads or writes is exactly what the HN determined it needs. Re-checking coherency at the SN would duplicate the HN and, worse, could disagree with it.
• Memory is not always current. Under a dirty-sharing protocol (MOESI-style), the newest value of a line can live in a cache, not in memory. So the SN's stored copy is not authoritative on its own — only the HN, having gathered snoop data, knows the true current value. That is precisely why the SN must not serve requesters directly.
• Separation buys flexibility. Keeping coherency out of the SN lets CHI put the same coherency protocol in front of DRAM, HBM, or an I/O region without changing the protocol — the SN is swappable storage.

The rule that falls out:

The SN completes to the Home Node, never to a requester. Because memory may be stale relative to a dirty cache, the SN's data is only valid once the HN has merged it with snoop results. Any path that lets the SN answer an RN directly bypasses the point of coherence — and returns stale data. This is the SN's one inviolable architectural constraint.

10. Transaction Walkthrough — the HN forwards a miss to the SN

CPU0 reads line A; no cache holds it (a clean miss), so the data must come from memory.

1. Request and serialize. CPU0's RN sends ReadShared to the HN; the HN serializes line A and consults its directory.
2. Directory miss. The directory shows no RN holds line A — the current value is in memory. No snoop is needed.
3. Forward to SN. The HN sends a read request to the SN-F that backs line A's address.
4. SN completes. The SN-F reads its memory and returns the data on DAT to the HN (not to CPU0).
5. HN responds. The HN forwards the data to CPU0 (CompData), records CPU0 as a sharer in the directory, and completes on CompAck.

The SN did the raw read and handed the result to the HN. Note step 4: even on a clean miss where memory is current, the SN completes to the HN, which stays the single responder to the RN. Consistency of the path matters more than shortcutting the easy case.

11. RTL / Hardware View — an SN memory completer

The SN is memory with a completion handshake. Here is a representative completer: it performs a read or write and pulses a completion the next cycle. Simplified — a small word memory, single request per cycle, no coherency logic (by design).

// Representative SN memory completer (educational, coherency-agnostic by design).
// Completes reads and writes the Home Node forwards: on a write it stores the
// data; on a read it returns the stored word. One completion per request. There
// is deliberately NO directory, snoop, or cache-state logic here.
module sn_completer #(
  parameter int WORDS = 16
)(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        req_valid,
  input  logic        req_write,               // 1 = write, 0 = read
  input  logic [3:0]  req_addr,
  input  logic [31:0] req_wdata,
  output logic        resp_valid,               // completion pulse
  output logic [31:0] resp_rdata                // read data (on read completion)
);
  logic [31:0] mem [0:WORDS-1];
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      resp_valid <= 1'b0;
    end else begin
      resp_valid <= req_valid;                  // one completion per request
      if (req_valid && req_write) mem[req_addr] <= req_wdata;   // store
      resp_rdata <= mem[req_addr];              // registered read data
    end
  end
endmodule

The same behavior in Verilog-2001:

// Representative SN memory completer (Verilog-2001).
module sn_completer #(
  parameter WORDS = 16
)(
  input         clk, rst_n,
  input         req_valid, req_write,
  input  [3:0]  req_addr,
  input  [31:0] req_wdata,
  output reg        resp_valid,
  output reg [31:0] resp_rdata
);
  reg [31:0] mem [0:WORDS-1];
  always @(posedge clk or negedge rst_n)
    if (!rst_n) resp_valid <= 1'b0;
    else begin
      resp_valid <= req_valid;
      if (req_valid && req_write) mem[req_addr] <= req_wdata;
      resp_rdata <= mem[req_addr];
    end
endmodule

And in VHDL:

-- Representative SN memory completer (VHDL).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity sn_completer is
  generic ( WORDS : integer := 16 );
  port (
    clk, rst_n : in  std_logic;
    req_valid  : in  std_logic;
    req_write  : in  std_logic;
    req_addr   : in  std_logic_vector(3 downto 0);
    req_wdata  : in  std_logic_vector(31 downto 0);
    resp_valid : out std_logic;
    resp_rdata : out std_logic_vector(31 downto 0)
  );
end entity;
 
architecture rtl of sn_completer is
  type mem_t is array(0 to WORDS-1) of std_logic_vector(31 downto 0);
  signal mem : mem_t;
begin
  process(clk, rst_n)
    variable a : integer range 0 to WORDS-1;
  begin
    if rst_n = '0' then
      resp_valid <= '0';
    elsif rising_edge(clk) then
      a := to_integer(unsigned(req_addr));
      resp_valid <= req_valid;
      if req_valid = '1' and req_write = '1' then mem(a) <= req_wdata; end if;
      resp_rdata <= mem(a);
    end if;
  end process;
end architecture;

All three are pure storage-plus-completion: one completion per request, data stored on write and returned on read, and not a line of coherency logic — because that lives at the Home Node.

12. Verification View — one completion per request

Two properties: every request produces exactly one completion, and a written value reads back (memory correctness).

// Bind to sn_completer.
// 1. Every accepted request produces a completion exactly one cycle later.
property p_one_completion;
  @(posedge clk) disable iff (!rst_n) req_valid |=> resp_valid;
endproperty
assert property (p_one_completion);
 
// 2. A write is observable by a later read of the same address (storage works).
//    (Written as a value-tracking check in a directed testbench.)
property p_write_then_read;
  @(posedge clk) disable iff (!rst_n)
    (req_valid && req_write) |=> !$isunknown(resp_rdata);
endproperty
assert property (p_write_then_read);

The system point, beyond the two checks:

The SN's correctness is local — store what you are told, return what you stored, complete once per request. It makes no coherency claims, and that is exactly right: a coherency check here would be redundant with the Home Node and could contradict it. The one thing the SN must never do is expose its data on a path that skips the Home Node, because its stored copy may be stale relative to a dirty cache. Local storage correctness plus "always complete to the HN" is the SN's entire contract.

• What it proves: one completion per request; stored data reads back.
• What it does not prove: anything about coherency — the SN makes no such guarantee.
• Bug signature: a completion count that does not match requests, or (architecturally) SN data reaching an RN without the HN (the DebugLab).

13. Testbench — write then read back

Writes a value, reads it back, and checks the completion timing.

module tb_sn_completer;
  logic clk = 0, rst_n, req_valid, req_write;
  logic [3:0] req_addr;
  logic [31:0] req_wdata, resp_rdata;
  logic resp_valid;
  int errors = 0;
 
  sn_completer #(.WORDS(16)) dut (.*);
  always #5 clk = ~clk;
 
  task automatic access(input logic wr, input logic [3:0] a, input logic [31:0] d, input string tag);
    req_valid = 1; req_write = wr; req_addr = a; req_wdata = d;
    @(posedge clk); #1;
    if (!resp_valid) begin errors++; $display("FAIL [%s] no completion", tag); end
    else $display("PASS [%s] %s addr=%0d rdata=%h", tag, wr ? "WR" : "RD", a, resp_rdata);
    req_valid = 0;
  endtask
 
  initial begin
    rst_n = 0; @(posedge clk); rst_n = 1; req_valid = 0;
    access(1, 4'd5, 32'hCAFE_0005, "write 5");
    access(0, 4'd5, 32'h0,         "read 5");   // expect CAFE_0005
    if (resp_rdata !== 32'hCAFE_0005) begin errors++; $display("FAIL readback %h", resp_rdata); end
    else $display("PASS readback = CAFE_0005");
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS [write 5] WR addr=5 rdata=xxxxxxxx
PASS [read 5] RD addr=5 rdata=cafe0005
PASS readback = CAFE_0005
ALL TESTS PASSED

14. DebugLab — the SN that answers the requester directly

cyc  event                              note
 12  HN issues SnpUnique to CPU1        CPU1 holds the line DIRTY
 12  SN returns memory data to RN       via the "fast path" -> STALE
 15  CPU1 supplies dirty data to HN     the true current value, too late

15. Common Mistakes

• Expecting the SN to do coherency. Assumption: memory tracks sharers. Bug: duplicating or contradicting the HN. Prevention: the SN is coherency-agnostic; the HN owns coherency.
• Letting the SN answer requesters. Assumption: memory can serve reads directly. Bug: stale data past a dirty cache (the DebugLab). Prevention: the SN completes to the HN only.
• Assuming memory is always current. Assumption: DRAM holds the newest value. Bug: reading stale memory under dirty sharing. Prevention: the newest value may be in a cache; only the HN knows.
• Confusing SN-F and SN-I by capability. Assumption: one is coherent, one is not. Bug: mis-modeling. Prevention: neither does coherency; they differ by address space (memory vs I/O).
• Thinking the SN is optional detail. Assumption: storage is trivial. Bug: missing completion/ordering requirements. Prevention: the SN still must complete correctly and stay behind the HN.
• Adding a fast path. Assumption: skipping the HN saves latency safely. Bug: bypasses the point of coherence. Prevention: no path may bypass the HN for coherent data.

16. Engineering Checklist

• Treat the SN as pure storage plus completion — no coherency logic.
• Route SN traffic through the Home Node; the SN never talks to a requester.
• Pick SN-F for coherent memory, SN-I for I/O / peripheral space.
• Remember memory may be stale relative to a dirty cache — only the HN reconciles.
• Ensure one completion per request back to the HN.
• Reject any "fast path" that lets SN data reach an RN without the HN.

17. Key Takeaways

• The Subordinate Node (SN) — historically the slave node — is the storage endpoint behind a Home Node; it completes the reads and writes the HN forwards.
• The SN is coherency-agnostic: no directory, no snoop, no cache state — coherency lives entirely at the HN.
• Variants split by address space: SN-F (coherent memory) and SN-I (I/O / peripheral); neither does coherency.
• The SN completes to the Home Node, never to a requester — because memory can be stale relative to a dirty cache.
• The clean separation — coherency at the HN, storage at the SN — lets one protocol front any storage.
• With the SN, the RN → HN → SN triad is complete; the model here is representative.

18. Quick Revision

The Slave / Subordinate Node (SN). The SN is the storage endpoint behind a Home Node: it completes the reads and writes the HN forwards, returning data on DAT or an acknowledgement on RSP — one completion per request. It is coherency-agnostic: no directory, no snoop, no cache state; the HN already resolved coherency. Variants: SN-F (coherent memory / DRAM behind HN-F) and SN-I (I/O / peripheral behind HN-I) — neither does coherency, they differ by address space. The inviolable rule: the SN completes to the Home Node, never directly to a requester, because memory may be stale relative to a dirty cache and only the HN reconciles the two. Clean split — coherency at the HN, storage at the SN — completes the RN → HN → SN triad. Representative model.

Coming Next

Chapter 4.5 — Fully Distributed Architecture. The three node types are now in hand — but a single Home Node holding one directory would just recreate Module 3's central bottleneck. The next chapter shows how CHI distributes coherency: many Home Nodes, each owning a slice of the address space and its own directory, spread across the mesh. We will see why sharding the point of coherence — rather than centralizing it — is what lets CHI scale, and how the address-to-Home-Node map ties it together.