Fully Distributed Architecture

Module 4 · Chapter 4.5 · CHI Architecture Overview

Project thread — 4.1–4.4 built the node types; this chapter distributes them. Many Home Nodes, each owning an address slice, spread across the mesh from 3.7. 4.6 turns to the links between nodes.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Explain why a single central Home Node / directory would recreate Module 3's bottleneck.
• Describe how CHI shards the address space across many Home Nodes, each with its own directory.
• Define address interleaving and why fine-grained striping balances load.
• Explain why each address having exactly one Home Node removes any need for inter-HN coordination.
• Trace two requests to different addresses proceeding in parallel at different Home Nodes.
• Implement a representative HN ownership / interleaving decoder in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

This chapter is where CHI's architecture pays off. You have the node types (4.1–4.4) and the mesh (3.7); distribution is what turns them into something that actually scales. A learner who stops at "the Home Node is the point of coherence" has half the picture — the other half is that there are many Home Nodes, each owning a slice, so coherency happens in parallel across the whole address space.

It is also a real design lever. How the address space is interleaved across Home Nodes decides whether load spreads evenly or piles onto one node. Getting the interleaving granularity right is a concrete architectural choice with a measurable performance cliff — exactly the kind of decision this chapter equips you to make.

3. Key Terms

4. Previous Chapter Connection

Chapters 4.2–4.4 built the three node types, and 4.3 made the Home Node the Point of Coherence and Serialization for an address range. But it left one question open: what if there were just one Home Node for everything?

That would be a disaster — every coherent access, from every core, funneling through one node, one directory, one serialization point. It is Module 3's central bottleneck wearing a CHI badge. This chapter resolves it: CHI uses many Home Nodes, each owning a slice of addresses, distributed across the mesh. The Point of Coherence is not one place — it is one place per address, and those places are spread out.

5. Core Concept — shard the point of coherence

CHI distributes coherency by partitioning the address space.

• Many Home Nodes, each owning a slice. The address space is split across N Home Nodes. Each HN is the Point of Coherence and Serialization — and holds the directory — for only its slice of addresses. No HN knows or tracks any other slice.
• The System Address Map routes to the owner. A request's address is mapped, by the SAM, to the one Home Node that owns it (its target Node ID). The mesh routes the packet there. The map is global and identical everywhere (Chapter 3.8).
• Distributed physically. Home Nodes are spread across the mesh routers, so their traffic spreads over the fabric rather than converging on one point.
• Interleaved for balance. The map interleaves addresses — fine-grained, at cache-line granularity — so consecutive lines land on different HNs and load spreads evenly across a contiguous workload.

The property that makes it clean:

Every address has exactly one Home Node. So two Home Nodes never need to coordinate about a single line — each owns its addresses completely. Coherency for different addresses proceeds fully in parallel at different HNs, with no cross-HN handshake. The distribution is embarrassingly parallel across addresses, which is why sharding the Point of Coherence scales instead of merely moving the bottleneck.

6. Engineering Mental Model — many registrars, one per record range

Extend the registrar model. A single registrar for an entire national archive would be swamped — every citizen queuing at one desk.

• Instead, open many registrar offices, each responsible for a range of record numbers (an address slice). Your record has exactly one office that owns it; you go there.
• Interleave the record numbers so that consecutive records are handled by different offices — otherwise everyone researching one popular topic (a contiguous range) would crowd a single office (a hotspot). Fine-grained interleaving keeps every office equally busy.
• Because each record has exactly one office, offices never phone each other about a single record — no coordination, no shared lock across offices. Different records are handled entirely independently, in parallel.

Many registrars, each authoritative over its slice, is CHI's distributed coherency. One registrar per record; many records handled at once.

7. Engineering Diagram — distributed Home Nodes on the mesh

Four directories, spread out, each owning a quarter of the addresses. A request for a slice-0 line and a request for a slice-2 line never touch the same Home Node — they run in parallel.

8. Address Interleaving

How addresses map to Home Nodes decides whether the distribution actually balances. Two granularities:

Fine-grained striping is the point: a workload marching through a contiguous buffer touches consecutive cache lines, and fine interleaving sends those consecutive lines to different Home Nodes, spreading the coherency work. Coarse interleaving would send the whole buffer to one Home Node — recreating the bottleneck. Two facts to carry: interleaving granularity is a load-balance decision, and fine-grained (line-level) is what makes distribution pay off.

9. Why One HN Per Address Simplifies Everything

The single-owner rule is not just for routing — it removes a whole class of complexity.

• No inter-HN coordination for a line. Because address A has exactly one Home Node, that HN alone serializes and tracks A. No other HN holds any state for A, so there is never a need for two HNs to agree on A's order or sharers. Each line's coherency is self-contained at its owner.
• Parallelism is free. Accesses to different addresses hit different HNs and proceed with no interaction. The system's coherency throughput scales with the number of Home Nodes, because they never wait on each other.
• Correctness is local. Each HN enforces the same per-address rules (serialize, directory, targeted snoop) 4.3 described — just over its slice. Distribution adds no new coherency rules; it replicates the existing one across slices.

The contrast that makes this elegant:

Some directory schemes must coordinate between directories when a line's state is split across them — adding protocol complexity and latency. CHI avoids that entirely: one address, one Home Node, all its state in one place. Distribution buys scalability without buying distributed-directory coordination, because the address space is partitioned so no line is ever shared between Home Nodes.

10. Transaction Walkthrough — two requests, two Home Nodes, in parallel

CPU0 reads line A (slice 0); CPU1 writes line B (slice 2). The addresses interleave to different Home Nodes.

1. Map. The SAM maps A to HN slice 0 and B to HN slice 2 — different Home Nodes.
2. Route. CPU0's REQ routes to HN0; CPU1's REQ routes to HN2. Independent paths across the mesh.
3. Serialize independently. HN0 serializes and processes A; HN2 serializes and processes B. Neither locks anything the other needs.
4. Snoop and complete in parallel. HN0 snoops A's holders and returns data to CPU0; HN2 snoops B's holders and completes CPU1's write. Both happen at the same time.
5. No interaction. At no point do HN0 and HN2 communicate — different addresses, different owners, disjoint state.

The two transactions share nothing. Add more Home Nodes and more such transactions run concurrently — coherency throughput scaling with the fabric.

11. RTL / Hardware View — Home Node ownership and interleaving

The mechanism behind distribution is the map from address to owning Home Node, and each HN's check of whether it owns an address. Here is that decoder, with fine-grained interleaving. Representative and simplified — a fixed line-bit stripe, no programmable SAM.

// Representative distributed-HN ownership / interleaving (educational).
// Fine-grained striping: interleave by low (cache-line) address bits so
// consecutive 64B lines map to different Home Nodes -> balanced load. Each HN
// checks whether it owns the address.
module hn_owns #(
  parameter int NUM_HN = 4,                       // number of Home Nodes
  parameter int MY_HN  = 0                          // this HN's ID
)(
  input  logic [31:0] addr,
  output logic [1:0]  home_hn,                      // owning HN (system address map)
  output logic        i_own                          // does THIS HN own the address?
);
  // Line address bits [7:6] pick the HN. Adjacent lines (addr + 64) differ in
  // bit 6, so they land on different HNs -> fine-grained, balanced striping.
  assign home_hn = addr[7:6];
  assign i_own   = (home_hn == MY_HN[1:0]);
endmodule

The same behavior in Verilog-2001:

// Representative distributed-HN ownership / interleaving (Verilog-2001).
module hn_owns #(
  parameter NUM_HN = 4,
  parameter MY_HN  = 0
)(
  input  [31:0] addr,
  output [1:0]  home_hn,
  output        i_own
);
  assign home_hn = addr[7:6];
  assign i_own   = (home_hn == MY_HN[1:0]);
endmodule

And in VHDL:

-- Representative distributed-HN ownership / interleaving (VHDL).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity hn_owns is
  generic ( NUM_HN : integer := 4; MY_HN : integer := 0 );
  port (
    addr    : in  std_logic_vector(31 downto 0);
    home_hn : out std_logic_vector(1 downto 0);
    i_own   : out std_logic
  );
end entity;
 
architecture rtl of hn_owns is
  signal hn : std_logic_vector(1 downto 0);
begin
  hn      <= addr(7 downto 6);                       -- fine-grained line stripe
  home_hn <= hn;
  i_own   <= '1' when hn = std_logic_vector(to_unsigned(MY_HN, 2)) else '0';
end architecture;

All three compute the same owning Home Node from the address, and each HN checks ownership against its ID. Interleaving on low line-address bits is what spreads consecutive lines across HNs; interleave on the wrong bits and the balance collapses — the DebugLab.

12. Verification View — exactly one Home Node per address

Two properties: every address maps to a valid HN, and — the key invariant — exactly one HN owns any given address.

// Bind across NUM_HN instances of hn_owns for the same address.
// 1. The computed owner is always a valid Home Node ID.
property p_valid_owner;
  @(*) home_hn < NUM_HN;
endproperty
 
// 2. Exactly one Home Node owns any address: summing i_own across all HNs = 1.
//    (Checked in a testbench that instantiates all NUM_HN and sums i_own.)
//    own_count == 1 for every address.

The system point, beyond the checks:

The whole distributed model rests on one invariant: each address has exactly one owning Home Node — no fewer (an unowned address is unreachable), no more (two owners is a split brain, Chapter 3.8). A deterministic map guarantees exactly one. Everything else — parallel coherency, no inter-HN coordination — follows from single ownership. The second decision, which bits the map uses, does not affect correctness at all; it only affects load balance. Correctness comes from single ownership; performance comes from fine-grained interleaving.

• What it proves: a valid, single owner per address — the distribution is well-formed.
• What it does not prove: that load is balanced — that depends on interleaving granularity versus the workload (the DebugLab).
• Bug signature: two HNs claiming one address (split coherence), or one HN saturated while others idle (a hotspot).

13. Testbench — check single ownership and striping

Instantiates all four Home Nodes and confirms exactly one owns each address, and that consecutive lines stripe across HNs.

module tb_hn_owns;
  logic [31:0] addr;
  logic [1:0]  home_hn [0:3];
  logic        i_own   [0:3];
  int errors = 0;
 
  genvar g;
  generate
    for (g = 0; g < 4; g++) begin : hns
      hn_owns #(.NUM_HN(4), .MY_HN(g)) dut (.addr(addr), .home_hn(home_hn[g]), .i_own(i_own[g]));
    end
  endgenerate
 
  task automatic check(input logic [31:0] a, input string tag);
    int own_count; addr = a; #1;
    own_count = i_own[0] + i_own[1] + i_own[2] + i_own[3];
    if (own_count !== 1) begin errors++; $display("FAIL [%s] own_count=%0d", tag, own_count); end
    else $display("PASS [%s] addr=%h -> HN %0d", tag, a, home_hn[0]);
  endtask
 
  initial begin
    check(32'h0000_0000, "line 0");
    check(32'h0000_0040, "line 1 (next line -> different HN)");
    check(32'h0000_0080, "line 2");
    check(32'h0000_00C0, "line 3");
    // adjacent lines must stripe to different HNs
    addr = 32'h0000_0000; #1; if (home_hn[0] == 2'd0) ; addr = 32'h0000_0040; #1;
    if (home_hn[0] == 2'd0) begin errors++; $display("FAIL striping: line 0 and 1 same HN"); end
    else $display("PASS striping: adjacent lines -> different HNs");
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS [line 0] addr=00000000 -> HN 0
PASS [line 1 (next line -> different HN)] addr=00000040 -> HN 1
PASS [line 2] addr=00000080 -> HN 2
PASS [line 3] addr=000000c0 -> HN 3
PASS striping: adjacent lines -> different HNs
ALL TESTS PASSED

14. DebugLab — coarse interleaving turns one Home Node into a hotspot

map = addr[31:30]  (coarse, high bits)
addr        home_hn   note
0x1000_0000    0       buffer base
0x1000_0040    0       next line -> same HN
0x1000_0080    0       still HN 0
...                    entire 256MB region -> HN 0 only

15. Common Mistakes

• Imagining one central Home Node. Assumption: the Point of Coherence is one place. Bug: expecting a bottleneck (or building one). Prevention: it is one place per address; there are many Home Nodes.
• Coarse interleaving. Assumption: any address split works. Bug: a contiguous workload hotspots one HN (the DebugLab). Prevention: fine-grained, line-granular striping.
• Expecting inter-HN coordination. Assumption: Home Nodes talk to each other about a line. Bug: over-complex mental model. Prevention: one HN per address means no coordination for a line.
• Confusing correctness with balance. Assumption: single ownership guarantees performance. Bug: a valid but unbalanced map. Prevention: single ownership is correctness; interleaving is performance.
• Inconsistent maps. Assumption: each requester can map addresses its own way. Bug: split coherence (Chapter 3.8). Prevention: one identical System Address Map everywhere.
• Placing all HNs at one mesh spot. Assumption: distribution is only logical. Bug: physical hotspot on the fabric. Prevention: spread Home Nodes across the mesh.

16. Engineering Checklist

• Shard the address space across many Home Nodes, each owning a slice and its directory.
• Route every request to its address's owning HN via the System Address Map.
• Interleave on low, cache-line bits — fine-grained — for balanced load.
• Confirm exactly one Home Node owns each address (no gaps, no overlaps).
• Rely on no inter-HN coordination for a single line — one address, one owner.
• Spread Home Nodes physically across the mesh, not clustered at one point.

17. Key Takeaways

• CHI distributes coherency: many Home Nodes, each owning a slice of the address space and its own directory, spread across the mesh.
• A single central Home Node / directory would recreate Module 3's bottleneck — distribution is what scales.
• Every address has exactly one Home Node, so Home Nodes never coordinate over a single line — coherency for different addresses runs fully in parallel.
• Fine-grained (line-level) interleaving spreads consecutive lines across HNs and balances load; coarse interleaving hotspots one node.
• Single ownership is correctness; interleaving granularity is performance — get both right.
• The System Address Map must be global and identical; the model here is representative.

18. Quick Revision

Fully distributed architecture. A single central Home Node / directory would recreate Module 3's bottleneck, so CHI distributes: many Home Nodes, each the Point of Coherence for a slice of the address space and holding that slice's directory, spread across the mesh. The System Address Map routes each address to its one owning Home Node. Fine-grained interleaving (low cache-line bits) sends consecutive lines to different HNs, balancing load; coarse interleaving (high bits) concentrates a contiguous workload on one HN — a hotspot. Because every address has exactly one Home Node, no two HNs coordinate over a line, and coherency for different addresses runs fully in parallel — scaling with the number of HNs. Single ownership is correctness; interleaving is performance. Representative model.

Coming Next

Chapter 4.6 — Point-to-Point Links. Distribution spreads coherency across many nodes — but those nodes still have to be wired together. Unlike AHB's shared bus, CHI connects components with point-to-point links carrying flits with credited flow control. The next chapter details CHI's link layer: how two adjacent components exchange flits, why point-to-point links replace the shared bus entirely, and how the link layer underpins the packet-based protocol the rest of the module builds on.