The Interconnect Fabric

Module 5 · Chapter 5.9 · CHI System Components — module finale

Project thread — Module 5 built every node; this chapter places them on the fabric and closes the module. Module 6 opens up the four channels those nodes exchange.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Describe how RN, HN, and SN nodes attach to the fabric and route by Node ID.
• Compare ring, crossbar, and mesh from the node's point of view — latency, bandwidth, area.
• Explain why node placement on the mesh shapes real performance.
• Explain why the four channels ride separate virtual channels — message-class deadlock.
• Distinguish message-class (protocol) deadlock from routing deadlock (Chapter 3.7).
• Implement a representative per-class virtual-channel allocator in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

Module 5 built the nodes; this chapter places them on the wires. It is where node roles become a floorplan: which router each RN, HN, and SN attaches to, how far apart they sit, and how that distance becomes latency. Getting placement and topology right is a large part of a CHI-based SoC's performance.

It also introduces one of the subtlest correctness issues in the whole protocol: message-class deadlock. If the four channels share fabric buffers, the protocol can wedge even when routing is perfectly deadlock-free. Understanding why — and the virtual channels that prevent it — is essential, and it sets up Module 6's channels directly.

3. Key Terms

4. Previous Chapter Connection

Chapter 3.7 made the topology case in the abstract: bus to ring to crossbar to mesh, judged by bisection bandwidth and hop count, with dimension-order routing to avoid routing deadlock. It had no nodes yet — only "traffic."

Now the nodes are built. This chapter revisits the same fabric knowing exactly who rides it: RN-F CPUs, distributed HN-F homes, SN-F memory. It asks the node's questions — where do I attach, how far to my Home Node, which topology serves my traffic — and surfaces a hazard 3.7 could not: the four message classes can deadlock the fabric even when routing cannot. It is the fabric chapter, grounded in the node taxonomy.

5. Core Concept — nodes on the fabric, channels in lanes

The fabric, seen from the nodes, is three ideas.

• Nodes attach and route by ID. Every RN, HN, and SN attaches at a fabric port and has a Node ID. A transaction is a sequence of packets between node IDs — RN → HN (request), HN → RN (snoop), HN → SN (memory) — and the fabric routes each by its TgtID, over whatever topology is underneath.
• Topology shapes distance. The topology (ring, crossbar, mesh) sets the hop count between any two nodes, and thus latency and bandwidth. A mesh puts most nodes a few hops apart and scales; a ring is simple but grows linearly; a crossbar is one hop but costs area.
• Placement matters. Where on the mesh each node sits changes average latency: memory (SN) near the edge PHYs, Home Nodes distributed for balance (Chapter 4.5), CPU clusters (RN) grouped. Placement is a real floorplan decision, not an afterthought.
• Channels ride separate lanes. The four message classes — REQ, RSP, SNP, DAT — travel on separate virtual channels (or physical networks) on each link. This is not optional: sharing buffers across classes creates a cyclic dependency that deadlocks the protocol.

The synthesis:

The fabric carries the node model physically: nodes attach by ID, topology and placement set the distances between them, and the four channels ride independent lanes so no message class can block another. Routing deadlock (Chapter 3.7) is avoided by turn rules; message-class deadlock is avoided by virtual channels. A correct fabric needs both — and the nodes of Module 5 are what actually sit on it.

6. Engineering Mental Model — a campus with separate mail streams

Picture the SoC as a campus and the fabric as its road network.

• Every building (node) has an address (Node ID) and sits at an intersection (fabric port). Deliveries travel between buildings by address, over the road grid (mesh).
• Where you put buildings matters: put the loading docks (memory) at the campus edge near the highway, spread the mailrooms (Home Nodes) so no one is far from theirs, cluster the offices (CPUs). A bad layout means long trips.
• Critically, run separate delivery streams for different mail types — outbound letters (REQ), receipts (RSP), inquiries (SNP), and parcels (DAT) — each with its own trucks and docks. If they shared docks, a pile-up of parcels could block the receipts needed to clear the parcels — a jam that never resolves (message-class deadlock). Separate streams keep each flowing.

Separate streams on a well-laid-out grid — that is the CHI fabric.

7. Engineering Diagram — nodes placed on the mesh

A requester's traffic goes to its Home Node (a hop or two down), and on a miss the Home Node reaches memory at the edge. Placement makes those paths short.

8. Topologies from the Node View

The same topologies of Chapter 3.7, judged now by what they mean for nodes.

And placement on the winning topology (mesh) follows the traffic:

• Home Nodes distributed so directory load and hop distance spread (Chapter 4.5).
• Memory (SN) at the edges near the DRAM PHYs.
• CPU clusters (RN) grouped, close to the Home Nodes they use most.

Two facts to carry: the mesh is the only topology with no disqualifying term at scale (latency √N, area N, bandwidth grows), and placement turns that topology into real numbers — a well-placed node is a few hops from what it needs, a badly-placed one is a hotspot.

9. Channels on the Fabric — virtual channels and deadlock

Here is the hazard 3.7 could not show, because it had no message classes.

• Four classes share the links. REQ, RSP, SNP, and DAT all traverse the same physical mesh. If they also share the same buffers on each link, a cyclic dependency forms: a REQ occupies a buffer and can only drain by generating a RSP — but the RSP needs a buffer that REQs have filled. Neither moves. The fabric deadlocks even though routing is fine.
• Virtual channels break the cycle. Give each message class its own buffer lane (virtual channel) on every link. Now a RSP always has buffers to drain into, independent of how full the REQ lane is, so the dependency cycle cannot close. The classes are physically independent even while sharing wires.
• This is a different deadlock. Chapter 3.7's deadlock was routing (turn cycles in the channel-dependency graph), fixed by dimension-order routing. This is message-class (protocol) deadlock, from a cyclic dependency between message types, fixed by per-class virtual channels. A correct fabric prevents both.

The point to carry:

The four CHI channels are not just a protocol abstraction — they must be independent on the fabric, each with its own virtual channel, or the protocol deadlocks. A response class must always be able to make progress regardless of request congestion, because responses are what free the resources requests hold. Separate lanes per message class is the structural guarantee, and it is why Module 6 treats the four channels as orthogonal.

10. Transaction Walkthrough — a request across the mesh

CPU0 (RN-F at the top-left) reads a line whose Home Node is at the middle-right.

1. Attach and issue. RN-F builds a REQ packet (TgtID = the Home Node) and injects it at its fabric port, on the REQ virtual channel.
2. Route by ID. The mesh routes the packet by Node ID, hop by hop (dimension-order), to the Home Node's router — a few hops given the placement.
3. Home acts, on its own lanes. The HN-F snoops a holder on the SNP channel and, on a miss, reaches memory (SN-F at the edge) — each message class on its own virtual channel, so none blocks another.
4. Responses return. Data comes back on the DAT channel and completion on RSP, routed by Node ID to CPU0 — again on independent lanes.
5. Latency is placement. The total latency is dominated by hop counts: RN to HN, HN to SN, and back. Good placement kept each leg short; separate VCs kept each leg unblocked.

Every leg was a routed packet on a per-class lane. Topology set the hop counts; virtual channels kept the classes from deadlocking.

11. RTL / Hardware View — per-class virtual-channel allocation

The structural guarantee is one rule: a message may only enqueue into its own class's buffer. Here is that allocation. Representative and combinational.

// Representative per-message-class VC allocation (educational).
// Each of the four CHI channel classes (REQ/RSP/SNP/DAT) has its OWN virtual-
// channel buffer on the link. A message may enqueue only if ITS class's VC has
// space -- never gated by another class -- which breaks the cyclic buffer
// dependency that causes message-class (protocol) deadlock.
module vc_alloc (
  input  logic [1:0] msg_class,      // 0 REQ, 1 RSP, 2 SNP, 3 DAT
  input  logic [3:0] vc_free,        // free bit per class VC
  output logic       accept          // may enqueue into this class's VC
);
  // Gated ONLY by this message's own class VC -- independence is the whole point.
  assign accept = vc_free[msg_class];
endmodule

The same behavior in Verilog-2001:

// Representative per-message-class VC allocation (Verilog-2001).
module vc_alloc (
  input  [1:0] msg_class,
  input  [3:0] vc_free,
  output       accept
);
  assign accept = vc_free[msg_class];
endmodule

And in VHDL:

-- Representative per-message-class VC allocation (VHDL).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity vc_alloc is
  port (
    msg_class : in  std_logic_vector(1 downto 0);
    vc_free   : in  std_logic_vector(3 downto 0);
    accept    : out std_logic
  );
end entity;
 
architecture rtl of vc_alloc is
begin
  accept <= vc_free(to_integer(unsigned(msg_class)));
end architecture;

All three gate a message solely on its own class's VC. That a RSP's admission never depends on the REQ lane's fullness is exactly what keeps responses flowing when requests congest — the anti-deadlock property.

12. Verification View — a class is admitted only by its own VC

Two properties: a message is admitted iff its own class VC is free, and its admission is independent of other classes' fullness.

// Bind to vc_alloc.
// 1. Admission tracks only this message's own class VC.
property p_own_vc_only;
  @(*) accept == vc_free[msg_class];
endproperty
 
// 2. Independence: with this class's VC free, admission holds no matter how
//    full the OTHER classes' VCs are (they cannot block this class).
property p_class_independent;
  @(*) vc_free[msg_class] |-> accept;   // other bits of vc_free are irrelevant
endproperty

The system point, beyond the two checks:

Message-class independence is the fabric's protocol-deadlock guarantee. The cycle that deadlocks a shared-buffer fabric is: REQ holds a buffer → needs a RSP to drain → RSP needs a buffer → buffers full of REQs. Per-class VCs cut the cycle by making the RSP lane's space independent of the REQ lane's, so a response can always drain regardless of request congestion. The property "admission depends only on your own class" is the machine-checkable form of "no message class can block another" — the exact condition that forbids message-class deadlock.

• What it proves: admission is per-class and independent of other classes.
• What it does not prove: routing deadlock-freedom (Chapter 3.7) or end-to-end liveness (needs the full flow).
• Bug signature: a message blocked by another class's buffer → a fabric that wedges under load (the DebugLab).

13. Testbench — each class admitted only by its own VC

Checks that a class is admitted when its VC is free and blocked when only its VC is full, regardless of others.

module tb_vc_alloc;
  logic [1:0] msg_class;
  logic [3:0] vc_free;
  logic accept;
  int errors = 0;
 
  vc_alloc dut (.*);
 
  task automatic check(input logic [1:0] cl, input logic [3:0] free, input logic exp, input string tag);
    msg_class = cl; vc_free = free; #1;
    if (accept !== exp) begin errors++; $display("FAIL [%s] accept=%b exp=%b", tag, accept, exp); end
    else $display("PASS [%s] accept=%b", tag, accept);
  endtask
 
  initial begin
    // RSP (class 1) VC free, REQ (class 0) VC full -> RSP still admitted.
    check(2'd1, 4'b0010, 1'b1, "RSP free, REQ full -> RSP admitted");
    // RSP VC full, others free -> RSP blocked (only its own VC matters).
    check(2'd1, 4'b1101, 1'b0, "RSP full -> RSP blocked");
    // DAT (class 3) free -> admitted regardless of others.
    check(2'd3, 4'b1000, 1'b1, "DAT free -> admitted");
    check(2'd0, 4'b0000, 1'b0, "REQ full -> blocked");
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS [RSP free, REQ full -> RSP admitted] accept=1
PASS [RSP full -> RSP blocked] accept=0
PASS [DAT free -> admitted] accept=1
PASS [REQ full -> blocked] accept=0

14. DebugLab — shared buffers across message classes

buffer pool   contents          note
  full        REQ, REQ, REQ...   requests occupy every buffer
  waiting     RSP / DAT blocked  responses cannot enter -> requests cannot drain

15. Common Mistakes

• Sharing buffers across classes. Assumption: one pool is simpler. Bug: message-class deadlock (the DebugLab). Prevention: separate virtual channels per class.
• Conflating the two deadlocks. Assumption: deadlock-free routing is enough. Bug: protocol deadlock survives. Prevention: prevent routing and message-class deadlock.
• Ignoring placement. Assumption: any node position works. Bug: long paths, hotspots. Prevention: place memory at edges, distribute Home Nodes, group CPUs.
• Choosing a ring at scale. Assumption: a ring is simple enough. Bug: O(N) latency at many nodes. Prevention: mesh for many-core.
• Forgetting nodes route by ID. Assumption: the fabric knows coherence. Bug: misplaced logic. Prevention: the fabric routes Node IDs; coherence is at nodes.
• Treating channels as one lane. Assumption: the four channels are just labels. Bug: coupling and deadlock. Prevention: they are independent traffic classes.

16. Engineering Checklist

• Attach each RN / HN / SN at a fabric port with a Node ID; route by TgtID.
• Choose a mesh for many-core; know the ring/crossbar trade-offs.
• Place memory at edges, distribute Home Nodes, group CPU clusters.
• Give each message class (REQ/RSP/SNP/DAT) its own virtual channel.
• Prevent both routing deadlock (turn rules) and message-class deadlock (VCs).
• Keep coherence at the nodes; the fabric only routes.

17. Key Takeaways

• Nodes (RN/HN/SN) attach at fabric ports with Node IDs; every transaction is routed packets between them.
• Topology (ring/crossbar/mesh) sets node-to-node latency and bandwidth; the mesh scales.
• Placement turns topology into real numbers — memory at edges, Home Nodes distributed, CPUs grouped.
• The four channels must ride separate virtual channels, or a cyclic buffer dependency deadlocks the protocol.
• Message-class deadlock (between message types) is distinct from routing deadlock (turn cycles) — a fabric must prevent both.
• This closes Module 5: every node kind placed on the structure that carries their packets; the model here is representative.

18. Quick Revision

The interconnect fabric. From the node's-eye view: RN/HN/SN attach at fabric ports with Node IDs, and every transaction is routed packets between them (RN→HN, HN→RN, HN→SN) by TgtID over the topology. Ring (O(N) latency), crossbar (1 hop, O(N²) area), mesh (O(√N), scales) trade differently; placement on the mesh — memory at edges, Home Nodes distributed, CPUs grouped — sets real latency. The four channels (REQ/RSP/SNP/DAT) must ride separate virtual channels: sharing buffers lets full REQ buffers block the RSP/DAT that would drain them — a message-class (protocol) deadlock, distinct from Chapter 3.7's routing deadlock. A correct fabric prevents both (turn rules + per-class VCs). Closes the node taxonomy onto the physical structure. Representative model; Module 6 opens the four channels.

Coming Next

Module 6 — CHI Channels. Module 5 built the nodes and placed them on the fabric; the virtual channels at the end of this chapter hinted at what comes next. Module 6 opens up the four CHI channels — REQ, RSP, DAT, SNP — in full: their packet structures, the routing rules every transaction rides on, and why the channels are orthogonal. It begins with a channel overview that turns the four traffic classes you just saw kept independent on the fabric into the precise message types that carry every CHI transaction.