Layered Architecture

Module 4 · Chapter 4.8 · CHI Architecture Overview — module finale

Project thread — Module 4 built nodes, links, and packets. This chapter stacks them into CHI's layers and closes the module. Module 5 returns to each node kind in full depth.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Name CHI's three layers — Protocol, Network, Link — and what each owns.
• Map the module's ideas to layers: nodes/coherence to Protocol, packets/routing to Network, flits/credits to Link.
• Explain why each layer treats the others' concerns as opaque.
• Justify the separation: independence, reuse across topologies, and layer-localized debugging.
• Recognize a layering violation — one layer depending on another's content.
• Implement a representative network-layer packet framer in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

Layering is how the CHI specification is organized, so knowing the layers tells you where any rule lives: a coherence rule is Protocol, a routing rule is Network, a flow-control rule is Link. That map makes the spec navigable and makes debugging tractable — a symptom points at a layer.

It is also the idea that ties Module 4 together. Nodes, packets, and flits were introduced separately; the layered stack shows they are one design seen at three levels. And the discipline of layering — each layer minding only its concern — is what lets CHI run over different fabrics and evolve. Internalize the stack and the whole architecture snaps into place.

3. Key Terms

4. Previous Chapter Connection

The last three chapters each described one layer without naming it. Chapters 4.2–4.5 built the nodes and their coherence behavior — that is the Protocol layer. Chapter 4.7 described packets routed by Node ID — that is the Network layer. Chapter 4.6 described flits moving with credits — that is the Link layer.

This chapter names the stack and makes the boundaries explicit. It is a synthesis, not new machinery: it shows that coherence, routing, and transport are three separable concerns, and that keeping them separate is what makes CHI implementable, reusable, and debuggable. With the stack in place, Module 4's mental model is complete.

5. Core Concept — three layers, three concerns

CHI separates a message's journey into three layers, each owning one question.

• Protocol layer — what a message means. Coherence lives here: transaction types (ReadShared, WriteBack…), cache states, the RN/HN/SN behaviors, snoop generation and response, and ordering via CompAck. It generates and consumes protocol messages and knows nothing about how they are routed or transported.
• Network layer — where a message goes. This layer addresses each message with SrcID/TgtID and delivers it across the fabric by Node ID. It routes; it does not interpret coherence. To it, the protocol content is opaque payload with a destination attached.
• Link layer — how a message crosses a link. This layer frames packets into flits and moves them between two adjacent components with credited (L-Credit) flow control, per channel. It guarantees flits arrive intact and in order; it does not know what they mean or where they ultimately go.
• Physical layer — beneath the Link layer, the actual wires and signaling that carry flits.

The synthesis:

A message descends the stack at the sender — Protocol builds it, Network addresses it, Link flitizes and transmits it — and ascends at the receiver in reverse. Each layer touches only its own fields and treats the rest as opaque. That discipline is why coherence (Protocol) does not depend on topology (Network) or flit width (Link), and why the same protocol runs over a mesh, a ring, or a crossbar.

6. Engineering Mental Model — writing, addressing, shipping

Sending a parcel through a company has three roles, and CHI's layers mirror them exactly.

• The author writes the letter's contents and decides what it says (Protocol). They do not care how it travels.
• The mailroom puts it in an envelope and writes the destination address (Network). They never read the letter; they only need the address.
• The courier carries envelopes between depots, one van-load at a time, checking there is room at the next depot before sending (Link, with credits). They never open envelopes or read addresses beyond "next hop."

Each role does its job with no knowledge of the others' internals. Swap couriers (a different link) or reroute through new depots (a different topology) and the author's letters are unchanged. That independence — author, mailroom, courier — is CHI's layering.

7. Engineering Diagram — the CHI layer stack

Three layers, three concerns, one message passing through all of them. The horizontal cuts are the whole design: each is a clean interface, so what is above never depends on what is below.

8. What Each Layer Owns

The division of responsibility, laid out.

Two facts to carry: each layer has a distinct set of fields/units it owns, and a layer only reads its own. The Network layer routes on SrcID/TgtID and never on the opcode; the Link layer moves flits and never on their meaning. Fields are the contract between layers.

9. Why Layering Matters

The separation is not bookkeeping — it buys three concrete properties.

• Independence. Coherence (Protocol) is defined without reference to topology or flit width. You can change the mesh dimensions, the flit size, or the number of Home Nodes without touching the coherence protocol — because those live in different layers.
• Reuse across fabrics. The same Protocol layer rides any Network/Link that carries its packets. A mesh, a ring, or a crossbar are Network/Link choices; the coherence protocol above is unchanged. One protocol, many fabrics.
• Layer-localized debugging. A symptom points at a layer: garbled coherence state is Protocol; a misrouted packet is Network; a dropped or overrun flit is Link. Bugs do not smear across the stack — if the layers are kept clean.

The rule that protects all three:

A layer must use only its own fields. When a lower layer reaches up into an upper layer's content — the Network routing on the protocol opcode, say — the layers couple, and every benefit evaporates: the protocol can no longer change opcodes freely, the fabric can no longer be swapped, and a bug now spans two layers. Keeping each layer opaque to the others is what makes the architecture hold together.

10. Transaction Walkthrough — a request down and up the stack

CPU0's RN issues a ReadShared. Follow it descending the sender's stack and ascending the Home Node's.

1. Protocol (build). The RN's Protocol layer creates a ReadShared message with an opcode, address, and TxnID — pure coherence intent.
2. Network (address). The Network layer stamps SrcID (this RN) and TgtID (the owning Home Node, from the address map), forming a routable packet. It does not read the opcode.
3. Link (flitize + send). The Link layer frames the packet into flit(s) and transmits with a credit, hop by hop, never interpreting the payload.
4. Link (receive). At the Home Node, the Link layer reassembles the flits into a packet and returns credits — knowing nothing of coherence.
5. Network (deliver). The Network layer sees TgtID matches this node and hands the packet up — without reading the opcode.
6. Protocol (act). The Home Node's Protocol layer reads the opcode and address and runs coherence: serialize, directory, snoop, respond.

Each layer did its one job and passed the message on. The opcode was read only at the two Protocol layers; the Node IDs only at the two Network layers; the flits only at the two Link layers.

11. RTL / Hardware View — a network-layer packet framer

The clean boundary between Protocol and Network is where a message becomes routable. Here is that framer: it takes an opaque protocol message and stamps routing fields, without interpreting coherence. Representative and simplified — a fixed field layout.

// Representative Network-layer packet framer (educational).
// Takes an opaque protocol message (opcode + address, meaning owned by the
// Protocol layer) and stamps routing fields: SrcID (this node) and TgtID (the
// owning Home Node from the System Address Map). The Network layer adds
// addressing; it does NOT interpret the opcode.
module net_framer #(
  parameter int MY_ID = 0                        // this node's SrcID
)(
  input  logic [3:0]  msg_opcode,                 // from Protocol layer (opaque here)
  input  logic [43:0] msg_addr,
  input  logic [3:0]  msg_txnid,
  input  logic [2:0]  tgt_from_sam,               // owning HN, from the address map
  output logic [63:0] pkt                          // routable network packet
);
  // Assemble {opcode, SrcID, TgtID, TxnID, addr, pad} — opcode passes through
  // unchanged; the framer never branches on its value.
  assign pkt = {msg_opcode, MY_ID[2:0], tgt_from_sam, msg_txnid, msg_addr, 6'd0};
endmodule

The same behavior in Verilog-2001:

// Representative Network-layer packet framer (Verilog-2001).
module net_framer #(
  parameter MY_ID = 0
)(
  input  [3:0]  msg_opcode,
  input  [43:0] msg_addr,
  input  [3:0]  msg_txnid,
  input  [2:0]  tgt_from_sam,
  output [63:0] pkt
);
  assign pkt = {msg_opcode, MY_ID[2:0], tgt_from_sam, msg_txnid, msg_addr, 6'd0};
endmodule

And in VHDL:

-- Representative Network-layer packet framer (VHDL).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity net_framer is
  generic ( MY_ID : integer := 0 );
  port (
    msg_opcode   : in  std_logic_vector(3 downto 0);
    msg_addr     : in  std_logic_vector(43 downto 0);
    msg_txnid    : in  std_logic_vector(3 downto 0);
    tgt_from_sam : in  std_logic_vector(2 downto 0);
    pkt          : out std_logic_vector(63 downto 0)
  );
end entity;
 
architecture rtl of net_framer is
begin
  -- opcode passes through unchanged; the framer never branches on it
  pkt <= msg_opcode
         & std_logic_vector(to_unsigned(MY_ID, 3))
         & tgt_from_sam
         & msg_txnid
         & msg_addr
         & "000000";
end architecture;

All three stamp routing fields onto the protocol message and pass the opcode through unchanged, without branching on it. That the framer never inspects the opcode is the layering discipline in one line — the Network layer routes, it does not interpret.

12. Verification View — the network layer carries protocol content unchanged

Two properties: the routing fields are stamped correctly, and the protocol payload survives framing untouched.

// Bind to net_framer (MY_ID = 3).
// 1. Routing fields are stamped correctly: SrcID = MY_ID, TgtID = SAM result.
property p_routing_fields;
  @(*) (pkt[59:57] == MY_ID) && (pkt[56:54] == tgt_from_sam);
endproperty
 
// 2. The protocol payload passes through unchanged (opaque to the Network layer).
property p_payload_preserved;
  @(*) (pkt[63:60] == msg_opcode) && (pkt[49:6] == msg_addr) && (pkt[53:50] == msg_txnid);
endproperty

The system point, beyond the two checks:

The layering contract is: the Network layer adds addressing and preserves protocol content bit-for-bit, never branching on it. Property 2 — payload preserved regardless of opcode value — is the machine-checkable statement of "the Network layer does not interpret coherence." If a future change made the framer behave differently for some opcode, property 2 would fail, catching the layering violation at the boundary. Clean layers are not just tidy; they are verifiable.

• What it proves: routing fields correct; protocol payload preserved for any opcode.
• What it does not prove: downstream routing or coherence — those are other layers' checks.
• Bug signature: a layer whose behavior depends on another layer's field (the DebugLab).

13. Testbench — frame a message, confirm payload survives

Frames several messages and checks the routing fields and payload preservation across opcodes.

module tb_net_framer;
  logic [3:0]  msg_opcode, msg_txnid;
  logic [43:0] msg_addr;
  logic [2:0]  tgt_from_sam;
  logic [63:0] pkt;
  int errors = 0;
 
  localparam int MY_ID = 3;
  net_framer #(.MY_ID(MY_ID)) dut (.*);
 
  task automatic check(input logic [3:0] op, input logic [2:0] tg,
                       input logic [3:0] tx, input logic [43:0] a, input string tag);
    msg_opcode = op; tgt_from_sam = tg; msg_txnid = tx; msg_addr = a; #1;
    if (pkt[63:60] !== op || pkt[59:57] !== MY_ID || pkt[56:54] !== tg ||
        pkt[53:50] !== tx || pkt[49:6] !== a) begin
      errors++; $display("FAIL [%s] pkt=%h", tag, pkt);
    end else $display("PASS [%s] src=%0d tgt=%0d op=%h preserved", tag, MY_ID, tg, op);
  endtask
 
  initial begin
    check(4'h2, 3'd5, 4'd7, 44'hABCDE, "ReadShared");
    check(4'hF, 3'd0, 4'd1, 44'h12345, "different opcode (still preserved)");
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS [ReadShared] src=3 tgt=5 op=2 preserved
PASS [different opcode (still preserved)] src=3 tgt=0 op=f preserved
ALL TESTS PASSED

14. DebugLab — routing on the opcode: a layering violation

opcode   TgtID   routed to   note
 0x2       5        HN5       known opcode -> correct
 0xF       5        (dropped) NEW opcode -> router has no case -> lost

15. Common Mistakes

• Blurring the layers. Assumption: it is all one protocol. Bug: rules placed in the wrong layer, coupling. Prevention: coherence is Protocol, routing is Network, flits are Link — keep them separate.
• Routing on protocol content. Assumption: the fabric can peek at opcodes. Bug: misroute on new opcodes (the DebugLab). Prevention: route on TgtID only.
• Link inspecting meaning. Assumption: the link can prioritize by transaction type. Bug: link coupled to protocol. Prevention: the Link layer moves opaque flits; priority is signaled by its own fields.
• Assuming one topology. Assumption: CHI is a mesh, period. Bug: baking mesh assumptions into Protocol. Prevention: topology is Network/Link; the Protocol is topology-independent.
• Debugging across layers. Assumption: a bug is everywhere. Bug: wasted effort. Prevention: localize by symptom — coherence (Protocol), routing (Network), flits (Link).
• Treating physical as protocol. Assumption: wires and coherence are one concern. Bug: conflating PHY issues with protocol. Prevention: the Physical layer is separate.

16. Engineering Checklist

• Place every rule in its layer: coherence → Protocol, routing → Network, flits → Link.
• Route on TgtID only; never branch the fabric on protocol opcodes.
• Keep the Link layer payload-opaque; carry priority in its own fields, not the opcode.
• Preserve protocol content unchanged across the Network layer.
• Localize bugs by layer from the symptom.
• Keep the Protocol layer topology-independent so it rides any fabric.

17. Key Takeaways

• CHI stacks into three layers: Protocol (coherence — what), Network (routing by Node ID — where), Link (flits + credits — how), over the Physical layer.
• The module's ideas map cleanly: nodes → Protocol, packets → Network, flits → Link.
• Each layer uses only its own fields and treats the others as opaque — the fields are the inter-layer contract.
• The separation buys independence (coherence free of topology/flit width), reuse (one protocol, many fabrics), and layer-localized debugging.
• A layering violation — like routing on the opcode — couples layers and breaks all three benefits; route on TgtID alone.
• With the stack in place, Module 4's model is complete; the model here is representative.

18. Quick Revision

Layered architecture. CHI has three layers over the physical wires. Protocol owns coherence — transactions, cache states, RN/HN/SN behavior, snoops, CompAck ordering (WHAT a message means). Network owns routing — it addresses packets by Node ID (SrcID/TgtID) and delivers them across the fabric (WHERE it goes), never reading the opcode. Link owns transport — flits with L-Credit, per-channel flow control between neighbors (HOW it crosses a link), never reading meaning. The module maps on: nodes → Protocol, packets → Network, flits → Link. Each layer touches only its own fields and treats others as opaque, buying independence (coherence free of topology), reuse (one protocol, many fabrics), and layer-localized debugging. Routing on the opcode is a layering violation — route on TgtID alone. Representative model; Module 5 details each node kind.

Coming Next

Module 5 — CHI System Components. Module 4 gave you the architecture: nodes, links, packets, and the layered stack. Module 5 zooms back into the nodes and details the full taxonomy — RN-F, RN-D, HN-F, HN-I, SN-F, SN-I — one kind at a time, with the exact role, state, and behavior of each, plus the cache agents and fabric topologies they sit on. It opens with RN-F, the fully coherent Request Node: the CPU-cluster requester that participates completely in coherency, the archetype every other node is defined against.