Packet-Based Design

Module 4 · Chapter 4.7 · CHI Architecture Overview

Project thread — 4.6 carried flits over links; this chapter gives those flits meaning as packets, and contrasts them with AXI's signals. 4.8 closes the module with the layered stack.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Contrast packet-based (CHI) with signal-based (AXI) transaction coordination.
• List a packet's self-describing fields — opcode, SrcID, TgtID, TxnID, address, data.
• Explain how the fabric routes a packet by TgtID without inspecting its contents.
• Trace a CHI read as a sequence of packets between nodes.
• Justify why packets, not signals, suit a distributed, multi-hop fabric.
• Implement a representative packet receive/decode block in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

Packets are the abstraction the entire rest of CHI is written in. Transaction flows, snoop sequences, retries — all are described as packets moving between nodes. If you still think in AXI's wires and handshakes, those descriptions read as noise. Once you think in packets — who sent it, to whom, which transaction, what opcode — every flow becomes a short, readable script.

It also crystallizes the shift this module has been building. Nodes (4.1–4.5), links (4.6), and now packets: the mental model is complete when you see a transaction as self-describing messages routed across a fabric, not signals toggling on a shared bus. That is the lens for everything that follows.

3. Key Terms

4. Previous Chapter Connection

Chapter 4.6 showed how flits move across point-to-point links with credited flow control — but the link layer only cares about moving flits, not what they mean. This chapter is the layer above: what those flits say.

The distinction matters. The link layer guarantees a packet's flits arrive intact and in order; the packet's fields then carry the transaction's meaning to the destination. Nodes (4.1–4.5) exchange these packets; links (4.6) carry their flits; and this chapter names the abstraction that ties them together — and contrasts it with the signal-based world of AXI it replaced.

5. Core Concept — meaning in fields, not wires

The difference between AXI and CHI is where a transaction's meaning lives.

• AXI is signal-based. Meaning is encoded in which wire carries a value and when: an address on AWADDR, data on WDATA, coordinated by a VALID/READY handshake per channel. The receiver reconstructs the transaction by tracking wire state across cycles. It works beautifully point to point — but signals cannot be routed across a network.
• CHI is packet-based. A transaction is a sequence of self-describing packets. Each packet carries its own opcode (what), SrcID/TgtID (from/to), TxnID (which transaction), address, and, on DAT, data. The fabric routes a packet by its TgtID without inspecting the rest; the receiver reads meaning from fields, needing no memory of wire state.
• Packets are routable and self-contained. Because a packet names its destination and carries its context, it can traverse a multi-hop fabric, be reordered and re-correlated by TxnID, and be flitized over any link — none of which a wire-level signal can do.

The synthesis:

AXI asks "what is on the wires this cycle, and where is the handshake?" CHI asks "what packet, from whom, to whom, for which transaction?" Moving meaning from wires-over-time into fields-in-a-packet is what lets CHI route, reorder, and scale across a distributed fabric. Packets are not a cosmetic change — they are the abstraction a network needs.

6. Engineering Mental Model — letters versus a telegraph line

AXI is a telegraph line between two stations: you convey a message by the pattern and timing of pulses on a shared wire, and both ends must stay in lock-step to interpret it. It is fast and direct — but it only connects those two stations, and only while both are synchronized.

CHI is the postal system: you write a letter (a packet) with a from-address, a to-address, a reference number, and the contents, and drop it in the network. The post office reads only the to-address to route it; it never opens the envelope. Letters travel any number of hops, arrive possibly out of order, and are matched up by their reference numbers.

• The telegraph's meaning is in timing on a wire; the letter's meaning is in fields on a page.
• You cannot mail a telegraph pulse across a country; you can mail a letter anywhere.

CHI chose letters because it is a network, not a wire.

7. Engineering Diagram — the anatomy of a packet

Every field has a job, and together they make the packet self-contained. The fabric touches only TgtID; the destination reads the rest.

8. A Transaction as a Packet Sequence

A CHI read is not a handshake — it is a short exchange of packets between nodes. CPU0 (RN0) reads a line the Home Node owns, and another requester (RN1) holds it.

Read it top to bottom: five packets, each self-describing, each routed by Node ID. RN0 never tracked a wire — it matched the returning CompData to its request by TxnID 5. That is the packet model in action.

9. Packet vs Signal

The two models compared directly.

Two facts to carry: signal-based coordination is stateful across cycles and local to two endpoints, while packet-based coordination is self-contained and routable. That is precisely why AXI suits a direct link and CHI suits a network.

10. Why Packets for a Distributed Fabric

The choice follows from everything Module 3 and this module established.

• Routing needs an address in the message. A distributed fabric (many Home Nodes, a mesh) must route each message to a specific node. Only a packet carrying a TgtID can be routed; a bare signal cannot.
• Concurrency needs correlation, not wires. Many transactions in flight, completing out of order, must be told apart. SrcID + TxnID in each packet do this; per-channel wire state cannot scale to it.
• Layering needs self-description. Packets let the protocol layer define meaning while the link layer moves flits — the clean separation of 4.6. A packet can be flitized over any link and routed over any topology.
• Scale needs packet-switching. A NoC (Chapter 3.7) is a packet-switched network; packets are its native currency. Signal buses do not scale to it.

The through-line:

Every reason CHI exists — distribution, concurrency, layering, NoC scale — requires messages that name their destination and carry their own context. That is a packet. AXI's signals were the right abstraction for a point-to-point link; CHI's packets are the right abstraction for a distributed, routed fabric. The mental model this module built — nodes exchanging packets over channels and links — is complete.

11. RTL / Hardware View — receiving and decoding a packet

A packet-based node's front door reads a packet's fields: is this for me (TgtID), and what does it say. Here is that receive/decode. Representative and simplified — a fixed illustrative field layout, combinational decode.

// Representative CHI packet receive/decode (educational, illustrative layout).
// A node reads a self-describing packet: it routes to this node by TgtID and
// extracts the fields. Meaning is in the fields, not in wire state over cycles.
module chi_packet_rx #(
  parameter int MY_ID = 4                        // this node's Node ID
)(
  input  logic [63:0] pkt,
  input  logic        pkt_valid,
  output logic        for_me,                     // TgtID matches this node
  output logic [3:0]  opcode,
  output logic [2:0]  src_id,
  output logic [3:0]  txn_id,
  output logic [43:0] addr
);
  wire [2:0] tgt_id = pkt[56:54];                  // routing field
  assign for_me = pkt_valid && (tgt_id == MY_ID[2:0]);
  assign opcode = pkt[63:60];
  assign src_id = pkt[59:57];
  assign txn_id = pkt[53:50];
  assign addr   = pkt[49:6];
endmodule

The same behavior in Verilog-2001:

// Representative CHI packet receive/decode (Verilog-2001).
module chi_packet_rx #(
  parameter MY_ID = 4
)(
  input  [63:0] pkt,
  input         pkt_valid,
  output        for_me,
  output [3:0]  opcode,
  output [2:0]  src_id,
  output [3:0]  txn_id,
  output [43:0] addr
);
  wire [2:0] tgt_id = pkt[56:54];
  assign for_me = pkt_valid & (tgt_id == MY_ID[2:0]);
  assign opcode = pkt[63:60];
  assign src_id = pkt[59:57];
  assign txn_id = pkt[53:50];
  assign addr   = pkt[49:6];
endmodule

And in VHDL:

-- Representative CHI packet receive/decode (VHDL).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity chi_packet_rx is
  generic ( MY_ID : integer := 4 );
  port (
    pkt       : in  std_logic_vector(63 downto 0);
    pkt_valid : in  std_logic;
    for_me    : out std_logic;
    opcode    : out std_logic_vector(3 downto 0);
    src_id    : out std_logic_vector(2 downto 0);
    txn_id    : out std_logic_vector(3 downto 0);
    addr      : out std_logic_vector(43 downto 0)
  );
  end entity;
 
architecture rtl of chi_packet_rx is
  signal tgt_id : std_logic_vector(2 downto 0);
begin
  tgt_id <= pkt(56 downto 54);
  for_me <= '1' when (pkt_valid = '1' and tgt_id = std_logic_vector(to_unsigned(MY_ID, 3))) else '0';
  opcode <= pkt(63 downto 60);
  src_id <= pkt(59 downto 57);
  txn_id <= pkt(53 downto 50);
  addr   <= pkt(49 downto 6);
end architecture;

All three read a self-describing packet: route by TgtID, extract the rest. Nothing depends on wire history — the packet carries its whole meaning. Note that correlating a response needs both SrcID and TxnID together, which is the DebugLab.

12. Verification View — routing and correlation use the right fields

Two properties: the node accepts a packet only when addressed to it, and decode is a pure function of the packet.

// Bind to chi_packet_rx (MY_ID = 4).
// 1. Accept a packet only when its TgtID targets this node.
property p_for_me_iff_targeted;
  @(*) for_me == (pkt_valid && (pkt[56:54] == MY_ID));
endproperty
 
// 2. Decode is deterministic: same packet in -> same fields out.
property p_decode_stable;
  @(posedge clk) (pkt == $past(pkt)) |-> (opcode == $past(opcode) && txn_id == $past(txn_id));
endproperty
assert property (p_decode_stable);

The system point, beyond the two checks:

A packet is self-describing, but only if you read the right fields for the job. The fabric routes on TgtID alone — it must never depend on any other field, or a packet could be misdelivered. And a receiver correlating a response must use SrcID together with TxnID, because TxnID is unique only per source: two different requesters may both use TxnID 5. Read TgtID for routing, the pair (SrcID, TxnID) for correlation — using the wrong field, or too few, is the classic packet-model bug.

• What it proves: routing keys on TgtID; decode is deterministic.
• What it does not prove: that a receiver correlates by the full (SrcID, TxnID) pair — that is its bookkeeping (the DebugLab).
• Bug signature: packets misrouted (wrong routing field) or responses aliased (correlating by TxnID alone).

13. Testbench — decode and route a packet

Builds a packet targeted at this node and checks the decoded fields and for_me.

module tb_chi_packet_rx;
  logic [63:0] pkt;
  logic pkt_valid, for_me;
  logic [3:0] opcode, txn_id;
  logic [2:0] src_id;
  logic [43:0] addr;
  int errors = 0;
 
  localparam int MY_ID = 4;
  chi_packet_rx #(.MY_ID(MY_ID)) dut (.*);
 
  task automatic check(input logic [3:0] op, input logic [2:0] s, input logic [2:0] tg,
                       input logic [3:0] tx, input logic exp_forme, input string tag);
    pkt = {op, s, tg, tx, 44'hABCDEF, 6'h0}; pkt_valid = 1; #1;   // 4+3+3+4+44+6 = 64 bits
    if (for_me !== exp_forme || opcode !== op || src_id !== s || txn_id !== tx) begin
      errors++; $display("FAIL [%s] for_me=%b op=%h src=%h txn=%h", tag, for_me, opcode, src_id, txn_id);
    end else $display("PASS [%s] for_me=%b op=%h src=%h txn=%h", tag, for_me, opcode, src_id, txn_id);
  endtask
 
  initial begin
    check(4'h2, 3'd1, 3'd4, 4'd5, 1'b1, "targeted at me (tgt=4)");
    check(4'h2, 3'd1, 3'd6, 4'd5, 1'b0, "targeted elsewhere (tgt=6)");
    check(4'h8, 3'd2, 3'd4, 4'd9, 1'b1, "different opcode/src/txn");
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS [targeted at me (tgt=4)] for_me=1 op=2 src=1 txn=5
PASS [targeted elsewhere (tgt=6)] for_me=0 op=2 src=1 txn=5
PASS [different opcode/src/txn] for_me=1 op=8 src=2 txn=9

14. DebugLab — correlating a response by TxnID alone

tracker key   SrcID  TxnID  note
    5           1       5    RN1's transaction
    5           2       5    RN2's transaction -> SAME KEY, collides

15. Common Mistakes

• Reasoning in wires. Assumption: CHI has AXI-like signals to track. Bug: looking for handshakes instead of packets. Prevention: think packets — from, to, opcode, TxnID.
• Routing on the wrong field. Assumption: any field can route. Bug: misdelivery. Prevention: the fabric routes on TgtID only.
• Correlating by TxnID alone. Assumption: TxnID is globally unique. Bug: cross-source collision (the DebugLab). Prevention: correlate by (SrcID, TxnID).
• Expecting the fabric to understand packets. Assumption: routers interpret opcodes. Bug: misplaced coherence logic. Prevention: the fabric reads only TgtID; meaning is acted on at nodes.
• Assuming in-order per channel. Assumption: packets arrive in issue order. Bug: mis-matching out-of-order responses. Prevention: re-correlate by ID; order is not guaranteed across the fabric.
• Treating packets as cosmetic. Assumption: packets are just AXI with headers. Bug: missing why they enable routing/scale. Prevention: packets are what make a distributed fabric possible.

16. Engineering Checklist

• Describe every transaction as a sequence of packets between nodes.
• Read each packet's fields — opcode, SrcID, TgtID, TxnID, addr, data.
• Route on TgtID only; never depend on other fields for delivery.
• Correlate responses by the pair (SrcID, TxnID), not TxnID alone.
• Expect out-of-order delivery; re-correlate rather than assume order.
• Keep coherence/meaning at nodes; the fabric only routes.

17. Key Takeaways

• CHI is packet-based: a transaction is a sequence of self-describing packets, not a signal handshake on wires.
• A packet carries opcode (what), SrcID/TgtID (from/to), TxnID (correlate), addr, and data — everything needed to route and act.
• The fabric routes by TgtID without inspecting contents; the destination reads meaning from fields, not wire state.
• Correlate responses by (SrcID, TxnID) — TxnID is unique only per source.
• Packets, not signals, are what make CHI routable, concurrent, layered, and NoC-scalable — the abstraction a network needs.
• With nodes, links, and packets in hand, the CHI mental model is complete; the model here is representative.

18. Quick Revision

Packet-based design. AXI is signal-based — meaning in wires and per-channel VALID/READY handshakes, point to point. CHI is packet-based — a transaction is a sequence of self-describing packets, each carrying opcode (what), SrcID/TgtID (from/to), TxnID (correlate), addr, and data. The fabric routes on TgtID alone, never reading the rest; the receiver reads meaning from fields, not wire state over cycles. Correlate responses by the pair (SrcID, TxnID) — TxnID is unique only per source. A CHI read is five packets (REQ → SNP → RSP/DAT → CompData → CompAck), routed by Node ID and re-correlated by ID, possibly out of order. Packets are what make CHI routable, concurrent, layered, and NoC-scalable — the abstraction a distributed fabric needs. Representative model.

Coming Next

Chapter 4.8 — Layered Architecture. Nodes, links, and packets are in hand — the final architecture chapter stacks them. CHI is organized into layers: a Protocol layer that defines coherence and transactions, a Network layer that routes packets by Node ID, and a Link layer that moves flits with credited flow control. The next chapter details what each layer owns and why the separation matters — closing Module 4 and setting up Module 5, where we return to each node type in full depth.