Point-to-Point Links

Module 4 · Chapter 4.6 · CHI Architecture Overview

Project thread — 4.5 distributed the nodes; this chapter wires them with point-to-point links. 4.7 then shows how those links carry packets instead of signal handshakes.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Contrast CHI's point-to-point links with AHB's shared bus.
• Define a flit as the flow-control unit and relate it to a packet (one or more flits).
• Explain credited (L-Credit) flow control and why it needs no shared stall signal.
• Describe why back-pressure in CHI is per link and per channel.
• Trace a multi-flit data packet moving across a link one flit per credit.
• Implement a representative link-layer flit sender in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

The link layer is the floor CHI's whole protocol stands on. Every packet you will trace in later modules is really a sequence of flits moving across point-to-point links under credited flow control. Knowing how a link works — flits, credits, per-channel back-pressure — is what lets you reason about throughput, deadlock, and buffering, not just correctness.

It is also the clearest embodiment of CHI's break from AHB. Where AHB serialized everyone through one shared medium with a global HREADY, CHI gives each pair of components a private link with local credits. That shift — from shared and arbitrated to private and credited — is why CHI scales, and this chapter is where you see it at the wire level.

3. Key Terms

4. Previous Chapter Connection

Chapter 4.5 spread coherency across many Home Nodes on the mesh. But "on the mesh" hides a question: how are two adjacent components actually connected, and how does one send to the other without overrunning it?

The answer is the link layer. The mesh of Chapter 3.7 is built from point-to-point links between routers and components, and each link runs the credited flit protocol this chapter details. The credit counter you built in Chapter 4.1 was one piece of it; here we assemble the whole picture — links, flits, per-channel credits — and contrast it with the shared bus it replaces.

5. Core Concept — private links, flits, and credits

CHI's link layer has three ideas.

• Point-to-point links. Each connection joins exactly two adjacent components — a component and its router, or two routers. It is not shared: no arbitration for the wires, and transfers on different links happen concurrently. Adding components adds links, not contention.
• Flits as the unit. The link moves flits (flow-control units). A protocol message becomes one or more flits: REQ, RSP, and SNP messages are typically a single flit; a DAT message carrying a cache line is several data flits. The link layer transfers flits; the protocol layer assembles them into packets.
• Credited flow control. The receiver grants the sender L-Credits equal to its available buffer slots. The sender may launch a flit only while it holds a credit; the receiver returns a credit as it frees a slot. This is per channel — REQ, RSP, SNP, DAT each have their own credits on the link.

Why this replaces the shared bus:

On a shared bus, one medium serializes everyone and a single HREADY stalls all masters at once. On a CHI link, each connection is private and each channel is independently credited, so back-pressure is local: a full DAT buffer withholds DAT credits without touching REQ, and a busy neighbor slows only its own link. There is no shared stall signal and no wire arbitration — which is exactly why the fabric scales.

6. Engineering Mental Model — couriers with prepaid slots

Picture each link as a dedicated courier route between two buildings.

• The route is private — this courier serves only these two buildings, so it never waits in line behind traffic to elsewhere (point-to-point, no arbitration).
• Goods move in standard boxes (flits). A small item is one box; a large shipment (a cache line) is several boxes sent one after another.
• Before shipping, the courier checks prepaid delivery slots at the destination (credits): it may send only as many boxes as the destination has confirmed room for, and the destination issues a new slot each time it unpacks one. No dock ever overflows, and no global "stop everything" signal is needed — just local slot-counting.
• Each kind of goods (REQ, RSP, SNP, DAT) has its own slots, so a backlog of one kind never blocks another.

Many private courier routes, each metered by prepaid slots, is CHI's link layer. Compare AHB: one shared truck everyone queues for, halted by a single dispatcher.

7. Engineering Diagram — a point-to-point link

One private link, two directions of information: flits forward, credits back. Multiply this across every adjacent pair in the mesh and you have CHI's transport — no shared medium anywhere in it.

8. Point-to-Point vs Shared Bus

The contrast with AHB (Chapter 3.2) is the whole point.

Two facts to carry: no arbitration and no shared stall means one busy link or channel cannot freeze the fabric, and capacity grows with links rather than contention growing with masters. Everything the shared bus did globally, the link does locally.

9. Flits and Credited Flow Control

Two mechanics deserve a closer look.

• Flits. A flit is the fixed unit the link transfers and meters. Framing a message into flits lets the link move a small control message (one flit) and a large data payload (several flits) with the same credit machinery. The receiver knows a packet is complete when it sees the packet's last flit — flit framing is what delimits packets on the link.
• Credits. The receiver advertises buffer space as credits; the sender spends one per flit and the receiver returns one per freed slot. Because the credit count never exceeds the buffer depth (Chapter 4.1), a held credit guarantees a free slot — overflow is impossible. And because credits are per channel, DAT back-pressure never blocks REQ.

The combination:

Flits give the link a uniform unit to transfer and delimit; credits give it back-pressure without a shared signal. Together they let every link run at full rate, independently, with buffers that provably never overflow. The protocol layer above hands down packets and never worries about wire-level flow control — the link layer guarantees it.

10. Transaction Walkthrough — a data packet crosses a link

A Home Node sends a cache line (four DAT flits) to a Request Node over their link.

1. Packetize. The 64-byte line becomes a 4-flit DAT packet at the link layer.
2. Check credit. The sender has DAT credits for B's buffer. With a credit available, it launches flit 0 and spends a credit.
3. Stream flits. Each cycle it holds a DAT credit, it sends the next flit — 1, 2, then 3 — spending a credit per flit. If credits run out mid-packet, it pauses until B returns one; the pause affects only this DAT channel.
4. Mark the last. On flit 3, the sender asserts last flit, telling B the packet is complete. B assembles the four flits into the line.
5. Return credits. As B drains flits from its buffer into the protocol layer, it returns DAT credits, letting the sender continue future packets.

The whole transfer used only local DAT credits on one link. Meanwhile REQ, RSP, and SNP flits moved on their own credits, and other links carried other packets — all concurrently.

11. RTL / Hardware View — a link-layer flit sender

The link sender's job is to stream a packet's flits, one per credit, and mark the last. Here it is for a fixed-size packet. Representative and simplified — flit sequencing and credit gating, building on the credit counter of Chapter 4.1.

// Representative link-layer flit sender (educational).
// Streams an NFLITS-flit packet, launching one flit per cycle while it has both
// a flit remaining and a credit. Marks the final flit as `flit_last` so the
// receiver can delimit the packet. Credits come from a Chapter 4.1 credit counter.
module link_flit_sender #(
  parameter int NFLITS = 4                      // flits per packet (e.g. a cache line)
)(
  input  logic clk,
  input  logic rst_n,
  input  logic pkt_start,                        // begin a new packet
  input  logic have_credit,                      // a link credit is available
  output logic flit_valid,                        // launch a flit this cycle
  output logic flit_last,                         // this flit is the packet's last
  output logic busy                               // a packet is in progress
);
  logic [2:0] remaining;                          // flits left to send
  wire send = (remaining != 3'd0) && have_credit; // send if a flit remains + credit
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n)                                 remaining <= 3'd0;
    else if (pkt_start && remaining == 3'd0)    remaining <= NFLITS[2:0];  // load packet
    else if (send)                              remaining <= remaining - 3'd1;
  end
 
  assign flit_valid = send;
  assign flit_last  = send && (remaining == 3'd1);   // last flit: one remaining
  assign busy       = (remaining != 3'd0);
endmodule

The same behavior in Verilog-2001:

// Representative link-layer flit sender (Verilog-2001).
module link_flit_sender #(
  parameter NFLITS = 4
)(
  input       clk, rst_n,
  input       pkt_start, have_credit,
  output      flit_valid, flit_last,
  output      busy
);
  reg  [2:0] remaining;
  wire send = (remaining != 3'd0) & have_credit;
 
  always @(posedge clk or negedge rst_n)
    if (!rst_n)                              remaining <= 3'd0;
    else if (pkt_start && remaining == 3'd0) remaining <= NFLITS[2:0];
    else if (send)                           remaining <= remaining - 3'd1;
 
  assign flit_valid = send;
  assign flit_last  = send & (remaining == 3'd1);
  assign busy       = (remaining != 3'd0);
endmodule

And in VHDL:

-- Representative link-layer flit sender (VHDL).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity link_flit_sender is
  generic ( NFLITS : integer := 4 );
  port (
    clk, rst_n  : in  std_logic;
    pkt_start   : in  std_logic;
    have_credit : in  std_logic;
    flit_valid  : out std_logic;
    flit_last   : out std_logic;
    busy        : out std_logic
  );
end entity;
 
architecture rtl of link_flit_sender is
  signal remaining : unsigned(2 downto 0) := (others => '0');
  signal snd       : std_logic;
begin
  snd <= '1' when (remaining /= 0 and have_credit = '1') else '0';
 
  process(clk, rst_n)
  begin
    if rst_n = '0' then
      remaining <= (others => '0');
    elsif rising_edge(clk) then
      if pkt_start = '1' and remaining = 0 then
        remaining <= to_unsigned(NFLITS, 3);
      elsif snd = '1' then
        remaining <= remaining - 1;
      end if;
    end if;
  end process;
 
  flit_valid <= snd;
  flit_last  <= '1' when (snd = '1' and remaining = 1) else '0';
  busy       <= '1' when remaining /= 0 else '0';
end architecture;

All three stream exactly NFLITS flits per packet, one per credit, and raise flit_last on the final flit so the receiver can delimit the packet. The last-flit marker is small and easy to get wrong — the DebugLab.

12. Verification View — every packet is exactly NFLITS flits with one last

Two properties: a flit only launches with a credit, and the last-flit marker fires exactly on the final flit.

// Bind to link_flit_sender (NFLITS = 4).
// 1. Never launch a flit without a credit.
property p_no_flit_without_credit;
  @(posedge clk) disable iff (!rst_n) flit_valid |-> have_credit;
endproperty
assert property (p_no_flit_without_credit);
 
// 2. flit_last is asserted exactly when the last flit is sent (remaining == 1).
property p_last_on_final_flit;
  @(posedge clk) disable iff (!rst_n)
    flit_last == (flit_valid && (remaining == 3'd1));
endproperty
assert property (p_last_on_final_flit);

The system point, beyond the two checks:

Flit framing is what tells the receiver where a packet ends. If flit_last fires on the wrong flit, the receiver's packet boundary is wrong: too early and it assembles a truncated packet (a short cache line) and mistakes the next flit for a new packet's start; too late and it waits for a flit that never comes. Correct framing — last flit marked on exactly the last flit — plus credit-gated sending is the link layer's entire contract. Get both right and the protocol layer above sees clean, complete packets.

• What it proves: flits are credit-gated and the packet is delimited on the true last flit.
• What it does not prove: end-to-end ordering across links or channels (protocol-layer properties).
• Bug signature: a receiver assembling short or run-on packets — a misplaced last-flit marker.

13. Testbench — stream a packet, watch the credits gate it

Starts a 4-flit packet, withholds a credit mid-stream, and checks flit count and last.

module tb_link_flit_sender;
  logic clk = 0, rst_n, pkt_start, have_credit;
  logic flit_valid, flit_last, busy;
  int flits = 0, lasts = 0;
 
  link_flit_sender #(.NFLITS(4)) dut (.*);
  always #5 clk = ~clk;
 
  always @(posedge clk) if (flit_valid) begin flits++; if (flit_last) lasts++; end
 
  initial begin
    rst_n = 0; pkt_start = 0; have_credit = 1; @(posedge clk); rst_n = 1;
    pkt_start = 1; @(posedge clk); pkt_start = 0;   // start a 4-flit packet
    @(posedge clk);                                  // flit 1 sent
    have_credit = 0; @(posedge clk); @(posedge clk); // stall: no credit -> no flits
    have_credit = 1;                                 // resume
    repeat (5) @(posedge clk);                        // finish remaining flits
    if (flits == 4 && lasts == 1)
      $display("PASS 4 flits, 1 last (credit stall respected)");
    else
      $display("FAIL flits=%0d lasts=%0d", flits, lasts);
    $finish;
  end
endmodule

Expected output:

PASS 4 flits, 1 last (credit stall respected)

14. DebugLab — the last flit marked one flit too soon

flit  remaining(at send)  flit_last  note
  0        4                 0
  1        3                 0
  2        2                 1        <- WRONG: marked last too early
  3        1                 0        receiver thinks this starts a new packet

15. Common Mistakes

• Thinking CHI has a shared bus. Assumption: an interconnect is a bus. Bug: expecting arbitration and a global stall. Prevention: CHI uses private point-to-point links with local credits.
• Merging channel credits. Assumption: one credit pool per link. Bug: DAT back-pressure blocking REQ. Prevention: credits are per channel.
• Confusing flit with packet. Assumption: one message is one flit. Bug: mishandling multi-flit DAT. Prevention: a packet is one or more flits; data is several.
• Misframing the last flit. Assumption: any last-flit timing works. Bug: truncated / run-on packets (the DebugLab). Prevention: mark the last flit on exactly the final flit.
• Expecting a global stall. Assumption: something like HREADY stalls the fabric. Bug: designing for a signal that does not exist. Prevention: back-pressure is local, via credits.
• Sending without a credit. Assumption: send when data is ready. Bug: buffer overflow (Chapter 4.1). Prevention: no credit, no flit.

16. Engineering Checklist

• Treat each connection as a private point-to-point link, not a shared medium.
• Move data as flits; frame each packet and mark its last flit precisely.
• Gate every flit on a credit; keep credits at or below the receiver's buffer depth.
• Keep credits per channel — REQ / RSP / SNP / DAT independent.
• Expect local back-pressure only — no shared stall signal.
• Verify multi-flit packets are exactly N flits with one last-flit marker.

17. Key Takeaways

• CHI connects components with dedicated point-to-point links — no shared medium, no wire arbitration.
• The link layer moves flits (flow-control units); a packet is one or more flits — control messages one, a cache line several.
• Credited (L-Credit) flow control gates every flit on a held credit; credits are per channel, so one channel's back-pressure never blocks another.
• Back-pressure is local to a link and channel — there is no shared stall signal like AHB's HREADY.
• Flit framing delimits packets: the last-flit marker must fire on exactly the final flit, or the receiver mis-frames.
• Point-to-point links plus credits are why the fabric scales; the model here is representative.

18. Quick Revision

Point-to-point links. CHI's link layer connects adjacent components with dedicated point-to-point links — no shared bus, no arbitration. It moves flits (flow-control units); a packet is one or more flits (REQ/RSP/SNP usually one, a cache line several DAT flits). Credited (L-Credit) flow control: the receiver grants credits equal to its buffer slots, the sender launches a flit only against a held credit, the receiver returns a credit per freed slot — so buffers never overflow. Credits are per channel (REQ/RSP/SNP/DAT independent), so back-pressure is local; there is no shared HREADY-style stall. Flit framing delimits packets — the last-flit marker must fire on exactly the final flit, or the receiver truncates and desyncs. Private links plus local credits are why the fabric scales. Representative model.

Coming Next

Chapter 4.7 — Packet-Based Design. Point-to-point links carry flits — but what those flits mean is the next abstraction up. The final architecture chapter contrasts CHI's packet-based design with AXI's signal-based handshakes: where AXI coordinates on wires (VALID/READY per channel), CHI sends self-describing packets whose fields carry the whole transaction. We will see why packets, not signals, are the right abstraction for a distributed, routed, credited fabric — and how they close out the CHI mental model this module built.