Link Flow Control

Module 14 · Chapter 14.2 · CHI Flow Control

Project thread — 14.1 was the credit mechanism on one channel. 14.2 replicates it per channel; 14.3 is deadlock avoidance across channels.

1. Learning Outcomes

By the end of this chapter you should be able to:

• State that each CHI channel (REQ, RSP, SNP, DAT) has its own buffer and credit pool.
• Explain that credits are per-channel — a credit reserves a slot in one channel's buffer.
• Describe why a credit can never be spent on a different channel.
• Explain how per-channel accounting lets channels make progress independently.
• Diagnose the overflow from a shared credit count or cross-using credits.
• Implement a representative per-channel credit block in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

The credit mechanism (Chapter 14.1) protects one buffer. A CHI link has several — one per channel — and a single shared count cannot protect them, because a credit only means "room" for the specific buffer it was granted against. A response credit says a response slot is free; it says nothing about the request buffer. Spend it to send a request and you send into a buffer that may be full — an overflow, exactly the failure credits exist to prevent.

Per-channel accounting also buys independence, which is a correctness property, not just performance. Because each channel has its own credits, a channel that stalls — its buffer full, no credits returning — stalls alone; the others keep moving. That separation is what makes deadlock avoidance possible (Chapter 14.3): responses can drain even when requests are blocked. Collapse the channels into one pool and you lose both the overflow protection and the independence. This chapter is why flow control must be replicated per channel.

3. Key Terms

4. Previous Chapter Connection

Chapter 14.1 established the credit mechanism for a channel; this chapter is the multi-channel reality. CHI's channels — REQ, RSP, SNP, DAT (Chapter 6.1) — are physically separate flit streams, each landing in its own receive buffer.

The independence has appeared before without a name. Data flows on DAT while responses flow on RSP (Chapter 13.3); a snoop travels on SNP while the request sits on REQ (Chapter 9.2). Those channels never shared a queue — and this chapter is why: separate buffers, separate credits. It also sets up Chapter 14.3, where the ordering of channel dependencies prevents deadlock — a rule that only makes sense because the channels are independently flow-controlled in the first place. This chapter is the per-channel foundation those rely on.

5. Core Concept — one credit pool per channel

Each CHI channel has its own receive buffer, so each has its own credit pool, and a credit is valid only for the channel it was granted against.

• Separate buffers. REQ, RSP, SNP, and DAT each land in a distinct buffer at the receiver — there is no shared queue.
• Separate credit pools. Each channel runs the credit mechanism (Chapter 14.1) independently — its own count, its own grants, its own returns.
• Credits are channel-specific. A REQ credit reserves a REQ-buffer slot. It can be spent only to send a REQ flit — never a RSP, SNP, or DAT flit.
• Independent progress. Because the pools are separate, a channel that runs out of credit stalls only itself; the other channels, with their own credits, keep moving.

The synthesis:

A CHI link is several independent channels (REQ, RSP, SNP, DAT), each with its own receive buffer and therefore its own credit pool. A credit is channel-specific — it reserves a slot in one channel's buffer and may be spent only on that channel. Per-channel accounting protects each buffer separately and lets channels make progress independently. Sharing or cross-using credits breaks both — a flit sent against another channel's credit overflows its own channel's buffer.

6. Engineering Mental Model — separate parking lots

Think of four separate parking lots (channel buffers), one per department, each with its own count of free spaces.

• Each lot issues its own parking passes (credits) up to its own capacity. A pass for Lot A guarantees a space in Lot A — and nowhere else.
• A driver heading to Lot A must hold a Lot A pass. A pass for Lot B is useless here: Lot A might be full, and the Lot B pass says nothing about Lot A's spaces.
• Because each lot tracks its own passes against its own spaces, one lot filling up turns away only its own drivers; the other three lots keep admitting cars.
• Now imagine a single shared pass pool for all four lots. A driver holds a pass because Lot B had a free space, drives to full Lot A anyway, and there is nowhere to park — a car blocks the entrance. That blocked car is an overflowed channel buffer.

A pass must match the lot. A credit must match the channel. One shared pool loses the guarantee that a pass means a space where you are going.

7. Engineering Diagram — four channels, four credit loops

Four channels, four independent credit loops into four separate buffers. No arrow crosses from one channel's credits to another's buffer — that crossing is exactly the bug. Each loop is the Chapter 14.1 mechanism, replicated and kept apart.

8. Per-Channel vs Shared Accounting

The contrast that defines link flow control.

The rule to carry: a credit must name the buffer it reserves, and the only way to guarantee that is one pool per buffer. A shared count cannot say which buffer has room, so a credit drawn from it is meaningless the moment channels differ in occupancy — and they always do. Per-channel pools make each credit a precise promise about a specific buffer, which is the only kind of promise the no-overflow invariant (Chapter 14.1) can rest on.

9. Why Credits Cannot Cross Channels

The prohibition, made explicit.

• A credit is a buffer reservation. From Chapter 14.1, a credit is a reservation for one slot in a specific buffer. Channels have different buffers.
• Occupancy differs per channel. At any instant the REQ buffer might be full while the DAT buffer is empty — the channels carry different traffic at different rates.
• A cross-used credit reserves the wrong buffer. Spending a DAT credit to send a REQ flit reserves a DAT slot for a REQ flit — but the REQ flit needs a REQ slot, which may not exist. The REQ buffer overflows.
• Independence requires separation. Even short of overflow, a shared pool couples the channels: a busy channel drains the pool and starves idle channels, destroying the independent progress the design needs.

The point to carry:

Multiplexing several logical streams over one physical link tempts a designer to economize with a single credit pool — one counter is simpler than four. But flow-control credits are not fungible, because they are claims against distinct physical resources. This is the same category of error as treating memory addresses in different address spaces as interchangeable: the number looks the same, but it names a different thing. A credit's entire meaning is "buffer X has a free slot," so a credit is only spendable where X is the buffer you are about to write. Channels exist precisely so that different traffic classes do not contend for the same buffer — collapsing their credits silently reintroduces the contention the channel separation was designed to remove, and does so in the one place (flow control) where the cost is a dropped flit rather than merely a slow one.

10. Sending on Four Channels — independent budgets

A node with traffic on all four channels, each with its own credit budget.

1. REQ credits exhausted, DAT credits available. The node has sent its REQ budget; the REQ count is 0. Its DAT count is still positive.
2. A new REQ flit waits. With 0 REQ credits, the node cannot send the REQ flit — it waits for a REQ credit to return. (Only the REQ channel is stalled.)
3. A DAT flit sends freely. The node sends a data flit against its DAT credit — the REQ stall does not block DAT. Independent progress.
4. A REQ credit returns. The receiver frees a REQ slot; the REQ count → 1. The waiting REQ flit can now send.
5. No cross-use anywhere. At no point did the node spend a DAT credit to send the REQ flit — that would have overflowed the REQ buffer.

The REQ stall was contained to REQ, and the other channels flowed because each had its own budget. The DebugLab is a node with a shared count that sends the REQ flit against leftover DAT credit — into a full REQ buffer.

11. RTL / Hardware View — per-channel credit counters

Each channel has its own counter; a send on a channel requires that channel's credit. Representative — an array of independent counters.

// Representative per-channel link-credit block (educational).
// One credit counter PER channel. A send on channel c requires credits[c] > 0, and
// decrements/increments credits[c] ONLY. Credits never cross channels: a credit for
// one channel can never authorize a send on another.
module chi_link_credits #(parameter NCH = 4, parameter CW = 4) (
  input  logic            clk, rst_n,
  input  logic [NCH-1:0]  credit_return,   // per-channel returns
  input  logic [NCH-1:0]  want_send,       // per-channel send requests
  input  logic [CW-1:0]   init_credits [NCH],
  output logic [NCH-1:0]  can_send,        // per-channel: a credit is available
  output logic [NCH-1:0]  do_send          // per-channel: a flit is sent
);
  logic [CW-1:0] cr_q [NCH];
 
  always_comb begin
    for (int c = 0; c < NCH; c++) begin
      can_send[c] = (cr_q[c] != 0);              // this channel's own credit
      do_send[c]  = want_send[c] && can_send[c];  // no cross-channel authorization
    end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int c = 0; c < NCH; c++) cr_q[c] <= init_credits[c];
    end else begin
      for (int c = 0; c < NCH; c++) begin
        // Each channel accounts independently against its OWN buffer.
        case ({credit_return[c], do_send[c]})
          2'b10:   cr_q[c] <= cr_q[c] + 1'b1;
          2'b01:   cr_q[c] <= cr_q[c] - 1'b1;
          default: cr_q[c] <= cr_q[c];
        endcase
      end
    end
  end
endmodule

The same behavior in Verilog-2001 (two explicit channels for clarity):

// Representative per-channel link-credit block (Verilog-2001, REQ + DAT shown).
module chi_link_credits #(parameter CW = 4) (
  input  wire          clk, rst_n,
  input  wire          req_return, dat_return, req_want, dat_want,
  input  wire [CW-1:0] req_init, dat_init,
  output wire          req_can, dat_can, req_send, dat_send
);
  reg [CW-1:0] req_q, dat_q;
  assign req_can  = (req_q != {CW{1'b0}});
  assign dat_can  = (dat_q != {CW{1'b0}});
  assign req_send = req_want & req_can;   // REQ credit only
  assign dat_send = dat_want & dat_can;   // DAT credit only
 
  always @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      req_q <= req_init; dat_q <= dat_init;
    end else begin
      if (req_return & ~req_send)      req_q <= req_q + 1'b1;
      else if (~req_return & req_send) req_q <= req_q - 1'b1;
      if (dat_return & ~dat_send)      dat_q <= dat_q + 1'b1;
      else if (~dat_return & dat_send) dat_q <= dat_q - 1'b1;
    end
  end
endmodule

And in VHDL (REQ + DAT shown):

-- Representative per-channel link-credit block (VHDL, REQ + DAT shown).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity chi_link_credits is
  generic ( CW : integer := 4 );
  port (
    clk, rst_n                       : in  std_logic;
    req_return, dat_return           : in  std_logic;
    req_want, dat_want               : in  std_logic;
    req_init, dat_init               : in  unsigned(CW-1 downto 0);
    req_can, dat_can                 : out std_logic;
    req_send, dat_send               : out std_logic
  );
end entity;
 
architecture rtl of chi_link_credits is
  signal req_q, dat_q   : unsigned(CW-1 downto 0) := (others => '0');
  signal req_has, dat_has : std_logic;
  signal rs, ds         : std_logic;
begin
  req_has  <= '0' when req_q = 0 else '1';   -- a REQ credit is available
  dat_has  <= '0' when dat_q = 0 else '1';   -- a DAT credit is available
  rs       <= req_want and req_has;          -- REQ credit only
  ds       <= dat_want and dat_has;          -- DAT credit only
  req_can  <= req_has;
  dat_can  <= dat_has;
  req_send <= rs;
  dat_send <= ds;
 
  process (clk, rst_n)
  begin
    if rst_n = '0' then
      req_q <= req_init; dat_q <= dat_init;
    elsif rising_edge(clk) then
      if req_return = '1' and rs = '0' then req_q <= req_q + 1;
      elsif req_return = '0' and rs = '1' then req_q <= req_q - 1; end if;
      if dat_return = '1' and ds = '0' then dat_q <= dat_q + 1;
      elsif dat_return = '0' and ds = '1' then dat_q <= dat_q - 1; end if;
    end if;
  end process;
end architecture;

All three keep a separate counter per channel and gate each channel's send on its own credit — no channel's credit can authorize another's send. The DebugLab is a single shared counter driving every channel.

12. Verification View — a send needs that channel's own credit

The properties that keep channels independent: each send consumes only its channel's credit, and no channel oversends.

// Bind to chi_link_credits (NCH channels).
// 1. A send on channel c requires channel c's own credit.
property p_send_needs_own_credit(int c);
  @(posedge clk) disable iff (!rst_n)
    do_send[c] |-> (cr_q[c] != 0);
endproperty
 
// 2. A send on channel c never changes another channel's count.
property p_no_cross_channel_effect(int c, int d);
  @(posedge clk) disable iff (!rst_n)
    (c != d && do_send[c] && !credit_return[d] && !do_send[d]) |=> (cr_q[d] == $past(cr_q[d]));
endproperty
 
// 3. Each channel independently never sends at zero (per-channel no-overflow).
//    Replicated over all channels c.

The system point, beyond the checks:

The verification story mirrors the design story: the single-channel proof of Chapter 14.1 is replicated, once per channel, with an added obligation that the replicas do not interfere. That non-interference property — a send on channel c leaves channel d's count untouched — is the formal statement of independence, and it is what the whole module ultimately relies on. Once you have it, deadlock avoidance (Chapter 14.3) can reason about channels one at a time, backpressure (Chapter 14.4) can throttle one channel without freezing the rest, and congestion on the data channel cannot silently strangle the response channel. Conversely, a single shared counter fails non-interference by construction: every send touches the one count that every channel reads, so the channels are maximally coupled, and no per-channel reasoning above it is sound. Independence is not a bonus property here; it is the load-bearing assumption of everything built on top.

• What it proves: each send needs its own channel's credit; sends do not affect other channels.
• What it does not prove: the cross-channel ordering rules — that is deadlock avoidance (Chapter 14.3).
• Bug signature: a send on one channel changing another's count — a shared or cross-used pool.

13. Testbench — a full REQ channel must not borrow DAT credit

Exhausts REQ credit while DAT has slack and checks REQ cannot send.

module tb_chi_link_credits;
  localparam CW = 4;
  logic clk = 0, rst_n = 0;
  logic req_return = 0, dat_return = 0, req_want = 0, dat_want = 0;
  logic [CW-1:0] req_init, dat_init;
  logic req_can, dat_can, req_send, dat_send;
  int errors = 0;
 
  chi_link_credits #(.CW(CW)) dut (.*);
  always #5 clk = ~clk;
 
  initial begin
    req_init = 4'd1;   // REQ has just 1 credit
    dat_init = 4'd4;   // DAT has plenty
    @(posedge clk) rst_n = 1;
 
    // Spend the single REQ credit.
    @(posedge clk) req_want = 1;
    @(posedge clk) req_want = 0;   // REQ count now 0
 
    // Now REQ wants to send again while DAT has spare credit.
    @(posedge clk) req_want = 1; dat_want = 1;
    #1;
    if (req_send) begin errors++; $display("FAIL REQ sent with 0 REQ credit (borrowed DAT?)"); end
    else $display("PASS REQ blocked: no REQ credit despite DAT slack");
    if (!dat_send) begin errors++; $display("FAIL DAT could not send despite its own credit"); end
    else $display("PASS DAT sends on its own credit, independent of REQ");
 
    @(posedge clk) req_want = 0; dat_want = 0;
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS REQ blocked: no REQ credit despite DAT slack
PASS DAT sends on its own credit, independent of REQ
ALL TESTS PASSED

14. DebugLab — a shared credit pool across channels

shared_credits = 3   (total across REQ + DAT pools)
REQ buffer: FULL (0 real slots)     DAT buffer: 3 free slots
node wants to send a REQ flit -> shared_credits = 3 > 0 -> SEND
  -> REQ flit lands in REQ buffer -> REQ buffer has no room -> OVERFLOW -> flit dropped
the 3 "credits" were DAT slots, not REQ slots
correct: req_credits = 0 -> REQ send BLOCKED; dat_credits = 3 -> DAT free

15. Common Mistakes

• A shared credit pool. Assumption: one count is simpler. Bug: cross-channel overflow (the DebugLab). Prevention: one pool per channel.
• Cross-using credits. Assumption: a credit is a credit. Bug: send into a full buffer. Prevention: spend a channel's credit only on that channel.
• Sizing all channels alike. Assumption: equal buffers. Bug: mismatch to real capacity. Prevention: size each pool to its buffer.
• Coupling channel stalls. Assumption: stall one, stall all. Bug: lost independence. Prevention: per-channel accounting.
• Ignoring starvation. Assumption: a shared pool is fair. Bug: hot channel blocks idle ones. Prevention: separate pools.
• Forgetting a channel. Assumption: only data needs credits. Bug: unprotected REQ/RSP/SNP buffers. Prevention: a pool for every channel.

16. Engineering Checklist

• Give each channel (REQ, RSP, SNP, DAT) its own credit pool.
• Match each pool to that channel's own buffer capacity.
• Gate each channel's sends on its own credit — never cross-use.
• Account decrements/increments per channel.
• Confirm one channel's stall does not block the others.
• Confirm no send on a channel changes another channel's count.

17. Key Takeaways

• A CHI link is several independent channels, each with its own buffer.
• Each channel has its own credit pool — credits are per-channel.
• A credit reserves a slot in one channel's buffer; it can be spent only there.
• Per-channel accounting gives independent progress — one stall stalls one channel.
• A shared pool lets one channel overflow another and starves idle channels.
• Replicate the credit mechanism per channel; the model here is representative.

18. Quick Revision

Link flow control. A CHI link is not one channel but several independent ones — REQ, RSP, SNP, DAT — each with its own receive buffer. Because each channel has its own buffer, each has its own credit pool, running the Chapter 14.1 credit mechanism independently. Credits are per-channel: a REQ credit reserves a REQ-buffer slot and can be spent only to send a REQ flit — never a RSP, SNP, or DAT flit. This gives independent progress — a channel that exhausts its credit stalls only itself, while the others keep flowing — which is the foundation deadlock avoidance (Chapter 14.3) and backpressure (Chapter 14.4) build on. The failure to avoid: a shared credit count (or cross-using one channel's credit on another). A shared count is nonzero whenever any channel has room, so it can authorize a send into a full channel — overflowing that channel's buffer and dropping the flit — and a busy channel draining the shared pool starves idle ones. A credit must name the buffer it reserves; one pool per buffer is the only way to keep that promise. Representative model; 14.3 covers deadlock avoidance across channels.

Coming Next

Chapter 14.3 — Deadlock Avoidance. Independent channels can still deadlock if their dependencies form a cycle. Chapter 14.3 covers deadlock avoidance — the channel-dependency rules that keep the request, response, snoop, and data channels acyclic, why a response must always be able to drain without waiting on a new request, and how a single dependency cycle can freeze the whole fabric under congestion.