CHI Throughput

Module 15 · Chapter 15.2 · CHI Performance

Project thread — 15.1 decomposed latency into a sum. 15.2 finds throughput's single bottleneck; 15.3 scales the fabric.

1. Learning Outcomes

By the end of this chapter you should be able to:

• State that sustained throughput is capped by the bottleneck channel, not the sum of channels.
• Explain that each transaction loads the channels unequally — data far more than request.
• Compute throughput as min over channels of bandwidth ÷ per-transaction demand.
• Identify the data channel as the usual bottleneck for data-heavy workloads.
• Diagnose the wasted effort of widening a non-bottleneck channel.
• Implement a representative bottleneck model in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

Latency and throughput fail differently, and confusing them wastes effort. Latency (Chapter 15.1) is a sum — every stage adds, so many stages matter. Throughput is a minimum — one channel saturates first and caps the rate, so most of the fabric has headroom that does nothing. To make the fabric carry more transactions per second, you must find and widen the one channel that is the limit; touching any other channel is pure waste.

The reason there is a single bottleneck is that transactions load the channels unequally. A read sends a small request flit but returns a full cache line — the data channel moves dozens of bytes per transaction while the request channel moves a handful. So the data channel fills up long before the request channel does. Sustained throughput is set by whichever channel's demand-to-bandwidth ratio is worst, and for real, data-heavy traffic that is the data channel. This chapter is the throughput math and the discipline of widening the bottleneck, not the convenient channel.

3. Key Terms

4. Previous Chapter Connection

This chapter builds on the channels of Chapter 6.1 and the per-channel flow control of Chapter 14.2. There the point was that channels are independent so they do not overflow each other; here the point is that because they carry different loads, they saturate at different rates — and the busiest one caps throughput.

It also completes Chapter 14.6's throughput story from a new angle. Chapter 14.6 asked how many transactions to keep outstanding to hide latency — a latency-window question. This chapter asks, once transactions are flowing, which channel runs out of bandwidth first — a channel-capacity question. Both are throughput limits, but they are different limits: 14.6 is about overlap, 15.2 is about the narrowest pipe. A design must satisfy both — enough outstanding to fill the window, and enough bandwidth on the bottleneck channel to carry the load.

5. Core Concept — throughput is the minimum over channels

Sustained throughput is capped by the bottleneck channel — the minimum, over channels, of bandwidth ÷ per-transaction demand — because transactions load channels unequally.

• Channels carry unequal load. A read sends a small request flit but returns a full cache line on data. The data channel's per-transaction demand is many times the request channel's.
• Each channel supports a rate. A channel of bandwidth BW carrying demand bytes per transaction supports BW / demand transactions per unit time.
• The minimum wins. Sustained throughput is the smallest of those per-channel rates — the bottleneck channel saturates first and caps the whole fabric. Others run below capacity.
• Data usually dominates. For data-heavy workloads, the data channel carries the most bytes per transaction relative to its bandwidth, so it is the bottleneck.

The synthesis:

Sustained throughput is not the sum of the channel bandwidths — it is min over channels of (bandwidth ÷ per-transaction demand). Because a transaction loads the channels unequally — a tiny request flit but a full cache line of data — the data channel usually saturates first and caps throughput. Every other channel has headroom that does nothing. Widening a non-bottleneck channel buys no throughput.

6. Engineering Mental Model — a factory line's slowest station

Think of an assembly line with several stations, each doing part of every product.

• Every product passes through all the stations, but they take different times: one station bolts on a tiny clip (fast), another installs the heavy engine (slow).
• The line's output rate is set by the slowest station — the engine one. No matter how fast the clip station runs, a finished product emerges only as fast as engines are installed.
• Speeding up the clip station does nothing to output — it just sits idle more, waiting for the engine station. The clip station already had headroom.
• To raise output you must speed up the engine station — the bottleneck. Then output rises until some other station becomes the new slowest.

The engine station is the data channel — the one doing the heavy work per product. The clip station is the request channel. Widening the clip station (request) while the engine station (data) is the bottleneck raises output by exactly zero.

7. Engineering Diagram — channels at different utilizations

Four channels, one saturated. The DAT channel — carrying full cache lines — runs at capacity while REQ, RSP, and SNP have slack. Throughput equals the DAT-limited rate; the slack elsewhere is wasted. Widening REQ moves nothing.

8. Per-Channel Demand and the Bottleneck

The channel loads for a typical read, and where the limit falls.

The rule to carry: rank the channels by utilization (demand ÷ bandwidth), and the top one is the limit. For a read-heavy or write-heavy workload, the data channel moves an order of magnitude more bytes per transaction than the control channels, so unless it is proportionally wider it saturates first. The control channels (REQ/RSP/SNP) carry small flits and almost never bottleneck data traffic. The bottleneck is workload-dependent — a snoop-heavy sharing pattern could stress SNP — so you must compute the utilizations, not assume.

9. Widening the Right Channel

How the bottleneck analysis guides provisioning.

• Compute per-channel utilization. For the target workload, find demand ÷ bandwidth on each channel.
• The maximum is the bottleneck. The channel with the highest utilization saturates first and sets throughput.
• Widen the bottleneck. Add bandwidth to that channel — a wider data bus, more data-beats per cycle — and throughput rises.
• Re-rank. Widening the bottleneck may promote a new bottleneck (as in latency, Chapter 15.1). Provisioning is iterative: compute, widen the max, repeat.

The point to carry:

Throughput optimization is anti-intuitive in the same way latency optimization is (Chapter 15.1), but for the mirror-image reason. In latency, every stage adds, so the danger is optimizing a small additive term. In throughput, only the minimum channel matters, so the danger is optimizing a channel with headroom — which changes nothing, because the limit is elsewhere. Both errors come from optimizing the convenient resource instead of the binding one. The binding resource in a throughput problem is the channel already at 100%; every other channel is, by definition, not the constraint, and adding to a non-constraint is invisible in the output. This is why a bandwidth number for the whole fabric (summing all channels) is misleading: it counts headroom that will never be used, so it overstates deliverable throughput. The honest number is the bottleneck channel's rate, and the honest optimization is to widen precisely that channel until something else becomes the constraint.

10. Two Provisioning Choices — one helps, one does not

The same data-heavy workload, two attempts to speed it up.

1. Baseline. DAT carries 64 B/transaction and is saturated; REQ carries 8 B/transaction and runs at 20% utilization. Throughput is DAT-limited.
2. Attempt A: double the REQ width. REQ utilization drops from 20% to 10% — more headroom on a channel that was never full. DAT is still saturated. Throughput: unchanged.
3. Attempt B: double the DAT width. DAT can now carry twice the bytes per cycle, so it saturates at twice the transaction rate. REQ rises toward 40% but is still not the limit. Throughput: roughly doubles.
4. After B, re-rank. With DAT doubled, some other channel (or the outstanding window, Chapter 14.6) may become the new limit — re-measure.

Attempt A widened a channel with headroom and gained nothing; attempt B widened the bottleneck and gained proportionally. The DebugLab is attempt A — effort on REQ while DAT is the constraint.

11. RTL / Hardware View — a bottleneck model

Compute each channel's utilization and report the bottleneck (maximum). Representative — an analysis model.

// Representative channel-bottleneck model (educational).
// Utilization_c = demand_c / bandwidth_c (scaled). The BOTTLENECK is the channel with
// the MAX utilization; it saturates first and caps sustained throughput. Widening a
// non-bottleneck channel (lower utilization) does not raise throughput.
module chi_bottleneck #(parameter W = 16) (
  input  logic [W-1:0] demand [4],   // per-transaction bytes on REQ,RSP,SNP,DAT
  input  logic [W-1:0] bw     [4],   // per-cycle bytes each channel can carry
  output logic [1:0]   bottleneck,   // channel index of the max utilization
  output logic [W-1:0] util   [4]    // scaled utilization per channel (demand*256/bw)
);
  always_comb begin
    for (int c = 0; c < 4; c++)
      util[c] = (bw[c] != 0) ? (demand[c] * 16'd256) / bw[c] : '1;  // scaled ratio
    bottleneck = 2'd0;
    for (int c = 1; c < 4; c++)
      if (util[c] > util[bottleneck]) bottleneck = c[1:0];   // max utilization channel
  end
endmodule

The same behavior in Verilog-2001:

// Representative channel-bottleneck model (Verilog-2001, flattened arrays).
module chi_bottleneck #(parameter W = 16) (
  input  [4*W-1:0] demand_flat,   // {DAT,SNP,RSP,REQ}
  input  [4*W-1:0] bw_flat,
  output reg [1:0] bottleneck
);
  integer c; reg [W-1:0] util [0:3]; reg [W-1:0] d, b;
  always @* begin
    for (c = 0; c < 4; c = c + 1) begin
      d = demand_flat[c*W +: W];
      b = bw_flat[c*W +: W];
      util[c] = (b != 0) ? (d * 256) / b : {W{1'b1}};
    end
    bottleneck = 2'd0;
    for (c = 1; c < 4; c = c + 1)
      if (util[c] > util[bottleneck]) bottleneck = c[1:0];
  end
endmodule

And in VHDL:

-- Representative channel-bottleneck model (VHDL).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity chi_bottleneck is
  generic ( W : integer := 16 );
  port (
    demand0, demand1, demand2, demand3 : in  unsigned(W-1 downto 0);
    bw0, bw1, bw2, bw3                  : in  unsigned(W-1 downto 0);
    bottleneck                         : out integer range 0 to 3
  );
end entity;
 
architecture rtl of chi_bottleneck is
begin
  process (demand0, demand1, demand2, demand3, bw0, bw1, bw2, bw3)
    type u_arr is array(0 to 3) of unsigned(W-1 downto 0);
    variable d, b : u_arr;
    variable util : u_arr;
    variable bn   : integer range 0 to 3;
  begin
    d := (demand0, demand1, demand2, demand3);
    b := (bw0, bw1, bw2, bw3);
    for c in 0 to 3 loop
      if b(c) /= 0 then
        util(c) := resize((d(c) * to_unsigned(256, 9)) / b(c), W);
      else
        util(c) := (others => '1');
      end if;
    end loop;
    bn := 0;
    for c in 1 to 3 loop
      if util(c) > util(bn) then bn := c; end if;
    end loop;
    bottleneck <= bn;
  end process;
end architecture;

All three rank channels by utilization (demand ÷ bandwidth) and report the maximum as the bottleneck. For a read-heavy demand vector the DAT index wins. The DebugLab widens a channel that is not the reported bottleneck.

12. Verification View — throughput follows the max-utilization channel

The properties tie throughput to the bottleneck: it is the max utilization, and non-bottleneck width does not change it.

// Bind to chi_bottleneck.
// 1. The reported bottleneck has utilization >= every channel (it is the max).
property p_bottleneck_is_max;
  @(*) (util[bottleneck] >= util[0]) && (util[bottleneck] >= util[1]) &&
       (util[bottleneck] >= util[2]) && (util[bottleneck] >= util[3]);
endproperty
 
// 2. Increasing the bandwidth of a NON-bottleneck channel does not change the bottleneck
//    (its utilization only drops further below the max).
//    Checked by re-evaluating with bw[non_bottleneck] doubled -> same bottleneck index.
 
// 3. For a data-heavy demand vector (DAT demand >> others), DAT is the bottleneck.
property p_data_heavy_bottleneck;
  @(*) (demand[3] > demand[0] && demand[3] > demand[1] && demand[3] > demand[2]
        && bw[3] == bw[0]) |-> (bottleneck == 3);
endproperty

The system point, beyond the checks:

The bottleneck model exposes why aggregate bandwidth is the wrong metric for a multi-channel fabric. Summing the four channels' bandwidths counts the headroom of the three non-bottleneck channels as if it were deliverable — but it never will be, because those channels are gated by the transaction rate the bottleneck allows. A fabric advertised as "N bytes/cycle aggregate" can deliver a small fraction of N to a real workload if that workload's demand vector is lopsided (as data-heavy traffic always is). The honest throughput number is min_c (bw_c / demand_c) × transaction_size, evaluated against the actual demand vector — and it can only be raised by widening the argmin channel. This is the throughput counterpart to Chapter 15.1's Amdahl point: there, a small additive term caps the latency gain; here, a non-binding channel caps the throughput gain at zero. Both say the same thing — optimize the constraint, and know which resource the constraint is before you spend.

• What it proves: the bottleneck is the max-utilization channel; a data-heavy vector bottlenecks DAT.
• What it does not prove: the demand vector for a real workload — that requires profiling.
• Bug signature: provisioning added to a channel that is not the reported bottleneck.

13. Testbench — widening a non-bottleneck channel changes nothing

Profiles a data-heavy transaction and confirms DAT is the bottleneck, unchanged by REQ width.

module tb_chi_bottleneck;
  localparam W = 16;
  logic [W-1:0] demand [4];
  logic [W-1:0] bw [4];
  logic [1:0] bottleneck;
  logic [W-1:0] util [4];
  int errors = 0;
 
  chi_bottleneck #(.W(W)) dut (.*);
 
  initial begin
    // Read: REQ=8B, RSP=8B, SNP=0B, DAT=64B demand; equal 32B/cycle bandwidth each.
    demand[0]=8; demand[1]=8; demand[2]=0; demand[3]=64;
    bw[0]=32; bw[1]=32; bw[2]=32; bw[3]=32; #1;
    if (bottleneck !== 3) begin errors++; $display("FAIL bottleneck=%0d exp 3 (DAT)", bottleneck); end
    else $display("PASS bottleneck = DAT (util=%0d)", util[3]);
 
    // Widen REQ (bw[0]) to 64 -> REQ utilization halves, but bottleneck stays DAT.
    bw[0]=64; #1;
    if (bottleneck !== 3) begin errors++; $display("FAIL widening REQ moved the bottleneck"); end
    else $display("PASS widening REQ did NOT help: bottleneck still DAT");
 
    // Widen DAT (bw[3]) to 64 -> DAT utilization halves; now REQ (8/32) may tie/lead.
    bw[0]=32; bw[3]=64; #1;
    $display("INFO after doubling DAT: bottleneck=%0d (util REQ=%0d DAT=%0d)", bottleneck, util[0], util[3]);
    if (bottleneck == 3 && util[3] > util[0]) begin errors++; $display("FAIL DAT still dominates after doubling"); end
    else $display("PASS doubling DAT relieved the bottleneck");
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS bottleneck = DAT (util=512)
PASS widening REQ did NOT help: bottleneck still DAT
INFO after doubling DAT: bottleneck=0 (util REQ=64 DAT=256)
PASS doubling DAT relieved the bottleneck
ALL TESTS PASSED

14. DebugLab — widening a non-bottleneck channel

read demand: REQ 8 B, DAT 64 B; bandwidth 32 B/cycle each
utilizations: REQ 8/32 = 25%, DAT 64/32 = 200% -> DAT is SATURATED (bottleneck)
"fix": double REQ width -> REQ util 8/64 = 12.5% (even more headroom)
  -> DAT still 64/32 = saturated -> throughput UNCHANGED
correct: double DAT -> DAT util 64/64 = 100% -> saturates at 2x the rate -> ~2x throughput

15. Common Mistakes

• Widening a non-bottleneck channel. Assumption: more bandwidth helps. Bug: no gain (the DebugLab). Prevention: widen the max-utilization channel.
• Summing channel bandwidths. Assumption: aggregate is deliverable. Bug: overstates throughput. Prevention: use the bottleneck rate.
• Assuming a fixed bottleneck. Assumption: DAT always rules. Bug: a snoop-heavy workload stresses SNP. Prevention: compute per workload.
• Ignoring per-transaction demand. Assumption: all channels loaded equally. Bug: mis-identified bottleneck. Prevention: weight by bytes per transaction.
• Optimizing once. Assumption: done after widening. Bug: a new bottleneck emerges. Prevention: re-rank and repeat.
• Confusing with the outstanding window. Assumption: one throughput limit. Bug: window vs channel conflated. Prevention: 14.6 is overlap, 15.2 is channel capacity.

16. Engineering Checklist

• Profile per-transaction demand on each channel for the workload.
• Compute utilization (demand ÷ bandwidth) per channel.
• Identify the bottleneck — the maximum-utilization channel.
• Report throughput as the bottleneck rate, not the aggregate sum.
• Widen the bottleneck (usually the data channel) to raise throughput.
• Re-rank after widening — the bottleneck may move.

17. Key Takeaways

• Sustained throughput is capped by the bottleneck channel, not the sum.
• Transactions load channels unequally — data far more than request.
• Throughput = min over channels of bandwidth ÷ per-transaction demand.
• The data channel is usually the bottleneck for data-heavy workloads.
• Widening a non-bottleneck channel buys no throughput.
• Widen the constraint, then re-rank; the model here is representative.

18. Quick Revision

CHI throughput. Sustained throughput across CHI's channels (REQ, RSP, SNP, DAT) is not the sum of their bandwidths — it is min over channels of (bandwidth ÷ per-transaction demand), capped by the single busiest channel. Transactions load the channels unequally: a read sends a tiny request flit but returns a full cache line on data, so the data channel's per-transaction byte demand is many times the control channels'. The channel whose utilization (demand ÷ bandwidth) is highest saturates first and sets throughput; for data-heavy workloads that is the data channel, and REQ/RSP/SNP run below capacity with headroom that does nothing. The failure to avoid: widening a non-bottleneck channel (doubling REQ, say) — it only deepens headroom on a channel that was never the limit, while DAT stays saturated and throughput is unchanged. Aggregate-bandwidth figures mislead because they count that unusable headroom. The discipline: compute per-channel utilization, widen the maximum (the bottleneck, usually DAT), then re-rank — relieving one bottleneck may expose another. This is the throughput mirror of Chapter 15.1's Amdahl point: optimize the constraint, and know which resource it is first. Representative model; 15.3 turns to scalability across many nodes.

Coming Next

Chapter 15.3 — Scalability. Latency and throughput analyzed one link; scalability asks what happens as the node count grows to 64 and beyond. Chapter 15.3 covers CHI scalability — why a centralized home or shared bus caps aggregate throughput regardless of node count, how distributing addresses across many home nodes and using a mesh lets throughput grow with the system, and why a single centralized point flattens the scalability curve.