CHI Latency Anatomy

Module 15 · Chapter 15.1 · CHI Performance

Project thread — Module 14 hid latency behind pipelining. 15.1 decomposes it; 15.2 turns to throughput.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Name the serial stages of a CHI round-trip — issue, traversal, lookup, access, return.
• State that total latency is the sum of those stages.
• Explain that one stage usually dominates — typically the memory access on a miss.
• Apply the Amdahl principle — optimizing a minor stage yields negligible gain.
• Diagnose wasted effort spent on a non-dominant stage.
• Implement a representative latency accumulator in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

Latency is the number that limits a single thread's progress, and Module 14 showed you can hide it with enough outstanding transactions — but only up to the point where the outstanding window is full. Beyond that, and for any latency-sensitive access, you must actually reduce it. You cannot reduce what you cannot see, and a round-trip presented as one opaque delay hides which part is worth attacking.

Decomposition is the tool. A CHI transaction is a chain of stages — issue, interconnect traversal, home lookup, the access, data return — and the total is their sum. Because the stages are wildly unequal (a memory read dwarfs an interconnect hop), the total is usually dominated by one term. The engineering payoff is the Amdahl consequence: effort spent on a small term barely moves the total, while the same effort on the dominant term pays off proportionally. This chapter teaches you to see the breakdown, so your optimization lands where the time actually is.

3. Key Terms

4. Previous Chapter Connection

Every stage in this decomposition is something you have already built. Issue is a request flit (Chapter 7.1); traversal is routing across the mesh, hop by hop (Chapter 6.7); home lookup is the directory access (Chapter 11.1); the access is a snoop (Chapter 9.1) or a memory read (Chapter 13.5); data return is the DAT-channel path back (Chapter 13.3). This chapter does not add a mechanism — it adds up the ones you know.

It also completes Chapter 14.6's picture. There, latency was a single number L to hide with outstanding transactions; here that L is opened into its parts. And the two connect: reducing the dominant term shrinks L, which (by Little's Law) reduces the outstanding count needed to saturate a link. Understanding the anatomy of L is therefore both a latency win and a throughput enabler — the reason this performance module opens here.

5. Core Concept — a sum of serial stages, one dominant

A CHI round-trip is the sum of serial stages, and because the stages are unequal, one usually dominates — attack that one.

• The stages are serial. Issue → interconnect traversal → home lookup → the access (snoop or memory) → data return. Each must finish before the next, so the total is their sum.
• The access is usually dominant. On a directory miss, the memory read is typically the largest term — often an order of magnitude beyond a single interconnect hop. On a hit to a nearby peer, a snoop is far cheaper.
• Amdahl governs optimization. Cutting a minor stage changes the total only by that stage's small share. Cutting the dominant stage changes the total proportionally. Effort must follow the dominant term.
• Measure to attribute. You cannot know the dominant term without decomposing the measured latency — guessing (or optimizing the visible-but-small stage) wastes effort.

The synthesis:

A CHI transaction's latency is the sum of issue, interconnect traversal, home lookup, the access (snoop or memory read), and data return. The stages are unequal, so one — usually the memory access on a miss — dominates. By Amdahl's principle, only reducing the dominant term meaningfully cuts the total; optimizing a minor stage is effort spent for negligible end-to-end gain.

6. Engineering Mental Model — a road trip with one long leg

Think of a trip made of several legs, with one leg far longer than the rest.

• The trip is: walk to the car (issue), drive across town to the highway (interconnect traversal), check the map (home lookup), the long highway drive to the next city (the access — a memory read), and drive back home (data return).
• The total time is the sum of the legs. But the legs are wildly different: the town driving is minutes, the highway leg is hours.
• If you want to get there faster, shaving thirty seconds off the walk to the car is pointless — the highway leg swamps it. The only leg worth optimizing is the long one: take a faster route, or avoid the trip entirely (cache the destination).
• An engineer who spends a week "optimizing the walk to the car" — polishing a stage that is 1% of the trip — has done real work for no measurable result. The clock is set by the highway.

The highway leg is the memory access; the walk to the car is the interconnect hop you were tempted to optimize because it was easy to see. Attack the long leg.

7. Engineering Diagram — the round-trip, stage by stage

Five serial stages, one large. The issue, traversal, lookup, and return are each a handful of cycles; the memory read is many. The total is their sum, and it tracks the memory term. The DebugLab optimizes one of the small stages and wonders why the total did not move.

8. The Latency Stages Compared

Each stage and its typical weight.

The rule to carry: the total is a sum, and the sum is set by its largest term. On a cache miss, that term is the memory read — so the biggest latency lever is avoiding the miss (caching, a system-level cache, Chapter 13.2) or speeding the memory path. On a hit to a close peer, the access is cheap and traversal may become the largest term — so distance (home placement, topology) matters. The dominant term is workload-dependent; you must measure which stage rules before optimizing.

9. Attacking the Dominant Term

How decomposition guides optimization.

• Decompose first. Break the measured round-trip into per-stage cycles. Only then is the dominant term visible.
• Identify the maximum. Find the largest stage. On misses it is usually memory; on hits it may be traversal.
• Attack that stage. Cut the dominant term — add a system-level cache to reduce memory accesses, place homes closer to reduce traversal, or widen the memory path.
• Re-measure. Reducing the old dominant term may promote a new dominant term. Optimization is iterative: decompose, attack the max, repeat.

The point to carry:

Amdahl's law is usually stated for parallel speedup, but its latency form is just as sharp: if a stage is a fraction f of the total, then no matter how much you optimize it, you cannot reduce the total by more than f. Shave a stage that is 5% of the round-trip to zero and you have gained 5% — the theoretical ceiling of that effort, rarely reached. This is why "easy" optimizations are so often worthless: the stage that is easy to attack is frequently easy because it is small and well-understood, while the dominant term (a memory miss) is large because it is hard. The discipline is to resist optimizing what is convenient and instead measure, find the term with the largest f, and attack that — even when it is the harder engineering problem. A decomposition is the artifact that keeps you honest, because it shows the f of each stage and makes the wasteful optimization obvious before you spend a week on it.

10. Two Reads — miss-dominated vs hit-dominated

The same fabric, two workloads with different dominant terms.

1. Miss-heavy workload. Reads mostly miss the directory and hit memory. Stages: issue 2, traverse 8, lookup 3, memory 120, return 8 — total 141 cycles.
2. The dominant term is memory (120 of 141). Shaving traversal from 8 to 4 saves 4 cycles — a 3% improvement. Adding a system-level cache that turns 80% of misses into hits removes most of the 120 — a transformative win.
3. Hit-heavy workload. A system-level cache now serves most reads. Stages: issue 2, traverse 8, lookup 3, SLC access 12, return 8 — total 33 cycles.
4. The dominant term is now traversal (8+8 of 33). With memory gone, distance rules — home placement and topology become the lever. The optimization target moved.

The right optimization depended on the decomposition. In workload 1, attacking traversal (the tempting, easy stage) wastes effort; attacking memory transforms it. The DebugLab is optimizing traversal in workload 1.

11. RTL / Hardware View — a latency accumulator

Sum the per-stage cycle counts and identify the dominant stage. Representative — a measurement model, not a datapath.

// Representative latency accumulator (educational).
// Sum the per-stage cycle counts into a total, and report which stage is DOMINANT
// (the maximum). Optimization effort should target the dominant stage -- reducing a
// minor stage changes the total only by that stage's small share (Amdahl).
module chi_latency_accum #(parameter W = 16) (
  input  logic [W-1:0] t_issue,     // issue / serialization
  input  logic [W-1:0] t_traverse,  // interconnect traversal (both ways lumped here)
  input  logic [W-1:0] t_lookup,    // home directory lookup
  input  logic [W-1:0] t_access,    // snoop or memory read (usually dominant)
  input  logic [W-1:0] t_return,    // data return traversal
  output logic [W+2:0] total,       // sum of stages
  output logic [2:0]   dominant,    // index of the largest stage (0..4)
  output logic [W-1:0] dominant_val
);
  logic [W-1:0] s [5];
  always_comb begin
    s[0]=t_issue; s[1]=t_traverse; s[2]=t_lookup; s[3]=t_access; s[4]=t_return;
    total = t_issue + t_traverse + t_lookup + t_access + t_return;
    dominant     = 3'd0;
    dominant_val = s[0];
    for (int i = 1; i < 5; i++)
      if (s[i] > dominant_val) begin dominant = i[2:0]; dominant_val = s[i]; end
  end
endmodule

The same behavior in Verilog-2001:

// Representative latency accumulator (Verilog-2001).
module chi_latency_accum #(parameter W = 16) (
  input  [W-1:0] t_issue, t_traverse, t_lookup, t_access, t_return,
  output [W+2:0] total,
  output reg [2:0]   dominant,
  output reg [W-1:0] dominant_val
);
  assign total = t_issue + t_traverse + t_lookup + t_access + t_return;
  integer i;
  reg [W-1:0] s [0:4];
  always @* begin
    s[0]=t_issue; s[1]=t_traverse; s[2]=t_lookup; s[3]=t_access; s[4]=t_return;
    dominant = 3'd0; dominant_val = s[0];
    for (i = 1; i < 5; i = i + 1)
      if (s[i] > dominant_val) begin dominant = i[2:0]; dominant_val = s[i]; end
  end
endmodule

And in VHDL:

-- Representative latency accumulator (VHDL).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity chi_latency_accum is
  generic ( W : integer := 16 );
  port (
    t_issue, t_traverse, t_lookup, t_access, t_return : in unsigned(W-1 downto 0);
    total        : out unsigned(W+2 downto 0);
    dominant     : out integer range 0 to 4;
    dominant_val : out unsigned(W-1 downto 0)
  );
end entity;
 
architecture rtl of chi_latency_accum is
  type stage_arr is array(0 to 4) of unsigned(W-1 downto 0);
begin
  process (t_issue, t_traverse, t_lookup, t_access, t_return)
    variable s : stage_arr;
    variable dom : integer range 0 to 4;
    variable domv : unsigned(W-1 downto 0);
  begin
    s := (t_issue, t_traverse, t_lookup, t_access, t_return);
    total <= resize(t_issue,W+3) + resize(t_traverse,W+3) + resize(t_lookup,W+3)
             + resize(t_access,W+3) + resize(t_return,W+3);
    dom := 0; domv := s(0);
    for i in 1 to 4 loop
      if s(i) > domv then dom := i; domv := s(i); end if;
    end loop;
    dominant     <= dom;
    dominant_val <= domv;
  end process;
end architecture;

All three sum the stages and report the dominant one. The dominant stage is where optimization pays — the DebugLab is reducing a non-dominant stage and seeing the total barely change.

12. Verification View — the total tracks the dominant term

The properties tie the total to its parts: the total is the sum, and the dominant stage is the maximum.

// Bind to chi_latency_accum.
// 1. Total is exactly the sum of the stages.
property p_total_is_sum;
  @(*) total == (t_issue + t_traverse + t_lookup + t_access + t_return);
endproperty
 
// 2. The reported dominant value is >= every stage (it is the maximum).
property p_dominant_is_max;
  @(*) (dominant_val >= t_issue) && (dominant_val >= t_traverse) &&
       (dominant_val >= t_lookup) && (dominant_val >= t_access) &&
       (dominant_val >= t_return);
endproperty
 
// 3. Reducing a NON-dominant stage cannot cut the total by more than that stage (Amdahl).
//    (checked as a bound: delta_total <= delta_stage for any single-stage reduction)

The system point, beyond the checks:

The accumulator makes an analytical truth mechanical: the total's sensitivity to a stage is exactly 1 — reduce that stage by k cycles and the total drops by k, no more. So the most you can gain from a stage is its whole value, and the most you can gain from the easy stages combined is their combined (small) share. This reframes optimization as a budgeting problem: list the stages by size, and your total achievable speedup from any subset is capped by that subset's sum. A decomposition is therefore not just diagnostic but predictive — before writing a line of RTL you can compute the ceiling of a proposed optimization and decide whether it is worth doing. Most wasted performance work skips this step: it optimizes a stage without first checking the stage's fraction of the total, and so cannot distinguish a 3% ceiling from a 90% one until the work is already done.

• What it proves: the total is the sum; the dominant value is the maximum stage.
• What it does not prove: which stage dominates in a given workload — that requires measurement.
• Bug signature: effort targeting a stage whose value is a small fraction of total.

13. Testbench — the dominant stage is the one to attack

Feeds a miss-dominated profile and confirms memory is the dominant term.

module tb_chi_latency_accum;
  localparam W = 16;
  logic [W-1:0] t_issue, t_traverse, t_lookup, t_access, t_return;
  logic [W+2:0] total;
  logic [2:0] dominant;
  logic [W-1:0] dominant_val;
  int errors = 0;
 
  chi_latency_accum #(.W(W)) dut (.*);
 
  initial begin
    // Miss-dominated profile: memory access (stage 3) is by far the largest.
    t_issue = 2; t_traverse = 8; t_lookup = 3; t_access = 120; t_return = 8; #1;
    if (total !== 141) begin errors++; $display("FAIL total=%0d exp 141", total); end
    else $display("PASS total=%0d", total);
    if (dominant !== 3) begin errors++; $display("FAIL dominant=%0d exp 3 (memory)", dominant); end
    else $display("PASS dominant stage = memory access (%0d cycles)", dominant_val);
 
    // Optimizing traversal (stage 1) from 8 -> 4 barely moves the total: 141 -> 137.
    t_traverse = 4; #1;
    $display("INFO traversal 8->4: total %0d (saved only %0d of 141)", total, 141-total);
    if (total !== 137) begin errors++; $display("FAIL total=%0d exp 137", total); end
 
    // Attacking memory (stage 3) 120 -> 20 (e.g. SLC hit) transforms it: 137 -> 37.
    t_traverse = 8; t_access = 20; #1;
    $display("INFO memory 120->20: total %0d (saved %0d of 141)", total, 141-total);
    if (total !== 41) begin errors++; $display("FAIL total=%0d exp 41", total); end
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS total=141
PASS dominant stage = memory access (120 cycles)
INFO traversal 8->4: total 137 (saved only 4 of 141)
INFO memory 120->20: total 41 (saved 100 of 141)
ALL TESTS PASSED

14. DebugLab — optimizing a non-dominant stage

round-trip = issue 2 + traverse 8 + lookup 3 + MEMORY 120 + return 8 = 141 cycles
optimization: traverse 8 -> 4  (halved the interconnect stage, real work)
  -> total 141 -> 137  (1.4 cycles per hop... 4 cycles total, ~3% end-to-end)
memory (120 of 141 = 85%) untouched -> workload still memory-bound
Amdahl ceiling for the traverse stage: at most 8/141 = ~5.7% even if reduced to zero
correct: attack memory -> add SLC -> 120 -> 20 -> total 41 (71% reduction)

15. Common Mistakes

• Optimizing the easy stage. Assumption: any reduction helps. Bug: negligible gain (the DebugLab). Prevention: attack the dominant term.
• Not decomposing. Assumption: latency is one number. Bug: blind optimization. Prevention: break it into stages.
• Ignoring the memory term. Assumption: the fabric is the bottleneck. Bug: memory-bound workload untouched. Prevention: measure — misses dominate.
• Assuming a fixed dominant stage. Assumption: memory always rules. Bug: hit-heavy workload is traversal-bound. Prevention: re-measure per workload.
• Optimizing once and stopping. Assumption: done after one pass. Bug: a new dominant term emerges. Prevention: iterate — decompose, attack, repeat.
• Confusing latency with throughput. Assumption: same metric. Bug: wrong optimization. Prevention: latency is a sum (15.1); throughput is a bottleneck (15.2).

16. Engineering Checklist

• Decompose the round-trip into issue, traversal, lookup, access, return.
• Measure each stage in cycles.
• Identify the dominant (largest) stage.
• Confirm its fraction of the total — the optimization ceiling.
• Attack the dominant term (cache the miss, place homes closer, widen memory).
• Re-measure — the dominant term may move after optimization.

17. Key Takeaways

• A CHI round-trip is the sum of issue, traversal, lookup, access, and return.
• The stages are unequal — one usually dominates.
• On a miss, the memory access is typically dominant; on a hit, traversal may be.
• Amdahl: optimizing a stage caps the gain at that stage's fraction of the total.
• Attacking a minor stage wastes effort; attack the dominant term.
• Decompose, find the max, attack, re-measure; the model here is representative.

18. Quick Revision

CHI latency anatomy. A CHI transaction's round-trip latency is the sum of serial stages: issue/serialization at the requester, interconnect traversal to the home (hops × per-hop), home lookup (the directory), the access — a snoop to a peer or, on a directory miss, a memory read at the SN — and data return traversal. The stages are unequal, so one dominates: on a miss it is almost always the memory access, an order of magnitude beyond a single hop; on a hit to a close peer, traversal may dominate instead. Amdahl's latency form is the lever: a stage that is fraction f of the total can, even reduced to zero, cut the total by at most f — so optimizing a minor stage yields negligible end-to-end gain. The failure to avoid: optimizing the easy, visible stage (shaving interconnect hops) while the dominant memory term goes untouched, so the round-trip barely moves. The discipline: decompose the measured latency, rank the stages, attack the dominant term (cache the miss, add a system-level cache, place homes closer), then re-measure — the dominant term may move. Representative model; 15.2 turns from latency to throughput.

Coming Next

Chapter 15.2 — CHI Throughput. Latency is a sum of stages; throughput is set by a single bottleneck. Chapter 15.2 covers CHI throughput — how sustained throughput across the request, response, snoop, and data channels is capped by the busiest channel relative to its bandwidth, why the data channel usually saturates first for data-heavy workloads, and why widening a non-bottleneck channel buys nothing.