HBM-Based Systems

26.3 §8 made HBM the binding node and stopped. This chapter opens it up — and the first finding is that "HBM is the bottleneck" is usually wrong even when the memory system is the thing measuring slow.

1. The One-Sentence Model

HBM bandwidth exists only if the complete path can feed it — request generation, local queue, die-to-die link, memory-controller die, channel, DRAM, and the entire way back — and any stage can bind first.

Two consequences shape the whole chapter. Aggregate HBM bandwidth is a sum over many independent channels (§6), so it is available only to a workload that spreads across them; a workload that lands on a quarter of the channels gets a quarter of the number, with the memory system idle and blameless (§9).

And when memory controllers live on a different die, the die-to-die edge is in the path twice — outbound for requests, inbound for responses (§13, §16). A read is not one crossing. It is two, in opposite directions, with different payload sizes.

2. What This Chapter Owns

26.3 owned the graph and the scheduler. This chapter owns the memory path, and it owns four things nothing before it did:

HBM's channel structure as an architectural fact (§6–§7) — why the headline number is a sum and what that implies.

Address-to-channel distribution (§8–§10), including the mapping mistake that costs three quarters of the bandwidth while every component behaves correctly.

The outstanding-window arithmetic (§14–§15) — why a path with ample bandwidth at both ends can still idle.

And bottleneck localisation (§19–§21): telling "HBM is slow" apart from "nothing is reaching HBM", which look identical in a throughput number and opposite in a queue-depth counter.

3. Sourcing

4. Claim-vs-Evidence

Four readings.

Row 1 is the most architecturally useful documented figure in this chapter. "Eight 12-Hi stacks, 16 × 512-bit controllers" is a ratio: two controllers per stack. It says the memory system is subdivided below the stack, which is the whole premise of §7.

Row 2 is marked Class G deliberately — it is my arithmetic, not NVIDIA's statement. 8 TB/s ÷ 8 = 1 TB/s per stack is a valid division of documented numbers and still a derived figure, and labelling it as inference is the difference between analysis and quiet fabrication.

Rows 7 and 8 carry a source limitation I could not remove. JEDEC blocked me. I could have quoted the numbers cleanly and nobody would have noticed — which is exactly why the caveat is attached to every use, and why §7's argument is built to survive being wrong about them.

And rows 4–6 are Class A and quietly important. AMD documents that address space is grouped onto stacks in selectable ways and that locality "remains on the same AID". That is a shipping product confirming §8's premise: which channel a request lands on is an architectural decision with measurable consequences.

5. The Path

Three readings.

Eight stages, and only one of them is "HBM". Yet every one of them, when it binds, produces the same top-level observation: memory is slow. The counters in §21 exist because that observation carries no information.

Rows 3 and 7 are the same physical link in opposite directions, and they saturate at different times because a read's request is small and its response is a full line (§16). A design that sizes the crossing on aggregate bandwidth without splitting by direction will get one of them wrong.

And row 2 is the one that surprises people. A path can be limited by neither endpoint — ample source, ample memory, ample link — and still idle because too few transactions may be in flight (§14). Nothing is saturated, and the bandwidth is simply not there.

6. What HBM Actually Is

HBM's headline bandwidth is not delivered by one wide port. It is delivered by many independent channels operating concurrently — and that structural fact drives everything in §7–§10.

Three readings, and the middle one is the chapter's engine.

"Independent" and "not necessarily synchronous" is the load-bearing phrase. Channels are not a wide bus that any request can use. A request goes to one channel, and it gets that channel's share of the bandwidth — not the stack's.

So the headline figure is a sum over channels. "~819 GB/s per device" means all channels busy simultaneously. A workload using a quarter of the channels gets a quarter of the number, and the memory system is behaving perfectly. §7 is the arithmetic and §9 is how a design lands there by accident.

And HBM4's direction confirms the diagnosis rather than changing it. HBM4 doubles the channel count and doubles the interface width. The industry's answer to needing more bandwidth is more parallelism, which means distribution across channels matters more with each generation, not less — a bad mapping wastes a bigger fraction of 32 channels than of 16.

7. Aggregate Bandwidth Is a Sum, Not a Resource

Take one documented figure and divide it correctly.

NVIDIA documents 8 TB/s across eight stacks (Class B), so 1 TB/s per stack (Class G — my arithmetic, §4 row 2). That is already a more useful number than the headline, because a request can only be served by the stack that owns its address.

Now go one level further, with explicitly illustrative structure.

Three readings.

The last row is the entire chapter in one line. 2 TB/s measured against an 8 TB/s specification, with no fault anywhere. Not a broken controller, not a slow link, not a bad DRAM part — a workload that touched a quarter of the channels.

And the fault is invisible from every endpoint. The compute die sees slow memory. The memory system sees 32 channels working hard and 96 idle. Neither view alone identifies the problem; only per-channel occupancy does (§12).

Which reframes what a memory architect is actually responsible for. Not "provide 8 TB/s" — provide 8 TB/s to the access pattern the workload has. Those are different engineering problems, and the second one is mostly about §8.

8. Address to Channel

Generic memory-controller structure, stated as generic and not as any product's implementation (§3).

And one Class-A product data point (§4 rows 4–5): AMD documents NPS1 / NPS2 / NPS4 grouping HBM stacks into one, two or four pools, with NPS4 forming pools from "pairs of HBM stacks". That is the first two rows of this table, made explicit and software-selectable in a shipping product — evidence that address-to-memory grouping is a real architectural control, not a textbook abstraction.

Two readings.

Row 4 is where bandwidth is won or lost, and it is the one row that is almost never documented (§23). The channel-selection function decides whether a workload spreads or piles up.

And rows 1–2 versus row 4 operate at completely different scales. Pool selection is coarse and often software-visible; channel selection is fine, happens per request, and is invisible to everyone — which is why §9's failure survives review.

9. Wrong RTL — Low-Bit Striping and Stride Aliasing

// WRONG. ILLUSTRATIVE. Channel selection by a contiguous slice of low address
// bits. This is the first thing anyone writes, it is uniform for sequential
// access, and it is uniform for random access. It is catastrophic for STRIDED
// access — which is what a tiled matrix or a strided tensor read produces.
localparam int N_CH   = 16;
localparam int CH_LSB = 8;              // 256-byte granule, illustrative
 
// BUG: the channel index is a fixed 4-bit window of the address. Any access
// stream whose stride is a multiple of (N_CH << CH_LSB) = 4096 bytes leaves
// this field CONSTANT — every request lands on the SAME channel.
assign ch_sel = req_addr[CH_LSB +: 4];

Architecture. One wire. No state, no logic, no cost — and that is precisely why it survives review.

State. None.

Event. Evaluated per request.

Contract. It assumes the address bits it selects vary across the request stream. That assumption is true for sequential and random access and false for strided access, which is the dominant pattern in tiled linear algebra.

Failure — the arithmetic. Channel granule is 2^8 = 256 bytes and there are 16 channels, so the pattern repeats every 16 × 256 = 4096 bytes.

A 4 KB stride — a page, a row of a large matrix, a plausible tensor slice — collapses the memory system onto ONE channel.

Failure — the timeline.

Root cause. The channel index was derived from address bits that the workload holds constant. Uniformity was verified against sequential and random patterns — the two patterns that cannot expose this bug — and never against a strided one.

And the reason it is so destructive is that the warm-up phase measures full bandwidth. The system demonstrably works. Only the specific access pattern that matters is broken.

DV/debug. The signature is unambiguous and requires per-channel counters to see at all (§12): one channel at maximum occupancy while the rest sit at zero. Aggregate utilisation reports ~6 % and offers no explanation. A memory-side scheduler tuning effort here would be weeks spent optimising a channel that is already saturated.

10. Corrected — Mapping That Resists Aliasing

// CORRECT. ILLUSTRATIVE (§9). Fold HIGH address bits into the channel index so
// no single stride can hold the selector constant. XOR-folding is a standard
// technique; the specific bit choice must be validated against the ACCESS
// PATTERNS THE PRODUCT ACTUALLY HAS, not against uniform random traffic.
localparam int N_CH   = 16;
localparam int CH_LSB = 8;
 
logic [3:0] ch_base, ch_fold1, ch_fold2;
 
assign ch_base  = req_addr[CH_LSB    +: 4];
assign ch_fold1 = req_addr[CH_LSB+ 4 +: 4];
assign ch_fold2 = req_addr[CH_LSB+12 +: 4];
 
// Folding widely-separated bit fields means a stride must alias in SEVERAL
// fields at once to collapse the selector — far harder to hit accidentally.
assign ch_sel = ch_base ^ ch_fold1 ^ ch_fold2;
 
// MANDATORY. English: a live request's channel selection never changes.
// The mapping is a pure function of the address, so this catches anyone
// making it depend on state — occupancy, retry, or a runtime remap — which
// would send a request and its bookkeeping to different places.
a_ch_sel_stable: assert property (
  @(posedge clk) disable iff (!rst_n)
    (req_live && $stable(req_addr)) |-> $stable(ch_sel)
);

Architecture. Three address fields XOR-folded. Cost is a handful of gates; the design work is entirely in choosing the fields.

State. None — and keeping it stateless is a deliberate contract (see the assertion).

Event. Per request, purely combinational from the address.

Contract. The mapping must be a pure function of the address. Making it depend on occupancy is tempting — send it to the emptiest channel — and wrong: the response path, the bookkeeping and any subsequent access to the same line must all agree on where it lives. A load-balanced mapping is not a mapping; it is a routing decision made without a memory.

Failure. XOR-folding reduces stride aliasing; it does not eliminate it. An adversarial stride that aliases in all three fields simultaneously still collapses. The honest claim is "far harder to hit accidentally", not "safe" — and the way to know is to replay the product's real address traces through the mapping and histogram the channel index, which is a day of work and the only real answer.

DV/debug. The test that matters is not a random-traffic uniformity check — random traffic passes with §9's broken mapping too. The test is a strided sweep across power-of-two strides from one granule to several megabytes, asserting that no stride concentrates more than a bounded fraction on any channel. §9's bug is invisible to every test except that one.

11. Illustrative RTL — the Memory Route Descriptor

// ILLUSTRATIVE ONLY. What one outstanding memory operation must carry across a
// die boundary. The grouping is the lesson: SEMANTIC identity is owned by the
// requester and stable for the operation's life; PLACEMENT is derived once from
// the address; TRANSPORT facts belong to the link and are never inputs to the
// other two (26.1 §13, 25.5 §11).
typedef struct packed {
  // SEMANTIC — stable for the operation's whole life
  logic [SEM_W-1:0]   sem_id;
  logic [GEN_W-1:0]   generation;      // which use of this id
  logic [2:0]         op;              // read / write / atomic class
  logic [DOM_W-1:0]   order_domain;    // §17 — what it is ordered against
  logic [LEN_W-1:0]   bytes;           // response size — §16 needs this
  // PLACEMENT — a pure function of the address (§10)
  logic [ADDR_W-1:0]  addr_token;      // opaque; not an architectural address
  logic [CTL_W-1:0]   controller_id;
  logic [CH_W-1:0]    channel_id;
  logic [EPOCH_W-1:0] cfg_epoch;       // the config this was mapped under
  // TRANSPORT — link-owned; NEVER written by placement or semantic logic
  logic [ATT_W-1:0]   attempt;
} mem_op_t;

Architecture. One record, three clearly separated field groups, so the ownership boundary is visible in the type rather than living in a reviewer's head.

State. One entry per outstanding operation. The table's size is the outstanding limit — which is the number §14 shows to be decisive.

Event. Constructed at acceptance on the requester side; attempt increments on retransmission. No transport event writes sem_id, generation, controller_id or channel_id.

Contract. bytes is present because the response path must be accounted separately from the request path (§16). A 64-byte read is one small outbound message and one 64-byte inbound message, and a design that tracks only "operations" cannot size either direction.

Failure. If channel_id were re-derived at response time rather than carried, a configuration change mid-flight would return the response through a different mapping — completing correctly against the wrong bookkeeping, and corrupting every per-channel counter in §12 while the data itself stays valid.

DV/debug. attempt in a trace is what stops an engineer reporting a duplicate. One sem_id, attempt = 3, one retirement is the system working (26.1 §13) — and channel_id in the same trace is what makes §9's aliasing visible in a capture rather than only in a counter.

12. Illustrative RTL — Per-Channel Occupancy

// ILLUSTRATIVE (§11). Per-channel accounting is the ONLY instrument that
// distinguishes "memory is saturated" from "memory is idle and one channel is
// on fire" (§9). Aggregate utilisation cannot tell those apart, and they need
// opposite responses.
localparam int N_CH = 16;
 
logic [15:0] occupancy_q     [N_CH];   // outstanding at this channel
logic [15:0] occ_max_q       [N_CH];   // high-water mark
logic [47:0] issued_q        [N_CH];
logic [47:0] completed_q     [N_CH];
logic [47:0] busy_cyc_q      [N_CH];   // cycles with occupancy != 0
logic [47:0] full_cyc_q      [N_CH];   // cycles at the per-channel limit
logic        ovf_sticky_q;             // any counter saturated — see below
 
function automatic logic [47:0] sat48(logic [47:0] a);
  sat48 = (a == 48'hFFFF_FFFF_FFFF) ? a : (a + 48'd1);
endfunction
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n || diag_clear) begin
    ovf_sticky_q <= 1'b0;
    for (int c = 0; c < N_CH; c++) begin
      occupancy_q[c] <= '0; occ_max_q[c]   <= '0;
      issued_q[c]    <= '0; completed_q[c] <= '0;
      busy_cyc_q[c]  <= '0; full_cyc_q[c]  <= '0;
    end
  end else begin
    for (int c = 0; c < N_CH; c++) begin
      // ONE signed next-state expression, so an issue and a completion in the
      // same cycle net correctly instead of losing one (25.6 §10).
      occupancy_q[c] <= occupancy_q[c]
                      + 16'(issue_fire  && (issue_ch  == c[CH_W-1:0]))
                      - 16'(cmpl_fire   && (cmpl_ch   == c[CH_W-1:0]));
 
      if (occupancy_q[c] > occ_max_q[c]) occ_max_q[c] <= occupancy_q[c];
 
      if (issue_fire && (issue_ch == c[CH_W-1:0])) issued_q[c]    <= sat48(issued_q[c]);
      if (cmpl_fire  && (cmpl_ch  == c[CH_W-1:0])) completed_q[c] <= sat48(completed_q[c]);
 
      if (occupancy_q[c] != '0)              busy_cyc_q[c] <= sat48(busy_cyc_q[c]);
      if (occupancy_q[c] == CH_LIMIT[15:0])  full_cyc_q[c] <= sat48(full_cyc_q[c]);
 
      // Saturation must be VISIBLE. A silently pinned counter turns a
      // diagnostic into a lie, and a wrapping one is worse.
      if (issued_q[c] == 48'hFFFF_FFFF_FFFF) ovf_sticky_q <= 1'b1;
    end
  end
end

Architecture. Six numbers per channel. At sixteen channels that is 96 counters — cheap, and it is the difference between a one-read diagnosis and a two-week investigation.

State. Occupancy is live; the rest accumulate over a window. occ_max survives the moment, which matters because the pathological condition may be transient and the average will hide it.

Event. Issue and completion are distinct accepted events, and the signed next-state expression is why a same-cycle pair does not lose an update — the 25.6 §10 hazard, which in a per-channel counter produces slow drift that eventually looks like a leak.

Contract. Saturating, never wrapping, and saturation is reported via ovf_sticky_q. A wrapped counter produces a negative apparent rate and sends an investigation in exactly the wrong direction; a silently pinned counter is a diagnostic that lies while looking healthy.

Failure. The realistic mistake is a single aggregate occupancy counter. It cannot represent §9 at all: one channel at limit and fifteen at zero averages to a comfortable-looking number, and the bug is arithmetically invisible.

DV/debug. Two signatures, opposite causes. full_cyc high on one channel, near zero elsewhere → §9's aliasing. full_cyc high on all channels → genuine memory saturation, and only then is memory actually the bottleneck. A third pattern — all busy_cyc low with the source backed up — means nothing is reaching memory, which is §19.

13. When D2D Binds Before HBM

If memory controllers live on a different die from the requester, the die-to-die edge is in the path — and it can bind first. AMD's documentation notes that latency "is minimized because it remains on the same AID" (Class A, §4 row 6) — confirming that not all memory traffic takes the same path length in a real product.

Setup. Requester on die A; controllers and HBM on die B. 64-byte lines.

Step 1 — what a read costs each direction. Per 64 bytes delivered: 16 B outbound, 64 B inbound.

Step 2 — inbound direction. 400 GB/s ÷ 64 B per line = 6.25 G lines/s, i.e. 400 GB/s of useful data.

Step 3 — outbound direction. 400 GB/s ÷ 16 B = 25 G requests/s — four times what inbound can return. Outbound is nowhere near binding.

Step 4 — the minimum. min(HBM 1000, inbound 400) = 400 GB/s.

Result: 40 % of HBM's capability, and the memory system is idle 60 % of the time.

Three readings.

The binding stage is the inbound direction of the die-to-die link, and it binds by a factor of 2.5×. Every HBM tuning effort in this system is wasted work.

The asymmetry is 4:1 and entirely structural (§16). A read is a small question and a large answer. A crossing sized on "aggregate bandwidth" without a direction split will be comfortable outbound and starved inbound.

And this is invisible from both ends. The compute die sees slow memory; the memory system sees light load. Only a stage-by-stage counter walk finds it (§21) — and §19 is exactly this failure presented as a debug problem.

14. The Outstanding Window

A path with enough bandwidth at every stage still delivers nothing if too little is in flight. The bandwidth-delay product (21.5 §26), applied to memory.

required_outstanding ≈ target_request_rate × round_trip_latency

Units matter and are where this goes wrong. Express the rate in operations per nanosecond and latency in nanoseconds, and the product is a dimensionless count of operations.

Worked example, illustrative. Target 400 GB/s of 64-byte lines, round trip 500 ns.

Step 1 — target rate in operations. 400 GB/s ÷ 64 B = 6.25 operations/ns.

Step 2 — required in flight. 6.25 ops/ns × 500 ns = 3125 operations outstanding.

Step 3 — per channel. Across 16 channels that is ≈ 196 per channel, if the workload spreads evenly (§7 — and it will not if §9's bug is present).

Step 4 — what happens with fewer. With only 256 operations outstanding package-wide: 256 ÷ 500 ns = 0.512 ops/ns → × 64 B = 32.8 GB/s. Eight per cent of target, with every stage under-utilised and nothing saturated.

Three readings.

Nothing is broken in step 4, and that is the point. Source ready, link idle, memory idle. The path is limited by how much may be in flight, and no utilisation counter anywhere reads high.

Latency enters as a multiplier on the required resource. Doubling round-trip latency doubles the ID table needed to sustain the same bandwidth. A latency regression therefore shows up as a bandwidth regression — and gets investigated as one, which is the wrong search.

And §9 makes this worse in a way that compounds. If a workload aliases onto four channels, the effective per-channel concurrency requirement quadruples while the ID table stays the same size. Bad mapping and a small outstanding window fail together and look like one bandwidth problem.

15. Wrong RTL — Too Few IDs

// WRONG. ILLUSTRATIVE. An 8-entry outstanding table. Sized during early
// bring-up when the memory model returned in ~40 ns, where 8 was ample.
// Silicon latency is ~500 ns. The parameter was never revisited, because
// nothing about it is wrong — it is just far too small.
localparam int N_ID = 8;              // BUG: sized against model latency
 
logic [N_ID-1:0] id_busy_q;
logic            can_issue;
 
assign can_issue = (id_busy_q != {N_ID{1'b1}});   // stall when all IDs busy
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) id_busy_q <= '0;
  else begin
    if (issue_fire) id_busy_q[alloc_id] <= 1'b1;
    if (cmpl_fire)  id_busy_q[cmpl_id]  <= 1'b0;
  end
end

Architecture. An 8-deep ID pool gating issue. Structurally correct — every mechanism works exactly as intended.

State. One busy bit per ID.

Event. Allocate on issue, free on completion.

Contract. It assumes 8 IDs cover the round trip. That assumption was measured and true against a fast behavioural model, and it silently became false when real latency arrived.

Failure — the arithmetic. At 500 ns round trip with 8 IDs and 64-byte lines:

8 ops ÷ 500 ns = 0.016 ops/ns → × 64 B = 1.02 GB/s.

Against a 400 GB/s path. A factor of ~390.

Failure — the timeline.

Root cause. A concurrency resource was sized against a latency that did not survive to silicon. The bug is a parameter, and parameters are exactly what reviews skim.

And it is invisible to every conventional diagnostic, because the classic bottleneck signature is something is full. Here nothing is full. Link idle, memory idle, source ready and stalled — the discriminator is the source's stall reason, not any utilisation number.

Corrected sizing. From §14: N_ID ≥ target_rate × round_trip, plus margin for latency variance. For 400 GB/s at 500 ns that is ≈ 3125, which is a genuinely large table — and if that cost is unacceptable, the honest conclusion is that the target bandwidth is unachievable at that latency, not that the table can be smaller. Publishing the trade is the engineering; hiding it is how §15 ships.

DV/debug. Instrument cycles stalled on no-free-ID as a first-class counter alongside utilisation. That single number distinguishes this failure from every other cause of low bandwidth, and it costs one counter.

16. The Response Direction

§13 found the inbound direction binding at a 4:1 asymmetry. That deserves its own treatment, because it is the stage most often left out of a bandwidth budget.

Three readings.

Read and write load opposite directions, so the binding direction depends on the workload phase. A package tuned on a read-heavy benchmark can be inbound-limited in one phase and outbound-limited in another — and a single aggregate number describes neither.

"Aggregate bandwidth" hides this completely. A crossing described as "10 TB/s" (§4 row 10) does not state whether that is per direction or summed. I therefore refuse to use that figure in arithmetic (§3), because the wrong reading changes every conclusion by 2×.

And the practical rule is a budget, not a number. Size each direction against the phase that stresses it, and state which phase that was. A crossing sized on the average of a mixed workload is sized for a workload that never runs.

17. Ordering and Dependency Domains

Ordering constraints cost bandwidth when applied more broadly than the semantics require.

Two readings.

A global ordering rule defeats §7's entire premise. Aggregate bandwidth comes from concurrent channels; forcing a total order across all traffic makes them serial. You can lose most of a memory system's bandwidth to an ordering rule that no consumer actually needs.

Which is why order_domain is a field in §11's descriptor rather than an implicit global property. Making the domain explicit is what allows two unrelated streams to use two channels at once — and it is a semantic decision that must come from the architecture's own definition, not from a generic assumption about memory ordering that no source here establishes (§23).

18. Three Kinds of Backpressure

They look identical at the top level and require opposite responses.

Three readings.

Rows 1 and 2 both show a full upstream queue and no credits, and they are completely different problems. The controller queue depth separates them: deep means memory is genuinely behind; shallow means the crossing is the limit and memory is starving.

Rows 3 and 4 both show available credits and a shallow controller queue, and they differ in whether the source has work. Row 3 is "nothing to do"; row 4 is §15's disaster, where the source is desperate to issue and cannot.

And row 5 is the one that only per-channel counters resolve (§12). It presents exactly like row 1 in every aggregate view. The distinguishing evidence is the distribution across channels, which no aggregate metric contains.

19. Failure Case — HBM Looks Slow, HBM Is Idle

Reported symptom: the system achieves 55 % of expected memory bandwidth.

Conclusion: the inbound direction of the die-to-die link binds — §13's arithmetic, observed.

Three readings.

Step 2 ends the investigation that most teams start. An empty controller queue means memory is waiting for work. Every hour spent on memory-scheduler tuning after step 2 is wasted, and step 2 costs one register read.

Steps 4–5 versus step 2 form the discriminator. Upstream full and downstream empty means the constraint is exactly at the boundary between them (25.9 §12's first-divergence principle applied to a pipeline of stages).

And step 6 is what turns a located problem into an actionable one. "The link is the bottleneck" invites the answer "add bandwidth". "The inbound direction is saturated while outbound runs at 25 %" invites the correct answers: rebalance the directions, compress or coalesce responses, or move the consumer closer to the data — and it explains why doubling the link symmetrically would waste half the addition.

20. Failure Case — D2D Looks Slow, the Mapping Is Wrong

Reported symptom: die-to-die utilisation is low and bandwidth is poor. The opposite evidence pattern.

Conclusion: channel aliasing. The source cannot produce completions fast enough to fill the link because its requests are queued behind one saturated channel.

Three readings.

Steps 1–3 form a pattern that looks like "nothing is wrong". Low utilisation, no credit pressure, empty upstream queue. A design that stops here concludes the workload is simply not memory-intensive — and ships.

Step 6 is the only step that could have found this, and it is impossible without §12's per-channel counters. Every aggregate view in the system reports mild, unremarkable load.

And step 7 is what makes the fix confident rather than speculative. A stride histogram of the actual address trace names the mechanism: 4 KB stride against a 4 KB mapping period (§9). Then §10's XOR-fold is a targeted fix that can be validated by replaying the same trace, instead of a hopeful change to a hash function.

21. The Instrumentation That Localises Any of This

Seven staged counters, read in order. The first stage whose long-run rate falls below target is the bottleneck (25.9 §12).

Plus three that are not rates and are worth as much:

cycles stalled on no-free-ID (§15) — the only counter that identifies an outstanding-window limit, because that failure makes nothing look full.

full_cyc per channel (§12) — the only counter that separates saturation from concentration.

Inbound bytes split payload vs header (§19 step 7) — separates "the link is carrying real data" from "the link is carrying overhead".

And the discipline that makes the set work is reading it in order. Counters 4 and 5 must be per channel, not aggregated — an aggregate at stage 4 hides §9 completely, and stage 4 is precisely where §9 lives.

22. Assertions Worth Writing

// ILLUSTRATIVE (§11). Four properties, each with an English invariant and a
// specific bug it catches.
 
// English: every accepted request creates exactly one live entry, and that
// entry's channel is the one derived from its address. Catches placement being
// re-derived, reassigned, or load-balanced after acceptance (§10 Contract).
a_placement_stable: assert property (
  @(posedge clk) disable iff (!rst_n)
    (op_live[chk] && $past(op_live[chk]))
      |-> (ops_q[chk].channel_id == $past(ops_q[chk].channel_id))
);
 
// English: a completion retires an entry only if that entry is live AND the
// generation matches. Catches a stale completion from a previous use of the
// same id retiring an unrelated in-flight operation.
a_completion_is_live_gen: assert property (
  @(posedge clk) disable iff (!rst_n)
    cmpl_fire |-> (op_live[cmpl_idx] &&
                   (ops_q[cmpl_idx].generation == cmpl_gen))
);
 
// English: per-channel occupancy equals issues minus completions for that
// channel. Catches the same-cycle increment/decrement loss that produces slow
// drift and eventually looks like a leak (§12 Event lens).
a_occupancy_conserved: assert property (
  @(posedge clk) disable iff (!rst_n)
    occupancy_q[chk_ch] == (issued_q[chk_ch] - completed_q[chk_ch])
);
 
// English: outstanding operations never exceed the ID table. Catches an
// allocation path that bypasses the can_issue gate — which produces id reuse
// while an operation is still live, and silently wrong data.
a_outstanding_bounded: assert property (
  @(posedge clk) disable iff (!rst_n)
    ($countones(id_busy_q) <= N_ID)
);

Architecture. Four properties covering placement stability, completion identity, counter conservation and resource bounding — the four ways this design can be wrong without being visibly wrong.

State. All read existing design state. a_occupancy_conserved is the only one needing counters that exist for diagnostics anyway, which makes it nearly free.

Event. a_completion_is_live_gen triggers on every completion, so it is non-vacuous whenever traffic flows. a_placement_stable requires an entry to remain live across two cycles — coverage must confirm that actually happens rather than assuming it (25.8 §22).

Contract. a_occupancy_conserved encodes the invariant that the diagnostic counters mean what §12 claims. An unchecked counter is a number, not evidence — and every conclusion in §19–§21 rests on those numbers being right.

Failure. a_outstanding_bounded catches the most dangerous bug in the chapter: ID reuse while an operation is live. That produces a completion matched to the wrong operation — wrong data, delivered with every integrity check passing (21.6 §14).

DV/debug. All four are simulation-suitable. a_occupancy_conserved over sixteen channels is a reasonable formal target if bound to a symbolic channel index; proving it across all channels simultaneously is an unnecessary expense — a scoping decision worth making before the proof stalls, not after.

23. What Public Sources Do Not Disclose

Mandatory. Everything below is something this chapter could have invented and sounded more authoritative for inventing.

Two readings.

The right-hand column is where all the engineering in this chapter lives, and none of it is public for any product. §9's aliasing, §14's window, §16's direction split — every one of them depends on a parameter no vendor publishes.

Public material reviewed does not establish any right-column item for any package discussed here. What the framework gives you instead is the list of questions, and — critically — the counters that answer them in your own silicon (§21). You cannot read AMD's hash function. You can absolutely histogram your own channel index.

24. Common Misconceptions

"HBM gives you its headline bandwidth." §7: that figure is a sum over all channels simultaneously. A quarter of the channels gives a quarter of the number, with nothing broken.

"Low measured bandwidth means memory is the bottleneck." §19: controller queues empty at 55 % of target means memory is waiting. The constraint was the inbound crossing.

"More channels always helps." §6: HBM4 doubles channels to 32 — which also doubles what a bad mapping wastes. More parallelism raises the value of good distribution, it does not remove the need for it.

"A uniformity test proves the mapping is fine." §9: sequential and random traffic both pass with the broken mapping. Only a strided sweep exposes it.

"Balance channels dynamically by occupancy." §10 Contract: placement must be a pure function of the address, or the response path and any later access disagree about where a line lives.

"Bandwidth at both ends is enough." §14–§15: 8 outstanding IDs at 500 ns delivers ~1 GB/s on a 400 GB/s path, with nothing full anywhere.

"A read and a write cost the same." §16: they load opposite directions — a read is a small request and a large response.

"Aggregate link bandwidth is the number to budget." §16: it does not state direction. A crossing sized on the average of a mixed workload is sized for a workload that never runs.

"HBM plus chiplets implies UCIe." §3: Infinity Fabric and NV-HBI are documented. That inference is 22.1 §12's error in memory clothing.

25. Understanding Check

26. Summary

Six things.

HBM bandwidth exists only if the whole path can feed it (§1, §5). Eight stages, and only one of them is HBM — but all eight produce the same top-level symptom.

Aggregate bandwidth is a sum over independent channels (§6–§7). Illustratively: 8 TB/s package → 1 TB/s per stack → 62.5 GB/s per channel, which is what one request actually contends for. A quarter of the channels gives a quarter of the number, with nothing broken.

Address mapping decides whether a workload spreads (§9–§10). A 4 KB stride against a 4 KB mapping period collapses onto one channel — 6.25 % — and sequential and random tests both pass.

Two crossings, opposite directions, different sizes (§13, §16). A read is a small request and a large response; the inbound direction can bind at 40 % of HBM's capability while outbound runs at a quarter.

Concurrency is a resource (§14–§15). required_outstanding ≈ rate × latency — 3125 operations for 400 GB/s at 500 ns, and an 8-entry table delivers ~1 GB/s with nothing full anywhere.

And the counters are the chapter (§12, §21). Per-channel occupancy, the seven staged rates read in order, and stalled on no-free-ID. Aggregate metrics cannot distinguish saturation from concentration from starvation — and those three need opposite fixes.