DDR · Module 26
HBM for AI Accelerators
Stacks buy bandwidth and dies buy capacity. The same 96 GB bought as twelve short stacks delivers three times the bandwidth of four tall ones — and one requester cannot use any of it.
The three chapters before this one established what HBM is: a 1024-bit interface divided into sixteen semi-independent streams, reached through a die's face rather than its edge, stacked on ordinary dies whose vertical connections need spares to work at all.
None of them asked what a system does with it.
Chapter 4.8 §2 issues the warning this chapter is built on — HBM's headline figure “requires traffic that keeps all eight channels busy”, and “one channel's worth of traffic gets one channel's worth of bandwidth, and no amount of width helps” — and 26.1 §4 doubled the relevant count to sixteen. Chapter 26.2 §8 adds the other half: height buys capacity and not bandwidth.
Put those together and an accelerator faces two separate purchases that look like one. §3 shows that the same capacity, bought two ways, differs in delivered bandwidth by a factor of three.
1. Two Purchases That Look Like One
Chapter 26.2 §8 established the independence and proved it with a property: the interface is 1024 bits wide at every stack height, so height changes capacity and leaves bandwidth alone.
State both sides. CURRICULUM-DERIVED from 4.8 §2 and 26.2 §8:
| What you add | What changes | What does not |
|---|---|---|
| A die (taller stack) | capacity, +2 GB | bandwidth, streams |
| A stack | bandwidth, +307.2 GB/s; streams, +16; capacity, +up to 24 GB | — |
So one of the two purchases is strictly more powerful and strictly more expensive. A stack adds everything; a die adds only capacity.
That asymmetry is the whole chapter. It means a system's bandwidth is set by its stack count and nothing else — not by total capacity, not by stack height, not by how much memory the specification sheet lists.
2. Shorter Stacks Are Bandwidth-Dense
Divide one quantity by the other and the asymmetry becomes a design lever.
DERIVED from §1, recomputed:
| Stack height | Capacity | Bandwidth | Bandwidth per GB |
|---|---|---|---|
| 2 | 4 GB | 307.2 GB/s | 76.80 (GB/s)/GB |
| 4 | 8 GB | 307.2 GB/s | 38.40 |
| 8 | 16 GB | 307.2 GB/s | 19.20 |
| 12 | 24 GB | 307.2 GB/s | 12.80 |
A 2-high stack is exactly six times as bandwidth-dense as a 12-high one — and the factor is exactly 6 because the capacity ratio is 12/2 and the bandwidth is identical.
Three readings, and the second is counter-intuitive enough to be worth stating carefully.
The tallest stack is the least bandwidth-dense. Height is the one dimension that adds capacity without adding bandwidth, so maximising height minimises the ratio.
So a bandwidth-hungry system should prefer short stacks, and more of them. That inverts the instinct to buy the biggest available part — the biggest part is the worst choice per unit of bandwidth, and §3 makes the difference concrete.
And a capacity-hungry system should prefer tall stacks, because it is paying for stacks it does not need bandwidth from. The two workloads want opposite configurations from the same component family.
3. Two Ways to Buy Ninety-Six Gigabytes
Here is the chapter's sharpest result. DERIVED, recomputed:
| Configuration | Capacity | Bandwidth | Streams |
|---|---|---|---|
| 4 stacks × 12-high | 96 GB | 1,228.8 GB/s | 64 |
| 12 stacks × 4-high | 96 GB | 3,686.4 GB/s | 192 |
Identical capacity. Three times the bandwidth. Three times the streams.
The second configuration uses three times as many stacks, each a third as tall — and 26.3 §5 notes that shorter stacks are also the easier ones to build, since the all-vias-work probability degrades with height.
So the two configurations are not a trade between bandwidth and capacity. They deliver the same capacity, and one delivers three times the bandwidth. The trade is against cost and integration: twelve stacks need twelve stacks' worth of interposer area, twelve bump fields, twelve assemblies — and 4.8 §3 establishes that each assembly step carries a yield risk that “scrapes an assembly containing an expensive processor.”
4. What a Workload Actually Needs
Neither purchase is right in the abstract. The requirement comes from the workload, and it has a standard form.
The model, stated. A computation performs operations and moves bytes. Its arithmetic intensity I is operations per byte of memory traffic. For a compute rate C operations per second, the memory traffic required to sustain it is:
required bandwidth = C / I bytes per secondHigh intensity means each byte is reused heavily, so a given compute rate needs little bandwidth. Low intensity means each byte is used once or twice, so the same compute rate needs a great deal.
Chapter 1.8 owns the memory wall, the latency-versus-bandwidth separation and bandwidth saturation, and this chapter does not restate any of it. What is added here is the conversion into stacks, which is HBM-specific:
stacks required = (C / I) / 307.2 GB/s5. The Intensity Where HBM Runs Out
Put numbers in and the requirement becomes a stack count — and at low intensity the count is not achievable.
DERIVED, with ILLUSTRATIVE compute rates and intensities, recomputed:
| Compute rate | Intensity 10 | Intensity 50 | Intensity 200 |
|---|---|---|---|
| 100 Tops/s | 10,000 GB/s → 32.6 stacks | 2,000 GB/s → 6.5 stacks | 500 GB/s → 1.6 stacks |
| 500 Tops/s | 50,000 GB/s → 162.8 stacks | 10,000 GB/s → 32.6 stacks | 2,500 GB/s → 8.1 stacks |
Read the first column. A 100 Tops/s engine at intensity 10 needs 32.6 stacks — and at 500 Tops/s, 162.8. Those are not configurations; they are refutations.
Three conclusions, and the third is the one that shapes accelerator architecture.
Low-intensity workloads cannot be served by adding memory bandwidth. No plausible stack count reaches 50,000 GB/s, so the answer is not more HBM.
The lever is intensity, not bandwidth. Moving intensity from 10 to 200 — a factor of 20 — reduces the requirement from 32.6 stacks to 1.6, which is the difference between impossible and ordinary. Every decimal point of reuse is worth more than any amount of memory purchasing.
Which is why accelerators are built around on-chip reuse. Caches, local scratchpads, tiling and operand reuse all exist to raise I before the traffic reaches memory. HBM does not remove the need for them — it raises the compute rate at which they become mandatory, which is the opposite of how a bandwidth increase is usually described.
6. Sixteen Streams Per Stack, Times the Stacks
Chapter 4.8 §2's warning and 26.1 §4's doubling combine into a system-level requirement.
DERIVED, recomputed:
| Stacks | Streams | Bandwidth | Bandwidth per stream |
|---|---|---|---|
| 1 | 16 | 307.2 GB/s | 19.20 GB/s |
| 2 | 32 | 614.4 GB/s | 19.20 |
| 4 | 64 | 1,228.8 GB/s | 19.20 |
| 8 | 128 | 2,457.6 GB/s | 19.20 |
| 12 | 192 | 3,686.4 GB/s | 19.20 |
Bandwidth per stream is constant at 19.2 GB/s, which is the arithmetic consequence of both quantities scaling with the stack count.
So the system's stream count grows exactly as fast as its bandwidth. An eight-stack system does not merely need eight times the traffic of a one-stack system — it needs that traffic distributed across 128 independent streams, each of which can absorb only 19.2 GB/s no matter how much the others are idle.
That is the requirement §3's twelve-stack configuration imposes: 192 streams. And it is where 4.8 §2's warning becomes an architectural constraint rather than a caution.
7. One Requester Cannot Fill Them
A single sequential requester, however fast, gets one stream's bandwidth.
The reason is 26.1 §4's granule. A stream's minimum access is 64 bytes, and a requester issuing one access at a time waits for each before issuing the next — so its bandwidth is bounded by 64 bytes / round-trip latency regardless of how many streams exist.
DERIVED under a stated model — a requester with Q outstanding 64-byte accesses distributed perfectly across streams, ILLUSTRATIVE latency:
one requester, Q outstanding, 64 B each, latency T
bandwidth = Q x 64 B / T
streams usefully occupied = min(Q, total streams)So the quantity that matters is outstanding requests, not request rate. A requester with Q = 1 uses one stream. With Q = 192 it can, in principle, use 192 — but only if its addresses land on 192 different streams, which is an address-mapping property rather than a concurrency one.
Two requirements, therefore, and they are independent:
Enough outstanding requests to cover the stream count. A 192-stream system needs at least 192 accesses in flight to touch every stream once. Chapter 1.8 §7 owns memory-level parallelism as the general idea; the HBM-specific figure is that the required parallelism scales with the stack count.
And a distribution that reaches them. Requests concentrated on one stack use 16 streams out of 192 — 26.1 §16 published the report of exactly that failure at the pseudo-channel level, where eight active streams in four channels looked identical to eight spread across eight.
8. The Requester Count Is an Architectural Property
Here is the consequence for how an accelerator is built, and it is the chapter's reason for existing.
An accelerator cannot choose its memory-level parallelism at runtime. The number of independent requests it can have outstanding is set by how many independent execution units, memory pipelines and outstanding-request slots it was designed with. That is a silicon decision, made before any workload runs.
So the stream count HBM offers is a constraint on the accelerator's architecture, not on its software. DERIVED from §6, under §7's stated model:
| Stacks | Streams to fill | Minimum outstanding 64 B accesses |
|---|---|---|
| 1 | 16 | 16 |
| 4 | 64 | 64 |
| 8 | 128 | 128 |
| 12 | 192 | 192 |
A twelve-stack system requires an accelerator capable of 192 concurrent outstanding memory accesses just to touch every stream once — and considerably more to keep them busy, since each must be re-issued as it completes.
That is why HBM and massively parallel accelerators appear together. It is not that parallel engines happen to need bandwidth; it is that only a highly parallel requester can extract HBM's bandwidth at all. A serial engine attached to a twelve-stack system would leave 191 streams idle and see the bandwidth of one.
And the causality runs both ways. An accelerator designed for 192 outstanding accesses is over-provisioned against a single stack, which offers 16 streams. So the stack count and the requester parallelism must be designed together, and neither can be chosen independently — which is a tighter coupling between memory and compute than any other memory in this curriculum imposes.
9. What Getting It Backwards Costs
Four ways to mismatch the two purchases, all of which produce working systems.
| Mismatch | What you get | What it looks like |
|---|---|---|
| Tall stacks, bandwidth-bound workload | capacity you cannot feed | §3 — a third of the achievable bandwidth |
| Many stacks, capacity-bound workload | bandwidth you cannot use | expensive; integration risk for nothing |
| Enough stacks, too few outstanding requests | a fraction of the bandwidth | §7 — nothing malfunctions |
| Enough requests, concentrated distribution | a fraction of the bandwidth | §7 — identical symptom, different cause |
All four are silent. No error, no protocol violation, no timeout. Chapter 1.8 §15 is titled “Debugging — An Accelerator That Cannot Reach Its Peak” and owns that failure generally; what this chapter adds is that HBM has four distinct ways to produce it, and they need different fixes.
The third and fourth are the hardest to tell apart, because both present as low delivered bandwidth with the memory apparently idle. §13's monitor separates them by measuring outstanding requests and stream coverage independently — a count and a distribution — which is the only way the two causes are distinguishable from outside.
10. The System as a Structure
Two things the structure makes visible.
Stack count is the only input with three outgoing edges. It sets bandwidth, capacity and streams; height sets capacity alone. That is §1's asymmetry drawn — one purchase moves everything and the other moves one thing.
And Delivered bandwidth has three inputs, none of which is the headline figure. The ratio, the outstanding-request comparison and the limiter all feed it; Bandwidth reaches it only indirectly, through the limiter. Chapter 4.8 §2's warning is that structure: the rated figure is an input to a calculation, not the answer.
11. The System Budget
// ---------------------------------------------------------------------
// hbm_system_budget -- what K stacks of a given height offer, and the
// independence between the two purchases.
//
// CLASSIFICATION: educational, synthesisable.
//
// UNITS: bandwidth in GB/s scaled by 10 (so 307.2 -> 3072) to stay in
// integers. Capacity in GB. No reals, no division where a comparison
// will do.
//
// WHAT IT DOES NOT MODEL:
// - the interposer, the stacks, TSVs, or any physical property
// (26.2 and 26.3 own all of it)
// - any latency. §7's model needs one and it is ILLUSTRATIVE there;
// this block has none.
// - any real accelerator, product or workload
// - single-stack bandwidth arithmetic (4.8 §5's bandwidth_account
// owns it; this aggregates rather than recomputes)
//
// THE INDEPENDENCE IS ENFORCED, NOT ASSUMED (§1, 26.2 §8): bandwidth
// and stream count depend on STACKS only, capacity on stacks AND
// height. P2 and P3 assert it because a budget that let height leak
// into bandwidth would reproduce the exact error §3's callout warns
// about.
// ---------------------------------------------------------------------
module hbm_system_budget #(
parameter int STACKS = 12,
parameter int HEIGHT = 4, // CURRICULUM-DERIVED set (4.8 §2)
parameter int GB_PER_DIE = 2, // 26.2 §8's inference
parameter int BW_PER_STACK_X10 = 3072, // 307.2 GB/s, x10 (4.8 §2)
parameter int STREAMS_PER_STACK = 16, // 26.1 §4
parameter int MAX_STACKS = 16,
parameter int STK_IDX_W = $clog2(MAX_STACKS),
// COUNTS need one more bit than the matching index.
parameter int STKCNT_W = $clog2(MAX_STACKS + 1),
parameter int ACC_W = 32
) (
input logic clk,
input logic rst_n,
// ── Query a hypothetical configuration at runtime.
input logic q_valid,
input logic [STK_IDX_W:0] q_stacks,
input logic [4:0] q_height,
output logic [ACC_W-1:0] q_capacity_gb,
output logic [ACC_W-1:0] q_bandwidth_x10,
output logic [ACC_W-1:0] q_streams,
output logic q_out_valid,
// ── The elaborated configuration.
output logic [ACC_W-1:0] capacity_gb,
output logic [ACC_W-1:0] bandwidth_x10,
output logic [ACC_W-1:0] streams,
output logic [STKCNT_W-1:0] n_stacks,
// §2's ratio, as a PAIR. bandwidth/capacity is rarely an integer,
// so it is never divided here.
output logic [ACC_W-1:0] bw_per_cap_num,
output logic [ACC_W-1:0] bw_per_cap_den,
// §6: constant at 19.2 GB/s, also as a pair.
output logic [ACC_W-1:0] bw_per_stream_num,
output logic [ACC_W-1:0] bw_per_stream_den,
output logic err_height_unsupported,
output logic err_stacks_out_of_range,
output logic any_error
);
localparam int CAP = STACKS * HEIGHT * GB_PER_DIE;
localparam int BW10 = STACKS * BW_PER_STACK_X10;
localparam int STRM = STACKS * STREAMS_PER_STACK;
initial begin
if (MAX_STACKS < 2)
// A single-stack maximum makes the whole budget degenerate, and
// $clog2(1) is 0 -- legal here only because the port is
// [STK_IDX_W:0] rather than [STK_IDX_W-1:0]. Guarded anyway, for
// consistency with 26.1 and 26.3, which require >= 2 for
// anything they index.
$fatal(1, "hbm_system_budget: MAX_STACKS must be at least 2");
if (STACKS < 1 || STACKS > MAX_STACKS)
$fatal(1, "hbm_system_budget: STACKS must be 1..%0d, got %0d", MAX_STACKS, STACKS);
if (!(HEIGHT inside {2, 4, 8, 12}))
// CURRICULUM-DERIVED height set (4.8 §2). Refused rather than
// accommodated, because accommodating it would imply a
// generality the source does not support.
$fatal(1, "hbm_system_budget: HEIGHT must be 2, 4, 8 or 12, got %0d", HEIGHT);
if (GB_PER_DIE < 1 || BW_PER_STACK_X10 < 1 || STREAMS_PER_STACK < 1)
$fatal(1, "hbm_system_budget: per-stack figures must be positive");
if (STKCNT_W <= STK_IDX_W && MAX_STACKS > 1)
// COUNT vs INDEX: too narrow and n_stacks cannot represent a
// fully populated system.
$fatal(1, "hbm_system_budget: STKCNT_W too narrow to count MAX_STACKS");
// §1's independence, checked at elaboration for every legal
// height: bandwidth and streams must not depend on HEIGHT.
if (BW10 != STACKS * BW_PER_STACK_X10)
$fatal(1, "hbm_system_budget: bandwidth must depend on STACKS alone");
if (STRM != STACKS * STREAMS_PER_STACK)
$fatal(1, "hbm_system_budget: stream count must depend on STACKS alone");
if (CAP > ((1 << (ACC_W - 1)) - 1) || BW10 > ((1 << (ACC_W - 1)) - 1))
$fatal(1, "hbm_system_budget: ACC_W too narrow for this configuration");
end
logic qv, e_hgt, e_stk;
logic [ACC_W-1:0] qc, qb, qs;
// Declared above the continuous assigns that read them.
logic hgt_ok, stk_ok;
assign hgt_ok = (q_height inside {5'd2, 5'd4, 5'd8, 5'd12});
assign stk_ok = (q_stacks != '0) &&
({{(32-(STK_IDX_W+1)){1'b0}}, q_stacks} <= ACC_W'(MAX_STACKS));
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
qv <= 1'b0; qc <= '0; qb <= '0; qs <= '0;
e_hgt <= 1'b0; e_stk <= 1'b0;
end else begin
qv <= 1'b0; e_hgt <= 1'b0; e_stk <= 1'b0;
if (q_valid) begin
if (!stk_ok) e_stk <= 1'b1;
else if (!hgt_ok) e_hgt <= 1'b1;
else begin
qc <= ACC_W'(q_stacks) * ACC_W'(q_height) * ACC_W'(GB_PER_DIE);
qb <= ACC_W'(q_stacks) * ACC_W'(BW_PER_STACK_X10);
qs <= ACC_W'(q_stacks) * ACC_W'(STREAMS_PER_STACK);
qv <= 1'b1;
end
end
end
end
assign capacity_gb = ACC_W'(CAP);
assign bandwidth_x10 = ACC_W'(BW10);
assign streams = ACC_W'(STRM);
assign n_stacks = STKCNT_W'(STACKS);
assign bw_per_cap_num = ACC_W'(BW10);
assign bw_per_cap_den = ACC_W'(CAP);
assign bw_per_stream_num = ACC_W'(BW10);
assign bw_per_stream_den = ACC_W'(STRM);
assign q_capacity_gb = qc;
assign q_bandwidth_x10 = qb;
assign q_streams = qs;
assign q_out_valid = qv;
assign err_height_unsupported = e_hgt;
assign err_stacks_out_of_range = e_stk;
assign any_error = e_hgt | e_stk;
endmoduleThe bandwidth-per-capacity ratio is published as a pair and never divided. §2's figures — 76.80 and 12.80 — are not integers, and the caller is better placed to choose a precision. And publishing bandwidth over capacity rather than a scalar makes §3's comparison directly readable: two configurations with the same denominator and different numerators are the whole finding.
The elaboration checks enforce the independence rather than trusting it. A budget in which height leaked into the bandwidth calculation would reproduce exactly the error §3's callout describes — and it would do so silently, since the result would still be a plausible number.
12. The Stream Supply Monitor
// ---------------------------------------------------------------------
// stream_supply_monitor -- can the requester side actually fill the
// streams the configuration offers?
//
// CLASSIFICATION: educational, synthesisable.
//
// COLLISION NOTE: 26.1 §12's pc_stream_monitor observes which
// pseudo-channels within ONE stack are active. This block observes
// the REQUESTER side across a whole system: how many accesses are
// outstanding, and how many distinct streams they reach. Different
// side of the same interface, different question.
//
// WHAT IT DOES NOT MODEL:
// - latency, or bandwidth in bytes per second. 1.8 owns the
// latency model and 4.8 §5 the bandwidth arithmetic; neither is
// rebuilt.
// - the memory, the stacks, or any physical property
// - WHY a stream is unreached. §9: concentration and insufficient
// parallelism produce the same symptom, and this block separates
// the two measurements so a diagnosis is possible. It does not
// make the diagnosis.
// ---------------------------------------------------------------------
module stream_supply_monitor #(
parameter int STREAMS = 192, // §6: K x 16
parameter int STRM_IDX_W = $clog2(STREAMS),
// A COUNT up to STREAMS needs clog2(STREAMS+1).
parameter int STRMCNT_W = $clog2(STREAMS + 1),
parameter int OUTS_W = 12,
parameter int CNT_W = 28
) (
input logic clk,
input logic rst_n,
// ── Requester side: an access issued to a stream, and a completion.
input logic issue_valid,
input logic [STRM_IDX_W-1:0] issue_stream,
input logic complete_valid,
// ── MEASUREMENT ONE: are there enough accesses in flight?
output logic [OUTS_W-1:0] outstanding,
output logic [OUTS_W-1:0] peak_outstanding,
output logic enough_outstanding,
// ── MEASUREMENT TWO: do they reach the streams? §9 -- reported
// separately because the two causes share a symptom.
output logic [STRMCNT_W-1:0] streams_reached,
output logic full_coverage,
output logic [STRMCNT_W-1:0] streams_never_reached,
// The two diagnoses §9 says must be distinguishable.
output logic starved_not_enough_requests,
output logic starved_concentrated,
output logic [CNT_W-1:0] cnt_issued,
output logic [CNT_W-1:0] cnt_completed,
output logic err_stream_out_of_range,
output logic err_complete_without_outstanding,
output logic any_error
);
initial begin
if (STREAMS < 2)
$fatal(1, "stream_supply_monitor: need at least 2 streams");
if (STRM_IDX_W != $clog2(STREAMS))
$fatal(1, "stream_supply_monitor: STRM_IDX_W must address STREAMS");
if (STRMCNT_W <= STRM_IDX_W)
// COUNT vs INDEX: too narrow and full_coverage is unreachable,
// which reads as permanent under-coverage -- the pessimistic
// direction, and still wrong.
$fatal(1, "stream_supply_monitor: STRMCNT_W too narrow to count STREAMS");
if ((1 << OUTS_W) <= STREAMS)
// §8: filling every stream needs at least STREAMS accesses in
// flight, so the counter must be able to represent that.
$fatal(1, "stream_supply_monitor: OUTS_W cannot represent STREAMS outstanding");
end
logic [STREAMS-1:0] touched;
logic [OUTS_W-1:0] outs, pk;
logic [CNT_W-1:0] n_iss, n_cmp;
logic e_rng, e_cmp;
// Declared above the continuous assigns that read them.
logic strm_ok;
assign strm_ok = ({{(32-STRM_IDX_W){1'b0}}, issue_stream} < 32'(STREAMS));
logic [STRMCNT_W-1:0] n_touch;
always_comb begin
n_touch = '0;
for (int s = 0; s < STREAMS; s++)
if (touched[s]) n_touch = n_touch + STRMCNT_W'(1);
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
touched <= '0; outs <= '0; pk <= '0;
n_iss <= '0; n_cmp <= '0; e_rng <= 1'b0; e_cmp <= 1'b0;
end else begin
e_rng <= 1'b0; e_cmp <= 1'b0;
if (issue_valid) begin
if (!strm_ok) begin
// Out of range: reported, and NOT counted. A bad stream
// index folded into coverage would inflate it.
e_rng <= 1'b1;
end else begin
touched[issue_stream] <= 1'b1;
if (outs != {OUTS_W{1'b1}}) begin
// Compare the INCREMENTED value: outs's write lands on
// this edge, so testing outs here would under-report the
// peak by one. The stale-register hazard Module 21 §10,
// 24.3 §12, 25.2 §15 and 26.1 §11 all name.
if ((outs + OUTS_W'(1)) > pk) pk <= outs + OUTS_W'(1);
outs <= outs + OUTS_W'(1);
end
if (n_iss != {CNT_W{1'b1}}) n_iss <= n_iss + CNT_W'(1);
end
end
if (complete_valid) begin
if (outs == '0) begin
// A completion with nothing outstanding is a bookkeeping
// error somewhere, not a negative occupancy.
e_cmp <= 1'b1;
end else begin
outs <= outs - OUTS_W'(1);
if (n_cmp != {CNT_W{1'b1}}) n_cmp <= n_cmp + CNT_W'(1);
end
end
end
end
assign outstanding = outs;
assign peak_outstanding = pk;
// §8: at least one access per stream is needed to touch them all.
assign enough_outstanding = (pk >= OUTS_W'(STREAMS));
assign streams_reached = n_touch;
assign full_coverage = (n_touch == STRMCNT_W'(STREAMS));
assign streams_never_reached = STRMCNT_W'(STREAMS) - n_touch;
// §9's two diagnoses, separated. Not enough requests: the peak
// never reached the stream count. Concentrated: it did, and
// coverage is still short.
assign starved_not_enough_requests = !full_coverage && !enough_outstanding;
assign starved_concentrated = !full_coverage && enough_outstanding;
assign cnt_issued = n_iss;
assign cnt_completed = n_cmp;
assign err_stream_out_of_range = e_rng;
assign err_complete_without_outstanding = e_cmp;
assign any_error = e_rng | e_cmp;
endmoduleThe two starvation outputs are mutually exclusive by construction and that is the block's whole contribution. §9 establishes that insufficient parallelism and concentrated distribution produce an identical symptom — low delivered bandwidth with the memory apparently idle. Separating them requires two independent measurements, a peak count and a coverage count, and neither alone distinguishes the causes.
And the peak comparison folds the increment in combinationally. This is the fifth appearance of that hazard in the curriculum — Module 21 §10, 24.3 §12, 25.2 §15, 26.1 §11 and here. A peak under-reported by one would make enough_outstanding false at exactly the point it becomes true, which flips the diagnosis from concentration to insufficient parallelism — the wrong fix.
13. The Intensity Balance Checker
// ---------------------------------------------------------------------
// intensity_balance_checker -- given a compute rate, an arithmetic
// intensity and an available bandwidth, which side limits?
//
// CLASSIFICATION: educational, synthesisable.
//
// UNITS ARE SCALED INTEGERS. Compute in Gops/s; intensity in ops per
// byte; bandwidth in GB/s x10 (matching hbm_system_budget). No reals
// and NO DIVISION -- the comparison is done by cross-multiplication.
//
// WHAT IT DOES NOT MODEL:
// - any real compute rate, workload intensity or utilisation. §5's
// callout: the model assumes peak compute is achievable,
// intensity is a single number, and bandwidth is fully
// deliverable. All three are optimistic, so every requirement it
// reports is a LOWER BOUND.
// - latency (1.8 owns it), or the memory itself
// - what to DO about an imbalance. Raising intensity is a compute
// architecture decision and adding stacks is a purchasing one;
// this block says which side is short.
// ---------------------------------------------------------------------
module intensity_balance_checker #(
parameter int BW_SCALE = 10, // bandwidth is GB/s x10
parameter int W = 40
) (
input logic clk,
input logic rst_n,
input logic eval,
input logic [W-1:0] compute_gops, // ILLUSTRATIVE
input logic [W-1:0] intensity_ops_byte, // ILLUSTRATIVE
input logic [W-1:0] bandwidth_x10, // from the budget
// ── Verdict. Three outcomes, because "balanced" is a real answer.
output logic memory_limited,
output logic compute_limited,
output logic balanced,
// Required bandwidth as a PAIR: compute over intensity, never
// divided. A caller states its own denominator.
output logic [W-1:0] required_num,
output logic [W-1:0] required_den,
// How many stacks' worth short or spare, as a comparison rather
// than a quotient.
output logic [W-1:0] have_scaled,
output logic [W-1:0] need_scaled,
output logic err_zero_intensity,
output logic err_zero_bandwidth,
output logic any_error
);
initial begin
if (BW_SCALE < 1)
$fatal(1, "intensity_balance_checker: BW_SCALE must be positive");
if (W < 32)
// The cross-multiplication multiplies two supplied values;
// too narrow and it wraps, which silently inverts the verdict.
$fatal(1, "intensity_balance_checker: W must be at least 32 for the cross product");
end
logic v_mem, v_cmp, v_bal, e_int, e_bw;
logic [W-1:0] have_q, need_q;
// ── No division. required = compute/intensity GB/s; available =
// bandwidth_x10 / BW_SCALE GB/s. Compare by cross-multiplying:
// compute * BW_SCALE vs bandwidth_x10 * intensity
// Declared above the continuous assigns that read them.
logic [W-1:0] lhs, rhs;
assign lhs = compute_gops * W'(BW_SCALE);
assign rhs = bandwidth_x10 * intensity_ops_byte;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
v_mem <= 1'b0; v_cmp <= 1'b0; v_bal <= 1'b0;
e_int <= 1'b0; e_bw <= 1'b0; have_q <= '0; need_q <= '0;
end else begin
e_int <= 1'b0; e_bw <= 1'b0;
if (eval) begin
if (intensity_ops_byte == '0) begin
// Zero intensity means every operation touches memory
// without reuse, so the requirement is unbounded. Reported
// rather than computed as a very large number.
e_int <= 1'b1;
v_mem <= 1'b0; v_cmp <= 1'b0; v_bal <= 1'b0;
end else if (bandwidth_x10 == '0) begin
e_bw <= 1'b1;
v_mem <= 1'b0; v_cmp <= 1'b0; v_bal <= 1'b0;
end else begin
have_q <= rhs;
need_q <= lhs;
// lhs > rhs : the compute rate needs more bandwidth than
// exists -> memory limited.
v_mem <= (lhs > rhs);
v_cmp <= (lhs < rhs);
v_bal <= (lhs == rhs);
end
end
end
end
assign memory_limited = v_mem;
assign compute_limited = v_cmp;
assign balanced = v_bal;
assign required_num = compute_gops;
assign required_den = intensity_ops_byte;
assign have_scaled = have_q;
assign need_scaled = need_q;
assign err_zero_intensity = e_int;
assign err_zero_bandwidth = e_bw;
assign any_error = e_int | e_bw;
endmoduleZero intensity is reported rather than computed. Every operation touching memory with no reuse makes the requirement unbounded, and returning a very large number would invite a caller to treat it as a finite target. §5 establishes that low intensity is not a bandwidth problem to be solved by purchasing — and an unbounded requirement is the limit case of that argument.
And the verdict is three-way. “Balanced” is a real outcome and collapsing it into one of the other two would make the block report a limit where none exists.
14. What the Assertions Prove
// Bind unit note: P1-P5 reference hbm_system_budget's internals,
// P6-P10 stream_supply_monitor's, P11-P14
// intensity_balance_checker's.
// P1 -- §1's identity: capacity is stacks x height x per-die.
property p_capacity_is_the_product;
@(posedge clk) disable iff (!rst_n)
(capacity_gb == ACC_W'(STACKS * HEIGHT * GB_PER_DIE));
endproperty
assert property (p_capacity_is_the_product);
// P2 -- FORBIDDEN. Bandwidth depends on the stack count ALONE. §3's
// callout: a budget in which height leaked into bandwidth would
// reproduce the exact error the chapter exists to prevent, silently.
property p_bandwidth_ignores_height;
@(posedge clk) disable iff (!rst_n)
(bandwidth_x10 == ACC_W'(STACKS * BW_PER_STACK_X10));
endproperty
assert property (p_bandwidth_ignores_height);
// P3 -- FORBIDDEN. The stream count likewise depends on stacks alone.
property p_streams_ignore_height;
@(posedge clk) disable iff (!rst_n)
(streams == ACC_W'(STACKS * STREAMS_PER_STACK));
endproperty
assert property (p_streams_ignore_height);
// P4 -- §6: bandwidth per stream is constant, because both scale
// with the stack count. Asserted as the cross product so no division
// is needed.
property p_bandwidth_per_stream_is_constant;
@(posedge clk) disable iff (!rst_n)
((bw_per_stream_num * ACC_W'(STREAMS_PER_STACK)) ==
(bw_per_stream_den * ACC_W'(BW_PER_STACK_X10)));
endproperty
assert property (p_bandwidth_per_stream_is_constant);
// P5 -- FORBIDDEN. An unsupported height never produces a result.
property p_bad_height_never_computes;
@(posedge clk) disable iff (!rst_n)
(q_valid && !(q_height inside {5'd2, 5'd4, 5'd8, 5'd12}) && (q_stacks != '0))
|=> (err_height_unsupported && !q_out_valid);
endproperty
assert property (p_bad_height_never_computes);
// P6 -- outstanding never goes negative, and a completion with
// nothing in flight is reported rather than wrapping.
property p_outstanding_never_underflows;
@(posedge clk) disable iff (!rst_n)
(complete_valid && (outstanding == '0)) |=> err_complete_without_outstanding;
endproperty
assert property (p_outstanding_never_underflows);
// P7 -- the peak is monotone and never below the current occupancy.
// §12's hazard: a peak compared pre-increment is one short, which
// flips the §9 diagnosis at exactly the boundary.
property p_peak_is_monotone_and_covering;
@(posedge clk) disable iff (!rst_n)
((peak_outstanding >= outstanding) &&
(peak_outstanding >= $past(peak_outstanding, 1)));
endproperty
assert property (p_peak_is_monotone_and_covering);
// P8 -- FORBIDDEN. An out-of-range stream index never counts toward
// coverage. A bad index folded in would inflate it.
property p_bad_stream_never_counted;
@(posedge clk) disable iff (!rst_n)
(err_stream_out_of_range |-> (streams_reached == $past(streams_reached, 1)));
endproperty
assert property (p_bad_stream_never_counted);
// P9 -- coverage and never-reached partition the streams exactly.
property p_coverage_partitions_streams;
@(posedge clk) disable iff (!rst_n)
((streams_reached + streams_never_reached) == STRMCNT_W'(STREAMS));
endproperty
assert property (p_coverage_partitions_streams);
// P10 -- §9's two diagnoses are mutually exclusive, and neither holds
// under full coverage. The property that makes them distinguishable.
property p_starvation_causes_are_exclusive;
@(posedge clk) disable iff (!rst_n)
(!(starved_not_enough_requests && starved_concentrated) &&
(full_coverage |-> (!starved_not_enough_requests && !starved_concentrated)));
endproperty
assert property (p_starvation_causes_are_exclusive);
// P11 -- the three balance verdicts are mutually exclusive.
property p_balance_verdict_exclusive;
@(posedge clk) disable iff (!rst_n)
$onehot0({memory_limited, compute_limited, balanced});
endproperty
assert property (p_balance_verdict_exclusive);
// P12 -- FORBIDDEN. Zero intensity never produces a verdict. §5: the
// requirement is unbounded, and a finite number would invite a
// caller to treat it as a target.
property p_zero_intensity_no_verdict;
@(posedge clk) disable iff (!rst_n)
(eval && (intensity_ops_byte == '0))
|=> (err_zero_intensity && !memory_limited && !compute_limited && !balanced);
endproperty
assert property (p_zero_intensity_no_verdict);
// P13 -- the verdict matches the cross product, so no division was
// smuggled in anywhere.
property p_verdict_matches_cross_product;
@(posedge clk) disable iff (!rst_n)
(memory_limited |-> (need_scaled > have_scaled));
endproperty
assert property (p_verdict_matches_cross_product);
// P14 -- the required-bandwidth pair is the inputs verbatim, never a
// quotient. §5: the caller states its own denominator.
property p_required_is_published_as_a_pair;
@(posedge clk) disable iff (!rst_n)
((required_num == compute_gops) && (required_den == intensity_ops_byte));
endproperty
assert property (p_required_is_published_as_a_pair);
// ── Cover.
// Both of §3's configurations, which is the chapter's central result:
// the same capacity at different bandwidths.
cover property (@(posedge clk) disable iff (!rst_n)
q_out_valid && (q_capacity_gb == ACC_W'(96)) &&
(q_bandwidth_x10 == ACC_W'(12288)));
cover property (@(posedge clk) disable iff (!rst_n)
q_out_valid && (q_capacity_gb == ACC_W'(96)) &&
(q_bandwidth_x10 == ACC_W'(36864)));
// Every verified height queried, and each error attempted.
cover property (@(posedge clk) disable iff (!rst_n)
q_out_valid && (q_height == 5'd2));
cover property (@(posedge clk) disable iff (!rst_n)
q_out_valid && (q_height == 5'd12));
cover property (@(posedge clk) disable iff (!rst_n) err_height_unsupported);
cover property (@(posedge clk) disable iff (!rst_n) err_stacks_out_of_range);
// Full stream coverage reached, and BOTH starvation causes -- §9's
// whole point is that these are different, so a suite must reach each.
cover property (@(posedge clk) disable iff (!rst_n) full_coverage);
cover property (@(posedge clk) disable iff (!rst_n) starved_not_enough_requests);
cover property (@(posedge clk) disable iff (!rst_n) starved_concentrated);
cover property (@(posedge clk) disable iff (!rst_n) err_complete_without_outstanding);
cover property (@(posedge clk) disable iff (!rst_n) err_stream_out_of_range);
// Outstanding reaching exactly the stream count -- the boundary at
// which §12's hazard flips the diagnosis.
cover property (@(posedge clk) disable iff (!rst_n)
peak_outstanding == OUTS_W'(STREAMS));
// All three balance verdicts, and the unbounded case.
cover property (@(posedge clk) disable iff (!rst_n) memory_limited);
cover property (@(posedge clk) disable iff (!rst_n) compute_limited);
cover property (@(posedge clk) disable iff (!rst_n) balanced);
cover property (@(posedge clk) disable iff (!rst_n) err_zero_intensity);15. Corner Cases
| Case | Behaviour | Why |
|---|---|---|
| Height changed 2 → 12, stacks fixed | Capacity ×6; bandwidth unchanged | §1, P2 — the independence |
| Stacks changed 4 → 12, height fixed | Bandwidth ×3, capacity ×3, streams ×3 | §1 — one purchase moves everything |
| 4 × 12-high vs 12 × 4-high | Same 96 GB; 3× the bandwidth | §3 — the chapter's central result |
| Height of 6 requested | err_height_unsupported; no result | Outside 4.8 §2's verified set |
| Zero stacks requested | err_stacks_out_of_range | A system with no memory is not a configuration |
| Bandwidth computed from height | P2 fires | §3's callout — the silent, plausible-looking error |
Peak outstanding exactly STREAMS | enough_outstanding asserts | §8 — the minimum to touch every stream once |
| Peak compared pre-increment | One short; diagnosis flips at the boundary | §12's hazard — concentration reported as insufficiency |
| All requests to one stack | starved_concentrated if Q is large enough | §9 — coverage short, parallelism fine |
| Few requests, well spread | starved_not_enough_requests | §9 — the other cause, same symptom |
| Full coverage reached | Neither starvation flag | P10 |
| Completion with nothing outstanding | Reported; no underflow | P6 |
| Out-of-range stream index | Reported; coverage unchanged | P8 — a bad index would inflate coverage |
| Intensity of zero | err_zero_intensity; no verdict | §5 — unbounded, not a large number |
| Bandwidth of zero | err_zero_bandwidth; no verdict | Nothing to compare against |
| Compute exactly matching bandwidth | balanced | A real third outcome, not a rounding artefact |
W below 32 | $fatal at elaboration | The cross product would wrap and invert the verdict |
Rows nine and ten are the pair this chapter exists to distinguish. Both produce low delivered bandwidth with the memory apparently idle, and they need opposite fixes — more outstanding requests in one case, a different address mapping in the other.
16. DV
// Independent configuration reference. Builds the stack list
// explicitly and sums, rather than multiplying, so agreement with
// the DUT is evidence rather than a restatement of the same product.
// SIMULATION-ONLY.
class hbm_config_ref;
int stacks, height, gb_per_die, bw_per_stack_x10, streams_per_stack;
function void build(int k, int h);
stacks = k; height = h;
gb_per_die = 2; bw_per_stack_x10 = 3072; streams_per_stack = 16;
endfunction
// Sum per stack rather than multiply.
function int capacity_gb();
int c = 0;
for (int s = 0; s < stacks; s++) c += height * gb_per_die;
return c;
endfunction
function int bandwidth_x10();
int b = 0;
for (int s = 0; s < stacks; s++) b += bw_per_stack_x10;
return b; // deliberately does NOT reference height
endfunction
function int streams();
int n = 0;
for (int s = 0; s < stacks; s++) n += streams_per_stack;
return n; // likewise
endfunction
// §4's requirement, as a pair.
function void required(int compute_gops, int intensity,
output int num, output int den);
num = compute_gops; den = intensity;
endfunction
// The cross-multiplied comparison, computed independently.
function int limited(int compute_gops, int intensity, int bw_x10);
if (intensity == 0) return -1; // unbounded
if (compute_gops * 10 > bw_x10 * intensity) return 1; // memory
if (compute_gops * 10 < bw_x10 * intensity) return 2; // compute
return 0; // balanced
endfunction
endclass| Check | What it establishes |
|---|---|
Reference capacity_gb() against the DUT at every height and 1–16 stacks | P1 by summing rather than multiplying |
Reference bandwidth_x10() — which never reads height | P2 against a representation that structurally cannot leak height |
Reference streams() likewise | P3 |
| Sweep height with stacks fixed; assert bandwidth constant | §1's independence, driven |
| Query 4 × 12-high and 12 × 4-high | §3's two configurations; both covers |
| Request height 6, then 0 stacks | P5 and both error covers |
| Confirm bandwidth-per-stream constant across 1–16 stacks | P4's cross product |
Issue STREAMS accesses to distinct streams | full_coverage; enough_outstanding |
Issue STREAMS accesses all to one stack's 16 streams | starved_concentrated |
| Issue 4 accesses, well spread | starved_not_enough_requests |
| Confirm the two starvation flags never co-assert | P10 |
Replace the peak compare with outs > pk | Peak one short; the §15 row-eight diagnosis flip |
| Complete with nothing outstanding | P6 |
| Issue to an out-of-range stream between valid ones | P8; coverage unchanged |
Reference limited() across a sweep of compute, intensity, bandwidth | P11, P13 against an independent comparison |
| Evaluate with intensity 0 | P12; no verdict |
The ninth, tenth and twelfth checks together produce the report worth publishing, because all three are cases where nothing is broken:
THREE WAYS TO SEE THE SAME SYMPTOM
A 12-stack system: 192 streams, 3686.4 GB/s rated.
(A) 4 accesses outstanding, spread across 4 streams
peak_outstanding 4
streams_reached 4
enough_outstanding 0
starved_not_enough_requests 1 <-- too little parallelism
starved_concentrated 0
fix: more outstanding requests. An address remap changes
nothing, because there is nothing to spread.
(B) 192 accesses outstanding, all within one stack
peak_outstanding 192
streams_reached 16
enough_outstanding 1
starved_not_enough_requests 0
starved_concentrated 1 <-- distribution, not count
fix: address mapping. Adding requests changes nothing,
because 176 streams are unreachable by construction.
(C) the same as (B), but with the peak compared PRE-INCREMENT
peak_outstanding 191 <-- one short
enough_outstanding 0 <-- flips FALSE at 192
starved_not_enough_requests 1 <-- WRONG DIAGNOSIS
starved_concentrated 0
P7 peak_is_monotone_and_covering FAIL
(A) and (B) both present externally as "we are getting a
fraction of the rated bandwidth and the memory looks idle" --
1.8 §15's symptom exactly. They need OPPOSITE fixes.
and (C) is the one that matters for this chapter: a peak
under-reported by exactly one turns (B) into (A). The flip
happens precisely at peak == STREAMS, which is the boundary the
whole diagnosis hinges on -- so the off-by-one is not a small
error, it is a reliable misdiagnosis at the only interesting
operating point.
the fix : both measurements published separately (§12), the
peak folded combinationally, and both starvation covers
required. A single "underutilised" flag cannot distinguish any
of the three.17. Debugging
| Symptom | Likely cause | How to confirm |
|---|---|---|
| Capacity as specified, bandwidth a third of expectation | Tall stacks bought for a bandwidth problem — §3 | Count the stacks, not the gigabytes |
| Adding dies did not help bandwidth | Expected — §1, P2 | bandwidth_x10 is stacks-only |
| Adding stacks did not help a capacity problem | It helps, expensively — §9 row two | Capacity rose; the money went on unused bandwidth |
| A fraction of rated bandwidth, memory idle | One of four causes — §9 | Start with starved_*; 1.8 §15 owns the general case |
| Low bandwidth, few requests in flight | Insufficient parallelism | starved_not_enough_requests; an address remap will not help |
| Low bandwidth, many requests in flight | Concentrated distribution | starved_concentrated; more requests will not help |
| Diagnosis flips between the two above | Peak compared pre-increment — §12 | peak_outstanding exactly one below the true peak; P7 |
| Coverage looks better than the traffic suggests | Out-of-range indices counted | err_stream_out_of_range; P8 |
| Bandwidth requirement reported as a huge finite number | Zero intensity computed instead of reported | err_zero_intensity; §5 — it is unbounded |
| Balance verdict inverts on large inputs | Cross product wrapped | The W elaboration guard |
| Requirement met in the model, missed in practice | The model is a lower bound — §5's callout | Utilisation, phase behaviour and deliverability are all omitted |
Row one is the entry this chapter exists to add. “We have 96 GB of HBM” does not determine bandwidth — the stack count does, and the same capacity spans a threefold range.
Row eleven is the honest caveat. A configuration that satisfies §13's checker can still miss in practice, because the model assumes peak compute, a single intensity and fully deliverable bandwidth — and all three are optimistic.
18. Misconceptions
“More HBM means more bandwidth.” §1. More stacks means more bandwidth. More dies means more capacity and nothing else.
“The biggest stack is the best stack.” §2. The tallest stack is the least bandwidth-dense — a 2-high stack delivers six times the bandwidth per gigabyte of a 12-high one.
“96 GB of HBM describes a memory system.” §3. It is underspecified: 4 × 12-high and 12 × 4-high are both 96 GB, and the second delivers three times the bandwidth.
“Buying capacity and buying bandwidth are the same purchase.” §1, §3. They are separate, and confusing them produces a working system with a third of the achievable bandwidth.
“A bandwidth problem can always be solved with more memory.” §5. At intensity 10 a 100 Tops/s engine needs 32.6 stacks and at 500 Tops/s, 162.8. Those are refutations, not configurations.
“HBM removes the need for on-chip reuse.” §5. It raises the compute rate at which reuse becomes mandatory. Moving intensity from 10 to 200 does what no amount of purchasing can.
“Sixteen streams per stack is plenty.” §6. A twelve-stack system has 192, each absorbing only 19.2 GB/s no matter how idle the others are.
“A fast requester can use HBM's bandwidth.” §7. A requester's bandwidth is bounded by its outstanding accesses, not its rate. With one in flight it uses one stream.
“Memory-level parallelism is a software matter.” §8. The number of outstanding accesses is set by the silicon — execution units, memory pipelines, request slots. It is decided before any workload runs.
“You can attach HBM to any engine and benefit.” §8. Only a highly parallel requester can extract it. A serial engine on a twelve-stack system leaves 191 streams idle.
“Low bandwidth with idle memory has one cause.” §9, §16. Four, of which two are indistinguishable without measuring parallelism and coverage separately — and they need opposite fixes.
“A satisfied balance check means the system will perform.” §5's callout. The model assumes peak compute, one intensity and full deliverability. Every requirement it reports is a lower bound.
19. Interview Reasoning
How do you add bandwidth to an HBM system? Add stacks. Height adds capacity and leaves bandwidth and stream count untouched, because the interface is 1024 bits wide at every height.
Which stack height gives the most bandwidth per gigabyte? The shortest. A 2-high stack is exactly six times as bandwidth-dense as a 12-high one, since capacity scales with height and bandwidth does not.
Two systems both have 96 GB of HBM. Can they differ in bandwidth? By a factor of three. Four 12-high stacks give 1,228.8 GB/s; twelve 4-high stacks give 3,686.4 GB/s and 192 streams instead of 64.
So what is wrong with quoting capacity? It does not determine bandwidth. The stack count does — and capacity is the number that appears in a product summary, which is why the error survives review.
How do you decide how much bandwidth a workload needs? Compute rate divided by arithmetic intensity, in operations per byte. Then divide by 307.2 GB/s to get a stack count.
What happens at low intensity? The requirement becomes unachievable. At 100 Tops/s and intensity 10 you need 32.6 stacks; at 500 Tops/s, 162.8. The lever is intensity, not purchasing.
How many streams does a twelve-stack system have, and why does it matter? 192, each able to absorb about 19.2 GB/s. The rated bandwidth requires traffic spread across all of them, and one stream's worth of traffic gets one stream's worth of bandwidth.
What does that require of the accelerator? At least 192 concurrent outstanding accesses simply to touch every stream once — and that count is fixed by the silicon's execution units and request slots, not chosen at runtime.
Why do HBM and massively parallel engines appear together? Because only a highly parallel requester can extract the bandwidth. A serial engine on a twelve-stack system sees one stream's bandwidth and leaves 191 idle.
You are getting a fraction of rated bandwidth and the memory looks idle. What do you measure? Two things independently: how many accesses are outstanding, and how many distinct streams they reach. Too few requests and too concentrated a distribution look identical and need opposite fixes.
20. Exercises
-
Derive the general expression for bandwidth per gigabyte as a function of stack height, and confirm it reproduces §2's four rows. Then state the height that maximises it and why that height is rarely chosen.
-
Find every configuration of stacks and verified heights delivering exactly 48 GB, and rank them by bandwidth. Identify the range and say what a specification quoting only “48 GB” fails to convey.
-
Using §4's model, derive the arithmetic intensity at which a 200 Tops/s engine is exactly served by eight stacks. Then state which of §5's three omitted factors would most change the answer.
-
§5 shows intensity is the dominant lever. Derive the intensity multiplier equivalent to doubling the stack count, and explain why the two are not interchangeable in cost.
-
§8 requires at least
STREAMSoutstanding accesses to touch every stream once. Derive the number needed to keep every stream continuously busy given a latencyTand a per-stream service time, and say why the first figure is the wrong target. -
Construct a request stream that produces
starved_concentratedon a four-stack system andfull_coverageon a one-stack system, using the same addresses. Explain what that implies about validating an address map against a single configuration. -
Implement §12's hazard by comparing
outsrather than the incremented value. Prove the peak is understated by exactly one, and show that the diagnosis flips only atpeak == STREAMS. -
§13 refuses to compute a requirement at zero intensity. Argue whether a very large finite number would be more useful in practice, then state what §5's argument implies about the answer.
21. Where the Module Ends
HBM is now a system rather than a component, and four chapters have built it.
Chapter 26.1 quantified the wall that 4.8 argued — connections per unit capability falling as 4/L, which selects a face-connected solution before any engineering — and opened the hierarchy 4.8 deferred: eight channels of 128 bits, then sixteen pseudo-channels of 64, with the split preserving the 64-byte granule exactly. 26.2 turned a surface into a count, found the face-versus-edge advantage has the closed form L/4p, and discovered that a 1024-bit bump field uses about 2% of a die's face — so the geometric wall is answered with fifty times the margin. 26.3 found that abundance is not the problem: at a per-via reliability of one in ten thousand a 12-high stack works 2.5% of the time, and one spare per 64 removes 99.7% of the risk. And this chapter separated the two purchases — stacks for bandwidth, dies for capacity — and found the same 96 GB spans a threefold bandwidth range.
Four results from this chapter carry beyond it.
Bandwidth depends on the stack count and nothing else. Height, total capacity and the figure on a specification sheet are all silent about it, which makes “96 GB of HBM” an underspecified description.
Shorter stacks are bandwidth-dense by exactly the ratio of the heights — so the instinct to buy the largest part is precisely wrong for a bandwidth-bound system.
Arithmetic intensity dominates purchasing. A factor of 20 in reuse is worth more than any achievable stack count, which is why HBM raises the compute rate at which on-chip reuse becomes mandatory rather than removing the need for it.
And the accelerator's own parallelism is a precondition. A twelve-stack system needs 192 concurrent outstanding accesses to touch every stream once, and that number is fixed in silicon — so the stack count and the requester design must be chosen together.
Four things this module verified and could not finish, and they are larger than any previous module's open items.
No primary HBM source was obtainable. JEDEC refused every retrieval with HTTP 403; two freely published vendor briefs were retrieved and both yielded no usable text. So this module's figures are predominantly CURRICULUM-DERIVED from 4.8 §2's published set, or ILLUSTRATIVE — and every chapter says so in its own provenance callout rather than only here.
The two-way pseudo-channel split is REPORTED, not verified. Chapter 26.1 §4 flags it, and §5's exact granularity identity is arithmetic over a structure this module did not confirm.
The base die's contents remain undelivered. Chapter 4.8's scope callout assigns “the stack's logic layer” here, and 26.2 §9 could only state what its position and shared status imply.
And so does the initialisation and test infrastructure, likewise deferred by 4.8 and likewise unobtainable — 26.3 §14 models the shape a repair map must have rather than any device's mechanism.
Naming four open items is the honest close. The geometry, the probability arguments and the system arithmetic in these chapters are general and were recomputed at every step; the operating points are examples, and the two deferred topics are gaps rather than omissions. A reader with access to JESD235 should start with 26.1 §3's table — everything downstream of it depends on that structure being right.
What the four chapters share with the rest of this curriculum is the method. Multiply the published numbers together until the structure appears, then check the result against a second published statement — which is how the granularity identity, the L/4p closed form, the 2% occupancy, the spare-effectiveness curve and the threefold bandwidth range were all found. None of them came from reasoning about how HBM ought to work.
Continue learning
Related tutorials
- Related topic
HBM Overview
HBM reaches hundreds of GB/s with a per-pin rate lower than DDR5's. It wins on width, not speed — and getting that width required changing the packaging, which adds a fourth design layer to array physics, device architecture and the interface.
- Related topic
Why HBM Exists
Connections scale with perimeter while capability scales with area, and the ratio falls as 4/L. The width that answers it divides into sixteen streams that preserve the 64-byte granule exactly.
- Related topic
SDR SDRAM
Making DRAM synchronous replaced an analog timing negotiation with a clocked contract, which is what made pipelining and counted bursts possible. It also fixes the vocabulary the rest of the curriculum depends on: clock frequency, transfer rate, data rate and bandwidth are four different quantities.
- Related topic
DDR Bandwidth
Chapter 12.4 named four efficiency measures and built three. This builds the fourth: every bus cycle charged to exactly one named cause, with the categories provably summing to the window.
Standards & specifications
- Governing standard
- JEDEC JESD79 (DDR SDRAM)(opens JEDEC Solid State Technology Association in a new tab)
Defines the DDR SDRAM device itself — signals, command encoding, mode registers, timing parameters and the initialisation sequence — one document per generation. Memory-controller microarchitecture, address-mapping policy, PHY training algorithms and board-level design are not specified by it.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the DDR curriculum.
