Skip to content
VLSI Mentor

DDR · Module 26

HBM for AI Accelerators

Stacks buy bandwidth and dies buy capacity. The same 96 GB bought as twelve short stacks delivers three times the bandwidth of four tall ones — and one requester cannot use any of it.

The three chapters before this one established what HBM is: a 1024-bit interface divided into sixteen semi-independent streams, reached through a die's face rather than its edge, stacked on ordinary dies whose vertical connections need spares to work at all.

None of them asked what a system does with it.

Chapter 4.8 §2 issues the warning this chapter is built on — HBM's headline figure “requires traffic that keeps all eight channels busy”, and “one channel's worth of traffic gets one channel's worth of bandwidth, and no amount of width helps” — and 26.1 §4 doubled the relevant count to sixteen. Chapter 26.2 §8 adds the other half: height buys capacity and not bandwidth.

Put those together and an accelerator faces two separate purchases that look like one. §3 shows that the same capacity, bought two ways, differs in delivered bandwidth by a factor of three.

1. Two Purchases That Look Like One

Chapter 26.2 §8 established the independence and proved it with a property: the interface is 1024 bits wide at every stack height, so height changes capacity and leaves bandwidth alone.

State both sides. CURRICULUM-DERIVED from 4.8 §2 and 26.2 §8:

What you addWhat changesWhat does not
A die (taller stack)capacity, +2 GBbandwidth, streams
A stackbandwidth, +307.2 GB/s; streams, +16; capacity, +up to 24 GB

So one of the two purchases is strictly more powerful and strictly more expensive. A stack adds everything; a die adds only capacity.

That asymmetry is the whole chapter. It means a system's bandwidth is set by its stack count and nothing else — not by total capacity, not by stack height, not by how much memory the specification sheet lists.

2. Shorter Stacks Are Bandwidth-Dense

Divide one quantity by the other and the asymmetry becomes a design lever.

DERIVED from §1, recomputed:

Stack heightCapacityBandwidthBandwidth per GB
24 GB307.2 GB/s76.80 (GB/s)/GB
48 GB307.2 GB/s38.40
816 GB307.2 GB/s19.20
1224 GB307.2 GB/s12.80

A 2-high stack is exactly six times as bandwidth-dense as a 12-high one — and the factor is exactly 6 because the capacity ratio is 12/2 and the bandwidth is identical.

Three readings, and the second is counter-intuitive enough to be worth stating carefully.

The tallest stack is the least bandwidth-dense. Height is the one dimension that adds capacity without adding bandwidth, so maximising height minimises the ratio.

So a bandwidth-hungry system should prefer short stacks, and more of them. That inverts the instinct to buy the biggest available part — the biggest part is the worst choice per unit of bandwidth, and §3 makes the difference concrete.

And a capacity-hungry system should prefer tall stacks, because it is paying for stacks it does not need bandwidth from. The two workloads want opposite configurations from the same component family.

3. Two Ways to Buy Ninety-Six Gigabytes

Here is the chapter's sharpest result. DERIVED, recomputed:

ConfigurationCapacityBandwidthStreams
4 stacks × 12-high96 GB1,228.8 GB/s64
12 stacks × 4-high96 GB3,686.4 GB/s192

Identical capacity. Three times the bandwidth. Three times the streams.

The second configuration uses three times as many stacks, each a third as tall — and 26.3 §5 notes that shorter stacks are also the easier ones to build, since the all-vias-work probability degrades with height.

So the two configurations are not a trade between bandwidth and capacity. They deliver the same capacity, and one delivers three times the bandwidth. The trade is against cost and integration: twelve stacks need twelve stacks' worth of interposer area, twelve bump fields, twelve assemblies — and 4.8 §3 establishes that each assembly step carries a yield risk that “scrapes an assembly containing an expensive processor.”

4. What a Workload Actually Needs

Neither purchase is right in the abstract. The requirement comes from the workload, and it has a standard form.

The model, stated. A computation performs operations and moves bytes. Its arithmetic intensity I is operations per byte of memory traffic. For a compute rate C operations per second, the memory traffic required to sustain it is:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   required bandwidth  =  C / I        bytes per second

High intensity means each byte is reused heavily, so a given compute rate needs little bandwidth. Low intensity means each byte is used once or twice, so the same compute rate needs a great deal.

Chapter 1.8 owns the memory wall, the latency-versus-bandwidth separation and bandwidth saturation, and this chapter does not restate any of it. What is added here is the conversion into stacks, which is HBM-specific:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   stacks required  =  (C / I)  /  307.2 GB/s

5. The Intensity Where HBM Runs Out

Put numbers in and the requirement becomes a stack count — and at low intensity the count is not achievable.

DERIVED, with ILLUSTRATIVE compute rates and intensities, recomputed:

Compute rateIntensity 10Intensity 50Intensity 200
100 Tops/s10,000 GB/s → 32.6 stacks2,000 GB/s → 6.5 stacks500 GB/s → 1.6 stacks
500 Tops/s50,000 GB/s → 162.8 stacks10,000 GB/s → 32.6 stacks2,500 GB/s → 8.1 stacks

Read the first column. A 100 Tops/s engine at intensity 10 needs 32.6 stacks — and at 500 Tops/s, 162.8. Those are not configurations; they are refutations.

Three conclusions, and the third is the one that shapes accelerator architecture.

Low-intensity workloads cannot be served by adding memory bandwidth. No plausible stack count reaches 50,000 GB/s, so the answer is not more HBM.

The lever is intensity, not bandwidth. Moving intensity from 10 to 200 — a factor of 20 — reduces the requirement from 32.6 stacks to 1.6, which is the difference between impossible and ordinary. Every decimal point of reuse is worth more than any amount of memory purchasing.

Which is why accelerators are built around on-chip reuse. Caches, local scratchpads, tiling and operand reuse all exist to raise I before the traffic reaches memory. HBM does not remove the need for them — it raises the compute rate at which they become mandatory, which is the opposite of how a bandwidth increase is usually described.

6. Sixteen Streams Per Stack, Times the Stacks

Chapter 4.8 §2's warning and 26.1 §4's doubling combine into a system-level requirement.

DERIVED, recomputed:

StacksStreamsBandwidthBandwidth per stream
116307.2 GB/s19.20 GB/s
232614.4 GB/s19.20
4641,228.8 GB/s19.20
81282,457.6 GB/s19.20
121923,686.4 GB/s19.20

Bandwidth per stream is constant at 19.2 GB/s, which is the arithmetic consequence of both quantities scaling with the stack count.

So the system's stream count grows exactly as fast as its bandwidth. An eight-stack system does not merely need eight times the traffic of a one-stack system — it needs that traffic distributed across 128 independent streams, each of which can absorb only 19.2 GB/s no matter how much the others are idle.

That is the requirement §3's twelve-stack configuration imposes: 192 streams. And it is where 4.8 §2's warning becomes an architectural constraint rather than a caution.

7. One Requester Cannot Fill Them

A single sequential requester, however fast, gets one stream's bandwidth.

The reason is 26.1 §4's granule. A stream's minimum access is 64 bytes, and a requester issuing one access at a time waits for each before issuing the next — so its bandwidth is bounded by 64 bytes / round-trip latency regardless of how many streams exist.

DERIVED under a stated model — a requester with Q outstanding 64-byte accesses distributed perfectly across streams, ILLUSTRATIVE latency:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   one requester, Q outstanding, 64 B each, latency T
      bandwidth  =  Q x 64 B / T

   streams usefully occupied  =  min(Q, total streams)

So the quantity that matters is outstanding requests, not request rate. A requester with Q = 1 uses one stream. With Q = 192 it can, in principle, use 192 — but only if its addresses land on 192 different streams, which is an address-mapping property rather than a concurrency one.

Two requirements, therefore, and they are independent:

Enough outstanding requests to cover the stream count. A 192-stream system needs at least 192 accesses in flight to touch every stream once. Chapter 1.8 §7 owns memory-level parallelism as the general idea; the HBM-specific figure is that the required parallelism scales with the stack count.

And a distribution that reaches them. Requests concentrated on one stack use 16 streams out of 192 — 26.1 §16 published the report of exactly that failure at the pseudo-channel level, where eight active streams in four channels looked identical to eight spread across eight.

8. The Requester Count Is an Architectural Property

Here is the consequence for how an accelerator is built, and it is the chapter's reason for existing.

An accelerator cannot choose its memory-level parallelism at runtime. The number of independent requests it can have outstanding is set by how many independent execution units, memory pipelines and outstanding-request slots it was designed with. That is a silicon decision, made before any workload runs.

So the stream count HBM offers is a constraint on the accelerator's architecture, not on its software. DERIVED from §6, under §7's stated model:

StacksStreams to fillMinimum outstanding 64 B accesses
11616
46464
8128128
12192192

A twelve-stack system requires an accelerator capable of 192 concurrent outstanding memory accesses just to touch every stream once — and considerably more to keep them busy, since each must be re-issued as it completes.

That is why HBM and massively parallel accelerators appear together. It is not that parallel engines happen to need bandwidth; it is that only a highly parallel requester can extract HBM's bandwidth at all. A serial engine attached to a twelve-stack system would leave 191 streams idle and see the bandwidth of one.

And the causality runs both ways. An accelerator designed for 192 outstanding accesses is over-provisioned against a single stack, which offers 16 streams. So the stack count and the requester parallelism must be designed together, and neither can be chosen independently — which is a tighter coupling between memory and compute than any other memory in this curriculum imposes.

9. What Getting It Backwards Costs

Four ways to mismatch the two purchases, all of which produce working systems.

MismatchWhat you getWhat it looks like
Tall stacks, bandwidth-bound workloadcapacity you cannot feed§3 — a third of the achievable bandwidth
Many stacks, capacity-bound workloadbandwidth you cannot useexpensive; integration risk for nothing
Enough stacks, too few outstanding requestsa fraction of the bandwidth§7 — nothing malfunctions
Enough requests, concentrated distributiona fraction of the bandwidth§7 — identical symptom, different cause

All four are silent. No error, no protocol violation, no timeout. Chapter 1.8 §15 is titled “Debugging — An Accelerator That Cannot Reach Its Peak” and owns that failure generally; what this chapter adds is that HBM has four distinct ways to produce it, and they need different fixes.

The third and fourth are the hardest to tell apart, because both present as low delivered bandwidth with the memory apparently idle. §13's monitor separates them by measuring outstanding requests and stream coverage independently — a count and a distribution — which is the only way the two causes are distinguishable from outside.

10. The System as a Structure

How an accelerator's HBM configuration and its own parallelism combine to determine delivered bandwidth, in four rows. The top row holds the four inputs: the number of stacks, the stack height in dies, the accelerator's compute rate, and the workload's arithmetic intensity in operations per byte. The second row holds four derived quantities: the total bandwidth which depends on the stack count alone, the total capacity which depends on stacks times height, the total stream count at sixteen per stack, and the bandwidth the workload requires which is the compute rate divided by the intensity. The third row holds the three things that decide what is actually delivered: the bandwidth-per-capacity ratio which is higher for shorter stacks, the number of outstanding requests the accelerator can sustain against the streams available, and which side is limiting — the memory or the compute. The bottom row is delivered bandwidth. Edges run from the stack count into bandwidth, capacity and streams, from the height into capacity, from the compute rate and the intensity into the required bandwidth, from bandwidth and capacity into the ratio, from streams into the outstanding-request comparison, from the required bandwidth and the available bandwidth into the limiter, and from the ratio, the request comparison and the limiter into delivered bandwidth.Stack count Kthe only bandwidth lever— §1Stack heightcapacity only — 26.2 §8Compute rate CILLUSTRATIVE alwaysIntensity Iops per byte — §4BandwidthK x 307.2 GB/s — §1CapacityK x height x 2 GB — §1StreamsK x 16 — §6Bandwidth neededC / I — §4Bandwidth per GBhigher for short stacks —§2Outstanding vsstreams192 needed at K=12 — §8Which side limitsmemory or compute — §5Delivered bandwidthnot the headline figure —4.8 §212

Two things the structure makes visible.

Stack count is the only input with three outgoing edges. It sets bandwidth, capacity and streams; height sets capacity alone. That is §1's asymmetry drawn — one purchase moves everything and the other moves one thing.

And Delivered bandwidth has three inputs, none of which is the headline figure. The ratio, the outstanding-request comparison and the limiter all feed it; Bandwidth reaches it only indirectly, through the limiter. Chapter 4.8 §2's warning is that structure: the rated figure is an input to a calculation, not the answer.

11. The System Budget

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// hbm_system_budget -- what K stacks of a given height offer, and the
// independence between the two purchases.
//
// CLASSIFICATION: educational, synthesisable.
//
// UNITS: bandwidth in GB/s scaled by 10 (so 307.2 -> 3072) to stay in
// integers. Capacity in GB. No reals, no division where a comparison
// will do.
//
// WHAT IT DOES NOT MODEL:
//   - the interposer, the stacks, TSVs, or any physical property
//     (26.2 and 26.3 own all of it)
//   - any latency. §7's model needs one and it is ILLUSTRATIVE there;
//     this block has none.
//   - any real accelerator, product or workload
//   - single-stack bandwidth arithmetic (4.8 §5's bandwidth_account
//     owns it; this aggregates rather than recomputes)
//
// THE INDEPENDENCE IS ENFORCED, NOT ASSUMED (§1, 26.2 §8): bandwidth
// and stream count depend on STACKS only, capacity on stacks AND
// height. P2 and P3 assert it because a budget that let height leak
// into bandwidth would reproduce the exact error §3's callout warns
// about.
// ---------------------------------------------------------------------
module hbm_system_budget #(
  parameter int STACKS        = 12,
  parameter int HEIGHT        = 4,       // CURRICULUM-DERIVED set (4.8 §2)
  parameter int GB_PER_DIE    = 2,       // 26.2 §8's inference
  parameter int BW_PER_STACK_X10 = 3072, // 307.2 GB/s, x10 (4.8 §2)
  parameter int STREAMS_PER_STACK = 16,  // 26.1 §4
  parameter int MAX_STACKS    = 16,
  parameter int STK_IDX_W     = $clog2(MAX_STACKS),
  // COUNTS need one more bit than the matching index.
  parameter int STKCNT_W      = $clog2(MAX_STACKS + 1),
  parameter int ACC_W         = 32
) (
  input  logic                  clk,
  input  logic                  rst_n,

  // ── Query a hypothetical configuration at runtime.
  input  logic                  q_valid,
  input  logic [STK_IDX_W:0]    q_stacks,
  input  logic [4:0]            q_height,

  output logic [ACC_W-1:0]      q_capacity_gb,
  output logic [ACC_W-1:0]      q_bandwidth_x10,
  output logic [ACC_W-1:0]      q_streams,
  output logic                  q_out_valid,

  // ── The elaborated configuration.
  output logic [ACC_W-1:0]      capacity_gb,
  output logic [ACC_W-1:0]      bandwidth_x10,
  output logic [ACC_W-1:0]      streams,
  output logic [STKCNT_W-1:0]   n_stacks,

  // §2's ratio, as a PAIR. bandwidth/capacity is rarely an integer,
  // so it is never divided here.
  output logic [ACC_W-1:0]      bw_per_cap_num,
  output logic [ACC_W-1:0]      bw_per_cap_den,
  // §6: constant at 19.2 GB/s, also as a pair.
  output logic [ACC_W-1:0]      bw_per_stream_num,
  output logic [ACC_W-1:0]      bw_per_stream_den,

  output logic                  err_height_unsupported,
  output logic                  err_stacks_out_of_range,
  output logic                  any_error
);

  localparam int CAP   = STACKS * HEIGHT * GB_PER_DIE;
  localparam int BW10  = STACKS * BW_PER_STACK_X10;
  localparam int STRM  = STACKS * STREAMS_PER_STACK;

  initial begin
    if (MAX_STACKS < 2)
      // A single-stack maximum makes the whole budget degenerate, and
      // $clog2(1) is 0 -- legal here only because the port is
      // [STK_IDX_W:0] rather than [STK_IDX_W-1:0]. Guarded anyway, for
      // consistency with 26.1 and 26.3, which require >= 2 for
      // anything they index.
      $fatal(1, "hbm_system_budget: MAX_STACKS must be at least 2");
    if (STACKS < 1 || STACKS > MAX_STACKS)
      $fatal(1, "hbm_system_budget: STACKS must be 1..%0d, got %0d", MAX_STACKS, STACKS);
    if (!(HEIGHT inside {2, 4, 8, 12}))
      // CURRICULUM-DERIVED height set (4.8 §2). Refused rather than
      // accommodated, because accommodating it would imply a
      // generality the source does not support.
      $fatal(1, "hbm_system_budget: HEIGHT must be 2, 4, 8 or 12, got %0d", HEIGHT);
    if (GB_PER_DIE < 1 || BW_PER_STACK_X10 < 1 || STREAMS_PER_STACK < 1)
      $fatal(1, "hbm_system_budget: per-stack figures must be positive");
    if (STKCNT_W <= STK_IDX_W && MAX_STACKS > 1)
      // COUNT vs INDEX: too narrow and n_stacks cannot represent a
      // fully populated system.
      $fatal(1, "hbm_system_budget: STKCNT_W too narrow to count MAX_STACKS");
    // §1's independence, checked at elaboration for every legal
    // height: bandwidth and streams must not depend on HEIGHT.
    if (BW10 != STACKS * BW_PER_STACK_X10)
      $fatal(1, "hbm_system_budget: bandwidth must depend on STACKS alone");
    if (STRM != STACKS * STREAMS_PER_STACK)
      $fatal(1, "hbm_system_budget: stream count must depend on STACKS alone");
    if (CAP > ((1 << (ACC_W - 1)) - 1) || BW10 > ((1 << (ACC_W - 1)) - 1))
      $fatal(1, "hbm_system_budget: ACC_W too narrow for this configuration");
  end

  logic             qv, e_hgt, e_stk;
  logic [ACC_W-1:0] qc, qb, qs;

  // Declared above the continuous assigns that read them.
  logic hgt_ok, stk_ok;
  assign hgt_ok = (q_height inside {5'd2, 5'd4, 5'd8, 5'd12});
  assign stk_ok = (q_stacks != '0) &&
                  ({{(32-(STK_IDX_W+1)){1'b0}}, q_stacks} <= ACC_W'(MAX_STACKS));

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      qv <= 1'b0; qc <= '0; qb <= '0; qs <= '0;
      e_hgt <= 1'b0; e_stk <= 1'b0;
    end else begin
      qv <= 1'b0; e_hgt <= 1'b0; e_stk <= 1'b0;
      if (q_valid) begin
        if (!stk_ok)      e_stk <= 1'b1;
        else if (!hgt_ok) e_hgt <= 1'b1;
        else begin
          qc <= ACC_W'(q_stacks) * ACC_W'(q_height) * ACC_W'(GB_PER_DIE);
          qb <= ACC_W'(q_stacks) * ACC_W'(BW_PER_STACK_X10);
          qs <= ACC_W'(q_stacks) * ACC_W'(STREAMS_PER_STACK);
          qv <= 1'b1;
        end
      end
    end
  end

  assign capacity_gb   = ACC_W'(CAP);
  assign bandwidth_x10 = ACC_W'(BW10);
  assign streams       = ACC_W'(STRM);
  assign n_stacks      = STKCNT_W'(STACKS);
  assign bw_per_cap_num    = ACC_W'(BW10);
  assign bw_per_cap_den    = ACC_W'(CAP);
  assign bw_per_stream_num = ACC_W'(BW10);
  assign bw_per_stream_den = ACC_W'(STRM);
  assign q_capacity_gb   = qc;
  assign q_bandwidth_x10 = qb;
  assign q_streams       = qs;
  assign q_out_valid     = qv;
  assign err_height_unsupported  = e_hgt;
  assign err_stacks_out_of_range = e_stk;
  assign any_error = e_hgt | e_stk;
endmodule

The bandwidth-per-capacity ratio is published as a pair and never divided. §2's figures — 76.80 and 12.80 — are not integers, and the caller is better placed to choose a precision. And publishing bandwidth over capacity rather than a scalar makes §3's comparison directly readable: two configurations with the same denominator and different numerators are the whole finding.

The elaboration checks enforce the independence rather than trusting it. A budget in which height leaked into the bandwidth calculation would reproduce exactly the error §3's callout describes — and it would do so silently, since the result would still be a plausible number.

12. The Stream Supply Monitor

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// stream_supply_monitor -- can the requester side actually fill the
// streams the configuration offers?
//
// CLASSIFICATION: educational, synthesisable.
//
// COLLISION NOTE: 26.1 §12's pc_stream_monitor observes which
// pseudo-channels within ONE stack are active. This block observes
// the REQUESTER side across a whole system: how many accesses are
// outstanding, and how many distinct streams they reach. Different
// side of the same interface, different question.
//
// WHAT IT DOES NOT MODEL:
//   - latency, or bandwidth in bytes per second. 1.8 owns the
//     latency model and 4.8 §5 the bandwidth arithmetic; neither is
//     rebuilt.
//   - the memory, the stacks, or any physical property
//   - WHY a stream is unreached. §9: concentration and insufficient
//     parallelism produce the same symptom, and this block separates
//     the two measurements so a diagnosis is possible. It does not
//     make the diagnosis.
// ---------------------------------------------------------------------
module stream_supply_monitor #(
  parameter int STREAMS   = 192,         // §6: K x 16
  parameter int STRM_IDX_W = $clog2(STREAMS),
  // A COUNT up to STREAMS needs clog2(STREAMS+1).
  parameter int STRMCNT_W = $clog2(STREAMS + 1),
  parameter int OUTS_W    = 12,
  parameter int CNT_W     = 28
) (
  input  logic                    clk,
  input  logic                    rst_n,

  // ── Requester side: an access issued to a stream, and a completion.
  input  logic                    issue_valid,
  input  logic [STRM_IDX_W-1:0]   issue_stream,
  input  logic                    complete_valid,

  // ── MEASUREMENT ONE: are there enough accesses in flight?
  output logic [OUTS_W-1:0]       outstanding,
  output logic [OUTS_W-1:0]       peak_outstanding,
  output logic                    enough_outstanding,

  // ── MEASUREMENT TWO: do they reach the streams? §9 -- reported
  //    separately because the two causes share a symptom.
  output logic [STRMCNT_W-1:0]    streams_reached,
  output logic                    full_coverage,
  output logic [STRMCNT_W-1:0]    streams_never_reached,

  // The two diagnoses §9 says must be distinguishable.
  output logic                    starved_not_enough_requests,
  output logic                    starved_concentrated,

  output logic [CNT_W-1:0]        cnt_issued,
  output logic [CNT_W-1:0]        cnt_completed,

  output logic                    err_stream_out_of_range,
  output logic                    err_complete_without_outstanding,
  output logic                    any_error
);

  initial begin
    if (STREAMS < 2)
      $fatal(1, "stream_supply_monitor: need at least 2 streams");
    if (STRM_IDX_W != $clog2(STREAMS))
      $fatal(1, "stream_supply_monitor: STRM_IDX_W must address STREAMS");
    if (STRMCNT_W <= STRM_IDX_W)
      // COUNT vs INDEX: too narrow and full_coverage is unreachable,
      // which reads as permanent under-coverage -- the pessimistic
      // direction, and still wrong.
      $fatal(1, "stream_supply_monitor: STRMCNT_W too narrow to count STREAMS");
    if ((1 << OUTS_W) <= STREAMS)
      // §8: filling every stream needs at least STREAMS accesses in
      // flight, so the counter must be able to represent that.
      $fatal(1, "stream_supply_monitor: OUTS_W cannot represent STREAMS outstanding");
  end

  logic [STREAMS-1:0]  touched;
  logic [OUTS_W-1:0]   outs, pk;
  logic [CNT_W-1:0]    n_iss, n_cmp;
  logic                e_rng, e_cmp;

  // Declared above the continuous assigns that read them.
  logic strm_ok;
  assign strm_ok = ({{(32-STRM_IDX_W){1'b0}}, issue_stream} < 32'(STREAMS));

  logic [STRMCNT_W-1:0] n_touch;
  always_comb begin
    n_touch = '0;
    for (int s = 0; s < STREAMS; s++)
      if (touched[s]) n_touch = n_touch + STRMCNT_W'(1);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      touched <= '0; outs <= '0; pk <= '0;
      n_iss <= '0; n_cmp <= '0; e_rng <= 1'b0; e_cmp <= 1'b0;
    end else begin
      e_rng <= 1'b0; e_cmp <= 1'b0;

      if (issue_valid) begin
        if (!strm_ok) begin
          // Out of range: reported, and NOT counted. A bad stream
          // index folded into coverage would inflate it.
          e_rng <= 1'b1;
        end else begin
          touched[issue_stream] <= 1'b1;
          if (outs != {OUTS_W{1'b1}}) begin
            // Compare the INCREMENTED value: outs's write lands on
            // this edge, so testing outs here would under-report the
            // peak by one. The stale-register hazard Module 21 §10,
            // 24.3 §12, 25.2 §15 and 26.1 §11 all name.
            if ((outs + OUTS_W'(1)) > pk) pk <= outs + OUTS_W'(1);
            outs <= outs + OUTS_W'(1);
          end
          if (n_iss != {CNT_W{1'b1}}) n_iss <= n_iss + CNT_W'(1);
        end
      end

      if (complete_valid) begin
        if (outs == '0) begin
          // A completion with nothing outstanding is a bookkeeping
          // error somewhere, not a negative occupancy.
          e_cmp <= 1'b1;
        end else begin
          outs <= outs - OUTS_W'(1);
          if (n_cmp != {CNT_W{1'b1}}) n_cmp <= n_cmp + CNT_W'(1);
        end
      end
    end
  end

  assign outstanding      = outs;
  assign peak_outstanding = pk;
  // §8: at least one access per stream is needed to touch them all.
  assign enough_outstanding = (pk >= OUTS_W'(STREAMS));
  assign streams_reached  = n_touch;
  assign full_coverage    = (n_touch == STRMCNT_W'(STREAMS));
  assign streams_never_reached = STRMCNT_W'(STREAMS) - n_touch;
  // §9's two diagnoses, separated. Not enough requests: the peak
  // never reached the stream count. Concentrated: it did, and
  // coverage is still short.
  assign starved_not_enough_requests = !full_coverage && !enough_outstanding;
  assign starved_concentrated        = !full_coverage &&  enough_outstanding;
  assign cnt_issued    = n_iss;
  assign cnt_completed = n_cmp;
  assign err_stream_out_of_range          = e_rng;
  assign err_complete_without_outstanding = e_cmp;
  assign any_error = e_rng | e_cmp;
endmodule

The two starvation outputs are mutually exclusive by construction and that is the block's whole contribution. §9 establishes that insufficient parallelism and concentrated distribution produce an identical symptom — low delivered bandwidth with the memory apparently idle. Separating them requires two independent measurements, a peak count and a coverage count, and neither alone distinguishes the causes.

And the peak comparison folds the increment in combinationally. This is the fifth appearance of that hazard in the curriculum — Module 21 §10, 24.3 §12, 25.2 §15, 26.1 §11 and here. A peak under-reported by one would make enough_outstanding false at exactly the point it becomes true, which flips the diagnosis from concentration to insufficient parallelism — the wrong fix.

13. The Intensity Balance Checker

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// intensity_balance_checker -- given a compute rate, an arithmetic
// intensity and an available bandwidth, which side limits?
//
// CLASSIFICATION: educational, synthesisable.
//
// UNITS ARE SCALED INTEGERS. Compute in Gops/s; intensity in ops per
// byte; bandwidth in GB/s x10 (matching hbm_system_budget). No reals
// and NO DIVISION -- the comparison is done by cross-multiplication.
//
// WHAT IT DOES NOT MODEL:
//   - any real compute rate, workload intensity or utilisation. §5's
//     callout: the model assumes peak compute is achievable,
//     intensity is a single number, and bandwidth is fully
//     deliverable. All three are optimistic, so every requirement it
//     reports is a LOWER BOUND.
//   - latency (1.8 owns it), or the memory itself
//   - what to DO about an imbalance. Raising intensity is a compute
//     architecture decision and adding stacks is a purchasing one;
//     this block says which side is short.
// ---------------------------------------------------------------------
module intensity_balance_checker #(
  parameter int BW_SCALE = 10,           // bandwidth is GB/s x10
  parameter int W        = 40
) (
  input  logic              clk,
  input  logic              rst_n,

  input  logic              eval,
  input  logic [W-1:0]      compute_gops,        // ILLUSTRATIVE
  input  logic [W-1:0]      intensity_ops_byte,  // ILLUSTRATIVE
  input  logic [W-1:0]      bandwidth_x10,       // from the budget

  // ── Verdict. Three outcomes, because "balanced" is a real answer.
  output logic              memory_limited,
  output logic              compute_limited,
  output logic              balanced,

  // Required bandwidth as a PAIR: compute over intensity, never
  // divided. A caller states its own denominator.
  output logic [W-1:0]      required_num,
  output logic [W-1:0]      required_den,

  // How many stacks' worth short or spare, as a comparison rather
  // than a quotient.
  output logic [W-1:0]      have_scaled,
  output logic [W-1:0]      need_scaled,

  output logic              err_zero_intensity,
  output logic              err_zero_bandwidth,
  output logic              any_error
);

  initial begin
    if (BW_SCALE < 1)
      $fatal(1, "intensity_balance_checker: BW_SCALE must be positive");
    if (W < 32)
      // The cross-multiplication multiplies two supplied values;
      // too narrow and it wraps, which silently inverts the verdict.
      $fatal(1, "intensity_balance_checker: W must be at least 32 for the cross product");
  end

  logic v_mem, v_cmp, v_bal, e_int, e_bw;
  logic [W-1:0] have_q, need_q;

  // ── No division. required = compute/intensity GB/s; available =
  //    bandwidth_x10 / BW_SCALE GB/s. Compare by cross-multiplying:
  //      compute * BW_SCALE   vs   bandwidth_x10 * intensity
  //    Declared above the continuous assigns that read them.
  logic [W-1:0] lhs, rhs;
  assign lhs = compute_gops  * W'(BW_SCALE);
  assign rhs = bandwidth_x10 * intensity_ops_byte;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      v_mem <= 1'b0; v_cmp <= 1'b0; v_bal <= 1'b0;
      e_int <= 1'b0; e_bw <= 1'b0; have_q <= '0; need_q <= '0;
    end else begin
      e_int <= 1'b0; e_bw <= 1'b0;
      if (eval) begin
        if (intensity_ops_byte == '0) begin
          // Zero intensity means every operation touches memory
          // without reuse, so the requirement is unbounded. Reported
          // rather than computed as a very large number.
          e_int <= 1'b1;
          v_mem <= 1'b0; v_cmp <= 1'b0; v_bal <= 1'b0;
        end else if (bandwidth_x10 == '0) begin
          e_bw <= 1'b1;
          v_mem <= 1'b0; v_cmp <= 1'b0; v_bal <= 1'b0;
        end else begin
          have_q <= rhs;
          need_q <= lhs;
          // lhs > rhs : the compute rate needs more bandwidth than
          // exists -> memory limited.
          v_mem <= (lhs >  rhs);
          v_cmp <= (lhs <  rhs);
          v_bal <= (lhs == rhs);
        end
      end
    end
  end

  assign memory_limited  = v_mem;
  assign compute_limited = v_cmp;
  assign balanced        = v_bal;
  assign required_num    = compute_gops;
  assign required_den    = intensity_ops_byte;
  assign have_scaled     = have_q;
  assign need_scaled     = need_q;
  assign err_zero_intensity = e_int;
  assign err_zero_bandwidth = e_bw;
  assign any_error = e_int | e_bw;
endmodule

Zero intensity is reported rather than computed. Every operation touching memory with no reuse makes the requirement unbounded, and returning a very large number would invite a caller to treat it as a finite target. §5 establishes that low intensity is not a bandwidth problem to be solved by purchasing — and an unbounded requirement is the limit case of that argument.

And the verdict is three-way. “Balanced” is a real outcome and collapsing it into one of the other two would make the block report a limit where none exists.

14. What the Assertions Prove

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Bind unit note: P1-P5 reference hbm_system_budget's internals,
// P6-P10 stream_supply_monitor's, P11-P14
// intensity_balance_checker's.

// P1 -- §1's identity: capacity is stacks x height x per-die.
property p_capacity_is_the_product;
  @(posedge clk) disable iff (!rst_n)
    (capacity_gb == ACC_W'(STACKS * HEIGHT * GB_PER_DIE));
endproperty
assert property (p_capacity_is_the_product);

// P2 -- FORBIDDEN. Bandwidth depends on the stack count ALONE. §3's
// callout: a budget in which height leaked into bandwidth would
// reproduce the exact error the chapter exists to prevent, silently.
property p_bandwidth_ignores_height;
  @(posedge clk) disable iff (!rst_n)
    (bandwidth_x10 == ACC_W'(STACKS * BW_PER_STACK_X10));
endproperty
assert property (p_bandwidth_ignores_height);

// P3 -- FORBIDDEN. The stream count likewise depends on stacks alone.
property p_streams_ignore_height;
  @(posedge clk) disable iff (!rst_n)
    (streams == ACC_W'(STACKS * STREAMS_PER_STACK));
endproperty
assert property (p_streams_ignore_height);

// P4 -- §6: bandwidth per stream is constant, because both scale
// with the stack count. Asserted as the cross product so no division
// is needed.
property p_bandwidth_per_stream_is_constant;
  @(posedge clk) disable iff (!rst_n)
    ((bw_per_stream_num * ACC_W'(STREAMS_PER_STACK)) ==
     (bw_per_stream_den * ACC_W'(BW_PER_STACK_X10)));
endproperty
assert property (p_bandwidth_per_stream_is_constant);

// P5 -- FORBIDDEN. An unsupported height never produces a result.
property p_bad_height_never_computes;
  @(posedge clk) disable iff (!rst_n)
    (q_valid && !(q_height inside {5'd2, 5'd4, 5'd8, 5'd12}) && (q_stacks != '0))
      |=> (err_height_unsupported && !q_out_valid);
endproperty
assert property (p_bad_height_never_computes);

// P6 -- outstanding never goes negative, and a completion with
// nothing in flight is reported rather than wrapping.
property p_outstanding_never_underflows;
  @(posedge clk) disable iff (!rst_n)
    (complete_valid && (outstanding == '0)) |=> err_complete_without_outstanding;
endproperty
assert property (p_outstanding_never_underflows);

// P7 -- the peak is monotone and never below the current occupancy.
// §12's hazard: a peak compared pre-increment is one short, which
// flips the §9 diagnosis at exactly the boundary.
property p_peak_is_monotone_and_covering;
  @(posedge clk) disable iff (!rst_n)
    ((peak_outstanding >= outstanding) &&
     (peak_outstanding >= $past(peak_outstanding, 1)));
endproperty
assert property (p_peak_is_monotone_and_covering);

// P8 -- FORBIDDEN. An out-of-range stream index never counts toward
// coverage. A bad index folded in would inflate it.
property p_bad_stream_never_counted;
  @(posedge clk) disable iff (!rst_n)
    (err_stream_out_of_range |-> (streams_reached == $past(streams_reached, 1)));
endproperty
assert property (p_bad_stream_never_counted);

// P9 -- coverage and never-reached partition the streams exactly.
property p_coverage_partitions_streams;
  @(posedge clk) disable iff (!rst_n)
    ((streams_reached + streams_never_reached) == STRMCNT_W'(STREAMS));
endproperty
assert property (p_coverage_partitions_streams);

// P10 -- §9's two diagnoses are mutually exclusive, and neither holds
// under full coverage. The property that makes them distinguishable.
property p_starvation_causes_are_exclusive;
  @(posedge clk) disable iff (!rst_n)
    (!(starved_not_enough_requests && starved_concentrated) &&
     (full_coverage |-> (!starved_not_enough_requests && !starved_concentrated)));
endproperty
assert property (p_starvation_causes_are_exclusive);

// P11 -- the three balance verdicts are mutually exclusive.
property p_balance_verdict_exclusive;
  @(posedge clk) disable iff (!rst_n)
    $onehot0({memory_limited, compute_limited, balanced});
endproperty
assert property (p_balance_verdict_exclusive);

// P12 -- FORBIDDEN. Zero intensity never produces a verdict. §5: the
// requirement is unbounded, and a finite number would invite a
// caller to treat it as a target.
property p_zero_intensity_no_verdict;
  @(posedge clk) disable iff (!rst_n)
    (eval && (intensity_ops_byte == '0))
      |=> (err_zero_intensity && !memory_limited && !compute_limited && !balanced);
endproperty
assert property (p_zero_intensity_no_verdict);

// P13 -- the verdict matches the cross product, so no division was
// smuggled in anywhere.
property p_verdict_matches_cross_product;
  @(posedge clk) disable iff (!rst_n)
    (memory_limited |-> (need_scaled > have_scaled));
endproperty
assert property (p_verdict_matches_cross_product);

// P14 -- the required-bandwidth pair is the inputs verbatim, never a
// quotient. §5: the caller states its own denominator.
property p_required_is_published_as_a_pair;
  @(posedge clk) disable iff (!rst_n)
    ((required_num == compute_gops) && (required_den == intensity_ops_byte));
endproperty
assert property (p_required_is_published_as_a_pair);

// ── Cover.
// Both of §3's configurations, which is the chapter's central result:
// the same capacity at different bandwidths.
cover property (@(posedge clk) disable iff (!rst_n)
                  q_out_valid && (q_capacity_gb == ACC_W'(96)) &&
                  (q_bandwidth_x10 == ACC_W'(12288)));
cover property (@(posedge clk) disable iff (!rst_n)
                  q_out_valid && (q_capacity_gb == ACC_W'(96)) &&
                  (q_bandwidth_x10 == ACC_W'(36864)));
// Every verified height queried, and each error attempted.
cover property (@(posedge clk) disable iff (!rst_n)
                  q_out_valid && (q_height == 5'd2));
cover property (@(posedge clk) disable iff (!rst_n)
                  q_out_valid && (q_height == 5'd12));
cover property (@(posedge clk) disable iff (!rst_n) err_height_unsupported);
cover property (@(posedge clk) disable iff (!rst_n) err_stacks_out_of_range);
// Full stream coverage reached, and BOTH starvation causes -- §9's
// whole point is that these are different, so a suite must reach each.
cover property (@(posedge clk) disable iff (!rst_n) full_coverage);
cover property (@(posedge clk) disable iff (!rst_n) starved_not_enough_requests);
cover property (@(posedge clk) disable iff (!rst_n) starved_concentrated);
cover property (@(posedge clk) disable iff (!rst_n) err_complete_without_outstanding);
cover property (@(posedge clk) disable iff (!rst_n) err_stream_out_of_range);
// Outstanding reaching exactly the stream count -- the boundary at
// which §12's hazard flips the diagnosis.
cover property (@(posedge clk) disable iff (!rst_n)
                  peak_outstanding == OUTS_W'(STREAMS));
// All three balance verdicts, and the unbounded case.
cover property (@(posedge clk) disable iff (!rst_n) memory_limited);
cover property (@(posedge clk) disable iff (!rst_n) compute_limited);
cover property (@(posedge clk) disable iff (!rst_n) balanced);
cover property (@(posedge clk) disable iff (!rst_n) err_zero_intensity);

15. Corner Cases

CaseBehaviourWhy
Height changed 2 → 12, stacks fixedCapacity ×6; bandwidth unchanged§1, P2 — the independence
Stacks changed 4 → 12, height fixedBandwidth ×3, capacity ×3, streams ×3§1 — one purchase moves everything
4 × 12-high vs 12 × 4-highSame 96 GB; 3× the bandwidth§3 — the chapter's central result
Height of 6 requestederr_height_unsupported; no resultOutside 4.8 §2's verified set
Zero stacks requestederr_stacks_out_of_rangeA system with no memory is not a configuration
Bandwidth computed from heightP2 fires§3's callout — the silent, plausible-looking error
Peak outstanding exactly STREAMSenough_outstanding asserts§8 — the minimum to touch every stream once
Peak compared pre-incrementOne short; diagnosis flips at the boundary§12's hazard — concentration reported as insufficiency
All requests to one stackstarved_concentrated if Q is large enough§9 — coverage short, parallelism fine
Few requests, well spreadstarved_not_enough_requests§9 — the other cause, same symptom
Full coverage reachedNeither starvation flagP10
Completion with nothing outstandingReported; no underflowP6
Out-of-range stream indexReported; coverage unchangedP8 — a bad index would inflate coverage
Intensity of zeroerr_zero_intensity; no verdict§5 — unbounded, not a large number
Bandwidth of zeroerr_zero_bandwidth; no verdictNothing to compare against
Compute exactly matching bandwidthbalancedA real third outcome, not a rounding artefact
W below 32$fatal at elaborationThe cross product would wrap and invert the verdict

Rows nine and ten are the pair this chapter exists to distinguish. Both produce low delivered bandwidth with the memory apparently idle, and they need opposite fixes — more outstanding requests in one case, a different address mapping in the other.

16. DV

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Independent configuration reference. Builds the stack list
// explicitly and sums, rather than multiplying, so agreement with
// the DUT is evidence rather than a restatement of the same product.
// SIMULATION-ONLY.
class hbm_config_ref;
  int stacks, height, gb_per_die, bw_per_stack_x10, streams_per_stack;

  function void build(int k, int h);
    stacks = k; height = h;
    gb_per_die = 2; bw_per_stack_x10 = 3072; streams_per_stack = 16;
  endfunction

  // Sum per stack rather than multiply.
  function int capacity_gb();
    int c = 0;
    for (int s = 0; s < stacks; s++) c += height * gb_per_die;
    return c;
  endfunction

  function int bandwidth_x10();
    int b = 0;
    for (int s = 0; s < stacks; s++) b += bw_per_stack_x10;
    return b;   // deliberately does NOT reference height
  endfunction

  function int streams();
    int n = 0;
    for (int s = 0; s < stacks; s++) n += streams_per_stack;
    return n;   // likewise
  endfunction

  // §4's requirement, as a pair.
  function void required(int compute_gops, int intensity,
                         output int num, output int den);
    num = compute_gops; den = intensity;
  endfunction

  // The cross-multiplied comparison, computed independently.
  function int limited(int compute_gops, int intensity, int bw_x10);
    if (intensity == 0) return -1;                    // unbounded
    if (compute_gops * 10 >  bw_x10 * intensity) return 1;   // memory
    if (compute_gops * 10 <  bw_x10 * intensity) return 2;   // compute
    return 0;                                         // balanced
  endfunction
endclass
CheckWhat it establishes
Reference capacity_gb() against the DUT at every height and 1–16 stacksP1 by summing rather than multiplying
Reference bandwidth_x10() — which never reads heightP2 against a representation that structurally cannot leak height
Reference streams() likewiseP3
Sweep height with stacks fixed; assert bandwidth constant§1's independence, driven
Query 4 × 12-high and 12 × 4-high§3's two configurations; both covers
Request height 6, then 0 stacksP5 and both error covers
Confirm bandwidth-per-stream constant across 1–16 stacksP4's cross product
Issue STREAMS accesses to distinct streamsfull_coverage; enough_outstanding
Issue STREAMS accesses all to one stack's 16 streamsstarved_concentrated
Issue 4 accesses, well spreadstarved_not_enough_requests
Confirm the two starvation flags never co-assertP10
Replace the peak compare with outs > pkPeak one short; the §15 row-eight diagnosis flip
Complete with nothing outstandingP6
Issue to an out-of-range stream between valid onesP8; coverage unchanged
Reference limited() across a sweep of compute, intensity, bandwidthP11, P13 against an independent comparison
Evaluate with intensity 0P12; no verdict

The ninth, tenth and twelfth checks together produce the report worth publishing, because all three are cases where nothing is broken:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  THREE WAYS TO SEE THE SAME SYMPTOM
    A 12-stack system: 192 streams, 3686.4 GB/s rated.

  (A) 4 accesses outstanding, spread across 4 streams
        peak_outstanding              4
        streams_reached               4
        enough_outstanding            0
        starved_not_enough_requests   1   <-- too little parallelism
        starved_concentrated          0
        fix: more outstanding requests. An address remap changes
             nothing, because there is nothing to spread.

  (B) 192 accesses outstanding, all within one stack
        peak_outstanding            192
        streams_reached              16
        enough_outstanding            1
        starved_not_enough_requests   0
        starved_concentrated          1   <-- distribution, not count
        fix: address mapping. Adding requests changes nothing,
             because 176 streams are unreachable by construction.

  (C) the same as (B), but with the peak compared PRE-INCREMENT
        peak_outstanding            191   <-- one short
        enough_outstanding            0   <-- flips FALSE at 192
        starved_not_enough_requests   1   <-- WRONG DIAGNOSIS
        starved_concentrated          0
        P7 peak_is_monotone_and_covering   FAIL

    (A) and (B) both present externally as "we are getting a
    fraction of the rated bandwidth and the memory looks idle" --
    1.8 §15's symptom exactly. They need OPPOSITE fixes.

    and (C) is the one that matters for this chapter: a peak
    under-reported by exactly one turns (B) into (A). The flip
    happens precisely at peak == STREAMS, which is the boundary the
    whole diagnosis hinges on -- so the off-by-one is not a small
    error, it is a reliable misdiagnosis at the only interesting
    operating point.

    the fix : both measurements published separately (§12), the
    peak folded combinationally, and both starvation covers
    required. A single "underutilised" flag cannot distinguish any
    of the three.

17. Debugging

SymptomLikely causeHow to confirm
Capacity as specified, bandwidth a third of expectationTall stacks bought for a bandwidth problem — §3Count the stacks, not the gigabytes
Adding dies did not help bandwidthExpected — §1, P2bandwidth_x10 is stacks-only
Adding stacks did not help a capacity problemIt helps, expensively — §9 row twoCapacity rose; the money went on unused bandwidth
A fraction of rated bandwidth, memory idleOne of four causes — §9Start with starved_*; 1.8 §15 owns the general case
Low bandwidth, few requests in flightInsufficient parallelismstarved_not_enough_requests; an address remap will not help
Low bandwidth, many requests in flightConcentrated distributionstarved_concentrated; more requests will not help
Diagnosis flips between the two abovePeak compared pre-increment — §12peak_outstanding exactly one below the true peak; P7
Coverage looks better than the traffic suggestsOut-of-range indices countederr_stream_out_of_range; P8
Bandwidth requirement reported as a huge finite numberZero intensity computed instead of reportederr_zero_intensity; §5 — it is unbounded
Balance verdict inverts on large inputsCross product wrappedThe W elaboration guard
Requirement met in the model, missed in practiceThe model is a lower bound — §5's calloutUtilisation, phase behaviour and deliverability are all omitted

Row one is the entry this chapter exists to add. “We have 96 GB of HBM” does not determine bandwidth — the stack count does, and the same capacity spans a threefold range.

Row eleven is the honest caveat. A configuration that satisfies §13's checker can still miss in practice, because the model assumes peak compute, a single intensity and fully deliverable bandwidth — and all three are optimistic.

18. Misconceptions

“More HBM means more bandwidth.” §1. More stacks means more bandwidth. More dies means more capacity and nothing else.

“The biggest stack is the best stack.” §2. The tallest stack is the least bandwidth-dense — a 2-high stack delivers six times the bandwidth per gigabyte of a 12-high one.

“96 GB of HBM describes a memory system.” §3. It is underspecified: 4 × 12-high and 12 × 4-high are both 96 GB, and the second delivers three times the bandwidth.

“Buying capacity and buying bandwidth are the same purchase.” §1, §3. They are separate, and confusing them produces a working system with a third of the achievable bandwidth.

“A bandwidth problem can always be solved with more memory.” §5. At intensity 10 a 100 Tops/s engine needs 32.6 stacks and at 500 Tops/s, 162.8. Those are refutations, not configurations.

“HBM removes the need for on-chip reuse.” §5. It raises the compute rate at which reuse becomes mandatory. Moving intensity from 10 to 200 does what no amount of purchasing can.

“Sixteen streams per stack is plenty.” §6. A twelve-stack system has 192, each absorbing only 19.2 GB/s no matter how idle the others are.

“A fast requester can use HBM's bandwidth.” §7. A requester's bandwidth is bounded by its outstanding accesses, not its rate. With one in flight it uses one stream.

“Memory-level parallelism is a software matter.” §8. The number of outstanding accesses is set by the silicon — execution units, memory pipelines, request slots. It is decided before any workload runs.

“You can attach HBM to any engine and benefit.” §8. Only a highly parallel requester can extract it. A serial engine on a twelve-stack system leaves 191 streams idle.

“Low bandwidth with idle memory has one cause.” §9, §16. Four, of which two are indistinguishable without measuring parallelism and coverage separately — and they need opposite fixes.

“A satisfied balance check means the system will perform.” §5's callout. The model assumes peak compute, one intensity and full deliverability. Every requirement it reports is a lower bound.

19. Interview Reasoning

How do you add bandwidth to an HBM system? Add stacks. Height adds capacity and leaves bandwidth and stream count untouched, because the interface is 1024 bits wide at every height.

Which stack height gives the most bandwidth per gigabyte? The shortest. A 2-high stack is exactly six times as bandwidth-dense as a 12-high one, since capacity scales with height and bandwidth does not.

Two systems both have 96 GB of HBM. Can they differ in bandwidth? By a factor of three. Four 12-high stacks give 1,228.8 GB/s; twelve 4-high stacks give 3,686.4 GB/s and 192 streams instead of 64.

So what is wrong with quoting capacity? It does not determine bandwidth. The stack count does — and capacity is the number that appears in a product summary, which is why the error survives review.

How do you decide how much bandwidth a workload needs? Compute rate divided by arithmetic intensity, in operations per byte. Then divide by 307.2 GB/s to get a stack count.

What happens at low intensity? The requirement becomes unachievable. At 100 Tops/s and intensity 10 you need 32.6 stacks; at 500 Tops/s, 162.8. The lever is intensity, not purchasing.

How many streams does a twelve-stack system have, and why does it matter? 192, each able to absorb about 19.2 GB/s. The rated bandwidth requires traffic spread across all of them, and one stream's worth of traffic gets one stream's worth of bandwidth.

What does that require of the accelerator? At least 192 concurrent outstanding accesses simply to touch every stream once — and that count is fixed by the silicon's execution units and request slots, not chosen at runtime.

Why do HBM and massively parallel engines appear together? Because only a highly parallel requester can extract the bandwidth. A serial engine on a twelve-stack system sees one stream's bandwidth and leaves 191 idle.

You are getting a fraction of rated bandwidth and the memory looks idle. What do you measure? Two things independently: how many accesses are outstanding, and how many distinct streams they reach. Too few requests and too concentrated a distribution look identical and need opposite fixes.

20. Exercises

  1. Derive the general expression for bandwidth per gigabyte as a function of stack height, and confirm it reproduces §2's four rows. Then state the height that maximises it and why that height is rarely chosen.

  2. Find every configuration of stacks and verified heights delivering exactly 48 GB, and rank them by bandwidth. Identify the range and say what a specification quoting only “48 GB” fails to convey.

  3. Using §4's model, derive the arithmetic intensity at which a 200 Tops/s engine is exactly served by eight stacks. Then state which of §5's three omitted factors would most change the answer.

  4. §5 shows intensity is the dominant lever. Derive the intensity multiplier equivalent to doubling the stack count, and explain why the two are not interchangeable in cost.

  5. §8 requires at least STREAMS outstanding accesses to touch every stream once. Derive the number needed to keep every stream continuously busy given a latency T and a per-stream service time, and say why the first figure is the wrong target.

  6. Construct a request stream that produces starved_concentrated on a four-stack system and full_coverage on a one-stack system, using the same addresses. Explain what that implies about validating an address map against a single configuration.

  7. Implement §12's hazard by comparing outs rather than the incremented value. Prove the peak is understated by exactly one, and show that the diagnosis flips only at peak == STREAMS.

  8. §13 refuses to compute a requirement at zero intensity. Argue whether a very large finite number would be more useful in practice, then state what §5's argument implies about the answer.

21. Where the Module Ends

HBM is now a system rather than a component, and four chapters have built it.

Chapter 26.1 quantified the wall that 4.8 argued — connections per unit capability falling as 4/L, which selects a face-connected solution before any engineering — and opened the hierarchy 4.8 deferred: eight channels of 128 bits, then sixteen pseudo-channels of 64, with the split preserving the 64-byte granule exactly. 26.2 turned a surface into a count, found the face-versus-edge advantage has the closed form L/4p, and discovered that a 1024-bit bump field uses about 2% of a die's face — so the geometric wall is answered with fifty times the margin. 26.3 found that abundance is not the problem: at a per-via reliability of one in ten thousand a 12-high stack works 2.5% of the time, and one spare per 64 removes 99.7% of the risk. And this chapter separated the two purchases — stacks for bandwidth, dies for capacity — and found the same 96 GB spans a threefold bandwidth range.

Four results from this chapter carry beyond it.

Bandwidth depends on the stack count and nothing else. Height, total capacity and the figure on a specification sheet are all silent about it, which makes “96 GB of HBM” an underspecified description.

Shorter stacks are bandwidth-dense by exactly the ratio of the heights — so the instinct to buy the largest part is precisely wrong for a bandwidth-bound system.

Arithmetic intensity dominates purchasing. A factor of 20 in reuse is worth more than any achievable stack count, which is why HBM raises the compute rate at which on-chip reuse becomes mandatory rather than removing the need for it.

And the accelerator's own parallelism is a precondition. A twelve-stack system needs 192 concurrent outstanding accesses to touch every stream once, and that number is fixed in silicon — so the stack count and the requester design must be chosen together.

Four things this module verified and could not finish, and they are larger than any previous module's open items.

No primary HBM source was obtainable. JEDEC refused every retrieval with HTTP 403; two freely published vendor briefs were retrieved and both yielded no usable text. So this module's figures are predominantly CURRICULUM-DERIVED from 4.8 §2's published set, or ILLUSTRATIVE — and every chapter says so in its own provenance callout rather than only here.

The two-way pseudo-channel split is REPORTED, not verified. Chapter 26.1 §4 flags it, and §5's exact granularity identity is arithmetic over a structure this module did not confirm.

The base die's contents remain undelivered. Chapter 4.8's scope callout assigns “the stack's logic layer” here, and 26.2 §9 could only state what its position and shared status imply.

And so does the initialisation and test infrastructure, likewise deferred by 4.8 and likewise unobtainable — 26.3 §14 models the shape a repair map must have rather than any device's mechanism.

Naming four open items is the honest close. The geometry, the probability arguments and the system arithmetic in these chapters are general and were recomputed at every step; the operating points are examples, and the two deferred topics are gaps rather than omissions. A reader with access to JESD235 should start with 26.1 §3's table — everything downstream of it depends on that structure being right.

What the four chapters share with the rest of this curriculum is the method. Multiply the published numbers together until the structure appears, then check the result against a second published statement — which is how the granularity identity, the L/4p closed form, the 2% occupancy, the spare-effectiveness curve and the threefold bandwidth range were all found. None of them came from reasoning about how HBM ought to work.

Continue learning

Standards & specifications

Governing standard
JEDEC JESD79 (DDR SDRAM)(opens JEDEC Solid State Technology Association in a new tab)

Defines the DDR SDRAM device itself — signals, command encoding, mode registers, timing parameters and the initialisation sequence — one document per generation. Memory-controller microarchitecture, address-mapping policy, PHY training algorithms and board-level design are not specified by it.

This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.

Where this fits

Part of the DDR curriculum.