HBM Overview

Every chapter so far has moved data through a package's external connections, and each has spent a lever on making those connections faster, better-timed or better-organised. Transfers per cycle. Prefetch depth. Calibration and training. Overlap. Granularity-preserving splits. Swing and clocking.

This chapter starts from a requirement none of them can reach. An accelerator wanting hundreds of gigabytes per second from its memory cannot get there by either of the two moves available, and understanding why both are closed is the whole setup.

Raising the per-pin rate is closed by energy. Chapter 4.7 §1 established that switching energy scales with the square of the swing, which is why low swing is the great energy lever. Run that argument the other way: a higher rate means more transitions per second, and it also means a larger swing is needed to maintain margin over a channel whose losses worsen with frequency. Energy per bit therefore rises with rate, and for a device already at a thermal ceiling, buying bandwidth with rate buys it at the worst possible exchange.

Adding pins is closed by geometry. A conventional package brings its signals out through connections arranged around its perimeter or across the face where it meets the board — and either way, the count is bounded by an area or a length, at a pitch set by what can be manufactured and assembled reliably. Die capability scales with area; external connections scale with perimeter. Those grow at different rates, and past a certain point width simply stops being available.

So the bottleneck stops being electrical and becomes geometric. No amount of signalling cleverness resolves "there is nowhere to put more connections", and that is a different kind of wall than anything in the first seven chapters.

1. The Response — Go Vertical, and Move the Channel onto Silicon

Two changes, and they are both packaging changes.

Stack the DRAM dies and connect them vertically. Instead of one die whose signals must exit through its package, several dies are stacked and connected by through-silicon vias — conductors passing vertically through the body of each die. A via's footprint is small and the available area for them is the die's face, not its edge. So the connection count scales with area rather than with perimeter, which is precisely the constraint that was binding.

HBM stacks are specified in several heights — 2, 4, 8 and 12 dies high in the generation this chapter's figures come from — with a logic die at the base of the stack handling the external interface.

Put the stack on an interposer beside the processor. Rather than routing to a socket across a board, the stack sits on a silicon interposer: a substrate carrying very fine, very dense wiring between the stack and the processor, both mounted on it side by side.

That changes the channel's character completely. A board trace is centimetres long with connector discontinuities; an interposer trace is short and on silicon, at a wiring pitch a printed circuit board cannot approach. Thousands of connections become physically routable, and each one is electrically easy in the way Chapter 4.7 §1 showed matters most.

2. The Numbers, and the Insight Inside Them

The published standard this chapter draws on specifies, per stack:

A 1024-bit wide interface, divided into 8 independent channels. Up to 24 GB of capacity, from stacks up to 12 dies high. Up to about 307 GB/s per device, at 2.4 Gb/s per pin.

Check the arithmetic, because it is self-verifying and the check is the lesson:

   1024 bits  ×  2.4 Gb/s per pin  =  2457.6 Gb/s
                                    ÷ 8           =  307.2 GB/s     ✓

Now compare the per-pin rate against the mainstream chain. A DDR5 device operating at 6400 MT/s transfers at 6.4 Gb/s per pin. HBM's 2.4 Gb/s per pin is well under half that.

And the 8 channels matter as much as the 1024 bits. A single 1024-bit channel would have a granularity no requester could use: Chapter 4.5 §1's arithmetic applies without mercy, and 1024 bits × any burst is enormous. Dividing the width into 8 independent channels keeps each channel's granularity usable while the aggregate width delivers the bandwidth — which is DDR5's channel-split reasoning applied at a much larger scale, and for both of the reasons that chapter and 4.7 identified: granularity control and independent request streams.

Eight independent channels also mean the aggregate figure is conditional in the way 4.5 and 4.6 established. 307 GB/s requires traffic that keeps all eight channels busy. One channel's worth of traffic gets one channel's worth of bandwidth, and no amount of width helps.

3. What It Costs

HBM's costs are larger and stranger than any other generation's in this module, and they are worth taking seriously because they explain why it did not simply displace everything else.

Cost 1 — packaging becomes a design layer, with its own failure modes. Stacking, vias, interposer assembly and the co-packaging of stack and processor are all manufacturing steps with their own yields. A defect in any of them scrapes an assembly containing an expensive processor, so yield loss compounds across parts that were individually good. That is a cost structure no other memory in this module has, and it is a large part of why HBM is reserved for applications whose bandwidth requirement justifies it.

Cost 2 — capacity is fixed and the memory is not replaceable. Chapter 4.7's trade, more absolutely: the memory is part of the processor's package. No upgrade, no repair, no reconfiguration, and the capacity available per stack bounds the system regardless of what the application wants.

Cost 3 — heat, and it lands on the one component least able to take it. This is the most interesting cost, because it is a collision between two things this curriculum has already established separately.

Stacking puts dies where heat cannot easily escape. A die in the middle of a stack is surrounded by other dies; its thermal path to anywhere useful runs through them.

And Chapter 2.3 §4 established that DRAM retention time falls as temperature rises, so a hotter device needs refreshing more often — which consumes more time and more energy, both of which produce more heat.

So stacking is thermally worst for the component that is most sensitive to temperature, and the sensitivity is self-reinforcing. A hot stack refreshes more, which heats it further. Thermal management is therefore not an implementation detail of HBM but a first-order architectural constraint, and it bounds how tall a stack is useful and how hard it can be driven — often more tightly than the electrical limits do.

Cost 4 — the bandwidth is conditional, at a larger scale than before. Eight channels need eight streams of traffic. A requester that cannot generate them gets a fraction of the figure, with nothing malfunctioning.

4. The Generation, With Layers Labelled

Four rows say "Packaging", and no earlier chapter has a single one. That is what makes HBM different in kind rather than in degree — it is the only generation in this module whose defining changes are not to the interface or to the device's internal organisation.

And the last row still reads "unchanged." Seven chapters, two lineages, a fourth design layer, and a sixteen-fold change in interface width — and underneath all of it is still a 1T1C cell with a destructive read, restoration, and mandatory refresh.

5. RTL — Bandwidth Arithmetic With Its Units Attached

Problem

Two things, and together they close the module's argument.

The peak bandwidth figure is elaboration-time arithmetic, and the arithmetic has units that must be carried through explicitly — because Chapter 4.1 §10 established that a memory-system budget hides factors of two and eight in exactly two places: the transfers-per-cycle term and the bits-to-bytes conversion.

The achieved bandwidth is a measurement, and it can be an arbitrarily small fraction of the peak. Wide interfaces make this worse, not better: eight independent channels need eight streams, and idle channels contribute nothing.

Classification

SYNTHESIZABLE RTL, with a COMPILE-TIME bandwidth derivation. The utilisation accounting is real controller telemetry; the bandwidth derivation generates no logic and exists to make a units error impossible to build.

What it does not model, and the list matters here. No TSVs, no stacking, no interposer, no thermal behaviour, no refresh, no array, no data, no beats, no timing. Every one of HBM's defining properties is physical, and none of them is representable in RTL — which is the cleanest illustration in this module of why the classification discipline exists. A module about packaging has almost no digital content, and pretending otherwise would be the fake-RTL error this curriculum has refused throughout.

Interface

ch_busy reports, per channel, whether it transferred this cycle. window_done pulses with busy_acc and util_pct valid, giving achieved utilisation over a window. The peak figures are elaboration-time parameters.

How to simulate it

vlog bandwidth_account.sv tb_bandwidth_account.sv then vsim -c tb_bandwidth_account -do "run -all"; VCS vcs -sverilog bandwidth_account.sv tb_bandwidth_account.sv && ./simv; Xcelium xrun -sv bandwidth_account.sv tb_bandwidth_account.sv.

// ─────────────────────────────────────────────────────────────────────────
// BANDWIDTH ACCOUNT.
// Classification: SYNTHESIZABLE RTL + COMPILE-TIME BANDWIDTH DERIVATION.
//
// Two halves, and together they are the module's closing argument:
//
//   COMPILE-TIME: peak bandwidth derived from width and per-pin rate with
//   every unit conversion written out and checked. Chapter 4.1 Section 10
//   established that memory budgets hide errors in exactly two places --
//   the transfers-per-cycle term and the bits-to-bytes divide -- so both
//   are explicit here and a shortfall against the target will not elaborate.
//
//   RUNTIME: ACHIEVED utilisation, measured per channel. A wide interface
//   makes the peak figure MORE conditional, not less: eight channels need
//   eight streams of traffic, and an idle channel contributes nothing.
//
// WHAT THIS DOES NOT MODEL: TSVs, die stacking, the interposer, thermal
// behaviour, refresh, the array, data, beats or timing. EVERY DEFINING
// PROPERTY OF HBM IS PHYSICAL and none is representable in RTL. A chapter
// about packaging has almost no digital content, and this block's job is to
// be honest about that rather than to invent some.
// ─────────────────────────────────────────────────────────────────────────
module bandwidth_account #(
  // Total interface width in bits, across all channels.
  parameter int DATA_BITS   = 1024,
  parameter int CHANNELS    = 8,
  // Per-pin data rate in megabits per second. 2400 is the figure this
  // chapter's arithmetic uses.
  parameter int RATE_MBPS   = 2400,
  // The aggregate the design must reach, in GB/s. Elaboration fails below it.
  parameter int TARGET_GBPS = 307,
  // Cycles per telemetry window.
  parameter int WINDOW      = 8,
  parameter int ACC_W       = 24,
  parameter int WIN_W       = (WINDOW <= 1) ? 1 : $clog2(WINDOW)
) (
  input  logic               clk,
  input  logic               rst_n,

  // Per channel: this channel transferred data this cycle.
  input  logic [CHANNELS-1:0] ch_busy,

  output logic               window_done,
  // Channel-cycles busy in the completed window. Peak is WINDOW*CHANNELS.
  output logic [ACC_W-1:0]   busy_acc,
  // Achieved utilisation as a whole percentage. See the truncation note.
  output logic [6:0]         util_pct
);

  // ── COMPILE-TIME BANDWIDTH DERIVATION. Generates no hardware.
  //    The unit chain is written out one step at a time DELIBERATELY: each
  //    line has a named unit, so a reader can check the conversion rather
  //    than trusting a single composite expression.
  localparam longint BITS_PER_SEC  = longint'(DATA_BITS)
                                   * longint'(RATE_MBPS)
                                   * 64'd1000000;              // bits/s
  localparam longint BYTES_PER_SEC = BITS_PER_SEC / 64'd8;      // bytes/s
  localparam longint PEAK_GBPS     = BYTES_PER_SEC / 64'd1000000000;

  // 1024 bits x 2400 Mb/s = 2 457 600 Mb/s = 2.4576e12 bits/s
  //                       -> 3.072e11 bytes/s -> 307 GB/s.
  // That the derived figure matches the standard's own published number is
  // the arithmetic checking itself, and is why the chain is spelled out.

  if (DATA_BITS < 1) begin : g_bits_min
    initial $fatal(1, "bandwidth_account: DATA_BITS must be >= 1");
  end
  if (CHANNELS < 1) begin : g_ch_min
    initial $fatal(1, "bandwidth_account: CHANNELS must be >= 1");
  end
  if ((DATA_BITS % CHANNELS) != 0) begin : g_ch_split
    initial $fatal(1, "bandwidth_account: DATA_BITS must divide evenly across CHANNELS");
  end
  if (RATE_MBPS < 1) begin : g_rate_min
    initial $fatal(1, "bandwidth_account: RATE_MBPS must be >= 1");
  end
  if (PEAK_GBPS < longint'(TARGET_GBPS)) begin : g_target
    initial $fatal(1, "bandwidth_account: peak %0d GB/s below target %0d GB/s",
                   PEAK_GBPS, TARGET_GBPS);
  end
  if (WINDOW < 1) begin : g_win_min
    initial $fatal(1, "bandwidth_account: WINDOW must be >= 1");
  end

  // ── Channel-cycles busy this cycle. A population count over ch_busy:
  //    the aggregate is the SUM across channels, which is exactly why one
  //    idle channel is a permanent subtraction from the achievable figure.
  logic [ACC_W-1:0] busy_now;
  always_comb begin
    busy_now = '0;
    for (int c = 0; c < CHANNELS; c++) begin
      busy_now = busy_now + ACC_W'(ch_busy[c]);
    end
  end

  logic [ACC_W-1:0] acc_q;
  logic [WIN_W-1:0] win_cnt;
  logic             win_last;
  assign win_last = (win_cnt == WIN_W'(WINDOW - 1));

  // Saturating, computed one bit wide to expose the carry.
  logic [ACC_W:0]   a_sum;
  logic [ACC_W-1:0] a_sat;
  // Percentage numerator, computed SEVEN BITS WIDER than the accumulator.
  // Multiplying an ACC_W-bit total by 100 needs about 7 more bits, and at
  // large WINDOW x CHANNELS an ACC_W-wide product would wrap -- which would
  // report a LOW utilisation from a high one, the one direction a
  // utilisation meter is most likely to be believed in.
  logic [ACC_W+6:0] pct_num;
  always_comb begin
    a_sum   = {1'b0, acc_q} + {1'b0, busy_now};
    a_sat   = a_sum[ACC_W] ? {ACC_W{1'b1}} : a_sum[ACC_W-1:0];
    pct_num = (ACC_W+7)'(a_sat) * (ACC_W+7)'(100);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      acc_q       <= '0;
      win_cnt     <= '0;
      window_done <= 1'b0;
      busy_acc    <= '0;
      util_pct    <= '0;
    end else begin
      window_done <= 1'b0;

      if (win_last) begin
        // ── Publish the completed window into SEPARATE output registers and
        //    restart the accumulator in the same cycle.
        //
        //    This is the structural fix for the window-boundary hazard named
        //    in Chapter 4.4 Section 6: because the published value and the
        //    working accumulator are different registers, restarting cannot
        //    drop the cycle that closed the window. The earlier block shared
        //    one register and had to lose a request; separating them costs a
        //    few flops and removes the race entirely.
        busy_acc    <= a_sat;
        window_done <= 1'b1;
        win_cnt     <= '0;
        acc_q       <= '0;

        // Integer percentage. TRUNCATES: 20 of 32 channel-cycles is 62.5%
        // and reports 62. Truncation rather than rounding is deliberate --
        // a utilisation figure that rounds UP flatters the design, and
        // Chapter 4.7 Section 8 argued an instrument should never err in the
        // direction its reader is hoping for.
        util_pct <= 7'(pct_num / (ACC_W+7)'(WINDOW * CHANNELS));
      end else begin
        acc_q   <= a_sat;
        win_cnt <= win_cnt + 1'b1;
      end
    end
  end

endmodule

Combinational logic

A population count over ch_busy and a saturating add. The population count is the architectural content: aggregate bandwidth is the sum across channels, so an idle channel is a permanent subtraction from the achievable figure and no activity elsewhere compensates.

Sequential logic

An accumulator, a window counter, and separate published output registers — which is the structural resolution of the hazard 4.4 §6 had to name and live with.

Cycle trace

CHANNELS = 4, WINDOW = 8 — reduced from the chapter's 8 channels purely for legibility. Channels 0 and 1 always busy, channel 2 busy on alternate cycles, channel 3 idle throughout:

20 channel-cycles of 32 possible: util_pct = 62.

Now attribute the loss, which is the point. Channel 3 was idle for all 8 cycles, costing 8 channel-cycles — 25% of the peak, lost to one channel having no traffic at all. Channel 2's alternating pattern cost 4 more. The other two channels were perfect and it did not matter.

That is the closing lesson of the module in one table. A 1024-bit interface is eight 128-bit channels, and one idle channel is a hard 12.5% ceiling reduction that no amount of width, rate or scheduling recovers. Width multiplies the peak; it also multiplies the number of ways to fail to reach it.

Simulation

A directed test should drive all channels busy and confirm 100%; drive one channel permanently idle and confirm the utilisation caps at (CHANNELS-1)/CHANNELS; confirm the window pulse every WINDOW cycles with totals matching an independent sum; confirm the published registers are not disturbed by the accumulator restarting; confirm truncation on a fractional percentage; and attempt elaboration with a width and rate below TARGET_GBPS and confirm it fails.

That last item tests the build rather than the simulation, and as in 4.6 §5, an elaboration check nobody has watched fire may not work.

Synthesis

A population count, a saturating adder, two counters and a divide. The divide is the one thing to watch — dividing by WINDOW * CHANNELS is free when that product is a power of two and a real divider otherwise, so choosing a power-of-two window is worth doing for a reason that has nothing to do with the statistics.

Corner cases

CHANNELS == 1 reduces to a single-channel utilisation meter. WINDOW == 1 pulses every cycle and reports 0% or 100% per channel-cycle, which is correct and useless — a good reminder that a window too short measures noise. ch_busy all zero gives 0%. A window product that is not a power of two forces a real divider. And a parameterisation below the bandwidth target does not elaborate, which is the check doing its job.

Verification

What DV must prove: busy_acc equalling an independently computed sum of set bits over the window; the published registers stable across the restart; utilisation truncating downward; the per-channel cap when a channel is idle; saturation rather than wrap; and the elaboration checks firing — including the bandwidth target, which is the one that encodes the architecture.

Debugging

If utilisation reads 100% when a channel is visibly idle, check the population count's loop bound against CHANNELS — a bound one short silently excludes the last channel and it is the highest-numbered one that is most often the idle one. If the percentage is consistently high by a small amount, check for rounding where truncation was intended. If busy_acc is one cycle's worth low, check that the cycle closing the window is included in a_sum before publication rather than after the restart. If the derived peak does not match a published figure, check the units before doubting the source — a factor of 8 is the bits-to-bytes divide and a factor of 1000 or 1024 is a decimal-versus-binary prefix.

Limitations

No TSVs, no stacking, no interposer, no thermal behaviour — HBM's defining properties, all absent, for the reason the header gives. No refresh, so the unavailability Chapter 2.3 mandates is invisible and the true achievable ceiling is below what this block reports. No data and no beats: ch_busy is asserted by something else and this block trusts it. No distinction between a channel busy with wanted data and a channel busy with granularity waste — so 100% utilisation here is fully compatible with delivering a fraction of it usefully, and combining this measurement with 4.4 §6's useful-byte counters is the only way to get a complete picture.

6. Three Assertions Worth Writing

// VERIFICATION-ONLY, inside bandwidth_account.

// P1 -- the accumulator is the sum of set bits over the window. Written
// against an independently computed reference rather than against the
// design's own adder, because an assertion that reuses the logic it is
// checking proves only that the logic equals itself.
logic [ACC_W-1:0] ref_acc;
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n)           ref_acc <= '0;
  else if (win_last)    ref_acc <= '0;
  else                  ref_acc <= ref_acc + ACC_W'($countones(ch_busy));
end

property p_acc_matches_reference;
  @(posedge clk) disable iff (!rst_n)
    (!win_last && (acc_q != {ACC_W{1'b1}}))
      |=> (acc_q == (ref_acc + ACC_W'($countones($past(ch_busy)))));
endproperty
assert property (p_acc_matches_reference);

// P2 -- the published window total is never disturbed by the accumulator
// restarting. This is the property that the SEPARATE REGISTERS exist to
// guarantee, and it is the one Chapter 4.4's shared-register version could
// not have satisfied.
property p_published_stable;
  @(posedge clk) disable iff (!rst_n)
    !window_done |=> (busy_acc == $past(busy_acc));
endproperty
assert property (p_published_stable);

// P3 -- THE DIRECTIONAL PROPERTY. Reported utilisation never exceeds the
// true utilisation. It is asserted as an inequality rather than an equality
// DELIBERATELY: truncation makes the reported figure sometimes lower than
// the true one, and that is correct behaviour. What must never happen is
// the reverse.
property p_util_never_flatters;
  @(posedge clk) disable iff (!rst_n)
    window_done |-> ((util_pct * (WINDOW * CHANNELS)) <= (busy_acc * 100));
endproperty
assert property (p_util_never_flatters);

// P4 -- an idle channel is a hard ceiling. The architectural property: with
// any channel never busy, utilisation cannot reach 100%. Without it, a
// design that ignored ch_busy and reported 100% would satisfy P2 and P3.
property p_idle_channel_caps;
  @(posedge clk) disable iff (!rst_n)
    (window_done && (ch_busy != {CHANNELS{1'b1}}))
      |-> (util_pct < 7'd100);
endproperty
assert property (p_idle_channel_caps);

P1's reference model is the lesson. An assertion written as acc_q == acc_q_computed_the_same_way proves nothing — a property must compute its expectation independently of the logic it is checking, or it is a tautology with good intentions. Here $countones and a separate accumulator provide that independence, and it is worth the few extra lines.

P3 is asserted as an inequality, and the direction is chosen. Truncation means the reported figure is sometimes lower than the truth, which is correct, so an equality would fail on correct behaviour. The property that matters is one-sided: the instrument must never over-report. When a design deliberately errs in one direction, assert the direction rather than the value — that is the property you actually care about, and it is strictly stronger than an approximate equality with a tolerance.

What none of them prove — and this is the module's final and largest verification gap. Nothing about stacking, TSVs, the interposer, assembly yield, or thermal behaviour. Thermal behaviour is HBM's dominant physical constraint, per §3, and it is unreachable by any assertion in any simulation of any digital block. It is established by thermal modelling, by measurement on real assemblies, and by the interaction with the refresh requirement that Chapter 2.3 describes.

Stating that plainly is the point of having classified every block in this module. A green regression on this block is evidence that a utilisation counter works. It is evidence about nothing else, and the gap between it and "HBM will work" is the largest in the entire module.

7. Utilisation in Cycles

ch_busy[3] is flat at zero, and that single flat trace costs 8 of the 12 lost channel-cycles. Channels 0 and 1 were perfect for the entire window and it did not help — because aggregate bandwidth is a sum across channels, not a maximum over them.

That is the module's closing observation stated as a waveform. Every generation in this module bought bandwidth by adding a dimension of parallelism: edges, then prefetch width, then bank groups, then sub-channels, then eight channels and a thousand bits. Every one of those dimensions is also a new way to leave capability unused — and the wider the interface, the more of the peak a single unused dimension costs.

What the figure does not claim. No refresh, so the true ceiling is lower. No distinction between busy-with-wanted-data and busy-with-waste, so 100% here would still be compatible with delivering a fraction usefully. And nothing about stacking, thermals or the interposer.

8. Common Misconceptions

"HBM is faster than DDR5." Wrong mental model: higher bandwidth means faster signalling. Engineering action: assuming HBM's per-pin rate exceeds DDR5's; expecting lower access latency; expecting worse energy per bit because "faster costs more". Observable failure / bad conclusion: exactly backwards on both. HBM's per-pin rate is under half DDR5's, so its energy per bit is lower, not higher — and its access latency is not the thing it improves. An engineer holding this model cannot explain why HBM is used in thermally limited parts, since on this model it would be the worst possible choice. Correct model: HBM's pins are slower. It delivers roughly six times a 64-bit DDR5 channel's bandwidth by being sixteen times wider at under half the rate, and the low rate is a benefit the width bought — because energy per bit rises with rate. Prevention: divide the bandwidth by the width to get the per-pin rate before comparing anything. Aggregate bandwidth says nothing about per-pin speed.

"HBM's advantage is the stacking." Wrong mental model: stacking is the mechanism. Engineering action: attributing the bandwidth to vertical integration in itself; expecting stacking alone to deliver it; not accounting for the interposer. Observable failure / bad conclusion: missing that a stack still has to reach the processor, and that a board-level connection would destroy the benefit — so the interposer is not an implementation detail but half the architecture. Also missing why the thermal cost exists. Correct model: stacking with TSVs makes connections scale with die area instead of package perimeter, which is what makes 1024 bits routable at all. The interposer is what makes them electrically usable, by replacing centimetres of board and a connector with short, dense silicon wiring. Both are required, and both are packaging changes. Prevention: ask where the signals go after they leave the stack. If the answer is a board, the width is unusable.

"A wider interface is straightforwardly better." Wrong mental model: width is pure benefit. Engineering action: budgeting the aggregate figure without checking whether the requesters can fill every channel; treating a single wide channel as equivalent to several narrow ones. Observable failure / bad conclusion: a system reaching a fraction of the quoted bandwidth with nothing malfunctioning — and, if the width were not divided into channels, granularity so large that almost every access wastes most of what it transfers. Correct model: width multiplies the peak and multiplies the number of ways to miss it. 1024 bits is divided into 8 independent channels precisely because a single 1024-bit channel's granularity would be unusable — and eight channels require eight streams of traffic, so one idle channel is a hard 12.5% ceiling reduction nothing recovers. Prevention: compute granularity per channel, and ask how many independent request streams the requesters actually produce.

"HBM's main challenge is signal integrity, like every other memory interface." Wrong mental model: the hardest problem is electrical, as it was for DDR3. Engineering action: budgeting engineering effort toward signalling and training; treating thermal management as a mechanical afterthought; sizing stack height by electrical limits. Observable failure / bad conclusion: a design whose stack height or drive level is electrically fine and thermally unviable — and a refresh budget that under-predicts badly, because retention falls with temperature and a hot stack refreshes more, which heats it further. Correct model: the interposer channel is electrically easy — short, dense, on silicon, no connector. The dominant constraint is thermal, and it lands on the component least able to tolerate it: a die inside a stack has its thermal path through other dies, and DRAM retention falls as temperature rises, so the sensitivity is self-reinforcing. Prevention: ask which constraint binds first. Moving the channel onto silicon solved the electrical problem and created a thermal one, and the design effort follows the binding constraint.

9. Debugging — An HBM System Delivers a Fraction of Its Rated Bandwidth

Symptom. A system with HBM delivers well below its rated aggregate bandwidth on a workload intended to stream. Functionally correct, no errors.

With eight channels, the first question is always distributional, because a wide interface's most common failure is that some of it is unused.

Mechanism 1 — traffic is not spread across all channels. Inspect: per-channel utilisation, which is §5's ch_busy accounting directly. Expected evidence: some channels near 100% and others near zero. Discriminator: the per-channel distribution. This is first because it is one counter read per channel and because it is the most common cause — and because the loss is exactly computable from the distribution, so it either accounts for the shortfall or it does not.

Mechanism 2 — the address mapping concentrates traffic. Inspect: which address bits select the channel, against the workload's stride. Expected evidence: a stride that leaves the channel field constant or cycling through a subset. Discriminator: whether the workload is balanced but the mapping is not. Distinguished from mechanism 1 by cause rather than symptom — mechanism 1 is the observation, this is one of its causes, and the other is a requester that genuinely only touches part of memory. Reading the mapping against the stride separates them, and only one is fixable in configuration.

Mechanism 3 — granularity waste. Inspect: useful bytes over transferred bytes per channel. Expected evidence: all channels near 100% busy with delivered useful bandwidth far below. Discriminator: all channels busy, yet still short. This is the case §5's limitations warned about explicitly: this block's 100% is fully compatible with delivering a fraction usefully, and only combining it with useful-byte counters distinguishes them. A 128-bit channel has a substantial granularity of its own.

Mechanism 4 — thermal throttling, and the refresh burden that comes with it. Inspect: stack temperature under load and whether the refresh rate is being raised accordingly. Expected evidence: a ceiling that falls as the system warms, with no change in the access pattern. Discriminator: does the ceiling move with temperature while the distribution stays constant? This is HBM's characteristic mechanism, per §3, and the self-reinforcing part matters: more refresh means less availability and more heat. A ceiling that degrades over minutes of sustained load rather than instantly is this mechanism's signature.

Mechanism 5 — the requester cannot generate enough concurrency. Inspect: outstanding requests per channel. Expected evidence: channels evenly but lightly loaded, with few accesses in flight. Discriminator: evenly distributed and lightly loaded. Cleanly distinct from mechanism 1's uneven distribution and mechanism 3's saturation. Eight channels cannot help a requester that keeps one access outstanding — Chapter 1.8 §7's point, and the fix is upstream of memory entirely.

Discrimination, cheapest first. Read per-channel utilisation. Its shape separates three mechanisms at once: uneven points at distribution or mapping, even and saturated points at granularity waste, even and light points at insufficient concurrency. Then read useful-over-transferred bytes. Then check whether the ceiling moves with temperature.

The reasoning lesson, and it is the module's. The shape of a distribution is more informative than its total. One number — aggregate achieved bandwidth — cannot distinguish five mechanisms in four different layers. Eight numbers, one per channel, separate most of them immediately, because each mechanism leaves a different signature in the distribution rather than in the sum.

That generalises far beyond memory. Whenever a system's capability comes from parallelism, the per-unit distribution is the diagnostic and the aggregate is merely the score. Every generation in this module added a dimension of parallelism, and therefore added a dimension to measure — which is why 4.4 built efficiency counters, 4.5 split its stall causes, 4.6 tracked per-sub-channel occupancy, and this chapter counts per channel. The telemetry grew for exactly the same reason the hardware did.

10. Interview Reasoning

"Is HBM faster than DDR5?" Its pins are slower — under half the per-pin rate. It delivers roughly six times a 64-bit DDR5 channel's bandwidth by being sixteen times wider: 1024 bits against 64, so sixteen times the width against 0.375 of the rate. And the low rate is a benefit rather than a limitation, because energy per bit rises with rate, so a wide-and-slow interface moves each bit more cheaply than a narrow-and-fast one delivering the same aggregate. For a device at a thermal ceiling, which every accelerator is, that is the entire point. The arithmetic checks itself: 1024 bits times 2.4 Gb/s per pin divided by 8 is about 307 GB/s, which is what the standard states.

"Why couldn't DDR5 just be made wider?" Because the limit is geometric, not electrical. A conventional package brings its signals out through connections bounded by a perimeter or a face, at a pitch set by what can be manufactured and assembled — while die capability scales with area. Those grow at different rates, so past a point there is simply nowhere to put more connections, and no signalling improvement changes that. HBM goes around it by stacking dies connected with through-silicon vias, so the connection count scales with die area instead of package perimeter, and by placing the stack on a silicon interposer beside the processor so the wiring is short and dense enough to be usable. Both are packaging changes, which is the layer none of the earlier generations treated as a variable.

"Why is HBM's width divided into eight channels rather than being one wide channel?" Granularity. Minimum access size is width times burst length, and 1024 bits times any burst is far larger than anything a requester wants — so a single wide channel would waste most of every transfer. Dividing into eight independent channels keeps each channel's granularity usable while the aggregate width supplies the bandwidth. It also gives eight independent request streams, which is the concurrency that converts a fixed access latency into throughput. The cost is that the aggregate figure now requires eight streams of traffic: one idle channel is a hard 12.5% reduction in the achievable ceiling that nothing else recovers.

"What is HBM's dominant physical constraint?" Thermal, not electrical — which is the reverse of every earlier generation in this module. Moving the channel onto a short, dense silicon interposer with no connector made the electrical problem comparatively easy, and stacking created a thermal one: a die in the middle of a stack has its thermal path through other dies. And it lands on the component least able to tolerate it, because DRAM retention time falls as temperature rises, so a hot stack must refresh more often — which costs availability and energy, both of which produce more heat. The sensitivity is self-reinforcing, which is why thermal management bounds stack height and drive level more tightly than the electrical limits do.

"An HBM system delivers a fraction of its rated bandwidth with no errors. How do you narrow it down?" Read per-channel utilisation first, because the shape of that distribution separates three causes immediately. Uneven — some channels saturated and others idle — means traffic is not spread, and the next question is whether the workload is unbalanced or the address-to-channel mapping is, since only the second is fixable in configuration. Even and saturated means all channels are busy and the bytes are being wasted, so the next measurement is useful bytes over transferred bytes; a 128-bit channel has substantial granularity of its own. Even and lightly loaded means the requester cannot generate enough concurrency, and the fix is upstream of memory entirely. After that, check whether the ceiling falls as the stack warms, which is HBM's characteristic mechanism — retention falls with temperature, so refresh burden rises, and a ceiling that degrades over minutes of sustained load rather than instantly is that signature. The general point is that one aggregate number cannot distinguish five mechanisms in four layers, and eight per-channel numbers distinguish most of them at a glance.

11. Engineering Check

Educational figures drawn from the standard's published values: 1024-bit interface, 8 channels, 2.4 Gb/s per pin.

1. Bits per channel, and aggregate bandwidth? 1024 ÷ 8 = 128 bits per channel. Aggregate: 1024 × 2.4 × 10⁹ ÷ 8 = 307.2 × 10⁹ bytes/s = about 307 GB/s — which matches the standard's own published figure, and a derivation that reproduces its source is the only kind worth trusting.

2. Per-channel bandwidth? 128 bits × 2.4 Gb/s ÷ 8 = 38.4 GB/s, and 8 × 38.4 = 307.2. ✓

3. Compare per-pin rate with a DDR5 device at 6400 MT/s. DDR5: 6.4 Gb/s per pin. HBM: 2.4 Gb/s per pin — about 37% of it. HBM's bandwidth advantage is 1024 ÷ 64 = 16× width against 2.4 ÷ 6.4 = 0.375× rate, giving 16 × 0.375 = 6× the aggregate. Width did all the work and then some had to be given back on rate.

4. What would a 64-bit interface need per pin to reach 307 GB/s? 307.2 × 10⁹ × 8 ÷ 64 = 38.4 × 10⁹ bits/s = 38.4 Gb/s per pin — six times DDR5's rate and sixteen times HBM's. This is the question the whole chapter answers: that rate is not achievable on a socketed channel at any energy per bit a thermally limited part could sustain, which is why the width had to come from somewhere, and why packaging was the only place left.

5. A workload keeps 6 of 8 channels fully busy and 2 idle. Achieved bandwidth? 6 ÷ 8 = 75%, so 307.2 × 0.75 = about 230 GB/s. 77 GB/s lost to two idle channels, with every other channel perfect and nothing malfunctioning.

6. The same workload, but each access uses only 64 of the 128 bits per channel. Achieved useful bandwidth? 230 × 50% = about 115 GB/s — 37% of the rated figure, from two independent and multiplicative losses. Neither is visible in the other's metric: per-channel utilisation reads 75% and says nothing about waste; useful-byte fraction reads 50% and says nothing about idle channels. Two numbers were required to see it, which is §9's lesson and the module's.

12. Summary

HBM answers a requirement neither earlier lineage can reach, and it does so because both of their levers are closed. Raising the per-pin rate is closed by energy — energy per bit rises with rate, and a thermally limited part cannot buy bandwidth at that exchange. Adding pins is closed by geometry — external connections scale with a package's perimeter while die capability scales with area, so width simply stops being available. The bottleneck stops being electrical and becomes geometric.

The response is two packaging changes. Stacking dies connected by through-silicon vias makes the connection count scale with die area instead of package perimeter. Placing the stack on a silicon interposer beside the processor replaces centimetres of board and a connector with short, dense wiring on silicon. Both are required: stacking makes the width routable and the interposer makes it usable.

The result inverts the module's assumption. A 1024-bit interface, divided into 8 independent channels, at 2.4 Gb/s per pin — under half DDR5's per-pin rate — for about 307 GB/s per device, a figure that derives exactly from 1024 × 2.4 ÷ 8 and matches its source. HBM's pins are slower, and the low rate is a benefit the width bought. Every earlier chapter took width as fixed by packaging and spent its ingenuity on rate; HBM makes width the variable and reduces the rate. The assumption that width was fixed lived in a layer nobody was treating as a design variable.

And that is why this chapter adds a fourth layer to Chapter 4.1's three. Array physics, device architecture, and external interface and PHY were enough for seven chapters. HBM's defining changes are in packaging, and it is the only generation in this module of which that is true.

The costs are larger and stranger than any other generation's. Packaging becomes a design layer with its own yields, so a defect scraps an assembly containing an expensive processor. Capacity is fixed and the memory unreplaceable. The bandwidth is conditional on eight streams of traffic. And the dominant constraint becomes thermal — which lands on the component least able to tolerate it, because a die inside a stack has its thermal path through other dies and DRAM retention falls as temperature rises, so a hot stack refreshes more, which heats it further. The sensitivity is self-reinforcing, and it bounds stack height more tightly than any electrical limit.

The module's closing observation. Every generation here bought bandwidth by adding a dimension of parallelism — edges, prefetch width, bank groups, sub-channels, channels and width. Every one of those dimensions is also a new way to leave capability unused, and the wider the interface, the more of the peak a single unused dimension costs. So the telemetry grew for exactly the same reason the hardware did, and the shape of a distribution became more informative than its total.

And the array is unchanged. Eight chapters, two lineages, a fourth design layer, a sixteen-fold change in width — and underneath all of it, still a 1T1C cell that loses its charge, destroys what it reads, and must be refreshed forever.

13. What Comes Next

Module 4 traced why DDR is shaped the way it is: eight bottlenecks, eight architectural responses, and a set of mechanisms — prefetch, strobes, termination, calibration, training, bank groups, channel splits, power states, stacking — each introduced only to the depth that explained why a generation needed it.

Module 5, DDR Architecture Overview, stops asking why and starts describing what. It sets out the organisation this module kept forward-linking to: the DDR device's top-level structure, banks and their per-bank row buffers, bank groups in proper detail rather than as 4.5's causal argument, ranks as groupings of devices sharing a select, channels, the DIMM forms a module actually takes, and the end-to-end memory-subsystem view from processor through controller and PHY to the module. Every later module references that picture, and this module's job was to make it inevitable rather than arbitrary.

Return to SDR SDRAM for the rate vocabulary the whole module depends on, DDR5 for the channel split this chapter scales up, or The Memory Wall Problem for why peak bandwidth predicts so little — a claim this chapter has now demonstrated six different ways. Module 26 is the HBM architecture in full. The full path is on the DDR tutorials index.