GPU Memory Bottlenecks

Chapter 1.4 ended with a prediction: every engine acquires memory close to it, because proximity is what buys bandwidth, and that memory becomes an island — fast for its owner and fixed in size.

This chapter is what happens when the workload outgrows the island. It is deliberately not a GPU architecture tutorial; it is about one property of one resource, and about a failure mode that is routinely misdiagnosed because it wears the costume of a different one.

1. The One-Sentence Model

A device can have extraordinary local memory bandwidth and still be limited, because bandwidth answers "how fast can the resident data be supplied" and capacity answers "can the data be resident at all". They are independent, they fail differently, and no amount of the first fixes the second.

Hold that as two separate failures with two separate symptoms:

BANDWIDTH-BOUND   the working set fits, and cannot be supplied fast enough
                  → device busy, memory system saturated, hit rate high
 
CAPACITY-BOUND    the working set does not fit at all
                  → device idle, link busy, hit rate collapses

The second is the subject of this chapter, and the reason it is worth a chapter is that its most visible symptom — a busy interconnect — looks exactly like a bandwidth problem to anyone watching the wrong counter.

2. What This Chapter Owns

Not here: what conventional device attach provides (Ch 1.6), or coherent attach as a mechanism (Ch 1.7). No CXL mechanism appears in this chapter.

3. Why Device-Local Memory Is Fast and Small

The two properties have one cause, which is why they cannot be optimised independently.

Memory placed close to a compute die can be very wide and very short-reach. Width is where the bandwidth comes from — thousands of signals in parallel rather than tens — and short reach is what makes that many signals electrically affordable. Both require the memory to be physically near the die: on the same package, on an interposer, or on a short dedicated channel.

Physical proximity is a finite resource. There is only so much perimeter, so much interposer area, so many stacks that fit and can be cooled and powered. So:

The property that makes device-local memory fast — proximity — is the property that bounds how much of it there can be.

That is not a design flaw awaiting a fix. It is the trade, and it is the same trade Chapter 1.1 §10 described on the placement axis. A device designer who wants more capacity gets it by moving memory further away, which costs the bandwidth that motivated the local memory in the first place.

4. What Occupies Device Memory

Any statement of the form "a model needs X GiB" is a statement about a specific implementation, so this section names the categories rather than sizes, and the worked example below labels every number as illustrative.

For a large-model inference or training workload, device memory is typically claimed by several distinct things at once:

• Parameters. The weights themselves, at whatever numerical precision the deployment uses. Precision is a first-order capacity decision: halving bytes-per-parameter halves this term exactly.
• Activations and intermediates. Values produced by one stage and consumed by the next. This term scales with batch size and with how much the implementation chooses to keep rather than recompute.
• Optimiser and gradient state during training. Frequently a multiple of the parameter term rather than a fraction of it, depending on the optimiser.
• Per-request runtime state during inference — for attention-based models, the cached keys and values, which grow with sequence length and with the number of concurrent requests.
• Workspace and fragmentation. Scratch buffers for individual kernels, plus allocator overhead. Small per item, and not small in aggregate.

Two consequences matter more than any specific size.

Several of these grow with things the hardware does not control. Batch size, sequence length and concurrency are serving decisions, so device capacity is consumed by choices made after the part shipped.

They compete. Capacity spent on a larger batch is capacity not available for per-request state. Serving systems spend real engineering effort on that allocation, which is a strong signal that the resource is genuinely scarce.

5. Worked Example — Does It Fit?

The arithmetic is trivial and the discipline is in labelling. Everything below is illustrative: the numbers are chosen for round arithmetic, not measured from a product.

GIVEN (illustrative)
  parameters                    P = 34e9
  bytes per parameter           b = 2          (a 16-bit format)
  device local memory           C = 48 GiB = 51.54e9 bytes
 
PARAMETER FOOTPRINT
  P × b = 34e9 × 2 = 68e9 bytes  ≈ 68 GB
 
DOES IT FIT?
  68e9 > 51.54e9   →  NO, by 16.5e9 bytes  (≈ 32% over)

The parameters alone exceed capacity before a single activation, optimiser slot or per-request buffer is counted. Now notice what does not help:

double the local bandwidth   → still does not fit
double the link bandwidth    → still does not fit
double the compute           → still does not fit
halve bytes per parameter    → 34e9 bytes, now fits with 17.5e9 to spare
add a second device          → 96 GiB aggregate, but see Section 8

Only two of those five change the answer, and one of them changes the numerics of the application. That is what makes capacity a different kind of constraint from bandwidth: bandwidth shortfalls degrade performance continuously, and capacity shortfalls are a cliff.

6. When It Does Not Fit

If the working set exceeds local capacity, something must move. The standard response is to keep a subset resident and migrate the rest on demand — which converts a capacity problem into a bandwidth and latency problem, and this is exactly where the misdiagnosis starts.

The loop is simple and expensive:

access block  →  resident?  ── yes ──→  fast local access
                     │
                     no
                     ↓
              evict something  →  migrate block in  →  access it
                                        ↑
                     the evicted block is needed again later, and returns

The cost has two parts, and both are worth naming because they are measured differently.

Bandwidth cost. Every migrated byte crosses the link and is written into device memory — Chapter 1.3 §7 applies unchanged, including the amplification. If a block is evicted and refetched k times, its transfer cost is paid k times.

Latency cost. The engine waits. Unlike a bandwidth shortfall, which slows a running computation, a residency miss can stop one — and the stall is the link round trip, not the local memory latency.

7. Measured: Fitting Against Not Fitting

Section 12 builds a residency tracker with a bounded number of resident blocks. Running the same access pattern against two working-set sizes isolates capacity from everything else — same hardware, same link, same block size, same code:

=== working set = 4 blocks, local memory holds 4 (FITS) ===
  accesses=28 hits=25 migrations=3   hit rate=89%  migrated=768 KiB
 
=== working set = 8 blocks, local memory holds 4 (DOES NOT FIT) ===
  accesses=24 hits=4  migrations=20  hit rate=16%  migrated=5120 KiB

Doubling the working set past capacity took the hit rate from 89% to 16% and multiplied migrated bytes by 6.7×. Nothing else changed. There is no bandwidth number that repairs the second row, because the second row is not a bandwidth failure — it is the same data being fetched again and again because there is nowhere to keep it.

That contrast is the chapter in one measurement, and it generalises past this toy: once the working set exceeds capacity, traffic scales with how often blocks are re-fetched rather than with how much data the algorithm actually needs.

8. Aggregate Capacity Is Not a Pool

The obvious response to "it does not fit" is "add another device". That adds capacity to the machine and does not necessarily add usable capacity to the problem.

Four devices with 48 GiB each hold 192 GiB in total. Whether a 68 GB working set fits in that 192 GiB depends entirely on whether the data and the computation can be partitioned so each device's share fits in its 48 GiB — and on what crossing the partition costs.

aggregate capacity  =  N × C            arithmetic, always true
usable capacity     =  depends on partitioning and on cross-device traffic

Three complications, stated without overclaiming any particular platform's behaviour:

A device's memory is local to that device. Another device reaching it does so over an interconnect, not over its own memory bus — so a partition that requires frequent cross-device access converts a capacity win into a bandwidth and latency cost.

Partitioning is not free and not always possible. Some structures split cleanly across devices; others require every device to see the same data, which duplicates rather than divides it. Duplication consumes the capacity that adding devices was supposed to provide.

Adding devices adds compute you may not need. Chapter 1.1 §7 made this point about sockets, and it is sharper here: buying four accelerators to obtain capacity means paying for four accelerators' worth of silicon and power to solve a memory problem.

9. The Hardware of a Device Memory Manager

Sections 10 to 12 build three structures that any design managing a finite local memory needs: admission control that refuses work which cannot fit, a residency table that knows what is present and what is on its way, and instrumentation that distinguishes a capacity failure from a bandwidth one.

10. RTL 1 — Capacity Admission

Purpose

Capacity is a resource that can be reserved, and the interesting bug is reserving against the wrong quantity.

// Admission control for a finite local memory. A work unit is admitted only if
// its working-set reservation fits in what is left. The subtlety the design
// exists to expose: "what is left" must account for reservations that are
// granted but not yet released, not just for memory already occupied.
module capacity_admit #(
  parameter int unsigned CAP_MIB = 48,
  parameter int unsigned SIZE_W  = 16
) (
  input  logic               clk,
  input  logic               rst_n,
  input  logic               adm_req,
  input  logic [SIZE_W-1:0]  adm_size_mib,
  output logic               adm_grant,
  input  logic               rel_valid,
  input  logic [SIZE_W-1:0]  rel_size_mib,
  output logic [SIZE_W-1:0]  reserved_q,
  output logic [SIZE_W-1:0]  headroom,
  output logic               overcommit_err
);
  assign headroom  = SIZE_W'(CAP_MIB) - reserved_q;
  // Admission compares against RESERVED, not against "occupied". A request is
  // granted only when it fits in the headroom remaining after every
  // outstanding reservation is honoured — see Debug Lab 1.
  assign adm_grant = adm_req && (adm_size_mib <= headroom);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      reserved_q     <= '0;
      overcommit_err <= 1'b0;
    end else begin
      if (reserved_q > SIZE_W'(CAP_MIB)) overcommit_err <= 1'b1;
      // Grant and release in the same cycle must both apply; treating them as
      // independent statements loses one of them.
      case ({adm_grant, rel_valid})
        2'b10: reserved_q <= reserved_q + adm_size_mib;
        2'b01: reserved_q <= reserved_q - rel_size_mib;
        2'b11: reserved_q <= reserved_q + adm_size_mib - rel_size_mib;
        default: reserved_q <= reserved_q;
      endcase
    end
  end
endmodule

Architectural role, state and contract

The gate between a scheduler that wants to run work and a memory that may not hold it. State is one reservation total plus a sticky overcommit flag. The contract is that reserved_q never exceeds CAP_MIB — and because admission is a comparison against headroom, that contract is structural rather than checked after the fact.

Cycle behaviour and simulation evidence

Four requests against a 48-unit memory, sampled in the cycle each request is presented — verbatim from the Icarus run:

=== EXP1: admission against a 48-unit local memory ===
  request 20 units: grant=1  reserved=0   headroom=48
  request 20 units: grant=1  reserved=20  headroom=28
  request 20 units: grant=0  reserved=40  headroom=8    <-- refused
  request  8 units: grant=1  reserved=40  headroom=8    <-- exactly fits
  final reserved=48 headroom=0 overcommit_err=0

The third line is the whole module. A request for 20 units against 8 units of headroom is refused, not queued and not partially granted, and refusing it is what keeps the invariant true. The fourth line shows the boundary case working: a request that exactly consumes the remaining headroom is admitted.

Synthesis and failure shape

One accumulator with a comparator and a subtractor, plus one error flop. The characteristic failure is comparing against occupancy rather than reservation, which admits work whose memory has been promised to somebody else — Debug Lab 1, where it produces overcommit under concurrency and never under a directed test.

11. RTL 2 — Residency and Eviction

Purpose

To know what is resident, what is on its way, and what must leave to make room — and to make oversubscription measurable rather than anecdotal.

// Which blocks of the working set are in local memory, and what happens when
// the working set is larger than local memory can hold.
//
// State per block, two bits:
//   00 ABSENT   not local, no migration in flight
//   01 PENDING  migration requested, not yet complete
//   10 RESIDENT local copy present and usable
//
// PENDING is what suppresses duplicate migration requests. RESIDENT_MAX is
// what makes oversubscription visible: once that many blocks are resident, a
// new arrival must evict one, and a working set larger than RESIDENT_MAX
// therefore misses on every access no matter how fast the link is.
module residency_tracker #(
  parameter int unsigned NBLK         = 8,
  parameter int unsigned RESIDENT_MAX = 4
) (
  input  logic                    clk,
  input  logic                    rst_n,
  input  logic                    acc_valid,
  input  logic [$clog2(NBLK)-1:0] acc_blk,
  output logic                    hit,
  output logic                    need_migrate,
  output logic                    dup_suppressed,
  input  logic                    mig_done_valid,
  input  logic [$clog2(NBLK)-1:0] mig_done_blk,
  output logic                    evict_valid,
  output logic [$clog2(NBLK)-1:0] evict_blk,
  output logic [1:0]              state_q [NBLK],
  output logic [$clog2(NBLK+1)-1:0] resident_cnt_q,
  output logic                    bad_done_err
);
  localparam logic [1:0] ABSENT = 2'b00, PENDING = 2'b01, RESIDENT = 2'b10;
  localparam int unsigned BW = $clog2(NBLK);
 
  logic [BW-1:0] victim_q;          // round-robin eviction pointer
 
  assign hit            = acc_valid && (state_q[acc_blk] == RESIDENT);
  assign need_migrate   = acc_valid && (state_q[acc_blk] == ABSENT);
  assign dup_suppressed = acc_valid && (state_q[acc_blk] == PENDING);
 
  // Space must be made when a migration completes into a full local memory.
  assign evict_valid = mig_done_valid
                       && (resident_cnt_q >= $clog2(NBLK+1)'(RESIDENT_MAX))
                       && (state_q[victim_q] == RESIDENT);
  assign evict_blk   = victim_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int unsigned i = 0; i < NBLK; i++) state_q[i] <= ABSENT;
      resident_cnt_q <= '0;
      victim_q       <= '0;
      bad_done_err   <= 1'b0;
    end else begin
      // A completion for a block that was not pending means the tracker and
      // the migration engine disagree about what is in flight.
      if (mig_done_valid && (state_q[mig_done_blk] != PENDING)) bad_done_err <= 1'b1;
 
      if (need_migrate) state_q[acc_blk] <= PENDING;
 
      if (mig_done_valid) begin
        state_q[mig_done_blk] <= RESIDENT;
        if (evict_valid) begin
          state_q[victim_q] <= ABSENT;
          victim_q <= (victim_q == BW'(NBLK-1)) ? '0 : victim_q + 1'b1;
          // one in, one out: resident count unchanged
        end else begin
          resident_cnt_q <= resident_cnt_q + 1'b1;
        end
      end
    end
  end
endmodule

Why PENDING earns its bit

Without a PENDING state, every access to a block that is absent issues a migration — including accesses to a block whose migration is already in flight. The result is duplicate transfers of the same data: link bandwidth spent fetching something that was already coming, and two completions racing to update one entry.

Measured, verbatim:

=== EXP2: first touch, duplicate suppression, then a hit ===
  touch blk0 (cold)    : hit=0 migrate=1 suppressed=0
  touch blk0 (pending) : hit=0 migrate=0 suppressed=1  <-- no 2nd migration
  touch blk0 (resident): hit=1 migrate=0 suppressed=0

Three states, three distinct behaviours, and the middle one is the one a two-state design gets wrong.

Cycle behaviour, contract, synthesis

resident_cnt_q never exceeds RESIDENT_MAX — a completion into a full memory evicts as it inserts, so the count is unchanged rather than incremented. A completion for a block that is not PENDING is a protocol error and is latched. Synthesis is 2 × NBLK state flops, a small counter, a victim pointer and comparators; the round-robin victim is deliberately the simplest possible policy, and a real design would choose the victim on recency or on a cost model.

Failure shape

A tracker that drops the PENDING state duplicates migrations. One that increments the resident count on an evicting completion overcommits local memory. One that evicts a block whose migration is still in flight loses the data entirely.

12. RTL 3 — Instrumentation That Separates the Two Failures

Purpose

The chapter's central claim is that capacity and bandwidth failures look alike from the outside. This is the counter set that tells them apart.

// Instrumentation for a device memory manager. `hits` and `migrations` are
// counted separately so the hit rate is computable, and migrated bytes are
// accumulated so the traffic caused by NOT fitting is directly visible.
module mem_stats #(
  parameter int unsigned CNT_W   = 32,
  parameter int unsigned BLK_KIB = 256
) (
  input  logic clk,
  input  logic rst_n,
  input  logic acc_valid,
  input  logic hit,
  input  logic need_migrate,
  input  logic dup_suppressed,
  input  logic wait_cycle,           // engine blocked waiting for a migration
  output logic [CNT_W-1:0] accesses_q,
  output logic [CNT_W-1:0] hits_q,
  output logic [CNT_W-1:0] migrations_q,
  output logic [CNT_W-1:0] suppressed_q,
  output logic [CNT_W-1:0] wait_cycles_q,
  output logic [CNT_W-1:0] migrated_kib_q
);
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      accesses_q <= '0; hits_q <= '0; migrations_q <= '0;
      suppressed_q <= '0; wait_cycles_q <= '0; migrated_kib_q <= '0;
    end else begin
      if (acc_valid)      accesses_q    <= accesses_q + 1'b1;
      if (hit)            hits_q        <= hits_q + 1'b1;
      if (dup_suppressed) suppressed_q  <= suppressed_q + 1'b1;
      if (wait_cycle)     wait_cycles_q <= wait_cycles_q + 1'b1;
      if (need_migrate) begin
        migrations_q   <= migrations_q + 1'b1;
        migrated_kib_q <= migrated_kib_q + CNT_W'(BLK_KIB);
      end
    end
  end
endmodule

The measurement that matters

Same access pattern, same hardware, two working-set sizes — the run from Section 7, reproduced here as the instrumentation output it actually is:

Read the last row against the second-to-last. Six times the traffic, for a quarter of the hit rate, on the same code. A performance engineer looking only at link utilisation sees a busy link and concludes the link is the bottleneck. The hit rate is what says otherwise, and the hit rate requires the tracker to be instrumented.

Contract and failure shape

Counters advance only on the events they name — a migration counted when it is requested rather than when it is issued over-reports if requests can be suppressed, which is why suppressed_q exists as a separate counter rather than being folded into either bucket.

13. Assertions

Bind-ready properties. Icarus does not support concurrent assertions, so these were not executed; the mapping table gives the procedural check that verified each.

// C1 — reservations never exceed capacity. The single most important property
// in the manager: violating it means work was admitted that cannot run.
a_no_overcommit: assert property (@(posedge clk) disable iff (!rst_n)
  reserved_q <= CAP_MIB);
 
// C2 — admission is genuinely gated, not advisory.
a_admit_fits: assert property (@(posedge clk) disable iff (!rst_n)
  adm_grant |-> (adm_size_mib <= headroom));
 
// R1 — residency never exceeds what local memory can hold.
a_resident_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  resident_cnt_q <= RESIDENT_MAX);
 
// R2 — no duplicate migration for a block already on its way. This is the
// property that pays for the PENDING state.
a_no_dup_migrate: assert property (@(posedge clk) disable iff (!rst_n)
  need_migrate |-> (state_q[acc_blk] != PENDING));
 
// R3 — a completion arrives only for a block that was requested.
a_done_was_pending: assert property (@(posedge clk) disable iff (!rst_n)
  mig_done_valid |-> (state_q[mig_done_blk] == PENDING));
 
// R4 — a block being evicted must currently be resident, never pending.
// Evicting a block whose migration is in flight loses the data outright.
a_evict_resident_only: assert property (@(posedge clk) disable iff (!rst_n)
  evict_valid |-> (state_q[evict_blk] == RESIDENT));

One finding from writing the testbench is worth recording. An early version completed a migration in the same cycle it was requested, and bad_done_err fired — correctly, because the block is only marked PENDING on that edge. The RTL was right and the testbench was wrong. That is the useful shape of R3: it catches a stimulus model that does not respect the design's own timing, which is a very common source of false failures in migration verification.

14. Debug Lab

Each failure was produced by injecting the bug into the real module and re-running the testbench. The output is actual simulator output.

// A work unit fits if it is no larger than the device's memory.
assign adm_grant = adm_req && (adm_size_mib <= SIZE_W'(CAP_MIB));

=== BUG 1: admission compares against CAPACITY, not headroom ===
  after admit 20: reserved=20
  after admit 20: reserved=40
  after admit 20: reserved=60  <-- 60 units reserved in a 48-unit memory
  overcommit_err=1

assign headroom  = SIZE_W'(CAP_MIB) - reserved_q;
assign adm_grant = adm_req && (adm_size_mib <= headroom);

// Anything not resident needs to be brought in.
assign need_migrate = acc_valid && (state_q[acc_blk] != RESIDENT);

=== BUG 2: no PENDING state, so a pending block is re-requested ===
  migrations issued for ONE block while it was in flight: 5
  (a correct tracker issues 1 and suppresses the other 4)

assign need_migrate   = acc_valid && (state_q[acc_blk] == ABSENT);
assign dup_suppressed = acc_valid && (state_q[acc_blk] == PENDING);

// Make room: evict whatever the victim pointer indicates.
assign evict_valid = mig_done_valid && (resident_cnt_q >= RESIDENT_MAX);
assign evict_blk   = victim_q;                    // no state check

assign evict_valid = mig_done_valid
                     && (resident_cnt_q >= RESIDENT_MAX)
                     && (state_q[victim_q] == RESIDENT);

15. Diagnosing Capacity Against Bandwidth

The two failures share a symptom — the device is not delivering what its specification suggests — and diverge on every counter that matters. The two right-hand columns below read bandwidth-bound and capacity-bound.

The two rows at the bottom are the cheapest experiments available and they are close to decisive.

Reduce the working set and watch the cliff. Lower precision, a smaller batch, or a shorter sequence reduces the footprint. If throughput improves far more than the arithmetic reduction accounts for, the system was capacity-bound and you have just moved it under the threshold. If it improves roughly in proportion, it was compute or bandwidth bound.

Watch the hit rate, not the link. A busy link is the consequence of a capacity failure, not its cause. Section 12's measurement is the reason instrumenting residency matters: 89% against 16% on identical code is unambiguous, and link utilisation alone would have shown "busy" in both cases.

16. How This Appears in Real Engineering

Accelerator architect

The capacity-versus-bandwidth trade is the defining decision of the memory subsystem, and it is made once. How much memory fits within reach, at what bandwidth, and what is the intended working-set envelope? What happens to parts whose customers exceed that envelope — is there a migration path, and is it fast enough to be useful or merely fast enough to be disappointing? The uncomfortable question is whether the part is sized for the workloads it will run in three years or the ones it was benchmarked on.

System architect

Whether a node's aggregate device memory is usable for the target workload, which is a partitioning question rather than an arithmetic one. Whether the host link is provisioned for the migration traffic that an oversubscribed device will generate. And whether buying devices for capacity — with all the compute and power that implies — is the cheapest way to obtain capacity, which is the question the rest of this track exists to reopen.

RTL and microarchitecture engineer

The three structures in Sections 10 to 12: admission against headroom, residency with a pending state, eviction that respects it. The recurring theme is that each needs one more state than the naive design has — headroom rather than capacity, three states rather than two, an eviction check rather than a bare pointer — and that every Debug Lab in this chapter is the missing one.

Verification engineer

The interesting stimulus is concurrency and pressure, not correctness of a single migration. Multiple reservations outstanding simultaneously. Repeated access to a block during its migration window. Eviction pressure with migrations in flight. Completion arriving in the same cycle as the request, which should be rejected. Reset with pending migrations. And working sets deliberately set to just under, exactly at, and just over capacity — the behaviour changes character at that boundary and the boundary is where the bugs live.

Performance engineer

Hit rate is the headline number and it is frequently not instrumented. Migrated bytes per unit of useful work exposes the refetch multiplier — Chapter 1.3's amplification ratio applied to residency. Stall cycles attributed to migration waits separate "the device is slow" from "the device is waiting". And the experiment that settles the argument is reducing the footprint and watching whether the response is proportional or a cliff.

Machine-learning systems engineer

Capacity allocation is a serving decision: batch size, sequence length, concurrency and precision all consume the same finite resource, and they trade against each other. Knowing which term dominates for a given deployment is what makes the difference between tuning that works and tuning that moves the problem. The hardware constraint is fixed; the software choices that fill it are not.

17. Common Misconceptions

18. Interview Reasoning

19. Summary

Device-local memory is fast because it is close, and small for the same reason. That single trade produces every difficulty in this chapter.

Capacity and bandwidth are independent failures. Bandwidth-bound means resident data cannot be supplied fast enough; capacity-bound means the data cannot be resident. The first degrades continuously; the second is a cliff, and its most visible symptom — a saturated host link — is easily misread as the first. The distinguishing measurement is the residency hit rate, and the distinguishing experiment is to shrink the working set and see whether the response is proportional or a step change.

The arithmetic is unglamorous and decisive. Parameters at two bytes each against a fixed capacity either fit or do not, and only two interventions change the answer: reduce bytes per element, or partition across devices. More bandwidth, more link, and more compute do not. When the working set does not fit, traffic stops scaling with the data the algorithm needs and starts scaling with how often blocks are refetched — measured here as 89% hit rate and 768 KiB against 16% and 5,120 KiB, on identical code.

Aggregate capacity across devices is a sum, not an address space. Whether it is usable depends on whether the problem partitions and on what crossing the boundary costs, and buying devices to obtain capacity means buying compute and power to solve a memory problem.

In hardware, each of the three structures needs exactly one more state than the naive version: admission against headroom rather than capacity, residency with a pending state rather than resident-or-not, and eviction that checks that state rather than sweeping a pointer. Every Debug Lab in this chapter is the missing one, and each is silent under directed testing and loud under pressure.

The durable form: bandwidth decides how fast resident data moves; capacity decides whether it is resident. Only one of those is fixed by a faster memory.

20. What Comes Next

Chapters 1.4 and 1.5 have established the shape of the problem from the device's side: engines proliferate, each acquires memory close to it, and that memory becomes a bounded island whose overflow is expensive.

Everything so far has assumed the device attaches the way devices have always attached — as an I/O endpoint that the host configures and that moves data into and out of host memory. Chapter 1.6 examines that attachment model directly: what it does extremely well, and which of the requirements accumulated across Module 1 sit outside it. The answer turns out to be about semantics rather than bandwidth, which is why more link speed has not resolved any of it.

Chapter 1.7 then makes the case for the alternative, and Module 2 finally names it.

For related material, the PCIe track covers accelerator attach and copy-compute overlap, host memory access and DMA engines; Chapter 1.1 has the capacity and bandwidth axes in full. The path is on the CXL tutorials index.