Memory Expansion for AI

22.1 §9 ended on a cliff: a working set larger than the accelerator's cache blends to 205 ns against 10 — a factor of twenty — and the boundary is exact.

This chapter is about where that boundary actually is. The working set is larger than almost everyone's first estimate, because a model's parameters are one term of four and the other three grow with the batch rather than with the model. And when it does not fit, CXL-attached DRAM is the answer that adds capacity on a link whose bandwidth does not grow with the capacity behind it.

1. The Engineering Problem — Capacity Is Not The Only Axis

Parameters are 20% of a training footprint. Optimiser state, activations and gradients are the rest, and a sizing taken from a model card undersizes by a factor of five. Section 5.

The far tier is further. A footprint twice the local memory blends to 200 ns against 100 — a 100% penalty — and a flat model reports the local number. Section 6.

Capacity scales and bandwidth does not. Eight modules behind one link give twice the gigabytes and half the bandwidth per gigabyte of four. Section 7.

Which bytes go far decides everything. A placer that fills local memory with cold data sends 90% of accesses to the far tier. Section 8.

Expansion helps a model that did not fit and hurts one that did. Section 9.

And the batch is a memory decision. Activation memory scales with batch size, so the largest batch that fits is arithmetic rather than tuning. Section 11.

This chapter against 20.2, stated precisely. That one owns pooled capacity as a fleet resource. This one owns one node's memory hierarchy under an AI workload — and section 15 shows an expansion where the footprint fits and nothing got faster.

2. The One-Sentence Model

Memory expansion helps an AI workload when the footprint fits, every term of that footprint was counted, the link carries the bandwidth the workload wants, hot data stays local, page migration pays for itself, and CXL capacity is not priced as bare DRAM — and every defect below is an expansion where the capacity fits and the workload is slower.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real deployment is a framework allocator, a NUMA policy, a page-migration daemon, a memory controller and a capacity planner, and none of that is reproduced. What is reproduced is the arithmetic each of them has to get right.

Each model is built twice — a correct build and a broken build selected by a parameter. The broken builds here are all the flat-memory model: one latency, one bandwidth, capacity that is just a number. Every one was accurate when a machine had one kind of memory, and every one stops being accurate the moment it has two.

Figure 1 — The dashed path is section 5's failure: a capacity decision made from the one term a model card publishes. The two tiers on the right are section 6, and which bytes land in which is section 8.

5. RTL 1 — Parameters Are One Term Of Four

// RTL 1 - what a model actually needs in memory. Parameters are one term of
// several, and the others grow with batch size rather than with the model.
module working_set #(parameter int PARAMS_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        size_it,
  input  logic [15:0] params_gb, optimiser_gb, activations_gb, gradients_gb,
  output logic [15:0] total_gb, params_share_pct,
  output logic        params_dominate,
  output logic [7:0]  n_sizings, n_underestimated,
  output logic        undersize_err
);
  logic [15:0] true_total;
  logic [31:0] s_q;
  assign true_total = params_gb + optimiser_gb + activations_gb + gradients_gb;
  // Sizing from the parameter count alone is what a model card invites.
  assign total_gb = (PARAMS_ONLY != 0) ? params_gb : true_total;
  assign s_q = (true_total == 16'd0) ? 32'd0
             : (({16'd0, params_gb} * 32'd100) / {16'd0, true_total});
  assign params_share_pct = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  assign params_dominate = (params_share_pct >= 16'd50);
  // A memory sizing that omits terms the workload will actually allocate.
  assign undersize_err = size_it && (total_gb < true_total);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_sizings <= 8'd0; n_underestimated <= 8'd0;
    end else if (size_it) begin
      n_sizings <= n_sizings + 8'd1;
      if (total_gb < true_total) n_underestimated <= n_underestimated + 8'd1;
    end
  end
endmodule

Five sizings.

Three sizings undersized by the params-only model.

Row one is the number that surprises people, and it is not close. A 70-billion-parameter model at two bytes per parameter is 140 GB of weights — and training it needs optimiser state at roughly twice that, plus activations and gradients. The parameters are a fifth of what has to be resident, and the four-fifths that is not is exactly the part a model card never mentions.

Row two is why the same model is a completely different sizing problem for inference. No optimiser, no gradients — 260 GB instead of 680, and parameters now genuinely dominate at 53%. The same weights, the same hardware, and a 2.6x difference in what has to fit.

Why the broken build is not a strawman. The parameter count is the number that is published, cited and compared. It is the one figure everyone has, and using it as a memory sizing is not carelessness — it is using the only input available. The fix is not a better estimate; it is three more measurements.

And the three missing terms do not scale with the one that is published. Optimiser state scales with the parameters, but activations and gradients scale with the batch — so a sizing derived from the model alone is not merely low by a constant factor, it is low by an amount that changes with a runtime setting. There is no multiplier that converts a parameter count into a footprint, which is why section 11 treats the batch as a separate model rather than a term in this one.

6. RTL 2 — The Far Tier Is Further

// RTL 2 - the tier. CXL-attached DRAM is further away than local DRAM, and the
// blended latency is what the workload actually sees.
module memory_tier #(parameter int IGNORE_TIER_COST = 0) (
  input  logic clk, rst_n,
  input  logic        access,
  input  logic [15:0] local_gb, cxl_gb, footprint_gb,
  input  logic [15:0] local_ns, cxl_ns,
  output logic [15:0] local_share_pct, blended_ns, penalty_pct,
  output logic        fits_local,
  output logic [7:0]  n_accesses, n_spilled,
  output logic        flat_model_err
);
  logic [31:0] l_q, b_q, p_q;
  logic [15:0] resident_local;
  assign fits_local = (footprint_gb <= local_gb);
  assign resident_local = fits_local ? footprint_gb : local_gb;
  assign l_q = (footprint_gb == 16'd0) ? 32'd100
             : (({16'd0, resident_local} * 32'd100) / {16'd0, footprint_gb});
  assign local_share_pct = (l_q > 32'd65535) ? 16'hFFFF : l_q[15:0];
  assign b_q = (IGNORE_TIER_COST != 0) ? {16'd0, local_ns}
             : ((({16'd0, local_ns} * {16'd0, local_share_pct})
               + ({16'd0, cxl_ns} * (32'd100 - {16'd0, local_share_pct}))) / 32'd100);
  assign blended_ns = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  assign p_q = ((local_ns == 16'd0) || (blended_ns <= local_ns)) ? 32'd0
             : ((({16'd0, blended_ns} - {16'd0, local_ns}) * 32'd100) / {16'd0, local_ns});
  assign penalty_pct = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  // A spilled footprint costed at the local latency.
  assign flat_model_err = access && !fits_local && (blended_ns == local_ns);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_accesses <= 8'd0; n_spilled <= 8'd0;
    end else if (access) begin
      n_accesses <= n_accesses + 8'd1;
      if (!fits_local) n_spilled <= n_spilled + 8'd1;
    end
  end
endmodule

Five windows. 512 GB local at 100 ns, CXL at 300 ns.

Three spilled windows; the flat model reported the local latency on all three.

A footprint twice the local memory costs twice the latency, and the arithmetic is a straight blend — half the accesses at 100 ns, half at 300. The penalty grows sub-linearly after that: three times the footprint is a 134% penalty rather than 200%, because the local share falls from a half to a third rather than to zero.

Row five is not hypothetical and it is why the floor exists. A machine with a slow local tier and fast CXL-attached memory blends to better than local, and without the floor the penalty calculation underflows. Section 18 records that this case was missing until a mutation asked for it.

7. RTL 3 — Capacity Scales, Bandwidth Does Not

// RTL 3 - capacity against bandwidth. CXL-attached DRAM adds capacity on a link
// whose bandwidth does not grow with the capacity behind it.
module capacity_bandwidth #(parameter int SCALE_BW_WITH_CAP = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [15:0] modules, gb_per_module, link_gbps, demand_gbps,
  output logic [15:0] total_gb, available_gbps, gbps_per_gb,
  output logic        bandwidth_met,
  output logic [7:0]  n_assess, n_starved,
  output logic        bw_overclaim_err
);
  logic [31:0] c_q, b_q, r_q;
  assign c_q = {16'd0, modules} * {16'd0, gb_per_module};
  assign total_gb = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  // The link is one link however much capacity hangs off it.
  assign b_q = (SCALE_BW_WITH_CAP != 0) ? ({16'd0, link_gbps} * {16'd0, modules})
                                        : {16'd0, link_gbps};
  assign available_gbps = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  assign r_q = (total_gb == 16'd0) ? 32'd0
             : (({16'd0, available_gbps} * 32'd100) / {16'd0, total_gb});
  assign gbps_per_gb = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  assign bandwidth_met = (available_gbps >= demand_gbps);
  // Claiming bandwidth that scales with the capacity behind one link.
  assign bw_overclaim_err = assess && (available_gbps > link_gbps);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assess <= 8'd0; n_starved <= 8'd0;
    end else if (assess) begin
      n_assess <= n_assess + 8'd1;
      if (!bandwidth_met) n_starved <= n_starved + 8'd1;
    end
  end
endmodule

Five assessments. 128 GB modules on a 400 Gbps link.

One bandwidth-starved assessment, and the scaling model overclaimed on every multi-module case.

Doubling the capacity halves the bandwidth per gigabyte, exactly. That is the shape of the whole trade: CXL adds capacity behind a link, and the link does not widen because more DRAM appeared behind it. A workload that is capacity-bound is helped; a workload that is bandwidth-bound is helped by nothing at all, and 22.3 is where that distinction becomes the whole chapter.

Row four is the sanity case that makes the broken build a strawman only at scale. At one module the two models agree — 400 Gbps either way — so a single-module prototype validates a capacity plan that is wrong by 4x at four modules and 8x at eight.

This is the sharpest test in the chapter for whether expansion is the right tool. A workload bound by capacity gets everything it wanted; a workload bound by bandwidth gets nothing and then gets worse, because each additional module dilutes the per-gigabyte rate further. The two look identical on a capacity plan and opposite in production, and the counter that separates them is link utilisation rather than resident bytes.

8. RTL 4 — Which Bytes Go Far

// RTL 4 - placement. Not every byte of a working set is touched equally, so
// which bytes go to the far tier decides the blended cost.
module tier_placement #(parameter int PLACE_BLINDLY = 0) (
  input  logic clk, rst_n,
  input  logic        place,
  input  logic [15:0] hot_gb, cold_gb, local_gb,
  input  logic [7:0]  hot_access_pct,
  output logic [15:0] hot_local_gb, hot_far_gb, far_access_pct,
  output logic        placed_well,
  output logic [7:0]  n_placements, n_poor,
  output logic        blind_placement_err
);
  logic [15:0] cold_local;
  // A blind placer fills local memory with whatever comes first, which on a
  // large cold region is cold data.
  assign cold_local = (PLACE_BLINDLY != 0)
                      ? ((cold_gb > local_gb) ? local_gb : cold_gb) : 16'd0;
  assign hot_local_gb = ((local_gb - cold_local) > hot_gb) ? hot_gb
                                                           : (local_gb - cold_local);
  // No floor is needed: hot_local_gb is a minimum against hot_gb, so it can
  // never exceed it and the subtraction cannot underflow.
  assign hot_far_gb = hot_gb - hot_local_gb;
  // The share of accesses that reach the far tier: hot bytes that spilled carry
  // the hot access rate with them.
  assign far_access_pct = (hot_gb == 16'd0) ? 8'd0
                        : (({8'd0, hot_access_pct} * hot_far_gb) / hot_gb);
  assign placed_well = (hot_far_gb == 16'd0);
  // Hot data placed in the far tier while local memory holds cold data.
  assign blind_placement_err = place && (hot_far_gb != 16'd0) && (cold_local != 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_placements <= 8'd0; n_poor <= 8'd0;
    end else if (place) begin
      n_placements <= n_placements + 8'd1;
      if (!placed_well) n_poor <= n_poor + 8'd1;
    end
  end
endmodule

Four placements. 128 GB hot at a 90% access rate, 512 GB cold, 256 GB local.

One poor placement in the correct placer, two in the blind one.

Row one is the entire argument for placement awareness. The same hardware, the same working set, the same local capacity — and 90% of accesses going far instead of none, because local memory was filled with the cold 512 GB before the hot 128 GB was considered.

Row two is the honest limit of good placement. With only 64 GB local and 128 GB hot, half the hot data spills whatever the placer does, and 45% of accesses go far — not a placement failure, a capacity one. The distinction matters because the two have different fixes, and blind_placement_err requires cold data to be occupying local memory for exactly that reason.

9. RTL 5 — Expansion Helps What Did Not Fit

// RTL 5 - what expansion buys. A model that did not fit runs; a model that fitted
// runs slower. The gain is the first case and the loss is the second.
module expansion_value #(parameter int ASSUME_ALL_BENEFIT = 0) (
  input  logic clk, rst_n,
  input  logic        eval,
  input  logic [15:0] footprint_gb, local_gb, expanded_gb,
  input  logic [15:0] blended_ns, local_ns,
  output logic        ran_before, runs_after, newly_possible,
  output logic [15:0] slowdown_pct,
  output logic        worth_expanding,
  output logic [7:0]  n_evals, n_worth,
  output logic        false_value_err
);
  logic [31:0] s_q;
  assign ran_before = (footprint_gb <= local_gb);
  assign runs_after = (footprint_gb <= expanded_gb);
  assign newly_possible = runs_after && !ran_before;
  assign s_q = ((local_ns == 16'd0) || (blended_ns <= local_ns)) ? 32'd0
             : ((({16'd0, blended_ns} - {16'd0, local_ns}) * 32'd100) / {16'd0, local_ns});
  assign slowdown_pct = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  // Expansion is worth it when it makes something possible, not when it makes
  // something that already ran slower.
  assign worth_expanding = (ASSUME_ALL_BENEFIT != 0) ? runs_after : newly_possible;
  // Claiming value on a model that already fitted and is now slower.
  assign false_value_err = eval && worth_expanding && ran_before
                           && (slowdown_pct != 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_worth <= 8'd0;
    end else if (eval) begin
      n_evals <= n_evals + 8'd1;
      if (worth_expanding) n_worth <= n_worth + 8'd1;
    end
  end
endmodule

Seven evaluations. 512 GB local, 1536 GB expanded, a 220 ns blend against 100 ns local.

Two evaluations worth expanding; the assuming model claimed value twice.

Row one and row two are the same expansion and opposite verdicts. A model that could not run now runs, at a 120% slowdown — an infinite improvement over not running. A model that already fitted now runs 120% slower for no benefit whatsoever, because its data has been spread across a tier it did not need.

That is the failure mode of an indiscriminate expansion, and it is why worth_expanding is newly_possible rather than runs_after. Adding CXL memory to a node and letting the allocator use it for everything makes every workload on that node slower except the ones that could not run.

10. Waveform — A Working Set Spilling Into The Far Tier

The blended row rises smoothly and that is the misleading part. There is no cliff, no error and no event — the workload simply gets slower as it grows, at a rate a flat model has no term for. The only observable is the gap between two rows, and one of them is a model rather than a measurement.

11. RTL 6 — The Batch Is A Memory Decision

// RTL 6 - the batch. Activation memory scales with batch size, so the largest
// batch that fits is a memory question rather than a throughput one.
module batch_sizing #(parameter int IGNORE_ACTIVATIONS = 0) (
  input  logic clk, rst_n,
  input  logic        size_it,
  input  logic [15:0] fixed_gb, act_gb_per_sample, batch, capacity_gb,
  output logic [15:0] act_gb, total_gb, max_batch,
  output logic        fits,
  output logic [7:0]  n_sizings, n_oom,
  output logic        oom_err
);
  logic [31:0] a_q, m_q;
  assign a_q = {16'd0, act_gb_per_sample} * {16'd0, batch};
  assign act_gb = (IGNORE_ACTIVATIONS != 0) ? 16'd0
                : ((a_q > 32'd65535) ? 16'hFFFF : a_q[15:0]);
  assign total_gb = fixed_gb + act_gb;
  assign fits = (total_gb <= capacity_gb);
  // The largest batch whose activations fit what is left after the fixed cost.
  assign m_q = (act_gb_per_sample == 16'd0) ? 32'd65535
             // The '=' half of this comparison is behaviourally redundant:
             // at equality the subtraction below yields zero anyway. It is kept
             // because it states the intent, not because it changes the result.
             : ((capacity_gb <= fixed_gb) ? 32'd0
               : (({16'd0, capacity_gb} - {16'd0, fixed_gb}) / {16'd0, act_gb_per_sample}));
  assign max_batch = (m_q > 32'd65535) ? 16'hFFFF : m_q[15:0];
  // A batch that will not fit, sized as if it would.
  assign oom_err = size_it && (batch > max_batch) && fits;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_sizings <= 8'd0; n_oom <= 8'd0;
    end else if (size_it) begin
      n_sizings <= n_sizings + 8'd1;
      if (batch > max_batch) n_oom <= n_oom + 8'd1;
    end
  end
endmodule

Six sizings. 400 GB fixed, 8 GB per sample, 600 GB capacity.

Three sizings exceeded the largest batch; the ignore-activations model called two of them fitting.

Batch size is the only knob most practitioners have, and it is a memory knob before it is a throughput knob. 25 samples fit and 26 do not, exactly, because activations are linear in the batch. The out-of-memory that terminates a training run at hour nine is this arithmetic, and it is fully predictable before the run starts.

Row four and row six are the two ways there is no room at all: capacity below the fixed cost, and capacity exactly equal to it. Both give a maximum batch of zero, and section 18 records that the second was the one that made a guard's = half provably redundant.

12. RTL 7 — Migration Has To Pay For Itself

// RTL 7 - migration. A tiered system that moves pages between tiers pays for the
// move, and a page that moves back is a move that bought nothing.
module page_migration #(parameter int IGNORE_MIGRATION = 0) (
  input  logic clk, rst_n,
  input  logic        window,
  input  logic [15:0] promotions, demotions, cost_us_each,
  input  logic [15:0] saved_us_per_promotion,
  output logic [15:0] migrations, migration_us, true_migration_us, benefit_us,
  output logic        net_positive,
  output logic [7:0]  n_windows, n_negative,
  output logic        thrash_err
);
  logic [31:0] m_q, c_q, b_q, t_q;
  assign m_q = {16'd0, promotions} + {16'd0, demotions};
  assign migrations = (m_q > 32'd65535) ? 16'hFFFF : m_q[15:0];
  assign c_q = (IGNORE_MIGRATION != 0) ? 32'd0
             : ({16'd0, migrations} * {16'd0, cost_us_each});
  assign migration_us = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  // What the migrations really cost, against what the model chose to charge.
  assign t_q = {16'd0, migrations} * {16'd0, cost_us_each};
  assign true_migration_us = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign b_q = {16'd0, promotions} * {16'd0, saved_us_per_promotion};
  assign benefit_us = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  assign net_positive = (benefit_us >= migration_us);
  // A migration policy declared positive whose real cost exceeds its benefit.
  assign thrash_err = window && net_positive && (true_migration_us > benefit_us);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_windows <= 8'd0; n_negative <= 8'd0;
    end else if (window) begin
      n_windows <= n_windows + 8'd1;
      if (benefit_us < true_migration_us) n_negative <= n_negative + 8'd1;
    end
  end
endmodule

Four windows. 5 µs per migration.

One net-negative window; the ignore-migration model called it positive.

Row two is thrashing, and the signature is demotions matching promotions. A page promoted to local memory and demoted back has been moved twice and bought nothing — and a policy doing that at scale spends five times what it saves while every individual decision looks locally reasonable.

Only promotions carry a benefit and every migration carries a cost, which is the asymmetry the model exists to express. A demotion is pure overhead: it is the system admitting a previous promotion was wrong.

Figure 3 — Both allocators place the same 640 GB into the same 256 GB of local memory. The difference is which 256 GB, and it is worth 90% of the accesses against none — a decision no capacity counter can see.

13. RTL 8 — What The Capacity Costs

// RTL 8 - what expansion costs per gigabyte. Local DIMM slots run out, and past
// that point capacity costs a CXL module, a link and its power.
module expansion_economics #(parameter int COUNT_DRAM_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        cost_it,
  input  logic [15:0] gb_wanted, slots_free, gb_per_dimm,
  input  logic [15:0] dram_cost_per_gb, cxl_overhead_per_gb,
  output logic [15:0] local_gb, cxl_gb, cxl_cost, total_cost, cost_per_gb,
  output logic        all_local,
  output logic [7:0]  n_costings, n_needs_cxl,
  output logic        undercost_err
);
  logic [31:0] lc_q, cc_q, pg_q;
  logic [15:0] local_capacity, bare_dram_cost;
  assign local_capacity = slots_free * gb_per_dimm;
  assign local_gb = (gb_wanted > local_capacity) ? local_capacity : gb_wanted;
  // No floor is needed: local_gb is a minimum against gb_wanted, so it can
  // never exceed it and the subtraction cannot underflow.
  assign cxl_gb = gb_wanted - local_gb;
  assign all_local = (cxl_gb == 16'd0);
  // CXL capacity costs the DRAM plus the module, the link and its power.
  assign lc_q = {16'd0, local_gb} * {16'd0, dram_cost_per_gb};
  assign cc_q = (COUNT_DRAM_ONLY != 0)
              ? ({16'd0, cxl_gb} * {16'd0, dram_cost_per_gb})
              : ({16'd0, cxl_gb} * ({16'd0, dram_cost_per_gb} + {16'd0, cxl_overhead_per_gb}));
  assign cxl_cost = (cc_q > 32'd65535) ? 16'hFFFF : cc_q[15:0];
  assign total_cost = ((lc_q + cc_q) > 32'd65535) ? 16'hFFFF : (lc_q + cc_q);
  assign pg_q = (gb_wanted == 16'd0) ? 32'd0
              : (({16'd0, total_cost} * 32'd100) / {16'd0, gb_wanted});
  assign cost_per_gb = (pg_q > 32'd65535) ? 16'hFFFF : pg_q[15:0];
  assign bare_dram_cost = (({16'd0, cxl_gb} * {16'd0, dram_cost_per_gb}) > 32'd65535)
                          ? 16'hFFFF : (({16'd0, cxl_gb} * {16'd0, dram_cost_per_gb}));
  assign undercost_err = cost_it && (cxl_gb != 16'd0)
                         && (cxl_cost == bare_dram_cost)
                         && (cxl_overhead_per_gb != 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_costings <= 8'd0; n_needs_cxl <= 8'd0;
    end else if (cost_it) begin
      n_costings <= n_costings + 8'd1;
      if (!all_local) n_needs_cxl <= n_needs_cxl + 8'd1;
    end
  end
endmodule

Four costings. Four free slots of 64 GB, 10 per GB of DRAM, 4 per GB of CXL overhead.

Three costings needed CXL capacity; the DRAM-only model undercosted two.

Row three is where the cost curve steps. 256 GB is all local; 257 GB needs a CXL module, a link, a slot for it and its power — for one gigabyte. The marginal cost of the 257th gigabyte is not 14, it is the whole module, and the per-gigabyte figure only makes sense once a module is amortised.

Row four is the honest bound. With no CXL overhead the two models agree exactly, so the DRAM-only model is not wrong in principle — it is wrong by the size of the overhead, which is a number somebody has to supply.

14. RTL 9 — The Alternative Is More Nodes

// RTL 9 - the alternative. Capacity that does not fit one node can be split
// across nodes instead, and that has its own cost.
module scale_out_compare #(parameter int IGNORE_COMM = 0) (
  input  logic clk, rst_n,
  input  logic        compare,
  input  logic [15:0] footprint_gb, node_gb,
  input  logic [15:0] expand_ns, comm_ns_per_step,
  input  logic [7:0]  remote_share_pct,
  output logic [15:0] nodes_needed, scaleout_ns, expand_blend_ns,
  output logic        expansion_better,
  output logic [7:0]  n_compares, n_scaleout,
  output logic        comm_ignored_err
);
  logic [31:0] n_q, s_q;
  assign n_q = (node_gb == 16'd0) ? 32'd0
             : (({16'd0, footprint_gb} + {16'd0, node_gb} - 32'd1) / {16'd0, node_gb});
  assign nodes_needed = (n_q > 32'd65535) ? 16'hFFFF : n_q[15:0];
  // Splitting across nodes adds communication on the remote share of accesses.
  assign s_q = (IGNORE_COMM != 0) ? 32'd0
             : (({16'd0, comm_ns_per_step} * {24'd0, remote_share_pct}) / 32'd100);
  assign scaleout_ns = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  assign expand_blend_ns = expand_ns;
  assign expansion_better = (expand_blend_ns <= scaleout_ns) || (nodes_needed <= 16'd1);
  // A scale-out plan costed with no communication at all.
  assign comm_ignored_err = compare && (scaleout_ns == 16'd0)
                            && (remote_share_pct != 8'd0) && (comm_ns_per_step != 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_compares <= 8'd0; n_scaleout <= 8'd0;
    end else if (compare) begin
      n_compares <= n_compares + 8'd1;
      if (!expansion_better) n_scaleout <= n_scaleout + 8'd1;
    end
  end
endmodule

Five comparisons. 512 GB nodes, a 400 ns communication step, a 220 ns expanded blend.

Two comparisons favoured scaling out; the ignore-communication model called it free on four.

The remote share is the variable and it is a property of the partitioning. A model split so that most accesses stay node-local pays little for the split; one split so that most accesses cross pays 320 ns and expansion wins. The same footprint, the same nodes, and opposite answers depending on how the model was partitioned — which is 22.4's subject.

Row two is why expansion_better carries the single-node exemption. A footprint that fits one node has nothing to split, so the comparison is vacuous and expansion is trivially the answer. Without the exemption the model would recommend scaling out a workload that already fits.

Figure 4 — The first gate is the one that surprises: a workload that already fits is made worse by expansion, so the correct answer for it is to do nothing. Every other path that does not reach the bottom-left produces a node with more memory and no more throughput.

15. RTL 10 — Expansion Assembled

// RTL 10 - memory expansion assembled. Everything that must hold before adding
// CXL-attached DRAM makes an AI workload better rather than merely larger.
module expansion_model #(parameter int CAPACITY_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       capacity_sufficient, // the footprint fits
  input  logic       footprint_complete,  // every term is in the sizing
  input  logic       bandwidth_sized,     // the link carries the demand
  input  logic       placement_aware,     // hot data stays local
  input  logic       migration_positive,  // moving pages pays for itself
  input  logic       cost_accounted,      // CXL capacity is not DRAM-priced
  output logic       expansion_helps,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_helps,
  output logic       false_help_err
);
  assign fail_mask[0] = ~capacity_sufficient;
  assign fail_mask[1] = ~footprint_complete;
  assign fail_mask[2] = ~bandwidth_sized;
  assign fail_mask[3] = ~placement_aware;
  assign fail_mask[4] = ~migration_positive;
  assign fail_mask[5] = ~cost_accounted;
  // The capacity-only build checks that the footprint fits and calls expansion a
  // win, which is what a capacity spreadsheet does.
  assign expansion_helps = (CAPACITY_ONLY != 0) ? capacity_sufficient
                                                : (fail_mask == 6'd0);
  assign false_help_err = evaluate && expansion_helps && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_helps <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (expansion_helps) n_helps <= n_helps + 8'd1;
    end
  end
endmodule

One helpful expansion of six, and four false claims.

The capacity-only definition is a spreadsheet, and it is right about exactly one of the six. Row two is the one that ships: the footprint fits because it was sized from the parameter count, and the four-fifths that was not counted does not fit at all.

16. Quantitative Reasoning

Working set. 140 GB of parameters in a 680 GB training footprint — 20%. The same weights for inference need 260 GB, and parameters then dominate at 53%.

Tier blend. A footprint twice the local memory: 200 ns against 100, a 100% penalty. Three times: 234 ns and 134%.

Capacity against bandwidth. Four 128 GB modules on a 400 Gbps link give 0.78 Gbps per GB; eight give 1024 GB and 0.39 — half.

Placement. 128 GB hot at a 90% access rate: correct placement sends 0% of accesses far, a blind one sends 90%.

Value. An 800 GB model that could not run now runs at a 120% slowdown; a 256 GB model that already ran is 120% slower for nothing.

Batch. 400 GB fixed and 8 GB per sample in 600 GB: the largest batch is 25, exactly, and 32 needs 656 GB.

Migration. 100 promotions and 100 demotions at 5 µs each saving 2 µs: 1000 µs spent, 200 µs saved — five times its benefit.

Economics. 512 GB with four 64 GB slots free: 256 local and 256 CXL, and the CXL half costs 3584 against a bare-DRAM 2560. The 257th gigabyte needs a whole module.

Scale-out. 800 GB across two 512 GB nodes at a 30% remote share costs 120 ns against a 220 ns expanded blend — scaling out wins. At an 80% remote share it costs 320 and expansion wins.

The assembled model. Six properties, six configurations, one helps. The capacity-only definition reported five.

17. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Working set. The parameters-alone case is asserted as correct for the params-only model, which is what stops the model being a strawman.

chk(pTg == 16'd140,  "so the params-only model is right");
chk(pUe == 1'b0,     "and undersizes nothing");

The dominance threshold is driven at exactly 50%.

Tier. The faster-far-tier case is asserted to floor rather than wrap.

chk(tBn == 16'd75, "blending to 75 ns, better than local");
chk(tPp == 16'd0,  "so the penalty floors at zero rather than wrapping");

Capacity. The single-module case is asserted to make both models agree, and a demand exactly at the link rate is driven.

Placement. The capacity-limited spill is asserted as not a blind placement.

chk(kPw == 1'b0,   "which is not a good placement");
chk(kBe == 1'b0,   "though nothing cold is taking local space");

Value. The exactly-fits-local case is driven, and a blend faster than local is asserted to floor.

Batch. The largest batch is asserted exactly, and both zero-room cases are driven.

Migration. Break-even is asserted positive.

Economics. The one-gigabyte-over case is driven, and the no-overhead case is asserted to make both models agree.

Scale-out. An exact tie is asserted to favour expansion.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 255 checks across two testbenches, 130 on the front five models and 125 on the back five, all passing on the unmutated sources.

18. Mutation Testing

Forty-nine mutations were injected one at a time.

49 injected, 49 killed, after ten survivors across three rounds — the largest survivor set of the batch, and the rounds are the finding.

Eight survivors appeared at once, and two of them were unreachable guards. The floor on hot_far_gb and the floor on cxl_gb both protect subtractions whose left operand is already a minimum against the right — hot_local_gb is a minimum against hot_gb, and local_gb is a minimum against gb_wanted. Neither can underflow by construction. Both were deleted with a comment saying why, and both mutations were replaced with ones that are not equivalent — counting the quantity from the wrong side of the subtraction.

Six were thresholds and floors never driven. A far tier faster than local, a demand exactly at the link rate, a footprint exactly the size of local memory, a blend faster than local, capacity exactly the fixed cost, and an expanded blend exactly equal to the scale-out cost. Three of the six are "the far thing is faster than the near thing", which is not a perverse input — a machine with slow local memory and a fast attached tier is a real configuration.

Then three more survivors appeared that had been killed in round one, because the added stimulus moved the counter totals their kills depended on. This is batch 020's §4.5 exactly, and re-running the complete set after every stimulus change is the only reason they were caught.

And one boundary turned out to be untestable. The capacity guard capacity_gb <= fixed_gb survives becoming < because at equality the subtraction below yields zero anyway — the two expressions produce identical results at the only input that distinguishes them. The = half is behaviourally redundant, and the resolution was to say so in a comment and replace the mutation with one that changes the compared term rather than the operator. A guard that states intent without changing behaviour is worth keeping and not worth mutating.

19. Verification Strategy

What a testbench for a tiered-memory model must cover.

The far thing being faster than the near thing. Three of this chapter's floors needed it, and it is a real configuration rather than a perverse input. Any model with a "near" and a "far" term needs the inverted case driven.

Re-run the complete mutation set after every stimulus change. Three mutations reverted from killed to survivor in this chapter alone.

Check whether a floor can fire before writing it. Two guards here protected subtractions whose operands are already ordered by a minimum. A floor on an expression that cannot underflow is documentation — keep it or delete it deliberately, but do not mutate it as though it were logic.

Distinguish a redundant operator half from a redundant term. The capacity guard's = case changes nothing because the arithmetic below already yields zero there. That is different from a redundant term, and it calls for a comment rather than a deletion.

The cases that are correct and look like failures. A params-only sizing that is right because there is nothing else. A capacity-limited spill that no placer could have avoided. A single-node footprint with nothing to split. A break-even migration policy.

What a real deployment needs that these models do not have. Temporal locality — a working set whose hot region moves during training. Allocator behaviour — which tier a framework actually requests. Interference — other tenants on the same link, which is 19.4 §7 arriving here.

20. Synthesis and Implementation Reality

The tier is visible to software and that is the whole design. CXL-attached memory appears as a separate NUMA node with its own latency and bandwidth, reported by the device — which means section 8's placement is a NUMA policy decision rather than anything the hardware arbitrates.

Bandwidth per gigabyte falls because the link is fixed and the capacity is not. Section 7's ratio is the number that decides whether a workload should use CXL memory at all, and it gets worse with every module added. A capacity-bound workload improves; a bandwidth-bound one does not, and adding modules makes the second worse.

Page migration is a kernel daemon with a sampling cost. Section 12's per-migration cost is a page copy plus a TLB shootdown across every core that mapped it, and the detection cost — sampling access bits to decide what is hot — is charged whether or not anything moves.

Activation memory is a framework property, not a model property. Section 11's per-sample figure depends on activation checkpointing, which trades recomputation for memory. The largest batch is therefore a function of a software setting, and the same model on the same hardware has several correct answers.

The step at the DIMM slot boundary is real and it is large. Section 13's 257th gigabyte needs a module, a slot, a link and its power. Capacity planning past that point is discrete, which is why the per-gigabyte figure only becomes meaningful once a module is amortised.

21. Silicon Observability

"Accesses per tier" is the one that has to be separate from capacity per tier. A tier holding 50% of the bytes and serving 5% of the accesses is a good outcome; the same capacity split serving 50% of the accesses is section 8's failure — and a capacity counter reports them identically.

22. Debug Lab

Symptom. A training node is given 1 TB of CXL-attached memory on top of 512 GB local. The model that previously failed to start now trains. Throughput is 31% of what the same model achieves on a node with 1.5 TB of local DRAM, and the team expected around 80%.

Step 1 — is the footprint what was predicted? Peak resident by category: parameters 140 GB, optimiser 280, activations 190, gradients 140 — 750 GB. The sizing predicted 680. Close enough; section 5 is not the gap.

Step 2 — is it the tier blend? Resident per tier: 512 GB local, 238 GB attached. Local share 68%, predicted blend 164 ns against 100 local. That predicts roughly 60% of local-DRAM throughput, not 31%. Section 6 is real and is not the whole story.

Step 3 — is it accesses rather than capacity? Accesses per tier: 41% of accesses go to the attached tier, which holds 32% of the bytes. The attached tier is serving disproportionately, which means the wrong bytes are there. Section 8.

Step 4 — which bytes? Hot-region size against local capacity: the optimiser state is 280 GB and is touched once per step; the activations are 190 GB and are touched many times per step. The allocator put activations in the attached tier and optimiser state locally, because activations were allocated later.

Step 5 — is migration fixing it? Promotions 4.1 million, demotions 3.9 million over the run. Section 12's thrashing signature exactly — the daemon is moving pages back and forth as each step touches them, spending more than it saves.

The finding. Three properties failed together and only one was predicted. The tier blend was modelled and accepted; the placement was not modelled at all, and the migration daemon's response to bad placement made it worse rather than better.

The fix, in order. Pin the activations to local memory and the optimiser state to the attached tier — an explicit NUMA policy rather than allocation order. Then disable migration, because with correct placement it has nothing to fix and its sampling cost is pure overhead. Expected result: local share of accesses rises to roughly 90%, and the blend approaches section 6's prediction.

What made this hard. The capacity model was right, the tier model was right, and the workload still ran at half what they jointly predicted — because both are stated in bytes and the thing that was wrong is stated in accesses.

23. Design Review

1. What is the full working set, by category? Parameters, optimiser, activations, gradients — four measurements, not one. Section 5.

2. Is the workload capacity-bound or bandwidth-bound? Expansion helps the first and hurts the second. Section 7.

3. What is the bandwidth per gigabyte after expansion? It halves every time the capacity doubles behind one link. Section 7.

4. Which allocations go to the attached tier, and who decides? If the answer is "the allocator, by order", section 22.

5. Is there an accesses-per-tier counter, separate from capacity-per-tier? Section 21, and section 22 turns on it.

6. What is the largest batch that fits, arithmetically? It is computable before the run. Section 11.

7. Are promotions and demotions counted separately? Matching counts are thrashing. Section 12.

8. Does the workload already fit? If so, expansion makes it slower for nothing. Section 9.

9. What does attached capacity cost per gigabyte, including the module and the link? Not the DRAM price. Section 13.

10. Which of the six properties does the team believe "we added memory" implies? Section 15 exists because the answer is the capacity.

24. How This Appears In Real Engineering

A capacity planner sizing a training node does section 5 and gets it wrong by 4.9x if the input is a model card. The four measurements exist — a framework will report peak resident by category — and the gap between "the number everyone cites" and "the number that has to fit" is the single most common sizing error in the field.

A performance engineer meets section 22, and the lesson generalises: capacity models are stated in bytes and performance is decided by accesses. A tier holding a third of the data and serving 41% of the traffic is a placement problem that no capacity counter can express.

A kernel or runtime team owns sections 8 and 12 together, and they interact badly. A migration daemon is a response to bad placement, and on a workload with a large regularly-touched region it converts a steady penalty into a thrashing one. Correct placement makes migration unnecessary; incorrect placement makes it harmful.

A procurement team runs into section 13's step. Capacity below the DIMM-slot boundary is cheap and linear; one gigabyte past it costs a module. The planning question is not "how much memory" but "how many modules", and the two produce very different answers at the margin.

25. Common Misconceptions

"A 70B model needs 140 GB." For inference, 260 GB. For training, 680. Parameters are 20% of a training footprint. Section 5.

"CXL memory is a bit slower." A footprint twice the local memory blends to 200 ns against 100 — a 100% penalty. Section 6.

"More modules means more bandwidth." More capacity on the same link: 0.78 Gbps/GB at four modules, 0.39 at eight. Section 7.

"The allocator will figure out placement." By allocation order, which put the activations far and the optimiser state near. Sections 8 and 22.

"Adding memory can't hurt." A model that already fitted runs 120% slower for no benefit. Section 9.

"Batch size is a throughput knob." It is a memory knob first: 25 samples fit, 26 do not. Section 11.

"Page migration fixes bad placement." It converts a steady penalty into thrashing — 1000 µs spent to save 200. Section 12.

"CXL capacity costs the DRAM price." Plus the module, the link and its power — 14 per GB against 10. Section 13.

"Scaling out is always worse than expanding." At a 30% remote share, scaling out wins. Section 14.

"The footprint fits, so the expansion worked." One property of six. Section 15.

26. Interview Reasoning

Q. How much memory does a 70-billion-parameter model need to train?

Not 140 GB. Parameters at two bytes each are 140, optimiser state is roughly twice that, plus activations and gradients — around 680 GB, so parameters are a fifth. The follow-up worth reaching: inference needs 260 GB of the same weights, because there is no optimiser state and no gradients, so the same model is two very different sizing problems.

Q. You add 1 TB of CXL memory to a 512 GB node. What gets faster?

Nothing that already fitted — those workloads get slower, because their data is now spread across a tier they did not need. What gets faster is anything that previously did not run at all, and the honest framing is that expansion converts an impossibility into a slowdown.

Q. Eight CXL modules instead of four. What happens to bandwidth?

Nothing — it is the same link. Capacity doubles and bandwidth per gigabyte halves, from 0.78 to 0.39 Gbps/GB. That is the test for whether expansion helps a given workload: capacity-bound workloads improve, bandwidth-bound ones do not, and adding modules makes the second case worse.

Q. Your tiered node runs at half the throughput the blend model predicted. Where do you look?

Accesses per tier, against capacity per tier. A tier holding 32% of the bytes and serving 41% of the accesses means the hot data is in the wrong place — and a capacity counter reports that identically to a correct placement. The model is in bytes and the performance is in accesses.

Q. Promotions and demotions are both around four million. What does that tell you?

Thrashing. Every demotion is the system admitting a promotion was wrong, and matched counts mean the policy is moving pages back and forth. The sharper point: only promotions carry a benefit and every migration carries a cost, so a balanced policy is a losing one by construction.

Q. When is scaling out better than expanding memory?

When the remote-access share after partitioning is low enough that communication costs less than the tier penalty. At a 30% remote share and a 400 ns step, scale-out costs 120 ns against a 220 ns expanded blend — scale-out wins. At 80% it costs 320 and expansion wins. The deciding variable is how the model was partitioned, not the hardware.

27. Exercises

1. Extend RTL 1 with an activation-checkpointing parameter and show how much of the footprint it trades for recomputation.

2. Give RTL 2 a per-tier bandwidth term as well as a latency term, and find the footprint at which bandwidth rather than latency binds.

3. Add a second link to RTL 3 and show at what module count two links restore the bandwidth-per-gigabyte of the original four.

4. Make RTL 4's hot region move over time and show that a static placement degrades while a migrating one thrashes.

5. Combine RTL 5 and RTL 2: find the local share at which expansion's slowdown outweighs the benefit of running at all.

6. Extend RTL 6 to gradient accumulation and show that it decouples the effective batch from the memory-limited one.

7. Add a sampling cost to RTL 7 that is charged whether or not a page moves, and find the access-rate threshold at which detection alone is a loss.

8. Model section 13's step explicitly: plot cost against capacity and mark every DIMM-slot and module boundary.

9. Combine RTL 9 with 21.3 §5's rack boundary and show how the remote-access cost changes when nodes are in different racks.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the expansion it catches that the current mask calls helpful.

28. Summary

22.1 ended on a cache that could not hold the working set. This is how large that working set actually is, and what reaching the rest costs.

Parameters are one term of four. 140 GB of weights in a 680 GB training footprint — 20% — and the four-fifths that is not published is the part that has to fit.

The far tier is further. A footprint twice the local memory blends to 200 ns against 100, and a flat model reports 100 for the whole run with nothing erroring.

Capacity scales and bandwidth does not. Four modules give 0.78 Gbps per GB; eight give twice the capacity and 0.39 — which is the test for whether expansion helps this workload at all.

Which bytes go far decides everything. A blind placer sent 90% of accesses to the far tier while local memory held cold data — and section 22 is that failure at 41%, invisible to every capacity counter on the node.

Expansion helps what did not fit and hurts what did. An 800 GB model runs at a 120% slowdown instead of not running; a 256 GB model runs 120% slower for nothing.

The batch is a memory decision. 400 GB fixed and 8 GB per sample in 600 GB gives a largest batch of exactly 25, computable before the run that would otherwise die at hour nine.

Migration is a response to bad placement and it can be worse than the problem. Matched promotions and demotions spent 1000 µs to save 200.

The 257th gigabyte costs a module, not 14 — and CXL capacity at bare DRAM pricing undercosts by 40%.

Scaling out wins at a low remote share and loses at a high one, decided by how the model was partitioned rather than by the hardware.

Ten mutation survivors across three rounds — two unreachable floors deleted, three kills that reverted when stimulus changed, and one boundary where the correct and mutated expressions coincide because the arithmetic below already yields zero.

The footprint fitting is one property of six. The definition a capacity spreadsheet uses called five of six expansions helpful when one was.

22.3 — CXL in GPU Systems takes the bandwidth-per-gigabyte number from section 7 and asks where CXL sits against memory that is an order of magnitude faster still.