AI Workloads on Expanded Memory

11.4 built the placement loop and left one thing unexamined: the workload. It assumed a hot set that is measurable, reasonably stable, and smaller than local memory.

This chapter takes a class of workload where all three assumptions are contested.

1. The Engineering Problem — A Workload That Breaks Every Assumption

Memory expansion is usually justified by AI workloads, and they are the worst case for almost every mechanism in this module.

The working set is not smaller than local memory. For a large model it may be larger than local memory by an order of magnitude, which means the tiering policy is not choosing which minority of pages to promote — it is choosing which majority to leave behind.

The access pattern is structured, not random, and the structure varies. A weight matrix is streamed sequentially; an embedding table is gathered from scattered indices; activations are reused intensively for a short window and then never touched again. These are three completely different demands on the same memory system, in the same process, often in the same millisecond.

Phases are sharp. A training step has distinct stages, and the working set changes at each boundary — not by drifting, but by being replaced.

And there is a periodic bulk write — a checkpoint — that must not starve everything else while it runs.

None of these is a reason expansion fails. Each is a specific demand, and the design either meets it or does not. This chapter identifies which mechanism each pattern stresses, and measures what it costs when the mechanism is missing.

2. The One-Sentence Model

Shape, not rate, decides whether a workload tolerates expanded memory. A long round trip is amortised over a large sequential transfer and paid in full for a small scattered one — so the number that matters is not accesses per second but bytes used over bytes moved.

Call it efficiency, not bandwidth. A memory system moving data at full rate that nobody wanted is not fast; it is busy.

3. What This Chapter Owns

The overlap with Module 22 is the one that matters, and it is a boundary of viewpoint.

Deferred:

Module 22 owns AI systems; this chapter owns the memory system's view of an AI workload. The distinction is concrete: 22.2 asks how should an AI system use expanded memory, and this chapter asks what do these access patterns prove or break about the expansion design built in 11.1 through 11.4. Nothing here discusses accelerators, GPUs, model architectures or training topologies.

4. Teaching-model boundary

5. RTL 1 — Shape Decides Everything

    // A sequential burst moves many lines per request; a scattered access
    // moves one line and uses part of it.
    case (shape)
      A_SEQ:     moved_this = 16'(LINE_BYTES * BURST_LINES);
      A_STRIDE:  moved_this = 16'(LINE_BYTES * 2);
      default:   moved_this = UNDERREPORT_MOVED ? 16'd1 : 16'(LINE_BYTES);
    endcase

=== EXP1: shape, not rate, decides whether expansion works ===
  8 sequential bursts : moved=4096 used=2040 accesses=8
  bytes moved matched an independent oracle                  : ok
  + 8 scattered       : moved=4608 used=2104 scattered=8
  bytes moved still matched the oracle                       : ok
  the memory system never moved less than was consumed       : ok
  under-report abuse : overfetch flag=1

Sixteen accesses, and the two halves are not comparable. Eight sequential bursts moved 4096 bytes and the workload used essentially all of them. Eight scattered accesses moved 512 more bytes and the workload used 64 — eight bytes out of every sixty-four-byte line.

That is the entire argument. A long round trip is amortised over a sequential burst: one request, one latency, many useful bytes. It is paid in full for a scattered access: one request, one latency, one line fetched, a fraction used.

This is why "is the workload memory-bound?" is the wrong question. Two workloads with identical bandwidth demand can differ completely in whether expanded memory suits them, and the discriminator is shape.

6. RTL 2 — A Gather Needs Its Elements Together

If scattered access is the hard case, this is what makes it survivable.

    // The credit freed by this cycle's completion is usable this cycle.
    room  = NO_SLOT_BOUND ? 1'b1
          : SERIAL ? ((inf_q - (completes ? 8'd1 : 8'd0)) == 8'd0)
                   : ((inf_q - (completes ? 8'd1 : 8'd0)) < SLOTS[7:0]);

=== EXP5: a gather needs its elements in flight together ===
  24 cycles of demand : parallel issued=16 done=8 inflight=8
                        serial   issued=2 done=1 inflight=1
  the parallel gather issued far more in the same time       : ok
  the serial gather never had more than one outstanding      : ok
  the parallel gather held exactly its full slot count in flight : ok
  no-slot-bound abuse : inflight=12 overflow flag=1

Eight completions against one, in the same 24 cycles.

A gather of N scattered elements can finish in roughly one round trip if all N requests are outstanding together, or in N round trips if they are issued one at a time. Nothing about the memory changes — only how many requests are in flight.

That is 9.6's concurrency argument arriving at its most extreme case. For sequential access, concurrency hides latency. For gather, concurrency is the only thing standing between a workload that tolerates expanded memory and one that does not — and the required depth is the gather width, which is a workload property rather than a tuning constant.

7. Waveform — Parallel and Serial Gather, Same Demand

Teaching-model timing derived from the simplified RTL in this chapter. Not CXL wire timing.

Both engines wait exactly one round trip for their first completion — latency is identical. What differs is that the parallel engine spent that latency accumulating three elements and the serial engine spent it accumulating one.

A gather is a latency-hiding problem, and the only thing that hides latency is having more work outstanding.

7b. RTL 3 — Streaming Must Not Evict Reuse

    // Streaming data is installed only by a design that does not know better.
    install = acc_valid && !already && (KEEP_STREAMED || !is_streaming);
    bypass  = acc_valid && !already && !install;

=== EXP4: streamed data must not evict reused data ===
  6 reusable pages : resident=6 installed=6
  4 more reusable  : resident=8 installed=10 evictions=2
  residency stopped at its capacity of 8                     : ok
  + 6 streamed     : resident=8 installed=10 bypassed=6 evictions=2
  keep-streamed variant : resident=8 installed=16 evictions=8 pollution=1
  the streamed pages bypassed rather than installing         : ok
  and the reusable pages stayed resident at full capacity    : ok

Two evictions against eight. The bypassing design kept its eight reusable pages and let the streamed data pass through; the keep-streamed variant installed all sixteen pages and evicted eight times, throwing out data that would have been reused to make room for data that never will be.

This is the pattern-mixing problem in one measurement. An AI step streams weights and reuses activations in the same process. A residency policy that cannot distinguish them will let the largest and least reusable structure evict the smallest and most reusable one — and the streaming data gets no benefit from being cached, because by definition it is not touched again.

A page touched once should not occupy space that a page touched a hundred times could use. The mechanism is a bypass, and the information it needs — will this be reused? — is something the workload knows and the memory system does not.

8. RTL 4 — Phases Must Be Confirmed, Not Guessed

    // A single noisy window is not a phase change. Confirmation is what
    // separates a real transition from measurement noise.
    phase_change = big_change && (NO_CONFIRM || (confirm_q >= CONFIRM_SAMPLES[7:0]-8'd1));

=== EXP2: a phase change must be confirmed, not guessed ===
  one noisy window : confirmed phases 1 -> 1, false starts 0 -> 1 | no-confirm 1 -> 3
  a single noisy window was not treated as a phase change    : ok
  and it was recorded as a false start                       : ok
  the no-confirm variant declared one immediately            : ok
  sustained change : confirmed phases=2 last working set=30
  a sustained change was confirmed as a real phase           : ok

One noisy window produced zero confirmed phases in the correct design and two in the variant.

A phase change is expensive to act on: it invalidates the current placement and triggers a wave of migrations. 11.4 rate-limits those migrations, but the rate limiter cannot distinguish a real phase change from a noisy measurement — that has to happen here, at the detector.

Confirmation is the hysteresis of phase detection. It is the same idea as 11.1's pressure band and 11.4's threshold gap: the system commits to seeing a condition more than once before paying to act on it.

The false-start counter is what makes the tuning visible. A high false-start count means the threshold is too tight for the workload's measurement noise, and it is the difference between "the workload is stable" and "the detector is discarding real changes".

9. RTL 5 — Prefetch Only When It Will Be Used

    // With no history yet, allow prefetching so accuracy can be learned.
    acc     = (tot_q == 8'd0) ? 8'd100 : 8'((hit_q * 8'd100) / tot_q);
    allowed = NO_THROTTLE || (acc >= MIN_ACCURACY);

=== EXP3: prefetch is only worth issuing when it will be used ===
  20 predictions, 20% accurate : accuracy=0% issued=2 throttled=18
  unthrottled variant          : issued=20
  the gate throttled prefetching once accuracy dropped       : ok
  the unthrottled variant kept issuing regardless            : ok
  unsolicited-miss abuse : wasted=1 issued=0 flag=1

Prefetch is the standard answer to a long round trip, and on expanded memory it is the most dangerous one.

An accurate prefetch converts a full round trip into a hit. An inaccurate one consumes bandwidth, a concurrency slot and a cache line — all three of the resources the expanded path is short of — to deliver data nobody wants.

The gate issued 2 of 20 and throttled 18 once measured accuracy fell. The unthrottled variant issued all 20, and on a sequential stream that is correct behaviour; on the scattered gather where prediction fails, it is pure waste at exactly the moment the system can least afford it.

Accuracy must be measured, not assumed, and it must be a recent rate — the history saturates and halves rather than accumulating forever, for the same reason 11.4's hotness counters decay.

10. RTL 6 — A Checkpoint Must Not Starve Everything Else

    burst_allowed = UNLIMITED || (quota_q < BURST_QUOTA[7:0]);
    burst_grant   = burst_req && burst_allowed;
    // Steady traffic wins whenever the burst is not allowed to proceed.
    steady_grant  = steady_req && !burst_grant;

=== EXP6: a checkpoint burst must not starve steady traffic ===
  burst and steady both demanding : burst=12 steady=12 starved=12
  unlimited variant               : burst=24 steady=0 starved=24 flag=1
  steady traffic kept making progress alongside the burst    : ok
  the unlimited burst starved steady traffic completely      : ok

Twelve and twelve against twenty-four and zero.

A checkpoint is a large periodic write that has every right to exist and no right to consume the machine. Unmetered, it took every cycle and the steady traffic received nothing for the whole window — which in a real system means the workload stalls entirely for the duration of the checkpoint rather than slowing down.

The metered burst is slower to complete and does not stop anything else. That is the correct trade, and it is the same shape as 11.3's per-host credits: bounding one consumer's share so the others remain predictable.

Note the starvation detector measures consecutive starvation, not total. A metered burst blocks steady traffic often and never for long; an unmetered one blocks it continuously. Only the consecutive measure separates them — the totals look similar.

11. RTL 7 — Bytes Used Over Bytes Moved

=== EXP7: bytes used over bytes moved is the number that matters ===
  20 accesses : moved=1280 used=720 efficiency=56%
  oracle      : moved=1280 used=720
  efficiency is below 100% because half the lines were scattered : ok
  efficiency was exactly 56%                                 : ok
  count-moved-as-used variant : efficiency=100% (always perfect)
  inflate-used abuse : used=5100 moved=1280 flag=1

Fifty-six percent. The memory system moved 1280 bytes and the workload consumed 720. Nearly half the traffic on the expanded path delivered data nobody asked for.

This is the single number this chapter exists to produce. A bandwidth counter reports the memory system working at full rate. An efficiency counter reports that nearly half of that work was wasted — and the two are perfectly compatible.

The count-moved-as-used variant reports 100%, always. It is not a strawman: counting the bytes you fetched as the bytes you delivered is the natural implementation if nobody specified otherwise, and the resulting metric can never report a problem.

Efficiency is actionable in a way bandwidth is not. It rises with larger transfers, better placement, accurate prefetch and bypassed streaming — every mechanism in this module — and it falls when the access shape and the memory system are mismatched.

12. Quantitative Reasoning

Illustrative, with stated assumptions. No figure describes any real workload or product.

The amortisation argument

Take an illustrative 250 ns round trip, a 64-byte line and a 512-byte sequential burst:

sequential : 250 ns for 512 useful bytes  -> 0.49 ns per useful byte
scattered  : 250 ns for   8 useful bytes  -> 31.3 ns per useful byte
ratio      : 64x

Sixty-four times the cost per useful byte, on the same memory with the same latency. That ratio is the shape effect, and it is why a workload's suitability for expanded memory cannot be judged from its bandwidth figure.

Gather concurrency

For a gather of N elements at round-trip R:

serial   : N x R
parallel : R + (N / issue_rate)

With N = 64 and R = 250 ns:

serial   : 64 x 250 ns = 16.0 us
parallel : 250 ns + 64 cycles ~ 0.31 us
ratio    : ~52x

The required outstanding depth is the gather width. That is a workload property, and it is why a pool sized from average bandwidth demand will throttle gathers specifically — the very pattern that needs concurrency most.

Efficiency and effective bandwidth

With efficiency E and raw bandwidth B, the bandwidth the workload actually receives is:

B_effective = B x E

From §11's measured 56%:

32 GB/s raw x 0.56 = 17.9 GB/s useful

Fourteen gigabytes per second of the link's capacity delivered bytes nobody used. No component was slow and no counter was wrong; the access shape and the transfer granularity were mismatched.

The cost of cache pollution

From §7b: 8 evictions against 2, over 16 accesses.

bypassing    : 8 reusable pages retained, 2 evictions
keep-streamed: 8 reusable pages churned,  8 evictions

Every streamed page that installs evicts a page that would have been reused. With a hot set already at capacity, admitting streaming data converts a working cache into a very expensive FIFO.

Prefetch break-even

A prefetch saves a round trip when used and costs bandwidth, a slot and a line when not. With hit probability p:

benefit = p x R
cost    = (1 - p) x (line transfer + occupied slot)

Below roughly 50% accuracy the cost dominates, which is where §9's threshold comes from — not a tuning constant but the point at which prefetching stops paying, and exactly the regime a scattered gather produces.

13. Assertions

Icarus Verilog 13.0 is the only simulator installed. It does not execute concurrent SVA — property blocks are unsupported, unique/priority qualities parsed but ignored — so every property below is synthesisable checker logic verified procedurally, not executed SVA. 41 assertions.

Safety

Liveness

Performance goals — not correctness

14. Mutation Testing

Twenty-five mutations. Twenty-five killed.

First run: 19 of 25 — the largest escape count in this batch, and it sorted the same way as the other four chapters:

Four conservation laws were unreachable by construction — bytes moved cannot fall below bytes used, a wasted prefetch cannot exceed issued ones, outstanding cannot exceed slots, and bytes used cannot exceed bytes moved. Each needed a deliberately broken instance (UNDERREPORT_MOVED, COUNT_UNSOLICITED, NO_SLOT_BOUND, INFLATE_USED) so the checker fires without illegal behaviour reaching the instance under test.

Three real RTL defects, all found by the baseline

A wasted prefetch counted with no prefetch behind it. n_wasted incremented on every reported miss, whether or not a prefetch had been issued, so the count could exceed issued prefetches — which its own conservation law then flagged. The corrected version requires an issue before a waste can be attributed.

A starvation condition that could never fire. The check compared total starved cycles against total burst grants plus a margin, which the unlimited variant satisfied comfortably while starving steady traffic on every single cycle. Rewritten to measure consecutive starvation, it fires immediately — and the lesson generalises: for a fairness property, the run length is the signal and the total is not.

A parameter that changed the output and not the counter. NO_CONFIRM altered the combinational phase_change output while the counter still required confirmation, so the variant was unobservable in telemetry. The counting path now follows the same rule as the output. A configuration that changes behaviour must change what the counters report, or the two disagree and the telemetry lies.

And two testbench defects

A test that reported outcomes for prefetches never issued, which is what exposed the waste-counting defect above — the testbench was driving hit and miss independently of whether the gate had issued anything.

Stale expectations after extending a test. Adding installs past capacity changed the resident count from 6 to 8 and the keep-streamed install count from 12 to 16, and two assertions still carried the old numbers. They were corrected to the measured values rather than the remembered ones.

15. Verification Strategy

The oracle accumulates bytes independently of the design's shape table. The testbench adds 64 × 8 for a sequential burst and 64 for a scattered access by its own arithmetic; a testbench that consulted the design's moved_this would agree with a wrong table.

The coverage cross is access shape × phase state × prefetch accuracy: sequential/strided/scattered, crossed with steady/transitioning, crossed with accurate/inaccurate. The scattered × transitioning × inaccurate point is the worst case in the whole module — a gather during a phase change with prefetch mispredicting — and it is where efficiency, concurrency and pollution all fail together.

16. Synthesis and Implementation Reality

The gather pool is the expensive structure, and it is expensive in exactly the way that matters: it must be as deep as the widest gather the workload issues, and that depth costs storage per outstanding element plus a priority encoder that grows with it.

The bypass bit is the cheapest and the hardest. One bit costs nothing; knowing what to put in it requires the workload to declare whether data will be reused, which is information the memory system cannot derive and the software must provide.

17. Silicon Observability

Efficiency is the one number to add first. Bandwidth counters show a memory system working hard; efficiency shows whether the work was useful, and the gap between them is where every mechanism in this module either pays for itself or does not.

Gather peak in flight against gather width is second. If the peak is pinned below the width, the pool is throttling exactly the pattern that most needs concurrency — and no device-side change will help.

18. Debug Lab

  20 accesses : moved=1280 used=720 efficiency=56%
  efficiency is below 100% because half the lines were scattered : ok

efficiency_pct = (us_q * 100) / mv_q;              // used over moved
if (us_q > mv_q) efficiency_err <= 1'b1;           // and a conservation law

  24 cycles of demand : parallel issued=16 done=8 inflight=8
                        serial   issued=2 done=1 inflight=1

  + 6 streamed     : resident=8 installed=10 bypassed=6 evictions=2
  keep-streamed variant : resident=8 installed=16 evictions=8 pollution=1

install = acc_valid && !already && !is_streaming;
bypass  = acc_valid && !already && is_streaming;

  one noisy window : confirmed phases 1 -> 1, false starts 0 -> 1 | no-confirm 1 -> 3
  a single noisy window was not treated as a phase change    : ok

phase_change = big_change && (confirm_q >= CONFIRM_SAMPLES-1);

  20 predictions, 20% accurate : accuracy=0% issued=2 throttled=18
  unthrottled variant          : issued=20

acc     = (hit_q * 100) / tot_q;
allowed = (acc >= MIN_ACCURACY);

  burst and steady both demanding : burst=12 steady=12 starved=12
  unlimited variant               : burst=24 steady=0 starved=24 flag=1

burst_allowed = (quota_q < BURST_QUOTA);
steady_grant  = steady_req && !burst_grant;

  unsolicited-miss abuse : wasted=1 issued=0 flag=1
  counting a miss with no prefetch behind it was caught      : ok

if (miss && (iss_q != 16'd0)) wst_q <= wst_q + 16'd1;
if (wst_q > iss_q) waste_err <= 1'b1;

  one noisy window : confirmed phases 1 -> 1 | no-confirm 1 -> 3
  the no-confirm variant declared one immediately            : ok

if (phase_change) begin   // the counter follows the same condition as the output
  phase_q <= phase_q + 16'd1;

19. Design Review

1. What is the efficiency — bytes used over bytes moved — and is it instrumented at all?
2. What is the widest gather the workload issues, and how deep is the outstanding pool?
3. Is peak gather concurrency reaching the gather width, or is the pool throttling it?
4. Can streaming data bypass residency, and who supplies the reuse hint?
5. How many samples confirm a phase change, and what is the false-start rate?
6. Is prefetch accuracy measured, and is it a recent rate or a lifetime average?
7. What throttles prefetch when accuracy falls?
8. What quota limits the checkpoint burst, and is starvation measured as a run length?
9. Which counters distinguish the access shapes from one another?
10. Does every configuration option move at least one counter?
11. Which of these numbers exist in silicon rather than only in a model?

20. How This Appears in Real Engineering

Architect. Owns the efficiency target and the gather-width assumption, both of which size the pool. Judging suitability from a bandwidth figure is the characteristic mistake.

RTL designer. Owns the gather pool depth and its priority encoder — the dominant cost here — and the wide byte accumulators that make efficiency measurable.

DV engineer. Owns scattered and phased stimulus. Sequential streams make a shallow pool, an inaccurate prefetcher and a polluting cache policy all look fine.

Software/runtime engineer. Owns the reuse hint and the gather issue pattern. The bypass bit costs nothing in hardware and cannot be derived without them.

Performance engineer. Owns efficiency as the headline metric and must resist bandwidth as a proxy for it — the two are perfectly compatible and mean opposite things.

Silicon validation. Owns the shape-mixed benchmark. Every mechanism here fails only when patterns are mixed, and single-pattern testing is the default.

What each needs from the others: the pool depth depends on the widest gather software issues; the bypass depends on a hint only software has; the efficiency target depends on the transfer granularity hardware chose. Any one missing and the others cannot be judged.

21. Common Misconceptions

22. Interview Reasoning

23. Exercises

1. Calculation. A round trip is 250 ns and a line is 64 bytes. Compute cost per useful byte for a 1 KB sequential burst fully consumed, and for a scattered access using 4 bytes of a line. Then compute the efficiency of a workload that is 70% sequential and 30% scattered by access count.

2. Analysis. Bandwidth counters show the expanded path saturated and the workload's throughput is a third of expectation. Explain what single measurement distinguishes a genuine bandwidth limit from a shape problem, and state the values you would expect in each case.

3. RTL task. Extend gather_engine to support two independent gathers in flight, each with its own slot allocation. State what prevents one gather from consuming the other's slots, and the failure mode if that protection is omitted.

4. Assertion task. Write the property proving a wasted prefetch implies an issued prefetch. Then explain why this property caught a real defect that inspection did not, and what class of metric always needs a law of this kind.

5. Performance calculation. A gather of 128 elements runs against a 300 ns round trip with an outstanding pool of 32. Compute the completion time, then the pool depth needed to complete it in two round trips, and state which counter would reveal the shortfall in silicon.

6. Testbench design. Design the stimulus that distinguishes a metered checkpoint burst from an unmetered one. Explain why total starved cycles cannot distinguish them and what must be measured instead.

7. Debug task. A workload's throughput collapses whenever a large sequential pass runs, and the sequential pass itself is not slow. Give your investigation order, the two defects in this chapter that produce that signature, and the measurement that separates them.

8. Design review. A colleague proposes removing the prefetch accuracy throttle, arguing that a wasted prefetch only costs bandwidth and bandwidth is plentiful on the expanded path. Give the strongest version of that argument, name the two other resources it ignores, and state the workload shape that makes it fail.

24. Summary

Shape, not rate, decides whether a workload tolerates expanded memory.

• A round trip is amortised over a sequential burst and paid in full for a scattered access — an illustrative 64× difference in cost per useful byte on identical memory.
• A gather needs its elements outstanding together: 8 completions against 1 in the same window, and the required pool depth is the gather width.
• Streaming data must bypass residency. Installing it evicted reusable pages eight times against two, and gained the stream nothing.
• Phase changes must be confirmed, not guessed — one noisy window produced zero confirmed phases with confirmation and two without.
• Prefetch must be throttled on measured, recent accuracy. Below roughly 50% it consumes bandwidth, a slot and a line to deliver data nobody wants.
• A checkpoint must be metered: 12/12 metered against 24/0 unmetered, and starvation must be measured as a run length, not a total.
• Bytes used over bytes moved is the number that matters — 56% measured, meaning a 32 GB/s link delivering under 18 GB/s of useful data while every bandwidth counter reads healthy.
• Verification: 25 of 25 mutations killed, 41 assertions. The baseline found three real RTL defects — a waste counter that could exceed its own issues, a starvation check comparing totals where run length is the signal, and a parameter that changed behaviour without moving any counter.

Module 11 — Memory Expansion is complete. 11.1 established the weld, 11.2 made capacity addressable, 11.3 made a device shareable, 11.4 built the placement loop, and this chapter showed that whether any of it pays depends on what the workload asks for.

Next: Module 12 — Memory Pooling, which takes the shared device of 11.3 and asks what changes when there are many hosts, many devices, and a fabric between them.