Capacity Scaling

11.1 established the weld: capacity is a product of channels, slots and density, fixed when the board was designed, and surplus behind one socket cannot serve demand behind another.

This chapter asks what actually changes when capacity arrives over a link instead of over channels — and the first answer is not about bandwidth or latency. It is about naming.

1. The Engineering Problem — Capacity You Cannot Name Is Not Capacity

A device with a terabyte of DRAM is attached to a host. The link trains. The device enumerates. Software reports the same memory it had before.

Nothing failed. The memory is there, powered, and working. It has no address, so no instruction can reach it, and from the workload's point of view it does not exist.

That gap — between installed and addressable — is where capacity scaling actually happens, and it introduces three problems the local memory system never had:

Every address must resolve to exactly one place. With one memory controller this was trivial. With local memory and several expanded regions it is a decode problem, and getting it wrong means either an address that goes nowhere or, worse, an address two regions both claim.

Capacity now appears and disappears. A local DIMM is present from power-on. An expanded region is configured, enabled, and can be disabled again — under live traffic.

Capacity is no longer uniform. Two addresses in the same address space now have materially different service times, and the system has to know which is which for every request it issues.

2. The One-Sentence Model

Addressable capacity is the sum of live regions, and every address must resolve to exactly one of them. Installed capacity is a hardware fact; addressable capacity is a configuration fact; and the gap between them is memory that exists and cannot be used.

Call it name it or lose it. Everything below follows from the requirement that an address resolves — uniquely, to an enabled region, with a known latency class.

3. What This Chapter Owns

Deferred:

This chapter deliberately does not decompose latency. It establishes that a latency class exists and must be tracked per request — which is a correctness and accounting problem — and leaves the hop-by-hop decomposition to Module 18.

4. Teaching-model boundary

5. RTL 1 — Exactly One Target

The decode is the first thing expansion adds, and it has two failure modes that are opposites.

      // A disabled region must not match. Enable is part of the decode, not a
      // downstream filter -- a filtered match still consumed a priority slot.
      if ((IGNORE_ENABLE || enabled[r])
          && (acc_addr >= base[r]) && (acc_addr <= limit[r])) begin
        match[r] = 1'b1;
        if (nmatch == 0) first_idx = r[1:0];
        nmatch++;
      end

=== EXP1: one address, exactly one target ===
  boundaries walked : local=4 expanded=4 unmapped=1
  independent oracle: local=4 expanded=4 unmapped=1
  local and expanded hits matched an independent oracle      : ok
  every region included both its own boundaries              : ok
  addr 250 with region 2 disabled : hit=0 unmapped=1 | ignore-enable hit=1
  a disabled region does not match, and the access is unmapped : ok
  overlapping regions, addr 260 : overlap flag=1
  two regions claiming one address was caught                : ok

Zero targets and two targets are both failures, and only one of them is loud.

An address matching nothing is unmapped — it has no correct answer, and the only correct response is to report it. An address matching two regions is worse: the priority logic silently picks one, the access succeeds, and whichever region lost the race has an address range that quietly belongs to someone else.

The overlap checker is written unguarded on purpose. Guarding it with the check-enable parameter would switch it off on the only configuration that can trip it — the self-disabling-checker defect this track has now seen four times, and it is always written by someone narrowing a check "sensibly".

Enable belongs inside the decode. Filtering a match afterwards looks equivalent and is not: a disabled region that still matches has consumed a priority slot, so an address it should never have claimed now resolves to it rather than falling through to a lower-priority region or reporting unmapped.

Every boundary was walked — first and last address of all four regions. That matters because the <= versus < limit comparator is invisible to any address comfortably inside a region, and it silently loses one line per region.

6. Waveform — A Region Disappearing Under an Address

Teaching-model timing derived from the simplified RTL in this chapter. Not CXL wire timing.

Compare cycle 4 with cycle 8. The address is identical and the outcome is opposite, because the enable mask changed between them. Local memory never behaves this way — a DIMM does not stop being addressable while the machine runs — and it is the single biggest behavioural difference expansion introduces.

That is why §9's enable FSM exists: if capacity can be withdrawn, the withdrawal has to be sequenced.

7. RTL 2 — Interleave Decides Whether Capacity Is Reachable

Once capacity spans several devices, the address bits chosen to select between them determine both bandwidth and, less obviously, whether the capacity can be reached at all.

  assign way = HIGH_ORDER_SEL ? high_sel : low_sel;
  // Whatever selects the way must be removed from the per-device offset.
  assign dev_offset = KEEP_SELECT_BITS
                    ? acc_addr[13:0]
                    : {acc_addr[15:GRAN_LOG2+2], acc_addr[GRAN_LOG2-1:0]};

=== EXP2: interleave decides whether capacity is reachable ===
  16 consecutive granules : ways=4/4/4/4
  the stream spread exactly evenly                           : ok
  high-order-select variant : way0=16 (a sequential stream never leaves it)
  addr 0 selects way 0 / addr 4 selects way 1 / addr 8 selects way 2 : ok
  addr 16 : way=0 offset=4 | keep-select-bits offset=16
  addr 16 wraps back to way 0                                : ok
  and its per-device offset advanced to 4                    : ok
  the keep-select-bits variant left the select bits in the offset : ok

Two independent failures, and the second is the quiet one.

Selecting on high-order bits makes a sequential stream stay on one device for an enormous run — 16 of 16 in the measured contrast — so three devices idle while one queues. That is a bandwidth failure, and it is at least visible as poor throughput.

Leaving the select bits in the per-device offset is a capacity failure. Each device then only ever sees offsets whose select bits match its own index, so it reaches a quarter of its own DRAM while the host believes all of it is addressable. Note the measured values: with the bits removed, address 16 maps to offset 4; with them left in, it maps to offset 16 — the offsets spread out and the device's address space is used sparsely.

An even distribution proves nothing

The distribution was 4/4/4/4 and matched the oracle — and that check alone would have passed a decode that shifts the select bits by one position, because such a decode is also perfectly even over a strided stream. It simply assigns the wrong addresses to each device.

Only the exact way and exact offset for known addresses pin a decode down. Evenness is a statistic; a decode is a function. This is the second batch in which that distinction has been the difference between a passing test and a real check.

8. RTL 3 — Latency Class Belongs to the Request

Expansion makes the address space non-uniform, and the moment it does, every performance counter has to answer a new question: which kind of memory was this?

  always_comb begin
    used_cls = RECLASSIFY_AT_DONE ? current_class : cls_q[retire_id];
  end
  ...
      if (issue) begin
        cls_q[issue_id]  <= issue_class;   // captured at issue, not at retire

=== EXP3: latency class belongs to the request, not the map ===
  issued expanded, map changed, then retired : correct exp done=1 local done=0
  reclassify-at-done variant                : exp done=0 local done=1 stale flag=1
  the request was attributed to the class it actually traversed : ok
  the reclassifying variant charged it to the wrong class    : ok
  measured age of the expanded request : 5 cycles
  the expanded request accumulated exactly its real age      : ok

A request that traversed expanded memory must be charged to expanded memory, even if the map changed while it was in flight.

The reclassifying variant looks harmless — it reads the current configuration at completion, which is simpler and needs no per-slot storage. It charged an expanded access to the local bucket, and the consequence is not a rounding error: the local latency average is inflated by a request that never touched local memory, and the expanded average is deflated by its absence. Both numbers move in the direction that makes expansion look better than it is.

The class must be captured at issue, in the slot, alongside the age. That is a few bits per outstanding transaction, and it is what makes the two latency populations separable at all.

9. RTL 4 — A Longer Round Trip Needs More In Flight

      case ({do_issue && !FREE_ON_ISSUE, do_free})
        2'b10: inflight_q <= inflight_q + 4'd1;
        2'b01: inflight_q <= inflight_q - 4'd1;
        default: ;                              // both or neither: hold
      endcase

=== EXP4: a longer round trip needs more requests in flight ===
  6 issues, 8 slots  : inflight=6 peak=6 throttled=0
  same 6 into a 2-slot pool : issued=2 throttled=4 ready=0
  the 2-slot pool issued 2 and throttled 4                   : ok
  and it deasserted ready once full, so upstream can see the limit : ok
  3 cycles of issue+response together : inflight 6 -> 6
  simultaneous issue and response left the in-flight count unchanged : ok
  response for an unallocated slot : double-free flag=1

This is 9.6's concurrency argument arriving at the capacity problem. Expanded memory has a longer round trip, so sustaining the same bandwidth requires proportionally more requests in flight — and a pool sized for local latency throttles expanded traffic while reporting no error at all.

The 2-slot pool issued 2 and throttled 4 on identical demand. Nothing is broken; the capacity is addressable and the device is fine. The pool is simply too small for the latency, and the only visible symptom is bandwidth that will not rise.

ready must fall when the pool is full, or upstream cannot see the limit and the throttle is invisible outside the block. That output is how a concurrency limit becomes observable rather than merely effective.

The simultaneous issue-and-response test is there because the two-assignment counter defect has now appeared eleven times in this track. One issue and one completion in the same cycle must leave the count unchanged; two separate non-blocking assignments make it fall, and the in-flight count drifts downward until anything gated on it acts early.

10. RTL 5 — Capacity May Not Appear or Vanish Under Traffic

=== EXP5: capacity may not appear or vanish under traffic ===
  a region is not addressable before configuration           : ok
  configured is still not addressable                        : ok
  only an enabled region is addressable                      : ok
  disable with 3 outstanding : state=3 addressable=0 wait=4 | instant variant state=0 vanish flag=1
  a draining region accepts nothing new                      : ok
  the instant-disable variant made capacity vanish under traffic : ok
  drained : state=0
  the region left only after outstanding reached zero        : ok

Configured is not addressable, and that distinction is the whole state machine. A region that has been described to the host but not enabled is dark capacity — it exists, its size is known, and no instruction can reach it.

Disable is a sequence, not a bit. This is the third structural appearance of the drain barrier in this curriculum, after 9.6's fence and 10.2's ownership handover, and it has the same shape every time: the cost is the work outstanding, and the fast variant that skips it corrupts something.

Here the corruption is specific. The instant-disable variant returns to OFF with three accesses still outstanding — those accesses were issued against memory that no longer answers, and they have no correct completion.

11. RTL 6 — Addressable Means Live

      counted = COUNT_CONFIGURED ? configured[r] : live[r];
      ...
      // Capacity that exists and is configured but cannot be addressed.
      if (configured[r] && !live[r]) d_sum += size[r];

=== EXP6: addressable capacity counts LIVE regions only ===
  sizes 128/128/512/512, live=0111 : addressable=768 local=256 expanded=512 dark=512
  addressable is 128+128+512 = 768                           : ok
  split 256 local and 512 expanded                           : ok
  the two halves account for the whole                       : ok
  512 GB is configured but dark                              : ok
  count-configured variant : addressable=1280 (512 GB software cannot use)
  double-count abuse : local=768 expanded=512 addressable=768 flag=1

768 addressable, 512 dark. The count-configured variant reports 1280 — a number that is arithmetically defensible and operationally false, because 512 GB of it cannot be reached by any instruction.

Dark capacity is the metric worth adding. It is the difference between "we installed a terabyte" and "software can use a terabyte", and the gap is usually a configuration or enablement problem rather than a hardware one. A platform reporting only installed capacity cannot distinguish a device that failed to enable from one that was never fitted.

The balance law — local plus expanded equals addressable — is what makes the split trustworthy. The DOUBLE_COUNT_EXP abuse instance counts expanded capacity into both halves and is caught immediately; without the law, a ledger can report a plausible total whose parts do not add up, and every derived percentage is then wrong.

12. RTL 7 — Count the Accepted Event

=== EXP7: count traffic on the accepted event ===
  offered=24 accepted local=8 expanded=8 stalled=8
  oracle : offered=24 local=8 expanded=8 stalled=8
  accepted local/expanded matched the oracle                 : ok
  accepted traffic is strictly less than offered             : ok
  count-offered abuse : offered=3 counted=3 flag=0
  and reports 100% acceptance on a path that accepted nothing : ok
  count-ready abuse : counted=3 offered=0 flag=1

The count-offered variant is the instructive one because it is self-consistent. Its own conservation law holds — it never counts more than it offered — and it still reports 100% acceptance on a path that accepted nothing. A metric can be internally coherent and measure the wrong event.

The rule is the same one AXI, PCIe and every credited interface teach: an event happened when valid && ready, not when valid. After expansion this matters more, because the local and expanded paths stall at different rates, and counting offers hides exactly the difference you added the counter to see.

Read it as three independent failure points on one path. A wrong decode sends the access nowhere or to two places; a wrong interleave strands three quarters of each device; an undersized pool throttles everything downstream of it. None of the three raises an error, and all three present as "expanded memory is disappointing".

13. Quantitative Reasoning

Illustrative, with stated assumptions. No figure below describes a real device.

Addressable versus installed

From §11:

installed   = 128 + 128 + 512 + 512 = 1280 GB
addressable = 128 + 128 + 512       =  768 GB
dark        =                          512 GB   (40% of installed)

Forty percent of the memory is present and unusable. No hardware fault would show that; only a live-versus-configured ledger does.

Concurrency for a longer round trip

Using 9.6's relation, concurrency equals bandwidth × round-trip time ÷ transaction size. Take a local round trip of 80 ns and an expanded one of 250 ns, both at 32 GB/s with 64-byte accesses:

local    : 32e9 x  80e-9 / 64 =  40 outstanding
expanded : 32e9 x 250e-9 / 64 = 125 outstanding
ratio    : 3.1x

A pool sized for local memory delivers roughly a third of the bandwidth on expanded memory — and reports no error, exactly as the 2-slot pool in §9 throttled 4 of 6 requests while behaving correctly.

Mixed-latency averages

With a fraction p of accesses served from expanded memory:

L_avg = (1 - p) x L_local + p x L_expanded
      = (1 - p) x 80 ns   + p x 250 ns

The mean is linear in placement, which is why placement policy is a first-class design concern rather than an optimisation — and why 11.5 returns to it for workloads whose p is not a free choice.

What interleaving is worth

From §7, a sequential stream across 4 ways:

spread across 4 devices : 4 concurrent media streams
all on one device       : 1 concurrent media stream

A 4× throughput difference decided entirely by which address bits select the device — same capacity, same link, same demand.

Reachable capacity when select bits are retained

From §7's offsets: with the select bits left in, each device only sees offsets whose bits match its index.

ways                    = 4
fraction of each device reachable = 1/4
usable capacity         = 25% of installed

The host believes all of it is addressable, which is why this defect presents as data corruption at high addresses rather than as an out-of-memory condition.

14. Assertions

Icarus Verilog 13.0 is the only simulator installed here. It does not execute concurrent SVA — property blocks are unsupported and unique/priority case qualities are parsed but ignored — so every property below is synthesisable checker logic verified procedurally, not executed SVA. 52 assertions.

Safety

Liveness

Performance goals — not correctness

15. Mutation Testing

Twenty-six mutations. Twenty-six killed.

First run: 22 of 26. The four escapes sorted the same way as the previous chapter's, which is now a reliable enough pattern to plan around:

Simultaneous issue and response was never driven, so the two-assignment in-flight counter survived — its eleventh appearance in this track, and it survives fill-then-drain stimulus every single time.

req_ready was never asserted on. The pool's issue behaviour was checked thoroughly and its output was not, so a mutation that leaves ready permanently high passed. A block's outputs are part of its contract even when internal behaviour is correct.

Two conservation laws were unreachable by construction — local + expanded always equals addressable, and accepted always ≤ offered, on any correct design. Rather than delete them, DOUBLE_COUNT_EXP and COUNT_READY_ONLY abuse instances were added, deliberately breaking each law on a separate instance so the checker fires without illegal stimulus reaching the instance under test.

Two testbench defects the baseline caught

A combinational output sampled after deassert. The disabled-region check read hit and unmapped after lowering acc_valid, so it read the idle value and reported a failure on a correct design. An isolated probe showed the decoder behaving perfectly. The fix samples while the input is still asserted — and this is the same defect class that appeared in 9.2.

An expectation asserted as a bound. The expanded request's age was checked as >= 6 when the true value is 5. A bound would have accepted a design that inflated the age arbitrarily; the assertion now checks the exact measured value.

16. Verification Strategy

The reference model is an address-map table, not a copy of the decode. The testbench resolves each address by a lookup over the intended map and compares against the design's priority loop. A testbench that reproduced the loop would agree with a wrong loop.

The coverage cross is region state × address position × traffic class: OFF/CONFIG/LIVE/DRAIN, crossed with first/interior/last/unmapped, crossed with local/expanded. The DRAIN × interior × expanded point is where an access is issued to a region that is about to stop answering, and it is unreachable without directed stimulus.

17. Synthesis and Implementation Reality

The decode is the timing-critical item. Every access pays for the compare tree, and adding regions widens it — which is the practical reason real systems favour a small number of large regions over many small ones, quite apart from any protocol consideration.

The free-slot priority encoder is the second. It sits between "a request arrived" and "a request issued", and its depth grows with pool size — so the pool you need for expanded latency is also the pool that is hardest to close timing on.

18. Silicon Observability

The pair to fight for is throttled requests against in-flight peak. Together they say whether the concurrency pool is the bottleneck, which is the single most common reason expanded capacity underperforms while every component reports healthy.

Max per-device offset is the cheap detector for the retained-select-bits defect — one register per device, and a value far above the device's real size is proof the offset was never reconstructed.

19. Debug Lab

  sizes 128/128/512/512, live=0111 : addressable=768 dark=512
  512 GB is configured but dark                              : ok

counted = live[r];                                  // not configured[r]
if (configured[r] && !live[r]) d_sum += size[r];    // and report the gap

  overlapping regions, addr 260 : overlap flag=1
  two regions claiming one address was caught                : ok

// Unguarded on purpose: guarding with the check-enable would switch it off
// on the only configuration that can trip it.
if (acc_valid && (nmatch > 1)) overlap_err <= 1'b1;

  addr 16 : way=0 offset=4 | keep-select-bits offset=16
  and its per-device offset advanced to 4                    : ok
  the keep-select-bits variant left the select bits in the offset : ok

assign dev_offset = {acc_addr[15:GRAN_LOG2+2], acc_addr[GRAN_LOG2-1:0]};

  issued expanded, map changed, then retired : correct exp done=1 local done=0
  reclassify-at-done variant : exp done=0 local done=1 stale flag=1

if (issue) cls_q[issue_id] <= issue_class;   // capture at issue
used_cls = cls_q[retire_id];                 // use it at retire

  6 issues, 8 slots  : inflight=6 peak=6 throttled=0
  same 6 into a 2-slot pool : issued=2 throttled=4 ready=0

  disable with 3 outstanding : state=3 addressable=0 | instant variant state=0 vanish flag=1
  the instant-disable variant made capacity vanish under traffic : ok

R_DRAIN: if (outstanding == 4'd0) st_n = R_OFF;
if ((st_q == R_DRAIN) && (st_n == R_OFF) && (outstanding != 4'd0))
  vanished_err <= 1'b1;

  16 consecutive granules : ways=4/4/4/4
  addr 0 selects way 0 / addr 4 selects way 1 / addr 8 selects way 2 : ok

wire [1:0] low_sel = acc_addr[GRAN_LOG2+1 -: 2];

  offered=24 accepted local=8 expanded=8 stalled=8
  count-offered abuse : offered=3 counted=3 flag=0
  and reports 100% acceptance on a path that accepted nothing : ok

assign accepted = valid && ready;
assign counted  = accepted;

20. Design Review

1. How many regions does the decode support, and where does the compare tree sit in the timing path?
2. What happens to an address that matches no region?
3. What proves no two regions can claim the same address?
4. Is enable part of the decode or a filter after it?
5. Which address bits select the device, and are they removed from the offset?
6. What is the maximum per-device offset observed, and does it match the device's size?
7. Where is the latency class captured, and what happens if the map changes in flight?
8. What is the outstanding pool sized from — local or expanded round-trip time?
9. Does ready deassert when the pool is full?
10. What must drain before a region can be disabled, and what proves it did?
11. Does the platform report installed or addressable capacity — and is dark capacity visible?

21. How This Appears in Real Engineering

Architect. Owns the region layout and the interleave granularity, and the trade between few large regions (shallow decode, coarse control) and many small ones (deep decode in the critical path).

RTL designer. Owns the decode's timing and the pool's priority encoder. Both sit in the request path and both grow with the flexibility the architect asked for.

DV engineer. Owns boundaries, overlaps, disabled regions, mid-flight reconfiguration and simultaneous issue/response. Every mutation that survived this chapter's first run died to one of those.

Firmware engineer. Owns the configure-then-enable sequence and the honest capacity report. Dark capacity is almost always visible here first.

OS/runtime engineer. Owns placement, which sets p in the mixed-latency mean — the one term in the equation software controls.

Performance engineer. Owns the concurrency argument, and must resist "expanded memory is slow" when the measurement shows a pool sized for local latency.

Silicon validation. Owns the full-range sweep that catches retained select bits and over-advertised regions — cheap, and almost never run to completion.

What each needs from the others: the pool size depends on the architect's worst-case round trip; the decode depth depends on the region count firmware wants; the placement policy depends on the latency split RTL is able to report. Get any one wrong and expanded capacity underperforms with every component reporting healthy.

22. Common Misconceptions

23. Interview Reasoning

24. Exercises

1. Calculation. A system has 4 local channels of 64 GB and two expanded devices of 512 GB each. One expanded device is configured but not enabled. Compute installed, addressable and dark capacity. Then compute the mixed-latency mean at 80 ns local and 250 ns expanded when 25% of accesses fall in expanded memory.

2. Analysis. Per-device counters are perfectly balanced, the imbalance metric is zero, and sequential bandwidth is half of expected. Explain why the balanced counters do not exonerate the decode, and state the single directed measurement that settles it.

3. RTL task. Extend region_decoder to report all matching regions rather than the first, and to refuse the access entirely when more than one matches. State the timing consequence of the refusal being combinational, and what changes if the decode is registered.

4. Assertion task. Write the property proving the per-device offset is a bijection onto device memory. Explain why a per-address check is insufficient alone, and what additional structure the testbench needs to prove the mapping is one-to-one.

5. Testbench design. Design the stimulus that distinguishes a latency-class-at-issue design from a latency-class-at-completion one. Explain why a static address map cannot distinguish them, and what the stimulus must do instead.

6. Performance calculation. A device sustains 32 GB/s with 64-byte accesses. Local round trip is 80 ns; expanded is 300 ns. Compute the outstanding depth required for each. Then compute the achieved expanded bandwidth if the pool holds 48 entries, and state which counter would have revealed this in silicon.

7. Debug task. Writes above a certain address alias onto lower addresses on a 4-way interleaved system. Give your investigation order, the two defects in this chapter that both produce that symptom, and the single measurement that distinguishes them.

8. Design review. A colleague proposes supporting 64 configurable regions instead of 4, arguing it costs only comparators. Give the strongest version of that argument, then state what it does to the request path, what it does to the verification surface, and what you would propose instead.

25. Summary

Addressable capacity is the sum of live regions, and every address must resolve to exactly one.

• Installed is not addressable. The measured ledger showed 1280 GB installed, 768 addressable and 512 GB dark — configured, described, and unreachable by any instruction.
• Exactly one target. Zero matches is loud; two matches is silent, because priority always produces an answer. Assert the match count.
• Enable belongs inside the decode, not after it — a filtered match has already consumed a priority slot.
• Interleave decides bandwidth and reachable capacity. High-order select put 16 of 16 accesses on one device; retained select bits leave each device reaching a quarter of its own DRAM while the host believes otherwise.
• An even distribution proves nothing about a decode. Only the exact way and offset for known addresses do.
• Latency class belongs to the request, captured at issue. Reclassifying at completion moves samples between buckets in the flattering direction.
• A longer round trip needs proportionally more requests in flight — roughly 3× here — and a pool sized for local latency throttles silently.
• Capacity may not vanish under traffic. Disable is a drain barrier, the third appearance of that shape in this curriculum.
• Verification: 26 of 26 mutations killed, 52 assertions. All four first-run escapes were stimulus gaps or unreachable checkers; the baseline separately caught a testbench sampling a combinational output after deassert, and an expectation written as a bound where an exact value existed.

Next: 11.3 — Memory Resource Sharing, which removes the assumption that a region has one owner, and asks what changes when a small number of hosts share a single expanded device.