Server Architectures

11.3 made a shared device safe: one owner per range, per-host budgets, fair arbitration, contained faults, scrubbed handover.

This chapter takes those mechanisms as given and asks a different question. What does the server actually look like?

1. The Engineering Problem — A Machine With Two Kinds of Memory

Once a server has both local DRAM and expanded memory, it has stopped being a machine with a memory system. It is a machine with two memory systems that share one address space and differ in latency, bandwidth and cost.

That is not a configuration detail. It changes what the word "memory" means to everything above it:

A page now has a location, and the location matters. Two addresses that look identical to software may differ by 3× in access time. Which one a page sits in is now a decision somebody has to make, continuously.

The decision is not static. A workload's hot set moves. A page that was hot at boot may be cold an hour later, and the machine that placed it perfectly at boot is placing it badly now.

Moving a page costs more than leaving it. Migration consumes bandwidth on both tiers and makes the page unavailable while it moves. A policy that migrates too eagerly spends its entire memory bandwidth on movement.

And local memory is no longer sized to the workload. It is sized to the part of the workload that needs to be fast — which is a different, smaller, and much harder number to know.

2. The One-Sentence Model

A tiered server places pages, and placement is a control loop. Measure which pages are hot, move the hot ones close and the cold ones away, do it atomically, and rate-limit the whole thing — because the loop's own cost is paid in the bandwidth it is trying to optimise.

Call it measure, move, but not too often. Every mechanism in this chapter is one term of that loop, and the loop is unstable without all four.

3. What This Chapter Owns

Deferred:

This chapter is about one node. It has no fabric and no rack. It also introduces NUMA-like placement, page migration and hot/cold classification architecturally — as hardware mechanisms and the state they require — and stops short of operating-system policy, allocator design or scheduler integration, which are software subjects this curriculum does not own.

4. Teaching-model boundary

5. RTL 1 — The Tier Belongs to the Page

    // The placement table is the authority. Inferring the tier from the page
    // number assumes a layout that migration immediately invalidates.
    t = GUESS_BY_ADDR ? tier_e'((page_id < 4'd8) ? T_LOCAL : T_EXPANDED)
                      : placement[page_id];

=== EXP1: the tier is a property of the page, not the address ===
  12 pages, layout matches address : local=8 expanded=4
  routing matched an independent oracle                      : ok
  the guess-by-address variant agrees while layout matches   : ok
  page 2 moved to expanded : correct routes expanded=1 | guess-by-address expanded=0
  the placement table routed the moved page correctly        : ok
  page 3 in motion : correct defer=1 | serve-migrating defer=0 flag=1
  an access to a page in motion was deferred                 : ok

Read the third line carefully: the guess-by-address variant agrees. While the layout matches the address ranges, inferring a page's tier from its address number produces exactly the right answer, and any test written before the first migration passes.

The moment page 2 moves, the two disagree. The placement table routes it to expanded; the address-based guess still says local, and sends the access to memory that no longer holds the page.

This is the defining property of a tiered machine: the tier is state, not a function of the address. A system that treats it as a function is correct until the first migration and wrong forever after.

A page in motion has no home at all. It is not in the old tier — the copy has begun — and not reliably in the new one. The only correct response is to defer the access, and the SERVE_MIGRATING variant that answers anyway was caught reading from a location that is no longer authoritative.

6. Waveform — A Page Changing Tier Under Live Traffic

Teaching-model timing derived from the simplified RTL in this chapter. Not CXL wire timing.

Four cycles in which a perfectly healthy page could not be read. That is the price of atomicity, and it is why migration has to be rare enough to be worth it — the deferral is invisible in throughput averages and extremely visible to whatever was waiting.

7. RTL 2 — Hotness Must Saturate and Decay

      if (do_decay) begin
        for (int unsigned p = 0; p < PAGES; p++)
          if (hot[p] != 8'd0) hot[p] <= hot[p] - 8'd1;
      end else if (touch) begin
        // Saturate: a counter that wraps makes the hottest page read coldest.
        if (NO_SATURATE || (hot[page_id] < MAXCOUNT))
          hot[page_id] <= hot[page_id] + 8'd1;
      end

=== EXP2: a hotness counter must saturate and decay ===
  20 touches on page 0 : hot=15 (max 15) | no-saturate hot=18
  the counter saturated at its maximum                       : ok
  and was caught exceeding it, so threshold comparisons are meaningless : ok
  after idle decay : hot=13 decays=4 | no-decay hot=15
  an idle page cooled down                                   : ok
  the no-decay variant stayed hot forever                    : ok
  pages 1 and 5 equally hot : p1=15 p5=15 hottest=1
  a tie resolves to the earlier page, not the last one scanned : ok

Two independent requirements, and each fails differently.

Without saturation, the counter runs past its declared maximum — 18 against a stated maximum of 15 — and every threshold comparison built on that scale becomes meaningless. Continue far enough and it wraps, at which point the hottest page in the system reads as the coldest and the placement policy actively moves the wrong page.

Without decay, hotness is a lifetime total rather than a recent rate. A page touched heavily once at startup stays maximally hot forever, and the policy is optimising for a workload phase that ended hours ago. Decay is what makes the metric describe the present.

The tie-break is the subtle one. Two equally hot pages must resolve deterministically, and the natural loop resolves to whichever index was scanned last if the comparison is >= rather than >. That looks like a triviality and is not: under a workload with many equally-warm pages, a last-index bias concentrates every migration decision on one end of the page space.

8. RTL 3 — Migration Must Be Atomic

      // Repointing the placement entry before the copy finished exposes a
      // half-copied page as if it were authoritative. A copy that COMPLETES on
      // this very edge is legal, so copied_q -- which this edge is still
      // setting -- cannot be the whole condition.
      if ((st_q == M_COPY) && (st_n == M_SWITCH)
          && !copied_q && !(cnt_q >= COPY_CYCLES[7:0]-8'd1))
        torn_err <= 1'b1;

=== EXP3: a migration is atomic from the accessor's view ===
  the migration entered its copy phase                       : ok
  and the page is marked in motion                           : ok
  mid-copy : state=1 commit=0 | switch-early state=0 commit=0 torn=1
  the correct engine has not committed mid-copy              : ok
  the switch-early variant repointed a half-copied page      : ok
  after copy : migrations=1 copy cycles=8 torn=0
  the copy took exactly 8 cycles                             : ok

A page has one home before the migration and one after. During it, none. The placement entry may be repointed only when the copy is complete — repointing early publishes a half-copied page as authoritative, and readers get a mixture of old and new contents with no error anywhere.

A checker that fired on correct behaviour

The first version of this checker read copied_q directly:

if ((st_q == M_COPY) && (st_n == M_SWITCH) && !copied_q) torn_err <= 1'b1;

It failed the baseline on a correct design. copied_q is set by the same clock edge on which the state transitions, so at the moment of transition it still reads zero — the checker was reading state that the transition it was checking was still creating.

This is the same shape that has now appeared in three separate chapters of this curriculum, in both directions: a checker too narrow to fire on a real violation, and — as here — one too broad, firing on legal behaviour. The corrected condition admits the copy that completes on the transition edge itself.

The general rule is worth stating once: whenever a checker examines a transition, ask which of the values it reads are being written by that same transition.

9. RTL 4 — Migration Must Be Rate Limited

    ready  = (NO_COOLDOWN || (cool_q == 8'd0))
             && (NO_BUDGET || (used_q < BUDGET[7:0]));

=== EXP4: migration must be rate limited and hysteretic ===
  40 evals oscillating : governed promote=2 demote=2 suppressed=36
  no-cooldown variant  : promote=3 demote=3
  no-budget variant    : promote=20 demote=20 thrash flag=1
  the governor suppressed migrations during cooldown         : ok
  the unlimited variant exceeded its declared budget and was caught : ok
  cold remote page, 20 evals : promotes went 2 -> 2
  a page below the promotion threshold was never promoted    : ok

Forty evaluations of a page oscillating around both thresholds. The governed policy moved it four times; the ungoverned one moved it forty.

Every one of those forty movements consumed bandwidth on both tiers and made the page unavailable while it copied. The policy would have spent the machine's memory bandwidth moving one page back and forth, and the workload would have observed only that memory got slower.

Two independent limiters, because they stop different failures. The cooldown stops a single page oscillating; the budget stops the population of pages from collectively saturating the migration path even when no individual page is thrashing. A design with only one of them fails in the other's direction.

Note the suppressed counter: 36. That number is the policy's own restraint made visible, and it is what distinguishes "the workload is stable" from "the governor is holding back a storm" — two situations that look identical if you only count completed migrations.

10. RTL 5 — Size Local Memory to the Hot Set

    // Hot pages take local memory first; whatever does not fit goes remote.
    hr = (hot_set_gb < local_gb) ? hot_set_gb : local_gb;
    hx = (hot_set_gb > local_gb) ? (hot_set_gb - local_gb) : 16'd0;
    // The right target is the hot set, not the footprint.
    tl = SIZE_TO_FOOTPRINT ? footprint_gb : hot_set_gb;

=== EXP5: size local memory to the hot set, not the footprint ===
  local=256 hot=192 footprint=1024 : hot resident=192 hot remote=0 wasted local=64
  the whole hot set fits locally                             : ok
  64 GB of local memory holds cold pages                     : ok
  the right local target is the hot set                      : ok
  the size-to-footprint variant targets the whole footprint  : ok
  local=128 hot=192 : hot resident=128 hot remote=64 undersized flag=1
  64 GB of hot pages were pushed to expanded memory          : ok

This is the sizing question the whole module has been building toward, and it has a precise answer.

Local memory should hold the hot set. Sizing it to the footprint — 1024 GB here — defeats the purpose of expansion entirely: you have bought all the local DRAM you were trying to avoid buying. Sizing it below the hot set — 128 against 192 — pushes 64 GB of genuinely hot pages onto the slow tier, and those pages will be touched constantly.

The interesting case is the first one: 256 local against a 192 GB hot set. It works, and 64 GB of local DRAM is holding cold pages. That is not free — it is the most expensive memory in the machine doing the least valuable job — and it is the number a capacity review should be looking at.

The undersized flag is the alarm that matters, because hot pages on the slow tier is the condition that makes users say the machine is slow, and it is invisible in any average.

11. RTL 6 — Two Tiers, Two Bandwidths

=== EXP6: the two tiers have independent bandwidth ===
  both tiers demanded for 32 cycles : local=32 expanded=8 total=40
  serialised variant                : local=32 expanded=0 total=32
  both tiers carried traffic concurrently                    : ok
  independent paths beat a serialised one                    : ok
  serialising cost the expanded tier its share               : ok

40 against 32 — and the whole difference is the expanded tier's contribution, which the serialised design loses entirely.

Expanded memory does not only add capacity; it adds a second independent path to memory. A workload that uses both tiers concurrently gets the sum of their bandwidths. One that places everything in one tier gets that tier's bandwidth and leaves the other idle.

The serialised variant is what a shared queue in front of both tiers does. It is a natural design — one request pipeline, one arbiter, two destinations — and it silently converts two independent bandwidths into one shared one. Here the expanded tier received nothing, because the local tier was always ready and always won.

That is a real architectural constraint on where the queueing structure sits, and it is invisible unless both tiers are driven simultaneously.

12. RTL 7 — Where a Page Lives Versus Why It Moved

=== EXP7: separate where a page lives from why it moved ===
  accesses=30 expanded=10 promotes=3 demotes=3 churn=3
  access counts matched an independent oracle                : ok
  expanded accesses are a strict subset of all accesses      : ok
  churn was recorded on every cycle that moved a page both ways : ok
  merge-tiers variant : expanded=30 of 30 accesses
  count-always abuse : expanded=279 accesses=0 flag=1

Access counts describe placement. Migration counts describe policy. Confusing them hides the failure mode that matters most.

A machine with 30% of accesses hitting expanded memory might be perfectly placed — that could be exactly the cold fraction — or it might be badly placed and thrashing. Access counts alone cannot tell. What separates them is churn: promotions and demotions of the same page class occurring together, which is the signature of a policy fighting itself.

Churn is the counter that would have caught the ungoverned policy in §9. Its access distribution would have looked reasonable throughout, while it moved a page forty times.

13. Quantitative Reasoning

Illustrative, with stated assumptions. No figure describes a real product.

The placement equation

With a fraction p of accesses served from expanded memory, and using 11.2's illustrative 80 ns local and 250 ns expanded:

L_avg = (1 - p) x 80 ns + p x 250 ns

Placement quality is the whole difference between 88 ns and 165 ns on identical hardware. No component changed; only which pages sat where.

What a migration costs

From §6 and §8: 8 copy cycles, 4 cycles of deferred access.

cost per migration = copy bandwidth + deferral of accesses to that page
benefit per migration = (accesses after) x (L_expanded - L_local)

A migration pays for itself only if the page is touched enough afterwards. With a 170 ns saving per access, a migration costing the equivalent of 100 accesses needs the page touched more than 100 times before it cools. That is the arithmetic behind the promotion threshold — not a tuning constant but a break-even point.

Thrash cost

From §9: 40 evaluations, 4 governed migrations, 40 ungoverned.

governed   :  4 migrations, 36 suppressed
ungoverned : 40 migrations
ratio      : 10x the migration bandwidth for the same workload

Ten times the movement for a page that ends up where it started. Migration bandwidth comes out of the same budget as the workload's own traffic, so the cost is not merely wasted — it is subtracted from useful work.

Sizing local memory

From §10, with a 192 GB hot set:

Undersized — hot pages forced onto the slow tier:

Oversized — local memory holding cold pages:

At 192 GB — exactly the hot set — both columns are zero, which is the only configuration that wastes nothing in either direction.

The optimum is the hot set, and both directions cost. Undersizing puts hot pages on the slow tier; oversizing puts the machine's most expensive memory under cold pages. The hot set is measurable — it is 11.1's distinct-page count over a window — which is what makes this a calculation rather than a guess.

Aggregate bandwidth

From §11: local 32, expanded 8 over 32 cycles.

independent paths : 32 + 8 = 40 accesses
serialised path   : 32 + 0 = 32 accesses
uplift            : 25%

Expansion adds bandwidth as well as capacity, but only if the request path lets both tiers work at once. A shared queue converts a 25% uplift into nothing.

14. Assertions

Icarus Verilog 13.0 is the only simulator installed. It does not execute concurrent SVA — property blocks are unsupported, unique/priority qualities are parsed but ignored — so every property below is synthesisable checker logic verified procedurally, not executed SVA. 45 assertions.

Safety

Liveness

Performance goals — not correctness

15. Mutation Testing

Twenty-five mutations. Twenty-five killed.

First run: 21 of 25. Two assertion-precision gaps, one stimulus gap, one checker needing an abuse instance — the same distribution as the three chapters before it.

The hottest output was never asserted. It was computed, it was correct, and nothing checked it, so a >= comparison biasing the selection to the last-scanned index passed. Killing it needed a deliberate tie between two pages, which is a stimulus most workloads never produce and no random test constructs on purpose.

The promotion threshold was never tested against a cold page. Every evaluation in the original stimulus had hotness above the threshold whenever the page was remote, so a design ignoring the threshold entirely behaved identically.

The budget checker needed the unlimited instance asserted. It was present and never checked.

A real RTL defect, found by the baseline

The torn-page checker described in §8 failed on a correct design because it read copied_q on the edge that sets it. That is a genuine design defect in the checker, not a testbench problem — and it was caught by the baseline run rather than by mutation testing, which is now the fourth such case in this curriculum. Mutation testing verifies the checkers; only running legal stimulus verifies the design.

Two testbench defects

An assertion about wrapping that could not occur. The no-saturate variant was asserted to "wrap and read colder", but with 20 touches on an 8-bit counter it reached 18 — it exceeded its declared maximum without wrapping. The assertion was rewritten to check what actually happens and matters: the counter passes its stated maximum, so every threshold comparison built on that scale is meaningless.

A serialisation test where nothing contended. The two tiers were configured with rates that never collided, so serialising them cost nothing and the test proved nothing. Reconfiguring so the local tier is ready every cycle made the contention real — and the expanded tier then received zero.

16. Verification Strategy

The reference model is a placement table maintained independently in the testbench, updated only when the design commits a migration. Comparing the design's routing against a table the testbench derived from the address would reproduce exactly the defect the chapter is about.

The coverage cross is page tier × hotness band × governor state: local/expanded/migrating, crossed with cold/warm/hot, crossed with ready/cooldown/budget-exhausted. The migrating × hot × budget-exhausted point is a page being accessed heavily while in motion and unable to be moved again — where deferral, thrash and starvation interact — and only directed stimulus reaches it.

17. Synthesis and Implementation Reality

The hotness array is the honest scalability limit. One counter per page is impossible at real page counts, and production tiering uses sampling, hashing or coarse region counters instead. The invariants taught — saturate, decay, deterministic tie-break — survive that change of mechanism; the implementation shown does not scale, and the chapter says so rather than implying otherwise.

The decay sweep is the structure people underestimate. Decaying every counter periodically is either a wide parallel operation or a background walk that takes many cycles, and if the walk is slower than the workload's phase changes the metric lags reality.

18. Silicon Observability

The pair to fight for is hot-pages-resident-remotely against wasted-local-capacity. Together they say whether local memory is the right size and in which direction it is wrong — the single most consequential sizing decision in a tiered server, and one that no average reports.

Churn against completed migrations is the second pair. High churn with low net movement means the policy is oscillating; that is the field signature of a missing cooldown, and it costs one comparator to detect.

19. Debug Lab

  page 2 moved to expanded : correct routes expanded=1 | guess-by-address expanded=0
  the placement table routed the moved page correctly         : ok

t = placement[page_id];   // the table is the authority

  mid-copy : state=1 commit=0 | switch-early state=0 commit=0 torn=1
  the switch-early variant repointed a half-copied page       : ok

M_COPY: if (cnt_q >= COPY_CYCLES-1) st_n = M_SWITCH;   // complete first
commit = (st_q == M_SWITCH);                            // then repoint

  40 evals oscillating : governed promote=2 demote=2 suppressed=36
  no-budget variant    : promote=20 demote=20 thrash flag=1

ready = (cool_q == 8'd0) && (used_q < BUDGET);   // both limits, both needed

  20 touches on page 0 : hot=15 (max 15) | no-saturate hot=18
  and was caught exceeding it, so threshold comparisons are meaningless : ok

if (hot[page_id] < MAXCOUNT) hot[page_id] <= hot[page_id] + 8'd1;
if (hot[page_id] > MAXCOUNT) wrap_err <= 1'b1;

  after idle decay : hot=13 decays=4 | no-decay hot=15
  an idle page cooled down                                    : ok
  the no-decay variant stayed hot forever                     : ok

if (do_decay) for (p...) if (hot[p] != 0) hot[p] <= hot[p] - 1;

  both tiers demanded for 32 cycles : local=32 expanded=8 total=40
  serialised variant                : local=32 expanded=0 total=32

  local=128 hot=192 : hot resident=128 hot remote=64 undersized flag=1
  64 GB of hot pages were pushed to expanded memory           : ok

hx = (hot_set_gb > local_gb) ? (hot_set_gb - local_gb) : 16'd0;
if (hx != 16'd0) undersized_err <= 1'b1;

  pages 1 and 5 equally hot : p1=15 p5=15 hottest=1
  a tie resolves to the earlier page, not the last one scanned : ok

if (hot[p] > hot[hottest]) hottest = p;   // strict: earlier index wins ties

20. Design Review

1. Where does the router get a page's tier — a table, or the address?
2. What happens to an access issued while its page is migrating?
3. Do hotness counters saturate, and what is the declared maximum?
4. How often do they decay, and how does that compare with the workload's phase duration?
5. How is a tie between equally hot pages resolved?
6. Can the placement table be repointed before a copy completes?
7. What limits the migration rate — a cooldown, a budget, or both?
8. What is the gap between the promote and demote thresholds?
9. Is local memory sized to the hot set or to the footprint?
10. Can both tiers be in flight simultaneously, or do they share a queue?
11. Does any counter distinguish good placement from a policy that is thrashing?

21. How This Appears in Real Engineering

Architect. Owns the local:expanded ratio, and must get the hot set from measurement rather than assumption. Sizing to the footprint defeats the purpose; sizing below the hot set defeats the machine.

RTL designer. Owns the placement table lookup in the access path and the hotness array's area. The decay sweep is the structure most often underestimated.

DV engineer. Owns the post-migration tests. Routing, tie-breaks and torn pages are all invisible before the first migration, and boot-time testing is the default.

Firmware engineer. Owns the promote and demote thresholds and the migration budget. The threshold gap is a hysteresis band, exactly as in 11.1's pressure FSM.

OS/runtime engineer. Owns what actually initiates migration and how deferred accesses are absorbed. This chapter provides the mechanism and stops short of the policy.

Performance engineer. Owns the placement equation and the churn counter, and must resist attributing to the tiers what is actually a shared-queue problem.

Silicon validation. Owns the deliberate-tie test and the both-tiers-simultaneously benchmark — neither of which arises naturally.

What each needs from the others: the ratio depends on a measured hot set; the thresholds depend on the migration cost RTL can report; the placement policy depends on hotness that saturates and decays. A gap in any one makes the loop unstable rather than merely suboptimal.

22. Common Misconceptions

23. Interview Reasoning

24. Exercises

1. Calculation. A workload has a 1 TB footprint and a measured 180 GB hot set. Local memory is 256 GB. Compute hot pages resident, hot pages remote, and wasted local capacity. Then recompute for a local size of 128 GB and state which configuration you would ship and what it costs.

2. Analysis. A tiered server shows 30% of accesses hitting expanded memory and stable throughput. Explain why this is insufficient to conclude that placement is good, name the counter that settles it, and state what value would indicate a policy fighting itself.

3. RTL task. Extend hotness_counter to use a two-level scheme: a small saturating counter per page plus a coarse counter per region. State the area saving at realistic page counts and what precision is lost, and identify which of this chapter's invariants the coarse level can no longer enforce.

4. Assertion task. Write the property that proves the placement table is never repointed before a copy completes. Explain why the completion flag cannot be read directly in that property, and what the correct formulation is.

5. Performance calculation. Local access is 80 ns and expanded is 250 ns. A migration costs 8 copy cycles at 1 GHz plus 4 deferred accesses. Compute the number of subsequent accesses needed for a promotion to pay for itself, and express that as a hotness threshold on a 0–15 scale with a decay interval of 16 cycles.

6. Testbench design. Design the stimulus that distinguishes a > tie-break from a >= one in the hottest-page scan. Explain why a random access pattern will essentially never distinguish them, and what the minimum deliberate stimulus is.

7. Debug task. A tiered server develops wrong-tier routing only after several hours of uptime, and rebooting fixes it temporarily. Give your investigation order, the two defects in this chapter that both produce that signature, and the single test that separates them.

8. Design review. A colleague proposes removing the migration cooldown, arguing that the promote and demote thresholds are far enough apart to prevent oscillation on their own. Give the strongest version of that argument, state the workload shape that defeats it, and describe the counter that would show it happening.

25. Summary

A tiered server places pages, and placement is a control loop.

• The tier is state, not a function of the address. An address-derived tier is correct until the first migration and wrong forever after — and every boot-time test passes.
• A page in motion has no home. Accesses must be deferred; the measured trace deferred four, which is the visible price of atomicity.
• Hotness must saturate and decay. Without saturation it exceeded its declared maximum at 18 of 15 and eventually inverts the policy; without decay it describes a workload phase that has ended.
• Ties must break deterministically, or a one-character >= concentrates every migration decision at one end of the page space.
• Migration must be atomic and rate limited. The ungoverned policy moved a page ten times as often for the same end state, spending workload bandwidth on movement.
• Size local memory to the hot set. Below it, hot pages go remote — 64 GB in the measured case. Above it, the most expensive memory in the machine holds cold pages.
• Two tiers are two bandwidths — 40 against 32, a 25% uplift — unless a shared queue serialises them, in which case the expanded tier gets nothing.
• Churn, not access counts, distinguishes good placement from thrash.
• Verification: 25 of 25 mutations killed, 45 assertions. The baseline caught a real RTL defect — a torn-page checker that read the completion flag on the edge that sets it, firing on correct behaviour — plus a testbench asserting a wrap that could not occur and a serialisation test in which nothing contended.

Next: 11.5 — AI Workloads on Expanded Memory, which takes a workload class whose access pattern contests almost every assumption this chapter relies on.