Cloud Architectures on CXL

23.1 priced pooling. 23.2 priced composing. 23.3 showed that the metric both were judged by can move thirty points without the fleet moving at all.

This chapter is what an operator does with all three at once, and the finding that organises it is a single decision made early and never revisited: how wide is the pool?

Widen it and the reach grows, which is the entire economic case. The blast radius grows by exactly the same factor, and the latency grows with the switch tiers the widening crossed — one decision, three consequences, and only the first appears on the business case.

1. The Engineering Problem — Scope Is Three Decisions Wearing One Hat

A fleet-wide pool reaches 512 sockets and takes 512 down. The same widening adds a switch tier: 175 ns becomes 250. Section 5.

Oversubscription is the business model, not a mistake. A 1024 GB pool committed to 2048 serves a 1536 GB simultaneous demand 512 GB short. Section 6.

And a breach costs money the price list does not carry. Four breaches at 1000 turn a 40% margin into exactly 20%; eight turn it into nothing. Section 7.

Live migration changes shape. A 128 GB guest with 16 GB on the host copies 1280 ms of memory and rebinds the rest. Section 8.

A pool strands too. Sixteen racks with 256 GB spare between them can still refuse a 32 GB request, because the rack it is in has 16. Section 12.

This chapter against 23.1, stated precisely. That one owns whether to pool at all. This one owns how an operator running many tenants across many racks actually deploys one — which is why sections 6, 7, 10 and 13 have no counterpart there: they are all consequences of selling the capacity rather than merely owning it.

2. The One-Sentence Model

A deployed pool is a sound architecture when its scope is costed in reach, blast radius and latency together, the oversubscription is admissible against simultaneous demand, near and far carry their own SLOs, a failed rack can be absorbed, and a bad configuration cannot reach the fleet — and every defect below is a pool in production that is a liability rather than a capability.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real cloud is a control plane, a placement service, a billing system, a fabric manager and a fleet-management pipeline, and none of that is reproduced. What is reproduced is the arithmetic each performs, and the shape of the mistake when it does not.

Each model is built twice — a correct build and a broken build selected by a parameter. Every broken build here is a claim made in a deployment proposal: fleet-wide reach at rack-local cost, a commitment that is backed, a migration that is free, one latency class, an independent tenant population. Each is the version of the architecture that gets approved, which is precisely why it needs a model rather than an argument.

Figure 1 — Sixteen times the reach, sixteen times the blast radius, and 43% more latency, from one line in a design document. The middle node is what the business case contains, and the two on the right are what the operations team inherits.

5. RTL 1 — Scope Sets Reach, Blast Radius And Latency Together

// RTL 1 - the scope of a pool. Widening it raises the reach, and raises the
// blast radius and the latency by exactly the same decision.
module pool_scope #(parameter int IGNORE_SCOPE_COST = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic        fleet_wide,
  input  logic [15:0] sockets_per_rack, racks, base_ns, hop_ns,
  output logic [15:0] reach_sockets, true_blast, blast_sockets, latency_ns,
  output logic [1:0]  tiers,
  output logic        blast_acceptable,
  output logic [7:0]  n_assessments, n_wide,
  output logic        scope_cost_hidden_err
);
  logic [31:0] r_q, l_q;
  logic [1:0]  true_tiers;
  assign r_q = {16'd0, sockets_per_rack} * {16'd0, racks};
  assign reach_sockets = fleet_wide
        ? ((r_q > 32'd65535) ? 16'hFFFF : r_q[15:0]) : sockets_per_rack;
  // A wider scope crosses a second switch tier and shares a wider failure unit.
  assign true_tiers = fleet_wide ? 2'd2 : 2'd1;
  assign true_blast = reach_sockets;
  assign tiers = (IGNORE_SCOPE_COST != 0) ? 2'd1 : true_tiers;
  assign blast_sockets = (IGNORE_SCOPE_COST != 0) ? sockets_per_rack : true_blast;
  assign l_q = {16'd0, base_ns} + ({14'd0, tiers} * {16'd0, hop_ns});
  assign latency_ns = (l_q > 32'd65535) ? 16'hFFFF : l_q[15:0];
  assign blast_acceptable = (blast_sockets <= 16'd64);
  // A pool's reach claimed at a wider scope than its cost was costed at.
  assign scope_cost_hidden_err = assess && (blast_sockets < true_blast);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assessments <= 8'd0; n_wide <= 8'd0;
    end else if (assess) begin
      n_assessments <= n_assessments + 8'd1;
      if (!blast_acceptable) n_wide <= n_wide + 8'd1;
    end
  end
endmodule

Six assessments. Racks of 32 sockets, a 100 ns base and 75 ns per switch tier.

Two blast radii were too wide when the scope is costed; the scope-free model reported none.

The scope-free model is the proposal every operator has read: fleet-wide reach with rack-local latency and rack-local failure containment. Nothing about it is dishonest — each of the three numbers is true of some configuration, and only the combination is impossible.

Row five is the case that makes the tier cost visible on its own. A fleet-wide scope containing exactly one rack has the reach of a rack-local pool and still crosses two tiers, because the scope is a routing decision rather than a population count. Declaring the wider scope costs the latency immediately and delivers the reach only as racks are added.

Row three is the design point most large operators land on, and section 14's flowchart explains why: a two-rack scope doubles the reach against a rack-local pool while keeping the blast radius inside an acceptance threshold that a fleet-wide scope misses by eight times.

6. RTL 2 — Oversubscription Is The Business Model

// RTL 2 - oversubscription. A pool is committed to more than it holds, which is
// the business model, and a simultaneous-demand event is what it costs.
module oversubscription_ratio #(parameter int ASSUME_BACKED = 0) (
  input  logic clk, rst_n,
  input  logic        admit,
  input  logic [15:0] pool_gb, committed_gb, demand_gb,
  output logic [15:0] ratio_pct, served_gb, shortfall_gb,
  output logic        safe,
  output logic [7:0]  n_admissions, n_short,
  output logic        commitment_assumed_backed_err
);
  logic [31:0] r_q;
  assign r_q = (pool_gb == 16'd0) ? 32'd65535
             : (({16'd0, committed_gb} * 32'd100) / {16'd0, pool_gb});
  assign ratio_pct = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  // A commitment is only backed to the extent the pool holds it.
  assign served_gb = (ASSUME_BACKED != 0) ? demand_gb
                   : ((demand_gb > pool_gb) ? pool_gb : demand_gb);
  // served_gb is a minimum against demand_gb, so this cannot underflow.
  assign shortfall_gb = demand_gb - served_gb;
  assign safe = (shortfall_gb == 16'd0);
  // Demand above the pool, served in full.
  assign commitment_assumed_backed_err = admit && (demand_gb > pool_gb)
                                         && (served_gb == demand_gb);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_admissions <= 8'd0; n_short <= 8'd0;
    end else if (admit) begin
      n_admissions <= n_admissions + 8'd1;
      if (!safe) n_short <= n_short + 8'd1;
    end
  end
endmodule

Six admissions. A 1024 GB pool.

Three admissions were short; the backed model reported none.

Two-to-one is not a defect; it is the product. Every cloud sells more capacity than it holds, on the same statistical bet section 11 models — tenants do not peak together. The failure is not oversubscribing; it is treating the commitment as though the pool had to be able to honour all of it at once.

Row one is the boundary the whole admission policy defines. A simultaneous demand exactly equal to the pool is served in full with nothing to spare, and one gigabyte more is a breach. Admission control is the machinery that keeps the fleet on the correct side of that line, and section 7 prices what happens when it does not.

Row six is the event that has to be planned for rather than avoided. Every tenant drawing its full entitlement simultaneously leaves half the commitment unserved, and a two-to-one ratio guarantees that outcome by construction — the question is only how often it occurs and what it costs.

The breach is two cycles of an eight-cycle window and it costs a credit for the whole window. That asymmetry — a brief excursion priced as a full-period penalty — is why section 7's economics are so sensitive to the number of breach events rather than to their duration.

7. RTL 3 — A Breach Costs Money The Price List Does Not Carry

// RTL 3 - what oversubscription earns and what a breach costs. The margin is
// the second minus the first, and only one of them is on the price list.
module oversubscription_economics #(parameter int IGNORE_BREACH_COST = 0) (
  input  logic clk, rst_n,
  input  logic        price,
  input  logic [15:0] committed_gb, rate_per_gb, pool_gb, cost_per_gb,
  input  logic [15:0] breaches, penalty_each,
  output logic [15:0] revenue, pool_cost, breach_cost, net, margin_pct,
  output logic        profitable,
  output logic [7:0]  n_pricings, n_unprofitable,
  output logic        breach_ignored_err
);
  logic [31:0] r_q, p_q, b_q, m_q;
  logic [15:0] gross;
  assign r_q = {16'd0, committed_gb} * {16'd0, rate_per_gb};
  assign revenue = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  assign p_q = {16'd0, pool_gb} * {16'd0, cost_per_gb};
  assign pool_cost = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  // A breach of a committed entitlement costs a credit, which a price list that
  // only carries the rate does not show.
  assign b_q = (IGNORE_BREACH_COST != 0) ? 32'd0
             : ({16'd0, breaches} * {16'd0, penalty_each});
  assign breach_cost = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  assign gross = (revenue > pool_cost) ? (revenue - pool_cost) : 16'd0;
  assign net = (gross > breach_cost) ? (gross - breach_cost) : 16'd0;
  assign m_q = (revenue == 16'd0) ? 32'd0
             : (({16'd0, net} * 32'd100) / {16'd0, revenue});
  assign margin_pct = (m_q > 32'd65535) ? 16'hFFFF : m_q[15:0];
  assign profitable = (margin_pct >= 16'd20);
  // Breaches that happened and cost nothing.
  assign breach_ignored_err = price && (breaches != 16'd0)
                              && (penalty_each != 16'd0) && (breach_cost == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_pricings <= 8'd0; n_unprofitable <= 8'd0;
    end else if (price) begin
      n_pricings <= n_pricings + 8'd1;
      if (!profitable) n_unprofitable <= n_unprofitable + 8'd1;
    end
  end
endmodule

Six pricings. 2048 GB committed at 10, served from a 1024 GB pool costing 12 per GB.

Three pricings were unprofitable; the breach-free model reported one.

Four breaches a year halve the margin and eight erase it. Revenue is 20,480 and the pool costs 12,288, so the gross is 8192 — and the penalty schedule is the same order of magnitude as the entire gross, which is what makes admission control a financial control rather than an operational one.

Row six is the exemption that keeps the check honest. A breach with no contractual penalty costs nothing, and reporting no breach cost for it is correct — the error is a penalty that exists and is not counted, which is why breach_ignored_err requires both a breach and a penalty.

Row four is the floor and it matters commercially. Penalties larger than the gross do not make the business owe money in this model; they take the margin to zero and stop, because a service credit is capped at the fee. Modelling it as an unbounded loss would be arithmetically tidier and contractually wrong.

8. RTL 4 — Live Migration Changes Shape

// RTL 4 - live migration with pooled memory. What sits in the pool rebinds and
// what sits on the host has to be copied, and only the second costs bandwidth.
module live_migration #(parameter int ASSUME_ALL_REBINDS = 0) (
  input  logic clk, rst_n,
  input  logic        migrate,
  input  logic [15:0] local_gb, pooled_gb, link_gbps, rebind_ms, budget_ms,
  output logic [15:0] moved_gb, move_ms, total_ms,
  output logic        within_budget,
  output logic [7:0]  n_migrations, n_over,
  output logic        copy_ignored_err
);
  logic [31:0] m_q, t_q;
  // Pooled pages rebind to the destination; host-local pages are copied.
  assign moved_gb = (ASSUME_ALL_REBINDS != 0) ? 16'd0 : local_gb;
  assign m_q = (link_gbps == 16'd0) ? 32'd65535
             : (({16'd0, moved_gb} * 32'd8000) / {16'd0, link_gbps});
  assign move_ms = (m_q > 32'd65535) ? 16'hFFFF : m_q[15:0];
  assign t_q = {16'd0, move_ms} + {16'd0, rebind_ms};
  assign total_ms = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign within_budget = (total_ms <= budget_ms);
  // Host-local pages that were never copied.
  assign copy_ignored_err = migrate && (local_gb != 16'd0) && (moved_gb == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_migrations <= 8'd0; n_over <= 8'd0;
    end else if (migrate) begin
      n_migrations <= n_migrations + 8'd1;
      if (!within_budget) n_over <= n_over + 8'd1;
    end
  end
endmodule

Six migrations. A 128 GB guest, a 250 ms rebind, a 2000 ms budget.

Two migrations exceeded the budget; the all-rebinds model reported one.

Pooling makes migration cheaper and not free, and the difference is the split. A guest with 16 GB on the host and 112 in the pool copies an eighth of its footprint — a genuine and large improvement — and a guest with everything local copies all of it, at which point the pool has contributed nothing to the migration at all.

Row three is the endpoint the architecture is aiming at. A guest whose memory is entirely pooled migrates in the rebind time alone, and that is the property that makes pooled memory operationally interesting to a cloud rather than merely cheaper: it turns a bandwidth-bounded operation into a control-plane one.

Row four is the failure that looks like a stall. With no destination link the copy never completes, and the model reports an unbounded time rather than a small one — which is the honest answer and the one an operator needs, because a migration that cannot proceed is a different incident from one that is slow.

9. RTL 5 — Near And Far Are Different Products

// RTL 5 - near and far are different products. A pool sold as one class quotes
// the near latency and delivers whichever class the page landed in.
module capacity_class #(parameter int ONE_CLASS = 0) (
  input  logic clk, rst_n,
  input  logic        place,
  input  logic        placed_far,
  input  logic [15:0] slo_ns, near_ns, far_ns,
  output logic [15:0] achieved_ns, margin_ns,
  output logic        meets_slo,
  output logic [7:0]  n_placements, n_missed,
  output logic        class_collapsed_err
);
  // One class means one quoted latency, and it is the flattering one.
  assign achieved_ns = (ONE_CLASS != 0) ? near_ns : (placed_far ? far_ns : near_ns);
  assign margin_ns = (slo_ns > achieved_ns) ? (slo_ns - achieved_ns) : 16'd0;
  assign meets_slo = (achieved_ns <= slo_ns);
  // A page placed in the far class and costed at the near one.
  assign class_collapsed_err = place && placed_far && (far_ns != near_ns)
                               && (achieved_ns == near_ns);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_placements <= 8'd0; n_missed <= 8'd0;
    end else if (place) begin
      n_placements <= n_placements + 8'd1;
      if (!meets_slo) n_missed <= n_missed + 8'd1;
    end
  end
endmodule

Seven placements. A near class at 100 ns.

Two placements missed the SLO; the one-class model reported none.

Selling a pool as one product is a pricing decision that the placement engine then has to satisfy and cannot. A tenant on a 200 ns SLO whose pages land in a 250 ns class misses it every time, and nothing in the tenant's request expressed which class it needed because the catalogue has one entry.

Row seven is the configuration that makes the check honest, and it is real: a "far" pool on the same switch tier as the near one delivers the same latency, so placing a page there costs nothing. The error is collapsing classes that differ, not having more than one name for the same thing — and section 17 records that this case was the one a mutation demanded.

Rows three and four are the two boundaries the SLO test defines, driven separately: an SLO relaxed to exactly the far latency, and a far class tightened to exactly the SLO. They are arithmetically identical and operationally opposite — one is a product decision and the other is an engineering one.

10. RTL 6 — A Staged Rollout Bounds What A Bad Config Reaches

// RTL 6 - a staged rollout bounds what a bad configuration reaches. Rolling to
// the fleet at once makes the fleet the test.
module staged_rollout #(parameter int ALL_AT_ONCE = 0) (
  input  logic clk, rst_n,
  input  logic        roll,
  input  logic [15:0] fleet_racks, staged_racks,
  input  logic [7:0]  defect_rate_pct,
  output logic [15:0] exposed_racks, affected_racks,
  output logic        contained,
  output logic [7:0]  n_rollouts, n_uncontained,
  output logic        staging_ignored_err
);
  logic [31:0] a_q;
  // A stage is only a stage if it is smaller than the fleet.
  assign exposed_racks = (ALL_AT_ONCE != 0) ? fleet_racks
                       : ((staged_racks > fleet_racks) ? fleet_racks : staged_racks);
  assign a_q = ({16'd0, exposed_racks} * {24'd0, defect_rate_pct}) / 32'd100;
  assign affected_racks = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign contained = (affected_racks <= 16'd16);
  // A staged rollout that reached the whole fleet anyway.
  assign staging_ignored_err = roll && (staged_racks < fleet_racks)
                               && (exposed_racks == fleet_racks);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_rollouts <= 8'd0; n_uncontained <= 8'd0;
    end else if (roll) begin
      n_rollouts <= n_rollouts + 8'd1;
      if (!contained) n_uncontained <= n_uncontained + 8'd1;
    end
  end
endmodule

Six rollouts. A 256-rack fleet.

Three rollouts were uncontained when the stage bounds it; five when the fleet is the stage.

A pool configuration is fleet-wide state and that is what makes staging non-negotiable. 23.2 §12 established the fabric manager as a single point of failure for fleet growth; this is the same component as a single point of failure for fleet correctness, and the only defence is that a bad change reaches eight racks before it reaches 256.

Row two is the case that makes the check subtle. A good configuration rolled to the whole fleet affects nothing, so the outcome is fine — and staging_ignored_err still fires, correctly, because the process was wrong and the outcome was luck. A check that only fires when something breaks cannot distinguish the two.

Row five is the plan rather than a failure, and the model says so: the final stage of a staged rollout is the fleet, and reaching it deliberately is not the same as starting there.

11. RTL 7 — The Pooling Gain Is A Bet On Correlation

// RTL 7 - the pooling gain is a bet on tenants not peaking together, and the
// correlation between them is the only term that decides whether it pays.
module demand_correlation #(parameter int ASSUME_INDEPENDENT = 0) (
  input  logic clk, rst_n,
  input  logic        size_it,
  input  logic [15:0] tenants, mean_each_gb, peak_each_gb, pool_gb,
  input  logic [7:0]  correlation_pct,
  output logic [15:0] mean_total_gb, peak_total_gb, required_gb,
  output logic        sufficient,
  output logic [7:0]  n_sizings, n_short,
  output logic        correlation_ignored_err
);
  logic [31:0] m_q, p_q, r_q;
  assign m_q = {16'd0, tenants} * {16'd0, mean_each_gb};
  assign mean_total_gb = (m_q > 32'd65535) ? 16'hFFFF : m_q[15:0];
  assign p_q = {16'd0, tenants} * {16'd0, peak_each_gb};
  assign peak_total_gb = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  // Requirement rises from the aggregate mean toward the sum of peaks as the
  // tenants become correlated. Assuming independence keeps it at the mean.
  assign r_q = (ASSUME_INDEPENDENT != 0) ? {16'd0, mean_total_gb}
             : ({16'd0, mean_total_gb}
                + ((({16'd0, peak_total_gb} - {16'd0, mean_total_gb})
                    * {24'd0, correlation_pct}) / 32'd100));
  assign required_gb = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  assign sufficient = (required_gb <= pool_gb);
  // Correlated tenants sized as if they were independent.
  assign correlation_ignored_err = size_it && (correlation_pct != 8'd0)
                                   && (peak_total_gb > mean_total_gb)
                                   && (required_gb == mean_total_gb);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_sizings <= 8'd0; n_short <= 8'd0;
    end else if (size_it) begin
      n_sizings <= n_sizings + 8'd1;
      if (!sufficient) n_short <= n_short + 8'd1;
    end
  end
endmodule

Six sizings. Sixteen tenants averaging 32 GB and peaking at 64, into a 768 GB pool.

Two sizings were short when correlation is modelled; one when independence is assumed.

Correlation is the only term in the pooling business case, and it is the one nobody measures. At zero correlation the pool needs the aggregate mean; at full correlation it needs the sum of peaks; and the entire saving lives in the space between, which is a statistical property of the tenant population rather than of the hardware.

Row three is the design point and it is uncomfortably close to the edge. A pool sized at 768 GB is sufficient at exactly 50% correlation and short at 51 — and correlation is not stationary. A public holiday, a viral event or a regional outage moves the whole population together, which is precisely when the pool is most needed.

Row five is the honest degenerate case. Tenants whose peak equals their mean have no spread for correlation to act on, so the requirement is the same at any correlation — and pooling them saves nothing, which is 23.1 §6's finding arriving through a different door.

12. RTL 8 — A Pool Strands Too

// RTL 8 - a pool strands too. Capacity free in one rack's pool is not capacity
// another rack can have, so a fleet with spare can still refuse a request.
module pool_stranding #(parameter int POOL_HAS_NO_STRAND = 0) (
  input  logic clk, rst_n,
  input  logic        request,
  input  logic [15:0] racks, pool_per_rack_gb, used_per_rack_gb, short_gb,
  output logic [15:0] fleet_pool_gb, fleet_used_gb, fleet_spare_gb, rack_free_gb,
  output logic        servable,
  output logic [7:0]  n_requests, n_denied,
  output logic        scope_ignored_err
);
  logic [31:0] p_q, u_q;
  assign p_q = {16'd0, racks} * {16'd0, pool_per_rack_gb};
  assign fleet_pool_gb = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign u_q = {16'd0, racks} * {16'd0, used_per_rack_gb};
  assign fleet_used_gb = (u_q > 32'd65535) ? 16'hFFFF : u_q[15:0];
  assign fleet_spare_gb = (fleet_pool_gb > fleet_used_gb)
                          ? (fleet_pool_gb - fleet_used_gb) : 16'd0;
  assign rack_free_gb = (pool_per_rack_gb > used_per_rack_gb)
                        ? (pool_per_rack_gb - used_per_rack_gb) : 16'd0;
  // A request is served from the rack's own pool, not from the fleet's total.
  assign servable = (POOL_HAS_NO_STRAND != 0) ? (short_gb <= fleet_spare_gb)
                                              : (short_gb <= rack_free_gb);
  // A request served from capacity that is in another rack.
  assign scope_ignored_err = request && servable && (short_gb > rack_free_gb);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_denied <= 8'd0;
    end else if (request) begin
      n_requests <= n_requests + 8'd1;
      if (!servable) n_denied <= n_denied + 8'd1;
    end
  end
endmodule

Five requests. Sixteen racks with 64 GB pools each, 48 GB used per rack.

Three requests were denied when the rack is the scope; two when the fleet is.

This is 23.1 §5's stranding one level up, and it is why section 5's scope decision matters so much. The fleet holds 256 GB of spare pool capacity, none of which can serve a rack whose own pool has 16 GB free — the strand moved from the socket to the rack and did not disappear.

Row four is the case that separates a scope problem from a capacity problem. A 300 GB request fits neither the rack nor the fleet, so widening the scope would not help — and telling the two apart is the whole operational value of reporting both numbers, because one is fixed by cabling and the other by buying.

Row one is what a fleet dashboard shows as healthy. 256 GB spare across sixteen racks reads as a quarter-empty pool, and the request in front of you is refused — which is 23.3 §8's aggregation fallacy expressed as an allocation failure rather than as a number.

Figure 3 — The dashboard shows 256 GB spare and the request is refused. The dashed edges are the whole problem: capacity that is counted and cannot be reached, which is the same sentence 23.1 opened with about a socket.

13. RTL 9 — A Fleet Must Be Able To Absorb A Failed Rack

// RTL 9 - evacuation headroom. A fleet has to be able to absorb a failed rack,
// and the capacity reserved for that is not capacity the pool can sell.
module evacuation_headroom #(parameter int IGNORE_EVACUATION = 0) (
  input  logic clk, rst_n,
  input  logic        plan,
  input  logic [15:0] racks, pool_per_rack_gb, used_per_rack_gb, racks_to_survive,
  output logic [15:0] total_pool_gb, used_gb, evac_reserve_gb, usable_pool_gb,
  output logic [15:0] effective_util_pct,
  output logic        headroom_ok,
  output logic [7:0]  n_plans, n_short,
  output logic        evacuation_ignored_err
);
  logic [31:0] t_q, u_q, e_q, p_q;
  assign t_q = {16'd0, racks} * {16'd0, pool_per_rack_gb};
  assign total_pool_gb = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign u_q = {16'd0, racks} * {16'd0, used_per_rack_gb};
  assign used_gb = (u_q > 32'd65535) ? 16'hFFFF : u_q[15:0];
  assign e_q = {16'd0, racks_to_survive} * {16'd0, used_per_rack_gb};
  assign evac_reserve_gb = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  // Capacity held back to absorb a failure is not capacity the pool can sell.
  assign usable_pool_gb = (IGNORE_EVACUATION != 0) ? total_pool_gb
                        : ((total_pool_gb > evac_reserve_gb)
                           ? (total_pool_gb - evac_reserve_gb) : 16'd0);
  assign p_q = (usable_pool_gb == 16'd0) ? 32'd65535
             : (({16'd0, used_gb} * 32'd100) / {16'd0, usable_pool_gb});
  assign effective_util_pct = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign headroom_ok = (used_gb <= usable_pool_gb);
  // An evacuation reserve that was never held back.
  assign evacuation_ignored_err = plan && (racks_to_survive != 16'd0)
                                  && (usable_pool_gb == total_pool_gb);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_short <= 8'd0;
    end else if (plan) begin
      n_plans <= n_plans + 8'd1;
      if (!headroom_ok) n_short <= n_short + 8'd1;
    end
  end
endmodule

Seven plans. Sixteen racks with 64 GB pools, 32 GB used per rack — 1024 held, 512 used.

Three plans were short when the reserve is held back; the no-evacuation model reported none.

Effective utilisation is the number a cloud actually runs against, and it is the fleet's usage divided by the capacity the fleet can sell — not by what it contains. Fifty percent becomes 66% the moment an eight-rack failure budget is honoured, and nothing about the workload changed.

Row four is the state a growing fleet drifts into invisibly. A twenty-rack failure budget on this fleet leaves 384 GB sellable against 512 in use, so the fleet is over its own evacuation policy and every dashboard that divides by the total reads 50%. 23.3 §9 called this headroom being spent; here it is spent by a policy nobody re-derived after the fleet grew.

Row seven is what a mis-scaled policy field produces, and the floor is why the model reports zero sellable rather than a wrapped 65,280. Section 17 records that this case was the one a surviving mutation demanded, and it is exactly the class 23.3 §17 named.

14. RTL 10 — Cloud Architectures On CXL Assembled

// RTL 10 - cloud architectures on CXL assembled. Everything that must hold
// before a deployed pool is an architecture rather than a purchase.
module cloud_cxl_model #(parameter int POOL_DEPLOYED = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       pool_deployed,          // pooled capacity is in production
  input  logic       scope_costed,           // reach, blast and latency together
  input  logic       oversubscription_bounded,// the commitment is admissible
  input  logic       classes_separated,      // near and far have their own SLOs
  input  logic       evacuation_reserved,    // a failed rack can be absorbed
  input  logic       rollout_staged,         // a bad config does not reach the fleet
  output logic       sound,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_sound,
  output logic       false_soundness_err
);
  assign fail_mask[0] = ~pool_deployed;
  assign fail_mask[1] = ~scope_costed;
  assign fail_mask[2] = ~oversubscription_bounded;
  assign fail_mask[3] = ~classes_separated;
  assign fail_mask[4] = ~evacuation_reserved;
  assign fail_mask[5] = ~rollout_staged;
  // The deployed-only build reports that the pool is in production, which is
  // what a programme milestone records.
  assign sound = (POOL_DEPLOYED != 0) ? pool_deployed : (fail_mask == 6'd0);
  assign false_soundness_err = evaluate && sound && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_sound <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (sound) n_sound <= n_sound + 8'd1;
    end
  end
endmodule

One sound configuration of six, and four false claims.

"Pooled memory is in production" is a programme milestone and it is right about one of the six. Row four is the one that produces the worst incident: everything about the pool is correct and a rack failure has nowhere to evacuate to, which converts a single-rack fault into a fleet-wide capacity event.

Figure 4 — Four gates and one deployment. Three of the four rejections are fixed by narrowing something — the scope, the ratio, the catalogue — which is why large operators converge on small pools rather than large ones, and why section 5's two-rack design point is where the industry actually sits.

15. Quantitative Reasoning

Scope. A fleet-wide pool across sixteen racks reaches 512 sockets, at 250 ns, with a blast radius of 512. Rack-local reaches 32, at 175 ns, with a blast radius of 32.

Oversubscription. A 1024 GB pool committed to 2048 is two-to-one; a 1536 GB simultaneous demand is served 512 GB short, and a full 2048 GB draw is 1024 short.

Economics. Revenue 20,480 against a pool costing 12,288: four breaches at 1000 leave exactly a 20% margin, eight leave nothing.

Migration. A 128 GB guest with 16 GB host-local copies 1280 ms and rebinds the rest; with 64 GB local it copies 5120 and misses a 2000 ms budget.

Classes. A 200 ns SLO served from a 250 ns far class misses every time, and the one-class model reports the 100 ns near latency.

Rollout. A bad config on eight racks of 256 affects eight; rolled to the fleet it affects 256.

Correlation. Sixteen tenants averaging 32 GB and peaking at 64 need 512 GB uncorrelated, 768 at half correlation, and 1024 fully correlated.

Pool stranding. Sixteen racks with 256 GB spare between them refuse a 32 GB request, because the rack holding it has 16.

Evacuation. A 1024 GB fleet using 512 runs at 50% by the total and 66% against a sellable pool that reserves eight racks — and 133% when it reserves twenty.

The assembled model. Six properties, six configurations, one sound. The deployment milestone reported five.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Every inclusive threshold is driven at exactly equal, and every floor is driven past its boundary — the two rules this batch has converged on.

Scope. A blast radius of exactly 64 is constructed from a two-rack fleet-wide pool, and a one-rack fleet-wide scope is asserted to cost two tiers anyway.

chk(pGr == 16'd32,  "one rack reaches 32 sockets either way");
chk(pGl == 16'd250, "but a fleet-wide scope still crosses two tiers");

Oversubscription. A demand exactly equal to the pool is driven, and a commitment against no pool is asserted to report an unbounded ratio.

Economics. A margin of exactly 20% is constructed from four breaches, and penalties exceeding the gross are asserted to floor the net at zero.

Migration. A total of exactly 2000 ms is constructed from 28 GB at 128 Gbps, and the entirely-pooled guest is asserted as not an ignored copy.

Classes. Both SLO boundaries are driven separately, and a far class exactly as fast as the near one is asserted as not a collapse.

chk(cGa == 16'd100, "a far class at the near latency delivers 100 ns");
chk(cGe == 1'b0,    "and placing far costs nothing when the classes coincide");

Rollout. A stage of exactly sixteen racks is driven at the containment threshold, a stage larger than the fleet is asserted capped, and the final stage is asserted as the plan rather than an error.

Correlation. A requirement of exactly the pool size is constructed at 50% correlation, and a population with no spread is asserted unaffected by correlation.

Pool stranding. A request exactly the size of the rack's free capacity is driven, and a request larger than the fleet's spare is asserted denied by both models.

Evacuation. A reserve leaving exactly the used capacity is driven, and a reserve larger than the fleet is asserted to floor at zero sellable.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 288 checks across two testbenches, 147 on the front five models and 141 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Sixty-two mutations were injected one at a time. 62 injected, 62 killed, after two survivors.

Both survivors were the class 23.3 §17 named, and the rule it produced caught them immediately.

Survivor 1 — a guard for a configuration the model is not named after. Section 9's collapse check requires the far and near classes to differ. Every case had them differing, because the model is about the difference — so the guard read as dead. A far class exactly as fast as the near one is a real deployment, a pool on the same switch tier, and driving it kills the mutation.

Survivor 2 — a floor for a policy field larger than the fleet. Section 13's reserve guard needed an evacuation budget exceeding the pool. The nearest case reserved exactly the fleet, where the guard and the bare subtraction agree at zero — so the equality case was present and the past-boundary case was not. Forty racks of reserve against a sixteen-rack fleet kills it, and without the guard it wraps to 65,280.

Survivor 2 is worth stating as a general point. Batch 022's second chapter added drive every ceiling off its boundary; this is the same idea for a floor. A guard at a > b is not tested by a == b — the equality case makes the guard and the unguarded expression agree, which is precisely why it feels like coverage and is not.

The complete set was re-run after every stimulus change, per standing discipline, and all sixty-two held.

18. Verification Strategy

What a testbench for a deployment model must cover.

Drive the configuration the model is not named after. A latency-class model needs classes that coincide. A saving model needs a loss. A guard exists for the case outside the model's own premise, which is exactly the case a testbench built around the premise omits.

Drive floors past their boundary, not at it. Equality makes a floor and its unguarded expression agree. Only strictly past the boundary distinguishes them.

The cases where the deployment proposal is right. A rack-local scope. A one-to-one commitment. An entirely pooled guest. A near placement. Coinciding classes. A final stage that is the fleet. A tenant population with no spread. Seven cases across nine models, each exempted explicitly, and each a real configuration.

Separate the process failure from the outcome failure. Section 10's staged rollout fires its error on a good configuration rolled to the whole fleet, because the process was wrong and the outcome was luck. A check that only fires when something breaks cannot tell those apart.

Counters as a second signature. Ten models, ten pairs of totals, differing in all ten — two wide against none, three short against none, three denied against two, three uncontained against five.

What a real cloud needs that these models do not have. Time and feedback. Correlation is not stationary; evacuation budgets are re-derived after growth and rarely before; breach events cluster. Every model here is a snapshot of a system whose interesting behaviour is that it drifts, and section 26 exercise 9 is the closest this chapter comes.

19. Synthesis and Implementation Reality

Section 5's scope is cabling and routing, decided at build time. It is the least reversible decision in the module — 23.1 §19 said the same about hop count, and this is that decision with a tenant population attached.

Section 6's admission control is a control-plane service on the hot path of every allocation, and its correctness is a financial control. It is also the component that has to be right during exactly the events that stress it, which is the hardest availability requirement in the chapter.

Section 8's migration split is a hypervisor property. Which pages are host-local and which are pooled is decided by the memory manager at allocation time, and a guest that has been running for weeks has a split nobody chose — which makes migration cost a function of history rather than of configuration.

Section 9's classes need a catalogue entry each, which is a product decision before it is an engineering one. The placement engine can only honour a distinction the request expresses, so a single SKU makes correct placement impossible regardless of how good the allocator is.

Section 10's staging is fleet-management practice, and pooled memory raises its stakes rather than changing its mechanics: a bad pool configuration is fleet-wide state in a way a bad host configuration is not.

Section 13's evacuation reserve is a policy number that must be recomputed as the fleet grows, because it scales with per-rack usage. A budget set once at commissioning is wrong by construction a year later, and section 21 is what that looks like.

20. Silicon Observability

"Correlation between tenant demands" is the counter this module has been circling since 23.1 §6 and nobody collects. It is the single term that decides whether pooling saves anything, it is a property of the tenant population rather than of the hardware, and it is not stationary — which means a one-off study at design time answers a question that will have a different answer by the time the pool is in production.

21. Debug Lab

Symptom. An operator deploys a fleet-wide pool across 256 racks. It runs for a year without incident. Then, over one weekend: a switch failure takes 512 sockets offline, a regional traffic event breaches 40 tenants' entitlements, and the fleet cannot evacuate the failed racks because there is nowhere to put them. All three within nine hours.

Step 1 — the switch failure. Blast radius at the deployed scope: 512 sockets share a second-tier switch. Section 5, and the design document recorded the reach — 512 sockets of pooled capacity — without recording that the same number is the failure unit. The number was known and was on the wrong line of the page.

Step 2 — the breaches. Simultaneous demand peak: 1536 GB against a 1024 GB pool, at a two-to-one commitment. Section 6, and admission control did exactly what it was configured to do. The commitment was never wrong; the assumption behind it was.

Step 3 — why now? Tenant demand correlation, over the event window: 0.72, against a design assumption of independence. Section 11. A regional traffic event moves the whole population together, and the pool was sized for 512 GB when 0.72 correlation requires 880 — which is why a year of quiet was not evidence.

Step 4 — the evacuation. Evacuation reserve against current per-rack usage: the policy reserves two racks' worth, set at commissioning when per-rack usage was 32 GB. It is now 58. Section 13, and the reserve is a fixed rack count against a usage figure that nearly doubled — the policy was correct and its inputs went stale.

Step 5 — and the effective utilisation. Used against sellable: the dashboard reads 71% against the total. Against the sellable pool with a correct two-rack reserve at today's usage, it reads 89%. 23.3 §9 and section 13 together: the fleet has been over its own headroom policy for months and no report divided by the right denominator.

Step 6 — was any of it a surprise? Every one of the four numbers was computable at design time from data the operator had. The correlation was the only one that needed measurement, and it is the only one nobody had a counter for.

The finding. One scope decision produced three of the four failures. A fleet-wide scope set the blast radius at 512, put the whole tenant population in one correlated pool, and made the evacuation budget a fleet-level policy that drifted. A two-rack scope — section 5's row three — would have bounded all three at the cost of some pooling gain.

The fix. Re-scope to two-rack pools, which caps the blast radius at 64 and puts smaller, less correlated tenant groups in each pool. Re-derive the evacuation reserve from current usage and put it on a schedule. Add a correlation counter, because it is the only term in the business case that was never measured and the only one that moved.

What made this hard. Nothing failed for a year, and then three things failed together because they share a cause. The scope decision was made once, early, by people reasoning about reach — and reach was the only one of its three consequences anyone wrote down.

22. Design Review

1. What is the pool's scope, and what blast radius and latency does that scope imply? Three numbers from one decision. Section 5.

2. What is the oversubscription ratio, and what happens at a full simultaneous draw? A two-to-one pool leaves half the commitment unserved by construction. Section 6.

3. What does a breach cost, and how many can the margin absorb? Four halve it and eight erase it. Section 7.

4. What fraction of a guest's memory is host-local, and how does that vary with guest age? It decides migration cost and nobody chose it. Section 8.

5. Does the catalogue have a near and a far entry, or one entry? Placement can only honour a distinction the request expresses. Section 9.

6. Is a pool configuration change staged, and is the staging enforced? Fleet-wide state needs it more than host state does. Section 10.

7. What is the measured correlation of tenant demand, and when was it last measured? The only term in the business case. Section 11.

8. Is free pool capacity reported per rack or summed? 256 GB spare and a 32 GB request refused. Section 12.

9. When was the evacuation reserve last re-derived from current per-rack usage? It scales with the fleet and the policy does not. Section 13.

10. Which of the six properties does "pooled memory is in production" imply? Section 14 exists because the answer is the first one only.

23. How This Appears In Real Engineering

A platform architect owns section 5, and the trap is that the decision is made in a document rather than in a review. Reach is the number the proposal is about; blast radius and latency are consequences nobody has to state, and section 21 is a year later.

A capacity and pricing function owns sections 6, 7 and 11 together, and they form one question: what ratio can be sold, given a correlation nobody has measured, against a penalty schedule that is the same size as the gross margin. Two of those three inputs are contractual and the third is empirical, which is why the third is usually assumed.

A fleet operations team owns sections 10 and 13, and both are process rather than architecture. The evacuation reserve is the one that fails silently: it is correct when set and wrong later, with no event marking the transition, which makes a scheduled re-derivation the only defence.

A hypervisor or virtualisation team owns section 8, and the finding is uncomfortable: migration cost is a function of a guest's allocation history, so the same guest type migrates cheaply or expensively depending on what the memory manager did weeks ago. Making the split a policy rather than an accident is the available fix.

24. Common Misconceptions

"A fleet-wide pool is strictly better — more reach." And sixteen times the blast radius, and a second switch tier. Section 5.

"Oversubscription is a risk we should eliminate." It is the product. The question is the ratio and the admission policy. Section 6.

"A breach is an availability problem." It is a financial one — eight breaches erase the margin. Section 7.

"Pooled memory makes live migration free." It makes it cheaper by the pooled fraction. A guest with everything host-local gains nothing. Section 8.

"Memory is memory." Near and far are different products with different SLOs, and one catalogue entry cannot express the difference. Section 9.

"It is a configuration change, not a deployment." A pool configuration is fleet-wide state. Section 10.

"Tenants are independent." Until a regional event, which is when the pool is most needed. Section 11.

"The fleet has 256 GB spare." In fifteen other racks. Section 12.

"We are at 71% utilisation." Against the total. Against what you can sell, 89%. Section 13.

"The pool is in production." One property of six. Section 14.

25. Interview Reasoning

Q. How wide should a memory pool be?

Narrower than the reach argument suggests. Widening raises the reach, and raises the blast radius by the same factor and the latency by the switch tiers crossed — 512 sockets and 250 ns fleet-wide against 32 sockets and 175 ns rack-local. Most large operators land at two or four racks, because that doubles or quadruples the reach while keeping the failure unit inside an acceptance threshold.

Q. Is oversubscribing a memory pool wrong?

No — it is the business model, and every cloud does it. The failure is treating the commitment as backed: a 1024 GB pool committed to 2048 leaves half the commitment unserved when everyone draws at once, and that outcome is guaranteed by the ratio rather than caused by a fault. The engineering is admission control and the economics are the penalty schedule.

Q. Why did a year of quiet not prove the pool was sized correctly?

Because the sizing depends on tenant correlation and a quiet year is a low-correlation year. Sixteen tenants averaging 32 GB need 512 GB uncorrelated and 1024 fully correlated — and a regional event moves the whole population together, which is exactly when the pool is most needed and least sufficient.

Q. What does pooled memory do to live migration?

It splits the guest's footprint into pages that rebind and pages that must be copied. A 128 GB guest with 16 GB host-local copies an eighth of itself — a real improvement — and one with everything host-local gains nothing. The follow-up worth knowing: the split is a function of allocation history, so the same guest type varies.

Q. Your fleet dashboard shows 256 GB spare and an allocation just failed. Explain.

The 256 is a sum across racks and the allocation is served from one rack's pool. If each rack has 16 GB free, the largest allocation is 16 — the capacity is free, idle, and in the wrong rack. It is the aggregation fallacy expressed as an allocation failure, and the fix is a per-rack counter replacing the fleet total.

Q. Why can a fleet be at 71% utilisation and out of headroom?

Because it is 71% of what the fleet contains and the fleet cannot sell all of it. Capacity reserved to absorb a failed rack is not sellable, so against the sellable pool the same fleet is at 89% — and the reserve scales with per-rack usage, so a budget set at commissioning is wrong once usage grows.

26. Exercises

1. Extend RTL 1 to a three-tier fabric and find the scope at which the latency tax exceeds a workload's tolerance.

2. Give RTL 2 an admission controller that rejects commitments above a threshold, and find the threshold that bounds breaches at four a year.

3. Combine RTL 3 and RTL 7: find the oversubscription ratio that maximises net margin for a given correlation.

4. Make RTL 4's host-local fraction a function of guest age and show how migration cost drifts.

5. Extend RTL 5 to three classes and price the catalogue complexity against the placement accuracy it buys.

6. Give RTL 6 a multi-stage schedule and find the stage sizes that minimise total exposure for a given confidence.

7. Make RTL 7's correlation a time series with a rare high-correlation event, and size the pool for a target breach rate.

8. Combine RTL 8 and RTL 1: show that narrowing the scope reduces the blast radius and increases the stranding, and find the balance.

9. Model section 21 end to end: a fleet-wide scope, a correlation event, a stale evacuation budget and a switch failure in one weekend.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the deployment it catches that the current mask calls sound.

27. Summary

Module 23 opened with a quarter of a fleet's DRAM bought and unreachable, and pooled it. It then composed machines from several pools, and then found that the metric both were judged by can move thirty points without the fleet moving. This chapter is an operator holding all three at once, and the finding that organises it is one decision.

Scope sets reach, blast radius and latency together. A fleet-wide pool across sixteen racks reaches 512 sockets, takes 512 down, and costs 250 ns; a rack-local one reaches 32, takes 32 down, and costs 175. Only the first of the three appears on the business case.

Oversubscription is the product, not a defect. A 1024 GB pool committed to 2048 serves a full simultaneous draw 1024 GB short — guaranteed by the ratio — and admission control is what keeps the fleet on the right side of the line.

A breach costs money the price list does not carry. Four breaches at 1000 take a 40% margin to exactly 20%, and eight take it to nothing — a penalty schedule the same size as the entire gross.

Pooling makes migration cheaper and not free. A 128 GB guest with 16 GB host-local copies 1280 ms and rebinds the rest; one with everything local gains nothing at all.

Near and far are different products. A 200 ns SLO served from a 250 ns class misses every time, and a catalogue with one entry cannot express a distinction the placement engine is then asked to honour.

A pool configuration is fleet-wide state. Eight racks affected against 256, and the staging check fires even when the outcome was fine — because the process was wrong and the outcome was luck.

The gain is a bet on correlation and nobody measures it. Sixteen tenants need 512 GB uncorrelated, 768 at half correlation and 1024 fully correlated — and correlation is not stationary, which is why a quiet year proves nothing.

A pool strands too. Sixteen racks with 256 GB spare between them refuse a 32 GB request, because the rack holding it has 16 — 23.1's opening sentence, one level up.

And a fleet must be able to absorb a failed rack. 50% utilisation by the total is 66% against a sellable pool reserving eight racks, and 133% reserving twenty — a policy that is correct when set and wrong once the fleet grows.

Two mutations survived on guards for cases outside the models' own premises — a far class exactly as fast as the near one, and an evacuation reserve larger than the fleet. The rule the batch has converged on covered both: drive the configuration the model is not named about, and drive floors past their boundary rather than at it.

Deploying a pool is one property of six. The milestone a programme reports called five of six architectures sound when one was — and section 21 is an operator whose scope decision produced three separate weekend incidents that shared a single cause.

Module 23 is complete. From stranding in 23.1, through composition in 23.2 and measurement in 23.3, to deployment here — the same finding at four scales: capacity you cannot reach is not capacity, and every architecture that fixes it moves the boundary rather than removing it.

24.1 — CXL Protocol Engines leaves the data centre entirely and goes back inside the silicon: the RTL that implements .io, .cache and .mem, and how one link's traffic is dispatched to three engines that share nothing but a physical layer.