Memory Disaggregation

Module 22 spent five chapters on one finding at five scales: capacity is not throughput. Module 23 turns the same lens on the data centre, and the first thing it finds is stranger than a throughput problem.

A quarter of a fleet's DRAM is typically bought, powered, and unreachable. Not idle — unreachable. It is soldered to a socket whose workload does not need it, and no other socket in the building can borrow a byte of it. That is not an efficiency problem; it is a topology problem, and it is the reason CXL exists at all.

This chapter is about what fixes it, and about the eight things that have to be true before the fix is a gain rather than a rearrangement.

1. The Engineering Problem — Memory Is Bought With The Socket

A quarter of the fleet's DRAM is stranded. Sixteen sockets fitted with 40 GB each, using 30 — 160 GB of 640 that nobody can reach. Section 5.

Pooling wins because peaks do not coincide. Sixteen sockets each peaking at 64 GB need 1024 provisioned separately and 640 as a pool — a 37% saving. Section 6.

And the failure domain grows with the sharing. Eight devices serving 64 sockets means one device failure takes eight sockets, not one. Section 7.

Every pooled access crosses a switch. 100 ns local becomes 250 ns at two hops — a 150% tax on every read. Section 8.

Memory is handed out in regions, not bytes. A 6 GB request in a 16 GB region wastes 62% of what it occupies. Section 9.

This chapter against 22.2, stated precisely. That one owns how much memory one workload needs. This one owns why a fleet with enough memory in aggregate is short on individual sockets — a question that has no single-node form, which is why sections 7, 12 and 13 have no counterpart there.

2. The One-Sentence Model

Disaggregation improves a fleet when the strand is genuinely recovered, the capacity really is pooled rather than relabelled, the fabric hop is inside the access budget, one device failure is survivable, bandwidth is quoted per socket rather than per device, and reclaim time is in the capacity plan — and every defect below is a fleet that moved its DIMMs and bought nothing.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real disaggregated fleet is a fabric manager, a switch, a set of memory devices, a hypervisor and a scheduler, and none of that is reproduced. What is reproduced is the arithmetic each of them has to get right, and the shape of the mistake when it does not.

Each model is built twice — a correct build and a broken build selected by a parameter. Every broken build here is the direct-attached mental model applied to a pool. The memory is all mine. The device is all mine. A failure affects me alone. Each was true when the DIMM was in the socket, and each is false the moment it is not — which is why they survive the transition invisibly.

Figure 1 — The dashed edge on the left is the whole problem: 160 GB that exists, draws power, and cannot be used. The dashed edge on the right is what sections 8 through 13 price, and it is the reason the gain is not automatic.

5. RTL 1 — Memory Bought With A Socket Strands With It

// RTL 1 - stranded memory. A socket's DRAM is bought with the socket, so memory
// the socket cannot use is memory nobody can use.
module stranded_memory #(parameter int ASSUME_FULLY_USED = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [15:0] sockets, dram_per_socket_gb, used_per_socket_gb,
  output logic [15:0] fitted_gb, used_gb, stranded_gb, stranded_pct,
  output logic        efficient,
  output logic [7:0]  n_assessments, n_wasteful,
  output logic        strand_hidden_err
);
  logic [31:0] f_q, u_q, p_q;
  assign f_q = {16'd0, sockets} * {16'd0, dram_per_socket_gb};
  assign fitted_gb = (f_q > 32'd65535) ? 16'hFFFF : f_q[15:0];
  // A model that assumes full utilisation reports the fitted capacity as used.
  assign u_q = (ASSUME_FULLY_USED != 0) ? {16'd0, fitted_gb}
             : ({16'd0, sockets} * {16'd0, used_per_socket_gb});
  assign used_gb = (u_q > 32'd65535) ? 16'hFFFF : u_q[15:0];
  // used_gb never exceeds fitted_gb while used_per_socket_gb is at or below
  // dram_per_socket_gb, and the guard below covers the case where it is not.
  assign stranded_gb = (fitted_gb > used_gb) ? (fitted_gb - used_gb) : 16'd0;
  assign p_q = (fitted_gb == 16'd0) ? 32'd0
             : (({16'd0, stranded_gb} * 32'd100) / {16'd0, fitted_gb});
  assign stranded_pct = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign efficient = (stranded_pct <= 16'd20);
  // Memory that is fitted and unused, reported as used.
  assign strand_hidden_err = assess && (fitted_gb != 16'd0)
                             && (used_per_socket_gb < dram_per_socket_gb)
                             && (stranded_gb == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assessments <= 8'd0; n_wasteful <= 8'd0;
    end else if (assess) begin
      n_assessments <= n_assessments + 8'd1;
      if (!efficient) n_wasteful <= n_wasteful + 8'd1;
    end
  end
endmodule

Six assessments. Sixteen sockets fitted with 40 GB each — 640 GB in the rack.

Two assessments were wasteful; the fully-used model reported none.

"Stranded" is a stronger word than "idle" and the distinction is the chapter. Idle memory is available to whatever needs it next. Stranded memory is available to exactly one socket, and if that socket's workload has been sized correctly it will never need it. The 160 GB in row one is not waiting to be used; it is structurally unusable.

Row four is what good packing achieves and why it is not enough. A carefully bin-packed fleet still strands 10%, because the packing has to leave headroom for growth on every socket independently. Pooling lets one headroom serve sixteen sockets, which is section 6's arithmetic.

Why the broken build is not a strawman. A capacity report built from purchase records reports fitted capacity, and it is correct about what was bought. Utilisation requires a different measurement entirely — per-socket, sampled, and often absent — so a fleet with no utilisation telemetry reports exactly the broken build's numbers and has no way of knowing better.

6. RTL 2 — Pooling Wins Because Peaks Do Not Coincide

// RTL 2 - pooling against per-socket provisioning. A pool is provisioned for the
// aggregate peak; sockets are each provisioned for their own peak.
module pool_provisioning #(parameter int PROVISION_FOR_SUM = 0) (
  input  logic clk, rst_n,
  input  logic        provision,
  input  logic [15:0] sockets, peak_per_socket_gb, aggregate_peak_gb,
  output logic [15:0] per_socket_total_gb, pooled_total_gb, saved_gb, saved_pct,
  output logic        pooling_wins,
  output logic [7:0]  n_provisions, n_wins,
  output logic        no_saving_err
);
  logic [31:0] t_q, s_q;
  assign t_q = {16'd0, sockets} * {16'd0, peak_per_socket_gb};
  assign per_socket_total_gb = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  // A pool sized to the sum of the peaks has bought nothing; the whole point is
  // that the peaks do not coincide.
  assign pooled_total_gb = (PROVISION_FOR_SUM != 0) ? per_socket_total_gb
                                                    : aggregate_peak_gb;
  assign saved_gb = (per_socket_total_gb > pooled_total_gb)
                    ? (per_socket_total_gb - pooled_total_gb) : 16'd0;
  assign s_q = (per_socket_total_gb == 16'd0) ? 32'd0
             : (({16'd0, saved_gb} * 32'd100) / {16'd0, per_socket_total_gb});
  assign saved_pct = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  assign pooling_wins = (saved_pct >= 16'd25);
  // A pool sized to the sum of the peaks, reported as a pool.
  assign no_saving_err = provision && (aggregate_peak_gb < per_socket_total_gb)
                         && (saved_gb == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_provisions <= 8'd0; n_wins <= 8'd0;
    end else if (provision) begin
      n_provisions <= n_provisions + 8'd1;
      if (pooling_wins) n_wins <= n_wins + 8'd1;
    end
  end
endmodule

Five provisionings. Sixteen sockets each peaking at 64 GB.

Two wins on the aggregate peak; none on the sum of peaks.

The entire economic case for pooling is one statistical fact: peaks do not coincide. Sixteen sockets that each could need 64 GB do not all need it at the same moment, so the aggregate peak is 640 rather than 1024. If they did coincide, pooling would save exactly nothing — which row two states and which the model treats as a legitimate answer rather than an error.

Row two is therefore the honest limit and it is a real workload class. A fleet running sixteen copies of the same batch job, synchronised, peaks together by construction. Pooling such a fleet buys a fabric and saves nothing, and no amount of good engineering elsewhere changes it.

The sum-of-peaks build is the failure of provisioning a pool defensively. Faced with a pool and no confidence in the correlation, the safe answer is to size for the worst case — and the worst case is the sum. That is a correct capacity decision and a complete waste of a pool, which is why no_saving_err fires only when the aggregate peak really was lower.

7. RTL 3 — The Failure Domain Grows With The Sharing

// RTL 3 - the blast radius. Disaggregated memory serves many sockets, so one
// failed memory device takes down everything holding a page on it.
module blast_radius #(parameter int IGNORE_SHARING = 0) (
  input  logic clk, rst_n,
  input  logic        fail_it,
  input  logic [15:0] sockets, devices, replicas,
  output logic [15:0] sockets_per_device, true_affected, affected_sockets, survivors,
  output logic        contained,
  output logic [7:0]  n_failures, n_uncontained,
  output logic        radius_ignored_err
);
  logic [31:0] s_q, a_q;
  assign s_q = (devices == 16'd0) ? 32'd0
             : ({16'd0, sockets} / {16'd0, devices});
  assign sockets_per_device = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  // The truth is computed unconditionally, so the check below does not depend on
  // the build being tested getting the replica count right either.
  assign a_q = (replicas == 16'd0) ? {16'd0, sockets_per_device}
             : ({16'd0, sockets_per_device} / {16'd0, replicas});
  assign true_affected = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  // A model that ignores sharing charges one socket for a device failure, which
  // is the direct-attached answer.
  assign affected_sockets = (IGNORE_SHARING != 0) ? 16'd1 : true_affected;
  assign survivors = (sockets > affected_sockets) ? (sockets - affected_sockets)
                                                  : 16'd0;
  assign contained = (affected_sockets <= 16'd1);
  // A shared device's failure costed as fewer sockets than it really takes.
  assign radius_ignored_err = fail_it && (affected_sockets < true_affected);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_failures <= 8'd0; n_uncontained <= 8'd0;
    end else if (fail_it) begin
      n_failures <= n_failures + 8'd1;
      if (!contained) n_uncontained <= n_uncontained + 8'd1;
    end
  end
endmodule

Six device failures. Sixty-four sockets.

Three failures were uncontained; the direct-attached model reported none.

Consolidation and blast radius are the same number read in two directions. Eight devices instead of sixty-four is the saving; eight sockets instead of one is the price. They cannot be separated — the sharing that recovers the strand is the sharing that widens the failure.

Replication buys the radius back and costs the capacity again. Two-way replication halves the affected sockets to four and doubles the capacity needed to hold anything replicated. Eight-way replication contains the failure completely and gives back the entire saving — which is row five, and it is the reason replication is applied selectively rather than universally.

The check compares against a separately computed truth. true_affected is derived unconditionally, so a broken build cannot hide behind a replica count it also got wrong. This is the pattern 22.3 §11 established after a broken build passed its own consistency check by being wrong twice in the same direction, and it appears twice in this chapter — here and in section 9.

8. RTL 4 — Every Pooled Access Crosses A Switch

// RTL 4 - the latency tax. Memory that is not on the socket is reached through
// a switch, and every hop is added to every access.
module latency_tax #(parameter int IGNORE_HOP = 0) (
  input  logic clk, rst_n,
  input  logic        access,
  input  logic [15:0] local_ns, switch_ns, hops,
  output logic [15:0] pooled_ns, added_ns, tax_pct,
  output logic        acceptable,
  output logic [7:0]  n_accesses, n_slow,
  output logic        hop_ignored_err
);
  logic [31:0] p_q, t_q;
  // Each hop through the fabric is added to the access, in both directions of
  // the round trip the switch_ns figure already describes.
  assign p_q = (IGNORE_HOP != 0) ? {16'd0, local_ns}
             : ({16'd0, local_ns} + ({16'd0, hops} * {16'd0, switch_ns}));
  assign pooled_ns = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  // pooled_ns is local_ns plus a non-negative term, so this cannot underflow.
  assign added_ns = pooled_ns - local_ns;
  assign t_q = (local_ns == 16'd0) ? 32'd0
             : (({16'd0, added_ns} * 32'd100) / {16'd0, local_ns});
  assign tax_pct = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign acceptable = (tax_pct <= 16'd100);
  // A pooled access costed as if the memory were on the socket.
  assign hop_ignored_err = access && (hops != 16'd0) && (switch_ns != 16'd0)
                           && (pooled_ns == local_ns);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_accesses <= 8'd0; n_slow <= 8'd0;
    end else if (access) begin
      n_accesses <= n_accesses + 8'd1;
      if (!acceptable) n_slow <= n_slow + 8'd1;
    end
  end
endmodule

Six accesses. A 100 ns local access against a fabric charging 75 ns per hop.

Two accesses were too slow; the no-hop model reported none.

This is the cost side of the trade and it is charged on every access. The strand is recovered once; the hop is paid billions of times a second, which is why a 25% capacity saving can be a net loss and section 13 has to compute both.

Row four is the argument against deep fabrics. 21.3 §7 priced switch depth for a fabric; here the same depth is a 300% memory-access tax, which no workload with any locality sensitivity survives. A pool two hops away is a different product from a pool four hops away.

Row six is the case that makes the model honest about its own limits. With no measured local latency there is no baseline, so the tax is undefined rather than zero — the model reports the added nanoseconds and declines to express them as a percentage. A ratio against an unmeasured denominator is not a small error; it is not a number.

The strand is recovered once and the tax is paid forever. That asymmetry is the single most important thing to hold about disaggregation: a one-time capacity gain against a per-access latency cost, and section 13 is the only place the two meet on the same scale.

9. RTL 5 — Memory Is Handed Out In Regions

// RTL 5 - allocation granularity. A pool hands out fixed regions, so a request
// that does not fill one still occupies it.
module allocation_granularity #(parameter int ASSUME_BYTE_EXACT = 0) (
  input  logic clk, rst_n,
  input  logic        alloc_it,
  input  logic [15:0] request_gb, granularity_gb,
  output logic [15:0] regions, true_allocated_gb, allocated_gb, waste_gb, waste_pct,
  output logic        acceptable,
  output logic [7:0]  n_allocations, n_wasteful,
  output logic        rounding_lost_err
);
  logic [31:0] r_q, a_q, w_q;
  assign r_q = (granularity_gb == 16'd0) ? 32'd0
             : (({16'd0, request_gb} + {16'd0, granularity_gb} - 32'd1)
                / {16'd0, granularity_gb});
  assign regions = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  // The truth is computed unconditionally so the check below does not depend on
  // the build being tested.
  assign a_q = (granularity_gb == 16'd0) ? {16'd0, request_gb}
             : ({16'd0, regions} * {16'd0, granularity_gb});
  assign true_allocated_gb = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign allocated_gb = (ASSUME_BYTE_EXACT != 0) ? request_gb : true_allocated_gb;
  assign waste_gb = (allocated_gb > request_gb) ? (allocated_gb - request_gb) : 16'd0;
  assign w_q = (allocated_gb == 16'd0) ? 32'd0
             : (({16'd0, waste_gb} * 32'd100) / {16'd0, allocated_gb});
  assign waste_pct = (w_q > 32'd65535) ? 16'hFFFF : w_q[15:0];
  assign acceptable = (waste_pct <= 16'd20);
  // An allocation reported smaller than the regions it actually occupies.
  assign rounding_lost_err = alloc_it && (allocated_gb < true_allocated_gb);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_allocations <= 8'd0; n_wasteful <= 8'd0;
    end else if (alloc_it) begin
      n_allocations <= n_allocations + 8'd1;
      if (!acceptable) n_wasteful <= n_wasteful + 8'd1;
    end
  end
endmodule

Six allocations.

Two allocations were wasteful; the byte-exact model reported none.

A pool has its own internal stranding and it is a different mechanism. Section 5's strand comes from memory attached to the wrong socket; this one comes from memory attached to the right socket and rounded up. A fleet that recovers 25% by pooling and loses 20% to region rounding has recovered five percent.

Row three is the worst case and it is one gigabyte from the best. Seventeen gigabytes occupies two regions and wastes fifteen. A request one byte over a boundary costs an entire region, which is why the granularity is one of the most consequential numbers in a pool's specification and is almost never discussed.

The check compares against a separately computed true_allocated_gb, for the same reason section 7 does. A model that reports byte-exact allocation and computes its waste from that report finds no waste — its internal arithmetic is perfectly consistent and entirely wrong, and only an unconditionally derived truth catches it.

10. RTL 6 — Reclaim Is Not Instant

// RTL 6 - reclaim is not instant. Memory returned to a pool must be scrubbed
// before it is handed to another tenant, and the pool is short while it is.
module reclaim_time #(parameter int INSTANT_RECLAIM = 0) (
  input  logic clk, rst_n,
  input  logic        reclaim,
  input  logic [15:0] freed_gb, scrub_gbps, churn_per_hour,
  output logic [15:0] scrub_s, unavailable_s_per_hour, unavailable_pct,
  output logic        acceptable,
  output logic [7:0]  n_reclaims, n_costly,
  output logic        free_reclaim_err
);
  logic [31:0] s_q, u_q, p_q;
  assign s_q = (INSTANT_RECLAIM != 0) ? 32'd0
             : ((scrub_gbps == 16'd0) ? 32'd65535
                : (({16'd0, freed_gb} * 32'd8) / {16'd0, scrub_gbps}));
  assign scrub_s = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  assign u_q = {16'd0, scrub_s} * {16'd0, churn_per_hour};
  assign unavailable_s_per_hour = (u_q > 32'd65535) ? 16'hFFFF : u_q[15:0];
  assign p_q = ({16'd0, unavailable_s_per_hour} * 32'd100) / 32'd3600;
  assign unavailable_pct = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign acceptable = (unavailable_pct <= 16'd10);
  // Memory handed straight back to the pool without being scrubbed.
  assign free_reclaim_err = reclaim && (freed_gb != 16'd0) && (scrub_gbps != 16'd0)
                            && (scrub_s == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reclaims <= 8'd0; n_costly <= 8'd0;
    end else if (reclaim) begin
      n_reclaims <= n_reclaims + 8'd1;
      if (!acceptable) n_costly <= n_costly + 8'd1;
    end
  end
endmodule

Six reclaims. 256 GB returned at a time, scrubbed at 80 Gbps.

Three reclaims were costly; the instant-reclaim model reported none.

Memory returned to a pool cannot be handed to the next tenant until it is zeroed, and that is a bandwidth-bounded operation on a large quantity of DRAM. The pool is genuinely short during it — the capacity exists, is powered, and is unavailable, which is section 5's stranding recreated inside the mechanism that was supposed to fix it.

Row three is the churn wall. A pool serving short-lived workloads reclaims constantly, and at 27% unavailability the effective pool is three-quarters of the purchased one. Churn rate belongs in a pool's capacity plan as directly as capacity does, and it almost never is.

Row two is the argument for a fast scrub path, and it is a hardware decision: ten times the scrub bandwidth turns 13% into 1%. Device-side scrub engines exist for exactly this reason, and the number that justifies them is this one.

11. RTL 7 — A Pooled Device's Bandwidth Is Shared

// RTL 7 - shared bandwidth. A pooled device's bandwidth is divided among the
// sockets using it, so a socket's share falls as the pool is subscribed.
module bandwidth_share #(parameter int ASSUME_DEDICATED = 0) (
  input  logic clk, rst_n,
  input  logic        access,
  input  logic [15:0] sockets_active, device_gbps, demand_gbps,
  output logic [15:0] share_gbps, delivered_gbps, shortfall_gbps,
  output logic        demand_met,
  output logic [7:0]  n_accesses, n_starved,
  output logic        contention_ignored_err
);
  logic [31:0] s_q;
  assign s_q = (ASSUME_DEDICATED != 0) ? {16'd0, device_gbps}
             : ((sockets_active == 16'd0) ? {16'd0, device_gbps}
                : ({16'd0, device_gbps} / {16'd0, sockets_active}));
  assign share_gbps = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  assign delivered_gbps = (share_gbps > demand_gbps) ? demand_gbps : share_gbps;
  // delivered_gbps is a minimum against demand_gbps, so this cannot underflow.
  assign shortfall_gbps = demand_gbps - delivered_gbps;
  assign demand_met = (shortfall_gbps == 16'd0);
  // A shared device's bandwidth quoted as if one socket owned it.
  assign contention_ignored_err = access && (sockets_active > 16'd1)
                                  && (share_gbps == device_gbps);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_accesses <= 8'd0; n_starved <= 8'd0;
    end else if (access) begin
      n_accesses <= n_accesses + 8'd1;
      if (!demand_met) n_starved <= n_starved + 8'd1;
    end
  end
endmodule

Five accesses. An 800 Gbps device, each socket wanting 400.

Two accesses were starved; the dedicated model reported none.

Capacity divides gracefully and bandwidth does not. Section 6's pool serves sixteen sockets from 640 GB because they need capacity at different times. Bandwidth is needed at the same time by everyone, so a device's rate divides by the number of sockets actually using it, and the divisor is a runtime property nobody controls.

Row three is the sizing rule. Two sockets on an 800 Gbps device get exactly what they asked for; a third makes all three short. The subscription ratio is the specification, and quoting a pooled device by its own bandwidth is describing a machine with one socket on it.

This is section 5's problem inverted, and it is worth stating. Direct-attached memory strands capacity and never contends for bandwidth. Pooled memory recovers the capacity and introduces the contention — the two failure modes are duals, and choosing between them is choosing which one the workload tolerates.

Figure 3 — One device, two prices. The top path is section 11's bandwidth contention, paid whenever more than a couple of sockets are active. The bottom path is section 7's blast radius, paid once and catastrophically. Both are consequences of the same consolidation that recovered the strand.

12. RTL 8 — What Is Available To You Is Not The Pool

// RTL 8 - a pool is shared. What is available to one socket is the pool minus
// what every other socket currently holds, not the pool.
module pool_contention #(parameter int ASSUME_ISOLATED = 0) (
  input  logic clk, rst_n,
  input  logic        request,
  input  logic [15:0] pool_gb, others_using_gb, request_gb,
  output logic [15:0] available_gb, granted_gb, shortfall_gb,
  output logic        grantable,
  output logic [7:0]  n_requests, n_denied,
  output logic        isolation_assumed_err
);
  logic [15:0] free_gb;
  // What is left after everybody else. A model that assumes isolation offers the
  // whole pool to every socket, which is the direct-attached mental model.
  assign free_gb = (pool_gb > others_using_gb) ? (pool_gb - others_using_gb) : 16'd0;
  assign available_gb = (ASSUME_ISOLATED != 0) ? pool_gb : free_gb;
  assign granted_gb = (request_gb > available_gb) ? available_gb : request_gb;
  // granted_gb is a minimum against request_gb, so this cannot underflow.
  assign shortfall_gb = request_gb - granted_gb;
  assign grantable = (shortfall_gb == 16'd0);
  // A shared pool offered whole to a socket while other sockets held part of it.
  assign isolation_assumed_err = request && (others_using_gb != 16'd0)
                                 && (available_gb == pool_gb);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_denied <= 8'd0;
    end else if (request) begin
      n_requests <= n_requests + 8'd1;
      if (!grantable) n_denied <= n_denied + 8'd1;
    end
  end
endmodule

Five requests. A 1024 GB pool, asking for 512.

Three requests were denied; the isolated model denied none.

A pool's capacity is a fleet property and a socket's availability is a moment. The same 1024 GB pool grants 512 at one instant and nothing at another, with no configuration change and no failure. Direct-attached memory never did this — the DIMM was there or it was not — and the difference is the single largest operational change disaggregation introduces.

Row five is what a real telemetry pipeline delivers. Sockets report their holdings independently and asynchronously, so the sum can briefly exceed the pool. The floor turns a nonsensical negative into an honest zero, and a model without it reports a hugely positive availability from an unsigned wrap — the failure mode this batch's predecessors met repeatedly.

Section 11 and this section are the same contention in two currencies. There, bandwidth divided among active sockets; here, capacity divided among holding sockets. A pool is short of both, at different moments, for different reasons, and a capacity plan that models one and not the other is half a plan.

13. RTL 9 — What Disaggregation Costs

// RTL 9 - what disaggregation costs. Fewer gigabytes of DRAM, plus a fabric that
// direct-attached memory does not need, against the same usable capacity.
module disaggregation_tco #(parameter int CAPEX_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        price,
  input  logic [15:0] direct_gb, pooled_gb, dram_cost_per_gb, fabric_cost,
  input  logic [15:0] used_gb,
  output logic [15:0] direct_cost, pooled_cost, direct_per_usable, pooled_per_usable,
  output logic        pooling_cheaper,
  output logic [7:0]  n_pricings, n_pool_wins,
  output logic        fabric_ignored_err
);
  logic [31:0] d_q, p_q, du_q, pu_q;
  assign d_q = {16'd0, direct_gb} * {16'd0, dram_cost_per_gb};
  assign direct_cost = (d_q > 32'd65535) ? 16'hFFFF : d_q[15:0];
  // The fabric is a cost direct attach does not carry, and a capital-only view
  // of the DIMM count leaves it out.
  assign p_q = ({16'd0, pooled_gb} * {16'd0, dram_cost_per_gb})
             + ((CAPEX_ONLY != 0) ? 32'd0 : {16'd0, fabric_cost});
  assign pooled_cost = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign du_q = (used_gb == 16'd0) ? 32'd65535
              : (({16'd0, direct_cost} * 32'd1000) / {16'd0, used_gb});
  assign direct_per_usable = (du_q > 32'd65535) ? 16'hFFFF : du_q[15:0];
  assign pu_q = (used_gb == 16'd0) ? 32'd65535
              : (({16'd0, pooled_cost} * 32'd1000) / {16'd0, used_gb});
  assign pooled_per_usable = (pu_q > 32'd65535) ? 16'hFFFF : pu_q[15:0];
  assign pooling_cheaper = (pooled_per_usable < direct_per_usable);
  // A pooled cost that does not carry the fabric it needs.
  assign fabric_ignored_err = price && (fabric_cost != 16'd0)
                              && (pooled_cost == ({16'd0, pooled_gb}
                                                  * {16'd0, dram_cost_per_gb}));

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_pricings <= 8'd0; n_pool_wins <= 8'd0;
    end else if (price) begin
      n_pricings <= n_pricings + 8'd1;
      if (pooling_cheaper) n_pool_wins <= n_pool_wins + 8'd1;
    end
  end
endmodule

Five pricings. 1024 GB direct-attached against a 768 GB pool, at 10 per GB, serving 768 GB of demand.

Two pooling wins when the fabric is costed; four when it is not.

The fabric is the whole question and a DIMM count cannot see it. Fewer gigabytes of DRAM is a real and easily computed saving; the switch, the cabling, the fabric manager and the power they draw are a cost direct attach simply does not have. Row three is a fabric expensive enough to lose, and it is not an extreme number.

Row five is the break-even and it is the figure to compute first. At a fabric cost of 2560 the two are exactly equal, which means every fabric cheaper than a quarter of the DRAM it replaces wins and every fabric dearer loses — a single ratio that decides the architecture before any latency argument is made.

The capital-only model is what a procurement process naturally produces, because the DRAM and the fabric are bought from different vendors on different budgets under different approval paths. Nothing about it is careless; it is a reporting boundary that happens to fall exactly where the decision is.

14. RTL 10 — Memory Disaggregation Assembled

// RTL 10 - memory disaggregation assembled. Everything that must hold before a
// pool improves a fleet rather than only moving where the DIMMs are.
module disaggregation_model #(parameter int UTILISATION_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       strand_recovered,   // utilisation actually improved
  input  logic       capacity_pooled,    // there is a pool, not per-socket DRAM
  input  logic       latency_budgeted,   // the fabric hop is in the access budget
  input  logic       blast_radius_bounded,// one device failure is survivable
  input  logic       bandwidth_honest,   // per-socket share, not the device total
  input  logic       reclaim_accounted,  // scrub time is in the capacity plan
  output logic       improves,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_improves,
  output logic       false_gain_err
);
  assign fail_mask[0] = ~strand_recovered;
  assign fail_mask[1] = ~capacity_pooled;
  assign fail_mask[2] = ~latency_budgeted;
  assign fail_mask[3] = ~blast_radius_bounded;
  assign fail_mask[4] = ~bandwidth_honest;
  assign fail_mask[5] = ~reclaim_accounted;
  // The utilisation-only build reports the number a capacity planner reports.
  assign improves = (UTILISATION_ONLY != 0) ? strand_recovered : (fail_mask == 6'd0);
  assign false_gain_err = evaluate && improves && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_improves <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (improves) n_improves <= n_improves + 8'd1;
    end
  end
endmodule

One improving configuration of six, and four false claims.

The utilisation-only definition is exactly what a capacity planner reports, and it is right about one of the six. A rising utilisation figure is the headline every disaggregation programme is measured on, and it is compatible with a fleet that is slower, more fragile, bandwidth-starved and shorter than it was.

Row two deserves naming because it is the most common form of the failure: memory relabelled as a pool without a fabric that lets a second socket reach it. Utilisation rises because the accounting changed, and nothing physical did.

Figure 4 — Four exits and only one builds a pool. Three of the four rejections are cheap to compute and are almost never computed — the strand is a telemetry question, the correlation is a statistics question, and the fabric ratio is arithmetic. The expensive question, whether the workload tolerates the hop, is the only one that needs an experiment.

15. Quantitative Reasoning

Stranding. Sixteen sockets at 40 GB using 30: 160 GB stranded of 640 — 25%. A well-packed fleet still strands 10%.

Pooling. Sixteen sockets peaking at 64 GB each need 1024 GB separately and 640 as a pool — a 37% saving, and exactly zero if the peaks coincide.

Blast radius. Eight devices for 64 sockets: one device failure takes eight sockets, four with two-way replication, one with eight-way — which gives the whole saving back.

Latency. 100 ns local, 250 ns at two hops — a 150% tax, and 400 ns at four hops.

Granularity. A 6 GB request in a 16 GB region wastes 62% of what it occupies; 17 GB in the same region wastes 15 GB.

Reclaim. 256 GB scrubbed at 80 Gbps is 25 s; twenty reclaims an hour makes the pool 13% unavailable, and forty makes it 27%.

Bandwidth. An 800 Gbps device across four active sockets gives 200 Gbps each against a demand of 400.

Contention. A 1024 GB pool with 768 GB held elsewhere offers 256 GB to a 512 GB request.

Cost. 13,333 per thousand usable gigabytes direct against 11,333 pooled — 15% cheaper, and break-even at a fabric costing a quarter of the DRAM it replaces.

The assembled model. Six properties, six configurations, one improves. The utilisation-only definition reported five.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Every inclusive threshold in this chapter is driven at exactly equal, constructed rather than swept — see section 18 for why that was done before the mutations asked.

Stranding. A fleet stranding exactly 20% is driven, and the empty fleet is asserted as not a hidden strand.

chk(sGs == 16'd128, "using 32 of 40 GB strands 128 GB");
chk(sGp == 16'd20,  "exactly twenty percent");
chk(sGe == 1'b1,    "which is exactly efficient");

Pooling. The coincident-peaks case is asserted as a truth rather than an error, and a saving of exactly 25% is driven.

Blast radius. Direct attach and full replication are both asserted as cases the direct-attached model gets right, and a two-socket device is driven as the smallest sharing that is still sharing.

chk(rGt == 16'd1, "eight-way replication takes exactly one socket");
chk(rGe == 1'b0,  "and both models now agree");

Latency. A tax of exactly 100% is driven, and the unmeasured-local case is asserted to report added nanoseconds without a ratio.

Granularity. An allocation wasting exactly 20% is constructed from a 64 GB request against 20 GB regions, and the unstated granularity is asserted as not a rounding loss.

Reclaim. Unavailability of exactly 10% is constructed from 180 GB at 80 Gbps, twenty times an hour.

Bandwidth. A share exactly equal to the demand is driven at two sockets.

Contention. Availability exactly equal to the request is driven, and the over-reported holding is asserted to floor at zero rather than wrap.

Cost. The fabric cost at which the two options are exactly equal is constructed, and asserted not cheaper.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 284 checks across two testbenches, 150 on the front five models and 134 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Sixty-seven mutations were injected one at a time.

67 injected, 67 killed, with no survivors on the first run.

That is the first zero-survivor chapter in this track, and it is a direct consequence of a rule recorded rather than a rule described. Batch 021 produced five survivors that were all the same omission: an inclusive threshold never driven at equality. Here, every <= and >= had its equality case constructed while the testbench was being written — a 20% strand from using 32 of 40 GB, a 25% saving from an aggregate peak of 768, a 100% latency tax from two 50 ns hops, a 20% allocation waste from 64 GB against 20 GB regions, a 10% reclaim overhead from 180 GB at 80 Gbps, and a break-even fabric cost of 2560.

None of those is a value a sweep would find. Each was solved for backwards from the threshold, which is a minute of arithmetic per model and the difference between sixty-seven kills and sixty-two.

Two models compare against a separately computed truth rather than against their own output — section 7's true_affected and section 9's true_allocated_gb. Both exist because 22.3 §11 established that a broken build wrong in one place is wrong consistently downstream, so a check written in the model's own terms passes by construction. Neither is a survivor here because neither was ever written the weak way.

18. Verification Strategy

What a testbench for a fleet-capacity model must cover.

Solve for every threshold's equality case before writing the stimulus. This is the batch-021 carry-forward and it produced the first clean mutation run in the track. Each equality is a small algebra problem: what request against what region size wastes exactly 20%? Answering it takes a minute and removes an entire class of survivor.

Compute the truth unconditionally when the broken build could corrupt the comparison. Two models here do it, and neither can be fooled by a build that is wrong twice in the same direction.

The cases where the direct-attached model is right. Every socket full. Peaks that coincide. One device per socket. Full replication. A single active socket. An empty pool. Six cases across five models where the broken build is correct — and each is a real configuration a fleet can be in, which is why the checks exempt them explicitly rather than by omission.

Drive inputs the telemetry pipeline produces and the system should not. Section 12's over-reported holding — sockets summing to more than the pool because they report asynchronously — is the case that makes the floor necessary, and no healthy fleet generates it.

Counters as a second signature. Ten models, ten pairs of totals, and the two builds differ in every one: two wasteful against none, three uncontained against none, two pooling wins against four. A single output can coincide by accident; ten pairs cannot.

What a real fleet needs that these models do not have. Time. Every model here is a snapshot, and the interesting behaviour of a pool is temporal — fragmentation accumulating over weeks, holdings correlating during an incident, reclaim queues building faster than they drain. Section 27 exercise 9 is the closest this chapter comes.

19. Synthesis and Implementation Reality

Nothing in sections 5, 6, 12 or 13 is hardware. They are arithmetic a fabric manager or a capacity planner performs, and their failure mode is a purchase order rather than a timing violation.

Section 7's blast radius is a placement policy. Which sockets' pages land on which device is decided by the fabric manager's allocator, and the difference between a radius of eight and a radius of two is a constraint in that allocator rather than a hardware property.

Section 8's hop is the one number fixed at design time. Switch latency is silicon, and hop count is cabling — neither changes after deployment. A pool two hops away cannot become a pool one hop away, which makes the topology decision the least reversible one in the chapter.

Section 9's granularity is a device and driver property together. The device exposes a region size and the host allocator rounds to it; changing either changes the waste, and a large granularity is chosen for exactly the reason it hurts — fewer, larger regions are cheaper to track.

Section 10's scrub is a device-side engine. DRAM cannot be handed between tenants without being cleared, and the rate at which it clears is a hardware capability that a pool's churn budget depends on entirely.

Section 11's contention is arbitration inside the device. How a multi-headed device divides its bandwidth among heads is a QoS mechanism, and equal division is only the default — 19.2 is where a different policy is priced.

20. Silicon Observability

"Time-to-available" is the counter this chapter needs and no direct-attached fleet ever had. Free capacity and grantable capacity are the same number when memory is soldered to a socket and different numbers in a pool, because section 10's scrub sits between them. A dashboard reporting free capacity in a pool is reporting a number that cannot be allocated, and the gap is invisible without a second counter.

21. Debug Lab

Symptom. A fleet moves 640 GB of per-socket DRAM to a 640 GB pool. The capacity dashboard shows utilisation rising from 75% to 96% — the programme's headline metric, met. Six weeks later, application p99 latency is up 40%, two incidents have taken eight servers each, and the team cannot get a 512 GB allocation granted at peak.

Step 1 — did the strand actually go? Fitted against resident, per socket: fitted is now 0 per socket and 640 in the pool, resident 614. The strand is genuinely recovered — section 5's number moved, and the headline is honest.

Step 2 — where is the p99 coming from? Access latency by target: 94% of accesses are one hop, 6% are two. The one-hop accesses are at 175 ns against the old 100. Section 8's tax is being paid on everything, and it is bigger than the p99 regression, which means the regression is smaller than the tax — the workload absorbed most of it.

Step 3 — the two-hop 6%. Those are at 250 ns and they are concentrated in one service. The allocator placed that service's pages on a device across the second switch tier, which nothing in the request expressed. Section 8's row one, and it is a placement bug rather than a topology one.

Step 4 — the incidents. Sockets holding pages per device: eight per device, unreplicated. Section 7 exactly — two device failures, eight servers each, and the blast radius was never computed because the direct-attached fleet's radius was one and nobody restated it.

Step 5 — the refused allocations. Pool free capacity at request time: 26 GB at peak, against 614 resident. But total resident is 614 of 640, so free is 26 — the pool is simply full. Section 12, and it is the predictable consequence of sizing the pool at the aggregate peak with no headroom for the peak being wrong.

Step 6 — and the scrub. Time-to-available against free capacity: free reads 26 GB, grantable reads 4 GB. Section 10's scrub queue holds the rest. The dashboard's free-capacity number was never allocatable.

The finding. Every symptom is a property this chapter models, none of them contradicts the headline metric, and the headline metric is true. Utilisation rose from 75% to 96% and the fleet got worse.

The fix, in the order the numbers say. Add replication for the services that cannot lose eight servers — which costs capacity, so the pool has to grow. Pin the two-hop service's pages to a first-tier device. Size the pool at the aggregate peak plus the scrub queue plus a genuine margin, which is section 6 plus section 10 rather than section 6 alone. The programme's saving falls from 37% to roughly 20%, and it is now real.

What made this hard. The metric the programme was measured on improved, on the first day, permanently, and correctly. Nothing in the reporting could have surfaced any of the six findings, because five of them need counters a direct-attached fleet had no reason to have.

22. Design Review

1. What is the measured strand, per socket, from telemetry rather than purchase records? If nobody has it, section 5 is unanswerable and the programme has no baseline.

2. Do the peaks coincide? If they do, pooling buys a fabric and saves nothing. Section 6.

3. How many sockets will hold pages on one device, and is that survivable? The consolidation ratio is the blast radius. Section 7.

4. How many hops, and what is the measured per-hop cost? 150% on every access at two hops. Section 8.

5. What is the region granularity, and what does the request distribution look like against it? A 6 GB request in a 16 GB region wastes 62%. Section 9.

6. What is the reclaim churn, and the scrub bandwidth? A pool can be 27% unavailable while reporting itself healthy. Section 10.

7. How many sockets share one device's bandwidth at peak? Capacity divides gracefully; bandwidth does not. Section 11.

8. Is the pool sized to the aggregate peak, or to the aggregate peak plus the scrub queue plus a margin? Section 21 is what the first answer produces.

9. What does the fabric cost, as a fraction of the DRAM it replaces? Break-even is a quarter. Section 13.

10. Which of the six properties does the team believe "utilisation went up" implies? Section 14 exists because the answer is none of the other five.

23. How This Appears In Real Engineering

A capacity-planning function owns sections 5, 6 and 13, and the hardest part is section 5: the strand cannot be computed from what was bought. It needs per-socket resident-memory telemetry sampled over a long enough window to see the peak, and a fleet without it is guessing at the size of the prize.

A fabric or platform team owns sections 7, 8 and 11 — the three costs consolidation introduces. All three scale with the consolidation ratio in the same direction, which means the ratio that maximises the capacity saving also maximises the blast radius, the contention and (through switch tiers) the latency. There is an optimum and it is rarely at either end.

A reliability function meets section 7 as a step change rather than a gradient. A direct-attached fleet's memory blast radius is one, by construction, and nobody writes it down because it is not a number anyone had to think about. Disaggregation makes it a design parameter overnight, and the first incident is where most organisations discover that.

An operations team lives in sections 10 and 12, and both are new failure modes rather than worse versions of old ones. A grant that is refused because another socket holds the memory has no direct-attached analogue at all — the closest equivalent is a server that will not boot, which is a different conversation with a different escalation path.

24. Common Misconceptions

"Stranded memory is idle memory." Idle memory can be used by whatever needs it. Stranded memory can be used by exactly one socket. Section 5.

"Pooling always saves capacity." Only when the peaks do not coincide. Section 6.

"Fewer, bigger memory devices is simpler." It is also a blast radius of eight instead of one. Section 7.

"The latency hit is a one-time cost." The strand is recovered once; the hop is paid on every access forever. Section 8.

"We allocated 6 GB." You occupied 16. Section 9.

"Freed memory is available memory." Not until it is scrubbed — a pool can be 13% unavailable while reporting itself free. Section 10.

"The device does 800 Gbps." Across four active sockets it does 200 each. Section 11.

"The pool has 1024 GB." You can have 256 of it, right now, because of what everyone else is holding. Section 12.

"Pooling is cheaper — fewer DIMMs." Not once the fabric is on the same line. Break-even is a quarter. Section 13.

"Utilisation went up, so it worked." One property of six, and section 21 is the other five. Section 14.

25. Interview Reasoning

Q. Why is a quarter of a data centre's DRAM typically unusable?

Because it is bought with the socket. Memory soldered to a server whose workload needs 30 of its 40 GB leaves 10 GB that no other server can reach — not idle, structurally unreachable. The follow-up worth knowing: even a carefully packed fleet strands about 10%, because every socket needs its own growth headroom.

Q. When does pooling save nothing?

When the peaks coincide. Sixteen sockets that each peak at 64 GB need 640 pooled if the peaks are spread and 1024 if they are simultaneous — and a synchronised batch fleet is exactly the second case. The correlation between sockets is the entire economic argument, and it is a measurement rather than an assumption.

Q. What does consolidation cost that a capacity model does not show?

Three things, all scaling the same way. Blast radius — eight devices for sixty-four sockets means a device failure takes eight servers. Bandwidth contention — the device's rate divides by the sockets actively using it. And the fabric hop, at 150% of the local access latency for two hops. All three get worse as the consolidation ratio gets better.

Q. Your fleet's memory utilisation went from 75% to 96% and everything got worse. What happened?

Utilisation measures one of the six things that has to be true. The other five — the fabric hop in the latency budget, a survivable blast radius, honest per-socket bandwidth, reclaim time in the capacity plan, and capacity genuinely pooled rather than relabelled — are all compatible with a rising utilisation figure. The metric is true and insufficient, which is the hardest kind to argue with.

Q. Why can a pool report free capacity you cannot allocate?

Because memory returned by one tenant must be scrubbed before another gets it, and scrubbing is bandwidth-bounded on a large quantity of DRAM. Free and grantable are the same number in a direct-attached fleet and different numbers in a pool — a pool with high churn can be a quarter unavailable while every dashboard reads healthy.

Q. When does disaggregation not pay?

When the fabric costs more than a quarter of the DRAM it replaces — that is the break-even in the model, and it is arithmetic anyone can do before committing. The latency question needs an experiment; the cost question does not, and it rules out more deployments than people expect.

26. Exercises

1. Extend RTL 1 to a distribution of per-socket usage rather than a single figure, and show that the mean understates the strand.

2. Make RTL 2's aggregate peak a function of a correlation coefficient, and find the correlation at which pooling stops winning.

3. Combine RTL 3 and RTL 2: for a fixed capacity budget, find the consolidation ratio that maximises the saving subject to a blast-radius limit.

4. Give RTL 4 a hit rate — some fraction of accesses served locally — and find the rate at which the average access returns to 100 ns.

5. Drive RTL 5 with a realistic request-size distribution and find the granularity that minimises total waste.

6. Add a scrub queue to RTL 6 and show that a churn rate above the scrub rate makes the pool shrink without bound.

7. Replace RTL 7's equal division with a weighted one and show what a QoS policy costs the unprioritised sockets.

8. Make RTL 8's others_using_gb a time series and count how often a fixed request would have been refused over a day.

9. Model section 21 end to end: strand recovered, hop tax paid, blast radius realised, and the pool full at peak.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the fleet it catches that the current mask calls improving.

27. Summary

Module 22 found the same thing at five scales: capacity is not throughput. Module 23 opens with a stranger finding — a quarter of a fleet's capacity is not even capacity, because it is attached to the wrong socket and no amount of demand elsewhere can reach it.

Memory bought with a socket strands with it. Sixteen sockets fitted with 40 GB using 30 strand 160 GB of 640 — and a carefully packed fleet still strands 10%, because every socket needs its own headroom.

Pooling wins on one statistical fact. Sixteen sockets peaking at 64 GB need 1024 GB separately and 640 as a pool — a 37% saving, and exactly nothing if the peaks coincide.

The failure domain grows with the sharing. Eight devices for sixty-four sockets means one failure takes eight sockets — and eight-way replication contains it perfectly by giving the entire saving back.

The strand is recovered once and the hop is paid forever. 100 ns local becomes 250 at two hops — a 150% tax on every access, the asymmetry that decides whether the trade is worth making.

A pool strands memory too, by a different mechanism. A 6 GB request in a 16 GB region occupies 16 and wastes 62% of it, and a request one gigabyte past a boundary costs a whole further region.

Freed is not available. 256 GB scrubs in 25 seconds, and twenty reclaims an hour makes the pool 13% unavailable while every dashboard reads free.

Capacity divides gracefully and bandwidth does not. An 800 Gbps device across four active sockets delivers 200 each against a demand of 400 — and the divisor is a runtime property nobody controls.

And the pool you can see is not the pool you can have. 1024 GB with 768 held elsewhere offers 256 GB to a 512 GB request, at that instant, with nothing broken.

The fabric is the whole cost question. 11,333 per thousand usable gigabytes pooled against 13,333 direct — and break-even at a fabric costing a quarter of the DRAM it replaces, which is arithmetic available before anything is bought.

Sixty-seven mutations, sixty-seven killed, no survivors on the first run — because every inclusive threshold's equality case was solved for while the testbench was written rather than after a mutation asked. That is batch 021's carry-forward doing exactly what a recorded rule is supposed to do.

Recovering the strand is one property of six. The definition a capacity planner reports called five of six fleets improved when one was — and section 21 is a fleet whose headline metric rose from 75% to 96% while everything that mattered got worse.

23.2 — Composable Infrastructure takes the pool from here and asks the next question: if memory can be attached to any socket, what else can, and what does a server become when its parts are assembled at boot rather than at purchase?