Datacenter Architecture

12.1 through 12.3 treated the pool as one flat set of units in one place.

Every invariant in those chapters still holds here. What changes is that the pool now has a geography, and geography decides what a failure costs.

1. The Engineering Problem — The Pool Is Not In One Place

At chassis scale, where an allocation sits is an efficiency question. At rack scale it stops being one.

A device failure has a radius, and placement sets it. Four allocations on one device and four spread across four devices consume identical capacity, and one device failure takes four of them or one. Nothing in the capacity accounting distinguishes the two arrangements.

Capacity and reachability stop being the same statement. A host that cannot reach an enclosure does not have a small amount of capacity there; it has none. Two hosts can look at the same pool, both correctly, and report different amounts of free memory.

Distance costs. Spreading allocations to limit the blast radius moves them away from the hosts using them, and the access cost rises. The safest placement and the fastest placement are not the same placement.

And a pool that can survive a failure is a pool forbidden to use all its capacity. Headroom held back to absorb a device loss is capacity that exists, is healthy, and may not be granted. That is a deliberate cost, and a pool that reports it as available is promising something it has already decided not to deliver.

2. The One-Sentence Model

Scale changes the blast radius, not the invariants. Conservation, unique ownership, generation safety and isolation are exactly as they were — what rack scale adds is that every allocation now has a location, and location decides who is affected when something breaks.

Call it the same rules, with a map. A pool without recorded placement can still be correct and cannot answer the only question anyone asks after a failure: who is affected.

3. What This Chapter Owns

Deferred:

And nothing here is a fabric. Reachability is modelled as a matrix that says which hosts can reach which enclosures. How that connectivity is built, discovered, managed or made redundant is Modules 15 and 16, and this chapter deliberately treats it as a given.

4. Teaching-model boundary

The hierarchy below — 16 units on 4 devices in 2 enclosures — is a teaching scale, chosen so every number in the chapter is checkable by hand. Real pools are larger by orders of magnitude and the arithmetic does not change.

The reachability matrix is likewise a teaching structure. It expresses the fact that connectivity is not uniform without claiming anything about how CXL establishes it. Nothing below should be read as a CXL topology or a fabric-manager mechanism.

5. RTL 1 — Placement Is State

//   16 units -> 4 devices (4 units each) -> 2 enclosures (2 devices each)
module placement_map #(parameter int UNITS = 16) (
  input  logic [3:0] q_unit,
  output logic       q_valid, output logic [1:0] q_owner, q_dev, output logic q_enc,
  output logic [2:0] d_used0, d_used1, d_used2, d_used3,
  output logic [3:0] e_used0, e_used1,
  output logic [2:0] dev_spread,          // how many devices this pool spans
  output logic       place_err
);
  // Placement is structural here: the unit index names its device and its
  // enclosure. In a real pool this is a stored mapping; what matters is that
  // it EXISTS, because without it a failure has no bounded answer.
  assign q_dev = q_unit[3:2];
  assign q_enc = q_unit[3];

dev_spread counts how many devices the pool actually occupies, and it is the cheapest exposure metric there is. Measured on four allocations placed two ways:

Same capacity, same hosts, same everything the accounting can see.

Placing a unit that is already placed is refused and reported — measured place_err=1 with the original owner unchanged, the same rule as 12.1's ownership table applied to physical units.

6. RTL 2 — Reachable Is Not The Same As Free

  // The capacity a host can see is not the pool total: it is the sum over the
  // enclosures it can reach. Two hosts looking at the same pool can correctly
  // report different amounts of free capacity.
  logic [4:0] vis;
  always_comb begin
    vis = 5'd0;
    if (m_q[req_host][0]) vis = vis + {1'b0, enc_free0};
    if (m_q[req_host][1]) vis = vis + {1'b0, enc_free1};
  end
 
  // Reachability is tested FIRST. A host that cannot reach an enclosure is not
  // short of capacity there -- it has none, and reporting a capacity failure
  // sends an operator to add memory that host still could not use.
  assign reach_short = req_en && !reachable;
  assign cap_short   = req_en && reachable &&
                       (req_size > (req_enc ? enc_free1 : enc_free0));

Measured on one pool with 16 free units:

Both numbers are correct. A pool that publishes a single free-capacity figure is answering a question nobody asked, because the number a host needs is the one it can actually consume.

The two refusals are separated for the same reason the two in 12.1 were. A host refused for reach has ample capacity in front of it that it cannot use; adding memory to that enclosure changes nothing. Measured: fail_reach=1 with fail_cap=0 for a host asking into an enclosure it cannot reach, and the reverse for a reachable enclosure that is short.

The second width defect lived here. visible_cap was four bits summing two four-bit enclosure totals, so 8 plus 8 read as 0 — a host that could reach everything appeared to be able to reach nothing.

7. Waveform — One Device Failure, And Everything It Restates

Transcribed from the printed cycle trace.

Three things happen that are worth separating.

Cycles 2 and 3: draining, not failed. The device still serves what it holds and accepts nothing new. f_drain rises and f_fail stays low, and usable capacity is unchanged because nothing has been lost yet. A design with only a failed state has no way to express a planned evacuation.

Cycle 4: the failure restates the pool. Usable falls from 32 to 24 and utilisation rises from 62% to 83% — with the same 20 units allocated. Nothing was granted; the denominator shrank.

Cycle 6: headroom removes what is left. Holding 4 units back for the next failure takes grantable from 4 to 0 while usable stays at 24. The capacity is healthy, present, and not available.

8. RTL 3 — What A Failure Actually Costs

  always_comb begin
    ha = 4'd0; hh = 4'd0;
    for (k = 5'd0; k < UNITS[4:0]; k = k + 5'd1)
      if (v[k] && (k >= {1'b0, fail_dev, 2'b00}) && (k < ({1'b0, fail_dev, 2'b00} + 5'd4))) begin
        ha = ha + 4'd1;
        hh[o[k]] = 1'b1;
      end
  end

Two numbers, and they are not related by anything except placement.

Spreading reduced allocations lost by a factor of four and did nothing for hosts affected, because in that case there was only ever one host. The third row is the one that matters at rack scale: a single device holding one allocation each for four different hosts puts every one of them in the blast radius of one failure.

An empty device has no blast radius at all, measured at zero on both counters — which sounds obvious and is the case a for loop with a wrong bound gets wrong.

The total-lost counter is accumulated combinationally and added once, for the reason 12.3 established the hard way: a nonblocking increment inside a loop keeps only the last iteration and under-reports in proportion to the severity of the failure.

9. RTL 4 — Capacity After A Loss

  assign unavailable = ({5'b0, failed_map[0]} + {5'b0, failed_map[1]}
                      + {5'b0, failed_map[2]} + {5'b0, failed_map[3]}) * DEV_CAP[5:0];
  assign usable = physical - unavailable;
 
  // Utilisation is measured against USABLE capacity, not physical. After a
  // failure the denominator shrinks, and reporting against physical flatters
  // the system precisely when it is least healthy.
  //
  // The intermediate is 16 bits on purpose. Verilog sizes an expression from
  // its operands, so `allocated * 7'd100` is evaluated in seven bits and 2000
  // wraps to 80 -- a percentage that is wrong by a factor of thirty and looks
  // entirely plausible.
  logic [15:0] util_num, util_q;
  always_comb begin
    util_num = {10'b0, allocated} * 16'd100;
    util_q   = (usable != 6'd0) ? (util_num / {10'b0, usable}) : 16'd0;
  end

Measured, with 20 units allocated throughout:

Against physical capacity it would still read 62% after the failure — the flattering number, reported at the moment the pool is least able to absorb anything.

grantable subtracts both what is already allocated and the headroom held back on purpose: measured 4 units grantable with no headroom, and 0 once 4 are reserved, with usable unchanged at 24 in both cases.

And overcommit_err fires when allocated capacity exceeds what survives — measured at 20 allocated on 16 usable after a second device is lost. That is not a policy problem; it means allocations exist on capacity that is gone, and something has to be migrated or killed.

10. RTL 5 — Admission While Degraded

A degraded pool is not a smaller healthy pool. Parts of it are unusable and parts are being emptied on purpose.

  // Order matters and encodes an operational meaning. A failed device cannot
  // serve anything; a draining device still serves what it holds but takes
  // nothing new; only then is capacity the question.
  assign fail_failed   = req_en &&  dev_failed;
  assign fail_draining = req_en && !dev_failed &&  dev_draining;
  assign fail_cap      = req_en && !dev_failed && !dev_draining && (req_size > dev_free);
  assign grant         = req_en && !dev_failed && !dev_draining && (req_size <= dev_free);

Four outcomes, four counters, measured one each over the run:

The draining state is what makes planned maintenance possible. A device being evacuated must keep serving its existing allocations while accepting no new ones; collapsing it into either failed or healthy removes the only state in which an operator can empty a device without an outage.

An exact fit is granted, not refused — measured, a request for exactly the free capacity succeeds. That case had to be driven deliberately: without it the boundary can be off by one in either direction and every other test still passes.

11. RTL 6 — Packing And Spreading, Chosen Explicitly

    // PACK: the lowest-indexed device that fits. Fills devices in order and
    // leaves whole devices empty, which is what you want if you intend to
    // power one down or hold it as failure headroom.
    // SPREAD: the least-loaded device that fits. Limits how much any single
    // failure can take, and leaves no device empty.

On free capacity of 4, 6, 8 and 2 units with a two-unit request:

Different devices, same pool state, same request. Neither is wrong.

Packing is right when you intend to act on empty devices — power one down, hold it as failure headroom, or reserve it for a large contiguous request. Spreading is right when the cost of a single failure dominates. The measurement in section 8 is the argument for spreading; the measurement in section 13 is the argument against.

Both policies skip failed devices, and neither selects anything for a zero-size request. That last case is not pedantry: a policy that returns a device for a zero-size request will hand out a placement for an allocation that does not exist.

12. RTL 7 — What Spreading Costs

module locality_cost #(parameter int LOCAL = 4, parameter int REMOTE = 10) (
  input  logic       acc_en, input logic host_enc, target_enc,
  output logic [4:0] cost,
  output logic [15:0] n_local, n_remote, total_cost, avg_cost_x10
);
  assign is_remote = (host_enc != target_enc);
  assign cost = is_remote ? REMOTE[4:0] : LOCAL[4:0];

With illustrative costs of 4 for a local access and 10 for a remote one — these are teaching values and describe no measured system:

This is the price of the spreading policy in section 11, and it is why the argument cannot be settled by blast radius alone. A placement that limits a device failure to one allocation may put most accesses in the far enclosure, and the workload pays for that on every access rather than once per failure.

The average is scaled by ten before the division. A mean reported as a truncated integer would show 4, 7 and 8 for these three cases — hiding exactly the movement the metric exists to reveal.

13. RTL 8 and 9 — The Capacity You Are Not Allowed To Use

  // survive_n is three bits; part-selecting [5:0] from it reads past the end
  // of the vector and yields x. Zero-extend instead.
  assign required = {3'b0, survive_n} * DEV_CAP[5:0];
 
  // The ceiling utilisation a pool may run at and still absorb the stated
  // number of failures. Running above it is a decision, not an accident, and
  // it should be visible as one.

Measured on a 32-unit pool of four devices:

A four-device pool that must survive one device loss can never exceed 75% utilisation. That is the entire economic cost of resilience, in one number, and it is a design decision rather than an accident.

It compounds after a failure. With one device already lost, still holding 8 units back for the next one, usable is 24 and the ceiling is 66% — a tighter constraint on a smaller pool.

Running above the ceiling is detected: 28 units allocated of 32 while claiming to survive one loss raises cannot_absorb_err, because the pool is asserting a resilience property it can no longer deliver. The boundary is exact and needed its own test — 24 allocated leaves precisely the 8-unit reserve, spare is zero, and the loss can still be absorbed.

The final model reconciles two independent views of where capacity sits — counted from the placement map and from the accounting — per enclosure and in total. Measured: a disagreement in either enclosure is detected and named, and a total that does not match the sum of the enclosures is detected separately.

14. Quantitative Reasoning

Five numbers describe a rack-scale pool, and four of them do not exist at chassis scale.

Blast radius, in two dimensions.

allocations_lost = allocations placed on the failed device
hosts_affected   = distinct owners among them

Measured: 4 and 1 for a packed single-host arrangement, 1 and 1 for a spread one, and 4 and 4 for four hosts sharing one device. The two numbers move independently, and only the second predicts how many people notice.

Utilisation, against the right denominator.

U = allocated / usable          where usable = physical - unavailable

Measured 62% healthy and 83% after one device loss, with the allocations unchanged. Against physical capacity it reads 62% in both cases — a number that is technically true and useless, because it is highest-confidence exactly when the pool is least able to absorb anything.

Utilisation ceiling under a resilience promise.

max_U = (usable - survive_n x device_capacity) / usable

Measured 100%, 75% and 50% for absorbing zero, one and two device losses on a four-device pool, and 66% for absorbing one more after one has already gone. This is the price of resilience stated as a percentage of capacity that may never be granted.

Visible capacity, per host.

visible(h) = sum of free capacity over enclosures h can reach

Measured 16 for one host and 8 for another on the same pool at the same instant. Both correct. A single pool-wide free figure answers neither host's question.

Average access cost, as a function of placement.

avg = (n_local x local_cost + n_remote x remote_cost) / (n_local + n_remote)

Measured 4.0, 7.0 and 8.5 for all-local, half-remote and three-quarters-remote mixes at illustrative costs of 4 and 10.

The two policies pull in opposite directions and the numbers say so. Spreading takes the blast radius from 4 allocations to 1; it also moves accesses into the far enclosure and takes the average cost from 4.0 toward 8.5. There is no placement that optimises both, and the honest framing is that a rack-scale pool chooses which failure it prefers: a rare expensive one, or a continuous small one paid on every access.

15. Assertions

Written as SystemVerilog for the reader, executed procedurally — see section 17.

Usable capacity never exceeds physical.

property p_usable_bounded;
  @(posedge clk) disable iff (!rst_n)  usable <= physical;
endproperty

Per-device totals sum to the pool occupancy.

property p_placement_conserved;
  @(posedge clk) disable iff (!rst_n)
    (d_used0 + d_used1 + d_used2 + d_used3) == (e_used0 + e_used1);
endproperty

The blast radius is confined to the failed device.

property p_blast_confined;
  @(posedge clk) disable iff (!rst_n)
    fail_en |-> (hit_allocs <= UNITS_PER_DEV);
endproperty

A host never sees capacity it cannot reach.

property p_visible_reachable;
  @(posedge clk) disable iff (!rst_n)
    !m_q[req_host][1] |-> (visible_cap <= enc_free0);
endproperty

A reach failure implies the capacity was there.

property p_reach_not_capacity;
  @(posedge clk) disable iff (!rst_n)  fail_reach |-> !fail_cap;
endproperty

Exactly one admission outcome per request.

property p_one_outcome;
  @(posedge clk) disable iff (!rst_n)
    req_en |-> ($countones({grant, fail_failed, fail_draining, fail_cap}) == 1);
endproperty

A failed device is never selected by any policy.

property p_never_place_on_failed;
  @(posedge clk) disable iff (!rst_n)  found |-> !failed_map[chosen];
endproperty

Grantable capacity excludes the reserve.

property p_headroom_withheld;
  @(posedge clk) disable iff (!rst_n)
    grantable <= (usable - allocated - headroom);
endproperty

Allocations never outlive the capacity they sit on.

property p_no_overcommit;
  @(posedge clk) disable iff (!rst_n)  allocated <= usable;
endproperty

105 assertion sites — 98 across the nine models and 7 in the waveform trace. All pass.

16. Mutation Testing

55 mutations injected, 55 killed, 0 surviving.

Representative kills:

Nine mutations did not die on the first run.

The first two are the instructive pair. A test in which every device holds the same amount cannot detect a device counted from the wrong place, because the wrong answer and the right answer are the same number. Making the per-device occupancy uneven — 0, 2, 3 and 1 — killed both immediately.

The seventh was genuinely equivalent and was replaced rather than papered over. Removing the load comparison from one branch changed nothing, because the next comparison corrected it; the useful mutation removes the comparison from all four and makes the policy pick the last eligible device instead of the emptiest.

The taxonomy holds for an eighth consecutive batch.

17. Verification Strategy

Tool reality. Icarus Verilog 13.0 — no concurrent SVA, so every property has an executable procedural counterpart, and every run is bounded by a hard timeout.

Read the warnings. This chapter is the strongest argument in the batch for that. Four defects in these models were width or select errors, and two of them were announced by the compiler before any test ran:

Verilog sizes an expression from its operands, not from its destination. Three of these four follow directly from that rule, and none of them produces an error at runtime — they produce plausible numbers. The hardened checker introduced in 12.2, which treats x as a failure rather than a silent pass, is what caught the fourth.

Independent oracles.

Coverage. The points that matter are placement shape, failure count, reach configuration, and the position of allocated capacity relative to the ceiling. The crosses worth driving are placement shape crossed with failure — the only way to reach the packed-versus-spread divergence — and reach configuration crossed with request target, which is what separates the two refusal counters. A cross of unit index against host is noise.

18. Synthesis and Implementation Reality

Placement is a table, and at rack scale it is a large one. The structural mapping here — the unit index names its device — is a teaching simplification. A real pool stores it:

placement bits = UNITS x (log2(DEVICES) + log2(ENCLOSURES))

That grows linearly with capacity and is read on every allocation decision and every failure analysis.

The per-device tally is the timing problem, and it is the same shape as the derived counters in 12.1: a wide combinational reduction over the whole map, recomputed continuously. The resolutions are the same — pipeline it and accept a reporting latency, or maintain incremented counters and reconcile them periodically, which is what the reconciliation model exists for.

The reachability matrix is small and its cost is in where it sits. Hosts times enclosures is a handful of bits, but the check is on the allocation path and must complete before a placement decision, so it is a lookup rather than a scan.

The blast-radius scan is not on any critical path and can be as slow as it likes. It runs on failure, which is rare, and its answer is consumed by software. This is worth stating because it is the one structure here that should not be optimised — correctness and completeness matter and latency does not.

Headroom arithmetic is trivial — a multiply by a small constant and two subtractions — and the only thing to get right is the width, which section 17 covers at length.

19. Silicon Observability

The most valuable pair is hit_allocs against n_hit_hosts. They are recorded per failure and they are the only record of what placement policy actually bought. A pool that spread its allocations and still lost four hosts to one device learns something no capacity metric can tell it.

20. Debug Lab

  healthy      : physical=32 usable=32 utilisation=62%
  one device lost : unavailable=8 usable=24 utilisation=83%

assign usable = physical - unavailable;
util_q = (usable != 6'd0) ? (util_num / {10'b0, usable}) : 16'd0;

  4 hosts, 1 device                     | device 0 lost -> 4 allocations, 4 hosts

if (v[k] && (k >= {1'b0, fail_dev, 2'b00}) && (k < ({1'b0, fail_dev, 2'b00} + 5'd4))) begin
  ha = ha + 4'd1;
  hh[o[k]] = 1'b1;     // owners, not just allocations
end

  visible capacity: host 0 = 16 units, host 1 = 8 units (same pool)

assign reach_short = req_en && !reachable;
assign cap_short   = req_en && reachable && (req_size > (req_enc ? enc_free1 : enc_free0));

  uneven placement: devices hold 2, 3, 1 -> enclosures 2 and 4

logic [4:0] j;                                  // wide enough for the offsets
u2 = u2 + {2'b0, v_q[5'd8  + j]};

  degraded admission : failed=1 draining=1 capacity=1

assign fail_failed   = req_en &&  dev_failed;
assign fail_draining = req_en && !dev_failed &&  dev_draining;

  grantable    : 4 with no headroom, 0 holding 4 back

always_comb g = $signed({2'b0, usable}) - $signed({2'b0, allocated})
                                        - $signed({2'b0, headroom});

  survive 1 loss : required=8 spare=0 max utilisation=75%
  28 of 32 allocated while claiming to survive a loss : err=1

assign headroom_ok = (sp >= 0);
if (!headroom_ok) cannot_absorb_err <= 1'b1;

  reconciliation : enclosure drift=1 (enc 1)  total drift=1

assign d0 = (place_e0 != acct_e0);
assign d1 = (place_e1 != acct_e1);
assign dt = (({2'b0, place_e0} + {2'b0, place_e1}) != total_alloc);

21. Design Review

What a reviewer should attack first.

Whether placement is recorded at all. If the allocator does not store which device holds each unit, the blast radius of any failure is the whole pool by definition, and no analysis after the fact can narrow it.

The utilisation denominator. Ask what it is measured against. If the answer is physical capacity, the number is highest exactly when the pool is least healthy and the failure is invisible in the metric.

The draining state. Ask how a device is taken out for maintenance. If the only states are healthy and failed, every planned evacuation is either an outage or a no-op.

Grantable versus usable. Ask which one is published to schedulers. Publishing usable promises capacity the pool has already decided to withhold.

The resilience ceiling, enforced rather than documented. Ask what stops utilisation crossing it. If the answer is a runbook, it will be crossed.

Every width in the file. Four defects in these models were width or select errors, two of them announced by the compiler. This is the specific class to grep for.

What is deliberately not here. No fabric, no topology, no discovery, no manager — Modules 15 and 16. No latency modelling: the local and remote costs are illustrative weights that make a tradeoff visible, not a performance model, which is Module 18. No TCO, no fleet economics, no composability — 12.5 for the economics of pooling and Module 23 for the datacentre architecture built on it.

22. How This Appears in Real Engineering

In architecture review, the recurring argument is packing against spreading, and it is usually conducted without numbers. It has two: the blast radius and the average access cost, and they move in opposite directions. A review that produces a policy without both is choosing by preference.

In RTL design, this chapter's defect class is width. Verilog sizes an expression from its operands rather than its destination, so a percentage can be wrong by a factor of thirty and look plausible. Two of the four defects here were announced by the compiler and the warnings were initially skimmed.

In verification, the trap is symmetric stimulus. Every device holding the same amount cannot detect a device counted from the wrong place, because the wrong answer equals the right one. Uneven occupancy is not a nicety; it is what makes per-partition counters testable at all.

In bring-up, the surprise is that utilisation moves without any allocation activity. It is correct behaviour and it looks alarming, and an operator who has not seen the denominator explained will chase an allocation storm that never happened.

In operations, the question is why can this host not allocate when the pool is half empty, and at rack scale the answer is often reachability rather than capacity. That is a connectivity change, not a purchase, and only per-host visible capacity distinguishes them.

23. Common Misconceptions

"Rack scale needs new invariants." It needs the same ones. Conservation, unique ownership, generation safety and isolation are unchanged from 12.1 through 12.3; what is added is a location per allocation.

"Spreading reduces the blast radius." It reduces how much any one host loses — measured, 4 allocations to 1. It does nothing about how many hosts a device touches, and a device carrying one allocation each for four hosts affects all four.

"Free capacity is free capacity." Not per host. The same pool showed 16 units to one host and 8 to another, both correct, because reachability is not uniform.

"Utilisation is utilisation." Against physical capacity it reads 62% before and after a device loss. Against usable capacity it reads 62% then 83%. The second is the one that predicts a refusal.

"A failed device and a draining device are the same thing." One cannot serve anything; the other serves what it holds and takes nothing new. Without the second, planned maintenance is an outage.

"Headroom is wasted capacity." It is the price of the resilience the pool claims. A four-device pool surviving one loss is capped at 75% utilisation, and capacity above that line was never really available.

"The compiler warnings are noise." Two of the four defects in this chapter's models were printed by the compiler before any test ran, and both produced plausible numbers at runtime.

"Equal test coverage per device is thorough." Equal occupancy per device is the one arrangement in which a mis-indexed per-device counter cannot be detected.

24. Interview Reasoning

25. Exercises

1. Calculation. A pool has 8 devices of 64 units each. Compute the utilisation ceiling for surviving one, two and three device losses. Then repeat for 16 devices of 32 units at the same total capacity, and state which arrangement pays less for the same resilience and why.

2. Analysis. A device fails and the blast-radius report shows 6 allocations and 6 hosts. Explain what placement produced that, what the two numbers would have been under a spreading policy, and why spreading might not have improved the second one.

3. RTL task. Extend placement_map so placement is a stored mapping rather than derived from the unit index. State the storage cost, what must now be validated on every allocation, and the new failure mode the stored version introduces that the derived version cannot have.

4. Assertion task. Write the property proving the blast radius is confined to the failed device. Then explain why checking only the allocation count is insufficient, and construct the failure on which a count-only check passes while the host set is wrong.

5. Design task. Add a third enclosure with partial reachability — reachable by two of four hosts. State how visible capacity changes per host, what the admission path must now check, and how the placement policy should choose between an enclosure that is closer and one that is more widely reachable.

6. Testbench design. Design the stimulus that verifies per-device occupancy counters. Explain why equal occupancy on every device cannot detect a mis-indexed counter, and state the minimum arrangement that can.

7. Debug task. A pool reports 62% utilisation before and after a device failure. Give the defect, explain why it produces a plausible number rather than an obvious error, and state the two measurements that expose it.

8. Design review. A colleague proposes removing the failure headroom, arguing that migration can relocate allocations after a failure and reserving capacity in advance is wasteful. Give the strongest version of that argument, name the state it assumes exists, and state the measurement that decides whether the argument holds for a given pool.

26. Summary

Scale changes the blast radius, not the invariants.

• Placement decides the cost of a failure. Four allocations packed lose 4 allocations and 1 host; spread they lose 1 and 1; four hosts sharing one device lose 4 and 4. Capacity accounting cannot distinguish any of them.
• The invariants are unchanged. Conservation, unique ownership, generation safety and isolation carry over intact from 12.1 to 12.3. What is added is a location.
• Reachable is not free. The same pool showed 16 units to one host and 8 to another, both correct, and a reach failure is fixed by connectivity rather than by memory.
• Utilisation must use the usable denominator. The same 20 allocated units read 62% healthy and 83% after one device loss; against physical capacity both read 62%.
• Draining is its own state. A device being evacuated serves what it holds and takes nothing new — without it, planned maintenance is an outage.
• Grantable is not usable. Holding 4 units back took grantable from 4 to 0 while usable stayed at 24.
• Resilience is a utilisation ceiling: 100%, 75% and 50% for absorbing zero, one and two device losses, and 66% for absorbing one more after a loss. Running above it raises cannot_absorb_err.
• Spreading is paid for on every access. Average cost moved 4.0 to 7.0 to 8.5 as accesses shifted from local to three-quarters remote, which is the argument against the policy that section 8 argues for.
• Verification: 105 assertion sites, 55 of 55 mutations killed, zero surviving. Nine first-run escapes were six stimulus gaps, two unobserved outputs and one equivalent mutation that was replaced rather than recorded. The baseline found four width defects — two of which the compiler had already printed.

Next: 12.5 Memory-Pooling Benefits and Challenges, which stops building mechanisms and asks whether the whole arrangement pays — stranded capacity recovered against latency added, utilisation gained against shared fate, and what evidence would settle it either way.