Cloud Data Centres

29.1 asked which tier and 29.2 asked what the device actually is. This chapter asks the question the other two were building towards — at fleet scale, does any of it pay? — and it is the case where the weak definition is not a technical claim at all but an appeal to somebody else's judgement.

The hyperscalers are deploying it. Probably true of somebody. It contains no quantity: not which operator, not what fraction of what fleet, not for which service, not what it recovered, and not what it cost to run.

1. The Engineering Problem — Deploying Is A Verb Without A Number

The claim has no quantity in it. Five quantities needed with one given is four missing, on a claim where only a quarter is attributed to anybody who could be asked. Section 5.

A pilot is not a fleet. Forty racks of nine hundred is four percent, and calling it a deployment costs nothing until somebody plans against it. Section 6.

Stranded memory is the actual argument. Five hundred and twelve gigabytes per host with three hundred allocated strands two hundred and twelve on every host — forty-one percent of the memory, bought and unrentable. Section 7.

A pool is an over-subscription bet. Twenty-four tenants peaking at a hundred and twenty against a two-thousand-and-forty-eight gigabyte pool is a hundred and forty percent subscribed and eight hundred and thirty-two short if the peaks coincide. Section 8.

A fleet is run on its tail. A mean that improved from two hundred to a hundred and eighty while the ninety-ninth percentile went from nine hundred to fourteen hundred is a five-hundred-unit regression that the mean does not show. Section 9.

And a pool serving many hosts is a domain containing all of them. Sixty-four hosts of two hundred is a sixty-four-host blast radius, not a device failure. Section 10.

Why this chapter closes the module's first three. A training cluster is one operator's decision and a card is one device's properties. A fleet is where both get multiplied by a number large enough to change which mistakes matter — and where the arguments stop being about memory and start being about utilisation, availability and who carries the pager.

2. The One-Sentence Model

A cloud-fleet case study is sound when an operator is named, when the fraction of the fleet is stated, when the stranding it recovers is measured, when the tail latency is stated rather than the mean, when the blast radius is stated, and when the operational cost is counted — and "the hyperscalers are deploying it" is bit 0.

3. What This Chapter Owns

Some vocabulary, because fleet arguments use words that sound technical and are economic.

Stranded memory is memory a host has and cannot rent. A machine whose cores are all allocated while half its memory is free has stranded that memory: it is present, powered, paid for, and unsellable until a core frees up. It is the single most-cited motivation for memory pooling, and section 7 is it as a number.

A pool is shared capacity behind an attach, drawn on by several hosts. Its value comes from the same statistical argument as any shared resource — that not everybody peaks at once — and section 8 is that argument as a bet with a size.

The blast radius is how many hosts stop when one thing fails. At fleet scale this replaces reliability as the interesting quantity, because at sufficient scale everything fails and the only question is how much goes with it.

And a service class is a group of workloads with a common shape. A fleet does not run one workload; it runs many, and section 12 is about which of them a pool is actually for.

4. Teaching-Model Boundary

This is a case-study chapter and the boundary matters most here, because the subject is other people's deployments.

Every model computes a property of a decision, not a claim about any operator. No figure in this chapter is a measurement of any real fleet, service or company. Nine hundred racks, five hundred and twelve gigabytes per host, twenty-four tenants — all are teaching figures chosen to make one relationship visible.

Nothing here is attributed to a named organisation, and no number should be read as one. The chapter deliberately does not name operators, because naming one and attaching an invented figure to it is precisely the failure section 5 is about. What is publicly established and used here is only this: that memory stranding in production fleets is a documented and widely discussed motivation for pooling, and that CXL is the interconnect most often proposed to address it. No stranding percentage, deployment size or roadmap in this chapter belongs to anybody.

Each model is built twice from one source. A parameter selects between the measured build, which counts what the deployment rests on, and the deploying build, which treats the verb as the finding. Every section's headline number is the gap between them.

5. RTL 1 — Deploying Is A Verb Without A Number

Start with the sentence, because it is the only part of most fleet arguments that actually travels.

"Deploying" covers one rack and a million servers equally. It is true the moment anybody powers on a single test system, and it stays true through every scale above that. A claim that is satisfied at both ends of a six-order-of-magnitude range has not located anything, and the range is exactly what a reader needs.

There is a second failure stacked on the first: the claim is usually unattributed. "The hyperscalers" is a plural with no members, which means there is nobody to ask.

// RTL 1 - "the hyperscalers are deploying it" is a claim with no quantity in it.
// Deploying is a verb that covers one rack and a million servers equally, and
// the properties that decide whether it means anything are all numbers the
// sentence does not contain.
module deployment_has_no_quantity #(parameter int DEPLOYING_IS_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] quantities_needed, quantities_given, named_operators, claims,
  output logic [15:0] missing, given_ok, attribution_pct, quantified_pct,
  output logic        claim_quantified,
  output logic [7:0]  n_evals, n_vague,
  output logic        claim_err
);
  logic [31:0] a_q, q_q;
  logic [15:0] true_missing;
  logic        truly_vague;
  assign given_ok = (quantities_given > quantities_needed)
                  ? quantities_needed : quantities_given;
  assign true_missing = quantities_needed - given_ok;
  assign missing = (DEPLOYING_IS_ENOUGH != 0) ? 16'd0 : true_missing;
  // How much of the claim is attributed to somebody who could be asked.
  assign a_q = (claims == 16'd0) ? 32'd0
             : (({16'd0, named_operators} * 32'd100) / {16'd0, claims});
  assign attribution_pct = (a_q > 32'd100) ? 16'd100 : a_q[15:0];
  assign q_q = (quantities_needed == 16'd0) ? 32'd100
             : (({16'd0, given_ok} * 32'd100) / {16'd0, quantities_needed});
  assign quantified_pct = (DEPLOYING_IS_ENOUGH != 0) ? 16'd100 : q_q[15:0];
  assign claim_quantified = (missing == 16'd0) && (quantities_needed != 16'd0);
  assign truly_vague = (true_missing != 16'd0);
  assign claim_err = evaluate && truly_vague && claim_quantified;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_vague <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_vague) n_vague <= n_vague + 8'd1;
    end
  end
endmodule

Five quantities needed with one given, in a claim where two of eight statements name an operator, leaves four quantities missing, a quarter attributed, and a fifth of the claim quantified.

Figure 1 — the claim that cannot be checked because there is nobody to check with. The upper path is an appeal to authority with the authority left as a plural. The lower path asks two things — how much, and whose — and both are answerable in principle and unanswered in practice, which is the difference between a claim and a rumour with a technology attached.

The second case is the claim done properly. Every quantity given and every statement attributed reports a fully quantified claim, both builds agree, and neither alarms — which is what a conference talk with numbers in it looks like.

The fourth case separates the two failures, and it is the one worth dwelling on. More operators named than there are claims saturates the attribution at a hundred percent while three quantities are still missing — a claim can be perfectly attributed and still say nothing, because naming who said it is not the same as saying what they said.

The fifth case is the sentence in its natural state. Nothing quantified and nobody named, which is the form the claim actually travels in.

The degenerate case bounds it: no claim enumerated at all reports nothing missing and nothing quantified, which is silence rather than vagueness.

The five quantities are worth naming, because "quantities needed" is otherwise as vague as the claim it replaces. They are the five the rest of this chapter builds — the fleet fraction, the stranding recovered, the tail effect, the blast radius, and the operational cost. Every one of them is a number the deploying operator would have, and none of them survives the trip into the sentence that gets repeated.

This weak definition differs in kind from the other two in Module 29, and the difference is worth being precise about. 29.1's and 29.2's are claims about a system that a reader could in principle go and measure. This one is a claim about other people's behaviour, which a reader cannot measure at all — they can only ask the people, who are not named. The claim has therefore been constructed, accidentally, to be unfalsifiable by anybody who hears it.

And it would survive its own refutation. If every deployment it refers to were being quietly rolled back, "the hyperscalers are deploying it" would still have been true at the time somebody said it. A claim whose truth is unaffected by the outcome of the thing it describes cannot be evidence for doing that thing, which is the whole of section 5 in one sentence.

The fix is the same two questions in every case. Which operator, and how much. Not because the answers are secret — operators publish a great deal — but because the sentence discards them on the way to the meeting, and asking restores them at no cost.

6. RTL 2 — A Pilot Is Not A Fleet

The second thing, and the first number the claim should have contained.

A deployment is a fraction. One rack is a deployment; the whole estate is a deployment; and the difference between them is everything a reader wants to know. The fraction is a division nobody performs on their own announcement, because the numerator is impressive on its own and the denominator is not flattering.

The model also asks a second question that is easy to skip: how far past the pilot has it actually gone?

// RTL 2 - a pilot is not a fleet. The fraction of the estate a technology has
// actually reached is the difference between an experiment and a deployment,
// and it is a division nobody performs on their own announcement.
module fleet_fraction #(parameter int A_PILOT_IS_A_FLEET = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] fleet_racks, deployed_racks, pilot_racks, rollout_cost,
  output logic [15:0] deployed_ok, remaining, fleet_pct, rollout_total,
  output logic [15:0] beyond_pilot,
  output logic        fleet_wide,
  output logic [7:0]  n_evals, n_partial,
  output logic        fleet_err
);
  logic [31:0] p_q, c_q;
  logic [15:0] true_remaining;
  logic        truly_partial;
  assign deployed_ok = (deployed_racks > fleet_racks) ? fleet_racks : deployed_racks;
  assign true_remaining = fleet_racks - deployed_ok;
  assign remaining = (A_PILOT_IS_A_FLEET != 0) ? 16'd0 : true_remaining;
  assign p_q = (fleet_racks == 16'd0) ? 32'd100
             : (({16'd0, deployed_ok} * 32'd100) / {16'd0, fleet_racks});
  assign fleet_pct = (A_PILOT_IS_A_FLEET != 0) ? 16'd100 : p_q[15:0];
  // What finishing the rollout would still cost.
  assign c_q = {16'd0, true_remaining} * {16'd0, rollout_cost};
  assign rollout_total = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  // How far the rollout has travelled past the pilot it started as. Published
  // as a number rather than folded into the truth: a fleet that IS its pilot is
  // a real configuration, and the measured build must not alarm on it.
  assign beyond_pilot = (deployed_ok > pilot_racks) ? (deployed_ok - pilot_racks) : 16'd0;
  assign fleet_wide = (remaining == 16'd0) && (fleet_racks != 16'd0);
  assign truly_partial = (true_remaining != 16'd0);
  assign fleet_err = evaluate && truly_partial && fleet_wide;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_partial <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_partial) n_partial <= n_partial + 8'd1;
    end
  end
endmodule

Forty racks of nine hundred, where the pilot was twenty-five, is four percent of the fleet, fifteen racks beyond the pilot, and six thousand eight hundred and eighty units still to spend.

The second case is the real thing. The whole fleet deployed, with eight hundred and seventy-five racks beyond the pilot and nothing left to spend — both builds agree and neither alarms, which is what a finished rollout reads like.

The third case is the one my own review put in and it is a real configuration rather than a corner. A fleet that is exactly its own pilot is fleet-wide, because the fleet is that size — and the measured build must not alarm on it. A small operator whose entire estate is twenty-five racks has genuinely deployed fleet-wide, and an earlier version of this model called that a failure. Section 17 records why that was wrong.

The fourth case is the boundary and it is strict on purpose. One rack short of the fleet is ninety-nine percent and is not fleet-wide, because the remaining rack is somebody's production.

The sixth case is the clamp with the shape that matters. A rollout cost past the counter saturates on nine thousand racks with nothing deployed — which is the honest picture of a rollout that has been announced and not begun.

The distance past the pilot is the output that separates two very different four-percent deployments, and it is why the model publishes it rather than stopping at the fraction. Forty racks where the pilot was twenty-five means the thing has been extended into production and survived; forty racks where the pilot was forty means it has not left the lab. Both are four percent, and they are not the same result.

The remaining cost is the number that makes a fraction actionable. A four-percent deployment with a cheap rollout ahead of it is most of the way to a decision; the same fraction with six thousand eight hundred and eighty units still to spend is a decision that has barely started being made. A fraction alone says where something is; the fraction and the remaining cost say whether it is going anywhere.

And the fraction moves in one direction over time, which is the honest caveat on this section and the reason it belongs to a case study rather than to a specification. A four-percent deployment today may be a sixty-percent deployment in two years, so the number needs a date attached. A fleet fraction quoted without a date is a fact about a moment being used as a fact about a technology — which is a milder version of the same substitution section 5 is about.

7. RTL 3 — Stranded Memory Is The Actual Argument

The third thing, and the one that makes the whole idea worth having.

A host that has allocated all its cores while half its memory sits free has stranded that memory. It is installed, powered, cooled and paid for, and it cannot be rented to anybody because the thing that would rent it needs a core that is gone. At fleet scale that is a very large amount of capital doing nothing, and it is the reason pooling is interesting rather than merely possible.

// RTL 3 - stranded memory is the actual economic argument. A server whose cores
// are exhausted while its memory is half free has paid for memory it cannot
// rent, and the amount is measurable per host and enormous per fleet.
module stranded_memory #(parameter int NOTHING_IS_STRANDED = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] gb_per_host, gb_allocated, hosts, gb_cost,
  output logic [15:0] stranded_per_host, stranded_total, stranded_pct, wasted_spend,
  output logic        fully_used,
  output logic [7:0]  n_evals, n_stranded,
  output logic        strand_err
);
  logic [31:0] t_q, p_q, w_q;
  logic [15:0] alloc_ok, true_stranded, true_total;
  logic        truly_stranded;
  assign alloc_ok = (gb_allocated > gb_per_host) ? gb_per_host : gb_allocated;
  assign true_stranded = gb_per_host - alloc_ok;
  assign stranded_per_host = (NOTHING_IS_STRANDED != 0) ? 16'd0 : true_stranded;
  assign t_q = {16'd0, true_stranded} * {16'd0, hosts};
  assign true_total = (t_q > 32'd9999) ? 16'd9999 : t_q[15:0];
  assign stranded_total = (NOTHING_IS_STRANDED != 0) ? 16'd0 : true_total;
  assign p_q = (gb_per_host == 16'd0) ? 32'd0
             : (({16'd0, true_stranded} * 32'd100) / {16'd0, gb_per_host});
  assign stranded_pct = p_q[15:0];
  assign w_q = {16'd0, true_total} * {16'd0, gb_cost};
  assign wasted_spend = (w_q > 32'd9999) ? 16'd9999 : w_q[15:0];
  assign fully_used = (stranded_per_host == 16'd0) && (gb_per_host != 16'd0);
  assign truly_stranded = (true_stranded != 16'd0);
  assign strand_err = evaluate && truly_stranded && fully_used;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_stranded <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_stranded) n_stranded <= n_stranded + 8'd1;
    end
  end
endmodule

Five hundred and twelve gigabytes per host with three hundred allocated, across eighteen hosts, strands two hundred and twelve per host and three thousand eight hundred and sixteen in total — forty-one percent, at seven thousand six hundred and thirty-two units of unrentable spend.

Figure 2 — the argument that makes the rest of the chapter worth having. Both paths agree what the host contains; they disagree about whether it is earning. The per-host figure is the one that motivates and the fleet figure is the one that funds, and a proposal carrying only the first has understated its own case by the host count.

The second case is the fleet that does not need any of this. Every gigabyte allocated strands nothing, both builds agree, and neither alarms — and an operator whose fleet looks like that has no stranding problem to solve.

The third case is the boundary, and it is deliberately unforgiving. One gigabyte short of full is not fully used, and the percentage rounds to nothing — a fleet can be ninety-nine point eight percent allocated and still be leaving capacity on the table, which is why the section reports the absolute figure alongside the fraction.

The fifth case is the clamp at fleet scale. Nothing allocated across nine hundred hosts saturates both the total and the spend, which is the arithmetic reason the fleet figure is the one that gets a project funded.

Stranding happens because machines are built in fixed ratios and demand does not arrive in them. A host ships with so many cores and so much memory, and the tenants that land on it want cores and memory in whatever proportion their own workloads need. When the two ratios disagree, one resource runs out first and the other is stranded — and which one runs out first varies by host, by hour and by tenant mix.

That is why the fix is not simply "buy less memory". The same fleet that strands memory on one host is memory-bound on another, and a fleet-wide reduction would strand cores instead. Pooling is attractive precisely because it addresses the variance rather than the average, which is also why section 8's bet is the load-bearing part: pooling converts a stranding problem into an over-subscription problem, and the second one has to be sized.

There is a competing answer that a serious proposal has to beat, and section 19 names it: better bin-packing. A scheduler with more freedom about where to place work strands less memory without any new hardware at all. The stranding figure motivates both answers equally, so quoting it proves that something should be done rather than that this thing should be done.

The per-host and fleet figures do different jobs and both belong in the proposal. The per-host number is the one an engineer can verify on a machine in front of them, which is what makes the argument credible. The fleet number is the one that is large enough to fund a project, and a proposal carrying only the first has understated its own case by a factor of the host count.

8. RTL 4 — A Pool Is An Over-Subscription Bet

The fourth thing, and the one that decides whether the recovery is real or borrowed.

Pooling recovers stranded memory by selling it more than once. That works while the tenants' peaks do not coincide, and it is a bet on a statistical property of the workload mix rather than a property of the hardware. The bet has a size, and the size is the sum of simultaneous peaks against what the pool actually holds.

// RTL 4 - a pool is an over-subscription bet. Capacity is sold on the
// assumption that peaks do not coincide, and the bet is sound only while the
// sum of simultaneous demands stays inside what the pool actually holds.
module oversubscription_bet #(parameter int PEAKS_NEVER_COINCIDE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] pool_gb, tenants, peak_each, typical_each,
  output logic [15:0] typical_demand, peak_demand, shortfall, subscribe_pct,
  output logic        bet_holds,
  output logic [7:0]  n_evals, n_short,
  output logic        bet_err
);
  logic [31:0] t_q, k_q, s_q;
  logic [15:0] true_peak, true_short;
  logic        truly_short;
  assign t_q = {16'd0, tenants} * {16'd0, typical_each};
  assign typical_demand = (t_q > 32'd9999) ? 16'd9999 : t_q[15:0];
  assign k_q = {16'd0, tenants} * {16'd0, peak_each};
  assign true_peak = (k_q > 32'd9999) ? 16'd9999 : k_q[15:0];
  assign peak_demand = (PEAKS_NEVER_COINCIDE != 0) ? typical_demand : true_peak;
  assign true_short = (true_peak > pool_gb) ? (true_peak - pool_gb) : 16'd0;
  assign shortfall = (PEAKS_NEVER_COINCIDE != 0) ? 16'd0 : true_short;
  assign s_q = (pool_gb == 16'd0) ? 32'd999
             : (({16'd0, true_peak} * 32'd100) / {16'd0, pool_gb});
  assign subscribe_pct = (s_q > 32'd999) ? 16'd999 : s_q[15:0];
  assign bet_holds = (shortfall == 16'd0) && (tenants != 16'd0);
  assign truly_short = (true_short != 16'd0);
  assign bet_err = evaluate && truly_short && bet_holds;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_short <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_short) n_short <= n_short + 8'd1;
    end
  end
endmodule

Twenty-four tenants at a hundred and twenty peak and sixty typical against a two-thousand-and-forty-eight gigabyte pool is fourteen hundred and forty typical, two thousand eight hundred and eighty at the coincident peak — eight hundred and thirty-two short, a hundred and forty percent subscribed.

The second case is a pool that is not a bet at all. Ten tenants against the same pool covers even the coincident peak at fifty-eight percent subscribed — safe, and leaving capacity unsold, which is the conservative end of the trade.

The boundary cases are the pair that defines the edge. A pool exactly the size of the coincident peak holds at a hundred percent subscribed, and one gigabyte under does not. A fleet operating at exactly a hundred percent has no headroom for a tenant that grows.

The fifth case is the clamp and it is the shape of an aggressive bet. Nine hundred tenants saturates both demands and reports four hundred and eighty-eight percent subscribed — a pool sold five times over, which is a position somebody may take deliberately and should take knowingly.

The sixth case is the one the mutation campaign forced and section 17 explains why. A subscription past its ceiling — a pool a twelfth the size of its peak demand — saturates the ratio, and it exists because the zero-guard case that also reports the ceiling value does not actually exercise the clamp.

The model computes the coincident peak deliberately, and that is the conservative choice rather than the realistic one. In practice tenants do not all peak simultaneously, which is the entire basis of the technique — so a pool sized for the coincident peak is a pool that has not over-subscribed at all and has recovered nothing. The useful reading is therefore not "the shortfall must be zero" but "this is what the bet is exposed to", and an operator choosing to run at a hundred and forty percent is making a defensible decision with a number attached.

What makes the bet fail is correlation, and correlation is the input the model does not take. Tenants whose demand is driven by a shared external cause — a time of day, a shopping event, a news cycle — peak together, and for those the coincident peak is not a worst case but a weekly occurrence. The arithmetic here assumes independence, and a proposal that has not checked that assumption has computed the right number about the wrong population.

The typical demand output exists to show what is being given up. Fourteen hundred and forty against a two-thousand-and-forty-eight gigabyte pool means that in the ordinary case six hundred gigabytes sit unused — which is the stranding of section 7 reappearing inside the pool that was built to eliminate it. Pooling does not remove stranding; it moves it somewhere it can be shared, and how much remains is the difference between the two demand figures.

9. RTL 5 — A Fleet Is Run On Its Tail

The fifth thing, and the one that decides whether the deployment survives contact with a service objective.

Service objectives are written against percentiles, not means. A change that improves the average while lengthening the tail has improved the number a benchmark reports and regressed the number the business is held to. At fleet scale that distinction is not academic, because a tail event is a real request belonging to a real customer, and there are enough requests that the ninety-ninth percentile happens constantly.

// RTL 5 - a fleet is run on its tail. Mean latency is what a benchmark reports
// and the ninety-ninth percentile is what a service objective is written
// against, so a mean that improved while the tail grew is a regression.
module tail_not_mean #(parameter int MEAN_IS_THE_NUMBER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] mean_before, mean_after, p99_before, p99_after,
  output logic [15:0] mean_delta, p99_delta, tail_ratio, regression,
  output logic        objective_met,
  output logic [7:0]  n_evals, n_regressed,
  output logic        tail_err
);
  logic [31:0] r_q;
  logic [15:0] true_regression;
  logic        truly_regressed;
  assign mean_delta = (mean_after > mean_before) ? (mean_after - mean_before) : 16'd0;
  assign p99_delta  = (p99_after > p99_before) ? (p99_after - p99_before) : 16'd0;
  assign r_q = (mean_after == 16'd0) ? 32'd999
             : (({16'd0, p99_after} * 32'd100) / {16'd0, mean_after});
  assign tail_ratio = (r_q > 32'd999) ? 16'd999 : r_q[15:0];
  assign true_regression = p99_delta;
  assign regression = (MEAN_IS_THE_NUMBER != 0) ? mean_delta : true_regression;
  assign objective_met = (regression == 16'd0);
  assign truly_regressed = (true_regression != 16'd0);
  assign tail_err = evaluate && truly_regressed && objective_met;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_regressed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_regressed) n_regressed <= n_regressed + 8'd1;
    end
  end
endmodule

A mean from two hundred to a hundred and eighty with a tail from nine hundred to fourteen hundred is no mean regression, a five-hundred-unit tail regression, and a tail nearly eight times the mean.

The second case is the change that genuinely worked. Both improved, no regression, both builds agree, and neither alarms.

The fifth case is the one that keeps the section honest, and it is the only case in this chapter where the weak view is stricter than the measured one. The mean got worse and the tail did not — the measured build passes it, because the objective is written on the tail, and the mean-is-the-number build fails a run that the objective passes. A weak model is not always a permissive one, and a section that only ever showed the weak build over-claiming would be arguing rather than modelling.

The boundary pair is exact. Nothing moved meets the objective; one unit of tail regression does not.

The seventh case is the clamp that section 17 records. A tail ratio past its ceiling saturates, and it was added because the unmeasured-mean case reports the same ceiling value by a different route.

The tail ratio is the output that says how much a fleet should care. A service whose ninety-ninth percentile is twice its mean has a well-behaved distribution and can absorb a small tail movement; one whose tail is eight times its mean is already queueing somewhere, and any added distance lands on a distribution that is poorly conditioned to receive it. The ratio is a property of the service before the change, which makes it the cheapest available predictor of whether the change is safe for that service.

The mechanism behind Figure 3's divergence is not mysterious and it is worth stating, because it is what makes the two rows move in opposite directions rather than merely at different rates. Added distance means more requests must be in flight to sustain the same bandwidth. When the available concurrency runs out, requests queue. Queueing adds a small amount to almost every request and a large amount to a few — which lowers nothing in the mean while stretching the tail, and the mean can still fall because the common case genuinely did get faster.

So the two rows are not two opinions about one quantity. They are correct measurements of different things, and the operator is contractually held to one of them. A rollout review that reads the other is not being sloppy; it is reading a real number that happens not to be the one that matters, which is a much harder mistake to catch than an error.

Section 9's fifth case is in the chapter to prevent the obvious over-correction. If the tail is the number, then a mean regression with no tail movement is not a failure — and the measured build says so, while the mean-based build fails a run that the objective passes. The rule is "read the number you are held to", not "the tail is always worse."

10. RTL 6 — A Pool Serving Many Hosts Is A Domain Containing All Of Them

The sixth thing, and the one that changes the availability model rather than the performance one.

At fleet scale, reliability stops being the interesting number. With enough devices, everything fails on a schedule; what matters is how much stops when one does. A pool that serves sixty-four hosts is a sixty-four-host failure, and no component reliability figure changes that.

// RTL 6 - a pool serving many hosts is a failure domain containing all of them.
// At fleet scale the question is not whether it fails but how much stops when
// it does, and the answer is a host count rather than a probability.
module blast_radius #(parameter int A_POOL_IS_A_DEVICE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] hosts_served, pools, hosts_total, service_value,
  output logic [15:0] served_ok, radius, independent_hosts, outage_value,
  output logic        radius_bounded,
  output logic [7:0]  n_evals, n_wide,
  output logic        blast_err
);
  logic [31:0] v_q;
  logic [15:0] true_radius;
  logic        truly_wide;
  assign served_ok = (hosts_served > hosts_total) ? hosts_total : hosts_served;
  assign true_radius = served_ok;
  assign radius = (A_POOL_IS_A_DEVICE != 0) ? 16'd1 : true_radius;
  assign independent_hosts = hosts_total - served_ok;
  assign v_q = {16'd0, radius} * {16'd0, service_value};
  assign outage_value = (v_q > 32'd9999) ? 16'd9999 : v_q[15:0];
  assign radius_bounded = (radius <= 16'd1) && (hosts_total != 16'd0);
  // A pool serving one host is a device; serving many is a domain.
  assign truly_wide = (true_radius > 16'd1) && (pools != 16'd0);
  assign blast_err = evaluate && truly_wide && radius_bounded;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_wide <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_wide) n_wide <= n_wide + 8'd1;
    end
  end
endmodule

A pool serving sixty-four of two hundred hosts is a sixty-four-host blast radius, a hundred and thirty-six hosts independent of it, and one thousand nine hundred and twenty units of service value at risk.

The second case is the bounded configuration. A pool serving one host has a one-host radius — which is a card rather than a pool, and it is the configuration 29.2 is about.

The boundary case is deliberately strict. A pool serving two hosts is already unbounded, because the second host is somebody who did not previously depend on the first host's memory.

The sixth case is the guard the model needs. Hosts served with no pool described does not alarm in either build, because with no pool there is no pooling claim to check — the same shape as 29.2 section 9's empty interleave set, and it exists so the measured build cannot alarm on itself.

At fleet scale, component reliability stops being the useful question. With enough devices, the probability that some device fails today approaches one, so a specification figure for mean time between failures does not tell an operator anything they can act on. What they can act on is how much stops, and that is a host count available from an inventory rather than a statistic requiring a population.

The independent-host count is the output an availability model actually needs. Two hundred hosts with a pool serving sixty-four is not two hundred independent units; it is a hundred and thirty-six plus one failure domain of sixty-four. An availability model that counted two hundred was not slightly optimistic — it was modelling a machine that does not exist, and the discrepancy only becomes visible during an incident.

The radius is also a placement decision rather than a fixed property, which is what makes this section actionable. An operator chooses which hosts a pool serves. Aligning the pool's membership with an existing failure domain costs some pooling efficiency and preserves the availability model; spreading it across domains maximises the efficiency and quietly couples things that were deliberately decoupled. Both are legitimate; only one of them is usually chosen on purpose.

11. RTL 7 — The Saving Is What Is Left After The Infrastructure

The seventh thing, and the one that turns section 7's large number into a decision.

Recovering stranded memory requires building something. Switches, cables, controllers, rack space that could have held a revenue-generating server. A gigabyte recovered at a cost greater than the gigabyte was worth is not a saving, and the arithmetic is a subtraction that section 7's headline figure invites people to skip.

// RTL 7 - the saving is what is left after the infrastructure that produced it.
// A pool needs switching, cabling, controllers and rack space, and a recovered
// gigabyte that cost more to recover than to buy was not a saving.
module recovered_not_free #(parameter int RECOVERY_IS_THE_SAVING = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] recovered_gb, gb_value, infra_cost, rack_penalty,
  output logic [15:0] gross_saving, total_cost, net_saving, net_loss,
  output logic        saving_real,
  output logic [7:0]  n_evals, n_negative,
  output logic        save_err
);
  logic [31:0] g_q, c_q;
  logic [15:0] true_gross, true_cost;
  logic        truly_negative;
  assign g_q = {16'd0, recovered_gb} * {16'd0, gb_value};
  assign true_gross = (g_q > 32'd9999) ? 16'd9999 : g_q[15:0];
  assign gross_saving = true_gross;
  assign c_q = {16'd0, infra_cost} + {16'd0, rack_penalty};
  assign true_cost = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign total_cost = (RECOVERY_IS_THE_SAVING != 0) ? 16'd0 : true_cost;
  // Reported against the cost this build admits to, so the recovery-only view
  // stays coherent; the truth below still uses the real cost.
  assign net_saving = (true_gross > total_cost) ? (true_gross - total_cost) : 16'd0;
  assign net_loss   = (total_cost > true_gross) ? (total_cost - true_gross) : 16'd0;
  assign saving_real = (RECOVERY_IS_THE_SAVING != 0) ? (true_gross != 16'd0)
                                                     : (true_gross > true_cost);
  assign truly_negative = (true_cost >= true_gross) && (true_gross != 16'd0);
  assign save_err = evaluate && truly_negative && saving_real;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_negative <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_negative) n_negative <= n_negative + 8'd1;
    end
  end
endmodule

Eight hundred gigabytes recovered at six, against four thousand of infrastructure and fifteen hundred of rack penalty, is four thousand eight hundred of gross against five thousand five hundred of cost — seven hundred of net loss.

The second case is the deployment worth doing. Seven thousand two hundred of gross against three thousand five hundred of cost is three thousand seven hundred net, both builds agree, and neither alarms.

The boundary case is strict for the same reason every cost boundary in this batch is. Gross exactly equal to cost is not a saving, because a change that nets to zero on the two quantities measured is a loss on the ones that are not — which is section 13, and it is the next section for a reason.

The sixth case is the honest zero and it is a real outcome. Nothing recovered at all makes the entire build-out a loss, which is what happens when a pool is built for a fleet that section 7 would have shown was not stranded.

The rack penalty is the term that is most often missing, and it is not a small one. Space that holds pooling infrastructure is space that could have held a revenue-generating server, so the cost is not merely what the equipment cost but what the displaced machine would have earned. In a facility that is power- or space-constrained — which is most of them — that opportunity cost can exceed the equipment cost outright, and a proposal counting only capital expenditure has counted the smaller half.

The gross figure is where section 7's large number cashes out, and it shrinks on the way. Not all stranded memory is recoverable: some is stranded on hosts the pool does not reach, some is fragmented too finely to sell, and some belongs to tenants who will not accept the added latency. The recovered figure is a subset of the stranded figure, and a proposal that uses the second where it means the first has overstated the gross before the costs are even subtracted.

Which is why the boundary case is strict. Breaking even on the two quantities this model measures is a loss once section 13's operational cost is counted — and section 13 is the next section precisely because it is the term that turns marginal deployments negative. The model deliberately stops before it, so the two can be read separately and the reader can see how little headroom a break-even build-out actually has.

The coherence note applies here as in 29.1 and 29.2: the recovery-only build reports its net against the zero cost it claims, so it cannot publish a free build-out and a loss at once. The truth it is checked against still uses the real cost.

12. RTL 8 — Which Service Classes Actually Gain

The eighth thing, and the one that says where in a fleet the pool belongs.

A fleet runs many kinds of workload. Pooled capacity helps the ones whose memory demand varies — that is where the over-subscription bet has something to work with — and harms the ones whose latency objective is tight, because section 9's tail is where the added distance shows up. The classification is per service class, and a fleet-wide answer has to be wrong about part of the fleet.

// RTL 8 - which service classes gain. A fleet runs many kinds of workload, and
// pooled capacity helps the ones whose memory demand varies while harming the
// ones whose latency objective is tight.
module service_classes #(parameter int THE_WHOLE_FLEET = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] classes, elastic_classes, latency_tight, gain_each,
  output logic [15:0] suited, unsuited, total_gain, suited_pct,
  output logic        suits_fleet,
  output logic [7:0]  n_evals, n_unsuited,
  output logic        class_err
);
  logic [31:0] g_q, p_q;
  logic [15:0] el_ok, lt_ok, true_suited, true_unsuited;
  logic        truly_unsuited;
  assign el_ok = (elastic_classes > classes) ? classes : elastic_classes;
  assign lt_ok = (latency_tight > classes) ? classes : latency_tight;
  assign true_suited = (el_ok > lt_ok) ? (el_ok - lt_ok) : 16'd0;
  assign suited = (THE_WHOLE_FLEET != 0) ? classes : true_suited;
  assign true_unsuited = classes - true_suited;
  assign unsuited = (THE_WHOLE_FLEET != 0) ? 16'd0 : true_unsuited;
  assign g_q = {16'd0, true_suited} * {16'd0, gain_each};
  assign total_gain = (g_q > 32'd9999) ? 16'd9999 : g_q[15:0];
  assign p_q = (classes == 16'd0) ? 32'd100
             : (({16'd0, true_suited} * 32'd100) / {16'd0, classes});
  assign suited_pct = p_q[15:0];
  assign suits_fleet = (unsuited == 16'd0) && (classes != 16'd0);
  assign truly_unsuited = (true_unsuited != 16'd0);
  assign class_err = evaluate && truly_unsuited && suits_fleet;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_unsuited <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_unsuited) n_unsuited <= n_unsuited + 8'd1;
    end
  end
endmodule

Fourteen classes with nine elastic and four latency-tight is five net suited, nine not, at two hundred and twenty-five units of gain — just over a third.

The second case is the fleet the technology is for. Every class elastic is suited entirely, at six hundred and thirty units of gain, both builds agree, and neither alarms.

The boundary case decides a marginal fleet. Equally elastic and latency-tight nets to nothing, because the gains and the harms cancel.

The fourth case is the direction nobody proposes and it is the expensive one. A mostly latency-tight fleet gains nothing at all, and the deploying view still calls it a sweep — the pool was built, is drawing power, and serves nothing that benefits.

The degenerate case bounds it: an unwritten fleet suits nothing, because nobody enumerated the classes the claim is about.

Elasticity and latency sensitivity are close to independent, which is why the classification needs both axes rather than one. A batch analytics job is elastic and latency-insensitive — the best possible candidate. An in-memory cache fronting a user request is inelastic and latency-tight — the worst. But a service can be elastic and latency-tight at once, and those are the ones where the decision is genuinely hard and where a one-axis classification gives a confident wrong answer.

The netting is the same arithmetic the other two Module 29 chapters use, and it says the same thing: a latency-tight class is not merely unhelped, it is harmed, so the harms subtract rather than failing to add. An evenly split fleet nets to zero rather than to half, and a proposal counting only the classes that improved has counted half its own result.

At fleet scale the classification has an operational consequence the smaller cases do not. Deciding per class means the placement system has to know the classification and honour it — which is a scheduler feature, a label somebody has to maintain, and a way for a service to be mislabelled. The per-class answer is correct and it is not free to act on, and that cost belongs in section 13 rather than being assumed away here.

And classifications drift. A service that was elastic at one scale becomes latency-tight when it starts fronting something interactive, and nothing about that change announces itself to the placement system. A fleet decision made on a classification needs the classification re-checked, which is one more thing section 13 has to have a tool for.

13. RTL 9 — A Fleet Pays For What It Has To Operate

The ninth thing, and the one that decides whether the deployment survives its second year.

Every new device class adds failure modes. Each needs a way to detect it, a way to diagnose it, a runbook, an alert, and somebody who understands it at three in the morning. None of that appears in a capacity saving, and at fleet scale it is the cost that compounds — because the tooling has to work across every machine, not just the ones in the pilot.

// RTL 9 - a fleet pays for what it has to operate. Every new device class adds
// failure modes to diagnose, tooling to build and people to carry a pager, and
// none of that appears in a capacity saving.
module operational_cost #(parameter int OPERATIONS_ARE_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] new_failure_modes, tools_needed, tools_built, per_mode_cost,
  output logic [15:0] tool_gap, ops_cost, covered_pct, built_ok,
  output logic        operable,
  output logic [7:0]  n_evals, n_uncovered,
  output logic        ops_err
);
  logic [31:0] c_q, p_q;
  logic [15:0] true_gap, true_cost;
  logic        truly_uncovered;
  assign built_ok = (tools_built > tools_needed) ? tools_needed : tools_built;
  assign true_gap = tools_needed - built_ok;
  assign tool_gap = (OPERATIONS_ARE_FREE != 0) ? 16'd0 : true_gap;
  assign c_q = {16'd0, new_failure_modes} * {16'd0, per_mode_cost};
  assign true_cost = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign ops_cost = (OPERATIONS_ARE_FREE != 0) ? 16'd0 : true_cost;
  assign p_q = (tools_needed == 16'd0) ? 32'd100
             : (({16'd0, built_ok} * 32'd100) / {16'd0, tools_needed});
  assign covered_pct = (OPERATIONS_ARE_FREE != 0) ? 16'd100 : p_q[15:0];
  assign operable = (tool_gap == 16'd0) && (new_failure_modes != 16'd0);
  assign truly_uncovered = (true_gap != 16'd0);
  assign ops_err = evaluate && truly_uncovered && operable;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_uncovered <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_uncovered) n_uncovered <= n_uncovered + 8'd1;
    end
  end
endmodule

Seven new failure modes with nine tools needed and four built is five tools missing, one thousand seven hundred and fifty units of operational cost, and forty-four percent covered.

The second case is the deployment that is ready. Every tool built closes the gap and reports it operable — and note that the operational cost is still one thousand seven hundred and fifty, because building the tooling does not make the failure modes go away. It makes them survivable.

The fifth case is a state worth naming. Failure modes with no tooling enumerated reports no gap and calls it operable — operable by an absence of requirements, which is what a deployment looks like before anybody has asked what could go wrong.

The sixth case is the guard, and it is the mirror of the fifth. Tooling missing but no new failure modes is not operable and does not alarm, because with nothing new that can fail there is no operability claim to contradict.

The operational cost is the only one of the chapter's nine quantities that grows after the deployment succeeds. Capital costs are paid once; stranding is recovered continuously; but every machine added to the pooled estate adds failure surface, and every year of operation adds incidents that need diagnosing by people who have to learn a device class that did not exist in their training. A saving booked in year one is spent quietly in years two and three, and nothing in the capacity model shows it.

The second case makes the point the section most needs to land. Building every tool closes the gap and makes the deployment operable — and the operational cost is unchanged at one thousand seven hundred and fifty. Tooling does not remove failure modes; it makes them survivable. A proposal that treats "we will build monitoring" as the answer to the operational cost has confused the two: the monitoring is what makes the cost payable, not what makes it go away.

The fifth case is the state most proposals are actually in, and it is worth reading carefully because the model calls it operable. No tooling was enumerated, so there is no gap — the deployment is operable by an absence of requirements rather than by a presence of capability. That is not a modelling artefact; it is exactly what a plan looks like before anybody has run the exercise of asking what can go wrong, and the honest response to a report of "no tool gap" is to ask whether anybody wrote the list.

And the failure modes are the input hardest to get right in advance, which section 20 records as partly unknowable. A device class at scale produces failures nobody predicted, and the only reliable way to enumerate them is to run some. That is an argument for staging a rollout rather than for skipping the estimate — a staged rollout converts an unknowable number into a measured one before the fleet-wide commitment is made.

14. RTL 10 — A Cloud-Fleet Case Study Assembled

Nine sections of inputs. This one puts them together.

// RTL 10 - a cloud-fleet case study assembled. Nine sections of inputs, one
// summary. "The hyperscalers are deploying it" is bit 0: true of somebody
// somewhere, and one sixth of a deployment.
module fleet_case_signoff #(parameter int DEPLOYING_IS_THE_ANSWER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic        operator_named, fraction_stated, stranding_measured,
  input  logic        tail_stated, radius_stated, operations_costed,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_evals, n_sound, n_claimed,
  output logic        signoff_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~operator_named;
  assign fail_mask[1] = ~fraction_stated;
  assign fail_mask[2] = ~stranding_measured;
  assign fail_mask[3] = ~tail_stated;
  assign fail_mask[4] = ~radius_stated;
  assign fail_mask[5] = ~operations_costed;
  assign conditions_met = {15'd0, operator_named} + {15'd0, fraction_stated}
                        + {15'd0, stranding_measured} + {15'd0, tail_stated}
                        + {15'd0, radius_stated} + {15'd0, operations_costed};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp: conditions_met sums six one-bit values, so the quotient cannot
  // exceed a hundred and a ceiling would be unreachable code.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  // The "they are deploying it" view reads bit 0 and stops.
  assign claimed = (DEPLOYING_IS_THE_ANSWER != 0) ? operator_named : truly_sound;
  assign sound = claimed;
  assign signoff_err = evaluate && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When an operator has been named and any one of the other five fails, the assembled model reports that the case study is not sound and the deploying view reports a case study.

Across the eight evaluations, the assembled model calls one case study sound and the deploying view calls seven of eight a case study.

The bit order is by how much of the deployment each condition carries. Bit 1 is first among the five because a fraction is what turns an anecdote into a deployment. Bit 2 is the argument for doing it at all. Bit 3 is what decides whether it can stay. Bits 4 and 5 are the two costs that arrive after the decision, which is why they are last and why they are the ones that go missing.

"The hyperscalers are deploying it" is bit 0, and it is the weakest of the three weak definitions in Module 29. 29.1's names a category containing its own opposite and 29.2's describes the packaging. This one is not a claim about the technology at all — it is a claim about other people's behaviour, offered in place of a reason, and it would remain true if every one of those deployments were failing.

The five other bits fail independently. A fraction can be stated by somebody who never measured the stranding it was meant to recover. The stranding can be measured with no idea what the peaks do when they coincide. The tail can be stated by a team that never asked how many hosts one pool serves. And the operational cost is the one that fails last and hurts longest, because it is the only one of the six that keeps accruing after the deployment is declared finished.

The right-hand terminal is somebody else's behaviour, and that is what makes this weak definition different in kind from the other two. A claim about a device can be checked against the device. A claim that other people are doing something can only be checked against those people, who are not in the room, are not named, and have not published the numbers — so the claim is unfalsifiable in practice while sounding like evidence.

15. Quantitative Reasoning

Four quantities of five missing, in a claim a quarter attributed and a fifth quantified.

Four percent of the fleet deployed — forty racks of nine hundred, fifteen of them beyond the pilot, with six thousand eight hundred and eighty units still to spend.

Two hundred and twelve gigabytes stranded per host and three thousand eight hundred and sixteen across eighteen, at forty-one percent and seven thousand six hundred and thirty-two units of unrentable spend.

A hundred and forty percent subscribed and eight hundred and thirty-two gigabytes short if twenty-four tenants peak together.

A five-hundred-unit tail regression under a mean that improved, with the tail nearly eight times the mean.

A sixty-four-host blast radius out of two hundred, at one thousand nine hundred and twenty units of service value.

Seven hundred units of net loss on a recovery worth four thousand eight hundred that cost five thousand five hundred to build.

Nine service classes of fourteen unsuited, with five suited at two hundred and twenty-five units of gain.

Five tools of nine missing and one thousand seven hundred and fifty units of operational cost, at forty-four percent covered.

One case study of eight sound; the deploying view counts seven. The assembled model's summary, and the chapter's.

16. Assertions

The testbenches carry 474 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported nothing on either testbench, and this is the second chapter running where all five scripted checks passed on their first run.

Both builds are asserted on every degenerate case. No claim enumerated, no fleet described, no fleet measured, no tenants described, nothing measured at all, no estate described, nothing costed, no classes enumerated, and nothing enumerated at all.

Every clamp that an input can reach is driven past its limit exactly once, by a case chosen so the clamp actually fires. More quantities given than needed, more operators named than claims, more racks deployed than the fleet holds, a rollout cost past its counter, a stranded total and a wasted spend past theirs, a typical and a peak demand past theirs, a subscription ratio past its ceiling, a tail ratio past its ceiling, more hosts served than exist, an outage value past its counter, a gross saving and a build-out cost past theirs, a gain past its counter, more tools built than needed, and an operational cost past its counter. The two emphasised are section 17's finding.

Every threshold is asserted on both sides of its boundary. The fleet at one rack short and at complete; the pool at exactly the coincident peak and one gigabyte under; the tail at no movement and at one unit; the blast radius at one host and at two; the saving at exactly break-even.

Every error output is checked in both directions in every case, including section 9's fifth case, which is the one place in the chapter where the weak build is stricter than the measured one — it fails a run the objective passes, and both builds correctly stay quiet.

17. Mutation Testing

126 mutations, 126 killed. Sixty-three against the first testbench, sixty-three against the second. The first run killed a hundred and twenty-four and left two, and the two shared a cause that is the sharpest of the three findings this batch has produced.

Both survivors were clamps whose zero-guard default equals the clamp ceiling. Section 8's subscription ratio returns 999 when the pool is unsized, and its clamp is > 999. Section 9's tail ratio does the same. The stimulus had a case for each guard, the output read 999, and the clamp branch had never executed — because 999 is not greater than 999.

That is why it survived a campaign that has caught every other clamp in three chapters. The testbench output looked exactly like a clamp firing. A reader checking coverage by reading the printed values would see a saturated figure in a degenerate case and tick the box, and the two produce identical output by different paths.

The rule the batch has carried since 026 is for every clamp, drive the input past the limit once. This is the second clause it has needed in two chapters, and the two are related: 29.2 found that a symmetric pair is two clamps rather than one, and this chapter finds that a guard returning the ceiling value is not the clamp firing. Both are failures of the same kind — an output that looks covered because something else produced the same number.

Three distinct masking relationships now, and it is worth separating them because the correct response differs each time. 29.1's equivalent mutant could not be killed by any stimulus and had to be withdrawn. 29.2's survivor was a real hole closed by one case. These two were real holes hidden by a coincidence of values, and the diagnostic question that separates them is the same one every time: is there an input that would make the mutated line behave differently? For the equivalent mutant the answer was no. Here it was yes, and it took a pool a twelfth the size of its demand and a tail twenty times its mean to find it.

One defect was caught by reading before the campaign ran. Section 6's truth signal originally folded in the pilot comparison — remaining != 0 || deployed <= pilot — which meant a fleet that is its own pilot made the measured build alarm on itself. A small operator whose entire estate is twenty-five racks has genuinely deployed fleet-wide, and the model was calling that a failure. The fix was to publish the distance past the pilot as a number rather than fold it into the truth, which is the same separation 29.2 section 9's capacity guard makes: the truth must describe what happened, and the interesting-but-not-wrong observations belong in the outputs.

18. Verification Strategy

Ask which operator and what fraction. Section 5 and section 6. Both are one question and neither is usually answerable.

Measure the stranding before proposing the fix. Section 7. If the fleet is not stranded, nothing below this matters.

Size the bet against coincident peaks, not typical demand. Section 8.

Read the tail, not the mean. Section 9. The mean can improve through the whole regression.

Count how many hosts one pool serves. Section 10. That is the failure, not the device.

Subtract the build-out from the recovery. Section 11. Both numbers exist.

Classify the service classes. Section 12. The answer is per class.

Count the new failure modes and the tools that do not exist yet. Section 13.

And ask all six of somebody who can be named. Section 5 again, which is where it started.

19. Synthesis and Implementation Reality

Stranding is a scheduling artefact as much as a hardware one. It arises because cores and memory are allocated in fixed ratios per machine while demand arrives in varying ratios, and a scheduler with more freedom strands less. Pooling is one answer and better bin-packing is another, and a proposal that has not compared them has not made its case.

The over-subscription bet is the same bet every shared resource makes, and it fails the same way: correlated demand. Tenants whose peaks are driven by a common external event — a time of day, a sale, a news cycle — are not independent, and section 8's arithmetic assumes they are.

The tail regression in section 9 has a specific mechanism and it is 9.6's: a longer round trip needs more requests in flight to sustain bandwidth, and when the concurrency runs out requests queue. Queueing does not move a mean much and moves a tail a great deal, which is why the two rows in Figure 3 diverge rather than track.

The blast radius interacts with placement and scheduling, because an operator can choose which hosts a pool serves. Spreading a pool across hosts that belong to different failure domains defeats the purpose of those domains; concentrating it within one preserves them at the cost of some pooling efficiency.

And the operational cost is the one that is genuinely hard to estimate in advance, because the failure modes a new device class produces are not fully knowable until it has been at scale for a while — which is an argument for staging a rollout rather than for skipping the estimate.

20. Silicon Observability

Free, and already collected. Allocated memory per host against installed memory per host. Section 7 — this is telemetry every fleet already has, which makes the chapter's central argument the cheapest number in it.

Free, and from an inventory. Rack counts, host counts, and how many hosts a pool serves. Sections 6 and 10.

Cheap, and already collected. Mean and percentile latency per service. Section 9 needs the percentile, which is usually recorded and sometimes not reported.

Moderate. Per-tenant peak and typical demand. Section 8 needs a distribution rather than an average, and whether the peaks are correlated needs a joint one.

Moderate, and owned elsewhere. Infrastructure and rack-opportunity cost. Section 11's second column lives with the people who plan capacity.

Expensive. Per-service-class elasticity and latency sensitivity. Section 12 needs each class characterised against both axes.

Expensive, and partly unknowable in advance. The new failure modes and the tooling they require. Section 13 is the estimate that improves most with a staged rollout.

21. Debug Lab

A pooled deployment is live and a service is missing its objective.

Step 1 — read the tail, not the mean. Section 9. If the mean looks fine, that is not evidence; it is the symptom.

Step 2 — check whether the pool is over-subscribed right now. Section 8. A coincident peak explains a tail directly.

Step 3 — check which service class is affected. Section 12. If it is latency-tight, it should not have been on the pool.

Step 4 — check how many hosts share the pool. Section 10. A tail on many hosts at once is a pool event; a tail on one is a host event.

Step 5 — check the stranding the pool was built to recover. Section 7. If it was small, section 11's arithmetic was never going to work and the deployment is the problem rather than the tuning.

Step 6 — check what tooling does not exist. Section 13. If the answer to steps 1 to 5 is "we cannot tell", that is section 13's gap rather than a performance problem.

Steps 1 and 2 are both already instrumented and between them explain most instances.

22. Design Review

Which operator, and what fraction of which fleet?

How much memory is stranded per host, and across how many hosts?

What is the sum of coincident peaks against the pool size?

What did the ninety-ninth percentile do — not the mean?

How many hosts does one pool serve?

What did the build-out cost, against what the recovery is worth?

Which service classes are elastic and which are latency-tight?

What new failure modes does this add, and which tools do not exist yet?

And who, by name, can be asked about all of the above?

23. How This Appears In Real Engineering

The failure is a decision made on somebody else's authority, and it propagates because the authority is never named precisely enough to check.

The most common shape is section 5 end to end. A strategy is set because "the hyperscalers are doing it." No operator is named, no fraction is quoted, and the fleet that adopts it has a different workload mix, a different stranding profile and a different service objective from whoever the sentence was about.

The second is section 6. A pilot is reported as a deployment. Four percent of a fleet is a real and valuable result, and describing it as a deployment means the next team plans a dependency on capacity that exists in one datacentre.

The third is section 9 and it is the one that gets caught late. The rollout dashboard shows the mean improving. Every review passes, the service objective is quietly being missed the whole time, and the regression is found by a customer rather than by the rollout.

The fourth is section 10 discovered during an incident. A pool fails and sixty-four hosts stop. The availability model said the hosts were independent because they are separate machines, and the pool underneath them was not in the model.

The fifth is section 13 and it is the slowest. The deployment succeeds and the operational load never goes down, because the tooling gap was never closed and every incident on the new device class is a bespoke investigation. Two years later the saving is real and nobody can find it in a budget.

The pattern is that two of the six inputs are already collected and the decision is usually made without either of them — which is the most frustrating version of this batch's recurring shape.

24. Common Misconceptions

"The hyperscalers are deploying it." Which one, and how much of what? Section 5.

"It is deployed." In four percent of the fleet. Section 6.

"Pooling saves memory." It recovers stranding, which first has to be measured. Section 7.

"Not everybody peaks at once." Size the bet on the assumption that they might. Section 8.

"Average latency improved." The tail is what you are held to. Section 9.

"A pool failure is a device failure." It is a sixty-four-host failure. Section 10.

"We recovered 800 GB." At a build-out cost of 5,500. Section 11.

"It works for the fleet." For the elastic classes. Section 12.

"The hardware is deployed, so we are done." Five of nine tools do not exist. Section 13.

25. Interview Reasoning

"Why are cloud operators interested in CXL?" The honest answer is one number: stranded memory. A host whose cores are fully allocated while half its memory is free cannot rent that memory, and at fleet scale that is a large amount of installed capital earning nothing. Pooling lets several hosts draw on shared capacity, so memory that would have been stranded on one machine can be sold to another. That is the argument, and it is an economic one rather than a performance one.

"How would you evaluate whether it is worth doing?" Measure the stranding first, because if the fleet is not stranded nothing else matters — and that measurement is telemetry most operators already collect. Then size the pool against the sum of coincident peaks rather than typical demand. Then subtract the build-out cost from what the recovered capacity is worth. Three numbers, and the first one decides whether to compute the other two.

"What would make you reject it?" A latency-tight service mix, because the pool adds distance and the tail is what the objective is written against. An uncorrelated-peaks assumption that does not hold, because that is what the over-subscription depends on. And a blast radius the availability model cannot absorb — a pool serving sixty-four hosts is a sixty-four-host failure, however reliable the pool is.

"Why do you keep saying the tail rather than the mean?" Because a mean can improve through an entire regression. Added distance means more requests in flight to sustain bandwidth, and when concurrency runs out requests queue — queueing barely moves a mean and moves a tail a great deal. A rollout dashboard built on means will show continuous improvement while the service objective is being missed.

"Somebody told me the hyperscalers are all doing this." That sentence contains no quantity. Which operator, what fraction of their fleet, for which service class, against what measured stranding, and at what tail cost? All five are answerable and none is in the claim — and it would remain true if every one of those deployments were being rolled back.

"What gets forgotten?" The operational cost. A new device class brings failure modes that need detection, diagnosis, runbooks and people who understand them, and that bill keeps accruing after the capacity saving has been booked. It is the only one of the six conditions that gets more expensive the longer the deployment succeeds.

26. Exercises

1. A claim needs 7 quantities, gives 2, and 3 of 11 statements name an operator. Compute the missing count, the attribution and the quantified fraction.

2. 2,400 racks, 180 deployed, a 60-rack pilot, 12 per rack to roll out. Compute the fraction, the distance past the pilot and the remaining cost.

3. 768 GB per host, 410 allocated, 40 hosts, 3 per GB. Compute the stranding per host, the fleet total and the unrentable spend. What allocation halves it?

4. A 4,096 GB pool, 40 tenants, 150 peak and 70 typical. Compute both demands, the shortfall and the subscription. How many tenants does the pool actually cover at peak?

5. Mean 300 → 260, p99 1,100 → 1,900. Compute both deltas and the tail ratio. Which number would you report, and why?

6. A pool serving 96 of 400 hosts at 45 each. Compute the radius, the independent count and the value at risk. What does splitting it into three pools give?

7. 1,500 GB recovered at 5, infrastructure 4,200, rack penalty 2,000. Compute the gross, the cost and the net. At what recovery does it break even?

8. 22 classes, 13 elastic, 6 latency-tight, 70 each. Compute suited, unsuited and gain.

9. 11 new failure modes, 14 tools needed, 6 built, 300 per mode. Compute the gap, the cost and the coverage.

10. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify its position using the rule that the ordering is by how much of the deployment each condition carries.

27. Summary

Deploying is a verb without a number, and it is true at one rack and at a million servers equally.

A pilot is not a fleet — four percent is a real result and describing it as a deployment makes somebody else plan wrongly.

Stranded memory is the actual argument, and it is telemetry most operators already have.

A pool is an over-subscription bet whose size is the sum of coincident peaks, not the typical demand.

A fleet is run on its tail, and a mean can improve through an entire regression.

A pool serving many hosts is a domain containing all of them, which changes the availability model rather than the performance one.

The saving is what is left after the infrastructure that produced it, and both numbers exist.

Elastic classes gain and latency-tight ones do not, so the answer is per service class.

A fleet pays for what it has to operate, and that bill is the one that keeps accruing.

Six bits, and "the hyperscalers are deploying it" is one of them. One case study of eight is sound; the deploying view counts seven.