Resource Utilisation

23.1 ended on a fleet whose utilisation rose from 75% to 96% and got worse. 23.2 ended on a programme reporting 94% of instances composed while the thing it promised never happened. Both deferred the same question, and this chapter is it: what does a utilisation number actually say?

The answer is that it says whatever its numerator, denominator, weighting and observation window were chosen to say — and those four choices span a range wide enough to report the same fleet at 25% or at 96% without anyone lying.

That is not a reason to distrust the number. It is a reason to specify it, and this chapter is the specification.

1. The Engineering Problem — Four Choices, One Number

Allocated is not touched. A fleet with 150 of 200 GB reserved and 50 touched is 75% or 25%, and both are computed correctly. Section 5.

A mean is not a provisioning number. A fleet averaging 40% and peaking at 95% has five points of spare capacity, not sixty. Section 6.

The denominator decides the answer. 80 GB used of 100 allocated in a 200 GB fleet is 80% or 40%. Section 7.

And averaging percentages is not averaging. A full 64 GB node beside a quarter-used 512 GB one averages to 62% and the fleet is 33% used. Section 8.

Some unused capacity is required. A fleet at 95% needing 20 points of headroom is already failing, and a metric that calls it excellent has inverted its own objective. Section 9.

This chapter against 23.1 and 23.2, stated precisely. Those own what pooling and composing cost. This one owns the measurement both were judged by — which is why every model here is about a number rather than about a mechanism, and why section 14's assembled model scores a dashboard rather than a fleet.

2. The One-Sentence Model

A rising utilisation figure means the fleet improved when it measures what was touched rather than what was reserved, is based on the peak rather than the mean, is weighted by capacity and by time, preserves the headroom the fleet requires, and credits the gain to what actually caused it — and every defect below is a number that went up while nothing did.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real telemetry stack is agents, a time-series database, a rollup pipeline and a dashboard, and none of that is reproduced. What is reproduced is the arithmetic each performs, and the shape of the mistake when it does not.

Each model is built twice — a correct build and a broken build selected by a parameter. Every broken build here is a defensible measurement choice, which is what makes this chapter different from the two before it. Counting reservations is what an allocator can see. Averaging an interval is what a sampler produces. Dividing by the allocation is what a tenant cares about. None of them is a bug; each is the wrong statistic for the question being asked.

Figure 1 — The same fleet at 75%, 25% and 80%, with no error anywhere. Each number answers a different question, and the failure is not computing one of them — it is putting one on a dashboard labelled "utilisation" and making a purchasing decision from it.

5. RTL 1 — Allocated Is Not Touched

// RTL 1 - allocated is not used. A tenant that reserves memory and touches a
// quarter of it is reported at full utilisation by anything counting reservations.
module utilisation_definition #(parameter int ALLOCATED_IS_USED = 0) (
  input  logic clk, rst_n,
  input  logic        report,
  input  logic [15:0] fitted_gb, allocated_gb, touched_gb,
  output logic [15:0] alloc_pct, touched_pct, gap_pct,
  output logic        honest,
  output logic [7:0]  n_reports, n_misleading,
  output logic        reservation_counted_err
);
  logic [31:0] a_q, t_q;
  logic [15:0] eff_touched;
  assign a_q = (fitted_gb == 16'd0) ? 32'd0
             : (({16'd0, allocated_gb} * 32'd100) / {16'd0, fitted_gb});
  assign alloc_pct = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  // Counting the reservation as the usage is what an allocator's own view does.
  assign eff_touched = (ALLOCATED_IS_USED != 0) ? allocated_gb : touched_gb;
  assign t_q = (fitted_gb == 16'd0) ? 32'd0
             : (({16'd0, eff_touched} * 32'd100) / {16'd0, fitted_gb});
  assign touched_pct = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  // alloc_pct is never below touched_pct while touched_gb is at or below
  // allocated_gb, and the guard below covers the case where it is not.
  assign gap_pct = (alloc_pct > touched_pct) ? (alloc_pct - touched_pct) : 16'd0;
  assign honest = (gap_pct <= 16'd10);
  // Memory reserved and never touched, reported as used.
  assign reservation_counted_err = report && (fitted_gb != 16'd0)
                                   && (touched_gb < allocated_gb)
                                   && (touched_pct == alloc_pct);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reports <= 8'd0; n_misleading <= 8'd0;
    end else if (report) begin
      n_reports <= n_reports + 8'd1;
      if (!honest) n_misleading <= n_misleading + 8'd1;
    end
  end
endmodule

Six reports. 200 GB fitted, 150 GB reserved.

Two reports were misleading; the reservation model reported none.

A fifty-point gap is the normal state of a fleet, not a pathology. Tenants reserve for their peak and run at their mean; the allocator sees the reservation and the hardware sees the touch. Both numbers are correct measurements of different things, and the one that belongs on a capacity plan is the second.

Row three is the case that makes the distinction concrete. A 75% allocated fleet touching nothing is a fleet with 200 GB of idle DRAM and a dashboard reading three-quarters full — and it is exactly the state 23.1 §5 called stranding, arrived at from the opposite direction.

Why the broken build is not a strawman. An allocator knows what it granted and does not know what was read; counting reservations requires no instrumentation and counting touches requires page-level accounting the hypervisor may not expose. The cheap number is the reservation, so the reservation is what most fleets report.

6. RTL 2 — A Mean Is Not A Provisioning Number

// RTL 2 - a mean is not a provisioning number. Capacity has to cover the peak,
// and an interval average hides exactly the moment that sets it.
module sampling_interval #(parameter int SAMPLE_THE_MEAN = 0) (
  input  logic clk, rst_n,
  input  logic        sample,
  input  logic [15:0] mean_pct, peak_pct,
  output logic [15:0] provisioning_basis, spare_pct,
  output logic        safe,
  output logic [7:0]  n_samples, n_unsafe,
  output logic        peak_hidden_err
);
  // Provisioning is set by the peak; an averaged sample reports the mean.
  assign provisioning_basis = (SAMPLE_THE_MEAN != 0) ? mean_pct : peak_pct;
  assign spare_pct = (provisioning_basis < 16'd100)
                     ? (16'd100 - provisioning_basis) : 16'd0;
  assign safe = (spare_pct >= 16'd10);
  // A peak that the reported number does not contain.
  assign peak_hidden_err = sample && (peak_pct > mean_pct)
                           && (provisioning_basis == mean_pct);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_samples <= 8'd0; n_unsafe <= 8'd0;
    end else if (sample) begin
      n_samples <= n_samples + 8'd1;
      if (!safe) n_unsafe <= n_unsafe + 8'd1;
    end
  end
endmodule

Five samples. A fleet averaging 40%.

Two samples were unsafe; the averaging model reported none.

Provisioning covers the peak and a mean is what a sampler naturally produces. Every rollup in a time-series pipeline averages by default — five-second samples into one-minute buckets into one-hour buckets — and each rollup discards exactly the moment that decides the capacity.

Row one is a factor of twelve on the number that matters. Five points of spare against sixty is the difference between a fleet one incident from exhaustion and one with room to grow, and both are honest summaries of the same hour.

Row four is what a real pipeline delivers. A peak above capacity comes from a counter read across a resize, a double-counted allocation, or a sampling race — and the floor turns it into zero spare rather than an unsigned wrap, which is the class 23.1 §12 met with over-reported holdings.

The rollup is where the information is lost and it is lost silently. A pipeline storing means at every tier can never reconstruct the peak, so the decision to keep a maximum alongside the mean has to be made before the data anyone needs has been discarded.

7. RTL 3 — The Denominator Decides The Answer

// RTL 3 - the denominator decides the answer. Usage against what was allocated
// and usage against what was fitted are different numbers about the same fleet.
module denominator_choice #(parameter int DIVIDE_BY_ALLOCATED = 0) (
  input  logic clk, rst_n,
  input  logic        report,
  input  logic [15:0] fitted_gb, allocated_gb, used_gb,
  output logic [15:0] vs_allocated_pct, vs_fitted_pct, reported_pct,
  output logic        honest,
  output logic [7:0]  n_reports, n_misleading,
  output logic        flattering_denominator_err
);
  logic [31:0] a_q, f_q;
  assign a_q = (allocated_gb == 16'd0) ? 32'd0
             : (({16'd0, used_gb} * 32'd100) / {16'd0, allocated_gb});
  assign vs_allocated_pct = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign f_q = (fitted_gb == 16'd0) ? 32'd0
             : (({16'd0, used_gb} * 32'd100) / {16'd0, fitted_gb});
  assign vs_fitted_pct = (f_q > 32'd65535) ? 16'hFFFF : f_q[15:0];
  // A fleet number has to be against what the fleet contains, not against what
  // somebody asked for.
  assign reported_pct = (DIVIDE_BY_ALLOCATED != 0) ? vs_allocated_pct
                                                   : vs_fitted_pct;
  assign honest = (reported_pct == vs_fitted_pct);
  // The larger of the two ratios quoted as the fleet's utilisation.
  assign flattering_denominator_err = report && (vs_allocated_pct > vs_fitted_pct)
                                      && (reported_pct == vs_allocated_pct);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reports <= 8'd0; n_misleading <= 8'd0;
    end else if (report) begin
      n_reports <= n_reports + 8'd1;
      if (!honest) n_misleading <= n_misleading + 8'd1;
    end
  end
endmodule

Six reports. 200 GB fitted, 80 GB used.

None misleading when the fleet is the denominator; five of six when the allocation is.

"A hundred percent utilised" in row three describes 50 GB of a 200 GB fleet. Both ratios are correct and they answer different questions: against the allocation is a tenant's question — am I using what I asked for — and against the fitted capacity is the operator's. A capacity plan built from the tenant's ratio buys hardware to satisfy people who are already satisfied.

Row two is the only case where the two coincide, and it is the case a fully-committed fleet is in. That is worth naming because it is the state most people picture when they say "utilisation": everything allocated, so the distinction has no room to appear.

Row six is the sharpest. A fleet with nothing fitted reports 80% against its allocations, which is a number about a fleet that does not exist. The fleet ratio's guard returns zero and the allocation ratio's does not, and the difference is only visible because both are computed.

8. RTL 4 — Averaging Percentages Is Not Averaging

// RTL 4 - averaging the averages. A fleet's utilisation is total used over total
// fitted, not the mean of per-node percentages.
module aggregation_fallacy #(parameter int AVERAGE_THE_AVERAGES = 0) (
  input  logic clk, rst_n,
  input  logic        report,
  input  logic [15:0] node_a_gb, node_a_used_gb, node_b_gb, node_b_used_gb,
  output logic [15:0] a_pct, b_pct, simple_avg_pct, weighted_pct, reported_pct,
  output logic        honest,
  output logic [7:0]  n_reports, n_misleading,
  output logic        unweighted_err
);
  logic [31:0] a_q, b_q, w_q;
  logic [15:0] total_gb, total_used;
  assign a_q = (node_a_gb == 16'd0) ? 32'd0
             : (({16'd0, node_a_used_gb} * 32'd100) / {16'd0, node_a_gb});
  assign a_pct = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign b_q = (node_b_gb == 16'd0) ? 32'd0
             : (({16'd0, node_b_used_gb} * 32'd100) / {16'd0, node_b_gb});
  assign b_pct = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  assign simple_avg_pct = (a_pct + b_pct) / 16'd2;
  assign total_gb = node_a_gb + node_b_gb;
  assign total_used = node_a_used_gb + node_b_used_gb;
  assign w_q = (total_gb == 16'd0) ? 32'd0
             : (({16'd0, total_used} * 32'd100) / {16'd0, total_gb});
  assign weighted_pct = (w_q > 32'd65535) ? 16'hFFFF : w_q[15:0];
  // A fleet figure must weight each node by the capacity it contributes.
  assign reported_pct = (AVERAGE_THE_AVERAGES != 0) ? simple_avg_pct : weighted_pct;
  assign honest = (reported_pct == weighted_pct);
  // Nodes of different sizes averaged as if they were the same size.
  assign unweighted_err = report && (node_a_gb != node_b_gb)
                          && (reported_pct == simple_avg_pct)
                          && (simple_avg_pct != weighted_pct);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reports <= 8'd0; n_misleading <= 8'd0;
    end else if (report) begin
      n_reports <= n_reports + 8'd1;
      if (!honest) n_misleading <= n_misleading + 8'd1;
    end
  end
endmodule

Six reports.

None misleading when weighted by capacity; four of six when the percentages are averaged.

Row five is the number this section exists for. A small full node beside a large idle one averages to 50% and the fleet is five percent used — a factor of ten, from an operation that looks like taking an average and is not one.

Rows two and three are the two ways the fallacy vanishes, and they matter because they are why it survives review: when the nodes are the same size, or when they are at the same rate, the simple average is exactly right. A homogeneous fleet can average percentages for years without consequence, and the error appears the day a different-sized node is added.

Row six is a distinct effect and the one that produced a mutation survivor. At equal capacities, per-node rounding can still separate the two figures by a point — 33 and 66 average to 49 while the fleet is exactly half used. That is not the weighting fallacy; it is integer division, and section 17 records why telling them apart mattered.

Figure 3 — Ten times apart, from two nodes and one arithmetic choice. The dashed edges are what "average the readings" throws away: the capacity each reading stands for, which is the only thing that makes the readings comparable.

9. RTL 5 — Headroom Is Not Waste

// RTL 5 - headroom is not waste. Some unused capacity is required for burst and
// for failover, so a hundred percent utilised fleet is a fleet that is already
// failing.
module headroom_accounting #(parameter int HEADROOM_IS_WASTE = 0) (
  input  logic clk, rst_n,
  input  logic        check,
  input  logic [15:0] used_pct, required_headroom_pct,
  output logic [15:0] target_pct, slack_pct,
  output logic        healthy,
  output logic [7:0]  n_checks, n_unhealthy,
  output logic        headroom_denied_err
);
  // The usable target is a hundred percent minus the headroom the fleet needs.
  assign target_pct = (HEADROOM_IS_WASTE != 0) ? 16'd100
                    : ((required_headroom_pct < 16'd100)
                       ? (16'd100 - required_headroom_pct) : 16'd0);
  assign slack_pct = (target_pct > used_pct) ? (target_pct - used_pct) : 16'd0;
  assign healthy = (used_pct <= target_pct);
  // A required headroom treated as capacity to be filled.
  assign headroom_denied_err = check && (required_headroom_pct != 16'd0)
                               && (target_pct == 16'd100);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_checks <= 8'd0; n_unhealthy <= 8'd0;
    end else if (check) begin
      n_checks <= n_checks + 8'd1;
      if (!healthy) n_unhealthy <= n_unhealthy + 8'd1;
    end
  end
endmodule

Seven checks. A fleet needing twenty points of headroom.

Four checks were unhealthy; the headroom-is-waste model reported none.

This is the only model in the chapter where the metric's direction is wrong rather than its value. Every other section reports a number that is too high or too low; here, a fleet that is 100% utilised is a fleet that has already failed, and a metric maximising utilisation is maximising toward the failure.

Row four is the honest exemption. A fleet that genuinely needs no headroom — a fixed, non-bursting, non-failover-protected workload — can run to 100%, and headroom_denied_err requires a real requirement to exist before it fires. The error is spending headroom that was required, not the absence of headroom.

Row seven is what a policy field above 100% produces, and it is the input that made this guard's mutation killable — section 17 records the reachability argument. A requirement to reserve more than the fleet contains is a configuration error, and the correct response is a target of zero rather than an unsigned wrap to 65,516.

10. RTL 6 — Readings Are Not Equally Long

// RTL 6 - readings are not equally long. A high reading held for six minutes and
// a low one held for fifty-four do not average to the middle.
module time_weighted #(parameter int COUNT_SAMPLES = 0) (
  input  logic clk, rst_n,
  input  logic        report,
  input  logic [15:0] high_pct, high_min, low_pct, low_min,
  output logic [15:0] total_min, counted_pct, weighted_pct, reported_pct,
  output logic        honest,
  output logic [7:0]  n_reports, n_misreported,
  output logic        equal_weight_err
);
  logic [31:0] w_q;
  assign total_min = high_min + low_min;
  assign counted_pct = (high_pct + low_pct) / 16'd2;
  assign w_q = (total_min == 16'd0) ? 32'd0
             : ((({16'd0, high_pct} * {16'd0, high_min})
               + ({16'd0, low_pct} * {16'd0, low_min})) / {16'd0, total_min});
  assign weighted_pct = (w_q > 32'd65535) ? 16'hFFFF : w_q[15:0];
  // Each reading counts for the time it was held, not for being a reading.
  assign reported_pct = (COUNT_SAMPLES != 0) ? counted_pct : weighted_pct;
  assign honest = (reported_pct == weighted_pct);
  // Readings of different durations counted as if they were the same duration.
  assign equal_weight_err = report && (high_min != low_min)
                            && (reported_pct == counted_pct)
                            && (counted_pct != weighted_pct);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reports <= 8'd0; n_misreported <= 8'd0;
    end else if (report) begin
      n_reports <= n_reports + 8'd1;
      if (!honest) n_misreported <= n_misreported + 8'd1;
    end
  end
endmodule

Five reports. Ninety percent and ten percent, in some proportion, over an hour.

None misreported when time is the weight; four of five when the readings are.

The count average is 50% in every row and the truth ranges from 10 to 82. That is the whole section: averaging readings gives each measurement equal weight regardless of how long it described the fleet, and a pipeline sampling irregularly — on change, on scrape, on a jittered timer — produces readings of wildly unequal duration.

Row four is the degenerate form and it happens constantly. A reading captured and immediately superseded describes the fleet for zero minutes and gets half the weight. On-change sampling produces exactly this: a brief spike emits a reading, the recovery emits another, and a count average treats a two-second excursion as half the hour.

Row two is where the two agree, and the algebra says something useful: at equal durations the time average reduces exactly to the count average, including under integer division. That is not an approximation, and it is why a strictly-periodic sampler can average readings safely.

11. RTL 7 — A Gain Has Several Causes

// RTL 7 - a utilisation gain has several causes. Crediting all of it to the pool
// is what makes a programme look better than the thing it changed.
module recovery_attribution #(parameter int CREDIT_EVERYTHING = 0) (
  input  logic clk, rst_n,
  input  logic        attribute,
  input  logic [15:0] total_gain_pct, from_pool_pct, from_rightsizing_pct,
  input  logic [15:0] from_consolidation_pct,
  output logic [15:0] sum_of_causes_pct, credited_to_pool_pct, overclaim_pct,
  output logic        consistent, honest,
  output logic [7:0]  n_attributions, n_overclaimed,
  output logic        credit_taken_err
);
  assign sum_of_causes_pct = from_pool_pct + from_rightsizing_pct
                           + from_consolidation_pct;
  assign consistent = (sum_of_causes_pct == total_gain_pct);
  // Crediting the whole gain to the change being defended is the failure.
  assign credited_to_pool_pct = (CREDIT_EVERYTHING != 0) ? total_gain_pct
                                                         : from_pool_pct;
  assign overclaim_pct = (credited_to_pool_pct > from_pool_pct)
                         ? (credited_to_pool_pct - from_pool_pct) : 16'd0;
  assign honest = (overclaim_pct == 16'd0);
  // A gain with several causes credited entirely to one of them.
  assign credit_taken_err = attribute && (total_gain_pct > from_pool_pct)
                            && (credited_to_pool_pct == total_gain_pct);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_attributions <= 8'd0; n_overclaimed <= 8'd0;
    end else if (attribute) begin
      n_attributions <= n_attributions + 8'd1;
      if (!honest) n_overclaimed <= n_overclaimed + 8'd1;
    end
  end
endmodule

Six attributions. A twenty-one point utilisation gain.

None overclaimed when the cause is credited; three of six when the programme is.

A capacity programme rarely runs alone. Pooling lands at the same time as a rightsizing campaign and a consolidation push, because they are all funded by the same efficiency initiative — and the utilisation figure moves once, for three reasons, into a report defending one of them.

Row three is the case worth stating plainly. The pool contributed nothing and the gain was twenty-one points; crediting it to the pool is not exaggeration, it is a complete inversion, and nothing in the utilisation number itself can distinguish it from row five.

Row six is the consistency check earning its place. Causes summing to sixteen against a twenty-one point gain means something is unattributed — which is an honest and useful state to report, and quite different from an attribution that is wrong. The model reports both.

12. RTL 8 — Capacity Held By Workloads That Are Gone

// RTL 8 - capacity held by workloads that are not doing anything. Allocated to
// something that no longer runs is not utilisation, however the allocator sees it.
module false_utilisation #(parameter int IGNORE_ZOMBIES = 0) (
  input  logic clk, rst_n,
  input  logic        audit,
  input  logic [15:0] fitted_gb, allocated_gb, zombie_gb,
  output logic [15:0] live_gb, alloc_pct, live_pct, inflation_pct,
  output logic        acceptable,
  output logic [7:0]  n_audits, n_inflated,
  output logic        zombie_counted_err
);
  logic [31:0] a_q, l_q, i_q;
  // Capacity held by workloads that are gone still shows as allocated.
  assign live_gb = (IGNORE_ZOMBIES != 0) ? allocated_gb
                 : ((allocated_gb > zombie_gb) ? (allocated_gb - zombie_gb) : 16'd0);
  assign a_q = (fitted_gb == 16'd0) ? 32'd0
             : (({16'd0, allocated_gb} * 32'd100) / {16'd0, fitted_gb});
  assign alloc_pct = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign l_q = (fitted_gb == 16'd0) ? 32'd0
             : (({16'd0, live_gb} * 32'd100) / {16'd0, fitted_gb});
  assign live_pct = (l_q > 32'd65535) ? 16'hFFFF : l_q[15:0];
  // alloc_pct is never below live_pct, because live_gb never exceeds allocated_gb.
  assign inflation_pct = alloc_pct - live_pct;
  assign acceptable = (inflation_pct <= 16'd10);
  // Capacity held by a dead workload, counted as live.
  assign zombie_counted_err = audit && (zombie_gb != 16'd0)
                              && (live_gb == allocated_gb);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_audits <= 8'd0; n_inflated <= 8'd0;
    end else if (audit) begin
      n_audits <= n_audits + 8'd1;
      if (!acceptable) n_inflated <= n_inflated + 8'd1;
    end
  end
endmodule

Six audits. 200 GB fitted, 192 GB allocated.

Three audits were inflated; the zombie-blind model reported none.

A dead workload's allocation is indistinguishable from a live one's to the thing that granted it. The allocator has a record; nothing in that record expires. A fleet at 96% allocated with a quarter of it held by jobs that finished is a fleet at 72%, and no amount of correct arithmetic on the allocation table reveals it.

This is section 5's problem with a different cause and the same shape, and the pairing is deliberate: there, capacity was reserved by something alive and unused; here it is reserved by something that no longer exists. The measurement fix is different — one needs page-level touch accounting, the other needs a liveness join against the scheduler — which is why they are separate models.

Row three is the state a long-running fleet drifts into. Every leaked allocation is permanent by default, so zombie capacity accumulates monotonically unless something reaps it, and a rising utilisation figure on an old fleet is as likely to be leakage as growth.

13. RTL 9 — A Gain Is Worth Nothing Until It Retires A Unit

// RTL 9 - a utilisation gain saves money only when it removes a whole unit.
// Capacity is bought in servers, and a fleet does not buy nine tenths of one.
module utilisation_value #(parameter int ASSUME_LINEAR = 0) (
  input  logic clk, rst_n,
  input  logic        value,
  input  logic [15:0] before_gb, after_gb, unit_gb,
  output logic [15:0] units_before, units_after, freed_gb, saved_units,
  output logic        worthwhile,
  output logic [7:0]  n_valuations, n_worthwhile,
  output logic        fractional_saving_err
);
  logic [31:0] b_q, a_q, s_q;
  assign b_q = (unit_gb == 16'd0) ? 32'd0
             : (({16'd0, before_gb} + {16'd0, unit_gb} - 32'd1) / {16'd0, unit_gb});
  assign units_before = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  assign a_q = (unit_gb == 16'd0) ? 32'd0
             : (({16'd0, after_gb} + {16'd0, unit_gb} - 32'd1) / {16'd0, unit_gb});
  assign units_after = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign freed_gb = (before_gb > after_gb) ? (before_gb - after_gb) : 16'd0;
  // A linear model turns freed gigabytes into a fraction of a server, which is
  // not a thing a fleet can stop buying.
  assign s_q = (ASSUME_LINEAR != 0)
             ? ((unit_gb == 16'd0) ? 32'd0 : ({16'd0, freed_gb} / {16'd0, unit_gb}))
             : ((units_before > units_after)
                ? ({16'd0, units_before} - {16'd0, units_after}) : 32'd0);
  assign saved_units = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  assign worthwhile = (saved_units >= 16'd1);
  // A saving reported smaller than the whole units the fleet can actually retire.
  assign fractional_saving_err = value && (units_before > units_after)
                                 && (saved_units < (units_before - units_after));

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_valuations <= 8'd0; n_worthwhile <= 8'd0;
    end else if (value) begin
      n_valuations <= n_valuations + 8'd1;
      if (worthwhile) n_worthwhile <= n_worthwhile + 8'd1;
    end
  end
endmodule

Seven valuations. Servers holding 256 GB each.

Three valuations were worthwhile; the linear model found two.

Row two is the row every efficiency programme needs on its first slide. A ten percent reduction in demand that retires no server saves nothing at all — not a tenth of a server, not a tenth of the power, nothing. Capacity is bought and decommissioned in whole units, and the gain is worth zero until it crosses one.

Row three is the opposite mistake and it is under-claiming. Freeing 500 GB retires two 256 GB servers, not one — the linear model divides the freed bytes by the unit size and gets one, because it ignores that the fleet was rounded up at both ends. Both errors come from the same missing ceiling.

Row seven is a fleet that grew, driven because the guard exists for it: without it, units_before - units_after underflows and reports a saving from a regression. That is the input that made the guard's mutation killable, and section 17 records the argument.

14. RTL 10 — Resource Utilisation Assembled

// RTL 10 - resource utilisation assembled. Everything that must hold before a
// rising utilisation number means the fleet got better.
module utilisation_model #(parameter int NUMBER_WENT_UP = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       number_rose,        // the reported figure went up
  input  logic       measures_touched,   // usage, not reservations
  input  logic       peak_not_mean,      // the provisioning basis is the peak
  input  logic       capacity_weighted,  // aggregated by capacity and by time
  input  logic       headroom_preserved, // the required slack is not counted as waste
  input  logic       attributed_honestly,// the gain is credited to what caused it
  output logic       improved,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_improved,
  output logic       false_gain_err
);
  assign fail_mask[0] = ~number_rose;
  assign fail_mask[1] = ~measures_touched;
  assign fail_mask[2] = ~peak_not_mean;
  assign fail_mask[3] = ~capacity_weighted;
  assign fail_mask[4] = ~headroom_preserved;
  assign fail_mask[5] = ~attributed_honestly;
  // The number-went-up build is the dashboard, and the dashboard is the metric
  // every capacity programme is measured on.
  assign improved = (NUMBER_WENT_UP != 0) ? number_rose : (fail_mask == 6'd0);
  assign false_gain_err = evaluate && improved && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_improved <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (improved) n_improved <= n_improved + 8'd1;
    end
  end
endmodule

One improving configuration of six, and four false claims.

The dashboard is right about exactly one thing: whether the number went up. Every other property is a choice made upstream in the pipeline, invisible at the point the number is read, and each one alone is enough to make the rise meaningless.

Row four deserves separate mention because its failure is the worst kind: the number rose because the headroom was consumed, which is a fleet moving toward an outage reported as a fleet moving toward efficiency. The metric and the objective are pointing in opposite directions.

Figure 4 — Four questions, and a figure that fails any of them is not a worse number — it is a number about something else. None of the three rejections is a defect to be fixed in the dashboard; each is a property of how the data was collected, which is upstream of anywhere the figure is displayed.

15. Quantitative Reasoning

Allocated against touched. 200 GB fitted, 150 reserved, 50 touched: 75% or 25%, a fifty-point gap, both correct.

Sampling. A fleet averaging 40% and peaking at 95% has five points of spare capacity, and the averaging model reports sixty — a factor of twelve.

Denominator. 80 GB used of 100 allocated in a 200 GB fleet: 80% or 40%; of 50 allocated, 100% or 25%.

Aggregation. A full 64 GB node beside a quarter-used 512 GB one: 62% averaged, 33% weighted. Beside an idle 1024 GB node: 50% averaged, 5% weighted — ten times.

Headroom. A fleet at 95% needing twenty points is at a target of 80 and failing, which the waste model reports as healthy.

Time weighting. 90% for six minutes and 10% for fifty-four: 50% counted, 18% time-weighted. Reverse the durations and the truth is 82% while the counted figure does not move.

Attribution. A twenty-one point gain of which seven came from the pool: crediting all of it overclaims by fourteen, and when the pool caused none of it, by twenty-one.

Zombies. 192 GB allocated with 48 held by dead workloads: 96% or 72% — a twenty-four point inflation.

Value. 1100 to 900 GB retires one server; 1000 to 900 retires none; 1000 to 500 retires two where a linear model finds one.

The assembled model. Six properties, six configurations, one improved. The dashboard reported five.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Every inclusive threshold is driven at exactly equal, and every ceiling at both a remainder and a whole multiple — the two carry-forwards this batch has accumulated.

Definition. A gap of exactly ten points is constructed from 130 GB touched of 200 fitted.

chk(dGt == 16'd65, "touching 130 of 200 is 65 percent");
chk(dGg == 16'd10, "exactly a ten point gap");
chk(dGh == 1'b1,   "which is exactly honest");

Sampling. Spare capacity of exactly ten points is driven, and a peak above capacity is asserted to floor at zero.

Denominator. The fully-allocated case is asserted as one where both ratios are honest, and a fleet with nothing fitted is asserted to report zero rather than a ratio of a fleet that does not exist.

Aggregation. Equal capacities, equal rates, and equal capacities with rounding divergence are all driven separately.

chk(gGs == 16'd49, "which average to 49 after rounding");
chk(gGw == 16'd50, "while the fleet is exactly half used");
chk(gBe == 1'b0,   "at equal capacities the divergence is rounding, not weighting");

Headroom. Usage exactly at the target is driven, a requirement of exactly 100% is driven, and a requirement of 120% is asserted to produce a target of zero.

Time weighting. Equal durations are asserted to make the two averages agree, and a zero-duration reading is driven with both builds asserted.

Attribution. A gain entirely caused by the pool is asserted as one both models get right, and causes that do not sum to the gain are asserted inconsistent.

Zombies. An inflation of exactly ten points is constructed from 20 GB dead, and an over-reported zombie figure is asserted to floor at zero live.

Value. A whole-unit reduction is asserted to make both models agree, and a fleet that grew is asserted to report no saving.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 293 checks across two testbenches, 146 on the front five models and 147 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Sixty-two mutations were injected one at a time. 62 injected, 62 killed, after four survivors.

All four survivors were guards whose reachability lives in the input pipeline rather than in the arithmetic — a class this batch has not seen before, and one that behaves differently from both previous ones.

Survivor 1 — reachable through integer rounding. Section 8's unweighted_err carries an equal-size guard. Algebraically, equal capacities make the simple and weighted averages identical, so the guard reads as dead. Integer division breaks the identity: two nodes of 300 GB at 100 and 200 GB used give per-node percentages of 33 and 66, averaging to 49, while the fleet is exactly 50. Driving that case kills the mutation and demonstrates that an algebraic proof of redundancy is not a proof about the implementation.

Survivor 2 — reachable through a configuration error. Section 9's target guard survives because no case set a headroom requirement above 100%. That is not a number a sane policy contains, and it is exactly what a mis-scaled field or a fraction stored as a percentage produces. Without the guard it wraps to 65,516.

Survivor 3 — reachable through a degenerate sample. Section 10's duration guard survives because the zero-duration case was only asserted on the correct build. Adding the broken build's assertion on that case kills it with no new stimulus at all — the case was present and only half-observed.

Survivor 4 — reachable through a regression. Section 13's retirement guard survives because no case had the fleet grow. units_before - units_after underflows when it does, and the check fires on a fleet that got worse. A "gain" model must be driven with a loss.

The general rule these four produce. Batch 021's rule was drive every inclusive threshold at equality. This batch's second chapter added drive every ceiling off its boundary. These four add the third: when a guard resists a reachability argument, look outside the arithmetic — at integer rounding, at malformed configuration, at degenerate samples, and at the opposite of the thing being measured. Three of the four are inputs the model's own domain says should not exist, and all three are inputs a real pipeline produces.

The complete set was re-run after every stimulus change, per standing discipline, and all sixty-two held.

18. Verification Strategy

What a testbench for a measurement model must cover.

Prove reachability outside the arithmetic before declaring a guard dead. All four survivors here looked redundant under algebra and were reachable under integer division, bad configuration, degenerate input, or a reversed sign of the quantity being measured.

Assert both builds on degenerate cases. Survivor 3 needed no new stimulus — only the second assertion on a case already driven. A case that is present and half-observed is a case that is half-tested.

Drive the complement of the thing being measured. A saving model needs a loss. A gain model needs a regression. The guard exists precisely for the case the model is not named after.

The cases where the broken build is right. A tenant touching everything reserved. A perfectly flat workload. A fully allocated fleet. Equal-capacity nodes. Equal-duration readings. A gain the pool really caused. A fleet needing no headroom. A whole-unit reduction. Eight cases across nine models, each exempted explicitly, and each a state a real fleet is genuinely in.

Counters as a second signature. Ten models, ten pairs of totals, differing in all ten — two misleading against none, none against five, none against four, three inflated against none.

What a real telemetry stack needs that these models do not have. Every model here takes its inputs as given. A real pipeline's hardest problems are upstream of all of them: agents that miss scrapes, counters that reset, and rollups that have already discarded what section 6 needs. Section 26 exercise 8 is the closest this chapter comes.

19. Synthesis and Implementation Reality

Nothing in this chapter is hardware. Every model is arithmetic a telemetry pipeline or a capacity planner performs, and the failure mode is a decision rather than a timing violation.

Section 5's numerator is the expensive one. Reservations are free — the allocator already has them. Touched pages need either hypervisor accounting or hardware access-bit sampling, which is a real cost and the reason most fleets report the cheap number.

Section 6's rollup is where the peak is destroyed, and the fix is a schema decision made before any data exists: keep a maximum alongside the mean at every tier. Retrofitting it does not recover the past.

Section 8's weighting is a query, not a collector. The data to compute it correctly is almost always present; the dashboard averages because averaging a column is one function call and weighting it is a join. This is the cheapest of the six to fix and among the most commonly wrong.

Section 9's headroom is a policy number that must live beside the metric. A utilisation figure with no target attached is uninterpretable, and the target is the only thing that turns a percentage into a judgement.

Section 12's liveness join is the second expensive one. Identifying zombie allocations means correlating the allocator's table against the scheduler's view of what is running, across systems that often disagree about identity. The reaper is easier to build than the report.

20. Silicon Observability

"The definition itself, versioned" is the counter that has no analogue in the other chapters of this module. A utilisation figure's meaning changes when the numerator, denominator, weighting or window changes, and those changes are invisible in the time series — a step in the graph looks identical whether the fleet changed or the pipeline did. Versioning the definition is the only thing that lets a year-old number be compared with today's.

21. Debug Lab

Symptom. A capacity team is asked to justify a 30% hardware reduction. Their dashboard shows fleet memory utilisation at 88%, up from 61% a year ago, and the pool programme is credited with the rise. A reduction is approved. Six weeks after it lands, the fleet is dropping allocations during every daily peak.

Step 1 — what is the numerator? Allocated against touched: the figure counts allocations. Touched bytes are 54% of fitted. Section 5, and the two are thirty-four points apart.

Step 2 — what is the denominator? Against fitted capacity or against allocations: against fitted, correctly. Section 7 is not the problem here, which is worth establishing rather than assuming.

Step 3 — how is it aggregated? Per-node percentages, averaged. The fleet has two node sizes since last year's refresh — 512 GB and 1 TB — and the smaller ones run hotter. Section 8: capacity-weighted, the figure is 79%, not 88%.

Step 4 — mean or peak? The pipeline rolls five-minute means into hourly means into a daily mean. The daily peak is 96%. Section 6, and the 30% reduction was sized against a number nine points below the number that matters, on a fleet that already had no room.

Step 5 — is there headroom in the target? The policy requires 15 points for failover. Nobody attached it to the metric, so the dashboard's "88%, healthy" was never compared against a target of 85. Section 9, and by the peak figure the fleet was already over.

Step 6 — was the rise the pool's? Three initiatives ran that year: the pool, a rightsizing campaign, and a JVM heap-tuning effort. Section 11 — the pool's share of the 27-point rise is nine points, and the report credited it with all twenty-seven.

Step 7 — and how much of the allocation is live? Allocation age against last-touch: 11% of allocated bytes have not been touched in ninety days. Section 12. The reaper has been broken since the pool migration changed the allocation record format.

The finding. Six defects, all in the measurement, none in the fleet. Corrected: touched rather than allocated, capacity-weighted, at the peak, against an 85% target, the figure is not 88% with room to cut — it is over target at peak with 11% of its allocations dead. The right action was a reaper fix and no reduction at all.

What made this hard. Every individual step is defensible. Averaging per-node percentages is what the dashboard tool does by default. Rolling up means is what the time-series database does by default. Counting allocations is what the allocator can see. Nobody chose any of it, and the compounded error was thirty points in the direction that justified the decision already being made.

22. Design Review

1. What is the numerator — bytes reserved or bytes touched? A fifty-point difference. Section 5.

2. What is the denominator — fitted capacity or allocations? A tenant's question and an operator's question. Section 7.

3. Is the fleet figure weighted by capacity, or an average of per-node percentages? Ten times, on a heterogeneous fleet. Section 8.

4. Is it weighted by time, or an average of readings? A count average is 50% whether the truth is 10 or 82. Section 10.

5. Is it a mean or a peak, and at which rollup tier was that decided? Provisioning covers the peak, and the rollup deletes it. Section 6.

6. What target is it compared against, and where does that number live? A percentage with no target is uninterpretable. Section 9.

7. How much of the allocation has not been touched in ninety days? Section 12, and it accumulates monotonically.

8. What else changed in the period this gain is being credited to? Section 11.

9. How many whole servers does this gain retire? The only figure that becomes money. Section 13.

10. Is the definition versioned beside the number? A step in the graph looks the same whether the fleet changed or the pipeline did. Section 20.

23. How This Appears In Real Engineering

A capacity-planning function owns all ten questions and typically inherits the answers from a dashboard nobody specified. The single highest-value change is attaching the target to the metric — section 9 — because a percentage with no target cannot be acted on and a percentage with one cannot be misread as far.

A telemetry or observability team owns sections 6 and 10, and both are schema decisions rather than query decisions. A rollup that keeps only means, and a reading stored without its duration, destroy information that no downstream query can recover — which makes them the two decisions on this list that have to be right the first time.

A platform team defending a programme meets section 11, and the incentive is uncomfortable rather than subtle: the attribution that makes the programme look best is also the one nobody has the data to dispute. Recording concurrent initiatives with their start dates costs nothing and is almost never done.

A finance function meets section 13, and it is the one section where the engineering and the accounting agree exactly. Capacity is bought and retired in whole units, so a gain that retires none is worth zero — which is a more familiar idea in finance than in engineering, and a useful common language between them.

24. Common Misconceptions

"The fleet is 75% utilised." 75% allocated. 25% touched. Section 5.

"We average 40%, so there is plenty of room." You peak at 95% and have five points. Section 6.

"We are at 100% utilisation." Of your allocations, which are a quarter of the fleet. Section 7.

"The average node is 62% used." The fleet is 33% used. Section 8.

"Higher utilisation is better." Not past the target — a full fleet is a failing one. Section 9.

"We averaged the readings." Each reading described the fleet for a different length of time. Section 10.

"The pool raised utilisation 21 points." Seven of them. Section 11.

"96% is allocated." A quarter of it to workloads that finished. Section 12.

"We cut demand 10%." And retired no servers, so you saved nothing. Section 13.

"Utilisation went up, so the fleet improved." One property of six. Section 14.

25. Interview Reasoning

Q. Your dashboard says the fleet is 75% utilised. What do you ask?

What the numerator is. Reserved bytes and touched bytes differ by fifty points on a typical fleet — 150 GB reserved of 200 fitted with 50 touched is 75% or 25%, both correctly computed. The follow-up is the denominator: against fitted capacity or against the allocations, which is another factor of two.

Q. Two nodes: a 64 GB one that is full and a 1 TB one that is idle. What is the fleet's utilisation?

Five percent — 64 GB used of 1088 fitted. Averaging the two percentages gives 50%, which is ten times wrong, and it is what almost every dashboard does by default because averaging a column is one function call and weighting it is a join.

Q. Why can averaging a utilisation time series be wrong even when each reading is right?

Because readings are not equally long. A 90% reading held for six minutes and a 10% reading held for fifty-four time-average to 18% and count-average to 50% — and irregular or on-change sampling makes durations wildly unequal. The reading needs its duration stored beside it or the average cannot be computed later.

Q. Should a capacity team maximise utilisation?

Up to the target, and the target is a hundred percent minus the headroom the fleet requires for burst and failover. A fleet at 100% with a 20-point requirement has already failed, so a metric that maximises utilisation without a target is pointed at the outage.

Q. A fleet's utilisation rose 21 points the year the memory pool landed. How much did the pool cause?

Unknown from that number alone, and typically not all of it — efficiency initiatives are funded together and land together. If rightsizing contributed nine points and consolidation five, the pool's share is seven, and crediting it with twenty-one is a three-times overclaim that nobody has the data to dispute unless the initiatives were recorded with their dates.

Q. You cut demand ten percent. What did you save?

Possibly nothing. Capacity is bought in whole servers, so a reduction from 1000 GB to 900 GB across 256 GB servers retires none of them — the fleet still needs four. The only figure that becomes money is whole units retired, and a ten percent gain is worth zero until it crosses one.

26. Exercises

1. Extend RTL 1 with a per-tenant breakdown and find which tenant contributes most of the fleet's reservation gap.

2. Give RTL 2 a full distribution rather than a mean and a peak, and find the percentile at which provisioning should be set for a given risk.

3. Combine RTL 3 and RTL 4: show that choosing the flattering denominator and averaging the averages compound rather than cancel.

4. Extend RTL 4 to N nodes and show that the fallacy's error grows with the spread of node sizes.

5. Make RTL 5's required headroom a function of failure rate and recovery time, and derive it rather than assuming it.

6. Drive RTL 6 with a realistic on-change sampling pattern and measure the error a count average introduces.

7. Extend RTL 7 to overlapping initiatives with different start dates and attribute a gain across them.

8. Model a rollup pipeline that keeps only means and show that RTL 2's peak cannot be recovered from it at any later tier.

9. Combine RTL 8 and RTL 13: find the zombie fraction at which reaping alone retires a whole server.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the fleet it catches that the current mask calls improved.

27. Summary

23.1 and 23.2 both ended on a programme whose metric improved while the fleet did not. This chapter is why that is so easy: a utilisation figure is four independent choices, and each one alone spans a factor of two to twelve.

Allocated is not touched. 150 GB reserved of 200 fitted with 50 touched is 75% or 25% — a fifty-point gap, and the cheap number to collect is the wrong one.

A mean is not a provisioning number. A fleet averaging 40% and peaking at 95% has five points of spare, not sixty — and every rollup tier that stores a mean destroys the peak permanently.

The denominator decides the answer. 80 GB used of 100 allocated in a 200 GB fleet is 80% to a tenant and 40% to an operator, and only one of them should buy hardware.

Averaging percentages is not averaging. A full 64 GB node beside an idle 1 TB one averages to 50% while the fleet is 5% used — and the fallacy is invisible on a homogeneous fleet, right up to the first refresh.

Headroom is not waste. A fleet at 95% needing twenty points is failing, and a metric that maximises utilisation without a target is aimed at the outage.

Readings are not equally long. 90% for six minutes and 10% for fifty-four is 18%, not 50% — and the count average does not move when the durations reverse and the truth becomes 82%.

A gain has several causes. Seven points of a twenty-one point rise came from the pool; crediting all of it is a three-times overclaim that nobody can dispute unless the concurrent initiatives were written down.

Allocations outlive their workloads. 48 GB of 192 held by jobs that finished is 96% allocated and 72% live — and zombie capacity accumulates monotonically unless something reaps it.

And a gain is worth nothing until it retires a whole unit. Cutting 1000 GB to 900 across 256 GB servers saves nothing at all, while cutting to 500 retires two where a linear model finds one.

Four mutations survived because their guards looked algebraically dead and were reachable through the input pipeline — integer rounding at equal capacities, a headroom policy above 100%, a zero-duration reading, and a fleet that grew. The rule the batch has now converged on is one sentence: when a guard resists a reachability argument, look outside the arithmetic.

A number going up is one property of six. The dashboard called five of six fleets improved when one was — and section 21 is a team that approved a 30% hardware reduction from a figure that was thirty points optimistic in exactly the direction the decision needed.

23.4 — Cloud Architectures on CXL takes all three chapters of this module to the scale they were written for: how hyperscalers actually deploy pooled and composed capacity, and which of these fifteen failure modes their architectures are shaped to avoid.