Future Datacentres on CXL

15.1 through 15.4 built the machinery: a manager, a shape, an inventory, a binding.

This chapter asks the question those four exist to answer — what is it all for, and what does the answer require of the hardware?

Not a forecast. Every claim below is a number a model produced, including the ones that argue against composability.

1. The Engineering Problem — The Vision Is A Set Of Numbers, Not A Direction

"Composable infrastructure" is usually argued as a direction. Stated as engineering, it is six measurements, and three of them are costs.

Stranding is the problem, and it has a definition. Idle capacity and unmet demand at the same instant, separated by a server boundary. Not "low utilisation" — a specific, checkable coincidence. Section 5 builds it.

Pooling recovers some of it, and the amount is a ratio. Section 6 measures utilisation of installed capacity, because that is what was paid for.

The bill is a shared failure domain. A dedicated device failure takes down one server. A pooled one takes down everything bound to it, and that number grows silently. Section 8 bounds it.

Tenants on shared hardware interfere. Without a reservation, one tenant's burst is another's latency, and neither can see why. Section 10 builds the floor.

Pooled memory is further away, and whether that matters depends entirely on how much of the workload touches it — a placement question the hardware can measure and cannot answer. Section 11.

And it stops somewhere. Four ceilings, and which one is hit first is the one that actually limits the design. Section 12.

This chapter against the rest of Module 15, stated precisely. Those chapters own how the fabric works. This one owns what it is worth and what it costs, as arithmetic. If a section here could be moved into 15.1 through 15.4 without loss, it is in the wrong chapter.

2. The One-Sentence Model

Composability trades a boundary for a failure domain — it recovers stranded capacity by removing the server boundary, and pays for it in blast radius, tenant interference, access latency, and overhead — and the decision is whether the first number is bigger than the sum of the other four.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

This is the chapter where a teaching-model boundary matters most, because the subject invites overclaiming.

What the models are: arithmetic over two servers, four hosts, eight devices, two latency tiers, and four ceilings. They make each argument checkable — a claim that pooling recovers capacity becomes an expression that can be wrong, and section 18 breaks each one 107 different ways.

What they are not: a datacentre. There is no workload model, no failure-rate model, no cost model, and no time. A real estate has thousands of servers, correlated demand, and a capital-expenditure argument this chapter does not attempt.

What transfers: the definitions. Stranding as a coincidence rather than a utilisation figure. Blast radius as a bounded count. Net saving as gross minus overhead. Those are structural.

What does not: every magnitude. A 37 percent gross saving is what these parameters produce, not what a rack produces.

5. RTL 1 — What Stranding Actually Is

module stranded_capacity #(parameter int POOLED = 0) (
  input  logic clk, rst_n,
  input  logic       sample,
  input  logic [7:0] demand_a, demand_b,     // what two servers need
  input  logic [7:0] installed_a, installed_b,
  output logic [7:0] served_a, served_b,
  output logic [7:0] stranded, unmet,
  output logic       waste_err,
  output logic [7:0] n_samples, total_stranded, total_unmet, peak_stranded
);
  // Dedicated: each server can only reach its own memory.
  assign ded_a = (demand_a <= installed_a) ? demand_a : installed_a;
  assign ded_b = (demand_b <= installed_b) ? demand_b : installed_b;
  assign pool_fits = (pool_want <= pool_have);
  // Pooled: a server is served from the whole estate, so a boundary no longer
  // decides whether the memory it needs is reachable.
  assign served_a = ((POOLED != 0) && pool_fits) ? demand_a : ded_a;
  // Idle capacity and unmet demand at the same instant is the definition of
  // stranding: the memory exists and cannot be reached by what needs it.
  assign waste_err = sample && (stranded != 8'd0) && (unmet != 8'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_samples <= 8'd0; total_stranded <= 8'd0; total_unmet <= 8'd0;
      peak_stranded <= 8'd0;
    end else if (sample) begin
      n_samples      <= n_samples + 8'd1;
      total_stranded <= total_stranded + stranded;
      total_unmet    <= total_unmet + unmet;
      if (stranded > peak_stranded) peak_stranded <= stranded;
    end
  end
endmodule

And the two definitions the parameter switches between:

  // Dedicated: what one server cannot use, the other cannot reach.
  assign stranded = (POOLED != 0)
                  ? ((pool_want >= pool_have) ? 8'd0 : (pool_have - pool_want))
                  : (idle_a + idle_b);
  assign unmet    = (POOLED != 0)
                  ? ((pool_want <= pool_have) ? 8'd0 : (pool_want - pool_have))
                  : (short_a + short_b);

  stranding: dedicated stranded=24 unmet=16 waste=1 | pooled stranded=8 unmet=0

Server A needs 48 and holds 32. Server B needs 8 and holds 32. 24 units sit idle while 16 units of demand go unserved, at the same instant, in the same rack.

That coincidence is the definition, and it is why waste_err is a conjunction. The testbench drives both halves alone to prove neither is sufficient:

The third and fourth rows are the ones that keep the argument honest. Idle capacity alone is not stranding, and unmet demand alone is not stranding. Pooling fixes exactly one of the four rows, and a case for it built on utilisation figures alone is claiming the other three.

The fourth row is worth stating plainly: when total demand exceeds the estate, the pooled build's unmet demand is also 32. Pooling relocates capacity; it does not create it.

6. RTL 2 — What Pooling Recovers

module pool_efficiency (
  input  logic clk, rst_n,
  input  logic        tick_ev,
  input  logic [7:0]  used, installed,
  input  logic        pooled_mode,
  output logic [15:0] n_ticks, total_used, total_installed,
  output logic [7:0]  util_pct, dedicated_util_pct, pooled_util_pct,
  output logic [15:0] ded_used, ded_inst, pool_used, pool_inst
);
  logic [31:0] w_all, w_ded, w_pool;
  assign w_all  = {16'd0, total_used} * 32'd100;
  assign w_ded  = {16'd0, ded_used}   * 32'd100;
  assign w_pool = {16'd0, pool_used}  * 32'd100;
  assign util_pct = (total_installed == 16'd0) ? 8'd0
                  : (w_all / {16'd0, total_installed});
  assign dedicated_util_pct = (ded_inst == 16'd0) ? 8'd0
                            : (w_ded / {16'd0, ded_inst});
  assign pooled_util_pct = (pool_inst == 16'd0) ? 8'd0
                         : (w_pool / {16'd0, pool_inst});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_ticks <= 16'd0; total_used <= 16'd0; total_installed <= 16'd0;
      ded_used <= 16'd0; ded_inst <= 16'd0;
      pool_used <= 16'd0; pool_inst <= 16'd0;
    end else if (tick_ev) begin
      n_ticks         <= n_ticks + 16'd1;
      total_used      <= total_used + {8'd0, used};
      total_installed <= total_installed + {8'd0, installed};
      // Kept apart, so the two modes can be compared rather than averaged.
      if (pooled_mode) begin
        pool_used <= pool_used + {8'd0, used};
        pool_inst <= pool_inst + {8'd0, installed};
      end else begin
        ded_used <= ded_used + {8'd0, used};
        ded_inst <= ded_inst + {8'd0, installed};
      end
    end
  end
endmodule

  efficiency: dedicated=25% pooled=75% estate=38% over 20 samples

Utilisation is measured against installed capacity, not against used capacity. Measured against what was used it reads 100 percent always, which is a metric that can never say anything — and that is one of the mutations.

Three figures, and the third is the one people forget. The estate-wide 38 percent sits between the two modes and is what a mixed estate actually runs at. A rack part-pooled and part-dedicated does not get the pooled number.

The empty-estate guard returns 0, not 100, asserted before any sample. An estate that has installed nothing is not fully utilised.

7. Waveform — Eight Cycles Of Two Tenants

Transcribed from the printed trace. Both builds see one stimulus stream.

Read nr_run against grant_b. The correct build's longest starvation run is 1; the unreserved build's is 6 and was still climbing when the burst ended. Neither tenant can see the other in either case.

8. RTL 3 — What One Failure Takes Down

module blast_radius #(parameter int NO_SPREAD_LIMIT = 0) (
  input  logic clk, rst_n,
  input  logic       bind_ev, fail_ev,
  input  logic [1:0] host_id,
  output logic [3:0] bound_hosts,
  output logic [2:0] radius,
  output logic       accept,
  output logic       over_spread_err,
  output logic [7:0] n_bound, n_refused, n_failures, worst_radius
);
  logic [3:0] hosts_q;
  logic [2:0] cnt;
  localparam logic [2:0] MAX_SPREAD = 3'd2;
  assign cnt = {2'd0, hosts_q[0]} + {2'd0, hosts_q[1]}
             + {2'd0, hosts_q[2]} + {2'd0, hosts_q[3]};
  // A limit on how many hosts may depend on one device is a limit on what one
  // failure can take out.
  assign accept = bind_ev && !hosts_q[host_id]
                  && ((cnt < MAX_SPREAD) || (NO_SPREAD_LIMIT != 0));
  assign over_spread_err = accept && (cnt >= MAX_SPREAD);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      hosts_q <= 4'd0;
      n_bound <= 8'd0; n_refused <= 8'd0; n_failures <= 8'd0; worst_radius <= 8'd0;
    end else begin
      if (accept) begin
        hosts_q[host_id] <= 1'b1;
        n_bound <= n_bound + 8'd1;
        if ({5'd0, cnt} + 8'd1 > worst_radius) worst_radius <= {5'd0, cnt} + 8'd1;
      end else if (bind_ev) n_refused <= n_refused + 8'd1;
      // A failure takes down everything currently bound to the device.
      if (fail_ev) begin
        n_failures <= n_failures + 8'd1;
        hosts_q <= 4'd0;
      end
    end
  end
endmodule

  blast    : radius=2 worst=2 refused=3 | unlimited build radius=0 worst=4

This is the bill for section 5, and it is the section that gets left out of the pitch.

A dedicated device failure takes down one server, by construction. A pooled device failure takes down everything bound to it — and in the unlimited build that reached four hosts, which is the entire estate. The correct build refused three binds to keep the number at two.

The limit is a design decision that has to be made explicitly, because the alternative is not "no limit" — it is a limit set by however many hosts happened to need capacity. That number only grows, silently, and nothing about the fabric's operation calls attention to it.

Two properties the testbench had to be strengthened to cover: the same host binding twice is not a second dependant (tested while there was still room, so the spread limit is not what refuses it), and a failure actually clears the dependency set.

9. RTL 4 — Tenants On Shared Hardware

module tenant_interference #(parameter int NO_RESERVATION = 0) (
  input  logic clk, rst_n,
  input  logic       req_a, req_b,
  output logic       grant_a, grant_b,
  output logic [3:0] credits_a, credits_b,
  output logic       starved_err,
  output logic [7:0] n_served_a, n_served_b, n_denied_b, max_starve
);
  localparam logic [3:0] RESERVED = 4'd2;
  logic [3:0] ca_q, cb_q;
  logic [7:0] st_q;
  // A reservation is a floor, not a share: tenant B has credits of its own,
  // however hard tenant A pushes. Without one, B simply loses to A.
  assign grant_b = req_b && ((NO_RESERVATION == 0) ? (cb_q != 4'd0) : !req_a);
  assign grant_a = req_a && !(req_b && grant_b);
  assign starved_err = req_b && !grant_b;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      ca_q <= 4'd4; cb_q <= RESERVED; st_q <= 8'd0;
      n_served_a <= 8'd0; n_served_b <= 8'd0;
      n_denied_b <= 8'd0; max_starve <= 8'd0;
    end else begin
      if (grant_a) n_served_a <= n_served_a + 8'd1;
      if (grant_b) begin
        n_served_b <= n_served_b + 8'd1;
        if (cb_q != 4'd0) cb_q <= cb_q - 4'd1;
      end else if (cb_q < RESERVED) cb_q <= cb_q + 4'd1;   // the floor refills
      if (starved_err) begin
        n_denied_b <= n_denied_b + 8'd1;
        st_q <= st_q + 8'd1;
        if (st_q + 8'd1 > max_starve) max_starve <= st_q + 8'd1;
      end else st_q <= 8'd0;
    end
  end
endmodule

  tenants : A served=9 B served=12 B starve run=1 | no-reservation build denied B=20, run=20

Twenty denials in a row against a worst run of one, on the identical stimulus.

The reservation is a floor, not a share. It does not promise B any particular fraction; it promises that B is never denied for longer than its refill interval. That distinction is what makes it implementable — a share requires knowing all the tenants, a floor requires knowing only this one.

starved_err fires on the request, not on the shortage. A tenant that is not asking is not being starved, and a monitor without that term alarms on every idle tenant.

10. RTL 5 — What A Pooled Access Costs

module latency_tier #(parameter int IGNORE_TIER = 0) (
  input  logic clk, rst_n,
  input  logic        access, is_pooled,
  input  logic [7:0]  local_ns, pooled_ns,
  output logic [15:0] access_ns,
  output logic [15:0] n_local, n_pooled, total_ns, max_ns,
  output logic [7:0]  mean_ns, pooled_share_pct, slowdown_pct
);
  logic [31:0] w_share, w_slow;
  logic [15:0] n_all;
  assign n_all   = n_local + n_pooled;
  assign mean_ns = (n_all == 16'd0) ? 8'd0 : (total_ns / n_all);
  assign w_share = {16'd0, n_pooled} * 32'd100;
  assign w_slow  = ({24'd0, mean_ns} - {24'd0, local_ns}) * 32'd100;
  // IGNORE_TIER prices every access as if it were local, which is how a
  // placement decision gets made on a model that cannot be wrong.
  assign access_ns = (IGNORE_TIER != 0) ? {8'd0, local_ns}
                   : (is_pooled ? {8'd0, pooled_ns} : {8'd0, local_ns});
  assign pooled_share_pct = (n_all == 16'd0) ? 8'd0 : (w_share / {16'd0, n_all});
  assign slowdown_pct = (local_ns == 8'd0) ? 8'd0
                      : ((mean_ns <= local_ns) ? 8'd0 : (w_slow / {24'd0, local_ns}));
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_local <= 16'd0; n_pooled <= 16'd0; total_ns <= 16'd0; max_ns <= 16'd0;
    end else if (access) begin
      if (is_pooled) n_pooled <= n_pooled + 16'd1;
      else           n_local  <= n_local + 16'd1;
      total_ns <= total_ns + access_ns;
      if (access_ns > max_ns) max_ns <= access_ns;
    end
  end
endmodule

At 100ns local and 250ns pooled:

  tiers   : 10% pooled, mean=115ns (15% slower), worst=250ns | untiered model reports 100ns

Ten percent of accesses to a 2.5× slower tier costs 15 percent. The same model driven to a mostly-pooled workload costs more than 100 percent. Same hardware, same two tiers; the entire difference is where the workload's accesses land, which is a placement decision the fabric can measure and cannot make.

That is the honest form of the pooled-memory latency argument. It is not "pooled memory is slow" or "the overhead is acceptable" — it is a number that depends on one property of the workload, and the model reports that property directly as pooled_share_pct.

IGNORE_TIER is the model that prices everything as local. It reports 100ns and zero slowdown for every workload, including the one that is twice as slow. A placement decision made on a model that cannot be wrong is a decision made on nothing.

11. RTL 6 — Where It Stops

module scale_limit #(parameter int NO_HEADROOM_CHECK = 0) (
  input  logic clk, rst_n,
  input  logic       add_host, add_device, add_link, add_route,
  output logic [7:0] hosts, devices, links, routes,
  output logic [3:0] limit_mask,   // which ceiling has been reached
  output logic       accept,
  output logic       over_limit_err,
  output logic [7:0] n_added, n_refused, first_limit
);
  localparam logic [7:0] MAX_HOSTS=8'd8, MAX_DEV=8'd16,
                         MAX_LINKS=8'd12, MAX_ROUTES=8'd24;
  logic [7:0] fl_q;
  logic       any_add;
  assign any_add = add_host || add_device || add_link || add_route;
  assign first_limit = fl_q;
  assign limit_mask = {routes >= MAX_ROUTES, links >= MAX_LINKS,
                       devices >= MAX_DEV,   hosts >= MAX_HOSTS};
  assign accept = (add_host   && !limit_mask[0])
                || (add_device && !limit_mask[1])
                || (add_link   && !limit_mask[2])
                || (add_route  && !limit_mask[3])
                || (any_add && (NO_HEADROOM_CHECK != 0));
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      hosts <= 8'd0; devices <= 8'd0; links <= 8'd0; routes <= 8'd0;
      fl_q <= 8'd0; n_added <= 8'd0; n_refused <= 8'd0;
    end else begin
      if (accept) begin
        if (add_host)   hosts   <= hosts + 8'd1;
        if (add_device) devices <= devices + 8'd1;
        if (add_link)   links   <= links + 8'd1;
        if (add_route)  routes  <= routes + 8'd1;
        n_added <= n_added + 8'd1;
      end else if (any_add) n_refused <= n_refused + 8'd1;
      // Which ceiling was hit FIRST is the one that limits the design; it is
      // latched once and never rewritten by the ones that follow.
      if ((fl_q == 8'd0) && (limit_mask != 4'd0)) begin
        if      (limit_mask[0]) fl_q <= 8'd1;
        else if (limit_mask[1]) fl_q <= 8'd2;
        else if (limit_mask[2]) fl_q <= 8'd3;
        else                    fl_q <= 8'd4;
      end
    end
  end
endmodule

  scale   : first ceiling=3 mask=101 refused=5 | no-headroom build grew past it=1

Four ceilings, and the useful output is not the mask — it is which one was hit first. In this fabric, links. The host ceiling was reached later, and by then the fabric had already stopped growing for a different reason.

first_limit is latched and never overwritten, and that is one of the mutations: a "first limit" that later ceilings can rewrite reports whichever constraint is checked first in the priority chain, not whichever one actually stopped the design.

The distinction decides where engineering effort goes. A fabric limited by links and a fabric limited by hosts need completely different work, and after both ceilings are reached they are indistinguishable from the mask alone.

12. RTL 7 — Capacity That Cannot Be Powered

module power_envelope #(parameter int IGNORE_POWER = 0) (
  input  logic clk, rst_n,
  input  logic        activate, deactivate,
  input  logic [7:0]  watts_each,
  output logic [7:0]  active_devices, watts_used, watts_headroom,
  output logic        accept, over_budget_err,
  output logic [7:0]  n_active_peak, n_refused,
  // 8 devices x 32GB is 256, which does not fit in eight bits.
  output logic [15:0] capacity_gb, usable_gb
);
  localparam logic [7:0] BUDGET = 8'd100;
  localparam logic [7:0] GB_EACH = 8'd32;
  localparam logic [7:0] INSTALLED = 8'd8;
  assign next_watts = {8'd0, act_q + 8'd1} * {8'd0, watts_each};
  // Everything installed, and only what the budget can actually run.
  assign capacity_gb = {8'd0, INSTALLED} * {8'd0, GB_EACH};
  assign usable_gb   = {8'd0, act_q}     * {8'd0, GB_EACH};
  assign accept = activate && (act_q < INSTALLED)
                  && ((next_watts <= {8'd0, BUDGET}) || (IGNORE_POWER != 0));
  assign over_budget_err = accept && (next_watts > {8'd0, BUDGET});
  // Saturating: headroom goes to zero, never negative.
  assign watts_headroom = (watts_used >= BUDGET) ? 8'd0 : (BUDGET - watts_used);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      act_q <= 8'd0; n_active_peak <= 8'd0; n_refused <= 8'd0;
    end else begin
      if (accept) begin
        act_q <= act_q + 8'd1;
        if (act_q + 8'd1 > n_active_peak) n_active_peak <= act_q + 8'd1;
      end else if (activate) n_refused <= n_refused + 8'd1;
      else if (deactivate && act_q != 8'd0) act_q <= act_q - 8'd1;
    end
  end
endmodule

  power   : 4 of 8 active, 128GB usable of 256GB installed, refused=1 | ignore-power build over budget=1

Half the installed capacity cannot be turned on. At 25 watts per device and a 100-watt budget, four of eight devices fit — and no capacity report shows this, because 256GB is installed.

There are two ceilings, and the testbench drives both. At 25 watts each, the power budget binds and the installed count does not. At 10 watts each, all eight fit the budget and the installed count becomes the limit. Which one binds is a property of the deployment, not of the design.

capacity_gb is sixteen bits, and that is not decoration. Eight devices at 32GB is 256, which does not fit in eight bits — and 8'd256 is also zero, so an eight-bit check against it passes vacuously. That defect was in the model until the transcript printed "128GB usable of 0GB installed" and the check that should have caught it had been comparing zero against zero.

13. RTL 8 — What Can Actually Be Promised

module composable_sla #(parameter int BEST_EFFORT = 0) (
  input  logic clk, rst_n,
  input  logic       admit_req,
  input  logic [7:0] req_gb, req_bw,
  output logic [7:0] committed_gb, committed_bw,
  output logic       admit, oversubscribed_err,
  output logic [7:0] n_admitted, n_refused, peak_gb, peak_bw
);
  localparam logic [7:0] POOL_GB = 8'd128;
  localparam logic [7:0] POOL_BW = 8'd64;
  // Both dimensions, because a tenant that fits by capacity and not by
  // bandwidth is a tenant whose guarantee is already broken.
  assign fits = (next_gb <= {1'b0, POOL_GB}) && (next_bw <= {1'b0, POOL_BW});
  assign admit = admit_req && (fits || (BEST_EFFORT != 0));
  assign oversubscribed_err = admit && !fits;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      committed_gb <= 8'd0; committed_bw <= 8'd0;
      n_admitted <= 8'd0; n_refused <= 8'd0; peak_gb <= 8'd0; peak_bw <= 8'd0;
    end else if (admit_req) begin
      if (admit) begin
        committed_gb <= next_gb[7:0];
        committed_bw <= next_bw[7:0];
        n_admitted <= n_admitted + 8'd1;
        if (next_gb[7:0] > peak_gb) peak_gb <= next_gb[7:0];
        if (next_bw[7:0] > peak_bw) peak_bw <= next_bw[7:0];
      end else n_refused <= n_refused + 8'd1;
    end
  end
endmodule

  sla     : admitted=2 refused=2, 96GB and 64 BW committed | best-effort build oversubscribed=1

A guarantee is only a guarantee if the thing that would break it is refused. Two tenants were refused for two different reasons — one fit by capacity and not bandwidth, the other by bandwidth and not capacity — and each half of the condition is driven alone, because a mutation dropping either leaves the other passing.

The boundary is driven exactly: a tenant filling the remaining bandwidth precisely is admitted. <=, not <. One off refuses the tenant that exactly fits, which is the tenant an admission controller exists to accept.

BEST_EFFORT admits everything and commits 80 units of a 64-unit bandwidth pool. Nothing fails at that moment; the promise is broken at the instant it is made, and it presents later as tenants missing targets they were told they would meet.

14. RTL 9 — What A Rack Actually Recovers

module rack_recovery (
  input  logic        clk, rst_n,
  input  logic        tick_ev,
  input  logic [7:0]  ded_installed, pooled_installed, demand_served,
  input  logic [7:0]  transition_overhead, fabric_overhead,
  output logic [15:0] t_ded, t_pool, t_served, t_over,
  output logic [7:0]  gross_saving_pct, net_saving_pct, overhead_pct
);
  logic [31:0] w_gross, w_net, w_over;
  logic [15:0] saved, net_saved;
  assign saved     = (t_pool >= t_ded) ? 16'd0 : (t_ded - t_pool);
  // The overhead has to come out of the saving, or the saving is a headline.
  assign net_saved = (saved <= t_over) ? 16'd0 : (saved - t_over);
  assign gross_saving_pct = (t_ded == 16'd0) ? 8'd0 : (w_gross / {16'd0, t_ded});
  assign net_saving_pct   = (t_ded == 16'd0) ? 8'd0 : (w_net   / {16'd0, t_ded});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      t_ded <= 16'd0; t_pool <= 16'd0; t_served <= 16'd0; t_over <= 16'd0;
    end else if (tick_ev) begin
      t_ded    <= t_ded + {8'd0, ded_installed};
      t_pool   <= t_pool + {8'd0, pooled_installed};
      t_served <= t_served + {8'd0, demand_served};
      // Both halves of the overhead, or the saving is a headline.
      t_over   <= t_over + {8'd0, transition_overhead} + {8'd0, fabric_overhead};
    end
  end
endmodule

  recovery: gross=37% overhead=7% net=29% of the dedicated baseline

37 percent gross, 29 percent net. The gross number is the one in the pitch; the net one is the number that decides whether to build it, and the eight points between them are 15.4's transition overhead plus the fabric's own cost.

Both subtractions are saturating, and the testbench drives the case that needs it: a rack where pooling installed more than the dedicated baseline reports a saving of zero, not a large positive number from an underflowed subtraction. A cost model that can report a negative saving as a positive one is a cost model that will, on exactly the deployment where it matters.

15. RTL 10 — The Composable Datacentre Assembled

module dc_top #(parameter int ADMIT_ANYWAY = 0) (
  input  logic clk, rst_n,
  input  logic       request,
  input  logic       pool_has_room, power_has_room, radius_ok, scale_ok,
  output logic       admit,
  output logic [3:0] refused_by,
  output logic       broken_promise_err,
  output logic [7:0] n_requests, n_admitted, n_refused, n_broken
);
  assign refused_by = {~scale_ok, ~radius_ok, ~power_has_room, ~pool_has_room};
  assign admit = request && ((refused_by == 4'd0) || (ADMIT_ANYWAY != 0));
  // Admitted against a constraint that cannot carry it: a promise made and
  // already broken at the moment it was made.
  assign broken_promise_err = admit && (refused_by != 4'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_admitted <= 8'd0; n_refused <= 8'd0; n_broken <= 8'd0;
    end else if (request) begin
      n_requests <= n_requests + 8'd1;
      if (admit) n_admitted <= n_admitted + 8'd1;
      else       n_refused  <= n_refused + 8'd1;
      if (broken_promise_err) n_broken <= n_broken + 8'd1;
    end
  end
endmodule

  assembled: 1 of 5 admitted, mask for scale=8 | admit-anyway build broke 4 promises

Five requests. One meets every constraint; four each fail exactly one, and the mask names which. Each bit position is asserted individually — a mask with two fields transposed passes any check that only asks whether it is non-zero.

ADMIT_ANYWAY admits all five and breaks four promises doing it. Every one of them succeeds at the moment it is made.

The constraint with no natural advocate is blast radius. Pool capacity is visible to whoever is allocating. Power is visible to whoever is racking. The fabric ceiling is visible to whoever adds a device. Nothing about normal operation makes the number of hosts depending on one device visible to anybody — which is why section 8 exists as a counter rather than a review item.

16. Quantitative Reasoning

Every number is printed by a model, not asserted here — including the ones that argue against composability.

Four worth a sentence.

24 idle and 16 short, simultaneously. That is the whole case for composability, and it is a coincidence rather than a utilisation figure. Three of the four demand patterns tested do not exhibit it.

2 hosts against 4. The bill. And it grows silently: nothing in normal operation makes it visible.

15% against over 100%. Same two tiers, same hardware. The entire difference is where the workload's accesses land.

37% gross, 29% net. Eight points of difference, and every serious version of this argument is about those eight points rather than the 37.

17. Assertions

Every property is an immediate check written as cond !== 1'b1, sampled after a settle.

18. Mutation Testing

107 mutations, one at a time, each required to make the baseline print RESULT: FAIL.

107 of 107 were killed.

The first run killed 93 and left 14 survivors:

Two of the equivalences are worth recording because they were not obvious. A ternary whose false branch equals its true branch at the boundary is unbreakable there: (demand <= installed) ? demand : installed and (demand < installed) ? demand : installed are identical when they are equal. And once the stimulus was changed so links became the first ceiling, a mutation on the host branch of the priority chain became unreachable — fixing one gap closed the path to another mutation, and it had to be re-aimed rather than left.

The second row is the pattern of this chapter. Four mutations survived because every measurement was taken once: one pooling comparison, one utilisation phase, one saving calculation. Each needed a second phase with genuinely different numbers — a pooled estate larger than the dedicated one, a per-device power draw low enough that the installed count binds instead of the budget.

A representative sample:

19. Verification Strategy

Parameterised twin builds. POOLED, NO_SPREAD_LIMIT, NO_RESERVATION, IGNORE_TIER, NO_HEADROOM_CHECK, IGNORE_POWER, BEST_EFFORT, ADMIT_ANYWAY. Every comparison in section 16 is one source under one parameter, driven by one stimulus stream — so "pooling recovers capacity" is a measured difference rather than a claim.

Independent oracles. Stranding, mean access cost, and two-ceiling admission are each checked against a plain-integer function with no reference to the design's expression.

At least two cases per ratio. Four mutations survived a single measurement. A utilisation, a saving, or a share computed once cannot distinguish the right denominator from several wrong ones.

The cases that argue the other way. Idle capacity with nobody short. Unmet demand with nothing idle. A pooled estate larger than the dedicated one. A demand pattern that pooling cannot help. A chapter that only drives the cases supporting its thesis is a chapter that has proved nothing.

Saturating arithmetic, driven to saturation. Both subtractions in rack_recovery and the power headroom are guarded, and each guard is reached by a deliberate case rather than left as a defensive habit.

Print what you assert on. The width bug in section 12 passed its own check — 8'd256 truncates to 0 exactly as the design did — and was caught only because the transcript said "of 0GB installed" where a human could read it.

Delta discipline. Every combinational sample follows a settle, and registered outputs like first_limit are sampled a cycle after the condition that sets them.

20. Synthesis and Implementation Reality

Almost none of this chapter is hardware, and that is the point of the section.

These are accounting models. stranded_capacity, pool_efficiency and rack_recovery are arithmetic a capacity planner runs, not logic anything synthesises. They appear as RTL because expressing them as executable, mutation-tested code turns "pooling recovers stranded capacity" into a claim that can be checked and can be wrong — and section 18 shows it wrong 107 ways.

Three of the ten are genuinely hardware-adjacent. tenant_interference is a credit scheme, and a real one lives in a device's request scheduler. power_envelope is a power-management controller. scale_limit is a set of table-size comparisons in a switch. All three are small.

blast_radius is a policy check in the fabric manager, and it is the one worth building even though nothing forces it. A counter of hosts bound per device, with a configurable ceiling, is a few registers — and it is the only mechanism that makes a number visible which nothing else in the system reports.

The reservation is the expensive one. A credit floor per tenant means per-tenant state in a request scheduler, and the cost scales with tenant count rather than with device count. That is why the model expresses it as a floor rather than a share: a floor needs state for the tenant being protected, and a share needs state for all of them.

Every percentage here is firmware. Combinational division appears because the chapter is about what to measure.

The widths are the model's, and one of them was wrong. Section 12's capacity_gb needed sixteen bits and had eight. In a real capacity model the equivalent quantity is petabytes and the same mistake is available in 32 bits.

21. Silicon Observability

Three to alarm on.

worst_radius above its limit means the failure domain grew past what was designed for. It is the one number here that no operational signal produces — a fabric with an oversized blast radius behaves identically to one without, right up until the failure.

max_starve growing across runs means a tenant is being denied for longer and longer. It is the early form of a tenant that will miss its targets, visible before any latency metric moves.

broken_promise_err non-zero at all means the admission controller admitted something the system cannot carry. Every such admission succeeds at the moment it is made.

22. Debug Lab

22.1 Utilisation is low and adding memory does not help

Symptom. Estate-wide memory utilisation is 30 percent. Workloads are still short of memory. Adding memory to the short servers helps briefly and the figure gets worse.

The reading. waste_err — not the utilisation figure.

The diagnosis. If waste_err is firing, this is stranding: capacity is idle and demand is unmet at the same instant, and the two are separated by a boundary. Adding memory to the short servers raises installed capacity without touching the coincidence, so utilisation falls and the shortage returns.

The case where it is not. If waste_err is clear, one of two things is true, and section 5's table distinguishes them. Idle with nothing short is over-provisioning — the fix is buying less, and pooling will not raise utilisation. Short with nothing idle is under-provisioning — the fix is buying more, and pooling cannot create capacity.

Three different problems, one utilisation figure, and one conjunction that separates them.

22.2 One device failed and half the estate went down

Symptom. A single pooled device failed. Far more hosts were affected than anyone expected.

The reading. worst_radius, which should have been known before the failure.

The diagnosis. The blast radius grew past what the design assumed, one bind at a time, and nothing reported it. Every individual bind was legitimate.

Why this one is different from every other entry in this lab. The information was available continuously and cost a counter to expose. There is no operational symptom of an oversized blast radius — the fabric behaves identically — so it is knowable in advance and invisible in practice, which is the worst combination.

22.3 A tenant misses its targets and the device is not busy

Symptom. One tenant's latency is poor. The shared device reports moderate utilisation. The tenant's own metrics show nothing unusual except the latency.

The reading. n_denied_b and max_starve for that tenant, and whether another tenant was bursting.

The diagnosis. Interference. The device is not busy on average; it is fully occupied during another tenant's bursts, and the affected tenant is denied for their entire duration. Figure 2 is exactly this — twenty consecutive denials on a device that looks half-idle over any longer window.

Why averages hide it. The starvation run is the metric, not the utilisation. A device at 50 percent utilisation can be denying one tenant 100 percent of the time it asks.

22.4 Pooled memory made things slower than the model predicted

Symptom. A workload moved to pooled memory. The slowdown is far worse than the projection.

The reading. pooled_share_pct measured, against whatever the projection assumed.

The diagnosis, in two sub-cases.

If the measured share is much higher than assumed, the projection was right about the hardware and wrong about the workload. Section 10's two runs differ by a factor of seven in slowdown on identical hardware; the entire difference is where the accesses land.

If the measured share matches the projection and the slowdown still exceeds it, check whether the projection used a tiered model at all. IGNORE_TIER reports zero slowdown for every workload, and a projection built on it is not wrong about this workload — it is incapable of being wrong about any.

23. Design Review

1. Is stranding measured as a coincidence, or inferred from a utilisation figure? Three of the four demand patterns in section 5 have low utilisation and are not stranding.

2. What is the net saving, and what overhead was subtracted from the gross? If the answer is the gross number, the overhead has not been measured.

3. How many hosts may depend on one device, and who enforces it? "As many as need capacity" is a limit, just not a chosen one.

4. What happens to every host bound to a pooled device when it fails? And is that number knowable before the failure?

5. Does each tenant have a floor, or does the fabric rely on tenants not bursting simultaneously? A floor needs state for the tenant being protected; a share needs it for all of them.

6. What fraction of the workload's accesses cross the fabric, and was it measured or assumed? It is the only input that matters, and it varies by more than the hardware does.

7. Does the latency model distinguish tiers? A model that cannot report a slowdown will not.

8. Which ceiling does the fabric hit first — hosts, devices, links, or routes? After two are reached they are indistinguishable, and they need different work.

9. How much installed capacity can actually be powered simultaneously? And which binds first, the budget or the installed count?

10. Is a request admitted only if every constraint can carry it, and does a refusal say which one refused? A promise made against a constraint that cannot carry it is broken at the instant it is made.

24. How This Appears In Real Engineering

The case is made on utilisation and the problem is stranding. Utilisation is easy to measure and easy to quote; the coincidence in section 5 requires instrumenting both halves at the same instant. Teams that measure only utilisation adopt pooling for workloads it cannot help, and the figure gets worse.

The gross saving is what gets funded. The overhead in section 14 — transition time from 15.4, fabric cost from 15.2 — arrives later, from different teams, after the decision.

Blast radius has no owner. Capacity has an owner. Power has an owner. The fabric's ceilings have an owner. The number of hosts depending on one device is nobody's metric, grows monotonically, and is only ever discovered by a failure.

Tenant interference is diagnosed as a tenant problem. It presents inside one tenant's metrics, on a device that looks unbusy, and the natural first move is to investigate the tenant that is suffering rather than the one that is bursting.

Latency projections are made with untiered models because that is what exists. The projection is then not wrong about the workload — it is structurally incapable of being wrong about any workload, which is worse than being wrong.

Power is discovered at rack time. The capacity was bought, installed, and counted. Half of it cannot be turned on, and no capacity report has a column for that.

25. Common Misconceptions

"Low utilisation means capacity is stranded." No. Stranding is idle capacity and unmet demand at the same instant. Section 5's table has three low-utilisation rows that are not stranding, and pooling helps exactly one of them.

"Pooling increases capacity." It relocates it. When total demand exceeds the estate, the pooled model's unmet demand is identical to the dedicated model's — 32 units in both.

"Pooling is a strict improvement." It trades a boundary for a failure domain. Section 8's unlimited build put the entire estate behind one device, and nothing about normal operation would have shown that.

"The blast radius is bounded because we would notice." Nothing reports it. Every bind that widens it is individually legitimate, and a fabric with an oversized failure domain behaves identically to one without.

"Tenants on a shared device share it fairly." Only if something makes them. Section 9's unreserved build denied one tenant every single cycle of the other's burst, on a device that was not saturated over any longer window.

"Pooled memory adds a fixed overhead." It adds a per-access cost that only matters in proportion to how many accesses cross the fabric. Ten percent pooled cost 15 percent; mostly-pooled cost over 100 percent. Same hardware.

"The capacity is installed, so it is available." Half of section 12's installed capacity could not be powered. Both ceilings are real, and which one binds is a property of the deployment.

"Best effort is a reasonable fallback." It admitted 80 units of a 64-unit bandwidth pool. Nothing failed at that moment; the promise was already broken, and it presents later as tenants missing targets they were told they would meet.

26. Interview Reasoning

Q1. What is the engineering case for composable infrastructure, stated as a measurement? Idle capacity and unmet demand at the same instant, separated by a server boundary. It is a coincidence, not a utilisation figure, and it is the only thing pooling fixes.

Q2. Why is that a conjunction rather than a low-utilisation threshold? Because idle capacity with nobody short is over-provisioning, and demand short with nothing idle is under-provisioning. Both look like poor utilisation and neither is helped by pooling.

Q3. A rack runs at 30 percent utilisation and workloads are short of memory. Is that stranding? Only if the two are true at the same instant. Check the conjunction. If it is, pooling helps; if the idleness and the shortage are at different times, it does not.

Q4. Does pooling create capacity? No. When total demand exceeds the estate, the pooled model's unmet demand is exactly the dedicated model's. It removes a boundary; it does not add memory.

Q5. Why measure utilisation against installed capacity rather than used? Against used it reads 100 percent always. Installed is what was paid for, and it is the denominator that can say something.

Q6. What does composability cost? A shared failure domain, tenant interference, per-access latency in proportion to the pooled share, and transition overhead. The decision is whether the recovered capacity exceeds the sum.

Q7. What is blast radius and why does it need a limit? The number of hosts a single device failure takes down. Without a limit it is set by however many hosts happened to need capacity, it grows monotonically, and nothing reports it.

Q8. Why is it the constraint that gets forgotten? It has no natural advocate. Capacity, power and fabric ceilings are each visible to somebody during normal operation. The number of hosts depending on one device is visible to nobody until it fails.

Q9. Two tenants on one device. What goes wrong by default? One tenant's burst is the other's starvation. Section 9's unreserved build denied tenant B on every cycle of a twenty-cycle burst.

Q10. Why a floor rather than a fair share? A floor promises that a tenant is never denied for longer than its refill interval, and it needs state only for the tenant being protected. A share needs state for every tenant, and it scales with tenant count.

Q11. Why does the starvation monitor fire on the request rather than the shortage? A tenant that is not asking is not being starved. Without that term the monitor alarms on every idle tenant, constantly.

Q12. A tenant misses its latency targets on a device at 50 percent utilisation. Explain. The device is not busy on average and is fully occupied during another tenant's bursts. The starvation run is the metric; the average hides it completely.

Q13. How much does pooled memory cost in latency? It depends entirely on the pooled share of accesses. Ten percent to a 2.5× tier costs 15 percent; a mostly-pooled workload on identical hardware costs over 100 percent.

Q14. What is wrong with a latency model that does not distinguish tiers? It reports the same number for both of those workloads. It is not wrong about a particular workload — it is incapable of being wrong about any, which makes every projection built on it meaningless.

Q15. What determines whether a workload should use pooled memory? Its access distribution, which the fabric can measure and cannot decide. The hardware's job is to report pooled_share_pct accurately; the placement decision is above it.

Q16. Four ceilings are reached. Which one limits the design? Whichever was reached first. After two are hit they are indistinguishable from the mask, and a fabric limited by links needs different work from one limited by hosts.

Q17. Why must the first-limit record never be overwritten? Because a later ceiling rewrites it to whichever constraint the priority chain checks first, which is a property of the code rather than of the fabric.

Q18. 256GB is installed and 128GB is usable. What happened? The power budget. At 25 watts per device and 100 watts available, four of eight run. No capacity report has a column for this, because 256GB genuinely is installed.

Q19. Which binds first, the power budget or the installed count? It depends on the per-device draw, and both are real. At 25 watts the budget binds; at 10 watts all eight fit and the installed count binds instead.

Q20. What is an admission controller actually promising? That the resources it committed exist. It is only a promise if the request that would break it is refused — and it must check every dimension, because a tenant that fits by capacity and not bandwidth has a guarantee that is already broken.

Q21. Why check both capacity and bandwidth? Because they run out independently. Two tenants in section 13 were refused for opposite reasons, and a check on either dimension alone admits one of them.

Q22. Best-effort admission committed 80 units of a 64-unit pool. When does that fail? Later. It succeeds at the moment it is made, which is what makes it dangerous — the failure appears as tenants missing targets, far from the admission decision that caused it.

Q23. Gross saving 37 percent, net 29. Which do you present? The net. The eight points between them are transition overhead and fabric cost, and every serious version of this argument is about those eight points.

Q24. Why must the saving calculation saturate? Because a pooled estate can be larger than the dedicated baseline. Without saturation the subtraction underflows and reports a huge positive saving on precisely the deployment where there is none.

Q25. A check compared an 8-bit signal against 8'd256 and passed while the signal was zero. Explain. 8'd256 truncates to zero as well. Neither the design nor the check was right, and they agreed. Only the printed transcript caught it.

Q26. What does that tell you about writing checks? Print the numbers you assert on, and read them. A check and a design can share the same mistake, and the assertion cannot detect what the literal also gets wrong.

Q27. Why did four mutations survive a testbench that measured everything? Because it measured everything once. A ratio computed from a single sample cannot distinguish the right denominator from several wrong ones; they all agree when the numbers coincide.

Q28. Why does this chapter drive cases that argue against pooling? Because a chapter that only drives cases supporting its thesis has proved nothing. Idle capacity with nobody short, demand exceeding the whole estate, and a pooled estate larger than the dedicated one are all in the stimulus.

Q29. If you could add one counter to a composable system, which? worst_radius. Every other cost here has some operational symptom. An oversized blast radius has none — the fabric behaves identically until the failure, and the number was knowable the whole time.

Q30. Summarise the trade in one sentence. Composability exchanges a server boundary for a failure domain: it recovers the capacity the boundary stranded, and it pays in blast radius, interference, access latency and overhead — and the only honest form of the argument is the net number.

27. Exercises

1. Extend stranded_capacity to four servers and find a demand pattern where pooling recovers nothing despite all four being under-utilised. Explain what that pattern has in common across the four.

2. Add correlation to the demand: make both servers peak at the same time. Show what happens to the pooled estate's unmet demand, and state the assumption pooling actually depends on.

3. Give blast_radius a per-device limit derived from the fabric's redundancy in 15.2. Justify the number you choose from that chapter's SPOF sweep.

4. Change tenant_interference from a floor to a weighted share. Measure the extra state per tenant, and identify the workload for which the floor is better.

5. Add a third latency tier to latency_tier. Show that the slowdown is a weighted sum, and find the pooled share at which a two-tier projection becomes misleading.

6. Parameterise scale_limit so the ceilings can be reached in any order. Confirm first_limit records the right one for every ordering.

7. Combine power_envelope with composable_sla: refuse a tenant whose capacity fits but whose devices cannot all be powered simultaneously. Show which existing check this is not.

8. Take the 107-mutation suite and remove every second measurement phase. Confirm the four single-sample survivors return, and list every other ratio in Module 15 with the same weakness.

28. Summary

Composability trades a server boundary for a failure domain.

• What it recovers: 24 units idle while 16 went unserved, in the same rack, at the same instant. That coincidence is the definition, and three of the four demand patterns tested do not exhibit it.
• What it does not: create capacity. When demand exceeded the estate, pooled and dedicated left the same 32 units unserved.
• What it costs in failure: a blast radius of 2 held by refusing three binds, against 4 — the whole estate — when nothing held it.
• What it costs in interference: a starvation run of 1 against 20, on identical stimulus, on a device that was not saturated.
• What it costs in latency: 15 percent from a 10 percent pooled share, and over 100 percent from a mostly-pooled one. Same hardware; the difference is entirely the workload.
• What it costs in overhead: 37 percent gross becomes 29 percent net, and every serious version of the argument is about those eight points.

And two ceilings that are not about the idea at all: half the installed capacity could not be powered, and the fabric stopped growing at its link count rather than its host count — which is a different engineering problem from the one the mask alone would suggest.

107 mutations, 107 killed. Four survived a testbench that measured everything once — a ratio from a single sample cannot tell the right denominator from a wrong one. And one check passed while both it and the design were wrong, because 8'd256 truncates to zero exactly as the design did. Print the numbers you assert on.

Module 15 is complete: a manager that owns the fabric, a shape worth having, an inventory that can be trusted, a binding that is safe to make, and — here — the arithmetic that says whether any of it is worth doing.