CXL · Module 29
Composable Infrastructure in Practice
Composing whole servers, whole devices and fractions of a device are three different systems. This chapter builds the granularity, the bind sequence, fragmentation, the binding lifecycle and its leak, the control-plane dependency, where stranding moves to, which workloads gain and what flexibility costs standing.
29.4 closed on four things wearing one word. This chapter closes Module 29 on a word that wears a whole architecture — and it is the one where the gap between the marketing sentence and the engineering is widest.
The infrastructure is composable. Composing whole servers, whole devices and fractions of a device are three different systems with three different allocators, three different failure models and three different control planes. The sentence is satisfied by all three, and by a rack where somebody can move a cable.
1. The Engineering Problem — Composition Is An Allocator, And Allocators Have Physics
"Composable" names no granularity, no fabric and no lifecycle. Three axes with one stated leaves two unstated, on an inventory where only a quarter of the resources can actually be bound. Section 5.
The allocation unit rounds every request up. A fifty-gigabyte request against a sixteen-gigabyte unit takes four units and serves sixty-four: fourteen gigabytes rounded away on every request, five hundred and sixty across forty of them. Section 6.
Composition is a sequence, not an assignment. A bind walks discover, reserve, program, verify and ready — five steps, fifteen cycles at three cycles a step — and a request arriving while one is in flight must be held rather than dropped. Section 7.
Free capacity is not allocatable capacity. A sixteen-slot pool with eight slots free and none of them adjacent cannot satisfy a four-slot request. Section 8.
A binding outlives the host that asked for it. When a host dies the resource does not become free — it becomes orphaned, still recorded as belonging to a host that no longer exists, and it stays there until something reclaims it. Section 9.
And the control plane is a new dependency. Thirty-six healthy resources with the orchestrator down are thirty-six resources that are unreachable rather than unavailable — a different incident with a different fix. Section 10.
Why this closes the module. 29.1 asked which tier, 29.2 what the device is, 29.3 whether the fleet arithmetic works and 29.4 which of four sharing modes. Composability is where all four questions arrive at once, because a composable estate is one that answers them differently every time somebody presses a button.
2. The One-Sentence Model
A composable-infrastructure case study is sound when the composability is named, when the granularity is stated, when the bind latency is stated, when fragmentation is measured rather than assumed away, when reclamation is stated, and when the control-plane dependency is stated — and "it is composable" is bit 0.
3. What This Chapter Owns
| Ground | Owner |
|---|---|
| Which memory tier a cluster means | 29.1 |
| The expansion card as a device | 29.2 |
| Whether a fleet deployment pays | 29.3 |
| Which of four sharing modes | 29.4 |
| How a shared pool is sized | 27.5 |
| Resources bound and unbound on demand | this chapter |
Some vocabulary, because the three granularities are the chapter and one word covers all of them.
Composing at server granularity means assembling a machine from whole chassis-level parts. It is the oldest form, it needs no new fabric, and its allocation unit is enormous.
Composing at device granularity means binding whole devices — a card, a module — to a host across a fabric. This is where CXL appears, and the allocation unit is whatever a device is.
Composing at sub-device granularity means binding a fraction of a device, which needs the device itself to support partitioning. It is the granularity that makes fragmentation interesting, because a partial allocation leaves a remainder.
And the control plane is whatever decides. An orchestrator that maps requests onto resources, holds the bindings, and is a dependency of every host that draws on it — which is section 10.
4. Teaching-Model Boundary And Source Discipline
This is a case-study chapter and the boundary matters most here, because "composable infrastructure" is a term with real products behind it.
Every model in this chapter is a teaching model. None of them is production orchestration logic, a real allocator, or an implementation of any vendor's control plane. They are small enough to read in one screen and are built to expose exactly one invariant each.
Nothing here is attributed to a named organisation or product. Sixteen slots, a three-cycle step, a two-hundred-and-sixty-gigabyte unreachable pool — every figure is an illustrative teaching number chosen to make one relationship visible.
Where this chapter touches real CXL, it is limited to what is publicly established: that the specification defines memory devices reachable over a link, that such devices can be bound to a host, and that multi-host topologies exist. This chapter does not state any opcode, register field, timing guarantee, allocation granularity or negotiation sequence from the specification, because those are normative details and the models here are not derived from them.
| Claim class | How it is marked |
|---|---|
| Publicly established architecture | stated plainly, no numbers attached |
| Teaching abstraction | called a teaching model in the RTL header |
| Illustrative parameter | any concrete figure in a model or table |
| Simulator-derived result | quoted from a testbench run and asserted |
| Derived arithmetic | shown with its inputs so it can be rechecked |
Each model is built twice from one source. A parameter selects between the measured build, which counts what the deployment rests on, and a weak build that treats the marketing sentence as the finding. Every section's headline number is the gap between them.
5. RTL 1 — "Composable" Names No Granularity, Fabric Or Lifecycle
Start with the sentence, because the three axes below it are what a reader actually needs.
The three axes are granularity, fabric and lifecycle. What unit is bound; what carries the binding; and what happens to it when the host goes away. A description that gives one of the three has given a third of a system, and the two it omitted are the ones that decide whether the estate works.
// RTL 1 - "the infrastructure is composable" names no granularity, no fabric
// and no lifecycle. Composing whole servers, whole devices, and fractions of a
// device are three different systems; the sentence is satisfied by all three.
//
// TEACHING MODEL - not production orchestration logic. Combinational except for
// the two evaluation counters, which exist so a testbench can prove the model
// was exercised rather than merely wired.
module composable_claim #(parameter int ANY_BINDING_IS_COMPOSABLE = 0) (
input logic clk, rst_n,
input logic evaluate,
input logic [15:0] axes_present, axes_stated, resources_total, resources_bindable,
output logic [15:0] axes_unstated, stated_ok, bindable_pct, specified_pct,
output logic claim_specified,
output logic [7:0] n_evals, n_vague,
output logic claim_err
);
logic [31:0] b_q, s_q;
logic [15:0] true_unstated;
logic truly_vague;
assign stated_ok = (axes_stated > axes_present) ? axes_present : axes_stated;
assign true_unstated = axes_present - stated_ok;
assign axes_unstated = (ANY_BINDING_IS_COMPOSABLE != 0) ? 16'd0 : true_unstated;
// How much of the inventory can actually be bound to an arbitrary host.
assign b_q = (resources_total == 16'd0) ? 32'd0
: (({16'd0, resources_bindable} * 32'd100) / {16'd0, resources_total});
assign bindable_pct = (b_q > 32'd100) ? 16'd100 : b_q[15:0];
assign s_q = (axes_present == 16'd0) ? 32'd100
: (({16'd0, stated_ok} * 32'd100) / {16'd0, axes_present});
assign specified_pct = (ANY_BINDING_IS_COMPOSABLE != 0) ? 16'd100 : s_q[15:0];
assign claim_specified = (axes_unstated == 16'd0) && (axes_present != 16'd0);
assign truly_vague = (true_unstated != 16'd0);
assign claim_err = evaluate && truly_vague && claim_specified;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
n_evals <= 8'd0; n_vague <= 8'd0;
end else if (evaluate) begin
n_evals <= n_evals + 8'd1;
if (truly_vague) n_vague <= n_vague + 8'd1;
end
end
endmoduleThree axes with one stated, on an inventory where ten of forty resources are bindable, leaves two axes unstated, a quarter of the inventory bindable, and a third of the claim specified.
| Fact | Value |
|---|---|
| Axes present | 3 |
| Axes stated | 1 |
| Resources | 40 |
| Bindable | 10 |
| Unstated | 2 |
| Specified | 33% |
Figure 1 — the word that three architectures answer to. The upper path is true under all three granularities, which is exactly why it identifies none of them. The lower path returns two numbers, and the second is the one that ends arguments: a quarter of this inventory can actually be bound to an arbitrary host, so three quarters of what is being called composable is not.
The second case is the description done properly: every axis stated and everything bindable, both builds agreeing and neither alarming.
The fourth case is the clamp on the inventory side, and it is a sanity check rather than a finding: more bindable resources claimed than exist saturates at a hundred percent.
The degenerate case bounds it — nothing enumerated reports nothing unstated and nothing specified, which is silence rather than a bad description.
6. RTL 2 — The Allocation Unit Rounds Every Request Up
The second thing, and the first that is a number rather than a distinction.
A pool is carved into units, and a request that is not a whole number of units takes the next one up. That rounding is capacity that is allocated, billed to nobody, and unusable by anybody else. It is the tax the granularity charges, and it is paid on every request rather than once.
The invariant the model enforces: served_gb is always a whole number of units and never less than the request. A model that could serve less than was asked would be describing a different failure.
// RTL 2 - the allocation unit decides everything downstream. A pool carved in
// units larger than a request rounds every request up, and the rounding is
// capacity nobody can use and nobody is billed for.
//
// TEACHING MODEL. The invariant: served_gb is always a whole number of units
// and is never less than the request.
module compose_granularity #(parameter int GRANULARITY_IS_FREE = 0) (
input logic clk, rst_n,
input logic evaluate,
input logic [15:0] request_gb, unit_gb, requests, pool_gb,
output logic [15:0] units_used, served_gb, rounded_waste, waste_total,
output logic exact_fit,
output logic [7:0] n_evals, n_rounded,
output logic gran_err
);
logic [31:0] u_q, s_q, t_q;
logic [15:0] true_units, true_served, true_waste;
logic truly_rounded;
// Ceiling division: a request of 1 GB against a 16 GB unit still takes a unit.
assign u_q = (unit_gb == 16'd0) ? 32'd0
: (({16'd0, request_gb} + {16'd0, unit_gb} - 32'd1) / {16'd0, unit_gb});
assign true_units = (u_q > 32'd9999) ? 16'd9999 : u_q[15:0];
assign units_used = true_units;
assign s_q = {16'd0, true_units} * {16'd0, unit_gb};
assign true_served = (s_q > 32'd9999) ? 16'd9999 : s_q[15:0];
assign served_gb = true_served;
assign true_waste = (true_served > request_gb) ? (true_served - request_gb) : 16'd0;
assign rounded_waste = (GRANULARITY_IS_FREE != 0) ? 16'd0 : true_waste;
assign t_q = {16'd0, true_waste} * {16'd0, requests};
assign waste_total = (GRANULARITY_IS_FREE != 0) ? 16'd0
: ((t_q > 32'd9999) ? 16'd9999 : t_q[15:0]);
assign exact_fit = (rounded_waste == 16'd0) && (request_gb != 16'd0);
assign truly_rounded = (true_waste != 16'd0);
assign gran_err = evaluate && truly_rounded && exact_fit;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
n_evals <= 8'd0; n_rounded <= 8'd0;
end else if (evaluate) begin
n_evals <= n_evals + 8'd1;
if (truly_rounded) n_rounded <= n_rounded + 8'd1;
end
end
endmoduleA fifty-gigabyte request against a sixteen-gigabyte unit, forty times, takes four units, serves sixty-four, rounds fourteen away per request and five hundred and sixty in total.
| Fact | Value |
|---|---|
| Request | 50 GB |
| Unit | 16 GB |
| Units taken | 4 |
| Served | 64 GB |
| Rounded away | 14 GB |
| Across 40 | 560 GB |
The third case is the worst case for this unit size and it is the one that decides a granularity. A one-gigabyte request against a sixteen-gigabyte unit serves sixteen and rounds fifteen away — a fifteen-sixteenths tax, which is what happens when a unit is chosen for the largest expected request and the smallest one arrives.
The second case is the configuration to aim for: a request that is a whole number of units rounds nothing, both builds agree, and neither alarms.
The fourth case is the honest zero. No unit size stated rounds nothing, because nobody said what the unit was — and the model declines to invent one.
Two clamps are driven independently here and section 18 explains why that mattered: the unit count saturates on a sixty-thousand-gigabyte request against a one-gigabyte unit, and the served figure saturates separately on a twelve-thousand-gigabyte request against a sixteen-gigabyte unit, where seven hundred and fifty units at sixteen gigabytes each exceeds the counter.
7. RTL 3 — Composition Is A Sequence, Not An Assignment
The third thing, and the first model in this chapter with real state.
Binding a resource is not writing a register. It is a walk: discover what is there, reserve it so nobody else takes it, program the fabric so the host can reach it, verify that the path works, and only then report ready. Each step takes time, and a request that arrives during one has to go somewhere.
// RTL 3 - composition is a sequence, not an assignment. A bind walks discover ->
// reserve -> program -> verify -> ready, each step takes cycles, and a request
// arriving while one is in flight must be held rather than dropped.
//
// TEACHING MODEL. Sequential, with real state.
// State remembered : the current step, the cycle counter within it, and one
// pending request (a single-entry skid).
// Invariant : `ready` pulses exactly once per accepted request.
// Reset semantics : asynchronous reset returns to IDLE, clears the pending
// slot, and zeroes every counter - no binding survives it.
// Concurrency : a request arriving in the same cycle as a completion is
// accepted, not lost (see the simultaneous-event test).
// Backpressure : `busy` is the flow-control signal; a second request while
// one is already pending is REJECTED and counted, so an
// overrun is visible rather than silent.
module compose_sequencer #(parameter int BIND_IS_INSTANT = 0) (
input logic clk, rst_n,
input logic req,
input logic [15:0] step_cycles,
output logic [2:0] step,
output logic busy, ready, reject, pending_held,
output logic [15:0] last_latency,
output logic [7:0] n_accepted, n_ready, n_rejected,
output logic seq_err
);
localparam logic [2:0] IDLE = 3'd0, DISCOVER = 3'd1, RESERVE = 3'd2,
PROGRAM = 3'd3, VERIFY = 3'd4, DONE = 3'd5;
logic [2:0] st;
logic [15:0] cyc, elapsed;
logic pending;
assign pending_held = pending;
logic step_done;
assign step = st;
assign busy = (st != IDLE);
// A step of zero cycles still takes one cycle to leave: the model never
// advances two steps in one clock, which is what makes `ready` countable.
assign step_done = (cyc + 16'd1 >= step_cycles);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
st <= IDLE; cyc <= 16'd0; elapsed <= 16'd0; pending <= 1'b0;
ready <= 1'b0; reject <= 1'b0; last_latency <= 16'd0;
n_accepted <= 8'd0; n_ready <= 8'd0; n_rejected <= 8'd0;
end else begin
ready <= 1'b0;
reject <= 1'b0;
// Accept, hold, or reject. A request is rejected only when one is already
// waiting behind an in-flight bind - that is the overrun case.
if (req) begin
if (st == IDLE && !pending) begin
n_accepted <= n_accepted + 8'd1;
st <= DISCOVER; cyc <= 16'd0; elapsed <= 16'd0;
end else if (!pending) begin
n_accepted <= n_accepted + 8'd1;
pending <= 1'b1;
end else begin
reject <= 1'b1;
n_rejected <= n_rejected + 8'd1;
end
end
if (st != IDLE) begin
elapsed <= elapsed + 16'd1;
if (step_done) begin
cyc <= 16'd0;
case (st)
DISCOVER: st <= RESERVE;
RESERVE : st <= PROGRAM;
PROGRAM : st <= VERIFY;
VERIFY : st <= DONE;
default : st <= IDLE;
endcase
if (st == DONE) begin
ready <= 1'b1;
n_ready <= n_ready + 8'd1;
// +1 because `elapsed` is incremented by a non-blocking assignment
// in this same cycle: reading it bare publishes a latency one cycle
// short of the truth. (Baseline defect 2, found before mutation.)
last_latency <= (BIND_IS_INSTANT != 0) ? 16'd0 : (elapsed + 16'd1);
// A request that arrived while this one ran starts immediately.
if (pending) begin
pending <= 1'b0; st <= DISCOVER; cyc <= 16'd0; elapsed <= 16'd0;
end else begin
st <= IDLE;
end
end
end else begin
cyc <= cyc + 16'd1;
end
end
end
end
// Safety: the model must never report a completed bind that took no time.
assign seq_err = ready && (last_latency == 16'd0) && (BIND_IS_INSTANT == 0);
endmoduleState remembered. The current step, the cycle counter within it, and one pending request. That pending slot is a single-entry skid buffer, and its depth is a design decision with visible consequences.
Invariant (safety). ready pulses exactly once per accepted request. Never twice, never for a request that was rejected.
Reset semantics. Asynchronous reset returns the walk to IDLE, clears the pending slot and zeroes every counter. No binding survives reset — a sequencer that came out of reset mid-walk would be programming a fabric on behalf of a request nobody remembers making.
Concurrency. A request arriving in the same cycle a bind completes is accepted, not lost. The testbench drives exactly that case.
Backpressure. busy is the flow-control signal. A second request while one is in flight is held; a third, while one is already held, is rejected and counted — so an overrun is visible on a counter rather than silently dropped.
Five steps at three cycles each is fifteen cycles of bind latency, and at the fastest legal step size it is five.
| Fact | Value |
|---|---|
| Steps | 5 |
| Cycles per step | 3 |
| Bind latency | 15 cycles |
| At 1 cycle/step | 5 cycles |
| Skid depth | 1 request |
| Third request | rejected |
Figure 2 — the three fates of a request. The straight path through the middle is the one everybody designs for. The upper path is the one that makes the design usable — a request that arrives at a busy sequencer is held rather than lost — and the lower path is the one that makes the overrun visible. A sequencer with no reject path drops the third request and reports nothing, which is the failure mode section 22's first lab is about.
The one-deep skid is deliberate and is worth defending. A deeper queue hides the overrun rather than removing it: at some depth the queue fills, and the only question is whether the design tells anybody. A depth of one with a counted reject is more honest than a depth of eight with a silent drop.
8. RTL 4 — Free Capacity Is Not Allocatable Capacity
The fourth thing, and the one that makes a capacity dashboard misleading.
A pool reports how much is free. A request needs a contiguous run. Those are different numbers, and the gap between them is fragmentation. A pool can be half free and unable to satisfy a quarter-sized request, and nothing in a free-capacity figure hints at it.
// RTL 4 - free capacity is not allocatable capacity. A pool with plenty free but
// scattered cannot satisfy a contiguous request, and the gap between the two
// numbers is fragmentation.
//
// TEACHING MODEL. Sequential, with a real occupancy bitmap.
// State remembered : one bit per slot - 1 = allocated, 0 = free.
// Invariant : free_slots always equals the number of zero bits, and
// largest_run is never greater than free_slots.
// Reset semantics : asynchronous reset frees every slot. A composable pool
// that survived reset with stale bits would leak on the
// first allocation after it.
// Concurrency : an allocate and a release in the same cycle are both
// honoured; the bitmap is updated with one expression so
// the two cannot race.
// Synthesis note : largest_run is computed by a combinational scan over the
// bitmap. That scan is O(SLOTS) logic and is the reason a
// real allocator keeps a maintained run-length structure
// rather than recomputing - see the chapter text.
module fragmentation_tracker #(
parameter int SLOTS = 16,
parameter int FREE_MEANS_ALLOCATABLE = 0
) (
input logic clk, rst_n,
input logic alloc, release_slot,
input logic [3:0] alloc_id, release_id,
input logic [15:0] request_run,
output logic [15:0] free_slots, largest_run, shortfall,
output logic [SLOTS-1:0] occupancy,
output logic can_place,
output logic [7:0] n_ops, n_blocked,
output logic frag_err
);
logic [SLOTS-1:0] occ;
integer i;
logic [15:0] run, best, freec;
logic truly_blocked;
assign occupancy = occ;
// Combinational scan: free count and longest contiguous free run.
always_comb begin
run = 16'd0; best = 16'd0; freec = 16'd0;
for (i = 0; i < SLOTS; i = i + 1) begin
if (!occ[i]) begin
freec = freec + 16'd1;
run = run + 16'd1;
if (run > best) best = run;
end else begin
run = 16'd0;
end
end
end
assign free_slots = freec;
// The weak build reports the free COUNT as if it were placeable, which is the
// whole error this section is about.
assign largest_run = (FREE_MEANS_ALLOCATABLE != 0) ? freec : best;
assign shortfall = (request_run > largest_run) ? (request_run - largest_run) : 16'd0;
assign can_place = (shortfall == 16'd0) && (request_run != 16'd0);
// The truth is computed from `best` regardless of the parameter, which is what
// lets the model detect its own weak build.
// No `request_run != 0` guard: for unsigned operands `request_run > best`
// already implies it, so the guard would be unreachable code and an
// equivalent mutant. Caught by domcheck BEFORE the campaign.
assign truly_blocked = (request_run > best);
assign frag_err = (alloc | release_slot) && truly_blocked && can_place;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
occ <= '0; n_ops <= 8'd0; n_blocked <= 8'd0;
end else begin
if (alloc || release_slot) begin
n_ops <= n_ops + 8'd1;
if (truly_blocked) n_blocked <= n_blocked + 8'd1;
end
// Separate ids, so a simultaneous allocate and release of DIFFERENT slots
// both land in one cycle. When they name the SAME slot the release is
// applied last and wins - a documented arbitration choice, not an
// accident, and the testbench asserts both orderings.
if (alloc) occ[alloc_id] <= 1'b1;
if (release_slot) occ[release_id] <= 1'b0;
end
end
endmoduleState remembered. One bit per slot: set means allocated, clear means free.
Invariant. free_slots always equals the number of zero bits, and largest_run is never greater than free_slots.
Reset semantics. Reset frees every slot. An allocator that survived reset holding stale bits would hand out a slot somebody else believes they own — which is the worst failure in this chapter, because it produces two owners for one resource and nothing reports it.
Concurrency. Allocate and release carry separate slot ids, so a simultaneous allocate of one slot and release of another both land in the same cycle. When they name the same slot, the release is applied last and wins — a documented arbitration choice, and the testbench asserts both orderings.
Synthesis note. largest_run is a combinational scan across the whole bitmap. That scan is O(SLOTS) of logic on the critical path, and it is exactly why a production allocator maintains a run-length structure incrementally instead of recomputing. Section 20 takes that further.
Sixteen slots with every odd one allocated is eight slots free and a longest run of one — so a four-slot request is three short and cannot be placed.
| Fact | Value |
|---|---|
| Slots | 16 |
| Free | 8 |
| Longest run | 1 |
| Request | 4 slots |
| Shortfall | 3 |
| Placeable | no |
The same eight free slots in a different shape place the request immediately. Release four adjacent slots out of a full pool and the free count is four — half what the checkerboard had — and the request places. The count went down and the capability went up, which is the whole of this section in one comparison.
9. RTL 5 — A Binding Outlives The Host That Asked For It
The fifth thing, and the one that leaks.
When a host dies, the resource it held does not become free. It becomes attributed to a host that no longer exists: not faulty, not available, and invisible to any check that asks whether the resource is healthy. Something has to notice and reclaim it, and that something is a separate mechanism with its own failure modes.
// RTL 5 - a binding outlives the host that asked for it unless something
// reclaims it. The resource is not broken and not free: it is attributed to a
// host that no longer exists, and nothing in the data path notices.
//
// TEACHING MODEL. Sequential, with the state that matters.
// State remembered : per-resource binding state and the owning host id.
// Invariant (safety) : a resource is never handed to a second host while it
// is still bound to a first.
// Invariant (liveness) : under the stated assumption that the reaper runs, a
// resource whose host died EVENTUALLY returns to FREE.
// With the reaper disabled that guarantee is withdrawn
// and the resource is leaked - which is the point.
// Reset semantics : asynchronous reset frees everything and clears owners.
// Concurrency : a host-loss and a voluntary release in the same cycle
// resolve to FREE exactly once, not twice.
module bind_lifecycle #(parameter int HOSTS_NEVER_DIE = 0) (
input logic clk, rst_n,
input logic bind_req, release_req, host_lost, reap_tick,
input logic [3:0] host_id,
output logic [1:0] state,
output logic [3:0] owner,
output logic resource_free, leaked,
output logic [7:0] n_bound, n_released, n_reaped, n_leaked,
output logic life_err
);
localparam logic [1:0] FREE = 2'd0, BOUND = 2'd1, ORPHANED = 2'd2;
logic [1:0] st;
logic [3:0] own;
assign state = st;
assign owner = own;
assign resource_free = (st == FREE);
// An orphaned resource is leaked: attributed to a dead host, not reclaimable
// by anybody else, and invisible to a free-capacity count that only asks
// whether the resource is faulty.
assign leaked = (st == ORPHANED);
// Safety violation: the model claims free while still owned by somebody.
assign life_err = resource_free && (st != FREE);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
st <= FREE; own <= 4'd0;
n_bound <= 8'd0; n_released <= 8'd0; n_reaped <= 8'd0; n_leaked <= 8'd0;
end else begin
case (st)
FREE: if (bind_req) begin
st <= BOUND; own <= host_id; n_bound <= n_bound + 8'd1;
end
BOUND: begin
// Priority is explicit: a host that is gone cannot also have
// issued a clean release, so host_lost is resolved first and
// the two cannot both fire a transition in one cycle.
if (host_lost && (HOSTS_NEVER_DIE == 0)) begin
st <= ORPHANED;
n_leaked <= n_leaked + 8'd1;
end else if (release_req || host_lost) begin
st <= FREE; own <= 4'd0;
n_released <= n_released + 8'd1;
end
end
ORPHANED: begin
// Liveness depends entirely on this tick arriving.
if (reap_tick) begin
st <= FREE; own <= 4'd0; n_reaped <= n_reaped + 8'd1;
end
end
default: st <= FREE;
endcase
end
end
endmoduleState remembered. The binding state and the owning host id.
Safety invariant. A resource is never handed to a second host while it is still bound to a first. The testbench drives exactly that attack: bind to host 7, then request a bind for host 9, and assert the owner is still 7.
Liveness invariant, with its assumption stated. An orphaned resource eventually returns to FREE — assuming the reaper runs. That assumption is load-bearing, and the testbench proves the withdrawal rather than the guarantee: with no reaper tick the resource sits orphaned through ten cycles, owner still recorded, and only returns when the tick arrives.
Concurrency. A host-loss and a voluntary release in the same cycle resolve to exactly one transition. Host-loss has priority — a host that is gone cannot also have issued a clean release — and the counters prove only one of them advanced.
The walk is FREE to BOUND on a bind, BOUND to ORPHANED on a host loss, ORPHANED to FREE on a reap.
| Fact | Value |
|---|---|
| States | FREE, BOUND, ORPHANED |
| Owner on orphan | retained |
| Reported free | no |
| Reported faulty | no |
| Returns without a reaper | never |
| Reaper priority vs release | host-loss wins |
The owner is retained on purpose and it is the most important line in the model. A resource that cleared its owner when the host died would be indistinguishable from a free one, and the leak would be undetectable. Keeping the dead host's id is what makes the leak visible to telemetry — section 21 is about reading exactly that.
10. RTL 6 — The Control Plane Is A New Dependency
The sixth thing, and the one that changes what an outage means.
Composition puts an orchestrator between a workload and the resources it needs. A host that could previously start on its own now cannot: the resources are healthy, powered and idle, and unreachable. That is a different incident from a resource failure, with a different fix and a different blast radius, and a capacity dashboard reports them identically.
// RTL 6 - the orchestrator is a new dependency. Composition puts a control
// plane between a workload and the resources it needs, and a workload that
// could previously start on its own now cannot.
//
// TEACHING MODEL. The distinction this model draws is between resources that
// are UNAVAILABLE and resources that are merely UNREACHABLE because the thing
// that grants them is down - two very different incident reports.
module orchestrator_dependency #(parameter int CONTROL_PLANE_IS_FREE = 0) (
input logic clk, rst_n,
input logic evaluate,
input logic [15:0] hosts, resources_healthy, orchestrator_up, start_value,
output logic [15:0] startable, blocked, blocked_value, available_pct,
output logic starts_independently,
output logic [7:0] n_evals, n_blocked,
output logic dep_err
);
logic [31:0] v_q, p_q;
logic [15:0] true_startable, true_blocked;
logic truly_blocked;
// Healthy resources are reachable only while the control plane is up.
assign true_startable = (orchestrator_up != 16'd0) ? resources_healthy : 16'd0;
assign startable = (CONTROL_PLANE_IS_FREE != 0) ? resources_healthy : true_startable;
assign true_blocked = resources_healthy - true_startable;
assign blocked = (CONTROL_PLANE_IS_FREE != 0) ? 16'd0 : true_blocked;
// Computed from the REPORTED blocked count, not the true one: a build that
// claims nothing is blocked must not simultaneously publish a blocked value.
// The truth below still uses true_blocked, which is what preserves the
// model's ability to detect its own weak build.
assign v_q = {16'd0, blocked} * {16'd0, start_value};
assign blocked_value = (v_q > 32'd9999) ? 16'd9999 : v_q[15:0];
assign p_q = (hosts == 16'd0) ? 32'd100
: (({16'd0, startable} * 32'd100) / {16'd0, hosts});
assign available_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
assign starts_independently = (blocked == 16'd0) && (resources_healthy != 16'd0);
assign truly_blocked = (true_blocked != 16'd0);
assign dep_err = evaluate && truly_blocked && starts_independently;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
n_evals <= 8'd0; n_blocked <= 8'd0;
end else if (evaluate) begin
n_evals <= n_evals + 8'd1;
if (truly_blocked) n_blocked <= n_blocked + 8'd1;
end
end
endmoduleThirty-six healthy resources with the control plane down are thirty-six blocked, at five thousand four hundred units of start value, and zero availability.
| Fact | Value |
|---|---|
| Hosts | 40 |
| Healthy resources | 36 |
| Control plane | down |
| Startable | 0 |
| Blocked | 36 |
| Value blocked | 5,400 |
The second case is the same estate with the plane up: all thirty-six startable, both builds agreeing, and neither alarming.
The third case is the distinction the section exists for. No healthy resources and no control plane reports nothing blocked — because nothing was available to be blocked. That is unavailable, not unreachable, and the model keeps them apart rather than adding them together.
11. RTL 7 — Composition Moves Stranding Rather Than Abolishing It
The seventh thing, and the one that decides whether any of it paid.
29.3 established stranding as the motivation. This section is what happens to it afterwards. Capacity that was stranded per host becomes capacity stranded in the fabric: reserved and not yet bound, rounded up by section 6's granularity, or sitting in a pool the requesting host cannot reach.
// RTL 7 - composition moves stranding, it does not abolish it. Capacity that was
// stranded per host becomes capacity stranded in the fabric: reserved, bound,
// rounded up, or held by a pool the requesting host cannot reach.
//
// TEACHING MODEL. Both figures are computed and published so the NET is visible;
// a description that reports only the recovered half has reported a benefit.
module stranding_moves #(parameter int COMPOSITION_ENDS_STRANDING = 0) (
input logic clk, rst_n,
input logic evaluate,
input logic [15:0] host_stranded, fabric_reserved, rounding_waste, unreachable_gb,
output logic [15:0] recovered, new_stranded, net_gain, net_loss,
output logic composition_pays,
output logic [7:0] n_evals, n_negative,
output logic strand_err
);
logic [31:0] s_q;
logic [15:0] true_new;
logic truly_negative;
assign recovered = host_stranded;
assign s_q = {16'd0, fabric_reserved} + {16'd0, rounding_waste}
+ {16'd0, unreachable_gb};
assign true_new = (s_q > 32'd9999) ? 16'd9999 : s_q[15:0];
assign new_stranded = (COMPOSITION_ENDS_STRANDING != 0) ? 16'd0 : true_new;
// Reported against the stranding this build admits to, so the ends-stranding
// view stays internally coherent rather than publishing a zero and a loss.
assign net_gain = (recovered > new_stranded) ? (recovered - new_stranded) : 16'd0;
assign net_loss = (new_stranded > recovered) ? (new_stranded - recovered) : 16'd0;
assign composition_pays = (COMPOSITION_ENDS_STRANDING != 0) ? (recovered != 16'd0)
: (recovered > true_new);
assign truly_negative = (true_new >= recovered) && (recovered != 16'd0);
assign strand_err = evaluate && truly_negative && composition_pays;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
n_evals <= 8'd0; n_negative <= 8'd0;
end else if (evaluate) begin
n_evals <= n_evals + 8'd1;
if (truly_negative) n_negative <= n_negative + 8'd1;
end
end
endmoduleNine hundred gigabytes of host stranding recovered, against four hundred reserved plus three hundred rounded plus two hundred and sixty unreachable, is nine hundred and sixty newly stranded — a net loss of sixty.
| Fact | Value |
|---|---|
| Host stranding recovered | 900 GB |
| Fabric reserved | 400 GB |
| Rounding waste | 300 GB |
| Unreachable | 260 GB |
| Newly stranded | 960 GB |
| Net | −60 GB |
The second case is the composition that genuinely pays — three hundred and eighty newly stranded against nine hundred recovered, five hundred and twenty net — and it is not a hard configuration to reach. The difference between the two is entirely in the three cost terms, which is where a design has leverage.
The fourth case is the one that keeps the model honest: nothing recovered at all makes the entire new stranding a loss, and neither build alarms, because there is no recovery to contradict.
12. RTL 8 — Which Workload Shapes Actually Gain
The eighth thing, and the one that says where composability belongs.
A workload whose resource mix varies between runs gains from composition. One whose mix is fixed pays for a fabric and a control plane to deliver the same shape every time. The classification is per workload, and an estate-wide answer has to be wrong about part of the estate.
// RTL 8 - which workload shapes gain. A workload whose resource mix varies
// between runs gains from composition; one with a fixed mix pays the fabric and
// the orchestrator for flexibility it never exercises.
module where_composability_wins #(parameter int EVERY_WORKLOAD_GAINS = 0) (
input logic clk, rst_n,
input logic evaluate,
input logic [15:0] workloads, varying_mix, fixed_mix, gain_each,
output logic [15:0] suited, unsuited, total_gain, suited_pct,
output logic suits_all,
output logic [7:0] n_evals, n_unsuited,
output logic shape_err
);
logic [31:0] g_q, p_q;
logic [15:0] var_ok, fix_ok, true_suited, true_unsuited;
logic truly_unsuited;
assign var_ok = (varying_mix > workloads) ? workloads : varying_mix;
assign fix_ok = (fixed_mix > workloads) ? workloads : fixed_mix;
assign true_suited = (var_ok > fix_ok) ? (var_ok - fix_ok) : 16'd0;
assign suited = (EVERY_WORKLOAD_GAINS != 0) ? workloads : true_suited;
assign true_unsuited = workloads - true_suited;
assign unsuited = (EVERY_WORKLOAD_GAINS != 0) ? 16'd0 : true_unsuited;
assign g_q = {16'd0, true_suited} * {16'd0, gain_each};
assign total_gain = (g_q > 32'd9999) ? 16'd9999 : g_q[15:0];
assign p_q = (workloads == 16'd0) ? 32'd100
: (({16'd0, true_suited} * 32'd100) / {16'd0, workloads});
assign suited_pct = p_q[15:0];
assign suits_all = (unsuited == 16'd0) && (workloads != 16'd0);
assign truly_unsuited = (true_unsuited != 16'd0);
assign shape_err = evaluate && truly_unsuited && suits_all;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
n_evals <= 8'd0; n_unsuited <= 8'd0;
end else if (evaluate) begin
n_evals <= n_evals + 8'd1;
if (truly_unsuited) n_unsuited <= n_unsuited + 8'd1;
end
end
endmoduleEighteen workloads with eleven varying and five fixed is six net suited, twelve not, at three hundred and ninety units of gain — a third of the estate.
| Fact | Value |
|---|---|
| Workloads | 18 |
| Varying mix | 11 |
| Fixed mix | 5 |
| Suited | 6 |
| Unsuited | 12 |
| Suited | 33% |
The fourth case is the expensive direction and it is common. A mostly fixed estate gains nothing at all, and the weak view still calls it a sweep — the fabric was built, the control plane is running, and every workload gets exactly the shape it would have had anyway.
13. RTL 9 — Flexibility Has A Standing Bill
The ninth thing, and the one that is paid whether or not anything moves.
A composable estate pays for fabric ports, a control plane and the operational surface of both, every hour. That cost does not scale with how often anything is recomposed — it is there when the estate is completely static. A bill that only appears when something is recomposed has hidden the larger half.
// RTL 9 - flexibility has a standing bill. A composable estate pays for fabric
// ports, a control plane, and the operational surface of both, every hour,
// whether or not anything is recomposed that hour.
//
// TEACHING MODEL. The rate/benefit comparison is deliberately per-period so the
// STANDING cost is separable from the per-composition one - a distinction a
// single total hides.
module flexibility_bill #(parameter int FLEXIBILITY_IS_FREE = 0) (
input logic clk, rst_n,
input logic evaluate,
input logic [15:0] recompositions, value_each, standing_cost, per_event_cost,
output logic [15:0] event_cost, total_cost, benefit, net_benefit,
output logic flexibility_pays,
output logic [7:0] n_evals, n_negative,
output logic bill_err
);
logic [31:0] e_q, t_q, b_q;
logic [15:0] true_event, true_total, true_benefit;
logic truly_negative;
assign e_q = {16'd0, recompositions} * {16'd0, per_event_cost};
assign true_event = (e_q > 32'd9999) ? 16'd9999 : e_q[15:0];
assign event_cost = true_event;
assign t_q = {16'd0, standing_cost} + {16'd0, true_event};
assign true_total = (t_q > 32'd9999) ? 16'd9999 : t_q[15:0];
// The weak build drops the STANDING cost and keeps only the per-event one,
// which is what a bill that only appears when something is recomposed looks
// like.
assign total_cost = (FLEXIBILITY_IS_FREE != 0) ? true_event : true_total;
assign b_q = {16'd0, recompositions} * {16'd0, value_each};
assign true_benefit = (b_q > 32'd9999) ? 16'd9999 : b_q[15:0];
assign benefit = true_benefit;
assign net_benefit = (true_benefit > total_cost) ? (true_benefit - total_cost) : 16'd0;
assign flexibility_pays = (FLEXIBILITY_IS_FREE != 0) ? (true_benefit > true_event)
: (true_benefit > true_total);
assign truly_negative = (true_total >= true_benefit) && (true_benefit != 16'd0);
assign bill_err = evaluate && truly_negative && flexibility_pays;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
n_evals <= 8'd0; n_negative <= 8'd0;
end else if (evaluate) begin
n_evals <= n_evals + 8'd1;
if (truly_negative) n_negative <= n_negative + 8'd1;
end
end
endmoduleTwenty recompositions at a hundred and forty, against sixteen hundred standing and seventy per event, is fourteen hundred of event cost, three thousand total, against two thousand eight hundred of benefit — a loss of two hundred.
| Fact | Value |
|---|---|
| Recompositions | 20 |
| Value each | 140 |
| Standing cost | 1,600 |
| Per-event cost | 70 |
| Total cost | 3,000 |
| Benefit | 2,800 |
Double the recompositions and it pays. Forty events gives five thousand six hundred of benefit against four thousand four hundred of cost — twelve hundred net — because the standing cost amortises and the per-event cost does not. That is the shape of the whole decision: composability rewards estates that actually recompose.
The third case is the one that ends projects. An estate that never recomposes pays the sixteen hundred standing cost and gains nothing at all, and the weak build sees no cost whatsoever because nothing triggered an event.
14. RTL 10 — A Composable-Infrastructure Case Study Assembled
Nine sections of inputs. This one puts them together.
// RTL 10 - a composable-infrastructure case study assembled. Nine sections of
// inputs, one summary. "The infrastructure is composable" is bit 0: true of
// three different granularities, and one sixth of a deployment.
module composable_signoff #(parameter int COMPOSABLE_IS_THE_ANSWER = 0) (
input logic clk, rst_n,
input logic evaluate,
input logic composability_named, granularity_stated, latency_stated,
input logic fragmentation_measured, reclamation_stated, control_plane_stated,
output logic [5:0] fail_mask,
output logic [15:0] conditions_met, sound_pct,
output logic sound,
output logic [7:0] n_evals, n_sound, n_claimed,
output logic signoff_err
);
logic [31:0] s_q;
logic truly_sound, claimed;
assign fail_mask[0] = ~composability_named;
assign fail_mask[1] = ~granularity_stated;
assign fail_mask[2] = ~latency_stated;
assign fail_mask[3] = ~fragmentation_measured;
assign fail_mask[4] = ~reclamation_stated;
assign fail_mask[5] = ~control_plane_stated;
assign conditions_met = {15'd0, composability_named} + {15'd0, granularity_stated}
+ {15'd0, latency_stated} + {15'd0, fragmentation_measured}
+ {15'd0, reclamation_stated} + {15'd0, control_plane_stated};
assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
// No clamp: conditions_met sums six one-bit values, so the quotient cannot
// exceed a hundred and a ceiling would be unreachable code.
assign sound_pct = s_q[15:0];
assign truly_sound = (fail_mask == 6'd0);
assign claimed = (COMPOSABLE_IS_THE_ANSWER != 0) ? composability_named : truly_sound;
assign sound = claimed;
assign signoff_err = evaluate && !truly_sound && claimed;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
n_evals <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
end else if (evaluate) begin
n_evals <= n_evals + 8'd1;
if (truly_sound) n_sound <= n_sound + 8'd1;
if (claimed) n_claimed <= n_claimed + 8'd1;
end
end
endmoduleThe stimulus walks all six bits one at a time. When composability has been named and any one of the other five fails, the assembled model reports that the case study is not sound and the composable view reports a case study.
| Bit | Condition, and the section that builds it |
|---|---|
| 0 | Composability was named at all — §14 |
| 1 | The granularity was stated — §6 |
| 2 | The bind latency was stated — §7 |
| 3 | Fragmentation was measured — §8 |
| 4 | Reclamation was stated — §9 |
| 5 | The control-plane dependency was stated — §10 |
Across the eight evaluations, the assembled model calls one case study sound and the composable view calls seven of eight a case study.
The bit order is by how much of the deployment each condition carries. Bit 1 is first among the five because granularity decides the allocator, the fragmentation and the rounding tax all at once. Bit 2 is what a bind costs in time. Bit 3 is why a free-capacity figure lies. Bits 4 and 5 are the two failure modes that belong to somebody else — the reaper and the orchestrator — which is why they go missing.
"It is composable" is bit 0, and it closes Module 29's set of four weak definitions. 29.1's names a category containing its own opposite; 29.2's describes the packaging; 29.3's is a claim about other people; this one is a claim about a capability that three different architectures deliver in three incompatible ways.
15. Quantitative Reasoning
Every figure below is either a teaching parameter chosen to make a relationship visible, or a result derived from those parameters and asserted by the testbench. None is a measurement of any real system.
Two axes of three unstated, on an inventory a quarter of which is bindable — a third of the claim specified.
Fourteen gigabytes rounded away per request and five hundred and sixty across forty, from a fifty-gigabyte request against a sixteen-gigabyte unit. Ceiling division: four units, sixty-four gigabytes served.
Fifteen cycles of bind latency — five steps at three cycles each — and five cycles at the fastest legal step size.
Eight slots free with a longest run of one, so a four-slot request is three short. Four adjacent slots free out of a full pool places the same request on half the free capacity.
Twenty-three allocator operations, sixteen of them issued while the pool could not satisfy the request — traced by hand over the bitmap rather than read from the design.
Four cycles orphaned with the dead host still recorded as owner, and unbounded without a reaper.
Thirty-six resources blocked at five thousand four hundred units of start value, with the control plane down and every resource healthy.
Nine hundred and sixty gigabytes newly stranded against nine hundred recovered — a net loss of sixty, from four hundred reserved plus three hundred rounded plus two hundred and sixty unreachable.
Twelve workloads of eighteen unsuited, with six suited at three hundred and ninety units of gain.
Three thousand of cost against two thousand eight hundred of benefit at twenty recompositions, turning to twelve hundred net at forty — because sixteen hundred of it is standing cost that amortises.
16. Verification Method
This section is the method rather than the results, because the method is what makes the results worth anything.
The order of work
Every model went through the same sequence, and the sequence matters:
legal baseline → corner cases → simultaneous events → resource boundaries → invalid and error inputs → PASS → mutation campaign
A mutation campaign on a failing baseline proves nothing. If the design is already wrong, a mutation that makes it differently wrong still fails the same assertions, and the kill is counted for the wrong reason. Six defects were found before a single mutation was injected, and section 19 lists them.
Independent oracles
Expected values are never derived by copying a DUT expression. Each one is reasoned from the model's specification and recorded at the call site. Three examples:
| Model | Oracle, reasoned independently |
|---|---|
| m3 latency | five states × three cycles per state = 15 |
| m4 fragmentation | eight zero bits, none adjacent → longest run 1 |
| m4 allocator | 8 + 2 + 9 + 4 = 23 ops, blocked from the eighth |
| m5 lifecycle | FREE → BOUND → ORPHANED → FREE, read off the transition list |
If the oracle and the design disagree, either could be wrong — which is why chkv prints both numbers on failure. That is exactly how m3's latency defect was caught: the oracle said fifteen, the design said fourteen, and the design was wrong.
X and Z are rejected explicitly
A check written if (!condition) lets an X-valued condition pass silently. Both check tasks reject unknowns:
| Task | Behaviour on X/Z |
|---|---|
chk(c, what) | c !== 1'b1 — an X condition fails |
chkv(got, exp, what) | ^got === 1'bx → explicit X/Z failure before any comparison |
Safety, liveness and performance are separated
These are three different kinds of claim and conflating them is how a performance target becomes a false correctness bug.
Safety — something bad never happens. A second host never takes a resource still bound to a first. ready never pulses twice for one request. The measured build never claims a request is placeable when it is not. Safety needs no assumptions.
Liveness — something good eventually happens, under stated assumptions. An accepted bind reaches DONE, assuming the clock runs and reset is not asserted. An orphaned resource returns to FREE, assuming the reaper runs. The second assumption is the one this chapter is about, and the testbench proves its withdrawal: ten cycles with no reaper and the resource never returns.
Performance — how fast, how much, how fair. Fifteen cycles of bind latency is a target. A pool that places a four-slot request is a capability. Neither is a correctness property, and a design that misses either is slow or small rather than broken.
17. Assertions
The testbenches carry 484 checks — 237 in tb1 across the first five models, 247 in tb2 across the last five.
Every output of every model is asserted as a value, in both builds. The output-listing step reports nothing on either testbench. That includes the three weak-build instances of the sequential models, which section 18 explains were missing entirely at first.
Every simulator-derived value printed to the reader is also asserted. The displayed-value gate scans 168 printed references and reports zero derived values without a check behind them. Stimulus inputs that the testbench itself drove are excluded, because those are constants the author set rather than results the simulator produced.
Reset is verified on every sequential model, and not only at time zero: reset is asserted again mid-walk with a bind in flight, mid-allocation with live occupancy bits, and mid-binding with a live owner recorded. In each case the model must come back empty.
Simultaneous events are driven rather than assumed. An allocate and a release of different slots in one cycle; an allocate and a release of the same slot, where the documented arbitration is asserted; a host loss and a voluntary release in one cycle, where exactly one counter must advance.
Invalid operations are driven. A release on a resource nobody holds must be a no-op. A bind request while already bound must not change the owner. A zero-slot request is not a placeable request.
Single-cycle pulses are latched by continuous monitors rather than sampled. ready, reject and both builds' error outputs are caught by an always @(posedge clk) monitor and asserted outside any conditional — section 18 explains why that change was forced.
18. Mutation Testing
128 mutations injected, 128 killed. Two further mutations were withdrawn as proven equivalent and are not counted as kills.
| Family | Count |
|---|---|
| Ownership, leak and stale state | 14 |
| Sequencer walk, backpressure and reset | 16 |
| Allocator bitmap and arbitration | 14 |
| Clamp inverted or removed | 26 |
| Parameter-selected branches swapped | 18 |
| Guard or zero-case flipped | 17 |
| Arithmetic reversed or wrong operator | 16 |
| Counter inverted or double-stepped | 14 |
Seven mutations survived the first run. Every one was classified before anything was changed.
Two were equivalent and were withdrawn
st == IDLE && !pending → st == IDLE. pending is only ever set in a cycle where the walk is already running, and is cleared on the same edge it restarts — so pending implies the walk is not idle and the second conjunct is redundant. No legal input distinguishes the two. Rather than merely withdraw it, the model now exposes pending_held and the testbench runs a continuous monitor asserting the invariant, so the redundancy is proven rather than assumed.
run > best → run >= best. At the equality the assignment writes best = run where best already equals run — a no-op. Withdrawn.
Three were genuine gaps in my own checkers
reject never pulsing survived because the testbench only ever asserted that reject was low. It never asserted it high at the overrun.
ready counted but never pulsed survived for a worse reason. The latency check lived inside if (cRdy === 1'b1). With ready suppressed the branch never ran, and the check passed vacuously. This is precisely the failure the batch rule warns about: a check nested inside a condition the mutation itself suppresses can never fail. Both are now caught by continuous monitors that latch the pulse and are asserted outside any conditional.
A release on a FREE resource binding it survived because releasing something you do not hold was never tested. An invalid-operation case now covers it.
One exposed a structural hole
host_lost no longer releasing in the HOSTS_NEVER_DIE build survived because that build was never instantiated. All three sequential models — the sequencer, the allocator and the lifecycle — had only their measured build wired up. The two-build contrast that every combinational model in this chapter carries had been silently omitted from all three, and only a mutation targeting a branch that the missing build would have taken could find it. All three weak builds are now instantiated and contrasted.
One exposed a hole in the tooling itself
A counter inversion in section 11 survived because three negative cases out of six evaluations is exactly half — the inverted counter reaches the same total, so the mutation was equivalent given that stimulus. A seventh, non-negative case makes it observable and the mutation is now a genuine kill.
The larger finding is why the split checker did not catch it in advance. This chapter switched its assertion idiom from chk(x == N) to chkv(x, N, "...") for the X/Z rejection described in section 16. Three of the scripted checks read expected values straight out of the testbench source — and all three silently stopped matching. Two of them reported a confident zero.
A checker that cannot parse its input reports "clean", not "cannot read this." That is the same masking relationship this track has been chasing inside the RTL, appearing in the verification tooling instead. All three checks are now idiom-agnostic and were re-verified against four already-published chapters, which still report zero.
19. Baseline Defects Found Before Mutation
Six defects were found by baseline verification, before any mutation was injected. They are reported separately from mutation survivors because they are a different kind of evidence: survivors measure the testbench, baseline failures measure the design.
RTL defects — in the teaching models themselves
m3 published a bind latency one cycle short. last_latency <= elapsed captured the counter before its own non-blocking increment landed, so a bind that genuinely takes fifteen cycles reported fourteen. Found because the independent oracle said fifteen. Fixed to elapsed + 1, and a mutation now guards the regression.
m4 carried a dead guard. (request_run > best) && (request_run != 0) — for unsigned operands the first conjunct already implies the second, so the guard was unreachable code and any mutation of it would have been equivalent. Found by a scripted check before the campaign ran, which is the first time that class has been caught prospectively in this track rather than by a survivor.
m6's weak build contradicted itself. It reported blocked = 0 and blocked_value = 5400 in the same evaluation, because the value was computed from the true count while the count was parameter-selected. A weak build that contradicts itself is broken rather than weak — a reader dismisses it instead of being misled by it. Found only because the output-listing check demanded a standalone assertion on that net.
Testbench defects — mine, not the design's
Stimulus driven at the sampling edge. A single request was sampled twice, reporting two accepted binds. Fixed by driving every stimulus #1 after the edge, never on it.
A combinational output read before it settled. Four checks failed reading can_place and shortfall in the same time step as request_run was driven. Fixed with #1 after every bare combinational assignment.
Simulator constraint — a tool limit, not a fault
Icarus Verilog 13.0 rejects ref arguments on tasks — sorry: Reference ports not supported yet. The pulse helper was inlined. This is recorded as a constraint alongside the existing list, not as a defect in anything.
20. Synthesis And Implementation Reality
These are teaching models, but the shapes they use have real implementation consequences and it is worth being explicit about which.
The bitmap scan in section 8 is the expensive one. largest_run is a combinational loop across every slot, computing a running length and a maximum. That is a chain of comparators and adders proportional to the slot count, sitting on the path between the occupancy registers and any logic that consumes can_place. At sixteen slots it is unremarkable; at a few thousand it is the critical path.
Which is why a production allocator does not recompute it. The usual structures maintain run lengths incrementally — a free list ordered by size, a buddy structure, or a coarse bitmap over a fine one — so an allocation updates a small amount of state instead of rescanning. The teaching model recomputes precisely because recomputation is obviously correct and obviously unaffordable, which makes the tradeoff visible.
Register cost is dominated by the bitmap and the owner fields. One bit per slot plus one host id per bindable resource. A sub-device granularity multiplies the slot count and therefore both the storage and the scan.
The sequencer's skid depth is a register decision with a protocol consequence. Depth one costs a flop and a valid bit; deeper costs a small FIFO. The depth does not remove the overrun, it moves it — at some depth the queue fills, and the design choice is whether that is reported.
Reset cost. Every model here uses asynchronous reset on state that must not survive it. The bitmap is the largest such structure, and a synchronous-reset implementation of it would be cheaper in area and would need a guarantee that no allocation is accepted before the clear completes.
Width discipline. Every product in these models is taken in thirty-two bits before being clamped back to sixteen. A sixteen-bit multiply of two sixteen-bit quantities wraps silently, and a wrapped capacity figure is worse than an absent one — it is a small plausible number where a large one belonged.
No area, frequency or power figures are given anywhere in this chapter, because none was measured and inventing them would be exactly the overclaim section 4 rules out.
21. Silicon Observability
The question to ask of every mechanism here: if this failed after tapeout, what evidence would an engineer need?
| Telemetry | What it exposes |
|---|---|
n_accepted vs n_ready | binds accepted but never completed — a stuck walk |
n_rejected | overrun: requests arriving faster than binds complete |
last_latency | bind time drift as the fabric grows |
free_slots vs largest_run | fragmentation, and nothing else shows it |
n_blocked | how often the pool could not satisfy a request |
state + owner | a leaked resource and which dead host holds it |
n_leaked vs n_reaped | the reaper falling behind, or not running |
blocked vs healthy count | unreachable versus unavailable |
The pair that matters most is free capacity against longest run. A dashboard reporting only the first will show a healthy pool that cannot place anything, and there is no other signal that distinguishes the two. If only one counter can be afforded, make it the run length, because free capacity can be inferred approximately and contiguity cannot be inferred at all.
The owner field on an orphaned resource is the debugging evidence, not the bug. It is tempting to clear it on host loss for tidiness. Doing so makes an orphan indistinguishable from a free resource, and the leak becomes invisible to every tool. Retaining a dead host id is deliberate telemetry.
n_leaked minus n_reaped is the standing leak, and it is the single number an operator should alarm on. A value that grows monotonically means the reaper is not running; a value that oscillates means it is running but losing.
What different patterns imply:
A rising n_rejected with flat last_latency means arrival rate grew, not that binds got slower. The fix is upstream rate limiting, not a faster sequencer.
A rising last_latency with flat n_rejected means the fabric got slower — more hops, more devices to program — and the overrun has not started yet but will.
free_slots high and largest_run low is fragmentation, and no amount of adding capacity fixes it. Compaction or a coarser allocation unit does.
n_blocked high with largest_run healthy means the requests grew, not the pool shrank.
22. DebugLabs
Lab 1 — Binds succeed, but one request in every burst disappears
Symptom. Under bursty load a host occasionally never receives its resource. No error is logged and the orchestrator shows the request as issued.
Evidence. n_accepted and n_ready match. n_rejected is non-zero and climbing with burst size. The missing requests correlate exactly with bursts of three or more arriving inside one bind window.
Hypothesis. The sequencer holds one pending request and rejects any further ones. The rejection is real, counted and correct — and nothing upstream is listening to it.
Investigation. Compare n_accepted + n_rejected against the orchestrator's issued count. If they match, nothing was lost at the hardware boundary and the request was refused rather than dropped. Then check whether the requester has any path for a reject response.
Root cause. A one-deep skid with a correctly reported overrun, feeding a requester that treats every issued request as accepted. The hardware is right and the contract is not implemented on the other side.
Fix. Either handle reject upstream by retrying, or apply backpressure using busy before issuing. Deepening the queue is the tempting fix and the wrong one — it raises the burst size that fails without removing the failure.
Recurrence check. A test that issues three requests inside one bind window and asserts both that reject pulses and that the requester retried. The mutation that removes the reject pulse must fail this test.
Telemetry. n_rejected alone, alarmed on any non-zero value in a system whose requester does not implement retry.
Lab 2 — The pool reports forty percent free and allocations fail
Symptom. Capacity dashboards show plenty free. Allocation requests fail with no capacity available.
Evidence. free_slots is eight of sixteen. largest_run is one. n_blocked is climbing on every request. No slot is faulty and no device is degraded.
Hypothesis. Fragmentation. The free capacity is real and scattered, and the requests need contiguity.
Investigation. Read largest_run against free_slots over time. If free capacity is stable while the longest run decays, allocations and releases are interleaving in a pattern that leaves isolated holes. Dump the occupancy bitmap and look at the shape rather than the count.
Root cause. A workload mix that allocates and frees single slots in an order that never leaves adjacent gaps — the checkerboard in section 8, reached organically.
Fix. Architectural, not a bug fix. Either coarsen the allocation unit so a request cannot leave a one-slot hole, or add compaction, or allow scatter-gather placement so contiguity is no longer required. Each is a different product.
Recurrence check. A test that builds the checkerboard explicitly and asserts a request smaller than the free count cannot be placed — then asserts that the same free count in contiguous form can.
Telemetry. largest_run as a first-class metric beside free_slots. A dashboard with only the latter cannot show this.
Lab 3 — Capacity shrinks over weeks with no failures
Symptom. Usable capacity declines slowly. No device reports a fault. Restarting the orchestrator recovers some of it.
Evidence. n_leaked exceeds n_reaped and the gap grows monotonically. Several resources report a state of ORPHANED with owner ids belonging to hosts that no longer exist.
Hypothesis. The reaper is not running, or is running and failing.
Investigation. Sample n_leaked − n_reaped over time. Monotonic growth means the reaper never runs. Oscillation with an upward trend means it runs and cannot keep pace. Then check whether the reap path requires the orchestrator — if it does, lab 4 applies.
Root cause. Host losses orphan resources correctly, and the reclamation tick is either absent or gated behind something that is itself down.
Fix. Make reclamation independent of the thing whose failure creates the work. A reaper that requires the control plane to be healthy cannot clean up after a control-plane outage.
Recurrence check. A test that orphans a resource and then runs for a bounded number of cycles with no reap tick, asserting the resource has not returned — proving the liveness guarantee is conditional — followed by the tick and the return.
Telemetry. n_leaked − n_reaped as a standing gauge with an absolute alarm, plus the owner field on every orphaned resource so the dead host can be identified.
Lab 4 — Everything is healthy and nothing will start
Symptom. A whole availability zone cannot launch workloads. Every resource reports healthy. No hardware alarm anywhere.
Evidence. blocked equals the full healthy count. startable is zero. The control-plane reachability signal is down.
Hypothesis. This is an unreachable incident, not an unavailable one.
Investigation. Compare healthy resources against startable resources. If healthy is high and startable is zero, the resources are fine and the thing that grants them is not. A capacity dashboard reports these identically and a correct one does not.
Root cause. Composition introduced a dependency that did not previously exist. A host that could once start on its own now requires a grant.
Fix. Architectural: a fallback that lets previously-bound resources continue, or a cached binding that survives a control-plane restart, or an explicit acceptance that the control plane is a tier-zero dependency with the availability engineering that implies.
Recurrence check. A test that holds every resource healthy, drops the control plane, and asserts blocked equals the healthy count while no resource reports a fault.
Telemetry. Startable and healthy as two separate counters. Their difference is the blast radius of a control-plane outage, in resources and in blocked start value.
Lab 5 — A resource is bound to two hosts
Symptom. Two hosts write the same memory and corrupt each other. Both believe they own it exclusively.
Evidence. The owner field shows one host. The other host's binding record shows itself. The allocator's occupancy bit is set once.
Hypothesis. A binding survived a reset, or an allocation was granted while the resource was still bound.
Investigation. Check whether the allocator's reset clears occupancy. Check whether a bind request is accepted in the BOUND state. Reproduce by asserting reset mid-binding and immediately requesting an allocation.
Root cause. Either state that outlived reset — the allocator came back believing slots were free while a host still held them — or a missing guard on the bind path. Both produce two owners and neither reports an error.
Fix. Asynchronous reset clearing the occupancy bitmap and the owner fields, and a bind path that only accepts from FREE. Both are in the models; both have mutations covering their removal.
Recurrence check. A test that binds to host 7, requests a bind for host 9, and asserts the owner is still 7 and the bind counter did not advance. Separately, a test that asserts reset with live occupancy and asserts the bitmap comes back clear.
Telemetry. A sticky fault bit set whenever a bind is accepted in a non-FREE state. It should never assert; if it ever does, the evidence is captured even if the corruption is found much later.
Lab 6 — Bind latency doubled after a fabric expansion
Symptom. Composition is noticeably slower after adding devices. Nothing fails.
Evidence. last_latency has roughly doubled. n_rejected is still zero. n_accepted and n_ready match.
Hypothesis. The walk has more to do per step — more devices to discover, more fabric to program — so each step takes longer.
Investigation. Compare latency against the device count across the expansion. Linear growth points at the discover and program steps; a step change points at a topology change adding a hop.
Root cause. Bind latency is a function of fabric size, and the fabric grew. This is a performance property, not a correctness one — no invariant is violated and nothing is broken.
Fix. If the latency now exceeds a target, the options are parallelising discovery, caching topology between binds, or accepting it. Framing it as a bug leads to the wrong investigation.
Recurrence check. A performance regression test with an explicit target and a comment saying it is a target. A liveness test that the bind eventually completes is separate and must not be conflated with it.
Telemetry. last_latency tracked as a distribution rather than a value, because a mean hides the tail — which 29.3 section 9 covers at fleet scale.
Lab 7 — The recomposition saving never appears in the budget
Symptom. A composable deployment delivered its technical goals. The capacity saving cannot be found in any financial report.
Evidence. Recomposition count is low. Standing fabric and control-plane costs are being paid every hour. Fragmentation and rounding waste are both non-zero.
Hypothesis. The estate is paying the standing bill of section 13 and earning the per-event benefit too rarely to cover it — and the stranding of section 11 moved rather than vanished.
Investigation. Count recompositions per period. Compute standing cost against per-event benefit at the observed rate. Separately, add fabric reserve, rounding waste and unreachable capacity, and compare against the host stranding that was recovered.
Root cause. Two independent shortfalls that look like one. The flexibility is under-exercised, and the stranding relocated into the fabric.
Fix. Either raise the recomposition rate so the standing cost amortises, or reduce it — coarser fabric reservation, a granularity that rounds less, better reachability. Which of the two to attack depends on which number is larger, and the model publishes both.
Recurrence check. A test that drives a recomposition count below break-even and asserts the model reports no net benefit, and a second that drives it above and asserts it does.
Telemetry. Recompositions per period beside standing cost, and the three stranding terms reported individually rather than as a total.
Lab 8 — A held request is never served
Symptom. Occasionally a request is accepted, busy clears, and the resource never arrives.
Evidence. n_accepted exceeds n_ready by exactly the number of stuck requests. The sequencer is idle. Nothing is rejected.
Hypothesis. The pending slot was set and never consumed — a held request that the walk forgot about.
Investigation. Check whether the pending flag is clear while the walk is idle. That pair is an invariant: a held request implies the walk is running. If pending is set with the walk idle, the restart path that consumes it did not fire.
Root cause. The completion path restarts the walk for a pending request but fails to clear the pending flag, or clears the flag without restarting. Either leaves a request in limbo.
Fix. Consume and clear in the same transition, and assert the invariant continuously rather than at sample points.
Recurrence check. A continuous monitor asserting that a held request never coexists with an idle walk, running for the whole simulation rather than at chosen moments. This chapter runs exactly that monitor, and it is what proves the redundant guard in section 18 is genuinely redundant.
Telemetry. n_accepted − n_ready as a standing gauge. It should return to zero whenever the sequencer is idle; a non-zero value at idle is a lost request.
23. Design Review
Which of the three granularities is this — server, device, or sub-device?
What is the allocation unit, and what does a typical request round up to?
How long does a bind take, and what happens to a request that arrives during one?
How deep is the skid, and does anything upstream listen to reject?
What is the longest contiguous free run right now, not the free capacity?
What happens to a binding when its host dies, and what reclaims it?
Does the reclamation path depend on the control plane whose failure creates the work?
Which hosts cannot start if the orchestrator is down, and what is that worth?
Where did the host stranding go — fabric reserve, rounding, or unreachable?
How often does this estate actually recompose, and what is the standing bill?
24. How This Appears In Real Engineering
The failure is a capability word standing in for an architecture, and it survives because every party hears the granularity they were expecting.
The most common shape is section 5 straight through. A programme adopts composable infrastructure; the platform team implements device-granularity binding; an application team writes against sub-device fractions; both are consistent with the document, and the mismatch surfaces when the first request for half a device is refused.
The second is section 8 and it is the one that wastes a quarter. A capacity dashboard shows headroom and allocations fail. Nothing is faulty and nothing is full. The free capacity is real, scattered, and useless for the request shape in question — and no metric on the dashboard distinguishes a healthy pool from a fragmented one.
The third is section 9 and it is discovered slowly. Usable capacity declines over weeks. Hosts die, their bindings orphan correctly, and nothing reclaims them. Every individual event is handled properly; the aggregate is a leak.
The fourth is section 10 and it is discovered all at once. The control plane goes down and an entire zone cannot launch, with every resource healthy. The incident is filed against the memory fabric because that is what was recently added, and the cause is a dependency nobody drew on the availability diagram.
The fifth is section 13 and it is the quietest. The deployment works and the saving never appears. The estate recomposes rarely, the standing bill is paid hourly, and the stranding moved into the fabric where a different team owns it.
Four of these five are cheap to check and none is visible in the sentence that introduced the architecture.
25. Common Misconceptions
"The infrastructure is composable." At which of three granularities? Section 5.
"We allocate what the workload asks for." Rounded up to the unit. Section 6.
"Binding is instant." Five steps and fifteen cycles. Section 7.
"The pool is forty percent free." With a longest run of one. Section 8.
"A dead host releases its resources." It orphans them. Section 9.
"Everything is healthy, so everything can start." Not without the orchestrator. Section 10.
"Composition eliminates stranding." It relocates it. Section 11.
"Flexibility helps every workload." The ones whose mix varies. Section 12.
"We only pay when we recompose." The standing bill is hourly. Section 13.
26. Interview And Design-Review Questions
Architecture
1. What does "composable infrastructure" actually commit you to? Three axes: granularity — server, device or sub-device; the fabric that carries the binding; and the lifecycle, meaning what happens when the host goes away. A description giving one of the three has given a third of a system, and the two omitted are usually the ones that decide whether it works.
2. Why does granularity decide more than it looks like it does? It sets the rounding tax on every request, the fragmentation behaviour of the pool, and whether the device itself needs partitioning support. A sixteen-gigabyte unit serving one-gigabyte requests wastes fifteen-sixteenths, and a unit small enough to avoid that produces a bitmap large enough to make the allocator's scan a critical path.
3. Where does CXL sit in this? It provides a link over which a memory device can be reached by a host, which makes device-granularity binding possible without a physical recable. It does not supply the allocator, the control plane, or the reclamation path — those are system software, and they are where this chapter's failures live.
4. Why is composition an allocator problem rather than a protocol problem? Because every hard question here — rounding, fragmentation, contiguity, ownership, reclamation — is a property of how resources are tracked and handed out. The link protocol is necessary and settles none of them.
5. What is the difference between an unavailable resource and an unreachable one? Unavailable means the resource is faulty or in use. Unreachable means it is healthy and idle and the thing that grants it is down. Different incident, different fix, different blast radius — and a capacity dashboard reports them identically unless it is built not to.
6. Why should reclamation not depend on the control plane? Because the control plane's own failure is a major source of orphaned bindings. A reaper that requires it to be healthy cannot clean up after exactly the outage that created the most work for it.
7. When is composability the wrong architecture? When the estate's resource mix is fixed. The fabric and control plane are paid for hourly, the flexibility is never exercised, and every workload receives the shape it would have had from static provisioning.
8. What breaks first as you scale the slot count? The allocator's contiguity tracking. A full bitmap scan is fine at sixteen slots and untenable at thousands, which is why real allocators maintain run lengths incrementally instead of recomputing.
RTL and microarchitecture
9. What state does a bind sequencer have to remember? The current step, the position within it, and any request being held. The last of those is the one people forget, and forgetting it means a request arriving during a bind is lost.
10. Why is the pending slot one deep rather than eight? Depth does not remove the overrun, it raises the burst size that triggers it. A depth of one with a counted reject is more honest than a depth of eight with a silent drop. The design question is whether the overrun is reported, not whether it can be made rare.
11. What must reset do to a bind sequencer? Return it to idle, clear the pending slot, and zero the counters. A sequencer emerging from reset mid-walk is programming a fabric for a request nobody remembers.
12. What must reset do to an allocator? Clear the occupancy bitmap. An allocator that survives reset believing slots are free while hosts still hold them will hand out a resource twice, and nothing in the data path reports it.
13. Why does the lifecycle model retain the owner id after the host dies? Because clearing it makes an orphan indistinguishable from a free resource. The dead host id is the only evidence the leak happened, and retaining it is deliberate telemetry rather than untidiness.
14. A resource is orphaned. Is that a safety or a liveness problem? Neither by itself — it is correct behaviour. The liveness property is that it eventually returns to free, and that holds only under the assumption that a reaper runs. If the reaper never runs, no safety property is violated and the capacity is gone anyway.
15. How do you handle an allocate and a release arriving in the same cycle? Decide explicitly and document it. Different slots: both land. Same slot: pick one and assert it. This chapter applies the release last, so release wins, and the testbench asserts that ordering rather than discovering it.
16. Why take every product in thirty-two bits? A sixteen-by-sixteen multiply wraps silently in sixteen bits. A wrapped capacity figure is worse than a missing one — it is a small, plausible number where a large one belonged, and nothing flags it.
17. What is the invariant linking the pending flag and the walk state? A held request implies the walk is running. It is what makes the idle-and-pending combination impossible, and asserting it continuously is what proves a guard elsewhere in the design is redundant rather than merely untested.
18. How would you make largest_run affordable at scale? Maintain it rather than recompute it: a size-ordered free list, a buddy allocator, or a hierarchical bitmap where a coarse level summarises a fine one. All trade exact answers or update cost against the scan.
19. What is the cost of sub-device granularity in RTL terms? More slots, so a wider bitmap, a longer scan and more owner state — and it requires the device to support partitioning at all, which is a device capability rather than a fabric one.
20. Why is busy a better flow-control signal than a queue? Because it composes. A requester that respects busy never creates an overrun; a requester that fills a queue creates one that is merely deferred.
Verification
21. Why must the baseline pass before mutation testing starts? Because a mutation on a broken design still fails the same assertions, and the kill is counted for the wrong reason. A perfect mutation score proves the checkers detect changes — it says nothing about whether the thing being changed was right.
22. What is an independent oracle and why does it matter? An expected value reasoned from the specification rather than copied from the design. Five states at three cycles is fifteen — derived from the state list, not from the counter expression. If the oracle merely restates the design, the test cannot fail when the design is wrong.
23. Give an example where the oracle caught a real defect. The bind latency. The oracle said fifteen cycles; the model reported fourteen because it captured a counter before its own non-blocking increment landed. A checker built from the design's own arithmetic would have agreed with the bug.
24. How do you stop an X-valued condition passing a test? Compare against a known value rather than truthiness: c !== 1'b1 fails on X, where if (!c) does not. For data, reject unknowns explicitly before comparing — an X reduction on the result, checked first.
25. Why is a check nested inside a conditional dangerous? Because a defect that suppresses the condition also suppresses the check. A latency assertion inside if (ready) cannot fail when the bug is that ready never asserts. Latch the event in a continuous monitor and assert the flag outside any branch.
26. What is a mutation survivor and what do you do first? A mutation the tests did not detect. First classify it, before changing anything: stimulus gap, unreachable state, equivalent mutant, masking relationship, imprecise observation, missing checker, or invalid mutation. The action differs completely between them.
27. What is the question that separates an equivalent mutant from a stimulus gap? Does any legal input make the mutated expression observably different? If no, it is equivalent — withdraw it and do not manufacture stimulus. If yes, the stimulus has a hole.
28. Give an example of an equivalent mutant from this chapter. run > best loosened to run >= best. At the equality the assignment writes a value that is already there, so no input distinguishes them. It is withdrawn and not counted as a kill.
29. When is a counter-inversion mutation equivalent? When the interesting cases are exactly half the total, so inverting the condition reaches the same count. It is equivalent because of the stimulus rather than the code, and one more case of either kind makes it observable.
30. How do you verify a liveness property in simulation? You cannot prove it, so you bound it and state the assumption. Run for a defined number of cycles and assert the good thing happened. Equally important, prove the withdrawal: remove the assumption and assert the good thing does not happen, which is what shows the assumption was load-bearing.
31. What should reset testing cover beyond time zero? Reset asserted with live state. A bind in flight, an allocator with occupancy set, a resource with a live owner. Reset at time zero only proves the initial values are right.
32. How do you test invalid operations? Drive them and assert nothing changed. Release a resource nobody holds. Bind one that is already bound. Request zero slots. Each must be a no-op, and each is a place where a real design quietly does something.
33. What does functional coverage look like for the allocator? Cross the occupancy shape against the request size: full, empty, contiguous-free, fragmented-free, and request sizes below, equal to and above the longest run. The interesting bin is fragmented-free with a request the free count could satisfy.
34. What would you add to this testbench if you had more time? A constrained-random allocate/release stream with a reference model tracking the bitmap independently, and a scoreboard comparing longest-run after every operation. The directed cases here prove specific properties; random streams find the orderings nobody thought of.
35. How do you know your scripted checks are still working? Run them against known-good material. This chapter changed its assertion idiom and silently disabled three checks — two of which then reported a confident zero. A checker that cannot parse its input reports clean, not unreadable.
Debugging and silicon
36. A host's request vanishes under burst load. Where do you look first? n_accepted against n_ready against n_rejected. If accepted plus rejected equals what was issued, nothing was lost at the hardware boundary and the request was refused — which means the requester is not handling reject.
37. The pool is forty percent free and allocations fail. What is the one metric that explains it? Longest contiguous run. Free capacity cannot distinguish a healthy pool from a fragmented one, and no other counter exposes contiguity.
38. Capacity declines over weeks with no faults. What do you measure? Leaked minus reaped. Monotonic growth means the reaper never runs; oscillation with an upward trend means it runs and cannot keep up.
39. An entire zone cannot start and every resource is healthy. What is the diagnosis? Healthy against startable. High healthy with zero startable is a control-plane outage, not a resource problem — the resources are unreachable rather than unavailable.
40. Two hosts corrupt the same memory. What are the two candidate causes? A binding that survived a reset, so the allocator believes a held slot is free; or a missing guard allowing a bind while already bound. Both produce two owners and neither raises an error on its own.
41. Bind latency doubled after expanding the fabric. Is that a bug? No. Bind time is a function of fabric size and the fabric grew. It may now miss a target, which is a performance problem with performance remedies. Filing it as a correctness bug sends the investigation to the wrong place.
42. If you could keep only one counter from this chapter, which? Longest contiguous run. Free capacity can be approximated from other signals; contiguity cannot be inferred at all, and its absence is the failure that looks most like having no failure.
43. What sticky fault bit would you add? One set whenever a bind is accepted in a non-free state. It should never assert. If it ever does, the evidence survives long after the corruption is found.
Performance and economics
44. Bind latency is fifteen cycles. Is that a requirement? No, it is a measurement against a target. Correctness is that the bind completes exactly once and that no resource gets two owners. Confusing the two turns a capacity-planning conversation into a bug hunt.
45. When does a composable estate pay for itself? When recompositions are frequent enough to amortise the standing fabric and control-plane cost, and when the stranding recovered from hosts exceeds what is newly stranded as fabric reserve, rounding waste and unreachable capacity. Both conditions, independently.
46. Composition recovered nine hundred gigabytes. Why might that not be a saving? Because the fabric reserved four hundred, rounding wasted three hundred and two hundred and sixty are unreachable — nine hundred and sixty newly stranded against nine hundred recovered. The stranding moved rather than vanished, and only the net matters.
27. Exercises
1 — Architecture. An estate composes at device granularity over a fabric with no reclamation path. Identify which of the six sign-off conditions it fails and argue which failure costs most in the first year of operation.
2 — Quantitative. A pool is carved into 32 GB units and serves requests of 8, 40, 64 and 100 GB in equal proportion. Compute the rounding waste per request class and the weighted average tax. What single unit size minimises it, and what does that choice cost the allocator?
3 — RTL implementation. Extend the bind sequencer with a two-deep pending queue. State what must change in the reject logic, what new invariant replaces "a held request implies the walk is running", and whether the overrun becomes less likely or merely less frequent.
4 — Assertions. Write the assertions for the allocator that would have caught a reset which failed to clear the occupancy bitmap. State which is safety and which is liveness, and explain why one of them cannot be written as a simple equality.
5 — Testbench design. The lifecycle model's liveness property depends on a reaper. Design the stimulus that proves the guarantee is conditional rather than absolute, and explain why asserting only the successful reclamation would be insufficient.
6 — Waveform analysis. Given Figure 3, state what an observer monitoring only free and a health check would conclude about the resource between cycles 2 and 5, and what evidence distinguishes that state from genuine availability.
7 — Debugging. A fleet reports n_accepted minus n_ready sitting at 3 with the sequencer idle and n_rejected at zero. Produce a hypothesis, the next measurement you would take, and the RTL defect most likely responsible.
8 — Coverage design. Define a functional coverage model for the allocator that would have found the checkerboard fragmentation case without anybody thinking of it. Specify the bins and the cross, and identify which bin is the interesting one.
28. Summary
"Composable" names no granularity, fabric or lifecycle — three architectures answer to the word and they are not interchangeable.
The allocation unit rounds every request up, and the tax is paid per request rather than once.
Composition is a sequence, not an assignment: five steps, fifteen cycles, and a request arriving during one must be held or counted as rejected.
Free capacity is not allocatable capacity — eight scattered slots cannot place a four-slot request that four adjacent ones can.
A binding outlives the host that asked for it, retaining the dead owner's id, and returns only if a reaper runs.
The control plane is a new dependency, and it makes healthy resources unreachable rather than unavailable.
Composition moves stranding into fabric reserve, rounding waste and unreachable capacity — and only the net is a saving.
A varying resource mix gains and a fixed one pays for nothing.
Flexibility has a standing bill that is paid whether or not anything is ever recomposed.
Six bits, and "it is composable" is one of them. One case study of eight is sound; the composable view counts seven.
Continue learning
Related tutorials
- Related topic
Shared Memory Pools
A pool is not several hosts sharing memory. It is a capacity inventory with correctness obligations: counted, owned, placed, and provably conserved. Why free capacity is not allocatable capacity, and what hardware has to hold to make the distinction.
- Related topic
CXL 2.0 Memory Pooling
Pooling is the feature CXL 2.0 exists for. This chapter builds the capacity saving, logical-device limits, block granularity, fragmentation, rebind cost, stranded capacity, hot-add ordering, the pool's own blast radius, fabric-manager accounting divergence and the assembled model.
- Related topic
AI Training Clusters in Practice
A training cluster has several memory tiers that differ by orders of magnitude. This chapter builds the tier claim, the capacity wall, the bandwidth floor, residency, the scale boundary, offload economics, the stall cost, where the attach actually wins and why the fast-tier floor does not move.
- Related topic
Memory Expansion Cards
A card is a link-attached device with its own controller, not a slot. This chapter builds the latency tier, why bandwidth comes from the link rather than the media, page placement, the interleaved failure domain, promotion cost, the slot and power bill, where a card wins and what it is really compared against.
Standards & specifications
- Governing standard
- CXL Specification (CXL Consortium)(opens CXL Consortium in a new tab)
Defines CXL.io, CXL.cache and CXL.mem, and the coherence and memory-pooling behaviour built on them. System design and deployment topology are not mandated.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the CXL curriculum.
