Multi-Host Systems

12.2 built an allocator that decides where capacity goes and runs the lifecycle that makes it usable. Every allocation it produced was anonymous.

This chapter gives them owners, and then asks what happens when an owner stops existing.

1. The Engineering Problem — The Owner Is Not A Constant

A single-host pool can treat ownership as a formality. There is one answer to whose is this, it never changes, and nothing has to be checked.

Once several hosts share the pool, ownership becomes the load-bearing structure, and three things about it are uncomfortable.

Ownership is reused. Slot 5 belongs to host 1, then to nobody, then to host 1 again. Those are different allocations that share a name, and an event belonging to the first must never be applied to the second. That is not a hypothetical: it is what happens every time a completion is in flight while its allocation is returned.

Ownership is enforced, not just recorded. A table saying host 1 owns units 0 through 7 is a claim. Something on the access path has to turn it into a rule, and that something has to distinguish this address belongs to nobody from this address belongs to somebody else — because the second is a security event and the first is a configuration error.

And owners disappear. A host can leave cleanly, reset, or simply stop responding. Its allocations do not vanish with it; they become capacity that is owned by nobody and grantable to nobody, which is worse than either. Somebody has to notice, and somebody has to reclaim.

None of this existed in 12.2, where the allocator's only correspondent was itself.

2. The One-Sentence Model

Identity is what makes reuse safe. A slot, a host and a generation together name one allocation exactly once in the life of the pool — and any event that cannot produce all three is an event that must not be applied.

Call it name it, or lose it. Owner alone is not enough, because the same host is handed the same slot again. Slot alone is not enough, because slots are reused. Only the triple is unique.

3. What This Chapter Owns

Deferred:

The boundary with 11.3 is the one to state clearly. That chapter shared one device among a small, fixed set of hosts with a table written once at configuration time. Here the host set is not fixed — hosts join, drain, fail and are replaced — and the allocations move between them at runtime. Everything in this chapter exists because that host set changes.

And nothing here is about coherency. Every allocation below has exactly one owner at a time. Two hosts reading and writing the same region under a coherence protocol is Modules 13 and 14, and this chapter's isolation guard exists precisely to make sure that situation never arises by accident.

4. Teaching-model boundary

Publicly available CXL material describes pooling in terms of capability rather than mechanism, so this chapter does not attempt to reconstruct a specified protocol. No pool-management commands, no host-identity encodings, no reassignment timing and no recovery sequences are attributed to CXL.

What does transfer is the obligation set, and it is not CXL-specific. Any system that hands a reusable resource to changing participants has to solve identity, isolation, accounting and the disappearance of a holder. The mechanisms below are one coherent way to do it, built here so the invariants can be verified rather than asserted.

5. RTL 1 — Owner, Slot, Generation

module alloc_table #(parameter int SLOTS = 8, parameter int GEN_BITS = 3) (
  input  logic clk, rst_n,
  input  logic       alloc_en, input logic [2:0] alloc_slot, input logic [1:0] alloc_host,
  input  logic       ret_en,   input logic [2:0] ret_slot,
  input  logic [1:0] ret_host, input logic [GEN_BITS-1:0] ret_gen,
  output logic       alloc_grant, ret_grant, gen_wrap,
  output logic       stale_gen_err, wrong_owner_err, double_alloc_err, unknown_ret_err,
  output logic [7:0] n_alloc, n_ret, n_stale
);
  logic owner_ok, gen_ok;
  assign owner_ok = valid_q[ret_slot] && (owner_q[ret_slot] == ret_host);
  assign gen_ok   = valid_q[ret_slot] && (gen_q[ret_slot]   == ret_gen);
 
  assign alloc_grant = alloc_en && !valid_q[alloc_slot];
  // BOTH identities must match. Owner alone is not enough: the same host can
  // be handed the same slot again, and its own stale event would be accepted.
  assign ret_grant   = ret_en && owner_ok && gen_ok;

The generation advances on every reuse of the slot, so the same slot carries a different number each time it is handed out. Measured: slot 5's first use is generation 1, and its next use is generation 2.

Four rejections, each with its own counter, because they send an engineer to four different places:

Measured: allocating an occupied slot yields grant=0 with double_alloc_err=1, and the original owner is untouched.

The table also publishes alloc_gen — the generation a grant would carry — before the grant happens, because whatever issues the allocation has to record the identity it is about to hand out. Asserting that this matches the slot's generation afterwards is what killed the mutation that reported the current generation instead of the next one.

6. RTL 2 — Late Events Are Why Any Of This Matters

Without delay there is no such thing as a stale event. The completion pipeline is what makes the hazard reachable rather than theoretical.

module late_event #(parameter int DELAY = 4, parameter int GEN_BITS = 3) (
  input  logic clk, rst_n,
  input  logic       in_valid, input logic [2:0] in_slot,
  input  logic [1:0] in_host,  input logic [GEN_BITS-1:0] in_gen,
  output logic       out_valid, output logic [2:0] out_slot,
  output logic [1:0] out_host,  output logic [GEN_BITS-1:0] out_gen,
  output logic [3:0] in_flight
);

A completion is issued carrying the identity that was live when it was issued. Four cycles later it arrives — and by then the slot has been returned and handed back to the same host at a new generation. Measured: the late completion is rejected as stale and the new allocation survives it.

7. Waveform — One Slot, Two Lives

Transcribed from the printed cycle trace.

Follow the gen row against owner. The owner never changes — it is host 3 in both lives of the slot. Only the generation distinguishes them, and at cycle 7 that is the entire defence: a return quoting generation 1 arrives while generation 2 is live, stale_err rises, and owner_err stays flat because there is nothing wrong with the owner.

The valid row shows what would happen without the guard. The return would be accepted, valid would fall, and host 3 would lose an allocation it is actively using — to its own stale message.

8. RTL 3 — Hosts Have Lifecycles Too

module host_registry #(parameter int HOSTS = 4, parameter int DRAIN_LIMIT = 8) (
  input  logic clk, rst_n,
  input  logic       cmd_en, input logic [1:0] cmd_host,
  input  logic [1:0] cmd,                        // 0=join 1=ack 2=leave 3=fail
  input  logic [3:0] held0, held1, held2, held3,
  output logic [2:0] st0, st1, st2, st3,
  output logic [2:0] n_active,
  output logic       gone_pulse, output logic [1:0] gone_host,
  output logic [3:0] gone_held,
  output logic       drain_stuck_err
);

Five states, and the distinction that matters is between two ways of leaving.

A clean departure waits. The host says it is leaving, enters DRAINING, and stays there until it has released everything. Measured: it held two allocations, remained draining while it held them, and became GONE with gone_held=0 — no orphans.

A failure does not wait. Measured: a host failing while holding five allocations is gone at once and reports gone_held=5 — five orphans created by one event. Whatever it held is now the pool's problem.

And a drain that never finishes is a real state, not an impossible one. A host that says it is leaving and then stops releasing will sit in DRAINING forever. drain_stuck_err names it after a bounded wait — measured at 1. Without it the pool has capacity in a state with no exit and no alarm, and the only symptom is a slow decline in what can be granted.

9. RTL 4 — Recording Ownership Is Not Enforcing It

module range_guard #(parameter int SLOTS = 4) (
  input  logic       acc_valid, input logic [1:0] acc_host, input logic [5:0] acc_addr,
  output logic       acc_permit, acc_deny, hit,
  output logic [7:0] n_permit, n_unmapped, n_foreign
);
  // Unmapped and foreign are different failures. Unmapped means the address
  // belongs to nobody; foreign means it belongs to somebody else, which is an
  // isolation violation and a security event.
  logic unmapped, foreign;
  assign unmapped   = acc_valid && !found;
  assign foreign    = acc_valid &&  found && (fowner != acc_host);
  assign acc_permit = acc_valid &&  found && (fowner == acc_host);
  assign acc_deny   = unmapped || foreign;

The boundaries are tested from both sides, one unit at a time:

Over the run: 5 permitted, 3 foreign, 3 unmapped.

Keeping the two denials apart is not bookkeeping. An unmapped access is usually a configuration error — a host addressing capacity nobody assigned. A foreign access is a host reaching into another host's memory, which is an isolation failure and is reported to different people with different urgency. A single deny counter tells you neither, and the rate of one hides the rate of the other.

Clearing a mapping removes the permission with it, measured: an address that was permitted becomes unmapped the cycle after its mapping is cleared.

10. RTL 5 — The Pool Has Capacity, And Not For You

  // Pool capacity is tested first: a request the pool cannot satisfy at all is
  // not a quota problem, and telling an operator to raise a quota that would
  // not have helped is worse than saying nothing.
  logic pool_short, quota_short;
  assign pool_short  = req_en && (req_n > pool_free);
  assign quota_short = req_en && !pool_short
                       && ((used[req_host] + req_n) > quota[req_host]);
  assign fail_pool   = pool_short;
  assign fail_quota  = quota_short;
  assign grant       = req_en && !pool_short && !quota_short;

The measurement that matters:

Over the run, 6 grants, 3 quota refusals, 1 pool refusal, with exactly one outcome per request.

A host that is refused sees a refusal either way. The operator does not: one of these is fixed by raising a limit and the other is fixed by buying memory, and a merged counter recommends the wrong one half the time.

Exactly filling a quota is legal — a host may take precisely its remaining headroom, and is refused on the next unit. The per-host usage is a single combined next-state, because a grant and a release can name the same host in the same cycle; measured, a host granted 1 while releasing 3 lands correctly at 7.

11. RTL 6 — Handing An Allocation From One Host To Another

Reassignment is where every mechanism in this chapter is needed at once.

  // The old owner keeps its access only until REVOKE. The new owner gets it
  // only at ENABLE. There is no state in which both are true, which is the
  // property the whole machine exists to guarantee.
  assign old_enabled = (st_q == S_IDLE) || (st_q == S_REVOKE);
  assign new_enabled = (st_q == S_ENABLE);

The testbench asserts on every cycle that the two are never both high, and the measured run reports overlap_err=0 across a full reassignment that quiesced for 5 cycles and scrubbed for 3.

Note where new_enabled is not asserted: at GRANT. The record exists there — the new owner is written into the table — and the access still does not. Recording ownership and enabling access are separate steps, and collapsing them is the same mistake 12.2 found between OWN and ACTIVATE, now with a second host on the other side of it.

post_revoke_err fires when the old owner attempts an access after revocation — measured at 1. It distinguishes the revoke did not take effect from the scrub did not work, which look identical from the new owner's side and are different bugs.

12. RTL 7 — Capacity Owned By Nobody

      // A departure orphans every allocation that host still held, in one
      // step. Walking them one at a time would leave a window in which some
      // of a dead host's allocations are still recorded as live.
      if (gone_pulse) begin
        for (i=0;i<SLOTS;i=i+1)
          if (v_q[i] && (o_q[i] == gone_host)) orphan_map[i] <= 1'b1;
        n_orphan_total <= n_orphan_total + {4'd0, newly};
      end

Orphans are reclaimed one at a time in a fixed order, and the state carries an age:

      // Age is a property of the ORPHAN STATE, not of any one slot: it counts
      // how long the pool has held capacity it can neither use nor give away.
      if (found) begin
        age_q <= age_q + 8'd1;
        if (age_q >= AGE_LIMIT[7:0]-8'd1) orphan_stuck_err <= 1'b1;
      end else begin
        age_q <= 8'd0;
      end

Measured: three orphans created in one step, all three reclaimed, and the age resets to zero once the state clears. When nothing reclaims them, the age reaches the limit and orphan_stuck_err=1 — because capacity stranded indefinitely needs a name, not just a number.

13. RTL 8 and 9 — Fairness, And Owners Who Should Not Exist

Two hosts asking for the last extent is the normal case.

  // Explicit four-way round robin from the pointer. Written out rather than
  // looped: the two-bit pointer arithmetic wraps by construction, and there is
  // no index arithmetic on a loop variable to get wrong.
  always_comb begin
    found = 1'b0; sel = 2'd0;
    if (avail) begin
      if      (req[ptr_q])          begin found = 1'b1; sel = ptr_q;          end
      else if (req[ptr_q + 2'd1])   begin found = 1'b1; sel = ptr_q + 2'd1;   end
      else if (req[ptr_q + 2'd2])   begin found = 1'b1; sel = ptr_q + 2'd2;   end
      else if (req[ptr_q + 2'd3])   begin found = 1'b1; sel = ptr_q + 2'd3;   end
    end
  end

Measured: two hosts contending for ten grants split them 5 and 5; four hosts saturating the arbiter were each served 3 times with a worst wait of 3 cycles, and multi_grant_err stayed 0 throughout.

The wait counter is a run length, not a total — a standing lesson from earlier modules that applies exactly here. A host served plenty overall can still be waiting far too long right now, and only the run length shows it.

The last model checks something no single-host pool can get wrong:

      // A host that is neither active nor draining must own nothing.
      if ((st[gi] != H_ACTIVE) && (st[gi] != H_DRAINING) && (tab[gi] != 6'd0)) begin
        ghost = 1'b1; gh3 = gi;
      end
      if (usd[gi] != tab[gi]) drift = 1'b1;

A ghost owner is a host that has left and is still recorded as owning capacity. Measured: setting a host to ABSENT while the table still credits it raises ghost_owner_err and names the host. A quota drift is the quota accounting disagreeing with the table — measured separately.

The draining case is the one that needs care, and it needed a deliberate test: a DRAINING host legitimately still owns capacity, because that is what draining means. Treating it as a ghost produces a false alarm on every clean departure, and the mutation that does so survived the first run precisely because no test ever sampled a draining host.

14. Quantitative Reasoning

Generation width is a protection window, not a formality.

protected reuses = 2 ** GEN_BITS

With the three-bit generation used here, the guard distinguishes eight successive uses of a slot. On the ninth, the generation aliases back and an event old enough to have survived eight reuses becomes acceptable again. Measured: after eight reuses the counter wraps and says so.

The width has to be chosen against the maximum event lifetime, not against convenience:

GEN_BITS  >=  log2( max_event_lifetime / min_reuse_interval )

A completion that can survive four cycles against a slot that can be reused every three cycles needs to distinguish at least two lives; a management operation that can be outstanding for milliseconds against a slot reused microseconds apart needs far more. The wrap must be reported either way, because a silent wrap converts a guaranteed rejection into a probabilistic one.

Blast radius is a per-host quantity. A host's failure orphans exactly what it held — measured, 5 allocations from one event — and the pool cannot bound that number without recording ownership per allocation. This is the multi-host analogue of the placement argument in 12.1: concentration decides exposure.

Quota is not fairness, and neither is service count. Four hosts saturating the arbiter were each served 3 times, which is fair by count. Fairness by waiting is the worst run length, measured at 3 cycles and bounded by the host count — and a scheme can be perfectly fair by count while one host waits far longer than the others in bursts.

Usable capacity under host churn:

grantable = pool_free  -  orphaned_not_yet_reclaimed  -  reserved_mid_lifecycle

Both subtractions are invisible to a naive free-capacity counter. The orphan age is what makes the first one bounded rather than permanent.

15. Assertions

Written as SystemVerilog for the reader, executed procedurally — see section 17.

A return must match owner and generation.

property p_identity_complete;
  @(posedge clk) disable iff (!rst_n)
    ret_grant |-> (owner_q[ret_slot] == ret_host) && (gen_q[ret_slot] == ret_gen);
endproperty

A stale event never also grants.

property p_stale_never_grants;
  @(posedge clk) disable iff (!rst_n)
    stale_gen_err |-> !ret_grant;
endproperty

Reuse advances the generation.

property p_gen_advances;
  @(posedge clk) disable iff (!rst_n)
    alloc_grant |=> (gen_q[$past(alloc_slot)] != $past(gen_q[$past(alloc_slot)]));
endproperty

A slot is never handed to a second owner.

property p_single_owner;
  @(posedge clk) disable iff (!rst_n)
    (alloc_en && valid_q[alloc_slot]) |-> !alloc_grant;
endproperty

A host may only reach its own range.

property p_isolation;
  @(posedge clk) disable iff (!rst_n)
    acc_permit |-> (hit_owner == acc_host);
endproperty

A quota refusal implies the pool had the capacity.

property p_quota_implies_capacity;
  @(posedge clk) disable iff (!rst_n)
    fail_quota |-> (req_n <= pool_free);
endproperty

The two owners are never enabled together.

property p_no_overlap;
  @(posedge clk) disable iff (!rst_n)
    !(old_enabled && new_enabled);
endproperty

A departed host owns nothing.

property p_no_ghost;
  @(posedge clk) disable iff (!rst_n)
    sample_en && (st[h] != H_ACTIVE) && (st[h] != H_DRAINING) |-> (tab[h] == 0);
endproperty

At most one grant per cycle.

property p_one_grant;
  @(posedge clk) disable iff (!rst_n)
    $countones(gnt) <= 1;
endproperty

119 assertion sites — 113 across the nine models and 6 in the waveform trace. All pass.

16. Mutation Testing

57 mutations injected, 57 killed, 0 surviving.

Representative kills:

Seven mutations did not die on the first run. Not one was a missing checker in the design.

The first is the most instructive. A test that violates two rules at once cannot tell you which rule caught it. The wrong-owner test also quoted a dead generation, so a design with no owner check at all rejected it correctly and looked fine. Isolating each rule requires a stimulus that breaks exactly one.

The taxonomy holds for a seventh consecutive batch: stimulus gaps, unreachable or unobserved checkers, equivalent mutations, and late sampling. Never a checker that was simply absent.

17. Verification Strategy

Tool reality. Icarus Verilog 13.0 — no concurrent SVA, so every property above has an executable procedural counterpart. It also rejects an enum-valued ternary without a cast, and emits inferred-sensitivity warnings for part-selects inside always_comb; the models here move the final part-select to a continuous assign and use sized loop counters so the compile is silent.

Independent oracles.

Coverage. The points that matter are host state, reuse depth, event age relative to reuse, quota position, and the concurrency shape of the cycle. The crosses worth driving are reuse crossed with event age — the only way to reach the stale-generation case deliberately — and host state crossed with capacity held, which is what reaches both the orphan path and the ghost-owner check. A cross of host against slot index is noise.

The scoreboard for a multi-host pool is keyed on the triple, not on the slot. Two entries with the same slot and different generations are different allocations, and a scoreboard keyed on slot alone silently merges them — which is the same defect the design is being tested for, reproduced in the testbench.

18. Synthesis and Implementation Reality

Identity is width, and width is everywhere. Each allocation entry carries an owner field and a generation field, and both are replicated across every entry:

table bits = SLOTS x (1 + log2(HOSTS) + GEN_BITS)

Generation width trades directly against area, and section 14 shows it also trades against safety. This is a real design decision, not a default.

Every event carries identity too. The completion pipeline stores slot, host and generation per stage, so its cost scales with the delay it models. A deeper pipeline is a longer window in which a stale event can exist and more storage to describe the events in it.

The range guard is the timing problem. It compares an access address against every installed range in parallel, which is a CAM-like structure on the access path — the most latency-sensitive place in the design. Practical implementations reduce it: fewer, larger ranges; a base-and-mask form that replaces a comparison with a masked equality; or a lookup indexed by high address bits. Each narrows what placements are expressible, which is a policy cost paid for timing.

The registry and the arbiter are cheap — a handful of states and a two-bit pointer. The reconciliation scan grows with host count and is a candidate for sampling rather than continuous evaluation, exactly as in 12.1.

No gate counts are offered. The structural claims — table width grows with the log of the host count plus the generation width, guard cost grows with the number of simultaneously enforceable ranges — hold regardless of process.

19. Silicon Observability

n_stale deserves a note. A non-zero stale count is not a bug — it is the guard doing exactly what it exists for, and a pool reporting zero stale events under load is more likely to have a broken counter than a perfect design. What matters is the rate, and gen_wrap, which says the guard has stopped being a guarantee.

20. Debug Lab

  host 1 own gen-1 return     : stale_gen_err=1 wrong_owner_err=0

assign ret_grant = ret_en && owner_ok && gen_ok;

assign wrap_now = alloc_grant && (&gen_q[alloc_slot]);
if (wrap_now) gen_wrap <= 1'b1;    // sticky: a pulse is unobservable

  host 3 fails outright      : orphaned=5 allocations

if (gone_pulse) begin
  for (i=0;i<SLOTS;i=i+1)
    if (v_q[i] && (o_q[i] == gone_host)) orphan_map[i] <= 1'b1;
  n_orphan_total <= n_orphan_total + {4'd0, newly};
end

  host 2 disappears           : orphans=3  total recorded=3

always_comb begin
  newly = 4'd0;
  for (bi = 4'd0; bi < SLOTS[3:0]; bi = bi + 4'd1)
    if (gone_pulse && v_q[bi] && (o_q[bi] == gone_host) && !orphan_map[bi])
      newly = newly + 4'd1;
end
// ... then, once:
n_orphan_total <= n_orphan_total + {4'd0, newly};

  access checks               : permit=5 foreign=3 unmapped=3

assign foreign    = acc_valid &&  found && (fowner != acc_host);
assign acc_permit = acc_valid &&  found && (fowner == acc_host);

  host 0 at quota, pool has 32 free : fail_quota=1 fail_pool=0

assign pool_short  = req_en && (req_n > pool_free);
assign quota_short = req_en && !pool_short
                     && ((used[req_host] + req_n) > quota[req_host]);

  reassign: quiesced 5 cycles, scrubbed 3, overlap_err=0
  access after revoke               : post_revoke_err=1

assign old_enabled = (st_q == S_IDLE) || (st_q == S_REVOKE);
assign new_enabled = (st_q == S_ENABLE);

  departed host still owning  : ghost_owner_err=1 host=1
  quota vs table              : quota_drift_err=1

if ((st[gi] != H_ACTIVE) && (st[gi] != H_DRAINING) && (tab[gi] != 6'd0)) begin
  ghost = 1'b1; gh3 = gi;
end

21. Design Review

What a reviewer should attack first.

The identity on the event, not just in the table. Ask what a completion carries. If it carries a slot and a host but not a generation, the design cannot reject a stale event from a host that was handed the same slot again — and that is a busy host's normal behaviour, not a corner case.

The generation width. Ask for the maximum event lifetime and the minimum reuse interval, and check the width against their ratio. Then ask what happens on wrap. If the answer is "it will not wrap", ask what reports it when it does.

The enforcement point. A range recorded in a table is not enforcement. Ask where an access is actually checked, and whether a denial distinguishes foreign from unmapped.

The failure path, separately from the drain path. Ask what happens to allocations when a host stops responding rather than leaving. If the answer describes a drain, the design has no failure path.

The orphan bound. Ask what limits how long capacity can sit orphaned. If nothing does, the pool has a state with no exit.

Any counter incremented inside a loop. This batch found one. It is invisible with a single matching item and under-reports in proportion to severity.

What is deliberately not here. No coherency — every allocation has exactly one owner at a time, and two hosts sharing a region under a coherence protocol is Modules 13 and 14. No physical placement, enclosures or device failure domains — 12.4. No fabric manager, discovery or topology — Module 15.

22. How This Appears in Real Engineering

In architecture review, the argument that recurs is whether generations are worth their width. They cost bits in every table entry and every in-flight event, and the failure they prevent is rare and intermittent. The answer is that the failure is rare, silent, and produces a host losing an allocation it is actively using — which is indistinguishable from memory corruption from the host's side.

In RTL design, the two recurring defects are both in this chapter. A validity check that tests one field where two are needed, and an accumulation written as a nonblocking increment inside a loop. Neither is exotic and both survive light testing.

In verification, the trap is a test that violates two rules at once. It passes against a design missing either check, and it proves nothing about either. Every identity rule needs a stimulus that breaks exactly that rule.

In bring-up, the ghost owner is the one that costs days. Capacity slowly stops being grantable, every live host's accounting is correct, and nothing is wrong except that the registry and the table disagree about a host that left.

In operations, the question that gets asked is why can host 3 not allocate when the pool is half empty — and the answer is either a quota, an orphan backlog, or a reservation stuck mid-lifecycle. Three different counters, three different fixes, and a pool that publishes only free capacity answers none of them.

23. Common Misconceptions

"The owner field is the identity." It names who, not which. The same host holding the same slot twice needs a generation to tell its two allocations apart.

"Generations only matter when capacity moves between hosts." Backwards. A cross-host stale event is caught by the owner check; the generation is what catches same-host reuse, which is the more common pattern.

"A stale-event counter reading zero means the design is clean." Under load it more likely means the counter is broken. Completions racing reuse is normal, and the guard rejecting them is the mechanism working.

"A generation counter cannot wrap in practice." It wraps after exactly two-to-the-N reuses of one slot, and hot slots churn. Measured here after eight reuses of a three-bit counter.

"A host leaving and a host failing are the same event." One waits until the host has released everything; the other does not wait for anything. Measured, a clean departure produced zero orphans and a failure produced five.

"Recording ownership enforces it." A table is a claim. Something on the access path has to turn it into a rule, and it has to distinguish a foreign access from an unmapped one.

"Quota is fairness." Quota is a ceiling. Four hosts served three times each was fair by count while the worst wait was still three cycles, and neither number is the other.

"An orphan will get cleaned up eventually." Only if something bounds it. Without an age and an alarm, orphaned capacity is a state with no exit and no symptom other than a slow decline.

24. Interview Reasoning

25. Exercises

1. Calculation. A slot can be reused every 200 ns. A management operation can remain outstanding for 12 µs. Compute the minimum generation width that keeps rejection guaranteed, then state the table cost for 4096 slots at that width and at one bit less.

2. Analysis. A pool reports a steady rate of stale events and no wrap. A second pool reports zero stale events under identical load. Explain which is more likely to be correct, what you would measure to confirm it, and what a third reading — stale events plus a set wrap alarm — would mean.

3. RTL task. Extend alloc_table so a return may be issued by a designated reclaim agent on behalf of a departed host. State exactly which identity checks must still apply, which must be relaxed, and the new failure mode the relaxation introduces.

4. Assertion task. Write the property proving that a departed host owns no capacity. Then explain why it must exempt the draining state, and construct the sequence on which the unexempted version produces a false alarm.

5. Design task. Replace the range guard's magnitude comparisons with a base-and-mask scheme. State what placements become inexpressible, how the allocator must change to work within them, and what the guard now costs on the access path.

6. Testbench design. Design the stimulus that proves the owner check and the generation check are each doing work. Explain why a single test that violates both rules proves neither, and give the minimum pair of sequences required.

7. Debug task. A host failure orphaning twelve allocations is reported as one. Give the defect class, explain why it is invisible when hosts hold a single allocation each, and state the general rule you would add to code review.

8. Design review. A colleague proposes reclaiming a failed host's allocations immediately, arguing that a host that is gone cannot possibly still be issuing accesses. Give the strongest version of that argument, name the mechanism from 12.2 that it discards, and describe the failure it produces.

26. Summary

Identity is what makes reuse safe.

• Slot plus owner does not name an allocation. Slot 5 was host 1's twice — generation 1 then generation 2 — and only the generation tells the two apart.
• Generations exist for same-host reuse. Host 1's own generation-1 return arriving against generation 2 was rejected with stale_gen_err=1 and wrong_owner_err=0: the owner check could not have caught it.
• The guard has a finite window. Three bits protect eight reuses; the ninth aliases, and the wrap is reported as a sticky alarm rather than a pulse.
• Late events are why any of this matters. A completion issued four cycles earlier arrived after the slot had been returned and reissued, and was rejected while the new allocation survived.
• Leaving and failing are different events. A clean drain produced zero orphans; a failure produced five in one instant, and a drain that never finishes is named by drain_stuck_err rather than waited on forever.
• Recording ownership is not enforcing it. Measured 5 permitted, 3 foreign, 3 unmapped — and foreign versus unmapped is an isolation event versus a configuration error.
• Quota is not capacity. A host at its quota with 32 units free reports fail_quota with fail_pool clear, and the two refusals have opposite remedies.
• Reassignment never enables both owners. Five cycles of quiesce, three of scrub, overlap_err=0 — and the new owner is still not enabled at GRANT, where its record already exists.
• Orphans must be bounded. Three orphans created in one step and reclaimed; an unreclaimed one raised orphan_stuck_err at the age limit.
• A departed host must own nothing, and only a reconciliation between the registry and the table can say so — with the draining state exempted, or it fires on every clean departure.
• Verification: 119 assertion sites, 57 of 57 mutations killed, zero surviving. The baseline found a nonblocking increment inside a loop that under-reported orphan counts in proportion to the severity of the failure, and a demonstration that proved the owner field rather than the generation guard.

Next: 12.4 Datacenter Architecture, which keeps every mechanism in this chapter and changes the one thing assumed throughout it — that the pool is in one place. Physical placement, enclosures, reachability, and what a device failure costs when the pool spans a rack.