Multi-Tenant Environments

19.3 built five boundaries between two tenants on one device. Everything in it is necessary and none of it scales by itself.

A hyperscaler does not run two tenants on one device. It runs thousands of tenants across thousands of devices, placed by a scheduler, sized against quotas that deliberately sum past the capacity, sharing bandwidth that nothing physically partitions, and reclaimed constantly as workloads come and go. Every one of those is a decision that isolation cannot make for you — and every one of them can break a tenant on a device where all five boundaries of 19.3 hold perfectly.

1. The Engineering Problem — Isolation Does Not Scale By Itself

Six things separate a working multi-tenant fleet from a set of isolated pairs.

Placement is a policy question, not a capacity question. A tenant that fits on a device is not necessarily a tenant that may share it, and a scheduler that only asks whether it fits will co-locate things that were never meant to meet. Section 5.

Quotas that sum past the pool are the normal case, not a mistake. Oversubscription is how a pool earns its cost, and it works until every tenant uses what it was promised at once. Section 6.

Bandwidth is shared and nothing partitions it physically. One tenant's demand is another tenant's latency, and a share is only a share if an arbiter enforces it. Section 7.

A counter that does not name a tenant cannot be acted on. Aggregate telemetry tells an operator that something is wrong and nothing about who to talk to. Section 8.

Reclaiming capacity is an operation with a notice period. Taking memory from a running tenant without one is a fault the tenant has no way to handle. Section 9.

And how many tenants share a device is an availability decision. Packing them densely is efficient right up to the first device failure. Section 11.

This chapter against 19.3, stated precisely. That one owns what keeps two tenants apart on one device. This one owns what a fleet needs on top of that — and section 16 shows an environment where isolation is perfect and five other properties are not.

2. The One-Sentence Model

A multi-tenant environment is sound when its tenants are isolated, placed by policy rather than by fit, held to their bandwidth shares, individually attributable in telemetry, reclaimed with notice, and spread so that one device failure is not a fleet event — and every defect below is an environment with isolation and one of the other five missing.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model here is a small synchronous block that isolates one property. A real fleet control plane is a distributed system with a scheduler, a placement database, a fabric manager per rack and a telemetry pipeline, and none of that is reproduced. What the models do reproduce is the arithmetic each of those components has to get right, and the shortcut each of them is tempted by.

Each model is built twice, a correct build and a broken build selected by a parameter, and every broken build is a real design — usually one that was correct for a single-tenant fleet and was never revisited.

Figure 1 — Every device in the pool satisfies all five boundaries of 19.3. The dashed path is still open, and the telemetry edge is the one that decides whether an operator can see anything at all.

5. RTL 1 — Placement Is A Policy Question

The first model is the scheduler's admission decision, and it has two conditions that people routinely collapse into one.

// RTL 1 - admission. A tenant may only be placed on a device that has room for
// it AND that its policy permits it to share.
module tenant_admission #(parameter int CAPACITY_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        request,
  input  logic [15:0] want_gb, dev_free_gb,
  input  logic [3:0]  tenant_class, dev_class_mask,
  output logic        fits, class_allowed, admitted,
  output logic [7:0]  n_requests, n_rejected,
  output logic        policy_violation_err
);
  logic [3:0] class_bit;
  assign class_bit     = 4'b0001 << tenant_class[1:0];
  assign fits          = (want_gb <= dev_free_gb);
  assign class_allowed = |(dev_class_mask & class_bit);
  // The capacity-only build places any tenant anywhere it fits, which is how a
  // scheduler written for a homogeneous fleet behaves on a heterogeneous one.
  assign admitted = (CAPACITY_ONLY != 0) ? (request && fits)
                                         : (request && fits && class_allowed);
  // A tenant placed on a device its class was not permitted to share.
  assign policy_violation_err = admitted && !class_allowed;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_rejected <= 8'd0;
    end else if (request) begin
      n_requests <= n_requests + 8'd1;
      if (!admitted) n_rejected <= n_rejected + 8'd1;
    end
  end
endmodule

Five placement requests against a device with 128 GB free, permitting class 0.

Two rejections against one, and exactly one policy violation.

Both conditions are required and they fail independently. Row two is a tenant that fits and may not be there — a production workload on a device shared with a batch class, a regulated tenant on a device outside its compliance boundary, a customer with a contractual guarantee of sole tenancy. Row three is a tenant that may be there and does not fit. A scheduler that checks only capacity places row two; a scheduler that checks only policy accepts row three and then fails the allocation somewhere it is much harder to attribute.

Row four is the capacity boundary: exactly the free capacity fits, and a comparison written with the wrong operator refuses an allocation that would have succeeded.

Why the broken build is not a strawman. Capacity-only placement is correct for a homogeneous fleet, where every device is interchangeable and every tenant has the same class. It is also faster, simpler and easier to reason about. It becomes wrong the moment the fleet acquires a second device type or the business acquires a second tenant class, and neither of those events causes anybody to revisit the scheduler.

6. RTL 2 — Oversubscription Is Deliberate

Quotas that sum past the pool are not a bug. They are the entire economic argument for pooling, and they are also a promise the operator may not be able to keep.

// RTL 2 - oversubscription. Quotas that sum past the pool work until every
// tenant uses what it was promised.
module oversubscription #(parameter int PROMISE_ALL = 0) (
  input  logic clk, rst_n,
  input  logic        step,
  input  logic [15:0] pool_gb, sum_quota_gb, sum_used_gb,
  output logic        oversubscribed, pool_exhausted, can_promise_more,
  output logic [15:0] committed_gb, free_gb, ratio_x100,
  output logic [7:0]  n_steps, n_exhausted,
  output logic        broken_promise_err
);
  logic [31:0] r_q;
  assign oversubscribed = (sum_quota_gb > pool_gb);
  assign pool_exhausted = (sum_used_gb >= pool_gb);
  // What the operator has actually committed is the sum of quotas; what it can
  // deliver is the pool.
  assign committed_gb = sum_quota_gb;
  assign free_gb      = (sum_used_gb >= pool_gb) ? 16'd0 : (pool_gb - sum_used_gb);
  assign r_q = (pool_gb == 16'd0) ? 32'd0
                                  : (({16'd0, sum_quota_gb} * 32'd100) / {16'd0, pool_gb});
  assign ratio_x100 = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  // The promising build keeps admitting because a quota is not a reservation.
  assign can_promise_more = (PROMISE_ALL != 0) ? 1'b1 : !oversubscribed;
  // A pool that is out of capacity while quotas remain unfilled: every tenant
  // still inside its quota that now cannot allocate was promised something the
  // operator cannot deliver.
  assign broken_promise_err = step && pool_exhausted && oversubscribed;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_steps <= 8'd0; n_exhausted <= 8'd0;
    end else if (step) begin
      n_steps <= n_steps + 8'd1;
      if (pool_exhausted) n_exhausted <= n_exhausted + 8'd1;
    end
  end
endmodule

Eight steps against a 256 GB pool.

Six steps out of eight found the pool exhausted, and four of those broke a promise.

The distinction the model exists to make. Rows three and four both have an exhausted pool, and only one of them is a broken promise. A full pool with quotas summing to exactly the pool is a pool doing its job — every tenant received what it was promised and the capacity is gone. A full pool with quotas summing to 512 GB is 256 GB of commitment the operator cannot honour, and the tenants who discover it are the ones who happen to allocate last.

Row two is the state a well-run pool spends most of its life in: 200% committed and nothing wrong, because tenants do not use their quotas simultaneously. That is the bet, and the model makes it explicit rather than accidental.

Row five is the accounting boundary — more recorded as used than the pool holds, which happens when a release has not yet been reflected. Free capacity floors at zero rather than wrapping to 65,532.

7. RTL 3 — The Noisy Neighbour

Memory capacity is partitioned by a decoder. Bandwidth is not partitioned by anything, and that asymmetry is the source of the most common multi-tenant complaint there is.

// RTL 3 - a noisy neighbour. Bandwidth taken by one tenant is bandwidth another
// does not get, and a share is only a share if something enforces it.
module bandwidth_share #(parameter int NO_ARBITER = 0) (
  input  logic clk, rst_n,
  input  logic        cycle_en,
  input  logic [15:0] a_demand, b_demand, link_gbps,
  input  logic [7:0]  a_share_pct,
  output logic [15:0] a_cap, a_served, b_served, b_starved_by,
  output logic [7:0]  n_cycles, n_starved,
  output logic        starvation_err
);
  logic [31:0] cap_q;
  logic [15:0] left;
  assign cap_q = ({16'd0, link_gbps} * {24'd0, a_share_pct}) / 32'd100;
  assign a_cap = cap_q[15:0];
  // The arbitered build holds A to its share. Without one, A takes what it asks
  // for and B gets the remainder.
  assign a_served = (NO_ARBITER != 0) ? ((a_demand > link_gbps) ? link_gbps : a_demand)
                                      : ((a_demand > a_cap) ? a_cap : a_demand);
  assign left     = link_gbps - a_served;
  assign b_served = (b_demand > left) ? left : b_demand;
  assign b_starved_by = b_demand - b_served;
  // B receiving less than it asked for because A went past its own share.
  assign starvation_err = cycle_en && (b_served < b_demand) && (a_served > a_cap);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_cycles <= 8'd0; n_starved <= 8'd0;
    end else if (cycle_en) begin
      n_cycles <= n_cycles + 8'd1;
      if ((b_served < b_demand) && (a_served > a_cap)) n_starved <= n_starved + 8'd1;
    end
  end
endmodule

A 100 Gbps link, A entitled to 50%.

(each cell is arbitered / unarbitered)

One starvation event, and it takes both halves of the condition.

Three of those rows are B going short and only one is starvation, which is the whole point of the model. Row three: A takes the entire link and B receives nothing — but B asked for nothing, so nothing was taken from it. Row five: A stays inside its 50% share and B asks for 80, receiving 50 and going 30 short. That is a share working exactly as specified, and a monitor that flags every shortfall would page an operator for it.

Row four is the other direction. A asks for 20, well inside its cap, and B receives 80 — more than its own half. A share is a floor under contention, not a ceiling in general, and an arbiter that held B to 50 while A used 20 would be wasting a third of the link.

Why the broken build is not a strawman. No arbiter is the default. Bandwidth arbitration between logical devices costs silicon, adds latency to every request, and does nothing at all on a device that is not contended — which, in the design lab, it never is.

8. RTL 4 — A Counter That Names Nobody

The previous three models all produce events an operator needs to see. This one is about whether seeing them is useful.

// RTL 4 - attribution. A counter that does not name a tenant cannot be acted on.
module tenant_attribution #(parameter int AGGREGATE_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       event_valid,
  input  logic [3:0] tenant_id,
  input  logic [3:0] suspect_id,
  output logic       attributable, correct_suspect,
  output logic [7:0] n_events, n_attributed, t0_events, t1_events,
  output logic       misattribution_err
);
  // Aggregate telemetry knows an event happened and not to whom.
  assign attributable    = (AGGREGATE_ONLY == 0);
  assign correct_suspect = attributable ? (suspect_id == tenant_id) : 1'b0;
  // Acting on a tenant that did not cause the event.
  assign misattribution_err = event_valid && attributable && !correct_suspect;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_events <= 8'd0; n_attributed <= 8'd0; t0_events <= 8'd0; t1_events <= 8'd0;
    end else if (event_valid) begin
      n_events <= n_events + 8'd1;
      if (attributable) begin
        n_attributed <= n_attributed + 8'd1;
        if (tenant_id == 4'd0) t0_events <= t0_events + 8'd1;
        if (tenant_id == 4'd1) t1_events <= t1_events + 8'd1;
      end
    end
  end
endmodule

Five events, three from tenant 0 and two from tenant 1.

Five attributed against zero, three credited to tenant 0 and two to tenant 1.

The counts must differ for the telemetry to be testable. Three and two is deliberate: with a two-and-two split, a telemetry block that credits every event to the wrong tenant produces exactly the same numbers as one that gets it right, and no test can tell them apart. That is not a hypothetical — it is the mutation in section 18 that survived until the distribution was made uneven.

The aggregate build cannot misattribute, and that is the trap. It has a perfect misattribution record because it never names anybody, and an operator reading a dashboard of aggregate error counts has a number that goes up and no action available. Attribution is what converts telemetry into an operation.

9. RTL 5 — Eviction Needs A Notice Period

Capacity comes back to the pool one of two ways: the tenant releases it, or the operator takes it.

// RTL 5 - eviction. Reclaiming capacity from a tenant is an operation with a
// notice period, and taking it without one is a fault the tenant cannot handle.
module eviction_policy #(parameter int IMMEDIATE_RECLAIM = 0) (
  input  logic clk, rst_n,
  input  logic       reclaim_req, tenant_acked,
  input  logic [7:0] notice_cycles, elapsed_cycles,
  output logic       notice_expired, may_reclaim, graceful,
  output logic [7:0] n_reclaims, n_forced,
  output logic       abrupt_eviction_err
);
  assign notice_expired = (elapsed_cycles >= notice_cycles);
  // A graceful reclaim waits for the tenant to acknowledge, or for its notice
  // period to run out. The immediate build waits for neither.
  assign may_reclaim = (IMMEDIATE_RECLAIM != 0) ? 1'b1
                                                : (tenant_acked || notice_expired);
  assign graceful = tenant_acked || notice_expired;
  // Capacity taken from a tenant that neither acknowledged nor ran out of time.
  assign abrupt_eviction_err = reclaim_req && may_reclaim && !graceful;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reclaims <= 8'd0; n_forced <= 8'd0;
    end else if (reclaim_req) begin
      n_reclaims <= n_reclaims + 8'd1;
      if (may_reclaim && !graceful) n_forced <= n_forced + 8'd1;
    end
  end
endmodule

Four reclaims against a ten-cycle notice.

One forced eviction against none.

Two independent ways to become graceful. An acknowledgement means the tenant has flushed what it needed to; an expired notice means it had the opportunity and did not take it. Either is sufficient, and a model requiring both would hang on any tenant that has crashed.

The last row is the policy stated honestly: a zero-length notice is expired the instant it is issued, so the immediate reclaim and the correct reclaim behave identically. The model does not call that an error, because it is not one — it is an operator choosing a policy of no notice, which is a decision they are entitled to make and which the telemetry will record as graceful. The failure in row two is different in kind: a notice period was configured, and the reclaim did not wait for it.

10. Waveform — A Reclaim Running Against Its Notice

The forced count freezing at six is the detail worth reading. Nothing about the immediate build changed — it took the capacity on every cycle, exactly as it always does. What changed is that the reclaim became legitimate, so the same behaviour stopped being recorded as a fault. A counter that stops rising is not always a problem that stopped.

11. RTL 6 — Blast Radius Is A Packing Decision

How many tenants share a device is usually decided by a cost model. It is also an availability decision, and the two are in direct opposition.

// RTL 6 - blast radius at fleet scale. One device failing takes down every
// tenant on it, so how many tenants per device is an availability decision.
module failure_domain #(parameter int PACK_DENSELY = 0) (
  input  logic clk, rst_n,
  input  logic       fail,
  input  logic [7:0] tenants_total, devices_total,
  output logic [7:0] per_device, tenants_lost, survivors,
  output logic [7:0] loss_pct,
  output logic [7:0] n_failures, n_majority_loss,
  output logic       majority_loss_err
);
  logic [15:0] lp_q;
  // The dense build puts every tenant on as few devices as they fit on; the
  // spread build uses all of them.
  assign per_device = (devices_total == 8'd0) ? 8'd0
                    : ((PACK_DENSELY != 0) ? tenants_total
                                           : ((tenants_total + devices_total - 8'd1) / devices_total));
  assign tenants_lost = fail ? per_device : 8'd0;
  // No floor is needed: per_device is either tenants_total (packed) or a ceiling
  // division of it (spread), so tenants_lost can never exceed tenants_total and
  // the subtraction cannot underflow.
  assign survivors    = tenants_total - tenants_lost;
  assign lp_q = (tenants_total == 8'd0) ? 16'd0
                                        : (({8'd0, tenants_lost} * 16'd100) / {8'd0, tenants_total});
  assign loss_pct = (lp_q > 16'd255) ? 8'hFF : lp_q[7:0];
  // A single device failure costing more than half the fleet's tenants.
  assign majority_loss_err = fail && (loss_pct > 8'd50);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_failures <= 8'd0; n_majority_loss <= 8'd0;
    end else if (fail) begin
      n_failures <= n_failures + 8'd1;
      if (loss_pct > 8'd50) n_majority_loss <= n_majority_loss + 8'd1;
    end
  end
endmodule

Five device failures.

Packed, every one of those is 100% and four of five are a majority loss.

Exactly half is not a majority, and the fourth row is one tenant away. That boundary is where a fleet-sizing conversation actually happens: sixteen tenants on two devices is survivable in the sense that half the fleet remains, and seventeen on two is not, and the difference is a single placement decision made by a scheduler with no view of the availability model.

The second row is the rounding. Sixteen tenants over three devices is 5.33 per device, and the blast radius is six, not five — the ceiling, because some device holds the remainder and that is the one that will fail. A model that rounds down understates the blast radius on every fleet whose tenant count is not a multiple of its device count, which is nearly all of them.

The last row is the degenerate case: a fleet with no devices places nobody and loses nobody, and the guard is what stops a divide-by-zero from reaching a capacity planner's dashboard.

Figure 3 — The same sixteen tenants, two placements, and no device behaves differently in either. The blast radius is decided entirely to the left of the devices, by a scheduler with no view of the availability model.

12. RTL 7 — A Rebind Is Not Atomic

Section 6 moves capacity between tenants as arithmetic. In the device it is a sequence, and the sequence has states.

// RTL 7 - a rebind is not atomic. Capacity moving from one tenant to another
// passes through phases, and each phase says exactly who may reach it.
module rebind_sequence #(parameter int NO_INTERLOCK = 0) (
  input  logic clk, rst_n,
  input  logic       step,
  input  logic [2:0] phase,        // 0 idle 1 quiesced 2 unbound 3 scrubbed 4 bound
  input  logic       old_can_access, new_can_access,
  output logic       both_can_access, neither_owns, phase_consistent, safe_phase,
  output logic [7:0] n_steps, n_unsafe,
  output logic       double_binding_err
);
  assign both_can_access = old_can_access && new_can_access;
  assign neither_owns    = !old_can_access && !new_can_access;
  // Idle and quiesced belong to the old tenant; unbound and scrubbed belong to
  // nobody; only the bound phase grants the new tenant access.
  assign phase_consistent =
      ((phase == 3'd0) && old_can_access && !new_can_access)
   || ((phase == 3'd1) && old_can_access && !new_can_access)
   || ((phase == 3'd2) && neither_owns)
   || ((phase == 3'd3) && neither_owns)
   || ((phase == 3'd4) && !old_can_access && new_can_access);
  // The build without an interlock unbinds and binds in the same step, so it
  // never checks which phase it is in.
  assign safe_phase = (NO_INTERLOCK != 0) ? 1'b1 : phase_consistent;
  // Two tenants able to reach the same capacity while the rebind is called safe.
  assign double_binding_err = step && safe_phase && both_can_access;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_steps <= 8'd0; n_unsafe <= 8'd0;
    end else if (step) begin
      n_steps <= n_steps + 8'd1;
      if (both_can_access) n_unsafe <= n_unsafe + 8'd1;
    end
  end
endmodule

Six steps through the rebind sequence.

Two double bindings, and two distinct failure modes.

Both ends of the sequence fail the same way. Rows four and six are the interlock failure at opposite ends — the old tenant not yet unbound in the middle of the sequence, and the old tenant never unbound after the new one was bound. A test that drives only one of them passes a design that gets the other wrong, which is precisely the mutation that survived in section 18.

Row five is the other kind of inconsistency, and it is not a leak. Nobody owning capacity in a phase that requires an owner is a stall: the old tenant has lost access it should still have, and the new one has not gained it. No data crosses a boundary and a tenant is nonetheless broken. The model reports it as inconsistent and not as a double binding, because the two need different responses.

13. RTL 8 — Fairness Is A Promise With A Shape

Section 7 held one tenant to a fixed cap. Real fleets promise weighted shares, and the promise has to survive both tenants asking at once and one of them going idle.

// RTL 8 - fairness. Equal shares and weighted shares are different promises, and
// a scheduler that serves whoever asks first keeps neither.
module fair_scheduler #(parameter int FIRST_COME = 0) (
  input  logic clk, rst_n,
  input  logic        arb_en,
  input  logic [7:0]  a_weight, b_weight,
  input  logic [15:0] a_req, b_req, budget,
  output logic [15:0] a_entitled, a_grant, b_grant,
  output logic        a_over_entitlement,
  output logic [7:0]  n_arb, n_unfair,
  output logic        unfairness_err
);
  logic [15:0] wsum;
  logic [31:0] ent_q;
  logic [15:0] left;
  assign wsum  = {8'd0, a_weight} + {8'd0, b_weight};
  assign ent_q = (wsum == 16'd0) ? 32'd0
                                 : (({16'd0, budget} * {24'd0, a_weight}) / {16'd0, wsum});
  assign a_entitled = ent_q[15:0];
  // First-come serves A up to the whole budget. The weighted build holds A to
  // its entitlement whenever B is also asking.
  assign a_grant = (FIRST_COME != 0) ? ((a_req > budget) ? budget : a_req)
                 : ((b_req == 16'd0) ? ((a_req > budget) ? budget : a_req)
                                     : ((a_req > a_entitled) ? a_entitled : a_req));
  assign left    = budget - a_grant;
  assign b_grant = (b_req > left) ? left : b_req;
  assign a_over_entitlement = (a_grant > a_entitled);
  // A served past its entitlement while B was asking and went short.
  assign unfairness_err = arb_en && a_over_entitlement && (b_grant < b_req);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_arb <= 8'd0; n_unfair <= 8'd0;
    end else if (arb_en) begin
      n_arb <= n_arb + 8'd1;
      if (a_over_entitlement && (b_grant < b_req)) n_unfair <= n_unfair + 8'd1;
    end
  end
endmodule

Five arbitrations against a budget of 100.

(each result cell is A granted / B granted)

Zero unfair arbitrations under weights against three.

Row three is the work-conserving exception, and it is what makes a weighted share usable. With B idle, A receives the whole budget and is well past its entitlement — and that is not unfair, because nobody went short. A scheduler that held A to 75 while B asked for nothing would idle a quarter of the budget for no benefit at all.

Row five is the degenerate case: both weights zero, so nobody is entitled to anything, A receives nothing and B takes the budget as the remainder. The guard on wsum is what stops the division, and the behaviour it produces — no entitlement means no grant under contention — is at least defensible, which matters because a configuration error that zeroes a weight is not rare.

14. RTL 9 — The Policy Itself Has A Cost

Nine models of things a control plane should check. Every one of those checks runs on the allocation path.

// RTL 9 - the cost of the policy itself. Every check runs on the allocation
// path, and a policy engine that is too slow is a policy nobody enables.
module policy_cost #(parameter int SERIAL_CHECKS = 0) (
  input  logic clk, rst_n,
  input  logic        alloc,
  input  logic [7:0]  n_checks, per_check_us,
  input  logic [15:0] base_alloc_us,
  output logic [15:0] policy_us, total_us, overhead_pct,
  output logic        budget_met,
  input  logic [15:0] budget_us,
  output logic [7:0]  n_allocs, n_over_budget,
  output logic        budget_miss_err
);
  logic [15:0] ser, par;
  logic [31:0] ov_q;
  assign ser = {8'd0, n_checks} * {8'd0, per_check_us};
  // Parallel checks cost one check time, not their sum.
  assign par = (n_checks == 8'd0) ? 16'd0 : {8'd0, per_check_us};
  assign policy_us = (SERIAL_CHECKS != 0) ? ser : par;
  assign total_us  = base_alloc_us + policy_us;
  assign ov_q = (total_us == 16'd0) ? 32'd0
                                    : (({16'd0, policy_us} * 32'd100) / {16'd0, total_us});
  assign overhead_pct = (ov_q > 32'd65535) ? 16'hFFFF : ov_q[15:0];
  assign budget_met = (total_us <= budget_us);
  assign budget_miss_err = alloc && !budget_met;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_allocs <= 8'd0; n_over_budget <= 8'd0;
    end else if (alloc) begin
      n_allocs <= n_allocs + 8'd1;
      if (!budget_met) n_over_budget <= n_over_budget + 8'd1;
    end
  end
endmodule

A 100 microsecond base allocation, checks costing 10 microseconds each, a 200 microsecond budget.

The parallel build does not grow with the check count, which is the entire argument. Adding a seventh, eighth and ninth policy check to a serial engine adds 30 microseconds to every allocation in the fleet; adding them to a parallel one adds nothing, because they were never the critical path.

Row three is the budget boundary: exactly 200 microseconds meets a 200 microsecond budget. Row two is where the serial engine stops being an option, and it is the moment section 5 through section 13 finish being a list of good ideas and start being a latency problem.

This is why policy gets disabled. Not because anybody argued against it — because an allocation path that misses its budget produces timeouts somewhere else, and the fastest way to fix a timeout is to turn off the checks that caused it.

Figure 4 — Only the first gate is arithmetic, and it is the only one most schedulers implement. Each of the other three is a property this chapter shows failing on a fleet where capacity always fits.

15. RTL 10 — The Multi-Tenant Environment Assembled

Six properties, and isolation is one of them.

// RTL 10 - the multi-tenant environment assembled. Isolation is necessary and
// is not the whole of it.
module multi_tenant_model #(parameter int ISOLATION_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       tenants_isolated,   // every boundary of 19.3 holds
  input  logic       policy_admitted,    // placement respected the policy
  input  logic       shares_enforced,    // no tenant exceeds its bandwidth share
  input  logic       events_attributed,  // telemetry names a tenant
  input  logic       reclaim_graceful,   // eviction has a notice period
  input  logic       blast_bounded,      // one device is not the whole fleet
  output logic       sound,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_sound,
  output logic       false_soundness_err
);
  assign fail_mask[0] = ~tenants_isolated;
  assign fail_mask[1] = ~policy_admitted;
  assign fail_mask[2] = ~shares_enforced;
  assign fail_mask[3] = ~events_attributed;
  assign fail_mask[4] = ~reclaim_graceful;
  assign fail_mask[5] = ~blast_bounded;
  // The isolation-only build treats 19.3 as the whole problem.
  assign sound = (ISOLATION_ONLY != 0) ? tenants_isolated : (fail_mask == 6'd0);
  assign false_soundness_err = evaluate && sound && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_sound <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (sound) n_sound <= n_sound + 8'd1;
    end
  end
endmodule

Five configurations.

One sound against four, and three false claims.

Row four is the one to argue with. Every tenant is isolated, placed by policy, held to its share, individually attributable and reclaimed with notice — and the fleet is packed onto so few devices that a single failure takes a majority of it down. Nothing is wrong with any tenant. The environment is unsound anyway, and no property that examines a tenant pair can see it.

The weak model only notices when isolation itself fails, which is the property with the most attention on it, the most tooling around it and the most people watching. Every one of the other five is somebody's job and nobody's alarm.

16. Quantitative Reasoning

Numbers from the models, all of them teaching values.

Admission. Five placements, one class violation. A capacity-only scheduler placed a class-1 tenant on a class-0 device with 96 GB to spare — the capacity check passed with 300% headroom, which is exactly why the shortcut is invisible in testing.

Oversubscription. 512 GB of quota against a 256 GB pool: a 200% commitment ratio, and a pool that spends most of its life perfectly healthy in that state. Six of eight steps found it exhausted, and four of those were promises that could not be kept.

Noisy neighbour. A 100 Gbps link. Without an arbiter, A took 100 of 100 and B received nothing against a demand of 50 — a 50 Gbps shortfall for a tenant entitled to exactly that. With an arbiter, 50 and 50.

Attribution. Five events, three from one tenant and two from another. The aggregate build attributed zero of five, with a perfect misattribution record and nothing an operator could act on.

Eviction. A ten-cycle notice. The immediate build reclaimed on cycle zero, then recorded six forced evictions in six cycles, stopping only when the notice expired and made the same behaviour legitimate.

Blast radius. Sixteen tenants over four devices is a 25% loss per failure; packed onto one it is 100%. Sixteen over two is exactly half and survivable; seventeen over two is 52% and a majority loss, one tenant away.

Rebind. Six steps, two double bindings, at opposite ends of the sequence.

Fairness. A 3:1 weight over a budget of 100: 75 and 25 under weights, 100 and 0 under first-come. Three unfair arbitrations of five.

Policy cost. Twelve checks at 10 microseconds: 220 microseconds serial against 110 parallel, and a 200 microsecond budget that only one of them meets. Serial overhead 55%; parallel overhead 9%, independent of check count.

The assembled model. Six properties, five configurations, one sound. The isolation-only definition reported four.

17. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality rather than a bound.

Admission. Both conditions are asserted independently, and the capacity boundary is driven exactly.

chk(aFi == 1'b1, "class 1 also fits");
chk(aCa == 1'b0, "but the device does not permit it");
chk(aAd == 1'b0, "so the correct scheduler rejects it");
chk(bAd == 1'b1, "the capacity-only scheduler places it");

Oversubscription. The two states that both have an exhausted pool are asserted apart — one broken promise, one not.

chk(oOs == 1'b0,  "quotas back within the pool");
chk(oPe2 == 1'b1, "the pool is still full");
chk(oBp == 1'b0,  "but nothing more was promised than exists");

The accounting overshoot is asserted to floor rather than wrap.

chk(oFg == 16'd0, "and free capacity floors at zero");

Bandwidth. The three shortfalls that are not starvation are each asserted as correct behaviour, which is what stops a checker flagging a working share.

chk(wAs == 16'd50, "A takes exactly its cap");
chk(wBs == 16'd50, "so B gets the remaining 50");
chk(wBb == 16'd30, "30 short of what it asked for");
chk(wSe == 1'b0,   "which is not starvation, because A stayed inside its share");

Attribution. The per-tenant counts are asserted at unequal values, deliberately.

chk(tT0 == 8'd3, "three belonged to tenant 0");
chk(tT1 == 8'd2, "and two to tenant 1");

Eviction. Both routes to gracefulness are asserted separately, and the zero-length notice is asserted as policy rather than fault.

chk(eNx == 1'b1, "a zero-cycle notice is expired immediately");
chk(eAe == 1'b0, "and is called graceful, which is the policy talking");

Blast radius. The majority boundary is asserted at exactly half and one tenant past it.

chk(fLp == 8'd50, "exactly half the fleet");
chk(fMe == 1'b0,  "and half is not a majority");
chk(fLp == 8'd52, "a 52 percent loss");
chk(fMe == 1'b1,  "which is a majority");

Rebind. Both ends of the sequence are asserted, and the stall is asserted as inconsistent-but-not-a-leak.

chk(rPc == 1'b0, "which the idle phase does not permit");
chk(rBa == 1'b0, "but nothing is double-bound");

Fairness. The work-conserving exception is asserted as over-entitlement and not unfairness.

chk(hAo == 1'b1, "which is past its entitlement");
chk(hUe == 1'b0, "and is not unfair, because B asked for nothing");

Policy cost. The parallel build's independence from check count is asserted as an exact equality across two different check counts.

The assembled model. Every fail mask is asserted as an exact six-bit value.

Totals: 237 checks across two testbenches, 114 on the front five models and 123 on the back five, all passing on the unmutated sources.

18. Mutation Testing

Forty-seven mutations were injected one at a time. A survivor is a statement about the testbench or the design.

47 injected, 47 killed, after six survivors were diagnosed.

Survivors 1 and 2 — boundaries never driven. Removing the free-capacity floor survived because the testbench never drove more used than the pool holds; at exactly the pool size, the guarded and unguarded expressions both produce zero. Removing the divide-by-zero guard survived because a pool of zero capacity was never driven. Both are stimulus gaps at a boundary the correct code was written to handle and the test never reached.

Survivor 3 — a compound condition with a half never driven alone. Dropping a_served > a_cap from the starvation check survived: every case where B went short also had A over its cap. Driving A at exactly its cap with B asking for more makes B go short with A blameless, and the mutation dies. This is the case the model exists to distinguish, and it was not in the testbench.

Survivor 4 — a degenerate case that makes correct and mutated coincide. Crediting tenant 1's counter on tenant_id == 0 survived while the event distribution was two and two, because both counters ended at the same value either way. Making it three and two kills it. Equal counts are not a neutral choice of stimulus; they are a stimulus that cannot observe the output.

Survivor 5 — an unreachable guard. The floor on survivors never fired, because tenants_lost is per_device, which is either tenants_total or a ceiling division of it, and can therefore never exceed tenants_total. The guard was protecting against an underflow the arithmetic cannot produce. It was deleted, with a comment saying why, and replaced with a mutation that is not equivalent — counting survivors against the device total instead of the tenant total.

Survivor 6 — one end of a sequence tested and not the other. Allowing the old tenant access in the bound phase survived because the testbench drove the both-reachable case only in the unbound phase. Driving it in the bound phase too kills it. The interlock has two ends and they fail independently.

19. Verification Strategy

What a testbench for a real control plane must cover, beyond what these models reach.

Every compound condition, each half driven alone. Admission is fit and policy. Starvation is B short and A over. Unfairness is A over and B short. Abrupt eviction is reclaimed and not graceful. Four compound conditions, eight halves, and section 18 shows two of them were not separately driven.

The cases that look like failures and are correct. B going short because it asked for more than its share. A taking the whole budget because B is idle. A zero-length notice making an immediate reclaim graceful. A device-wide reset — from 19.3 — clearing regions of a host that asked for it. Each trips a naive checker, and a testbench that does not assert them as correct will accept a design that refuses them.

Unequal stimulus for per-entity counters. Any telemetry with one counter per tenant must be driven with different counts per tenant, or a block that credits the wrong one is invisible. This generalises: equal values are the worst possible stimulus for anything that routes.

Both ends of every sequence. A rebind fails at the start and at the finish, and passing one says nothing about the other.

The degenerate configurations. A pool of zero capacity. A fleet of zero devices. Both scheduler weights zero. Zero policy checks. Each is reachable through a configuration error, and each is where an ungurarded division produces an X that a capacity planner will read as a number.

What a real control plane needs that these models do not have. Concurrency between placement decisions — two schedulers admitting to the same device at once. Staleness — a placement made against a capacity view that has since changed. Partial failure — a fabric manager that accepted a rebind and died before completing it. Every one of those is where a fleet actually breaks, and none is visible in a combinational model.

20. Synthesis and Implementation Reality

Most of this chapter is control plane, and the parts that are not are the expensive ones.

Admission and quota arithmetic is software. It runs on a management processor at millisecond timescales and none of it is on a critical path in silicon. The cost is entirely the latency of section 14, and the fix is parallel evaluation rather than faster checks.

Bandwidth arbitration is hardware and it is not cheap. A weighted arbiter between sixteen logical devices needs per-device credit counters, a weight register file and an arbitration tree that resolves in a single cycle at the device's clock rate. It sits on every request, contended or not, and it is the reason unarbitered devices ship.

Per-tenant telemetry is a counter array. Sixteen logical devices times ten counters is 160 registers plus the read path, and the read path is the harder half — an operator polling telemetry must not perturb the traffic it is measuring, which means a shadow copy and a snapshot mechanism.

The rebind interlock is a small state machine with large consequences. Five phases, and each transition must be visible to both the fabric manager and the affected hosts. The failure mode in section 12 is not an arithmetic error; it is a transition taken before the previous one was acknowledged, which is the classic distributed-handshake bug expressed in silicon.

Blast radius is not implemented anywhere. It is a property of how a scheduler places tenants, and no device can enforce it. That is worth stating plainly: section 11 is the one property in this chapter that hardware cannot help with at all.

21. Silicon Observability

Demand alongside served is the one people leave out. A logical device served 10 Gbps is either a tenant asking for 10 or a tenant asking for 90 and being starved of 80, and served-only telemetry cannot tell those apart. It is also the harder counter to build, because demand that was never admitted was never counted anywhere.

22. Debug Lab

Symptom. A tenant reports its p99 latency has tripled. Nothing about its own allocation changed: same capacity, same device, same workload. Its error counters are zero. The device reports no faults and every boundary from 19.3 is intact.

Step 1 — is it the tenant? Compare its own bandwidth demand against the previous week. Unchanged. Whatever happened, it happened to the tenant rather than because of it.

Step 2 — who else is on the device? The placement database shows a second tenant admitted four days ago, at the point the latency changed. Both tenants fit; the device had 90 GB free.

Step 3 — was the placement legal? Check the class mask. The new tenant is a batch class and the device is marked for latency-sensitive workloads. The scheduler placed it anyway, which is section 5 — the capacity check passed with room to spare and the class check was never made.

Step 4 — is it actually bandwidth? Read served bandwidth per logical device. The new tenant is served 78 Gbps of a 100 Gbps link. Read demand per logical device: the complaining tenant demands 40 and is served 22. It is being starved of 18 Gbps, which is section 7, and served-only telemetry would have shown a tenant using 22 Gbps and looking unremarkable.

Step 5 — why did nothing alarm? The device has no arbiter. There is no share to exceed, so there is no threshold to cross, so there is no event to count. The absence of an arbiter is also the absence of the telemetry that would have detected the absence of an arbiter.

The finding. A policy violation at placement time, four days earlier, surfacing as a latency regression in a tenant that did nothing. Two of this chapter's properties failed and neither produced an event.

The fix, in order. Move the batch tenant — it should never have been placed there. Then add the class check to the scheduler, because the next one will land somewhere equally wrong. Then, on a longer timescale, add demand telemetry, because without it the next instance of this is another four-day investigation.

What made this hard. Every counter that exists read zero, correctly. The failure was in a decision made days earlier by a component that was not instrumented, and the symptom appeared in a tenant that had no visibility into any of it.

23. Design Review

1. Does the scheduler check policy, or only capacity? And if policy, which dimensions — class, compliance boundary, sole tenancy, hardware generation. Section 5.

2. What is the commitment ratio, and does anyone watch it? Oversubscription is fine and unmeasured oversubscription is not. Section 6.

3. Is there a bandwidth arbiter? If not, there is no noisy-neighbour telemetry either, because there is no share to exceed. Section 7.

4. Is demand recorded, or only what was served? Section 21, and section 22 is what it costs to answer no.

5. Is every counter per logical device? If not, an operator can see that something is wrong and never who. Section 8.

6. Does reclaim have a notice period, and does anything wait for it? Two questions, and the second is the one that fails. Section 9.

7. How many tenants share one device, and who decided? Usually a cost model, and usually not with an availability review. Section 11.

8. Is the rebind sequence interlocked at both ends? Both, not one. Section 12.

9. Are shares weighted or equal, and is the scheduler work-conserving? A non-work-conserving scheduler idles capacity under exactly the conditions a pool exists to exploit. Section 13.

10. What does the policy engine cost per allocation, and is it inside the budget at the check count you plan to reach? Section 14, and the honest version of this question is asked at twice the current check count.

24. How This Appears In Real Engineering

A fleet team adding a second device generation discovers the scheduler has no concept of device class, because it never needed one. Retrofitting it means labelling every device and every tenant, and reconciling the tenants already placed — a migration measured in months, caused by a check that would have cost one line at the start.

A capacity planning team owns the commitment ratio and is usually the only group that knows it. The ratio is an explicit risk position — a bet that tenants will not simultaneously use what they were promised — and it is worth writing down as a number rather than leaving it as a consequence of individual quota decisions.

An SRE team debugging a latency regression does what section 22 does, and how long it takes is decided entirely by whether per-tenant demand telemetry exists. With it, the investigation is an hour. Without it, the tenant is asked to reproduce the problem while somebody watches, which does not work because the noisy neighbour is not under anybody's control.

A device team implementing a fabric-manager interface owns the rebind interlock, and the pressure is always to collapse the phases. Each phase is a round trip to a management processor, and a rebind that takes five round trips is a rebind that shows up in a capacity-rebalancing latency budget. Section 12 is what the collapsed version costs.

25. Common Misconceptions

"If the tenants are isolated, the environment is sound." One property of six. Section 15 shows the isolation-only definition calling four of five configurations sound.

"A tenant that fits can be placed." Fitting and being permitted are two questions, and only one of them is arithmetic. Section 5.

"Oversubscription is a mistake." It is the economic argument for pooling. The mistake is not measuring it. Section 6.

"Bandwidth is shared fairly by default." Bandwidth is shared by whoever asks. Fairness requires an arbiter, which is silicon somebody chose to spend. Section 7.

"The error counters will tell us who caused it." Only if they are per tenant. An aggregate counter has a perfect misattribution record and no action attached. Section 8.

"Reclaiming capacity is just a rebind." It is a rebind plus a notice period plus something that waits for it, and the third part is the one that gets dropped. Section 9.

"Denser packing is more efficient." It is, until the first device failure, at which point it is a fleet event. Section 11.

"A rebind is atomic." Five phases, and two of them belong to nobody. Section 12.

"A share is a limit." A share is a floor under contention. A scheduler that enforces it when nobody is competing wastes the capacity a pool exists to share. Section 13.

"Policy checks are free." Twelve serial checks doubled the allocation latency and missed the budget. Section 14, and the consequence is that the policy gets turned off.

26. Interview Reasoning

Q. A tenant's latency triples and its own workload did not change. Where do you look?

Who else is on the device, and when they arrived. If the timing matches a placement, the question becomes whether that placement was legal and whether the new tenant is taking bandwidth. The follow-up worth asking is what telemetry you need: served bandwidth per logical device is not enough, because a starved tenant and an idle tenant both show low numbers. Demand is the counter that distinguishes them, and it is usually missing.

Q. Quotas on a pool sum to twice its capacity. Is that wrong?

No — that is what pooling is for, and a ratio of 100% means the pool is earning nothing. It becomes wrong when nobody knows the number, or when the pool exhausts while tenants are still inside their quotas. The distinction to draw is between a full pool that delivered everything promised and a full pool that cannot, and those look identical from a capacity number alone.

Q. Sixteen tenants and four devices. How do you place them?

Four each, if availability matters, and the reasoning is that one device failure costs 25% rather than 100%. The follow-up: what if the count does not divide? Ceiling, not floor — some device holds the remainder, and that is the one to size against. And the deeper follow-up: no device can enforce this. It is purely a scheduler property, which is why it is the one that gets forgotten.

Q. Why is a shared bandwidth arbiter worth the silicon?

Because without it there is no share, and without a share there is no threshold, and without a threshold there is no event — so the noisy-neighbour problem is not just unfixed, it is unmeasurable. That is the argument that wins: the arbiter buys observability as much as it buys fairness.

Q. What makes a bandwidth share different from a bandwidth cap?

A share is a floor under contention and a cap is a ceiling always. A work-conserving scheduler lets a tenant exceed its share when nobody else is asking; a cap idles the capacity. The follow-up is what that means for a checker: a tenant over its share is not automatically a fault, and a checker that says otherwise will flag correct behaviour constantly.

Q. A rebind moves capacity from tenant A to tenant B. What states does it pass through?

Quiesce A, unbind A, scrub, bind B. The interesting part is the invariant: in the middle phases the capacity belongs to nobody, and at no point may it belong to both. The follow-up is where the interlock fails — and the answer is at either end, because unbinding late and binding early produce the same violation from opposite directions.

27. Exercises

1. Extend RTL 1 to score candidate devices rather than accept or reject one, and show that the highest-scoring device is not always the one with the most free capacity.

2. Add per-tenant quotas to RTL 2 and compute, for a given commitment ratio, the exact number of simultaneously-active tenants at which the pool exhausts. Assert it.

3. Make RTL 3 work-conserving: A may exceed its cap when B is idle, and must be pushed back inside it within a bounded number of cycles when B begins asking. Assert the bound exactly.

4. Add a shadow snapshot to RTL 4 so telemetry can be read without perturbing the counters, and assert that a read during an event neither drops it nor counts it twice.

5. Give RTL 5 a notice period that varies per tenant class, and show that a fleet-wide reclaim completes in the time of the longest notice rather than the sum of them.

6. Extend RTL 6 to report the blast radius of a rack failure as well as a device failure, and show that spreading across devices within one rack improves one and not the other.

7. Turn RTL 7 into a state machine with explicit transitions and assert that no transition may be taken before the previous one is acknowledged. This is the real bug in section 12.

8. Extend RTL 8 to three tenants and show that the work-conserving exception is harder: with one idle tenant, the budget must be redistributed by weight among the rest, not given to whoever asks first.

9. Model the failure in section 14 end to end: raise the check count until the serial engine misses its budget, then show the resulting timeout, then show that disabling policy fixes the timeout and re-enables every defect in this chapter.

10. Add a seventh property to RTL 10 of your choosing. If it is implied by one of the existing six, say which. If it is not, give the configuration it catches that the current mask calls sound.

28. Summary

19.3 built five boundaries between two tenants. This chapter is what a fleet needs on top of them.

Placement is a policy question. A capacity-only scheduler placed a tenant on a device it was not permitted to share, with 300% capacity headroom — the check that passed was never the check that mattered.

Oversubscription is deliberate. 512 GB of quota against a 256 GB pool is a 200% commitment ratio and a healthy pool. Four of eight steps found it exhausted with promises outstanding, and those are different from a full pool that delivered everything.

Bandwidth is shared by whoever asks. Without an arbiter A took 100 of 100 Gbps and B received nothing against a demand of 50 — and three of five shortfalls were not starvation at all, which is why the condition needs both halves.

A counter that names nobody cannot be acted on. Five events, zero attributed, and a perfect misattribution record earned by never naming anyone.

Per-entity counters need unequal stimulus. A two-and-two split hid a telemetry block crediting the wrong tenant; three-and-two exposed it immediately.

Reclaim needs a notice period and something that waits for it. Six forced evictions in six cycles, stopping only when the notice made the same behaviour legitimate.

Blast radius is a packing decision no device can enforce. Sixteen tenants over four devices loses 25%; packed, 100%. Sixteen over two is exactly half and survivable; seventeen is 52% and a majority.

A rebind is not atomic, and its interlock fails at both ends — two double bindings in six steps, from opposite directions.

A share is a floor under contention, not a ceiling. A 3:1 weight gave 75 and 25; first-come gave 100 and 0, and being over an entitlement while nobody is competing is correct behaviour.

Policy has a latency, and that is why policy gets disabled. Twelve serial checks: 220 microseconds against a 200 budget, where the parallel build stayed at 110 regardless of check count.

Isolation is one property of six, and the isolation-only definition called four of five environments sound when one was.

Module 20 — CXL 2.0 turns to the generation that made all of this possible. 20.1 — CXL 2.0 Switching starts with the component every chapter since 19.3 has assumed: the switch that lets many hosts reach one device at all.