“Memory Pooling Is Just Virtualisation”

"Memory pooling is just virtualisation with extra steps" is the belief a software-shaped intuition produces, and it is the most sophisticated misconception in this module. The person holding it knows exactly how a hypervisor divides memory, and is reasoning correctly from that knowledge to a conclusion the knowledge does not support.

The question this chapter turns on:

Does the capacity change which machine it belongs to — and what has to happen for it to?

Virtualisation divides capacity a host already owns. Pooling changes which host owns a region of physical capacity. Those are different verbs applied to different nouns, and the second one needs a hardware path, an ordered sequence, and an authority that no single host has.

1. A Boundary Software Cannot Cross

The belief is assembled from an accurate model of the wrong layer.

Everything in the left column is real and is not pooling. The right column is a boundary — the boundary of the machine the hypervisor runs on — and crossing it is the entire content of the thing pooling does.

That boundary is a physical fact, which is why software cannot legislate its way over it: a page table entry is a promise that an address means something, and it is not a wire.

2. How To Use This Chapter

Each of the nine dimensions below is a working test of the claim, and every one answers the same seven questions:

3. The One-Sentence Model

Virtualisation re-maps capacity a host already owns and can already reach; pooling changes the owner of physical capacity, which needs a path the host did not have, an ordered detach-and-attach sequence, an authority above the host to decide it, and a failure domain that now spans more than one machine.

4. What This Chapter Owns

The boundary with 29.5 is worth stating. That chapter examines what composition costs when it works. This one examines whether the mechanism exists at all in software — which is a prior question, and the one somebody asks before they are willing to read the other.

5. Teaching-Model Boundary And Source Discipline

Every model in this chapter is a teaching model, and each computes a property of a CLAIM rather than of a platform.

Nothing in this chapter states a normative detail of any specification. No opcode, layout, field width, encoding, register definition, timing guarantee, hot-plug flow, management interface, negotiation step or specification revision appears anywhere — checked by a scan over the finished page as well as by writing the models that way.

No capacity, price, latency or product figure is attributed to anything real. Every number is an illustrative parameter in arbitrary units, and the model headers say so.

6. Test 1 — Can The Owner Change?

The claim under test. That pooling is partitioning done by different software.

What the claim would require. That capacity never changed host.

The measurement. A region four hosts can reach:

// RTL 1 - who owns the bytes, and can that change?
//
// Virtualisation divides capacity a host ALREADY OWNS among the things running
// on it. Pooling changes WHICH HOST owns a region of physical capacity. Those
// are different verbs applied to different nouns, and the count that separates
// them is how many hosts could ever own a given region: exactly one for
// virtualised capacity, more than one for pooled capacity.
//
//   BAD  : "it is just software deciding who gets the memory"
//   GOOD : count the hosts that could own this region; one means partitioning,
//          more than one means pooling
//
// TEACHING MODEL. Illustrative host counts. It is not a model of CXL or of any
// pooling implementation, and it contains no opcode, layout, field width,
// encoding, register definition, timing guarantee or specification revision
// from any published standard.
//
//   INITIALIZATION CONTRACT. Sequential; the state is the current owner.
//     power-on/reset  : owner_host = 0, meaning no host owns the region yet
//     initialisation  : `assign_to` sets the first owner - a one-shot per
//                       region in the sense that the first assignment is the
//                       one that makes the region usable at all
//     re-initialise   : a further `assign_to` is a REBIND and is legal while
//                       the region is owned; it is the operation virtualisation
//                       has no equivalent of, and the model counts it
//     telemetry       : moves_between_hosts must read zero on a partitioned
//                       system and non-zero on a pooled one
module who_owns_the_bytes #(parameter int SOFTWARE_DECIDES = 0) (
  input  logic clk, rst_n,
  input  logic       assign_to, release_it, assess,
  input  logic [7:0] target_host, reachable_hosts,
  output logic [7:0] owner_host, moves_between_hosts, n_assessments, n_misread,
  output logic       is_pooled, reported_pooled,
  output logic       own_err
);
  logic [7:0] own_q, mov_q;
  assign owner_host          = own_q;
  assign moves_between_hosts = mov_q;

  // The truth: a region is pooled when more than one host could own it. One
  // reachable host is a partitioned region however the software divides it.
  assign is_pooled = (reachable_hosts > 8'd1);

  // The whole review point: a reader for whom the deciding software is the
  // whole mechanism, so nothing is ever pooled.
  assign reported_pooled = (SOFTWARE_DECIDES != 0) ? 1'b0 : is_pooled;

  // SAFETY-OF-CLAIM VIOLATION: a region reachable by several hosts was
  // reported as ordinary partitioned capacity.
  assign own_err = assess && !reported_pooled && is_pooled;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      own_q <= 8'd0; mov_q <= 8'd0; n_assessments <= 8'd0; n_misread <= 8'd0;
    end else begin
      // ONE assignment, priority written down: a release beats a simultaneous
      // assignment, because a region being torn down must not acquire a fresh
      // owner in the same cycle.
      if (release_it)     own_q <= 8'd0;
      else if (assign_to) own_q <= target_host;
      else                own_q <= own_q;

      // A move is an assignment to a DIFFERENT host while the region is owned.
      // A first assignment from unowned is not a move, and re-assigning the
      // same host is not a move either.
      if (assign_to && !release_it && (own_q != 8'd0) && (target_host != own_q))
        mov_q <= (mov_q == 8'hFF) ? mov_q : mov_q + 8'd1;

      if (assess) begin
        n_assessments <= n_assessments + 8'd1;
        if (own_err) n_misread <= n_misread + 8'd1;
      end
    end
  end
endmodule

4 hosts can reach it : pooled=1 software_decides_says=0

The count that separates the two mechanisms is how many hosts could ever own a given region: exactly one for virtualised capacity, more than one for pooled capacity. The software-decides build reports nothing pooled, ever, because for it the deciding software is the whole mechanism.

The run drives the boundary in both directions — exactly one reachable host, which is partitioning; and exactly two, which is the smallest pool there is.

The ownership sequence is driven as well, and the definitions are the interesting part. A first assignment from unowned is not a move. Re-assigning the same host is not a move. An assignment to a different host while the region is owned is a move — and the model counts those separately, because "moves between hosts" is the number that reads zero on a partitioned system and non-zero on a pooled one.

Why the belief is reasonable. A hypervisor genuinely does decide who gets memory, and watching it do so is watching a control plane allocate capacity. The mechanism underneath is the part that is not visible from there.

What it costs to hold. A capacity plan that assumes rebalancing is a policy change, and a discovery that the policy has nowhere to execute.

What to say instead. "How many hosts could own this region? One is partitioning. More than one is a pool, and the difference is physical."

7. Test 2 — Software Can Only Map What The Hardware Can Reach

The claim under test. That the hypervisor can just map it.

What the claim would require. That mapping were sufficient.

The failure. A page table entry is a promise that an address means something. It is not a wire. If the physical capacity is on the other side of a link this host has no path to, no entry in any table makes it readable — the access does not arrive.

// RTL 2 - software can only map what the hardware can reach.
//
// A page table entry is a promise that an address means something. It is not a
// wire. If the physical capacity is on the other side of a link this host has
// no path to, no entry in any table makes it readable - the access does not
// arrive. Virtualisation operates entirely inside the set of things a host can
// physically reach; pooling changes that set. That is the whole difference,
// and it is a hardware difference.
//
//   BAD  : "the hypervisor can just map it"
//   GOOD : ask whether the host has a physical path to it FIRST, then ask
//          whether software has mapped it
//
// TEACHING MODEL. Two illustrative booleans per region.
//
//   INITIALIZATION CONTRACT (combinational + counters):
//     power-on/reset : counters zero; usability is a pure function
//     re-initialise  : not applicable
//     telemetry      : mapped and reachable are published SEPARATELY, which is
//                      what makes "mapped but unreachable" a diagnosable state
module physical_reachability #(parameter int MAPPING_IS_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic assess,
  input  logic mapped_by_software, physically_reachable,
  output logic [7:0] n_regions, n_unusable,
  output logic       actually_usable, mapped_not_reachable, reported_usable,
  output logic       map_err
);
  // The truth: a region is usable only when BOTH are true.
  assign actually_usable = mapped_by_software && physically_reachable;

  // The state the two readings disagree about, published on its own.
  assign mapped_not_reachable = mapped_by_software && !physically_reachable;

  // The whole review point: a reader for whom the mapping is the mechanism.
  assign reported_usable = (MAPPING_IS_ENOUGH != 0) ? mapped_by_software
                                                    : actually_usable;

  // SAFETY VIOLATION: a region was reported usable with no physical path.
  assign map_err = assess && reported_usable && !actually_usable;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_regions <= 8'd0; n_unusable <= 8'd0;
    end else if (assess) begin
      n_regions <= n_regions + 8'd1;
      if (map_err) n_unusable <= n_unusable + 8'd1;
    end
  end
endmodule

The measurement. A region mapped and not physically reachable:

mapped, not reachable : usable=0 mapped_not_reachable=1 mapping_is_enough_says=1

Virtualisation operates entirely inside the set of things a host can physically reach. Pooling changes that set. That is the whole difference, and it is a hardware difference.

The run drives all four states, and the two readings disagree in exactly one of them — mapped and unreachable. A disagreement confined to one cell of four is what makes this error survive review, because three quarters of the time the shortcut is right.

Why the belief is reasonable. In every system the believer has administered, everything the host could map was something the host could reach, so the two conditions have always moved together.

What it costs to hold. An architecture that assumes address space is the constraint, and a bring-up in which mapped memory produces no data.

What to say instead. "Is there a physical path to it? Ask that first, and ask about the mapping second."

8. Test 3 — Count The Stranded Bytes

The claim under test. That the hypervisor will balance it.

What the claim would require. That stranded capacity were recoverable in software.

Stranded capacity is capacity attached to a machine that is not using it, while another machine needs it. A hypervisor can redistribute memory inside the host it runs on and cannot move a byte across the boundary of the machine it runs on.

// RTL 3 - the capacity virtualisation cannot reach.
//
// Stranded capacity is capacity that is physically attached to a host which is
// not using it, while another host needs it. A hypervisor can redistribute
// memory INSIDE the host it runs on and cannot move a byte across the boundary
// of the machine it runs on. The stranded number is therefore invariant under
// virtualisation and is exactly what pooling is for.
//
//   BAD  : "the hypervisor will balance it"
//   GOOD : compute the stranded bytes, then ask which mechanism can move them
//
// TEACHING MODEL. Illustrative byte counts in arbitrary units. No capacity,
// density or figure from any specification or product appears.
//
//   INITIALIZATION CONTRACT (combinational + counters):
//     power-on/reset : counters zero
//     re-initialise  : not applicable
//     telemetry      : stranded_bytes is published beside recoverable_bytes,
//                      so a claim that software recovers it is checkable
module stranding_is_physical #(parameter int SOFTWARE_RECOVERS_IT = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [7:0]  attached_bytes, used_bytes, wanted_elsewhere,
  output logic [15:0] stranded_bytes, recoverable_bytes, movable_bytes,
  output logic [7:0]  n_assessments, n_overstated,
  output logic        anything_stranded, reported_recovered,
  output logic        strand_err
);
  logic [7:0] used_c;

  // A host cannot use more than is attached to it. Clamp rather than wrap.
  assign used_c = (used_bytes > attached_bytes) ? attached_bytes : used_bytes;

  // Stranded is what is attached to this host and not used by it.
  assign stranded_bytes = {8'd0, (attached_bytes - used_c)};

  // Movable is the part of that another host actually wants - the amount a
  // pooling mechanism could put to work. It is the minimum of the two.
  assign movable_bytes = (stranded_bytes > {8'd0, wanted_elsewhere})
                       ? {8'd0, wanted_elsewhere} : stranded_bytes;

  // The truth: virtualisation recovers NOTHING across a host boundary, so the
  // recoverable figure under software alone is zero.
  assign recoverable_bytes = (SOFTWARE_RECOVERS_IT != 0) ? stranded_bytes : 16'd0;

  assign anything_stranded = (stranded_bytes != 16'd0);

  // The whole review point: whether the reader believes software recovers it.
  assign reported_recovered = (recoverable_bytes != 16'd0);

  // SAFETY-OF-CLAIM VIOLATION: capacity across a host boundary was reported
  // recovered by a mechanism that cannot cross one.
  assign strand_err = assess && reported_recovered && anything_stranded;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assessments <= 8'd0; n_overstated <= 8'd0;
    end else if (assess) begin
      n_assessments <= n_assessments + 8'd1;
      if (strand_err) n_overstated <= n_overstated + 8'd1;
    end
  end
endmodule

The measurement. Two hundred units attached, sixty used, fifty wanted elsewhere:

200 attached, 60 used, 50 wanted elsewhere : stranded=140 movable=50 software_recovers=140

One hundred and forty stranded, of which fifty is what another host actually wants and a pooling mechanism could put to work. The honest build reports a software recovery of zero, which is not a pessimistic estimate — it is the number, because the mechanism cannot cross the boundary the capacity is on the wrong side of.

The software-recovers build reports the whole 140, which is the claim stated as a figure.

The stranded number is invariant under virtualisation, and that invariance is exactly what pooling is for.

Why the belief is reasonable. Hypervisors demonstrably do recover memory — from idle guests, through ballooning, by overcommitting. All of it happens inside one machine, and the constraint is invisible until a second machine is involved.

What it costs to hold. Stranded capacity left in the estimate as recoverable, and a capacity purchase to cover a shortfall that was already sitting idle one rack away.

What to say instead. "Compute the stranded bytes, then ask which mechanism can move them. Software moves none of it across a host boundary."

Figure 1 — the red box is the whole argument. Everything a hypervisor does happens to the left of it, and the capacity that is stranded is stranded because it is on the wrong side.

9. Test 4 — Name The Four Steps

The claim under test. That it just gets remapped.

What the claim would require. That an owner change were an assignment.

The failure. Moving a region from one host to another is a SEQUENCE: stop the old owner touching it, detach it, deal with whatever the old owner left in it, attach it to the new owner. Every step can fail and every step has to be ordered. Virtualisation has no step that does any of this, because it never changes which machine the capacity belongs to.

// RTL 4 - the sequence a region has to go through to change owner.
//
// Moving a region from one host to another is not an assignment. It is a
// SEQUENCE: stop the old owner touching it, detach it, deal with whatever the
// old owner left in it, attach it to the new owner. Every step can fail and
// every step has to be ordered. Virtualisation has no step that does any of
// this, because it never changes which machine the capacity belongs to.
//
//   BAD  : "it just gets remapped"
//   GOOD : name the four steps, and say what happens if the sequence is
//          interrupted between any two of them
//
// TEACHING MODEL. Sequential, and the four steps are abstract. No training,
// negotiation, hot-plug or reconfiguration flow from any specification appears.
//   Safety : a region is never attached to a fresh host while the previous
//            owner is still able to touch it.
//
//   INITIALIZATION CONTRACT:
//     power-on/reset : step 0, nothing quiesced, nothing detached
//     initialisation : `start_move` begins a sequence; it is a one-shot and a
//                      second one while a move is live is IGNORED rather than
//                      restarting, because restarting mid-sequence is how a
//                      region ends up attached twice
//     re-initialise  : `abort_move` returns to step 0 and is legal at any point
//                      in the sequence; it is idempotent and it DOMINATES a
//                      simultaneous step, because an abort a step could
//                      survive would not be an abort
//     partial undo   : `resume_owner` clears the quiesce WITHOUT clearing the
//                      detach - the old owner is brought back while the region
//                      is still detached. It is the hazard the attach
//                      conjunction refuses, and it is a different operation
//                      from an abort, which unwinds everything
//     telemetry      : steps_done is published, so an interrupted move is
//                      visible rather than inferred
module rebind_sequence #(parameter int REMAP_IS_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic start_move, do_quiesce, do_detach, do_reattach, abort_move,
  input  logic resume_owner, assess,
  output logic [7:0] steps_done, n_moves, n_unsafe,
  output logic       quiesced, detached, reattached, safe_to_attach,
  output logic       attach_allowed,
  output logic       seq_err
);
  logic q_q, d_q, r_q, live_q;
  logic [7:0] step_q, mv_q;

  assign quiesced   = q_q;
  assign detached   = d_q;
  assign reattached = r_q;
  assign steps_done = step_q;
  assign n_moves    = mv_q;

  // The truth: a fresh host may be attached only after the region has been
  // quiesced AND detached. Either one alone leaves the old owner able to touch
  // capacity the new owner now believes it holds.
  assign safe_to_attach = q_q && d_q;

  // The whole review point: a reader for whom the attach is the whole move.
  assign attach_allowed = (REMAP_IS_ENOUGH != 0) ? 1'b1 : safe_to_attach;

  // SAFETY VIOLATION: an attach was permitted before the sequence licensed it.
  assign seq_err = assess && do_reattach && attach_allowed && !safe_to_attach;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      q_q <= 1'b0; d_q <= 1'b0; r_q <= 1'b0; live_q <= 1'b0;
      step_q <= 8'd0; mv_q <= 8'd0; n_unsafe <= 8'd0;
    end else begin
      // ONE assignment per register, priority written down. An abort dominates
      // every step in the same cycle.
      if (abort_move)                   live_q <= 1'b0;
      else if (start_move && !live_q)   live_q <= 1'b1;
      else                              live_q <= live_q;

      // An operator can resume the old owner mid-sequence - a host is brought
      // back before the move completes. That clears the quiesce and leaves the
      // detach standing, which is the state `safe_to_attach`'s conjunction
      // exists to refuse. Without this input the state is unreachable and the
      // conjunct is untestable: a mutation campaign proved exactly that.
      if (abort_move)                       q_q <= 1'b0;
      else if (resume_owner)                q_q <= 1'b0;
      else if (do_quiesce && live_q)        q_q <= 1'b1;
      else                                  q_q <= q_q;

      if (abort_move)                       d_q <= 1'b0;
      else if (do_detach && live_q && q_q)  d_q <= 1'b1;
      else                                  d_q <= d_q;

      if (abort_move)                             r_q <= 1'b0;
      else if (do_reattach && attach_allowed)     r_q <= 1'b1;
      else                                        r_q <= r_q;

      if (abort_move)      step_q <= 8'd0;
      else if (live_q && (do_quiesce || do_detach || do_reattach))
                           step_q <= (step_q == 8'hFF) ? step_q : step_q + 8'd1;
      else                 step_q <= step_q;

      if (start_move && !live_q && !abort_move)
        mv_q <= (mv_q == 8'hFF) ? mv_q : mv_q + 8'd1;

      if (assess && seq_err) n_unsafe <= n_unsafe + 8'd1;
    end
  end
endmodule

The measurement. An attach with nothing quiesced:

reattach with nothing quiesced : safe=0 allowed=0 remap_is_enough_allows=1

The remap-is-enough build attaches the region to a fresh host while the old owner can still touch it, which is the failure the ordering exists to prevent. The run then drives the sequence properly — quiesce, detach, attach — and the sequencing build permits the attach at exactly the right moment.

The partial undo is the case worth keeping

The run drives a resume_owner event: the old owner brought back while the region is still detached. Quiesced falls, detached stands, and the attach must be refused.

That state is the only one in which the quiesced term of the safety conjunction does any work — and the model could not represent it at first. Section 18 records how the campaign found that, and why the answer was to change the model rather than the stimulus.

Why the belief is reasonable. From above, a move looks like a reassignment: the region was here and now it is there. The four steps are underneath, and none of them has a software analogue to build intuition from.

What it costs to hold. A move implemented as an assignment, and a region attached to two owners for as long as the old one takes to notice.

What to say instead. "Name the four steps, and say what happens if the sequence is interrupted between any two of them."

10. Test 5 — How Wide Is The Blast?

The claim under test. That the failure domain is the same.

What the claim would require. That a fault reached one machine either way.

A virtualised region has exactly one machine behind it, so a failure in that capacity takes down one machine. A pooled region has every host that could own it in its potential blast radius, and a failure in the shared path takes down as many as are attached.

// RTL 5 - how many hosts a fault can reach.
//
// A virtualised region has exactly one machine behind it, so a failure in that
// capacity takes down one machine. A pooled region has every host that could
// own it in its potential blast radius, and a failure in the shared path takes
// down as many of them as are attached. The containment domain is therefore
// different, and it is a hardware property rather than a policy one.
//
//   BAD  : "the failure domain is the same, it is still memory"
//   GOOD : count the hosts a single failure can reach, on each mechanism
//
// TEACHING MODEL. Illustrative host counts.
//
//   INITIALIZATION CONTRACT (combinational + counters):
//     power-on/reset : counters zero
//     re-initialise  : not applicable
//     telemetry      : blast_hosts is published, so a containment claim is a
//                      number rather than an assurance
module blast_radius #(parameter int SAME_FAILURE_DOMAIN = 0) (
  input  logic clk, rst_n,
  input  logic       assess, shared_path_fails,
  input  logic [7:0] attached_hosts, tolerated_hosts,
  output logic [7:0] blast_hosts, n_assessments, n_understated,
  output logic       contained, reported_contained,
  output logic       radius_err
);
  // A failure in the shared path reaches every attached host. With no failure
  // it reaches none.
  assign blast_hosts = shared_path_fails ? attached_hosts : 8'd0;

  // The truth: containment holds when the blast stays inside what the design
  // was built to tolerate.
  assign contained = (blast_hosts <= tolerated_hosts);

  // The whole review point: a reader who assumes one machine, always.
  assign reported_contained = (SAME_FAILURE_DOMAIN != 0)
                            ? (8'd1 <= tolerated_hosts) : contained;

  // SAFETY-OF-CLAIM VIOLATION: a blast wider than the tolerance was reported
  // contained, because the reader priced a single-machine failure.
  assign radius_err = assess && reported_contained && !contained;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assessments <= 8'd0; n_understated <= 8'd0;
    end else if (assess) begin
      n_assessments <= n_assessments + 8'd1;
      if (radius_err) n_understated <= n_understated + 8'd1;
    end
  end
endmodule

The measurement. Eight hosts attached, a tolerance of one, and the shared path fails:

8 hosts attached, 1 tolerated, shared path fails : blast=8 contained=0 same_domain_says=1

The same-failure-domain build prices a single-machine failure, which is correct for virtualised capacity and wrong by a factor of eight here.

The containment domain is a hardware property rather than a policy one, and the run drives a tolerance exactly equal to the blast — which contains it — and a tolerance of zero, where even one machine is too many and the weak build is right by accident.

Why the belief is reasonable. In a virtualised world the failure domain really is one machine, always, and there has never been a reason to ask how wide it is.

What it costs to hold. An availability design sized for single-machine failures, deployed on a topology where one shared path takes eight.

What to say instead. "Count the hosts a single failure can reach, on each mechanism. They are different numbers."

11. Test 6 — Who Is Allowed To Decide?

The claim under test. That the hypervisor allocates it.

What the claim would require. That a host could decide about capacity spanning hosts.

A hypervisor has authority over the machine it runs on and none at all over its neighbours. Deciding that a region moves from host 3 to host 7 is a decision neither host can make on its own, because each would be deciding about capacity the other believes it holds.

// RTL 6 - who is allowed to decide.
//
// A hypervisor has authority over the machine it runs on and none at all over
// its neighbours. Deciding that a region moves from host 3 to host 7 is a
// decision neither host can make on its own, because each would be deciding
// about capacity the other believes it holds. Pooling therefore needs an
// authority ABOVE the host, and that authority is a component that has to exist,
// be reachable, and be trusted.
//
//   BAD  : "the hypervisor allocates it"
//   GOOD : name the authority that can decide about capacity spanning two
//          hosts, and say what happens when it is unreachable
//
// TEACHING MODEL. Abstract authority flags; no management protocol, interface
// or component from any specification appears.
//
//   INITIALIZATION CONTRACT (combinational + counters):
//     power-on/reset : counters zero
//     re-initialise  : not applicable
//     telemetry      : decided_by is published per decision, so a decision
//                      taken at the wrong level is visible after the fact
module control_plane_authority #(parameter int HOST_CAN_DECIDE = 0) (
  input  logic clk, rst_n,
  input  logic assess, decision_needed,
  input  logic spans_hosts, fabric_authority_present,
  output logic [7:0] decided_by, n_decisions, n_unauthorised,
  output logic       authority_ok, reported_ok,
  output logic       auth_err
);
  // 0 nobody, 1 the host, 2 the fabric authority.
  assign decided_by = (!decision_needed)          ? 8'd0
                    : (spans_hosts && fabric_authority_present) ? 8'd2
                    : (!spans_hosts)              ? 8'd1
                    : 8'd0;

  // The truth: a decision spanning hosts needs the authority above them; a
  // decision inside one host does not.
  assign authority_ok = (!decision_needed)
                     || (spans_hosts ? fabric_authority_present : 1'b1);

  // The whole review point: a reader for whom the host decides everything.
  assign reported_ok = (HOST_CAN_DECIDE != 0) ? 1'b1 : authority_ok;

  // SAFETY VIOLATION: a cross-host decision was permitted with no authority
  // able to make it.
  assign auth_err = assess && reported_ok && !authority_ok;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_decisions <= 8'd0; n_unauthorised <= 8'd0;
    end else if (assess) begin
      n_decisions <= n_decisions + 8'd1;
      if (auth_err) n_unauthorised <= n_unauthorised + 8'd1;
    end
  end
endmodule

The measurement. A cross-host decision with no authority above the hosts:

cross-host decision, no authority : decided_by=0 ok=0 host_can_decide_says=1

Nobody can make this decision, and the host-can-decide build permits it anyway. The run drives all four cases — cross-host with an authority (the fabric decides), inside one host (the host decides), cross-host without one (nobody), and no decision needed at all — and the authority-naming build is right on every one.

Pooling therefore needs an authority ABOVE the host, and that authority is a component that has to exist, be reachable, and be trusted. A component whose absence is invisible until the first move.

Why the belief is reasonable. Within one machine the hypervisor is the authority, completely and without exception. Nothing in that experience suggests there is a level above it.

What it costs to hold. A pooling deployment with no control plane in the design, and a first move with nobody empowered to authorise it.

What to say instead. "Name the authority that can decide about capacity spanning two hosts, and say what happens when it is unreachable."

12. Test 7 — A Debt, Or Inventory?

The claim under test. That pooling is overcommitment with extra steps.

What the claim would require. That the two carried the same risk.

Overcommitment is the classic virtualisation move: promise more than exists and rely on nobody claiming it all at once. Pooling is the opposite shape — capacity that physically exists and is not yet assigned to anybody.

// RTL 7 - promising capacity you do not have, against having capacity you have
// not promised.
//
// Overcommitment is the classic virtualisation move: promise more than exists
// and rely on nobody claiming it all at once. Pooling is the opposite shape -
// capacity that physically exists and is not yet assigned to anybody. One
// carries the risk that a promise cannot be met; the other carries the risk
// that capacity sits idle. Calling them the same thing inverts the risk being
// managed.
//
//   BAD  : "it is overcommitment with extra steps"
//   GOOD : compare promised against held. Promised above held is
//          overcommitment; held above assigned is a pool
//
// TEACHING MODEL. Illustrative byte counts in arbitrary units.
//
//   INITIALIZATION CONTRACT (combinational + counters):
//     power-on/reset : counters zero
//     re-initialise  : not applicable
//     telemetry      : overcommitted_bytes and unassigned_bytes are published
//                      separately, because one is a debt and the other is
//                      inventory
module overcommit_vs_pool #(parameter int SAME_SHAPE = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [7:0]  promised_bytes, held_bytes, assigned_bytes,
  output logic [15:0] overcommitted_bytes, unassigned_bytes,
  output logic [7:0]  n_assessments, n_misjudged,
  output logic        is_overcommitted, is_pooled, reported_overcommitted,
  output logic        commit_err
);
  logic [7:0] assigned_c;

  // Nothing can be assigned that is not held.
  assign assigned_c = (assigned_bytes > held_bytes) ? held_bytes : assigned_bytes;

  // A debt: promised beyond what is held. Saturates at zero rather than
  // wrapping, because "negative overcommitment" is not a quantity.
  assign overcommitted_bytes = (promised_bytes > held_bytes)
                             ? {8'd0, (promised_bytes - held_bytes)} : 16'd0;

  // Inventory: held and not yet assigned to anybody.
  assign unassigned_bytes = {8'd0, (held_bytes - assigned_c)};

  assign is_overcommitted = (overcommitted_bytes != 16'd0);
  assign is_pooled        = (unassigned_bytes    != 16'd0);

  // The whole review point: a reader for whom both shapes are the same risk.
  assign reported_overcommitted = (SAME_SHAPE != 0)
                                ? (is_overcommitted || is_pooled)
                                : is_overcommitted;

  // SAFETY-OF-CLAIM VIOLATION: unassigned inventory was reported as a debt,
  // which is the risk register reading backwards.
  assign commit_err = assess && reported_overcommitted && !is_overcommitted;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assessments <= 8'd0; n_misjudged <= 8'd0;
    end else if (assess) begin
      n_assessments <= n_assessments + 8'd1;
      if (commit_err) n_misjudged <= n_misjudged + 8'd1;
    end
  end
endmodule

The measurement. Eighty promised, a hundred held, sixty assigned:

promised 80, held 100, assigned 60 : overcommitted=0 unassigned=40 same_shape_says=1

No debt and forty of inventory, and the same-shape build enters the inventory in the risk register as a debt. One carries the risk that a promise cannot be met; the other carries the risk that capacity sits idle. Calling them the same thing inverts the risk being managed.

The run drives all four combinations, including the one where both are present at once — a genuine debt and unassigned inventory, which is a real state and the one where the weak build happens to be right.

Why the belief is reasonable. Both look like "capacity accounting that does not add up to what the hardware has", and from a spreadsheet they are indistinguishable.

What it costs to hold. A risk register with the wrong entry, and mitigations designed for a shortfall when the exposure is idle capital.

What to say instead. "Promised above held is a debt. Held above assigned is inventory. They are opposite risks."

13. Test 8 — What Does The Pool Cost Per Access?

The claim under test. That memory is memory.

What the claim would require. That pooled bytes cost what near bytes cost.

// RTL 8 - pooled capacity is further away.
//
// Capacity a hypervisor hands to a guest is on the same machine and costs what
// it always cost. Capacity that arrives from a pool crossed a link to get
// there, and that crossing is a number. A plan that treats pooled bytes as
// interchangeable with near bytes is wrong by the added latency on every
// access to them, and the crossover - how much capacity relief is worth how
// much added latency - is the decision the plan is actually making.
//
//   BAD  : "memory is memory"
//   GOOD : state the added latency, state the access share that lands in the
//          pool, and find the point where the trade reverses
//
// TEACHING MODEL. All latencies are illustrative integers in arbitrary units.
// None is a CXL figure and none is attributed to any product.
//
//   INITIALIZATION CONTRACT (combinational + counters):
//     power-on/reset : counters zero
//     re-initialise  : not applicable
//     telemetry      : effective_lat is published beside near_lat so the cost
//                      of the pool is visible rather than assumed
module latency_is_not_free #(parameter int MEMORY_IS_MEMORY = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [7:0]  near_lat, pooled_lat, pool_share_pct, budget,
  output logic [15:0] added_lat, effective_lat,
  output logic [7:0]  n_evals, n_missed,
  output logic        within_budget, reported_within,
  output logic        lat_err
);
  logic [31:0] e_q;
  logic [7:0]  share_c;

  // A share is a percentage; anything above a hundred is a bad measurement.
  assign share_c = (pool_share_pct > 8'd100) ? 8'd100 : pool_share_pct;

  // The pooled path costs more, or the pool is pointless. Saturate at zero
  // rather than wrapping if a caller inverts them.
  assign added_lat = (pooled_lat > near_lat)
                   ? {8'd0, (pooled_lat - near_lat)} : 16'd0;

  // The weighted mean: near latency everywhere, plus the added cost on the
  // share of accesses that land in the pool. The product reaches
  // 255 x 100 = 25,500 in 32 bits before the division brings it back.
  assign e_q = {24'd0, near_lat}
             + (({16'd0, added_lat} * {24'd0, share_c}) / 32'd100);
  assign effective_lat = (e_q > 32'd65535) ? 16'd65535 : e_q[15:0];

  // The truth: the plan holds when the EFFECTIVE latency fits the budget.
  assign within_budget = (effective_lat <= {8'd0, budget});

  // The whole review point: a reader who budgets the near latency and treats
  // pooled bytes as the same bytes.
  assign reported_within = (MEMORY_IS_MEMORY != 0)
                         ? ({8'd0, near_lat} <= {8'd0, budget}) : within_budget;

  assign lat_err = evaluate && reported_within && !within_budget;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_missed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (lat_err) n_missed <= n_missed + 8'd1;
    end
  end
endmodule

The measurement. A near latency of 10, a pooled latency of 40, half the accesses landing in the pool, against a budget of 20:

near 10, pooled 40, 50% in pool, budget 20 : added=30 effective=25 memory_is_memory_says=1

The pool adds 30 on the accesses that reach it, and half of them gives an effective 25 — outside a budget of 20. The memory-is-memory build budgets the near latency and accepts the plan.

All latencies are illustrative integers in arbitrary units. None is a CXL figure and none is attributed to any product. The weighted mean is the durable part: capacity relief is bought with latency on the share of accesses that land in the pool, and the crossover is the decision the plan is actually making.

The run drives the boundary — a budget exactly equal to the effective latency, which holds — and both ends of the share, and an inverted case where the pooled path is somehow faster, which adds zero rather than a negative, because a bargain is not a quantity.

Why the belief is reasonable. Capacity a hypervisor hands to a guest is on the same machine and costs what it always cost. Nothing in a virtualised world has a distance.

What it costs to hold. A latency budget written for near memory, applied to a workload a share of whose accesses now cross a link.

What to say instead. "State the added latency and the share that lands in the pool. The effective number is a weighted mean, and it is the one the budget has to hold."

14. Test 9 — Write Down What Would Have To Be True

The claim under test. All of them, at once.

What the claim would require. Four conditions, and all four.

// RTL 9 - what would have to be true for "just virtualisation" to hold?
//
// Same discipline, fourth chapter running. Four conditions: capacity never
// changes host, software can map anything the host needs, stranded bytes are
// recoverable without hardware, and no authority above the host is required.
//
// TEACHING MODEL. Four illustrative booleans.
//
//   INITIALIZATION CONTRACT (combinational + counters):
//     power-on/reset : counters zero
//     re-initialise  : not applicable
//     telemetry      : met_pct beside the conjunction
module pooling_conditions #(parameter int MOSTLY_SOFTWARE = 0) (
  input  logic clk, rst_n,
  input  logic       assess,
  input  logic       capacity_never_moves, software_can_map_it,
  input  logic       stranding_recoverable, no_authority_needed,
  output logic [7:0] conditions_met, n_assessments, n_overclaims,
  output logic [15:0] met_pct,
  output logic       would_hold, claimed_holds,
  output logic       just_err
);
  logic [31:0] m_q;
  assign conditions_met = {7'd0, capacity_never_moves} + {7'd0, software_can_map_it}
                        + {7'd0, stranding_recoverable} + {7'd0, no_authority_needed};
  // No clamp: four one-bit values over four cannot exceed a hundred.
  assign m_q = ({24'd0, conditions_met} * 32'd100) / 32'd4;
  assign met_pct = m_q[15:0];
  assign would_hold = (conditions_met == 8'd4);
  assign claimed_holds = (MOSTLY_SOFTWARE != 0) ? (conditions_met >= 8'd3) : would_hold;
  assign just_err = assess && claimed_holds && !would_hold;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assessments <= 8'd0; n_overclaims <= 8'd0;
    end else if (assess) begin
      n_assessments <= n_assessments + 8'd1;
      if (just_err) n_overclaims <= n_overclaims + 8'd1;
    end
  end
endmodule

The measurement. Three of four met:

3 of 4 conditions : met=75% would_hold=0 mostly_software_says=1

Seventy-five percent, and a conjunction has no partial credit.

What to say instead. "Here are the four things that would have to be true. Any one of them failing settles it, and all four fail."

15. The Misconception Assembled

Nine tests, one summary.

// RTL 10 - the misconception examined. Nine tests, one summary.
// "Software decides who gets it" is bit 0: a true statement about a control
// plane, and one sixth of an argument about a mechanism.
module pool_review_signoff #(parameter int SOFTWARE_DECIDES_IS_PROOF = 0) (
  input  logic clk, rst_n,
  input  logic        review,
  input  logic        software_decides, owner_can_change, reachability_checked,
  input  logic        stranding_measured, sequence_named, conditions_checked,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_reviews, n_sound, n_claimed,
  output logic        mis_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~software_decides;
  assign fail_mask[1] = ~owner_can_change;
  assign fail_mask[2] = ~reachability_checked;
  assign fail_mask[3] = ~stranding_measured;
  assign fail_mask[4] = ~sequence_named;
  assign fail_mask[5] = ~conditions_checked;
  assign conditions_met = {15'd0, software_decides} + {15'd0, owner_can_change}
                        + {15'd0, reachability_checked} + {15'd0, stranding_measured}
                        + {15'd0, sequence_named} + {15'd0, conditions_checked};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp: six one-bit values over six cannot exceed a hundred.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  assign claimed = (SOFTWARE_DECIDES_IS_PROOF != 0) ? software_decides : truly_sound;
  assign sound = claimed;
  assign mis_err = review && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reviews <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (review) begin
      n_reviews <= n_reviews + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The measurement. Two views of the same argument:

the stranding was never measured : mask=001000 met=5 sound=83%
software decides who gets it : mask=111110 met=1 sound=16%

The first line is a serious argument with one condition unmet — bit 3, the stranding was never measured. Five of six, and the missing measurement is a subtraction on two numbers a platform already has.

The second line is the misconception. Bit 0 is clear — software does decide who gets it, in both mechanisms — and nothing else was checked. Sixteen percent of an argument, from a true statement about a control plane.

Figure 2 — bit 0 is true of both mechanisms, which is why it distinguishes neither. A premise shared by the two things you are trying to tell apart cannot tell them apart.

16. Quantitative Reasoning

Every figure here is a teaching parameter or a value derived from one and asserted by the testbench. None is a measurement of a real system, none is a price, and none is attributed to any product or operator.

Ownership, derived. A region reachable by one host is partitioned; by two or more, pooled. Two is the smallest pool there is, and the run drives exactly that boundary. The move counter distinguishes three cases that look alike: a first assignment from unowned is not a move, re-assigning the same host is not a move, and an assignment to a different host while owned is — so the counter reads zero on a partitioned system by construction.

The move counter's ceiling. An 8-bit counter saturates at 255, and the run drives 254 further moves to reach it. A counter that wraps turns "this region has moved constantly" into "this region has never moved", which is the direction a saturating counter refuses.

Reachability. Two booleans, four states, and the readings disagree in exactly one — mapped and unreachable. A shortcut that is right three quarters of the time is a shortcut that survives review.

Stranding, derived. Two hundred attached with sixty used strands 200 − 60 = 140. Another host wanting fifty makes min(140, 50) = 50 of it movable. The software-recoverable figure is zero — not an estimate, a consequence of the boundary. The general form: stranded = attached − used, movable = min(stranded, wanted elsewhere), and neither term contains a fleet size.

The sequence. Four steps, and the safety condition is a conjunction of two of them. Quiesced alone is not safe and detached alone is not reachable, which makes the resume-owner state — quiesced false, detached true — the only one in which the conjunction's first term does any work. Section 18 records that the model could not reach that state at first.

Blast radius, derived. Eight attached hosts against a tolerance of one is a blast of 8 against a budget of 1 — wrong by a factor of eight. The general form is blast = attached hosts on a shared-path failure, and virtualised capacity makes that number 1 by construction, which is why nobody used to ask.

Risk shape, derived. Promised 80 against held 100 gives a debt of 0; held 100 against assigned 60 gives inventory of 40. The two quantities are independent — the run drives all four combinations, including both at once — and the same-shape reading computes their union, which is right whenever a debt exists and wrong whenever only inventory does.

Latency, derived. A near latency of 10 and a pooled latency of 40 gives an added cost of 30 on the accesses that reach the pool. At a 50 percent share the effective latency is 10 + (30 × 50)/100 = 25. The general form is a weighted mean, near + added × share / 100, and it is the number a budget has to hold — not the near latency, and not the pooled one.

Conditions, derived. Four conditions with three met is 3 × 100 / 4 = 75 percent, and the claim requires four of four.

The sign-off arithmetic. Six conditions; five met is 5 × 100 / 6 = 83 percent under integer division, and one met is 100 / 6 = 16 percent.

17. Verification Method

Order of work

names.txt → ten models, each compiled alone → model-expressiveness review → boolean-tautology review → width review → structural gates → testbench → legal baseline → PASS → mutation campaign → re-baseline after every change → MDX assembled from the verified sources

A mutation campaign on a failing baseline is invalid, and both campaigns in this chapter ran against a green one.

Independent oracles

chkv prints got against expected, which is what lets an oracle be wrong out loud. In this chapter it caught one, mine, recorded in section 18.

X and Z rejected explicitly

chk(c, …) tests c !== 1'b1, so an X-valued condition fails rather than passing. chkv(got, exp, …) reduces the result and reports an explicit X/Z failure before comparing. The scripted output-connectivity gate returns zero on all ten models.

Pulses are latched, never sampled

Every evidence output — own_err, map_err, strand_err, seq_err, radius_err, auth_err, commit_err, lat_err, just_err, mis_err — is caught by a continuous always @(posedge clk) monitor into a sticky bit, asserted outside any conditional, in a window containing a clock edge.

Stimulus never lands on the active edge, and reset is released after it

step_clk is @(posedge clk); #1;, and reset release lands one delta after the edge.

Initialisation contracts are stated, not implied

Two models carry real sequential state, and both headers answer the same questions explicitly.

The ownership model's release dominates a simultaneous assignment, because a region being torn down must not acquire a fresh owner in the same cycle. The rebind model's abort dominates every step, and it carries a second, different operation — a partial undo that clears the quiesce and leaves the detach standing. Distinguishing a full abort from a partial undo is what makes the safety conjunction testable, and section 18 records that the model had neither at first.

Both builds are always instantiated

Every model has both its counting build and its shortcut build wired to the same stimulus, and the testbench asserts the internal figures on both.

Safety, liveness and performance kept apart

Safety — a region is never attached to a fresh host while the previous owner can still touch it. A region is never reported usable with no physical path. Capacity across a host boundary is never reported recovered by a mechanism that cannot cross one. None requires an assumption.

Liveness — nothing in this chapter is a liveness claim. The sequencing model deliberately refuses an attach, and whether an indefinitely refused attach is a hang is a design decision the model exposes rather than settles.

Performance — one model is explicitly a performance claim: the latency comparison is about a budget, and a plan outside its budget is slow rather than broken.

18. Baseline Defects Found Before Mutation

RTL defects — none at baseline. One found by the campaign, and it was a MODEL EXPRESSIVENESS GAP.

The ten models compiled clean and the testbench passed on its first run with 269 checks. The campaign then found that the rebind model could not represent the state its own safety conjunction exists to refuse.

detached was set only by a path already conditioned on quiesced, so detached implied quiesced and safe_to_attach = quiesced && detached equalled detached in every reachable state. No stimulus could separate them.

The fix was to the model, not to the testbench and not to the mutation. A resume_owner input was added — the old owner brought back while the region is still detached, which is a hazard a real rebind sequence has and this model did not. The state is now reachable, the conjunct is load-bearing, the mutation dies, and the chapter gained its sharpest sequencing case.

That is the §8 gate finding a defect one chapter after the gate was written, and it is the third instance of the class in two batches — after 30.7's correlation model and this batch's ordering model. A mutation that cannot be killed is not always an equivalent mutant; sometimes the model has no way to represent the experiment.

Testbench defects — none. Wrong oracles — one, mine.

A check whose expected value was invalidated by a loop inserted above it. The simultaneous release-versus-assign case asserted a move count of 1, from before a 254-iteration loop was added to drive the counter to its ceiling. The correct value is 255, saturated.

Second instance of this class in the batch. The habit that prevents it: re-derive every expectation that follows an inserted case, rather than assuming the insertion was local.

Coverage gaps found by the structural gates

The simwrite false positive is worth recording. It flagged a wrapped else if (…) whose condition sits on a different line from its assignment, which the gate read as an unguarded sibling statement. Line-continuation handling was added, both controls were re-run, and the gate was re-validated across all eight chapters of batches 030 to 032: zero hits, 345 registers scanned.

A gate that mis-flags correct code is a gate nobody runs, which is why both new gates this batch ship with a positive and a negative control.

One equivalent mutant, withdrawn by boolean algebra

The mutation turning start_move && !live_q into start_move could not be killed, and the reason is arithmetic rather than coverage. With the abort low, the original computes live | (start & ~live) and the mutant computes start ? 1 : live. Both reduce to live | start. No input separates them, in any state.

Withdrawn and not counted as a kill, and replaced with start_move && live_q, which makes a first start fail to open the sequence — so nothing can be quiesced, nothing detached, and the replacement dies on the first step of the first move.

The !live_q guard is still correct code. It expresses "a second start does not restart a live move" directly rather than relying on the reader to perform the reduction, and the separate guard on the move counter — which is not equivalent, and was killed — is what enforces the intent.

Compiler-warning findings

Under -Wall the ten models produce no truncation, select-width, signedness, cast, multiplication-width, shift-width or loop-width warnings.

Six width results were reasoned rather than trusted.

• attached_bytes − used_c and held_bytes − assigned_c are 8-bit subtractions that cannot underflow, because each subtrahend is clamped to its minuend first. The campaign kills both clamp-removal mutations.
• promised_bytes − held_bytes is guarded by promised > held rather than clamped — the same protection written the other way round — and the mutation that removes the guard is killed by the case where nothing is owed.
• added_lat × share_c reaches 255 × 100 = 25,500 in a 32-bit context before the division by 100. The share_c clamp bounds the second operand, and removing it is killed by a share of 150.
• mov_q + 1 and step_q + 1 saturate at 8'hFF explicitly. The move counter's saturation is driven to its ceiling by a 254-iteration loop.
• conditions_met × 100 / 4 and / 6 cannot exceed 100: a sum of one-bit values over its own denominator.

Simulator constraints

Icarus Verilog 13.0 rejects ref task arguments, carried forward.

19. Mutation Testing

73 mutations attempted, 1 withdrawn as equivalent, 72 non-equivalent injected, 72 killed. Zero unexplained survivors.

Four survivors on the first run, and three of them were in one model.

Three of four were in the only model with a real state machine

rebind_sequence is the only model here with an ordered state machine, and it produced three of the four survivors: an unreachable conjunct, an unisolated ordering precondition, and a boolean-equivalent guard.

Sequencing logic has more reachable states than a scoring model and therefore more states a stimulus can miss. That is an argument for driving sequences to their boundaries rather than through their happy path, and it is the same lesson the batch-030 coherency chapter reached from the other direction.

The fourth was a counter never driven to its ceiling

An 8-bit move counter, saturating at 255, with a stimulus that drove one move. 254 more were added in a loop, and the saturation is now asserted at the ceiling and one move past it.

The classification rule

Never add an assertion for a survivor before classifying it.

The fifth row is the one this chapter exercised, and distinguishing it from equivalent is the whole skill: an equivalent mutant means the code is tighter than it looks, and an expressiveness gap means the model is looser than the world.

20. Synthesis And Implementation Reality

These models are not meant to be synthesised. What follows is the honest reading of what the mechanisms they abstract cost, because "pooling is just software" is a cost argument and it deserves a cost answer.

A region that can change owner needs a routing decision that is not fixed at boot. That is the hardware difference in one sentence: a partitioned region's path is wired, and a pooled region's path is selected, which means a decode that is writable, an authority that can write it, and a path for the authority to reach it.

The detach-and-attach sequence is state, not a message. Four steps, each with its own completion, each interruptible, and a recovery that has to unwind partial progress. It is a small state machine and a large specification, and the specification is the expensive half.

An authority above the host is a component. It has to exist, be reachable from every host it decides about, survive the failure of any one of them, and be trusted by all of them. Its absence is invisible until the first move, which is the worst property a required component can have.

The blast radius is a topology property and it costs redundancy. A shared path that reaches eight hosts is a single point of failure for eight hosts, and containing it means duplicating the path — which is the cost line the same-failure-domain reading leaves out entirely.

The added latency is physical. A link crossing is a link crossing, and no amount of control-plane sophistication removes it. The weighted mean of section 13 is the number a capacity plan has to carry, and it is the only place in this chapter where the trade is quantitative rather than structural.

The counters and clamps cost almost nothing. A saturating 8-bit counter is a comparator and a mux; a subtraction guarded against underflow is one more comparator. Every one of them is killed by its own mutation, which is the cheapest evidence in the chapter that they are load-bearing.

No area, frequency or power figures appear in this chapter, because none was measured.

21. Silicon Observability

The second is the one that settles the chapter's argument in production. A move counter that reads zero on every region of a system is a system where nothing has ever changed owner — which is exactly what virtualisation looks like, and exactly what pooling does not.

Two counters must read permanently zero — unsafe attaches, and mapped-and-unreachable regions. Each costs almost nothing, and each catches a class of failure that otherwise presents as data corruption or a silent hang.

Publish counts, not percentages — the same conclusion 30.5 reached about denominators and 31.1 about device populations. "140 stranded, 50 wanted elsewhere" is exact and actionable; "70 percent utilisation" is neither.

22. DebugLabs

These labs debug decisions made from the misconception. The symptom is always a project that went wrong, and the root cause is always a mechanism assumed to exist in software.

Lab 1 — A rebalancing policy has nowhere to execute

Symptom. A capacity plan calls for rebalancing memory between hosts under load. At implementation time nobody can find the component that does it.

Evidence. The plan describes the policy in detail. It does not name the mechanism.

Hypothesis. The policy was designed against a hypervisor's capabilities and applied across a host boundary.

Investigation. Ask which component moves the region, and what sequence it runs. There is no answer.

Root cause. A partitioning mechanism assumed to work between machines.

Fix. Either scope the policy inside each host, or specify the pooling mechanism the plan requires.

Prevention. Count the hosts that could own a region. One is partitioning; more than one needs hardware.

Observability. Moves between hosts, per region. Zero on every region is a system where nothing can move.

Lab 2 — Mapped memory produces no data

Symptom. A host maps a region successfully and reads return nothing meaningful. No fault is reported.

Evidence. The page tables are correct. The region is on the other side of a link this host has no path to.

Hypothesis. Mapping was treated as sufficient.

Investigation. Ask whether there is a physical path before asking about the mapping. There is not.

Root cause. A page table entry is a promise that an address means something. It is not a wire.

Fix. Establish reachability first, and refuse the mapping otherwise.

Prevention. Publish mapped and reachable as two fields. Their disagreement is the whole failure, and it has exactly one cell in the truth table.

Observability. A mapped-and-unreachable counter. Permanently zero.

Lab 3 — Capacity was purchased to cover capacity that was already idle

Symptom. A shortfall triggers a purchase. Afterwards, a survey finds substantial idle memory one rack away.

Evidence. The plan recorded stranded capacity as recoverable by the hypervisor.

Hypothesis. Recovery was assumed across a boundary software cannot cross.

Investigation. Compute stranded per node and wanted-elsewhere per node. The minimum is substantial and the software-recoverable figure is zero.

Root cause. A hypervisor's demonstrated ability to reclaim memory inside a machine, generalised to between machines.

Fix. Separate the two figures in the plan, and name the mechanism for each.

Prevention. Compute the stranded bytes, then ask which mechanism can move them.

Observability. Stranded and wanted-elsewhere per node. Two subtractions on numbers a platform already has.

Lab 4 — A region was attached to two owners

Symptom. Data corruption immediately after a capacity move. Both hosts believe they hold the region.

Evidence. The move was implemented as a reassignment: the decode was rewritten and nothing else.

Hypothesis. A sequence was implemented as an assignment.

Investigation. Ask what happened to the old owner. Nothing did: it was never quiesced and never detached.

Root cause. Four steps collapsed into one, because from above a move looks like the region being here and then there.

Fix. Quiesce, detach, deal with the contents, attach — with a refusal at the attach until the first two have completed.

Prevention. Name the four steps, and say what happens if the sequence is interrupted between any two of them. The interruption case is the one that produces this failure.

Observability. Sequence step reached per in-flight move, and an unsafe-attach counter at permanently zero.

Lab 5 — An availability design was sized for one machine

Symptom. A single component failure takes down eight hosts. The availability model predicted one.

Evidence. The model was written for virtualised capacity, where the failure domain is one machine by construction.

Hypothesis. The blast radius was inherited rather than computed.

Investigation. Count the hosts attached to the shared path. Eight.

Root cause. A containment assumption that was true of the previous mechanism and is a property of topology rather than of policy.

Fix. Compute the blast radius per shared path, and either duplicate the path or accept the domain explicitly.

Prevention. Count the hosts a single failure can reach, on each mechanism. In a virtualised world the answer is always one, which is why nobody used to ask.

Observability. Attached hosts per shared path. The assurance becomes a number.

Lab 6 — The first move had nobody empowered to authorise it

Symptom. A pooling deployment reaches the point of its first capacity move and stops. Neither host will act.

Evidence. The design has a policy and no control plane.

Hypothesis. The hypervisor was assumed to be the authority, as it is within a machine.

Investigation. Ask who decides that a region moves from host 3 to host 7. Each host would be deciding about capacity the other believes it holds.

Root cause. A decision spanning hosts needs an authority above them, and within one machine the hypervisor really is that authority — so nothing in the intuition suggests a level above it exists.

Fix. Specify the authority: where it runs, how every host reaches it, what happens when it is unreachable.

Prevention. Name the authority, and say what happens when it is down. Its absence is invisible until the first move.

Observability. The deciding authority per decision. A decision taken at the wrong level is visible after the fact.

Lab 7 — A risk register carried the wrong entry

Symptom. Mitigations are designed for a capacity shortfall. The exposure that materialises is idle capital.

Evidence. The register lists "overcommitment" for a pooled deployment with capacity held and unassigned.

Hypothesis. Inventory was entered as a debt.

Investigation. Compute promised, held and assigned separately. Promised is below held — there is no debt — and held is above assigned, which is inventory.

Root cause. Two opposite risk shapes that look identical from a spreadsheet.

Fix. Three numbers, and two derived quantities with different names.

Prevention. Promised above held is a debt. Held above assigned is inventory. Calling them the same thing inverts the risk being managed.

Observability. Promised, held and assigned, as three published numbers.

Lab 8 — A latency budget was written for memory that is now further away

Symptom. A latency-sensitive workload degrades after a capacity increase that was supposed to be neutral.

Evidence. The budget was written against near memory. A share of accesses now lands in the pool.

Hypothesis. Pooled bytes were budgeted as near bytes.

Investigation. Measure the share of accesses that reach the pool, and the added latency of doing so. Compute the weighted mean.

Root cause. Memory treated as memory, where one of the two has a distance.

Fix. Budget the effective latency, and state the share the budget assumes.

Prevention. State the added latency and the pool share, and compute the weighted mean. The near latency is not the number the budget has to hold.

Observability. Effective latency per region. A budget becomes reviewable.

23. Coverage Reasoning

Coverage of an argument has the same failure mode as coverage of a design.

Four coverage models are worth keeping over any "it is just software" claim:

Reachability coverage. Mapped crossed with physically reachable, four cells. The cell that matters is mapped-and-unreachable, and a system in which the two have always moved together fills it never.

Sequence-state coverage. Quiesced crossed with detached, four cells. The cell that isolates the safety conjunction is quiesced-false, detached-true — and this chapter's model could not reach it until an input was added. An unreachable cell is a finding about the model, not about the coverage plan.

Risk-shape coverage. Overcommitted crossed with pooled, four cells including both-at-once. The cell where the shortcut reading is wrong is inventory without debt, and it is the normal state of a healthy pool.

Blast-radius coverage. Attached hosts crossed with tolerance, at, above and below. The boundary is a tolerance exactly equal to the blast, and it is the value that separates a containment claim from a coincidence.

The bin the shortcut build cannot hit is the most valuable bin in any model. In section 7 it is "mapped and unusable". In section 9 it is "quiesced cleared while detached stands". In section 12 it is "inventory with no debt". Each is unreachable in the shortcut build and trivial in the counting one.

24. How This Appears In Real Engineering

Capacity plans are written by people who have administered hypervisors, and a hypervisor's capabilities are the intuition they reason from. That intuition is correct and is scoped to one machine.

Policies are easier to write than mechanisms, so a plan that describes what should happen is complete long before anybody asks what makes it happen.

Mapping is the visible half of memory management. Reachability is established once, at design time, and never thought about again — until a region appears that a host cannot reach.

Recovery inside a machine is demonstrable and recovery between machines is not, so the evidence a planner has all points the same way.

Moves are drawn as arrows — the region was here, now it is there — and arrows have no steps.

Control planes are added late because nothing needs them until the first move, and the first move is usually after the architecture is signed off.

Availability models are inherited, and a model written when the failure domain was one machine by construction has no field for how many machines a path reaches.

And risk registers are written in a spreadsheet, where a debt and an inventory are both "capacity numbers that do not match", and only the sign tells them apart.

25. Where The Misconception Comes From

The knowledge behind it is real and detailed. Somebody who says this usually knows more about memory management than the person correcting them, which is why the correction has to be about the boundary rather than about the mechanism.

A hypervisor genuinely is the complete authority within its machine. There is no experience inside virtualisation that suggests a level above the host exists, because within a host there is not one.

Everything a hypervisor cannot do is invisible from where it stands. The boundary is not an error message; it is simply the edge of what the abstraction covers, and abstractions do not announce their edges.

Both mechanisms are decided by software. That premise is true of both, which is exactly why it cannot distinguish them — and it is the premise the belief rests on.

The observable outcomes look alike. Capacity was here and is now allocated there. The difference is in what had to happen underneath, and underneath is where nobody with a software-shaped intuition has had reason to look.

And the correction sounds like a technicality. "But the owner changes" is a sentence that needs a second and a third sentence — the path, the sequence, the authority — before it lands, and beliefs that need one sentence beat corrections that need four.

26. Common Misconceptions

"Pooling is just virtualisation with extra steps." The extra steps are a path, a sequence and an authority. Software has none of the three.

"It is just software deciding who gets the memory." True of both mechanisms, which is why it distinguishes neither.

"The hypervisor can just map it." A page table entry is a promise that an address means something. It is not a wire.

"The hypervisor will balance it." Inside its machine, completely. Across a host boundary, not one byte.

"It just gets remapped." Four steps: quiesce, detach, deal with the contents, attach. Every one can fail and every one has to be ordered.

"The failure domain is the same, it is still memory." Count the hosts a single shared-path failure reaches. Virtualised capacity makes that number one by construction.

"The hypervisor allocates it." It has authority over its own machine and none over its neighbours. A cross-host decision needs an authority above both.

"It is overcommitment with extra steps." Promised above held is a debt; held above assigned is inventory. Opposite risks.

"Memory is memory." One of the two has a distance. Budget the weighted mean, not the near latency.

"Most of the conditions hold." A conjunction has no partial credit, and all four fail.

"Software decides who gets it." That is bit 0, and it is worth one sixth of an argument about a mechanism.

27. Interview And Design-Review Questions

The boundary

1. State the difference between virtualisation and pooling in one sentence. Virtualisation divides capacity a host already owns; pooling changes which host owns it. Different verbs, different nouns.

2. Why can software not do the second one? Because the boundary it would have to cross is the edge of the machine it runs on, and that is a physical fact rather than a policy one.

3. What single count separates the two mechanisms? How many hosts could ever own a given region. One is partitioning; more than one is a pool.

4. What is the smallest pool there is? Two reachable hosts. It is a boundary worth driving explicitly, because "a pool" sounds like it needs a fleet.

5. Which premise does the misconception rest on, and why is it useless? That software decides who gets the capacity. It is true of both mechanisms, so it cannot tell them apart.

6. What makes this belief harder to correct than "CXL replaces PCIe"? The knowledge behind it is real and detailed, and correct within its scope. You are correcting the scope, not the knowledge.

Reachability and stranding

7. A hypervisor maps a region it cannot reach. What happens? The access does not arrive. A page table entry is a promise that an address means something; it is not a wire.

8. Which of the four mapped-versus-reachable states do the two readings disagree on? Exactly one: mapped and unreachable. A shortcut that is right three quarters of the time survives review.

9. Define stranded capacity. Capacity attached to a machine that is not using it, while another machine needs it.

10. Two hundred attached, sixty used, fifty wanted elsewhere. Give the three numbers. 140 stranded, 50 movable, and 0 recoverable by software — the third is a consequence of the boundary, not an estimate.

11. Why is the stranded number invariant under virtualisation? Because the mechanism that would reduce it cannot cross the boundary the capacity is on the wrong side of. That invariance is what pooling is for.

12. Why does a planner's evidence all point one way? Because recovery inside a machine is demonstrable and recovery between machines is not, so every observation they have is of the case that works.

The sequence and the authority

13. Name the four steps of a capacity move. Quiesce the old owner, detach the region, deal with what the old owner left in it, attach it to the new owner.

14. Which two must have completed before an attach is safe? Quiesced and detached, both. Either alone leaves the old owner able to touch capacity the new owner believes it holds.

15. Describe the state in which the quiesced term does real work. The old owner is resumed while the region is still detached — a partial undo rather than a full abort. Quiesced falls, detached stands, and the attach must be refused.

16. Why is that state worth modelling explicitly? Because without it the conjunction is unreachable in one direction and the term is untestable. This chapter's model could not reach it, and a mutation proved so.

17. What is the difference between an abort and a partial undo? An abort unwinds everything and returns to step zero. A partial undo changes one step's state and leaves the rest, which is the harder case and the one that produces the hazard.

18. Who can decide that a region moves from host 3 to host 7? Neither host — each would be deciding about capacity the other believes it holds. It needs an authority above both.

19. Why does nothing in a hypervisor intuition suggest that authority exists? Because within one machine the hypervisor is the complete authority. There is no level above it to have met.

20. What are the three properties that authority must have? It must exist, be reachable from every host it decides about, and be trusted by all of them — and its absence is invisible until the first move.

Domains, risk and cost

21. How wide is a virtualised region's failure domain? One machine, by construction. That is why an availability model inherited from virtualisation has no field for anything else.

22. Eight hosts on a shared path, tolerance of one. What is the blast radius? Eight, against a budget of one — wrong by a factor of eight.

23. What is the boundary case in a containment check? A tolerance exactly equal to the blast. It is the value that separates a containment claim from a coincidence.

24. Distinguish overcommitment from a pool. Overcommitment promises more than exists; a pool holds capacity that is not yet assigned. One risks a promise that cannot be met, the other risks idle capital.

25. Promised 80, held 100, assigned 60. Which is it? Neither a debt nor a shortfall: zero overcommitment and 40 of inventory. The two quantities are independent and can both be present.

26. Why do they look alike? From a spreadsheet both are capacity numbers that do not match, and only the sign of the comparison tells them apart.

27. Near 10, pooled 40, half the accesses in the pool. What is the effective latency? 10 + (30 × 50)/100 = 25. A weighted mean, and it is the number a budget has to hold.

28. Why is that not a control-plane problem? Because a link crossing is physical. No amount of control-plane sophistication removes the distance.

29. What does the pooled path cost when the share is zero? Nothing — the effective latency is the near latency. The cost is a function of the share, which is why the share has to be stated.

Method

30. State the disciplined way to handle this claim. Write the four conditions it requires, then check them. All four fail, so it is not a near miss.

31. Give the four conditions. Capacity never moves, software can map anything, stranding is recoverable in software, and no authority above the host is needed.

32. What makes a mutation campaign invalid? A failing baseline. Every mutation then fails for the reason the baseline does.

33. Distinguish an equivalent mutant from a model-expressiveness gap. An equivalent mutant means the code is tighter than it looks — no input separates them. An expressiveness gap means the model is looser than the world — the input exists in reality and the model cannot represent it.

34. Which did this chapter find, and what was the response to each? Both. The equivalent one was withdrawn and replaced; the expressiveness gap was fixed by adding an input to the model, not by extending the stimulus.

35. Why is "withdraw it" the wrong answer to an expressiveness gap? Because withdrawing removes the evidence that the model cannot teach the thing it claims to. The mutation is doing its job; the model is not.

36. Three of four survivors were in one model. What does that tell you? That sequencing logic has more reachable states than scoring logic, and therefore more states a stimulus can miss. Drive sequences to their boundaries, not through their happy path.

37. Why drive an 8-bit counter 254 times? Because a counter that wraps turns "this region has moved constantly" into "this region has never moved", and the saturation is only observable at the ceiling.

38. Your expected value was 1 and the result was 255. What is the lesson? Re-derive every expectation that follows an inserted case. An expectation carried across an insertion is an expectation about the old ordering.

39. A new checker mis-flags correct code. What do you do? Fix the checker before trusting any of its output, and keep a negative control so the fix is provable. A gate that mis-flags correct code is a gate nobody runs.

40. What false positive did this batch's simultaneous-write gate produce? A wrapped else if whose condition sits on a different line from its assignment, read as an unguarded sibling statement.

Saying it well

41. What would you say to a colleague who states this? Name the boundary. Everything they know is correct inside one machine, and the claim is about what happens between two.

42. What is the cheapest sentence that shows you understand the difference? "Virtualisation re-maps what a host owns; pooling changes which host owns it."

43. Why does the correction need three sentences and the belief needs one? Because the belief points at something visible — software allocating capacity — and the correction points at a path, a sequence and an authority, none of which is visible from where the believer stands.

44. Which telemetry field settles the argument in production? Moves between hosts, per region. Zero everywhere is a system where nothing has ever changed owner.

45. What does this chapter share with 31.1? Both refute a claim whose premises are true. There the premises are about the wrong subject; here they are about the wrong layer.

46. State the single question this chapter turns on. Does the capacity change which machine it belongs to — and what has to happen for it to?

28. Exercises

1 — Architecture · Intermediate. Builds: locating the boundary a claim crosses. Take the sentence "memory pooling is just virtualisation". Bounded scope: write the four conditions it requires, mark each true or false, and write the one-sentence correction that names the boundary rather than the mechanism. Hint: the correction has to survive the fact that everything your colleague knows is correct.

2 — Design review · Intermediate. Builds: separating a mapping from a path. Review a capacity design that specifies address-space layout in detail. Bounded scope: list the reachability questions it does not answer, give the four mapped-versus-reachable states, and name the single state the design's assumption erases. Hint: three of the four states are consistent with the assumption.

3 — Quantitative · Intermediate. Builds: computing what a mechanism could actually recover. A fleet of six nodes each has 256 units attached; usage is 200, 64, 240, 32, 192 and 96, and two nodes want 64 more each. Bounded scope: compute stranded per node, the fleet total, the movable total, and the figure a hypervisor can recover. Hint: the last number is not an estimate.

4 — Design · Advanced. Builds: specifying a sequence rather than an assignment. Specify the capacity-move sequence for a region changing owner. Bounded scope: name every step, state the precondition of each, define the safety condition for the attach, and say what an interruption between each adjacent pair leaves behind. Hint: one of the interruptions is not an abort, and it is the dangerous one.

5 — Architecture · Advanced. Builds: computing a containment domain instead of inheriting one. A shared path serves twelve hosts and the availability model was written for virtualised capacity. Bounded scope: compute the blast radius, state what the inherited model assumed, give the two ways to bring the design back inside its tolerance, and price each qualitatively. Hint: one of the two ways changes the topology and the other changes the promise.

6 — Design · Advanced. Builds: specifying the component whose absence is invisible. Specify the authority that decides cross-host capacity moves. Bounded scope: say where it runs, how each host reaches it, what happens when it is unreachable mid-sequence, and what telemetry proves a decision was taken at the right level. Hint: "unreachable mid-sequence" is the requirement that shapes the rest.

7 — Quantitative · Advanced. Builds: budgeting capacity that has a distance. A workload has a latency budget of 150 units. Near memory is 90; pooled memory is 300. Bounded scope: compute the maximum pool share the budget tolerates, state the effective latency at that share, and say what happens to your answer if the pooled latency improves by a third. Hint: solve for the share, then sanity-check it against the endpoints.

8 — Verification · Expert. Builds: telling an equivalent mutant from an expressiveness gap. Take a safety conjunction from your own design. Bounded scope: for each term, construct the input that drives it false with the others true; for any term where no such input is reachable, decide whether the code is tighter than it looks or the model is looser than the world, and state the evidence that distinguishes the two. Hint: ask whether the state exists in the real system. If it does and the model cannot reach it, the model is wrong.

29. Summary

Virtualisation divides capacity a host already owns. Pooling changes which host owns it. Different verbs, different nouns, and the second one needs a path, a sequence and an authority.

The boundary is the edge of the machine the hypervisor runs on, and it is a physical fact rather than a policy one.

Count the hosts that could own a region. One is partitioning; two is the smallest pool there is.

A page table entry is a promise that an address means something. It is not a wire. Establish the path first.

A hypervisor recovers stranded capacity inside its machine and not one byte across a host boundary, which is why the stranded number is invariant under virtualisation and is exactly what pooling is for.

A move is a sequence, not an assignment — quiesce, detach, deal with the contents, attach — and the dangerous interruption is the one that undoes a step rather than the whole thing.

A virtualised region's failure domain is one machine by construction. A pooled region's is every host on the shared path, and that is topology rather than policy.

A cross-host decision needs an authority above both hosts, and within a machine there is no such level to have met — which is why its absence is invisible until the first move.

Promised above held is a debt. Held above assigned is inventory. Opposite risks, identical from a spreadsheet.

Pooled capacity has a distance, and a budget has to hold the weighted mean rather than the near latency.

A conjunction has no partial credit, and here all four conditions fail rather than three.

Six conditions, and "software decides who gets it" is one of them. A statement true of both mechanisms distinguishes neither, and it is 16 percent of an argument.