Memory Resource Sharing

11.2 made expanded capacity addressable for one host: every address resolving to exactly one live region, with a known latency class and enough concurrency to reach it.

This chapter removes the assumption that there is one host.

1. The Engineering Problem — Stranding Is Only Fixed by Sharing

11.1 measured stranding: one node 30 short, three others holding 60 spare between them, and half the fleet's free memory unreachable. Giving every node its own expander does not fix that. It gives every node more memory to strand.

Stranding is only fixed when one pool of memory can serve more than one host. And the moment a second host can address the same device, four things that were previously free become design problems:

Ownership. With one host, every address belonged to it. With two, "who owns this range" is a question that must have exactly one answer, enforced in hardware.

Fairness. One host's traffic can consume the device's entire concurrency, and the other host sees no error — only latency.

Isolation. One host's fault must not stop the device serving the others.

Reuse. When a partition moves from one host to another, the second host must not be able to read the first one's data.

None of these is a protocol problem. They are the price of admission for sharing anything, and this chapter builds the hardware that pays it.

2. The One-Sentence Model

Sharing is partitioning plus policing. Every range has exactly one owner, every host has its own concurrency budget, arbitration advances only on work actually done, a fault is contained to the host that caused it, and memory is scrubbed before it changes hands.

Call it one owner, one budget, one blast radius. Each clause is a separate mechanism, and omitting any one of them makes the other three unsafe to rely on.

3. What This Chapter Owns

The boundary with Module 12 is the important one, and it is a boundary of scale and mechanism, not of subject.

This chapter has no fabric, no manager and no allocator. It has one device, a small fixed set of hosts, and a static partition table someone else configured. That is deliberate: the mechanisms below are the ones a pool depends on, and Module 12 cannot allocate safely unless partitioning, fairness and isolation already work.

4. Teaching-model boundary

Each gate answers a different question — may you, how much, whose turn, and are you healthy — and each has a failure mode invisible to the other three. That is why they are four mechanisms rather than one policy.

5. RTL 1 — Exactly One Owner

11.2 required every address to resolve to exactly one region. Sharing adds a second requirement on top: that region must belong to the host asking.

    owned_by_caller = (nmatch != 0) && (p_owner[first_idx] == acc_host);
    // Ownership is part of the permission decision, not a later filter.
    permit   = acc_valid && (nmatch != 0) && (NO_OWNER_CHECK || owned_by_caller);

=== EXP1: every range belongs to exactly one host ===
  each host inside its own partition, boundaries walked : permitted=12 denied=0
  all four partitions included both their own boundaries     : ok
  host 1 addressing host 2's range : permit=0 deny=1 | no-owner-check permit=1
  a cross-host access was denied                             : ok
  the no-owner-check variant permitted it                    : ok
  and was caught crossing a host boundary                    : ok
  overlapping partitions : double-owned flag=1
  two partitions covering one address was caught             : ok

Two hosts owning one range is not sharing — it is corruption. The word "sharing" in this chapter's title means sharing a device, not sharing a range. A range with two owners has no defined behaviour: both hosts write it, neither is wrong, and the data is whatever arrived last.

Ownership belongs in the permission decision, not after it, for the same reason enable belonged inside the decode in 11.2: a check applied afterwards is a filter, and a filter can be forgotten on one path while the decode is shared by all of them.

The NO_OWNER_CHECK variant is the trust model most systems start with — the host tells the device which range it wants and the device believes it. That is adequate when there is one host and catastrophic when there are two, and the transition between those two situations is a configuration change rather than a redesign.

6. RTL 2 — One Host Must Not Consume the Device

    has_room = ONE_POOL ? (shared_q < SHARED_POOL[15:0])
                        : (inflight[req_host] < PER_HOST[15:0]);

=== EXP2: one host must not consume the whole device ===
  host 0 issued 10 : per-host inflight=3 granted=3 blocked=7
  the per-host bound held host 0 at 3                        : ok
  one-pool variant : host 0 inflight=10 shared used=10
  the single-pool variant let one host take far more         : ok
  host 1 then asks : host1 inflight=1 (correct) | one-pool host1=1
  host 1 was still admitted despite host 0 saturating        : ok
  3 cycles of grant+response for host 1 : inflight 1 -> 1
  simultaneous grant and response left the per-host count unchanged : ok

This is the noisy-neighbour failure, and it produces no error at all.

With a single shared pool, host 0 took ten slots. It did nothing wrong — it had work and the device accepted it. Host 1 then arrives and finds a device that is technically healthy, correctly configured, and out of concurrency. Host 1 experiences that as latency, and there is nothing in its own telemetry to explain it.

A per-host bound converts an invisible problem into a visible one. Host 0 is held at 3, its excess requests are counted as blocked, and host 1 gets in immediately. The device is now slightly less efficient for a single busy host and dramatically more predictable for everyone else — which is the trade sharing always makes.

The simultaneous grant-and-response test is the two-assignment counter defect again, now per host. Its twelfth appearance in this track. Note it had to be driven on a host with room — host 0 sat at its limit, so no grant could occur there, and a test written against the saturated host would have proved nothing.

7. RTL 3 — Rotate on the Transfer, Not the Choice

      // The pointer advances on an accepted TRANSFER, never on the choice.
      if (ROTATE_ON_GRANT ? any : transfer)
        ptr_q <= 2'((winner + 1) % HOSTS);

=== EXP3: rotate on the transfer, not on the choice ===
  all four requesting, downstream ready : served=4/4/4/4 transfers=16
  round robin served all four equally                        : ok
  with a stalling downstream : correct served=6/6 | rotate-on-grant served=8/4
  the correct arbiter stayed fair through the stalls         : ok
  the rotate-on-grant variant developed a bias               : ok
  fixed-priority variant : host0=40 host3=0 starved flag=1

With no backpressure, both arbiters are perfectly fair. 4/4/4/4. Any fairness test run against an always-ready downstream passes on both designs, which is how this defect reaches production.

Add stalls and they diverge: 6/6 against 8/4. When the pointer advances on the arbiter's choice, a host granted during a stall cycle loses its turn without ever being served — and since stalls correlate with load, the bias appears exactly when fairness matters and is absent whenever anyone tests for it.

Fixed priority is the honest failure. Host 3 received nothing in forty transfers and the starvation flag fired. It is at least obvious; the rotate-on-grant bias is not.

8. Waveform — Fairness Diverging Under Backpressure

Teaching-model timing derived from the simplified RTL in this chapter. Not CXL wire timing.

Cycles 1 and 2 are identical in both designs — that is the whole problem. The divergence begins at cycle 3, the first stall, and by cycle 10 the fair arbiter has served 2/2/2/1 while the rotate-on-grant variant has served 3/2/0/1.

Host 2 was granted three times and transferred zero times. From the arbiter's own point of view it was treated perfectly fairly.

9. RTL 4 — A Fault Must Not Cross Hosts

    // Only a host that has faulted is refused. A global halt refuses everyone.
    blocked = acc_valid && (GLOBAL_HALT ? (|faulted_q) : faulted_q[acc_host]);

=== EXP4: one host's fault must not stop the others ===
  host 1 faulted, host 1 retries : correct serve=0 | global-halt serve=0
  the faulted host is refused                                : ok
  host 2 (never faulted) : correct serve=1 | global-halt serve=0
  an unaffected host is still served                         : ok
  the global-halt variant refused it too                     : ok
  and was caught inflicting collateral damage                : ok

Blast radius is the property that makes sharing acceptable to the people whose workloads are being shared.

A device that halts on any fault has converted one host's problem into everyone's. The failure is not subtle — it is a multi-tenant outage caused by a single tenant — and it is the reason a platform team will refuse to share a device at all if they cannot see the containment mechanism.

The global-halt variant is not a strawman. It is what a design does by default: a fault raises an error, the error stops the pipeline, and the pipeline serves everyone. Containing it requires the fault to be attributed to a host first, which requires the per-host attribution of §12 to exist before the isolation of §9 can work.

10. RTL 5 — Scrub Before Handing Over

      // Handing a partition to a new owner while it still holds old data.
      if ((st_q == H_FREE) && (st_n == H_ASSIGN) && dirty_q) leak_err <= 1'b1;

=== EXP5: a partition must be scrubbed before it changes hands ===
  release moves it to quiesce                                : ok
  and it still holds the previous owner's data               : ok
  quiesced : state=2 (2=SCRUB) | skip-scrub state=3
  after scrub : state=3 exposed=0 scrub cycles=6
  only after scrubbing is it free and clean                  : ok
  the scrub took exactly 6 cycles                            : ok
  assigned : handoffs=1 leak flag=0 | skip-scrub leak flag=1
  the skip-scrub variant leaked the previous owner's data    : ok

Reuse across a trust boundary is the one failure in this chapter that cannot be undone.

Every other defect here produces wrong behaviour that stops when the bug is fixed. This one produces a disclosure — the second host has already read the first host's data, and no subsequent fix retrieves it.

The sequence is quiesce, scrub, free, assign, and none of the four steps is optional. Quiesce first, because scrubbing memory with accesses in flight races against them. Scrub second, because free memory that still holds data is not free. Only then may it be assigned.

This is the fourth appearance of the drain-barrier shape in this curriculum — after 9.6's fence, 10.2's ownership handover and 11.2's region disable. Every one of them has a fast variant that skips the drain, and every one of those variants corrupts something. The pattern is now reliable enough to look for by default.

11. RTL 6 — A Service Floor Needs a Window

      // An instantaneous judgement calls every host starved on almost every
      // cycle, because only one host can be served at a time.
      if (INSTANT_SHARE) below_c[h] = !(served && (served_host == h[1:0]));
      else               below_c[h] = (cnt[h] < FLOOR[15:0]);

=== EXP6: a service floor is measured over a window ===
  open window : per-host served=24/6/0/0
  service was attributed to the host that received it        : ok
  hosts 2 and 3 received nothing                             : ok
  below-floor mask : windowed=1100 | instantaneous=1101
  the windowed mask names exactly the two starved hosts      : ok
  the instantaneous variant calls hosts starved every cycle  : ok

A share is meaningless on a single cycle. Only one host is served at a time, so an instantaneous judgement reports every other host as starved — the mask 1101 says three of four hosts are being starved on a cycle when the system is working perfectly.

The windowed mask names exactly the two hosts that actually received nothing: 1100. That is a usable signal — it can drive an alarm, a policy change, or a conversation with the tenant consuming the device.

The floor is per host, per window, and both parts matter. A floor without a window is noise; a window without a per-host floor tells you the device was busy without telling you for whom.

12. RTL 7 — Attribute Everything Per Host

=== EXP7: every counter attributed per host ===
  per host=6/6/6/6 total=24 denied=3
  oracle  : per host=6/6/6/6 total=24
  per-host attribution matched an independent oracle         : ok
  the per-host counts account for the whole                  : ok
  exactly 3 denials were counted, separately from accesses    : ok
  device-total-only variant : per host=0 total=24 flag=1
  aggregating away the host was caught by the attribution law : ok

A device-wide total cannot answer the only question sharing raises. "The device served 24 accesses" is true and useless; "host 0 served 6" is actionable.

The attribution law — per-host counts summing to the device total — is what makes the split trustworthy. The DEVICE_TOTAL_ONLY variant aggregates the host away and is caught immediately, because its per-host counts no longer account for the total it reports.

Note that denials are counted separately from accesses. On a shared device a denial is not a failed access — it is evidence about a host attempting something it does not own, which is a security signal rather than a performance one.

13. Quantitative Reasoning

Illustrative, with stated assumptions. No figure describes a real device.

What sharing recovers

From 11.1's stranding example: one node 30 short, three nodes with 60 spare.

private capacity  : unmet demand 30, stranded surplus 30
shared capacity   : unmet demand  0, stranded surplus 30

Sharing recovers exactly the overlap between surplus and deficit — 30 units here — and no more. The remaining 30 stays stranded because nothing demands it. That bound matters: sharing does not eliminate stranding, it eliminates the part of stranding that someone wanted.

The cost of a per-host bound

With 4 hosts, a per-host limit of 3 and a device pool of 12:

one host alone, shared pool  : up to 12 outstanding
one host alone, per-host cap :        3 outstanding
worst-case single-host loss  : 75% of achievable concurrency

A per-host bound costs a lone host three quarters of the device when nobody else is using it. That is the honest price of predictability, and it is why real designs give each host a reserved floor plus access to a shared surplus rather than a hard cap — a refinement this teaching model deliberately omits so the trade is visible.

Fairness loss under backpressure

From §7, with stalls on roughly half the cycles:

rotate on transfer : 6 / 6      (ratio 1.00)
rotate on grant    : 8 / 4      (ratio 2.00)

A 2:1 service imbalance between two hosts making identical demands, invisible until the downstream stalls. Scale that to a device shared by four tenants and one of them is paying for capacity it cannot reach.

Scrub cost

The measured scrub took 6 cycles for a teaching-sized partition. Scaled honestly:

scrub time ≈ partition size / scrub bandwidth

A 64 GB partition at 10 GB/s of scrub bandwidth is roughly 6.4 seconds during which the partition belongs to nobody. That is the real constraint on how dynamically memory can be reassigned, and it is a property of media bandwidth rather than of any protocol — which is why Module 12's allocation policies have to plan around it rather than assume instant handover.

Window sizing for a service floor

With a floor of 4 services per host per window and 4 hosts:

minimum window = hosts x floor = 16 services

A window shorter than that reports violations on a perfectly fair device, because there is not enough service in it to give everyone their floor. The window used here is 32 — twice the minimum — which leaves headroom for burstiness without hiding sustained starvation.

14. Assertions

Icarus Verilog 13.0 is the only simulator installed. It does not execute concurrent SVA — property blocks are unsupported, and unique/priority qualities are parsed but ignored — so every property below is synthesisable checker logic verified procedurally, not executed SVA. 48 assertions.

Safety

Liveness

Performance goals — not correctness

15. Mutation Testing

Twenty-four mutations. Twenty-four killed.

First run: 19 of 24. The five escapes sorted into the same two categories the previous two chapters produced:

Three values were printed and not asserted — the per-host window counts, the correct below-floor mask, and the denial count. All three appeared in the transcript, all three looked right to a reader, and none was checked. That is now the single most common escape cause across this batch, and it is worth stating plainly: a transcript line is not a test.

Two states were never reached — a partition's exact boundary addresses, and a grant coinciding with a response for the same host.

Two testbench defects the baseline caught

A test that could not have passed. The simultaneous grant-and-response test was first written against host 0 — which was already pinned at its per-host limit, so no grant could ever occur and the count could only fall. The test proved nothing about simultaneity. Rewriting it against a host with headroom made it meaningful.

A free-running window. The QoS measurement assumed its window started when the experiment did. The window counter free-runs from reset, so it had already rolled mid-measurement and the counts read 0/3/0/0 instead of 24/6/0/0. Synchronising to a window boundary first — the same fix 11.1 needed for its working-set window — made the measurement deterministic.

16. Verification Strategy

Fairness must be tested with backpressure. This is the single most important line in the plan: an always-ready downstream makes a biased arbiter and a fair one indistinguishable, and that is exactly the test everyone writes.

The coverage cross is host × partition state × downstream readiness: owner/non-owner, owned/quiescing/scrubbing/free, ready/stalled. The non-owner × scrubbing × stalled point is where an access from the wrong host arrives at a partition mid-handover under load, and nothing but directed stimulus reaches it.

17. Synthesis and Implementation Reality

The isolation mask is one flop per host and it is what makes sharing sellable. That ratio — trivial hardware, decisive property — is worth noticing, because it is usually omitted on the grounds that faults are rare.

The scrub is the only structure whose cost is not in gates. It occupies media bandwidth for a duration proportional to partition size, and during it the partition serves nobody. No amount of logic makes that faster.

18. Silicon Observability

Denials per host is the security counter, and it is the one most likely to be omitted. A host repeatedly addressing ranges it does not own is either misconfigured or misbehaving, and neither shows up in any performance metric.

Grants without transfers is the fairness counter. A high value means the arbiter is making choices that do not become work, which is precisely the condition under which a rotate-on-grant bias develops — and it is measurable before the bias shows up as a tenant complaint.

19. Debug Lab

  host 0 issued 10 : per-host inflight=3 granted=3 blocked=7
  one-pool variant : host 0 inflight=10 shared used=10

has_room = (inflight[req_host] < PER_HOST);   // per host, not per device

  all four requesting, downstream ready : served=4/4/4/4
  with a stalling downstream : correct served=6/6 | rotate-on-grant served=8/4

if (transfer) ptr_q <= (winner + 1) % HOSTS;   // not on `any`

  host 2 (never faulted) : correct serve=1 | global-halt serve=0
  and was caught inflicting collateral damage                : ok

if (acc_valid && fault) faulted_q[acc_host] <= 1'b1;
blocked = acc_valid && faulted_q[acc_host];    // not |faulted_q

  after scrub : state=3 exposed=0 scrub cycles=6
  assigned : handoffs=1 leak flag=0 | skip-scrub leak flag=1
  the skip-scrub variant leaked the previous owner's data     : ok

if ((st_q == H_FREE) && (st_n == H_ASSIGN) && dirty_q) leak_err <= 1'b1;

  below-floor mask : windowed=1100 | instantaneous=1101
  the windowed mask names exactly the two starved hosts       : ok

below_c[h] = (cnt[h] < FLOOR);        // per host, per window

  host 1 addressing host 2's range : permit=0 deny=1 | no-owner-check permit=1
  and was caught crossing a host boundary                     : ok

owned_by_caller = (nmatch != 0) && (p_owner[first_idx] == acc_host);
permit = acc_valid && (nmatch != 0) && owned_by_caller;

  per host=6/6/6/6 total=24
  per-host attribution matched an independent oracle          : ok
  the per-host counts account for the whole                    : ok

per[acc_host] <= per[acc_host] + 16'd1;
if (sum != total) attribution_err <= 1'b1;   // the law that catches it

  release moves it to quiesce                                 : ok
  and it still holds the previous owner's data                : ok
  with nothing outstanding it begins scrubbing                : ok

H_QUIESCE: if (outstanding == 4'd0) st_n = H_SCRUB;
if ((st_q == H_QUIESCE) && (st_n != H_QUIESCE) && (outstanding != 4'd0))
  early_release_err <= 1'b1;

20. Design Review

1. How is ownership of a range enforced, and is it inside the permission decision?
2. What proves no two partitions cover the same address?
3. Is concurrency bounded per host or per device?
4. What does one host saturating the device do to the others' latency?
5. Does the arbitration pointer advance on a grant or on a transfer?
6. How was fairness tested — with or without backpressure?
7. What is the blast radius of a fault raised by one host?
8. What must happen between one host releasing a partition and another receiving it?
9. How long does a scrub take at production partition sizes, and who waits?
10. Over what window is a service floor measured, and is it at least hosts × floor?
11. Do per-host counters sum to the device total?

21. How This Appears in Real Engineering

Architect. Owns the partition granularity and the per-host credit split, and the decision between hard caps and reserved floors with a shared surplus.

RTL designer. Owns the owner compare in the permission path and the rotating priority encoder. The isolation mask is one flop per host and is the cheapest high-value structure here.

DV engineer. Owns the two-host tests. Single-host verification cannot expose noisy-neighbour, fairness bias, or containment failures, and single-host verification is the default.

Firmware engineer. Owns the partition table and the handover sequence, and must not treat release as a register write.

Security engineer. Owns the scrub requirement and the denial counter. The scrub is the only failure here that cannot be undone.

Performance engineer. Owns the service-floor window and the grants-without-transfers counter, which predicts a fairness bias before any tenant complains.

Silicon validation. Owns the pattern-based handover test — write recognisable data, release, reassign, read — which is the only test that proves the scrub actually happened.

What each needs from the others: isolation depends on attribution, which depends on the host ID surviving to the counters; the handover sequence depends on the scrub bandwidth firmware plans around; the credit split depends on the tenancy model the architect assumed. A gap in any one makes the others unenforceable.

22. Common Misconceptions

23. Interview Reasoning

24. Exercises

1. Calculation. A device has 16 outstanding slots shared by 4 hosts. Compute the per-host cap for strict equal division, the concurrency a lone host then loses, and the cap that would give each host a reserved floor of 2 with the remaining 8 slots shared. State which you would ship and why.

2. Analysis. A tenant reports latency doubling at unpredictable times on a shared expander. Its own request rate is constant and the device reports no errors. Name the two counters that would confirm the cause, what values you would expect, and what you would tell the tenant if the counters disagreed with your hypothesis.

3. RTL task. Extend per_host_credits to implement a reserved floor plus a shared surplus: each host is guaranteed F slots and may use shared slots beyond that when free. State the new invariant this creates and the failure mode if the shared pool is released to the wrong host.

4. Assertion task. Write the property that proves the arbitration pointer advances only on an accepted transfer. Explain why a fairness property over service counts is insufficient, and construct a design that satisfies equal service counts while still being unfair.

5. Testbench design. Design the stimulus that distinguishes a fair arbiter from a rotate-on-grant one. Explain precisely why an always-ready downstream cannot distinguish them, and what the minimum stall pattern is that can.

6. Security task. Design the test that proves a partition was scrubbed before reassignment. Explain why reading zeroes is insufficient evidence on its own, and what the previous owner must write for the test to be conclusive.

7. Debug task. Two tenants on a shared device see intermittent corruption in each other's data. Give your investigation order across this chapter's alarms, the two distinct defects that both produce this symptom, and the single measurement that separates them.

8. Design review. A colleague proposes removing per-host credit bounds, arguing they waste concurrency when only one host is active and that tenants can be trusted to behave. Give the strongest version of that argument, state what it costs when it fails, and describe the mechanism you would propose instead.

25. Summary

Sharing is partitioning plus policing.

• Exactly one owner per range, enforced inside the permission decision. Two owners is not sharing — it is silent corruption.
• Per-host concurrency bounds, or one host consumes the device and the others see only latency with no error anywhere. The measured single pool let one host take 10 slots; the bound held it at 3 and admitted the second host immediately.
• Rotate on the transfer, never the grant. Identical demand gave 6/6 fair and 8/4 biased — and a perfect 4/4/4/4 for both when nothing stalled, which is why the default fairness test cannot find this.
• Contain the blast radius. One flop per host, and it is what makes sharing acceptable to the tenants being shared.
• Scrub before handing over. Quiesce, scrub, free, assign — the fourth drain-barrier in this curriculum, and the only failure here that cannot be undone.
• A service floor needs a window of at least hosts × floor; an instantaneous share reports almost everyone starved almost always.
• Attribute every counter per host, protected by a conservation law — a correct total with wrong attribution is the failure that survives review.
• Verification: 24 of 24 mutations killed, 48 assertions. Five first-run escapes: three values printed but never asserted, two states never reached. The baseline separately caught a test that could not have passed (targeting a host already at its limit) and a free-running window measured as though it were aligned.

Next: 11.4 — Server Architectures, which takes these mechanisms as given and asks what a server built around expanded memory actually looks like.