GPU Memory Sharing

29.1 asked which tier, 29.2 asked what the device is, and 29.3 asked whether the fleet arithmetic works. This chapter asks the question that sounds simplest and has the most answers.

The GPUs share memory. Four different things satisfy that sentence: giving every device its own copy, letting one device read another's, drawing on a pooled region, and keeping a structure coherent across all of them. They cost capacity, latency, bandwidth and coherence traffic respectively, and the sentence distinguishes none of them.

1. The Engineering Problem — Four Things Wear One Word

"Sharing" names no mode. Four modes present with one named leaves three unnamed, on a description where only a quarter of the regions called shared are actually kept coherent. Section 5.

Somebody keeps it coherent, and coherence is traffic. Eight sharers at three messages a write is twenty-one messages per write and forty-two hundred in total — a hundred and forty percent of the budget. Section 6.

A peer read and a read through a host are different distances. Three hundred nanoseconds peer-to-peer with two host hops at two hundred and fifty is eight hundred on a path described as direct. Section 7.

Replication and sharing spend different currencies. Eight devices with a ninety-six gigabyte buffer against eighty gigabytes each is sixteen over per device — or twenty-four hundred units of traffic if they share one copy instead. Section 8.

Synchronisation is serial in a system bought for parallelism. A hundred and twenty barriers at forty microseconds of skew is forty-eight hundred microseconds waited — fifty-five percent of the run parallel. Section 10.

And a shared region couples the devices that use it. Six devices of sixteen sharing three regions is six devices that are now one failure unit. Section 11.

Why this is the module's hardest case. The other three chapters each have one dominant quantity — a tier, a link, a stranding figure. This one has four modes, and the first job is to find out which one is being discussed before any number in the chapter has a subject.

2. The One-Sentence Model

A GPU-memory-sharing case study is sound when the sharing is named, when the mode is named rather than called "sharing", when somebody owns the coherence, when the path is stated as peer or through-host, when the synchronisation is counted, and when the failure coupling is stated — and "the GPUs share memory" is bit 0.

3. What This Chapter Owns

Some vocabulary, because the four modes are the chapter and one word covers them all.

Replication gives every device its own copy. It costs capacity — the same bytes stored as many times as there are devices — and it costs nothing at access time, which is why it is the default answer for anything read-only.

Peer access lets one device read another's memory directly. It costs a longer access and no capacity, and its defining question is whether the read really goes device-to-device or out to a host and back.

A pooled region is capacity neither device owns, reached across a link. It is 29.2's device seen from the accelerator side.

And coherent sharing keeps one structure consistent across devices that all write it. It is the only one of the four that generates traffic proportional to the writing, and it is the mode people mean least often and say most often.

4. Teaching-Model Boundary

This is a case-study chapter and the boundary is the one Module 29 has carried throughout.

Every model computes a property of a decision, not a claim about any product. No figure here is a measurement of any real accelerator, fabric or system. Three hundred nanoseconds of peer latency, an eight-device group, a ninety-six gigabyte buffer — all are teaching figures chosen to make one relationship visible.

Nothing is attributed to a named organisation. What is publicly established and used here is only this: that accelerators attach high-bandwidth memory directly, that vendor fabrics exist to connect a vendor's own devices to each other, and that CXL attaches memory over a link. No latency, bandwidth, capacity or topology figure in this chapter belongs to anybody's product.

Each model is built twice from one source. A parameter selects between the measured build, which counts what the design rests on, and the sharing build, which treats the word as the finding. Every section's headline number is the gap between them.

5. RTL 1 — Four Things Wear One Word

Start with the sentence, because every number below it is a number about one of four different things.

The four modes are not variations on a theme. Replication costs capacity and no latency. Peer access costs latency and no capacity. A pooled region costs both and buys flexibility. Coherent sharing costs traffic proportional to writes. A design that picked one and a reader who assumed another are describing different machines, and the sentence they share does not reveal the disagreement.

// RTL 1 - "the GPUs share memory" names no mode. Replicating a read-only buffer,
// reading a peer's memory directly, drawing on a pooled region and keeping a
// coherent shared structure are four different things with four different costs,
// and the sentence is satisfied by all of them.
module sharing_mode_claim #(parameter int ANY_MODE_IS_SHARING = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] modes_present, modes_named, regions_total, regions_coherent,
  output logic [15:0] modes_unnamed, named_ok, coherent_pct, pinned_pct,
  output logic        mode_identified,
  output logic [7:0]  n_evals, n_ambiguous,
  output logic        mode_err
);
  logic [31:0] c_q, p_q;
  logic [15:0] true_unnamed;
  logic        truly_ambiguous;
  assign named_ok = (modes_named > modes_present) ? modes_present : modes_named;
  assign true_unnamed = modes_present - named_ok;
  assign modes_unnamed = (ANY_MODE_IS_SHARING != 0) ? 16'd0 : true_unnamed;
  // How much of what is called shared is actually kept coherent.
  assign c_q = (regions_total == 16'd0) ? 32'd0
             : (({16'd0, regions_coherent} * 32'd100) / {16'd0, regions_total});
  assign coherent_pct = (c_q > 32'd100) ? 16'd100 : c_q[15:0];
  assign p_q = (modes_present == 16'd0) ? 32'd100
             : (({16'd0, named_ok} * 32'd100) / {16'd0, modes_present});
  assign pinned_pct = (ANY_MODE_IS_SHARING != 0) ? 16'd100 : p_q[15:0];
  assign mode_identified = (modes_unnamed == 16'd0) && (modes_present != 16'd0);
  assign truly_ambiguous = (true_unnamed != 16'd0);
  assign mode_err = evaluate && truly_ambiguous && mode_identified;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_ambiguous <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_ambiguous) n_ambiguous <= n_ambiguous + 8'd1;
    end
  end
endmodule

Four modes with one named, and five of twenty regions actually coherent, leaves three modes unnamed, a quarter coherent, and a quarter of the structure pinned down.

Figure 1 — the word that four designs answer to. The upper path is true under every one of them, which is exactly why it identifies none. The lower path returns two numbers, and the second is the more uncomfortable: a quarter of what this system calls shared is actually kept coherent, so three quarters of it is shared in some other sense that nobody has written down.

The second case is the description done properly. Every mode named and every region coherent reports an identified mode, both builds agree, and neither alarms.

The fourth case is the clamp on the coherence side and it is a sanity check rather than a finding. More coherent regions claimed than exist saturates at a hundred percent, because a subset cannot exceed its set.

The fifth case is the state the sentence usually describes. One mode, none named, nothing coherent — a system that shares something, in some way, with no consistency guarantee anybody has stated.

The degenerate case bounds it: nothing enumerated reports nothing unnamed and nothing identified, which is silence rather than ambiguity.

The four modes are worth separating precisely, because the word hides a genuine incompatibility. Replication and coherent sharing are not neighbouring points on a spectrum — they are opposite answers to the question what happens when a device writes? Under replication the write is local and the other copies are now stale, which is correct if the structure is read-only and catastrophic if it is not. Under coherent sharing the write is a message to everybody. A team that implemented one while a team that assumed the other reads the same document and finds out at integration.

The middle two modes differ in ownership rather than in mechanism. Peer access reads memory that a specific device owns; a pooled region reads memory that none of them does. That distinction sounds administrative and decides three things — who reclaims the capacity when a job ends, whose failure takes it out, and which device's memory-bandwidth budget the reads consume. None of those questions has an answer until the mode is named.

The coherence fraction is the section's second finding and it is the sharper one. Twenty regions described as shared with five actually kept coherent means three quarters of them are shared in a sense nobody wrote down — most often replication that was correct when the structure was read-only and has since acquired a writer. That is not a documentation problem; it is a correctness problem with a documentation symptom, and the count is the only thing that surfaces it before the results go wrong.

The fix costs one question and it is the cheapest in the chapter. Asking which of the four converts a sentence that four designs satisfy into a claim that one design satisfies, and every remaining section is then a number about that one.

6. RTL 2 — Somebody Keeps It Coherent, And Coherence Is Traffic

The second thing, and the one that turns the fourth mode into a number.

Coherence is not a property the interconnect provides for free. If several devices may write a line, every write has to reach every other holder of it, and that is a message count that scales with the number of sharers. The traffic is proportional to writes times sharers, and both of those are workload facts rather than protocol facts.

// RTL 2 - somebody has to keep it coherent, and coherence is traffic. Every
// writer of a shared line makes work for every reader of it, and the work is a
// message count rather than a property of the interconnect.
module coherence_owner #(parameter int IT_IS_JUST_COHERENT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] sharers, writes, msg_per_write, budget_msgs,
  output logic [15:0] msgs_per_write, total_msgs, over_budget, budget_pct,
  output logic        within_budget,
  output logic [7:0]  n_evals, n_over,
  output logic        coh_err
);
  logic [31:0] m_q, t_q, p_q;
  logic [15:0] true_per_write, true_total, true_over;
  logic        truly_over;
  // A write must reach every other sharer.
  assign m_q = (sharers == 16'd0) ? 32'd0
             : ({16'd0, sharers} - 32'd1) * {16'd0, msg_per_write};
  assign true_per_write = (m_q > 32'd9999) ? 16'd9999 : m_q[15:0];
  assign msgs_per_write = (IT_IS_JUST_COHERENT != 0) ? 16'd0 : true_per_write;
  assign t_q = {16'd0, true_per_write} * {16'd0, writes};
  assign true_total = (t_q > 32'd9999) ? 16'd9999 : t_q[15:0];
  assign total_msgs = (IT_IS_JUST_COHERENT != 0) ? 16'd0 : true_total;
  assign true_over = (true_total > budget_msgs) ? (true_total - budget_msgs) : 16'd0;
  assign over_budget = (IT_IS_JUST_COHERENT != 0) ? 16'd0 : true_over;
  assign p_q = (budget_msgs == 16'd0) ? 32'd999
             : (({16'd0, true_total} * 32'd100) / {16'd0, budget_msgs});
  assign budget_pct = (p_q > 32'd999) ? 16'd999 : p_q[15:0];
  assign within_budget = (over_budget == 16'd0);
  assign truly_over = (true_over != 16'd0);
  assign coh_err = evaluate && truly_over && within_budget;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_over <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_over) n_over <= n_over + 8'd1;
    end
  end
endmodule

Eight sharers with two hundred writes at three messages each is twenty-one messages per write and four thousand two hundred in total — twelve hundred over a three-thousand budget, at a hundred and forty percent.

The second case is the configuration that fits. Eight sharers at one message and a hundred writes costs seven hundred against a three-thousand budget, both builds agree, and neither alarms.

The third case is the one that shows where the cost comes from. A single sharer generates no coherence traffic at all — there is nobody to tell — which is the honest reason a design with one writer per region has no coherence problem to solve and should say so rather than claiming it solved one.

The boundary pair is exact: exactly at the budget is within it and one message past is not.

The seventh case is the ratio clamp, and it is one this batch's rules forced. A budget percentage past its ceiling — forty-two hundred messages against a budget of twenty — saturates, and it is a separate case from the unstated-budget one because the guard there returns the ceiling value by a different route. Section 17 explains why that distinction matters.

The scaling is the part that decides whether this mode is usable at all, and the model makes it visible by keeping the sharer count and the per-sharer cost separate. Traffic grows with the product of sharers and writes, so doubling the group doubles the cost of every write that was already happening. The link carrying that traffic does not double, which is why coherent sharing has a group size beyond which it stops being an option regardless of how the protocol is implemented.

That is the honest reason most real designs use one of the other three modes for the bulk of their data and reserve coherence for small, rarely-written structures — a flag, a counter, a work-queue head. Those are exactly the cases where the sharer count is high and the write count is low, which is the corner of the product where the arithmetic is comfortable.

The single-sharer case is worth dwelling on because it is the design pattern rather than a degenerate input. One writer per region generates no coherence traffic at all: there is nobody to tell. A system partitioned so that every region has exactly one writer has not solved the coherence problem — it has avoided having one, and that is usually the better engineering. The model reports zero, and a description that claims to have solved coherence when it actually partitioned around it has overstated what it did.

The budget input is the part the model takes on trust, and it deserves the same scepticism as any other. A coherence-message budget is a share of a link that other traffic also uses, so it is a negotiated number rather than a specification one — and a design sitting at a hundred percent of it is a design that has assumed nothing else will ever want the link.

7. RTL 3 — A Peer Read And A Read Through A Host Are Different Distances

The third thing, and the one most often assumed rather than measured.

"Peer-to-peer" is a claim about a path, and paths have hops. A read that genuinely goes device to device is one distance; a read that leaves the device, crosses to a host, and comes back is another, and the second is what happens whenever the topology does not actually provide the direct route the software is written for.

// RTL 3 - the path matters. A peer-to-peer read and a read that goes out to a
// host and back are different distances, and a description that says "shared"
// without saying which one has left out the number that decides the design.
module peer_path #(parameter int ALL_PATHS_ARE_PEER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] peer_ns, host_hop_ns, hops, accesses,
  output logic [15:0] path_ns, added_ns, total_us, path_ratio,
  output logic        path_is_direct,
  output logic [7:0]  n_evals, n_indirect,
  output logic        path_err
);
  logic [31:0] a_q, p_q, t_q, u_q;
  logic [15:0] true_added, true_path;
  logic        truly_indirect;
  assign a_q = {16'd0, host_hop_ns} * {16'd0, hops};
  assign true_added = (a_q > 32'd9999) ? 16'd9999 : a_q[15:0];
  assign added_ns = (ALL_PATHS_ARE_PEER != 0) ? 16'd0 : true_added;
  assign t_q = {16'd0, peer_ns} + {16'd0, true_added};
  assign true_path = (t_q > 32'd9999) ? 16'd9999 : t_q[15:0];
  assign path_ns = (ALL_PATHS_ARE_PEER != 0) ? peer_ns : true_path;
  assign p_q = (peer_ns == 16'd0) ? 32'd999
             : (({16'd0, true_path} * 32'd100) / {16'd0, peer_ns});
  assign path_ratio = (p_q > 32'd999) ? 16'd999 : p_q[15:0];
  // What the indirection costs over a run, in microseconds.
  assign u_q = ({16'd0, true_added} * {16'd0, accesses}) / 32'd1000;
  assign total_us = (u_q > 32'd9999) ? 16'd9999 : u_q[15:0];
  assign path_is_direct = (added_ns == 16'd0);
  assign truly_indirect = (true_added != 16'd0);
  assign path_err = evaluate && truly_indirect && path_is_direct;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_indirect <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_indirect) n_indirect <= n_indirect + 8'd1;
    end
  end
endmodule

Three hundred nanoseconds peer with two host hops at two hundred and fifty is five hundred added, eight hundred on the path, nearly three times a direct access, and two thousand microseconds of indirection over four thousand accesses.

Figure 2 — the difference between a path a program was written for and the path it gets. Both agree the access completes and both agree what it returns. They disagree about the distance, and the run total is where that disagreement becomes a schedule.

The second case is the genuinely direct path. No hops adds nothing and reports exactly a direct access — both builds agree and neither alarms, which is what a correctly described peer topology looks like.

The third case separates the two ways of being direct. Hops that cost nothing also add nothing, and the model counts the path as direct — because what matters is the time, not the topology diagram.

The boundary is strict: one nanosecond of indirection is not a direct path.

The sixth case is the ratio clamp driven independently of its guard, and the seventh is the run total driven past its own counter — both there for the reason section 17 gives.

The hop count is the cheapest unchecked number in this chapter, and that combination is what makes this failure so common. It is available from a topology diagram, it takes one minute to read, and it is almost never read — because "peer-to-peer" arrives in the vocabulary of the software that will use the path rather than the hardware that provides it. The software is written against an intent and the intent is not a measurement.

The three-times figure is the one to carry into a review, because it converts directly into which accesses may take this path. An access made once per kernel launch can afford three times the distance; one made in an inner loop cannot. The ratio is therefore the same kind of threshold 29.1 section 7 derives for a memory tier — a line between what may be moved and what may not — arriving here as a line between what may be reached remotely and what must be local.

The run total exists because a per-access figure is not persuasive. Five hundred nanoseconds sounds negligible and is; two thousand microseconds across four thousand accesses is a number somebody will act on, and they are the same fact. A review that sees only the per-access number will approve the path, which is why the model publishes both and why the multiplier is an input rather than an assumption.

And the third case is a genuine caution against over-reading the topology. Hops that cost nothing add nothing, and the model counts that path as direct — because what matters is the time, not the diagram. A design with a hop through a switch that adds no measurable latency has not been penalised, and a model that counted hops rather than nanoseconds would have penalised it wrongly.

8. RTL 4 — Replication Spends Capacity, Sharing Spends Bandwidth

The fourth thing, and the trade the first two modes represent.

There are two ways for several devices to use one buffer. Give each its own copy, which costs capacity on every device and nothing at access time; or let them all reach one copy, which costs nothing in capacity and puts every read on the link. Both are correct answers and they are answers to different constraints, so a design has to say which one it bought.

// RTL 4 - replication and sharing are the two answers, and they spend different
// currencies. Giving every device its own copy costs capacity; letting them
// reach one copy costs bandwidth. A design has to say which it bought.
module replicate_or_share #(parameter int COPIES_ARE_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] devices, buffer_gb, per_device_gb, reads_each,
  output logic [15:0] replicated_gb, copies_over, shared_traffic, capacity_pct,
  output logic        replication_fits,
  output logic [7:0]  n_evals, n_over,
  output logic        rep_err
);
  logic [31:0] r_q, s_q, p_q;
  logic [15:0] true_rep, true_over, avail;
  logic        truly_over;
  assign r_q = {16'd0, devices} * {16'd0, buffer_gb};
  assign true_rep = (r_q > 32'd9999) ? 16'd9999 : r_q[15:0];
  assign replicated_gb = true_rep;
  // Every device pays the buffer out of its own memory.
  assign avail = per_device_gb;
  assign true_over = (buffer_gb > avail) ? (buffer_gb - avail) : 16'd0;
  assign copies_over = (COPIES_ARE_FREE != 0) ? 16'd0 : true_over;
  // Sharing one copy instead moves the reads across the link.
  assign s_q = {16'd0, devices} * {16'd0, reads_each};
  assign shared_traffic = (s_q > 32'd9999) ? 16'd9999 : s_q[15:0];
  assign p_q = (avail == 16'd0) ? 32'd999
             : (({16'd0, buffer_gb} * 32'd100) / {16'd0, avail});
  assign capacity_pct = (p_q > 32'd999) ? 16'd999 : p_q[15:0];
  assign replication_fits = (copies_over == 16'd0) && (devices != 16'd0);
  assign truly_over = (true_over != 16'd0);
  assign rep_err = evaluate && truly_over && replication_fits;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_over <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_over) n_over <= n_over + 8'd1;
    end
  end
endmodule

Eight devices with a ninety-six gigabyte buffer against eighty gigabytes each is seven hundred and sixty-eight gigabytes replicated, sixteen over per device, or twenty-four hundred units of traffic shared instead — a hundred and twenty percent of a device's memory.

The second case is the buffer small enough to replicate. Forty gigabytes in eighty fits everywhere at half a device's memory, both builds agree, and neither alarms — and note that the shared-traffic figure is reported anyway, because the choice is still a choice.

The boundary pair is exact: a buffer exactly the size of a device fits, and one gigabyte more does not.

The sixth case is the capacity ratio driven past its ceiling with its guard not firing, which is the shape this batch's second rule exists for.

The degenerate case bounds it: an unwritten set does not fit anything, because nobody said how many devices there are.

Replication has a reputation problem that the arithmetic does not support. It is the unglamorous option, it wastes capacity by construction, and it is very often the right answer — because for a read-only structure that fits in a device's own memory it costs nothing at access time, generates no coherence traffic, creates no failure coupling, and needs no barriers. Four of the nine costs this chapter counts are zero under replication, which is a strong position for an option people apologise for choosing.

Its single failure mode is the one section 5 describes, and it is abrupt rather than gradual: replication is correct exactly as long as nobody writes. The moment a writer appears, every other copy is stale and nothing reports it — no error, no message, no counter. That is why the mode has to be named rather than assumed, because the assumption is safe until it silently is not.

Sharing's cost, by contrast, is continuous and visible. Every read crosses the link, the traffic figure is measurable, and the degradation is proportional. A design that chose sharing and got the sizing wrong is slow; a design that chose replication and got the access pattern wrong is wrong. Those are not comparable failure modes, and the choice between the two currencies is also a choice between those two ways of being caught.

The model reports both figures in every case — the replicated total and the shared traffic — including the cases where one of them clearly wins. That is deliberate: a decision made without the number for the road not taken is a preference, and the second figure costs nothing to compute once the first one has been.

9. RTL 5 — Access Is Asymmetric And The Asymmetry Is The Design

The fifth thing, and the one a flat address space hides.

Three tiers, three latencies. A device's own memory, a peer's, and a pooled region. Presenting them as one address space is a convenience for the programmer and a trap for the performance model, because the blend a kernel actually experiences depends entirely on what fraction of its accesses leave home.

// RTL 5 - access is asymmetric and the asymmetry is the design. A device's own
// memory, a peer's memory and a pooled region are three latencies, and a model
// that treats them as one has averaged away the thing being decided.
module access_asymmetry #(parameter int ONE_FLAT_SPACE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] local_ns, peer_ns, pooled_ns, peer_fraction,
  output logic [15:0] effective_ns, spread_ns, peer_penalty, pool_penalty,
  output logic        space_is_flat,
  output logic [7:0]  n_evals, n_asymmetric,
  output logic        sym_err
);
  logic [31:0] e_q;
  logic [15:0] frac_ok, true_effective, true_spread, hi_ns;
  logic        truly_asymmetric;
  assign frac_ok = (peer_fraction > 16'd100) ? 16'd100 : peer_fraction;
  // The effective latency a kernel sees is the blend of local and peer access.
  assign e_q = (({16'd0, local_ns} * (32'd100 - {16'd0, frac_ok}))
             +  ({16'd0, peer_ns}  * {16'd0, frac_ok})) / 32'd100;
  assign true_effective = (e_q > 32'd9999) ? 16'd9999 : e_q[15:0];
  assign effective_ns = (ONE_FLAT_SPACE != 0) ? local_ns : true_effective;
  assign peer_penalty = (peer_ns > local_ns) ? (peer_ns - local_ns) : 16'd0;
  assign pool_penalty = (pooled_ns > local_ns) ? (pooled_ns - local_ns) : 16'd0;
  // The spread between the nearest and furthest tier is the asymmetry itself.
  assign hi_ns = (pooled_ns > peer_ns) ? pooled_ns : peer_ns;
  assign true_spread = (hi_ns > local_ns) ? (hi_ns - local_ns) : 16'd0;
  assign spread_ns = (ONE_FLAT_SPACE != 0) ? 16'd0 : true_spread;
  assign space_is_flat = (spread_ns == 16'd0) && (local_ns != 16'd0);
  assign truly_asymmetric = (true_spread != 16'd0);
  assign sym_err = evaluate && truly_asymmetric && space_is_flat;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_asymmetric <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_asymmetric) n_asymmetric <= n_asymmetric + 8'd1;
    end
  end
endmodule

Two hundred local, six hundred peer, nine hundred pooled, with a quarter of accesses remote, is three hundred nanoseconds effective, a seven-hundred-nanosecond spread, a four-hundred peer penalty and a seven-hundred pool penalty.

The second case is a genuinely flat space. Every tier at two hundred reports no spread and no penalties, both builds agree, and neither alarms.

The third case is why the model takes the furthest tier rather than a fixed one. A pool nearer than a peer makes the peer the far tier, and the spread follows it — the asymmetry is a property of the arrangement, not of which name a tier has.

The fourth case is the one worth keeping so the section is not one-sided. Local slower than both remote tiers reports no penalties at all and an effective latency better than local alone — which happens on a device whose own memory is the constrained one, and a model that assumed local is always nearest would report it backwards.

The fifth case clamps the remote fraction, and the sixth drives the effective-latency ceiling, which section 17 records as a survivor the campaign found.

The blend is the number a kernel actually experiences, and it is the reason a flat address space is a trap rather than merely a simplification. Presenting three tiers as one is correct for addressing — the pointer works, the load completes — and wrong for every performance model built on top of it. A programmer reasoning about one latency is reasoning about a number that does not exist, and the number that does exist depends on a fraction nobody reported.

The remote fraction is the input that carries the whole section and it is the hardest of the three to obtain. The latencies are on specification sheets; the fraction is a property of how the data was laid out and how the kernel walks it, and it moves when either changes. A blend computed once for a kernel and reused after a layout change is a stale number wearing a precise one's clothes.

The spread matters separately from the blend, which is why the model publishes both. The blend says what the average access costs; the spread says how wrong a uniform assumption can be. A system with a small blend and a large spread is one where most accesses are local and the occasional remote one is very expensive — which is exactly the shape that produces a tail rather than a mean regression, and 29.3 section 9 is what that costs at fleet scale.

The fourth case is in the chapter to stop the section becoming an argument. Local is not always nearest. A device whose own memory is the constrained resource can genuinely be better off reaching a peer, and the model reports no penalty and an effective latency better than local alone. A model that assumed local is fastest would report that design backwards, and the whole point of computing the blend is that it does not need the assumption.

10. RTL 6 — Synchronisation Is Serial In A System Bought For Parallelism

The sixth thing, and the one that decides whether the sharing scales.

Shared state needs barriers, and a barrier is where the fastest device waits for the slowest. That wait is serial time in a machine bought for parallel time, and it multiplies by the number of barriers rather than by the number of devices — which makes it a property of the algorithm rather than of the hardware.

// RTL 6 - sharing needs synchronisation, and synchronisation is a serial cost
// in a system bought for parallelism. Every barrier is a point where the
// fastest device waits for the slowest, and the count is the design.
module synchronisation_cost #(parameter int BARRIERS_ARE_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] barriers, devices, skew_us, compute_us,
  output logic [15:0] wait_us, total_wait, parallel_pct, serial_us,
  output logic        scaling_holds,
  output logic [7:0]  n_evals, n_serialised,
  output logic        sync_err
);
  logic [31:0] w_q, t_q, p_q;
  logic [15:0] true_wait, true_total;
  logic        truly_serialised;
  // At a barrier the whole group waits out the slowest device's skew.
  assign w_q = (devices == 16'd0) ? 32'd0 : {16'd0, skew_us};
  assign true_wait = (w_q > 32'd9999) ? 16'd9999 : w_q[15:0];
  assign wait_us = (BARRIERS_ARE_FREE != 0) ? 16'd0 : true_wait;
  assign t_q = {16'd0, true_wait} * {16'd0, barriers};
  assign true_total = (t_q > 32'd9999) ? 16'd9999 : t_q[15:0];
  assign total_wait = (BARRIERS_ARE_FREE != 0) ? 16'd0 : true_total;
  assign serial_us = total_wait;
  assign p_q = ({16'd0, compute_us} + {16'd0, true_total} == 32'd0) ? 32'd100
             : (({16'd0, compute_us} * 32'd100)
                / ({16'd0, compute_us} + {16'd0, true_total}));
  assign parallel_pct = (BARRIERS_ARE_FREE != 0) ? 16'd100 : p_q[15:0];
  assign scaling_holds = (total_wait == 16'd0) && (devices != 16'd0);
  assign truly_serialised = (true_total != 16'd0);
  assign sync_err = evaluate && truly_serialised && scaling_holds;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_serialised <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_serialised) n_serialised <= n_serialised + 8'd1;
    end
  end
endmodule

A hundred and twenty barriers with forty microseconds of skew against six thousand of compute is four thousand eight hundred microseconds waited — fifty-five percent of the run parallel.

The second case is the configuration that scales. Perfectly matched devices wait nothing at any barrier, both builds agree, and neither alarms — which is the honest statement that barriers cost nothing when there is no skew to absorb.

The third case is strict on purpose. A single barrier is still a barrier, and the measured build says the scaling does not hold — ninety-nine percent parallel is not a hundred.

The fourth case is the guard. Barriers with no devices described waits nothing, and the model declines to call that a scaling claim.

The sixth case is the far end. All barrier and no compute reports nothing parallel at all, which is the shape of an algorithm that synchronises more than it computes.

The skew is the input that makes barriers expensive, and it is usually caused by the thing the barriers are there to coordinate. A device that waited on a remote read arrives late; the barrier then makes every other device wait for it; and the cost of one slow access is multiplied by the group size and by the barrier count. That is a feedback loop rather than an independent cost, and it is why section 10 sits after sections 7 and 9 rather than beside them.

The multiplier is the barrier count, not the device count, which is the structural point and the one that makes this an algorithm property. Adding devices does not add barriers; it adds candidates for being the slowest one at each barrier, which raises the expected skew but does not change how many times it is paid. An algorithm restructured to synchronise half as often halves this cost outright, and no hardware change available does that.

The third case is strict on purpose and it is worth defending. One barrier is still a barrier, and ninety-nine percent parallel is not a hundred. That sounds pedantic at one barrier and stops sounding pedantic at a hundred and twenty, which is the same figure compounded — and the model reports the same verdict at both ends so that the boundary is not something a reader has to locate for themselves.

The parallel fraction is also the number that makes the trade-off legible to somebody who did not read the rest of the chapter. Fifty-five percent parallel on eight devices is a machine doing rather less than four devices' worth of work, and that framing tends to move a conversation faster than a microsecond count does.

11. RTL 7 — A Shared Region Couples The Devices That Use It

The seventh thing, and the one that changes the failure model rather than the performance one.

Two devices that shared nothing were two failure units. Once they share a region, a fault in it takes both, and the count of coupled devices is the number an availability model needs. It is not a probability and it is not a reliability figure — it is how many things stop.

// RTL 7 - a shared region couples the devices that use it. Two accelerators
// that shared nothing were two failure units; once they share a region they are
// one, and the coupling is a count rather than a probability.
module failure_coupling #(parameter int SHARING_IS_ISOLATED = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] devices, sharing_devices, regions, job_value,
  output logic [15:0] coupled, independent, lost_value, coupled_pct,
  output logic        isolation_kept,
  output logic [7:0]  n_evals, n_coupled,
  output logic        iso_err
);
  logic [31:0] v_q, p_q;
  logic [15:0] share_ok, true_coupled;
  logic        truly_coupled;
  assign share_ok = (sharing_devices > devices) ? devices : sharing_devices;
  assign true_coupled = share_ok;
  assign coupled = (SHARING_IS_ISOLATED != 0) ? 16'd1 : true_coupled;
  assign independent = devices - share_ok;
  assign v_q = {16'd0, coupled} * {16'd0, job_value};
  assign lost_value = (v_q > 32'd9999) ? 16'd9999 : v_q[15:0];
  assign p_q = (devices == 16'd0) ? 32'd0
             : (({16'd0, true_coupled} * 32'd100) / {16'd0, devices});
  assign coupled_pct = p_q[15:0];
  assign isolation_kept = (coupled <= 16'd1) && (devices != 16'd0);
  // Sharing one region across more than one device is what couples them; with
  // no region declared there is no coupling claim to check.
  assign truly_coupled = (true_coupled > 16'd1) && (regions != 16'd0);
  assign iso_err = evaluate && truly_coupled && isolation_kept;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_coupled <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_coupled) n_coupled <= n_coupled + 8'd1;
    end
  end
endmodule

Sixteen devices with six sharing three regions is six coupled, ten independent, and two thousand four hundred units of job value that goes with the region — thirty-seven percent of the estate.

The second case is the isolated configuration. One device using the region is a one-device failure, both builds agree, and neither alarms.

The boundary is deliberately strict. Two devices sharing is already coupled, because the second is somebody who did not previously depend on the first.

The sixth case is the guard, and it is the same shape 29.2 section 9 and 29.3 section 10 both needed. Sharers with no region declared does not alarm in either build, because with no region there is no coupling claim to check — and without that guard the measured build alarms on itself.

The coupling is invisible in every artefact an availability model normally consults. The devices are separate cards in separate slots with separate power and separate failure statistics, and every one of those facts stays true after they start sharing. What changed is a software configuration, and it changed the failure domain without changing anything an inventory records.

The independent count is the output to carry, for the same reason 29.2 section 9's is: it answers the question actually being asked. Sixteen devices with six coupled is not sixteen failure units and it is not one — it is ten plus one group of six, and any plan built on either of the round numbers is wrong in a different direction.

The region count is what makes the coupling a decision rather than a fact. Three regions across six devices is one arrangement; six regions each used by two devices is a very different one with the same total sharing, and it produces three failure units of two rather than one of six. How the sharing is partitioned decides the blast radius, and that partitioning is usually chosen for convenience rather than for isolation because nobody computed what it cost.

And the boundary at two is deliberately unforgiving. A region used by two devices couples two devices, and the model says isolation was not kept — because the second device now depends on something it previously did not, and that is the whole of what coupling means.

12. RTL 8 — Which Access Patterns Gain From Sharing

The eighth thing, and the one that says where the technique belongs.

A read-mostly structure is the best case for sharing and a write-heavy one is the worst. Reads of a shared copy cost a longer access; writes cost that plus section 6's coherence traffic on every one. The classification is per pattern, and an answer given for the whole application has to be wrong about part of it.

// RTL 8 - which access patterns gain. A pattern that reads a large structure
// rarely gains from sharing it; one that writes a small structure constantly
// pays coherence on every write, and counting which is which is the decision.
module where_sharing_wins #(parameter int SHARING_ALWAYS_WINS = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] patterns, read_mostly, write_heavy, gain_each,
  output logic [15:0] suited, unsuited, total_gain, suited_pct,
  output logic        suits_all,
  output logic [7:0]  n_evals, n_unsuited,
  output logic        pat_err
);
  logic [31:0] g_q, p_q;
  logic [15:0] rd_ok, wr_ok, true_suited, true_unsuited;
  logic        truly_unsuited;
  assign rd_ok = (read_mostly > patterns) ? patterns : read_mostly;
  assign wr_ok = (write_heavy > patterns) ? patterns : write_heavy;
  assign true_suited = (rd_ok > wr_ok) ? (rd_ok - wr_ok) : 16'd0;
  assign suited = (SHARING_ALWAYS_WINS != 0) ? patterns : true_suited;
  assign true_unsuited = patterns - true_suited;
  assign unsuited = (SHARING_ALWAYS_WINS != 0) ? 16'd0 : true_unsuited;
  assign g_q = {16'd0, true_suited} * {16'd0, gain_each};
  assign total_gain = (g_q > 32'd9999) ? 16'd9999 : g_q[15:0];
  assign p_q = (patterns == 16'd0) ? 32'd100
             : (({16'd0, true_suited} * 32'd100) / {16'd0, patterns});
  assign suited_pct = p_q[15:0];
  assign suits_all = (unsuited == 16'd0) && (patterns != 16'd0);
  assign truly_unsuited = (true_unsuited != 16'd0);
  assign pat_err = evaluate && truly_unsuited && suits_all;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_unsuited <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_unsuited) n_unsuited <= n_unsuited + 8'd1;
    end
  end
endmodule

Sixteen patterns with ten read-mostly and five write-heavy is five net suited, eleven not, at two hundred and seventy-five units of gain — under a third.

The second case is the application the technique is for. Every pattern read-mostly is suited entirely, both builds agree, and neither alarms.

The boundary case nets to nothing, because the gains and the harms cancel exactly.

The fourth case is the expensive direction. A mostly write-heavy set gains nothing at all, and the sharing view still calls it a sweep — the sharing was implemented, the coherence traffic is real, and nothing benefits.

The netting is the same arithmetic the other Module 29 chapters use and it says the same uncomfortable thing. A write-heavy pattern is not merely unhelped by sharing; it is harmed, because every write now generates section 6's traffic on top of a longer access. So the harms subtract from the gains, and a set split evenly nets to zero rather than to half.

The classification is cheaper here than in the other chapters, which is worth saying because it makes the section actionable. Read-mostly versus write-heavy is a property a developer usually knows by inspection — a lookup table, a weights buffer, a constant — and does not need profiling to guess correctly. The failure is not that the classification is hard; it is that nobody is asked to make it, because the decision is taken at the level of "the application" rather than the structure.

And the boundary case is the one that decides a marginal design. Equally read-mostly and write-heavy nets to nothing, and a design at that balance has not been shown to benefit — which is a more useful review outcome than a small positive number, because it prompts the question of which structures are on which side.

13. RTL 9 — Sharing Does Not Make A Bigger Device

The ninth thing, and the arithmetic error that survives design reviews.

Reaching more memory changes what a kernel can address; it does not change how fast the device reads its own. Those are different quantities in different units of usefulness, and adding a shared path's bandwidth to a device's own produces a number that describes no achievable operation.

// RTL 9 - sharing does not make a bigger device. Reaching more memory changes
// what a kernel can address; it does not change the rate at which that device
// can read its own, and a plan that adds the two has added incompatible things.
module not_a_bigger_device #(parameter int SHARING_ADDS_RATE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] own_gbps, shared_gbps, needed_gbps, reachable_gb,
  output logic [15:0] claimed_gbps, real_gbps, shortfall, rate_pct,
  output logic        rate_met,
  output logic [7:0]  n_evals, n_short,
  output logic        rate_err
);
  logic [31:0] c_q, p_q;
  logic [15:0] true_claimed;
  logic        truly_short;
  // The weak build adds the shared path's rate to the device's own.
  assign c_q = {16'd0, own_gbps} + {16'd0, shared_gbps};
  assign true_claimed = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign claimed_gbps = (SHARING_ADDS_RATE != 0) ? true_claimed : own_gbps;
  // What a kernel reading its own memory actually gets is unchanged.
  assign real_gbps = own_gbps;
  assign shortfall = (needed_gbps > claimed_gbps) ? (needed_gbps - claimed_gbps) : 16'd0;
  assign p_q = (needed_gbps == 16'd0) ? 32'd100
             : (({16'd0, own_gbps} * 32'd100) / {16'd0, needed_gbps});
  assign rate_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign rate_met = (shortfall == 16'd0) && (needed_gbps != 16'd0);
  // No `needed != 0` guard: for unsigned operands `needed > own` already
  // implies it, so the guard would be unreachable code and an equivalent
  // mutant. (Batch 030 finding; `domcheck.py` now decides this shape.)
  assign truly_short = (needed_gbps > own_gbps);
  assign rate_err = evaluate && truly_short && rate_met;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_short <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_short) n_short <= n_short + 8'd1;
    end
  end
endmodule

Nine hundred own against sixty shared, with nine hundred and forty needed, is forty short — and the adds-rate view reports nine hundred and sixty and declares the requirement met.

The second case is the device that genuinely meets the rate, and the boundary pair is exact: exactly the needed rate is met, one short is not.

The fifth case is the clamp on the weak build's own claim, which is worth having because a wrong number should still be a bounded wrong number — a model whose error output wraps is harder to read than one that saturates.

The seventh case is the honest zero: an unstated requirement is trivially met and is not a met requirement.

The error this section models is not a misunderstanding of the hardware; it is a units error. Both numbers are bandwidths, both are measured in the same units, and adding them is arithmetically valid. What makes the sum meaningless is that no single operation draws on both — a kernel reading its own memory gets the first, a kernel reading across the link gets the second, and nothing gets the total.

That is why the error survives review. There is no step in the calculation to object to, and the result is a plausible-looking number in the right units. The only defence is to keep the two quantities in separate columns from the start, which is what the model does by publishing the claimed and real rates side by side rather than a single figure.

Sharing's real contribution is addressability, and that is worth stating positively. A kernel that could not run at all because its working set did not fit can now run. That is a large and genuine benefit, and it is a different benefit from running faster — a design that needs the first should claim the first, and one that quotes a summed bandwidth has claimed the second while delivering the first.

The clamp on the weak build's own figure is a small point of discipline worth recording. Even the wrong number is bounded, because a model whose error output wraps produces a second, unrelated confusion on top of the first — and the object of the exercise is to make the weak build's mistake legible, not to make it unreadable.

14. RTL 10 — A GPU-Memory-Sharing Case Study Assembled

Nine sections of inputs. This one puts them together.

// RTL 10 - a GPU-memory-sharing case study assembled. Nine sections of inputs,
// one summary. "The GPUs share memory" is bit 0: true of four different things,
// and one sixth of a design.
module sharing_signoff #(parameter int SHARING_IS_THE_ANSWER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic        sharing_named, mode_named, coherence_owned,
  input  logic        path_stated, sync_counted, coupling_stated,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_evals, n_sound, n_claimed,
  output logic        signoff_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~sharing_named;
  assign fail_mask[1] = ~mode_named;
  assign fail_mask[2] = ~coherence_owned;
  assign fail_mask[3] = ~path_stated;
  assign fail_mask[4] = ~sync_counted;
  assign fail_mask[5] = ~coupling_stated;
  assign conditions_met = {15'd0, sharing_named} + {15'd0, mode_named}
                        + {15'd0, coherence_owned} + {15'd0, path_stated}
                        + {15'd0, sync_counted} + {15'd0, coupling_stated};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp: conditions_met sums six one-bit values, so the quotient cannot
  // exceed a hundred and a ceiling would be unreachable code.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  // The "they share memory" view reads bit 0 and stops.
  assign claimed = (SHARING_IS_THE_ANSWER != 0) ? sharing_named : truly_sound;
  assign sound = claimed;
  assign signoff_err = evaluate && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When the sharing has been named and any one of the other five fails, the assembled model reports that the case study is not sound and the sharing view reports a case study.

Across the eight evaluations, the assembled model calls one case study sound and the sharing view calls seven of eight a case study.

The bit order is by how much of the design each condition carries. Bit 1 is first among the five because four different designs answer to the word, and until one is picked the other four conditions have no subject. Bits 2 and 3 are the two costs the chosen mode incurs. Bit 4 is what the sharing costs in scaling and bit 5 is what it costs in availability.

"The GPUs share memory" is bit 0, and it fails differently from Module 29's other three. 29.1's names a category containing its own opposite; 29.2's describes the packaging; 29.3's is a claim about other people. This one is a claim that is true under four mutually exclusive designs, so two engineers can agree on it completely and be building different machines — and neither is wrong until integration.

The five other bits fail independently. A mode can be named by somebody who never asked who keeps it coherent. The coherence can be owned with nobody having measured whether the peer path is a peer path. The path can be right while the barrier count quietly serialises the run. And the coupling is the one that fails last, because it belongs to whoever writes the availability model rather than to whoever chose the sharing.

The right-hand terminal says four designs answer to it, and that is what makes this weak definition the most expensive of the four in Module 29. The others produce an incomplete description. This one produces agreement between people who disagree, and the disagreement surfaces at integration rather than at review.

Two of the six bits are owned by somebody other than whoever chose the sharing, which is the recurring structural point across Module 29. The coherence owner is a hardware and firmware question; the failure coupling belongs to whoever writes the availability model. Both are consequences of a decision taken elsewhere, so they are discovered rather than chosen — and a case study that reaches four bits and stops has almost certainly stopped at those two.

Bit 3 is the one that is cheapest to check and most often skipped. Counting hops on a topology diagram takes a minute and settles whether a path advertised as peer-to-peer is one. It sits third rather than first only because a hop count has no meaning until the mode is named — under replication there is no remote path to count.

And bit 4 is the only one of the six that can get worse without anybody changing anything. Barrier skew grows as a job scales, as data grows, and as the sharing itself introduces the waits that produce it. A design signed off at fifty-five percent parallel does not stay there, which makes section 10 the condition most worth re-measuring rather than re-deriving.

15. Quantitative Reasoning

Three modes of four unnamed, with a quarter of the regions called shared actually coherent.

Twenty-one coherence messages per write and four thousand two hundred in total, against a three-thousand budget — a hundred and forty percent.

Five hundred nanoseconds added to a three-hundred-nanosecond peer access, nearly three times direct, at two thousand microseconds over the run.

Sixteen gigabytes over per device to replicate, or twenty-four hundred units of traffic to share — a hundred and twenty percent of a device's memory.

A seven-hundred-nanosecond spread across three tiers, with a quarter of accesses remote giving three hundred effective.

Four thousand eight hundred microseconds waited across a hundred and twenty barriers — fifty-five percent of the run parallel.

Six devices of sixteen coupled by three shared regions, at two thousand four hundred units of job value.

Eleven patterns of sixteen unsuited, with five suited at two hundred and seventy-five units of gain.

Forty gigabytes per second short, on a device the adds-rate view reports as meeting its requirement.

One case study of eight sound; the sharing view counts seven. The assembled model's summary, and the chapter's.

16. Assertions

The testbenches carry 465 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported nothing on either testbench, and all five scripted checks passed on their first run for the third chapter in succession — after batch 029's 29.2 and 29.3 established that as reachable rather than exceptional.

One check now runs that did not exist before this chapter. Every mutation's anchor is verified to match its target file exactly once, before every campaign and after any edit to a model. Section 17 records why.

Both builds are asserted on every degenerate case. Nothing enumerated, nothing described, nothing measured, an unwritten set, an unmeasured space, an unwritten run, an unwritten estate, an unwritten pattern set, and an unmeasured device.

Every clamp that an input can reach is driven past its limit by a case where the guard does not also fire. That distinction is this batch's inherited rule and it accounts for four separate cases here — the coherence budget ratio, the path ratio, the capacity ratio and the rate percentage all have a zero-guard whose default equals or approaches the ceiling, and each needed a case where the guard stays quiet and the quotient exceeds the limit on its own.

Every threshold is asserted on both sides of its boundary. The coherence budget at exactly full and one message past; the path at zero and one nanosecond of indirection; replication at exactly a device's memory and one gigabyte more; the rate at exactly needed and one short; the blast radius at one device and two.

Every error output is checked in both directions in every case. Each model's second case is a configuration where the sharing view happens to be right.

17. Mutation Testing

132 mutations, 132 killed. Sixty-seven against the first testbench, sixty-five against the second. The first run left two survivors and one stale anchor, and all three were instructive.

Two survivors were clamps never driven, both closed by one case each — section 9's effective-latency ceiling, which no case had exceeded because every latency in the stimulus was under ten thousand, and section 10's per-barrier wait, for the same reason. The separating question settled both immediately: an input exists that would make the mutated line behave differently, so they were stimulus holes rather than equivalent mutants.

The third finding is a new dominated-guard shape, and it is this chapter's contribution. Section 13's truth signal was written (needed > own) && (needed != 0). Removing the second conjunct survived — and it survived because for unsigned operands needed > own already implies needed != 0, so the guard is unreachable code and the mutation is equivalent by construction.

That is the third form of a relationship this track has now found three times: batch 027's dominated clamp (a percentage ceiling behind a minimum), batch 028's dominated guard ((A != 0) && (S != 0) where A is zero whenever S is), and now a redundant ordering guard. domcheck.py decides all three, and the new clause was self-tested against a file carrying one dead guard and two live ones before being trusted.

The > form is dead and the >= form is live, which is the part worth carrying. (A >= B) && (B != 0) is not redundant, because A >= 0 is satisfied at zero — so the guard is what excludes the degenerate case. Running the extended check across the four chapters already signed off flagged three of those, all in the >= form, and hand-triage confirmed every one is load-bearing. No published kill count is affected and no signed-off chapter needed a change.

One stale anchor, and it is worth recording because it was self-inflicted. Removing the dead guard from the model invalidated a mutation that targeted that exact line. The harness correctly reported ANCHOR x0 rather than counting it as a kill — but a stale anchor silently removes a mutation from the campaign, so the total looks right while the coverage is not. That is now a scripted check of its own: every anchor must match its file exactly once, verified after any model edit and before every campaign.

18. Verification Strategy

Ask which of the four modes. Section 5. Replication, peer access, a pooled region, or coherent sharing — they cost different currencies.

Ask who keeps it coherent and count the messages. Section 6. Sharers times writes.

Ask whether the peer path is a peer path. Section 7. Count the hops.

Ask whether the buffer is replicated or shared, and price both. Section 8.

Measure the three latencies and the remote fraction. Section 9. The blend is what a kernel sees.

Count the barriers and the skew. Section 10. This is an algorithm property, not a hardware one.

Count how many devices one region couples. Section 11.

Classify the access patterns as read-mostly or write-heavy. Section 12.

Keep the addressable capacity and the achievable rate in separate columns. Section 13.

19. Synthesis and Implementation Reality

Vendor fabrics between accelerators exist precisely to make section 7's path direct, which is 28.4 arriving as a topology: a link that owns both ends can offer a device-to-device route that a general attach reaches through a host.

Coherent sharing across devices is the mode with the sharpest scaling limit, because section 6's traffic grows with the sharer count while the link that carries it does not. That is why most real designs use one of the other three modes and reserve coherence for small structures.

The concurrency argument from 9.6 applies to every remote access here. A longer path needs more requests in flight to sustain bandwidth, so section 9's blend understates the problem for a device whose request pool was sized for local latency — the latency is what the model computes, and the throughput consequence is worse.

Section 10's skew is frequently caused by the sharing itself, which makes it a feedback rather than an independent input: a device waiting on a remote read arrives at the barrier late, and the barrier then makes everybody wait. A model that took skew as given understates a design where the sharing produced it.

And replication remains the right answer far more often than its reputation suggests. It is unglamorous, it costs capacity, and for a read-only structure that fits it has no access-time cost at all — which is what section 8's second case reports and section 12's read-mostly column is really measuring.

20. Silicon Observability

Free, and from a topology diagram. How many devices, how many share a region, how many hops on the path. Sections 7 and 11.

Free, and from a specification. The three latencies. Section 9 needs all three, and the pooled one is the one most often missing.

Cheap, and from a profiler. Barrier count and skew. Section 10 — already instrumented in most parallel runtimes.

Moderate. Writes per shared line and the sharer count. Section 6 needs the workload characterised, not just the topology.

Moderate. The remote access fraction. Section 9's blend is decided by it and nothing reports it by default.

Expensive. Per-pattern read-mostly versus write-heavy classification. Section 12 needs each structure's access mix.

Expensive, and usually assumed. Whether a path advertised as peer-to-peer actually is one under the topology as built. Section 7's hop count is cheap to obtain and rarely obtained.

21. Debug Lab

Devices were configured to share and the application is slower.

Step 1 — ask which mode was actually implemented. Section 5. If the answer is "sharing", stop here and find out.

Step 2 — count the hops on the path. Section 7. A peer read that goes through a host explains a large slowdown immediately and is a topology reading rather than a measurement.

Step 3 — check the barrier count and the skew. Section 10. Already instrumented, and it separates a communication problem from a synchronisation one.

Step 4 — count writes to shared lines. Section 6. Heavy writing on a coherent structure is the mode that scales worst.

Step 5 — check the remote access fraction. Section 9. If it is high, the blend is the answer and the fix is placement rather than protocol.

Step 6 — check whether anything was replicated that should have been shared, or the reverse. Section 8.

Steps 1 and 2 cost minutes and between them explain most instances.

22. Design Review

Which of the four modes is this, and where is that written down?

Who keeps it coherent, and how many messages does a write generate?

How many hops does a peer access take under the topology as built?

Is this buffer replicated or shared, and what does the other option cost?

What are the three latencies, and what fraction of accesses are remote?

How many barriers, and what is the skew at each?

How many devices does one shared region couple?

Which structures are read-mostly and which are write-heavy?

What is addressable, and separately, what rate is achievable?

23. How This Appears In Real Engineering

The failure is agreement between people who mean different things, and it survives review because everybody nods.

The most common shape is section 5 straight through. A design document says the devices share memory. The hardware team implements a pooled region; the software team writes code assuming coherent sharing; both are consistent with the document; and the disagreement is discovered when the results are wrong rather than slow.

The second is section 7 and it is a topology problem wearing a software label. Code is written for peer-to-peer access on a system whose topology routes it through a host. Everything works, nothing errors, and the access is three times the expected distance — and the hop count that would have shown it was available on a diagram from the start.

The third is section 10. Sharing is added and the run gets slower with no communication bottleneck visible. The cause is that the shared state needed barriers, the barriers exposed skew that was always there, and the serial fraction grew — which is an algorithm property that no interconnect change will fix.

The fourth is section 11 and it is found during an incident. A fault in a shared region takes six accelerators, in a system whose availability model counted sixteen independent devices because they are sixteen physical cards.

The fifth is section 13 and it is the quietest. A capacity plan adds the shared path's bandwidth to the device's own and reports a rate no kernel can achieve. The number is arithmetically valid and describes no operation.

The sixth shape is section 12 and it is the one that damages the technique's reputation rather than any single project. Sharing is applied across an application because it helped the structure somebody tried it on. The read-mostly structures improve, the write-heavy ones get worse, and the aggregate is a small unattributable number — after which the approach is described internally as "not worth it", when what actually happened is that it was applied to the structures it harms as well as the ones it helps.

The pattern is that four of these five are cheap to check and none of them is visible in the sentence that introduced the design. The mode is a question, the hop count is a diagram, the barrier count is already instrumented, and the coupling is a count of devices. The method costs an afternoon; the failures cost an integration, and the ratio is the argument for the checklist rather than for the analysis.

24. Common Misconceptions

"The GPUs share memory." Which of the four modes? Section 5.

"It is coherent." At how many messages per write? Section 6.

"It is peer-to-peer." How many hops? Section 7.

"Sharing saves memory." Compared with replicating, at a bandwidth cost. Section 8.

"It is one address space." With a seven-hundred-nanosecond spread. Section 9.

"More devices, more throughput." Until the barriers. Section 10.

"A device failure loses a device." It loses the sharing group. Section 11.

"Sharing helps the application." The read-mostly parts of it. Section 12.

"Now the device has more bandwidth." It has more addressable memory. Section 13.

25. Interview Reasoning

"How do accelerators share memory over CXL?" The useful answer refuses the premise: there are four different things that sentence covers, and they have different costs. Replication gives every device a copy and costs capacity. Peer access lets one read another's and costs distance. A pooled region is capacity neither owns, reached across a link. Coherent sharing keeps one structure consistent and costs traffic proportional to writes times sharers. Naming which one is the first half of any real answer.

"What decides whether coherent sharing scales?" The message count. A write to a shared line has to reach every other holder, so the traffic grows with the sharer count while the link carrying it does not. That is why most designs reserve coherence for small structures and use one of the other three modes for the bulk.

"Why might a peer-to-peer read be slow?" Because it may not be peer-to-peer. A read that leaves the device, crosses to a host and comes back is a different distance from one that goes device to device, and whether the topology provides the direct route is a property of how the system was built rather than of how the code was written. The hop count is on a diagram and is rarely consulted.

"Sharing was added and the job got slower with no bandwidth bottleneck. What happened?" Most likely synchronisation. Shared state needs barriers, a barrier is where the fastest device waits for the slowest, and that wait is serial time in a machine bought for parallel time. It multiplies by the barrier count, which is an algorithm property — so no interconnect change fixes it.

"What does sharing cost in availability?" Coupling. Devices that shared nothing were independent failure units; once they share a region, a fault in it takes all of them. That is a count rather than a probability, and an availability model built on the physical device count will be wrong by exactly the size of the sharing group.

"Does sharing give a device more bandwidth?" No. It gives it more addressable memory. The rate at which it reads its own memory is unchanged, and adding the shared path's bandwidth to the device's own produces a figure that describes no achievable operation — which is an arithmetic error that passes review because both numbers are real.

26. Exercises

1. 5 modes present, 2 named; 30 regions with 9 coherent. Compute the unnamed count, the coherent fraction and the pinned fraction.

2. 12 sharers, 400 writes, 2 messages each, a 6,000-message budget. Compute per-write, total, overage and the budget fraction. How many sharers fit the budget?

3. 250 ns peer, 3 hops at 180, 9,000 accesses. Compute the added latency, the path, the ratio and the run total.

4. 12 devices, a 150 GB buffer, 128 GB each, 400 reads each. Compute the replicated total, the overage and the shared traffic. Which would you choose?

5. 180 local, 700 peer, 1,100 pooled, 40% remote. Compute the effective latency, the spread and both penalties.

6. 300 barriers, 60 us of skew, 9,000 us of compute. Compute the total wait and the parallel fraction. What barrier count keeps it above 90%?

7. 24 devices, 9 sharing 4 regions, 350 per job. Compute the coupled count, the independent count and the value at risk.

8. 20 patterns, 13 read-mostly, 6 write-heavy, 60 each. Compute suited, unsuited and gain.

9. 1,200 own, 80 shared, 1,250 needed. Compute the real shortfall and what the adds-rate view would report.

10. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify its position using the rule that the ordering is by how much of the design each condition carries.

27. Summary

Four things wear one word — replication, peer access, a pooled region and coherent sharing — and they cost capacity, distance, both, and traffic.

Somebody keeps it coherent, and coherence is a message count that grows with sharers times writes.

A peer read and a read through a host are different distances, and which one a system provides is a topology fact.

Replication spends capacity and sharing spends bandwidth, and both are correct answers to different constraints.

Access is asymmetric, and a flat address space hides a spread that decides the performance model.

Synchronisation is serial in a machine bought for parallelism, and the barrier count is an algorithm property.

A shared region couples the devices that use it, which changes the availability model rather than the performance one.

Read-mostly patterns gain and write-heavy ones do not, so the classification is per pattern.

Sharing does not make a bigger device — it makes a device that can address more memory at the same rate.

Six bits, and "the GPUs share memory" is one of them. One case study of eight is sound; the sharing view counts seven.