CXL.mem Performance Implications

Chapter 9.1 through 9.5 established what CXL.mem obliges a device to guarantee. This chapter closes Module 9 by asking the question those five deferred: what does each of those guarantees cost?

1. The Engineering Problem — A Fast Link That Idles

Here is the result that ends most first-generation CXL memory bring-ups. A device is measured at roughly half its link bandwidth. The link is not saturated. The media is not saturated. The requester has work queued the entire time.

Every component is under-utilised, and yet the system is at its limit.

The instinct is to look for the slow part. There isn't one. The limit is structural: a transaction occupies a resource for its whole round trip, and if you have fewer of those resources than the round trip demands, the pipe idles no matter how fast anything is.

That is a different failure from "something is slow", it is invisible to every utilisation counter taken alone, and it is the first of six costs this chapter measures.

2. The One-Sentence Model

Every guarantee is paid for in concurrency. Ordering, atomicity, a defined visibility point, byte-granular writes and bounded queues each convert into either serialisation or storage — and the currency in both cases is how many transactions can be in flight at once.

Call it the concurrency ledger. Latency is what one transaction experiences; concurrency is what the system can overlap. Throughput is set by the second, not the first, and every promise in 9.3 is a withdrawal from the same account.

3. What This Chapter Owns

Performance is discussed in several places in this curriculum, and the boundaries matter.

Module 18 answers "where do the nanoseconds go" and "how do I model this". This chapter answers a narrower and earlier question: given the obligations Module 9 established, what is the bill? It is the last chapter of CXL.mem, not the first chapter of performance analysis.

4. The Ledger

Architectural. The arrows are cost relationships, not signals.

The single most useful reframing in this chapter: these are not six independent problems. They all debit the same account. A design that spends its concurrency on ordering has less left for latency hiding, and a design that spends it on partial writes has less left for everything else. That is why "which one is the bottleneck?" is usually the wrong question — the right one is "what is this system's concurrency, and who is spending it?"

5. Teaching-model boundary

6. RTL 1 — Throughput Is Set by Concurrency, Not by Latency

This is the chapter's foundation, and it is the result most often got backwards.

  // A slot freed by this cycle's completion is usable this cycle only if the
  // credit return is bypassed. Without the bypass the pipe idles one cycle per
  // completion -- a real and measurable loss of achievable concurrency.
  wire [7:0] eff_out = outstanding_q - ((CREDIT_BYPASS && completes) ? 8'd1 : 8'd0);
  assign can_issue = NO_LIMIT ? 1'b1 : (eff_out < BUDGET[7:0]);

=== EXP1: throughput is set by concurrency, not by latency ===
  budget 8,  latency 16 : issued=104 done=104 peak=8
  budget 16, latency 16 : issued=200 done=200 peak=16
  budget 8,  NO bypass  : issued=96 done=96 peak=8
  independent oracle    : issued=104 done=104
  the 8-slot config saturated at exactly 8           : ok
  the 16-slot config saturated at exactly 16         : ok
  issue count matched an independent oracle          : ok
  the 16-slot config issued every cycle              : ok
  the 8-slot config issued exactly 104               : ok
  near-doubling: 1.9x or better                      : ok
  the shortfall is exactly the 8-issue ramp          : ok
  without a credit bypass the same budget issues less : ok

Read the first two lines carefully: the latency is identical. Both configurations have a 16-cycle round trip. One completed 104 transactions, the other 200. The only difference is how many could be in flight at once.

That is the whole point. Latency did not change and throughput doubled, which means latency was never the throughput limit — concurrency was. A team that responds to low bandwidth by attacking latency is optimising a variable that, past a point, does not appear in the throughput equation at all.

Why the ratio is 1.92× and not 2×

The 8-slot run issued 104, not 100. It gets a free head start: for the first 16 cycles nothing has completed yet, so it issues 8 transactions before the steady-state rate takes over. The assertion checks the exact relationship — 200 == 104 × 2 − 8 — rather than the idealised double, because rounding a measured result to the number the theory predicted is how a model stops being checkable.

The one-cycle credit return costs 7.7%

The third line is the same 8-slot budget without the same-cycle credit bypass: 96 issues instead of 104. Reading occupancy from a register means the slot freed by a completion is not visible until the next cycle, so the pipe idles one cycle per completion.

Eight percent of a memory device's bandwidth, lost to a single missing bypass path. It is invisible in every counter — the budget is honoured, the peak is right, nothing errors — and it shows only when achieved bandwidth is compared against the bandwidth-delay product.

7. Waveform — The Requester Wants To Issue and Cannot

Teaching-model timing derived from the simplified RTL in this chapter. Not CXL wire timing.

Cycle 4 is the whole chapter in one column. want_issue is high — the requester has work. can_issue is low. No queue is full, no error is set, no component is slow. Three slots are held by transactions that are simply not back yet, and until one returns, the system does nothing.

From cycle 5 onward, outstanding is pinned at 3 and issued advances exactly in step with done. The completion rate has become the issue rate. That is what a concurrency-limited system looks like, and no amount of link or media speed changes it.

8. RTL 2 — What an Ordering Promise Costs

9.3 argued that per-address ordering is worth up to two orders of magnitude over global ordering. This is that claim, measured.

  wire same_addr_busy = addr_busy_q[req_addr];
  wire room           = (inflight_q < BUDGET[7:0]);
  wire order_ok       = NO_ORDER     ? 1'b1
                      : GLOBAL_ORDER ? (inflight_q == 8'd0)
                                     : !same_addr_busy;

=== EXP2: what an ordering promise costs ===
  8 distinct addresses : per-addr=8 global=1 no-order=8
  per-address ordering accepted all 8                : ok
  global ordering accepted only 1                    : ok
  global ordering blocked the other 7                : ok
  peak concurrency 8 vs 1                            : ok
  offered past capacity: per-addr inflight=8 (budget 8) | no-order inflight=8
  the capacity limit held at exactly 8               : ok
  the unordered variant stopped at the same capacity : ok
  retired addr 9, retried addr 8 : inflight=7 blocked delta=1
  address 8 was still blocked: one retire frees one address : ok

Eight versus one. Global ordering does not slow a transaction down; it prevents transactions from overlapping. The per-transaction latency is unchanged and the achievable throughput falls by 8×, which is the same shape as §6 — a concurrency loss wearing a latency disguise.

Two limits, not one. The unordered variant also stopped at 8, because dropping the ordering rule does not raise capacity. Ordering and capacity are independent constraints on the same account, and conflating them is a common sizing error: a team that removes an ordering restriction and sees no improvement concludes ordering was free, when in fact they were capacity-bound the entire time.

Scope is the whole design space here. One busy bit per address costs storage proportional to tracked addresses; one global busy bit costs almost nothing and 8× the throughput. That is the trade, stated plainly, and it is why 9.3 insisted the guarantee be per-address rather than global.

9. RTL 3 — A Barrier Costs the Window It Drains

  wire drained = (uncommitted_q == 8'd0);
  assign fence_done = fence_active_q && (RETIRE_EARLY || drained);

=== EXP3: a barrier costs the window it must drain ===
  4 accepted, none visible : uncommitted=4 max=4
  the invisible window is exactly 4                  : ok
  fence issued, 5 cycles later : correct wait=5 done=0 | early-variant wait=0
  the correct fence has still not completed          : ok
  it has been waiting for exactly 5 cycles           : ok
  the retire-early variant never waited at all       : ok
  after drain : uncommitted=0 fence wait=9 fences=1
  the fence waited exactly 9 cycles                  : ok

A barrier has almost no logic and a large cost, and the cost is not its own. The measured wait was 9 cycles for a window 4 deep — the fence spent every one of them waiting for other people's writes to become visible.

That has a direct design consequence that surprises people: making the write path deeper makes barriers slower. Buffering improves write throughput and lengthens the accepted-but-invisible window, so the same barrier now waits longer. A device tuned purely for streaming-write bandwidth can be worse on a workload that fences, and the two workloads will disagree about which firmware revision is better.

The retire-early variant waited zero cycles, which is exactly why it is tempting and exactly why it is wrong: it converts 9.5's correctness guarantee into a throughput number.

The requester is idle for the whole of that middle section, and none of the work it is waiting for is its own barrier's. The barrier is a few gates; its cost is the four writes ahead of it. That is why §9's counter-intuitive result holds — a deeper buffer improves write throughput and makes this picture longer.

10. RTL 4 — Sub-Line Writes Cost Twice

  // Media cost this cycle: every write costs one store; a partial adds a read.
  wire [1:0] media_this_cycle = wr_valid ? (needs_merge_read ? 2'd2 : 2'd1) : 2'd0;

=== EXP4: partial writes double the media cost ===
  8 full-line writes   : writes=8 partial=0 media ops=8
  8 full-line writes cost 8 media operations         : ok
  + 8 partial writes   : writes=16 partial=8 media ops=24 merge reads=8
  16 writes cost 24 media operations, not 16         : ok
  every partial write took exactly one merge read    : ok
  no-merge variant     : media ops=16 (silently lost bytes err=1)
  the no-merge variant looks 33% cheaper             : ok
  ...and flagged destroyed bytes to prove why        : ok

Sixteen writes, twenty-four media operations. At a 50% partial mix the media load is 1.5×; at 100% partial it is 2×.

This is the one cost on the list that software controls. Ordering, barriers and queueing are properties of the device; the partial-write fraction is a property of how the workload writes. Aligning and batching stores to whole lines halves the media traffic of the write stream, and that is a larger win than most device-side tuning available.

The no-merge variant is instructive as a trap. It reports 16 media operations against the correct design's 24 — it looks 33% cheaper on every performance counter. It is also silently destroying bytes the writer never named, which is why lost_merge_err exists next to the performance counters rather than in a separate correctness block. A performance improvement that beats the correct design by skipping work should always be read as a correctness question first.

11. RTL 5 — Why the Last of the Utilisation Costs the Most

  wire      serve_slot = (svc_cnt_q == 8'd0);
  wire      do_serve   = serve_slot && (level_q != 8'd0);
  assign ready = DROP_ON_FULL ? 1'b1 : (level_q < DEPTH[7:0]);

=== EXP5: the last of the utilisation costs the most ===
  utilisation 0.5 : arrived=30 served=30 peak=1 occupancy sum=60
  a half-loaded queue barely builds                  : ok
  overloaded      : arrived=105 served=90 peak=16 stalled=45
  the overloaded queue filled completely             : ok
  and pushed back on the source                      : ok
  nothing was lost: arrived = served + resident      : ok
  drop-on-full abuse : level=4 arrived=34 lost-arrival flag=1

At half load the queue peaked at 1. At overload it filled completely, stalled the source 45 times, and served 90 of 105 arrivals.

The engineering content is the shape, not the two endpoints. Occupancy — and therefore waiting time — does not rise linearly with load; it stays near zero and then goes vertical as the arrival rate approaches the service rate. A device measured at 50% load tells you almost nothing about its behaviour at 90%, which is why capacity planning done at comfortable load is worthless, and why the interesting measurements are always taken near the knee.

Conservation held throughout: arrived equals served plus resident. That is the law that separates back-pressure from loss — a queue that stalls its source is working correctly, and a queue that accepts and discards is not, even though both keep the reported occupancy comfortable.

12. RTL 6 — The Mean Hides the Transaction That Hurt

      if (finish) begin
        if (fin_age > THRESHOLD[15:0]) n_over_q <= n_over_q + 16'd1;
        if (!MEAN_ONLY && (fin_age > max_age_q)) begin
          max_age_q  <= fin_age;
          worst_id_q <= finish_id;
        end
      end

=== EXP6: the mean hides the transaction that hurt ===
  8 transactions : finished=8 max age=40 worst id=7 over-threshold=1 sum=54
  all eight finished                                 : ok
  exactly one crossed the threshold                  : ok
  the tail was identified as id 7                    : ok
  the worst case is far above the mean               : ok
  mean-only variant : max age=0 (it cannot name the outlier)

Eight transactions, total age 54, mean 6.75, worst 40. The outlier is roughly 6× the mean, and it is 74% of all the time spent.

A mean latency of 6.75 describes none of these transactions. Seven were fast and one was catastrophic, and the average is a number that no transaction experienced. This is why service-level arguments are made in percentiles: the mean is the statistic most likely to be reported and least likely to explain a complaint.

The mean-only variant reports a maximum of zero. It is not merely less precise — it cannot name the transaction that caused the problem, so a post-silicon investigation using it has no thread to pull. worst_id_q is the cheapest useful counter in this chapter: one register, and it turns "some requests are slow" into "this one was, here is its identity".

13. RTL 7 — Telemetry That Separates Slow From Idle

=== EXP7: telemetry that separates slow from idle ===
  accepted=21 completed=8 backpressure=6 link busy=20 media busy=6 idle=608
  oracle: accepted=21 completed=8 backpressure=6 idle=5
  acceptances matched an independent oracle          : ok
  back-pressure cycles matched the oracle            : ok
  idle cycles matched the oracle                     : ok
  conservation abuse : completed=1 accepted=0 flag=1

The counters exist to answer one question that a single utilisation figure cannot: is this device slow, or is it not being asked?

9.1 made this point about stalls; here it becomes a measurement. The last row is the one teams forget, and it is the most common real answer: the device is idle because nothing is asking it, usually because the requester ran out of the very outstanding slots §6 is about.

14. Quantitative Reasoning

Illustrative, with assumptions stated.

The concurrency requirement

The governing relation is Little's Law — concurrency equals rate times latency:

outstanding = bandwidth x round-trip time / transaction size

Using 9.1's teaching figures of 32 GB/s, a 250 ns round trip and 64-byte transactions:

32e9 x 250e-9 / 64 = 125 outstanding transactions

125 transactions must be in flight simultaneously to keep that pipe full. Not 125 per second — 125 at every instant. A device with 32 slots achieves 26% of the link regardless of how fast its media is, which is the arithmetic behind §6's result.

What each guarantee withdraws

Applying the measured ratios to that 125-slot budget:

Ordering is the largest single line item by an order of magnitude, which is why 9.3 spends a whole chapter narrowing its scope. The credit bypass is the smallest and the cheapest to fix — one bypass path for 8%.

What a barrier costs

The measured window was 4 deep and the fence waited 9 cycles. Scaled to the 125-slot budget, a barrier issued at full occupancy waits for the whole window to drain — roughly one full round trip, 250 ns, during which that requester issues nothing.

A workload fencing every 100 accesses therefore spends about 1% of its time fenced at this depth; one fencing every 10 spends 10%. The fence frequency is a software property and the window depth is a hardware one, and the cost is their product — neither team can see it alone.

The knee

With arrival rate λ and service rate μ, queue occupancy scales roughly as ρ/(1−ρ) where ρ = λ/μ. At ρ = 0.5 that is 1 — which is exactly the peak of 1 measured in §11. At ρ = 0.9 it is 9, and at ρ = 0.99 it is 99.

The last 9% of utilisation costs eleven times more queueing than the first 90%. That is the argument for provisioning to about 70% rather than to 95%, and it is arithmetic rather than caution.

15. Assertions

Icarus Verilog 13.0 does not support concurrent SVA here, so every property is synthesisable checker logic verified in simulation. 51 assertions.

Safety

Liveness

Performance goals — not correctness

The separation matters more in this chapter than in any other in the module. Every line in the third table is a goal: missing it means the device is slower than intended, not that it is wrong. Promoting one of them to an assertion produces a design that fails regression on a slow day, and teams that do it learn to ignore the failure — which is how a real assertion later gets ignored too.

16. Mutation Testing

Twenty-five mutations. Twenty-five killed.

The first run scored 18 of 25. All seven escapes were on the carry-forward list from Modules 7 through 9, and closing them required three distinct techniques.

Three needed abuse instances — the drop-on-full queue, the unlimited-budget requester, and a counter fed completions that were never accepted. A checker for an impossible state is unreachable on a correct design, so deleting it changes nothing; the fix is a deliberately misconfigured instance that makes the state reachable without the instance under test scoring its own failure.

Two were the transient-peak problem, and it took a genuine insight to kill. A peak implementation that reads last cycle's value still eventually records the true maximum, because the peak value persists for at least one more cycle and the lagging comparison catches up. The only stimulus that distinguishes them is one where the maximum is created on the last edge before the check — so the fix was not a new assertion but moving an existing one, ahead of an idle cycle that was hiding the bug.

Two were untested paths: capacity beyond the budget, and retiring one address while another stayed busy.

One real RTL defect, found by mutation testing

      if (!NO_LIMIT && (outstanding_q > BUDGET[7:0])) overflow_err <= 1'b1;

The overflow checker was guarded by !NO_LIMIT — and NO_LIMIT is the only configuration that can ever exceed the budget. The checker switched itself off on the one case it existed to catch. Corrected:

      // Exceeding the DECLARED budget is always a reportable event. Guarding
      // this with !NO_LIMIT would switch the checker off on the only
      // configuration that can ever trip it.
      if (outstanding_q > BUDGET[7:0]) overflow_err <= 1'b1;

This is the second appearance of this exact defect in Module 9 — 9.1's overflow_err had it too. A checker guarded by a condition correlated with the fault it detects is a self-disabling checker, and it passes review easily because the guard always looks like a sensible narrowing.

And two testbench defects, found by the baseline

A stimulus-versus-sampling race. The first testbench assigned inputs with blocking assignments in the same instant as the sampling edge, so the design read stale values — the ordering block appeared to reject every second request to a free address. The design was correct throughout. The fix is a driving discipline in which every stimulus change lands 1 ns after an edge:

  // Drive discipline: every stimulus change lands 1ns AFTER an edge, so the
  // DUT never samples a signal that the testbench is changing at that instant.
  task step; begin @(posedge clk); #1; end endtask

Two wrong expectations, both mine. I asserted that the unordered variant would exceed the capacity limit — it does not, because dropping ordering does not raise capacity — and that retiring one address would leave the in-flight count unchanged. The design was right both times. That is the value of writing the expectation before running: a testbench that agrees with the design on every point has usually been written after it.

17. Verification Plan

The coverage cross that matters is load × ordering scope × write shape: half/knee/overload, crossed with global/per-address/none, crossed with full-line/partial. The knee-with-per-address-ordering-and-partial-writes point is where all three costs interact, and it is the operating point real workloads occupy — so it must be directed, because random stimulus lands there almost never.

18. Silicon Observability

The pairing in rows two and three is the most valuable diagnostic in the chapter. A single "stall cycles" counter cannot distinguish a device that is saturated from a requester that has run out of tags — the two have opposite fixes, and teams routinely buy faster media to solve the second one. Splitting one counter by whether slots were available answers it immediately.

worst_id is the cheapest high-value counter here. One register turns an unactionable complaint into a specific transaction.

19. Debug Lab

localparam OUTSTANDING = 32;   // sized from "how many can the media absorb"

  budget 8,  latency 16 : issued=104
  budget 16, latency 16 : issued=200
  the 16-slot config issued every cycle              : ok

assign can_issue = (outstanding_q < BUDGET);   // registered occupancy only

  budget 8,  latency 16 : issued=104
  budget 8,  NO bypass  : issued=96
  without a credit bypass the same budget issues less : ok

wire [7:0] eff_out = outstanding_q - (completes ? 8'd1 : 8'd0);
assign can_issue   = (eff_out < BUDGET);

// "Ordering is the bottleneck" -- relax it and measure
wire order_ok = 1'b1;

  offered past capacity: per-addr inflight=8 (budget 8) | no-order inflight=8
  the unordered variant stopped at the same capacity : ok

localparam WRITE_BUFFER_DEPTH = 64;   // was 8 -- "more buffering is better"

  4 accepted, none visible : uncommitted=4 max=4
  after drain : uncommitted=0 fence wait=9 fences=1
  the fence waited exactly 9 cycles                  : ok

media_line <= wr_data;   // skip the merge read -- "partial writes were slow"

  no-merge variant     : media ops=16 (silently lost bytes err=1)
  the no-merge variant looks 33% cheaper             : ok
  ...and flagged destroyed bytes to prove why        : ok

localparam QUEUE_DEPTH = 16;   // "peak occupancy was 1 in every lab run"

  utilisation 0.5 : arrived=30 served=30 peak=1
  overloaded      : arrived=105 served=90 peak=16 stalled=45
  the overloaded queue filled completely             : ok

// telemetry: total age and count -- the mean is computed in software
age_sum_q <= age_sum_q + fin_age;

  8 transactions : finished=8 max age=40 worst id=7 over-threshold=1 sum=54
  the tail was identified as id 7                    : ok
  mean-only variant : max age=0 (it cannot name the outlier)

if (fin_age > max_age_q) begin
  max_age_q  <= fin_age;
  worst_id_q <= finish_id;
end
if (fin_age > THRESHOLD) n_over_q <= n_over_q + 1;

if (!NO_LIMIT && (outstanding_q > BUDGET)) overflow_err <= 1'b1;

  unlimited abuse probe : outstanding=6 declared budget=4 overflow=1
  the abuse instance exceeded its declared budget    : ok
  and the overflow checker reported it               : ok

// Exceeding the DECLARED budget is always reportable.
if (outstanding_q > BUDGET) overflow_err <= 1'b1;

20. Design Review

1. What is this device's concurrency, and does the measured peak reach it?
2. Is the outstanding table sized from the bandwidth-delay product or from media absorption?
3. Is a completion's credit usable in the cycle it frees?
4. What is the ordering scope, and what does the blocked-by-order counter say?
5. Can you distinguish a capacity stall from an ordering stall in silicon?
6. What does a barrier cost at full occupancy, and how often does the workload fence?
7. What fraction of writes are partial, and does partial == merge_reads hold?
8. At what utilisation was the queue sized, and where is the knee?
9. Do you report a maximum and an identity, or only a mean?
10. Which counters here are goals and which are correctness alarms?
11. Which of your checkers is unreachable in every configuration you run?

21. How This Appears in Real Engineering

Architecture. The concurrency budget is the first number to compute and the one most often derived from the wrong quantity. Bandwidth-delay product, not media absorption.

RTL. The credit-return bypass is one path worth 8%, and it is a real timing decision rather than a free win — the comparison moves into the issue path.

DV. The knee, the hot address, and the partial-write mix must all be directed. Random stimulus lands at low utilisation, spreads addresses, and aligns writes.

Performance. The two-way split of stall cycles — slots free versus slots exhausted — is the single most valuable counter pair, because the two have opposite fixes.

Software. The partial-write fraction and the fence rate are both software-controlled and both convert directly into device cost. They are the two levers that do not require new silicon.

Post-silicon. max_age with worst_id is what turns "occasionally slow" into a transaction you can chase.

22. Common Misconceptions

23. Interview Reasoning

24. Exercises

1. Calculation. A device targets 48 GB/s with a 200 ns round trip and 64-byte transactions. Compute the required outstanding depth. Then recompute it for a workload that is 40% partial writes and blocks 15% of accesses on same-address ordering, and state which of the two corrections costs more.

2. Analysis. A device reports peak outstanding equal to its budget, near-zero idle cycles, and bandwidth at 60% of its bandwidth-delay product. Name the two candidate explanations that remain, and the single counter that separates them.

3. RTL task. Extend outstanding_budget to report stall cycles split by cause — slots exhausted versus no request offered. State the storage cost and explain why one aggregate stall counter cannot be post-processed into the same information.

4. Waveform analysis. Using §7's trace, identify every cycle where the requester wanted to issue and could not, and compute the achieved issue rate over cycles 5 to 10. Then state what would change if the budget were 4 rather than 3, and what would not.

5. Verification plan. Write the directed test that distinguishes a correct max_uncommitted from one that lags by one cycle. Explain why a randomly generated accept/visible stream will essentially never distinguish them.

6. Design review. A colleague proposes deepening the write buffer 8× to improve streaming bandwidth, noting no correctness property changes. Give the strongest version of that argument, then name what regresses, the counter that would prove it, and the workload that must be in the regression suite for anyone to notice.

7. Tradeoff. Your device is capacity-bound at 8 outstanding. You can spend an equal area budget on doubling the outstanding table or on halving the round-trip time. Compute the throughput of each, state which wins, and identify the operating condition under which the answer reverses.

25. Summary

Every guarantee is paid for in concurrency.

• Throughput is set by concurrency, not latency. Identical 16-cycle latency, 104 versus 200 completions — the only difference was how many could be in flight.
• Size the outstanding table from bandwidth × round-trip time. 32 GB/s at 250 ns and 64 B needs 125 in flight; 32 slots yields 26% of the link no matter how fast anything is.
• A missing credit-return bypass costs 8% — 96 issues against 104 — and no counter reports it.
• Global ordering costs 8× concurrency against per-address, which is the most expensive single guarantee in Module 9.
• Ordering and capacity are independent limits. Relaxing a non-binding constraint proves only that it was not binding.
• A barrier costs the window it drains, so deeper write buffering makes every fence proportionally slower — one revision, two workloads, opposite verdicts.
• Sub-line writes cost twice: 16 writes became 24 media operations. It is the one cost software controls.
• Occupancy is non-linear in load — 1 entry at ρ = 0.5, 99 at ρ = 0.99 — so a comfortable measurement carries no information about the knee.
• The mean hides the tail: mean 6.75, worst 40, one transaction of eight consuming 74% of the time.
• Separate goals from assertions. A performance goal in an assertion teaches a team to ignore failing tests.
• Verification: 25 of 25 mutations killed, 51 assertions. Mutation testing found a self-disabling checker — overflow_err guarded by the only condition that could trip it, its second appearance in this module — and the baseline found a testbench sampling race plus two wrong expectations of my own.

Module 9 — CXL.mem is complete. 9.1 gave the contract, 9.2 the host's path, 9.3 the guarantees, 9.4 and 9.5 the flows, and this chapter the bill. Module 10 turns from what a device must guarantee to what kind of device it is — beginning with Type 1 devices, which cache host memory and own none of their own.