CXL.mem Question

27.3 took the protocol where the device holds the host's memory. This chapter takes the one that runs the other way — the host addressing memory that lives on a card — and it is the protocol most of the deployed CXL market actually uses.

Explain CXL.mem. — "it is just more memory". True in the only sense that matters and false in five others, and every one of the five is a decision somebody has to make before the machine works well.

1. The Engineering Problem — Addressable Is Not Equivalent

A load reaches it, and that is the whole architecture. A hundred sixty-four-byte accesses moved in four-kilobyte grains is nine percent of the traffic useful — which is what a transfer does and what a load does not. Section 5.

It is further away, and how much further decides whether it is memory. Ten times the latency of attached DRAM against a fourfold budget is six times past the point where the tier stops behaving like memory. Section 6.

The allocator has to be told it is slower. Four hundred far pages in a thousand-page system, undeclared, is eighty hot pages placed on the slow tier by an allocator doing exactly what it was built to do. Section 7.

Capacity is not bandwidth. Four gigabytes at a sixth of the bandwidth density halves the machine's bandwidth per gigabyte while doubling its memory, which changes what the machine is good at rather than making it better. Section 9.

The failure domain moved. Eight hosts each holding a hundred and twenty-eight gigabytes on one device is eight times the blast radius of the DIMM it replaced, at the same component reliability. Section 13.

This chapter against 27.3, stated precisely. That one owns a device holding the host's memory. This one owns a host holding the device's — which is why every model here is about what the host and the operating system must know rather than about what two caching agents must agree, and why section 14's weak definition is a statement about the address space.

2. The One-Sentence Model

Device memory is memory in the sense that matters when a load reaches it without a driver, when it is near enough that a load from it is still a load, when the allocator knows it is slower, when the capacity was not added without bandwidth, when the device manages the media behind it, and when the failure domain it creates is understood — and "it is just more memory" is one of those six.

3. What This Chapter Owns

The content first.

CXL.mem puts device memory into the host's physical address space. A load instruction issued by a host core reaches a DRAM chip on a card across a link, and returns a value, with no driver in the path, no descriptor, no completion queue and no software involvement of any kind. That sentence is the entire architecture and everything else is a consequence of it.

The device is a memory controller, not a peripheral. It owns the media behind its range: refresh, error correction, scrubbing, and signalling when it cannot answer correctly. The host does not manage the DRAM on a CXL device any more than it manages the DRAM behind another socket's memory controller.

The host may cache it. Lines of device memory sit in host caches exactly as lines of attached DRAM do, which is what makes it usable as memory rather than as a slow mapped region. That has a consequence the device must handle, and section 11 is about it.

It is further away, and the whole design depends on how much. A load from CXL memory takes longer than a load from attached DRAM — further down a link, through a device's controller, and back. What makes the architecture work is that the difference is a small multiple rather than an order of magnitude. If it were a hundred times further it would be a block device with an unusual programming model, and nobody would want it.

Type 3 is the common case and it is this protocol plus CXL.io. No CXL.cache, no accelerator, no coherent device attach — a memory expander. When an interviewer says "CXL device" without qualification they usually mean this.

It is worth saying why that is so, because "Type 3 is most of the market" sounds like a fact to memorise and is actually a consequence. A memory expander is the device with the shortest path from a real problem to a product. It needs no accelerator, no coherency agent, no novel programming model and no application changes — the memory appears, the operating system uses it, and the value is realised by software that was written before CXL existed. A Type 1 or Type 2 accelerator needs all of those things plus a workload that wants coherent device attach, which is a much longer chain of things that have to be true simultaneously.

What it is for, in two shapes. The first is capacity: a machine that needs more memory than its DIMM slots hold. The second, and the one that pays better, is pooling: memory bought for one host and unused by it, made available to a host that needs it. Section 12 is the arithmetic, and the pooling case is the argument that justifies the architecture's existence rather than merely its convenience.

Two things the host does that are worth naming. The first is address decoding: the host has to know which physical addresses belong to which device, which is programmed over CXL.io as 27.2 section 9 describes, and is the dependency that makes an io failure a memory failure at bring-up and not afterwards. The second is interleaving: a host can spread a contiguous physical range across several devices to aggregate their bandwidth, which works exactly as DRAM channel interleaving does and fails in exactly the way 26.6 section 9 describes when the access size is smaller than the granule.

Neither belongs in a ninety-second answer. Both belong in the next ninety, and knowing which is which is what 27.1 section 11 is about.

4. Teaching-Model Boundary

Every model in this chapter is a teaching model, not a memory system. It computes the one relationship the section is about and nothing else. There is no address decoder, no DRAM timing, no page table and no allocator anywhere in this file.

Each model is built twice from one source. A parameter selects between the measured build, which computes what the tier actually is, and the just-memory build, which computes what follows from treating it as indistinguishable from attached DRAM. Every section's headline number is the gap between them.

5. RTL 1 — It Is Memory, Not A Device Behind A Driver

Start with the property that makes the whole thing worth building, because it is also the one that is easiest to state and hardest to appreciate until you have written the alternative.

A load instruction reaches it. Not a descriptor, not a DMA engine, not a completion the driver polls for — a mov from an address. The distinction sounds like a convenience and is an architectural one, because it changes what granularity is possible.

A transfer moves whole grains. A DMA engine configured for four-kilobyte transfers moves four kilobytes whether the program wanted sixty-four bytes of it or all of it. A load moves what it asked for. For a workload that touches a large structure sparsely, that difference is the difference between a viable architecture and an unusable one.

The clamp in this model exists because both figures can saturate, and a saturated load figure against a saturated transfer figure reports no waste — correctly, since two numbers the counter cannot hold are not a measurable difference. The third stimulus case drives exactly that, and it is the honest limit of what a sixteen-bit model can say about a large transfer.

// RTL 1 - it is memory, not a device behind a driver. The whole of CXL.mem is
// that a load instruction reaches it, which is a different thing from a fast
// way to move bytes.
module addressable_memory #(parameter int BYTES_MOVED_IS_BYTES_MOVED = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] accesses, bytes_wanted, bytes_per_access, dma_grain,
  output logic [15:0] load_store_bytes, dma_bytes, waste_bytes, useful_pct,
  output logic        addressable,
  output logic [7:0]  n_evals, n_wasteful,
  output logic        not_memory_err
);
  logic [15:0] true_waste;
  logic [31:0] l_q, d_q, u_q;
  logic        truly_wasteful;
  // A load moves what it asked for. A transfer moves whole grains.
  assign l_q = {16'd0, accesses} * {16'd0, bytes_per_access};
  assign load_store_bytes = (l_q > 32'hFFFF) ? 16'hFFFF : l_q[15:0];
  assign d_q = {16'd0, accesses} * {16'd0, dma_grain};
  assign dma_bytes = (d_q > 32'hFFFF) ? 16'hFFFF : d_q[15:0];
  assign true_waste = (dma_bytes > load_store_bytes)
                      ? (dma_bytes - load_store_bytes) : 16'd0;
  assign waste_bytes = (BYTES_MOVED_IS_BYTES_MOVED != 0) ? 16'd0 : true_waste;
  assign u_q = (dma_bytes == 16'd0) ? 32'd100
             : (({16'd0, load_store_bytes} * 32'd100) / {16'd0, dma_bytes});
  assign useful_pct = (u_q > 32'd100) ? 16'd100 : u_q[15:0];
  assign addressable = (waste_bytes == 16'd0) && (bytes_wanted != 16'd0);
  assign truly_wasteful = (true_waste != 16'd0) && (bytes_wanted != 16'd0);
  assign not_memory_err = evaluate && truly_wasteful && addressable;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_wasteful <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_wasteful) n_wasteful <= n_wasteful + 8'd1;
    end
  end
endmodule

A hundred sixty-four-byte accesses is six thousand four hundred bytes of useful traffic. Moved in four-kilobyte grains it is four hundred kilobytes of traffic for six kilobytes of data — nine percent useful, and the bytes-are-bytes view reports no waste at all because from its point of view every byte moved was a byte moved.

Figure 1 — the same six kilobytes of demand, served two ways. The upper path is not a badly built DMA engine; it is a correctly built one moving the granularity it was configured for. The gap is entirely in the relationship between the access size and the grain, which is why the honest form of this argument is conditional: match the grain to the access and the two paths are identical, and the whole of CXL.mem's granularity advantage is the case where they cannot be matched.

The sixth case shows the ratio is a property of the grain rather than a fixed penalty. A two-hundred-and-fifty-six byte grain wastes three quarters, not ninety-one percent, and a grain that matches the access wastes nothing. This is the honest version of the argument: DMA is not inherently wasteful, it is wasteful when the access is finer than the grain, and the whole of the sparse-access case is that condition.

The second case is the one to concede, and conceding it makes the rest of the argument stronger. When the grain matches the access there is nothing between them — both builds report the same thing, the transfer is exactly as efficient as the load, and the only remaining difference is the software model. For dense access to a contiguous buffer, an explicit transfer is a perfectly good answer and this protocol is solving a problem you do not have.

Two degenerate cases bracket it. A workload that wants nothing is not being served wrongly however much is moved, and the model declines to report — a guard that stops it firing on every idle interface. And a transfer with no grain size configured moves nothing and wastes nothing, which is an instrument artefact rather than a perfect DMA engine.

There is a consequence of the load-reaches-it property that is worth having ready because it is the natural follow-up. A load that misses in cache blocks the core issuing it. There is no queue to hide behind, no completion to poll, no opportunity for software to do something else — which means section 6's latency lands directly on instruction-level parallelism rather than on a bandwidth number. That is the mechanism by which a far tier's latency becomes an application's problem, and it is also why the out-of-order window and the memory-level parallelism a workload can sustain matter more for a far tier than for attached memory.

6. RTL 2 — It Is Further Away, And How Much Further Decides Everything

The second fact, and the one that separates an answer that has read about CXL from one that understands why it exists.

Device memory is further away than attached DRAM. That is unavoidable: a load leaves the host, crosses a link, enters a device's memory controller, reaches DRAM, and comes back. What makes the architecture work is that this is a small multiple — the same order of magnitude as a load from another socket's memory, rather than a different kind of operation.

The importance of that is easy to miss because it is a quantitative claim underneath a qualitative-sounding one. If device memory were a hundred times further away, every property in this chapter would still be true and the thing would be useless: it would be a block device addressed by loads, and the correct software response would be to stop issuing loads to it. The architecture is viable at a small multiple and not at a large one, and knowing that is what makes the latency number a design constraint rather than a specification detail.

// RTL 2 - it is further away, and the whole architecture depends on how much
// further. CXL.mem is useful because a load from it is in the same order of
// magnitude as a load from attached DRAM; a hundred times further would be a
// block device with extra steps.
module latency_order #(parameter int MEMORY_IS_MEMORY = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] near_lat_ns, far_lat_ns, budget_x, accesses,
  output logic [15:0] ratio_x, over_by_x, stall_ns, budget_pct,
  output logic        latency_bounded,
  output logic [7:0]  n_measures, n_over,
  output logic        latency_err
);
  logic [15:0] true_over;
  logic [31:0] r_q, s_q, b_q;
  logic        truly_over;
  // How many times further away the far tier is.
  assign r_q = (near_lat_ns == 16'd0) ? 32'hFFFF
             : ({16'd0, far_lat_ns} / {16'd0, near_lat_ns});
  assign ratio_x = (r_q > 32'hFFFF) ? 16'hFFFF : r_q[15:0];
  assign true_over = (ratio_x > budget_x) ? (ratio_x - budget_x) : 16'd0;
  assign over_by_x = (MEMORY_IS_MEMORY != 0) ? 16'd0 : true_over;
  assign s_q = ({16'd0, far_lat_ns} > {16'd0, near_lat_ns})
             ? (({16'd0, (far_lat_ns - near_lat_ns)}) * {16'd0, accesses}) : 32'd0;
  assign stall_ns = (s_q > 32'hFFFF) ? 16'hFFFF : s_q[15:0];
  assign b_q = (budget_x == 16'd0) ? 32'd0
             : (({16'd0, ratio_x} * 32'd100) / {16'd0, budget_x});
  assign budget_pct = (b_q > 32'd999) ? 16'd999 : b_q[15:0];
  assign latency_bounded = (over_by_x == 16'd0);
  assign truly_over = (true_over != 16'd0) && (accesses != 16'd0);
  assign latency_err = measure && truly_over && latency_bounded;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_over <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (truly_over) n_over <= n_over + 8'd1;
    end
  end
endmodule

A far tier at ten times the near latency, against a budget of four, is six times past the point where the tier behaves like memory — two hundred and fifty percent of the budget. The memory-is-memory view reports it as bounded because from its point of view memory is memory.

The percentage saturates at nine hundred and ninety-nine rather than a hundred, for the same reason 27.1 section 5's does: exceeding the budget is the result here, and a clamp at the whole would delete the number the section exists to produce.

The sixth case is the one that explains why a latency problem waits. A far tier nobody accesses costs nothing today — the ratio is wrong, the budget is exceeded, and no stall is produced because nothing is reading it. A machine provisioned with a far tier and a workload that has not yet been placed on it will measure perfectly.

The degenerate case is instructive about the model rather than the protocol. A near tier measured at zero latency makes every far tier infinitely far, which is an instrument artefact rather than an architectural finding, and the model reports it as unbounded rather than dividing by zero and producing something confident and wrong.

What the budget should be is a fair question and the honest answer is that it depends on the workload, which is unsatisfying and true. A useful way to think about it: the budget is the ratio at which the workload's memory-level parallelism can still hide the latency. A workload with many independent accesses in flight tolerates a larger ratio, because the misses overlap; a pointer-chasing workload with one dependent access at a time tolerates almost none, because every cycle of extra latency is a cycle the core spends stopped. The same tier is comfortably within budget for one and hopelessly outside it for the other, on the same machine, at the same time.

7. RTL 3 — The Operating System Has To Be Told It Is Slower

The third fact, and the one that turns a correctly-built machine into a slow one.

An allocator places pages. Given memory it believes to be uniform, it places them uniformly — and a far tier presented as ordinary DRAM is memory the allocator believes to be uniform. The hot pages land on it in proportion to its capacity, because nothing told the allocator not to, and the allocator is doing exactly what it was built to do.

The mechanism for avoiding this is a declaration: the firmware describes the range as a distinct, slower tier, the operating system's memory manager reads that description, and page placement takes it into account. It is a data-structure entry rather than a piece of hardware, which is why it is so frequently the thing that was missed.

// RTL 3 - the operating system has to be told it is slower. Memory presented
// as ordinary DRAM is allocated from like ordinary DRAM, and the allocator
// puts hot pages on it because nothing said not to.
module tier_declaration #(parameter int A_PAGE_IS_A_PAGE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] pages_total, pages_hot, far_pages, tier_flag,
  output logic [15:0] far_capacity, hot_on_far, cold_on_far, cold_pct,
  output logic        tier_declared,
  output logic [7:0]  n_evals, n_misplaced,
  output logic        tier_err
);
  logic [15:0] spread_hot, true_misplaced;
  logic [31:0] h_q, p_q;
  logic        truly_misplaced;
  // The far tier holds what it holds.
  assign far_capacity = (far_pages > pages_total) ? pages_total : far_pages;
  // An allocator that has not been told spreads hot pages in proportion.
  assign h_q = (pages_total == 16'd0) ? 32'd0
             : (({16'd0, pages_hot} * {16'd0, far_capacity}) / {16'd0, pages_total});
  assign spread_hot = (h_q > {16'd0, pages_hot}) ? pages_hot : h_q[15:0];
  assign true_misplaced = (tier_flag != 16'd0) ? 16'd0 : spread_hot;
  // A page-is-a-page view counts no page as misplaced, declared or not.
  assign hot_on_far = (A_PAGE_IS_A_PAGE != 0) ? 16'd0 : true_misplaced;
  assign cold_on_far = (far_capacity > true_misplaced)
                       ? (far_capacity - true_misplaced) : 16'd0;
  assign p_q = (far_capacity == 16'd0) ? 32'd100
             : (({16'd0, cold_on_far} * 32'd100) / {16'd0, far_capacity});
  assign cold_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign tier_declared = (hot_on_far == 16'd0);
  assign truly_misplaced = (true_misplaced != 16'd0);
  assign tier_err = evaluate && truly_misplaced && tier_declared;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_misplaced <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_misplaced) n_misplaced <= n_misplaced + 8'd1;
    end
  end
endmodule

Four hundred far pages in a thousand-page system with two hundred hot pages, undeclared, puts eighty hot pages on the slow tier — twenty percent of the tier filled with exactly what should not be on it.

Figure 2 — the cheapest fix in the chapter. Nothing on the upper path is malfunctioning: the allocator was given memory it was told was uniform and it allocated uniformly, which is correct behaviour for the information it had. The difference between the two paths is a table entry describing the range, and the reason it goes missing is that it sits between the firmware team and the operating-system team and belongs to neither by default.

The second case is the fix and it costs nothing: the same system with the tier declared places no hot page on it. No hardware change, no workload change, one description.

The last case is the worst shape and it is worth reading carefully. A workload that is entirely hot fills the far tier completely with hot pages — zero percent correctly filled — and the page-is-a-page view is most wrong exactly there. That is not a pathological case; it is what a workload with a small, uniformly-accessed working set looks like, and it is the workload for which a far tier is least appropriate and most likely to be bought anyway.

The clamp in this model carries a real meaning too. A far tier configured larger than the system's total page count is what a configuration file that outlived its machine produces, and the model clamps it to the system rather than computing a tier with more pages than exist. Doing otherwise would produce a misplacement count larger than the number of hot pages, which is impossible and would be read as a model defect rather than as a configuration one.

The degenerate case added during mutation testing is about the instrument rather than the system. A page report naming hot pages in a system reporting no pages at all is an inconsistent snapshot — two counters read at different moments — and the model refuses to invent a misplacement from it. A model that did would fire on every system whose accounting was momentarily inconsistent, which is most of them, most of the time.

The declaration mechanism is worth naming precisely, because an interviewer may ask how it actually works. Firmware describes the memory in the platform's memory-affinity tables — the same structures that describe which memory belongs to which socket — with attributes indicating that the range is a separate, slower domain. The operating system reads those tables at boot and builds its memory topology from them, and its page-placement and migration policies then have something to act on.

The important property is that this happens once, at boot, from a static description. It is not negotiated and it is not discovered by measurement. If the description is wrong or missing, nothing later corrects it, and the operating system will behave correctly given information that is false — which is the shape of the whole failure.

8. RTL 4 — The Win Is In The Cold Pages

Section 7 is about the allocator not knowing. This one is about what it should do once it does, and it is the half of the argument that makes a far tier worth having at all.

A far tier holding a working set's cold pages costs almost nothing: the pages are rarely touched, the latency penalty is paid rarely, and the DRAM they displace is DRAM nobody had to buy. A far tier holding the hot pages costs on every access to them, and the capacity saving is bought at a price paid continuously.

The whole value proposition is that a working set is not uniform. If every page were equally hot, tiering would be a way of making some fraction of memory slower for no benefit. It works because real working sets have a temperature distribution, and the cold tail is often most of the capacity and almost none of the accesses.

// RTL 4 - the win is in the cold pages. A far tier holding a working set's
// cold half costs almost nothing and saves the price of the DRAM it replaces;
// the same tier holding the hot half costs every access to it.
module page_placement #(parameter int FILL_IT_WITH_ANYTHING = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] far_pages, hot_pages, hot_acc, cold_acc,
  output logic [15:0] hot_placed, cold_placed, cost_now, cost_best,
  output logic        placement_ok,
  output logic [7:0]  n_evals, n_misplaced,
  output logic        placement_err
);
  logic [15:0] best_hot;
  logic [31:0] n_q, b_q;
  logic        truly_worse;
  // Filling the tier with whatever is available takes the hot pages first.
  assign hot_placed  = (FILL_IT_WITH_ANYTHING != 0)
                       ? ((hot_pages > far_pages) ? far_pages : hot_pages) : 16'd0;
  assign cold_placed = far_pages - hot_placed;
  assign best_hot = 16'd0;
  assign n_q = ({16'd0, hot_placed} * {16'd0, hot_acc})
             + ({16'd0, cold_placed} * {16'd0, cold_acc});
  assign cost_now = (n_q > 32'hFFFF) ? 16'hFFFF : n_q[15:0];
  assign b_q = ({16'd0, best_hot} * {16'd0, hot_acc})
             + ({16'd0, (far_pages - best_hot)} * {16'd0, cold_acc});
  assign cost_best = (b_q > 32'hFFFF) ? 16'hFFFF : b_q[15:0];
  assign placement_ok = (cost_now <= cost_best);
  assign truly_worse = (hot_acc > cold_acc) && (far_pages != 16'd0)
                       && (hot_pages != 16'd0);
  assign placement_err = evaluate && truly_worse && (hot_placed != 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_misplaced <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_worse) n_misplaced <= n_misplaced + 8'd1;
    end
  end
endmodule

Four hundred far pages filled cold-first, against three hundred hot pages available, is four hundred units of access cost against fifteen thousand one hundred for the fill-with-anything policy — a factor of thirty-seven from an ordering decision.

The third case is the boundary and it is the honest limit of the whole idea. When hot and cold pages are accessed at the same rate the two policies cost the same, and there is no placement to get right. A workload with no temperature difference gains nothing from tiering and loses whatever the far tier's latency costs it.

The last case shows the scaling, and it runs the wrong way for anybody hoping the problem is self-limiting. The gap grows with the temperature difference — a thousand-to-one hot-to-cold ratio saturates the fill-with-anything cost before the model can express it. The more skewed the working set, the more a far tier is worth having and the more a bad placement policy costs.

Static placement is the simple case and it is not what a real system does. Page temperature changes, so a system that places well at allocation time drifts as the workload's access pattern moves, and the mechanism for correcting the drift is migration: promoting pages that have become hot to the near tier and demoting cold ones. That machinery has its own costs — tracking which pages are hot is not free, and moving a page is a copy plus a page-table update plus a shootdown — and a policy that migrates too eagerly spends more than the misplacement costs.

The model here is deliberately static because the static case is the one that has to be right first. A system that cannot place correctly at allocation time will not be rescued by migration, and a system that places well but never migrates is left with the drift. Both halves are needed and only one of them is a table entry.

9. RTL 5 — Capacity Is Not Bandwidth

The fourth fact, and the one that changes what a machine is good at rather than how fast it is.

A host with four DRAM channels has four channels of bandwidth. Adding a CXL memory device adds capacity and adds one link of bandwidth. The two do not scale together, which means the machine's bandwidth per gigabyte falls — and bandwidth per gigabyte is the quantity that determines whether a memory-bandwidth-bound workload runs well.

This is not a defect and it is not an argument against the architecture. It is a statement that the machine has changed shape: more capacity at lower bandwidth density, which is excellent for a capacity-bound workload and actively bad for a bandwidth-bound one. The mistake is not adding the tier; it is expecting a bandwidth-bound workload to improve because the machine has more memory.

// RTL 5 - capacity is not bandwidth. Four DRAM channels and one CXL link do
// not add capacity and bandwidth in the same proportion, and a tier that
// doubles the memory and adds a fifth of the bandwidth changes what the
// machine is good at.
module capacity_bandwidth #(parameter int MORE_MEMORY_IS_MORE_MEMORY = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] near_gb, near_bw, far_gb, far_bw,
  output logic [15:0] total_gb, total_bw, bw_per_gb_before, bw_per_gb_after,
  output logic        bandwidth_sized,
  output logic [7:0]  n_measures, n_diluted,
  output logic        bw_err
);
  logic [15:0] true_after;
  logic [31:0] a_q, b_q;
  logic        truly_diluted;
  assign total_gb = near_gb + far_gb;
  assign total_bw = near_bw + far_bw;
  assign b_q = (near_gb == 16'd0) ? 32'd0
             : ({16'd0, near_bw} / {16'd0, near_gb});
  assign bw_per_gb_before = (b_q > 32'hFFFF) ? 16'hFFFF : b_q[15:0];
  assign a_q = (total_gb == 16'd0) ? 32'd0
             : ({16'd0, total_bw} / {16'd0, total_gb});
  assign true_after = (a_q > 32'hFFFF) ? 16'hFFFF : a_q[15:0];
  // The more-memory view reports the bandwidth density it started with.
  assign bw_per_gb_after = (MORE_MEMORY_IS_MORE_MEMORY != 0)
                           ? bw_per_gb_before : true_after;
  assign bandwidth_sized = (bw_per_gb_after >= bw_per_gb_before);
  assign truly_diluted = (true_after < bw_per_gb_before) && (far_gb != 16'd0);
  assign bw_err = measure && truly_diluted && bandwidth_sized;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_diluted <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (truly_diluted) n_diluted <= n_diluted + 8'd1;
    end
  end
endmodule

Four gigabytes of attached memory at four hundred units of bandwidth is a hundred per gigabyte. Adding four gigabytes at sixty-four units gives eight gigabytes at four hundred and sixty-four — fifty-eight per gigabyte, a forty-two percent dilution — and the more-memory view reports the density the machine started with.

The sixth case is the shape a real memory expander takes. Sixty gigabytes of far memory on one link against four gigabytes of attached DRAM gives seven units per gigabyte against a hundred — a fourteenfold dilution — which is exactly right for a workload that needs to hold a large structure and touch it rarely, and catastrophic for one that streams.

The third case is the one that keeps the model honest. A far tier denser than the near memory raises the density, which is possible and does happen when the attached memory is small. The model reports it correctly rather than assuming every addition dilutes, because a model that only knew how to report dilution would be a slogan rather than a measurement.

The rule worth carrying out of this section is short: capacity scales with devices and bandwidth scales with links. A design that puts more capacity behind each link dilutes further; one that spreads the same capacity across more links dilutes less and costs more. Where a deployment sits on that line should follow from whether the workload is capacity-bound or bandwidth-bound, and that is a question with a measurable answer that is frequently not asked before the purchase.

There is a second-order effect worth mentioning because it cuts the other way. The far tier absorbs traffic that would otherwise have contended for the near memory's channels. A workload whose cold data now lives on the far tier leaves more of the attached memory's bandwidth for the hot data, so the density number overstates the harm for a workload with good temperature separation. That does not rescue a uniformly-hot workload, which is section 8's point, and it is why the two sections belong next to each other.

10. RTL 6 — The Device Is A Memory Controller

The fifth fact, and the one most often left out because it sounds like somebody else's problem.

A CXL memory device owns the media behind its range. Not the host — the device. It refreshes the DRAM, corrects the errors, decides when a word is unrecoverable, and signals poison when it cannot answer correctly. The host's memory controller does none of this for a CXL range, in the same way it does none of it for memory behind another socket.

The property this creates is the one to know: correctable errors accumulate, and scrubbing is what removes them. A range whose error rate exceeds its scrub rate accumulates standing errors, and a standing correctable error is a word one bit away from being uncorrectable. The failure is not the first error; it is the second one arriving in a word where the first has not been cleaned up.

// RTL 6 - the device is a memory controller. Correctable errors accumulate,
// scrubbing removes them, and a range whose scrub rate is below its error rate
// is a range where the second error in a word arrives before the first is
// cleaned up.
module media_management #(parameter int THE_HOST_HANDLES_ERRORS = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] err_rate_hr, scrub_rate_hr, hours, poison_ok,
  output logic [15:0] errs_made, errs_cleaned, standing_errs, clean_pct,
  output logic        media_managed,
  output logic [7:0]  n_evals, n_accumulating,
  output logic        media_err
);
  logic [15:0] true_standing;
  logic [31:0] m_q, c_q, k_q, p_q;
  logic        truly_accumulating;
  assign m_q = {16'd0, err_rate_hr} * {16'd0, hours};
  assign errs_made = (m_q > 32'hFFFF) ? 16'hFFFF : m_q[15:0];
  assign c_q = {16'd0, scrub_rate_hr} * {16'd0, hours};
  // Part-select of an expression is illegal, so the clamp lands in 32 bits
  // first and the narrowing is a separate assignment.
  assign k_q = (c_q > {16'd0, errs_made}) ? {16'd0, errs_made} : c_q;
  assign errs_cleaned = k_q[15:0];
  assign true_standing = errs_made - errs_cleaned;
  // A device that leaves errors to the host reports none standing of its own.
  assign standing_errs = (THE_HOST_HANDLES_ERRORS != 0) ? 16'd0 : true_standing;
  assign p_q = (errs_made == 16'd0) ? 32'd100
             : (({16'd0, errs_cleaned} * 32'd100) / {16'd0, errs_made});
  assign clean_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign media_managed = (standing_errs == 16'd0) && (poison_ok != 16'd0);
  assign truly_accumulating = (true_standing != 16'd0) && (hours != 16'd0);
  assign media_err = evaluate && truly_accumulating && media_managed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_accumulating <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_accumulating) n_accumulating <= n_accumulating + 8'd1;
    end
  end
endmodule

Ten correctable errors an hour against a scrub that clears four is six hundred standing errors after a hundred hours — forty percent cleaned — on a device the host-handles-errors view reports as having none.

The fifth case is worth noting because it is caught by both builds and is therefore not the interesting one. A device with no way to signal an uncorrectable error fails both views, because poison signalling is a binary capability rather than a rate. That failure does not ship; the accumulating one does, because it looks like a device that is working.

The last case is the limit and it is a real configuration. A device that does not scrub at all accumulates everything — a thousand standing errors after a hundred hours, nothing cleaned — and reports a clean range while it does. Patrol scrub is frequently disabled for performance or left unconfigured, and its absence produces no symptom until the second error arrives.

The poison mechanism deserves its own sentence because it is what makes an unrecoverable error survivable. When a device cannot return correct data for a line it returns it marked as poisoned rather than returning wrong data silently, and the host's machine-check machinery decides what to do — which is usually to kill the process that touched it rather than the machine. 26.4 section 7 is a whole section on what happens when that signalling is absent: the bad data is returned as ordinary data and is indistinguishable from correct data by anything downstream.

That is why the model treats poison capability as a separate term rather than folding it into the error rate. A device with a high error rate and working poison signalling is a device that degrades loudly. A device with a low error rate and no poison signalling is a device that will eventually corrupt something quietly, and the second is worse.

11. RTL 7 — The Host Caches This Memory, So The Device Must Get It Back

The sixth property, and the one that surprises people who have thought of CXL.mem as one-directional.

The host caches device memory. It has to, or the far tier would be unusably slow for anything with locality. That means at any moment some set of the device's lines are sitting in host caches, and the device cannot repurpose, re-map or reclaim those lines without asking the host to give them up first.

Asking has a mechanism and the mechanism has a capacity. A device that wants to hand a range to a different host, or re-map it, or take it offline for maintenance, must recall the lines the host is holding, and it can only have so many recalls outstanding at once. A device holding more cached lines than it has recall capacity for is a device that cannot reclaim its own memory on demand.

// RTL 7 - the host caches this memory, so the device must be able to get it
// back. A range the host is caching is a range the device cannot repurpose,
// re-map or reclaim without asking, and asking has a finite capacity.
module recall_capacity #(parameter int THE_HOST_GIVES_IT_BACK = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] host_cached, recall_slots, reclaims_wanted, stall_per_miss,
  output logic [15:0] recallable, unrecallable, reclaims_done, stall_cyc,
  output logic        recall_ok,
  output logic [7:0]  n_evals, n_stuck,
  output logic        recall_err
);
  logic [15:0] true_unrecallable;
  logic [31:0] s_q;
  logic        truly_stuck;
  // The device can ask for back as many lines as it has slots to ask with.
  assign recallable = (host_cached > recall_slots) ? recall_slots : host_cached;
  assign true_unrecallable = host_cached - recallable;
  assign unrecallable = (THE_HOST_GIVES_IT_BACK != 0) ? 16'd0 : true_unrecallable;
  assign reclaims_done = (reclaims_wanted > recallable) ? recallable : reclaims_wanted;
  assign s_q = {16'd0, true_unrecallable} * {16'd0, stall_per_miss};
  assign stall_cyc = (s_q > 32'hFFFF) ? 16'hFFFF : s_q[15:0];
  assign recall_ok = (unrecallable == 16'd0);
  assign truly_stuck = (true_unrecallable != 16'd0) && (reclaims_wanted != 16'd0);
  assign recall_err = evaluate && truly_stuck && recall_ok;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_stuck <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_stuck) n_stuck <= n_stuck + 8'd1;
    end
  end
endmodule

A thousand lines the host is caching against two hundred and fifty-six recall slots is seven hundred and sixty-eight lines the device cannot ask back, with two hundred and fifty-six of five hundred wanted reclaims completed.

The fifth case is the usual timing story and it is the same shape as sections 7, 10 and 27.3's capacity sections. A recall shortfall nobody exercises costs nothing today — the device is not trying to reclaim anything, the lines sit cached, the system is quiet. The cost arrives at the first repurpose, which in a pooled deployment is the operation the whole architecture exists to perform.

That is the connection worth making explicitly. Pooling is the reason to buy this and reclaim is how pooling works, so a device whose recall capacity is small relative to what the host caches is a device that is bad at precisely the thing it was bought for.

It is worth being clear that this is a CXL 3.0-era mechanism rather than something that was always there, because the question "how does the device get its memory back" has a different answer depending on which generation is being discussed. Earlier deployments handled reassignment by quiescing — stopping the host's use of the range entirely, which is correct and heavyweight. The back-invalidate path exists so that reclaim can happen without quiescing, which is what makes dynamic pooling practical rather than merely possible, and it is a good thing to know the shape of even if the details are outside a ninety-second answer.

12. RTL 8 — What It Buys Against Buying DIMMs

Nine sections about what the tier is. This one is about why anybody pays for it, and the answer that holds up is not the capacity one.

The capacity argument is real and weak. A machine that needs more memory than its slots hold can add a CXL device. That is useful and it competes with buying a machine with more slots, which is often cheaper.

There is a third argument that is weaker than pooling and stronger than raw capacity, and it is worth having because it is the one that applies to a single machine. A far tier lets a machine hold a working set that does not fit in its slots at any price. Some machines are slot-limited rather than budget-limited — a dense blade, a particular socket count — and for those the comparison is not against a bigger machine but against not running the workload at all. That is a narrow case and it is a real one, and it is the honest answer when somebody points out that DIMMs are cheap.

The pooling argument is the strong one. A fleet of hosts each provisioned for its peak has, at any moment, a great deal of memory that is bought, powered, and unused. That memory is stranded: it belongs to a host that does not need it and cannot be lent to one that does. Pooled memory is memory bought once instead of twice, and the saving is the stranded fraction rather than the marginal cost of a DIMM.

// RTL 8 - what it buys against buying DIMMs. Memory bought for a host and
// unused by it is memory paid for twice: once to buy and once to buy again
// somewhere it is needed, and reclaiming it is the argument for the tier.
module stranded_memory #(parameter int BUY_MORE_DIMMS = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] hosts, gb_per_host, gb_used_per_host, gb_needed,
  output logic [15:0] stranded_gb, reclaimed_gb, dimm_buy_gb, saving_pct,
  output logic        pooling_pays,
  output logic [7:0]  n_evals, n_stranded,
  output logic        cost_err
);
  logic [15:0] per_host_free, true_stranded;
  logic [31:0] t_q, s_q;
  logic        truly_stranded;
  // Memory a host owns and is not using.
  assign per_host_free = (gb_used_per_host > gb_per_host)
                         ? 16'd0 : (gb_per_host - gb_used_per_host);
  assign t_q = {16'd0, per_host_free} * {16'd0, hosts};
  assign true_stranded = (t_q > 32'hFFFF) ? 16'hFFFF : t_q[15:0];
  assign stranded_gb = true_stranded;
  // Pooling reclaims what is stranded, up to what is needed.
  assign reclaimed_gb = (BUY_MORE_DIMMS != 0) ? 16'd0
                      : ((true_stranded > gb_needed) ? gb_needed : true_stranded);
  assign dimm_buy_gb = gb_needed - reclaimed_gb;
  assign s_q = (gb_needed == 16'd0) ? 32'd0
             : (({16'd0, reclaimed_gb} * 32'd100) / {16'd0, gb_needed});
  assign saving_pct = (s_q > 32'd100) ? 16'd100 : s_q[15:0];
  assign pooling_pays = (reclaimed_gb != 16'd0);
  assign truly_stranded = (true_stranded != 16'd0) && (gb_needed != 16'd0);
  assign cost_err = evaluate && truly_stranded && (reclaimed_gb == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_stranded <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_stranded) n_stranded <= n_stranded + 8'd1;
    end
  end
endmodule

Ten hosts with two hundred and fifty-six gigabytes each, using two hundred, strand five hundred and sixty gigabytes across the fleet. Against a four-hundred-gigabyte demand elsewhere, all of it is met from waste: nothing bought, a hundred percent of the need reclaimed.

The second case is the honest argument against, and it belongs in any answer that claims to understand the trade. A fleet using everything it owns has nothing to pool. No stranded memory, no reclamation, no saving — and the far tier is then a pure capacity play competing against cheaper alternatives. Whether a real fleet looks like the first case or the second is an empirical question about provisioning practice rather than an architectural one.

The empirical answer, where it has been measured, tends to favour pooling — fleets provisioned per-host for per-host peaks strand a substantial fraction, because peaks do not coincide. But the number is a property of a particular fleet's workload mix and provisioning discipline, and an answer that quotes a specific stranded fraction as though it were a constant has overreached. The defensible form is that the saving equals the stranded fraction, that the stranded fraction is measurable, and that a deployment which has not measured it is guessing at the size of its own benefit.

The fourth case draws the other boundary. Stranded memory that nobody needs is waste, not an opportunity. Reclamation requires both a surplus and a demand, and a fleet with a great deal of idle memory and no unmet need has a provisioning problem that pooling does not solve.

13. RTL 9 — The Failure Domain Moved

The last property, and the one that is an availability decision disguised as a hardware choice.

A DIMM that fails takes down the host it is in. That is the failure domain everybody has designed around for decades, and every recovery mechanism in a data centre assumes it.

A pooled CXL memory device that fails takes down every host holding a range on it. The device is no less reliable than a DIMM — it may well be more so — but the consequence of its failure is multiplied by the sharing, and the sharing is the feature.

// RTL 9 - the failure domain moved. A DIMM that fails takes down one host.
// A pooled memory device that fails takes down every host holding a range on
// it, and the blast radius is a property of the sharing rather than of the
// hardware's reliability.
module failure_domain #(parameter int A_FAILURE_IS_A_FAILURE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] hosts_attached, gb_per_host, dimm_gb, redundancy,
  output logic [15:0] blast_gb, blast_hosts, dimm_blast_gb, ratio_x,
  output logic        failure_scoped,
  output logic [7:0]  n_evals, n_widened,
  output logic        domain_err
);
  logic [15:0] true_blast_hosts, true_blast_gb;
  logic [31:0] b_q, r_q;
  logic        truly_widened;
  // A device failure reaches every host holding a range on it, unless the
  // deployment provided a second copy somewhere.
  assign true_blast_hosts = (redundancy != 16'd0) ? 16'd0 : hosts_attached;
  assign b_q = {16'd0, true_blast_hosts} * {16'd0, gb_per_host};
  assign true_blast_gb = (b_q > 32'hFFFF) ? 16'hFFFF : b_q[15:0];
  assign blast_hosts = (A_FAILURE_IS_A_FAILURE != 0) ? 16'd1 : true_blast_hosts;
  assign blast_gb = (A_FAILURE_IS_A_FAILURE != 0) ? dimm_gb : true_blast_gb;
  assign dimm_blast_gb = dimm_gb;
  assign r_q = (dimm_gb == 16'd0) ? 32'd0
             : ({16'd0, true_blast_gb} / {16'd0, dimm_gb});
  assign ratio_x = (r_q > 32'hFFFF) ? 16'hFFFF : r_q[15:0];
  assign failure_scoped = (blast_gb <= dimm_gb);
  assign truly_widened = (true_blast_gb > dimm_gb) && (hosts_attached != 16'd0);
  assign domain_err = evaluate && truly_widened && failure_scoped;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_widened <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_widened) n_widened <= n_widened + 8'd1;
    end
  end
endmodule

Eight hosts each holding a hundred and twenty-eight gigabytes on one device is a thousand and twenty-four gigabytes lost together — eight times the domain of the DIMM it replaced — at the same component reliability.

The third case is the boundary and it is a useful thing to say. A single-host device has a DIMM's failure domain, exactly — which means the failure-domain argument is not an argument against CXL memory, it is an argument against sharing CXL memory without thinking about it. A dedicated expander changes nothing about availability.

The second case is the mitigation. A deployment with a second copy loses nothing, and the model reports zero — which is correct and is also the expensive answer, since the redundancy costs the capacity the pooling was buying. That tension is the real content of the section: pooling and redundancy pull in opposite directions on the same memory, and a deployment has to choose where on that line it sits.

The saturating case is worth a note about the model rather than the architecture. The ratio is computed from the saturated blast figure, so a very large deployment reports five hundred and eleven times a DIMM rather than the true multiple. A saturated ratio is still an enormous ratio, and reporting a wrapped one would have been the dangerous alternative.

There is a mitigation between "no redundancy" and "a full second copy" that is worth knowing about, because the binary framing makes the trade look worse than it is. A deployment can limit the sharing — four hosts per device instead of sixteen — which halves or quarters the blast radius at no capacity cost, trading pooling efficiency rather than capacity. Or it can arrange that the ranges on one device belong to hosts whose failure is already correlated, so that losing them together costs no more than losing them separately.

Both are topology decisions rather than hardware ones, both are free, and both require somebody to have computed the number in section 13's table before the racks are laid out.

14. RTL 10 — The CXL.mem Answer Assembled

Nine models. This one puts them in one place and makes the weak claim visible as what it is: one bit of six.

"It is just more memory" is true about the address space and says nothing about the other five, each of which is a decision somebody has to make.

// RTL 10 - the CXL.mem answer assembled. Everything that must hold before
// device memory is memory in the sense that matters, with "it is just more
// memory" as one of the six rather than the whole claim.
module mem_answer_signoff #(parameter int JUST_MORE_MEMORY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       addressable,      // a load reaches it, no driver involved
  input  logic       latency_bounded,  // it is near enough to still be memory
  input  logic       tier_declared,    // the allocator knows it is slower
  input  logic       bandwidth_sized,  // capacity was not added without bandwidth
  input  logic       media_managed,    // the device owns errors for that range
  input  logic       failure_scoped,   // the blast radius is understood
  output logic       answer_sound,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_sound,
  output logic       false_mem_err
);
  assign fail_mask[0] = ~addressable;
  assign fail_mask[1] = ~latency_bounded;
  assign fail_mask[2] = ~tier_declared;
  assign fail_mask[3] = ~bandwidth_sized;
  assign fail_mask[4] = ~media_managed;
  assign fail_mask[5] = ~failure_scoped;
  // The just-more-memory build stops at the first bit.
  assign answer_sound = (JUST_MORE_MEMORY != 0) ? addressable : (fail_mask == 6'd0);
  assign false_mem_err = evaluate && answer_sound && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_sound <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (answer_sound) n_sound <= n_sound + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When a load reaches the memory and any one of the other five fails, the assembled model reports that the answer is not sound and the just-memory build reports memory.

Across the eight evaluations, the assembled model calls one tier sound and the just-memory build calls six of them memory.

The bit order is by who has to do something about it. Bit 0 is the architecture. Bit 1 is a device and link property. Bit 2 is firmware and the operating system. Bit 3 is a provisioning decision. Bit 4 is the device vendor. Bit 5 is whoever owns availability — and it is last because it is the one that belongs to nobody in particular, which is why it is the one that goes unowned.

Figure 4 — the mask ordered by who has to act. The first decision belongs to the device and the link, the second to firmware, the third to whoever sized the machine, the fourth to the device vendor and the last to whoever owns availability — which is the one that is nobody's job by default and is therefore the one that goes unowned.

15. Quantitative Reasoning

Nine percent of the traffic useful, moving sixty-four byte accesses in four-kilobyte grains. Match the grain to the access and it is a hundred percent — the waste is a property of the mismatch, not of transfers.

Ten times the latency against a fourfold budget is six times past. The architecture is viable at a small multiple and not at a large one, and the multiple is the specification that matters.

Eighty hot pages on a four-hundred-page far tier, placed there by an allocator that was never told, in a system with two hundred hot pages in total.

Four hundred units against fifteen thousand one hundred — a factor of thirty-seven, from filling a tier cold-first instead of with whatever was available.

A hundred per gigabyte becomes fifty-eight, doubling capacity. On a realistic expander — sixty gigabytes on one link — it becomes seven.

Six hundred standing correctable errors after a hundred hours, at ten an hour against a scrub clearing four. A device that never scrubs stands at a thousand.

Seven hundred and sixty-eight lines the device cannot ask back, on a device with two hundred and fifty-six recall slots and a host caching a thousand lines.

Five hundred and sixty gigabytes stranded across ten hosts, which is more than the four hundred somebody else needed. Bought once instead of twice.

Eight times a DIMM's failure domain, at the same component reliability, from sharing one device across eight hosts.

One of eight tiers sound; the just-memory view counts six. The assembled model's summary, and the chapter's.

16. Assertions

The testbenches carry 519 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported twenty-three and one was a real gap: the stranding model's saving percentage, which is derived from the reclaimed capacity and is therefore parameter-dependent — a hundred percent in the measured build and zero in the buy-more-DIMMs one. It is the headline number of section 12 and it was unpinned.

Both builds are asserted on every degenerate case. A workload wanting nothing, a transfer with no grain, a near tier measured at zero, a system with no pages, a tier with no capacity, a device with no hours on it, a host caching nothing, a fleet of one, a device nobody is attached to.

Every clamp is driven past its limit exactly once. A far tier larger than the system, a usage figure above the memory installed, hot pages exceeding the tier, an error count that saturates, a stall that saturates, a stranded figure that saturates, a blast radius that saturates.

Every error output is checked in both directions in every case. Section 5's second case, section 8's third case, section 10's fifth case and section 12's second case exist to assert the quiet half — a grain matching the access, equal page temperatures, a device with no poison signalling, and a fleet with nothing to pool. In all four the measured build must stay silent, and the fourth is the honest argument against the architecture rather than a corner case.

17. Mutation Testing

108 mutations, 108 killed. Fifty-four against each testbench.

One mutation survived the first run, and it was the same shape as 27.3's: a missing stimulus case rather than an equivalent mutant. Flipping the zero-guard in the tier model — making a system with no pages report a saturated misplacement — changed nothing, because the only case with zero total pages also had zero hot pages, and the downstream clamp reduced the saturated value to zero.

The missing case is an inconsistent snapshot: a page report naming hot pages in a system reporting no pages at all, which is two counters read at different moments and is what a real accounting read looks like some of the time. Adding it killed the mutation and gave section 7 the paragraph about refusing to invent a misplacement from inconsistent data.

That is twice in two chapters that the survivor was a zero-guard whose degenerate case was under-specified: the guard was driven, but with every other input also at zero, so a downstream clamp hid the mutation. The lesson is narrower than "drive the degenerate input" and worth recording as its own rule: drive the degenerate input with the other inputs non-zero, because a guard tested in an all-zero configuration is tested against a clamp rather than against itself.

Both instances were found by the mutation campaign and by nothing else. The output-listing step passed, because the outputs were all pinned; the split check passed, because the counters were unbalanced; the error-exclusivity check passed, because the errors were live. The campaign is the only instrument in the set that asks whether an assertion is load-bearing, and these two chapters are the argument for running it even when everything cheaper is already clean.

18. Verification Strategy

Test with the access size the workload actually uses. Section 5's whole argument is about the relationship between access size and transfer grain, and a test using large accesses measures a case where the protocol's advantage does not apply.

Measure the latency ratio, not the latency. Section 6. The absolute number means nothing without the near-memory number beside it, and the ratio is what decides whether the tier behaves like memory.

Check the tier is declared before measuring anything. Section 7. A performance measurement on an undeclared tier measures the allocator's confusion rather than the tier.

Report bandwidth per gigabyte, not bandwidth. Section 9. Total bandwidth goes up when a tier is added; the quantity that predicts a bandwidth-bound workload's behaviour goes down.

Soak for the media. Section 10's accumulation takes hours to appear and is invisible in a functional test. Read the correctable-error and scrub counters over time rather than checking that errors are corrected.

Exercise reclaim, not just access. Section 11. A device that serves memory perfectly and cannot take it back is a device that fails at the operation pooling depends on, and no access-pattern test finds it.

Compute the blast radius at deployment time. Section 13. It is a multiplication and it is nobody's default responsibility.

19. Synthesis and Implementation Reality

The latency budget is set by the link and the device controller together, and the device's share is the part a device team controls. A controller optimised for throughput at the cost of latency has spent the budget the architecture depends on.

The tier description is firmware's job and it is the cheapest item on this list. Section 7's entire failure is prevented by a correct entry in a table, and it is missed because it sits between the firmware team and the operating system team.

Bandwidth per link is fixed and capacity per device is not. A device that maximises capacity per link maximises section 9's dilution, which is the right choice for a capacity-bound deployment and the wrong one otherwise. This is a product decision that looks like an engineering one.

Patrol scrub must be on and its rate must exceed the media's error rate. Section 10. Both halves are configuration, both are frequently left at defaults chosen for a different device.

Recall capacity should be sized against what the host will cache, not against what the device holds. Section 11. The two are different numbers and only one of them is in the device's own specification.

A pooled deployment needs an availability story before it needs a performance one. Section 13. The blast radius is a consequence of the topology, and the topology is chosen for capacity reasons by people not thinking about failures.

20. Silicon Observability

Free, already there. The device's capacity and the link's width and speed. These answer nothing except that the tier exists.

Cheap. Correctable error counts and scrub progress, which are section 10 directly and are the single most valuable counters a memory device exposes.

Cheap, on the host side. The memory tier description as the operating system sees it. Section 7 is a one-line check against what firmware intended.

Moderate. Per-tier access counts and latency distributions, which give sections 6 and 8. A mean latency is nearly useless here for the reason 26.6 section 7 gives.

Moderate. Recall or back-invalidate counters. Section 11 is invisible without them and they are frequently absent.

Expensive. A protocol analyser, which shows the access stream directly and answers sections 5, 6 and 11 by observation. 26.7 is about using one well.

Unobtainable. Which pages the allocator considers hot. Section 7's failure is a property of a decision made inside the operating system, and the practical substitute is to check the declaration rather than to observe the outcome.

21. Debug Lab

A machine with a CXL memory expander runs slower than the same machine without it.

Step 1 — check the tier is declared. Section 7. One read of what the operating system believes, against what firmware intended. This is minutes and it is the most common cause.

Step 2 — measure the latency ratio against attached memory. Section 6. If the ratio is far outside the budget, the tier is not usable as memory whatever else is true.

Step 3 — compute bandwidth per gigabyte before and after. Section 9. If the workload is bandwidth-bound, the machine got worse by design and no tuning fixes it.

Step 4 — check page placement against page temperature. Section 8. Even a declared tier can be filled badly if the policy is wrong.

Step 5 — if the complaint is correctness rather than speed, read the correctable-error and scrub counters. Section 10, then 26.4.

Step 6 — if the complaint is that a range cannot be reassigned, look at recall capacity. Section 11.

Steps 1 through 3 are all reads and together they resolve most occurrences of this symptom. The ordering is by cost, and it happens to also be the order of likelihood, which is unusual and worth exploiting.

It is worth noting what is not on this list. There is no step that involves attaching an instrument, opening a capture or taking a maintenance window, and that is not an accident of this particular symptom — it is a property of the failure modes this chapter owns. Five of the six bits are configuration facts or arithmetic, and configuration facts are read rather than measured. 26.7 prices the alternative at a window, a probe-loading risk and a capture covering one link, which is the right instrument for a signal-integrity problem and a very poor one for a table entry.

22. Design Review

What is the latency ratio to attached memory, and what ratio does the workload tolerate?

Does firmware describe the range as a distinct tier, and has anybody checked what the operating system does with that description?

What is bandwidth per gigabyte before and after, and is the target workload capacity-bound or bandwidth-bound?

Is patrol scrub enabled, and does its rate exceed the expected correctable-error rate?

Can the device signal poison, and has that path been tested?

What is the recall capacity, and how many lines will the host cache?

How many hosts share one device, and what is the blast radius of losing it?

If the argument is pooling, what is the measured stranded fraction across the fleet?

23. How This Appears In Real Engineering

The machine is provisioned, the expander is installed, the capacity appears, and the application is slower than before.

The cause is usually section 7 and the fix is a firmware table entry. What makes it expensive is that the symptom — everything slightly slower — points at nothing in particular, and the investigation usually starts with the device, which is working perfectly. The check that finds it is on the host, and it is a read of a data structure rather than a measurement.

The second shape is a capacity purchase that was really a bandwidth problem. The workload was memory-bandwidth-bound, somebody added capacity, and the bandwidth density fell. Section 9's arithmetic predicts this exactly and nobody ran it, because "more memory" does not sound like a change that could make a machine slower.

The third is the one that damages a fleet rather than a machine. A pooled deployment, sized for capacity, with eight hosts on one device and no redundancy. The device is reliable, the deployment runs for a year, and a single failure takes down eight hosts simultaneously in a way the recovery procedures — written for DIMM failures — do not anticipate. The blast radius was computable on day one and was nobody's job to compute.

There is a version of this that is worse and is becoming more common as pooling deployments grow. The blast radius is computed, correctly, at deployment time — and then the deployment grows. Hosts are added to the pool one at a time, each addition is individually unremarkable, and nothing recomputes the number that was checked once. A topology that was signed off at four hosts per device and has drifted to sixteen has quadrupled its blast radius without any single decision having been made to do so.

That is the general shape of an availability property in a system that grows incrementally, and the mitigation is the same as it always is: make the number a continuously-reported one rather than a design-time calculation. It is a division and it can be on a dashboard.

The pattern is that five of the six bits belong to somebody other than the device vendor, and the device vendor is the party everybody talks to. Firmware, the operating system, provisioning and availability each own one, and each of those owners sees a system that is working.

That diffusion has a practical consequence for how these get resolved, and it is the opposite of 27.3's. There, the difficulty was that no single instrument could see the fault. Here, every fault is plainly visible to whoever owns it — firmware can see the table, provisioning can see the density, availability can see the topology. The difficulty is that nobody is looking, because from each owner's position the system is behaving exactly as configured.

The fix is correspondingly different. 27.3 needs paired counters and an agreed way to read them together. This needs a checklist run once at deployment — six reads, none of them requiring an instrument — and the reason it does not happen is that a checklist for a machine that is working has no obvious owner either.

24. Common Misconceptions

"It's just more memory." In the address space, yes. Five other things have to hold. This is the chapter.

"It's a fast DMA." A load reaches it. That is a different granularity and a different software model. Section 5.

"Latency doesn't matter, it's still memory." It is memory because the latency is a small multiple. Section 6.

"The OS will figure it out." Not if nothing told it the tier is slower. Section 7.

"More memory makes the machine faster." More capacity at lower bandwidth density. For a bandwidth-bound workload that is worse. Section 9.

"The host handles memory errors." The device owns the media behind its range. Section 10.

"CXL.mem is one-directional." The host caches it, and the device must be able to get those lines back. Section 11.

"It's cheaper than DIMMs." The capacity argument is weak. The pooling argument is the strong one, and it depends on a measured stranded fraction. Section 12.

"It's as reliable as a DIMM." Probably. Its failure domain is larger by the number of hosts sharing it. Section 13.

25. Interview Reasoning

"Explain CXL.mem." Device memory in the host's physical address space, reachable by a load with no driver in the path, cacheable by host cores, with the device acting as the memory controller for that range. Type 3 devices are this plus CXL.io. That is the answer.

"Why not just use a fast NVMe device?" Because a load reaches this. The granularity is a cache line rather than a block, and there is no software in the path. For sparse access to a large structure that is the difference between viable and not.

"What decides whether it's usable as memory?" The latency ratio to attached DRAM. A small multiple works; an order of magnitude does not, and the architecture's viability is that quantitative claim.

"You add a CXL expander and the application gets slower. Why?" Most likely the tier was not declared and the allocator put hot pages on it. Second most likely, the workload was bandwidth-bound and the bandwidth density fell. Both are computable before the machine is built.

"Who handles memory errors on a CXL device?" The device. It scrubs, corrects and signals poison for its own range. A range whose error rate exceeds its scrub rate accumulates standing errors, which is the failure to look for.

"Why is pooling a better argument than capacity?" Because stranded memory is memory bought and unused, and pooling buys it once instead of twice. The capacity argument competes with buying a bigger machine; the pooling argument does not.

"What changes about availability?" The failure domain. Eight hosts sharing one device lose their memory together — eight times a DIMM's domain at the same reliability — and recovery procedures written for DIMM failures do not cover it.

26. Exercises

1. A workload issues 500 accesses of 128 bytes each. Compute the useful fraction at transfer grains of 128, 512 and 4,096 bytes. At which grain does the transfer stop being wasteful?

2. Attached memory answers in 90 ns and a far tier in 450 ns. Compute the ratio. If the budget is 3×, how far over is it? What far latency would just meet the budget?

3. A 2,000-page system has 600 hot pages and a 500-page far tier, undeclared. How many hot pages land on it? What fraction of the tier is correctly filled? Now declare the tier.

4. A 300-page far tier is filled hot-first from 400 hot pages, with hot pages accessed 80× and cold pages 2×. Compute both placement costs and the ratio between them.

5. A host has 8 GB of attached memory at 800 units of bandwidth. Compute the density. Now add 120 GB of far memory at 100 units and recompute. By what factor did the density change?

6. A device sees 25 correctable errors per hour and scrubs 30 per hour. After 200 hours, how many stand? Now halve the scrub rate and recompute.

7. Twenty hosts each own 512 GB and use 380 GB. Compute the stranded capacity. If 3 TB is needed elsewhere, how much is met from waste and how much must be bought?

8. Extend the assembled model with a seventh bit for a property this chapter does not cover. Justify its position using the rule that the ordering is by who has to do something about it.

27. Summary

A load reaches it, with no driver in the path. That is the architecture, and it is why the granularity is a cache line rather than a transfer grain.

It is further away, and a small multiple is what makes it memory. Ten times against a fourfold budget is six times past.

The allocator has to be told it is slower, or it places hot pages on it in proportion to capacity, doing exactly what it was built to do.

The win is in the cold pages — a factor of thirty-seven between filling a tier cold-first and filling it with whatever was available.

Capacity is not bandwidth. Doubling the memory halved the bandwidth per gigabyte, which changes what the machine is good at.

The device is the memory controller. Correctable errors accumulate, and a range that does not scrub fast enough is a range waiting for a second error.

The host caches it, so the device must be able to get it back — and reclaim is the operation pooling depends on.

Pooling is the strong argument and capacity is the weak one. Five hundred and sixty gigabytes stranded across ten hosts is memory bought once instead of twice.

The failure domain moved. Eight hosts on one device is eight times a DIMM's blast radius at the same reliability.

Six bits, and "it is just more memory" is one of them. One tier of eight is sound; the just-memory view counts six.