Memory Expanders

Module 16 built the switch. 15.4 built the binding that attaches a device to a host.

This chapter is about what is on the other end: the Type 3 memory expander, which is the dominant CXL product class and the one whose behaviour a host has the least visibility into.

1. The Engineering Problem — The Host Cannot See The Media

Six things make a memory expander different from the DIMM it is standing in for.

Everything the host knows about it, the device said. Type, capacity, media class — three separate claims, and only one of them is usually checked, which is how a host schedules against DRAM timing on a device that is not DRAM. Section 5.

It claims an address window, and an access outside it belongs to somebody else. That is the whole of address decode in one comparison, and it is one of the few things about the device the host can verify. Section 6.

The media is not the link. A host that models only the CXL round trip is modelling the part that does not vary and missing the part that does. Section 8.

Several expanders can sit behind one range. Every address must land on exactly one, and the mapping must be a function. Section 9.

Memory produces errors, and what separates a usable expander from an unusable one is whether a corrected error is distinguished from an uncorrected one — because they need opposite responses. Section 10.

And installed capacity is not usable capacity. Spare regions, device metadata and retired media all come off it, and a host planning against the installed number plans against memory it will never see. Section 11.

This chapter against 15.4, stated precisely. That chapter owns attaching a device to a host. This one owns what the device is and what it can be trusted to say about itself. If a section here could be moved into 15.4 without loss, it is in the wrong chapter.

2. The One-Sentence Model

A memory expander is a device whose media a host cannot inspect, so every property that matters must be reported, checked against something observable, and re-checked against what is delivered — and every defect below is one of reported, checked or delivered missing.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Eight-bit addresses, a 64-unit window, four-way interleave, single- and multi-bit ECC. A real expander has 48-bit addresses, far more capacity and a much richer error model.

What is faithful: the identity-as-a-claim structure, the window comparison and its boundaries, the media-versus-link latency split, the interleave-as-a-function property, the corrected-versus-uncorrected distinction, the usable-versus-installed capacity arithmetic, the three-way bandwidth bound, and reporting audited against delivery.

What is not: every width, every latency, and the error model's simplicity.

Every model is parameterised so the correct behaviour and a specific plausible failure are the same source under a different parameter.

5. RTL 1 — What An Expander Says It Is

module expander_id #(parameter int ASSUME_DRAM = 0) (
  input  logic clk, rst_n,
  input  logic       identify,
  input  logic [1:0] dev_type,      // 1 = Type 3 memory expander
  input  logic [1:0] media_class,   // 0 DRAM, 1 persistent, 2 tiered
  input  logic [7:0] rpt_capacity, observed_capacity,
  output logic       is_expander, media_known,
  output logic [1:0] assumed_media,
  output logic [7:0] usable_capacity,
  output logic       wrong_media_err, overclaim_err,
  output logic [7:0] n_ident, n_rejected, total_capacity
);
  assign is_expander = (dev_type == 2'd1);
  assign media_known = (media_class <= 2'd2);
  // ASSUME_DRAM schedules everything as DRAM, which is right for the common
  // case and wrong for every device that is not.
  assign assumed_media = (ASSUME_DRAM != 0) ? 2'd0 : media_class;
  assign usable_capacity = (rpt_capacity <= observed_capacity)
                         ? rpt_capacity : observed_capacity;
  // Scheduled against a media class the device does not have.
  assign wrong_media_err = identify && is_expander
                           && (assumed_media != media_class);
  assign overclaim_err   = identify && (rpt_capacity > observed_capacity);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_ident <= 8'd0; n_rejected <= 8'd0; total_capacity <= 8'd0;
    end else if (identify) begin
      n_ident <= n_ident + 8'd1;
      // Only a device that is an expander with a known media class counts.
      if (is_expander && media_known) total_capacity <= total_capacity + usable_capacity;
      else                            n_rejected <= n_rejected + 8'd1;
    end
  end
endmodule

Seven devices are presented:

  identity: 7 identified, 2 not expanders, 280GB counted | assume-DRAM build wrong media=1 overclaim=1

Three separate claims, three separate checks:

The capacity rule is the smaller of the two, in both directions, and the testbench drives both: a device reporting more than is observable is capped at what is observable, and one reporting less is taken at its word. Reporting conservatively is honest; the model must not silently promote it.

ASSUME_DRAM is the build that schedules every expander as DRAM. It is right for the common case, and section 20 explains why that makes it dangerous rather than safe.

The media check is gated on is_expander, and the testbench drives a non-expander with a non-default media class to prove it — otherwise a Type 2 accelerator's media field is being audited by a model that has no business reading it.

6. RTL 2 — The Window It Presents

module hdm_window #(parameter int NO_RANGE_CHECK = 0) (
  input  logic clk, rst_n,
  input  logic       access,
  input  logic [7:0] addr,
  input  logic [7:0] window_base, window_size,
  input  logic       window_valid,
  output logic       claim,
  output logic [7:0] local_addr,
  output logic       out_of_window_err, unconfigured_err,
  output logic [7:0] n_claimed, n_passed, n_bad
);
  // Nine bits: base + size on eight-bit operands wraps, and a wrapped top
  // admits everything above the base.
  logic [8:0] window_top;
  logic       in_window;
  assign window_top = {1'b0, window_base} + {1'b0, window_size};
  assign in_window  = window_valid && (addr >= window_base)
                      && ({1'b0, addr} < window_top);
  // NO_RANGE_CHECK claims every access, so the device answers for addresses
  // that belong to somebody else.
  assign claim      = access && (in_window || (NO_RANGE_CHECK != 0));
  assign local_addr = addr - window_base;
  assign out_of_window_err = claim && !in_window;
  assign unconfigured_err  = claim && !window_valid;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_claimed <= 8'd0; n_passed <= 8'd0; n_bad <= 8'd0;
    end else if (access) begin
      if (claim) n_claimed <= n_claimed + 8'd1;
      else       n_passed  <= n_passed + 8'd1;
      if (out_of_window_err) n_bad <= n_bad + 8'd1;
    end
  end
endmodule

  window  : claimed=2 passed=3 bad=0 | no-range-check build claimed out of window=3

Three boundaries, all driven:

• Exactly at the base — inside, translating to local address zero. >=, not >.
• Exactly at the top — outside. <, not <=. The window is half-open, and one off here makes two adjacent devices both claim one address.
• Below the base — outside, and passed on.

And one condition that is not a boundary at all: an unconfigured window claims nothing. A device that answers before it has been told what range it owns is answering for a range it was never given, and NO_RANGE_CHECK does exactly that.

window_top is nine bits. base + size on eight-bit operands wraps, and a wrapped top makes the range test admit everything above the base.

7. Waveform — Eight Accesses Through The Media's ECC

Transcribed from the printed trace. Both builds see one stimulus stream.

Read data_ok against merge_ok. The correct device refuses to return data it could not correct; the merging build returns it every time, and mg_silent names the three accesses where that happened.

8. RTL 3 — The Media Behind The Link

module media_latency #(parameter int LINK_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        access, is_hit,
  input  logic [7:0]  link_ns, media_hit_ns, media_miss_ns,
  output logic [15:0] access_ns,
  output logic [15:0] n_access, n_hits, total_ns, max_ns,
  output logic [7:0]  mean_ns, media_share_pct, hit_pct
);
  logic [15:0] media_ns;
  logic [31:0] w_media, w_hit;
  assign mean_ns = (n_access == 16'd0) ? 8'd0 : (total_ns / n_access);
  assign w_media = ({16'd0, access_ns} - {16'd0, {8'd0, link_ns}}) * 32'd100;
  assign w_hit   = {16'd0, n_hits} * 32'd100;
  assign hit_pct = (n_access == 16'd0) ? 8'd0 : (w_hit / {16'd0, n_access});
  assign media_ns  = is_hit ? {8'd0, media_hit_ns} : {8'd0, media_miss_ns};
  // LINK_ONLY prices every access at the link round trip, which is the same
  // number for every device and every access pattern.
  assign access_ns = (LINK_ONLY != 0) ? {8'd0, link_ns}
                                      : ({8'd0, link_ns} + media_ns);
  // What share of the access the MEDIA is, which is the part that varies.
  assign media_share_pct = (access_ns == 16'd0) ? 8'd0
                         : (w_media / {16'd0, access_ns});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_access <= 16'd0; n_hits <= 16'd0; total_ns <= 16'd0; max_ns <= 16'd0;
    end else if (access) begin
      n_access <= n_access + 16'd1;
      if (is_hit) n_hits <= n_hits + 16'd1;
      total_ns <= total_ns + access_ns;
      if (access_ns > max_ns) max_ns <= access_ns;
    end
  end
endmodule

At an 80ns link, a 40ns media hit and a 200ns media miss:

  media   : hit rate=30% mean=136ns worst=280ns media share=71% | link-only model reports 80ns

The media is 71 percent of a missing access and the link is the rest. A model that prices only the link is modelling the 80ns that is the same for every device and every access, and omitting the part that ranges from 40 to 200.

The worst access is 280ns — the link plus a media miss — and the link-only model's worst case never rises above 80. It reports the same number for a hit-heavy workload and a miss-heavy one, which makes it not wrong about a particular case but incapable of being wrong about any. This is the third chapter in this batch where that shape appears, after 16.2 and 16.4.

9. RTL 4 — Several Expanders Behind One Range

module interleave #(parameter int OVERLAP_WAY = 0) (
  input  logic clk, rst_n,
  input  logic       access,
  input  logic [7:0] addr,
  input  logic [1:0] n_ways,          // 1, 2 or 4 devices
  output logic [1:0] target_dev,
  output logic [3:0] claim_mask,
  output logic       multi_claim_err, no_claim_err,
  output logic [7:0] n_access, n_dev0, n_dev1, n_multi
);
  logic [1:0] sel;
  assign target_dev  = sel;
  assign no_claim_err = access && (claim_mask == 4'd0);
  assign sel = (n_ways == 2'd0) ? 2'd0
             : (n_ways == 2'd1) ? {1'b0, addr[0]}
             :                     addr[1:0];
  assign claim_mask = access
                    ? ((OVERLAP_WAY != 0) ? ((4'd1 << sel) | 4'd1)
                                          :  (4'd1 << sel))
                    : 4'd0;
  // Exactly one device must answer for an address.
  assign multi_claim_err = access &&
    (({2'd0,claim_mask[0]} + {2'd0,claim_mask[1]}
    + {2'd0,claim_mask[2]} + {2'd0,claim_mask[3]}) > 3'd1);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_access <= 8'd0; n_dev0 <= 8'd0; n_dev1 <= 8'd0; n_multi <= 8'd0;
    end else if (access) begin
      n_access <= n_access + 8'd1;
      if (claim_mask[0]) n_dev0 <= n_dev0 + 8'd1;
      if (claim_mask[1]) n_dev1 <= n_dev1 + 8'd1;
      if (multi_claim_err) n_multi <= n_multi + 8'd1;
    end
  end
endmodule

  interleave: dev0=4 dev1=4 multi-claim=0 | overlapping build multi-claim=10

The mapping must be a function: same address, same device, every time. The testbench sweeps all eight addresses against an independent plain-integer oracle, then reads one address twice and asserts the same device answers.

multi_claim_err is > 1 and no_claim_err is == 0, and together they say exactly one. Two devices answering for one address is two devices holding the same data, which is 13.3's two-owner problem arriving through the address decoder.

With no access, no device claims and nothing is unclaimed — driven explicitly, because a claim mask that ignores the access signal makes the device answer for addresses nobody asked about.

10. RTL 5 — The Errors The Media Produces

module media_error #(parameter int MERGE_ERRORS = 0) (...);
  assign corrected   = access && ecc_single && !ecc_multi;
  // MERGE_ERRORS treats an uncorrectable error as a corrected one, so bad
  // data is returned as good.
  assign uncorrected = access && ecc_multi && (MERGE_ERRORS == 0);
  assign data_ok     = access && !uncorrected;
  assign poison      = uncorrected;
  // Data returned as good with an uncorrectable error behind it.
  assign silent_corruption_err = access && ecc_multi && data_ok;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_access <= 8'd0; n_corrected <= 8'd0;
      n_uncorrected <= 8'd0; n_poisoned <= 8'd0;
    end else if (access) begin
      n_access <= n_access + 8'd1;
      if (corrected)   n_corrected   <= n_corrected + 8'd1;
      if (uncorrected) n_uncorrected <= n_uncorrected + 8'd1;
      if (poison)      n_poisoned    <= n_poisoned + 8'd1;
    end
  end
endmodule

  errors  : 1 corrected (25%) 2 uncorrectable 2 poisoned | merging build poisoned=0 silent=1

The two error classes need opposite responses and the model keeps them apart:

• Corrected — the data is good, and the count is a health signal. A rising corrected-error rate is a device on its way to producing uncorrectable ones, visible long before any data is lost.
• Uncorrectable — the data is not good, and it must be poisoned rather than returned. Poison is how the error reaches the host as an error rather than as a number.

The !ecc_multi term in corrected matters: an access with both flags is uncorrectable, not corrected, and the testbench drives that case. Counting it as corrected inflates the health signal with the events it is supposed to predict.

MERGE_ERRORS returns bad data as good and poisons nothing. silent_corruption_err is the only signal that names it — every other counter in that build looks healthier than the correct one's.

11. RTL 6 — Installed Is Not Usable

module usable_capacity #(parameter int REPORT_RAW = 0) (...);
  assign oh          = {1'b0, spare_gb} + {1'b0, metadata_gb} + {1'b0, retired_gb};
  assign actual_gb   = (oh >= {1'b0, installed_gb}) ? 8'd0
                     : (installed_gb - overhead_gb);
  // REPORT_RAW reports the installed capacity, which a host will allocate.
  assign reported_gb = (REPORT_RAW != 0) ? installed_gb : actual_gb;
  assign overreport_err = evaluate && (reported_gb > actual_gb);
  assign overhead_gb  = oh[7:0];
  assign overhead_pct = (installed_gb == 8'd0) ? 8'd0
                      : (weighted / {8'd0, installed_gb});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; worst_overhead <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (overhead_pct > worst_overhead) worst_overhead <= overhead_pct;
    end
  end
endmodule

  capacity: 64GB installed, 8GB overhead, 56GB usable | raw-reporting build reports 64GB, over-report=1

Three separate deductions, and the third grows over the device's life:

The testbench drives retirement rising from 2GB to 10GB and asserts the overhead rises to 25 percent, then drives it past the installed capacity entirely and asserts the usable amount is zero rather than an underflowed maximum.

REPORT_RAW reports 64GB where 56 are usable. Every allocation a host makes against those 8GB will fault, and it will fault at first touch rather than at allocation — which is why overreport_err has to exist on the device rather than being discovered by the workload.

12. RTL 7 — What An Expander Can Actually Deliver

module expander_bw #(parameter int LINK_IS_LIMIT = 0) (...);
  // Little's law: outstanding requests divided by latency bounds throughput.
  assign concurrency_bw = (latency_cycles == 4'd0) ? 8'd255
                        : ({4'd0, outstanding_max} * 8'd16) / {4'd0, latency_cycles};
  assign lo1 = (link_bw <= media_bw)   ? link_bw : media_bw;
  assign lo2 = (lo1 <= concurrency_bw) ? lo1     : concurrency_bw;
  assign achievable_bw = (LINK_IS_LIMIT != 0) ? link_bw : lo2;
  assign binding_term = (lo2 == link_bw)  ? 2'd0
                      : (lo2 == media_bw) ? 2'd1
                      :                     2'd2;
  // Reported more bandwidth than every bound allows.
  assign mismodel_err = sample && (achievable_bw > lo2);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_samples <= 8'd0; worst_bw <= 8'd255;
    end else if (sample) begin
      n_samples <= n_samples + 8'd1;
      if (achievable_bw < worst_bw) worst_bw <= achievable_bw;
    end
  end
endmodule

  bandwidth: link binds=0 media binds=1 concurrency binds=2, worst=4 | link-is-limit build reports 32

Three bounds, and the link is only sometimes the smallest. The testbench drives each of the three into the binding position in turn:

• The link at 32 with a 64 media and 64 concurrency.
• The media at 16 — half the link rate, which is ordinary for anything that is not DRAM.
• Concurrency at 4 — two outstanding requests over eight cycles of latency, which is Little's law and is the bound that surprises people.

That third one is the reason outstanding_max matters as much as the media does. A device with enormous media bandwidth and a shallow request queue delivers the queue depth divided by the latency, and no amount of link or media improves it.

LINK_IS_LIMIT reports 32 — the datasheet number — in every case, including the one where the device delivers 4.

13. RTL 8 — What The Tier Costs

module tier_cost (...);
  assign slowdown_pct = (local_ns == 8'd0) ? 8'd0
                      : ((mean_ns <= local_ns) ? 8'd0
                                               : (w_slow / {24'd0, local_ns}));
  // What the expander added to the host's capacity. A gain can exceed 100%:
  // 300 does not fit in eight bits.
  assign capacity_gain_pct = (cap_local == 16'd0) ? 16'd0
                           : (w_cap / {16'd0, cap_local});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_local <= 16'd0; n_exp <= 16'd0; total_ns <= 16'd0;
      cap_local <= 16'd0; cap_total <= 16'd0;
    end else if (tick) begin
      cap_local <= {8'd0, local_gb};
      cap_total <= {8'd0, local_gb} + {8'd0, expander_gb};
      if (access_local) begin
        n_local  <= n_local + 16'd1;
        total_ns <= total_ns + {8'd0, local_ns};
      end
      if (access_expander) begin
        n_exp    <= n_exp + 16'd1;
        total_ns <= total_ns + {8'd0, expander_ns};
      end
    end
  end
endmodule

  tier    : 7% on the expander, mean=101ns (12% slower) for 300% more capacity

Twelve percent slower for four times the capacity, at a seven percent expander share. The same model driven to a workload that lives on the expander is more than twice as slow — for exactly the same capacity gain, which is the point: the capacity is a property of the hardware and the slowdown is a property of the workload.

capacity_gain_pct is sixteen bits. A capacity gain routinely exceeds 100 percent — that is the entire reason to buy an expander — and an eight-bit percentage reports 300 as 44. The first draft of this model had exactly that, and 44 percent is a plausible enough number that nothing about it looks wrong.

The slowdown guard is reached deliberately: the reference local latency is raised above the measured mean, and the answer is zero rather than a wrapped maximum.

14. RTL 9 — What The Device Must Report

module dev_report (...);
  // Reporting less bandwidth or more latency than delivered is conservative
  // and fine. The other direction is a claim the device cannot keep.
  assign bw_gap  = (rpt_bw <= obs_bw) ? 8'd0 : (rpt_bw - obs_bw);
  assign lat_gap = (rpt_latency >= obs_latency) ? 8'd0
                 : (obs_latency - rpt_latency);
  assign report_honest = bw_ok && latency_ok;
  assign bw_ok         = (bw_gap == 8'd0);
  assign latency_ok    = (lat_gap == 8'd0);
  assign misreport_err = check && !report_honest;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_checked <= 8'd0; n_misreported <= 8'd0;
      worst_bw_gap <= 8'd0; worst_lat_gap <= 8'd0;
    end else if (check) begin
      n_checked <= n_checked + 8'd1;
      if (!report_honest) n_misreported <= n_misreported + 8'd1;
      if (bw_gap  > worst_bw_gap)  worst_bw_gap  <= bw_gap;
      if (lat_gap > worst_lat_gap) worst_lat_gap <= lat_gap;
    end
  end
endmodule

  report  : 2 of 4 dishonest, worst bw overclaim=8 worst latency underclaim=50ns

Honesty here is asymmetric, and that is what makes it a real check rather than an equality test. Reporting less bandwidth than delivered is conservative and fine; reporting more is a claim the device cannot keep. Reporting more latency than delivered is conservative; reporting less is not.

Both directions are driven, and the conservative case is asserted honest — a symmetric check would flag a device that under-promises and over-delivers, which is the behaviour you want.

Both halves are driven alone: a bandwidth overclaim with an honest latency, and a latency underclaim with honest bandwidth. A design checking one reports the other as honest.

15. RTL 10 — The Expander Assembled

module expander_top #(parameter int USE_ANYWAY = 0) (...);
  assign refused_by = {~report_ok, ~media_ok, ~window_ok, ~identified};
  assign usable = request && ((refused_by == 4'd0) || (USE_ANYWAY != 0));
  // Used against a claim nothing checked.
  assign unchecked_use_err = usable && (refused_by != 4'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_usable <= 8'd0;
      n_refused <= 8'd0; n_unchecked <= 8'd0;
    end else if (request) begin
      n_requests <= n_requests + 8'd1;
      if (usable) n_usable  <= n_usable + 8'd1;
      else        n_refused <= n_refused + 8'd1;
      if (unchecked_use_err) n_unchecked <= n_unchecked + 8'd1;
    end
  end
endmodule

  assembled: 1 of 5 usable, mask for report=8 | use-anyway build used 5 on 4 unchecked claims

Four checks, and the order is the order the chapter built them: identified, window configured, media class known, report honest. Each refuses on its own and the mask names which.

unchecked_use_err is the invariant: nothing is used on the strength of a claim nothing checked. It cannot fire in the correct build, and USE_ANYWAY uses all five devices, four of them on claims that failed.

The check with no natural advocate is the fourth. Identification is visible at enumeration; the window is visible at configuration; the media class is visible at scheduling. Nothing about normal operation compares what the device reported against what it delivered — which is why section 14 is a model rather than a paragraph.

16. Quantitative Reasoning

Three worth a sentence.

Media 71 percent, link 29. The link is the part everyone models and the part that does not vary. The media is the part that ranges from 40ns to 200ns and it is most of the access.

Achievable 4 against a 32 link rate. Two outstanding requests over eight cycles of latency. The datasheet number was eight times what the device delivered, and neither the link nor the media was the reason.

12 percent slower for 300 percent more capacity. That is the expander trade stated honestly — and the same hardware with a workload that lives on it is more than twice as slow, for exactly the same capacity.

17. Assertions

Every property is an immediate check written as cond !== 1'b1, sampled after a settle.

18. Mutation Testing

91 mutations, one at a time, each required to make the baseline print RESULT: FAIL.

91 of 91 were killed.

The first run killed 83 and left 8 survivors:

The most instructive were the width bugs found before mutation testing began, because they show the failure mode that mutation testing cannot catch: three separate eight-bit truncations, each producing a plausible number.

8'd270 became 14, so an expander access looked faster than the link. 8'd300 became 44, so a device reporting 300ns of latency looked like it was reporting 44 and therefore honest. And capacity_gain_pct declared eight bits reported a 300 percent gain as 44 percent — a modest, believable improvement, on a model whose entire purpose is to show that the gain is large.

A representative sample:

19. Verification Strategy

Parameterised twin builds. ASSUME_DRAM, NO_RANGE_CHECK, LINK_ONLY, OVERLAP_WAY, MERGE_ERRORS, REPORT_RAW, LINK_IS_LIMIT, USE_ANYWAY. Every comparison in section 16 is one source under one parameter, driven by one stimulus stream.

Independent oracles. Identity, the address window, the interleave target and the three-way bandwidth bound are each checked against a plain-integer function that shares no structure with the design.

Check the width before the logic. Three eight-bit truncations were found by reading printed transcripts, not by any assertion — because the assertions used the same truncated literals. Any value that can exceed 255, including a percentage that is a gain rather than a share, needs its width decided rather than defaulted.

Both directions of an asymmetric rule. A device reporting less capacity than observable, and more. A report claiming more bandwidth than delivered, and less. A conservative claim must be asserted honest, or the check is an equality test wearing a rule's clothes.

Drive the ungated case. A non-expander with a non-default media class. An access signal low with a valid address. A clean access with the silent-corruption monitor watching. Each of those terms reads as obviously necessary and none had been driven false.

Reach every guard. Zero latency in the concurrency bound. A measured mean below the reference latency. Retirement past the installed capacity. Each guard exists for a case, and the case has to be driven.

Delta discipline. Every combinational sample follows a settle.

20. Synthesis and Implementation Reality

The window decode is small and it is on every access. A base, a size, two comparators and a subtractor. It is the cheapest thing in the device and the only part of it a host can verify.

The media controller is the device. Everything else in this chapter is bookkeeping around a memory controller that is doing scheduling, refresh or wear management depending on the media. Section 8's 40-to-200ns range is that controller, and it is why a link-only model is modelling the wrong component.

ASSUME_DRAM is not a hypothetical. A host's memory scheduler has DRAM timing assumptions built into it — bank conflicts, row buffer locality, refresh windows. Presenting persistent or tiered media through the same interface and scheduling it identically is the default behaviour, not a bug someone introduced, and it is why the media class has to be reported and acted on.

Poison is a wire, and it has to reach the host. Section 10's poison output is a real signal in a real memory path, and the alternative — returning bad data — is not a failure mode the device can detect afterwards. This is the one place in the chapter where the correct behaviour costs a signal the merging build does not have.

The spare region is real capacity that is never usable. Section 11's overhead is media that exists, is powered, is paid for, and cannot be allocated. Reporting installed rather than usable is the easy implementation, and it converts a capacity-planning number into a fault the workload discovers.

Concurrency is the bandwidth bound teams miss. Link rate and media rate are both on datasheets. Outstanding-request depth divided by latency is not, and section 12 drives it into the binding position because it frequently is.

The divisions are firmware. Media share, hit rate, overhead percentage, capacity gain and the report gaps are computed by host software from raw counters the device exposes.

21. Silicon Observability

Three to alarm on.

silent_corruption_err non-zero at all means the device returned data it could not correct, as though it were correct. Nothing downstream can detect it, and every other counter on that device looks healthy.

ce_pct rising across a device's life is the one predictive signal here. Corrected errors are, by definition, not yet a problem — and a rising rate is a device on its way to producing uncorrectable ones, visible well before any data is lost.

worst_bw_gap or worst_lat_gap non-zero means the device claimed something it does not deliver. Every placement and scheduling decision the host has made since is based on it.

22. Debug Lab

22.1 A host allocated memory and touching it faults

Symptom. A host allocates capacity on an expander successfully. Accesses beyond some offset fault.

The reading. overreport_err and the gap between reported_gb and actual_gb.

The diagnosis. The device reported installed capacity rather than usable. The offset where faults begin is the usable capacity, and the difference is the spare region, the metadata and whatever media has been retired.

Why it appears late. The allocation succeeds, and the fault arrives when a workload first touches the top of the range — which may be days. overreport_err fires at evaluation time, on the device.

22.2 An expander is slower than the model and the link is idle

Symptom. Measured latency to an expander exceeds the projection. The CXL link shows plenty of headroom.

The reading. media_share_pct and hit_pct.

The diagnosis. The projection modelled the link. The media is 71 percent of a missing access in section 8, and it varies by a factor of five between a hit and a miss — neither of which the link model contains.

The sub-case that is not this. If hit_pct matches the projection's assumption and the latency is still high, the media itself is slower than reported — check worst_lat_gap on the device's own report.

22.3 Data came back wrong and nothing reported an error

Symptom. A host read data that was incorrect. No error was signalled anywhere.

The reading. silent_corruption_err and n_poisoned on the expander.

The diagnosis. The device merged its error classes and returned an uncorrectable error as good data. n_poisoned will be zero and n_uncorrected will be zero too, because the device counted them as corrected.

The confirming tell. n_corrected will be unusually high — it contains the uncorrectable errors as well. A corrected-error rate that rose sharply with no uncorrectable errors at all is the signature, because in a healthy device the two rise together.

22.4 Bandwidth is far below the link rate and the media is fast

Symptom. An expander delivers a fraction of its link rate. The media's rated bandwidth is well above what is measured.

The reading. binding_term, and outstanding_max against the measured latency.

The third row is the one that surprises, and it is the one no datasheet number predicts: two outstanding requests over eight cycles of latency delivers a fraction of both the link and the media, and improving either changes nothing.

23. Design Review

1. Which of the device's claims are checked against something observable? Capacity usually is. Media class and latency usually are not.

2. Is the address window half-open, and is the top boundary driven in test? One off makes two adjacent devices claim the same address.

3. Does the device claim anything before a window is configured? It should claim nothing at all.

4. Does the latency model include the media, or only the link? A link-only model reports the same number for every device and every access pattern.

5. In an interleaved set, does exactly one device claim each address? Not at least one, and not at most one.

6. Are corrected and uncorrectable errors counted separately, and is the uncorrectable case poisoned? Merging them returns bad data as good and inflates the health signal with the events it is meant to predict.

7. Does the device report installed or usable capacity? And does the usable figure fall as media is retired?

8. Which of the three bandwidth bounds is binding, and has anyone computed the concurrency one? Queue depth over latency is not on any datasheet.

9. Is the device's report audited against what it delivers, in both directions? Under-promising must pass; over-promising must not.

10. Can the host use the device on a claim that failed a check? That is the invariant, and it needs a state rather than a procedure.

24. How This Appears In Real Engineering

The media class is reported and ignored. The host's memory scheduler has DRAM assumptions built in, presenting anything through the same interface is the point of the standard, and scheduling it identically is the default. The failure is a performance one and it is attributed to the device.

Capacity is reported raw because usable is harder. The spare region and the metadata are known at manufacture; retirement is not, and a device that reports a falling capacity over its life is an integration problem nobody wants. Reporting installed converts it into a fault the workload finds.

Link-only latency models are used because the media number is not available. The link round trip is a specification value; the media latency is a device characteristic that varies by product, by access pattern and by wear. The model that exists gets used.

Error merging is a simplification that ships. Two error classes with two response paths is more work than one, and the corrected path is the common one. The result is a device that returns bad data and reports a healthy corrected-error rate.

The concurrency bound is discovered in silicon. Link and media bandwidth are procurement numbers. Outstanding-request depth is a microarchitectural detail, and the product of it and the latency is what the workload actually sees.

Reports are audited against delivery only after a discrepancy. Nothing in normal operation compares them, which is why section 14 has to be a model with counters rather than a review checklist.

25. Common Misconceptions

"The device reported its capacity, so that is its capacity." It reported a number. Section 5 caps it at what is observable in one direction and takes it at its word in the other, and section 11 subtracts the overhead the device knows about and the host does not.

"Installed capacity is usable capacity." 64GB installed, 8GB of overhead, 56 usable — rising to 16GB of overhead as media retires. A host planning against 64 plans against memory it will never see.

"CXL memory is slower because of the link." The link is 80ns and does not vary. The media is 40 to 200ns and is 71 percent of a missing access. The link is the part everyone models and the part that matters least.

"An expander is just slower DRAM." Only if its media class is DRAM. Scheduling persistent or tiered media against DRAM timing is section 5's ASSUME_DRAM, and it is the default behaviour rather than a mistake someone made.

"A corrected error is not worth counting." It is the only predictive signal here. A rising corrected-error rate is a device on its way to producing uncorrectable ones, visible before data is lost.

"An uncorrectable error is just a corrected one that failed." They need opposite responses: one returns data, the other must poison it. Merging them returns bad data as good, and no counter on that device shows anything wrong.

"The device's bandwidth is its link rate." It is the smallest of the link, the media, and the outstanding-request depth divided by the latency. Section 12's device delivered 4 against a link rate of 32, and neither the link nor the media was why.

"Under-reporting is as dishonest as over-reporting." No. A device that delivers more than it promised has kept its promise. A check that flags both is an equality test, and it penalises exactly the behaviour you want.

26. Interview Reasoning

Q1. What can a host actually verify about a memory expander? The address window, by accessing it. Almost nothing else — type, capacity, media class and latency are all reported, which is why each needs checking against something observable.

Q2. A device reports 128GB and 64GB is observable. What do you record? 64 as usable, 128 as claimed, and an overclaim. Both numbers, because the difference is the thing that will fault.

Q3. A device reports 40GB and 64GB is observable. What do you record? 40. Reporting less than is there is conservative, and promoting it silently is the model deciding it knows better than the device.

Q4. Why does the media class matter if the interface is the same? Because the host's memory scheduler has DRAM timing assumptions in it. Presenting persistent media through the same interface is the point of the standard; scheduling it identically is a performance failure attributed to the device.

Q5. Is the address window half-open or closed? Half-open: >= base and < base + size. Closed at the top makes two adjacent devices claim the same address, and both are individually correct.

Q6. What should a device claim before its window is configured? Nothing. A device answering for a range it was never given is answering for somebody else's.

Q7. Why is a link-only latency model useless rather than approximate? It reports the same number for every device and every access pattern. It is not wrong about a particular workload — it cannot be wrong about any, which is worse.

Q8. What fraction of an expander access is the media? 71 percent on a miss, in section 8's numbers. The link is 80ns and constant; the media ranges from 40 to 200 and is the part that varies.

Q9. Four expanders behind one address range. What must be true? Every address lands on exactly one. Not at least one, which admits two devices holding the same data, and not at most one, which admits an address nobody answers for.

Q10. And what makes it a function? The same address always the same device. Section 9 reads one address twice and asserts the same device answers, because an interleave that depends on anything but the address is not a mapping.

Q11. What separates a corrected error from an uncorrectable one? Whether the data is good. A corrected error returns data and is a health signal; an uncorrectable one must poison the data rather than return it.

Q12. What happens if a device merges them? It returns bad data as good, poisons nothing, and reports a healthy corrected-error rate that contains the uncorrectable errors. Nothing downstream can detect any of it.

Q13. What is the tell for that from outside the device? A corrected-error rate that rose sharply with no uncorrectable errors at all. In a healthy device the two rise together.

Q14. An access has both ECC flags set. Corrected or uncorrectable? Uncorrectable. Counting it as corrected inflates the health signal with exactly the events it is meant to predict.

Q15. Why is installed capacity not usable capacity? Spare, metadata and retired media all come off it. And the third one grows over the device's life, so the usable figure falls.

Q16. A device is retired past its installed capacity. What does it report? Zero usable. Not an underflowed maximum, which is what an unguarded subtraction produces on exactly the device where it matters.

Q17. What are the three bounds on an expander's bandwidth? The link, the media, and the outstanding-request depth divided by the latency. The smallest binds, and it is frequently the third.

Q18. Why is the concurrency bound the one teams miss? Link and media bandwidth are procurement numbers on datasheets. Queue depth is a microarchitectural detail, and its product with the latency is what the workload sees.

Q19. Twelve percent slower for 300 percent more capacity. Is that a good trade? It depends entirely on the access distribution. The same hardware with a workload that lives on the expander is more than twice as slow, for exactly the same capacity gain.

Q20. Why is capacity_gain_pct sixteen bits when every other percentage in this batch is eight? Because a gain is not a share. Four times the capacity is 300 percent, and an eight-bit output reports it as 44 — a plausible, modest-looking number, on a model whose whole purpose is to show the gain is large.

Q21. How would you have caught that? By reading the printed transcript. The assertion used the same truncated literal, so it agreed with the design. That is the failure mode a mutation suite cannot find.

Q22. 8'd270 — what is its value? 14. And a latency of 14ns makes an expander look faster than the link it sits behind, which is the sort of wrong answer that gets believed.

Q23. Is a device that under-reports its bandwidth dishonest? No. It delivers more than it promised, which is what you want. A symmetric equality check flags exactly the behaviour worth encouraging.

Q24. What are the two halves of an honest report? Bandwidth not overclaimed, and latency not underclaimed. Each is driven alone in test, because checking one reports the other as honest.

Q25. What is the expander's assembled invariant? Nothing is used on the strength of a claim nothing checked. It cannot fire in a correct design, which is what makes it an invariant.

Q26. Of the four assembled checks, which has no natural advocate? The report audit. Identification happens at enumeration, the window at configuration, the media class at scheduling. Nothing in normal operation compares what the device said against what it delivers.

Q27. A host allocates capacity and faults on touch. First reading? overreport_err, and the gap between reported and actual. The offset where faults begin is the usable capacity.

Q28. An expander is slow and the link is idle. Where do you look? media_share_pct and hit_pct. The projection almost certainly modelled the link, which is the constant part.

Q29. Which failure in this chapter would you page someone about? Silent corruption. Every other failure reports itself somewhere; that one returns bad data as good with a healthier-looking set of counters than a correct device.

Q30. If you could require one thing of a memory expander, what? That it reports usable capacity and that the figure falls as media retires. It is the number every capacity plan is built on, and reporting installed instead converts a planning input into a fault a workload discovers.

27. Exercises

1. Add a fourth media class to expander_id that the host does not recognise but the device claims is DRAM-compatible. Decide whether to schedule it and justify the answer from what can be verified.

2. Extend hdm_window to two disjoint windows on one device. Show which assertions still hold and what multi_claim_err must become.

3. Give media_latency a queue so a miss delays subsequent hits. Show that mean_ns no longer follows the hit rate linearly, and identify the mutation that becomes reachable.

4. Make interleave granular at more than one address unit. Derive the address bits the select must use and show the oracle changes with it.

5. Add a scrubbing engine to media_error that converts some corrected errors into retirements. Show the effect on ce_pct and on section 11's usable capacity.

6. Combine usable_capacity with 15.4's binding: a device whose usable capacity falls below what a host was already allocated. Decide what happens and justify it.

7. Extend expander_bw so the outstanding-request depth is per-host on a pooled device. Show which host's depth binds and what that means for 12.1's fairness.

8. Take the 91-mutation suite and change capacity_gain_pct back to eight bits. Confirm nothing fails, then find every other percentage in Modules 15, 16 and 17 that can exceed 100.

28. Summary

A memory expander is a device whose media a host cannot inspect, so every property that matters must be reported, checked against something observable, and re-checked against what is delivered.

• Reported: seven devices, two refused for two different reasons, and 280GB counted — verified rather than claimed. The assume-DRAM build scheduled persistent media on DRAM timing.
• Checked: the window is the one thing a host can verify. Half-open, driven at both boundaries, and claiming nothing before it is configured — against a build that claimed three addresses including one before any window existed.
• Delivered: the media is 71 percent of a missing access and ranges from 40ns to 200ns; the link is 80ns and constant. The link-only model reported 80ns for every workload it was given.
• And what the host cannot see at all: 8GB of a 64GB device is overhead before any media retires, and 16GB after. An expander delivering 4 against a link rate of 32, bounded by neither the link nor the media but by two outstanding requests over eight cycles.

The trade, stated honestly: 12 percent slower for 300 percent more capacity — and more than twice as slow for the same capacity, if the workload lives there.

91 mutations, 91 killed. But the three findings worth carrying were found before the mutation suite ran, by reading printed numbers: 8'd270 is 14, 8'd300 is 44, and a 300 percent capacity gain in an eight-bit output is 44 percent. Each produced a plausible answer, and the assertions agreed because they used the same truncated literals. A percentage that can exceed 100 is a width decision, not a default.

17.2 — Persistent-Memory Devices takes the expander as built and asks what changes when the media does not forget.