Memory Expansion Cards

29.1 asked which tier. This chapter is about the device that makes one of those tiers exist, and it is the case where the weak definition is not merely incomplete — it is a description of the packaging.

It is a memory card. True, and it names the shape: a rectangle that goes in a slot. What it does not name is the link in front of the media, the controller between them, the latency that link adds, where the operating system will put the pages, or what happens to the other cards when one of them fails.

1. The Engineering Problem — A Card Is A Device, Not A Slot

The shape is not the device. Six properties that matter with two stated leaves four unstated, on a description where seventy percent of the claims are about the form factor. Section 5.

The link adds a fixed amount to every access. Ninety nanoseconds local against a hundred and seventy of link plus eighty of media is a hundred and sixty added — nearly three times local latency. Section 6.

The bandwidth comes from the link, not the media. A hundred gigabytes per second of DRAM behind a sixty-four gigabyte link delivers sixty-four: thirty-six can never leave the card. Section 7.

What decides the card's value is where the pages land. Three hundred hot pages with two hundred placed near leaves a hundred hot pages on the far side, at four thousand units. Section 8.

Interleaving widens the failure domain in the same move that buys bandwidth. A four-way set of two hundred and fifty-six gigabyte cards makes one card's loss a one-thousand-and-twenty-four gigabyte event. Section 9.

And the capacity is not free capacity. Six cards at seventy-five watts against a four-hundred-watt budget is fifty watts over, with two slots left. Section 11.

Why this chapter sits after the training-cluster one. 29.1 established that a tier has to be named. This chapter is what naming one actually commits you to — a device with its own latency, its own bandwidth ceiling, its own failure domain and its own line in the chassis power budget, none of which is visible in the phrase that usually introduces it.

2. The One-Sentence Model

A memory-expansion-card case study is sound when the card is named, when the latency tier it creates is stated, when the bandwidth is sourced to the link rather than to the media, when the page placement is stated, when the failure domain is stated, and when the slot and power cost is counted — and "it is a memory card" is bit 0.

3. What This Chapter Owns

Some vocabulary, because the parts of a card are what the chapter is about and the word "card" hides all of them.

A Type 3 device is a memory expander — it presents capacity to the host and participates in no coherency of its own beyond what the host directs. That is the device class this chapter is about, and 27.6 is where the choice between the classes is made.

The link is the connection in front of the device, and its width and rate set the bandwidth ceiling regardless of what media is fitted behind it. Section 7 is that sentence as a number.

The controller is the logic between the link and the media. It is where the protocol becomes a memory access, and it contributes latency that neither the link nor the DRAM accounts for.

A NUMA node is how the operating system sees the result — a block of memory with a distance attached. The distance is advisory and the placement is a decision, which is section 8 and the reason a card's value is a software property as much as a hardware one.

4. Teaching-Model Boundary

This is a case-study chapter and the boundary is the same one 29.1 sets.

Every model computes a property of a decision, not a claim about any product. No figure here is a measurement of any real card, controller or chassis. Ninety nanoseconds of local latency, a seventy-five watt card, a two-hundred-and-fifty-six gigabyte module — all are teaching figures chosen to make one relationship visible, and none is a specification.

Where a real technology is named it is for a publicly established fact only. That CXL defines a memory-expander device class. That such a device is reached across a link and presents capacity to a host. That operating systems represent distant memory as NUMA nodes. Nothing here attributes a latency, a price or a design choice to any named vendor or part.

Each model is built twice from one source. A parameter selects between the measured build, which counts what the deployment rests on, and the a-card build, which treats the form factor as the description. Every section's headline number is the gap between them.

5. RTL 1 — The Shape Is Not The Device

Start with the description itself, because everything below is a property the description skipped.

"Memory card" names a form factor. It says the thing is a card and the thing is memory, both of which are visible from across the room. The properties that decide whether it belongs in a design are all invisible from there — the latency the link adds, the bandwidth ceiling, the failure domain, the power, the placement behaviour.

That is not a criticism of shorthand. It is an observation that the shorthand is usually where the description stops.

// RTL 1 - "it is a memory card" describes the form factor and not the device.
// A card is a link-attached device with its own controller; a slot is a fixed
// electrical relationship. The properties that matter belong to the second
// description, and the first one names none of them.
module card_is_not_a_slot #(parameter int A_CARD_IS_A_SLOT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] properties_that_matter, properties_stated, form_claims, total_claims,
  output logic [15:0] unstated, stated_ok, form_pct, described_pct,
  output logic        device_described,
  output logic [7:0]  n_evals, n_unstated,
  output logic        desc_err
);
  logic [31:0] f_q, d_q;
  logic [15:0] true_unstated;
  logic        truly_unstated;
  assign stated_ok = (properties_stated > properties_that_matter)
                   ? properties_that_matter : properties_stated;
  assign true_unstated = properties_that_matter - stated_ok;
  assign unstated = (A_CARD_IS_A_SLOT != 0) ? 16'd0 : true_unstated;
  // How much of the description is about the shape rather than the behaviour.
  assign f_q = (total_claims == 16'd0) ? 32'd0
             : (({16'd0, form_claims} * 32'd100) / {16'd0, total_claims});
  assign form_pct = (f_q > 32'd100) ? 16'd100 : f_q[15:0];
  assign d_q = (properties_that_matter == 16'd0) ? 32'd100
             : (({16'd0, stated_ok} * 32'd100) / {16'd0, properties_that_matter});
  assign described_pct = (A_CARD_IS_A_SLOT != 0) ? 16'd100 : d_q[15:0];
  assign device_described = (unstated == 16'd0) && (properties_that_matter != 16'd0);
  assign truly_unstated = (true_unstated != 16'd0);
  assign desc_err = evaluate && truly_unstated && device_described;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_unstated <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_unstated) n_unstated <= n_unstated + 8'd1;
    end
  end
endmodule

Six properties that matter with two stated, in a description where seven of ten claims are about the form, leaves four unstated and a third of the device described.

Figure 1 — the description that is entirely correct and decides nothing. The upper path's claims are all true and all verifiable by looking: it is a card, it has memory on it, it goes in a slot. The lower path asks for the six properties an architect would need, and four of them are not there — and note that the description could grow indefinitely along the top without ever producing one of them.

The second case is the description done properly. Every property stated and nothing about the form reports a fully described device, both builds agree, and neither alarms — which is what a good datasheet summary reads like and it takes six facts.

The third case is the clamp on the naming side. More properties stated than matter clamps to the properties that matter, because a description that covers extra ground has still covered all of this one.

The fourth case is the clamp on the claim side and it is the shape of marketing copy. More claims about the form than there are claims saturates at a hundred percent — a description wholly about the packaging — while three of four properties remain unstated.

The degenerate case bounds it: nothing enumerated at all reports nothing unstated and nothing described, which is a device nobody has written about rather than one written about badly.

It is worth naming the six properties, because "properties that matter" is otherwise as empty as the phrase it replaces. They are the five the rest of this chapter builds — the latency tier, the bandwidth ceiling, the placement behaviour, the failure domain, the slot and power draw — plus the capacity, which is the one everybody does state. Five of six are invisible from the outside of the box, and that ratio is the section's whole argument.

The form-factor description is durable for a specific reason: every claim in it is independently verifiable and none of it requires the device to be powered on. Somebody can confirm that it is a card, that it has DRAM on it, that it fits the slot, and that it is made by a real company, without measuring anything. A description made entirely of unfalsifiable-because-obvious claims feels thorough — there is a lot of it, all of it is true, and none of it is load-bearing.

The information is not secret, which is the frustrating part. Every one of the six properties appears in a datasheet or is one measurement away. The failure is a transmission failure: the datasheet has them, the design meeting does not, and the description that travelled between the two kept the part anybody could have guessed.

6. RTL 2 — The Link Adds A Fixed Amount To Every Access

The second thing, and the first property the description usually omits.

A card is reached across a link, and the link costs time on every access. So does the controller in front of the media. The sum is the latency tier the card creates, and it does not depend on which DRAM was fitted — a faster module behind the same link moves the number very little.

That is the property that decides which pages may live there, which is section 8.

// RTL 2 - the latency tier. A card is reached across a link, and the link adds
// a fixed amount to every access. That addition is the property that decides
// which pages may live there, and it does not depend on the DRAM fitted.
module latency_tier #(parameter int SAME_AS_LOCAL = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] local_ns, link_ns, media_ns, tolerance_ns,
  output logic [15:0] card_ns, added_ns, ratio_pct, over_tolerance,
  output logic        within_tolerance,
  output logic [7:0]  n_evals, n_over,
  output logic        lat_err
);
  logic [31:0] c_q, r_q;
  logic [15:0] true_card, true_added, true_over;
  logic        truly_over;
  assign c_q = {16'd0, link_ns} + {16'd0, media_ns};
  assign true_card = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign card_ns = (SAME_AS_LOCAL != 0) ? local_ns : true_card;
  assign true_added = (true_card > local_ns) ? (true_card - local_ns) : 16'd0;
  assign added_ns = (SAME_AS_LOCAL != 0) ? 16'd0 : true_added;
  assign r_q = (local_ns == 16'd0) ? 32'd999
             : (({16'd0, true_card} * 32'd100) / {16'd0, local_ns});
  assign ratio_pct = (r_q > 32'd999) ? 16'd999 : r_q[15:0];
  assign true_over = (true_added > tolerance_ns) ? (true_added - tolerance_ns) : 16'd0;
  assign over_tolerance = (SAME_AS_LOCAL != 0) ? 16'd0 : true_over;
  assign within_tolerance = (over_tolerance == 16'd0);
  assign truly_over = (true_over != 16'd0);
  assign lat_err = evaluate && truly_over && within_tolerance;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_over <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_over) n_over <= n_over + 8'd1;
    end
  end
endmodule

Ninety nanoseconds local against a hundred and seventy of link plus eighty of media is two hundred and fifty to reach the card, a hundred and sixty added, and a hundred nanoseconds beyond a sixty-nanosecond tolerance.

Figure 2 — the tier the card creates, which is the thing being bought whether or not anybody said so. Both paths agree there is memory and agree how much; they disagree about how far away it is. The ratio is the number to carry, because it converts directly into which pages can tolerate living there.

The second case is the one worth keeping so the section is not an argument against cards. A card closer than local memory reports nothing added at all — which happens when the local figure being compared against is itself a distant socket, and it is a legitimate configuration that the model must not penalise.

The boundary cases are the pair that decides an acceptance criterion. Added latency exactly at tolerance is within it, and one nanosecond past is not — a deployment sitting exactly on its tolerance has no margin for the variance a real access pattern has.

The fifth case is the honest limit. No local latency measured at all counts the whole card figure as added and saturates the ratio, which is the state of a comparison nobody has baselined.

The degenerate case bounds it: nothing measured reports nothing added and declines to call it a breach.

The model sums three terms and that is deliberate, because the two-term version is where the reasoning usually goes wrong. The link is one term and the media is another, and the controller between them is the third — the logic that turns a protocol transaction into a memory access, arbitrates it, and turns the answer back. It is a real design with real latency and it belongs to neither of the other two, so a calculation that adds "link plus DRAM" understates the result by an amount nobody has budgeted for.

That is also why fitting faster media moves the total so little. If the media term is the smallest of the three, halving it changes the sum by a fraction of the smallest part — which is section 7's argument arriving on the latency axis rather than the bandwidth one. The two sections are the same observation about where a card's behaviour actually comes from: the parts in front of the memory, not the memory.

The baseline deserves as much care as the measurement, and the fifth case exists to say so. "Local latency" is not one number — a nearby socket and a distant socket in the same machine differ, and comparing a card against the wrong one produces a ratio that is either flattering or damning without being right. The second case is the honest version of that: a card compared against a genuinely distant local access can legitimately report nothing added, and a model that refused to report it would be an advocacy tool rather than a measurement.

7. RTL 3 — The Bandwidth Comes From The Link, Not The Media

The third thing, and the sizing error that costs real money.

The link in front of a card is a fixed number of lanes at a fixed rate, and that product is the ceiling. Media fitted behind it that can go faster does not go faster; it waits. Bandwidth bought above the link is bandwidth that can never leave the card, and it is paid for in full.

// RTL 3 - the bandwidth comes from the link, not from the media. Fitting faster
// DRAM behind a fixed-width link buys nothing once the link is the limit, which
// is the single most common sizing error on an expansion card.
module link_limited_bandwidth #(parameter int MEDIA_SETS_RATE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] lanes, per_lane_gbps, media_gbps, demand_gbps,
  output logic [15:0] link_gbps, delivered, wasted_media, served_pct,
  output logic        demand_met,
  output logic [7:0]  n_evals, n_wasted,
  output logic        bw_err
);
  logic [31:0] l_q, s_q;
  logic [15:0] true_link, true_delivered, true_wasted;
  logic        truly_wasted;
  assign l_q = {16'd0, lanes} * {16'd0, per_lane_gbps};
  assign true_link = (l_q > 32'd9999) ? 16'd9999 : l_q[15:0];
  assign link_gbps = true_link;
  // What actually arrives is the smaller of the two, every time.
  assign true_delivered = (media_gbps > true_link) ? true_link : media_gbps;
  assign delivered = (MEDIA_SETS_RATE != 0) ? media_gbps : true_delivered;
  // Media rate bought above the link is rate that can never leave the card.
  assign true_wasted = (media_gbps > true_link) ? (media_gbps - true_link) : 16'd0;
  assign wasted_media = (MEDIA_SETS_RATE != 0) ? 16'd0 : true_wasted;
  assign s_q = (demand_gbps == 16'd0) ? 32'd100
             : (({16'd0, delivered} * 32'd100) / {16'd0, demand_gbps});
  assign served_pct = (s_q > 32'd100) ? 16'd100 : s_q[15:0];
  assign demand_met = (delivered >= demand_gbps) && (demand_gbps != 16'd0);
  assign truly_wasted = (true_wasted != 16'd0);
  assign bw_err = evaluate && truly_wasted && (wasted_media == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_wasted <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_wasted) n_wasted <= n_wasted + 8'd1;
    end
  end
endmodule

Sixteen lanes at four with a hundred gigabytes per second of media and eighty wanted delivers sixty-four — thirty-six stranded on the card, and eighty percent of the demand served.

The second case is a card sized correctly. Media slower than the link delivers the media rate, strands nothing, and meets the demand — both builds agree and neither alarms, which is the configuration a specification exercise should be aiming at.

The boundary case is where the money stops being wasted. Media exactly matching the link strands nothing and delivers everything, and it is the point past which additional media rate returns zero.

The fourth case is the clamp and it flips which side is limiting. A link rate past the counter saturates, and the media becomes the limit instead — which is the correct behaviour and a reminder that the ceiling is whichever of the two is lower rather than always the link.

The fifth case is the honest zero on the demand side. No demand stated is trivially served and is not a met demand, because nobody said what was wanted.

The last measured case is the error at full size. A card fitted far beyond its link strands a hundred and thirty-six gigabytes per second and serves a third of the demand — capacity and rate both bought, one of them entirely unusable.

The reason this error is common is that the two numbers live in different vocabularies. Media is specified in transfers per second per pin across some number of channels; a link is specified in lanes at a rate with an encoding overhead. Neither converts to the other by inspection, so a specification exercise that treats "fast DRAM" and "a wide link" as independently good choices can select both and discover the mismatch only under load.

The model's delivered is a minimum for a reason worth stating. Whichever of the two is lower is the answer, always, and neither being large rescues the other. That is why the fourth case matters more than it looks: when the link rate saturates, the media becomes the limit, and the model reports the media figure rather than continuing to blame the link. A model that always blamed the link would be right most of the time and wrong in exactly the configuration where somebody is about to buy a wider one.

And the stranded rate is paid for at full price. It is not a missed opportunity; it is a component selected, purchased, powered and cooled, whose extra capability cannot reach the host under any workload. Section 13's cost comparison is where that shows up in currency, and a card carrying stranded media is a card whose cost per usable gigabyte-per-second is worse than its datasheet suggests.

8. RTL 4 — What Decides A Card's Value Is Where The Pages Land

The fourth thing, and the one that makes a card a software decision.

The operating system sees a NUMA node with a distance. What it does with that is policy: pages land where the allocator put them, and unless something decided otherwise the placement is an accident of allocation order rather than of access frequency. A hot page on the far node pays section 6's added latency on every access.

// RTL 4 - the operating system sees a distant node, and a page placed on it by
// default rather than by decision is a page whose latency nobody chose. What
// decides a card's value is where the pages land, not how much it holds.
module page_placement #(parameter int PLACEMENT_IS_AUTOMATIC = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] pages, hot_pages, placed_local, miss_cost,
  output logic [15:0] hot_local, hot_remote, misplace_cost, local_pct,
  output logic        placement_ok,
  output logic [7:0]  n_evals, n_misplaced,
  output logic        place_err
);
  logic [31:0] c_q, p_q;
  logic [15:0] hot_ok, local_ok, true_local, true_remote;
  logic        truly_misplaced;
  assign hot_ok  = (hot_pages > pages) ? pages : hot_pages;
  assign local_ok = (placed_local > pages) ? pages : placed_local;
  // Only the hot pages that were placed locally are placed well.
  assign true_local = (hot_ok > local_ok) ? local_ok : hot_ok;
  assign hot_local = (PLACEMENT_IS_AUTOMATIC != 0) ? hot_ok : true_local;
  assign true_remote = hot_ok - true_local;
  assign hot_remote = (PLACEMENT_IS_AUTOMATIC != 0) ? 16'd0 : true_remote;
  assign c_q = {16'd0, true_remote} * {16'd0, miss_cost};
  assign misplace_cost = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign p_q = (pages == 16'd0) ? 32'd100
             : (({16'd0, local_ok} * 32'd100) / {16'd0, pages});
  assign local_pct = p_q[15:0];
  assign placement_ok = (hot_remote == 16'd0) && (hot_ok != 16'd0);
  assign truly_misplaced = (true_remote != 16'd0);
  assign place_err = evaluate && truly_misplaced && placement_ok;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_misplaced <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_misplaced) n_misplaced <= n_misplaced + 8'd1;
    end
  end
endmodule

A thousand pages with three hundred hot and two hundred placed near leaves a hundred hot pages on the card, at four thousand units — a fifth of the pages placed near.

The second case is placement done right. Every hot page near, with near capacity to spare, reports nothing remote — both builds agree and neither alarms, and the seven hundred cold pages on the card are exactly what the card is for.

The boundary case is the tight configuration. Exactly enough near capacity for the hot set places all of it and leaves no margin, which is a system that will start missing the moment the hot set grows.

The third and fourth cases are the two clamps and they guard opposite mistakes: more hot pages claimed than exist, and more near placement claimed than pages. Both clamp to the pages that exist, because a subset cannot exceed its set.

The fifth case is the default nobody chose. Nothing placed near at all puts every hot page on the card and saturates the miss cost — which is what happens when the capacity is added and the placement policy is left alone.

The degenerate case bounds it: no pages characterised reports nothing near and nothing remote and declines to call that a good placement.

This is the section that makes a card a software purchase, and it is the one most often treated as somebody else's problem. The hardware arrives correct: the capacity is present, the link trains, the NUMA node appears with a distance attached. Everything the hardware promised is delivered, and the workload is slower — because a distance in a table is advisory, and what actually decides latency is which node a page ended up on.

The default is not a policy, it is an accident. Absent a decision, pages land wherever the allocator was when the request arrived, which correlates with allocation order and not at all with access frequency. So the hot set is distributed roughly in proportion to capacity — which means the more capacity the card adds, the larger the fraction of hot pages that land on the slow side. The improvement makes the problem worse, which is the shape that makes it hard to diagnose.

The fifth case is that default drawn at full size, and it is worth reading as the realistic baseline rather than as a worst case: nothing placed near, every hot page on the card, and a saturated miss cost. That is not a misconfiguration — it is what happens when capacity is installed and nothing else is changed.

The miss cost multiplies by access frequency, which the model takes as an input rather than deriving. That is the honest boundary: a hot page is hot by some amount, and the difference between a page touched twice and one touched two thousand times per interval is the difference between a rounding error and the whole problem. The count is section 8; the multiplier is section 6's added latency, and it takes both to turn a placement into a cost.

9. RTL 5 — Interleaving Widens The Failure Domain

The fifth thing, and the one that is bought without being chosen.

Interleaving across cards buys bandwidth by spreading accesses. It also means an address range lives on several cards at once, so losing any one of them loses the range. The bandwidth and the failure domain are the same decision, and only the first half is usually stated.

// RTL 5 - interleaving across cards buys bandwidth and widens the failure
// domain in the same move. Every interleaved card is a card whose loss takes
// the whole interleave set with it.
module interleave_failure_domain #(parameter int WIDER_IS_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] cards, interleave_ways, capacity_each, outage_cost,
  output logic [15:0] domain_gb, ways_ok, exposure, independent_cards,
  output logic        domain_bounded,
  output logic [7:0]  n_evals, n_wide,
  output logic        dom_err
);
  logic [31:0] d_q, e_q;
  logic [15:0] true_domain;
  logic        truly_wide;
  assign ways_ok = (interleave_ways > cards) ? cards : interleave_ways;
  // Losing one card of an interleave set loses the set's whole capacity.
  assign d_q = {16'd0, ways_ok} * {16'd0, capacity_each};
  assign true_domain = (d_q > 32'd9999) ? 16'd9999 : d_q[15:0];
  assign domain_gb = (WIDER_IS_FREE != 0) ? capacity_each : true_domain;
  assign independent_cards = cards - ways_ok;
  assign e_q = {16'd0, domain_gb} * {16'd0, outage_cost};
  assign exposure = (e_q > 32'd9999) ? 16'd9999 : e_q[15:0];
  assign domain_bounded = (ways_ok <= 16'd1) && (cards != 16'd0);
  // The capacity guard matters: with nothing fitted, a wide set and a single
  // card both report a zero domain, and the measured build would alarm on
  // itself.
  assign truly_wide = (ways_ok > 16'd1) && (capacity_each != 16'd0);
  assign dom_err = evaluate && truly_wide && (domain_gb == capacity_each);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_wide <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_wide) n_wide <= n_wide + 8'd1;
    end
  end
endmodule

Eight cards in a four-way interleave of two hundred and fifty-six gigabytes each makes one card's loss a one-thousand-and-twenty-four gigabyte event, at three thousand and seventy-two units of exposure, with four cards outside the set.

The second case is the configuration that keeps the domain small. No interleaving bounds the domain to one card and leaves seven independent — at the cost of the bandwidth the interleave would have bought, which is the trade being made.

The third case is the clamp and it is the far end. More ways claimed than cards clamps to the cards present and makes the whole installation one domain, with nothing independent.

The last case is the one my own review added and the mutation campaign then confirmed was load-bearing. A three-way set with nothing fitted loses nothing, because there is no capacity to lose — and without that guard the measured build alarms on itself, since a zero domain and a one-card domain are the same number when the card holds nothing.

The trade this section prices is genuine and it is not one-sided. Interleaving is how a set of cards delivers more aggregate bandwidth than any one of them: consecutive addresses land on different devices, so sequential access is spread rather than serialised. That is a real and often necessary gain, and a chapter that presented interleaving as a mistake would be wrong.

What makes it worth a model is that the two halves are decided together and stated separately. The bandwidth appears in the performance discussion; the failure domain appears in the availability discussion; and the two discussions happen in different meetings with different people. The way count is one number that belongs in both, and the model's job is to put it there.

The independent-card count is the output to carry into an availability review, because it is the one that answers the question actually being asked. "How many cards can we lose?" has a different answer from "how many cards are there", and the gap between them is the interleave width. In the headline case there are eight cards, four of which are independent — so the machine has eight devices and five failure units, and any plan built on the first number is wrong.

And section 23 records how this is usually discovered, which is the part that makes it worth its place: not in a review at all, but during an outage, because the width was set by a platform default rather than by anybody's decision.

10. RTL 6 — Promotion Machinery Is Not Free

The sixth thing, and the one that turns section 8 from a policy into a cost.

Moving a page to the near tier costs a copy. Systems that migrate pages between tiers are doing real work to do it, and a page that is promoted and demoted repeatedly pays both copies and delivers nothing. The machinery that fixes the placement problem has its own bill.

// RTL 6 - promotion machinery is not free. Moving a page to the near tier costs
// a copy, and a page that is promoted and demoted repeatedly costs both copies
// and delivers nothing.
module promotion_cost #(parameter int MIGRATION_IS_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] promotions, demotions, copy_cost, accesses_saved,
  output logic [15:0] moves, move_cost, benefit, net_benefit,
  output logic        migration_pays,
  output logic [7:0]  n_evals, n_thrash,
  output logic        mig_err
);
  logic [31:0] m_q, c_q;
  logic [15:0] true_moves, true_cost;
  logic        truly_thrash;
  assign m_q = {16'd0, promotions} + {16'd0, demotions};
  assign true_moves = (m_q > 32'd9999) ? 16'd9999 : m_q[15:0];
  assign moves = true_moves;
  assign c_q = {16'd0, true_moves} * {16'd0, copy_cost};
  assign true_cost = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign move_cost = (MIGRATION_IS_FREE != 0) ? 16'd0 : true_cost;
  assign benefit = accesses_saved;
  // Reported against the cost this build admits to, so the free-migration view
  // stays coherent; the truth below still uses the real cost.
  assign net_benefit = (accesses_saved > move_cost)
                     ? (accesses_saved - move_cost) : 16'd0;
  assign migration_pays = (MIGRATION_IS_FREE != 0) ? (accesses_saved != 16'd0)
                                                   : (accesses_saved > true_cost);
  assign truly_thrash = (true_cost >= accesses_saved) && (true_moves != 16'd0);
  assign mig_err = evaluate && truly_thrash && migration_pays;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_thrash <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_thrash) n_thrash <= n_thrash + 8'd1;
    end
  end
endmodule

Four hundred promotions and three hundred and fifty demotions at eight, saving four thousand accesses, is seven hundred and fifty moves costing six thousand against four thousand of benefit — no net benefit at all.

The second case is migration working. Few demotions costs two thousand against four thousand of benefit — two thousand net, both builds agree, and neither alarms.

The boundary case is the one that decides whether to enable the machinery. Cost exactly equal to benefit is reported as not paying, because a mechanism that breaks even on the two quantities you measured is a loss on the ones you did not.

The sixth case is the honest positive with no machinery at all. A benefit with no moves is wholly net, both builds agree — which is what a correct initial placement looks like, and it is better than any amount of migration.

The same coherence note as 29.1 section 10 applies, and it is deliberate: the free-migration build reports its net benefit against the zero cost it claims, so it cannot publish a free migration and a loss at once. The truth it is checked against still uses the real cost.

The thrashing case is what the promotion and demotion counts are for, and it is why the model takes both rather than a single "migrations" figure. A workload whose hot set is stable produces many promotions and few demotions: pages move once and stay. A workload whose hot set rotates produces the two in near-equal numbers, and equal-and-large is the signature — the machinery is running continuously and the placement is no better at the end of it than at the start.

That is a diagnosis available from two counters, which makes it one of the cheapest readings in the chapter and the reason it is step 4 of the debug lab rather than step 8.

The section also sets a bound on how much section 8 can be fixed after the fact. Migration is a correction applied at runtime to a placement that was wrong at allocation, and every correction costs a copy. A system that places well initially pays nothing, which is what the sixth case reports — a whole benefit with no moves at all. That is not an argument against tiering software; it is the reason initial placement is worth getting right even when tiering software is available to rescue it.

11. RTL 7 — A Card Occupies A Slot And Draws Power

The seventh thing, and the one that is a chassis fact rather than a memory fact.

Capacity added on a card is capacity bought with a slot, watts and thermal headroom. The slot might have held an accelerator or a network card. The watts come out of a budget the chassis already allocated. None of that appears in a capacity figure, and all of it appears in whether the machine can actually be built.

// RTL 7 - a card occupies a slot, draws power and takes thermal budget. The
// capacity it adds is therefore not free capacity: it is capacity bought with
// resources the chassis also wanted for something else.
module slot_and_power_cost #(parameter int CAPACITY_IS_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] slots_total, slots_used, watts_each, watts_budget,
  output logic [15:0] slots_left, watts_drawn, watts_over, budget_pct,
  output logic        fits_chassis,
  output logic [7:0]  n_evals, n_over,
  output logic        chassis_err
);
  logic [31:0] w_q, p_q;
  logic [15:0] used_ok, true_drawn, true_over;
  logic        truly_over;
  assign used_ok = (slots_used > slots_total) ? slots_total : slots_used;
  assign slots_left = slots_total - used_ok;
  assign w_q = {16'd0, used_ok} * {16'd0, watts_each};
  assign true_drawn = (w_q > 32'd9999) ? 16'd9999 : w_q[15:0];
  assign watts_drawn = (CAPACITY_IS_FREE != 0) ? 16'd0 : true_drawn;
  assign true_over = (true_drawn > watts_budget) ? (true_drawn - watts_budget) : 16'd0;
  assign watts_over = (CAPACITY_IS_FREE != 0) ? 16'd0 : true_over;
  assign p_q = (watts_budget == 16'd0) ? 32'd999
             : (({16'd0, true_drawn} * 32'd100) / {16'd0, watts_budget});
  assign budget_pct = (p_q > 32'd999) ? 16'd999 : p_q[15:0];
  assign fits_chassis = (watts_over == 16'd0) && (slots_left != 16'd0);
  assign truly_over = (true_over != 16'd0);
  assign chassis_err = evaluate && truly_over && (watts_over == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_over <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_over) n_over <= n_over + 8'd1;
    end
  end
endmodule

Six cards of eight slots at seventy-five watts against a four-hundred-watt budget draws four hundred and fifty — fifty over, at a hundred and twelve percent of budget, with two slots left.

The second case is a configuration that fits. Four cards draws three hundred against four hundred, leaves four slots, and both builds agree — which is the answer a chassis review should produce.

The boundary case is the tight one. Exactly at the power budget fits, and a design sitting exactly on its thermal budget has no margin for a warm day or a degraded fan.

The fourth case is the other half of the constraint and it is the one the power figure hides. More cards claimed than slots leaves no slots, and the chassis does not fit another card even though the power is comfortably inside budget — two independent limits, and the model reports a failure when either binds.

The sixth case is the honest zero. No power budget stated puts the whole draw over an unstated budget and saturates the percentage, which is a chassis nobody has costed.

The two limits in this model are independent and that is the section's structural point. Slots and watts are not two views of the same constraint — a chassis can have slots free and no power for them, or power to spare and nowhere to put a card. The fourth case is the first of those and it is the one that surprises people: the power is comfortably inside budget, and the machine still cannot take another card, because the slots are gone.

Neither limit belongs to the person making the memory decision, which is why both go missing. The capacity requirement comes from an application team, the card selection from a memory architect, and the slot and thermal budget from whoever owns the platform — and the first two can reach agreement without the third being in the room. The model's job is to make the third a number rather than a veto that arrives late.

The thermal consequence is worth stating because it does not present as a memory problem. A chassis over its power budget throttles, and throttling shows up as unexplained slowness distributed across everything the machine is doing. The investigation starts wherever the most recent change was — which is the memory subsystem, because that is what was just installed — and the actual cause is a number on a platform specification that nobody added up.

And the boundary case deserves the same caution as every other boundary in this chapter. Exactly at budget fits, and a machine sitting exactly at its thermal budget has no margin for a warm aisle, a degraded fan or a card drawing slightly more than its typical figure.

12. RTL 8 — A Capacity-Bound Workload Gains; A Latency-Bound One Does Not

The eighth thing, and the one that says which workloads to put on it.

A card is a capacity device with a latency penalty, so it helps workloads limited by how much memory they have and harms workloads limited by how fast they can reach it. The classification is per workload, and an estate-wide answer has to be wrong about part of the estate.

// RTL 8 - where a card wins. A workload bounded by capacity and tolerant of the
// added latency gains; one bounded by latency does not, and counting which is
// which is what turns a product into a deployment.
module where_a_card_wins #(parameter int WINS_EVERYWHERE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] workloads, capacity_bound, latency_bound, gain_each,
  output logic [15:0] suited, unsuited, total_gain, suited_pct,
  output logic        suits_all,
  output logic [7:0]  n_evals, n_unsuited,
  output logic        fit_err
);
  logic [31:0] g_q, p_q;
  logic [15:0] cap_ok, lat_ok, true_suited, true_unsuited;
  logic        truly_unsuited;
  assign cap_ok = (capacity_bound > workloads) ? workloads : capacity_bound;
  assign lat_ok = (latency_bound > workloads) ? workloads : latency_bound;
  assign true_suited = (cap_ok > lat_ok) ? (cap_ok - lat_ok) : 16'd0;
  assign suited = (WINS_EVERYWHERE != 0) ? workloads : true_suited;
  assign true_unsuited = workloads - true_suited;
  assign unsuited = (WINS_EVERYWHERE != 0) ? 16'd0 : true_unsuited;
  assign g_q = {16'd0, true_suited} * {16'd0, gain_each};
  assign total_gain = (g_q > 32'd9999) ? 16'd9999 : g_q[15:0];
  assign p_q = (workloads == 16'd0) ? 32'd100
             : (({16'd0, true_suited} * 32'd100) / {16'd0, workloads});
  assign suited_pct = p_q[15:0];
  assign suits_all = (unsuited == 16'd0) && (workloads != 16'd0);
  assign truly_unsuited = (true_unsuited != 16'd0);
  assign fit_err = evaluate && truly_unsuited && suits_all;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_unsuited <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_unsuited) n_unsuited <= n_unsuited + 8'd1;
    end
  end
endmodule

Fifteen workloads with nine capacity-bound and four latency-bound is five net suited, ten not, at two hundred and fifty units of gain — a third of the estate.

The second case is the estate the product is for. Every workload capacity-bound is suited entirely at seven hundred and fifty units of gain, both builds agree, and neither alarms.

The boundary case is the one that decides a marginal estate. Equally capacity- and latency-bound nets to nothing, because the gains and the harms cancel — and an estate at that balance has not been shown to benefit.

The fourth case is the direction nobody proposes. A mostly latency-bound estate gains nothing at all, and the a-card view still calls it a sweep — the capacity really was installed, really is drawing power, and really is unused.

The degenerate case bounds it: an unwritten estate suits nothing, because nobody enumerated the workloads the claim is about.

The netting is the same arithmetic 29.1 section 12 uses and it says the same uncomfortable thing. A latency-bound workload is not merely unhelped by a card — it is harmed, because its pages are now further away and its limiting resource just got worse. So the harms subtract from the gains rather than failing to add to them, and an estate split evenly nets to zero rather than to half.

That is why a rollout decision cannot be made from a pilot on one workload. The pilot picks a workload that someone believed would benefit, which is a capacity-bound one, and it benefits. Extending that result to the estate assumes the estate looks like the pilot — and the estate contains the latency-bound workloads that the pilot deliberately did not choose.

The classification is also not stable, which is the honest limit on section 20's cost estimate. A workload can be capacity-bound at one working-set size and latency-bound at another, so the split is a measurement of the estate as it is configured today rather than a permanent property. A card justified by a classification is justified for as long as that classification holds, and a growth in one workload can move it across the line.

13. RTL 9 — The Comparison Is Against A Whole Machine

The ninth thing, and the one that decides whether the card should exist at all.

A card is not competing with a DIMM slot; it is competing with another server. The reason to add capacity on a card is that adding it any other way costs more — and the honest comparison is cost per gigabyte against the cost per gigabyte of simply buying another machine with memory in it. A card dearer per gigabyte than a whole server is a card nobody needs.

// RTL 9 - capacity per socket is what a card is bought for, and the comparison
// that decides it is against adding a whole machine rather than against adding
// a slot. A card that costs more per gigabyte than another server is a card
// nobody needs.
module capacity_per_socket #(parameter int MORE_IS_BETTER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] card_gb, card_cost, server_gb, server_cost,
  output logic [15:0] card_per_gb, server_per_gb, saving_per_gb, gb_added,
  output logic        card_cheaper,
  output logic [7:0]  n_evals, n_worse,
  output logic        cost_err
);
  logic [31:0] c_q, s_q;
  logic [15:0] true_card_rate, true_server_rate;
  logic        truly_worse;
  assign c_q = (card_gb == 16'd0) ? 32'd9999
             : ({16'd0, card_cost} / {16'd0, card_gb});
  assign true_card_rate = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign card_per_gb = true_card_rate;
  assign s_q = (server_gb == 16'd0) ? 32'd9999
             : ({16'd0, server_cost} / {16'd0, server_gb});
  assign true_server_rate = (s_q > 32'd9999) ? 16'd9999 : s_q[15:0];
  assign server_per_gb = true_server_rate;
  assign saving_per_gb = (true_server_rate > true_card_rate)
                       ? (true_server_rate - true_card_rate) : 16'd0;
  assign gb_added = card_gb;
  assign card_cheaper = (MORE_IS_BETTER != 0) ? (card_gb != 16'd0)
                                              : (true_card_rate < true_server_rate);
  assign truly_worse = (true_card_rate >= true_server_rate) && (card_gb != 16'd0);
  assign cost_err = evaluate && truly_worse && card_cheaper;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_worse <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_worse) n_worse <= n_worse + 8'd1;
    end
  end
endmodule

Five hundred and twelve gigabytes at four thousand and ninety-six against a thousand-and-twenty-four gigabyte server at six thousand one hundred and forty-four is eight per gigabyte against six — no saving at all.

The second case is the card worth buying. A genuinely cheaper card at four per gigabyte against six saves two, both builds agree, and neither alarms.

The boundary case is the marginal one. Exactly the same cost per gigabyte is reported as not cheaper, because equal price with added latency, an added failure domain and an added power draw is not a tie.

The fifth case is the comparison nobody made. No server figure at all saturates the server rate and reports the card as cheaper — anything beats a comparison nobody made, which is exactly how a card gets approved without this section being done.

The degenerate case bounds it: nothing costed saturates both rates and calls the card not cheaper, which is the absence of a decision rather than a decision.

The comparison is against a whole machine because that is the actual alternative. A design that needs more memory than a socket holds has two options: reach further with a card, or add another socket with its own memory attached. The second option brings cores, network and storage along with the memory, which is either a bonus or waste depending on whether they were needed — and it is why the comparison is genuinely hard rather than merely arithmetic.

Against a DIMM slot the card always loses, which is why that comparison gets made: a slot is cheaper, faster and has no failure domain of its own. But a slot that is already full is not an option, and the card exists precisely for the case where the slots have run out. Comparing against an alternative that is unavailable is how a decision gets justified without being made.

The fifth case is the version of that failure the model can detect, and it is the most common one in practice. With no server figure entered, the server rate saturates and the card is cheaper than it by an enormous margin — anything beats a comparison nobody made, and the model reports the card as cheaper because that is arithmetically what an unstated alternative produces.

What the model deliberately does not price is everything the other eight sections found. The added latency, the widened failure domain, the migration machinery, the slot, the watts. That is why the boundary case reports equal cost per gigabyte as not cheaper: a tie on the one axis this model measures is a loss once the axes it does not measure are counted, and a deployment should want a margin rather than a tie.

14. RTL 10 — A Memory-Expansion-Card Case Study Assembled

Nine sections of inputs. This one puts them together.

// RTL 10 - a memory-expansion-card case study assembled. Nine sections of
// inputs, one summary. "It is a memory card" is bit 0: true of the shape, and
// one sixth of a deployment.
module card_case_signoff #(parameter int A_CARD_IS_THE_ANSWER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic        card_named, latency_stated, bandwidth_sourced,
  input  logic        placement_stated, failure_domain_stated, chassis_costed,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_evals, n_sound, n_claimed,
  output logic        signoff_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~card_named;
  assign fail_mask[1] = ~latency_stated;
  assign fail_mask[2] = ~bandwidth_sourced;
  assign fail_mask[3] = ~placement_stated;
  assign fail_mask[4] = ~failure_domain_stated;
  assign fail_mask[5] = ~chassis_costed;
  assign conditions_met = {15'd0, card_named} + {15'd0, latency_stated}
                        + {15'd0, bandwidth_sourced} + {15'd0, placement_stated}
                        + {15'd0, failure_domain_stated} + {15'd0, chassis_costed};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp: conditions_met sums six one-bit values, so the quotient cannot
  // exceed a hundred and a ceiling would be unreachable code.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  // The "it is a memory card" view reads bit 0 and stops.
  assign claimed = (A_CARD_IS_THE_ANSWER != 0) ? card_named : truly_sound;
  assign sound = claimed;
  assign signoff_err = evaluate && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When the card has been named and any one of the other five fails, the assembled model reports that the case study is not sound and the a-card view reports a case study.

Across the eight evaluations, the assembled model calls one case study sound and the a-card view calls seven of eight a case study.

The bit order is by how much of the deployment each condition carries. Bit 1 is first among the five because the latency tier is what the card is — every placement, migration and workload decision below it is downstream of that number. Bit 2 is the ceiling that sizing gets wrong. Bit 3 is where the value is realised or lost. Bits 4 and 5 are the two costs that belong to somebody other than the memory architect, which is why they go missing.

"It is a memory card" is bit 0, and it fails differently from this batch's other weak definitions. 28.4's substitutes a true claim for an argument and 29.1's names a category containing its own opposite. This one describes the packaging — it is a claim about what the thing looks like offered in place of a claim about what it does, and it is the only one of the three that could be verified without powering the device on.

The five other bits fail independently. The latency can be stated by somebody who never checked the link width. The bandwidth can be sourced correctly with no placement policy at all. The placement can be right while the interleave quietly made eight cards into one failure domain. And the chassis cost is the one that fails last, because it belongs to the platform team and the memory decision was made before they saw it.

The right-hand terminal is a description of the shape, and that is the whole difference between this chapter and a datasheet. A datasheet contains every one of the five numbers; the description that reaches a design meeting usually contains none of them. The failure is not that the information does not exist — it is that it stopped travelling one step before the decision.

Two of the six bits belong to other teams, which is the structural reason they fail most often. The failure domain is set at the platform level, frequently by a firmware default; the slot and power budget belongs to whoever owns the chassis. Neither is a memory decision and both are consequences of one, so they are discovered rather than chosen — and a case study that reaches five bits and stops has almost certainly stopped at exactly those two.

Bit 3 is the one that decides whether any of it mattered. Placement is where the capacity becomes useful or becomes a slower copy of memory the machine already had, and it is the only bit of the six that can be got right or wrong after everything is installed. The other five are properties of the purchase; this one is a property of the running system, and it is the one section 21 checks first.

15. Quantitative Reasoning

Four properties of six unstated, from a description where seventy percent of the claims are about the form factor — a third of the device described.

A hundred and sixty nanoseconds added over a ninety-nanosecond local access, at two hundred and seventy-seven percent of local and a hundred nanoseconds past tolerance.

Thirty-six gigabytes per second stranded behind a sixty-four gigabyte link, with eighty percent of the demand served.

A hundred hot pages of three hundred left on the card, at four thousand units, with a fifth of the pages placed near.

A one-thousand-and-twenty-four gigabyte failure domain from a four-way interleave of two hundred and fifty-six gigabyte cards, at three thousand and seventy-two units of exposure.

Seven hundred and fifty page moves costing six thousand against four thousand of benefit — no net benefit at all.

Fifty watts over a four-hundred-watt budget, at a hundred and twelve percent, with two slots left.

Ten workloads of fifteen unsuited, with five suited at two hundred and fifty units of gain.

Eight per gigabyte against a server's six — a card dearer than the machine it was meant to avoid buying.

One case study of eight sound; the a-card view counts seven. The assembled model's summary, and the chapter's.

16. Assertions

The testbenches carry 460 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported nothing on either testbench, and this is the first chapter in the batch where all five scripted checks passed on their first run.

Both builds are asserted on every degenerate case. Nothing enumerated, nothing measured, nothing specified, no pages characterised, no installation described, an unmeasured migration, an unwritten chassis, an unwritten estate, and an uncosted card.

Every clamp that an input can reach is driven past its limit exactly once. More properties stated than matter, more form claims than claims, a card latency past its counter, a ratio past its ceiling, a link rate past its counter, a served percentage past a hundred, more hot pages than exist, more near placement than pages, a miss cost past its counter, more ways than cards, a domain and an exposure past theirs, a move count and a copy cost past theirs, a power draw and a budget percentage past theirs, a gain past its counter, and both the card rate and the server rate past theirs — the second of which is section 17's finding.

Every threshold is asserted on both sides of its boundary. The latency tolerance at exactly the limit and one nanosecond past; the media rate exactly matching the link; the hot set exactly filling the near capacity; the power exactly at budget; the migration exactly breaking even; the estate exactly balanced; the cost per gigabyte exactly equal.

Every error output is checked in both directions in every case. Each model's second case is a configuration where the a-card view happens to be right, and a model that alarmed on one would be unusable.

17. Mutation Testing

128 mutations, 128 killed. Sixty-two against the first testbench, sixty-six against the second. The first run killed a hundred and twenty-seven and left one.

The single survivor was a clamp I had tested on one side of a symmetric pair and not the other. Section 13 computes two cost-per-gigabyte figures with identical structure — one for the card, one for the server. I drove the card rate past its ceiling and never the server rate, so removing the server clamp changed nothing any assertion could see.

The rule this refines is one the batch has carried since 026: for every clamp, drive the input past the limit once. The clause it needed is that a symmetric pair is two clamps, not one. The card and server rates are the same four lines of arithmetic with different inputs, and that similarity is precisely what made the gap invisible — reading the model, the pair looks covered because one of them is.

This is a different failure from the equivalent mutant 29.1 found, and the contrast is the useful part. There, the survivor could not be killed by any stimulus and the mutation had to be withdrawn. Here the survivor was a genuine hole and one case closed it — a server of one gigabyte at fifty thousand, which saturates the rate and makes the clamp load-bearing. Telling the two apart is the whole skill: ask whether an input exists that would make the mutated line behave differently, and only if the answer is no is the mutant equivalent.

One defect was caught by reading before the campaign ran, and the campaign then proved it mattered. Section 9's truth signal was ways_ok > 1 with no capacity guard — so an interleaved set of cards with nothing fitted reported a wide domain of zero gigabytes, which equals the one-card domain of zero, and the measured build alarmed on itself. Adding the capacity guard fixed it, and the mutation that removes that guard again is killed by the three-way-set-with-nothing-fitted case, which exists for exactly that reason.

18. Verification Strategy

Ask for the six properties, not the form factor. Section 5.

Ask what the link and the controller add, and against what local baseline. Section 6.

Compute lanes times rate before looking at the media. Section 7. The ceiling is the lower of the two.

Ask what the page placement policy is. Section 8. The default is an accident of allocation order.

Ask how wide the interleave is. Section 9. That is the failure domain, and it was bought with the bandwidth.

Count the page moves and price them. Section 10.

Take the slot count and the watts to the platform team. Section 11. Both limits bind independently.

Classify each workload as capacity- or latency-bound. Section 12.

Compare cost per gigabyte against another whole machine. Section 13, not against a DIMM.

19. Synthesis and Implementation Reality

A memory expander's controller is a real design with real latency, and it sits between the link and the media adding to both. That is why section 6 sums three terms rather than two, and why a faster DRAM part moves the total less than people expect.

Expanded capacity needs more requests in flight to sustain bandwidth, which is 9.6's concurrency argument arriving at a card: the longer round trip means a request pool sized for local latency throttles the card's traffic while reporting no error at all. That is a failure section 7's arithmetic does not predict — the link is wide enough and the bandwidth still does not arrive.

Operating systems represent the card as a NUMA node, and the tiering machinery that promotes and demotes pages between nodes is section 10's subject. It works, it is not free, and whether it pays is a measurement rather than an assumption.

Interleaving is configured at the platform level, frequently by firmware defaults rather than by a decision, which is why section 9's failure domain is so often a surprise rather than a choice.

And the honest reading of section 13 is that the answer changes with the market. Cost per gigabyte on both sides of that comparison moves, so a card that is not worth buying this year may be next — which is an argument for recomputing the comparison rather than for settling it.

20. Silicon Observability

Free, and from a specification sheet. Lane count, per-lane rate, media rate, card capacity, watts per card. Sections 7 and 11.

Free, and from a platform document. Slot count and chassis power budget. Section 11.

Cheap, and from firmware settings. The interleave width. Section 9 — this is a configuration value somebody can read in minutes, and it is the single highest-value reading in the chapter relative to its cost.

Cheap, and from a memory-latency tool. The local and card latencies. Section 6 needs a baseline as well as a measurement, and the baseline is the half most often skipped.

Moderate, and needs instrumentation. The hot page count and where those pages actually are. Section 8 is the number that decides the card's value and the hardest of the cheap ones to obtain.

Moderate. Promotion and demotion counts. Section 10's machinery usually reports them if asked.

Expensive. Per-workload classification as capacity- or latency-bound. Section 12 needs each workload profiled against both limits.

Expensive, and owned elsewhere. The cost per gigabyte of a whole comparison machine. Section 13's second figure lives in procurement.

21. Debug Lab

Cards were installed and an application is slower.

Step 1 — check where the hot pages are. Section 8. This is the most common cause by a wide margin: the capacity arrived, the placement policy did not change, and hot pages landed on the far node.

Step 2 — check the added latency against the workload's tolerance. Section 6. If the workload is latency-bound, section 12 applies and no placement fix will help enough.

Step 3 — check lanes times rate against the media. Section 7. If the link is the ceiling, the bandwidth the card was bought for was never available.

Step 4 — check the promotion and demotion counts. Section 10. Equal and large numbers mean thrashing, and the machinery is now a cost rather than a fix.

Step 5 — check the interleave width. Section 9. It may explain a bandwidth figure, and it certainly changes the failure domain.

Step 6 — check the chassis power. Section 11. Throttling under a power cap presents as unexplained slowness and has nothing to do with memory.

Steps 1 and 5 are the two cheapest and between them explain most instances.

22. Design Review

Which six properties of this card do we actually know?

What does the link plus the controller add, and against which local baseline?

What is lanes times rate, and what is the media rate?

What places the hot pages, and what is placing them today?

How wide is the interleave, and what does one card's loss take with it?

How many promotions and demotions, and what does a copy cost?

How many slots and how many watts, and has the platform team seen both?

Which workloads are capacity-bound and which are latency-bound?

What is the cost per gigabyte against another whole machine?

23. How This Appears In Real Engineering

The failure is a device described by its packaging, and it propagates because the packaging is the part everybody can see.

The most common shape is section 8. Capacity is installed and nothing places the pages. The machine has more memory, the benchmark is slower, and the cause is that the allocator spread hot pages across a node two and a half times further away. Nothing is broken and nothing was configured.

The second is section 7 and it is the one that wastes money at purchase. Media is specified by rate rather than by capacity, a faster part is chosen for a card whose link cannot carry it, and a third of the rate is stranded on the board. The card meets its specification perfectly.

The third is section 9 and it is discovered during an outage. A firmware default set an interleave width nobody chose, one card fails, and a multi-card capacity range goes with it — in a system whose availability plan assumed a card was a card.

The fourth is section 11. The memory decision is made and the platform team sees the power number later. The chassis throttles, the symptom is unexplained slowness, and the investigation starts in the memory subsystem because that is what changed.

The fifth is section 13 and it is the quietest. No comparison is made at all. The card is evaluated against the alternative of doing nothing rather than against another machine, so it wins by default — and the model reports exactly that: anything beats a comparison nobody made.

The sixth shape is section 6 and it is the one that ends a project rather than degrading it. A latency-sensitive service is moved onto expanded capacity because the capacity figure was the only figure in the proposal. The service meets none of its latency objectives, the cause is a fixed per-access addition that no amount of tuning removes, and the capacity has to be rolled back — at which point the slots, the watts and the procurement are all already spent.

The pattern is that every one of these is cheap to check and none of them is visible in the phrase that introduced the device. Five of the six checks are a specification sheet, a firmware setting or a platform document away, and the sixth needs a profiler most estates already run. The method costs an afternoon and the failures cost a deployment, which is an unusually favourable ratio and the reason this chapter is a checklist rather than an argument.

24. Common Misconceptions

"It is a memory card." That is the shape. Section 5.

"Memory is memory." Plus a link and a controller. Section 6.

"We fitted faster DRAM, so it is faster." Not past the link. Section 7.

"The capacity is there, so the application can use it." Where are the hot pages? Section 8.

"A card failure loses a card." It loses the interleave set. Section 9.

"Tiering software handles the placement." At a copy per move. Section 10.

"It is just extra memory." It is a slot and seventy-five watts. Section 11.

"It helped this workload, so roll it out." To the capacity-bound ones. Section 12.

"It is cheaper than buying more DIMMs." Compare it to another server. Section 13.

25. Interview Reasoning

"What is a CXL memory expansion card?" The useful answer refuses the form-factor description: it is a link-attached memory device with a controller, and its defining properties are the latency the link and controller add, the bandwidth ceiling the link sets, and the fact that the operating system will see it as a distant NUMA node. The capacity is the thing being bought; those three are the terms it is bought on.

"How would you size one?" Lanes times per-lane rate first, because that is the ceiling — media fitted faster than the link is money that stays on the card. Then the added latency against what the workload tolerates. Then the hot-page count, because the capacity only pays if the cold pages are the ones that move to it.

"Why would a machine with a card installed get slower?" Almost always placement. The capacity arrived and nothing told the allocator that the new node is further away, so hot pages land there and pay the added latency on every access. The second candidate is the link being the real ceiling, and the third is the chassis throttling under a power cap that the extra cards pushed past.

"What does interleaving across cards cost?" The failure domain. It buys bandwidth by spreading an address range over several cards, which means losing any one of them loses the range. That is one decision with two consequences, and it is frequently set by a firmware default rather than chosen — so the availability plan and the actual failure domain can disagree without anybody having decided anything.

"Is page migration the answer to the placement problem?" Sometimes, and it has a bill. Every promotion and demotion is a copy, and a page that oscillates pays both directions and delivers nothing. The test is whether the accesses saved exceed the copies spent — and breaking even is a loss, because the machinery also adds a failure mode and a thing to debug.

"What should a card be compared against?" Another whole machine, not a DIMM slot. The reason to add capacity on a card is that adding it any other way costs more per gigabyte, so the comparison is cost per gigabyte against a server with memory in it. A card that is dearer than that — with added latency, an added failure domain and an added power draw on top — is a card nobody needs.

26. Exercises

1. A description has 9 properties that matter, 3 stated, and 12 of 16 claims about the form. Compute the unstated count, the form percentage and the described fraction.

2. Local is 110 ns; the link adds 200 and the media 90; tolerance is 80. Compute the card latency, the added latency, the ratio and the overage. What tolerance would this card meet?

3. 8 lanes at 8 GB/s with 140 GB/s of media and 120 wanted. Compute the link rate, delivered, stranded and served. What media rate wastes nothing?

4. 4,000 pages, 900 hot, 600 placed near, 35 per miss. Compute the hot-remote count and the miss cost. What near capacity places the whole hot set?

5. 12 cards, 6-way interleave, 512 GB each, 4 per GB of outage cost. Compute the domain, the independent count and the exposure. What does 2-way cost instead?

6. 900 promotions, 880 demotions, 6 per copy, 5,000 accesses saved. Compute the moves, the cost and the net. At what copy cost does it break even?

7. 16 slots, 11 used, 80 W each, a 700 W budget. Compute the draw, the overage and the slots left. How many cards fit?

8. 24 workloads, 15 capacity-bound, 7 latency-bound, 60 units each. Compute suited, unsuited and gain.

9. A 1,024 GB card at 6,000 against a 2,048 GB server at 10,240. Compute both rates and the saving. At what card cost do they tie?

10. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify its position using the rule that the ordering is by how much of the deployment each condition carries.

27. Summary

The shape is not the device — "memory card" names a rectangle and a slot, and none of the six properties that decide a design.

The link adds a fixed amount to every access, and so does the controller; the DRAM fitted changes the total less than expected.

The bandwidth comes from the link, not the media, and rate bought above the link never leaves the card.

What decides a card's value is where the pages land, which is a policy decision that has a default nobody chose.

Interleaving widens the failure domain in the same move that buys bandwidth, and only the bandwidth half is usually stated.

Promotion machinery is not free, and a page that oscillates pays both copies and delivers nothing.

A card occupies a slot and draws power, and both limits bind independently of each other.

A capacity-bound workload gains and a latency-bound one does not, so the classification is per workload.

The comparison is against a whole machine, not against a DIMM slot — and anything beats a comparison nobody made.

Six bits, and "it is a memory card" is one of them. One case study of eight is sound; the a-card view counts seven.