Real Industry Memory Devices

17.1 built an expander. 17.2 built a persistent one. 17.3 put several of them in a system and decided what goes where.

All three are models. This chapter is about the gap between a model and a device somebody is offering to sell you, and the discipline that closes it.

1. The Engineering Problem — A Datasheet Is Not A Measurement

Six things separate evaluating a real device from reasoning about a modelled one.

Most of what you know, the vendor said. Some claims can be measured on a bench, some only under conditions you cannot reproduce, and some cannot be checked at all. Recording all three the same way is how an evaluation ends up resting on the third kind. Section 5.

A device has to physically fit. Power, airflow and height are three independent budgets, and a device that fits one is routinely assumed to fit the others. Section 6.

Burst performance is not sustained performance. A device that meets its number for a second and throttles afterwards has a different number, and the benchmark that found the first one will not find the second. Section 7.

Generation is negotiated, and the loser sets the terms. A 3.0 host with a 2.0 device runs a 2.0 link, and the capabilities that needed 3.0 are simply absent. Section 9.

How many fit is not how many slots there are. Slots, host power and address space are three ceilings and the population is the smallest, not the first one anybody checked. Section 10.

And two devices of the same class are not the same device. The spread across a population is a property no class description carries, and a capacity plan made against the typical device is a plan some deployed device will not meet. Section 13.

This chapter against 17.3, stated precisely. That chapter owns deciding where data lives among memories you already have. This one owns deciding whether a memory should be there at all.

2. The One-Sentence Model

An evaluation is the process of converting claims into measurements, and its quality is exactly the fraction it converts — every defect below is a claim that reached a deployment decision without being measured.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Four devices, six qualification checks, three slot budgets and eight slots are small numbers chosen so every boundary is reachable and every result is verifiable on paper.

What is not simplified is the structure: claims separated by whether they are checkable, budgets checked independently rather than as a single "fits" bit, a sustained test distinct from a burst test, a negotiated generation that gates capability, a population that is the minimum of three ceilings, and a deployment gated on measurements rather than on the datasheet.

Two things are absent by design. There is no acceptance-sampling model — how many devices out of a lot you test, and what a failure rate means, is statistics rather than architecture. And there is no supply-chain or lifecycle model: revision changes, firmware updates and end-of-life are real evaluation inputs and none of them is a property of the device's behaviour.

5. RTL 1 — Not Every Claim Can Be Checked

// A datasheet claim is either checkable or it is not, and the two must not be
// recorded the same way.
module claim_class #(parameter int TRUST_ALL_CLAIMS = 0) (
  input  logic clk, rst_n,
  input  logic       record,
  input  logic [1:0] claim_kind,   // 0 measurable, 1 conditional, 2 unverifiable
  input  logic       bench_agrees,
  output logic       accepted, needs_test,
  output logic [7:0] n_claims, n_measured, n_assumed,
  output logic       unverified_accept_err
);
  logic measurable;
  assign measurable = (claim_kind == 2'd0);
  // A measurable claim is accepted when the bench agrees with it. The trusting
  // build accepts every claim as printed, which is what an evaluation does when
  // it has no bench.
  assign needs_test = measurable;
  assign accepted   = (TRUST_ALL_CLAIMS != 0) ? 1'b1
                                              : (measurable && bench_agrees);
  // Accepting a claim that was never measured.
  assign unverified_accept_err = record && accepted && !(measurable && bench_agrees);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_claims <= 8'd0; n_measured <= 8'd0; n_assumed <= 8'd0;
    end else if (record) begin
      n_claims <= n_claims + 8'd1;
      if (measurable && bench_agrees) n_measured <= n_measured + 8'd1;
      else if (accepted)              n_assumed  <= n_assumed + 8'd1;
    end
  end
endmodule

Four claims presented — one measurable and confirmed, one measurable and contradicted, one conditional, one unverifiable:

  claims: recorded=4 measured=1 assumed=0 | trusting assumed=3 unverified=3

One of four claims survived contact with a bench. Not because the device is bad — because three of the four claims were not the kind of thing a bench can settle. The three classes are worth naming carefully:

The distinction is not pedantry. 17.2 section 11 built a wear model whose endurance limit is exactly a class-three claim: no evaluation is going to write a device to death to check it, so it enters the decision as an assumption, and it should be recorded as one.

6. RTL 2 — Three Budgets, Not One

// A device must fit the slot it is going into: power, thermal and height are
// three separate budgets and the device must satisfy all three.
module slot_fit #(parameter int POWER_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] dev_watts, slot_watts,
  input  logic [15:0] dev_airflow_req, slot_airflow,
  input  logic [7:0]  dev_height, slot_height,
  output logic        fits,
  output logic [2:0]  fail_mask,
  output logic [7:0]  n_eval, n_fit,
  output logic        overcommit_err
);
  // One bit per budget, so a rejection names the budget that rejected it.
  assign fail_mask[0] = (dev_watts       > slot_watts);
  assign fail_mask[1] = (dev_airflow_req > slot_airflow);
  assign fail_mask[2] = (dev_height      > slot_height);
  // The power-only build checks the budget everyone checks and neither of the
  // two that strand devices in the field.
  assign fits = (POWER_ONLY != 0) ? ~fail_mask[0] : (fail_mask == 3'd0);
  // Declaring a fit while a budget is exceeded.
  assign overcommit_err = evaluate && fits && (fail_mask != 3'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_fit <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (fits) n_fit <= n_fit + 8'd1;
    end
  end
endmodule

Five devices against one slot:

  slot: evaluated=5 fit=2 | power-only fit=4 overcommits=0

Two fit, and the power-only evaluation passed four. The two it wrongly passed failed on airflow and on height — budgets that are invisible on a bench with the lid off and decisive in a populated chassis.

All three comparisons are strict, and the bench drives a device sitting exactly on every budget to prove it. A device drawing exactly the slot's rated power fits; >= would reject it, and rejecting every device that exactly meets a budget rejects the devices designed to the specification.

The failure mask is three bits for the reason established in 17.2 section 15 and 17.3 section 15: 3'b010 sends an engineer to the airflow and "does not fit" sends them to all three.

7. RTL 3 — Burst Is Not Sustained

// Sustained load is not burst load. A device that meets its number for a second
// and throttles afterwards has a different number.
module thermal_throttle #(parameter int NO_THROTTLE = 0) (
  input  logic clk, rst_n,
  input  logic        active,
  input  logic [15:0] full_bw, throttled_bw,
  input  logic [7:0]  temp_c, throttle_c,
  output logic        throttling,
  output logic [15:0] this_bw,
  output logic [31:0] delivered,
  output logic [15:0] n_cyc, n_throttled,
  output logic        burst_claim_err
);
  assign throttling = (NO_THROTTLE != 0) ? 1'b0 : (temp_c >= throttle_c);
  assign this_bw    = throttling ? throttled_bw : full_bw;
  // Delivering the full rate while over the throttle temperature is the claim a
  // burst benchmark makes on behalf of a device that cannot sustain it.
  assign burst_claim_err = active && (temp_c >= throttle_c) && (this_bw == full_bw);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      delivered <= 32'd0; n_cyc <= 16'd0; n_throttled <= 16'd0;
    end else if (active) begin
      delivered <= delivered + {16'd0, this_bw};
      n_cyc <= n_cyc + 16'd1;
      if (throttling) n_throttled <= n_throttled + 16'd1;
    end
  end
endmodule

Four cool cycles at 400, then six hot cycles:

  thermal: delivered=2500 throttled_cyc=6 | no-throttle delivered=4000 burst_claims=6

2500 against 4000 — a 37.5% shortfall that appears only after the device warms up. The first four cycles are identical in both builds. A benchmark that ran for four cycles would have measured 1600 and reported a device delivering exactly its rated 400 per cycle, and it would have been correct about everything it measured.

The throttle comparison is inclusive and driven exactly: at 80°C against an 80°C limit the device throttles, and at 79°C it does not. > instead of >= moves the throttle point by one degree and is invisible to any test that heats the device well past it.

8. Waveform — Ten Cycles Of Sustained Load

Transcribed from the printed trace. One stimulus stream, both builds.

9. RTL 4 — Generation Is Negotiated

// Two devices negotiate the highest generation BOTH support, and what is lost is
// the difference, not nothing.
module gen_negotiate #(parameter int ASSUME_HIGHEST = 0) (
  input  logic clk, rst_n,
  input  logic       link_up,
  input  logic [1:0] host_gen, dev_gen,   // 0 = 1.1, 1 = 2.0, 2 = 3.0
  output logic [1:0] run_gen, lost_gens,
  output logic       switching_ok, pooling_ok, sharing_ok,
  output logic [7:0] n_links, n_degraded,
  output logic       overclaim_err
);
  logic [1:0] lower;
  assign lower = (host_gen < dev_gen) ? host_gen : dev_gen;
  // The assuming build reports the host's generation whatever the device is,
  // which is right whenever they match and wrong in the direction that matters.
  assign run_gen   = (ASSUME_HIGHEST != 0) ? host_gen : lower;
  assign lost_gens = (host_gen > run_gen) ? (host_gen - run_gen) : 2'd0;
  // Capability by generation. Each is a real gate on what the fabric may do.
  assign switching_ok = (run_gen >= 2'd1);
  assign pooling_ok   = (run_gen >= 2'd1);
  assign sharing_ok   = (run_gen >= 2'd2);
  // Running at a generation the device does not support.
  assign overclaim_err = link_up && (run_gen > dev_gen);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_links <= 8'd0; n_degraded <= 8'd0;
    end else if (link_up) begin
      n_links <= n_links + 8'd1;
      if (lost_gens != 2'd0) n_degraded <= n_degraded + 8'd1;
    end
  end
endmodule

Four link-ups:

  generation: links=4 degraded=2 | assuming build overclaimed=2

The fourth row is the one that keeps the model honest. A 1.1 host with a 3.0 device also runs at 1.1, and nothing is lost relative to the host — the host was never going to do more. lost_gens measures degradation against what the host could have had, and the assuming build is correct here too, because the host is the lower of the two. Without that row, "the negotiation takes the higher" survives as a mutation.

The capability gates are the point. Section 9's "one generation lost" is not an abstraction: it is the difference between a fabric that can share memory between hosts and one that cannot, on a link that reports itself up and healthy either way.

10. RTL 5 — How Many Actually Fit

// How many devices actually fit: slots, host power and address space are three
// independent ceilings and the population is the smallest of them.
module population_limit (
  input  logic clk, rst_n,
  input  logic        plan,
  input  logic [7:0]  slots,
  input  logic [15:0] host_watts, per_dev_watts,
  input  logic [15:0] addr_gb, per_dev_gb,
  output logic [7:0]  by_slots, by_power, by_addr, max_devs,
  output logic [1:0]  binding,
  output logic [7:0]  n_plans,
  output logic        no_capacity_err
);
  logic [15:0] pw_q, pa_q;
  assign pw_q = (per_dev_watts == 16'd0) ? 16'hFFFF : (host_watts / per_dev_watts);
  assign pa_q = (per_dev_gb    == 16'd0) ? 16'hFFFF : (addr_gb    / per_dev_gb);
  assign by_slots = slots;
  assign by_power = (pw_q > 16'd255) ? 8'hFF : pw_q[7:0];
  assign by_addr  = (pa_q > 16'd255) ? 8'hFF : pa_q[7:0];
  assign max_devs = (by_slots < by_power)
                  ? ((by_slots < by_addr) ? by_slots : by_addr)
                  : ((by_power < by_addr) ? by_power : by_addr);
  // Which ceiling is binding. Reporting the number without the reason sends an
  // engineer to buy slots when the limit is watts.
  assign binding = (max_devs == by_slots) ? 2'd0
                 : ((max_devs == by_power) ? 2'd1 : 2'd2);
  assign no_capacity_err = plan && (max_devs == 8'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) n_plans <= 8'd0;
    else if (plan) n_plans <= n_plans + 8'd1;
  end
endmodule

  population: slots=8 power=10 addr=8 max=8 binding=0

Eight slots, 800 W at 75 W each, 4096 GB of address space at 512 GB each. Three ceilings of 8, 10 and 8, and the answer is 8.

The bench then moves the binding ceiling twice. Halving the host power makes it 4, bound by watts. Restoring the power and quartering the address space makes it 2, bound by address space. The device did not change and the number went from 8 to 4 to 2.

Reporting the number without the reason is the defect binding exists to prevent. "Four devices" sends somebody to buy more slots; "four devices, bound by power" sends them to the power budget, which is the only place the number can be improved.

This is 16.4 section 5's ceiling argument applied to device population rather than switch ports, and the structure is identical: several independent limits, the smallest one binds, and the one that binds must be recorded because the others are irrelevant until it moves.

11. RTL 6 — Qualification

// The checks a device passes before it may be deployed, and the order they must
// be run in.
module qualification #(parameter int SKIP_SUSTAINED = 0) (
  input  logic clk, rst_n,
  input  logic       run,
  input  logic       enumerates, window_ok, burst_ok, sustained_ok, thermal_ok, field_ok,
  output logic       qualified,
  output logic [5:0] fail_mask,
  output logic [7:0] n_run, n_passed,
  output logic       premature_pass_err
);
  assign fail_mask[0] = ~enumerates;
  assign fail_mask[1] = ~window_ok;
  assign fail_mask[2] = ~burst_ok;
  assign fail_mask[3] = ~sustained_ok;
  assign fail_mask[4] = ~thermal_ok;
  assign fail_mask[5] = ~field_ok;
  // The skipping build runs the burst test and not the sustained one, which is
  // the shortcut every schedule pressures an evaluation into taking.
  assign qualified = (SKIP_SUSTAINED != 0)
                   ? (enumerates && window_ok && burst_ok && thermal_ok && field_ok)
                   : (fail_mask == 6'd0);
  assign premature_pass_err = run && qualified && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_run <= 8'd0; n_passed <= 8'd0;
    end else if (run) begin
      n_run <= n_run + 8'd1;
      if (qualified) n_passed <= n_passed + 8'd1;
    end
  end
endmodule

Seven runs — all passing, then each check falsified alone:

  qualify: runs=7 passed=1 | skip-sustained passed=2

The six checks are ordered by how early they fail and how cheap they are to run:

The skipped check is number 3, and the choice is not arbitrary. It is the most expensive test in the list — it needs the device under load for a long time, in a thermally realistic environment — and it is the one whose absence is invisible in every other result. A device that passes 0, 1, 2, 4 and 5 looks fully qualified.

12. RTL 7 — Two Devices Of The Same Class Are Not The Same Device

// Two devices of the same class are not the same device. The spread across a
// population is a property the class does not carry.
module vendor_variance #(parameter int USE_TYPICAL = 0) (
  input  logic clk, rst_n,
  input  logic        sample,
  input  logic [15:0] d0_ns, d1_ns, d2_ns, d3_ns,
  output logic [15:0] best_ns, worst_ns, typical_ns, spread_ns,
  output logic [15:0] plan_ns,
  output logic [7:0]  spread_pct,
  output logic        plan_too_optimistic_err
);
  // ... best_ns and worst_ns selected by comparator tree, omitted for length
  assign sum_q = {16'd0, d0_ns} + {16'd0, d1_ns} + {16'd0, d2_ns} + {16'd0, d3_ns};
  assign typical_ns = sum_q[17:2];              // the mean of four, no divider
  assign spread_ns  = worst_ns - best_ns;
  // A capacity plan must be made against the worst device that will be deployed,
  // not the typical one. The typical build plans against the mean.
  assign plan_ns = (USE_TYPICAL != 0) ? typical_ns : worst_ns;
  // Sixteen bits: a spread can exceed 100 percent of the best device.
  assign pct_q = (best_ns == 16'd0) ? 32'd0
                                    : (({16'd0, spread_ns} * 32'd100) / {16'd0, best_ns});
  assign spread_pct = (pct_q > 32'd255) ? 8'hFF : pct_q[7:0];
  // Planning against a number some deployed device will not meet.
  assign plan_too_optimistic_err = sample && (plan_ns < worst_ns);
endmodule

Three populations of four devices:

The third population's typical device is 340 and its worst is 700. A plan built on 340 is a plan that 25% of the deployed fleet will not meet, and the typical figure is not wrong — it is the correct mean of a distribution with an outlier in it.

The typical_ns computation is sum_q[17:2], a two-bit right shift rather than a divider, which is the standard trick for a mean of four and is worth naming because it only works for powers of two.

spread_pct needed more than eight bits during development for the same reason 17.1 section 18's capacity_gain_pct did: a spread is not a share. 250% does not fit in a percentage anybody assumed was bounded at 100, and the third population exists in the bench specifically to drive it there.

The second population — four identical devices — is what makes the typical build's error visible as a conditional rather than a constant. With no spread, planning against the mean is planning against the worst, and plan_too_optimistic_err correctly goes quiet. A checker that fires on a uniform population would be firing on the case where the shortcut is safe.

13. RTL 8 — What A Deployed Device Tells You

// What a deployed device reports about itself, and the gap between a counter that
// exists and a counter that is read.
module field_telemetry #(parameter int NO_READBACK = 0) (
  input  logic clk, rst_n,
  input  logic        poll,
  input  logic [15:0] corrected, uncorrected, throttle_cyc, total_cyc,
  output logic [15:0] visible_corrected, visible_uncorrected, visible_throttle,
  output logic [15:0] throttle_pct, true_throttle_pct,
  output logic        health_ok, degraded,
  output logic [7:0]  n_polls, n_degraded_seen,
  output logic        blind_health_err
);
  // The no-readback build reports zeroes for every counter it never reads, which
  // is indistinguishable from a device with nothing to report.
  assign visible_corrected   = (NO_READBACK != 0) ? 16'd0 : corrected;
  assign visible_uncorrected = (NO_READBACK != 0) ? 16'd0 : uncorrected;
  assign visible_throttle    = (NO_READBACK != 0) ? 16'd0 : throttle_cyc;
  assign tp_q = (total_cyc == 16'd0) ? 32'd0
              : (({16'd0, visible_throttle} * 32'd100) / {16'd0, total_cyc});
  assign tt_q = (total_cyc == 16'd0) ? 32'd0
              : (({16'd0, throttle_cyc} * 32'd100) / {16'd0, total_cyc});
  assign throttle_pct      = tp_q[15:0];   // what the operator sees
  assign true_throttle_pct = tt_q[15:0];   // what is actually happening
  assign degraded  = (visible_uncorrected != 16'd0) || (throttle_pct > 16'd25);
  assign health_ok = !degraded;
  // Reporting health while the device's real counters say otherwise.
  assign blind_health_err = poll && health_ok
                            && ((uncorrected != 16'd0) || (true_throttle_pct > 16'd25));
  // ... poll counters omitted for length
endmodule

  telemetry: polls=4 degraded_seen=3 blind=0 | no-readback degraded_seen=0 blind=3

Three degraded polls seen, and zero. The device is the same device in both columns — uncorrected and throttle_cyc are the same inputs. The difference is entirely whether anybody read them.

The two throttle percentages are the model's whole argument. throttle_pct is what the operator sees and true_throttle_pct is what is happening, and in the no-readback build they are 0 and 40. A fleet dashboard built on the first number shows a green estate.

Two degradation conditions, each driven alone:

• One uncorrected error is a degradation regardless of throttling — 17.1 section 10's distinction, arriving as an operational signal.
• Forty percent throttled with zero errors is a degradation too, and it is the one an error-focused monitor misses entirely.

The threshold is driven exactly: 25% is not degraded, 26% is. A monitor whose threshold is off by one reports a different estate than the one anybody agreed on.

14. RTL 9 — Capacity Per Unit Cost Is Not The Number

// The cost model: capacity per dollar is not the number, because a device that
// cannot be populated to its slot count does not deliver its capacity.
module cost_model (
  input  logic clk, rst_n,
  input  logic        price,
  input  logic [15:0] gb_per_dev, cost_per_dev,
  input  logic [7:0]  populated, nominal_slots,
  output logic [31:0] nominal_gb, actual_gb, total_cost,
  output logic [15:0] gb_per_unit_cost,
  output logic [7:0]  shortfall_pct,
  output logic        nominal_claim_err
);
  assign nominal_gb = {16'd0, gb_per_dev} * {24'd0, nominal_slots};
  assign actual_gb  = {16'd0, gb_per_dev} * {24'd0, populated};
  assign total_cost = {16'd0, cost_per_dev} * {24'd0, populated};
  assign gpc_q = (total_cost == 32'd0) ? 32'd0 : (actual_gb / total_cost);
  assign gb_per_unit_cost = gpc_q[15:0];
  // The shortfall between the capacity a slot count implies and the capacity the
  // power and address budgets actually allow.
  assign sf_q = (nominal_gb == 32'd0) ? 32'd0
                                      : (((nominal_gb - actual_gb) * 32'd100) / nominal_gb);
  assign shortfall_pct = (sf_q > 32'd255) ? 8'hFF : sf_q[7:0];
  // Quoting the nominal capacity when the achievable population is lower.
  assign nominal_claim_err = price && (populated < nominal_slots);
endmodule

  cost: nominal=4096GB actual=4096GB shortfall=0% gb_per_cost=128

Eight slots fully populated deliver the nominal 4096 GB. Then the power budget from section 10 limits the population to four:

• Actual capacity: 2048 GB
• Shortfall: 50%
• nominal_claim_err: asserted

The cost per gigabyte did not change. Four devices at cost 4 each deliver 2048 GB, which is the same 128 GB per unit of cost as eight devices delivering 4096. What changed is the capacity the machine has, and a procurement decision made on cost-per-gigabyte alone would see no difference between the two situations.

total_cost is charged on populated, not nominal_slots — you do not buy the devices that will not fit. The mutation that charges for all eight is the more pessimistic error and is still an error.

15. RTL 10 — The Evaluation Assembled

// The evaluation assembled: from a claim on a page to a device in production.
module device_evaluation #(parameter int TRUST_DATASHEET = 0) (
  input  logic clk, rst_n,
  input  logic       decide,
  input  logic       claims_checked,   // the measurable claims were measured
  input  logic       slot_fits,        // power, airflow and height all fit
  input  logic       gen_adequate,     // the negotiated generation does what is needed
  input  logic       qualified,        // it passed the sustained tests
  input  logic       telemetry_ok,     // it reports what operations needs
  output logic       deploy,
  output logic [4:0] fail_mask,
  output logic [7:0] n_decisions, n_deployed,
  output logic       unqualified_deploy_err
);
  assign fail_mask[0] = ~claims_checked;
  assign fail_mask[1] = ~slot_fits;
  assign fail_mask[2] = ~gen_adequate;
  assign fail_mask[3] = ~qualified;
  assign fail_mask[4] = ~telemetry_ok;
  // The trusting build deploys on the datasheet, skipping the two gates that
  // require the device to be in the building.
  assign deploy = (TRUST_DATASHEET != 0) ? (slot_fits && gen_adequate)
                                         : (fail_mask == 5'd0);
  assign unqualified_deploy_err = decide && deploy && (fail_mask != 5'd0);
  // ... decision counters omitted for length
endmodule

Six decisions — all gates passing, then each falsified alone:

  evaluation: decisions=6 deployed=1 | datasheet-trusting deployed=4

One deployment out of six against four. The trusting build's three extra deployments are precisely the three gates that require the device to be physically present:

That is the shape of the whole chapter in one table. Everything a datasheet can tell you, the datasheet-trusting evaluation gets right. Everything that requires the device on a bench, it gets wrong — and it deploys four devices where one was justified.

16. Quantitative Reasoning

Every number is from a printed line above. None describes any shipping product.

Claims. Four recorded, one measured. The trusting evaluation assumed three and its measured count is identical.

Slot fit. Five devices, two fit. The power-only check passed four — two of them failing budgets it does not look at.

Sustained load. 4 × 400 + 6 × 150 = 2500 against a no-throttle 4000. A 37.5% shortfall, invisible for the first four cycles.

Generation. Four links, two degraded. A 3.0 host with a 2.0 device loses sharing; with a 1.1 device it loses switching and pooling too.

Population. ⌊800/75⌋ = 10 by power, 8 by slots, ⌊4096/512⌋ = 8 by address. min = 8. Drop the power to 300 W and it is 4; cut the address space to 1024 GB and it is 2.

Qualification. Seven runs, one pass. The build skipping the sustained test passed two.

Vendor spread. 200/220/240/700 gives a typical of 340 and a worst of 700 — a spread of 250% of the best device, on a population whose mean is unremarkable.

Telemetry. Four polls, three degraded — and zero seen by a monitor that does not read the counters, on identical device state.

Cost. 4096 GB nominal, 2048 achievable at four devices, a 50% shortfall — at an unchanged 128 GB per unit of cost.

Deployment. Six decisions, one deployment against the trusting build's four.

17. Assertions

Every property is an immediate check written as cond !== 1'b1, sampled after a settle. 171 assertion sites across two testbenches.

18. Mutation Testing

82 mutations, one at a time, each required to make the baseline print RESULT: FAIL.

82 of 82 were killed.

The first run killed 80 and left 2 survivors:

"Switching needs 3.0" survived because the bench checked switching_ok only where it was false. Three link-ups ran at generations where switching is available and none of them asserted it, so a mutation that raises the requirement went unnoticed. This is the same shape as 17.2's "a checker asserted only where it should fire is half-tested", applied to a capability rather than an error.

"Lost generations unguarded" was provably equivalent: run_gen is the minimum of host and device, so run_gen <= host_gen always holds and the guard on the subtraction can never change the result. The guard stays — it documents that the subtraction is unsigned and must not wrap — and the mutation was replaced with one that measures degradation against the device instead of the host.

A representative sample:

19. Verification Strategy

Two builds, one stimulus. The parameter is the only difference, so any divergence is attributable to one line.

Assert a capability where it holds, not only where it fails. The one real survivor came from checking switching_ok only at the generation where it was unavailable.

Drive every boundary exactly. A device at precisely the slot's power budget. Exactly the throttle temperature and one degree below. Exactly 25% throttled and 26%. Exactly the nominal population and one short.

Include the population where the shortcut is correct. Four identical devices make the typical-plan build right, and a checker that fired there would be firing on the safe case.

Include the configuration where the two builds agree. A 1.1 host with a 3.0 device: the assuming build gets the right answer, and only that row proves the negotiation takes the lower rather than the host's.

Assert exact values, so a model inconsistency surfaces as a failure. The telemetry gating bug appeared as four failed assertions with specific expected numbers. A bench asserting "degraded is plausible" would have passed.

20. Synthesis and Implementation Reality

None of this is hardware. These are executable specifications for an evaluation process, and their value is that the process has boundaries a spreadsheet does not force you to think about.

The slot budgets are not independent in practice. Airflow available to a slot depends on what is in the neighbouring slots, so slot_airflow is a function of the population rather than a constant — which means section 6 and section 10 are coupled, and a device that fits in isolation may not fit in a full chassis.

The sustained test is the expensive one, and that is why it gets cut. It needs the device under load for long enough to reach thermal steady state, in a chassis representative of production, with a workload representative of production. Every one of those three is negotiable under schedule pressure, and each negotiation moves the measured number.

Vendor spread needs a sample, and four is not one. Section 12 uses four devices because four is enough to demonstrate that a spread exists. Characterising a spread is a statistical exercise with a sample size, and the worst-case planning rule is what makes a small sample safe rather than a large one necessary.

Telemetry counters are frequently present and rarely read. The NO_READBACK build is not a hypothetical: it is the common case, where a device exposes registers that no monitoring agent polls. The gap this chapter models is between the device's capability and the fleet's use of it.

21. Silicon Observability

What a fleet needs from a deployed device, and where each requirement comes from.

The last row is the cheapest and most frequently missing. A host that records what generation it asked for rather than what the link settled on will report a fleet of 3.0 links, some of which are running at 2.0 with a device that never supported more — and the capability difference shows up later as a feature that does not work on some machines.

22. Debug Lab

Symptom: a device that benchmarked well in evaluation performs poorly in production.

Compare the benchmark duration against thermal steady state. If the benchmark was shorter, section 7's four-cycle window is what was measured. Re-run it long enough to reach steady state and compare.

Check the negotiated generation, not the configured one. A link running a generation below what was planned loses capabilities silently, and section 9's table says which ones.

Read the throttle counter. If it is high and nobody was reading it, the device has been telling you the answer since deployment — section 13's blind health report.

Compare this device against its siblings. If one device in a population is markedly worse, the evaluation measured a good sample and production received the spread — section 12. If they are all equally poor, the evaluation measured the wrong thing.

Check the population against the three ceilings. A machine with fewer devices than planned delivers proportionally less capacity, and section 10's binding says which budget to fix.

23. Design Review

Questions to ask of any device evaluation, each answerable from a model in this chapter.

Which datasheet claims did you measure, and which did you accept? If the evaluation reports one number for "claims verified", it cannot answer this.

How long was the sustained test, and did the device reach thermal steady state?

What generation did the link actually negotiate?

How many devices fit, and which of the three ceilings binds?

How many devices did you measure, and what was the spread? One device is a sample of one, and the plan should be built on the worst of the sample, not the mean.

Which telemetry counters does the fleet actually poll? A counter that exists and is not read is a counter that does not exist.

24. How This Appears In Real Engineering

Evaluation failures surface months after the decision, as a fleet that does not behave like the sample.

The characteristic case is a device that met its numbers in the lab and misses them in a rack. Almost always the lab test was shorter, cooler, or less densely populated than production — section 7's burst window and section 20's coupling between airflow and neighbours.

The second is a capability that works on some machines and not others. That is section 9: links that negotiated down, on hosts that recorded what they configured rather than what they got.

The third is a capacity plan that comes up short. Slots were counted, watts were not, and section 10's binding ceiling was a budget nobody was asked about.

The fourth is a fleet-wide monitoring gap discovered during an incident: the devices had been reporting degradation for months into a monitor that was not polling those registers.

25. Common Misconceptions

"The datasheet is the specification." It is a set of claims of three different kinds, and an evaluation's job is to sort them and measure the first kind.

"It fits the slot." It fits the slot's power budget. Airflow and height are separate, and section 6's power-only check passed two devices that fail one of the other two.

"We benchmarked it." For how long, at what temperature, in what chassis? Section 7's device delivers its rated number for four cycles and 37.5% less over ten.

"The link is up, so we have CXL 3.0." The link is up at whatever both ends support. A 2.0 device gives you a 2.0 link and no memory sharing, on a link that reports up.

"We have eight slots so we can fit eight devices." Only if power and address space also allow eight. The population is the minimum of three ceilings and the binding one is often not the one that was counted.

"We measured a device from this vendor." You measured a device. Section 12's third population has a typical of 340 and a worst of 700, and both numbers are honest descriptions of the same four devices.

"The device reports healthy." The monitor reports healthy. Section 13's two builds have identical device state and report three degradations and zero.

26. Interview Reasoning

Q1. How do you classify a datasheet claim? By whether it can be measured: directly, only under conditions you may not reproduce, or not at all. The third kind enters the decision as a risk, never as a fact.

Q2. Why separate "measured" from "accepted" in an evaluation? Because the two counts differ, and a single "verified" number reports the larger one. In section 5 both evaluations measured exactly one claim; only one of them also accepted three it had not.

Q3. A device fits the slot's power budget. What else do you check? Airflow and height, independently. Section 6's power-only check passed four of five devices and two of those failures would have been discovered in a rack.

Q4. Why are the slot comparisons strict rather than inclusive? Because a device drawing exactly the rated power fits. An inclusive comparison rejects every device designed precisely to the specification.

Q5. Your benchmark shows 400 units of bandwidth. What is your next question? For how long. Section 7's device delivers 400 for four cycles and 150 afterwards, and a short benchmark measures the first number correctly.

Q6. A 3.0 host and a 2.0 device. What generation runs, and what is lost? 2.0, and memory sharing between hosts. Switching and pooling survive. The link reports up either way.

Q7. A 1.1 host and a 3.0 device — what is lost? Nothing, relative to the host. The host was never going to do more, which is why degradation is measured against the host and not against the device.

Q8. Eight slots, 800W of host budget, 75W devices, 4TB of address space, 512GB devices. How many fit? Eight — bound by slots and address space equally, at ten by power. Halve the host power and it is four, bound by watts.

Q9. Why report which ceiling binds, not just the number? Because "four devices" sends somebody to buy slots. "Four devices, bound by power" sends them to the only place the number can change.

Q10. Which qualification check gets skipped, and why that one? The sustained test. It is the most expensive to run and the only one whose absence is invisible in every other result.

Q11. Four devices measure 200, 220, 240 and 700ns. What do you plan against? 700. The mean of 340 is an honest number that a quarter of the fleet will not meet.

Q12. When is planning against the typical device safe? When the spread is zero. Section 12's second population is four identical devices, and there the mean is the worst case.

Q13. A monitor reports a healthy fleet. What would make you doubt it? Whether it polls the counters. Section 13's two builds see identical devices and report three degradations and zero.

Q14. Is 40 percent throttled with no errors a problem? Yes, and it is the one an error-focused monitor misses. It is a degradation with no error attached to it.

Q15. Population drops from eight to four. Does cost per gigabyte change? No — 128GB per unit either way. The capacity halves, which is why cost per gigabyte alone cannot see the difference.

Q16. Which deployment gates cannot be answered from a datasheet? Claims measured, sustained qualification, and telemetry adequacy. Those three are precisely the ones a datasheet-trusting evaluation skips, and it deploys four devices where one was justified.

27. Exercises

1. Add a fourth claim class to RTL 1 for a claim that is measurable but only destructively. Where does it belong relative to "unverifiable", and what does an evaluation do with it?

2. Make slot_airflow in RTL 2 a function of how many neighbouring slots are populated. Which of the existing assertions become conditional?

3. In RTL 3, model a device that throttles in two stages rather than one. What does a benchmark that stops between the two stages report?

4. Extend RTL 4 so a device can support a generation partially — the link speed but not the feature set. Does run_gen still capture what the fabric can do?

5. Add a fourth ceiling to RTL 5 for available link bandwidth on the host. At what per-device bandwidth does it become binding before power does?

6. RTL 6's checks are unordered. Order them by cost and add early exit. Which check should run first, and does the failure mask still mean the same thing?

7. RTL 7 samples four devices. Derive the sample size at which the observed worst is within 10 percent of the population worst, and state the assumption that derivation requires.

8. In RTL 8, add a counter that the device exposes but the monitor polls only hourly. What class of degradation becomes invisible, and how would you detect it?

28. Summary

Evaluating a real device is the process of converting claims into measurements, and it has eight parts.

Claims come in three kinds and only one of them a bench can settle. Four claims recorded, one measured — and the trusting evaluation measured exactly as many while accepting three more.

A slot is three budgets. Five devices, two fit, and the power-only check passed four.

Burst is not sustained. 2500 delivered against a nominal 4000, with the first four cycles identical in both.

Generation is negotiated and the lower end wins. Two of four links degraded, and the capability lost is memory sharing on a link that reports up.

Population is the minimum of three ceilings — eight, then four, then two, on a device that never changed.

Qualification's most expensive test is the one whose absence is invisible. Seven runs, one pass, two passes without the sustained test.

A class is not a device. A population whose typical is 340 ns and whose worst is 700, and a plan built on the first number that a quarter of the fleet will miss.

And a counter nobody reads is a counter that does not exist. Three degraded polls and zero, on identical device state.

Module 18 — CXL Performance takes a device that has passed all of this and asks the two questions the evaluation deferred: where the latency actually comes from, and how much bandwidth the link can really carry.