Memory-Access Cost

18.1 decomposed the hardware path and 18.2 found the binding ceiling on it. Both measured the flit.

This chapter measures the load instruction — everything between an application touching a pointer and the flit those two chapters modelled. On a warm, mapped, well-placed access the two are nearly the same number. On every other kind of access they are not close.

1. The Engineering Problem — The Hop Latency Is One Term Of Several

Six things sit between a load and the wire.

Where the access lands decides what it costs. A cache hit, a TLB hit, a page-table walk and a page fault are four different prices, and only the second is the one 18.1 modelled. Section 5.

The TLB has a reach, and past it every access walks. A working set larger than what the TLB can map does not degrade gradually — it changes which of section 5's four outcomes is the common one. Section 6.

A page fault is paid once, which makes it either irrelevant or the entire cost, depending on a number nobody measures: how many times the page is used afterwards. Section 7.

A page allocated on one node and used from another pays the far cost forever. The placement decision is made once, at first touch, usually by a thread that is not the one that will use it. Section 9.

One software read moves a whole cache line, so a strided access moves a line per element and the bytes on the wire are a multiple of the bytes the program asked for. Section 10.

And the API is a cost. For a small access the call can be more than everything else combined, and the transfer size at which that stops being true is computable. Section 13.

This chapter against 18.1, stated precisely. That one owns the path from the host's port to the device's media. This one owns the path from the instruction to the host's port, plus everything the operating system does on the way.

2. The One-Sentence Model

The hardware latency is a lower bound on the software-visible cost, and the gap between them is the operating system — every defect below is a model that reports the bound as the answer.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Four access outcomes, one TLB, two NUMA nodes and a single prefetch decision are coarser than any real system. They are sized so every boundary is reachable and every number recomputable on paper.

What is not simplified is the structure: a cost selected by outcome rather than assumed, a reach compared against a working set, a one-off cost amortised over a use count, a placement compared between allocation and use, a byte count multiplied by a granularity, and a timeliness test on a prefetch.

Three things are absent by design. There is no cache-hierarchy model — section 5's "cache hit" is a single outcome, where a real system has three or four levels with their own costs. There is no huge-page fragmentation model: section 6 shows that huge pages extend reach and says nothing about whether you can get them. And coherency is out of scope — a .cache access may involve snoops whose cost belongs to 13.1.

5. RTL 1 — Four Outcomes, Four Prices

// A software memory access is not one event. Where it lands decides what it
// costs, and only the second case is the one 18.1 modelled.
module access_path #(parameter int HW_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        access,
  input  logic [1:0]  outcome,      // 0 cache hit, 1 TLB hit, 2 TLB miss, 3 page fault
  input  logic [15:0] cache_ns, tlb_ns, walk_ns, fault_ns,
  output logic [15:0] this_ns,
  output logic [31:0] total_ns,
  output logic [15:0] n_acc, n_fault,
  output logic        undercount_err
);
  logic [15:0] true_ns;
  always_comb begin
    case (outcome)
      2'd0:    true_ns = cache_ns;
      2'd1:    true_ns = tlb_ns;
      2'd2:    true_ns = walk_ns;
      default: true_ns = fault_ns;
    endcase
  end
  // The hardware-only model charges the CXL access cost for everything and knows
  // nothing about walks or faults.
  assign this_ns = (HW_ONLY != 0) ? tlb_ns : true_ns;
  // Charging less than the access actually cost.
  assign undercount_err = access && (this_ns < true_ns);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      total_ns <= 32'd0; n_acc <= 16'd0; n_fault <= 16'd0;
    end else if (access) begin
      total_ns <= total_ns + {16'd0, this_ns};
      n_acc <= n_acc + 16'd1;
      if (outcome == 2'd3) n_fault <= n_fault + 16'd1;
    end
  end
endmodule

Five accesses — a cache hit, a TLB hit, two walks and a fault — at 4, 300, 900 and 12,000 ns:

  path: total=14104ns acc=5 faults=1 | hw-only total=1500ns undercounts=3

14,104 ns against 1,500 — a factor of 9.4, and the access count is 5 in both models. The hardware-only model is exactly right on one of the five outcomes and wrong on three of the other four, in the same direction every time.

The four prices span three orders of magnitude, which is the whole reason the outcome has to be modelled rather than averaged:

Two walks are driven rather than one, deliberately. With exactly one walk and one fault, a model that counts walks as faults produces the same n_fault as one that counts faults — the mutation is invisible until the two populations differ in size.

6. RTL 2 — TLB Reach

// TLB reach: how much memory the TLB can map before every access walks.
module tlb_reach #(parameter int IGNORE_REACH = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] entries, page_kb, working_set_mb,
  output logic [31:0] reach_mb,
  output logic [7:0]  miss_pct,
  output logic        covered,
  output logic [7:0]  n_eval, n_thrash,
  output logic        reach_blind_err
);
  logic [31:0] ws_kb, mp_q;
  assign reach_mb = ({16'd0, entries} * {16'd0, page_kb}) / 32'd1024;
  assign ws_kb    = {16'd0, working_set_mb} * 32'd1024;
  assign covered  = (reach_mb >= {16'd0, working_set_mb});
  // The share of the working set the TLB cannot map. The ignoring build reports
  // zero misses whatever the working set is.
  assign mp_q = (IGNORE_REACH != 0) ? 32'd0
              : (covered ? 32'd0
                         : ((({16'd0, working_set_mb} - reach_mb) * 32'd100)
                            / {16'd0, working_set_mb}));
  assign miss_pct = (mp_q > 32'd255) ? 8'hFF : mp_q[7:0];
  // Reporting full coverage on a working set the reach does not cover.
  assign reach_blind_err = evaluate && !covered && (miss_pct == 8'd0);
  // ... evaluation counters omitted for length
endmodule

1536 entries against a 64 MB working set:

  tlb: reach=6MB covered=0 miss=90% | ignoring miss=0% blind=2

1536 four-kilobyte entries reach 6 MB. Against a 64 MB working set, 90% of it cannot be mapped, and every access to that 90% takes section 5's third path rather than its second — a 3× cost increase applied to nine accesses in ten.

The same 1536 entries backed by 2 MB pages reach 3072 MB, and the problem disappears entirely. That is the single largest lever in this chapter and it costs nothing on the wire: the memory is the same memory, at the same place, over the same link.

The coverage comparison is inclusive and driven exactly — a 6 MB reach covers a 6 MB working set and does not cover 7 MB, leaving 14% unmapped.

7. RTL 3 — A Fault Is Paid Once

// First touch: a page fault is paid once, and whether it matters depends
// entirely on how many times the page is used afterwards.
module first_touch #(parameter int IGNORE_FAULT = 0) (
  input  logic clk, rst_n,
  input  logic        touch, is_first,
  input  logic [15:0] fault_ns, steady_ns, uses,
  output logic [31:0] total_ns, amortised_ns,
  output logic [15:0] this_ns,
  output logic [7:0]  fault_share_pct,
  output logic        fault_dominates, hidden_fault_err
);
  logic [31:0] amt_q, sh_q;
  // The ignoring build charges the steady cost for the first touch too.
  assign this_ns = (IGNORE_FAULT != 0) ? steady_ns : (is_first ? fault_ns : steady_ns);
  // Cost per use once the one-off fault is spread over every use of the page.
  assign amt_q = (uses == 16'd0) ? 32'd0
               : (({16'd0, fault_ns} + ({16'd0, steady_ns} * {16'd0, uses}))
                  / {16'd0, uses});
  assign amortised_ns = amt_q;
  assign sh_q = (amortised_ns == 32'd0) ? 32'd0
              : (((amortised_ns - {16'd0, steady_ns}) * 32'd100) / amortised_ns);
  assign fault_share_pct = (sh_q > 32'd255) ? 8'hFF : sh_q[7:0];
  assign fault_dominates = (fault_share_pct > 8'd50);
  // Charging a first touch as though it were a steady-state access.
  assign hidden_fault_err = touch && is_first && (this_ns == steady_ns);
  // ... total omitted for length
endmodule

A 12,000 ns fault against a 300 ns steady access:

  fault: amortised=312ns share=3% dominates=0 | ignoring hid=1

Forty uses is the break-even. Below it the fault is the majority of what an access costs; above it, the steady-state number is a fair description. The dominance test is strict at 50 and the bench constructs exactly that case, because 40 uses is the only use count where the two comparisons differ.

This is the same amortisation argument as 17.3 section 13 — a one-off cost repaid out of repeated use — arriving here as a page fault rather than a migration. The structure is identical and so is the failure: a system that never measures uses-after-fault cannot tell a cheap fault from an expensive one.

The outer loop is the failure mode section 7 cannot express with a fixed use count. A page faulted in, used a handful of times, reclaimed under pressure and faulted again pays the 12,000 ns repeatedly and never amortises it. The steady-state number for such a page is not 312 ns and not 12,300 — it is whatever the reclaim rate makes it, and exercise 3 asks for exactly that model.

8. Waveform — Five Accesses, Four Different Prices

Transcribed from the printed trace. One stimulus stream, both builds.

The hw_ns row being flat is the point: it is the same 300 whatever the outcome row says, and the under row marks the three cycles where that is an undercount rather than an overcount.

9. RTL 4 — Where The Page Was Allocated

// A page allocated on one node and used from another pays the far cost on every
// access, for the life of the page.
module numa_placement #(parameter int FIRST_TOUCH_BLIND = 0) (
  input  logic clk, rst_n,
  input  logic        access,
  input  logic [1:0]  alloc_node, use_node,
  input  logic [15:0] local_ns, remote_ns,
  output logic [15:0] this_ns,
  output logic [31:0] total_ns,
  output logic [15:0] n_local, n_remote,
  output logic        is_remote, misplaced_err
);
  assign is_remote = (alloc_node != use_node);
  // The blind build assumes every page is local, which is what an allocator does
  // when it places on the allocating thread's node rather than the using one's.
  assign this_ns = (FIRST_TOUCH_BLIND != 0) ? local_ns
                                            : (is_remote ? remote_ns : local_ns);
  // Charging a remote access at the local rate.
  assign misplaced_err = access && is_remote && (this_ns == local_ns);
  // ... per-node counters omitted for length
endmodule

One local access and four remote, at 100 and 400 ns:

  numa: local=1 remote=4 total=1700ns | blind total=500ns misplaced=4

1700 ns against 500 — a factor of 3.4 on a placement decision made once, at allocation, by whichever thread touched the page first.

The blind build is not a strawman. Placing a page on the node of the thread that allocates it is a reasonable default and is right whenever the allocating and using threads are the same. The failure case is an initialisation thread that touches every page before handing the data to workers — at which point every page in the program is on one node and every worker but one is remote.

The remoteness test is a comparison the model can make and a real allocator, at the moment it must decide, cannot: it does not yet know which thread will use the page.

10. RTL 5 — One Read Moves A Line

// Amplification: one software read of N bytes moves a whole cache line, and a
// strided access moves a line per element.
module amplification #(parameter int ASSUME_DENSE = 0) (
  input  logic clk, rst_n,
  input  logic        read,
  input  logic [15:0] want_bytes, line_bytes, stride_bytes,
  output logic [31:0] moved_bytes,
  output logic [15:0] lines_touched, amp_x10,
  output logic        wasteful,
  output logic [15:0] n_reads,
  output logic        amp_blind_err
);
  logic [31:0] amp_q;
  logic [15:0] per_elem;
  // A stride at least as large as a line touches one line per element; a smaller
  // stride packs several elements into a line.
  assign per_elem = (stride_bytes >= line_bytes) ? line_bytes : stride_bytes;
  // Any access that moves a byte touches at least one line: integer division
  // alone floors a sub-line access to zero, which is a line count no access has.
  logic [15:0] lt_q;
  assign lt_q = (line_bytes == 16'd0) ? 16'd0 : ((want_bytes * per_elem) / line_bytes);
  assign lines_touched = ((lt_q == 16'd0) && (want_bytes != 16'd0)) ? 16'd1 : lt_q;
  // The dense assumption moves exactly what was asked for.
  assign moved_bytes = (ASSUME_DENSE != 0) ? {16'd0, want_bytes}
                                           : ({16'd0, want_bytes} * {16'd0, per_elem});
  assign amp_q = (want_bytes == 16'd0) ? 32'd0
               : ((moved_bytes * 32'd10) / {16'd0, want_bytes});
  assign amp_x10 = (amp_q > 32'd65535) ? 16'hFFFF : amp_q[15:0];
  assign wasteful = (amp_x10 > 16'd10);
  // Reporting no amplification on a strided access that has some.
  assign amp_blind_err = read && (stride_bytes > 16'd1) && (amp_x10 == 16'd10);
  // ... read counter omitted for length
endmodule

Eight elements against a 64-byte line:

The last row is the model's ceiling and it is real: a stride larger than a line still moves only a line per element. Beyond the line size, increasing the stride costs nothing extra on the wire, which is why the per_elem cap exists and why the mutation that removes it reports a 256× amplification that no memory system produces.

The first row is what makes the blindness check conditional. A unit-stride read has an amplification of exactly 1.0, and a model reporting 1.0 there is correct, not blind — which is why amp_blind_err requires stride > 1.

11. RTL 6 — Is The Hardware The Problem At All

// What fraction of the software-visible cost is the hardware path at all.
module cost_attribution #(parameter int BLAME_HARDWARE = 0) (
  input  logic clk, rst_n,
  input  logic        attribute,
  input  logic [15:0] hw_ns, tlb_ns, fault_ns, sw_ns,
  output logic [31:0] visible_ns,
  output logic [7:0]  hw_pct, sw_pct,
  output logic [1:0]  biggest_id,
  output logic [15:0] biggest_ns,
  output logic        hw_bound, misattribution_err
);
  logic [31:0] hp_q, sp_q;
  logic [15:0] a, b;
  assign visible_ns = {16'd0, hw_ns} + {16'd0, tlb_ns}
                    + {16'd0, fault_ns} + {16'd0, sw_ns};
  // ... percentage and largest-term selection omitted for length
  // The blaming build always names the hardware, whatever the split.
  assign hw_bound = (BLAME_HARDWARE != 0) ? 1'b1 : (biggest_id == 2'd0);
  // Calling an access hardware-bound when the hardware is not the largest term.
  assign misattribution_err = attribute && hw_bound && (biggest_id != 2'd0);
endmodule

  attribute: visible=500ns hw=60% sw=40% biggest=0 | blaming misattributed=2

Three splits, and the largest term moves across all three:

The second row is the one that matters. The hardware term did not change — it is the same 300 ns access — and it went from being 60% of the cost to 2% of it, because one page fault entered the sample. An investigation that starts from "the CXL access is slow" is starting from the 2%.

The bench moves the largest term to all three positions so the answer cannot be a fixed index, and the first row is the case where the blaming build is correct: when the hardware genuinely is the largest term, "hardware bound" is the right answer and misattribution_err correctly stays quiet.

12. RTL 7 — A Prefetch Has To Arrive In Time

// A prefetch that arrives in time removes the latency; one that does not costs
// bandwidth and removes nothing.
module prefetch_effect #(parameter int ASSUME_TIMELY = 0) (
  input  logic clk, rst_n,
  input  logic        access, prefetched,
  input  logic [15:0] issued_ahead_ns, need_ns, miss_ns, hit_ns,
  output logic [15:0] this_ns,
  output logic        timely, useless,
  output logic [15:0] n_pf, n_timely, n_wasted,
  output logic [31:0] wasted_bytes,
  input  logic [15:0] line_bytes,
  output logic        false_credit_err
);
  // A prefetch helps only if it was issued at least the access latency ahead.
  assign timely  = prefetched && (issued_ahead_ns >= need_ns);
  assign useless = prefetched && !timely;
  // The assuming build credits every prefetch with a hit.
  assign this_ns = (ASSUME_TIMELY != 0) ? (prefetched ? hit_ns : miss_ns)
                                        : (timely ? hit_ns : miss_ns);
  // Crediting a prefetch that could not have arrived in time.
  assign false_credit_err = access && useless && (this_ns == hit_ns);
  // ... prefetch counters omitted for length
endmodule

  prefetch: issued=5 timely=2 wasted=3 bytes=192 | assuming false credits=3

Three of five prefetches arrived too late to help, and each of them still moved a 64-byte line — 192 bytes of link bandwidth spent for nothing, on a link 18.2 has already established is a ceiling somebody is competing for.

The timeliness test is inclusive and driven exactly: issued 300 ns ahead of a 300 ns need is timely, and 299 ns ahead is not. A prefetch is not a probabilistic benefit here — it either had time or it did not, and the difference is one nanosecond.

The no-prefetch case is driven too, and it is neither timely nor useless. A model where useless means "not timely" reports every unprefetched access as a wasted prefetch, which turns the metric into a count of accesses.

13. RTL 8 — The API Is A Cost

// The API used to reach the memory is itself a cost, and for small accesses it
// can be the whole cost.
module api_overhead #(parameter int IGNORE_API = 0) (
  input  logic clk, rst_n,
  input  logic        call,
  input  logic [15:0] api_ns, per_byte_ns, bytes,
  output logic [31:0] payload_ns, total_ns, break_even_b,
  output logic [7:0]  api_share_pct,
  output logic        api_dominates, overhead_blind_err
);
  logic [31:0] sh_q;
  assign payload_ns = {16'd0, per_byte_ns} * {16'd0, bytes};
  // The ignoring build charges only the payload, which is right for a large
  // transfer and wrong for every small one.
  assign total_ns = (IGNORE_API != 0) ? payload_ns
                                      : (payload_ns + {16'd0, api_ns});
  // A build that does not charge the API reports no API share either: it has no
  // way to know there was one.
  assign sh_q = (IGNORE_API != 0) ? 32'd0
              : ((total_ns == 32'd0) ? 32'd0
                                     : (({16'd0, api_ns} * 32'd100) / total_ns));
  assign api_share_pct = (sh_q > 32'd255) ? 8'hFF : sh_q[7:0];
  assign api_dominates = (api_share_pct > 8'd50);
  // The transfer size at which the API cost equals the payload cost.
  assign break_even_b = (per_byte_ns == 16'd0) ? 32'd0
                      : ({16'd0, api_ns} / {16'd0, per_byte_ns});
  // Reporting no API share on a call that has one.
  assign overhead_blind_err = call && (api_ns != 16'd0) && (api_share_pct == 8'd0);
endmodule

A 2000 ns call at 2 ns per byte:

  api: payload=32000ns total=34000ns share=5% breakeven=1000B | ignoring blind=3

The break-even is 1000 bytes, and it is a single division: the call cost over the per-byte cost. Below it the API is the majority of what the operation costs; above it the payload is. That number is worth computing before choosing an access granularity and it is almost never computed.

The zero-cost-API case is driven and is not blindness: a free API genuinely has a zero share, and a checker that fired there would fire on every direct load. The api_ns != 0 term in the checker exists for exactly that case.

The ignoring build reports a zero API share and does not charge the API — the two are gated by the same parameter, because a model that does not charge a cost has no way to report its share. An earlier version gated only the charge and produced an ignoring build that somehow knew its own overhead, which four assertions caught.

14. RTL 9 — The Model Against The Measurement

// What the application measures against what the hop model predicts, and the
// ratio that says whether the model is usable.
module visible_vs_modelled (
  input  logic clk, rst_n,
  input  logic        compare,
  input  logic [15:0] modelled_ns, measured_ns,
  output logic [15:0] gap_ns,
  output logic [7:0]  error_pct,
  output logic        model_low, model_usable,
  output logic [7:0]  n_cmp, n_unusable,
  output logic        no_gap_err
);
  logic [31:0] ep_q;
  assign model_low = (modelled_ns < measured_ns);
  assign gap_ns    = model_low ? (measured_ns - modelled_ns)
                               : (modelled_ns - measured_ns);
  // Error as a share of what was actually measured, so a model half the truth
  // reports 50 rather than 100.
  assign ep_q = (measured_ns == 16'd0) ? 32'd0
              : (({16'd0, gap_ns} * 32'd100) / {16'd0, measured_ns});
  assign error_pct = (ep_q > 32'd255) ? 8'hFF : ep_q[7:0];
  assign model_usable = (error_pct <= 8'd20);
  // A model claiming to match exactly is a model that has not been compared.
  assign no_gap_err = compare && (modelled_ns == measured_ns) && (measured_ns != 16'd0);
  // ... comparison counters omitted for length
endmodule

  model-vs-measured: compares=7 unusable=3 gap=900ns error=75%

A 300 ns hop model against a 1200 ns measured access is 75% low — which is this whole chapter in one comparison. The gap is not noise; it is the four sections above it.

The bench drives both directions, because a model can be wrong high:

The gap_ns subtraction is guarded in both directions for the reason 18.1 section 14 established: an unsigned subtraction of a larger from a smaller wraps, and a model 300 ns high would otherwise report a 65,236 ns error.

no_gap_err is the model's oddest check and its most useful. An exact match between a model and a measurement is evidence the two were never independently obtained — and the empty case, where nothing was measured at all, is driven to prove the checker distinguishes "identical" from "no data".

15. RTL 10 — The Cost Model Assembled

// The software-visible cost model assembled: every term between the
// application's load instruction and the flit 18.1 measured.
module access_cost_model #(parameter int HW_PATH_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       hw_path,        // the hop latency of 18.1
  input  logic       tlb_modelled,   // walks are charged
  input  logic       faults_charged, // first touch is charged
  input  logic       placement_known,// remote pages cost the remote rate
  input  logic       api_charged,    // the access API is charged
  output logic       credible,
  output logic [4:0] fail_mask,
  output logic [7:0] n_eval, n_credible,
  output logic       hw_only_err
);
  assign fail_mask[0] = ~hw_path;
  assign fail_mask[1] = ~tlb_modelled;
  assign fail_mask[2] = ~faults_charged;
  assign fail_mask[3] = ~placement_known;
  assign fail_mask[4] = ~api_charged;
  // The hardware-path build knows the wire and nothing the operating system does.
  assign credible = (HW_PATH_ONLY != 0) ? hw_path : (fail_mask == 5'd0);
  assign hw_only_err = evaluate && credible && (fail_mask != 5'd0);
  // ... evaluation counters omitted for length
endmodule

  cost model: evaluated=6 credible=1 | hw-path-only credible=5

One credible model out of six, and the hardware-path build found five. That is the widest gap of any assembled model in this batch or the last, and the reason is structural: four of the five terms are things the operating system does, and a model built from the hardware specification has no way to see any of them.

17.4 section 15 found three gates a datasheet cannot answer and 18.1 section 15 found two terms a bench cannot see. This is the same shape at its widest: a model derived entirely from the interconnect specification passes one gate in five.

16. Quantitative Reasoning

Every number is from a printed line above. None describes any platform.

Outcomes. 4 + 300 + 900 + 900 + 12,000 = 14,104 ns over five accesses. The hop-only model: 5 × 300 = 1,500. A factor of 9.4, with identical access and fault counts.

TLB reach. 1536 × 4 KB = 6 MB; against 64 MB that is 90% unmapped. The same entries at 2 MB reach 3072 MB — a 512× increase for no change on the wire.

Fault amortisation. 12,000 ns spread over uses: 12,300 at one use, 700 at thirty, 600 at forty (exactly 50%), 420 at a hundred, 312 at a thousand.

NUMA. 100 + 4 × 400 = 1,700 ns against a blind 500. A factor of 3.4 from one placement decision.

Amplification. Eight elements: 1.0× contiguous, 8.0× at an 8-byte stride, 64.0× at 64 bytes — and still 64.0× at 256, because a line is the unit.

Attribution. The same 300 ns hardware term is 60% of a 500 ns access and 2% of a 12,500 ns one.

Prefetch. Five issued, two timely, three wasted, 192 bytes of link spent for nothing.

API. 2000 ns ÷ 2 ns per byte = 1000-byte break-even. At 64 bytes the call is 93%; at 16,000 it is 5%.

Model error. 300 modelled against 1200 measured is 75% low — and 75% of a real access is the four sections this chapter adds.

17. Assertions

Every property is an immediate check written as cond !== 1'b1, sampled after a settle. 185 assertion sites across two testbenches.

18. Mutation Testing

80 mutations, one at a time, each required to make the baseline print RESULT: FAIL.

80 of 80 were killed.

The first run killed 73 and left 5 survivors plus one stale anchor:

Three are worth stating individually.

"Walks counted as faults" survived on a stimulus with one of each. With exactly one walk and one fault, a model that counts the wrong one produces the same n_fault. The two populations had to differ in size before the counter could distinguish them — so the bench drives two walks against one fault.

The dominance boundary had to be solved for. fault_share > 50 against >= 50 differ on exactly one use count, and it is not a round number: 12,000 + 300u over u equals 600 only at u = 40. The representative values of 1, 100 and 1000 all step over it.

The provably-equivalent survivor was the stride cap at exactly the line size: at stride == line, >= and > both yield the line size, so the mutation cannot change any output. Replaced with one that removes the line-size division.

A representative sample:

19. Verification Strategy

Two builds, one stimulus. The parameter is the only difference.

Make the two populations differ in size before trusting a counter. One walk and one fault cannot distinguish a model that counts the wrong one.

Solve for the boundary. A 50% fault share exists only at 40 uses; a 20% model error only at 960 against 1200. Neither is reachable by adjusting a representative value.

Assert every output, including the ones that feel like bookkeeping. lines_touched was unobserved and wrong.

Drive the case where the shortcut is correct. A TLB hit, where the hop-only model is right. A unit-stride read, where the dense model is right. A free API, where a zero share is honest. A local access, where the blind allocator is right. In each case a checker that fired would fire on the common case.

Drive both directions of every comparison. A model low and a model high; a covered working set and an uncovered one.

Drive the empty case for every ratio. A zero measurement, a zero API cost, a zero use count — each divides by something the representative stimulus never makes zero.

20. Synthesis and Implementation Reality

None of this is hardware. These are executable specifications for a cost model. What is real is the instrumentation: hardware performance counters for TLB misses and page walks, kernel counters for faults, and NUMA statistics per node.

The four outcomes of section 5 are not directly observable per access. A real system samples: a PMU counts walks over an interval, the kernel counts faults, and the outcome mix is inferred. The mix is what the model needs, and it is available; the per-access outcome is not.

TLB reach is a design-time number and the working set is not. Section 6's comparison is easy to compute and hard to apply, because the working set is a property of the workload at a moment. Huge pages change the reach by two orders of magnitude and are the only lever that does not require changing the program.

First-touch placement is decided before the information needed to decide it exists. Section 9's allocator must place a page when it is first touched, and the thread that will use it may not have started. Every real policy is a guess, and the interesting question is what it costs when wrong — which is section 9's 3.4×.

Prefetch distance is the only tunable in section 12 and it is workload-specific. Issue too early and the line is evicted before use; too late and it is section 12's wasted 192 bytes. The correct distance is the access latency, which 18.1 showed is load-dependent — so the correct prefetch distance changes with utilisation.

21. Silicon Observability

The third and sixth rows are the ones systems lack. Fault counts are universal; uses-after-fault is not measured anywhere, which is why section 7's break-even of 40 is a number nobody can evaluate against their own workload. Similarly, prefetch issue counts are common and evicted-before-use counts are rare, which makes section 12's wasted 192 bytes invisible in practice.

22. Debug Lab

Symptom: an application on CXL memory is far slower than the link latency predicts.

Check the page-walk rate first. Section 6: if the working set exceeds TLB reach, a large fraction of accesses cost 3× the hop. Huge pages are the cheapest available fix and change nothing on the wire.

Check the fault rate, and whether it is still happening. Section 7: faults during a warm-up phase are amortised; faults in steady state are not. If the rate is not falling, pages are being reclaimed and re-faulted.

Compare allocation node against use node. Section 9: a 3.4× penalty on every access to a misplaced page, decided once, invisible afterwards.

Compare bytes moved against bytes requested. Section 10: a strided access at 64× amplification is using 64 times the bandwidth 18.2 budgeted.

Compute the attribution split before optimising anything. Section 11: the same hardware term is 60% of one access and 2% of another. If the fault term dominates, tuning the interconnect is work on the 2%.

Check the access granularity against the API break-even. Section 13: below 1000 bytes the call dominates, and batching is worth more than anything else on this list.

23. Design Review

What fraction of your accesses take each of section 5's four paths? If the answer is unknown, no latency number for this system means anything.

What is your TLB reach, and what is your working set? And are you using huge pages?

How many times is a page used after it is faulted in? Section 7's break-even is 40; if nobody knows the number, the fault cost is unbounded.

Which thread first-touches your data, and which threads use it?

What is your access stride against the line size?

What does one call of your access API cost, and what is the break-even transfer size?

When you say the access is slow, what fraction of it is the interconnect?

24. How This Appears In Real Engineering

Access-cost problems arrive as "CXL is slow", supported by a measurement that is correct and an attribution that is not.

The characteristic case is a first-run benchmark. Every page faults, every translation misses, and the measured access cost is section 5's fourth path throughout. Run it twice and the number changes by an order of magnitude, which is the tell.

The second is a working set that crossed the TLB reach. Nothing changed except size, performance fell sharply rather than gradually, and section 6's cliff is the explanation.

The third is an initialisation thread. Everything is allocated on one node by a setup phase, the workers are elsewhere, and every access in the program pays section 9's remote cost for the life of the process.

The fourth is a small-access API. The transfer size is below section 13's break-even and the program is paying more for the call than for the data — usually fixed by batching rather than by anything about the memory.

25. Common Misconceptions

"CXL memory access takes about 300ns." That is section 5's second outcome. The other three are 4, 900 and 12,000, and which one you get is an operating-system question.

"We measured the latency." On a warm run or a cold one? Section 5's total moves by 9.4× depending on the outcome mix.

"The TLB is a cache, so it degrades gradually." Its reach is a cliff. A working set inside it produces zero walks and one outside it produces walks proportional to the overflow.

"Page faults are noise." At one use per page a fault is 98% of the access cost. At a thousand uses it is 3%. Nothing about the fault changed.

"The memory is on CXL, so it is all equally far." Not if the allocation node and the use node differ. Section 9 is a 3.4× penalty that has nothing to do with CXL.

"We only read 8 bytes." You moved a line. At a 64-byte stride you moved 64 bytes per 8 requested, and 18.2's bandwidth ceiling applies to the 64.

"Prefetching helps." If it arrives in time. Three of five in section 12 did not, and each still spent a line of bandwidth.

"The API overhead is negligible." Below 1000 bytes it is the majority. The break-even is one division and almost nobody performs it.

26. Interview Reasoning

Q1. Someone says a CXL access takes 300ns. What do you ask? Which of the four outcomes. A cache hit is 4, a TLB hit is 300, a walk is 900 and a fault is 12,000 — and the mix is a property of the operating system, not the link.

Q2. A benchmark is 10x slower on its first run. What happened? Every page faulted and every translation missed. Section 5's fourth path throughout, amortised to nothing on the second run.

Q3. Your working set grows by 10 percent and performance falls by half. Why? It crossed the TLB reach. Section 6 is a cliff: inside the reach there are no walks, outside it there are walks proportional to the overflow.

Q4. What is the cheapest fix for that? Huge pages. The same entry count at 2MB reaches 512 times as far, and nothing about the memory, the link or the program changes.

Q5. Is a page fault expensive? It depends entirely on uses-after-fault. At 40 uses it is exactly half the amortised cost; below that it is the majority and above it a rounding error.

Q6. Why is uses-after-fault rarely measured? Because the decision that needs it — whether to fault a page in — happens before it can be known. Fault counts are everywhere; use counts after the fault are almost nowhere.

Q7. Your initialisation thread touches every page. What have you done? Placed the entire dataset on one node. Every worker on another node now pays the remote rate on every access, for the life of the process.

Q8. You read 8 bytes with a 64-byte stride. How much did you move? 64 bytes per element — 64x amplification. The bandwidth ceiling applies to what moved, not to what you asked for.

Q9. And with a 256-byte stride? Still 64x. A line is the unit, and beyond the line size a larger stride costs nothing more on the wire.

Q10. A profile says the memory access is 300ns and the operation takes 12,500ns. Where do you look? Not at the interconnect. The hardware is 2 percent of that access, and section 11's largest term is the page fault.

Q11. Your prefetch hit rate is high and performance is unchanged. Explain. The prefetches are arriving too late to remove the latency while still spending the bandwidth. A hit rate counts arrivals, not timeliness.

Q12. How far ahead should a prefetch be issued? At least the access latency — which, per 18.1, grows with utilisation. So the correct distance is not a constant.

Q13. When does an access API stop mattering? Above the break-even, which is the call cost over the per-byte cost. At 2000ns and 2ns per byte, that is 1000 bytes.

Q14. A model predicts 300ns and you measure 1200. Is the model wrong? It is 75 percent low, and the gap is the four software terms it does not have. It is not wrong about the hardware; it is incomplete about the access.

Q15. A model matches your measurement exactly. Good? Suspicious. An exact match between an independent model and an independent measurement is evidence they were not independent.

Q16. Which terms of an access-cost model can a hardware specification give you? One of five. The hop latency. Walks, faults, placement and the API are all operating-system and runtime behaviour, which is why the hop-only build in section 15 passes one gate in five.

27. Exercises

1. Extend RTL 1 with a three-level cache, each level its own outcome. Which existing assertions survive unchanged, and what does n_acc now fail to distinguish?

2. In RTL 2, make page_kb a mix of 4KB and 2MB pages rather than one size. What is the reach, and at what huge-page fraction does the working set become covered?

3. RTL 3 amortises over a use count. Make the use count decay — pages reclaimed and re-faulted — and find the reclaim rate at which the fault never amortises.

4. Add a migration to RTL 4 that moves a misplaced page after N remote accesses. Using 17.3 section 7's break-even, find the N at which migration is worth it.

5. In RTL 5, model a gather with an irregular stride. Does the per_elem cap still hold, and what is the worst-case amplification?

6. RTL 7 tests timeliness against a fixed need. Feed it 18.1 section 9's utilisation-dependent latency and find the utilisation at which a fixed prefetch distance becomes useless.

7. In RTL 8, add a per-call cost that falls with batch size. At what batch size does the API stop dominating a 64-byte access?

8. Add a sixth term to RTL 10 for coherency traffic on .cache accesses. Is it knowable from the hardware specification, and where does it belong in the mask?

28. Summary

A load instruction costs the hop latency plus everything the operating system does, and this chapter builds all of it.

Four outcomes, four prices. 14,104 ns of real access time reported as 1,500 by a model that is right on one path in four.

The TLB has a reach and it is a cliff. 1536 entries reach 6 MB against a 64 MB working set — 90% unmapped — and the same entries on huge pages reach 3072 MB.

A fault is paid once, which makes it 98% of an access at one use and 3% at a thousand. The break-even is exactly 40.

Placement is decided once and paid forever. 1,700 ns against 500 from a decision made by whichever thread touched the page first.

One read moves a line. 64× amplification at a 64-byte stride, and still 64× at 256 because the line is the unit.

A prefetch has to arrive in time. Three of five did not, spending 192 bytes of the ceiling 18.2 measured and removing nothing.

And the API is a cost — 93% of a 64-byte access, 5% of a 16,000-byte one, with a break-even that is one division away.

A hop-only model passes one gate of five, because four of the five are things the operating system does.

18.4 — Fabric Scaling takes this access and asks what happens to it when the fabric between the host and the memory is not one switch but many.