CXL Latency Anatomy

Module 17 built memory devices and decided where data should live. Every one of those chapters deferred the same question: where does the time actually go?

17.1 section 8 said a link-only model misses the media. 17.3 section 11 reported a mean that hid a tail. 17.4 section 7 measured a device for four cycles. This chapter is the one that owns all three.

1. The Engineering Problem — A Latency Is Not A Number

Six things separate a usable latency model from a number somebody measured once.

A hop has two parts and only one of them is constant. The fixed cost is what a bench measures; the queueing cost is what production adds, and it is the part that varies. Section 5.

A path is a sum of named segments. A total with no attribution cannot answer the only question anybody asks a latency model — which part got worse. Section 6.

The tail is not the mean, and budgets are met at the tail. A mean of 218 ns on a distribution whose slow requests take 2000 is a correct number that passes a budget the system misses. Section 7.

Queueing does not grow linearly. At 50% utilisation the wait equals the service time; at 90% it is nine times it. A linear model reports 0.9× where the real answer is 9×. Section 9.

Where you measure decides what you attribute. A host-only measurement charges the device for every nanosecond between the host and the device, including the switch's. Section 11.

And a retried transaction pays for the whole path twice, plus the timeout. A model that charges every transaction one clean traversal reports an unchanged mean while the system degrades. Section 12.

This chapter against 18.2, stated precisely. This one owns how long one transaction takes. That one owns how many can be in flight at once. They meet at Little's law and are otherwise disjoint.

2. The One-Sentence Model

A latency number is only useful if it says which parts it is made of and under what load it was taken — and every defect below is a number missing one of those two.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Six path segments, three budget allocations, a two-population latency distribution and integer-percent utilisation are all coarser than a real analysis. They are sized so every result can be recomputed on paper and every boundary is reachable in a short simulation.

What is not simplified is the structure: a per-hop term that grows with occupancy, a path that names its segments, a tail distinct from a mean, a utilisation curve with the right shape, an attribution that depends on measurement placement, and a budget checked per segment rather than in total.

Three things are deliberately absent. There is no distribution model — section 7 uses two populations rather than a histogram, because the argument is that the mean and the tail differ, not what shape the tail has. There is no coherency traffic: a .cache transaction may involve snoops whose latency is a different chapter's. And there is no software cost — the syscall, the page fault, the TLB miss — which is 18.3's ground.

5. RTL 1 — A Hop Has Two Parts

// A hop has a fixed part and a part that depends on how busy it is.
module hop_latency #(parameter int FIXED_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        traverse,
  input  logic [15:0] fixed_ns, per_queue_ns,
  input  logic [7:0]  queue_depth,
  output logic [15:0] hop_ns, queue_ns,
  output logic [31:0] total_ns,
  output logic [15:0] n_hops,
  output logic        underestimate_err
);
  // The queueing component is the part that grows with load. A fixed-only model
  // reports the unloaded number for every measurement it ever takes.
  assign queue_ns = per_queue_ns * {8'd0, queue_depth};
  assign hop_ns   = (FIXED_ONLY != 0) ? fixed_ns : (fixed_ns + queue_ns);
  // Reporting less than the hop actually costs.
  assign underestimate_err = traverse && (hop_ns < (fixed_ns + queue_ns));
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      total_ns <= 32'd0; n_hops <= 16'd0;
    end else if (traverse) begin
      total_ns <= total_ns + {16'd0, hop_ns};
      n_hops   <= n_hops + 16'd1;
    end
  end
endmodule

Three traversals at queue depths 0, 5 and 10, with a 40 ns fixed cost and 12 ns per queued entry:

  hop: total=300ns hops=3 | fixed-only total=120ns underestimates=2

The first traversal is identical in both builds. An empty queue adds nothing, so 40 ns is the right answer and the fixed-only model gets it. The second and third are 100 and 160, and the fixed-only model reports 40 for both.

That is the shape of every unloaded measurement ever taken: correct at zero load and wrong everywhere else, by an amount that grows with exactly the thing production has and a bench does not.

n_hops is 3 in both builds — the same traversals happened. The divergence is entirely in what each one was charged, which is the third time in this batch that a broken model's counters are indistinguishable from a correct one's.

6. RTL 2 — A Path Is A Sum Of Named Segments

// A path is a sum of named hops, and naming them is what makes a regression
// attributable.
module path_latency #(parameter int LUMP_TOGETHER = 0) (
  input  logic clk, rst_n,
  input  logic        issue,
  input  logic [15:0] host_ns, link_out_ns, switch_ns, link_in_ns, device_ns, media_ns,
  output logic [31:0] path_ns,
  output logic [15:0] biggest_ns,
  output logic [2:0]  biggest_id,
  output logic [7:0]  n_paths,
  output logic        unattributable_err
);
  logic [15:0] a, b, c;
  assign path_ns = {16'd0, host_ns} + {16'd0, link_out_ns} + {16'd0, switch_ns}
                 + {16'd0, link_in_ns} + {16'd0, device_ns} + {16'd0, media_ns};
  // The largest segment, and which one it is. A lumped model has a total and no
  // way to say where it went.
  assign a = (host_ns    > link_out_ns) ? host_ns    : link_out_ns;
  assign b = (switch_ns  > link_in_ns)  ? switch_ns  : link_in_ns;
  assign c = (device_ns  > media_ns)    ? device_ns  : media_ns;
  assign biggest_ns = (LUMP_TOGETHER != 0) ? 16'd0
                    : ((a > b) ? ((a > c) ? a : c) : ((b > c) ? b : c));
  // ... biggest_id selected from biggest_ns, omitted for length
  // A total with no attribution.
  assign unattributable_err = issue && (biggest_id == 3'd7);
  // ... path counter omitted for length
endmodule

30 + 25 + 60 + 25 + 45 + 120:

  path: total=305ns biggest=120ns id=5 | lumped biggest=0 unattributable=0

Both models agree the path is 305 ns. The totals are identical and always will be — the lumped model is not arithmetically wrong. What it cannot do is answer which segment, and biggest_id == 7 is the model saying so explicitly rather than returning a plausible index.

The bench moves the largest segment three times — media, then switch, then host — so the answer is never a fixed index. A model that always returns "the media" is right on the representative case and useless on the one that matters, which is the case where something changed.

7. RTL 3 — The Tail Is Not The Mean

// The tail is not the mean, and a budget is met or missed at the tail.
module tail_latency #(parameter int MEAN_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        sample,
  input  logic [15:0] n_fast, n_slow,
  input  logic [15:0] fast_ns, slow_ns,
  input  logic [15:0] budget_ns,
  output logic [31:0] total_ns, n_total,
  output logic [15:0] mean_ns, tail_ns, headline_ns,
  output logic        within_budget, budget_lie_err
);
  logic [31:0] mean_q;
  assign n_total  = {16'd0, n_fast} + {16'd0, n_slow};
  assign total_ns = {16'd0, n_fast} * {16'd0, fast_ns}
                  + {16'd0, n_slow} * {16'd0, slow_ns};
  assign mean_q  = (n_total == 32'd0) ? 32'd0 : (total_ns / n_total);
  assign mean_ns = mean_q[15:0];
  // The tail is the slow population's latency whenever any request is slow.
  assign tail_ns = (n_slow != 16'd0) ? slow_ns : fast_ns;
  // A budget is a statement about the tail. Judging it on the mean passes
  // systems whose slow requests miss it by a wide margin.
  assign headline_ns   = (MEAN_ONLY != 0) ? mean_ns : tail_ns;
  assign within_budget = (headline_ns <= budget_ns);
  // Declaring a budget met while the tail exceeds it.
  assign budget_lie_err = sample && within_budget && (tail_ns > budget_ns);
endmodule

990 requests at 200 ns, 10 at 2000 ns, against a 500 ns budget:

  tail: n=1000 mean=218ns tail=2000ns budget_met=0 | mean-only met=1 lie=0

218 against 2000. One percent of requests take ten times the mean, and the mean-only model declares a 500 ns budget comfortably met on a system where one request in a hundred misses it by a factor of four.

The bench drives three cases and the second two are what make the model honest:

The second row is the case where the shortcut is safe: with no slow population the mean is the tail, and budget_lie_err correctly stays quiet. The third drives the budget boundary exactly — a tail of precisely 500 ns is within a 500 ns budget, and 501 is not.

This is 17.3 section 11's argument with the budget attached: there the mean hid a tail, here the mean passes a requirement the tail fails.

8. Waveform — Latency Rising With Load

Transcribed from the printed trace. One stimulus stream, both builds.

9. RTL 4 — Queueing Is Not Linear

// Queueing grows with utilisation, and it grows faster near the top.
module queue_delay #(parameter int LINEAR_MODEL = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [7:0]  util_pct,        // 0..100
  input  logic [15:0] service_ns,
  output logic [15:0] wait_ns,
  output logic [15:0] factor_x10,      // the multiplier, times ten
  output logic        saturated,
  output logic [7:0]  n_eval,
  output logic        unbounded_err
);
  logic [31:0] f_q, w_q;
  // M/M/1 shape: wait = service * u / (1 - u). At 90 percent that is nine times
  // the service time; a linear model reports 0.9 times it.
  assign saturated = (util_pct >= 8'd100);
  assign f_q = saturated ? 32'd65535
             : ((LINEAR_MODEL != 0) ? (({24'd0, util_pct} * 32'd10) / 32'd100)
                                    : (({24'd0, util_pct} * 32'd10) / (32'd100 - {24'd0, util_pct})));
  assign factor_x10 = (f_q > 32'd65535) ? 16'hFFFF : f_q[15:0];
  assign w_q = ({16'd0, service_ns} * {16'd0, factor_x10}) / 32'd10;
  assign wait_ns = (w_q > 32'd65535) ? 16'hFFFF : w_q[15:0];
  // A queue at or past full utilisation has no bounded wait.
  assign unbounded_err = evaluate && saturated;
  // ... evaluation counter omitted for length
endmodule

A 100 ns service time at four utilisations:

At 90% utilisation the linear model is an order of magnitude low, and at 99% it is two. The two models agree at zero and diverge without limit, which is the worst possible shape for a modelling error: it is accurate exactly where nothing matters.

The u / (1 - u) form is the standard M/M/1 waiting-time shape, and the multiplier is carried times ten so a single decimal survives integer arithmetic. saturated is checked inclusively at 100% and the bench drives 99 and 100 adjacent to prove it: at 99% the answer is large and finite, at 100% there is no answer at all.

10. RTL 5 — Every Switch Is A Hop

// Each switch on the path adds a hop, and the hops are not free.
module switch_hops #(parameter int IGNORE_HOPS = 0) (
  input  logic clk, rst_n,
  input  logic        route,
  input  logic [3:0]  n_switches,
  input  logic [15:0] per_switch_ns, base_ns,
  output logic [31:0] path_ns,
  output logic [15:0] switch_total_ns,
  output logic [7:0]  switch_share_pct,
  output logic [7:0]  n_routes,
  output logic        hop_blind_err
);
  logic [31:0] sh_q;
  assign switch_total_ns = (IGNORE_HOPS != 0) ? 16'd0
                                              : (per_switch_ns * {12'd0, n_switches});
  assign path_ns = {16'd0, base_ns} + {16'd0, switch_total_ns};
  // What fraction of the path the switches are. On a short base path with
  // several switches this is most of it.
  assign sh_q = (path_ns == 32'd0) ? 32'd0
                                   : (({16'd0, switch_total_ns} * 32'd100) / path_ns);
  assign switch_share_pct = (sh_q > 32'd255) ? 8'hFF : sh_q[7:0];
  // Routing through switches and charging nothing for them.
  assign hop_blind_err = route && (n_switches != 4'd0) && (switch_total_ns == 16'd0);
  // ... route counter omitted for length
endmodule

A 150 ns base path at 60 ns per switch:

At four switches the fabric is the majority of the access. This is 16.4 section 5's multi-switch path expressed as latency: adding a switch buys ports and costs a hop, and beyond a certain depth the hops dominate the thing they were added to reach.

hop_blind_err requires switches to actually be present — a zero-switch path charging nothing for switches is correct, not blind. The bench drives that case explicitly, because a checker that fires on a direct-attached device would fire on the most common configuration there is.

11. RTL 6 — Where You Measure Decides What You Blame

// Where you measure decides what you attribute.
module measurement_point #(parameter int MEASURE_AT_HOST = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] sw_queue_ns, dev_queue_ns, wire_ns, media_ns,
  output logic [15:0] observed_ns, attributed_to_device_ns, true_device_ns,
  output logic [15:0] misattributed_ns,
  output logic        misattribution_err
);
  // The host sees one number: everything past its own port. Split at the switch
  // and the queueing there stops being charged to the device.
  assign true_device_ns = dev_queue_ns + media_ns;
  assign observed_ns    = sw_queue_ns + wire_ns + true_device_ns;
  // A host-only measurement attributes everything it observed to the device.
  assign attributed_to_device_ns = (MEASURE_AT_HOST != 0) ? observed_ns : true_device_ns;
  assign misattributed_ns = attributed_to_device_ns - true_device_ns;
  // Charging the device for time it did not spend.
  assign misattribution_err = measure && (misattributed_ns != 16'd0);
endmodule

80 ns of switch queueing, 25 ns of wire, 30 ns of device queueing, 120 ns of media:

  measure: observed=255ns true_device=150ns | host-only attributed=255ns misattributed=105ns

The device spends 150 ns and gets charged 255. The extra 105 is switch queueing and wire time — 41% of the attributed figure, belonging to parts of the system the device does not control and cannot be improved by replacing it.

The consequence is the whole point: an investigation that measures at the host concludes the device is slow. Replacing the device improves 150 ns worth of a 255 ns problem at best, and if the switch queueing is the thing that grew, replacing the device changes nothing at all.

The bench drives the configuration where the two measurements agree — no switch queueing, no wire time — and confirms misattribution_err goes quiet. A host-only measurement is not wrong in general; it is wrong exactly to the extent that something sits between the host and the device.

12. RTL 7 — A Retry Pays Twice, Plus The Timeout

// A retried transaction pays the whole path again, plus the timeout.
module retry_cost #(parameter int IGNORE_RETRY = 0) (
  input  logic clk, rst_n,
  input  logic        complete, was_retried,
  input  logic [15:0] path_ns, timeout_ns,
  output logic [15:0] this_ns,
  output logic [31:0] total_ns, n_done, n_retried,
  output logic [15:0] mean_ns,
  output logic        retry_hidden_err
);
  logic [31:0] mean_q;
  // A retried transaction costs the timeout plus a second traversal. The
  // ignoring build charges every transaction one clean path.
  assign this_ns = (IGNORE_RETRY != 0) ? path_ns
                 : (was_retried ? (timeout_ns + path_ns + path_ns) : path_ns);
  assign mean_q  = (n_done == 32'd0) ? 32'd0 : (total_ns / n_done);
  assign mean_ns = mean_q[15:0];
  // Charging a retried transaction as though it had not been retried.
  assign retry_hidden_err = complete && was_retried && (this_ns == path_ns);
  // ... completion counters omitted for length
endmodule

Three clean completions at 300 ns and one retried, with a 2000 ns timeout:

  retry: total=3500ns done=4 mean=875ns | ignoring total=1200ns mean=300ns hidden=1

One retry in four transactions moves the mean from 300 to 875. The retried transaction cost 2600 ns — the timeout, the failed traversal, and the successful one — and a model that charges it 300 reports an unchanged mean on a system whose real mean has nearly tripled.

n_retried is 1 in both builds. The retry was counted. It was simply not charged, which is the same structure as section 5's hop count and 17.3 section 13's amortisation counters: the event is visible and its cost is not.

The timeout_ns + path_ns + path_ns form is deliberate rather than 2 * path_ns + timeout_ns, because the two mutations that survive a careless version — charging one traversal, and charging no timeout — are each a plausible simplification somebody would make on purpose.

13. RTL 8 — A Budget Is Per Segment

// A latency budget is spent across segments, and the first segment to exceed its
// allocation is the one to fix.
module latency_budget #(parameter int TOTAL_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        check,
  input  logic [15:0] seg0_ns, seg1_ns, seg2_ns,
  input  logic [15:0] alloc0_ns, alloc1_ns, alloc2_ns,
  output logic [31:0] spent_ns, budget_ns,
  output logic [2:0]  over_mask,
  output logic        in_budget, over_total,
  output logic [7:0]  n_checks, n_over,
  output logic        blind_overrun_err
);
  assign over_mask[0] = (seg0_ns > alloc0_ns);
  assign over_mask[1] = (seg1_ns > alloc1_ns);
  assign over_mask[2] = (seg2_ns > alloc2_ns);
  assign spent_ns  = {16'd0, seg0_ns}   + {16'd0, seg1_ns}   + {16'd0, seg2_ns};
  assign budget_ns = {16'd0, alloc0_ns} + {16'd0, alloc1_ns} + {16'd0, alloc2_ns};
  assign over_total = (spent_ns > budget_ns);
  // Judging only the total lets one segment overrun while another underruns,
  // which is a design that meets its budget and misses its requirement.
  assign in_budget = (TOTAL_ONLY != 0) ? ~over_total : (over_mask == 3'd0);
  // A segment over its allocation while the total is within budget.
  assign blind_overrun_err = check && in_budget && (over_mask != 3'd0);
  // ... check counters omitted for length
endmodule

Segments of 80, 120 and 150 against allocations of 100, 100 and 200:

  budget: spent=350ns budget=400ns over_mask=010 | total-only blind=2

350 spent against a 400 budget, and segment 1 is over its allocation. The total-only check passes it. The design meets its aggregate number by underspending elsewhere, and the segment that overran is the one whose requirement came from somewhere — a coherency deadline, a device timeout, a software assumption — that the aggregate does not represent.

The bench drives seven configurations, and three matter beyond the representative one:

• Every segment within: both checks pass, blind_overrun_err quiet.
• Spent exactly the budget with every segment exactly at its allocation: both pass, and the boundary is inclusive on both the segment and the total.
• Every segment over: both checks fail, so the total-only check is not merely permissive — it is correct whenever the aggregate is also blown.

n_over is 4 for the per-segment check and 2 for the total-only one across those seven, which is the size of the gap between the two policies on one small sample.

14. RTL 9 — Attributing A Regression

// A latency regression is attributed by comparing segments, not totals.
module regression_attrib (
  input  logic clk, rst_n,
  input  logic        compare,
  input  logic [15:0] before0, before1, before2,
  input  logic [15:0] after0, after1, after2,
  output logic [15:0] d0, d1, d2, worst_delta,
  output logic [1:0]  culprit,
  output logic        regressed,
  output logic [7:0]  n_compares, n_regressions,
  output logic        no_culprit_err
);
  assign d0 = (after0 > before0) ? (after0 - before0) : 16'd0;
  assign d1 = (after1 > before1) ? (after1 - before1) : 16'd0;
  assign d2 = (after2 > before2) ? (after2 - before2) : 16'd0;
  assign worst_delta = (d0 >= d1) ? ((d0 >= d2) ? d0 : d2)
                                  : ((d1 >= d2) ? d1 : d2);
  assign culprit = (worst_delta == d0) ? 2'd0 : ((worst_delta == d1) ? 2'd1 : 2'd2);
  assign regressed = (worst_delta != 16'd0);
  // A regression with no segment to blame means the segmentation is wrong.
  assign no_culprit_err = compare && regressed && (worst_delta == 16'd0);
  // ... comparison counters omitted for length
endmodule

  regression: compares=4 regressions=2 worst=0ns culprit=0

Four comparisons:

The third row is the one that makes the model correct rather than merely plausible. Segment 2 improved and its delta is zero, not negative — the guard on each subtraction is what stops an unsigned improvement from wrapping to 65,000-odd and becoming the largest "regression" on the chart.

That guard is not defensive decoration. The mutation that removes it from d0 is killed by the third comparison alone, and an unguarded version would attribute every regression to whichever segment improved most.

15. RTL 10 — The Latency Model Assembled

// The latency model assembled: every term a credible number needs.
module latency_model #(parameter int UNLOADED_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       hops_counted,     // every switch on the path is charged
  input  logic       queueing_modelled,// the load-dependent term exists
  input  logic       media_included,   // the device's own time is in the number
  input  logic       retries_charged,  // retried transactions cost what they cost
  input  logic       tail_reported,    // the number quoted is the tail
  output logic       credible,
  output logic [4:0] fail_mask,
  output logic [7:0] n_eval, n_credible,
  output logic       false_confidence_err
);
  assign fail_mask[0] = ~hops_counted;
  assign fail_mask[1] = ~queueing_modelled;
  assign fail_mask[2] = ~media_included;
  assign fail_mask[3] = ~retries_charged;
  assign fail_mask[4] = ~tail_reported;
  // The unloaded build drops the two terms that only appear under load, which is
  // exactly the model a bench measurement produces.
  assign credible = (UNLOADED_ONLY != 0)
                  ? (hops_counted && media_included && tail_reported)
                  : (fail_mask == 5'd0);
  assign false_confidence_err = evaluate && credible && (fail_mask != 5'd0);
  // ... evaluation counters omitted for length
endmodule

Six evaluations — all terms present, then each dropped alone:

  model: evaluated=6 credible=1 | unloaded-only credible=3

One credible model out of six, and the bench-derived model found three. The two extra are queueing and retries — precisely the two terms that do not exist at zero load:

This is 17.4 section 15's structure applied to a model rather than a device, and it is the same lesson: everything that is knowable without load, a bench measurement gets right; everything that only appears under load, it cannot see at all.

16. Quantitative Reasoning

Every number is from a printed line above. None describes any platform.

Hop decomposition. 40 + 100 + 160 = 300 ns across three traversals. The fixed-only model: 3 × 40 = 120. Two and a half times, with an identical hop count.

Path. 30 + 25 + 60 + 25 + 45 + 120 = 305 ns, largest segment the media at 120 — 39% of the path.

Tail. 990 × 200 + 10 × 2000 = 218,000 ns over 1000 requests. Mean 218, tail 2000. The tail is 9.2× the mean and the budget is 500.

Queueing. At 50%, ×1.0. At 90%, ×9.0. At 99%, ×99.0. The linear model gives 0.5, 0.9 and 0.99 — a factor of 10 wrong at 90% and 100 at 99%.

Switch hops. 2 switches at 60 ns on a 150 ns base is 270 ns, 44% switch. Four is 390 ns, 61% switch.

Measurement point. Device spends 150 ns, host observes 255, 105 ns misattributed — 41% of the attributed figure.

Retry. 3 × 300 + 2600 = 3500 ns over four. Mean 875 against the ignoring model's 300. One retry in four moves the mean by 2.9×.

Budget. 80 + 120 + 150 = 350 against 400. Total within, segment 1 over. Across seven checks: 4 per-segment failures against 2 total failures.

Regression. Segment 0 rose 300 while segment 2 fell 50. Worst delta 300, culprit segment 0 — and segment 2's delta is 0, not a wrapped 65,486.

17. Assertions

Every property is an immediate check written as cond !== 1'b1, sampled after a settle. 182 assertion sites across two testbenches.

18. Mutation Testing

78 mutations, one at a time, each required to make the baseline print RESULT: FAIL.

78 of 78 were killed.

The first run killed 71 and left 7 survivors:

Four frozen-counter survivors in one run is the largest cluster of a single class in this batch, and the cause is uniform: n_paths, n_eval, n_routes and n_over were each accumulating correctly while nothing ever read them. Every one of the four models had its values checked exhaustively and its count checked not at all.

"Misattribution reported without measuring" survived because the bench never held measure low while the configuration would misattribute. The checker was verified only where it fires.

"Total boundary off by one" survived until the bench spent exactly the budget. spent > budget and spent >= budget differ on one value, and the first six configurations all stepped over it.

A representative sample:

19. Verification Strategy

Two builds, one stimulus. The parameter is the only difference.

Include the configuration where the shortcut is correct. An unloaded hop, a distribution with no slow population, a direct-attached path with no switches, a measurement with nothing between host and device. In each case the simplified model gets the right answer, and a checker that fired there would be firing on the common case.

Move the answer so it cannot be a fixed index. The largest path segment is driven to the media, the switch and the host in turn. The regression culprit is driven to two different segments.

Assert counters, not only values. Four survivors in this chapter were frozen counters whose accumulated values had been checked exhaustively.

Assert checkers on their quiet path. misattribution_err with no measurement running. hop_blind_err with no switches. budget_lie_err with no slow requests.

Drive every boundary exactly. A tail precisely at the budget. Utilisation at 99 and 100. A segment exactly at its allocation. Total spend exactly equal to the budget.

Guard every unsigned subtraction and then drive the case that needs the guard. Section 14's improving segment is the only stimulus that distinguishes a guarded delta from a wrapped one.

20. Synthesis and Implementation Reality

This is a model, not a datapath. Nothing here is synthesised. What is real is the instrumentation the model implies: per-segment timestamps, queue-depth sampling, and retry counting, each of which costs registers and each of which is the difference between a number you can act on and a number you can only quote.

Per-segment timestamps are the expensive part. Section 6 needs a timestamp at six points on the path, and the points are in different clock domains on different chips. Real decomposition is usually coarser — host-side and device-side, with the switch inferred by subtraction — which is exactly the granularity section 11 shows misattributes.

Queue depth must be sampled at the right moment. Section 5 charges per_queue_ns × depth for the depth at the moment of traversal, and an average depth over a window produces a different and smoother number that hides the bursts responsible for the tail.

The utilisation curve is a model, not a measurement. u / (1 - u) is the M/M/1 shape and real arrival processes are not Poisson. The transferable content is that the curve is convex and unbounded, not that this particular formula is right — a linear model is wrong in kind, not merely in coefficient.

Retry accounting needs the timeout to be visible. Section 12's 2600 ns is mostly timeout, and a system that counts retries without recording how long each one waited cannot reconstruct the cost.

21. Silicon Observability

The fourth row is the one most systems half-implement. Retry counters are common; cumulative retry time is rare, and without it section 12's 2600 ns is unrecoverable — the system knows a retry happened and not what it cost.

22. Debug Lab

Symptom: measured latency is much worse in production than on the bench.

Compare the load. Section 5's hop costs 40 ns unloaded and 160 at depth 10. If the bench ran unloaded, the numbers are not comparable and no further investigation is warranted until they are.

Check utilisation before anything else. Section 9: at 90% the queueing term is nine service times. A link that moved from 70% to 90% has roughly tripled its waiting time with no change to anything else.

Get a per-segment breakdown, or the coarsest one available. Section 14 needs a before and an after per segment. Two totals produce a magnitude and no direction.

Check where the measurement is taken. Section 11: a host-only measurement charges the device for switch queueing. If the switch got busier, the device will be blamed and replacing it will change nothing.

Look at the retry counter. Section 12: one retry in four moves a 300 ns mean to 875. A rising retry rate presents as a latency regression with no segment obviously at fault.

Compare the tail, not the mean. Section 7's system has a 218 ns mean and a 2000 ns tail. If complaints do not match the mean, the mean is not the number the complaints are about.

23. Design Review

What is the latency, and at what utilisation? A number without a load is a bench number.

Which segments does your model name? If the answer is "host to device", a regression in that span has no culprit.

Do you quote a mean or a percentile? And does the budget the number is checked against refer to the same statistic?

How many switch hops does the worst path have? Section 10's four-switch path is 61% fabric.

Where is the measurement taken, and what sits between it and the device?

Are retries charged, and is the timeout included in the charge?

Is the budget checked per segment or in total? Section 13's design meets its total and blows a segment allocation.

24. How This Appears In Real Engineering

Latency work arrives as a complaint that a system is slower than expected, with a measurement attached that is correct and unhelpful.

The characteristic case is a bench number that does not survive production. Almost always the bench ran at low utilisation and production does not, and section 9's convex curve means the discrepancy grows sharply with load rather than proportionally.

The second is a regression nobody can locate. A total moved and nothing else is instrumented, so the investigation proceeds by substitution — swap the device, swap the switch — rather than by measurement.

The third is a device blamed for the fabric. Section 11's misattribution is the standard version, and it is expensive because it produces a hardware change that does not help.

The fourth is a budget that is met on paper and missed in practice, either because the number quoted was a mean (section 7) or because the budget was checked in total while one segment overran (section 13).

25. Common Misconceptions

"CXL adds about X nanoseconds." It adds a fixed cost plus a load-dependent one plus a hop per switch. Section 5's hop costs 40 ns at zero load and 160 at depth 10.

"We measured it." At what utilisation? Section 5's two builds agree exactly once, at zero.

"Queueing adds maybe ten percent at high load." At 90% utilisation it adds nine times the service time. The linear intuition is the mutation this chapter kills.

"The device is slow." Measured where? Section 11's device spends 150 ns of a 255 ns host-observed figure.

"The average latency is fine." Section 7's average is fine and one request in a hundred misses the budget by 4×.

"We are within our latency budget." In total, or per segment? Section 13's design is within the total and over on segment 1, and the segment allocation is where the requirement actually lives.

"Retries are rare so they do not matter." One in four moved a 300 ns mean to 875. Rarity is not the same as cheapness, and the cost is mostly the timeout.

"Adding a switch adds a small hop." Two switches on a 150 ns base path are 44% of it. Four are 61%.

26. Interview Reasoning

Q1. Someone quotes a CXL latency. What are your first two questions? At what utilisation, and measured where. Without both, the number describes a bench.

Q2. Why is a fixed-cost-only latency model dangerous rather than approximate? Because it is exactly right at zero load and wrong by an unbounded amount everywhere else. It agrees with reality precisely where nothing is at stake.

Q3. What does a lumped total cost you? Attribution. It gives the size of a regression and no direction, so the investigation proceeds by swapping parts rather than by measuring them.

Q4. Mean 218ns, tail 2000ns, budget 500ns. Do you pass? No. Budgets are met at the tail. The mean passes and one request in a hundred misses by a factor of four.

Q5. Utilisation goes from 50% to 90%. What happens to queueing delay? It goes from one service time to nine. Not from 50% to 90% of something — nine times.

Q6. And from 90% to 99%? From nine service times to ninety-nine. The curve is convex and unbounded, which is why "high utilisation" without a number is not a specification.

Q7. You measure at the host and the number is bad. What can you conclude about the device? Very little. Section 11's host-side figure includes switch queueing and wire time — 105ns of 255 in that model, and none of it the device's.

Q8. Four switches at 60ns on a 150ns base path. What fraction is fabric? 240 of 390, so 61%. The fabric is the majority of the access, and improving the device improves the minority.

Q9. A retried transaction — what does it cost? The timeout, the failed traversal and the successful one. In section 12 that is 2600ns against a clean 300, and a model charging it 300 reports an unchanged mean.

Q10. Your retry counter is rising and your latency mean is flat. What does that tell you? That retries are not being charged. The count and the cost are separate instrumentation and the second one is usually missing.

Q11. A design meets its total latency budget. Is it correct? Not necessarily. Section 13's design meets 350 against 400 while segment 1 exceeds its allocation, and the allocation is where the requirement came from.

Q12. How do you attribute a latency regression? Per-segment deltas, and take the largest. Guard the subtractions — a segment that improved must contribute zero, not a wrapped maximum.

Q13. Why guard the subtraction rather than use a signed type? Either works. The guard makes the intent explicit at the point of the arithmetic, and the mutation that removes it is killed by the one comparison where a segment improved.

Q14. Which terms of a latency model does a bench measurement give you? Hops, media and whichever statistic you chose to report. Not queueing and not retries — the two that only exist under load, and the two the unloaded build in section 15 skips.

Q15. How would you instrument a system to make this chapter's models usable? Per-segment timestamps or the coarsest split available, queue depth sampled at traversal rather than averaged, utilisation per link, retry count and cumulative retry time, and a percentile rather than a mean.

Q16. Which single instrument would you add first? A percentile. It costs the least, and it is the one that decides whether the number you already have is the number your users are experiencing.

27. Exercises

1. Extend RTL 1 so per_queue_ns differs by protocol, as .io, .cache and .mem traffic would. Which assertions become per-protocol?

2. In RTL 2, add a seventh segment for a second switch hop. Does biggest_id still fit in three bits, and what happens to the comparator tree?

3. RTL 3 uses two populations. Replace it with four and derive the p99 rather than the tail. At what slow fraction do the two answers diverge?

4. RTL 4 uses the M/M/1 shape. Implement M/D/1, whose waiting time is half of it, and determine the utilisation at which the two differ by more than 100ns at a 100ns service time.

5. In RTL 5, make per_switch_ns depend on the switch's own utilisation using RTL 4. At what load does a two-switch path exceed a four-switch path at low load?

6. Extend RTL 6 with a third measurement point at the device. What can be attributed with three points that cannot with two?

7. RTL 7 charges one retry. Model a transaction retried twice and determine the retry rate at which the mean doubles.

8. Add a fourth segment to RTL 8 and make the allocations reallocatable — a segment may lend headroom to another. Which of blind_overrun_err's conditions still applies?

28. Summary

A CXL latency is a sum of named parts taken under a stated load, and this chapter builds every part.

A hop has a fixed term and a queueing term. 300 ns of real hop time reported as 120 by a model that is exactly right at zero load.

A path is its segments. Both models agree the path is 305 ns; only one can say the media is 120 of it.

Budgets are met at the tail. A mean of 218 passing a 500 ns budget that the 2000 ns tail misses by a factor of four.

Queueing is convex. ×1.0 at 50% utilisation, ×9.0 at 90%, ×99.0 at 99% — where a linear model says 0.5, 0.9 and 0.99.

Every switch is a hop. Two switches are 44% of a 270 ns path; four are 61% of a 390 ns one.

Where you measure decides what you blame. A device spending 150 ns charged 255 by a host-side measurement, 41% of it belonging to the fabric.

A retry pays twice plus the timeout. One in four moving a 300 ns mean to 875, on a model whose retry counter is correct.

And a budget is per segment. A design within its 400 ns total with a segment over its allocation, passed by a total-only check twice in seven.

18.2 — CXL Throughput takes the same path and asks the other question: not how long one transaction takes, but how many can be in flight at once, and which ceiling stops the next one.