Performance Bottlenecks

26.5 was about a path that carries nothing. This chapter is about a path that carries everything it was asked to and is still too slow — and the difference matters because the first has a broken component in it and the second does not.

Every hop has credit. Every route agrees. There is no cycle, no head-of-line victim, no leak. The link trained at full width and full speed, and the register says so. The application is getting a quarter of the number on the datasheet.

1. The Engineering Problem — Full Speed Is Not Full Throughput

A link runs at its signalling rate and delivers what is left after framing. Sixty-four payload bytes in an eighty-byte frame on a sixty-four gigabyte link is fifty-one gigabytes a second of goodput — a number that appears in no register anywhere. Section 5.

Throughput is what is in flight divided by how long each one takes. Eight requests outstanding on a thirty-two cycle round trip is twenty-five percent of the link, and the link is not the thing that is limiting it. Section 6.

An application waits for its slowest request, not its average one. A mean of a hundred cycles with a tail at eight hundred, against a five-hundred-cycle budget, is a perfectly healthy average that misses the budget on one request in a hundred. Section 7.

A bidirectional figure is two numbers a one-directional workload cannot both use. Sixty-four gigabytes a second aggregate on a read-only workload is thirty-two, and half the quoted figure was never available. Section 8.

Interleaving spreads a working set only when the accesses are big enough to land on more than one device. Sixty-four byte accesses on a two-hundred-and-fifty-six byte granule touch one device of four: twenty-five percent of what the interleave was built for. Section 9.

This chapter against 26.5, stated precisely. That one owns a fabric with a broken hop in it. This one owns a fabric with nothing broken in it at all — which is why every model here is about a quantity being measured wrongly rather than a component behaving wrongly, and why section 14's weak definition is a link-status register.

2. The One-Sentence Model

A link is delivering what it was quoted at when it trained at its full rate, enough requests are in flight to fill it, the slow requests are inside the budget, neither direction is over its own half, accesses land on every device the interleave spans, and the traffic that carries no payload is counted — and "it is running at full speed" is one of those six.

3. What This Chapter Owns

The boundary against 26.5 is the one worth being precise about, because the two chapters share a symptom and share almost no evidence. A fabric problem has a component in the wrong state. A performance problem has every component in exactly the state it was configured for, and the configuration is wrong for the workload. The instruments differ accordingly: a fabric problem is found by reading state, and a performance problem is found by reading counters over time.

There is a second boundary that trips people more often. Slow is not the same as inconsistent. A link delivering half its rate on every request is section 5 or 6. A link delivering full rate on ninety-nine requests and ten times the latency on the hundredth is section 7, and the two produce completely different application behaviour from averages that can be identical. Every model in this chapter publishes both a headline number and the thing that headline number hides.

4. Teaching-Model Boundary

Every model in this chapter is a teaching model, not a performance model. It computes the one relationship the section is about and nothing else. There is no queue, no arbiter, no scheduler, no cache and no traffic generator anywhere in this file.

Each model is built twice from one source. A parameter selects between the measured build, which computes what the link actually delivers, and the reported build, which computes what a particular instrument would say. The two are instantiated side by side against identical stimulus, and every section's headline number is the gap between them.

The error output in each model separates what is true of the link from what the build reports about it, and fires only when the second contradicts the first. That shape is what makes a mutation to either half visible rather than absorbed.

5. RTL 1 — Line Rate Is Not Goodput

Start with the number every conversation about CXL performance starts with, because it is the number most likely to be wrong by a stable, predictable, entirely invisible amount.

A link's line rate is how fast it signals. It is a property of the electrical layer, it is what the link trains to, and it is what the status register reports. It is also not a rate at which any payload moves, because every byte of payload travels inside a frame that carries headers, sequence numbers, CRC and — on a protocol that multiplexes three traffic classes over one link — the identification that says which class this flit belongs to.

Goodput is line rate scaled by the fraction of the frame that is payload. The difference is not an error bar; it is a constant, computable, entirely legitimate tax, and the only mistake available is failing to compute it before promising someone a number.

// RTL 1 - line rate against goodput. A link running at its full signalling
// rate delivers the payload left over after framing, and the two numbers are
// quoted in the same unit by people who mean different things.
module goodput_model #(parameter int LINE_RATE_IS_THROUGHPUT = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] line_rate, frame_bytes, payload_bytes, offered,
  output logic [15:0] carried_payload, overhead_bytes, goodput, efficiency_pct,
  output logic        rate_is_goodput,
  output logic [7:0]  n_measures, n_overstated,
  output logic        full_speed_err
);
  logic [31:0] g_q, e_q;
  logic        truly_lossy;
  // Payload cannot exceed the frame that carries it.
  assign carried_payload = (payload_bytes > frame_bytes) ? frame_bytes : payload_bytes;
  assign overhead_bytes  = frame_bytes - carried_payload;
  assign g_q = (frame_bytes == 16'd0) ? 32'd0
             : (({16'd0, line_rate} * {16'd0, carried_payload}) / {16'd0, frame_bytes});
  assign goodput = (LINE_RATE_IS_THROUGHPUT != 0) ? line_rate
                 : ((g_q > {16'd0, line_rate}) ? line_rate : g_q[15:0]);
  assign e_q = (line_rate == 16'd0) ? 32'd0
             : (({16'd0, goodput} * 32'd100) / {16'd0, line_rate});
  assign efficiency_pct = (e_q > 32'd100) ? 16'd100 : e_q[15:0];
  assign rate_is_goodput = (goodput >= line_rate);
  assign truly_lossy = (frame_bytes != 16'd0) && (overhead_bytes != 16'd0)
                       && (offered != 16'd0) && (line_rate != 16'd0);
  // The number being quoted is the signalling rate on a link that frames.
  assign full_speed_err = measure && truly_lossy && rate_is_goodput;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_overstated <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (rate_is_goodput && truly_lossy) n_overstated <= n_overstated + 8'd1;
    end
  end
endmodule

Sixty-four payload bytes inside an eighty-byte frame on a sixty-four gigabyte link is fifty-one gigabytes a second of goodput at seventy-nine percent efficiency. The line-rate view reports sixty-four at a hundred percent. Neither number is a mistake; one of them is an answer to a question nobody asked.

Figure 1 — the same link measured two ways. The register on the upper path is correct: the link signals at sixty-four. The payload calculation on the lower path is also correct: sixteen of every eighty bytes are framing, so fifty-one gigabytes a second of payload cross the link. The thirteen-gigabyte gap between them is not a defect, a margin or a measurement error — it is a design decision, computable before silicon exists, and reported by nothing.

The model refuses two inputs that a careless measurement supplies regularly. A payload larger than the frame that carries it is clamped to the frame, because a frame cannot carry more than it is — a counter that reports both independently will occasionally produce this, and a model that accepted it would compute negative overhead and a goodput above the line rate. And a link with no rate makes the comparison meaningless: goodput of zero is trivially equal to a line rate of zero, so the model requires a rate to exist before it will call anything overstated.

Two degenerate cases deserve a moment. A framed link with nothing offered to it is not an overstated link: it is idle, and the goodput figure describes what it would deliver rather than what it is delivering. A measurement with no frame size measures nothing at all, and both builds correctly decline to report. Between them these two cases cover most of the spurious alarms a naive efficiency monitor produces in the first week it is deployed.

The last case is the one to carry into a design review: half the frame as overhead halves the goodput. Efficiency is linear in payload fraction, which means it is entirely determined by a design decision — frame size against payload size — made long before anyone measures anything. Section 19 returns to this.

There is one more thing the model is quietly doing that is worth naming. The goodput calculation scales the line rate by the payload fraction and then clamps the result to the line rate, and the clamp is not decoration. A payload-to-frame ratio computed from two counters read at different instants can exceed one, and a model without the clamp would report a goodput above the signalling rate — physically impossible, and entirely believable to somebody reading a spreadsheet rather than the model. Reporting an impossible number is worse than reporting no number: it does not merely fail to help, it destroys the credibility of every other figure on the same page.

6. RTL 2 — Throughput Is Requests In Flight Divided By Latency

The second number, and the one that explains most CXL performance shortfalls that are not the first one.

A pipelined interconnect does not deliver bandwidth because it is fast. It delivers bandwidth because many requests are in flight simultaneously, each one taking a fixed round trip. The throughput that results is the number of bytes in flight divided by the time each one is in flight for, and if that product is smaller than the link, the link is not the limiter — the requestor is.

This is uncomfortable in practice because the symptom points at the wrong component. A host issuing eight outstanding reads on a device with a thirty-two-cycle round trip will measure a quarter of the link rate and will, quite reasonably, conclude that something about the device or the link is slow. Nothing is. The device is idle three quarters of the time, waiting for work.

// RTL 2 - outstanding requests against latency. Throughput on a pipelined
// interconnect is the number of requests in flight divided by the time each
// one takes, and a queue too shallow to hold them caps the link no matter what
// the link can do.
module outstanding_limit #(parameter int THE_LINK_SETS_THE_RATE = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] link_bw, outstanding, latency_cyc, bytes_per_req,
  output logic [15:0] little_bw, achieved_bw, depth_needed, achieved_pct,
  output logic        depth_ok,
  output logic [7:0]  n_measures, n_starved,
  output logic        shallow_queue_err
);
  logic [31:0] l_q, d_q, a_q;
  logic        truly_starved;
  // Bytes in flight divided by the round trip each one takes.
  assign l_q = (latency_cyc == 16'd0) ? 32'hFFFF
             : (({16'd0, outstanding} * {16'd0, bytes_per_req}) / {16'd0, latency_cyc});
  assign little_bw = (l_q > 32'hFFFF) ? 16'hFFFF : l_q[15:0];
  assign achieved_bw = (THE_LINK_SETS_THE_RATE != 0) ? link_bw
                     : ((little_bw > link_bw) ? link_bw : little_bw);
  // How many requests would have to be in flight to fill the link.
  assign d_q = (bytes_per_req == 16'd0) ? 32'd0
             : (({16'd0, link_bw} * {16'd0, latency_cyc}) / {16'd0, bytes_per_req});
  assign depth_needed = (d_q > 32'hFFFF) ? 16'hFFFF : d_q[15:0];
  assign a_q = (link_bw == 16'd0) ? 32'd0
             : (({16'd0, achieved_bw} * 32'd100) / {16'd0, link_bw});
  assign achieved_pct = (a_q > 32'd100) ? 16'd100 : a_q[15:0];
  assign depth_ok = (outstanding >= depth_needed);
  assign truly_starved = (link_bw != 16'd0) && (little_bw < link_bw);
  assign shallow_queue_err = measure && truly_starved && (achieved_bw == link_bw);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_starved <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (truly_starved) n_starved <= n_starved + 8'd1;
    end
  end
endmodule

Eight sixty-four-byte requests on a thirty-two-cycle round trip is sixteen gigabytes a second — twenty-five percent of a sixty-four gigabyte link, with thirty-two outstanding requests needed to fill it. The link-rate view reports sixty-four at a hundred percent, which is what the link would do if anybody asked it to.

The model publishes depth_needed deliberately, because it converts the diagnosis into an instruction. Knowing the link is at twenty-five percent is a complaint; knowing that thirty-two outstanding requests would fill it is a change someone can make. Every performance model in this chapter publishes at least one number that is an action rather than an observation.

Three degenerate inputs are driven. Zero latency is unbounded bandwidth, clamped to the link — a measurement that reports no round trip is an instrument artefact, and treating it as infinite throughput and then capping it is the only answer that does not divide by zero. Requests that carry no bytes deliver nothing, and the link-rate view still reports the full link. And nothing in flight at all is the limiting case of the section: an empty pipe on a perfect link, reported by a status register as a link running at full speed.

The last of those is worth stating plainly, because it is the cleanest possible statement of what the chapter is about. A completely idle link and a completely saturated link report the same line rate.

The relationship in this model is Little's law, and it is worth recognising under its name because it constrains every queueing system in the stack rather than just the interconnect. The practical form for an engineer is that any two of the three quantities determine the third, so a measurement of throughput and latency is a measurement of concurrency whether or not anybody counted it. Teams that internalise this stop asking whether the link is fast enough and start asking how many requests the requestor can keep in flight, which is a question with an answer in a register rather than an argument in a meeting.

The corollary is uncomfortable and worth stating. Reducing latency and increasing concurrency are interchangeable for throughput and are not interchangeable for anything else. A device that halves its round trip and a host that doubles its outstanding count produce the same bandwidth number and completely different behaviour for a latency-sensitive workload — which is section 7's subject, and the reason a bandwidth-only verification plan can bless the wrong fix.

7. RTL 3 — The Mean Hides The Tail

Everything so far has been about a number being too low. This section is about a number being exactly right and completely useless.

An application does not experience an average latency. It issues a request, waits, and proceeds — and if it issues a hundred requests and must have all hundred before it can continue, it experiences the slowest one. A distribution with a mean of a hundred cycles and a ninety-ninth percentile at eight hundred has an excellent mean, and an application that gathers a hundred results per operation will feel eight hundred on essentially every operation it performs.

This is the single largest gap between a performance measurement and a user experience in the whole of interconnect work, and it is invisible to any instrument that reports one number per interval.

// RTL 3 - the mean hides the tail. An application waits for its slowest
// request, not its average one, and a distribution with a long tail has a
// perfectly respectable mean.
module tail_latency #(parameter int THE_MEAN_IS_THE_LATENCY = 0) (
  input  logic clk, rst_n,
  input  logic        sample,
  input  logic [15:0] requests, mean_lat, p99_lat, budget_cyc,
  output logic [15:0] felt_lat, tail_ratio, slow_reqs, in_budget_pct,
  output logic        tail_bounded,
  output logic [7:0]  n_samples, n_over_budget,
  output logic        tail_hidden_err
);
  logic [15:0] true_p99;
  logic [31:0] r_q, b_q;
  logic        truly_long;
  // A tail below the mean is a measurement artefact, not a short tail.
  assign true_p99 = (p99_lat < mean_lat) ? mean_lat : p99_lat;
  assign felt_lat = (THE_MEAN_IS_THE_LATENCY != 0) ? mean_lat : true_p99;
  assign r_q = (mean_lat == 16'd0) ? 32'd1
             : (({16'd0, true_p99} * 32'd10) / {16'd0, mean_lat});
  assign tail_ratio = (r_q > 32'hFFFF) ? 16'hFFFF : r_q[15:0];
  // One request in a hundred is over the tail, by construction of p99.
  assign slow_reqs = (true_p99 > budget_cyc) ? (requests / 16'd100) : 16'd0;
  assign b_q = (requests == 16'd0) ? 32'd100
             : ((({16'd0, (requests - slow_reqs)}) * 32'd100) / {16'd0, requests});
  assign in_budget_pct = (b_q > 32'd100) ? 16'd100 : b_q[15:0];
  assign tail_bounded = (felt_lat <= budget_cyc);
  assign truly_long = (requests != 16'd0) && (true_p99 > budget_cyc);
  assign tail_hidden_err = sample && truly_long && tail_bounded;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_samples <= 8'd0; n_over_budget <= 8'd0;
    end else if (sample) begin
      n_samples <= n_samples + 8'd1;
      if (truly_long) n_over_budget <= n_over_budget + 8'd1;
    end
  end
endmodule

A mean of a hundred against a tail at eight hundred, on a five-hundred-cycle budget, is ten requests in a thousand outside the budget and ninety-nine percent inside it — and the mean view reports a hundred cycles and calls the tail bounded. The tail is eight times the mean, which the model publishes as a ratio because the ratio is the thing worth alarming on.

The clamp here is unusual and worth reading. A tail measured below the mean is a measurement artefact, not a short tail — percentiles and means computed over different windows, or a percentile estimated from too few samples, produce this regularly. The model raises the tail to the mean rather than believing it, because a distribution whose ninety-ninth percentile is below its mean does not exist.

Two degenerate cases bracket the section. A budget of zero is exceeded by the mean as well as the tail, and both builds fail together — the one case where the averaging view cannot hide anything, and correspondingly the one case nobody needs this section for. A mean of zero with a real tail is the opposite extreme and the strongest possible hiding place: the mean view reports zero latency on an interface with an eight-hundred-cycle tail, and calls it bounded. Instruments that report a mean of zero are usually reporting a counter that was not incrementing, which is exactly when the tail is most likely to be interesting.

The saturating case is the reason the ratio is clamped: a mean of one cycle against a tail of twenty thousand is a ratio the counter cannot hold, and a saturated ratio is still a reported tail, whereas a wrapped one would read as a healthy distribution.

The percentile arithmetic in this model is deliberately crude — one request in a hundred is above the ninety-ninth percentile by definition — and it is crude for a reason worth defending. A model that estimated a distribution shape would be making claims about a system it does not have, and would be wrong in ways that depend on the estimator rather than on the system. The one-in-a-hundred figure is true for every distribution, which makes the slow-request count a floor rather than a guess, and a floor is exactly what an argument about a latency budget needs.

8. RTL 4 — Aggregate Is Two Numbers A Workload Cannot Both Use

A CXL link is full duplex. Its bandwidth is frequently quoted as the sum of the two directions, which is correct, useful, and directly misleading for the majority of real workloads.

A memory-expansion workload is read-heavy. A write-back-heavy workload is the opposite. Neither is balanced, and neither can borrow the unused half of the link. The direction that is busy is capped at its own peak, and the direction that is idle contributes its full share to a headline figure that nothing will ever claim.

// RTL 4 - aggregate bandwidth against per-direction bandwidth. A link quoted
// as bidirectional adds two numbers that a one-directional workload cannot
// both use.
module direction_split #(parameter int AGGREGATE_IS_AVAILABLE = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] dir_peak, rd_demand, wr_demand, quoted_bw,
  output logic [15:0] aggregate_bw, rd_served, wr_served, served_pct, overquote,
  output logic        balanced_ok,
  output logic [7:0]  n_measures, n_skewed,
  output logic        direction_blind_err
);
  logic [15:0] demand_total, served_total;
  logic [31:0] s_q;
  logic        truly_skewed;
  assign aggregate_bw = dir_peak + dir_peak;
  // Neither direction can draw on the other's half of the quoted number.
  assign rd_served = (rd_demand > dir_peak) ? dir_peak : rd_demand;
  assign wr_served = (wr_demand > dir_peak) ? dir_peak : wr_demand;
  assign demand_total = rd_demand + wr_demand;
  assign served_total = (AGGREGATE_IS_AVAILABLE != 0)
                        ? ((demand_total > aggregate_bw) ? aggregate_bw : demand_total)
                        : (rd_served + wr_served);
  assign s_q = (demand_total == 16'd0) ? 32'd100
             : (({16'd0, served_total} * 32'd100) / {16'd0, demand_total});
  assign served_pct = (s_q > 32'd100) ? 16'd100 : s_q[15:0];
  assign overquote = (quoted_bw > served_total) ? (quoted_bw - served_total) : 16'd0;
  assign balanced_ok = (rd_demand <= dir_peak) && (wr_demand <= dir_peak);
  assign truly_skewed = (demand_total != 16'd0)
                        && ((rd_demand > dir_peak) || (wr_demand > dir_peak));
  // The quoted figure is the sum and the workload is one-directional.
  assign direction_blind_err = measure && truly_skewed
                               && (served_total > (rd_served + wr_served));

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_skewed <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (truly_skewed) n_skewed <= n_skewed + 8'd1;
    end
  end
endmodule

A read-only workload demanding sixty-four gigabytes a second on a link with thirty-two per direction is served thirty-two — fifty percent of its demand, with a thirty-two gigabyte overquote — while the aggregate view reports the demand fully served because the total fits inside the sum.

The third stimulus case is the honest one and it is why this mistake survives so long. When both directions saturate, the aggregate figure is right. Sixty-four of demand in each direction on a sixteen-per-direction link is served thirty-two either way, and both models report twenty-five percent. A benchmark that exercises both directions equally will never expose the error, which is precisely the benchmark most likely to be run.

Two degenerate cases keep the model honest. An idle link is balanced — no demand in either direction cannot be direction-limited, and a model that reported a skew because the quoted figure exceeded zero traffic would alarm on every idle link in the fleet. And a link with no per-direction rate serves nothing in either model, so the aggregate view has nothing to overstate.

The overquote output is the number to take to a capacity-planning conversation: it is the difference between what was promised and what this particular workload can actually draw, and for a one-directional workload on a symmetric link it is exactly half the headline figure.

Two things are often merged here and should not be. A link being full duplex is a real property and a real benefit — a read stream and its write-back stream genuinely do not contend, which is why the architecture is full duplex in the first place. The error is not in the link; it is in summarising a two-dimensional capability with a one-dimensional number and then comparing that number against a one-dimensional demand. Any workload whose direction mix differs from fifty-fifty draws less than the sum, and how much less is entirely determined by the mix.

9. RTL 5 — An Interleave Finer Than The Access Does Nothing

Interleaving is how a CXL memory pool turns four devices into one fast device. It works by placing consecutive granules on different devices, so that a stream of accesses is served in parallel. Whether that parallelism materialises depends on a relationship nobody checks: the size of an access against the size of a granule.

An access smaller than a granule lands entirely inside one granule and is therefore served by exactly one device. Four-way interleaving across four devices, with a granule larger than the access, delivers the bandwidth of one device. The interleave is configured, the devices are present, the accesses are spread across the address space — and every individual access is serialised on one of them.

// RTL 5 - access size against interleave granule. Interleaving spreads a
// working set across devices only when the accesses are large enough to land
// on more than one of them.
module granule_spread #(parameter int INTERLEAVED_MEANS_SPREAD = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] access_bytes, granule_bytes, devices, per_dev_bw,
  output logic [15:0] devices_touched, spread_bw, serialized_pct, achieved_pct,
  output logic        spread_ok,
  output logic [7:0]  n_measures, n_serialized,
  output logic        granule_err
);
  logic [15:0] true_touched, ideal_bw;
  logic [31:0] t_q, s_q, i_q, a_q;
  logic        truly_serial;
  // An access smaller than the granule lands on exactly one device.
  assign t_q = (granule_bytes == 16'd0) ? 32'd1
             : (({16'd0, access_bytes} + {16'd0, granule_bytes} - 32'd1)
                / {16'd0, granule_bytes});
  assign true_touched = (t_q > {16'd0, devices}) ? devices
                      : ((t_q == 32'd0) ? 16'd1 : t_q[15:0]);
  assign devices_touched = (INTERLEAVED_MEANS_SPREAD != 0) ? devices : true_touched;
  assign s_q = {16'd0, devices_touched} * {16'd0, per_dev_bw};
  assign spread_bw = (s_q > 32'hFFFF) ? 16'hFFFF : s_q[15:0];
  assign i_q = {16'd0, devices} * {16'd0, per_dev_bw};
  assign ideal_bw = (i_q > 32'hFFFF) ? 16'hFFFF : i_q[15:0];
  assign a_q = (ideal_bw == 16'd0) ? 32'd100
             : (({16'd0, spread_bw} * 32'd100) / {16'd0, ideal_bw});
  assign achieved_pct = (a_q > 32'd100) ? 16'd100 : a_q[15:0];
  assign serialized_pct = 16'd100 - achieved_pct;
  assign spread_ok = (devices_touched >= devices);
  assign truly_serial = (devices > 16'd1) && (true_touched < devices)
                        && (access_bytes != 16'd0);
  assign granule_err = measure && truly_serial && spread_ok;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_serialized <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (truly_serial) n_serialized <= n_serialized + 8'd1;
    end
  end
endmodule

Sixty-four byte accesses on a two-hundred-and-fifty-six byte granule across four devices touch one device: sixteen gigabytes a second of a possible sixty-four, seventy-five percent serialised. The interleaved view counts four devices and reports the full rate, because from a configuration dump four-way interleaving is exactly what is programmed.

The fix is a configuration change rather than a hardware change, which is what makes this worth catching: a granule at or below the access size spreads every access. The second stimulus case shows the working configuration — a kilobyte access on a two-hundred-and-fifty-six byte granule touches all four devices and delivers the full rate — and the third shows that an access larger than the whole interleave is clamped rather than credited with touching sixteen devices that do not exist.

Three degenerate cases are driven. No granule declared is one device, not none — a configuration read that returns zero for an unimplemented field must not be treated as infinite spreading. A single device cannot be serialised against itself, and the model says so rather than reporting a hundred percent serialisation on a system with nothing to interleave. And an access of no bytes lands on one device by arithmetic; the model explicitly refuses to call that a granularity fault, because a zero-byte access is an instrument artefact and reporting it would bury the real cases.

The last case is the subtle one. Devices that deliver no bandwidth produce a spread figure of zero and an achieved percentage of a hundred — a hundred percent of nothing — while the interleaved view happily reports four devices delivering nothing. The model still flags the serialisation, because the granule relationship is wrong regardless of what the devices can do.

Why this configuration is common rather than exotic deserves a sentence. Interleave granularity is usually chosen for locality, not for parallelism. A larger granule keeps a contiguous region on one device, which is good for sequential streaming and good for a device's own internal locality, and the engineer choosing it is optimising something real. Access size, meanwhile, is chosen by whoever wrote the application, usually much later and without reference to the interleave. Nothing connects the two decisions, no tool warns when they conflict, and the failure is the product of two reasonable choices made independently.

10. RTL 6 — Traffic That Carries No Payload

Every section so far has measured something the link delivers. This one measures what else is on the link while it delivers it.

A CXL link carries more than data. It carries snoops, completions without data, credit returns, and — depending on the traffic classes in use — the coherency traffic that makes the memory behave like memory. All of it occupies bandwidth. A utilisation counter that counts data bytes reports headroom that does not exist, and the headroom it reports is exactly the traffic it is not counting.

This matters most at the moment it matters most: capacity planning. A link measured at sixty percent utilisation looks like a link with forty percent of room to grow. If thirty of those forty percent are snoop and credit traffic that will grow with the workload, the real headroom is ten, and the growth plan is wrong by a factor of four.

// RTL 6 - bandwidth that carries no payload. Snoops, completions without data
// and credit returns all occupy the link, and a counter that measures only
// data traffic reports headroom the link does not have.
module overhead_traffic #(parameter int DATA_IS_THE_TRAFFIC = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] link_bw, data_bytes, snoop_bytes, credit_bytes,
  output logic [15:0] acct_bytes, hidden_bytes, headroom, real_headroom,
  output logic [15:0] util_pct,
  output logic        overhead_accounted,
  output logic [7:0]  n_measures, n_hidden,
  output logic        unaccounted_err
);
  logic [15:0] all_bytes, true_hidden;
  logic [31:0] u_q;
  logic        truly_hidden;
  assign true_hidden = snoop_bytes + credit_bytes;
  assign all_bytes   = data_bytes + true_hidden;
  assign acct_bytes  = (DATA_IS_THE_TRAFFIC != 0) ? data_bytes : all_bytes;
  assign hidden_bytes = all_bytes - acct_bytes;
  // Headroom is what is left of the link after everything actually on it.
  assign real_headroom = (all_bytes  > link_bw) ? 16'd0 : (link_bw - all_bytes);
  assign headroom      = (acct_bytes > link_bw) ? 16'd0 : (link_bw - acct_bytes);
  assign u_q = (link_bw == 16'd0) ? 32'd0
             : (({16'd0, acct_bytes} * 32'd100) / {16'd0, link_bw});
  assign util_pct = (u_q > 32'd100) ? 16'd100 : u_q[15:0];
  assign overhead_accounted = (hidden_bytes == 16'd0);
  assign truly_hidden = (true_hidden != 16'd0) && (link_bw != 16'd0);
  // The utilisation number was computed from the data counter alone.
  assign unaccounted_err = measure && truly_hidden && (acct_bytes == data_bytes);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_hidden <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (truly_hidden) n_hidden <= n_hidden + 8'd1;
    end
  end
endmodule

Sixty bytes of data, twenty of snoop and ten of credit on a hundred-byte link is ninety bytes of traffic and ten of headroom. The data counter reports sixty bytes, forty of headroom and sixty percent utilisation, with thirty bytes it never counted.

Figure 3 — the headroom a capacity plan is built from. The data counter is not broken and is not lying; it is counting data, which is what it was built to count. The thirty bytes it does not see are snoop and credit traffic that will grow with the workload rather than staying fixed, which is why the plan is wrong by a term rather than by a margin.

The over-subscribed case is where the error becomes dangerous rather than merely wrong. Eighty bytes of traffic on a sixty-four byte link has no headroom at all and is a hundred percent utilised, and the data counter reports fourteen bytes spare at seventy-eight percent. That link is dropping or stalling right now, and the instrument watching it says there is room.

The fifth case is the limit: a link carrying nothing but overhead reads as completely idle. Thirty bytes of snoop and credit traffic with no data at all is thirty percent utilisation to the traffic model and zero percent to the data counter. A coherency storm — many snoops, little data movement — produces exactly this shape, and the instrument most teams are watching will show a flat line through it.

Two guards keep the model from firing on nothing. A link with no rate cannot overstate its headroom, and a link carrying nothing but payload has nothing hidden, so both builds agree. Without those, every idle and every unconfigured link in a fleet would raise this alarm.

The distinction the model draws between the headroom it reports and the headroom that is really there is the shape of the whole section. Both are correct subtractions; they differ only in what was subtracted. A measurement is made wrong by a correct calculation over an incomplete input far more often than by bad arithmetic, and the second failure is much harder to catch because every intermediate value looks right. The only real defence is to name what the counter counts, in the dashboard, next to the number.

11. RTL 7 — The Bottleneck Moves

Nine sections identify what is slow. This one is about what happens after you fix it, and it is the section most likely to change a schedule.

A system has more than one limiter. Accelerating the one that dominates today shifts the load onto the next one, and the gain you actually realise is bounded by how much room the second limiter leaves. If the second limiter is close behind the first, a fix that looks like a forty-five-point improvement on paper delivers twenty, and the difference is discovered after the engineering has been spent.

// RTL 7 - the bottleneck moves. Speeding up the part that is slow is bounded
// by the part that is not, and the second limiter is often close enough to the
// first that the fix is not worth what it cost.
module limiter_shift #(parameter int FIX_THE_SLOW_PART = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] slow_pct, speedup_x, second_limit_pct, effort_weeks,
  output logic [15:0] fast_pct, capped_gain_pct, realised_pct, gain_per_week,
  output logic        gain_worth_it,
  output logic [7:0]  n_evals, n_capped,
  output logic        amdahl_err
);
  logic [15:0] share, remaining, ideal_gain;
  logic [31:0] c_q, g_q;
  logic        truly_capped;
  // The accelerated share cannot exceed the whole.
  assign share = (slow_pct > 16'd100) ? 16'd100 : slow_pct;
  assign fast_pct = 16'd100 - share;
  // What the fix would give if nothing else limited it.
  assign c_q = (speedup_x == 16'd0) ? 32'd0
             : ({16'd0, share} - ({16'd0, share} / {16'd0, speedup_x}));
  assign ideal_gain = (c_q > 32'd100) ? 16'd100 : c_q[15:0];
  // The second limiter caps what any fix to the first can deliver.
  assign remaining = (second_limit_pct > 16'd100) ? 16'd100 : second_limit_pct;
  assign capped_gain_pct = (ideal_gain > remaining) ? remaining : ideal_gain;
  assign realised_pct = (FIX_THE_SLOW_PART != 0) ? ideal_gain : capped_gain_pct;
  assign g_q = (effort_weeks == 16'd0) ? 32'd0
             : ({16'd0, realised_pct} / {16'd0, effort_weeks});
  assign gain_per_week = (g_q > 32'hFFFF) ? 16'hFFFF : g_q[15:0];
  assign gain_worth_it = (gain_per_week != 16'd0);
  assign truly_capped = (ideal_gain > remaining);
  // The projection ignores the limiter that will take over.
  assign amdahl_err = evaluate && truly_capped && (realised_pct == ideal_gain);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_capped <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_capped) n_capped <= n_capped + 8'd1;
    end
  end
endmodule

Sixty percent of the time in the slow part, accelerated fourfold, is a forty-five-point ideal gain. With a second limiter twenty points away, the fix delivers twenty points, not forty-five — two points per week of effort against a projected five. The single-limiter view projects the forty-five, because that is what the arithmetic says when nothing else is modelled.

The last case is the one that most often reverses a decision. A hundredfold speedup on the slow part still delivers twenty points, because the cap does not move with the speedup. Increasing the aggressiveness of a fix increases its cost, increases its projected gain, and leaves its delivered gain exactly where it was — which means the more ambitious version of the project has a strictly worse return than the modest one. Arguing that in a planning meeting requires the second limiter to be a number rather than an intuition, and that is the entire reason this model publishes it.

Two things follow that are easy to state and hard to act on. First, the second limiter has to be measured before the first one is fixed, because afterwards there is no way to distinguish a fix that underdelivered from a fix aimed at the wrong thing — both look like a disappointing benchmark. Second, the projection worth quoting is the capped one even when the cap is uncertain, because a projection that comes in low is survivable and a projection that comes in high spends its error in trust. Section 23 is about what that costs in practice.

The degenerate cases are quieter but load-bearing. A fix with no speedup is capped by nothing because it gains nothing. A slow part that is zero percent of the run time has nothing to accelerate. And a capped fix with no effort estimate still has its cap — the overstatement does not depend on the estimate, which means the projection is wrong before anybody has argued about how long it will take.

12. RTL 8 — The Measurement Window

The last of the measurement errors, and the one that most reliably makes a real problem invisible.

Utilisation is bytes divided by time. Which time is a choice, and the choice determines the answer. A link saturated for ten cycles out of a thousand carried the same bytes either way — but reported per burst it is a hundred percent utilised, and reported per window it is zero. The application experienced the burst. The monitoring system recorded the window.

// RTL 8 - the measurement window. A burst that saturates the link for a
// millisecond, averaged over a second, reads as a link with nothing on it -
// and the application felt every microsecond of the burst.
module window_averaging #(parameter int THE_AVERAGE_IS_THE_LOAD = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] link_bw, burst_cyc, window_cyc, burst_bytes,
  output logic [15:0] peak_bw, mean_bw, reported_bw, util_pct,
  output logic        window_honest,
  output logic [7:0]  n_measures, n_hidden_peaks,
  output logic        averaging_err
);
  logic [15:0] span_cyc;
  logic [31:0] p_q, m_q, u_q;
  logic        truly_peaked;
  // A burst cannot be longer than the window it was measured in.
  assign span_cyc = (burst_cyc > window_cyc) ? window_cyc : burst_cyc;
  assign p_q = (span_cyc == 16'd0) ? 32'd0
             : ({16'd0, burst_bytes} / {16'd0, span_cyc});
  assign peak_bw = (p_q > 32'hFFFF) ? 16'hFFFF : p_q[15:0];
  assign m_q = (window_cyc == 16'd0) ? 32'd0
             : ({16'd0, burst_bytes} / {16'd0, window_cyc});
  assign mean_bw = (m_q > 32'hFFFF) ? 16'hFFFF : m_q[15:0];
  assign reported_bw = (THE_AVERAGE_IS_THE_LOAD != 0) ? mean_bw : peak_bw;
  assign u_q = (link_bw == 16'd0) ? 32'd0
             : (({16'd0, reported_bw} * 32'd100) / {16'd0, link_bw});
  assign util_pct = (u_q > 32'd100) ? 16'd100 : u_q[15:0];
  assign window_honest = (reported_bw >= peak_bw);
  assign truly_peaked = (peak_bw > mean_bw) && (peak_bw >= link_bw);
  assign averaging_err = measure && truly_peaked && (reported_bw == mean_bw);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_hidden_peaks <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (truly_peaked) n_hidden_peaks <= n_hidden_peaks + 8'd1;
    end
  end
endmodule

A ten-cycle burst delivering six hundred and forty bytes on a sixty-four gigabyte link is the link at a hundred percent for ten cycles. Averaged over a thousand-cycle window it is zero, and the averaging view reports an idle link.

The sixth case is the important qualification, and it is what keeps this model from crying wolf. A burst that does not reach the link rate is not a hidden saturation. A sixty-four gigabyte burst on a hundred-and-twenty-eight gigabyte link is a busy moment on a link with headroom, and the model does not report it even though the averaging view still shows zero. The failure this section owns is specifically a saturated link that reads as idle, not any burst at all.

The saturating case is the most instructive. A one-cycle burst of sixty-five thousand bytes has a peak the counter cannot hold and a mean of sixty-five. Both builds report a hundred percent utilisation — and the two hundred percents are a thousandfold apart in what they describe. A percentage that matches is not a measurement that agrees.

The remedy is not a shorter window; it is more than one. A single window length answers exactly one question — is the link busy on this timescale — and a monitoring system with one window is permanently blind to every phenomenon faster than it. Two window lengths an order of magnitude apart cost almost nothing and turn an invisible burst into a visible ratio between them, which is simultaneously the detection and the measurement of how bursty the traffic is.

The clamp is the usual shape: a burst cannot outlast the window it was measured in, and a report that says otherwise is two counters read at different instants.

13. RTL 9 — Which Resource Saturates Is The Cheapest Signal

Every previous section identifies a candidate. This one is about choosing between them, because in performance work the cost is dominated by how many candidates you tune rather than how hard each one is.

Three candidate limiters and one utilisation read: if one of them is at ninety-five percent and the others are at forty and thirty, there is one candidate and the other two are a waste of a week. If all three are at fifty, there is no single bottleneck and tuning any one of them individually will produce a disappointing, confusing result.

// RTL 9 - which resource saturates is the cheapest signal. Three candidate
// limiters and one utilisation read turn a week of bisection into an
// afternoon of confirmation.
module saturation_attribution #(parameter int OPTIMISE_EVERYTHING = 0) (
  input  logic clk, rst_n,
  input  logic        plan_it,
  input  logic [15:0] util_a, util_b, util_c, tune_cost,
  output logic [15:0] hottest_util, candidates, isolate_cost, saving_pct,
  output logic        attributed,
  output logic [7:0]  n_plans, n_blind,
  output logic        saturation_blind_err
);
  logic [15:0] max_ab, true_candidates, blind_cost;
  logic [31:0] i_q, b_q, s_q;
  logic        truly_attributable;
  assign max_ab = (util_a > util_b) ? util_a : util_b;
  assign hottest_util = (max_ab > util_c) ? max_ab : util_c;
  // How many of the three are within reach of the hottest one.
  assign true_candidates = ((util_a == hottest_util) ? 16'd1 : 16'd0)
                         + ((util_b == hottest_util) ? 16'd1 : 16'd0)
                         + ((util_c == hottest_util) ? 16'd1 : 16'd0);
  assign candidates = (OPTIMISE_EVERYTHING != 0) ? 16'd3 : true_candidates;
  assign i_q = {16'd0, candidates} * {16'd0, tune_cost};
  assign isolate_cost = (i_q > 32'hFFFF) ? 16'hFFFF : i_q[15:0];
  assign b_q = 32'd3 * {16'd0, tune_cost};
  assign blind_cost = (b_q > 32'hFFFF) ? 16'hFFFF : b_q[15:0];
  assign s_q = (blind_cost == 16'd0) ? 32'd0
             : ((blind_cost > isolate_cost)
                ? ((({16'd0, (blind_cost - isolate_cost)}) * 32'd100)
                   / {16'd0, blind_cost}) : 32'd0);
  assign saving_pct = (s_q > 32'd100) ? 16'd100 : s_q[15:0];
  assign attributed = (candidates == 16'd1);
  assign truly_attributable = (true_candidates == 16'd1) && (hottest_util != 16'd0);
  // One resource is hottest and the plan still tunes all three.
  assign saturation_blind_err = plan_it && truly_attributable && (candidates > 16'd1);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_blind <= 8'd0;
    end else if (plan_it) begin
      n_plans <= n_plans + 8'd1;
      if (!attributed) n_blind <= n_blind + 8'd1;
    end
  end
endmodule

One resource at ninety-five percent against two well below it is one candidate, eight hours to confirm, sixty-six percent cheaper than tuning all three. The tune-everything plan takes twenty-four hours, saves nothing and attributes nothing.

The two counter-cases are what make the model usable rather than merely encouraging. Two resources tied at the top is not an attribution — the model reports two candidates, a thirty-three percent saving over tuning all three, and explicitly declines to say which. Three equally loaded resources is the one case where the blind plan is right: there is no single bottleneck, tuning all three is the correct plan, and the saving is zero. An attribution model that reported a winner in either of those cases would be worse than no model, because it would send a week of work at a resource chosen by rounding.

The degenerate case is an idle system, where the hottest resource is at zero percent and there is nothing to find. The model requires a non-zero hottest utilisation before it will attribute anything, which is the guard that stops it from confidently naming a bottleneck in a system that is not doing any work.

The free-counter case is the design-review argument, and it is the same argument as 26.5 section 13 in a different costume. Utilisation counters that cost nothing to read still attribute, and ignoring them is still the expensive plan. The saving is unquotable when the counters are free — there is no cost to compare against — and the attribution is worth exactly as much as it was.

14. RTL 10 — A Throughput Diagnosis Assembled

Nine models, nine independent claims. This one puts them in one place and makes the weak claim visible as what it is: one bit of six.

"The link is running at full speed" is what a link-status register reports, and it is not wrong — it is one of six conditions, and the only one of the six that any register anywhere reports.

// RTL 10 - a throughput diagnosis assembled. Everything that must hold before
// a link is delivering what it is quoted at, with "the link is running at full
// speed" as one of the six rather than the whole claim.
module throughput_signoff #(parameter int RUNNING_AT_FULL_SPEED = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       line_rate_ok,       // the link trained at its full rate
  input  logic       enough_outstanding, // requests in flight fill the pipe
  input  logic       tail_bounded,       // the slow requests are inside budget
  input  logic       direction_ok,       // neither direction is over its half
  input  logic       granule_ok,         // accesses land on every device
  input  logic       overhead_accounted, // non-payload traffic is counted
  output logic       throughput_sound,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_sound,
  output logic       false_speed_err
);
  assign fail_mask[0] = ~line_rate_ok;
  assign fail_mask[1] = ~enough_outstanding;
  assign fail_mask[2] = ~tail_bounded;
  assign fail_mask[3] = ~direction_ok;
  assign fail_mask[4] = ~granule_ok;
  assign fail_mask[5] = ~overhead_accounted;
  // The full-speed build is what a link-status register reports.
  assign throughput_sound = (RUNNING_AT_FULL_SPEED != 0)
                            ? line_rate_ok : (fail_mask == 6'd0);
  assign false_speed_err = evaluate && throughput_sound && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_sound <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (throughput_sound) n_sound <= n_sound + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When the link trained at its full rate and any one of the other five fails, the assembled model reports the failure and the register reports a link at full speed. Only when the link itself did not train do the two agree.

Across the eight evaluations the stimulus drives, the assembled model calls one link sound and the register view calls six of them full speed. The five it gets wrong are the five single-bit failures with the line-rate bit set, and every one of them is a real link that a real register has reported as healthy.

The bit order is again a cost order, but the costs are shaped differently from 26.5. Bit 0 is a register read. Bit 1 needs a latency measurement and an outstanding count. Bit 2 needs a distribution rather than a mean, which is the first bit that requires the monitoring system to have been built for it. Bits 3 and 4 are configuration reads against a workload description. Bit 5 needs counters most links do not implement. The cost is not the measurement; it is whether the instrument exists at all, and that is a decision made at design time. Section 22 is about making it correctly.

Figure 4 — the mask read as a triage order. The first two decisions are arithmetic on numbers that already exist and resolve most bandwidth complaints between them. The last two need an instrument that may not have been built, which is why they sit at the bottom and why section 22 is a design-review section rather than a debug one.

15. Quantitative Reasoning

Numbers from the models, stated so they can be argued with rather than admired.

Sixteen bytes of framing in eighty is twenty-one percent of the link. Not an error bar, not a measurement uncertainty — a fixed, computable tax determined by a frame-format decision, and absent from every register that reports a rate.

Eight outstanding requests on a thirty-two cycle round trip is twenty-five percent. Thirty-two would be a hundred. The ratio between what is in flight and what is needed is the whole diagnosis, and both numbers come from counters that most hosts already have.

A tail eight times the mean is ninety-nine percent inside budget. Ten requests in a thousand. If an operation needs a hundred results, it will encounter one of those ten on roughly two operations in three — the ninety-nine percent figure and the user experience are almost unrelated.

A one-directional workload gets half the quoted number. Exactly half, on a symmetric link, every time. The thirty-two gigabyte overquote in section 8 is not an approximation.

An interleave four times coarser than the access delivers a quarter of the devices. One of four, seventy-five percent serialised, on a correctly configured four-way interleave with four healthy devices present.

Thirty bytes of overhead on a hundred-byte link turns forty bytes of apparent headroom into ten. A factor of four on the number a capacity plan is built from, and the error grows with the workload rather than staying constant.

A fourfold speedup on sixty percent of the run time delivers twenty points, not forty-five. Two points per week against a projected five: the project is worth 40 percent of what the proposal said, and the difference is entirely in a number nobody was asked for.

A ten-cycle burst in a thousand-cycle window reads as zero. The link was at a hundred percent for ten cycles. Both statements are true and only one of them is in the dashboard.

One hot resource of three is sixty-six percent cheaper to fix. Eight hours against twenty-four, from a utilisation read that takes minutes.

One of eight links sound, six of eight at full speed. The assembled model's summary number, and the chapter's.

16. Assertions

The testbenches carry 529 checks across ten models, and their structure is deliberate.

Every output of every model is asserted as a value, in both builds. An output listing step runs before the mutation campaign and reports any output net connected to a model instance that never appears in a check. This chapter's first run reported eighteen, of which one was a real gap: the interleave model's serialised percentage, which is the broken build's own headline figure — it reads zero where the measured build reads seventy-five. The pattern is now familiar enough to predict: the output most likely to be unasserted is the one the broken build gets most spectacularly wrong, because the reviewer's attention goes to the measured build.

Both builds are asserted on every degenerate case. A link with no rate, a frame with no size, a measurement with no latency, a window with no length, an interleave with no granule, a system with no load, requests that carry no bytes, an effort estimate of zero. Every model has at least two.

Every clamp is driven past its limit exactly once. A payload larger than its frame, an in-flight figure beyond sixteen bits, a tail below the mean, demand above the aggregate, an access larger than the whole interleave, traffic beyond the link capacity, a slow share above a hundred percent, a burst longer than its window, a tuning cost that saturates the estimate.

Every error output is checked in both directions in every case. The measured build must never fire; the reported build must fire on exactly the cases the section owns and stay quiet elsewhere. Section 8's third case and section 12's sixth case exist specifically to assert the quiet half — situations where the reported build happens to be right, and a model that alarmed on them would be untrustworthy.

The counters are asserted at the end of each model's block. Seven measurements with two overstated; seven samples with four long tails; eight evaluations with three capped. A counter assertion catches a mutation that corrupts a decision without moving any single output far enough to fail a value check.

17. Mutation Testing

123 mutations, 123 killed. Fifty-nine against the first testbench, sixty-four against the second, both sets clean on the first run.

Four mutations were designed and then discarded as equivalent, and the reasoning is worth recording because three of the four had the same shape. Loosening a strict comparison to an inclusive one changes nothing unless some stimulus actually produces the equality, and in the outstanding-request model, the cap model and the granule model no case did. The fourth was subtler: perturbing the ceiling-division constant in the interleave model was absorbed entirely by the device clamp beneath it, so the mutant computed a different intermediate and an identical output. A replacement mutation needs its own reachability argument, and a mutation upstream of a clamp needs an argument that it escapes the clamp.

Three mutation families are worth calling out because each caught something a reviewer would not.

The zero-guard flip — turning a degenerate input's result from zero into a hundred, or from one into nine — is the family that grew most this batch, from nine in 26.5 to twelve here. Each of those guards exists because a real instrument supplies the degenerate value, and each flip produces a model that reports a confident full percentage on a measurement that contains nothing.

The signal substitution — the attribution model judging its plan by the evidence, the interleave model measuring the ideal against the achieved, the goodput model taking the mean where it should take the tail — is the family that most resembles a real bug, because it is what a careless rename produces.

The counter mutation changes no published value in any single cycle. Only an end-of-block count assertion catches it, and in the overhead model the obvious inversion produced an identical count by coincidence: seven measurements would have split three-and-three, and inverting the counter gave the same total. A counter mutation that yields the same total is an equivalent mutant wearing a disguise — and the disguise is the stimulus, not the code. The fix is a seventh measurement that unbalances the split, after which the inversion is killed on the first run. 26.7 section 17 gives the general rule.

18. Verification Strategy

A verification plan for performance inherits a structural difficulty: the quantity that matters is usually not the quantity that is measured, so a plan organised around "measure throughput" will pass while every one of these six bits is set.

Measure goodput, not line rate, and derive it rather than assuming it. The frame format is known at design time. Section 5's efficiency figure can be computed before silicon exists, and any measurement that disagrees with it is an instrument problem or a framing problem — both worth knowing about.

Sweep outstanding requests, not just bandwidth. A single-point bandwidth measurement cannot distinguish a link limit from a requestor limit. A sweep can: throughput that rises linearly with outstanding requests and then flattens shows exactly where the limiter changes hands, and where it flattens is depth_needed.

Report a distribution, never a mean. Section 7 is undetectable from a mean, and the cost of carrying a percentile in a performance harness is trivial compared to the cost of not having one when a tail appears. If only one number can be carried, carry the ninety-ninth percentile rather than the average.

Exercise one direction at a time, then both. Section 8's error is invisible to a symmetric benchmark and obvious to an asymmetric one. A plan that only runs the symmetric case is the reason this mistake ships.

Test the interleave with the access size the application actually uses. A sixty-four byte access and a kilobyte access on the same configuration produce a factor of four difference in delivered bandwidth. A performance test that uses large accesses will report the interleave working perfectly for an application that uses small ones.

Count all traffic, not just data. Section 10 needs a counter that most designs do not have, which makes it a design-review item rather than a test-plan item. Where the counter does not exist, the substitute is to measure delivered data against theoretical goodput and treat the gap as overhead — imprecise, and far better than assuming it is zero.

Measure at more than one window length. Section 12's burst is visible at ten cycles and invisible at a thousand. A harness that records a single window length will find whichever failures happen to match it.

19. Synthesis and Implementation Reality

The models are teaching models, but the counters they assume are real, and nearly all of them are cheap enough that their absence is a decision rather than a constraint.

Frame efficiency is fixed at frame-format definition. Section 5's twenty-one percent is not tunable later. The design-time decision — how much payload per frame — sets the ceiling on every performance number the product will ever quote, and it is usually made by the link team for link reasons.

Outstanding-request counters are two registers and are the highest-value performance observability a host can have. A count of requests in flight and a round-trip latency histogram resolve section 6 completely and section 13 substantially. They are small, they are debug-only in the sense that nothing functional depends on them, and they are among the first things cut.

A latency histogram is worth more than a latency counter. A sum and a count give a mean, which section 7 exists to discredit. A small number of buckets — eight is plenty — gives a percentile. The area difference is negligible and the information difference is the whole of section 7.

Directional counters must be separate. A single byte counter cannot distinguish section 8's case from a balanced workload, and merging them to save a register throws away the only evidence for a factor-of-two error.

Interleave granularity should be reported alongside the interleave width. A configuration dump that says "four-way interleaved" without the granule cannot be checked against an access size, which makes section 9 undetectable from configuration alone.

Non-data traffic needs its own counter. This is the one item on the list that is genuinely not free — it means instrumenting the snoop and credit paths as well as the data path — and it is the one that section 10 shows is worth the most at capacity-planning time.

Counters must be readable without perturbing the traffic they measure. A counter read that stalls the link makes every measurement in this chapter slightly wrong and section 12's burst measurement completely wrong.

20. Silicon Observability

What can be read from a real link, ordered by what it costs to get.

Free, already there. Link state, trained width, trained speed. This answers bit 0 of section 14's mask and nothing else, which is precisely the chapter's subject.

Cheap, if the counters exist. Bytes transferred per direction, requests issued, requests completed. These give goodput against line rate and, with a time base, throughput. Combined with an outstanding count they give section 6 directly.

Moderate. A latency histogram, or even a max-latency register with a sticky flag. The sticky maximum is the cheapest possible approximation of section 7 and catches the case where a tail exists at all, which is most of the value.

Moderate to expensive. Non-data traffic counters, which need instrumentation on paths that carry no payload and are therefore easy to overlook when the counter list is drawn up.

Expensive. A protocol analyser, which gives ground truth on framing, on burst structure and on traffic mix — everything in this chapter, on one link, for the duration of a capture. In performance work an analyser is unusually well-suited because the quantities are statistical rather than rare, and a short capture gives a good estimate. 26.7 is about capturing well.

Unobtainable. The application's own notion of which request it was waiting for. Every latency figure in this chapter is an interconnect measurement; the mapping from a tail latency to a user-visible stall lives above the interconnect and has to be supplied by whoever owns the workload.

21. Debug Lab

A host reports a CXL memory device delivering a quarter of its expected bandwidth. 26.5's checks have all passed: every hop has credit, the routes agree, there is no cycle, no shared-channel victim and no credit leak.

Step 1 — compute goodput from the frame format. Section 5. This is arithmetic, not a measurement, and it takes minutes. If the expectation was built on the line rate, the shortfall may be entirely explained here and the investigation is over.

Step 2 — read outstanding requests and round-trip latency. Section 6. Multiply and divide. If the product is a quarter of the link, the link is not the limiter and depth_needed says what to change. This step resolves more CXL bandwidth complaints than every other step combined, because the requestor is the most common limiter and the least common suspect.

Step 3 — if throughput is fine and the application is still slow, look at the distribution. Section 7. A healthy mean with a long tail is a completely different problem from a low mean, and the two are indistinguishable from a bandwidth number.

Step 4 — check the workload's direction mix against the per-direction rate. Section 8. A read-only workload on a symmetric link has a hard ceiling at half the quoted figure, and no amount of tuning moves it.

Step 5 — check the interleave granule against the access size. Section 9. This is two configuration reads and one division, and when it is wrong it is wrong by the interleave width.

Step 6 — check whether the utilisation figure counts non-data traffic. Section 10. If it does not, the headroom in the dashboard is wrong in the dangerous direction, and a capacity decision may already have been made on it.

Step 7 — if the numbers all look fine and the application disagrees, shorten the measurement window. Section 12. A burst that saturates the link for ten cycles in a thousand is real, is felt, and is invisible at the window length most monitoring uses.

Step 8 — before fixing anything, find the second limiter. Section 11. The gain is capped by it, and the cap determines whether the fix is worth doing at all.

The order is roughly the mask's bit order with one deliberate exception: section 11 is last in the list and first in the decision. Everything above it identifies a candidate; it decides whether the candidate is worth the engineering.

22. Design Review

Questions worth asking before a link exists, each cheap now and expensive later.

What is the frame efficiency, and is the quoted bandwidth a line rate or a goodput? If the answer is "line rate" and it appears in a customer-facing document, section 5's tax is about to become a support case.

Can I read outstanding requests and round-trip latency at the same instant? Two registers. Without them, section 6 is a guess.

Is there a latency histogram, or at minimum a sticky maximum? A mean cannot represent a tail, and a tail is what an application experiences.

Are byte counters per direction? A merged counter cannot see a factor-of-two ceiling.

Does the configuration report interleave granularity as well as width? Width without granule cannot be checked against an access size.

Is there any counter for traffic that carries no payload? If not, every utilisation figure the product reports is a lower bound of unknown tightness, and capacity plans built on it will be wrong in the optimistic direction.

At what window length does the monitoring system record utilisation, and can it be shortened? A fixed long window makes section 12's failure permanently invisible.

Does reading a counter perturb the link? If it does, every measurement above is slightly wrong and the burst measurement is badly wrong.

23. How This Appears In Real Engineering

The complaint is never "the goodput is lower than the line rate". It is "the device is slower than the datasheet", and it arrives with a benchmark number attached.

What follows is usually three conversations. The device team demonstrates the device meeting its specification. The link team demonstrates the link trained at full width and full speed. The application team demonstrates a benchmark returning a quarter of the expected figure. All three are correct, and none of them has looked at the same number.

The resolution is almost always arithmetic rather than investigation. Frame efficiency accounts for twenty percent. Outstanding requests account for most of the rest. Nobody's component was broken, and the days spent proving that were spent because the first question asked was what is wrong rather than what did we expect, and how did we compute it.

The second recurring shape is the capacity plan built on an optimistic headroom figure. A link at sixty percent utilisation is approved for a workload increase that will take it to ninety. It reaches saturation at what the dashboard calls seventy, because thirty percent of the link was never in the dashboard. The plan was not wrong by a margin; it was wrong by a term.

The third shape is the fix that underdelivers. A team spends eight weeks on a fourfold acceleration of the dominant cost and gains twenty points instead of forty-five. The work was competent, the speedup was real, and the projection was made without the second limiter in it. What makes this one organisationally expensive is that it damages trust in the next proposal — the following optimisation, correctly scoped and honestly projected, gets discounted because the last one missed.

24. Common Misconceptions

"The link is running at full speed, so we're getting full bandwidth." Full speed is one of six conditions and the only one a register reports. This is the chapter.

"The datasheet says 64 gigabytes a second." That is a line rate. Goodput is line rate times payload fraction. Section 5.

"The device is slow." If eight requests are in flight on a thirty-two-cycle round trip, the device is idle three quarters of the time. Section 6.

"Average latency is well inside budget." An application waits for its slowest request. Section 7.

"It's a 64-gigabyte link, so a read-heavy workload gets 64." It gets 32. Section 8.

"It's four-way interleaved, so we get four devices of bandwidth." Only if the access is at least as large as the granule. Section 9.

"We're at 60 percent, so there's 40 percent of headroom." Only if the counter counts everything on the link. Section 10.

"A fourfold speedup on 60 percent of the time gives 45 points." Only if nothing else limits it. Section 11.

"Utilisation is 5 percent, the link is nearly idle." Over what window? Section 12.

"Let's tune all three and see what helps." One utilisation read costs minutes and eliminates two of them. Section 13.

25. Interview Reasoning

"A CXL device is quoted at 64 gigabytes a second and a benchmark measures 16. Every fabric check passes. Walk me through it." Goodput first — frame efficiency accounts for a fixed fraction before anything else is considered. Then outstanding requests against latency, because a requestor limit is the most common cause and produces exactly this shape. The reasoning being tested is whether you compute an expectation before investigating a deviation.

"Mean latency is 100 cycles against a 500-cycle budget, and the application reports stalls. Explain." The application waits for its slowest request. A tail at the ninety-ninth percentile above the budget affects one request in a hundred, which for an operation gathering a hundred results is most operations. The follow-up is what instrument would show it, and the answer is a histogram or a sticky maximum, not a mean.

"You have a read-heavy workload on a full-duplex link quoted at 64 gigabytes a second aggregate. What is the ceiling?" Half, and it is a hard ceiling. The interesting part of the answer is why a symmetric benchmark will never reveal it, which is the reason the mistake survives to production.

"A system is 4-way interleaved across four healthy devices and delivers the bandwidth of one. What would you check?" The granule against the access size. An interleave finer than the access spreads nothing, and it is a configuration change rather than a hardware problem — which makes it one of the cheapest performance fixes available if it is found.

"You can make the dominant cost four times faster in eight weeks. How do you decide whether to do it?" Find the second limiter first, because it caps the realised gain regardless of how good the fix is. A strong answer notes that a more aggressive version of the same fix has a strictly worse return, since the cost rises and the delivered gain does not.

"A dashboard shows 5 percent link utilisation and the application reports stalls. Reconcile them." Window length. A burst saturating the link for ten cycles in a thousand is a hundred percent utilisation and 1 percent of a window average, and the application felt every cycle of it.

26. Exercises

1. A link signals at 100 gigabytes a second and carries 96 payload bytes in a 128-byte frame. Compute goodput and efficiency. Now halve the frame size with the same 32 bytes of header and recompute.

2. A host supports 16 outstanding 64-byte reads. Round-trip latency is 80 cycles. What throughput results? How many outstanding requests would fill a 64-gigabyte link, and what does that imply about the host?

3. A distribution has a mean of 200 cycles and a 99th percentile of 3,000 against a 1,000-cycle budget. How many requests in 10,000 miss? If an operation needs 50 results, roughly what fraction of operations are affected?

4. A full-duplex link is quoted at 128 gigabytes a second aggregate. A workload is 80 percent reads. What fraction of the quoted figure can it draw? At what read-write ratio does the quoted figure become achievable?

5. A pool is 8-way interleaved with a 512-byte granule. The application issues 128-byte accesses. How many devices does each access touch, and what fraction of the pool bandwidth is available? What granule would fix it?

6. A link measures 70 percent utilisation from a data counter. Snoop traffic is 15 percent of link capacity and credit returns are 8 percent. What is the real headroom, and by what factor is the capacity plan wrong?

7. 70 percent of run time is in a component you can make 10 times faster. The second limiter allows a 25-point gain. Compute the ideal and realised gains. At what speedup does the ideal gain first exceed the cap?

8. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify its position in the bit order using the rule that the cost of confirming a bit is dominated by whether the instrument exists.

27. Summary

A line rate is not a throughput. A link-status register answers one of six questions, and it is the only one of the six that any register answers.

Goodput is line rate times payload fraction, fixed at frame-format definition and absent from every register.

Throughput is what is in flight divided by how long each one takes. Eight requests on a thirty-two-cycle round trip is a quarter of the link, and the link is not the limiter.

An application waits for its slowest request. A mean of a hundred with a tail at eight hundred is ninety-nine percent inside budget and felt on most operations.

A bidirectional figure is two numbers a one-directional workload cannot both use. Exactly half, every time, and a symmetric benchmark will never show it.

An interleave finer than the access does nothing. Four devices, one touched, seventy-five percent serialised — and it is a configuration change to fix.

A data counter reports headroom that does not exist. Thirty bytes of overhead turns forty bytes of apparent headroom into ten, and the error grows with the workload.

The bottleneck moves. A fourfold speedup on sixty percent delivers twenty points, not forty-five, and a hundredfold speedup delivers the same twenty.

Utilisation is bytes divided by a time you chose. A ten-cycle burst in a thousand-cycle window reads as zero, and the application felt all ten.

Which resource saturates is the cheapest signal — sixty-six percent cheaper than tuning all three, from a read that takes minutes.

Six bits, and "running at full speed" is one of them. One link of eight is sound; a status register calls six of them full speed.

26.7 takes all of this to a bench, where the counters are whatever the silicon happened to implement.