CXL Throughput

18.1 asked how long one transaction takes. This chapter asks the other question: how many can be in flight at once, and what stops the next one.

They are not independent. Little's law joins them, and section 7 is where the two chapters meet.

1. The Engineering Problem — The Line Rate Is Not The Bandwidth

Six things separate an achievable rate from the number on the box.

The raw rate is an encoding ceiling, not a delivery. A flit carries a header whatever its size, and the payload fraction comes straight off the top before anything else happens. Section 5.

Three protocols share one link. .io, .cache and .mem are interleaved on the same wire, and a model that gives each of them the whole link is right only when one of them is running alone. Section 6.

Bandwidth needs concurrency, and the amount needed grows with latency. Little's law: at a fixed request size, doubling the latency doubles the outstanding requests needed to sustain the same rate. Section 7.

Reads and writes do not cost the same, so the achievable rate moves with the mix — the same argument 17.2 section 12 made about latency, arriving as bandwidth. Section 9.

The device has its own ceiling. A 400 Gbps link in front of a 250 Gbps media delivers 250, and the other 150 is link nobody can use. Section 10.

And credits cap throughput below both. Credits × bytes ÷ round trip is a fourth ceiling, and it is the one that is invisible in every static specification. Section 11.

This chapter against 18.1, stated precisely. That one owns the duration of one transaction. This one owns how many fit. Section 7 is the bridge and everything else is disjoint.

2. The One-Sentence Model

Achievable throughput is the minimum of several independent ceilings, and a bandwidth number is only useful if it says which one is binding — every defect below is a ceiling that was not modelled at all.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Four ceilings, three protocols, two-way read/write splits and integer-percent shares are coarser than a real analysis. They are sized so every result can be recomputed on paper and every boundary is reachable.

What is not simplified is the structure: an efficiency taken off the raw rate, a share per protocol, a concurrency requirement derived from latency, a delivered rate that is the min of link and media, a credit ceiling from the round trip, and an analysis that reports which ceiling binds and how far it can be raised.

Three things are absent by design. There is no bidirectional model — reads and writes are treated as a rate mix rather than as separate directions with their own link resources. There is no per-flit scheduling: the mix is a static share, where a real arbiter decides flit by flit and 16.3 owns that. And variance is out of scope — everything here is a mean rate, with 18.1 section 7 owning the distribution argument.

5. RTL 1 — The Raw Rate Is An Encoding Ceiling

// The raw line rate is a ceiling nothing reaches.
module link_ceiling #(parameter int RAW_IS_USABLE = 0) (
  input  logic clk, rst_n,
  input  logic        quote,
  input  logic [15:0] lanes, per_lane_gbps,
  input  logic [15:0] payload_b, flit_b,      // useful bytes per flit, total bytes
  output logic [31:0] raw_gbps, usable_gbps,
  output logic [7:0]  efficiency_pct,
  output logic [15:0] quoted_gbps,
  output logic        overquote_err
);
  logic [31:0] eff_q, use_q;
  assign raw_gbps = {16'd0, lanes} * {16'd0, per_lane_gbps};
  // Efficiency is payload over total. A flit carries a header whatever its size.
  assign eff_q = (flit_b == 16'd0) ? 32'd0
                                   : (({16'd0, payload_b} * 32'd100) / {16'd0, flit_b});
  assign efficiency_pct = (eff_q > 32'd255) ? 8'hFF : eff_q[7:0];
  assign use_q = (raw_gbps * {24'd0, efficiency_pct}) / 32'd100;
  assign usable_gbps = use_q;
  // The raw rate is what a marketing sheet quotes; usable is what arrives.
  assign quoted_gbps = (RAW_IS_USABLE != 0) ? raw_gbps[15:0] : usable_gbps[15:0];
  // Quoting more than the encoding can deliver.
  assign overquote_err = quote && ({16'd0, quoted_gbps} > usable_gbps);
endmodule

16 lanes at 32 Gbps, 60 payload bytes in an 80-byte flit:

  link: raw=512 eff=75% usable=384 | raw-is-usable quotes=512 overquotes=1

512 raw, 384 usable — 128 Gbps of the headline number is header. The overhead is not a loss under load or a degradation; it is arithmetic that applies to every flit the link has ever carried, including the first one at zero utilisation.

The bench drives a perfectly efficient encoding — 80 payload bytes of 80 — and both builds then agree at 512 with no overquote. The raw-is-usable model is not wrong in general; it is wrong exactly to the extent that the encoding has overhead, which is the same conditional shape as 18.1's fixed-only hop being correct at zero load.

6. RTL 2 — Three Protocols, One Link

// Three protocols share one link, and the mix decides what each gets.
module protocol_mix #(parameter int IGNORE_MIX = 0) (
  input  logic clk, rst_n,
  input  logic        allocate,
  input  logic [15:0] link_gbps,
  input  logic [7:0]  io_pct, cache_pct, mem_pct,
  output logic [15:0] io_gbps, cache_gbps, mem_gbps,
  output logic [7:0]  total_pct,
  output logic        oversubscribed, mix_valid,
  output logic [7:0]  n_alloc, n_rejected
);
  assign total_pct = io_pct + cache_pct + mem_pct;
  assign oversubscribed = (total_pct > 8'd100);
  assign mix_valid = !oversubscribed;
  // The ignoring build gives every protocol the whole link, which is right when
  // only one is active and wrong the moment two are.
  assign io_gbps    = (IGNORE_MIX != 0) ? link_gbps
                    : ((link_gbps * {8'd0, io_pct})    / 16'd100);
  assign cache_gbps = (IGNORE_MIX != 0) ? link_gbps
                    : ((link_gbps * {8'd0, cache_pct}) / 16'd100);
  assign mem_gbps   = (IGNORE_MIX != 0) ? link_gbps
                    : ((link_gbps * {8'd0, mem_pct})   / 16'd100);
  // ... allocation counters omitted for length
endmodule

20/30/50 of a 400 Gbps link:

  mix: io=80 cache=120 mem=200 total=100% | ignoring gives io=400

The ignoring build gives .io 400 and .mem 400 on a 400 Gbps link. Each answer is individually defensible — either protocol could have the whole link if the others were idle — and their sum is 1200 on a link that carries 400.

The oversubscription check is strict at 100: 100% is a valid mix and 101% is not. The bench drives both, because a >= comparison rejects the fully-allocated case, which is the case a capacity plan is trying to reach.

16.1 established that the three protocols must not share a queue inside a switch. This is the bandwidth counterpart: they do share the link, and the share each gets is a number somebody has to choose.

7. RTL 3 — Little's Law: Bandwidth Needs Concurrency

// Little's law: bandwidth needs concurrency, and the concurrency needed grows
// with latency.
module concurrency_required #(parameter int ASSUME_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] target_mbps, latency_ns, bytes_per_req,
  input  logic [15:0] outstanding,
  output logic [31:0] need_q,
  output logic [15:0] needed, achieved_mbps,
  output logic        sufficient,
  output logic [7:0]  n_eval, n_short,
  output logic        starved_err
);
  logic [31:0] ach_q;
  // requests in flight = target_rate * latency / bytes_per_req, with the units
  // arranged so MB/s * ns / bytes comes out dimensionless.
  assign need_q = (bytes_per_req == 16'd0) ? 32'd0
                : (({16'd0, target_mbps} * {16'd0, latency_ns})
                   / ({16'd0, bytes_per_req} * 32'd1000));
  assign needed = (need_q > 32'd65535) ? 16'hFFFF : need_q[15:0];
  // What the given concurrency actually achieves.
  assign ach_q = (latency_ns == 16'd0) ? 32'd0
               : (({16'd0, outstanding} * {16'd0, bytes_per_req} * 32'd1000)
                  / {16'd0, latency_ns});
  assign achieved_mbps = (ach_q > 32'd65535) ? 16'hFFFF : ach_q[15:0];
  // The assuming build never reports a concurrency shortfall.
  assign sufficient = (ASSUME_ENOUGH != 0) ? 1'b1 : (outstanding >= needed);
  // Claiming a target rate the outstanding count cannot sustain.
  assign starved_err = evaluate && sufficient && (outstanding < needed);
  // ... evaluation counters omitted for length
endmodule

An 8000 MB/s target at 500 ns latency with 64-byte requests:

  concurrency: needed=62 outstanding=64 achieved=8192MB/s short=3 | assuming starved=3

62 requests must be outstanding to sustain 8000 MB/s. With 64 in flight the target is met; with 32 the achieved rate is 4096 MB/s — almost exactly half, because Little's law is linear in concurrency.

The latency coupling is the reason this chapter and 18.1 are not independent:

Doubling the latency doubles the concurrency required for the same bandwidth. A system whose latency regressed — for any of 18.1's eight reasons — now needs twice the outstanding requests to hold its throughput, and if the queue depth is fixed, its throughput halves instead.

The sufficiency comparison is inclusive and driven exactly: 62 outstanding is enough and 61 is not, achieving 7808 MB/s of the 8000 target.

8. Waveform — Bandwidth Saturating With Concurrency

Transcribed from the printed trace. One stimulus stream, both builds.

9. RTL 4 — Reads And Writes Do Not Cost The Same

// Reads and writes do not cost the same on the wire, so the mix moves the
// achievable rate.
module rw_asymmetry #(parameter int SYMMETRIC = 0) (
  input  logic clk, rst_n,
  input  logic        model,
  input  logic [15:0] rd_gbps, wr_gbps,
  input  logic [7:0]  rd_pct,
  output logic [15:0] mixed_gbps, quoted_gbps,
  output logic [7:0]  wr_pct,
  output logic        overstated_err
);
  logic [31:0] mix_q;
  assign wr_pct = 8'd100 - rd_pct;
  // The blend: a mix is bounded by the slower component's share.
  assign mix_q = (({16'd0, rd_gbps} * {24'd0, rd_pct})
                + ({16'd0, wr_gbps} * {24'd0, wr_pct})) / 32'd100;
  assign mixed_gbps = mix_q[15:0];
  // The symmetric model quotes the read rate whatever the mix is.
  assign quoted_gbps = (SYMMETRIC != 0) ? rd_gbps : mixed_gbps;
  // Quoting more than the mix can deliver.
  assign overstated_err = model && (quoted_gbps > mixed_gbps);
endmodule

400 Gbps read, 200 Gbps write:

The symmetric model quotes 400 for every mix, and it is correct for exactly one of the three — the all-read case, which is the one a benchmark defaults to.

This is 17.2 section 12 in the other unit. There the asymmetry moved the mean latency by a factor of five; here it moves the achievable bandwidth by a factor of two, and in both cases the model that ignores it reports a number that does not respond to the workload at all.

10. RTL 5 — The Device Has Its Own Ceiling

// The device's own media has a ceiling, and the delivered rate is the smaller of
// the link and the media.
module media_ceiling (
  input  logic clk, rst_n,
  input  logic        deliver,
  input  logic [15:0] link_gbps, media_gbps,
  output logic [15:0] delivered_gbps, headroom_gbps,
  output logic        link_bound, media_bound,
  output logic [7:0]  n_deliver, n_media_bound,
  output logic        wasted_link_err
);
  assign delivered_gbps = (link_gbps < media_gbps) ? link_gbps : media_gbps;
  assign link_bound  = (link_gbps <= media_gbps);
  assign media_bound = (media_gbps < link_gbps);
  // Link capacity the media cannot use.
  assign headroom_gbps = media_bound ? (link_gbps - media_gbps) : 16'd0;
  // A faster link bought for a device that cannot fill the one it has.
  assign wasted_link_err = deliver && media_bound && (headroom_gbps != 16'd0);
  // ... delivery counters omitted for length
endmodule

  media: delivered=400 headroom=0 media_bound=1 of 3

Three configurations:

The middle row is the case that costs money: 150 Gbps of link bought and unusable, because the device behind it cannot produce data faster than 250. Upgrading the link changes nothing; upgrading the device changes everything, and the two are usually purchased by different people.

headroom_gbps is guarded on media_bound for the same reason 18.1 section 14 guards its deltas: link_gbps - media_gbps on the third row is an unsigned subtraction going negative, and an unguarded version reports 65,486 Gbps of wasted link on a perfectly balanced system.

The tie case — link exactly equal to media — is link_bound, not media_bound, and the bench drives it. A tie is not waste, and a system that reports one on every balanced configuration reports one constantly.

11. RTL 6 — Credits Cap It Below Both

// Credits cap throughput below both the link and the media.
module credit_ceiling #(parameter int IGNORE_CREDITS = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] credits, bytes_per_credit, rtt_ns,
  input  logic [15:0] link_gbps,
  output logic [31:0] credit_q,
  output logic [15:0] credit_gbps, achievable_gbps,
  output logic        credit_bound,
  output logic [7:0]  n_eval, n_credit_bound,
  output logic        credit_blind_err
);
  // credits * bytes / rtt, scaled so bytes-per-ns comes out in Gbps.
  assign credit_q = (rtt_ns == 16'd0) ? 32'd0
                  : (({16'd0, credits} * {16'd0, bytes_per_credit} * 32'd8)
                     / {16'd0, rtt_ns});
  assign credit_gbps = (credit_q > 32'd65535) ? 16'hFFFF : credit_q[15:0];
  // The ignoring build never lets credits bind, which is how a link with plenty
  // of raw bandwidth delivers a fraction of it for no visible reason.
  assign credit_bound    = (IGNORE_CREDITS != 0) ? 1'b0 : (credit_gbps < link_gbps);
  assign achievable_gbps = credit_bound ? credit_gbps : link_gbps;
  // Reporting the link rate while credits cap it lower.
  assign credit_blind_err = evaluate && (credit_gbps < link_gbps)
                            && (achievable_gbps == link_gbps);
  // ... evaluation counters omitted for length
endmodule

64 credits of 64 bytes over a 500 ns round trip, against a 400 Gbps link:

  credits: credit_gbps=65 bound=3 of 5 | ignoring blind=3

65 Gbps achievable on a 400 Gbps link. Six-sevenths of the link is unreachable, and nothing in the link's specification, the device's specification, or the media's rating says so. The number comes entirely from how many credits the receiver advertised and how long a credit takes to come back.

Two levers, both driven:

• More credits. 512 credits gives 524 Gbps and the constraint disappears.
• A shorter round trip. Halving it from 500 to 250 ns roughly doubles the credit bandwidth to 131.

The second is why this chapter and 18.1 meet twice. A latency regression does not only make transactions slower — it shrinks the credit bandwidth ceiling proportionally, and a system that was link-bound can become credit-bound without any change to its credits.

The boundary is strict and driven exactly: 400 credits over a 512 ns round trip is precisely 400 Gbps, which matches the link and does not bind it. 399 credits does.

Every stage of that loop is in the divisor. The flit's transit is one part of it; the receiver's holding time, the moment the slot is actually freed, and the return path are the rest. A design that counts only the wire time computes a round trip that is too short and a credit ceiling that is too high — and the error is in the optimistic direction, which is the one that produces a system delivering less than its model predicts.

The interaction with 16.3 is worth stating precisely. That chapter established that a credit must be returned exactly once — a second return inflates the pool and breaks the invariant. This chapter adds the performance half: a credit returned late does not break anything, and it lowers the ceiling by exactly the proportion of the delay. Correctness cares that the credit comes back; throughput cares when.

12. RTL 7 — Which Ceiling Is Binding, And By How Much

// Which of the four ceilings is binding, and how far it can be raised before
// something else takes over.
module binding_ceiling (
  input  logic clk, rst_n,
  input  logic        analyse,
  input  logic [15:0] link_gbps, media_gbps, credit_gbps, concurrency_gbps,
  output logic [15:0] achievable_gbps, second_gbps, gap_gbps,
  output logic [1:0]  binding_id,
  output logic [7:0]  n_analyse,
  output logic        no_binding_err
);
  logic [15:0] ab_min, ab_max, cd_min, cd_max;
  assign ab_min = (link_gbps   < media_gbps)       ? link_gbps        : media_gbps;
  assign ab_max = (link_gbps   < media_gbps)       ? media_gbps       : link_gbps;
  assign cd_min = (credit_gbps < concurrency_gbps) ? credit_gbps      : concurrency_gbps;
  assign cd_max = (credit_gbps < concurrency_gbps) ? concurrency_gbps : credit_gbps;
  assign achievable_gbps = (ab_min < cd_min) ? ab_min : cd_min;
  // The SECOND-smallest of the four, not the larger of the two pair minimums:
  // once the smallest is removed, the next one up is the other member of its own
  // pair or the smaller member of the other pair, whichever is less.
  assign second_gbps = (ab_min < cd_min)
                     ? ((ab_max < cd_min) ? ab_max : cd_min)
                     : ((cd_max < ab_min) ? cd_max : ab_min);
  // How far the binding ceiling can be raised before another one binds. Raising
  // it further than this buys nothing.
  assign gap_gbps    = second_gbps - achievable_gbps;
  assign binding_id  = (achievable_gbps == link_gbps)   ? 2'd0
                     : (achievable_gbps == media_gbps)  ? 2'd1
                     : (achievable_gbps == credit_gbps) ? 2'd2 : 2'd3;
  // Every ceiling equal means raising any one alone changes nothing.
  assign no_binding_err = analyse && (gap_gbps == 16'd0);
  // ... analysis counter omitted for length
endmodule

Link 400, media 250, credits 600, concurrency 800:

  ceiling: achievable=250 binding=1 second=400 gap=150

The media binds at 250, and raising it is worth exactly 150 Gbps. Past 400 the link takes over, so a device upgrade that doubles the media bandwidth delivers 150 of the 250 it promises.

Three configurations are driven, and the third is what makes the analysis honest: every ceiling at 300 produces a gap of zero and no_binding_err. There is no single thing to raise — every ceiling has to move together, which is a different and much more expensive engineering problem than a single binding constraint.

13. RTL 8 — Allocating Bandwidth Across Hosts

// Bandwidth allocated across hosts: a share is not a guarantee unless the shares
// sum to what exists.
module bw_allocation #(parameter int OVERSUBSCRIBE = 0) (
  input  logic clk, rst_n,
  input  logic        request,
  input  logic [15:0] link_gbps,
  input  logic [15:0] h0_gbps, h1_gbps, h2_gbps,
  output logic [31:0] promised_gbps,
  output logic [15:0] shortfall_gbps,
  output logic        granted, oversubscribed,
  output logic [7:0]  n_req, n_granted,
  output logic        overpromise_err
);
  logic [31:0] short_q;
  assign promised_gbps  = {16'd0, h0_gbps} + {16'd0, h1_gbps} + {16'd0, h2_gbps};
  assign oversubscribed = (promised_gbps > {16'd0, link_gbps});
  // The difference needs its own name: a part-select of an expression is not legal.
  assign short_q        = promised_gbps - {16'd0, link_gbps};
  assign shortfall_gbps = oversubscribed ? short_q[15:0] : 16'd0;
  // The oversubscribing build grants every request, which works until every host
  // uses its share at once.
  assign granted = (OVERSUBSCRIBE != 0) ? request : (request && !oversubscribed);
  // Granting more than the link can carry.
  assign overpromise_err = granted && oversubscribed;
  // ... request counters omitted for length
endmodule

  alloc: promised=350 shortfall=0 granted=2 of 4 | oversubscribing granted=4

Two grants of four against four. The oversubscribing allocator granted a 450 Gbps promise on a 400 Gbps link, and a 401 on the same link, and the shortfall is exact each time.

Oversubscription is a legitimate policy when hosts do not peak together — but it is a policy, and this model's point is that the two builds are indistinguishable until they are not. The correct allocator refused two requests it could have granted if the peaks never coincided; the oversubscribing one granted two it cannot honour if they do.

The boundary is strict: exactly 400 promised on a 400 link is not oversubscribed and is granted; 401 is not. This is the fourth consecutive chapter in which an inclusive boundary would have stranded the fully-allocated configuration.

14. RTL 9 — A Burst Is Not A Rate

// A burst is not a rate: what a link sustains depends on how long you ask for it.
module burst_vs_sustained #(parameter int BURST_IS_RATE = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] burst_gbps, sustained_gbps,
  input  logic [15:0] buffer_kb, window_us,
  output logic [15:0] burst_us, quoted_gbps,
  output logic [31:0] bytes_in_window,
  output logic        window_exceeds_burst,
  output logic        overquote_err
);
  logic [31:0] bu_q, bw_q;
  // How long the buffer can absorb the difference between burst and sustained.
  assign bu_q = ((burst_gbps <= sustained_gbps) || (burst_gbps == 16'd0)) ? 32'd65535
              : (({16'd0, buffer_kb} * 32'd8)
                 / {16'd0, (burst_gbps - sustained_gbps)});
  assign burst_us = (bu_q > 32'd65535) ? 16'hFFFF : bu_q[15:0];
  assign window_exceeds_burst = (window_us > burst_us);
  // Over a window longer than the burst can last, the rate is the sustained one.
  assign quoted_gbps = (BURST_IS_RATE != 0) ? burst_gbps
                     : (window_exceeds_burst ? sustained_gbps : burst_gbps);
  // ... bytes_in_window and the overquote check omitted for length
endmodule

400 Gbps burst, 250 Gbps sustained, a 512 KB buffer:

  burst: burst_us=27 quoted=400 | burst-is-rate quoted=400 overquotes=2

The buffer absorbs the 150 Gbps difference for 27 µs. Any measurement window shorter than that measures 400; any window longer measures something between 400 and 250, tending to 250.

The bench drives the boundary exactly — a 27 µs window does not exceed the burst, a 28 µs window does — and it drives the degenerate case where the burst rate equals the sustained rate, in which there is no burst at all and burst_us is effectively infinite.

This is 17.4 section 7 in bandwidth rather than in thermal terms, and the structure is identical: a short measurement is a correct measurement of something other than what will be experienced. There the limit was temperature; here it is buffer depth, and the arithmetic gives the exact window at which the answer changes.

15. RTL 10 — The Throughput Model Assembled

// The throughput model assembled: every ceiling a credible number needs.
module throughput_model #(parameter int LINK_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       encoding_charged,  // flit overhead is taken off the raw rate
  input  logic       mix_modelled,      // the three protocols share the link
  input  logic       media_bounded,     // the device's own ceiling is present
  input  logic       credits_bounded,   // the credit round trip is present
  input  logic       sustained_quoted,  // the number is a sustained one
  output logic       credible,
  output logic [4:0] fail_mask,
  output logic [7:0] n_eval, n_credible,
  output logic       false_ceiling_err
);
  assign fail_mask[0] = ~encoding_charged;
  assign fail_mask[1] = ~mix_modelled;
  assign fail_mask[2] = ~media_bounded;
  assign fail_mask[3] = ~credits_bounded;
  assign fail_mask[4] = ~sustained_quoted;
  // The link-only build knows the wire and nothing on either end of it.
  assign credible = (LINK_ONLY != 0)
                  ? (encoding_charged && sustained_quoted)
                  : (fail_mask == 5'd0);
  assign false_ceiling_err = evaluate && credible && (fail_mask != 5'd0);
  // ... evaluation counters omitted for length
endmodule

Six evaluations — all ceilings present, then each dropped alone:

  model: evaluated=6 credible=1 | link-only credible=4

One credible model out of six, and the link-only model found four. Its three extra are the mix, the media and the credits — every ceiling that is a property of the system the link is in rather than of the link:

This is the third assembled model in this batch with the same shape — 17.2 section 15, 17.4 section 15, 18.1 section 15 — and the same conclusion each time: the gates a simplified model passes are the ones it can see from where it stands.

16. Quantitative Reasoning

Every number is from a printed line above. None describes any platform.

Encoding. 16 × 32 = 512 raw. 60/80 = 75% efficient. 384 usable — 128 Gbps of header.

Mix. 20/30/50 of 400 is 80/120/200. The ignoring model gives each of them 400, summing to 1200 on a 400 Gbps link.

Concurrency. 8000 MB/s × 500 ns ÷ 64 B = 62 outstanding. At 32 outstanding: 4096 MB/s. At 1000 ns latency: 125 needed.

Asymmetry. 400 read, 200 write. At 50/50 the mix is 300; the symmetric model quotes 400 — 33% over. At all-write it is 200 against 400 — 100% over.

Media. 400 link, 250 media, delivered 250, 150 of link wasted.

Credits. 64 × 64 × 8 ÷ 500 = 65 Gbps on a 400 Gbps link. 512 credits: 524. Halving the round trip: 131. Six-sevenths of the link unreachable at 64 credits.

Binding ceiling. Link 400, media 250, credits 600, concurrency 800. Achievable 250, second 400, gap 150 — the exact value of a media upgrade.

Allocation. 100 + 250 + 100 = 450 on a 400 link: shortfall 50, refused. Four requests, two granted against four.

Burst. 512 KB ÷ (400 − 250) = 27 µs. A 27 µs window measures 400; a 28 µs window measures 250.

17. Assertions

Every property is an immediate check written as cond !== 1'b1, sampled after a settle. 169 assertion sites across two testbenches.

18. Mutation Testing

71 mutations, one at a time, each required to make the baseline print RESULT: FAIL.

71 of 71 were killed.

The first run killed 69 and left 2 survivors, both boundaries never driven exactly:

"Credit-bound boundary off by one" needed a credit configuration producing precisely the link rate. The three existing configurations gave 65, 524 and 131 against a 400 Gbps link and stepped over 400 entirely; 400 credits over a 512 ns round trip lands on it exactly.

"Window boundary off by one" needed a measurement window exactly as long as the burst. The burst lasts 27 µs and the bench drove 10 and 100.

Beyond mutation testing, one genuine modelling error was caught by the assertions themselves — second_gbps computed as the larger of the two pairwise minimums, reporting 600 where the correct second-smallest was 400. Four assertions with exact expected values failed on it. A bench asserting only second_gbps > achievable_gbps would have passed the broken model.

A representative sample:

19. Verification Strategy

Two builds, one stimulus. The parameter is the only difference.

Solve for the boundary rather than approaching it. Both survivors needed an input constructed to land exactly on a comparison. 400 credits over a 512 ns round trip exists in the bench for that reason alone.

Assert exact values so a modelling error surfaces as a failure. The second_gbps bug produced four failed assertions with specific expected numbers. A relational assertion would have passed it.

Include the configuration where the shortcut is correct. A 100%-efficient encoding, an all-read mix, a matched link and media, a credit count that covers the link, a window shorter than the burst. In every case the simplified model gets the right answer.

Include the degenerate configuration. Every ceiling equal, so no_binding_err fires. A burst rate equal to the sustained rate, so there is no burst.

Guard every unsigned subtraction and drive the case that needs the guard. headroom_gbps on a link-bound configuration, shortfall_gbps on an under-subscribed one.

Move the answer so it cannot be a fixed index. The binding ceiling is driven to the media and then to the link.

20. Synthesis and Implementation Reality

This is analysis, not a datapath. What is real is the instrumentation: bytes transferred per protocol, outstanding-request occupancy, credit-return latency, and the device-side delivered rate.

Credit-return latency is the hard measurement. Section 11's ceiling is credits × bytes ÷ rtt, and the round trip is not a static number — it is 18.1's path latency including its queueing term. The credit ceiling therefore moves with load, and a system measured at low load has a credit ceiling it will not have at high load.

Outstanding-request occupancy needs sampling, not averaging. Section 7's Little's law uses the mean occupancy, which is correct, but a system that reaches its required concurrency only in bursts has a mean that says it is fine and a distribution that says it is not.

The protocol mix is decided flit by flit. Section 6's static shares are the aggregate result of 16.3's arbiter running for a long time. The shares are an outcome, not a control, unless the arbiter is explicitly configured with them.

Efficiency is not one number. Section 5's 75% assumes every flit carries a full payload. Partial flits — a short write, an unaligned access — lower it further, and the effective efficiency is workload-dependent in a way a single constant cannot express.

21. Silicon Observability

The fourth row is the one that is almost never instrumented and the one that most often explains an unexplained shortfall. A link delivering a sixth of its rate with no errors, no throttling and no queue buildup is section 11, and the only way to see it is to time a credit's return.

22. Debug Lab

Symptom: a link delivers far less than its rated bandwidth, with no errors.

Compute the four ceilings before measuring anything else. Section 12: the achievable rate is their minimum, and three of the four can be computed from configuration alone.

Check outstanding requests against Little's law. Section 7: 8000 MB/s at 500 ns needs 62 in flight. If the queue depth is 32, the system is doing exactly what it was built to do.

Time a credit's return. Section 11: 64 credits of 64 bytes over 500 ns is 65 Gbps whatever the link is rated at. This is the ceiling that appears in no document.

Compare the device-side rate with the link-side rate. Section 10: if they match and both are below the link rating, the media is binding and the link is not the problem.

Check the read/write mix. Section 9: a benchmark that was all-read and a workload that is half-write differ by 33% before anything else is considered.

Vary the measurement window. Section 14: if the number falls as the window lengthens, a buffer was absorbing the difference and the sustained rate is the lower one.

Then ask what raising the binding ceiling is worth. Section 12's gap: raising the media past 400 buys nothing because the link takes over.

23. Design Review

What is your usable rate, and what efficiency did you assume?

What share of the link does each protocol get, and is that configured or emergent?

How many outstanding requests does your target rate require at your measured latency? If nobody has computed it, the target is aspirational.

What is your credit-return round trip, and what ceiling does it imply?

Is the device or the link binding? And if the device, what is a link upgrade worth?

Over what window was the number measured, and does it change if the window doubles?

If you raised the binding ceiling, what would take over, and at what value?

24. How This Appears In Real Engineering

Throughput problems arrive as "we are not getting the bandwidth we paid for", with a measurement that is accurate.

The characteristic case is a link delivering a fraction of its rate with nothing visibly wrong. Section 11's credit ceiling is the usual answer and the hardest to see, because every component reports itself healthy and no specification contains the binding number.

The second is a benchmark that does not survive the workload. Section 9's read/write mix and section 14's measurement window are the two usual causes, and both produce a correct measurement of a different question.

The third is an upgrade that does not help. Section 12's gap is the analysis that would have predicted it: a media upgrade past the link's rate buys the gap and nothing more.

The fourth is an oversubscribed allocation discovered during a peak. Section 13's two builds are indistinguishable until every host uses its share at once, which by construction is the worst possible moment to find out.

25. Common Misconceptions

"The link is 512 Gbps." The link signals at 512. It delivers 384 at 75% flit efficiency, and everything else comes off that.

"Each protocol can use the full link." Individually, yes. Simultaneously, no — section 6's ignoring model promises 1200 Gbps on a 400 Gbps link and every individual answer is defensible.

"More bandwidth needs a faster link." It needs the binding ceiling raised, and section 12's was the media. A faster link would have bought nothing.

"We have plenty of credits." Enough for what round trip? Section 11's 64 credits are 65 Gbps at 500 ns and 131 at 250 — the same credits.

"Latency and bandwidth are separate concerns." Little's law joins them, and so do credits. Doubling the latency doubles the concurrency needed and halves the credit ceiling.

"We measured 400 Gbps." Over what window? Section 14's 512 KB buffer sustains 400 for 27 µs and 250 thereafter.

"The benchmark showed full bandwidth." At what read/write mix? Section 9's all-read case is the one where the symmetric model is correct.

"Every host gets its guaranteed share." Only if the shares sum to something the link has. Section 13's oversubscribing allocator granted 450 on a 400 Gbps link.

26. Interview Reasoning

Q1. A 16-lane link at 32 Gbps per lane. What is the bandwidth? 512 raw. Usable depends on flit efficiency — at 60 payload bytes of 80, it is 384, and every other ceiling applies to the 384.

Q2. Three protocols on one link. How much does each get? Whatever share the arbiter gives it, summing to at most 100%. A model that gives each the whole link produces answers that are individually right and collectively impossible.

Q3. 8000 MB/s target, 500ns latency, 64-byte requests. How many outstanding? 62. Little's law — rate times latency over request size.

Q4. The latency doubles. What happens to the required concurrency? It doubles, to 125. If the queue depth is fixed, throughput halves instead.

Q5. Your read benchmark shows 400 Gbps. The workload is half writes at 200. What do you get? 300. The benchmark was correct about the all-read case, which is the one case where a symmetric model is right.

Q6. A 400 Gbps link in front of a 250 Gbps device. What do you deliver? 250, with 150 of link unusable. A faster link changes nothing.

Q7. 64 credits of 64 bytes, 500ns round trip. What is the ceiling? 65 Gbps — regardless of what the link is rated at. This is the ceiling that appears in no specification.

Q8. How do you raise it? More credits, or a shorter round trip. Halving the round trip roughly doubles it, which means a latency regression shrinks the credit ceiling proportionally.

Q9. Link 400, media 250, credits 600, concurrency 800. What is achievable and what is an upgrade worth? 250, bound by the media. Raising the media is worth 150 — past 400 the link takes over.

Q10. All four ceilings are equal. What do you do? Nothing single. Raising any one alone buys zero, which is a much more expensive problem than one binding constraint and worth knowing before the purchase order.

Q11. Why is the second-smallest ceiling worth computing? Because it is the value of raising the binding one. Without it, an upgrade is sized against the promise rather than against the gap.

Q12. Three hosts want 100, 250 and 100 on a 400 Gbps link. Do you grant it? Not as a guarantee — it oversubscribes by 50. Granting it is a policy that assumes the peaks do not coincide, and it should be an explicit one.

Q13. You measure 400 Gbps over 10µs and 250 over 100µs. Which is the bandwidth? 250. The buffer absorbed the difference for 27µs, and any window longer than that measures the sustained rate.

Q14. Which throughput ceiling is hardest to observe? Credits, because it is a product of a configuration value and a measured latency, and it moves with load.

Q15. What single instrument would you add to a system with unexplained low throughput? Credit-return round-trip timing. It is the ceiling that no specification carries and the one that most often explains the shortfall.

Q16. How are this chapter and the latency chapter connected? Twice. Little's law makes the required concurrency proportional to latency, and the credit ceiling is inversely proportional to it. A latency regression degrades throughput through both.

27. Exercises

1. Extend RTL 1 so efficiency depends on request size — a short write fills less of a flit. At what request size does effective efficiency fall below half the nominal?

2. In RTL 2, make the shares emergent from a round-robin arbiter rather than configured. Does the oversubscription check still mean anything?

3. RTL 3 uses a mean latency. Feed it 18.1 section 9's utilisation-dependent latency and find the concurrency at which the system becomes unstable.

4. In RTL 5, add a third ceiling for the device's internal interconnect. Does binding_id still fit in two bits when RTL 7 consumes it?

5. Make RTL 6's rtt_ns a function of utilisation using 18.1's queueing model. At what utilisation does the credit ceiling fall below the media ceiling?

6. RTL 7 reports the second-smallest ceiling. Extend it to report the full ordering, and determine what an upgrade plan needs that the second-smallest alone does not give it.

7. In RTL 8, add a peak-coincidence probability and compute the expected shortfall rather than the worst-case one. What does overpromise_err become?

8. RTL 9 uses a single buffer. Model two buffers in series and derive the burst duration. Is it the sum, the minimum, or neither?

28. Summary

Achievable CXL throughput is the minimum of several independent ceilings, and this chapter builds all of them.

The raw rate is an encoding ceiling. 512 signalled, 384 usable at 75% flit efficiency, and every other ceiling applies to the 384.

Three protocols share one link. 80/120/200 of 400 — against a model that gives each of them 400 and promises 1200.

Bandwidth needs concurrency, and the amount scales with latency. 62 outstanding requests for 8000 MB/s at 500 ns; 125 at 1000 ns.

Reads and writes differ, so the mix moves the rate: 400 all-read, 300 at fifty-fifty, 200 all-write, against a symmetric model that quotes 400 for all three.

The device has its own ceiling. 250 delivered behind a 400 Gbps link, with 150 of link nobody can use.

Credits cap it below both. 64 credits over a 500 ns round trip is 65 Gbps — six-sevenths of the link unreachable, from a number no specification contains.

And the analysis has to say which one binds and by how much. Media at 250, link at 400, a gap of exactly 150 — the entire value of a device upgrade, computable before it is bought.

18.3 — Memory-Access Cost takes the latency of 18.1 and the throughput of this chapter and asks what a single memory access costs software, end to end, once the page fault, the TLB and the scheduler are in the path.