CXL in GPU Systems

22.2 §7 established the number that governs this chapter: CXL adds capacity behind a link whose bandwidth does not grow with the capacity behind it. On a CPU node, where local DRAM runs at a few hundred gigabits per second, that trade is often worth making.

On a GPU node it is a different trade entirely, because the fast tier is not a few hundred gigabits — it is thousands. HBM, host DRAM and CXL are three tiers roughly sixteen times apart end to end, and a model that collapses the bottom two loses the distinction that decides where every byte should live.

This chapter is about where CXL sits in that hierarchy, which is not the hot path — and what it is genuinely good for once that is accepted.

1. The Engineering Problem — CXL Is Not The Fast Tier

There are three tiers, not two. HBM at 3200 Gbps, host DRAM at 400, CXL at 200 — a sixteen-times spread, and a two-tier model calls the bottom two one thing. Section 5.

HBM is small and it is the only fast tier. Filling it with weights because they were allocated first sends 80% of accesses to a slower tier. Section 6.

The roofline decides everything. The same 100 TFLOP/s achieves 100 from HBM and 12 from CXL — a factor of eight, from the tier alone. Section 7.

CXL's job is staging, not serving. 4 GB per step at 200 Gbps takes 160 ms, which a 200 ms step hides completely — and that is the entire value proposition. Section 8.

GPU-to-GPU traffic must not take the CXL path. A peer link at 900 Gbps against CXL at 200 is a 4.5x penalty for choosing wrongly. Section 9.

And a GPU's access to CXL memory crosses the host first. 450 ns against the host's own 300. Section 12.

This chapter against 22.1, stated precisely. That one owns whether coherent attach beats copying for an accelerator. This one owns where CXL sits in a hierarchy that already has a very fast tier — and section 15 shows a node with more memory and no more throughput.

2. The One-Sentence Model

CXL helps a GPU node when it adds capacity, the three tiers are modelled apart, HBM is reserved for the hot data, staging is hidden inside the step, GPU-to-GPU traffic takes the peer link, and node bandwidth is quoted by access weight rather than capacity weight — and every defect below is a node that holds more and computes no faster.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real GPU node is a memory hierarchy, a peer interconnect, a host root complex, a driver and a scheduler, and none of that is reproduced. What is reproduced is the arithmetic or the decision each has to get right.

Each model is built twice — a correct build and a broken build selected by a parameter. The broken builds here are all the CPU-node intuition applied to a GPU node: two tiers instead of three, capacity-weighted bandwidth, CXL as a memory tier rather than a staging tier. Each is reasonable on a machine whose fast tier is 400 Gbps and wrong on one whose fast tier is 3200.

Figure 1 — Four distinct rates on one node, spanning sixteen times. The dashed edges are section 5's failure: a model that treats the bottom two as interchangeable, when one is twice the other and they are used for entirely different things.

5. RTL 1 — Three Tiers, Not Two

// RTL 1 - the memory hierarchy a GPU actually has. HBM, host DRAM and CXL are
// three tiers an order of magnitude apart, and CXL is not the fast one.
module gpu_hierarchy #(parameter int TWO_TIER_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        classify,
  input  logic [15:0] hbm_gbps, host_gbps, cxl_gbps,
  output logic [15:0] fastest_gbps, slowest_gbps, spread_x,
  output logic [1:0]  tiers,
  output logic        cxl_is_fastest,
  output logic [7:0]  n_classified, n_three_tier,
  output logic        tier_collapsed_err
);
  logic [15:0] mid;
  logic [31:0] s_q;
  assign fastest_gbps = hbm_gbps;
  // Collapsing host DRAM and CXL into one tier is what a two-tier model does.
  assign tiers = (TWO_TIER_ONLY != 0) ? 2'd2 : 2'd3;
  assign mid = host_gbps;
  assign slowest_gbps = (TWO_TIER_ONLY != 0) ? mid : cxl_gbps;
  assign s_q = (slowest_gbps == 16'd0) ? 32'd0
             : ({16'd0, fastest_gbps} / {16'd0, slowest_gbps});
  assign spread_x = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  assign cxl_is_fastest = (cxl_gbps >= hbm_gbps);
  // A hierarchy with three distinct tiers reported as two.
  assign tier_collapsed_err = classify && (tiers == 2'd2) && (cxl_gbps != host_gbps);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_classified <= 8'd0; n_three_tier <= 8'd0;
    end else if (classify) begin
      n_classified <= n_classified + 8'd1;
      if (tiers == 2'd3) n_three_tier <= n_three_tier + 8'd1;
    end
  end
endmodule

Five classifications. HBM at 3200 Gbps, host DRAM at 400.

All five are three-tier in the correct model; the two-tier model collapsed four.

Row two is what makes the broken build honest rather than a strawman. When CXL and host DRAM genuinely run at the same rate, a two-tier model collapses nothing — it is simply describing the machine. The error is not "two tiers is wrong"; it is "two tiers when there are three", which is why tier_collapsed_err is gated on the rates actually differing.

Row three is worth stating because it is the future everyone assumes is already here. A CXL tier faster than HBM would reorder the hierarchy entirely, and every placement decision in this chapter would invert. That is not the machine anyone is buying today, and section 9's value model exists because it is not.

The sixteen-times spread is the number to carry forward. 22.2 §6 dealt with a 3x spread between local DRAM and CXL, and the blends were manageable. At sixteen times, the same arithmetic produces very different answers, and section 7 is where that becomes concrete.

Why the broken build is not a strawman. Two tiers is the right model for a CPU node — local DRAM and CXL, one step apart — and it is the model everybody arrives with. It survives onto a GPU node because the name of the near tier changes without the shape of the model changing, and the reason it fails there is not that two is too few in principle but that the two it keeps are the two that are closest together, collapsing the distinction that decides placement while preserving one that mostly does not.

6. RTL 2 — HBM Is The Only Fast Tier

// RTL 2 - what belongs in HBM. The GPU's own memory is small and fast, and
// filling it with anything but the hottest data wastes the only fast tier.
module hbm_residency #(parameter int FILL_BY_ORDER = 0) (
  input  logic clk, rst_n,
  input  logic        place,
  input  logic [15:0] hbm_gb, weights_gb, activations_gb, kv_gb,
  input  logic [7:0]  act_access_pct,
  output logic [15:0] act_in_hbm, act_spilled, spilled_access_pct,
  output logic        act_all_resident,
  output logic [7:0]  n_placements, n_spilled,
  output logic        wasted_hbm_err
);
  logic [15:0] taken_first;
  // Filling by order puts the weights in first because they are allocated first.
  assign taken_first = (FILL_BY_ORDER != 0)
                       ? ((weights_gb > hbm_gb) ? hbm_gb : weights_gb) : 16'd0;
  assign act_in_hbm = ((hbm_gb - taken_first) > activations_gb) ? activations_gb
                                                                : (hbm_gb - taken_first);
  // act_in_hbm is a minimum against activations_gb, so this cannot underflow.
  assign act_spilled = activations_gb - act_in_hbm;
  assign spilled_access_pct = (activations_gb == 16'd0) ? 8'd0
                            : (({8'd0, act_access_pct} * act_spilled) / activations_gb);
  assign act_all_resident = (act_spilled == 16'd0);
  // Activations spilled while the weights occupy HBM.
  assign wasted_hbm_err = place && (act_spilled != 16'd0) && (taken_first != 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_placements <= 8'd0; n_spilled <= 8'd0;
    end else if (place) begin
      n_placements <= n_placements + 8'd1;
      if (!act_all_resident) n_spilled <= n_spilled + 8'd1;
    end
  end
endmodule

Four placements. 80 GB of HBM, 140 GB of weights, 60 GB of activations at an 80% access rate.

One spilled placement in the correct model, two in the fill-by-order one.

Row one is 22.2 §8's argument with an order of magnitude more at stake. There, a blind placer cost 90% of accesses a 3x latency. Here it costs 80% of accesses a 16x bandwidth difference, because the tier below HBM is not host DRAM at 400 — for a spilled activation it is wherever the allocator put it.

Row two is the honest limit. With 40 GB of HBM and 60 GB of activations, a third spills whatever the placer does — that is a capacity failure rather than a placement one, and wasted_hbm_err requires the weights to be occupying HBM for exactly that reason.

Why the broken build is not a strawman. Weights are allocated first because they are loaded first — before the first batch exists, before any activation is computed. Allocation order and access frequency are unrelated, and an allocator with no hint follows the only order it has.

7. RTL 3 — The Roofline Decides Everything

// RTL 3 - the roofline. A GPU is bound by whichever of compute and bandwidth
// runs out first, and which tier supplies the bytes decides that.
module gpu_roofline #(parameter int ASSUME_COMPUTE_BOUND = 0) (
  input  logic clk, rst_n,
  input  logic        run,
  input  logic [15:0] tflops, bytes_per_flop, tier_gbps,
  output logic [15:0] demand_gbps, achieved_tflops, utilisation_pct,
  output logic        compute_bound,
  output logic [7:0]  n_runs, n_bw_bound,
  output logic        overclaim_err
);
  logic [31:0] d_q, a_q, u_q;
  assign d_q = {16'd0, tflops} * {16'd0, bytes_per_flop} * 32'd8;
  assign demand_gbps = (d_q > 32'd65535) ? 16'hFFFF : d_q[15:0];
  assign compute_bound = (demand_gbps <= tier_gbps);
  // No separate zero-demand guard is needed: a demand of zero is trivially at or
  // below any tier rate, so compute_bound already covers it and the division is
  // never reached with a zero divisor.
  assign a_q = (ASSUME_COMPUTE_BOUND != 0) ? {16'd0, tflops}
             : (compute_bound ? {16'd0, tflops}
               : (({16'd0, tflops} * {16'd0, tier_gbps}) / {16'd0, demand_gbps}));
  assign achieved_tflops = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign u_q = (tflops == 16'd0) ? 32'd0
             : (({16'd0, achieved_tflops} * 32'd100) / {16'd0, tflops});
  assign utilisation_pct = (u_q > 32'd65535) ? 16'hFFFF : u_q[15:0];
  // Claiming peak compute on a tier that cannot feed it.
  assign overclaim_err = run && !compute_bound && (achieved_tflops == tflops);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_runs <= 8'd0; n_bw_bound <= 8'd0;
    end else if (run) begin
      n_runs <= n_runs + 8'd1;
      if (!compute_bound) n_bw_bound <= n_bw_bound + 8'd1;
    end
  end
endmodule

Five runs. 100 TFLOP/s at two bytes per FLOP — a demand of 1600 Gbps.

Two bandwidth-bound runs; the compute-bound model claimed peak on both.

Twelve percent is the number this chapter exists to prevent. The same GPU, the same kernel, the same arithmetic — served from CXL it achieves an eighth of what it achieves from HBM. That is not a tier to run a hot loop from under any circumstances, and it is why section 8 reframes CXL's job entirely.

Row four is the ridge and it is where the sizing conversation lives. A demand exactly equal to the tier rate is compute-bound with no margin, and a workload with slightly worse reuse falls off it. Row five is the other extreme: perfect reuse means no bandwidth demand at all, and every tier serves it equally — which is the property a well-blocked matrix multiply has and a sparse operation does not.

The sensitivity here is worth stating precisely. Bytes per FLOP is a kernel property, and it varies by more than an order of magnitude across the kernels in one model — a blocked matmul and an element-wise normalisation differ enormously. A node sized against the average is over-provisioned for one and starved on the other, which is why section 21 asks for bytes-per-FLOP measured per kernel rather than per workload.

8. RTL 4 — CXL's Job Is Staging

// RTL 4 - what CXL is for on a GPU node. Not the hot path, but the capacity the
// hot path is staged from, and the staging rate is what matters.
module staging_tier #(parameter int TREAT_AS_HOT = 0) (
  input  logic clk, rst_n,
  input  logic        step,
  input  logic [15:0] stage_gb, cxl_gbps, step_ms,
  output logic [15:0] stage_ms, overlap_ms, exposed_ms,
  output logic        hidden,
  output logic [7:0]  n_steps, n_exposed,
  output logic        stall_err
);
  logic [31:0] s_q;
  // Bytes staged per step at the tier's rate, in milliseconds.
  assign s_q = (cxl_gbps == 16'd0) ? 32'd65535
             : (({16'd0, stage_gb} * 32'd8000) / {16'd0, cxl_gbps});
  assign stage_ms = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  // Staging overlaps with compute up to the length of the step.
  assign overlap_ms = (TREAT_AS_HOT != 0) ? 16'd0
                    : ((stage_ms > step_ms) ? step_ms : stage_ms);
  assign exposed_ms = stage_ms - overlap_ms;
  assign hidden = (exposed_ms == 16'd0);
  // Staging that does not fit inside the step it overlaps.
  assign stall_err = step && !hidden;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_steps <= 8'd0; n_exposed <= 8'd0;
    end else if (step) begin
      n_steps <= n_steps + 8'd1;
      if (!hidden) n_exposed <= n_exposed + 8'd1;
    end
  end
endmodule

Five steps. A 200 ms step, 200 Gbps of CXL.

Two exposed steps; the treat-as-hot model stalled on all five.

Row one is the entire case for CXL on a GPU node. 4 GB staged per step at a tier eight times slower than HBM, completely invisible, because it happens while the GPU computes. The tier's latency and bandwidth are irrelevant as long as the transfer finishes before the step does.

That reframing is what section 7 forces. CXL cannot serve the hot path — 12% utilisation — so the only question worth asking is whether it can feed the hot path fast enough to stay ahead of it. Row one says yes at 4 GB per step; row two says no at 8.

The treat-as-hot model is the failure of doing nothing. Not overlapping the staging is not a modelling error; it is a system that issues the transfer synchronously, and it converts an invisible cost into a 160 ms stall on every step. The fix is prefetch depth, and it is software.

9. RTL 5 — GPU-To-GPU Must Not Take The CXL Path

// RTL 5 - CXL against the GPU's own interconnect. A GPU-to-GPU link is faster
// than a CXL link, so peer traffic should not take the slow path.
module path_selection #(parameter int ALWAYS_CXL = 0) (
  input  logic clk, rst_n,
  input  logic        transfer,
  input  logic        peer_link_up, dest_is_gpu,
  input  logic [15:0] peer_gbps, cxl_gbps,
  output logic [15:0] chosen_gbps,
  output logic        uses_peer_link, best_available,
  output logic [7:0]  n_transfers, n_suboptimal,
  output logic        slow_path_err
);
  // A GPU-to-GPU transfer should take the peer link when it is up.
  assign uses_peer_link = (ALWAYS_CXL != 0) ? 1'b0 : (dest_is_gpu && peer_link_up);
  assign chosen_gbps = uses_peer_link ? peer_gbps : cxl_gbps;
  assign best_available = !(dest_is_gpu && peer_link_up) || uses_peer_link;
  // A GPU-to-GPU transfer sent over CXL while the peer link was available.
  assign slow_path_err = transfer && dest_is_gpu && peer_link_up && !uses_peer_link;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_transfers <= 8'd0; n_suboptimal <= 8'd0;
    end else if (transfer) begin
      n_transfers <= n_transfers + 8'd1;
      if (!best_available) n_suboptimal <= n_suboptimal + 8'd1;
    end
  end
endmodule

Four transfers. A 900 Gbps peer link against 200 Gbps of CXL.

Two slow paths taken by the always-CXL build; none by the correct one.

Both conditions are required and rows two and three show why. A peer link that is down is not an option, and a destination that is not a GPU cannot be reached over it — in both cases CXL is the best available path, not a mistake. slow_path_err fires only when a faster path existed and was not taken.

This is 21.2's peer-transfer argument arriving with a twist. There, the question was whether to remove the host from a transfer. Here there are two paths that both avoid the host, and the CXL one is 4.5 times slower — so the interesting failure is not going through the host, it is choosing the wrong direct path.

10. Waveform — Staging Hidden Behind Compute

The two scenarios move identical bytes over an identical tier. The only difference is when the transfer was issued, and it is the difference between a tier that costs nothing and one that costs 160 ms per step. CXL's usefulness on a GPU node is almost entirely a scheduling property.

11. RTL 6 — The KV Cache Decides Concurrency

// RTL 6 - the KV cache. Inference memory grows with sequence length and batch,
// and it is the term that decides how many requests a GPU can serve at once.
module kv_cache #(parameter int IGNORE_SEQUENCE = 0) (
  input  logic clk, rst_n,
  input  logic        size_it,
  input  logic [15:0] kb_per_token, seq_len, concurrency, spare_mb,
  output logic [15:0] kv_mb, max_concurrency, true_max_concurrency,
  output logic        fits,
  output logic [7:0]  n_sizings, n_over,
  output logic        oversubscribe_err
);
  logic [31:0] k_q, m_q, tp_q, tm_q;
  logic [15:0] true_per_request_kb;
  logic [15:0] per_request_kb;
  // Each request holds a cache proportional to its sequence length.
  assign per_request_kb = (IGNORE_SEQUENCE != 0) ? kb_per_token
                        : ((({16'd0, kb_per_token} * {16'd0, seq_len}) > 32'd65535)
                           ? 16'hFFFF : ({16'd0, kb_per_token} * {16'd0, seq_len}));
  assign k_q = ({16'd0, per_request_kb} * {16'd0, concurrency}) / 32'd1024;
  assign kv_mb = (k_q > 32'd65535) ? 16'hFFFF : k_q[15:0];
  assign fits = (kv_mb <= spare_mb);
  assign m_q = (per_request_kb == 16'd0) ? 32'd65535
             : (({16'd0, spare_mb} * 32'd1024) / {16'd0, per_request_kb});
  assign max_concurrency = (m_q > 32'd65535) ? 16'hFFFF : m_q[15:0];
  // The concurrency the cache really admits, against what the model computed.
  assign tp_q = {16'd0, kb_per_token} * {16'd0, seq_len};
  assign true_per_request_kb = (tp_q > 32'd65535) ? 16'hFFFF : tp_q[15:0];
  assign tm_q = (true_per_request_kb == 16'd0) ? 32'd65535
              : (({16'd0, spare_mb} * 32'd1024) / {16'd0, true_per_request_kb});
  assign true_max_concurrency = (tm_q > 32'd65535) ? 16'hFFFF : tm_q[15:0];
  // A concurrency the cache cannot hold, sized as if it could.
  assign oversubscribe_err = size_it && (concurrency > true_max_concurrency) && fits;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_sizings <= 8'd0; n_over <= 8'd0;
    end else if (size_it) begin
      n_sizings <= n_sizings + 8'd1;
      if (concurrency > max_concurrency) n_over <= n_over + 8'd1;
    end
  end
endmodule

Five sizings. 8 KB per token, a 2048-token sequence, 1024 MB spare.

One sizing exceeded the largest concurrency; the ignore-sequence model called it fitting.

Row four is the lever that matters for inference serving. A quarter of the sequence length gives four times the concurrency, because the cache is linear in both. That is a product decision — a shorter context window serves more users on the same GPU — and it is arithmetic rather than tuning.

The ignore-sequence model cannot detect its own error, which is why oversubscribe_err compares against a separately computed true maximum. A model that gets the per-request size wrong gets its own maximum wrong by the same factor, so its internal consistency check always passes. Section 18 records that this was a design change rather than a stimulus fix.

Figure 3 — Same GPU, same kernel, three tiers, and an eight-times spread in what is achieved. Nothing about the compute changed; only where the bytes came from. This is why section 8 stops asking CXL to serve and asks it to stage instead.

12. RTL 7 — A GPU Reaches CXL Through The Host

// RTL 7 - where CXL memory attaches. Behind the host, a GPU's access to CXL
// memory crosses the host root complex as well as the CXL link.
module attach_topology #(parameter int IGNORE_HOST_HOP = 0) (
  input  logic clk, rst_n,
  input  logic        access,
  input  logic        gpu_originated,
  input  logic [15:0] cxl_ns, host_hop_ns,
  output logic [15:0] gpu_ns, host_ns,
  output logic        symmetric,
  output logic [7:0]  n_accesses, n_gpu,
  output logic        hop_ignored_err
);
  // A host access to CXL memory crosses one link; a GPU access crosses the host
  // root complex first.
  assign host_ns = cxl_ns;
  assign gpu_ns = (IGNORE_HOST_HOP != 0) ? cxl_ns : (cxl_ns + host_hop_ns);
  assign symmetric = (gpu_ns == host_ns);
  // A GPU access to CXL memory costed as if it started at the host.
  assign hop_ignored_err = access && gpu_originated && symmetric
                           && (host_hop_ns != 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_accesses <= 8'd0; n_gpu <= 8'd0;
    end else if (access) begin
      n_accesses <= n_accesses + 8'd1;
      if (gpu_originated) n_gpu <= n_gpu + 8'd1;
    end
  end
endmodule

Three GPU-originated accesses; the ignore-hop model missed the hop on two.

A GPU's CXL access is not the host's CXL access, and the asymmetry is topological: the memory hangs off the host, so a GPU reaching it traverses the root complex first. A latency figure quoted from a host-side measurement understates the GPU-side cost by the hop, which at 150 ns is 50% of the number being quoted.

Row three is the configuration that removes it, and it is worth naming: a CXL device attached such that the GPU reaches it without the host hop. That is a topology choice made at board design, and it is the difference between 450 ns and 300 on every access.

13. RTL 8 — Quote Bandwidth By Access, Not Capacity

// RTL 8 - what CXL adds to a GPU node's capacity, and what it does to the
// node's bandwidth-weighted average.
module node_blend #(parameter int CAPACITY_WEIGHTED = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [15:0] hbm_gb, hbm_gbps, cxl_gb, cxl_gbps,
  input  logic [7:0]  hbm_access_pct,
  output logic [15:0] total_gb, capacity_avg_gbps, access_avg_gbps,
  output logic        honest,
  output logic [7:0]  n_assess, n_misleading,
  output logic        mislead_err
);
  logic [31:0] c_q, a_q;
  assign total_gb = hbm_gb + cxl_gb;
  // A capacity-weighted average is dominated by whichever tier is larger, which
  // is not where the accesses go.
  assign c_q = (total_gb == 16'd0) ? 32'd0
             : ((({16'd0, hbm_gbps} * {16'd0, hbm_gb})
               + ({16'd0, cxl_gbps} * {16'd0, cxl_gb})) / {16'd0, total_gb});
  assign capacity_avg_gbps = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  // The access-weighted average is what the workload actually experiences.
  assign a_q = (({16'd0, hbm_gbps} * {24'd0, hbm_access_pct})
              + ({16'd0, cxl_gbps} * (32'd100 - {24'd0, hbm_access_pct}))) / 32'd100;
  assign access_avg_gbps = (CAPACITY_WEIGHTED != 0) ? capacity_avg_gbps
                         : ((a_q > 32'd65535) ? 16'hFFFF : a_q[15:0]);
  assign honest = (access_avg_gbps != capacity_avg_gbps) || (hbm_gb == cxl_gb);
  // A node bandwidth quoted from capacity weights rather than access weights.
  assign mislead_err = assess && !honest;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assess <= 8'd0; n_misleading <= 8'd0;
    end else if (assess) begin
      n_assess <= n_assess + 8'd1;
      if (!honest) n_misleading <= n_misleading + 8'd1;
    end
  end
endmodule

Four assessments. 80 GB of HBM at 3200 Gbps, CXL at 200.

The capacity-weighted model misled on two of four.

417 against 3050 is a factor of seven on the same node, and both numbers are arithmetically correct. The capacity-weighted figure describes a node where accesses are spread evenly across bytes; the access-weighted figure describes the node anybody actually runs.

Row three is the case that makes the capacity model right, and it is worth being precise about: when the accesses really are spread in proportion to capacity, the two averages coincide and neither is misleading. The error is the assumption, not the arithmetic — and on a GPU node with 95% of accesses in HBM the assumption is off by a factor of seven.

14. RTL 9 — What CXL Is Worth Here

// RTL 9 - what CXL is worth on a GPU node. It buys capacity for the staged tier
// and buys nothing for the hot path, and the split decides the value.
module gpu_cxl_value #(parameter int ASSUME_HOT_PATH = 0) (
  input  logic clk, rst_n,
  input  logic        eval,
  input  logic [15:0] staged_benefit_pct, hot_path_benefit_pct,
  input  logic [7:0]  staged_share_pct,
  output logic [15:0] blended_benefit_pct,
  output logic        worth_adding,
  output logic [7:0]  n_evals, n_marginal,
  output logic        overclaim_err
);
  logic [31:0] b_q;
  logic [15:0] hot_benefit;
  // CXL does nothing for the hot path; a model claiming otherwise inflates it.
  assign hot_benefit = (ASSUME_HOT_PATH != 0) ? staged_benefit_pct
                                              : hot_path_benefit_pct;
  assign b_q = (({16'd0, staged_benefit_pct} * {24'd0, staged_share_pct})
              + ({16'd0, hot_benefit} * (32'd100 - {24'd0, staged_share_pct}))) / 32'd100;
  assign blended_benefit_pct = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  assign worth_adding = (blended_benefit_pct >= 16'd10);
  // Claiming a hot-path benefit CXL cannot deliver.
  assign overclaim_err = eval && (hot_benefit > hot_path_benefit_pct);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_marginal <= 8'd0;
    end else if (eval) begin
      n_evals <= n_evals + 8'd1;
      if (!worth_adding) n_marginal <= n_marginal + 8'd1;
    end
  end
endmodule

Five evaluations. A 40% benefit on staged work.

One evaluation was not worth adding; the hot-path model overclaimed on every one.

The hot-path benefit is zero and that is section 7's finding, not an assumption. CXL achieves 12% utilisation on a bandwidth-bound kernel — it does not accelerate the hot path, it cannot, and a value model that assigns it a hot-path benefit is claiming something the roofline forbids.

Row five is the exception worth stating. A workload whose "hot path" is itself capacity-limited — one that was thrashing before the capacity arrived — genuinely does benefit, and the model accepts a non-zero hot_path_benefit_pct when there is one. The error is asserting a benefit larger than the one supplied, which is what overclaim_err tests.

Figure 4 — The rightmost terminal is the one this chapter argues for and nobody wants to hear: hot data that does not fit HBM does not go somewhere slower, it means the workload is too large for the node. Section 7's twelve percent is why spilling the hot path is not a option, and section 22 is a deployment that spilled it anyway.

15. RTL 10 — CXL On A GPU Node Assembled

// RTL 10 - CXL on a GPU node assembled. Everything that must hold before adding
// CXL memory helps a GPU workload rather than merely enlarging the node.
module gpu_cxl_model #(parameter int CAPACITY_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       capacity_added,     // the node holds more
  input  logic       tiers_distinct,     // HBM, host and CXL are modelled apart
  input  logic       hbm_reserved,       // the hot data owns HBM
  input  logic       staging_hidden,     // staging overlaps the step
  input  logic       peer_path_used,     // GPU-to-GPU does not take CXL
  input  logic       bandwidth_honest,   // quoted by access weight, not capacity
  output logic       helps,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_helps,
  output logic       false_help_err
);
  assign fail_mask[0] = ~capacity_added;
  assign fail_mask[1] = ~tiers_distinct;
  assign fail_mask[2] = ~hbm_reserved;
  assign fail_mask[3] = ~staging_hidden;
  assign fail_mask[4] = ~peer_path_used;
  assign fail_mask[5] = ~bandwidth_honest;
  // The capacity-only build checks that the node holds more and calls it a win,
  // which is what a configuration sheet does.
  assign helps = (CAPACITY_ONLY != 0) ? capacity_added : (fail_mask == 6'd0);
  assign false_help_err = evaluate && helps && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_helps <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (helps) n_helps <= n_helps + 8'd1;
    end
  end
endmodule

One helpful configuration of six, and four false claims.

The capacity-only definition is a configuration sheet, and it is right about exactly one of the six. Row four is the one that produces the strangest symptom: everything about the memory hierarchy is correct, and GPU-to-GPU collectives are running at 200 Gbps instead of 900 because the addition of a CXL path changed which route the runtime selected.

16. Quantitative Reasoning

Hierarchy. HBM 3200, host DRAM 400, CXL 200 — a sixteen-times spread, and a two-tier model collapses two distinct tiers on four of five configurations.

HBM residency. 60 GB of activations at an 80% access rate: correct placement spills none; fill-by-order spills all 60 and sends 80% of accesses elsewhere.

Roofline. 100 TFLOP/s at two bytes per FLOP: 100 TFLOP/s from HBM, 25 from host DRAM, 12 from CXL — a factor of eight from the tier alone.

Staging. 4 GB per step at 200 Gbps is 160 ms, entirely hidden inside a 200 ms step. 8 GB is 320 ms and leaves 120 ms exposed.

Path selection. A 900 Gbps peer link against 200 Gbps of CXL — 4.5x for choosing wrongly, on two of four transfers.

KV cache. 8 KB per token over 2048 tokens in 1024 MB: 64 concurrent requests, and a quarter of the sequence gives 256.

Attach topology. A host access to CXL memory costs 300 ns; a GPU access costs 450, and a 400 ns hop makes it 700.

Node blend. 80 GB of HBM and 1024 GB of CXL: a capacity-weighted 417 Gbps against an access-weighted 3050 — a factor of seven, both correct.

Value. A 40% staged benefit at a 40% staged share blends to 16%; at a 10% share it is 4% and not worth adding.

The assembled model. Six properties, six configurations, one helps. The capacity-only definition reported five.

17. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Hierarchy. The case where CXL and host DRAM coincide is asserted as not a collapse.

chk(hSg == 16'd400, "CXL matching host DRAM");
chk(wTe == 1'b0,    "and the two-tier model collapses nothing real");

The exactly-at-HBM case is driven.

Residency. The capacity-limited spill is asserted as not wasted HBM.

Roofline. Exactly at the ridge is driven, and a zero-bandwidth-demand workload is asserted to achieve peak from any tier.

Staging. The exact step boundary is driven, and the no-tier case is asserted to report an unbounded time.

chk(sSm == 16'd200, "staging 5 GB takes exactly 200 ms");
chk(sEm == 16'd0,   "which the step exactly hides");

Path selection. Both reasons the peer link is unavailable are asserted as best available rather than as errors.

KV cache. The exact largest concurrency is driven, and the zero-sequence case is asserted unbounded.

Attach topology. The zero-hop case is asserted genuinely symmetric.

Node blend. The even-access case is asserted to make both averages coincide.

chk(nAa == 16'd1700, "at an even access split the access average is 1700");
chk(nCa == 16'd1700, "matching the capacity average");

Value. The threshold is driven at exactly 25% staged share, and a real hot-path benefit is asserted legitimate.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 224 checks across two testbenches, 118 on the front five models and 106 on the back five, all passing on the unmutated sources.

18. Mutation Testing

Forty-five mutations were injected one at a time.

45 injected, 45 killed, after three survivors and one design change.

A broken build that could not detect its own error — fixed at baseline. The KV-cache oversubscription check originally compared concurrency against the model's own max_concurrency. The ignore-sequence build computes both the per-request size and the maximum from the same wrong input, so its internal consistency check always passes — the error was structurally undetectable. The fix computes a true_max_concurrency from the real per-request size and compares against that, which is the same pattern 21.3 §11 uses for a degraded link count. A broken build that is wrong consistently cannot be caught by a check written in its own terms.

Survivor 1 — a threshold never driven. cxl_is_fastest surviving >= becoming > needed a CXL tier at exactly HBM's rate. Not a machine anyone ships, and exactly the boundary the comparison exists to define.

Survivor 2 — a provably redundant guard. The roofline's zero-demand guard survives removal because a demand of zero is trivially at or below any tier rate, so compute_bound already covers it and the division is never reached with a zero divisor. The guard was deleted with a comment saying so, and the mutation replaced with one that inverts the division instead.

Survivor 3 — a divide-by-zero guard with no stimulus. The staging model's zero-rate guard needed a tier that is not there at all. Driving it — a node where the CXL rate is zero — kills the mutation and produces the honest output: an unbounded staging time rather than a division.

Both a redundant guard and a necessary one, in the same batch chapter. Section 7's zero-demand guard is genuinely unreachable and was removed; section 8's zero-rate guard is reachable and was kept. The distinction is whether another condition already dominates the guarded case, and it has to be checked rather than assumed from the shape.

19. Verification Strategy

What a testbench for a GPU-node memory model must cover.

Check whether a guard is dominated before writing it. This chapter had two zero-guards: one redundant because compute_bound already covered it, one necessary because nothing else did. The shape is identical and the answer is opposite.

Broken builds that are wrong consistently. A model with a wrong per-request size and a wrong maximum derived from it passes every internal check. Compare against a separately computed truth, not against the model's own numbers.

The boundary where two tiers coincide. Several checks here distinguish "collapsed a real tier" from "described a machine with two tiers", and both need driving.

The cases that are correct and look like failures. A capacity-limited HBM spill. CXL chosen because the peer link is down. A capacity-weighted average on a node where accesses really are spread by capacity. A demand of zero achieving peak from every tier.

Rate boundaries at exactly equal. CXL at exactly HBM's rate, a demand at exactly the ridge, staging at exactly the step length, a cache at exactly the spare capacity.

What a real node needs that these models do not have. Concurrency between staging and compute for the same memory controller. Multi-GPU contention for one host root complex. Kernel-level blocking, which changes bytes-per-FLOP by an order of magnitude and is the input section 7 is most sensitive to.

20. Synthesis and Implementation Reality

HBM's rate is why the hierarchy is steep. A GPU's memory is on-package and very wide; a CXL link is a serial interface with a handful of lanes. The sixteen-times spread is not a design choice anyone made — it is two very different physical constructions, and no CXL generation closes it.

Staging is a DMA engine and a prefetch depth. Section 8's overlap is achieved by issuing the next step's transfer during the current step, which needs double-buffered destination memory in HBM — capacity that section 6 is already short of. The two models compete for the same resource.

The peer link and the CXL link are different fabrics with different drivers. Section 9's selection is a runtime decision made per transfer, and the failure mode is a runtime that treats a newly-present CXL path as a general-purpose route.

The host hop in section 12 is a board-level topology. Whether a GPU reaches CXL memory through the host root complex or more directly is decided by how the device is attached, not by anything either endpoint does.

The KV cache lives in HBM and competes with everything else there. Section 11's spare capacity is what section 6 did not use, so a serving deployment is choosing between weights resident, activations resident, and concurrency — three demands on one 80 GB tier.

21. Silicon Observability

"Staging overlap with compute" is the counter that has to be designed in. Section 8's entire value proposition is that the transfer is invisible, and an invisible cost has no natural observable — the only evidence that staging is working is a measurement of the thing not happening. A node reporting staging bytes but not staging overlap cannot distinguish section 8's row one from its row two.

22. Debug Lab

Symptom. A GPU inference node is given 1 TB of CXL memory to raise serving concurrency. Concurrency does rise, from 64 to 96 requests. Per-request latency rises 3.4x and total throughput falls. Nothing errors, and the CXL tier reports healthy.

Step 1 — is it the roofline? Achieved FLOP/s against peak: 89% during decode. The GPU is not bandwidth-starved, so section 7 is not the chapter.

Step 2 — is it staging? Staging transfers issued: zero. Nothing is being staged; the CXL memory is being used directly. Section 8 is not in play, and that is itself the finding — CXL is not acting as a staging tier here.

Step 3 — where is the KV cache? Resident by tier: 62% of the KV cache is in CXL memory. The allocator placed the additional concurrency's cache in the new tier, because that is where the free capacity was.

Step 4 — what does that cost per access? GPU-originated CXL access latency: 450 ns, against HBM's tens of nanoseconds. The KV cache is read on every token of every request, so it is the hottest structure in the workload.

Step 5 — check the hop. Host-side CXL latency reads 300 ns; the GPU-side figure is 450. Section 12's hop is 150 ns of the 450, and the capacity plan used the host-side number.

The finding. CXL capacity raised concurrency and the KV cache — the single hottest structure — was placed in the slowest tier, read once per token. Section 6's failure applied to inference, with section 12's hop making the quoted latency 50% optimistic on top.

The fix. Keep the KV cache in HBM and cap concurrency at what HBM holds — 64, which is where it was. Use the CXL capacity for weights of models not currently being served, staged into HBM on a model switch, which is section 8's pattern and the one this node should have been built around.

What made this hard. Concurrency went up, which was the goal, and every counter that was being watched improved. The regression was in a structure nobody thought of as memory-placed, at a latency nobody had measured from the GPU's side.

23. Design Review

1. How many tiers does the model have, and what are their rates? If the answer is two, section 5.

2. What is in HBM, and by what rule? If the answer is allocation order, section 6.

3. What is bytes-per-FLOP for the dominant kernel? It decides whether any tier below HBM is usable for the hot path. Section 7.

4. Is CXL a staging tier or a serving tier? Only the first works. Section 8.

5. Is the staging overlapped with compute, and is the overlap measured? An invisible cost needs a counter. Sections 8 and 21.

6. Does GPU-to-GPU traffic take the peer link? A 4.5x penalty for choosing wrongly. Section 9.

7. Is the CXL latency figure measured from the GPU or from the host? They differ by the host hop. Section 12.

8. Is node bandwidth quoted by access weight or by capacity weight? A factor of seven on the same node. Section 13.

9. Where does the KV cache live, and is it allowed to spill? Section 22 is what happens when it is.

10. Which of the six properties does the team believe "we added CXL" implies? Section 15 exists because the answer is the capacity.

24. How This Appears In Real Engineering

A system architect specifying a GPU node asks section 3's question — bytes per FLOP — before anything else, because it determines whether a tier below HBM can serve the hot path at all. On a well-blocked dense workload the answer is generous and CXL has room; on a sparse or memory-bound one, nothing below HBM is usable and CXL's only role is staging.

A runtime or framework team owns sections 6, 8 and 9 together, and they are all placement decisions with no hardware involvement. HBM residency is an allocator hint, staging overlap is a prefetch depth, and path selection is a route table — three software decisions that determine whether the hardware helps.

A serving team meets section 22. Raising concurrency by adding capacity works only if the per-request state stays in the fast tier, and the KV cache is both the largest per-request structure and the hottest. Concurrency and residency are the same budget, and section 11 is the arithmetic that connects them.

A benchmarking team runs into section 13. A node's "memory bandwidth" is not a single number once the node has tiers, and the capacity-weighted figure is the one a configuration sheet naturally produces. Quoting it is not dishonest, it is under-specified — and the access-weighted figure requires knowing the workload.

25. Common Misconceptions

"CXL memory is another memory tier." It is the third tier, sixteen times slower than the first. Section 5.

"The allocator will put the hot data in HBM." It puts what was allocated first, which is the weights. Section 6.

"The GPU is fast, so the tier does not matter much." 100 TFLOP/s from HBM, 12 from CXL — the tier is the whole difference. Section 7.

"CXL is too slow to be useful on a GPU node." It is too slow to serve and fast enough to stage — 4 GB per step entirely hidden. Section 8.

"Any direct path avoids the host, so they are equivalent." A peer link at 900 against CXL at 200. Section 9.

"CXL latency is 300 ns." From the host. From the GPU it is 450, because of the root-complex hop. Section 12.

"The node has 1.1 TB at 417 Gbps average." By capacity weight. By access weight it is 3050. Section 13.

"More capacity means more concurrency." Only if the per-request state stays in the fast tier. Sections 11 and 22.

"CXL will accelerate the workload." It has no hot-path benefit — that is what section 7's 12% means.

"We added CXL, so the node is better." One property of six. Section 15.

26. Interview Reasoning

Q. Where does CXL sit in a GPU node's memory hierarchy?

Third, behind HBM and host DRAM — roughly sixteen times slower than HBM end to end. The follow-up that matters: that spread is two different physical constructions, on-package wide memory against a serial link, so no CXL generation closes it and the placement conclusions are stable across generations.

Q. A kernel needs two bytes per FLOP at 100 TFLOP/s. What does each tier deliver?

A 1600 Gbps demand: HBM at 3200 serves it fully; host DRAM at 400 gives 25 TFLOP/s; CXL at 200 gives 12. That is the argument that CXL cannot serve a hot path — an eighth of peak — and the number to compute before proposing it as one.

Q. So what is CXL good for on a GPU node?

Staging. 4 GB moved per step at 200 Gbps takes 160 ms, and a 200 ms step hides it completely. The tier's speed becomes irrelevant as long as the transfer finishes before the step does — which turns a memory question into a scheduling question.

Q. Your GPU-to-GPU collectives got slower after adding CXL. Why?

Because the runtime started routing peer traffic over the new path. A peer link at 900 Gbps against CXL at 200 is a 4.5x penalty, and the failure is not "going through the host" — both paths avoid it — it is choosing the wrong direct path.

Q. Why is the CXL latency your GPU sees higher than the datasheet figure?

The datasheet figure is host-side. The memory hangs off the host, so a GPU access crosses the root complex first — 450 ns against 300 in the model. A capacity plan built on the host-side number is 50% optimistic on every GPU access.

Q. You added CXL to raise inference concurrency and latency got worse. Where do you look?

Where the KV cache went. It is the largest per-request structure and it is read once per token, so it is the hottest thing in the workload — and the allocator will put the new concurrency's cache wherever the free capacity is, which is the new tier. Concurrency and fast-tier residency are one budget, and raising the first without the second moves the hottest data to the slowest place.

27. Exercises

1. Extend RTL 1 to four tiers by splitting host DRAM into local and remote NUMA, and show which pairs a two-tier model can safely collapse.

2. Give RTL 2 a double-buffering requirement from section 8 and show that staging and residency compete for the same HBM.

3. Plot RTL 3's achieved rate against bytes-per-FLOP for all three tiers and mark each tier's ridge.

4. Add prefetch depth to RTL 4 and find the depth at which an 8 GB stage becomes hidden.

5. Extend RTL 5 to three paths — peer link, CXL, and host-mediated — and rank them for each destination type.

6. Combine RTL 6 with RTL 2: given an HBM budget, find the concurrency and residency split that maximises served tokens.

7. Make RTL 7's host hop a function of root-complex load and show how it degrades under multi-GPU contention.

8. Extend RTL 8 to report both averages always, and show that a single "node bandwidth" figure is under-specified by construction.

9. Model section 22 end to end: rising concurrency, KV cache spilling to CXL, and the resulting per-token latency.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the node it catches that the current mask calls helpful.

28. Summary

22.2 established that CXL adds capacity on a link that does not widen. On a GPU node the fast tier is sixteen times faster than that link, and every conclusion follows from the ratio.

Three tiers, not two. HBM 3200, host DRAM 400, CXL 200 — and a two-tier model collapsed a real distinction on four of five configurations.

HBM is the only fast tier and it is small. Filling it by allocation order put the weights in and sent 80% of accesses elsewhere.

The roofline decides everything. 100 TFLOP/s at two bytes per FLOP achieves 100 from HBM, 25 from host DRAM and 12 from CXL — a factor of eight from the tier alone, which is why CXL cannot serve a hot path.

Its job is staging. 4 GB per step at 200 Gbps is entirely hidden inside a 200 ms step, and the same bytes issued synchronously are a 160 ms stall — the same tier, the same transfer, and the difference is when it was issued.

GPU-to-GPU must not take the CXL path. 900 Gbps against 200 — 4.5x — and both paths avoid the host, so the failure is choosing wrongly between two direct routes.

A GPU reaches CXL memory through the host. 450 ns against the host's own 300, which makes a host-side latency figure 50% optimistic for every GPU access.

Node bandwidth is under-specified as one number. 417 Gbps by capacity weight against 3050 by access weight — a factor of seven, both arithmetically correct.

Concurrency and fast-tier residency are one budget. 64 requests at 2048 tokens, 256 at 512 — and section 22 is a node that raised concurrency by putting the KV cache in the slowest tier.

A broken build that is wrong consistently cannot check itself. The KV model's oversubscription test had to compare against a separately computed truth, because a wrong per-request size produces a wrong maximum that agrees with it.

And two identically shaped zero-guards had opposite answers — one dominated by compute_bound and removed, one reachable and kept.

Adding capacity is one property of six. The definition a configuration sheet uses called five of six nodes helped when one was.

22.4 — CXL for Large Language Models takes the working set from 22.2 and the hierarchy from this chapter to the scale where no single node's HBM is close to enough.