CXL for Large Language Models

22.3 established that a GPU node has three tiers sixteen times apart, and that CXL is the slow one — twelve percent of peak on a bandwidth-bound kernel, and useful for staging rather than serving.

This chapter takes that hierarchy to the workload that broke it. A large language model does not have a working set that happens to be large; it has a parameter count that is read in its entirety for every token it generates, which is the single worst access pattern for a capacity tier that anyone has yet deployed at scale.

The result is a chapter with an uncomfortable finding in it: CXL makes a model that would not run, run — and it makes almost none of them run faster. Both halves matter, and section 15 is what happens when a plan only holds the first.

1. The Engineering Problem — Holding A Model Is Not Serving One

The footprint is not the parameter count. 70 billion parameters at two bytes is 140 GB of weights and 560 GB to train — four times over. Section 5.

Precision is the cheapest lever there is. The same model is 140 GB, 70 GB or 35 GB, and only the last two fit an 80 GB node. Section 6.

Sharding uses every node's fast tier; tiering uses one. Eight nodes hold 640 GB in HBM; tiering to CXL keeps 80 GB resident and spills 480. Section 7.

Layers can be streamed because a transformer is a sequence. A 2 GB layer at 200 Gbps takes 80 ms, which 100 ms of compute hides entirely. Section 8.

And decode reads every active weight for every token. 8 GB per token is 50 tokens a second from HBM and three from CXL. Section 12.

This chapter against 22.2, stated precisely. That one owns how large a working set really is. This one owns a workload whose working set is the whole model, read per token — which is why section 12 exists and section 16 concludes what it does.

2. The One-Sentence Model

CXL serves a large language model when the footprint is counted completely, a precision is chosen, fast tiers are used before slow ones, layer fetches hide behind compute, the token rate the tier supports is the one being promised, and the slow tier holds only weights that are genuinely cold — and every defect below is a machine that loads a model it cannot serve.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real serving stack is a scheduler, a batching engine, a paged attention allocator, a set of fused kernels and a collective library, and none of that is reproduced. What is reproduced is the arithmetic each of them has to get right, and the shape of the mistake when it does not.

Each model is built twice — a correct build and a broken build selected by a parameter. The broken builds here are all the same mistake in different clothes: treating capacity as the question. A model fits, therefore it runs. A node holds it, therefore it serves it. That is the assumption sections 12 and 14 are built to destroy.

Figure 1 — The dashed edges are the terms a parameter count does not mention and section 5 makes explicit. A footprint reported as 140 GB and a footprint of 560 GB are the same model, and the difference is whether anyone is training it.

5. RTL 1 — The Footprint Is Not The Parameter Count

// RTL 1 - what a model costs to hold. Parameters are one term; training also
// holds gradients and optimizer state, and the total is several times the count.
module model_footprint #(parameter int WEIGHTS_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        size_it,
  input  logic        training,
  input  logic [15:0] params_b, bytes_per_param, opt_copies,
  output logic [15:0] weight_gb, extra_gb, total_gb, multiple_x,
  output logic        fits_one_node,
  input  logic [15:0] node_gb,
  output logic [7:0]  n_sizings, n_oversize,
  output logic        undercount_err
);
  logic [31:0] w_q, e_q, m_q;
  assign w_q = {16'd0, params_b} * {16'd0, bytes_per_param};
  assign weight_gb = (w_q > 32'd65535) ? 16'hFFFF : w_q[15:0];
  // Training holds a gradient and opt_copies of optimizer state per parameter.
  assign e_q = (WEIGHTS_ONLY != 0) ? 32'd0
             : (training ? ({16'd0, weight_gb} * (32'd1 + {16'd0, opt_copies}))
                         : 32'd0);
  assign extra_gb = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  assign total_gb = ((({16'd0, weight_gb} + {16'd0, extra_gb})) > 32'd65535)
                    ? 16'hFFFF : (weight_gb + extra_gb);
  assign m_q = (weight_gb == 16'd0) ? 32'd0
             : ({16'd0, total_gb} / {16'd0, weight_gb});
  assign multiple_x = (m_q > 32'd65535) ? 16'hFFFF : m_q[15:0];
  assign fits_one_node = (total_gb <= node_gb);
  // A training footprint reported as the weight footprint.
  assign undercount_err = size_it && training && (extra_gb == 16'd0)
                          && (weight_gb != 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_sizings <= 8'd0; n_oversize <= 8'd0;
    end else if (size_it) begin
      n_sizings <= n_sizings + 8'd1;
      if (!fits_one_node) n_oversize <= n_oversize + 8'd1;
    end
  end
endmodule

Six sizings. 70 billion parameters at two bytes each.

One sizing exceeded the machine; the weights-only model reported none.

Four times is the number that surprises people, and it is not exotic. A gradient is one copy of every parameter, and a first-and-second-moment optimizer is two more. The model that fits comfortably for serving does not fit at all for training, and it is the same file on disk.

Row two is why the weights-only model survives in practice. For inference it is exactly right, and inference is what most people size for. undercount_err is gated on training for precisely that reason — the error is not "you forgot the optimizer", it is "you forgot the optimizer while training", and a serving deployment is entitled to ignore it.

Row five is the boundary the fit test defines. A footprint exactly the size of the machine fits, with nothing left for activations, the KV cache, or the CUDA context — which is a machine that will fail on the first request. The arithmetic says yes and the deployment says no, and section 23 asks the question that separates them.

6. RTL 2 — Precision Is The Cheapest Lever

// RTL 2 - precision. Bytes per parameter is a choice, and it moves the whole
// footprint linearly, which is the cheapest lever in the chapter.
module precision_choice #(parameter int ASSUME_FP16 = 0) (
  input  logic clk, rst_n,
  input  logic        pick,
  input  logic [15:0] params_b, node_gb,
  input  logic [7:0]  bits_per_param,
  output logic [15:0] bits_used, footprint_gb, headroom_gb,
  output logic        fits,
  output logic [7:0]  n_picks, n_fitting,
  output logic        precision_ignored_err
);
  logic [31:0] f_q;
  // A model that assumes half precision ignores the quantisation entirely.
  assign bits_used = (ASSUME_FP16 != 0) ? 16'd16 : {8'd0, bits_per_param};
  assign f_q = ({16'd0, params_b} * {16'd0, bits_used}) / 32'd8;
  assign footprint_gb = (f_q > 32'd65535) ? 16'hFFFF : f_q[15:0];
  assign fits = (footprint_gb <= node_gb);
  assign headroom_gb = fits ? (node_gb - footprint_gb) : 16'd0;
  // A quantised model costed at half precision.
  assign precision_ignored_err = pick && (bits_per_param != 8'd0)
                                 && (bits_used != {8'd0, bits_per_param});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_picks <= 8'd0; n_fitting <= 8'd0;
    end else if (pick) begin
      n_picks <= n_picks + 8'd1;
      if (fits) n_fitting <= n_fitting + 8'd1;
    end
  end
endmodule

Five picks. 70 billion parameters against an 80 GB node.

Four fit in the full model; none in the fp16 one.

A factor of four on the largest single term in the plan, and it costs no hardware. Quantisation is the reason a model that needs 140 GB runs on a node with 80, and any capacity discussion that has not settled the precision first is discussing the wrong number.

Row four is a much larger model made to fit by precision alone. 160 billion parameters at four bits is exactly the 80 GB node — more than twice the parameters of row one in less than a third of the memory, which is the trade the whole quantisation literature exists to price in accuracy rather than bytes.

Why the broken build is not a strawman. Half precision was the default for years, and a great deal of tooling still reports footprints in it. The fp16 assumption is not wrong about arithmetic, it is stale about deployment — and it declares four perfectly deployable configurations impossible.

7. RTL 3 — Shard Before You Tier

// RTL 3 - shard or tier. A model too large for one node is split across nodes
// or spilled to a slower tier, and the two cost entirely different things.
module parallel_vs_tier #(parameter int TIER_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        plan,
  input  logic [15:0] total_gb, hbm_per_node_gb, nodes,
  output logic [15:0] sharded_capacity_gb, resident_gb, spilled_gb,
  output logic        sharded, all_resident,
  output logic [7:0]  n_plans, n_spilling,
  output logic        needless_spill_err
);
  logic [31:0] c_q;
  assign c_q = {16'd0, hbm_per_node_gb} * {16'd0, nodes};
  assign sharded_capacity_gb = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  // Sharding uses every node's fast tier; tiering uses only the local one.
  assign sharded = (TIER_ONLY != 0) ? 1'b0 : (nodes > 16'd1);
  assign resident_gb = sharded
        ? ((total_gb > sharded_capacity_gb) ? sharded_capacity_gb : total_gb)
        : ((total_gb > hbm_per_node_gb) ? hbm_per_node_gb : total_gb);
  // resident_gb is a minimum against total_gb, so this cannot underflow.
  assign spilled_gb = total_gb - resident_gb;
  assign all_resident = (spilled_gb == 16'd0);
  // Spilling to a slow tier while other nodes' fast tiers sat unused. No
  // separate node-count term is needed: with a single node the sharded capacity
  // is that node's own fast tier, so anything that spills is already larger than
  // the capacity and the last term excludes it.
  assign needless_spill_err = plan && !all_resident && !sharded
                              && (total_gb <= sharded_capacity_gb);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_spilling <= 8'd0;
    end else if (plan) begin
      n_plans <= n_plans + 8'd1;
      if (!all_resident) n_spilling <= n_spilling + 8'd1;
    end
  end
endmodule

Five plans. 80 GB of HBM per node.

Two plans spilled when sharding is available; four without it.

Row three is the case that makes the check honest. A single node genuinely has nowhere else to put the model, so spilling is not a mistake — it is the only option. The redundant node-count term this originally carried is documented in the comment above, and section 18 records why it was removed rather than kept for clarity.

Row two is the honest limit of sharding. Four nodes hold 320 GB and the model is 560, so 240 spills whichever strategy is chosen. Sharding is not a substitute for capacity; it is a way of using capacity that already exists, and that is why it is checked before tiering rather than instead of it.

And row five is the failure that has a name in every serving stack. A model that fits four nodes' HBM comfortably, kept on one node and spilled to CXL, because the tiering path was easier to configure than the sharding path. The fast memory was there and was not used.

8. RTL 4 — A Transformer Is A Sequence, So Stream It

// RTL 4 - streaming a layer at a time. A transformer is a sequence of layers,
// so the next one can be fetched while the current one computes.
module layer_streaming #(parameter int NO_PREFETCH = 0) (
  input  logic clk, rst_n,
  input  logic        run_layer,
  input  logic [15:0] layer_gb, tier_gbps, compute_ms, layers,
  output logic [15:0] fetch_ms, overlap_ms, exposed_ms, total_added_ms,
  output logic        streamable,
  output logic [7:0]  n_layers_run, n_exposed,
  output logic        stall_err
);
  logic [31:0] f_q, t_q;
  assign f_q = (tier_gbps == 16'd0) ? 32'd65535
             : (({16'd0, layer_gb} * 32'd8000) / {16'd0, tier_gbps});
  assign fetch_ms = (f_q > 32'd65535) ? 16'hFFFF : f_q[15:0];
  // Without prefetch the fetch is serial with the compute and hides nothing.
  assign overlap_ms = (NO_PREFETCH != 0) ? 16'd0
                    : ((fetch_ms > compute_ms) ? compute_ms : fetch_ms);
  assign exposed_ms = fetch_ms - overlap_ms;
  assign streamable = (exposed_ms == 16'd0);
  assign t_q = {16'd0, exposed_ms} * {16'd0, layers};
  assign total_added_ms = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign stall_err = run_layer && !streamable;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_layers_run <= 8'd0; n_exposed <= 8'd0;
    end else if (run_layer) begin
      n_layers_run <= n_layers_run + 8'd1;
      if (!streamable) n_exposed <= n_exposed + 8'd1;
    end
  end
endmodule

Five layers. An 80-layer model with 100 ms of compute per layer.

Two layers exposed with prefetch; four without it, adding 6400 ms to row one alone.

This is 22.3 §8's staging argument with the structure that makes it natural. A transformer is a strict sequence of layers, each depending only on the previous one's output, so the address of the next fetch is known before the current compute begins. There is no prediction, no locality heuristic and no cache — the schedule is static and perfectly known.

Row two is the sizing constraint that follows. A layer larger than what the tier can deliver in one compute step cannot be hidden, and the exposed remainder is paid once per layer. 60 ms across 80 layers is 4.8 seconds added to every forward pass, from a tier that reports itself healthy the entire time.

Row five is where the two builds agree. An empty layer fetches nothing, so there is nothing to hide and nothing to expose — and it is the only configuration in which the no-prefetch build is not making a mistake.

9. Waveform — Layer Fetch Under Layer Compute

The two scenarios move identical bytes over an identical tier. What separates them is a prefetch issued one layer ahead, which is a scheduling decision made in the serving framework and nowhere else. A tier's bandwidth cannot rescue a stack that fetches serially, and a tier's slowness cannot hurt one that does not.

10. RTL 5 — Prefill And Decode Are Different Machines

// RTL 5 - prefill and decode are different machines. Prefill reads the weights
// once for a whole prompt; decode reads all of them again for every token.
module phase_split #(parameter int ONE_PHASE = 0) (
  input  logic clk, rst_n,
  input  logic        serve,
  input  logic [15:0] weight_gb, prompt_tokens, output_tokens,
  output logic [15:0] prefill_reads, decode_reads, total_reads_gb,
  output logic        decode_dominates,
  output logic [7:0]  n_requests, n_decode_dominated,
  output logic        phase_collapsed_err
);
  logic [31:0] d_q, t_q;
  // Prefill streams the weights once regardless of prompt length.
  assign prefill_reads = 16'd1;
  // Decode reads them once per generated token. A single-phase model reuses the
  // prefill count and misses the whole of decode.
  assign decode_reads = (ONE_PHASE != 0) ? 16'd1 : output_tokens;
  assign t_q = ({16'd0, prefill_reads} + {16'd0, decode_reads}) * {16'd0, weight_gb};
  assign total_reads_gb = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign decode_dominates = (decode_reads > prefill_reads);
  // A request whose decode cost was folded into a single prefill pass.
  assign phase_collapsed_err = serve && (output_tokens > 16'd1)
                               && (decode_reads == 16'd1);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_decode_dominated <= 8'd0;
    end else if (serve) begin
      n_requests <= n_requests + 8'd1;
      if (decode_dominates) n_decode_dominated <= n_decode_dominated + 8'd1;
    end
  end
endmodule

Five requests. A 140 GB model.

Three requests were decode-dominated; the single-phase model saw none.

Prefill is compute-bound and decode is memory-bound, and they are the same model. Prefill processes the whole prompt in parallel and reads each weight once for all of it; decode processes one token at a time and reads every active weight again for each one. A machine tuned for the first is the wrong machine for the second.

Fifty times is not a modelling nicety. A hundred-token response reads the model a hundred and one times, and the entire question of which tier the weights live in is decided by that number rather than by the footprint. Capacity is a prefill question; bandwidth is a decode question, and section 12 is what happens when only the first is asked.

Row two is the case where the single-phase model is right, and it is worth naming because it is also the case people benchmark: one token generated, which is a prefill measurement wearing a decode label. A serving benchmark that generates one token measures the phase that is not the bottleneck.

11. RTL 6 — Most Of A Sparse Model Is Cold

// RTL 6 - mixture of experts. Only a few experts run per token, so most of the
// parameter count is cold, and cold parameters are what a slow tier is for.
module expert_routing #(parameter int ALL_EXPERTS_HOT = 0) (
  input  logic clk, rst_n,
  input  logic        route,
  input  logic [15:0] total_gb, experts, active_experts, hbm_gb,
  output logic [15:0] hot_gb, cold_gb, hot_pct,
  output logic        hot_set_fits,
  output logic [7:0]  n_routes, n_hot_spills,
  output logic        all_hot_err
);
  logic [31:0] h_q, p_q;
  logic [15:0] eff_active;
  // Treating every expert as hot is what a dense-model view of MoE does.
  assign eff_active = (ALL_EXPERTS_HOT != 0) ? experts : active_experts;
  assign h_q = (experts == 16'd0) ? 32'd0
             : (({16'd0, total_gb} * {16'd0, eff_active}) / {16'd0, experts});
  assign hot_gb = (h_q > 32'd65535) ? 16'hFFFF : h_q[15:0];
  // hot_gb is a share of total_gb with eff_active never above experts, so this
  // cannot underflow.
  assign cold_gb = total_gb - hot_gb;
  assign p_q = (total_gb == 16'd0) ? 32'd0
             : (({16'd0, hot_gb} * 32'd100) / {16'd0, total_gb});
  assign hot_pct = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign hot_set_fits = (hot_gb <= hbm_gb);
  // A sparse model costed as if every expert were resident.
  assign all_hot_err = route && (active_experts < experts)
                       && (eff_active == experts);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_routes <= 8'd0; n_hot_spills <= 8'd0;
    end else if (route) begin
      n_routes <= n_routes + 8'd1;
      if (!hot_set_fits) n_hot_spills <= n_hot_spills + 8'd1;
    end
  end
endmodule

Five routings. A 640 GB model of 64 experts against an 80 GB fast tier.

Two routings spilled the hot set; the dense view spilled four.

This is the one workload shape in the chapter where CXL is unambiguously the right answer. A 640 GB model with a 20 GB hot set has 620 GB of parameters that are not read for this token — and holding those in a capacity tier while the active experts sit in HBM is exactly what a capacity tier is for.

The catch is that "cold" is per-token, not per-deployment. Every expert is active for some token, so the cold set is a rolling selection rather than a fixed one, and the tier has to supply a newly-hot expert on demand. That turns section 8's streaming problem into a routing-latency problem, and the model here deliberately does not solve it — see section 19.

Row three is the dense model, and the dense view is right about it. all_hot_err requires active_experts < experts, because a model with no sparsity has genuinely no cold set. The error is applying a dense assumption to a sparse model, not holding a dense assumption.

Figure 3 — The only clean fit in the chapter. Three percent of the parameters are read for a given token, so 97% of them are a capacity problem rather than a bandwidth one — and a capacity tier is precisely the right place for a capacity problem.

12. RTL 7 — Decode Reads Everything, Every Token

// RTL 7 - decode throughput. Every generated token reads the active weights,
// so the serving tier's bandwidth sets the token rate directly.
module decode_bandwidth #(parameter int IGNORE_WEIGHT_READ = 0) (
  input  logic clk, rst_n,
  input  logic        decode,
  input  logic [15:0] active_gb, tier_gbps, target_tok_s,
  output logic [15:0] ms_per_token, tokens_per_sec,
  output logic        meets_target,
  output logic [7:0]  n_decodes, n_missed,
  output logic        rate_overclaim_err
);
  logic [31:0] m_q, t_q;
  assign m_q = (tier_gbps == 16'd0) ? 32'd65535
             : (({16'd0, active_gb} * 32'd8000) / {16'd0, tier_gbps});
  assign ms_per_token = (m_q > 32'd65535) ? 16'hFFFF : m_q[15:0];
  // A model that ignores the weight read sees no bandwidth limit at all.
  assign t_q = (IGNORE_WEIGHT_READ != 0) ? 32'd65535
             : ((ms_per_token == 16'd0) ? 32'd65535
                : (32'd1000 / {16'd0, ms_per_token}));
  assign tokens_per_sec = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign meets_target = (tokens_per_sec >= target_tok_s);
  // An unbounded token rate reported for a tier that has to supply weights.
  assign rate_overclaim_err = decode && (active_gb != 16'd0) && (tier_gbps != 16'd0)
                              && (tokens_per_sec == 16'hFFFF);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_decodes <= 8'd0; n_missed <= 8'd0;
    end else if (decode) begin
      n_decodes <= n_decodes + 8'd1;
      if (!meets_target) n_missed <= n_missed + 8'd1;
    end
  end
endmodule

Six decodes. 8 GB of active weights read per token.

Three decodes missed the target; the model that ignores the weight read missed none.

Three tokens a second is the number this chapter is really about. It is not slow in the sense of needing tuning — it is unusable, sixteen times short of a target that itself is modest, and no amount of batching or kernel work recovers it because the bytes have to cross the link once per token regardless.

This is 22.3 §7's roofline specialised to the worst possible reuse. There, the demand came from a bytes-per-FLOP ratio that a well-blocked kernel could improve. Here the ratio is fixed by the algorithm: every active parameter is read exactly once per generated token, and no blocking changes that.

Row four is the case the overclaim check has to exempt. A decode step with no active weights has no bandwidth bound, and reporting an unbounded rate for it is correct rather than a claim. rate_overclaim_err requires both a real read and a real tier, which is why it stays silent on rows four and five.

13. RTL 8 — Offloading Is Never Free

// RTL 8 - offloading layers. Moving layers off the fast tier frees capacity and
// costs time, and the freed capacity has to be worth the time it cost.
module offload_cost #(parameter int FREE_OFFLOAD = 0) (
  input  logic clk, rst_n,
  input  logic        offload,
  input  logic [15:0] layers_offloaded, layer_gb, tier_gbps, budget_ms,
  output logic [15:0] freed_gb, added_ms,
  output logic        within_budget,
  output logic [7:0]  n_offloads, n_over_budget,
  output logic        free_lunch_err
);
  logic [31:0] f_q, a_q;
  assign f_q = {16'd0, layers_offloaded} * {16'd0, layer_gb};
  assign freed_gb = (f_q > 32'd65535) ? 16'hFFFF : f_q[15:0];
  // Offloading is never free: the layers have to come back over the tier.
  assign a_q = (FREE_OFFLOAD != 0) ? 32'd0
             : ((tier_gbps == 16'd0) ? 32'd65535
                : (({16'd0, freed_gb} * 32'd8000) / {16'd0, tier_gbps}));
  assign added_ms = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign within_budget = (added_ms <= budget_ms);
  // Capacity freed at no reported cost.
  assign free_lunch_err = offload && (freed_gb != 16'd0) && (added_ms == 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_offloads <= 8'd0; n_over_budget <= 8'd0;
    end else if (offload) begin
      n_offloads <= n_offloads + 8'd1;
      if (!within_budget) n_over_budget <= n_over_budget + 8'd1;
    end
  end
endmodule

Five offloads. 2 GB layers against a 100 ms budget.

Two offloads exceeded the budget; the free-offload model reported none.

Offloading trades capacity for time at a fixed exchange rate, and the rate is the tier's bandwidth. There is no configuration in which it is free, and a plan that shows freed capacity without a corresponding time cost has not modelled the return path at all.

Row one is the shape that catches people. Offloading ten layers looks like a modest fraction of an 80-layer model, and it costs 800 ms per forward pass — which section 8 could have hidden had the fetches been overlapped, and cannot when they are not. The two models are the same transfer measured under different schedules, and section 20 says which knob controls which.

Row three is the trivial case and it matters for the check. Freeing nothing costs nothing, and that is not a free lunch — it is an empty plate. free_lunch_err requires capacity to have actually been freed.

14. RTL 9 — Cheaper Is Not Cheaper Per Token

// RTL 9 - tier against shard, priced. Adding CXL to one node is cheaper than
// adding nodes and serves far fewer tokens, so cost alone decides nothing.
module serving_economics #(parameter int IGNORE_RATE = 0) (
  input  logic clk, rst_n,
  input  logic        compare,
  input  logic [15:0] node_cost, cxl_add_cost, nodes,
  input  logic [15:0] cxl_tok_s, shard_tok_s,
  output logic [15:0] cxl_cost, shard_cost, cxl_per_mtok, shard_per_mtok,
  output logic        tier_cheaper_per_token,
  output logic [7:0]  n_compares, n_tier_wins,
  output logic        rate_ignored_err
);
  logic [31:0] sc_q, cp_q, sp_q;
  assign cxl_cost = node_cost + cxl_add_cost;
  assign sc_q = {16'd0, node_cost} * {16'd0, nodes};
  assign shard_cost = (sc_q > 32'd65535) ? 16'hFFFF : sc_q[15:0];
  assign cp_q = (cxl_tok_s == 16'd0) ? 32'd65535
              : (({16'd0, cxl_cost} * 32'd1000) / {16'd0, cxl_tok_s});
  assign cxl_per_mtok = (cp_q > 32'd65535) ? 16'hFFFF : cp_q[15:0];
  assign sp_q = (shard_tok_s == 16'd0) ? 32'd65535
              : (({16'd0, shard_cost} * 32'd1000) / {16'd0, shard_tok_s});
  assign shard_per_mtok = (sp_q > 32'd65535) ? 16'hFFFF : sp_q[15:0];
  // Comparing outlay instead of cost per token is what a budget sheet does.
  assign tier_cheaper_per_token = (IGNORE_RATE != 0) ? (cxl_cost < shard_cost)
                                : (cxl_per_mtok < shard_per_mtok);
  // The cheaper option declared on outlay while its cost per token is worse.
  assign rate_ignored_err = compare && tier_cheaper_per_token
                            && (cxl_per_mtok > shard_per_mtok);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_compares <= 8'd0; n_tier_wins <= 8'd0;
    end else if (compare) begin
      n_compares <= n_compares + 8'd1;
      if (tier_cheaper_per_token) n_tier_wins <= n_tier_wins + 8'd1;
    end
  end
endmodule

Five comparisons. One node at 100 plus CXL at 20, against sharded nodes at 100 each.

One tiering win on cost per token; four on outlay alone.

The tiered node is genuinely cheaper and genuinely worse. 120 against 400 is a real saving, and 40,000 against 8,000 per million tokens is a real five-times penalty — both numbers are correct, and only one of them is what anybody is buying.

Row two is the configuration where tiering wins outright, and it is section 11's model: a sparse model whose hot set fits HBM, tiered for the cold experts. The tiered node keeps the full token rate and costs a third as much. That is the deployment CXL was built for, and it is a specific one rather than a general case.

Row three is the honest comparison nobody makes. Against a single node, the tiered node is more expensive on outlay too — the outlay model gets it right, because the comparison stopped flattering the tier. The outlay model is not always wrong; it is wrong exactly when the rates differ, which is what row five's tie makes precise.

15. RTL 10 — CXL For Large Language Models Assembled

// RTL 10 - CXL for large language models assembled. Everything that must hold
// before a tiered node serves a model rather than merely holding one.
module llm_cxl_model #(parameter int FITS_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       fits_somewhere,      // the footprint lands on the machine
  input  logic       footprint_complete,  // optimizer state and precision counted
  input  logic       sharded_first,       // fast tiers used before slow ones
  input  logic       streaming_overlapped,// layer fetch hides behind compute
  input  logic       decode_rate_met,     // the tier supplies the token rate
  input  logic       cold_set_identified, // the slow tier holds only cold weights
  output logic       serves,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_serves,
  output logic       false_serve_err
);
  assign fail_mask[0] = ~fits_somewhere;
  assign fail_mask[1] = ~footprint_complete;
  assign fail_mask[2] = ~sharded_first;
  assign fail_mask[3] = ~streaming_overlapped;
  assign fail_mask[4] = ~decode_rate_met;
  assign fail_mask[5] = ~cold_set_identified;
  // The fits-only build checks that the model loads and calls that serving.
  assign serves = (FITS_ONLY != 0) ? fits_somewhere : (fail_mask == 6'd0);
  assign false_serve_err = evaluate && serves && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_serves <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (serves) n_serves <= n_serves + 8'd1;
    end
  end
endmodule

One serving configuration of six, and four false claims.

The fits-only definition is a model loader, and it is right about exactly one of the six — the one where nothing fits. Row four is the deployment this chapter is written against: everything about the memory plan is correct, the model loads, and it produces three tokens a second.

Figure 4 — Three of the four terminals are good outcomes and the fourth is a warning rather than a plan. A dense model with no cold set, streamed from a capacity tier, is the configuration section 12 prices at three tokens a second — and the flowchart reaches it only after every better option has been ruled out, which is the order sections 6, 7 and 11 exist to enforce.

16. Quantitative Reasoning

Footprint. 70 billion parameters at two bytes: 140 GB of weights, 560 GB to train — four times, from a gradient copy and two optimizer copies.

Precision. The same model is 140 GB at sixteen bits, 70 at eight, 35 at four. Four of five configurations fit an 80 GB node; the fp16 model said none did.

Shard against tier. 560 GB across eight 80 GB nodes is fully resident; the same model tiered on one node keeps 80 GB and spills 480.

Layer streaming. A 2 GB layer at 200 Gbps is 80 ms, entirely hidden by 100 ms of compute. Without prefetch it is 80 ms exposed per layer — 6400 ms across 80 layers.

Phases. A 140 GB model generating 100 tokens reads 14,140 GB; a single-phase model reports 280 — fifty times out.

Expert routing. 640 GB across 64 experts with 2 active: 20 GB hot, 620 cold, a 3% hot share — the one configuration that fits an 80 GB tier cleanly.

Decode bandwidth. 8 GB of active weights per token: 50 tokens a second from HBM, 6 from host DRAM, 3 from CXL.

Offload. Ten 2 GB layers frees 20 GB and costs 800 ms — eight times a 100 ms budget.

Economics. 120 against 400 in outlay, 40,000 against 8,000 per million tokens — cheaper and five times worse.

The assembled model. Six properties, six configurations, one serves. The fits-only definition reported five.

17. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Footprint. The serving case is asserted as not an undercount, and a footprint exactly the size of the machine is driven.

chk(fGt == 16'd140, "the served footprint is 140 GB");
chk(fGf == 1'b1,    "which exactly fills a 140 GB machine");

Precision. The unstated-precision case is asserted as not an ignored precision, and a model sized to exactly fill the node is driven.

Shard against tier. The single-node case is asserted as not a needless spill, and the four-node case that cannot fit either way is asserted the same way.

chk(sGs == 16'd240, "so 240 GB spills even when sharded");
chk(sGe == 1'b0,    "which is a capacity failure, not a needless spill");

Streaming. A fetch exactly as long as the compute is driven, and the empty layer is asserted streamable in both builds.

Phases. The single-token request is asserted as a case the single-phase model gets right, and the overflowing 500-token total is driven.

Expert routing. The dense model is asserted as a case the dense view gets right, and the exactly-filling eight-expert case is driven.

Decode bandwidth. A rate exactly at the target is driven, and both exemptions from the overclaim check are asserted.

chk(dGt == 16'd50,  "fifty tokens a second");
chk(dGmeet == 1'b1, "exactly meets a fifty-token target");

Offload. A return cost exactly equal to the budget is driven, and the zero-offload case is asserted as not a free lunch.

Economics. The exact per-token tie is driven, and the outlay model's claim there is asserted unfounded rather than wrong.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 249 checks across two testbenches, 127 on the front five models and 122 on the back five, all passing on the unmutated sources.

18. Mutation Testing

Fifty-four mutations were injected one at a time.

54 injected, 54 killed, after three survivors.

Survivor 1 — a term dominated by the terms beside it. needless_spill_err carried a nodes > 1 guard that survived removal. The reason is provable rather than empirical: with a single node the sharded capacity is that node's own fast tier, so anything that spills is already larger than the capacity, and the total_gb <= sharded_capacity_gb term excludes it. The same holds for zero nodes. The term was deleted with a comment saying so, and the mutation replaced with one that drops the residency test instead.

That is the second such finding in two chapters, after 22.3 §7's zero-demand guard. A guard that reads as defensive can be arithmetically dead, and the only way to tell is to ask what the other terms already exclude.

Survivors 2 and 3 — the same boundary in two places. fits_one_node and meets_target both survived >= becoming >, because no case drove a footprint exactly the size of the machine or a token rate exactly at the target. Both are the boundary the comparison exists to define, and both were added as stimulus rather than changed in the design.

Both boundaries are also the cases a real deployment lands on. A model sized to exactly fill a node is what a capacity planner produces, and a token rate quoted as exactly the SLA is what a contract says. Inclusive-or-exclusive at that point is not a detail; it is the difference between meeting the target and missing it.

19. Verification Strategy

What a testbench for a large-language-model placement stack must cover.

Ask what the other terms already exclude before adding a guard. Section 7's node-count term was dead given the capacity test beside it, and the only evidence was a mutation that would not die.

Drive every threshold at exactly equal. Two of three survivors here were the same omission in different models, which suggests it is systematic rather than incidental.

The cases where the broken build is correct. Weights-only for inference. fp16 when the model really is fp16. The dense view of a dense model. A single-token request. Outlay against a single node. Each is a case the check must exempt, and each is a case a real team is entitled to.

Separately computed truth for consistency checks. Not needed here, but 22.3 §11 shows the failure it prevents: a model wrong in one place and wrong consistently downstream passes every internal test.

Counters as a second signature. Every model reports how many cases it processed and how many failed, and the broken build's totals differ from the correct build's in every model — five against zero, four against two, three against none. A single output can agree by accident; a pair of totals across five cases does not.

What a real placement stack needs that these models do not have. Rolling expert residency, which section 11 explicitly defers. Batching, which amortises the decode read across concurrent requests and is the single largest omission in section 12. Pipeline parallelism, which changes the streaming schedule in section 8 entirely.

20. Synthesis and Implementation Reality

The footprint arithmetic in section 5 is a planning tool, not hardware. It lives in a capacity spreadsheet or a deployment controller, and its failure mode is a purchase order rather than a timing violation.

Precision is a compile-time decision with a runtime footprint. Section 6's factor of four is realised by quantised kernels, and the cost is accuracy rather than silicon — which is why it is a model-team decision that a platform team inherits.

Sharding and tiering are different software stacks. Section 7's ordering — fast tiers first — is a policy that has to be expressed somewhere, and the reason it is often violated is that the tiering path takes one configuration flag and the sharding path takes a collective library.

Section 8's overlap is a prefetch depth of one layer, which needs a second layer's worth of HBM to land in. On a node that is already short of HBM, the streaming plan competes with the residency plan for the same bytes — the identical conflict 22.3 §20 raised between staging and the KV cache.

Section 12's decode rate is the one number that is pure hardware. Active bytes divided by tier bandwidth, once per token, with no software able to change either term. Batching amortises it across requests and does not reduce it per request, which is why latency and throughput diverge in exactly this workload.

21. Silicon Observability

"Expert-fault latency" is the counter this chapter cannot supply and a real MoE deployment cannot do without. Section 11 models the cold set as a static share, which is right for capacity and wrong for latency — every expert is hot for some token, and the tail of the token-latency distribution is entirely made of tokens that routed to an expert not currently resident. A deployment reporting mean decode latency will not see it.

22. Debug Lab

Symptom. A 140 GB model is deployed on one node with 80 GB of HBM plus 512 GB of CXL, because the sharded four-node configuration was four times the cost. The model loads. Prefill latency is excellent. Generation runs at four tokens a second against a fifty-token target, and no counter reports an error.

Step 1 — is the model where it was meant to be? Resident bytes by tier: 80 GB in HBM, 60 GB in CXL. The placement is exactly as planned, which rules out a misconfiguration and rules in the plan.

Step 2 — is prefill or decode the problem? Achieved tokens per second, split by phase: prefill is comfortably within target, decode is at four. Section 10's split is the first thing to measure, and it immediately halves the search.

Step 3 — what does decode read? Weight bytes per generated token: the full 140 GB of active weights, because the model is dense. There is no cold set to exploit — section 11 does not apply to this model.

Step 4 — what can the tiers supply? 80 GB from HBM at 3200 Gbps is 200 ms; 60 GB from CXL at 200 Gbps is 2400 ms. Per token. The CXL-resident 43% of the model is supplying 92% of the time.

Step 5 — could streaming have hidden it? Layer fetches issued: the framework does prefetch, and the overlap counter shows fetches fully overlapped during prefill. During decode there is nothing to overlap with — a decode step is 20 ms of compute against 2400 ms of fetch, so section 8's mechanism has nothing to hide behind.

The finding. Section 8's streaming works during prefill and cannot work during decode, because decode's compute is three orders of magnitude shorter than its fetch. The capacity plan was correct, the placement was correct, the prefetch was correct, and the workload was the one shape none of it helps.

The fix. Two options, and they are the two the flowchart in section 15 offers. Quantise to eight bits — 70 GB, resident entirely in HBM, and the CXL tier becomes unnecessary. Or shard across four nodes at four times the outlay and, per section 14, a fifth of the cost per token. The team quantised.

What made this hard. Every subsystem behaved exactly as designed, and the only failing number was a business one. The defect was in the decision to tier a dense model at all, which is a choice made before any of the counters existed.

23. Design Review

1. Is this footprint for training or serving, and does it include optimizer state? A factor of four. Section 5.

2. What precision, and was it settled before the capacity was sized? A factor of four, for free. Section 6.

3. Does the footprint leave room for activations, the KV cache and the runtime? A footprint that exactly fills the node fits arithmetically and fails on the first request. Section 5.

4. Are there other nodes with unused fast memory? Shard before tiering. Section 7.

5. Is the model sparse, and what is the hot share? Three percent is the difference between a good CXL deployment and section 22. Section 11.

6. How many bytes does one generated token read? The only number that sets the decode rate. Section 12.

7. What token rate does the slowest resident tier support? If nobody has computed it, section 22 is the deployment.

8. Is the layer fetch overlapped, and with what? During decode there may be nothing to overlap with. Sections 8 and 22.

9. What does offloading cost in time, not just what does it free? There is no free offload. Section 13.

10. Is the comparison outlay or cost per token? Cheaper and five times worse are both true. Section 14.

24. How This Appears In Real Engineering

A model team choosing a precision is making the largest single capacity decision in the deployment, usually before any platform engineer is in the room. Section 6's factor of four decides whether a tiering conversation happens at all — and the cheapest capacity plan is frequently a quantisation ticket rather than a hardware one.

A platform team sizing a serving fleet meets sections 10 and 12 together. Prefill and decode have different bottlenecks, different tier requirements and different scaling behaviour, and a fleet sized on a single blended number is wrong for both. The industry's move toward disaggregated prefill and decode pools is exactly this observation turned into topology.

An infrastructure team evaluating CXL for AI should read sections 11 and 22 as a matched pair. Sparse models are the case that works — a rolling hot set that fits the fast tier and a large cold remainder that does not need bandwidth. Dense models are the case that does not, and no amount of good engineering elsewhere changes it.

A finance or capacity-planning function owns section 14, and the failure there is structural rather than careless: an outlay comparison is the one a budget system can compute, and cost per token requires a performance number that the budget system does not have. Supplying it is a platform-team responsibility.

25. Common Misconceptions

"The model is 140 GB." For serving. For training it is 560. Section 5.

"We need more memory." You may need fewer bits — a factor of four before buying anything. Section 6.

"CXL lets us run the model on one node." It lets you load it on one node. Sections 12 and 22.

"Tiering and sharding are alternatives." Sharding uses memory that already exists; tier only after it is exhausted. Section 7.

"Streaming solves the capacity problem." During prefill. During decode there is nothing to overlap with. Sections 8 and 22.

"Decode is just prefill repeated." Prefill reads the weights once; decode reads them per token — fifty times over for a short reply. Section 10.

"CXL is too slow for LLMs." For a dense model's decode path, yes. For a sparse model's cold experts it is exactly right. Sections 11 and 12.

"Offloading frees memory." It trades memory for time at the tier's bandwidth. Section 13.

"The tiered node is cheaper." In outlay. Per token it is five times worse. Section 14.

"It loads, so it serves." One property of six. Section 15.

26. Interview Reasoning

Q. How much memory does a 70-billion-parameter model need?

Two questions back: what precision, and training or serving? At two bytes it is 140 GB of weights; training adds a gradient and two optimizer copies for 560 GB. At four bits it is 35 GB. The parameter count alone does not determine the answer within a factor of sixteen.

Q. A model does not fit your node. What do you try, in what order?

Precision first, because it is free and linear — a factor of four. Then sharding, because it uses fast memory that already exists. Then tiering, and only if the model has a cold set. The order matters because each step is cheaper and faster than the next, and the common failure is jumping to the last one because it takes a single configuration flag.

Q. Why is CXL fine for a mixture-of-experts model and not for a dense one?

Because sparsity creates a genuine cold set. Two of sixty-four experts active per token means a 640 GB model has a 20 GB hot set and 620 GB that is not read for this token — capacity without bandwidth, which is exactly what a capacity tier supplies. A dense model reads every active weight per token, so nothing is cold.

Q. Your model loads and generates four tokens a second. Where do you look?

Bytes read per token against the bandwidth of the slowest tier holding any of them. If 60 GB of a dense model is on a 200 Gbps link, that is 2400 ms per token before anything else happens. The tier holding the smallest share of the model can supply the overwhelming majority of the latency, so share of capacity is the wrong thing to look at.

Q. Why can't prefetching hide it, when it hides the same fetch during prefill?

Because prefetching hides a fetch behind compute, and a decode step has almost no compute — twenty milliseconds against two and a half seconds of fetch. Prefill has a long compute phase to hide behind and decode does not, which is why the same mechanism works in one phase and is useless in the other.

Q. The tiered node costs a third of the sharded one. Is it cheaper?

In outlay, yes and genuinely. Per million tokens it is five times worse, because it serves a sixteenth of the rate. Both numbers are correct; the outlay number is the one a budget system can compute without a performance measurement, which is why it is usually the one that gets quoted.

27. Exercises

1. Extend RTL 1 with activation memory as a function of batch and sequence, and find the batch at which a footprint that "exactly fits" stops fitting.

2. Add an accuracy penalty per bit to RTL 2 and find the precision that minimises cost per token at a fixed quality floor.

3. Extend RTL 3 to price sharding's collective traffic, and find the node count at which sharding stops being free.

4. Give RTL 4 a prefetch depth greater than one and show how much HBM each additional layer of depth costs.

5. Split RTL 5 into separate prefill and decode rate models and find the output length at which decode overtakes prefill in total time.

6. Make RTL 6's cold set rolling: add an expert-fault rate and a fault latency, and show how the mean token time degrades.

7. Add batching to RTL 7 — the weight read is amortised across concurrent requests — and show that throughput improves while per-token latency does not.

8. Combine RTL 8 with RTL 4: offload only layers whose fetch can be hidden, and find how much can be freed for nothing.

9. Extend RTL 9 to a third option, a sparse model on a tiered node, and rank all three by cost per token.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the deployment it catches that the current mask calls serving.

28. Summary

22.3 established a hierarchy sixteen times apart with CXL at the bottom. This chapter put on it the workload with the worst possible reuse — a model read in its entirety for every token it generates — and the arithmetic is unforgiving.

The footprint is not the parameter count. 70 billion parameters at two bytes is 140 GB served and 560 GB trained, four times over, from a gradient copy and two optimizer copies.

Precision is the cheapest lever there is. The same model is 140, 70 or 35 GB, and four of five configurations fit an 80 GB node that the fp16 assumption said none did.

Shard before you tier. 560 GB across eight nodes is fully resident; tiered on one node it keeps 80 GB and spills 480 — and a 200 GB model that fits four nodes' HBM was spilled for nothing.

A transformer is a sequence, so its fetches are perfectly predictable. A 2 GB layer at 200 Gbps is entirely hidden by 100 ms of compute, and serially issued it adds 6400 ms across 80 layers.

Prefill and decode are different machines. A 140 GB model generating 100 tokens reads 14,140 GB, and a single-phase model reports 280 — fifty times out, which is the error that makes every tier decision look easier than it is.

Sparsity is the case that works. Two of sixty-four experts active gives a 20 GB hot set against 620 GB cold — capacity without bandwidth, which is precisely what CXL supplies.

And density is the case that does not. 8 GB of active weights per token is 50 tokens a second from HBM and three from CXL — sixteen times short, with no software able to change either term.

Offloading is never free. Ten 2 GB layers frees 20 GB and costs 800 ms, eight times the budget, and a plan showing capacity freed at no cost has not modelled the return path.

Cheaper is not cheaper per token. 120 against 400 in outlay, 40,000 against 8,000 per million tokens — both correct, and only one is what is being bought.

A guard that reads as defensive can be arithmetically dead. Section 7's node-count term was already excluded by the capacity test beside it — the second such finding in two chapters, and the only evidence was a mutation that would not die.

Loading is one property of six. The definition a model loader uses called five of six deployments serving when one was.

22.5 — AI Training Clusters takes this from one node to a rack, where the footprint from section 5 is spread across a fabric and the streaming schedule from section 8 becomes a collective.