AI Training Clusters in Practice

Module 28 compared CXL against other things. Module 29 asks where it actually ends up, and the first case is the one with the most written about it and the least of it checkable.

They use CXL for memory. The sentence is true of something and it identifies nothing: a training cluster has several memory tiers, they differ by more than an order of magnitude in bandwidth, and the sentence does not say which one — nor what did not fit, nor what the traffic costs when it lands somewhere slower.

1. The Engineering Problem — A Cluster Has Several Memories

"Memory" is not a tier. Four tiers present with one named leaves three the reader has to choose between, and the named one may be carrying six percent of the step. Section 5.

The capacity wall is a sum. Parameters, optimizer state and activations are resident at once: 120 plus 240 plus 80 against 80 gigabytes of fast tier is 360 over, at five and a half times the tier. Section 6.

A tier only helps if the traffic tolerates its bandwidth. Four hundred gigabytes per second wanted against sixty-four offered is sixteen percent served, and capacity is not the axis that fails. Section 7.

Only the hot part needs the fast tier. Three hundred gigabytes hot against eighty resident is two hundred and twenty evicted and eight hundred and eighty refetched across four reuses. Section 8.

An attach inside a node is not the fabric between nodes. Twenty links claimed for an attach with eight inside the node is twelve claimed outside its reach. Section 9.

And the fast-tier floor does not move. Adding five hundred and twelve gigabytes further away leaves a design eighty gigabytes short of a floor those gigabytes cannot reach, at three thousand units spent. Section 13.

Why this is the module's first case. Training is where the capacity pressure is most acute, most public and most often described in a sentence that has no tier in it. Every later case in Module 29 inherits the tier question from this one, because a deployment that has not said which memory it means has not said anything a later chapter can build on.

2. The One-Sentence Model

A training-cluster case study is sound when the attach is named, when the tier it serves is named rather than called "memory", when the capacity wall is computed as a sum, when the bandwidth floor is checked against the traffic that will land there, when the stall cost is computed, and when the scale boundary between a node and the fabric between nodes is stated — and "they use CXL for memory" is bit 0.

3. What This Chapter Owns

Some vocabulary, because the tiers are what the chapter is about and they are routinely collapsed into one word.

The fast tier is the memory attached directly to the accelerator die — high-bandwidth memory, stacked and in-package, and it is the tier a training step runs out of. Its defining property is bandwidth rather than capacity, and its defining limit is that there is not very much of it.

Host memory is the DRAM attached to the processor the accelerator is connected to. It is larger, slower, and reached across a link.

An attached tier is memory reached over a link rather than over a package — which is where CXL appears. It is larger again, slower again, and its bandwidth is set by the link rather than by the memory.

And the scale boundary is the line between a node and the fabric between nodes. Inside a node the connections are short and dense; between nodes they are a network. They are different problems and an attach lives on one side of that line, which is section 9.

4. Teaching-Model Boundary

This is a case-study chapter, and the boundary matters more here than anywhere else in the track.

Every model computes a property of a decision, not a claim about any company. None of the figures in this chapter is a measurement of any real cluster, product or deployment. A capacity of eighty gigabytes, a bandwidth of sixty-four, a stall of eighteen milliseconds — all of them are teaching figures chosen to make one relationship visible, and none should be read as a specification, a benchmark or a roadmap.

Where a real technology is named it is for a publicly established fact only. That high-bandwidth memory is what training accelerators attach directly. That NVIDIA's NVLink and AMD's Infinity Fabric are vendor fabrics connecting a vendor's own devices. That CXL attaches memory over a link. Nothing here attributes a number, a design choice or a plan to any named organisation, and any resemblance between a model's inputs and a real part is a coincidence of round numbers.

Each model is built twice from one source. A parameter selects between the measured build, which counts what the deployment actually rests on, and the memory-answer build, which treats "it is memory" as the finding. Every section's headline number is the gap between them.

5. RTL 1 — "Memory" Names No Tier

Start where the sentence starts, because the whole chapter is downstream of which tier is meant.

A training cluster has several memories and they are not interchangeable. The fast tier in the accelerator package, the host's DRAM, memory attached across a link, and storage below that. They differ by more than an order of magnitude in bandwidth and by more than two in capacity, which means a sentence that says "memory" has named a category containing its own opposite.

// RTL 1 - "they use CXL for memory" names no tier. A training cluster has
// several memory tiers and they differ by orders of magnitude in bandwidth, so
// a claim that does not say which one has not said anything measurable.
module memory_tier_claim #(parameter int ANY_TIER_IS_MEMORY = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] tiers_present, tiers_named, step_gb, cxl_gb,
  output logic [15:0] tiers_unnamed, cxl_share_pct, named_ok, claim_pct,
  output logic        tier_identified,
  output logic [7:0]  n_evals, n_ambiguous,
  output logic        tier_err
);
  logic [31:0] s_q, c_q;
  logic [15:0] true_unnamed;
  logic        truly_ambiguous;
  assign named_ok = (tiers_named > tiers_present) ? tiers_present : tiers_named;
  assign true_unnamed = tiers_present - named_ok;
  assign tiers_unnamed = (ANY_TIER_IS_MEMORY != 0) ? 16'd0 : true_unnamed;
  // The share of a step's bytes the named tier actually carries.
  assign s_q = (step_gb == 16'd0) ? 32'd0
             : (({16'd0, cxl_gb} * 32'd100) / {16'd0, step_gb});
  assign cxl_share_pct = (ANY_TIER_IS_MEMORY != 0) ? 16'd100
                       : ((s_q > 32'd100) ? 16'd100 : s_q[15:0]);
  // How much of the tier structure the claim actually pins down.
  assign c_q = (tiers_present == 16'd0) ? 32'd100
             : (({16'd0, named_ok} * 32'd100) / {16'd0, tiers_present});
  assign claim_pct = (ANY_TIER_IS_MEMORY != 0) ? 16'd100 : c_q[15:0];
  assign tier_identified = (tiers_unnamed == 16'd0) && (tiers_present != 16'd0);
  assign truly_ambiguous = (true_unnamed != 16'd0);
  assign tier_err = evaluate && truly_ambiguous && tier_identified;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_ambiguous <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_ambiguous) n_ambiguous <= n_ambiguous + 8'd1;
    end
  end
endmodule

Four tiers with one named, and the named tier carrying sixty of a thousand gigabytes, leaves three tiers unnamed, six percent of the step, and a quarter of the structure pinned down.

Figure 1 — why the sentence survives contact with an audience. The upper path is not lying; the cluster does use CXL, and what CXL attaches is memory. The claim is true and carries no information, because every tier in the diagram satisfies it equally. The lower path asks which one, and the two numbers it returns — three unnamed, six percent — are the two facts that decide whether the attach mattered.

The sixth case is the one that makes the section a measurement rather than a complaint. Over-claimed naming, nine tiers with four named, reports forty-four percent pinned down rather than a hundred: the model counts the tiers that exist, and naming a tier twice does not name a second one.

The fifth case bounds it in the other direction. More tiers named than the cluster has clamps to the tiers present and reports nothing unnamed, which is the right answer — a description that over-enumerates has still described everything there is.

The third case is the share clamp, and it is worth reading as a sanity check on the traffic figure rather than on the naming. More bytes on the tier than the step contains saturates at a hundred percent, because a tier cannot carry more of a step than the step has.

The degenerate case is the honest zero. No tier structure written down at all reports nothing unnamed and nothing identified, which is a deployment nobody has described rather than one described badly.

It is worth being concrete about how far apart the tiers actually are, because "several memory tiers" is otherwise as vague as the sentence it is replacing. The memory stacked on an accelerator package and memory reached across a link differ by roughly an order of magnitude in bandwidth and by rather more in capacity, and host DRAM sits between them on both axes. That spread is the whole reason the tiers are separate things: if they were within a factor of two of each other nobody would need to say which one, because the answer would not change any decision.

The spread also explains why the sentence is so durable. A claim that ranges over a set whose members differ by an order of magnitude is satisfied by the best case and the worst case equally, so no observation can embarrass it. Somebody who meant "we attached a large capacity tier for optimizer state" and somebody who meant "our accelerators have a lot of HBM" produce the same sentence, and a reader cannot tell them apart. Two engineers can agree on it completely and be describing different machines.

The fix is one question and it costs nothing, which is why this section is first rather than longest. Asking which tier converts an unfalsifiable sentence into a claim with a number behind it, and every remaining section of the chapter is a number about the tier that answer names. Without it they have no subject.

6. RTL 2 — The Capacity Wall Is A Sum

The second thing, and the first that is a real number rather than a distinction.

A training step holds three things in the fast tier at once. The parameters, the optimizer state that accompanies them, and the activations produced on the way forward and consumed on the way back. The sum is the requirement, and a description that quotes only the parameter count has quoted the smallest of the three.

That sum is the entire reason another tier is under discussion, which is why getting it as a sum rather than as a headline matters.

// RTL 2 - the capacity wall. A training step holds parameters, optimizer state
// and activations at once, and the sum is what has to fit. HBM is the tier it
// has to fit in, and the amount over is the whole reason another tier is being
// discussed at all.
module capacity_wall #(parameter int IT_ALL_FITS = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] param_gb, optimizer_gb, activation_gb, hbm_gb,
  output logic [15:0] need_gb, over_gb, hbm_pct, headroom_gb,
  output logic        fits,
  output logic [7:0]  n_evals, n_over,
  output logic        wall_err
);
  logic [31:0] n_q, p_q;
  logic [15:0] true_need, true_over;
  logic        truly_over;
  assign n_q = {16'd0, param_gb} + {16'd0, optimizer_gb} + {16'd0, activation_gb};
  assign true_need = (n_q > 32'd9999) ? 16'd9999 : n_q[15:0];
  assign need_gb = true_need;
  assign true_over = (true_need > hbm_gb) ? (true_need - hbm_gb) : 16'd0;
  assign over_gb = (IT_ALL_FITS != 0) ? 16'd0 : true_over;
  assign headroom_gb = (hbm_gb > true_need) ? (hbm_gb - true_need) : 16'd0;
  assign p_q = (hbm_gb == 16'd0) ? 32'd999
             : (({16'd0, true_need} * 32'd100) / {16'd0, hbm_gb});
  assign hbm_pct = (p_q > 32'd999) ? 16'd999 : p_q[15:0];
  assign fits = (over_gb == 16'd0);
  assign truly_over = (true_over != 16'd0);
  assign wall_err = evaluate && truly_over && fits;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_over <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_over) n_over <= n_over + 8'd1;
    end
  end
endmodule

One hundred and twenty plus two hundred and forty plus eighty against eighty gigabytes of fast tier is four hundred and forty needed, three hundred and sixty over, at five and a half times the tier.

The second case is what a fitting step looks like and it is worth keeping because it is the case the whole chapter is trying to reach. Sixty gigabytes in eighty leaves twenty of headroom at three quarters used, both builds agree, and neither alarms — a design that does not need another tier should not be told it does.

The fourth case is the boundary and it is the one that decides a configuration. Exactly eighty in eighty fits, reports no headroom, and reports a hundred percent — which is a step that will not survive a single increase in batch size, sequence length or parameter count, and the model says it fits because it does.

The fifth case is the shape that makes the section necessary. A requirement with no fast tier stated at all reports everything over and a saturated ratio, which is what a description that quotes a model size and never quotes the accelerator's memory amounts to.

The third case bounds the arithmetic honestly. A sum past the counter saturates rather than wrapping, and it still reports the overflow — a requirement too large to represent is still too large to fit, and the model must not report a small number because the big one did not fit in its own register.

The three terms behave differently and that is what makes the sum worth computing rather than estimating. The parameter count is fixed by the model and known before anything runs. The optimizer state is a multiple of it, and the multiple depends on which optimizer and which precision — it is arithmetic, not a measurement, but it is arithmetic somebody has to actually do. The activations are the term that moves, because they scale with batch size and sequence length and shrink if the implementation recomputes rather than stores them.

That asymmetry is why quoting the parameter count is not a rough approximation but a category error: the term that is easiest to quote is the one least likely to be the binding one, and the term that decides whether a configuration fits is the one that changes every time somebody adjusts a batch size.

It is also why the boundary case deserves more attention than a boundary case usually gets. A step that exactly fills the fast tier passes every check in this section and will fail the next time any of the three terms moves — and one of them moves whenever anybody tunes anything. A configuration reported as fitting at a hundred percent is a configuration with no margin, and the model says so by reporting zero headroom alongside the fit.

7. RTL 3 — A Tier Only Helps If The Traffic Tolerates Its Bandwidth

The third thing, and the one that decides whether the capacity was worth having.

Capacity is why a tier is added; bandwidth is whether the step can still run. Moving bytes to a larger, slower tier solves the wall in section 6 and creates a new problem in the same move, and the new problem is the one that shows up as a number nobody planned for.

The comparison is not close. The fast tier in an accelerator package and a link-attached tier differ by a large multiple, and the model reports that multiple because it is the fact that decides which traffic may be moved.

// RTL 3 - a tier only helps if the traffic it takes tolerates its bandwidth.
// Capacity is the reason a tier is added; bandwidth is what decides whether the
// step can still run when the traffic lands there.
module bandwidth_floor #(parameter int TIER_KEEPS_UP = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] needed_gbps, tier_gbps, hbm_gbps, offload_gb,
  output logic [15:0] shortfall_gbps, served_pct, hbm_ratio, drain_ms,
  output logic        keeps_up,
  output logic [7:0]  n_evals, n_short,
  output logic        bw_err
);
  logic [31:0] s_q, r_q, d_q;
  logic [15:0] true_short;
  logic        truly_short;
  assign true_short = (needed_gbps > tier_gbps) ? (needed_gbps - tier_gbps) : 16'd0;
  assign shortfall_gbps = (TIER_KEEPS_UP != 0) ? 16'd0 : true_short;
  assign s_q = (needed_gbps == 16'd0) ? 32'd100
             : (({16'd0, tier_gbps} * 32'd100) / {16'd0, needed_gbps});
  assign served_pct = (s_q > 32'd100) ? 16'd100 : s_q[15:0];
  // How many times faster the resident tier is than the added one.
  assign r_q = (tier_gbps == 16'd0) ? 32'd999
             : ({16'd0, hbm_gbps} / {16'd0, tier_gbps});
  assign hbm_ratio = (r_q > 32'd999) ? 16'd999 : r_q[15:0];
  // Time to move the offloaded bytes across the added tier once.
  assign d_q = (tier_gbps == 16'd0) ? 32'd9999
             : (({16'd0, offload_gb} * 32'd1000) / {16'd0, tier_gbps});
  assign drain_ms = (d_q > 32'd9999) ? 16'd9999 : d_q[15:0];
  assign keeps_up = (shortfall_gbps == 16'd0);
  assign truly_short = (true_short != 16'd0);
  assign bw_err = evaluate && truly_short && keeps_up;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_short <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_short) n_short <= n_short + 8'd1;
    end
  end
endmodule

Four hundred gigabytes per second wanted against sixty-four offered is three hundred and thirty-six short, sixteen percent served, a fifty-times gap, and three thousand one hundred and twenty-five milliseconds to move the offload once.

Figure 2 — the axis the capacity argument does not travel on. Both paths agree the bytes fit; they disagree about whether they can be reached in time. The fifty-times figure is the one to carry away, because it sets which traffic can move: something touched once per step can cross a tier fifty times slower, and something touched every microsecond cannot.

The second case is the honest positive and it is the whole design pattern. Fifty gigabytes per second wanted against sixty-four offered keeps up, both builds agree, and neither alarms — an offload sized to the tier's bandwidth is not a compromise, it is the correct use of the tier.

The boundary case is worth its place because it is where a capacity plan usually lands. Exactly matched keeps up, and a design sitting exactly on a tier's rate has no margin for the variance a real workload has.

The fourth case is the description that has not been done. A tier with no measured bandwidth reports everything short and saturates both derived figures, which is the state of most deployment descriptions at the point the decision is made.

The last case separates rate from volume, and it is a distinction the section needs. A very large offload saturates the drain time while the tier still keeps up with the rate — because keeping up is about gigabytes per second and the drain is about how long one pass takes, and confusing them produces a plan that is right about the steady state and wrong about the start-up.

The reason the fifty-times figure decides the design rather than merely describing it is that it converts directly into a reuse threshold. A byte that is touched once per step can afford to live fifty times further away, because the cost of fetching it is amortised over the whole step. A byte touched fifty times per step cannot, because the fetches do not amortise — they accumulate. The ratio is therefore not a quality score for the tier; it is the line between what may be moved and what may not, and section 8 is that line applied to an actual working set.

This is also where the capacity argument and the bandwidth argument stop being separable. Section 6 says something must move. This section says what may. A deployment that has answered the first without the second has decided to move bytes without deciding which ones, and the bytes that get moved are then chosen by whatever is easiest to move rather than by what tolerates the move — which is usually the opposite of the right answer, because the easiest thing to relocate is frequently the most frequently touched.

8. RTL 4 — Only The Hot Part Needs The Fast Tier

The fourth thing, and the one that makes the wall smaller than section 6 suggests.

A step does not touch all its bytes equally. Some are read repeatedly within a step and some are touched once, and only the first group needs the fast tier. The question is therefore not how large the working set is but how much of it is hot, and that is a measurable fraction rather than a property of the model size.

The hot bytes that do not fit are the expensive ones, because they are re-fetched on every reuse rather than once.

// RTL 4 - residency. Only the bytes a step touches repeatedly need the fast
// tier; the rest can live further away. The question a cluster actually asks is
// not how much memory it has but how much of it is hot.
module working_set_residency #(parameter int ALL_HOT_RESIDENT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] working_set_gb, hot_gb, hbm_gb, reuse_count,
  output logic [15:0] hot_resident, hot_evicted, hot_pct, refetch_gb,
  output logic        hot_fits,
  output logic [7:0]  n_evals, n_evicted,
  output logic        res_err
);
  logic [31:0] p_q, f_q;
  logic [15:0] hot_ok, true_resident, true_evicted;
  logic        truly_evicted;
  assign hot_ok = (hot_gb > working_set_gb) ? working_set_gb : hot_gb;
  assign true_resident = (hot_ok > hbm_gb) ? hbm_gb : hot_ok;
  assign hot_resident = (ALL_HOT_RESIDENT != 0) ? hot_ok : true_resident;
  assign true_evicted = hot_ok - true_resident;
  assign hot_evicted = (ALL_HOT_RESIDENT != 0) ? 16'd0 : true_evicted;
  assign p_q = (working_set_gb == 16'd0) ? 32'd0
             : (({16'd0, hot_ok} * 32'd100) / {16'd0, working_set_gb});
  assign hot_pct = p_q[15:0];
  // Hot bytes that did not fit are fetched again on every reuse.
  assign f_q = {16'd0, true_evicted} * {16'd0, reuse_count};
  assign refetch_gb = (f_q > 32'd9999) ? 16'd9999 : f_q[15:0];
  assign hot_fits = (hot_evicted == 16'd0) && (hot_ok != 16'd0);
  assign truly_evicted = (true_evicted != 16'd0);
  assign res_err = evaluate && truly_evicted && hot_fits;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_evicted <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_evicted) n_evicted <= n_evicted + 8'd1;
    end
  end
endmodule

Eight hundred gigabytes with three hundred hot against eighty resident is two hundred and twenty evicted and eight hundred and eighty refetched across four reuses — thirty-seven percent of the set hot.

The second case is the configuration the whole technique aims at. Sixty gigabytes hot in eighty of fast tier is wholly resident, nothing is refetched, and the other seven hundred and forty gigabytes of the working set can live anywhere — which is the honest, useful version of "they use CXL for memory" and takes four numbers to say.

The fourth case is the clamp and it is a modelling guard rather than a workload. More hot bytes claimed than the working set contains clamps to the set, because a subset cannot be larger than the set it is drawn from.

The fifth case is the far end and it is a real failure. A wholly hot set with no fast tier refetches a saturated volume, which is the shape of a workload that has no reuse structure to exploit — and the correct response is a different partitioning rather than a different tier.

The last case bounds it: an uncharacterised workload, nothing measured, reports nothing resident and nothing evicted and declines to call that a fit. An empty hot set is not a fitting one, and a deployment that has not measured its reuse has not earned the offload it is planning.

The reuse count is the multiplier that makes an eviction expensive rather than merely inconvenient, and it is the input most often assumed to be one. A hot byte that does not fit is not fetched once; it is fetched on every reuse, so the traffic the eviction generates is the evicted volume times the reuse count. Two hundred and twenty gigabytes evicted becomes eight hundred and eighty of traffic at a reuse of four, and that traffic lands on the tier section 7 just measured at a fiftieth of the rate.

The two sections compound, which is the practical point and the reason they sit next to each other. An eviction is a capacity event and a bandwidth event at the same time: it happens because the fast tier was too small, and it costs what it costs because the tier it falls back to is slow. Either fact alone is survivable; together they are what turns a working configuration into one that is slower than it was before the capacity was added.

And the hot fraction is a design choice more than a measurement, which is the part that makes the section actionable. Offload techniques that move optimizer state out of the fast tier are, precisely, decisions about what counts as hot. A workload does not arrive with a hot set; an implementation produces one, and a different partitioning produces a different one. Measuring the fraction tells you what the current implementation chose, and the useful next question is whether it chose well.

9. RTL 5 — An Attach Inside A Node Is Not The Fabric Between Nodes

The fifth thing, and the one that is most often quietly wrong in a description that is otherwise careful.

Scale-up and scale-out are different connections. Inside a node the links are short, dense, and known at design time; between nodes they are a network with switches, congestion and a different failure model. A memory attach lives on the first side of that line, and a claim that places it on the second has misdescribed every link it covers.

That is not pedantry: the two sides have different reach, different latency and different sharing semantics, and a capacity plan that crosses the line without noticing has planned for bandwidth it will not get.

// RTL 5 - scale-up is not scale-out. An attach inside a node and a fabric
// between nodes are different connections with different reach, and a claim
// that puts one where the other belongs has misplaced every link it describes.
module scale_boundary #(parameter int ONE_FABRIC_THROUGHOUT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] links_total, links_in_node, attach_links, hop_cost,
  output logic [15:0] links_between, misplaced, in_scope, misplace_cost,
  output logic        boundary_stated,
  output logic [7:0]  n_evals, n_misplaced,
  output logic        scope_err
);
  logic [31:0] c_q;
  logic [15:0] node_ok, true_between, true_misplaced;
  logic        truly_misplaced;
  assign node_ok = (links_in_node > links_total) ? links_total : links_in_node;
  assign true_between = links_total - node_ok;
  assign links_between = true_between;
  // An attach is a node-scope connection. Any of it claimed for the links
  // between nodes is claimed outside its reach.
  assign in_scope = (attach_links > node_ok) ? node_ok : attach_links;
  assign true_misplaced = attach_links - in_scope;
  assign misplaced = (ONE_FABRIC_THROUGHOUT != 0) ? 16'd0 : true_misplaced;
  assign c_q = {16'd0, true_misplaced} * {16'd0, hop_cost};
  assign misplace_cost = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign boundary_stated = (misplaced == 16'd0) && (links_total != 16'd0);
  assign truly_misplaced = (true_misplaced != 16'd0);
  assign scope_err = evaluate && truly_misplaced && boundary_stated;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_misplaced <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_misplaced) n_misplaced <= n_misplaced + 8'd1;
    end
  end
endmodule

Sixty-four links with eight inside a node and twenty claimed for the attach leaves fifty-six between nodes, eight in scope, twelve claimed outside its reach, at three hundred and sixty units.

The second case is the description done correctly. The attach claimed only inside the node misplaces nothing, both builds agree, and the fifty-six links between nodes are somebody else's problem — which is the right answer and is what a correct case study reads like.

The boundary case is the one to watch for in review. An attach exactly filling the node's link budget misplaces nothing and leaves no room, which is a design where every in-node link is committed and the next device added has nowhere to go.

The fifth case is the shape of the error at full size. The whole fabric claimed for the attach, with no links inside the node at all, misplaces every one of them and saturates the cost — which is what "the cluster is built on CXL" means when nobody has drawn the node boundary.

The degenerate case bounds it: a topology nobody enumerated states no boundary, because there is nothing to state it about.

The error is almost always a vocabulary problem rather than an engineering one, which is why it survives review so reliably. "Node" is used for a chassis, a rack, a pod and a failure domain by different people in the same organisation, and an attach that is correctly described as node-scope by somebody meaning a chassis is incorrectly described by somebody who reads "node" as a rack. Every number in the description stays right and the reach silently multiplies, and nobody is wrong at the moment they say it.

The consequence is specific and it is a bandwidth consequence. The connections inside a chassis are short and dense; a rack-scale connection crosses a switch. The plan assumed the first and gets the second, so the bandwidth is lower, the latency is higher and the concurrency needed to sustain either goes up — which lands straight on the request-pool problem 9.6 describes, where a pool sized for the shorter round trip throttles the longer one and reports no error at all.

The defence is to count links rather than to argue about words. A boundary that has been drawn on a diagram with a number of links on each side cannot be reinterpreted later, and the model's in-scope count is exactly that diagram reduced to one number.

10. RTL 6 — An Offload Pays Only If The Time It Buys Is Worth More

The sixth thing, and the one that turns a capability into a decision.

Moving capacity to a cheaper tier buys accelerator time, and accelerator time in a training cluster has a price everybody already knows. The tier has a price too, and the comparison is a subtraction rather than an argument.

The section exists because the capability is usually presented without either number, and a capability with no price attached always looks worth having.

// RTL 6 - offload economics. Moving capacity to a cheaper tier buys accelerator
// time only if the time it buys is worth more than the tier costs, and both
// halves are prices somebody already knows.
module offload_economics #(parameter int OFFLOAD_IS_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] offload_gb, tier_cost_per_gb, hours_saved, hour_cost,
  output logic [15:0] tier_cost, time_value, net_gain, net_loss,
  output logic        worth_it,
  output logic [7:0]  n_evals, n_negative,
  output logic        econ_err
);
  logic [31:0] c_q, v_q;
  logic [15:0] true_cost, true_value;
  logic        truly_negative;
  assign c_q = {16'd0, offload_gb} * {16'd0, tier_cost_per_gb};
  assign true_cost = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign tier_cost = (OFFLOAD_IS_FREE != 0) ? 16'd0 : true_cost;
  assign v_q = {16'd0, hours_saved} * {16'd0, hour_cost};
  assign true_value = (v_q > 32'd9999) ? 16'd9999 : v_q[15:0];
  assign time_value = true_value;
  // Gain and loss are reported against the cost this build admits to, so the
  // free-offload view stays internally coherent: it cannot publish a zero cost
  // and a loss in the same breath. The truth below is computed from the real
  // cost regardless, which is what lets the model detect its own weak build.
  assign net_gain = (true_value > tier_cost) ? (true_value - tier_cost) : 16'd0;
  assign net_loss = (tier_cost > true_value) ? (tier_cost - true_value) : 16'd0;
  assign worth_it = (OFFLOAD_IS_FREE != 0) ? (true_value != 16'd0)
                                           : (true_value > true_cost);
  assign truly_negative = (true_cost >= true_value) && (true_value != 16'd0);
  assign econ_err = evaluate && truly_negative && worth_it;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_negative <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_negative) n_negative <= n_negative + 8'd1;
    end
  end
endmodule

Four hundred gigabytes at twelve against thirty hours at a hundred is four thousand eight hundred of tier cost against three thousand of time value — eighteen hundred of net loss.

The second case is the genuine win and it is not hard to reach. A thousand of tier against four thousand of time is three thousand of net gain, both builds call it worth it, and neither alarms — an offload that buys more than it costs is exactly what the technique is for.

The boundary case is the one that decides marginal deployments. Cost exactly equal to value is reported as not worth it, because breaking even on a change that adds a tier, a failure mode and an integration cost is a loss once the things the model does not price are counted.

The last case is the honest zero. Nothing offloaded and nothing measured is not a win, and both builds agree — an uncosted offload has not been shown to pay, which is different from having been shown not to.

The two prices in this model are unusual in that both genuinely exist. Almost every cost model in this track asks for a number somebody has to estimate; this one asks for a memory price and an accelerator-hour price, and a training organisation knows both to a good precision because it is buying both continuously. The subtraction is therefore not an approximation of a decision — it is the decision, and the reason it is skipped is that neither number lives in the same system as the architecture discussion.

What the model deliberately does not price is the part that makes the boundary case strict. Adding a tier adds an integration, a failure mode, a software path and a thing to debug at three in the morning. None of that appears in the arithmetic, which is exactly why breaking even is reported as not worth it: a change that nets to zero on the two quantities you measured is a loss on the ones you did not. A deployment should want a margin, and the model's job is to say how large the margin currently is rather than to decide how large it should be.

One structural note, because it is this chapter's own correction. The free-offload build reports a zero tier cost, and it therefore reports its gain and loss against that zero rather than against the real cost. That is deliberate: a view that denies the cost must be internally coherent, or it publishes a zero cost and a loss in the same breath and no reader would believe it. The truth the model checks itself against is computed from the real cost regardless, which is what lets it detect its own weak build.

11. RTL 7 — An Accelerator Waiting On A Slower Tier Is Idle

The seventh thing, and the number that decides whether section 10's arithmetic was optimistic.

Time spent waiting on a tier is time the accelerator is not computing. In a cluster that idleness is not one device's problem: the same wait happens on every device, on every step, and the total is what the deployment actually paid.

Utilisation is the number the industry already tracks, which makes this the easiest of the chapter's quantities to obtain and the one most often left out of an offload proposal.

// RTL 7 - the stall cost. An accelerator waiting on a slower tier is idle, and
// in a cluster that idleness is multiplied by every device and every step. It
// is the number that decides whether an offload was a good idea.
module stall_cost #(parameter int NO_STALL = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] step_ms, stall_ms, steps, devices,
  output logic [15:0] util_pct, lost_ms, busy_ms, device_hours,
  output logic        util_ok,
  output logic [7:0]  n_evals, n_stalled,
  output logic        stall_err
);
  logic [31:0] u_q, l_q, h_q;
  logic [15:0] stall_ok, true_busy;
  logic        truly_stalled;
  assign stall_ok = (stall_ms > step_ms) ? step_ms : stall_ms;
  assign true_busy = step_ms - stall_ok;
  assign busy_ms = (NO_STALL != 0) ? step_ms : true_busy;
  assign u_q = (step_ms == 16'd0) ? 32'd100
             : (({16'd0, true_busy} * 32'd100) / {16'd0, step_ms});
  assign util_pct = (NO_STALL != 0) ? 16'd100 : u_q[15:0];
  assign l_q = {16'd0, stall_ok} * {16'd0, steps};
  assign lost_ms = (l_q > 32'd9999) ? 16'd9999 : l_q[15:0];
  // The same idleness happens on every device in the cluster at once.
  assign h_q = {16'd0, lost_ms} * {16'd0, devices};
  assign device_hours = (h_q > 32'd9999) ? 16'd9999 : h_q[15:0];
  assign util_ok = (util_pct >= 16'd90);
  assign truly_stalled = (stall_ok != 16'd0);
  assign stall_err = evaluate && truly_stalled && (util_pct == 16'd100);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_stalled <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_stalled) n_stalled <= n_stalled + 8'd1;
    end
  end
endmodule

Eighteen milliseconds of stall in a hundred-millisecond step, across fifty steps and eight devices, is eighty-two percent utilisation, nine hundred milliseconds lost per device and seven thousand two hundred across the cluster.

The second case is the configuration to aim for and the one the whole section is measured against. No stall at all is full utilisation with nothing lost, both builds agree, and neither alarms.

The boundary case is the one that decides an acceptance criterion. Exactly at the ninety percent bar passes, because the bar is inclusive — and a deployment sitting exactly on its acceptance threshold has no margin, which the next case makes concrete: one millisecond past the bar and it fails.

The last measured case is the multiplier that makes this a cluster problem rather than a device one. A small per-device loss across a large cluster saturates the total while every individual device is still comfortably above the bar — which is how a deployment passes its per-device acceptance test and still loses a large amount of aggregate time.

The degenerate case bounds it: an unmeasured step reports full utilisation, which is the absence of a measurement rather than the presence of a good one.

The device multiplier is what makes this a cluster chapter rather than a device one, and it is worth stating as a shape rather than a number. Utilisation is a per-device ratio and it is bounded: it cannot get worse than zero, and a few percent always sounds like a few percent. The aggregate loss is a product, and products of large numbers are not intuitive. The same eighteen milliseconds that reads as "eighty-two percent, acceptable" on one device is a quantity of accelerator time across a fleet that would never be approved if it were requested directly.

That mismatch is why acceptance criteria written per device pass deployments that lose badly. The criterion is not wrong — a device above its bar is genuinely fine — but the quantity the organisation actually cares about is the product, and no per-device threshold bounds a product. The model reports both for that reason, and the case with a small per-device loss across a large cluster exists specifically to show them disagreeing.

And the direction of the error is always the same. A per-device figure understates and never overstates, because the multiplier is greater than one. A deployment that checked only the per-device number has an optimistic answer, which is the worst available property for a number used to approve a change.

12. RTL 8 — A Capacity-Bound Stage Gains; A Bandwidth-Bound One Does Not

The eighth thing, and the one that says where in a pipeline the attach belongs.

A training pipeline is not one workload. Data loading, preprocessing, the forward pass, the backward pass, optimizer update, checkpointing — each has a different limiting resource, and a larger slower tier helps exactly the ones limited by capacity. For the ones limited by bandwidth the same move makes things worse, which means the answer is per stage rather than per cluster.

// RTL 8 - where the attach actually wins. A capacity-bound stage gains from a
// large slow tier; a bandwidth-bound one does not, and counting which stages
// are which is the difference between a deployment and a slogan.
module where_it_wins #(parameter int WINS_EVERYWHERE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] stages, capacity_bound, bandwidth_bound, gain_each,
  output logic [15:0] helped, unhelped, total_gain, helped_pct,
  output logic        helps_all,
  output logic [7:0]  n_evals, n_unhelped,
  output logic        win_err
);
  logic [31:0] g_q, p_q;
  logic [15:0] cap_ok, bw_ok, true_helped, true_unhelped;
  logic        truly_unhelped;
  assign cap_ok = (capacity_bound > stages) ? stages : capacity_bound;
  assign bw_ok  = (bandwidth_bound > stages) ? stages : bandwidth_bound;
  // Only a capacity-bound stage gains; a bandwidth-bound one is made worse by
  // the same move, so it is counted out rather than counted in.
  assign true_helped = (cap_ok > bw_ok) ? (cap_ok - bw_ok) : 16'd0;
  assign helped = (WINS_EVERYWHERE != 0) ? stages : true_helped;
  assign true_unhelped = stages - true_helped;
  assign unhelped = (WINS_EVERYWHERE != 0) ? 16'd0 : true_unhelped;
  assign g_q = {16'd0, true_helped} * {16'd0, gain_each};
  assign total_gain = (g_q > 32'd9999) ? 16'd9999 : g_q[15:0];
  assign p_q = (stages == 16'd0) ? 32'd100
             : (({16'd0, true_helped} * 32'd100) / {16'd0, stages});
  assign helped_pct = p_q[15:0];
  assign helps_all = (unhelped == 16'd0) && (stages != 16'd0);
  assign truly_unhelped = (true_unhelped != 16'd0);
  assign win_err = evaluate && truly_unhelped && helps_all;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_unhelped <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_unhelped) n_unhelped <= n_unhelped + 8'd1;
    end
  end
endmodule

Twelve stages with seven capacity-bound and three bandwidth-bound is four helped, eight not, a hundred and sixty units of gain — a third of the pipeline.

The second case is the strongest version of the deployment argument. Every stage capacity-bound is helped entirely, at four hundred and eighty units of gain, both builds agree, and neither alarms — a pipeline that is capacity-limited throughout is exactly what this technology is for and should be described as such.

The boundary case is the one that decides a marginal pipeline. Equally capacity- and bandwidth-bound nets to nothing helped, which is the right answer: the gains and the harms cancel, and a deployment at that balance has not been shown to benefit.

The fourth case is the direction nobody proposes. A mostly bandwidth-bound pipeline gains nothing at all, and the memory-answer view still calls it a sweep — which is the most expensive version of this mistake, because the capacity really was added and really is unused.

The degenerate case bounds it: an unwritten pipeline helps nothing, because nobody enumerated the stages the claim is about.

The stages differ more than the word "pipeline" suggests, which is why per-stage classification is worth the effort. Data loading and preprocessing are throughput problems over large volumes with little reuse. The forward and backward passes are bandwidth problems with intense reuse. The optimizer update touches a large state once. Checkpointing writes a large volume periodically and tolerates almost any latency. Those are four different answers to the same question, and a cluster-wide decision has to be wrong about at least three of them.

The netting in the model is the part to read carefully. Capacity-bound stages gain and bandwidth-bound stages are harmed by the same move, so the model subtracts rather than adding — which is why an equally divided pipeline nets to zero rather than to half. That is the honest arithmetic: the harm is not a smaller benefit, it is a benefit with the opposite sign, and a deployment that counts only the stages that improved has counted half of its own result.

It also explains the unattributable-small-gain failure that section 23 describes. A cluster-wide rollout onto a mixed pipeline produces a real gain on some stages and a real loss on others, and the aggregate is a small number with no obvious cause. The technique then acquires a reputation for being marginal, when what actually happened is that it was applied to the stages it harms as well as the ones it helps.

13. RTL 9 — An Added Tier Does Not Lower The Fast-Tier Floor

The ninth thing, and the one that most often survives a whole design review.

Whatever the step must touch at full rate still has to be in the fast tier. Adding capacity further away changes what the cluster can hold; it does not change what it must hold at speed. Those are different quantities, and a plan that subtracts the second from the first has bought capacity that cannot reach the problem.

// RTL 9 - an added tier does not lower the floor. Whatever a step must touch at
// full rate still has to be in the fast tier, so capacity added further away
// changes what a cluster can hold and not what it must hold.
module floor_unchanged #(parameter int TIER_REPLACES_FAST = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] fast_floor_gb, fast_present_gb, added_gb, per_gb_cost,
  output logic [15:0] still_short, total_capacity, wasted_spend, floor_pct,
  output logic        floor_met,
  output logic [7:0]  n_evals, n_short,
  output logic        floor_err
);
  logic [31:0] t_q, w_q, p_q;
  logic [15:0] true_short;
  logic        truly_short;
  assign true_short = (fast_floor_gb > fast_present_gb)
                    ? (fast_floor_gb - fast_present_gb) : 16'd0;
  // The weak build lets the added tier count against the fast-tier floor.
  assign still_short = (TIER_REPLACES_FAST != 0)
                     ? ((true_short > added_gb) ? (true_short - added_gb) : 16'd0)
                     : true_short;
  assign t_q = {16'd0, fast_present_gb} + {16'd0, added_gb};
  assign total_capacity = (t_q > 32'd9999) ? 16'd9999 : t_q[15:0];
  // Capacity bought to fix a floor it cannot reach is spend with no effect.
  assign w_q = (true_short != 16'd0)
             ? ({16'd0, added_gb} * {16'd0, per_gb_cost}) : 32'd0;
  assign wasted_spend = (w_q > 32'd9999) ? 16'd9999 : w_q[15:0];
  assign p_q = (fast_floor_gb == 16'd0) ? 32'd100
             : (({16'd0, fast_present_gb} * 32'd100) / {16'd0, fast_floor_gb});
  assign floor_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign floor_met = (still_short == 16'd0);
  assign truly_short = (true_short != 16'd0);
  assign floor_err = evaluate && truly_short && floor_met;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_short <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_short) n_short <= n_short + 8'd1;
    end
  end
endmodule

A three-hundred-and-twenty-gigabyte floor with two hundred and forty present and five hundred and twelve added further away is still eighty short, at three thousand and seventy-two units spent on capacity that cannot reach it.

The second case is the floor genuinely met and it is the answer the plan was aiming for. Three hundred and twenty present against a three-hundred-and-twenty floor is short by nothing, wastes nothing, and both builds agree — and note that the five hundred and twelve added is still there and still useful, just not for this.

The third case is the sharpest one in the model and it took a moment to see. An added tier too small to cover the shortfall even on the weak reading leaves both builds reporting a failure — the measured build says eighty short, the weak build says thirty short, and neither alarms. The two disagree about the amount while agreeing about the verdict, which is the most dangerous kind of agreement, because a reviewer who checks only the verdict will not notice that one of the two numbers is meaningless.

The fourth case is the shape at full cost. Five hundred gigabytes added against a four-hundred-gigabyte floor with no fast tier present at all saturates the wasted spend and leaves the entire floor short — capacity bought in quantity to fix a problem it is on the wrong side of.

The degenerate case bounds it: an unstated floor is trivially met, which is what a plan looks like before anybody has said what the step must touch at speed.

The confusion this section is about is a units confusion wearing a capacity costume. Total capacity and fast-tier capacity are both measured in gigabytes, which makes them look like the same quantity at different scales — so adding one to the other is arithmetically valid and physically meaningless. The floor is not a quantity of bytes; it is a quantity of bytes reachable at a rate, and the added tier fails the second half of that description no matter how large the first half is.

The wasted-spend figure exists to make the failure visible in a currency rather than in a contradiction. A plan that is eighty gigabytes short after adding five hundred and twelve has not merely made an error of reasoning; it has spent three thousand units on capacity that cannot reach the problem, and that number is the one that gets a plan revisited. An architectural objection gets argued with; a line item gets checked.

And the third case is the one to carry into a review. When both builds report a failure but disagree about its size, the verdict agrees and the number does not — so a reviewer who reads the verdict sees consensus and stops. That is the most dangerous state in the whole chapter, because every visible signal says the two readings concur, and the only way to catch it is to compare the amounts rather than the conclusions.

14. RTL 10 — A Training-Cluster Case Study Assembled

Nine sections of inputs. This one puts them together and shows the confident sentence for what it is.

// RTL 10 - a training-cluster case study assembled. Nine sections of inputs,
// one summary. "They use CXL for memory" is bit 0: true of something, and one
// sixth of a case study.
module cluster_case_signoff #(parameter int MEMORY_IS_THE_ANSWER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic        attach_named, tier_named, wall_computed,
  input  logic        bandwidth_checked, stall_computed, scope_stated,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_evals, n_sound, n_claimed,
  output logic        signoff_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~attach_named;
  assign fail_mask[1] = ~tier_named;
  assign fail_mask[2] = ~wall_computed;
  assign fail_mask[3] = ~bandwidth_checked;
  assign fail_mask[4] = ~stall_computed;
  assign fail_mask[5] = ~scope_stated;
  assign conditions_met = {15'd0, attach_named} + {15'd0, tier_named}
                        + {15'd0, wall_computed} + {15'd0, bandwidth_checked}
                        + {15'd0, stall_computed} + {15'd0, scope_stated};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp: conditions_met sums six one-bit values, so the quotient cannot
  // exceed a hundred and a ceiling would be unreachable code.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  // The "it uses CXL for memory" view reads bit 0 and stops.
  assign claimed = (MEMORY_IS_THE_ANSWER != 0) ? attach_named : truly_sound;
  assign sound = claimed;
  assign signoff_err = evaluate && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When the attach has been named and any one of the other five fails, the assembled model reports that the case study is not sound and the memory-answer view reports a case study.

Across the eight evaluations, the assembled model calls one case study sound and the memory-answer view calls seven of eight a case study.

The bit order is by how much of the deployment each condition carries. Bit 1 is first among the five because every other number in the chapter is a number about a tier, and until the tier is named none of them has a subject. Bits 2 and 3 are the capacity and the bandwidth, which is the whole trade. Bit 4 is what the trade cost and bit 5 is where it applies.

"They use CXL for memory" is bit 0, and it is the weakest of the three weak definitions this batch has built. 28.3's points at a trade-off that does not exist and 28.4's is a true claim standing in for an argument. This one is a true claim that names a category containing its own opposite — the fast tier is memory too, and so is storage, so the sentence is satisfied by every possible answer including the ones it was meant to rule out.

The five other bits fail independently. A tier can be named by somebody who never summed the three resident quantities. The wall can be computed by somebody who never checked what the moved traffic needs. The bandwidth can be checked without anybody costing the wait it produces. And the scale boundary is the one that fails last and quietest, because a description can be right about every number in it and still have placed them on the wrong side of the node.

The terminal on the right is labelled "a true sentence" deliberately. Every path that reaches it produces a claim nobody can contradict — the cluster does use an attach, and what it attaches is memory. The failure is not that the description is wrong; it is that it is unfalsifiable, and an unfalsifiable description cannot be used to decide anything, which is what a case study is for.

15. Quantitative Reasoning

Three tiers of four unnamed, with the named tier carrying six percent of the step and a quarter of the structure pinned down.

Three hundred and sixty gigabytes over an eighty-gigabyte fast tier, from a sum of one hundred and twenty, two hundred and forty and eighty — five and a half times the tier.

Three hundred and thirty-six gigabytes per second short, sixteen percent served, and a fifty-times gap between the two tiers.

Two hundred and twenty gigabytes of hot data evicted and eight hundred and eighty refetched across four reuses, on a set thirty-seven percent hot.

Twelve links of twenty claimed outside the attach's reach, at three hundred and sixty units.

Eighteen hundred units of net loss on an offload costing four thousand eight hundred and buying three thousand.

Eighty-two percent utilisation and seven thousand two hundred milliseconds lost across eight devices, from eighteen milliseconds of stall per step.

Eight stages of twelve unhelped, with four helped at a hundred and sixty units of gain.

Eighty gigabytes still short of the fast-tier floor after five hundred and twelve were added further away, at three thousand and seventy-two units spent.

One case study of eight sound; the memory-answer view counts seven. The assembled model's summary, and the chapter's.

16. Assertions

The testbenches carry 461 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported fifteen gaps on its first run and they were closed before the campaign — every one of them a weak-build output that is a shared fact rather than a parameter-selected one, and therefore the kind that is easy to assume is covered by its partner.

Both builds are asserted on every degenerate case. No tier structure, nothing required and nothing present, nothing measured at all, no working set, no topology enumerated, nothing offloaded, no step measured, no stages enumerated, and no floor stated.

Every clamp that an input can reach is driven past its limit exactly once. More tiers named than exist, more bytes on a tier than the step holds, a requirement past its counter, a ratio past its ceiling, a served percentage past a hundred, an absent tier's ratio and drain time, more hot bytes than the working set holds, a refetch volume past its counter, more in-node links than exist, a misplacement cost past its counter, a tier cost and a time value past theirs, a lost time and a cluster total past theirs, a gain past its counter, a total capacity past its counter, a wasted spend past its counter, and a floor percentage past a hundred.

Every threshold is asserted on both sides of its boundary. The ninety-percent utilisation bar is asserted at exactly ninety and at eighty-nine; the capacity wall at exactly full; the bandwidth floor at exactly matched; the offload at exactly break-even; the pipeline at exactly balanced.

Every error output is checked in both directions in every case. Section 5's second and sixth cases, section 6's second, fourth and seventh, section 7's second, third and seventh, section 8's second, third and seventh, section 9's second, third and sixth, section 10's second, fifth and seventh, section 11's second and seventh, section 12's second, sixth and seventh, and section 13's second, third and seventh exist to assert the quiet half. Each is a case where the memory-answer view happens to be right, and a model that alarmed on them would be unusable.

17. Mutation Testing

132 mutations, 132 killed. Sixty-four against the first testbench, sixty-eight against the second.

One mutation was withdrawn as equivalent rather than counted, and it is this chapter's finding.

Loosening section 6's overflow test from > to >= survived. The reflex is to treat a survivor as a stimulus gap and add a case at the equality — but the stimulus already had one, and adding more would never have killed it. The expression is (a > b) ? (a - b) : 0, and at the equality both branches evaluate to zero. The mutation is equivalent by construction, and no test can distinguish it.

The rule that follows is worth stating because the same shape appears eleven times in this chapter alone: a saturating difference has no observable boundary, so "boundary loosened" is not a valid mutation family for one. The boundary is real and it is observable — but only through a comparison output derived from the difference, like section 6's fits, which the stimulus does assert at exactly eighty in eighty. The mutation has to be injected where the boundary is read, not where it is computed.

That is the fourth distinct masking relationship this batch has found, after a clamp behind a minimum, a clamp behind an outer clamp, and a clamp behind itself. This one is a boundary behind a saturation, and it is the first that cannot be fixed by better stimulus — only by mutating somewhere else.

One defect was caught by reading rather than by the campaign, and it belongs in the same paragraph. Section 10's model reported its gain and loss against the real cost while publishing a zero cost, so the free-offload build claimed the tier was free and reported eighteen hundred of loss at the same time. A weak build that contradicts itself is not a weak build; it is a broken one, and a reader would have dismissed it rather than been misled by it. Folding the parameter selection through the gain and loss made it coherent, and the truth it checks itself against is still computed from the real cost — which is what preserves the model's ability to detect its own weak build.

18. Verification Strategy

Ask which tier. Section 5. The word "memory" is satisfied by every tier in the cluster including the one the attach was meant to relieve.

Sum the three resident quantities. Section 6. Parameters, optimizer state, activations — the parameter count alone is the smallest of them.

Check what the moved traffic needs against what the tier offers. Section 7. Capacity is why it was added; bandwidth is whether the step still runs.

Measure the hot fraction, not the working-set size. Section 8.

Draw the node boundary before counting links. Section 9.

Price the offload against the accelerator time it buys. Section 10. Both numbers already exist.

Compute the utilisation cost and multiply it by the cluster. Section 11. The per-device figure passes tests the aggregate does not.

Classify each pipeline stage as capacity- or bandwidth-bound. Section 12. The answer is per stage.

State the fast-tier floor separately from the total capacity. Section 13. Adding capacity further away does not lower it.

19. Synthesis and Implementation Reality

The fast tier's bandwidth advantage comes from being in the package, which is 28.3's subject arriving in a deployment: the shoreline and the stacking are what buy the rate, and a link-attached tier cannot have them however good its protocol is.

A link-attached tier's latency is a round trip across a link, which is 28.1's and 9.6's concurrency argument: sustaining bandwidth across a longer round trip needs proportionally more requests in flight, and a request pool sized for local latency throttles the expanded traffic while reporting no error.

Offload techniques that move optimizer state and parameters out of the fast tier are well established in software, and they are the mechanism by which section 8's hot-fraction argument is actually exploited — the technique decides what counts as hot, which is why the fraction is a design choice rather than a measurement of the model.

Vendor fabrics between accelerators and an open attach to memory coexist in large systems, which is 28.4 section 13 arriving as a topology: the scale-up domain and the memory attach are different connections with different requirements, and section 9 is the line between them.

And checkpointing is the traffic most often forgotten in a capacity plan. It is periodic, large, and tolerant of a slow tier — which makes it the best-fitting use of an attached tier in the whole pipeline and the one least likely to appear in the sentence that describes the deployment.

20. Silicon Observability

Free, and from a specification sheet. The fast tier's capacity and bandwidth, and the link's. Sections 6 and 7.

Free, and from a model definition. The parameter count, and from it the optimizer state. Section 6.

Cheap, and from a profiler. Step time and stall time. Section 11 — and this is the number a cluster already tracks, which makes it the cheapest real quantity in the chapter.

Cheap, and from a topology diagram. The node boundary and the link counts. Section 9.

Moderate. Activation memory, which depends on batch size, sequence length and what the implementation chooses to recompute rather than store. Section 6's third term is the one that moves.

Moderate, and requires instrumentation. The hot fraction. Section 8 needs reuse distance rather than a byte count, and nothing reports it by default.

Expensive. Per-stage classification as capacity- or bandwidth-bound. Section 12 needs each stage profiled against both limits, and a stage can change class with batch size.

Expensive, and usually estimated. The tier cost per gigabyte and the accelerator hour cost. Section 10's two prices exist but live in a finance system rather than a profiler.

21. Debug Lab

A cluster has added an attached memory tier and training is slower.

Step 1 — ask which tier the traffic went to and which tier it came from. Section 5. If the answer is "memory", nothing below this step can be done.

Step 2 — check utilisation before and after. Section 11. This is already instrumented and it localises the problem in one reading: if utilisation fell, the stall is the cause and step 3 applies; if it did not, the problem is elsewhere.

Step 3 — compare what the moved traffic needs against what the tier offers. Section 7. A shortfall here explains the whole symptom.

Step 4 — check whether what moved was hot. Section 8. Moving hot bytes is the single most common cause, and the refetch volume makes it obvious.

Step 5 — check the stage. Section 12. If the stage that slowed was bandwidth-bound, the offload was applied to the wrong part of the pipeline.

Step 6 — check the fast-tier floor. Section 13. If the plan subtracted the added capacity from the floor, the fast tier is still short and no amount of attached capacity will fix it.

Steps 1 and 2 take minutes and between them localise most instances.

22. Design Review

Which tier does this attach serve, and what fraction of a step's bytes land on it?

What is the sum of parameters, optimizer state and activations, and what is the fast tier?

What bandwidth does the traffic being moved require, and what does the tier offer?

What fraction of the working set is hot, and does the hot part fit?

Where is the node boundary, and are these links on the right side of it?

What does the offload cost, and what accelerator time does it buy?

What did utilisation do, and what is that across the cluster?

Which pipeline stages are capacity-bound and which are bandwidth-bound?

What is the fast-tier floor, stated separately from total capacity?

23. How This Appears In Real Engineering

The failure is a description that is true and undecidable, and it propagates because nobody can object to it.

The most common shape is section 5 straight through. A deployment is described as using CXL for memory, the description is repeated in a design document, a roadmap and a conference talk, and at no point does anyone ask which tier — so when a later team plans against it they plan against a sentence rather than a number.

The second is section 13 and it is the most expensive. A capacity plan subtracts the attached tier from the fast-tier requirement. The total looks sufficient, the parts are ordered, and the step still does not fit — because the thing that had to be fast is still not fast, and the capacity that was bought is on the wrong side of the problem.

The third is section 12. An offload is applied cluster-wide because it worked on one stage. The capacity-bound stages improve, the bandwidth-bound ones get worse, and the aggregate result is a small gain that nobody can attribute, which makes the technique look marginal when it was simply misapplied.

The fourth is section 11 at cluster scale. A per-device acceptance test passes — every device is above its utilisation bar — while the aggregate loss across thousands of devices is large. The test was correct and the wrong quantity was tested.

The fifth is section 9 and it is the quietest. A description places an attach on the wrong side of the node boundary, usually by describing a rack as though it were a node. Every number in the description is right; the reach is wrong; and the error surfaces as bandwidth that was planned for and does not arrive.

The pattern is that this chapter's inputs are all cheap and its usual descriptions contain none of them, and the entire method is asking for six numbers that already exist.

24. Common Misconceptions

"They use CXL for memory." Which tier? Section 5.

"The model is 120 GB so we need 120 GB." Parameters are the smallest of three resident quantities. Section 6.

"It is memory, so it will be fine." At a fiftieth of the bandwidth. Section 7.

"The working set is too big for HBM." Only the hot part needs to be there. Section 8.

"The cluster is built on CXL." Inside a node, or between them? Section 9.

"Cheaper memory is obviously worth it." Against the accelerator time it buys. Section 10.

"Utilisation only dropped a few percent." Multiplied by every device and every step. Section 11.

"It helped, so roll it out." To the capacity-bound stages. Section 12.

"We added 512 GB, so we have enough." Not in the tier that had to have it. Section 13.

25. Interview Reasoning

"Where does CXL fit in an AI training cluster?" The honest answer starts by refusing the question's premise: a cluster has several memory tiers and they differ by a large multiple, so the answer has to name one. An attached tier is a capacity tier — it is large, it is reached across a link, and its bandwidth is a fraction of what is in the accelerator package. So it fits where capacity is the limit and bandwidth is not: parameters and optimizer state that are touched once per step rather than continuously, checkpoint staging, and preprocessing — and not the activations and weights the inner loop is reading at full rate.

"How would you decide whether to do it?" Six numbers, all of which already exist. The sum of parameters, optimizer state and activations against the fast tier, which is the wall. What the traffic being moved actually needs against what the tier offers. The hot fraction of the working set, because only the hot part has to be resident. What the offload costs against the accelerator time it buys. What utilisation did, multiplied by the cluster. And which side of the node boundary the links are on.

"What is the most common mistake?" Subtracting the attached capacity from the fast-tier requirement. The floor does not move: whatever the step touches at full rate still has to be in the fast tier, so adding capacity further away changes what the cluster can hold and not what it must hold at speed. A plan that conflates them buys capacity that is on the wrong side of the problem.

"Why is the utilisation number the one you would ask for first?" Because it is already instrumented, it is a single reading, and it localises the problem immediately. Every millisecond of wait is a millisecond not computing, and the same wait happens on every device on every step — so a per-device figure that looks tolerable can be a large aggregate loss, and a per-device acceptance test can pass while the deployment is losing badly.

"Does it help the whole pipeline?" No, and that is the useful part of the answer. A pipeline has stages with different limiting resources. A larger slower tier helps the capacity-bound ones and harms the bandwidth-bound ones, so the classification is per stage — and a technique that was validated on one stage and rolled out to all of them produces a small unattributable gain that makes it look marginal.

"Is the memory attach the fabric between nodes?" No. It is a node-scope attach; the fabric between nodes is a network. They have different reach, latency and sharing semantics, and a plan that describes a rack as though it were a node has planned for bandwidth that will not arrive.

26. Exercises

1. A cluster has 5 tiers, 2 named, a 2,400 GB step and 90 GB on the named tier. Compute the unnamed count, the share and the fraction pinned down. How many tiers must be named to reach 100%?

2. 200 GB of parameters, 400 of optimizer state, 150 of activations, against 141 GB of fast tier. Compute the requirement, the overflow and the ratio. What activation figure would make it fit?

3. Traffic needs 900 GB/s; a tier offers 128; the fast tier runs at 4,000; the offload is 600 GB. Compute the shortfall, the served fraction, the ratio and the drain. What offload size drains in under one second?

4. An 1,800 GB set with 500 hot, 141 GB of fast tier, reused 6 times. Compute the resident, the evicted, the hot fraction and the refetch. What hot figure fits?

5. 128 links, 16 in a node, 40 claimed for an attach, 45 units per hop. Compute the misplaced count and its cost. What does it cost if the node is redefined as a rack of 4?

6. 900 GB at 9 per GB against 60 hours at 140. Compute cost, value and net. At what hour cost does it break even?

7. A 250 ms step with 40 ms of stall, 800 steps, 512 devices. Compute utilisation and the cluster total. What stall keeps utilisation at 95%?

8. 20 stages, 13 capacity-bound, 6 bandwidth-bound, 55 units each. Compute helped, unhelped and gain. Which six would you want named?

9. A 640 GB floor with 400 present and 1,024 added at 7 per GB. Compute the shortfall and the waste. What fast-tier figure meets the floor?

10. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify its position using the rule that the ordering is by how much of the deployment each condition carries.

27. Summary

"Memory" names no tier, and a cluster has several that differ by more than an order of magnitude.

The capacity wall is a sum of parameters, optimizer state and activations — not a parameter count.

A tier only helps if the traffic tolerates its bandwidth, and the gap to the fast tier is a large multiple.

Only the hot part needs the fast tier, which makes the wall smaller than the working set suggests.

An attach inside a node is not the fabric between nodes, and the boundary has to be drawn before the links are counted.

An offload pays only if the accelerator time it buys is worth more than the tier costs, and both prices already exist.

An accelerator waiting on a slower tier is idle, and the same wait happens on every device on every step.

A capacity-bound stage gains and a bandwidth-bound one does not, so the answer is per stage.

The fast-tier floor does not move when capacity is added further away.

Six bits, and "they use CXL for memory" is one of them. One case study of eight is sound; the memory-answer view counts seven.