AI Training Clusters

22.4 ended on a single node holding a model it could not serve. This chapter is the same failure at rack scale, where the machine is bought by peak capability and paid for by delivered work, and the gap between the two is where every number in this chapter lives.

Sixty-four nodes of a hundred teraflops each is 6400 teraflops on the purchase order. Spend half of every step communicating and it is 3200. Add a pipeline bubble, a straggler and a checkpoint and the honest figure keeps falling — not because anything broke, but because nothing in the list is optional.

This is also the chapter where CXL's role is smallest and most specific, and section 24 says exactly where it is.

1. The Engineering Problem — Peak Is Bought, Efficiency Is Earned

N nodes give N times the compute only if talking is free. Half a step spent on the collective is 50% efficiency and 3200 of 6400 teraflops. Section 5.

An all-reduce moves nearly twice the gradient, not once. Eight ranks move 14 GB of an 8 GB gradient, and sixty-four move fifteen. Section 6.

A pipeline is idle while it fills and drains. Eight stages on one microbatch is an 87% bubble; on sixty-four it is nine. Section 7.

A run has to write its whole state down, often. 560 GB at 80 Gbps is 56 seconds, which is 18% of a five-minute interval and 93% of a one-minute one. Section 8.

And a thousand year-long nodes fail every eight hours. Three times a day, losing 90 minutes of work at an hourly checkpoint. Section 9.

This chapter against 22.4, stated precisely. That one owns one node holding a model. This one owns many nodes finishing a step together — which is why sections 6, 7 and 11 have no single-node equivalent and section 17 is entirely about totals.

2. The One-Sentence Model

A training cluster scales when nodes are added, the collective is inside the step time, the pipeline has enough microbatches to fill, the checkpoint fits its interval, the slowest rank is near the mean, and the fabric carries the collective it is asked to carry — and every defect below is a cluster that cost more and trained no faster.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real training cluster is a scheduler, a collective library, a fabric manager, a checkpoint service and a fault-tolerance layer, and none of that is reproduced. What is reproduced is the arithmetic each of them has to get right, and the shape of the mistake when it does not.

Each model is built twice — a correct build and a broken build selected by a parameter. The broken builds here share one property: they are all what a specification sheet says. Peak teraflops. Full-bisection bandwidth. Node mean time between failures. Every one of them is a true number about a component and a false number about the cluster, which is why they are so hard to argue with.

Figure 1 — Peak on the left is a purchase order; achieved on the right is what trains a model. Every dashed edge is a term the purchase order does not carry, and sections 5 through 12 price them one at a time.

5. RTL 1 — N Nodes Give N Times The Compute Only If Talking Is Free

// RTL 1 - scaling a cluster. N nodes give N times the compute only if the
// communication between them costs nothing, and it never costs nothing.
module cluster_scale #(parameter int IGNORE_COMM = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] nodes, node_tflops, compute_ms, comm_ms,
  output logic [15:0] ideal_tflops, step_ms, efficiency_pct, achieved_tflops,
  output logic        scales_well,
  output logic [7:0]  n_measures, n_poor,
  output logic        comm_ignored_err
);
  logic [31:0] i_q, e_q, a_q;
  assign i_q = {16'd0, nodes} * {16'd0, node_tflops};
  assign ideal_tflops = (i_q > 32'd65535) ? 16'hFFFF : i_q[15:0];
  // Ignoring communication is what a peak-FLOPs figure does.
  assign step_ms = (IGNORE_COMM != 0) ? compute_ms : (compute_ms + comm_ms);
  assign e_q = (step_ms == 16'd0) ? 32'd0
             : (({16'd0, compute_ms} * 32'd100) / {16'd0, step_ms});
  assign efficiency_pct = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  assign a_q = ({16'd0, ideal_tflops} * {16'd0, efficiency_pct}) / 32'd100;
  assign achieved_tflops = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign scales_well = (efficiency_pct >= 16'd75);
  // A step time that does not include the communication it performed.
  assign comm_ignored_err = measure && (comm_ms != 16'd0) && (step_ms == compute_ms);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_poor <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (!scales_well) n_poor <= n_poor + 8'd1;
    end
  end
endmodule

Five measurements. Sixty-four nodes of a hundred teraflops — a peak of 6400.

Three measurements scaled poorly; the peak-FLOPs model reported one.

Half the machine, and nothing is broken. Every node is computing at full rate, the fabric is working, and the cluster delivers 3200 of the 6400 teraflops it was bought as. The efficiency is not a defect to be fixed; it is a property of the ratio between compute and communication, and only changing that ratio changes it.

Row four is the threshold and it is where the argument gets practical. Three hundred milliseconds of compute against a hundred of communication is 75% — the same collective, made affordable by a larger step. Bigger batches and bigger models communicate the same bytes over more compute, which is why scaling efficiency improves as models grow and degrades as clusters do.

Row five is the degenerate end. A step with no compute in it is all communication, and the efficiency is zero — and the peak-FLOPs model reports a step of zero length, which is the shape a badly-instrumented profiler produces when it times only the kernels.

6. RTL 2 — An All-Reduce Moves Nearly Twice The Gradient

// RTL 2 - the collective. An all-reduce over N ranks moves nearly twice the
// gradient across the fabric, not once, and the topology sets the factor.
module collective_cost #(parameter int IGNORE_TOPOLOGY = 0) (
  input  logic clk, rst_n,
  input  logic        collect,
  input  logic [15:0] bytes_gb, nodes, fabric_gbps, budget_ms,
  output logic [15:0] moved_gb, collective_ms,
  output logic        within_budget,
  output logic [7:0]  n_collectives, n_over_budget,
  output logic        understated_err
);
  logic [31:0] m_q, t_q;
  // A ring all-reduce moves 2(N-1)/N of the gradient; a flat model moves one.
  assign m_q = (IGNORE_TOPOLOGY != 0) ? {16'd0, bytes_gb}
             : ((nodes == 16'd0) ? 32'd0
                : ((32'd2 * {16'd0, bytes_gb} * ({16'd0, nodes} - 32'd1))
                   / {16'd0, nodes}));
  assign moved_gb = (m_q > 32'd65535) ? 16'hFFFF : m_q[15:0];
  assign t_q = (fabric_gbps == 16'd0) ? 32'd65535
             : (({16'd0, moved_gb} * 32'd8000) / {16'd0, fabric_gbps});
  assign collective_ms = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign within_budget = (collective_ms <= budget_ms);
  // A collective over more than two ranks costed as a single pass.
  assign understated_err = collect && (nodes > 16'd2) && (moved_gb == bytes_gb);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_collectives <= 8'd0; n_over_budget <= 8'd0;
    end else if (collect) begin
      n_collectives <= n_collectives + 8'd1;
      if (!within_budget) n_over_budget <= n_over_budget + 8'd1;
    end
  end
endmodule

Six collectives. An 8 GB gradient over an 800 Gbps fabric, against a 100 ms budget.

Three collectives exceeded the budget; the flat model reported one.

Two times minus a bit is the factor everybody forgets. A ring all-reduce is a reduce-scatter followed by an all-gather, and each moves (N−1)/N of the gradient — so the fabric carries 2(N−1)/N, approaching twice the gradient as the cluster grows and never falling below it beyond two ranks.

Row two is the case where the flat model is right, and it is worth being precise about. Over two ranks, 2(N−1)/N is exactly one, so the flat count is not an approximation — it is the answer. understated_err is gated on nodes > 2 for exactly that reason.

Row three is the ceiling. Sixty-four ranks move 15 GB of an 8 GB gradient, and a thousand ranks would move very slightly more. The factor saturates, which is the good news: the collective's cost grows with the gradient and the fabric, not with the cluster size, once past a handful of ranks.

7. RTL 3 — A Pipeline Is Idle While It Fills

// RTL 3 - the pipeline bubble. Splitting a model across stages leaves every
// stage idle while the pipeline fills and drains, and the microbatch count is
// the only thing that shrinks it.
module pipeline_bubble #(parameter int IGNORE_BUBBLE = 0) (
  input  logic clk, rst_n,
  input  logic        schedule,
  input  logic [15:0] stages, microbatches,
  output logic [15:0] bubble_pct, useful_pct,
  output logic        acceptable,
  output logic [7:0]  n_schedules, n_stalled,
  output logic        bubble_ignored_err
);
  logic [31:0] b_q;
  logic [15:0] denom;
  assign denom = microbatches + stages - 16'd1;
  // The bubble is (S-1)/(M+S-1) of every step. A model that ignores it reports
  // a pipeline as if it filled instantly.
  assign b_q = (IGNORE_BUBBLE != 0) ? 32'd0
             : ((denom == 16'd0) ? 32'd0
                : ((({16'd0, stages} - 32'd1) * 32'd100) / {16'd0, denom}));
  assign bubble_pct = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  // bubble_pct is a share of a total that includes it, so it never exceeds 100.
  assign useful_pct = 16'd100 - bubble_pct;
  assign acceptable = (bubble_pct <= 16'd20);
  // A multi-stage pipeline reported with no fill or drain at all.
  assign bubble_ignored_err = schedule && (stages > 16'd1) && (bubble_pct == 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_schedules <= 8'd0; n_stalled <= 8'd0;
    end else if (schedule) begin
      n_schedules <= n_schedules + 8'd1;
      if (!acceptable) n_stalled <= n_stalled + 8'd1;
    end
  end
endmodule

Seven schedules.

Two schedules stalled; the no-bubble model reported none.

The bubble is (S−1)/(M+S−1) and both terms are choices. More stages deepens it; more microbatches dilutes it, and the second is nearly free while the first is forced by the model's size. That asymmetry is the whole design rule: pipeline depth is a capacity decision and microbatch count is the compensation for it.

Rows one and four are the practical range. Thirty-two microbatches over eight stages costs 17%, sixty-four costs 9%, and the returns diminish as 1/M. Doubling the microbatches roughly halves the bubble, and past a certain point the memory the in-flight microbatches occupy costs more than the bubble saves — which is section 11's ground.

Row three is the case that makes the check honest. A single stage is not a pipeline, has no fill or drain, and reporting no bubble for it is correct. bubble_ignored_err requires stages > 1.

Figure 2 — Only the top branch produces teraflops. The other three are all real time on real hardware doing necessary things, and a peak figure counts none of them — which is why section 13's cost per useful FLOP is twice what the purchase order implies.

8. RTL 4 — A Run Has To Write Itself Down

// RTL 4 - the checkpoint. A training run has to write its whole state down
// often enough to survive a failure, and that write is not free.
module checkpoint_cost #(parameter int IGNORE_CHECKPOINT = 0) (
  input  logic clk, rst_n,
  input  logic        checkpoint,
  input  logic [15:0] footprint_gb, tier_gbps, interval_s,
  output logic [15:0] write_s, overhead_pct,
  output logic        affordable,
  output logic [7:0]  n_checkpoints, n_costly,
  output logic        free_write_err
);
  logic [31:0] w_q, o_q;
  // Writing the state down costs the footprint at the tier's rate.
  assign w_q = (IGNORE_CHECKPOINT != 0) ? 32'd0
             : ((tier_gbps == 16'd0) ? 32'd65535
                : (({16'd0, footprint_gb} * 32'd8) / {16'd0, tier_gbps}));
  assign write_s = (w_q > 32'd65535) ? 16'hFFFF : w_q[15:0];
  assign o_q = (interval_s == 16'd0) ? 32'd65535
             : (({16'd0, write_s} * 32'd100) / {16'd0, interval_s});
  assign overhead_pct = (o_q > 32'd65535) ? 16'hFFFF : o_q[15:0];
  assign affordable = (overhead_pct <= 16'd10);
  // A state write that took no time at all.
  assign free_write_err = checkpoint && (footprint_gb != 16'd0)
                          && (tier_gbps != 16'd0) && (write_s == 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_checkpoints <= 8'd0; n_costly <= 8'd0;
    end else if (checkpoint) begin
      n_checkpoints <= n_checkpoints + 8'd1;
      if (!affordable) n_costly <= n_costly + 8'd1;
    end
  end
endmodule

Six checkpoints. A 560 GB training state — 22.4 §5's footprint.

Four checkpoints were unaffordable; the no-checkpoint model reported one.

The checkpoint is the same footprint 22.4 §5 computed, written to something durable. 560 GB is not the model — it is the model plus gradients plus optimizer state, all of which have to survive a restart, which is why the training footprint rather than the serving one sets the write.

Row three is the tension this model exists to expose. A shorter interval loses less work per failure — section 9's arithmetic — and costs more overhead. Sixty seconds of interval against 56 seconds of write is a run that spends 93% of its life checkpointing, and the optimum is a balance between two costs that move in opposite directions.

Row two is the argument for a fast checkpoint tier, and it is where CXL has a genuine role — see section 24. Ten times the write bandwidth turns an 18% overhead into 1%, which then permits a much shorter interval, which then cuts section 9's lost work.

9. RTL 5 — A Thousand Year-Long Nodes Fail Every Eight Hours

// RTL 5 - failures at scale. A node that fails once a year fails every few
// hours once there are a thousand of them, and the run loses work each time.
module failure_recovery #(parameter int IGNORE_FAILURES = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [15:0] nodes, node_mtbf_h, interval_min,
  output logic [15:0] cluster_mtbf_h, failures_per_day, lost_min_per_day,
  output logic        tolerable,
  output logic [7:0]  n_assessments, n_lossy,
  output logic        scale_ignored_err
);
  logic [31:0] c_q, f_q, l_q;
  // A cluster fails as often as its nodes do, divided by how many there are.
  assign c_q = (IGNORE_FAILURES != 0) ? {16'd0, node_mtbf_h}
             : ((nodes == 16'd0) ? {16'd0, node_mtbf_h}
                : ({16'd0, node_mtbf_h} / {16'd0, nodes}));
  assign cluster_mtbf_h = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  assign f_q = (cluster_mtbf_h == 16'd0) ? 32'd65535
             : (32'd24 / {16'd0, cluster_mtbf_h});
  assign failures_per_day = (f_q > 32'd65535) ? 16'hFFFF : f_q[15:0];
  // A failure loses on average half the interval since the last checkpoint.
  assign l_q = ({16'd0, failures_per_day} * {16'd0, interval_min}) / 32'd2;
  assign lost_min_per_day = (l_q > 32'd65535) ? 16'hFFFF : l_q[15:0];
  assign tolerable = (lost_min_per_day <= 16'd60);
  // A cluster failure rate reported as a single node's.
  assign scale_ignored_err = assess && (nodes > 16'd1)
                             && (cluster_mtbf_h == node_mtbf_h);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assessments <= 8'd0; n_lossy <= 8'd0;
    end else if (assess) begin
      n_assessments <= n_assessments + 8'd1;
      if (!tolerable) n_lossy <= n_lossy + 8'd1;
    end
  end
endmodule

Seven assessments. Nodes whose parts last a year — 8760 hours each.

Four assessments were intolerable; the single-node model reported none.

Reliability divides. A component that is excellent on its own — a year between failures — is a component that fails every eight hours when there are a thousand of them, and nothing about the component changed. This is the single most counter-intuitive number in cluster design and it is elementary arithmetic.

Row three is the trade against section 8, in the opposite direction. Doubling the checkpoint interval halves the overhead and doubles the work lost per failure — 180 minutes a day instead of 90. The two models together define an optimum, and neither alone can find it.

Row seven is where the arithmetic stops being useful and starts being a verdict. At a scale where the mean time between failures is under an hour, the run spends more time restarting than training, and the answer is not a shorter interval — it is fault tolerance that does not restart the whole run, which is beyond what this model describes.

10. RTL 6 — A Step Finishes When Its Slowest Rank Finishes

// RTL 6 - the straggler. A synchronous step finishes when its slowest rank
// finishes, so the mean rank time is not the step time.
module straggler_step #(parameter int ASSUME_UNIFORM = 0) (
  input  logic clk, rst_n,
  input  logic        step,
  input  logic [15:0] mean_ms, slowest_ms,
  output logic [15:0] step_ms, waste_ms, waste_pct,
  output logic        tolerable,
  output logic [7:0]  n_steps, n_wasteful,
  output logic        uniform_assumed_err
);
  logic [31:0] p_q;
  // A synchronous step waits for the slowest rank, not the average one.
  assign step_ms = (ASSUME_UNIFORM != 0) ? mean_ms : slowest_ms;
  // step_ms is the slowest rank and mean_ms never exceeds it, so this cannot
  // underflow in the correct build; the uniform build makes the two equal.
  assign waste_ms = (step_ms > mean_ms) ? (step_ms - mean_ms) : 16'd0;
  assign p_q = (mean_ms == 16'd0) ? 32'd0
             : (({16'd0, waste_ms} * 32'd100) / {16'd0, mean_ms});
  assign waste_pct = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign tolerable = (waste_pct <= 16'd10);
  // A step time taken from the average rank while a slower one existed.
  assign uniform_assumed_err = step && (slowest_ms > mean_ms)
                               && (step_ms == mean_ms);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_steps <= 8'd0; n_wasteful <= 8'd0;
    end else if (step) begin
      n_steps <= n_steps + 8'd1;
      if (!tolerable) n_wasteful <= n_wasteful + 8'd1;
    end
  end
endmodule

Five steps. A 200 ms mean rank time.

Two steps were wasteful; the uniform model reported none.

A synchronous step is a maximum, not an average, and the distinction costs whatever the tail of the rank-time distribution costs. Sixty-three ranks finishing in 200 ms and one finishing in 260 gives a 260 ms step for everybody — the sixty-three wait.

Thirty percent from one slow rank in sixty-four is why tail latency is a cluster property. Improving the mean rank time by tuning kernels does nothing here; the only thing that helps is making the slowest rank faster, which is usually a different problem entirely — a hot node, a degraded link, a rank with an unlucky data shard.

Row two is the honest baseline the check needs. When the ranks really are uniform, taking the mean is taking the maximum, and neither model is wrong. uniform_assumed_err requires a genuinely slower rank to exist.

Half the cycles produce work and all eight are paid for. The waveform is section 5's efficiency, section 6's ring factor and section 10's straggler in one picture — and the peak-teraflops figure counts all eight cycles as compute.

11. RTL 7 — Training Holds Every Layer's Activations

// RTL 7 - activations. Training holds every layer's activations for the
// backward pass unless it agrees to recompute them, and the batch multiplies it.
module activation_memory #(parameter int IGNORE_ACTIVATIONS = 0) (
  input  logic clk, rst_n,
  input  logic        size_it,
  input  logic        recompute,
  input  logic [15:0] layers, batch, act_per_layer_mb, hbm_mb,
  output logic [15:0] act_mb, spare_mb,
  output logic        fits,
  output logic [7:0]  n_sizings, n_unfit,
  output logic        activations_free_err
);
  logic [31:0] a_q;
  logic [15:0] held_layers;
  // Recomputation holds one layer's activations instead of every layer's.
  assign held_layers = recompute ? 16'd1 : layers;
  assign a_q = (IGNORE_ACTIVATIONS != 0) ? 32'd0
             : ({16'd0, held_layers} * {16'd0, batch} * {16'd0, act_per_layer_mb});
  assign act_mb = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign fits = (act_mb <= hbm_mb);
  assign spare_mb = fits ? (hbm_mb - act_mb) : 16'd0;
  // Activations that occupy no memory at all.
  assign activations_free_err = size_it && (layers != 16'd0) && (batch != 16'd0)
                                && (act_per_layer_mb != 16'd0) && (act_mb == 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_sizings <= 8'd0; n_unfit <= 8'd0;
    end else if (size_it) begin
      n_sizings <= n_sizings + 8'd1;
      if (!fits) n_unfit <= n_unfit + 8'd1;
    end
  end
endmodule

Five sizings. Eighty layers, 16 MB of activation each, against 8192 MB of fast memory.

One sizing did not fit; the model that ignores activations reported none.

Eighty times, for one flag. Recomputation holds one layer's activations and recomputes the rest during the backward pass — trading roughly a third more compute for a factor of the layer count in memory. Almost every large training run uses it, and a capacity plan that does not say which assumption it made is off by that factor.

And this is section 7's other side. More microbatches shrink the bubble and more microbatches are more activations in flight, so the two models bound each other: the pipeline wants many, the memory permits few, and recomputation is what widens the window.

Row five is the guard's reason. An empty batch holds nothing, so activations costing nothing is correct rather than an omission — which is why activations_free_err requires layers, batch and per-layer size all to be non-zero.

12. RTL 8 — A Full-Bisection Number Is A Link Count

// RTL 8 - the fabric. A full-bisection number is a link count; an oversubscribed
// fabric delivers a fraction of it, and the collective sees the fraction.
module fabric_bisection #(parameter int ASSUME_FULL = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [15:0] nodes, link_gbps, oversub_x, demand_gbps,
  output logic [15:0] full_bisection_gbps, real_bisection_gbps,
  output logic        sufficient,
  output logic [7:0]  n_assessments, n_short,
  output logic        oversub_ignored_err
);
  logic [31:0] f_q, r_q;
  assign f_q = ({16'd0, nodes} * {16'd0, link_gbps}) / 32'd2;
  assign full_bisection_gbps = (f_q > 32'd65535) ? 16'hFFFF : f_q[15:0];
  // Oversubscription divides what the fabric can actually carry across the cut.
  assign r_q = (ASSUME_FULL != 0) ? {16'd0, full_bisection_gbps}
             : ((oversub_x == 16'd0) ? {16'd0, full_bisection_gbps}
                : ({16'd0, full_bisection_gbps} / {16'd0, oversub_x}));
  assign real_bisection_gbps = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  assign sufficient = (real_bisection_gbps >= demand_gbps);
  // An oversubscribed fabric quoted at its full-bisection number.
  assign oversub_ignored_err = assess && (oversub_x > 16'd1)
                               && (real_bisection_gbps == full_bisection_gbps);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assessments <= 8'd0; n_short <= 8'd0;
    end else if (assess) begin
      n_assessments <= n_assessments + 8'd1;
      if (!sufficient) n_short <= n_short + 8'd1;
    end
  end
endmodule

Six assessments. Sixteen nodes on 800 Gbps links — a full bisection of 6400.

Three fabrics fell short; the full-bisection model reported one.

A full-bisection figure is arithmetic on the endpoints, not a measurement of the fabric. Sixteen nodes with 800 Gbps links have 6400 Gbps across any cut if the switching layer can carry it, and an oversubscribed topology cannot — by construction and by design, because full bisection at scale is extremely expensive.

Four to one is a common and reasonable design point, and it is a factor of four on exactly the traffic section 6 generates. An all-reduce is the worst possible pattern for an oversubscribed fabric: it is all-to-all, it is synchronous, and it happens every step.

Row six is the boundary the sufficiency test defines, and it is the one a fabric is actually specified against — enough and no more, with no margin for the collective growing when someone raises the batch size.

13. RTL 9 — Cost Per Useful FLOP

// RTL 9 - what a cluster costs per useful FLOP. Peak capability is bought;
// efficiency decides how much of it is delivered, and only the second is spent.
module cluster_economics #(parameter int IGNORE_EFFICIENCY = 0) (
  input  logic clk, rst_n,
  input  logic        price,
  input  logic [15:0] nodes, node_cost, node_tflops, efficiency_pct,
  output logic [15:0] total_cost, peak_tflops, useful_tflops, cost_per_useful,
  output logic        acceptable,
  output logic [7:0]  n_pricings, n_costly,
  output logic        efficiency_ignored_err
);
  logic [31:0] c_q, p_q, u_q, k_q;
  assign c_q = {16'd0, nodes} * {16'd0, node_cost};
  assign total_cost = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  assign p_q = {16'd0, nodes} * {16'd0, node_tflops};
  assign peak_tflops = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  // Peak is what was bought; useful is what the cluster delivers.
  assign u_q = (IGNORE_EFFICIENCY != 0) ? {16'd0, peak_tflops}
             : (({16'd0, peak_tflops} * {16'd0, efficiency_pct}) / 32'd100);
  assign useful_tflops = (u_q > 32'd65535) ? 16'hFFFF : u_q[15:0];
  assign k_q = (useful_tflops == 16'd0) ? 32'd65535
             : (({16'd0, total_cost} * 32'd1000) / {16'd0, useful_tflops});
  assign cost_per_useful = (k_q > 32'd65535) ? 16'hFFFF : k_q[15:0];
  assign acceptable = (cost_per_useful <= 16'd1500);
  // A cluster priced at peak while its efficiency was below it.
  assign efficiency_ignored_err = price && (efficiency_pct < 16'd100)
                                  && (useful_tflops == peak_tflops);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_pricings <= 8'd0; n_costly <= 8'd0;
    end else if (price) begin
      n_pricings <= n_pricings + 8'd1;
      if (!acceptable) n_costly <= n_costly + 8'd1;
    end
  end
endmodule

Six pricings. Nodes at a hundred each, a hundred teraflops each.

Four pricings were unacceptable; the peak-priced model reported none.

Row five is the point of the model. A quarter of the cluster at the same efficiency costs exactly the same per useful teraflop — 2000 either way. Scale does not change the unit economics; efficiency does, and a cluster four times larger at half the efficiency is strictly worse than a smaller one that scales.

Doubling the cost per useful FLOP is the price of section 5's fifty percent, and it is the number that should appear beside every scaling decision. The purchase order says 6400 teraflops at 6400; what is being bought is 3200 teraflops at 6400.

The peak-priced model is exactly what a procurement process computes, because efficiency requires a measurement that does not exist until the cluster is built. That is the structural reason this failure is so common — the honest number is not available at the moment the decision is made.

14. RTL 10 — An AI Training Cluster Assembled

// RTL 10 - an AI training cluster assembled. Everything that must hold before
// adding nodes adds training throughput rather than only capability on paper.
module training_cluster_model #(parameter int SCALE_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       nodes_added,          // there is more hardware
  input  logic       comm_costed,          // the collective is in the step time
  input  logic       bubble_bounded,       // enough microbatches to fill the pipe
  input  logic       checkpoint_affordable,// the state write fits the interval
  input  logic       stragglers_bounded,   // the slowest rank is near the mean
  input  logic       bisection_sufficient, // the fabric carries the collective
  output logic       scales,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_scales,
  output logic       false_scale_err
);
  assign fail_mask[0] = ~nodes_added;
  assign fail_mask[1] = ~comm_costed;
  assign fail_mask[2] = ~bubble_bounded;
  assign fail_mask[3] = ~checkpoint_affordable;
  assign fail_mask[4] = ~stragglers_bounded;
  assign fail_mask[5] = ~bisection_sufficient;
  // The scale-only build counts the nodes and calls that scaling.
  assign scales = (SCALE_ONLY != 0) ? nodes_added : (fail_mask == 6'd0);
  assign false_scale_err = evaluate && scales && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_scales <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (scales) n_scales <= n_scales + 8'd1;
    end
  end
endmodule

One scaling configuration of six, and four false claims.

The scale-only definition is a node count, and it is right about exactly one of the six — the one where nothing was added. Row four is the deployment that costs the most to diagnose: the memory plan, the fabric and the schedule are all correct, and one rank in sixty-four is 30% slow.

Figure 4 — Four terminals, four different teams. The fabric branch is a network problem, the straggler branch is a per-node problem, the bubble branch is a scheduler configuration and the checkpoint branch is a storage one — and the first split, compute against everything else, is the only measurement that tells you which building to walk into.

15. Quantitative Reasoning

Scaling. Sixty-four nodes of a hundred teraflops peak at 6400. Half the step spent communicating gives 50% efficiency and 3200 achieved; three-quarters compute gives 75% and 4800.

The collective. An 8 GB gradient over eight ranks moves 14 GB, over sixty-four moves 15, and over two moves exactly 8. At 800 Gbps that is 140 ms against a flat model's 80.

The pipeline. Eight stages bubble 87% on one microbatch, 17% on thirty-two and 9% on sixty-four — the bubble falls as 1/M.

The checkpoint. 560 GB at 80 Gbps is 56 seconds — 18% of a five-minute interval and 93% of a one-minute one. At 800 Gbps it is 5 seconds and 1%.

Failures. A thousand year-long nodes have an eight-hour cluster mean time between failures, three failures a day and 90 minutes of work lost at an hourly interval — 180 at a two-hourly one.

Stragglers. One 260 ms rank among 200 ms ranks makes the step 260 — a 30% tax on every other rank.

Activations. Eighty layers at a batch of eight is 10,240 MB held; recomputation holds 128 — a factor of eighty.

The fabric. Sixteen nodes on 800 Gbps links bisect at 6400 in principle and 1600 at four-to-one — a factor of four on the exact traffic the collective generates.

Economics. 6400 spent, 3200 delivered: 2000 per useful teraflop against a peak-priced 1000 — and a quarter of the cluster at the same efficiency costs exactly the same.

The assembled model. Six properties, six configurations, one scales. The scale-only definition reported five.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Scaling. The efficiency exactly at the threshold is driven, and the no-communication case is asserted as not an omission.

chk(cGe == 16'd75, "300 ms of compute against 100 of communication is 75 percent");
chk(cGw == 1'b1,   "which exactly scales well");

The collective. The two-rank case is asserted as one both models get right, and a collective exactly filling the budget is driven.

chk(kGm == 16'd8,   "two ranks move the gradient once");
chk(kGt == 16'd100, "which at 640 Gbps is exactly the 100 ms budget");

The pipeline. A single stage is asserted as not an ignored bubble, the empty-schedule guard is driven, and a bubble exactly at the acceptance threshold is driven.

The checkpoint. Both the missing-interval and missing-tier guards are driven, and an overhead exactly at the threshold is asserted.

Failures. The interval at which the loss is exactly tolerable is driven, and the scale at which the cluster mean time between failures rounds to nothing is asserted unbounded.

Stragglers. The uniform case is asserted as one both models get right, and the tolerance boundary is driven exactly.

Activations. An activation set exactly filling the fast tier is driven, and the empty batch is asserted as not a free activation.

The fabric. Both reasons the real bisection equals the full one — one-to-one and unstated — are asserted as not ignored oversubscription.

Economics. A quarter-sized cluster at the same efficiency is asserted to cost exactly the same per useful teraflop.

chk(eGu == 16'd800,  "delivering 800 teraflops");
chk(eGk == 16'd2000, "at the same 2000 per useful teraflop");

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 256 checks across two testbenches, 134 on the front five models and 122 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Sixty-three mutations were injected one at a time.

63 injected, 63 killed, after two survivors.

Both survivors were the same omission. within_budget in section 6 and acceptable in section 7 each survived <= becoming >, because no case drove a collective exactly filling its budget or a bubble exactly at the acceptance threshold. Two more inclusive thresholds, in a batch that has now produced five of them across three chapters — and every one was a stimulus gap rather than a design defect.

That consistency is itself the finding. Nine of the ten models here have an inclusive threshold, and only the two that were never driven at equality survived. The failure mode is systematic, which means the fix is a habit rather than a fix: when a model has a <= or >=, drive the case where the two sides are equal, before the mutation asks for it.

Constructing the equality is arithmetic, not luck. Section 6's budget case needed a fabric rate that makes 8 GB take exactly 100 ms — 640 Gbps. Section 7's needed a stage and microbatch count with (S−1)/(M+S−1) exactly one fifth — five stages and sixteen microbatches. Neither is a value a random sweep would land on, which is why the mutation is the thing that finds it.

18. Verification Strategy

What a testbench for a cluster-scaling model must cover.

Drive every inclusive threshold at exactly equal, and construct the input to get there. Both survivors here were this, and the batch has now produced five across three chapters. It is the most reliable single source of surviving mutations in this material.

The cases where the specification-sheet model is right. Two ranks for the ring factor. A one-to-one fabric. A single stage for the bubble. Uniform ranks. A hundred percent efficiency. Each is a case the check must exempt, and each is a real configuration.

Both directions of a trade. Sections 8 and 9 move in opposite directions with the checkpoint interval — shorter costs overhead, longer costs lost work — and a testbench that drives only one direction cannot show that an optimum exists.

The degenerate end of every scale. A step with no compute. A cluster with no nodes. A fabric with no links. A schedule with no microbatches. Four of the ten guards here exist for exactly one such case, and each needed driving to kill its mutation.

Counters as a second signature. Every model reports totals for both builds, and they differ in all ten — three against one, four against none, seven assessments with four intolerable against zero. A single output can coincide; ten pairs of totals do not.

What a real cluster needs that these models do not have. Overlap between the collective and the backward pass, which is the single largest omission in section 5 — real stacks hide much of the all-reduce behind compute. Hierarchical collectives, which change section 6's factor. Elastic training, which changes section 9's conclusion entirely.

19. Synthesis and Implementation Reality

Nothing in this chapter is a hardware block. Every model is arithmetic that a scheduler, a profiler or a capacity planner performs, and its failure mode is a purchase decision or a job configuration rather than a timing violation.

Section 5's efficiency is measured, not designed. The only instrument that produces it is a profiler that times the whole step rather than the kernels, and the common instrumentation failure is timing only what runs on the accelerator — which reports exactly the broken build's number.

Section 6's ring factor is a property of the collective algorithm, and real libraries use tree, hierarchical and bandwidth-optimal variants that change the constant. What does not change is that it is greater than one for more than two ranks, which is the only claim the model makes.

Section 7's microbatch count is a single configuration value with a memory cost from section 11 attached. The two are set together or neither is set correctly.

Section 8's checkpoint tier is where CXL genuinely fits, and section 24 develops it: a capacity tier that absorbs a 560 GB state write faster than durable storage lets the interval shorten, which cuts section 9's lost work.

Section 12's oversubscription is fixed at the moment the fabric is cabled, and no software changes it. It is the only term in the chapter that cannot be tuned after the fact.

20. Silicon Observability

"Straggler identity over a window" is the counter that turns section 10 from a number into an action. A 30% straggler tax says the step is slow; knowing whether it is the same rank every step says whether to replace a node or investigate the data, and a mean-and-maximum report cannot distinguish them.

21. Debug Lab

Symptom. A cluster is doubled from 32 nodes to 64 to halve the time to train a model. Training time falls by 11%. Every node reports full utilisation, the fabric reports no errors, and the job completes correctly.

Step 1 — split the step. Compute against non-compute: at 32 nodes the step was 70% compute; at 64 it is 41%. Section 5's efficiency fell from 70 to 41 while the node count doubled — which is exactly the 11%, since 64 × 0.41 against 32 × 0.70 is a ratio of 1.17 before the other terms.

Step 2 — what grew? Fabric bytes per step: unchanged at 14 GB, because section 6's ring factor was already near its ceiling at 32 ranks and barely moved at 64. The collective is not moving more data.

Step 3 — so why is it slower? Fabric utilisation at the bisection cut: saturated during the collective at 64 nodes, not at 32. The same 14 GB is crossing a cut that now has twice as many nodes behind it, and the fabric is four-to-one oversubscribed — section 12, and the number that was quoted at purchase was the full-bisection one.

Step 4 — check the ranks. Per-rank step time at 64 nodes: mean 180 ms, maximum 244 ms, and the maximum is a different rank each step. That is a 36% straggler tax on top, and a rotating straggler means it is the data shard rather than a bad node — section 10, with section 20's identity counter answering which.

Step 5 — check the schedule. Microbatches per step: unchanged from the 32-node configuration. The pipeline is now eight stages instead of four, so section 7's bubble roughly doubled — from 9% to 17% — because nobody raised the microbatch count when the depth changed.

The finding. Three independent terms degraded at once, and each is small enough alone to dismiss. Doubling the cluster made the fabric the bottleneck, doubled the pipeline bubble and exposed a straggler that 32 ranks had absorbed — and none of them is a fault.

The fix. Raise the microbatch count to restore section 7's bubble, which section 11 permits because activation memory is per-stage and the stages got shallower. Balance the data shards to cut the straggler. The fabric is cabled and cannot be fixed — which is why section 13's cost per useful teraflop rose from 1400 to 2400, and the honest recommendation is that this cluster should not have been doubled.

What made this hard. Every subsystem was healthy, the job was correct, and the failure was distributed across three teams' areas with no single owner. The only measurement that pointed anywhere was the first one, and it is the one that is least often instrumented.

22. Design Review

1. What is the step's compute-to-communication ratio, measured end to end? If nobody has it, section 5 is unanswerable and so is everything else.

2. What does the collective actually move, against the gradient size? Nearly twice, for more than two ranks. Section 6.

3. How many microbatches, and was that number set when the pipeline depth was? The bubble is (S−1)/(M+S−1) and both terms are choices. Section 7.

4. How long does a checkpoint take, and how often? 56 seconds every 60 is a run that checkpoints. Section 8.

5. What is the cluster's mean time between failures, not the node's? It divides by the node count. Section 9.

6. What is the slowest rank's step time, and is it the same rank each step? The step is the maximum. Section 10.

7. Is recomputation on, and does the activation budget say so? A factor of the layer count. Section 11.

8. Is the fabric's bisection number a full-bisection figure or a measured one? Four-to-one is a factor of four on exactly the collective's traffic. Section 12.

9. What is the cost per useful teraflop, and against what measured efficiency? Scale does not improve it; efficiency does. Section 13.

10. Which of the six properties does the team believe "we doubled the cluster" implies? Section 14 exists because the answer is the node count.

23. How This Appears In Real Engineering

A capacity-planning function meets section 13's structural problem: efficiency is not knowable until the cluster exists, so the decision is made on peak and the bill arrives on delivered work. The only defence is a measured efficiency from a comparable existing cluster, which is why organisations that train continuously make much better purchasing decisions than those that do it once.

A distributed-training team owns sections 5, 7 and 10 together, and section 21 is their week. The three terms interact, they are owned by different configuration files, and the measurement that separates them — the step split — is the one that has to be built before it is needed.

A network team owns section 12, and the awkwardness is that their number is honest. A full-bisection figure is a correct statement about the links; it is simply not the number an all-reduce experiences on an oversubscribed fabric, and reconciling the two requires a measurement at the cut during the collective, which is not a standard counter.

A storage or platform team owns section 8, and it is the one place in the chapter with a clean engineering answer: make the checkpoint write faster and the interval can shorten, which cuts section 9's lost work without costing more overhead. That is section 24's ground.

A reliability function owns section 9, and the finding they have to communicate is uncomfortable and elementary: component reliability targets that are excellent per node are inadequate per cluster by the node count, and the response is fault tolerance rather than better components.

24. Where CXL Actually Fits Here

This chapter has priced nine terms and CXL has appeared in none of them. That is deliberate, and this section says where it does belong.

The checkpoint tier — section 8. A 560 GB state write is a pure capacity-and-bandwidth problem with no latency sensitivity whatsoever, which is the exact shape 22.3 §8 identified as CXL's strength. Absorbing the write into a CXL-attached capacity tier and draining it to durable storage in the background turns 56 seconds of blocking into a much shorter stall — and a cheaper checkpoint permits a shorter interval, which cuts section 9's lost work at both ends.

Activation offload — section 11. Activations are written once and read once, in a known order, with a known deadline. That is section 8's streaming pattern from 22.4 applied to training, and it lets a batch or a pipeline depth exist that HBM alone would not permit.

Not the collective — section 6. An all-reduce is latency-sensitive, synchronous and all-to-all, and it belongs on the accelerator fabric. Nothing in this chapter suggests routing it over CXL, and 22.3 §9's 4.5x penalty is the reason.

Not the hot path — section 5. The compute half of the step reads from HBM, and 22.3 §7 priced what happens when it does not: twelve percent of peak.

The honest summary is narrow and real. CXL's role in a training cluster is capacity for the two large, sequential, deadline-bounded flows — checkpoints and activations — and that is a genuine contribution to sections 8, 9 and 11 rather than a headline about scaling.

25. Common Misconceptions

"Twice the nodes, twice the throughput." Only if the communication is free. Section 5, and section 21 is 11%.

"The all-reduce moves the gradient." It moves nearly twice the gradient, for more than two ranks. Section 6.

"Pipeline parallelism is free capacity." It costs (S−1)/(M+S−1) of every step. Section 7.

"Checkpoints are a storage detail." 56 seconds every 60 is 93% overhead. Section 8.

"Our nodes are reliable — a year between failures." A thousand of them fail every eight hours. Section 9.

"The average rank time is the step time." The maximum is. Section 10.

"Activations are small." Eighty layers at a batch of eight is 10 GB, or 128 MB with recomputation. Section 11.

"The fabric does 6400 Gbps." At one-to-one. At four-to-one it does 1600 on exactly the collective's traffic. Section 12.

"A bigger cluster is more efficient." Scale does not change cost per useful FLOP; efficiency does. Section 13.

"We doubled the cluster." One property of six. Section 14.

26. Interview Reasoning

Q. You double a training cluster and it gets 11% faster. Where do you look first?

The step split — compute against everything else. If efficiency fell from 70% to 41%, that ratio alone explains it, and every subsequent question is about which non-compute term grew. It is one measurement and it eliminates three of four possible answers, which is why it should exist before it is needed.

Q. How much does an all-reduce move across the fabric?

2(N−1)/N of the gradient — nearly twice for any real rank count, exactly once at two ranks. The follow-up worth knowing: the factor saturates, so the collective's cost grows with the gradient and the fabric rather than with the cluster size.

Q. Why does a longer checkpoint interval sometimes make a run slower?

Because it loses more work per failure. Shorter intervals cost overhead, longer ones cost lost work, and the two move in opposite directions — which means there is an optimum, and it depends on both the write bandwidth and the cluster's failure rate.

Q. Your nodes have a one-year mean time between failures. Is a thousand-node cluster reliable?

It fails every eight hours. Reliability divides by the node count, so excellent per-component numbers are inadequate at scale by exactly the scale factor — and the answer is not better components, it is a training stack that survives a node loss without restarting.

Q. Sixty-three ranks finish a step in 200 ms and one takes 260. What is the step time?

260, for all sixty-four. A synchronous step is a maximum, so the tail of the rank-time distribution is the cluster's throughput. The follow-up that matters: whether it is the same rank every time, because a fixed straggler is a hardware ticket and a rotating one is a data-balance problem.

Q. What does CXL contribute to a training cluster?

Capacity for the two large sequential flows — the checkpoint write and activation offload. Both are bandwidth-and-capacity problems with known deadlines and no latency sensitivity, which is what a capacity tier is good at. It contributes nothing to the collective or the hot path, and claiming otherwise is the mistake 22.3 §7 prices.

27. Exercises

1. Extend RTL 1 so the collective partially overlaps the backward pass, and find the overlap fraction at which 64 nodes beat 32.

2. Replace RTL 2's ring with a hierarchical collective — intra-node then inter-node — and find the node count at which it wins.

3. Combine RTL 3 and RTL 7: for a fixed activation budget, find the microbatch count that minimises the bubble.

4. Join RTL 4 and RTL 5 into one model and solve for the checkpoint interval that minimises total lost time.

5. Make RTL 5's failures independent per node rather than a mean rate, and compare the expected loss.

6. Extend RTL 6 to a distribution of rank times and show how the step time grows with the number of ranks at a fixed variance.

7. Add a CXL offload tier to RTL 7 with a write bandwidth, and find the batch size it makes possible.

8. Extend RTL 8 to a two-level fabric with different oversubscription at each level, and find which level binds.

9. Model section 21 end to end: a cluster doubled, with the fabric, bubble and straggler terms all moving.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the cluster it catches that the current mask calls scaling.

28. Summary

22.4 ended on a node holding a model it could not serve. This chapter is the same gap at rack scale: peak capability is bought and delivered work is earned, and nine terms stand between them.

N nodes give N times the compute only if talking is free. Half a step on the collective is 50% efficiency — 3200 of 6400 teraflops — with nothing broken and every node at full rate.

An all-reduce moves nearly twice the gradient. Eight ranks move 14 GB of an 8 GB gradient, sixty-four move 15, and the factor saturates — so the collective's cost tracks the gradient and the fabric, not the cluster size.

A pipeline is idle while it fills. Eight stages bubble 87% on one microbatch and 9% on sixty-four — and the microbatch count is nearly free while the stage count is forced.

A run has to write itself down. 560 GB at 80 Gbps is 56 seconds: 18% of a five-minute interval and 93% of a one-minute one.

And a thousand year-long nodes fail every eight hours. Three times a day, 90 minutes of work lost at an hourly checkpoint and 180 at a two-hourly one — the same trade as the checkpoint, pulling the other way.

A step finishes when its slowest rank finishes. One 260 ms rank among 200 ms ranks taxes every other rank 30%.

Training holds every layer's activations. 10,240 MB at a batch of eight, or 128 with recomputation — a factor of eighty from one flag, and the thing that bounds the microbatch count.

A full-bisection number is a link count. 6400 Gbps in principle, 1600 at four-to-one — a factor of four on exactly the traffic the collective generates, fixed at the moment the fabric was cabled.

Scale does not change the unit economics. 2000 per useful teraflop at sixty-four nodes and exactly 2000 at sixteen — efficiency is the only term that moves it, and the peak-priced figure says 1000.

Two more inclusive thresholds survived their mutations for want of an equality case — five now across three chapters, every one a stimulus gap, and every one needing an input constructed rather than swept.

Adding nodes is one property of six. The definition a purchase order uses called five of six clusters scaling when one was.

And CXL's genuine role here is narrow. Not the collective, not the hot path — the checkpoint write and the activation offload, two large sequential flows with deadlines and no latency sensitivity, which is exactly what a capacity tier is for.

Module 22 is complete. From coherent accelerator attach in 22.1, through working-set growth in 22.2, the GPU hierarchy in 22.3 and model serving in 22.4, to the rack here — the same finding at five scales: capacity is not throughput, and the arithmetic that separates them is always available before the hardware arrives.