Ethernet for AI Training Clusters

Chapter 23.1 §10 derived that 32 large flows over 32 equal-cost paths reach 28.4% of an uplink group. A collective is the traffic pattern that makes that number binding rather than pessimistic.

Row one is the model most people carry and row two is the one Chapter 23.1 §10 actually computed. Independent flows fill whatever uplink they land on, so the group delivers 100 Gb/s times the number of non-empty uplinks — and with 32 balls in 32 bins, 20.4 of the 32 uplinks are non-empty. A collective cannot do that: a participant that finishes early has nowhere to go, because the next step needs the data the slowest participant has not sent yet.

A collective's bandwidth is the minimum over its members, not the sum and not the mean. Chapter 23.1's figure was already the minimum-based one — which makes it exactly right here and conservative everywhere else.

The cost, on a real job.

And the mechanism is a single uplink.

One uplink out of 32 on one leaf out of 32 sets the pace of every accelerator in the job. That is what "bound to the minimum" means in hardware, and Section 8 is the derivation.

Then the result that decides the architecture, and it is not the one people expect.

All-to-all is ECMP's best case and the ring is its worst, on the same fabric, in the same job, for the same reason: the ring has N flows and the all-to-all has N². Section 10 is the arithmetic and Section 12 answers which of Chapter 23.1 §10's four remedies survives.

1. Scope — The Adversarial Case for Chapter 23.1's Arithmetic

Scope: what a collective communication pattern is, why it converts a statistical inefficiency into a hard bound, and what a fabric has to add.

Not in scope: training itself. What the accelerators compute between collectives is a different discipline. This chapter treats a training step as an alternating sequence — compute, then a collective — and asks only what the collective costs.

Three facts define the whole chapter.

Fact 3 is the one that makes it unbearable rather than merely inefficient. Chapter 23.1 §9 established that ECMP pins a flow to a path for the flow's life, because re-hashing reorders it. A collective's flows live for the duration of the training run — hours or weeks — so the assignment drawn in the first millisecond is the assignment for the whole job. Two ring links that collided at start-up collide for eighty-nine hours.

And one thing this chapter is careful not to claim.

Scheduling is not the answer. Chapter 22.2 priced a time-aware schedule at 19.48% of the link for a two-window gate list, and showed that per-stream separation makes the gate-control list as long as the stream count. At 1 024 endpoints that is 131 072 flops of schedule state per port with the shadow copy — 9.3 times Chapter 19.7 §19's entire MAC receive datapath — and it would have to be recomputed whenever a job's participant set changed. Section 14 rules it out with that arithmetic rather than with an opinion.

The chapter's arc.

2. What a Collective Is, and Why It Is Not Traffic

A collective is a communication pattern with a completion condition that every participant shares. That is the whole of it, and everything else follows.

The four that matter, with their flow counts.

Column three is the chapter. A ring all-reduce over 1 024 participants produces 1 024 flows; an all-to-all over the same 1 024 produces 1 047 552. Those two numbers sit at opposite ends of Chapter 23.1 §10's efficiency curve, and they are produced by the same job on the same hardware in the same second.

Column four is worth deriving once, because the factor of two surprises people.

A ring all-reduce is two passes. The first — reduce-scatter — moves each participant's chunk around the ring, accumulating; after N−1 steps every participant holds the final value of one chunk. The second — all-gather — circulates those final chunks; after another N−1 steps everyone holds all of them.

steps = 2(N − 1), bytes per step = S / N, total = 2(N−1)/N × S

At N = 1 024 and S = 8 GB that is 15.984 GB per participant, which is almost exactly 2S — and independent of N, which is the property that makes a ring the standard choice. Doubling the participant count does not increase the bytes each one moves.

What it does increase is the number of links in the chain.

Row four is the fact this chapter exists to price, and it is a property of the algorithm, not of the fabric. A ring is a chain; a chain's throughput is its weakest link's, and nothing the fabric does changes that.

And the traffic's shape is nothing like ordinary data-centre traffic.

Every row moves in the direction that makes ECMP worse, which is why a fabric built and measured on general traffic disappoints on a training job without anything having changed about the fabric.

3. RTL 1 — The Collective Package and the Participant Set

// ---------------------------------------------------------------------
// collective_pkg -- a collective as arithmetic over a participant set.
//
// The one constant that matters is not a rate or a size: it is the
// REDUCTION OPERATOR used to combine the participants' rates. Every
// other chapter in this track has summed or averaged. A collective
// takes the MINIMUM, and Section 20's refused property is what happens
// when a design forgets that.
// ---------------------------------------------------------------------
package collective_pkg;

  // ---- the fabric, from Chapter 23.1 -----------------------------------
  localparam int UPLINKS_PER_LEAF = 32;
  localparam int RATE_GBPS        = 100;
  localparam int LEAVES           = 32;      // leaves this job spans
  localparam int ACC_PER_LEAF     = 32;
  localparam int PARTICIPANTS     = LEAVES * ACC_PER_LEAF;   // 1024

  // ---- the collective ---------------------------------------------------
  typedef enum logic [2:0] {
    COLL_BROADCAST  = 3'd0,
    COLL_ALLREDUCE  = 3'd1,   // ring: N flows -- ECMP's worst case
    COLL_ALLGATHER  = 3'd2,
    COLL_ALLTOALL   = 3'd3,   // N(N-1) flows -- ECMP's best case
    COLL_REDUCE     = 3'd4
  } collective_e;

  // Flows leaving ONE leaf, which is what the uplink group sees.
  function automatic int flows_per_leaf(collective_e c);
    case (c)
      COLL_ALLREDUCE, COLL_ALLGATHER: return ACC_PER_LEAF;
      COLL_ALLTOALL:                  return ACC_PER_LEAF * (PARTICIPANTS - 1);
      default:                        return ACC_PER_LEAF;
    endcase
  endfunction

  // Ring all-reduce: 2(N-1) steps of S/N bytes -- total is independent
  // of N, which is why a ring is the default choice.
  function automatic int bytes_per_participant_mb(collective_e c, int s_mb);
    case (c)
      COLL_ALLREDUCE: return (2 * (PARTICIPANTS - 1) * s_mb) / PARTICIPANTS;
      COLL_ALLGATHER,
      COLL_ALLTOALL:  return ((PARTICIPANTS - 1) * s_mb) / PARTICIPANTS;
      default:        return s_mb;
    endcase
  endfunction

  // ---- THE reduction operator ------------------------------------------
  // Every other chapter combined per-port quantities with a sum or a
  // mean. A collective does not get to.
  typedef enum logic [1:0] {
    RED_SUM  = 2'd0,   // wrong for a collective
    RED_MEAN = 2'd1,   // wrong for a collective
    RED_MIN  = 2'd2    // the only correct one
  } rate_reduction_e;

  localparam rate_reduction_e COLLECTIVE_REDUCTION = RED_MIN;

  // Chapter 23.1 Section 10's efficiency curve, as a lookup rather than a
  // simulation. Indexed by log2 of flows-per-uplink, values in per cent.
  //   1 flow/uplink -> 28.4 ; 2 -> 37.0 ; 4 -> 46.2 ; 8 -> 55.9
  //   20 -> 67.2 ; 100 -> 82.4
  localparam int EFF_PCT [6] = '{28, 37, 46, 56, 67, 82};

  function automatic int efficiency_pct(int flows, int uplinks);
    automatic int fpu = flows / uplinks;
    if (fpu <= 1)   return EFF_PCT[0];
    if (fpu <= 2)   return EFF_PCT[1];
    if (fpu <= 4)   return EFF_PCT[2];
    if (fpu <= 8)   return EFF_PCT[3];
    if (fpu <= 20)  return EFF_PCT[4];
    return EFF_PCT[5];
  endfunction

  // Section 12's remedies, as an enumeration, so a design has to name
  // which one it is relying on.
  typedef enum logic [2:0] {
    FIX_NONE         = 3'd0,
    FIX_STRIPE       = 3'd1,   // more, smaller flows
    FIX_FLOWLET      = 3'd2,   // re-hash at a gap
    FIX_LOAD_AWARE   = 3'd3,   // choose the least-loaded uplink
    FIX_SPRAY        = 3'd4,   // per-packet, plus a reorder buffer
    FIX_HIERARCHICAL = 3'd5    // reduce inside the leaf first
  } fabric_fix_e;

endpackage

Classification: a parameter package whose central declaration is a reduction operator rather than a number.

What it teaches: that COLLECTIVE_REDUCTION = RED_MIN is the only line in the package that is not arithmetic. Every earlier chapter in Modules 19 to 23 combined per-port quantities by summing — Chapter 19.7's counters, Chapter 23.1's bisection — or by averaging. A collective does not get to choose: its completion condition is that every participant has finished, so its rate is the minimum and every sum in the track's habits is now the wrong operator.

And it teaches that flows_per_leaf differs between collectives by a factor of a thousand. A ring all-reduce and an all-gather each produce one flow per accelerator — 32 per leaf. An all-to-all produces 32 × 1 023 = 32 736. Those land at opposite ends of Chapter 23.1 §10's curve, and a job runs both.

Deliberately simplified: EFF_PCT is a six-entry lookup where the underlying quantity is a smooth function of flows per uplink; the table rounds to whole per cent and the function is not linear between entries. bytes_per_participant_mb uses integer megabytes, so an 8 GB buffer is 8 192 and the division by 1 024 is exact by luck rather than by design. PARTICIPANTS is a compile-time constant where a real job's participant set changes when a node fails and the job restarts on a different set. And there is no notion of the ring's ordering — which accelerator is whose neighbour — even though Section 17 shows that ordering is the single most valuable thing a job scheduler controls.

Production implication: the missing ring ordering is the cheapest large win available and it belongs to the job scheduler rather than the fabric. A ring whose consecutive participants are on the same leaf never crosses the fabric for most of its links; a ring assigned in arbitrary order crosses it for all of them. With 32 accelerators per leaf and a ring ordered leaf-by-leaf, 31 of every 32 ring links stay inside a leaf switch and only 1 crosses the fabric — which is Section 17's hierarchical result arriving as a placement decision rather than an algorithm change. Give the scheduler the topology, and the fabric's hardest problem stops being posed.

// ---------------------------------------------------------------------
// slowest_member_tracker -- combine per-participant rates with the
// correct operator, and report how wrong the usual one would be.
//
// The block exists to make the gap between the mean and the minimum a
// measured quantity rather than an argument.
// ---------------------------------------------------------------------
module slowest_member_tracker
  import collective_pkg::*;
#(
  parameter int MEMBERS = 32        // ring links leaving one leaf
)(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        sample,
  input  logic [15:0] member_gbps [MEMBERS],

  output logic [15:0] min_gbps,
  output logic [15:0] mean_gbps,
  output logic [15:0] max_gbps,
  output logic [15:0] collective_gbps,      // = min, always
  output logic [7:0]  slowest_member,
  output logic [15:0] mean_over_min_x10,
  output logic        mean_would_overstate,
  output logic [31:0] c_samples,
  output logic [31:0] c_slowest_unchanged
);
  logic [31:0] total;
  logic [7:0]  prev_slowest;

  always_comb begin
    min_gbps       = 16'hFFFF;
    max_gbps       = 16'd0;
    total          = 32'd0;
    slowest_member = 8'd0;
    for (int i = 0; i < MEMBERS; i++) begin
      total = total + 32'(member_gbps[i]);
      if (member_gbps[i] < min_gbps) begin
        min_gbps       = member_gbps[i];
        slowest_member = 8'(i);
      end
      if (member_gbps[i] > max_gbps) max_gbps = member_gbps[i];
    end
    mean_gbps = 16'(total / MEMBERS);

    // The collective's rate is the minimum. Not the sum, not the mean.
    collective_gbps = min_gbps;

    mean_over_min_x10 = (min_gbps == 0) ? 16'hFFFF
                      : (mean_gbps * 16'd10) / min_gbps;
    mean_would_overstate = (mean_gbps > min_gbps);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_samples <= '0; c_slowest_unchanged <= '0; prev_slowest <= 8'd0;
    end else if (sample) begin
      c_samples <= c_samples + 32'd1;
      // Section 1's fact 3: an ECMP assignment is pinned for the flow's
      // life, so the SAME member is slowest for the whole job. A
      // stable index is the signature of a persistent collision.
      if (slowest_member == prev_slowest)
        c_slowest_unchanged <= c_slowest_unchanged + 32'd1;
      else
        c_slowest_unchanged <= '0;
      prev_slowest <= slowest_member;
    end
  end
endmodule

Classification: a reduction with three operators computed and only one used, so the difference is reportable.

What it teaches: that mean_over_min_x10 is the number to publish and it is never 1.0. With 32 ring links whose rates are set by ECMP, the mean is 28.4 Gb/s of usable share per link on average and the minimum is what the job runs at — and quoting the mean overstates the job's rate by the imbalance factor, which Chapter 23.1 §10 put at 3.52× for 32 flows on 32 paths.

And it teaches that c_slowest_unchanged is the diagnostic that distinguishes the two failure modes. A collective that is slow because of transient congestion has a slowest member that moves between samples. One that is slow because of a pinned hash collision has the same index, sample after sample, for hours — which is Chapter 23.1 §9's stickiness made visible in one counter.

Deliberately simplified: member_gbps is an instantaneous rate where a collective's steps are discrete and the useful quantity is step completion time, not bandwidth. The combinational loop over 32 members is a 32-way comparator tree, fine at this scale and not at 1 024. c_slowest_unchanged resets to zero on any change, so a slowest member that alternates between two indices reads as unstable when it is in fact two pinned collisions. And the block sees only one leaf's members — the job's true minimum is over all 1 024, which no leaf can compute.

Production implication: that last limitation is the one that decides where this block belongs. The collective's rate is the minimum over 1 024 participants and every leaf sees 32 of them, so a per-leaf minimum is a local minimum and can be far above the global one. Reporting it as the collective's rate is precisely Chapter 22.2 §20's twelfth taxonomy group — a guarantee that is a relation between nodes, evaluated at one. The global minimum has to be reduced across leaves, which is itself a small all-reduce, and the honest implementation runs it in the job's control plane at a rate of a few hertz rather than trying to compute it in a switch.

4. The Minimum, Not the Mean: Two Models of the Same 32 Flows

Take Chapter 23.1 §10's exact set-up — 32 large flows, 32 equal-cost uplinks, every link healthy — and run it under two models.

Model A — independent greedy flows. Each flow takes whatever its uplink will give. An uplink with k flows delivers 100 Gb/s regardless of k; an uplink with zero flows delivers nothing. So the group's aggregate is 100 Gb/s times the number of non-empty uplinks.

non-empty uplinks = n(1 − (1 − 1/n)^m)

Model B — a collective. Every participant is held to the same rate, because a participant that finishes its step early cannot start the next one: the next step needs data the slowest participant has not sent. So all 32 flows run at rate r, the busiest uplink carries max(k) × r ≤ 100, and:

r = 100 ÷ max(k), aggregate = 32 × r

The same flows, the same fabric, the same instant — and a factor of 2.25 between the two answers.

Chapter 23.1's 28.4% was computed under the equal-rate assumption. That assumption is conservative for ordinary traffic and exactly right for a collective, which is why this chapter can use that chapter's table unmodified.

Three consequences, and the third is the one that changes how a fabric is measured.

One — spare capacity is unusable. Model A's 63.9% comes from uplinks running full while others run empty. In a collective the empty ones stay empty, because the flow that would have used them is pinned elsewhere and the flows that could use them are all throttled to the minimum. The fabric has 2 291 Gb/s of idle uplink capacity and nothing that can reach it.

Two — the mean is not merely optimistic, it is uninformative. Averaged over the 32 ring links, the per-link share is 100 Gb/s; the per-link rate is 28.4. No participant experiences the mean and no participant experiences anything above the minimum.

Three — and improving one participant can make the collective slower. If a fabric change speeds up 31 links and slows one, the mean rises and the collective's rate falls. That is not a hypothetical: a load-balancing scheme that reassigns flows to even out utilisation can move a flow onto the busiest uplink, improving the aggregate and lengthening every step. Section 20's refused property is exactly this observation written as an assertion.

One calibration, so the two models are not confused later.

Row four against row five is why a fabric that benchmarks well can disappoint on a training job without anything having changed. The benchmark ran independent streams; the job runs a chain.

5. RTL 2 — The Ring All-Reduce Model

// ---------------------------------------------------------------------
// ring_allreduce_model -- 2(N-1) steps, S/N bytes each, and a step that
// completes only when its slowest link has.
//
// The block's purpose is to expose the two quantities a collective
// cares about and general traffic does not: the step's completion time,
// and which link set it.
// ---------------------------------------------------------------------
module ring_allreduce_model
  import collective_pkg::*;
#(
  parameter int MEMBERS   = 32,        // ring links leaving this leaf
  parameter int S_MB      = 8192       // gradient buffer, megabytes
)(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        start,
  input  logic [15:0] link_gbps [MEMBERS],

  output logic [31:0] step_bytes_kb,
  output logic [31:0] steps_total,
  output logic [31:0] step_time_us,
  output logic [31:0] collective_time_ms,
  output logic [31:0] ideal_time_ms,
  output logic [15:0] slowdown_x100,
  output logic [7:0]  pacing_member,
  output logic        running,
  output logic [31:0] c_steps_done
);
  localparam int N = PARTICIPANTS;

  logic [15:0] slowest;

  always_comb begin
    // 2(N-1) steps of S/N bytes. The total is 2(N-1)/N x S and is
    // independent of N, which is the ring's defining property.
    steps_total   = 32'(2 * (N - 1));
    step_bytes_kb = 32'((S_MB * 1024) / N);

    slowest = 16'hFFFF;
    pacing_member = 8'd0;
    for (int i = 0; i < MEMBERS; i++)
      if (link_gbps[i] < slowest) begin
        slowest = link_gbps[i]; pacing_member = 8'(i);
      end

    // A step completes when the SLOWEST link has delivered its chunk.
    // kB x 8 / (Gb/s) gives microseconds directly.
    step_time_us = (slowest == 0) ? 32'hFFFF_FFFF
                 : (step_bytes_kb * 8) / 32'(slowest);

    collective_time_ms = (step_time_us * steps_total) / 1000;
    ideal_time_ms      = ((step_bytes_kb * 8) / 32'(RATE_GBPS))
                         * steps_total / 1000;

    slowdown_x100 = (ideal_time_ms == 0) ? 16'd100
                  : 16'((collective_time_ms * 100) / ideal_time_ms);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n)      begin running <= 1'b0; c_steps_done <= '0; end
    else if (start)  begin running <= 1'b1; c_steps_done <= '0; end
    else if (running) begin
      c_steps_done <= c_steps_done + 32'd1;
      if (c_steps_done + 1 >= steps_total) running <= 1'b0;
    end
  end
endmodule

Classification: a step-time model whose only interesting input is a minimum.

What it teaches: that step_time_us divides by slowest and never by the mean, and that this single choice is the difference between a model that predicts a training job and one that does not. Every link in the ring transfers the same S/N bytes in every step; they all start together and the step is not over until the last one finishes. The 31 fast links sit idle waiting, which is Section 4's model B in one expression.

And it teaches that steps_total is 2(N−1) while step_bytes_kb is S/N, so their product is independent of N. A ring over 1 024 participants and a ring over 64 move the same bytes per participant — 15.984 GB against 15.750 for S = 8 GB — which is why the ring is chosen and why its flow count, not its byte count, is what hurts.

Deliberately simplified: the model assumes every link's rate is constant for the whole collective, which Chapter 23.1 §9's stickiness makes nearly true and which a congestion controller makes only approximately so. There is no propagation or per-step synchronisation overhead, which at 2 046 steps and a three-hop 1.635 µs fabric latency adds 3.3 ms — small against 4.5 seconds and not zero. MEMBERS is the links leaving one leaf, not the ring's 1 024, so pacing_member is a local index. And S_MB is fixed, where real training overlaps a collective with the backward pass and issues it in buckets rather than as one transfer.

Production implication: bucketing is the optimisation that makes the model's headline number soft, and it is worth being precise about what it does and does not fix. Splitting an 8 GB gradient into 32 buckets of 256 MB lets each bucket's all-reduce start as soon as its layer's gradients are ready, overlapping communication with the backward pass — so the exposed time is the part not hidden behind compute, not the whole 4.5 seconds. What bucketing does not change is the rate: each bucket still runs at the ring's slowest link, so a 3.52× slowdown on a collective that was 40% exposed becomes a 3.52× slowdown on 40% of the step. It reduces the damage and does not address the cause.

6. A Ring Is a Chain, and a Chain Is Its Slowest Link

Section 5's model divides by the minimum. This section is why no amount of fabric cleverness changes that.

The ring's completion condition, stated exactly.

Step s of a ring all-reduce cannot begin at any participant until step s−1 has completed at every participant — because the chunk a participant reduces in step s is the chunk its predecessor sent in step s−1.

That is a property of the algorithm. It is true on a perfect fabric, on an imperfect one, and on no fabric at all. The ring is a data dependency, and a data dependency does not negotiate.

Three things follow, in increasing order of how much they hurt.

One — the fast participants idle, and the idle time is exactly the imbalance.

Every step, 31 links wait for one. Over 2 046 steps, 1.576 ms of idling per step is 3.22 seconds — which is precisely Section 8's lost time, and it is spent by hardware that is otherwise perfectly healthy.

Two — the ring's rate is the minimum over all 1 024 links, not over one leaf's 32. Each leaf's 32 outbound ring links have their own local minimum; the job's rate is the minimum of the 32 local minima, which is lower again. That is the difference between a per-leaf measurement and a job-level one, and it is why Section 3's tracker is explicit that its minimum is local.

Three — and the ring cannot route around the slow link. A general network recovers from a congested path by moving traffic elsewhere; a ring link connects two specific participants and there is no other pair of participants that can do its work. Chapter 23.1 §5's 32 equal-cost paths are equal-cost for that ring link, and moving it is Chapter 23.1 §9's re-hash, which reorders.

The comparison that makes it concrete.

Row four is the one that turns an inefficiency into an outage-shaped event. Chapter 23.1 §9 established that a flow's path is pinned for the flow's life; a collective's flows live as long as the job, so an unlucky draw in the first millisecond of a two-week run is an unlucky draw for two weeks.

7. RTL 3 — The Stall Accountant

// ---------------------------------------------------------------------
// stall_accountant -- convert an imbalance into the currency a training
// programme actually budgets in: accelerator-seconds.
//
// Section 6's idle time is not a fabric metric. It is hardware that
// was bought, powered and scheduled, waiting. This block prices it.
// ---------------------------------------------------------------------
module stall_accountant
  import collective_pkg::*;
#(
  parameter int ACCELERATORS = PARTICIPANTS     // 1024
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [31:0] ideal_time_ms,
  input  logic [31:0] actual_time_ms,
  input  logic [31:0] steps_per_run,
  input  logic        collective_done,

  output logic [31:0] lost_ms_per_collective,
  output logic [31:0] accel_seconds_lost,
  output logic [31:0] accel_hours_per_run,
  output logic [31:0] wall_hours_per_run,
  output logic [15:0] slowdown_x100,
  output logic [15:0] exposed_pct,
  output logic [31:0] c_collectives,
  output logic [31:0] c_accel_seconds_total
);
  always_comb begin
    lost_ms_per_collective = (actual_time_ms > ideal_time_ms)
                           ? (actual_time_ms - ideal_time_ms) : 32'd0;

    // The conversion that makes the number arguable in a budget
    // meeting: every accelerator in the job waits for the whole of it.
    accel_seconds_lost = (lost_ms_per_collective * 32'(ACCELERATORS))
                         / 1000;

    wall_hours_per_run = (lost_ms_per_collective * steps_per_run)
                         / (1000 * 3600);
    accel_hours_per_run = wall_hours_per_run * 32'(ACCELERATORS);

    slowdown_x100 = (ideal_time_ms == 0) ? 16'd100
                  : 16'((actual_time_ms * 100) / ideal_time_ms);

    // Bucketing hides part of the collective behind the backward pass.
    // What it hides is TIME, not RATE -- Section 5's production note.
    exposed_pct = 16'd100;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_collectives <= '0; c_accel_seconds_total <= '0;
    end else if (collective_done) begin
      c_collectives         <= c_collectives + 32'd1;
      c_accel_seconds_total <= c_accel_seconds_total + accel_seconds_lost;
    end
  end
endmodule

Classification: a unit conversion, and the unit is the point.

What it teaches: that accel_seconds_lost is the only figure in this chapter that a programme will act on. A fabric engineer reports 28.4% ECMP efficiency; nobody outside the network team knows what that costs. The same fact expressed as 3 301 accelerator-seconds per all-reduce, 91 705 accelerator-hours over a 100 000-step run is a number with a price attached, and it is the same measurement multiplied by a constant.

And it teaches that exposed_pct is hard-wired to 100 and should not be. A real training step overlaps the collective with the backward pass by issuing it in buckets; only the unhidden part is lost. The block leaves the field at 100 deliberately, as an honest worst case, and a design that fills it in correctly reduces every number below it proportionally without touching the slowdown.

Deliberately simplified: all arithmetic is integer milliseconds, so a collective under a millisecond reads as zero. ACCELERATORS is the whole job where a failure that affects one leaf still stalls the whole job — correct here, and wrong for a job partitioned into independent sub-jobs. steps_per_run is an input with no notion of a run that is stopped early, which most are. And c_accel_seconds_total is a 32-bit accumulator that at 3 301 per collective wraps after 1.3 million collectives, which a long run reaches.

Production implication: the 32-bit wrap is real at this scale and it is worth choosing the width deliberately rather than by habit. A 100 000-step run accumulates 3.3 × 10⁸ accelerator-seconds of loss, which fits; a fleet-level accumulator across a thousand runs does not. The general rule the track has arrived at repeatedly — Chapter 19.7 §8's saturating counters, Chapter 22.1 §18's 64-second wrap — applies here with a new twist: the quantity being accumulated is a product of two large numbers, so its width is the sum of their widths and not either one's.

8. One Unlucky Uplink Stalls a Thousand Accelerators

The chapter's central quantitative claim, derived rather than asserted.

Set-up. A 1 024-accelerator ring all-reduce, 32 accelerators on each of 32 leaves, 32 uplinks per leaf at 100 Gb/s, an 8 GB gradient buffer. Every link in the fabric is up and every forwarding table is correct.

Each leaf sends 32 ring flows, ECMP hashes them across its 32 uplinks, and the busiest uplink carries k of them.

The slowdown is exactly k. Not approximately, not asymptotically — the ring runs at 100/k Gb/s and takes k times as long, because every other link is waiting for that one.

And k = 1 almost never happens. Thirty-two balls in thirty-two bins gives a busiest bin of 1 with probability 1.8 × 10⁻¹³; the expected maximum is 3.52, and the distribution is what matters:

So 87.8% of randomly assigned rings run at a third or a quarter of line rate, and the fabric reports every link healthy throughout.

The cost, expressed three ways.

Eighty-nine hours of wall clock on a run that should have taken thirty-six — and the cause is that thirty-two flows were assigned to thirty-two paths by a hash.

Two framings that make the size of this clear.

First, against the fabric's own capacity. The job is using 909 Gb/s of a 3 200 Gb/s uplink group per leaf, so 2 291 Gb/s per leaf — 73.3 Tb/s across the 32 leaves — is idle, powered, and unreachable. That is Chapter 23.1 §19's gap arriving with a workload attached.

Second, against what the accelerators cost. A 3.52× slowdown on the communication phase of a job whose accelerators are the most expensive items in the building is not a networking problem that the network team owns; it is a capital-efficiency problem that happens to have a networking cause — and Section 7's accelerator-hours is the unit that makes that legible.

The fabric is not congested. It is 71.6% idle, and the job is running at 28.4% of line rate, and both statements are true at the same instant.

9. RTL 4 — The All-to-All Model

// ---------------------------------------------------------------------
// all_to_all_model -- the same job, the same fabric, the same second,
// and the opposite end of Chapter 23.1 Section 10's efficiency curve.
//
// A ring produces N flows. An all-to-all produces N(N-1). That single
// difference moves the uplink group from 28.4% to 93.8%, and nothing
// about the fabric changed.
// ---------------------------------------------------------------------
module all_to_all_model
  import collective_pkg::*;
#(
  parameter int S_MB = 1024          // per-participant dispatch volume
)(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        start,

  output logic [31:0] flows_per_leaf_o,
  output logic [31:0] flows_per_uplink,
  output logic [15:0] efficiency_pct,
  output logic [31:0] egress_mb,
  output logic [31:0] time_ms,
  output logic [31:0] ideal_ms,
  output logic [15:0] slowdown_x100,
  output logic        is_ecmp_best_case,
  output logic [31:0] ring_flows_same_job
);
  always_comb begin
    // 32 accelerators per leaf, each sending to N-1 peers.
    flows_per_leaf_o = 32'(ACC_PER_LEAF) * 32'(PARTICIPANTS - 1);
    flows_per_uplink = flows_per_leaf_o / 32'(UPLINKS_PER_LEAF);

    efficiency_pct = 16'(efficiency_pct_fn(int'(flows_per_leaf_o),
                                           UPLINKS_PER_LEAF));

    egress_mb = 32'((S_MB * (PARTICIPANTS - 1)) / PARTICIPANTS);

    ideal_ms = (egress_mb * 8 * 1000) / (32'(RATE_GBPS) * 1000);
    time_ms  = (efficiency_pct == 0) ? 32'hFFFF_FFFF
             : (ideal_ms * 100) / 32'(efficiency_pct);

    slowdown_x100 = (ideal_ms == 0) ? 16'd100
                  : 16'((time_ms * 100) / ideal_ms);

    // With a thousand flows per uplink the imbalance is small, which
    // is the whole point of the comparison.
    is_ecmp_best_case = (flows_per_uplink > 32'd100);

    // The SAME job's ring all-reduce, for contrast.
    ring_flows_same_job = 32'(ACC_PER_LEAF);
  end

  function automatic int efficiency_pct_fn(int flows, int uplinks);
    return efficiency_pct(flows, uplinks);
  endfunction
endmodule

Classification: the same fabric model with one input changed by a factor of a thousand.

What it teaches: that flows_per_leaf_o and ring_flows_same_job are outputs of the same block for a reason. A training job that uses expert parallelism runs both collectives: an all-to-all to dispatch tokens to experts, and a ring all-reduce over the gradients. They produce 32 736 and 32 flows per leaf respectively, land at 93.8% and 28.4% efficiency, and run on the same uplinks within the same training step. Any statement about "the fabric's efficiency for AI traffic" that does not say which collective is meaningless.

And it teaches why the all-to-all is comfortable. With 1 023 flows per uplink, the balls-in-bins imbalance is 1 091 on the busiest against a mean of 1 023 — a ratio of 1.067 — because the relative fluctuation of a sum of m independent assignments falls as 1/√m. The law of large numbers is the all-to-all's load balancer, and it is free.

Deliberately simplified: the model assumes all N(N−1) flows are active simultaneously, which for a real all-to-all is true within a step and not across one. efficiency_pct comes from the package's six-entry lookup, so 32 736 flows and 3 200 flows both return 82 — the true figure at 32 736 is 93.8%, and the lookup's top bucket is too coarse. egress_mb ignores the message the participant sends to itself, correctly. And there is no notion of incast: every participant receives from N−1 peers simultaneously, which is a receive-side problem the model does not have.

Production implication: incast is the all-to-all's real hazard and it is on the other side of the link from everything in this chapter. 1 023 peers each sending to one receiver at once is Chapter 14.1's congestion at its sharpest: the receiving leaf's downlink to that accelerator is a single 100 Gb/s port with 1 023 senders aimed at it, and the buffer that absorbs the overshoot is the switch's, not the fabric's. An all-to-all's efficiency on the uplink side is 93.8% and its failure mode is entirely on the downlink, which is why a fabric tuned for all-reduce and a fabric tuned for all-to-all are tuned in different places.

10. All-to-All Is ECMP's Best Case and the Ring Is Its Worst

Two collectives, one job, one fabric, and a factor of 3.3 in efficiency between them.

Row four explains rows one to five in one line. The imbalance of m balls in n bins is driven by the relative fluctuation of a bin's count, which scales as 1/√(m/n). At one flow per uplink that fluctuation is of the same size as the mean; at a thousand flows per uplink it is about three per cent.

imbalance ≈ 1 + √(2 ln n ÷ (m/n))

The closed form tracks the simulation within about 8% across three orders of magnitude, which is enough to use it as a design rule: an uplink group needs of order 2 ln n flows per uplink to be within a few per cent of balanced, and at n = 32 that is about seven — so 222 flows per leaf, against the ring's 32.

The ring is not unlucky. It is structurally seven times short of the flow count its uplink group needs.

Which reframes the problem usefully. The question is not why is ECMP bad at all-reduce — ECMP is behaving exactly as specified. The question is why a collective with N participants produces only N flows, and the answer is that a ring is a one-dimensional decomposition of an all-reduce. A different decomposition produces a different flow count, which is Section 12's first remedy and Section 17's better one.

And the two collectives fail in different places, which matters for tuning.

Row five is worth stating plainly. Chapter 23.1's whole analysis was about an uplink group and a hash; the all-to-all's difficulty is neither. A fabric optimised entirely on that chapter's arithmetic will handle all-reduce better and all-to-all no differently, because the two failures do not share a mechanism.

11. RTL 5 — The Striping Controller

// ---------------------------------------------------------------------
// striping_controller -- Chapter 23.1 Section 10's first remedy, applied to
// a ring link.
//
// Split each ring link into c connections. The ring link's bandwidth is
// then the SUM of its connections' fair shares, and the collective's
// rate is the minimum over ring links. Section 12 has the measured
// curve; this block is the mechanism and its ordering hazard.
// ---------------------------------------------------------------------
module striping_controller
  import collective_pkg::*;
#(
  parameter int MEMBERS = 32,     // ring links leaving this leaf
  parameter int STRIPES = 8       // connections per ring link
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        chunk_start,
  input  logic [31:0] chunk_bytes,
  input  logic [15:0] stripe_gbps [MEMBERS*STRIPES],

  output logic [31:0] stripe_bytes,
  output logic [15:0] member_gbps [MEMBERS],
  output logic [15:0] collective_gbps,
  output logic [7:0]  pacing_member,
  output logic        reassembly_required,
  output logic [31:0] reorder_depth_bytes,
  output logic [15:0] connections_total,
  output logic [31:0] c_chunks
);
  always_comb begin
    stripe_bytes      = chunk_bytes / STRIPES;
    connections_total = 16'(MEMBERS) * 16'(STRIPES);

    collective_gbps = 16'hFFFF;
    pacing_member   = 8'd0;
    for (int m = 0; m < MEMBERS; m++) begin
      automatic logic [31:0] acc = 32'd0;
      for (int s = 0; s < STRIPES; s++)
        acc = acc + 32'(stripe_gbps[m*STRIPES + s]);
      member_gbps[m] = 16'(acc);
      if (member_gbps[m] < collective_gbps) begin
        collective_gbps = member_gbps[m];
        pacing_member   = 8'(m);
      end
    end

    // The cost that is not bandwidth: a chunk split across STRIPES
    // independent paths arrives out of order, and the receiver must
    // hold the gap. This is NOT the same as Chapter 23.1 Section 9's
    // reordering hazard -- these are separate connections carrying
    // disjoint byte ranges, so nothing is reordered WITHIN a
    // connection. What is required is reassembly, not reordering.
    reassembly_required = (STRIPES > 1);
    // Worst case: STRIPES-1 stripes complete and one lags by the
    // fabric's path skew.
    reorder_depth_bytes = stripe_bytes * 32'(STRIPES - 1);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n)           c_chunks <= '0;
    else if (chunk_start) c_chunks <= c_chunks + 32'd1;
  end
endmodule

Classification: a striping controller whose most important comment is that its hazard is not the one people expect.

What it teaches: that striping a ring link across c connections is not the reordering hazard Chapter 23.1 §9 warned about. That hazard was splitting a single flow's packets across paths, which delivers one byte stream out of order. Here each connection carries a disjoint byte range, so nothing is reordered within a connection — what the receiver must do is reassemble c independent, individually ordered streams into one buffer. That is a memory problem, not a transport one, and it is why striping is the cheapest of the remedies.

And it teaches that member_gbps is a sum while collective_gbps is a minimum, in the same block. The two reductions are correct at different levels: a ring link's bandwidth is the sum of its stripes because they run in parallel; the collective's rate is the minimum over ring links because they run in series. Getting either one wrong by habit is Section 20's refused class.

Deliberately simplified: stripe_gbps is an input where the actual share depends on how many other connections landed on the same uplink — the block consumes the result of the assignment rather than modelling it. The nested loop is MEMBERS × STRIPES = 256 additions combinationally, which is a model. reorder_depth_bytes assumes at most one lagging stripe where a real skew distributes across all of them. And there is no connection-establishment cost, which at 256 connections per leaf and a job that restarts on every failure is not free.

Production implication: the connection count is what limits this remedy in practice and it multiplies faster than people expect. Striping every ring link by 8 takes a 1 024-participant job from 1 024 connections to 8 192; striping by 32 takes it to 32 768. Each connection costs state in the NIC — a send queue, a receive queue, a congestion-control context, and a reassembly buffer — and a high-rate NIC's connection cache is finite. Exceeding it causes context thrashing, whose symptom is a throughput collapse that looks exactly like the fabric problem striping was deployed to fix. Size the stripe count against the NIC's cache, not against the efficiency curve, and measure the cache's hit rate before and after.

12. Which of Chapter 23.1's Four Remedies Survives

Chapter 23.1 §10 offered four. This section runs each against a 1 024-participant ring all-reduce and reports what it delivers.

Remedy 1 — more, smaller flows (striping). Measured, with each ring link split into c connections, the ring's rate being the minimum over ring links of the sum of their stripes' shares:

It survives, and it is the best return per unit of effort in the chapter. Eight connections per ring link — a library setting — takes the all-reduce from 4.503 s to 1.659 s, recovering 88% of the gap. The cost is 8 192 connections for the job and the NIC cache that must hold them.

Remedy 2 — flowlet switching. Chapter 23.1 §9 described it as re-hashing after a gap long enough to prevent reordering. A ring all-reduce has 2 046 natural gaps — one per step — so every step is a flowlet boundary and the mechanism fires constantly.

It survives in a weakened and still valuable form. It does not improve the expected rate — each step's minimum is still set by that step's maximum — but it converts a systematic penalty into a statistical one. A job that drew k = 6 no longer runs at 16.7 Gb/s for two weeks; it runs at the average of many draws. For a fleet, that removes the tail of unlucky jobs, which is worth a great deal and is not a speed-up.

Remedy 3 — load-aware path selection. This is the one the collective case rewards most.

It survives and it is transformative, for a reason specific to collectives: the flows are long-lived, equal-sized and start simultaneously, which is precisely the condition under which a greedy least-loaded assignment is optimal rather than merely better. General traffic — short, heavy-tailed, independently arriving — is the case where load-aware selection is hard to get right. A collective hands it the easy case.

Remedy 4 — larger link groups. Chapter 23.1 §10 already noted it does not remove the hash, and at this scale it is worse than neutral: doubling a leaf's uplinks from 32 to 64 halves the flows per uplink from 1.0 to 0.5, and Section 10's curve says the imbalance rises. It does not survive.

Summary, ranked by delivered rate.

Rows one and two reach the same number by completely different routes, and row two costs nothing at all, which is why Section 17 argues it should be tried first.

13. RTL 6 — Load-Aware Path Selection

// ---------------------------------------------------------------------
// load_aware_selector -- choose the least-loaded uplink at flow start,
// instead of hashing.
//
// Chapter 23.1 Section 9's stickiness is PRESERVED: the choice is made once,
// at flow establishment, and never revisited. What changes is only HOW
// the choice is made, so nothing is reordered and the correctness
// argument is untouched.
// ---------------------------------------------------------------------
module load_aware_selector
  import collective_pkg::*;
#(
  parameter int PATHS = UPLINKS_PER_LEAF
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        flow_start,
  input  logic [31:0] flow_hash,          // the fallback
  input  logic [15:0] declared_gbps,      // what the flow will offer
  input  logic        flow_end,
  input  logic [7:0]  ending_path,
  input  logic        load_aware_enable,

  output logic [7:0]  chosen_path,
  output logic [15:0] path_load_gbps [PATHS],
  output logic [15:0] busiest_gbps,
  output logic [15:0] lightest_gbps,
  output logic [15:0] imbalance_x10,
  output logic        choice_was_hash,
  output logic [31:0] c_flows_placed,
  output logic [31:0] c_perfect_placements,
  output logic        sticky_preserved
);
  logic [7:0] lightest_idx;

  always_comb begin
    lightest_idx  = 8'd0;
    lightest_gbps = 16'hFFFF;
    busiest_gbps  = 16'd0;
    for (int i = 0; i < PATHS; i++) begin
      if (path_load_gbps[i] < lightest_gbps) begin
        lightest_gbps = path_load_gbps[i];
        lightest_idx  = 8'(i);
      end
      if (path_load_gbps[i] > busiest_gbps) busiest_gbps = path_load_gbps[i];
    end

    choice_was_hash = !load_aware_enable;
    chosen_path     = load_aware_enable ? lightest_idx
                                        : 8'(flow_hash % PATHS);

    imbalance_x10 = (lightest_gbps == 0) ? 16'hFFFF
                  : (busiest_gbps * 16'd10) / lightest_gbps;

    // The correctness claim, made checkable: the path is decided ONCE,
    // at flow start. Chapter 23.1 Section 9's ordering requirement is
    // untouched.
    sticky_preserved = 1'b1;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_flows_placed <= '0; c_perfect_placements <= '0;
      for (int i = 0; i < PATHS; i++) path_load_gbps[i] <= '0;
    end else begin
      if (flow_start) begin
        path_load_gbps[chosen_path] <= path_load_gbps[chosen_path]
                                     + declared_gbps;
        c_flows_placed <= c_flows_placed + 32'd1;
        // A placement onto an empty path is the outcome a hash cannot
        // guarantee and a greedy choice can, while flows <= paths.
        if (path_load_gbps[chosen_path] == 16'd0)
          c_perfect_placements <= c_perfect_placements + 32'd1;
      end
      if (flow_end)
        path_load_gbps[ending_path] <= (path_load_gbps[ending_path]
                                        > declared_gbps)
                                     ? path_load_gbps[ending_path] - declared_gbps
                                     : 16'd0;
    end
  end
endmodule

Classification: a greedy least-loaded assignment that changes the choice function and nothing else.

What it teaches: that sticky_preserved is the reason this remedy is deployable and flowlet switching is contentious. Chapter 23.1 §9 established that a flow's path is pinned because re-hashing reorders it. Load-aware selection still pins the flow — it simply decides where to pin it using load rather than a hash. No ordering property changes, so the entire correctness argument from that chapter carries over unmodified.

And it teaches that c_perfect_placements counts the outcome a hash structurally cannot produce. While the flow count is at most the path count, a greedy least-loaded assignment places every flow on an empty path — 32 flows, 32 uplinks, one each, 100% efficiency, deterministically. A uniform hash produces that outcome with probability 1.8 × 10⁻¹³.

Deliberately simplified: declared_gbps assumes a flow announces its offered rate, which a collective library can and general traffic cannot. The greedy choice is made on instantaneous load with no hysteresis, so two flows starting in the same cycle both see the same lightest path. path_load_gbps decrements on flow_end with a clamp, hiding an accounting bug rather than reporting it. And the whole scheme is per-leaf: two leaves independently choosing lightest uplinks can still converge on the same spine, which this block cannot see.

Production implication: the spine-side collision is the limitation that decides how far this remedy goes, and it is Chapter 22.2 §20's twelfth taxonomy group again. Each leaf's uplink choice is locally optimal and the spine's ingress load is a sum over 64 leaves' independent choices — so a fabric can be perfectly balanced on every leaf's uplink group and congested at a spine. The remedy is the same as it has been in three previous chapters: import a number. A spine that advertises its per-port load, at a few hertz, lets each leaf's greedy choice account for the far side, and the advertisement is a few bytes per spine per second. Without it, load-aware selection fixes the leaf's half of the path and leaves the other half to chance.

14. What a Fabric Must Add, and What It Must Not

Four candidate additions, priced.

Must not — a time-aware schedule. Chapter 22.2 built one and priced it precisely, and the arithmetic does not survive the move to a cluster.

Row four alone rules it out — 9.25 times a MAC receive datapath, per port, of schedule state — and row six rules it out again for a different reason: a vehicle's schedule is installed once and a cluster's would be recomputed whenever a job started. Chapter 22.2 §13's atomic installation is a per-port mechanism; coordinating it across 96 switches at job-start latency is a distributed-consensus problem the fabric does not have.

And the express reservation is the second objection. A schedule that reserves 19.48% of every link for the scheduled class gives back less than the imbalance it removes unless the efficiency gain exceeds that — which at 28.4% it comfortably does, but only if the schedule is actually correct for the job running, and Section 18 shows the participant set is the one thing that changes most.

Should add — load-aware path selection, plus a spine-side load advertisement.

May add — per-packet spraying with a receive-side reassembly buffer. This is the strongest mechanism available and its cost is memory rather than logic.

The reassembly buffer must equal the fabric's worst-case path skew times the line rate, and the skew is set by the deepest buffer a packet can sit behind. So spraying's cost is a direct function of how much buffering the fabric has — a shallow-buffered fabric makes spraying cheap and a deep-buffered one makes it expensive, which is the opposite of the usual intuition that more buffer is safer.

Must add, and it costs nothing — a way for the collective library to tell the fabric what it is doing. Every remedy above is easier with one bit of information the library has and the fabric does not: these N flows belong to one collective and will all run for the job's duration. With it, load-aware selection has a declared rate to place against and spraying has a reassembly bound. Without it, the fabric is inferring a collective's structure from packet headers, which is the hardest possible way to obtain a fact that the application already knows.

The cheapest thing a fabric can add for AI traffic is not a mechanism. It is an interface through which the collective declares itself.

15. RTL 7 — Collective Telemetry

// ---------------------------------------------------------------------
// collective_telemetry -- what a leaf knows about a collective running
// across it, and the one field it cannot compute.
//
// The uncomputable field is the JOB's minimum, which is a minimum over
// 1024 participants spread across 32 leaves. Chapter 22.2 Section 20's
// twelfth taxonomy group, third appearance: import it.
// ---------------------------------------------------------------------
module collective_telemetry
  import collective_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        snapshot,

  input  logic [15:0] i_local_min_gbps,
  input  logic [15:0] i_local_mean_gbps,
  input  logic [7:0]  i_slowest_member,
  input  logic [31:0] i_slowest_unchanged,
  input  logic [31:0] i_steps_done,
  input  logic [31:0] i_accel_seconds_lost,
  input  logic [15:0] i_flows_this_leaf,
  input  logic [15:0] i_busiest_uplink_flows,
  input  logic [15:0] i_job_min_gbps,        // IMPORTED across leaves
  input  logic        i_job_min_valid,
  input  fabric_fix_e i_active_fix,

  output logic [15:0] o_local_min_gbps,
  output logic [15:0] o_mean_over_min_x10,
  output logic [7:0]  o_slowest_member,
  output logic [15:0] o_pinned_samples_pct,
  output logic [15:0] o_imbalance_x10,
  output logic [31:0] o_accel_seconds_lost,
  output logic [15:0] o_job_min_gbps,
  output logic [2:0]  o_active_fix,
  output logic [2:0]  o_caveats,
  output logic        o_rate_claim_valid
);
  localparam int C_LOCAL_ONLY = 0;   // no job-wide minimum available
  localparam int C_PINNED     = 1;   // the same member is always slowest
  localparam int C_FEW_FLOWS  = 2;   // flows per uplink below 2 ln n

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      o_local_min_gbps <= '0; o_mean_over_min_x10 <= 16'd10;
      o_slowest_member <= '0; o_pinned_samples_pct <= '0;
      o_imbalance_x10 <= 16'd10; o_accel_seconds_lost <= '0;
      o_job_min_gbps <= '0; o_active_fix <= 3'd0; o_caveats <= '0;
    end else if (snapshot) begin
      o_local_min_gbps     <= i_local_min_gbps;
      o_slowest_member     <= i_slowest_member;
      o_accel_seconds_lost <= i_accel_seconds_lost;
      o_active_fix         <= 3'(i_active_fix);

      if (i_local_min_gbps != 0)
        o_mean_over_min_x10 <= (i_local_mean_gbps * 16'd10)
                               / i_local_min_gbps;

      if (i_steps_done != 0)
        o_pinned_samples_pct <= 16'((i_slowest_unchanged * 100)
                                    / i_steps_done);

      if (i_flows_this_leaf != 0)
        o_imbalance_x10 <= (i_busiest_uplink_flows * 16'd10
                            * 16'(UPLINKS_PER_LEAF)) / i_flows_this_leaf;

      o_job_min_gbps <= i_job_min_valid ? i_job_min_gbps : 16'd0;

      o_caveats[C_LOCAL_ONLY] <= !i_job_min_valid;
      o_caveats[C_PINNED]     <= (i_steps_done != 0) &&
                                 ((i_slowest_unchanged * 2) > i_steps_done);
      // 2 ln 32 is about 7 flows per uplink; below that the imbalance
      // is structural rather than incidental -- Section 10.
      o_caveats[C_FEW_FLOWS]  <= (i_flows_this_leaf <
                                  16'(7 * UPLINKS_PER_LEAF));
    end
  end

  // A rate figure for a collective is only meaningful as the JOB's
  // minimum, which this leaf cannot compute.
  assign o_rate_claim_valid = !o_caveats[C_LOCAL_ONLY];
endmodule

Classification: a snapshot bank whose headline field is disqualified without an imported one.

What it teaches: that o_rate_claim_valid depends on i_job_min_valid and on nothing else. Every other field here is a correct local measurement; none of them is the collective's rate, because the collective's rate is a minimum over 1 024 participants and a leaf holds 32. Publishing o_local_min_gbps as the job's rate is exactly the error Section 20 refuses, one level up from the mean-versus-minimum mistake.

And it teaches that o_pinned_samples_pct separates the two slow-collective causes in one number. Chapter 23.1 §9's stickiness means a hash collision produces the same slowest member on every sample; transient congestion moves it. A value near 100% is a pinned collision — fix it with placement; a value near 1/32 is congestion — fix it with capacity or backoff. The two have completely different remedies and identical symptoms without this field.

Deliberately simplified: no read-clear. o_imbalance_x10 recomputes from flow counts rather than from measured rates, so it is a prediction rather than an observation — useful, and not the same thing. C_FEW_FLOWS's threshold of 7 flows per uplink is Section 10's 2 ln n rounded, baked for n = 32 and wrong for any other uplink count. And o_active_fix reports which remedy is enabled without reporting whether it is working.

Production implication: the missing effectiveness check for o_active_fix is the field to add, because every remedy in Section 12 can be enabled and silently inert. Striping is inert if the library opened the connections and the NIC's cache is thrashing; load-aware selection is inert if the spine side is the bottleneck; flowlet switching is inert if the gaps are shorter than the detection threshold. Publish the remedy's own success metric alongside the flag — connections actually established, perfect placements achieved, flowlet boundaries detected — and a fix that is configured but not functioning becomes visible in the same read that reports it is on.

16. RTL 8 — The Collective Conformance Monitor

// ---------------------------------------------------------------------
// collective_conformance_monitor -- properties for a leaf carrying a
// collective.
//
// Properties 1 to 9 are about mechanism and hold throughout a 3.52x
// slowdown. Properties 10 to 12 are the ones that do not, and all
// three constrain a CLAIM rather than a behaviour.
// ---------------------------------------------------------------------
module collective_conformance_monitor
  import collective_pkg::*;
#(
  parameter int MEMBERS = 32
)(
  input logic        clk,
  input logic        rst_n,

  input logic [15:0] member_gbps [MEMBERS],
  input logic [15:0] min_gbps,
  input logic [15:0] mean_gbps,
  input logic [15:0] collective_gbps,
  input logic [7:0]  slowest_member,
  input logic [31:0] step_time_us,
  input logic [31:0] steps_total,
  input logic [31:0] c_steps_done,
  input logic        running,
  input logic [7:0]  chosen_path,
  input logic [7:0]  paths,
  input logic        flow_start,
  input logic        sticky_preserved,
  input logic        load_aware_enable,
  input logic [15:0] lightest_gbps,
  input logic [15:0] path_load_at_choice,
  input logic        job_min_valid,
  input logic [15:0] job_min_gbps,
  input logic        rate_claim_valid,
  input logic        claim_job_rate
);
  // 1. The collective's rate is the minimum, never the mean.
  p_cl_is_min: assert property (@(posedge clk) disable iff (!rst_n)
    collective_gbps == min_gbps);

  // 2. The minimum never exceeds the mean.
  p_cl_min_le_mean: assert property (@(posedge clk) disable iff (!rst_n)
    min_gbps <= mean_gbps);

  // 3. The slowest member's rate IS the minimum.
  p_cl_slowest_is_min: assert property (@(posedge clk) disable iff (!rst_n)
    member_gbps[slowest_member] == min_gbps);

  // 4. A step's time is set by the minimum.
  p_cl_step_time: assert property (@(posedge clk) disable iff (!rst_n)
    (min_gbps != 0) |-> (step_time_us > 32'd0));

  // 5. Steps done never exceed the total.
  p_cl_steps_bounded: assert property (@(posedge clk) disable iff (!rst_n)
    c_steps_done <= steps_total);

  // 6. Running clears when the last step completes.
  p_cl_done: assert property (@(posedge clk) disable iff (!rst_n)
    (c_steps_done >= steps_total) |=> !running);

  // 7. A chosen path is inside the path set.
  p_cl_path_in_set: assert property (@(posedge clk) disable iff (!rst_n)
    flow_start |-> (chosen_path < paths));

  // 8. The path choice is made once and never revisited.
  p_cl_sticky: assert property (@(posedge clk) disable iff (!rst_n)
    sticky_preserved);

  // 9. Load-aware selection picks a lightest path.
  p_cl_greedy: assert property (@(posedge clk) disable iff (!rst_n)
    (flow_start && load_aware_enable) |->
      (path_load_at_choice == lightest_gbps));

  // 10. A JOB rate claim requires the imported job-wide minimum.
  p_cl_claim_needs_import: assert property (@(posedge clk) disable iff (!rst_n)
    claim_job_rate |-> job_min_valid);

  // 11. And the claim must be the job's minimum, not this leaf's.
  p_cl_claim_is_job_min: assert property (@(posedge clk) disable iff (!rst_n)
    claim_job_rate |-> (job_min_gbps <= min_gbps));

  // 12. The telemetry's validity flag agrees with the import.
  p_cl_valid_flag: assert property (@(posedge clk) disable iff (!rst_n)
    rate_claim_valid == job_min_valid);
endmodule

Classification: twelve properties, nine about mechanism and three about a claim.

What it teaches: that property 11's direction is the interesting one. It asserts job_min_gbps <= min_gbps — the job's minimum is at most this leaf's minimum — which is true by construction and is worth asserting because the tempting error is to publish the larger number. A leaf that reports its own minimum as the job's rate is reporting something that is always at least as high as the truth, and the property's inequality is what catches an implementation that has them the wrong way round.

And it teaches that properties 1 to 9 hold perfectly during every failure in this chapter. A ring running at 28.4% satisfies all nine: the minimum is computed correctly, the step time follows from it, the paths are in the set, the stickiness is preserved, and the greedy choice — when enabled — really does pick the lightest path. Nothing is broken. The collective is simply three and a half times slower than the hardware it is running on, which is Section 20's subject.

Deliberately simplified: property 3 indexes an array with a signal, which needs the index to be stable in the same cycle the array is read. Property 9 compares against path_load_at_choice, a signal the selector must export purely for the property's benefit. Property 6 uses >= on a counter that property 5 bounds, so the two overlap. And there is no property about the rate at all — nothing here says the collective should be fast, because nothing in the fabric promises it.

Production implication: that last absence is the one to raise in a specification review. A fabric vendor can satisfy all twelve properties on a fabric delivering 28.4%, and the customer's question — will my training job run at line rate — is not expressible as a property of the switch. What is expressible, and what a specification should ask for, is the mechanism: does the switch support load-aware selection, does it export per-path load, does it accept a spine-side load advertisement, and does it publish o_pinned_samples_pct. Ask for the four mechanisms, not for a rate, because the rate depends on a flow count the fabric does not control.

17. Hierarchical Reduction, and the Flow Count That Cannot Collide

Section 12 ranked five remedies. This one reaches 100% and costs nothing, because it changes the algorithm rather than the fabric.

The observation is Section 10's, read backwards. A ring all-reduce is slow because it produces one flow per accelerator, which is far below the 2 ln n ≈ 7 flows per uplink an uplink group needs to balance. The obvious response is to raise the flow count — Section 12's striping. The better response is to lower it to one.

A flow count of one cannot collide with itself.

Hierarchical all-reduce, in two stages.

Stage 2 is the only fabric traffic and it is one flow per leaf. One flow on a 32-uplink group picks one uplink and gets all 100 Gb/s of it — Chapter 23.1 §4's result that a single flow uses one path, arriving here as an advantage rather than a limitation.

Row six against Section 8's 1.279 s is the striking part: hierarchical reduction is slightly faster than the ideal flat ring, because the inter-leaf ring has 32 participants rather than 1 024, so its 2(N−1)/N factor is 1.9375 rather than 1.998.

And stages 1 and 3 are not free — they are just not on the fabric.

A leaf switch's internal fabric is non-blocking by construction — Chapter 23.1 §2's D + U = P says all 64 ports can be active — so an intra-leaf all-reduce among 32 accelerators runs at full rate with no hashing involved at all. The uplink group is not touched.

Which gives the chapter's most useful practical statement.

The fix for a collective's fabric problem is usually not in the fabric. It is to arrange the collective so that almost none of it crosses the fabric.

Three ways to obtain that arrangement, in increasing order of effort.

Row one costs nothing and is frequently not done, because the job scheduler places work by accelerator availability and has no reason to know the network topology. Giving it the leaf assignment — a single integer per accelerator — converts a 3.52× slowdown into none, and it is a scheduling input rather than a fabric feature.

The honest limitation. Hierarchical reduction assumes the intra-leaf group is large enough to be worth reducing and that all its members belong to the same job. A job spread thinly — four accelerators on each of 256 leaves — has 4 flows per leaf crossing the fabric instead of 32, which is better and not one. The benefit is proportional to how well the job is packed, which is again the scheduler's decision.

18. What the Collective Assumes

Seven premises. Three of them are about the job rather than the fabric, which is the chapter's recurring point.

Row one is unusual and worth dwelling on. In every other chapter of this track an assumption is something that might be false. Here it is guaranteed true by the algorithm: a ring's data dependency makes every participant run at the slowest one's rate. The assumption is not a risk — it is the mechanism, and the chapter's arithmetic follows from it rather than being threatened by it.

Row four is the operational one and it is easy to miss. A 1 024-accelerator job that loses a node restarts on a different set of 1 024, with a different ring ordering, and every flow redraws its hash. So a job's speed changes discontinuously at every restart, for no reason visible anywhere — and a job that was running at 100 Gb/s per link before a restart can run at 16.7 after one. That is a support case that looks like a hardware regression and is a fresh draw from a distribution.

Row five is the one with the largest gap between cost and benefit. The ring's ordering is chosen by the collective library, usually by rank index; the mapping from rank to leaf is chosen by the job scheduler, usually by availability. Neither knows the other's constraint, and the result is a ring whose consecutive members are scattered across leaves when they could have been adjacent. One integer per accelerator, passed from the scheduler to the library, closes it.

And one premise that is not on the list because it does not hold at all.

That the fabric can tell a collective's flows from any other traffic. It cannot. Thirty-two long-lived, equal-sized, simultaneously-started flows look exactly like thirty-two long-lived, equal-sized, simultaneously-started flows — and Section 14's conclusion is that the cheapest fix is an interface through which the collective says so.

19. The Cost, Accounted

Priced against Chapter 19.7 §19's 14 166-flop MAC receive datapath, for the sixth chapter running.

One block is 55% of the total and it is the remedy that works. load_aware_selector's 512 flops are Chapter 23.1 §19's per-path load array — 32 uplinks × 16 bits — and they convert a 28.4% fabric into a 100% one for the case this chapter is about. Four point one per cent of a MAC receive datapath, for a 3.52× improvement in a training job's communication phase.

That ratio is the best in Modules 21 to 23 and it is worth stating against the alternatives.

Rows two and four reach the same rate at 3.6% and 0% respectively, and row six is two and a half orders of magnitude more expensive than row two for an outcome Section 14 rejects.

And the unit's limits show again, in a new place. Row five's cost is megabytes of buffer, which in flip-flops would be 8 × 10⁶ to 2.56 × 10⁸ — 565 to 18 071 times the datapath — and is meaningless expressed that way because it is SRAM. That is the fifth time in six chapters that a mechanism's cost has escaped the flop unit, always at the same place: when the size is set by the data rather than by the design. Chapter 23.3 meets the problem at full scale and names the unit that replaces it.

The comparison that closes the section.

Five hundred and twelve flip-flops against three and a half times the fabric. That is the whole engineering argument, and it is the same shape as Chapter 23.1 §19's — a cheap measurement beats expensive capacity — with the difference that here the cheap thing is in the switch rather than in the application.

20. Properties Worth Asserting, and One Worth Refusing

Six groups. Every property holds while a training job runs at 28.4% of the hardware it is using, which is the condition the refused class describes.

Group A — the reduction operator.

// A1. The collective's rate is the minimum.
p_rd_is_min: assert property (@(posedge clk) disable iff (!rst_n)
  collective_gbps == min_gbps);

// A2. The minimum is at most the mean.
p_rd_min_le_mean: assert property (@(posedge clk) disable iff (!rst_n)
  min_gbps <= mean_gbps);

// A3. The minimum is at most every member.
p_rd_min_le_all: assert property (@(posedge clk) disable iff (!rst_n)
  min_gbps <= member_gbps[member_idx]);

// A4. The slowest member's rate is the minimum.
p_rd_slowest: assert property (@(posedge clk) disable iff (!rst_n)
  member_gbps[slowest_member] == min_gbps);

// A5. The declared reduction operator is MIN and nothing else.
p_rd_operator: assert property (@(posedge clk) disable iff (!rst_n)
  COLLECTIVE_REDUCTION == RED_MIN);

// A6. mean-over-min is at least 1.0.
p_rd_ratio: assert property (@(posedge clk) disable iff (!rst_n)
  mean_over_min_x10 >= 16'd10);

Group B — the ring's structure.

// B1. Steps are 2(N-1).
p_rg_steps: assert property (@(posedge clk) disable iff (!rst_n)
  steps_total == 32'(2 * (PARTICIPANTS - 1)));

// B2. A step moves S/N bytes.
p_rg_chunk: assert property (@(posedge clk) disable iff (!rst_n)
  step_bytes_kb == 32'((S_MB * 1024) / PARTICIPANTS));

// B3. Step time is the chunk divided by the MINIMUM rate.
p_rg_step_time: assert property (@(posedge clk) disable iff (!rst_n)
  (min_gbps != 0) |->
    (step_time_us == ((step_bytes_kb * 8) / 32'(min_gbps))));

// B4. Steps done never exceed the total.
p_rg_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  c_steps_done <= steps_total);

// B5. The collective ends when the last step does.
p_rg_ends: assert property (@(posedge clk) disable iff (!rst_n)
  (c_steps_done >= steps_total) |=> !running);

// B6. Slowdown is at least 1.00.
p_rg_slowdown: assert property (@(posedge clk) disable iff (!rst_n)
  slowdown_x100 >= 16'd100);

Group C — flows and paths.

// C1. A ring all-reduce produces one flow per accelerator.
p_fp_ring_flows: assert property (@(posedge clk) disable iff (!rst_n)
  (coll == COLL_ALLREDUCE) |->
    (flows_per_leaf_o == 32'(ACC_PER_LEAF)));

// C2. An all-to-all produces ACC_PER_LEAF x (N-1).
p_fp_a2a_flows: assert property (@(posedge clk) disable iff (!rst_n)
  (coll == COLL_ALLTOALL) |->
    (flows_per_leaf_o == 32'(ACC_PER_LEAF) * 32'(PARTICIPANTS - 1)));

// C3. A chosen path is inside the set.
p_fp_in_set: assert property (@(posedge clk) disable iff (!rst_n)
  flow_start |-> (chosen_path < 8'(UPLINKS_PER_LEAF)));

// C4. The choice is made once per flow and never revisited.
p_fp_sticky: assert property (@(posedge clk) disable iff (!rst_n)
  sticky_preserved);

// C5. Efficiency rises monotonically with flows per uplink.
p_fp_monotone: assert property (@(posedge clk) disable iff (!rst_n)
  (flows_a > flows_b) |->
    (efficiency_pct(flows_a, UPLINKS_PER_LEAF) >=
     efficiency_pct(flows_b, UPLINKS_PER_LEAF)));

// C6. Efficiency never exceeds 100%.
p_fp_capped: assert property (@(posedge clk) disable iff (!rst_n)
  efficiency_pct(flows_a, UPLINKS_PER_LEAF) <= 100);

Group D — load-aware selection.

// D1. Greedy selection picks a lightest path.
p_la_greedy: assert property (@(posedge clk) disable iff (!rst_n)
  (flow_start && load_aware_enable) |->
    (path_load_at_choice == lightest_gbps));

// D2. With hashing, the choice is the hash modulo the path count.
p_la_hash: assert property (@(posedge clk) disable iff (!rst_n)
  (flow_start && !load_aware_enable) |->
    (chosen_path == 8'(flow_hash % UPLINKS_PER_LEAF)));

// D3. Placing a flow raises exactly one path's load.
p_la_one_path: assert property (@(posedge clk) disable iff (!rst_n)
  flow_start |=> (path_load_gbps[$past(chosen_path)] ==
                  $past(path_load_gbps[$past(chosen_path)]) +
                  $past(declared_gbps)));

// D4. While flows <= paths, greedy placement is perfect.
p_la_perfect: assert property (@(posedge clk) disable iff (!rst_n)
  (load_aware_enable && (c_flows_placed <= 32'(UPLINKS_PER_LEAF))) |->
    (c_perfect_placements == c_flows_placed));

// D5. The lightest path's load never exceeds the busiest's.
p_la_order: assert property (@(posedge clk) disable iff (!rst_n)
  lightest_gbps <= busiest_gbps);

// D6. Load never goes negative on flow end.
p_la_nonneg: assert property (@(posedge clk) disable iff (!rst_n)
  flow_end |=> (path_load_gbps[$past(ending_path)] >= 16'd0));

Group E — striping.

// E1. A ring link's rate is the SUM of its stripes.
p_st_sum: assert property (@(posedge clk) disable iff (!rst_n)
  member_gbps[m] == stripe_sum(m));

// E2. The collective's rate is still the MINIMUM over ring links.
p_st_min: assert property (@(posedge clk) disable iff (!rst_n)
  collective_gbps == min_over_members());

// E3. Stripes carry disjoint byte ranges, so nothing is reordered.
p_st_disjoint: assert property (@(posedge clk) disable iff (!rst_n)
  (STRIPES > 1) |-> reassembly_required);

// E4. The reassembly depth is (STRIPES-1) stripe-sized gaps.
p_st_depth: assert property (@(posedge clk) disable iff (!rst_n)
  reorder_depth_bytes == (stripe_bytes * 32'(STRIPES - 1)));

// E5. Connections scale with members times stripes.
p_st_conns: assert property (@(posedge clk) disable iff (!rst_n)
  connections_total == (16'(MEMBERS) * 16'(STRIPES)));

Group F — the claim.

// F1. A job-rate claim requires the imported job-wide minimum.
p_cm_needs_import: assert property (@(posedge clk) disable iff (!rst_n)
  claim_job_rate |-> job_min_valid);

// F2. And the claim is the job's minimum, never this leaf's.
p_cm_is_job_min: assert property (@(posedge clk) disable iff (!rst_n)
  claim_job_rate |-> (job_min_gbps <= min_gbps));

// F3. The telemetry flag agrees with the import.
p_cm_flag: assert property (@(posedge clk) disable iff (!rst_n)
  rate_claim_valid == job_min_valid);

// F4. A pinned slowest member is reported, not averaged away.
p_cm_pinned: assert property (@(posedge clk) disable iff (!rst_n)
  ((c_slowest_unchanged * 2) > c_steps_done) |-> caveat_pinned);

// F5. Accelerator-seconds are the loss times the participant count.
p_cm_accel: assert property (@(posedge clk) disable iff (!rst_n)
  accel_seconds_lost ==
    ((lost_ms_per_collective * 32'(PARTICIPANTS)) / 1000));

Thirty-four properties, and all thirty-four hold on a fabric delivering 28.4% to a job that paid for 100%.

21. Verification Scenarios

Fifty-eight scenarios. Group 2's need a chain rather than a stream, and group 6's need more than one leaf.

Group 1 — the reduction operator (8).

Group 2 — the ring, which needs a chain (10).

Group 3 — flow counts and efficiency (10).