The Spine-Leaf Data-Centre Fabric

A modern fabric is two tiers of the same switch, wired so that every server is exactly three hops from every other. Everything about it follows from one part's port count.

Row eight is the property that made the design win and it is not a bandwidth property at all. A spine and a leaf are the same silicon, the same firmware and the same spare; only the cabling differs. Every scaling, sparing, qualification and procurement question collapses into one part number.

Then the honest part, which is that a non-blocking fabric does not deliver non-blocking throughput.

Thirty-two equal flows across thirty-two equal-cost paths deliver 28.4% of the group, because a hash chose the paths and a hash collides.

Every link is up. Every path is equal-cost. Every forwarding table is correct. The capacity is present and unreachable — and Section 20's rejected property is about the assertion that cannot tell the difference.

And one number that explains the topology's other half.

A fabric with no distinguished node has no failure that is worse than any other failure, which is Section 12's subject and is the second reason the design replaced the hierarchy it came from.

1. Scope — A Fabric With No Distinguished Node

Scope: the spine-leaf topology, where its numbers come from, what its oversubscription ratio does and does not promise, and why it is built from identical switches.

Not in scope: what is inside a switch. The forwarding pipeline, the buffer architecture and the table structures are Chapter 23.3's. This chapter treats a switch as a port count and a forwarding function and derives everything else from the topology.

Three properties define the design and only one of them is about bandwidth.

Property 1 is what people notice and property 3 is what makes the design operationally different. A three-tier tree has a core, and a core is a thing that can fail in a way nothing else can fail. A spine-leaf fabric has thirty-two spines and losing any one of them is the same event — 3.1% of the bisection, uniformly distributed across every pair of leaves.

And one thing the design deliberately gives up.

Locality. In a tree, two servers under the same access switch are close and two servers in different buildings are far. In a spine-leaf fabric every pair is either one hop or three, and the three-hop case is almost all of them — so there is no such thing as a nearby server, and no benefit to placing two communicating workloads together beyond a single rack.

That is a real loss and it was traded away on purpose. A uniform fabric is easier to reason about, easier to schedule onto, and easier to extend than one where placement matters — and modern schedulers place work anywhere, which only works because the fabric made everywhere equivalent.

The chapter's arc is four questions.

Question 3 is the chapter's hardest and it is where Module 21's instruments come back. Chapter 21.8 named three ceilings below the line rate; a fabric has a fourth, and it is the one in the table at the top of this chapter.

2. The Topology, Derived From One Switch's Port Count

Take a single switch with P ports. Everything else is arithmetic.

Step 1 — split a leaf's ports between down and up.

P = D + U where D ports face servers and U face spines

Step 2 — the number of spines is the number of uplinks. Each leaf sends exactly one uplink to each spine, so the mesh between tiers is complete:

spines = U

Step 3 — the number of leaves is a spine's port count. Each spine has one port per leaf:

leaves = P

Step 4 — and the server count falls out.

servers = leaves × D = P × D

Step 5 — the oversubscription ratio is the ratio of the two halves.

oversubscription = D ÷ U

With P = 64 at 100 Gb/s, three splits.

Read the last two columns together, because the trade is the whole design decision. Going from 1:1 to 7:1 adds 75% more servers and removes 75% of the bisection — and which is right depends entirely on whether the workload's traffic stays inside a rack.

Bisection bandwidth, derived, because it is the number most often quoted and least often computed.

bisection = leaves × U × rate ÷ 2

The division by two is the part people drop. Bisection is the capacity across a cut that separates the fabric into two halves; the leaf-to-spine links carry traffic in both directions, and only half of them cross any given cut. 64 leaves × 32 uplinks × 100 Gb/s ÷ 2 = 102.4 Tb/s.

And the scaling limit is sharp and worth stating.

Row four is why fabrics have a size at which they stop being simple. A two-tier fabric caps at P × D servers; beyond it a super-spine tier is added, path lengths become five hops for cross-pod traffic and three within a pod, and the "every pair is equivalent" property — property 3 of Section 1 — is lost. That boundary is the single most important number in a fabric design and it is P × D.

3. RTL 1 — The Fabric Package and the Uniform Switch

// ---------------------------------------------------------------------
// dcfabric_pkg -- a spine-leaf fabric as arithmetic over one number.
//
// Everything in this package is derived from PORTS and the split
// between down and up. The design's distinguishing property is not in
// any constant: it is that the SAME switch model is used in both
// tiers, so there is exactly one part number in the fabric.
// ---------------------------------------------------------------------
package dcfabric_pkg;

  // ---- the one switch ---------------------------------------------------
  localparam int PORTS      = 64;
  localparam int RATE_GBPS  = 100;

  // ---- the split --------------------------------------------------------
  localparam int DOWN_PORTS = 32;                 // toward servers
  localparam int UP_PORTS   = PORTS - DOWN_PORTS; // toward spines

  // ---- and everything else follows --------------------------------------
  localparam int SPINES     = UP_PORTS;           // one uplink per spine
  localparam int LEAVES     = PORTS;              // one spine port per leaf
  localparam int SERVERS    = LEAVES * DOWN_PORTS;

  // Bisection is HALF the leaf-to-spine capacity: the links are
  // bidirectional and only half of them cross any given cut. Dropping
  // the division by two is the commonest error in a fabric datasheet.
  localparam int BISECT_GBPS = (LEAVES * UP_PORTS * RATE_GBPS) / 2;

  // ---- roles ------------------------------------------------------------
  // Note that this enum describes CABLING, not silicon. The same part
  // is both.
  typedef enum logic {
    ROLE_LEAF  = 1'b0,
    ROLE_SPINE = 1'b1
  } fabric_role_e;

  // ---- paths ------------------------------------------------------------
  // Same leaf: one switch. Different leaves: leaf, spine, leaf.
  localparam int HOPS_SAME_LEAF = 1;
  localparam int HOPS_CROSS     = 3;

  // The number of equal-cost paths between two different leaves is the
  // number of spines, and Section 10 is about how the flows land on
  // them.
  localparam int ECMP_PATHS = SPINES;

  typedef enum logic [1:0] {
    FAIL_NONE      = 2'd0,
    FAIL_SPINE     = 2'd1,
    FAIL_LEAF      = 2'd2,
    FAIL_UPLINK    = 2'd3
  } fabric_fail_e;

  function automatic int oversubscription_x10();
    return (DOWN_PORTS * 10) / UP_PORTS;
  endfunction

  // Section 12: losing one spine of N removes 1/N of the bisection,
  // uniformly, for every pair of leaves.
  function automatic int spine_loss_pct_x10(int spines_up);
    if (spines_up >= SPINES) return 0;
    return ((SPINES - spines_up) * 1000) / SPINES;
  endfunction

  // The scaling wall. Beyond PORTS * DOWN_PORTS servers a third tier
  // is required and the uniform three-hop property is lost.
  localparam int MAX_SERVERS_TWO_TIER = PORTS * DOWN_PORTS;

endpackage

Classification: a package in which one enumeration describes cabling rather than hardware, and that is the design's whole economic argument.

What it teaches: that fabric_role_e has no hardware meaning. A spine and a leaf are the same part, the same firmware image and the same spare; the only difference is which ports have servers on them. That collapses the fabric's bill of materials to one line, its spares pool to one bin, and its qualification effort to one campaign — and none of those savings appears in a bandwidth calculation.

And it teaches that MAX_SERVERS_TWO_TIER is the number a fabric design lives or dies by. At P = 64 and a 1:1 split it is 2 048; beyond it a super-spine tier appears, cross-pod paths become five hops while intra-pod stay three, and Section 1's property 3 — every pair equivalent — is gone. The fabric stops being uniform at a number that is entirely a property of the switch somebody bought.

Deliberately simplified: DOWN_PORTS is a constant where real leaves mix rates — 48 ports of 25 Gb/s down and 8 of 100 up is the commonest arrangement of all and needs the ratio computed in bandwidth rather than in ports. BISECT_GBPS assumes every uplink is present and healthy, which Section 12 is about. ECMP_PATHS equals SPINES, which is true only when every spine is reachable from both leaves. And HOPS_CROSS counts switches rather than links, which is the convention that makes "three hops" the familiar phrase.

Production implication: the mixed-rate leaf is the configuration to model properly, because port counts lie about bandwidth. Forty-eight 25 Gb/s down ports is 1 200 Gb/s; eight 100 Gb/s up ports is 800 — an oversubscription of 1.5:1, not the 6:1 the port counts suggest. Computing the ratio from ports rather than from bandwidth is a factor-of-four error in the direction that makes a fabric look worse than it is, and it has caused real designs to buy uplink capacity they did not need. Always take the ratio of total down bandwidth to total up bandwidth.

// ---------------------------------------------------------------------
// uniform_switch_model -- one part, two roles, and the only thing that
// differs is what is plugged into it.
//
// The block exists to make the claim checkable: if a leaf and a spine
// need different behaviour anywhere, the fabric does not have one part
// number and the economic argument for the topology is weaker than
// advertised.
// ---------------------------------------------------------------------
module uniform_switch_model
  import dcfabric_pkg::*;
(
  input  logic          clk,
  input  logic          rst_n,
  input  fabric_role_e  role,

  input  logic [PORTS-1:0] port_link_up,
  input  logic [PORTS-1:0] port_faces_server,

  output logic [7:0]    ports_up,
  output logic [7:0]    ports_down_count,
  output logic [7:0]    ports_uplink_count,
  output logic [15:0]   down_gbps,
  output logic [15:0]   up_gbps,
  output logic [15:0]   oversub_x10,
  output logic          role_is_cabling_only,
  output logic          spine_has_servers      // a cabling error
);
  always_comb begin
    ports_up           = 8'd0;
    ports_down_count   = 8'd0;
    ports_uplink_count = 8'd0;

    for (int i = 0; i < PORTS; i++) begin
      if (port_link_up[i]) begin
        ports_up = ports_up + 8'd1;
        if (port_faces_server[i]) ports_down_count   = ports_down_count + 8'd1;
        else                      ports_uplink_count = ports_uplink_count + 8'd1;
      end
    end

    down_gbps = 16'(ports_down_count)   * 16'(RATE_GBPS);
    up_gbps   = 16'(ports_uplink_count) * 16'(RATE_GBPS);

    // The ratio is BANDWIDTH over BANDWIDTH, never ports over ports.
    oversub_x10 = (up_gbps == 0) ? 16'hFFFF
                                 : (down_gbps * 16'd10) / up_gbps;

    // The claim, made checkable: nothing in this block's behaviour
    // depends on `role`.
    role_is_cabling_only = 1'b1;

    spine_has_servers = (role == ROLE_SPINE) && (ports_down_count != 8'd0);
  end
endmodule

Classification: a port census with one output that is a claim about the design rather than a measurement of it.

What it teaches: that role is an input the block never reads, and that this is the point rather than an oversight. Every computation — the census, the bandwidths, the ratio — is identical for both tiers. If a future revision needs role in an expression, the fabric has two behaviours and therefore two qualification campaigns, and the economic argument in Section 6 weakens by exactly that much.

And it teaches that oversub_x10 divides bandwidth by bandwidth. A leaf with 48 down ports at 25 Gb/s and 8 up at 100 has a port ratio of 6:1 and a bandwidth ratio of 1.5:1 — a factor of four apart, and only the second one means anything.

Deliberately simplified: all ports are RATE_GBPS, so the mixed-rate case the production note just argued for cannot actually be expressed — a real model needs a per-port rate vector. port_faces_server is an input where a switch discovers it from neighbour discovery and can be wrong. The census is a combinational loop over 64 ports, which is fine as a model and is a 64-input adder tree in silicon. And role_is_cabling_only is a literal, which is honest: it is a claim the surrounding code must keep true, not a fact the block establishes.

Production implication: spine_has_servers is worth wiring to an alarm, because it is the commonest cabling error in a new fabric and it is silent. A server accidentally patched into a spine port works — the spine forwards for it, traffic flows, and nothing complains. What it does is destroy the topology's uniformity: that server's traffic reaches other leaves in one hop while everyone else takes three, its failure domain is a spine rather than a leaf, and a spine reboot takes it down. The detection is one comparison against a port's expected neighbour type, and without it the anomaly is found months later during a failure analysis.

4. Oversubscription Is a Ratio, Not a Promise

The ratio is quoted in every fabric datasheet and almost always misread. This section says exactly what it claims.

What D:U actually states.

If every server on a leaf transmits at line rate to servers on other leaves at the same time, the leaf's uplinks carry D ÷ U times more offered load than they can accept.

That is a statement about a single, specific, and rather unlikely traffic pattern, and four things it is not:

Row one is the one that makes the ratio useful when it is applied correctly. Traffic that stays within a leaf never touches an uplink at all — a 32-server rack whose servers mostly talk to each other has an effective oversubscription of infinity and it does not matter. So the honest form of the ratio includes a locality term:

uplink load = D × rate × (fraction of traffic leaving the leaf)

And modern workloads have almost no locality, which is why the industry moved toward 1:1: a scheduler that places work anywhere produces traffic that goes anywhere, and the locality term is close to 1.

Row two of the first table deserves its own treatment because it is the commonest practical confusion.

A single flow gets exactly one uplink, because Chapter 21.7 §8's ordering requirement runs all the way up the stack: striping one flow across 32 paths reorders it, and a reordered TCP flow collapses. So the fabric's 102.4 Tb/s is available to the aggregate and never to an individual, which is exactly Chapter 21.8 §5's goodput-versus-capacity distinction at fabric scale.

Which gives the three ratios a design should actually compute.

Row three is the chapter's most useful single number and nobody quotes it. A fabric sold as 1:1 delivers, to 32 large flows, the behaviour of a 3.5:1 fabric — and Section 10 derives it.

5. RTL 2 — The Leaf-Spine Topology Model

// ---------------------------------------------------------------------
// leaf_spine_topology -- the fabric as a function from a port count to
// a server count, a hop count and a path set.
//
// The three outputs that matter are: every cross-leaf path is the same
// length, the number of equal-cost paths is the number of spines, and
// the fabric stops scaling at PORTS * DOWN_PORTS.
// ---------------------------------------------------------------------
module leaf_spine_topology
  import dcfabric_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [15:0] src_leaf,
  input  logic [15:0] dst_leaf,
  input  logic [15:0] spines_healthy,
  input  logic [31:0] servers_requested,

  output logic [3:0]  hops,
  output logic [15:0] ecmp_paths,
  output logic        same_leaf,
  output logic [31:0] servers_supported,
  output logic        needs_third_tier,
  output logic [15:0] path_length_variance,
  output logic [31:0] bisection_gbps
);
  always_comb begin
    same_leaf = (src_leaf == dst_leaf);

    // The design's defining property: TWO values, and no others.
    hops = same_leaf ? 4'(HOPS_SAME_LEAF) : 4'(HOPS_CROSS);

    // Every healthy spine is an equal-cost path. There is no notion of
    // a better or worse spine, which is Section 6's argument.
    ecmp_paths = same_leaf ? 16'd1 : spines_healthy;

    servers_supported = 32'(MAX_SERVERS_TWO_TIER);
    needs_third_tier  = (servers_requested > servers_supported);

    // Zero, always, for a two-tier fabric. A third tier makes it
    // non-zero, and Section 1's property 3 is what that costs.
    path_length_variance = 16'd0;

    bisection_gbps = (32'(LEAVES) * 32'(spines_healthy)
                      * 32'(RATE_GBPS)) / 2;
  end
endmodule

Classification: a topology function whose most informative output is permanently zero.

What it teaches: that path_length_variance being zero is the product. In a three-tier tree the distance between two servers depends on where they are: same access switch, same aggregation block, or across the core — three different latencies, three different failure domains, and a placement decision that matters. In a two-tier fabric there are two values and one of them covers almost everything, so every pair of servers is interchangeable. That is what lets a scheduler place work anywhere, and it is a topology property rather than a software one.

And it teaches that ecmp_paths tracks spines_healthy rather than SPINES. A spine that is down is not a path, so a fabric's path diversity degrades gracefully — 32 paths become 31, the bisection falls by 3.1%, and nothing about the topology changes shape. Section 12 is the arithmetic and it is the second reason the design displaced the tree.

Deliberately simplified: spines_healthy is a count where a real fabric has per-leaf-pair reachability — a spine may be reachable from one leaf and not another, which makes the path set asymmetric and ECMP's assumptions false. hops counts switches rather than links or latency. There is no notion of a link being up but degraded — Chapter 21.4's whole fault space collapses into one bit here. And bisection_gbps recomputes from healthy spines but not from healthy uplinks, so a leaf that has lost four of its 32 uplinks is invisible.

Production implication: the missing per-leaf-pair reachability is what turns a graceful degradation into a silent one. A spine that is reachable from leaf A and not from leaf B is a path that ECMP on A believes exists, so A hashes flows onto it and they are dropped or take a longer route — and A's own view of the fabric says everything is healthy. The detection is a fabric-wide reachability matrix, which no single switch can compute; it is the twelfth taxonomy group of Chapter 22.2 §20 in a new setting — a guarantee that spans nodes, evaluated at one — and the practical remedy is the same: import something, in this case the spine's own neighbour list, and compare.

6. Why Every Switch Is the Same Part

The topology's least-discussed property is the one that decided its adoption, and it has nothing to do with packets.

What "one part number" buys.

Row five is the one that shaped procurement. A core switch in a tree is the largest and most expensive box in the building, it is bought in small numbers, its lead time is long, and its failure removes a large fraction of the network's capacity. Replacing it with thirty-two ordinary switches turns a capital decision into a commodity purchase.

And row six is what changed operations. A tree scales by replacing a tier with bigger boxes — a disruptive, planned, capital event. A fabric scales by adding more of the same part, up to the P × D wall of Section 2, and the addition is incremental.

The technical enabler is worth naming: the two tiers need identical behaviour.

Row four is the only difference and it is a matter of table contents rather than capability. A spine's table has one entry per leaf — 64 — where a leaf's has one per server plus one per remote prefix. The spine's table is smaller, so a part sized for the leaf role is comfortably sufficient for the spine role, and the uniformity holds in the direction that matters.

One honest qualification. Real fabrics are not always perfectly uniform: leaves often use a cheaper, lower-radix part with mixed rates — 48 ports at 25 Gb/s down and 8 at 100 up — while spines use a uniform high-rate part. That is two part numbers rather than one, and it is still two rather than three, with both being commodity. The argument survives in weakened form, and Section 3's bandwidth-not-ports rule is what makes the mixed-rate leaf analysable at all.

The spine-leaf fabric's central innovation is not a routing idea. It is that a large network can be built entirely out of a part that is small enough to be a commodity.

7. RTL 3 — The Bisection Accountant

// ---------------------------------------------------------------------
// bisection_accountant -- the number every fabric datasheet quotes,
// computed correctly, and then the number that actually matters.
//
// The division by two is the part datasheets drop. The ECMP efficiency
// term is the part nobody includes at all, and Section 10 shows it is
// worth a factor of three and a half.
// ---------------------------------------------------------------------
module bisection_accountant
  import dcfabric_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [15:0] leaves_present,
  input  logic [15:0] spines_healthy,
  input  logic [15:0] uplinks_healthy_per_leaf,
  input  logic [15:0] ecmp_efficiency_pct,   // from Section 9's model

  output logic [31:0] nominal_bisect_gbps,
  output logic [31:0] healthy_bisect_gbps,
  output logic [31:0] effective_bisect_gbps,
  output logic [15:0] healthy_pct,
  output logic [15:0] effective_pct,
  output logic        datasheet_number_is_double,
  output logic [31:0] per_leaf_uplink_gbps
);
  always_comb begin
    // What a datasheet SHOULD say.
    nominal_bisect_gbps = (32'(LEAVES) * 32'(UP_PORTS)
                           * 32'(RATE_GBPS)) / 2;

    // What survives the fabric's actual state.
    healthy_bisect_gbps = (32'(leaves_present)
                           * 32'(uplinks_healthy_per_leaf)
                           * 32'(RATE_GBPS)) / 2;

    // What the traffic can reach. Section 10's term, and the only one
    // of the three that predicts throughput.
    effective_bisect_gbps = (healthy_bisect_gbps
                             * 32'(ecmp_efficiency_pct)) / 100;

    healthy_pct = (nominal_bisect_gbps == 0) ? 16'd0
                : 16'((healthy_bisect_gbps * 100) / nominal_bisect_gbps);
    effective_pct = (nominal_bisect_gbps == 0) ? 16'd0
                  : 16'((effective_bisect_gbps * 100) / nominal_bisect_gbps);

    per_leaf_uplink_gbps = 32'(uplinks_healthy_per_leaf) * 32'(RATE_GBPS);

    // Always true, and it is a warning rather than a measurement: a
    // "bisection" quoted without the division by two is the aggregate
    // leaf-to-spine capacity, which is twice the bisection.
    datasheet_number_is_double = 1'b1;
  end
endmodule

Classification: three bisection figures, of which the third is the only one that predicts anything.

What it teaches: that the gap between healthy_bisect_gbps and effective_bisect_gbps is the chapter. A fabric with every link up and every spine healthy has 100% of its nominal bisection present; the same fabric carrying 32 large flows across a 32-uplink group reaches 28.4% of it. The first number is a property of the hardware; the second is a property of the hash, and only the second appears in a throughput measurement.

And it teaches why datasheet_number_is_double is asserted unconditionally. Bisection names the capacity across a cut that splits the fabric in half; every leaf-to-spine link is bidirectional and only half of them cross a given cut. A figure quoted without the division is the aggregate tier-to-tier capacity, which is a real number and a different one — and the two are routinely used interchangeably in marketing material.

Deliberately simplified: uplinks_healthy_per_leaf is one number for all leaves, so a fabric with one badly degraded leaf reports the same figure as one with the damage spread evenly — and those are very different situations. ecmp_efficiency_pct is an input rather than computed here, which is correct division of labour and means this block is only as good as Section 9's model. And there is no notion of traffic pattern: bisection is a worst-case cut figure, and a fabric whose traffic is all intra-leaf never approaches it.

Production implication: the per-leaf uniformity assumption is what makes a fabric's health dashboard lie in the most reassuring direction. A fabric of 64 leaves where one leaf has lost 16 of its 32 uplinks has 99.2% of its nominal bisection — the arithmetic is honest — and the servers on that one leaf have exactly half their uplink capacity. The aggregate figure is correct and useless; the figure that matters is the minimum across leaves, not the mean, and publishing the minimum alongside the total costs one comparator and prevents a class of investigation that begins with the fabric says it is healthy.

8. Every Path Is Three Hops, and That Is the Product

The topology's defining property, stated as a latency argument using Chapter 17.1's terms.

Two path lengths exist and no others.

So 98.5% of all server pairs are three hops apart and the remaining 1.5% are one. There is no middle case, no "nearby rack", and no placement decision that improves anything beyond keeping a pair in the same rack.

Priced with Chapter 17.1 §4's bounded terms, at 100 Gb/s.

Under two microseconds, bounded, for any pair of servers in a 2 048-server fabric. And the variance across pairs is zero — every three-hop path has the same terms.

Two observations follow and the second is the surprising one.

First: propagation is now the largest term. At 100 Gb/s a maximum frame serialises in 0.121 µs and 30 metres of fibre takes 0.150 — so cable length has overtaken frame length, which is the reverse of Chapter 22.2's automotive case where propagation was 0.076 µs against a 121 µs serialisation. The same two terms, four orders of magnitude apart in ratio, because the rate moved by a thousand.

Second: a cut-through fabric removes a third of the total. Chapter 12.6 established that cut-through drops the store-and-forward term; at three hops that is 0.364 µs of 1.635 — 22% — and it is why low-latency fabrics use it despite the cost of forwarding frames whose check sequence has not been verified.

And one thing the uniform path length does not give.

Uniform latency is not bounded latency. Chapter 17.1 §11's two unbounded terms — interference and queueing — are still unbounded here, and a fabric has no scheduling mechanism at all. The 1.635 µs is the bounded subtotal and the queueing term above it can be milliseconds.

That is the honest difference between this chapter's fabric and Chapter 22.2's vehicle. A vehicle bounds latency with a schedule and pays 19.48% of the link for it. A data-centre fabric does not bound latency at all — it makes the bounded part uniform and small, and leaves the unbounded part to congestion control. Neither approach is available to the other: a fabric cannot schedule 2 048 servers' traffic, and a vehicle cannot tolerate an unbounded tail.

9. RTL 4 — The ECMP Hash Distribution

// ---------------------------------------------------------------------
// ecmp_hash_distribution -- N equal-cost paths, and a hash that
// decides which flow takes which.
//
// Every path is correct. Every path is equal cost. Every forwarding
// table is right. And the throughput a set of flows achieves is set by
// the BUSIEST path, not the mean -- which is a balls-in-bins problem
// and not a networking one.
// ---------------------------------------------------------------------
module ecmp_hash_distribution
  import dcfabric_pkg::*;
#(
  parameter int PATHS = ECMP_PATHS      // 32
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        flow_arrives,
  input  logic [31:0] flow_hash,
  input  logic [15:0] flow_gbps,
  input  logic        epoch_reset,

  output logic [7:0]  chosen_path,
  output logic [15:0] path_load_gbps [PATHS],
  output logic [15:0] busiest_path_gbps,
  output logic [15:0] mean_path_gbps,
  output logic [15:0] imbalance_x10,
  output logic [15:0] efficiency_pct,
  output logic [31:0] c_flows,
  output logic        one_flow_one_path
);
  logic [31:0] total_gbps;

  assign chosen_path = 8'(flow_hash % PATHS);

  // The property Section 4 argued for: a flow uses ONE path, because
  // striping it would reorder it, and a reordered TCP flow collapses.
  assign one_flow_one_path = 1'b1;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || epoch_reset) begin
      c_flows <= '0; total_gbps <= '0;
      for (int i = 0; i < PATHS; i++) path_load_gbps[i] <= '0;
    end else if (flow_arrives) begin
      path_load_gbps[chosen_path] <= path_load_gbps[chosen_path] + flow_gbps;
      total_gbps <= total_gbps + 32'(flow_gbps);
      c_flows    <= c_flows + 32'd1;
    end
  end

  always_comb begin
    busiest_path_gbps = 16'd0;
    for (int i = 0; i < PATHS; i++)
      if (path_load_gbps[i] > busiest_path_gbps)
        busiest_path_gbps = path_load_gbps[i];

    mean_path_gbps = 16'(total_gbps / PATHS);

    imbalance_x10 = (mean_path_gbps == 0) ? 16'd10
                  : (busiest_path_gbps * 16'd10) / mean_path_gbps;

    // THE number. The group saturates when its busiest member does, so
    // the fraction of the group's capacity that is reachable is
    // mean over max.
    efficiency_pct = (busiest_path_gbps == 0) ? 16'd100
                   : (mean_path_gbps * 16'd100) / busiest_path_gbps;
  end
endmodule

Classification: a balls-in-bins model wearing a networking costume, and its last output is the fabric's real capacity.

What it teaches: that efficiency_pct is mean ÷ max and not anything about links. An uplink group saturates when its busiest member saturates; the capacity that can actually be used is therefore the mean divided by the maximum, and for a random assignment of flows to paths that ratio is a well-understood function of how many flows there are. Nothing in the fabric is broken when it reads 28%.

And it teaches that one_flow_one_path is a hard constraint rather than an implementation choice. Striping one flow across 32 paths would use the whole group — and would deliver the frames out of order, because the paths have different queue depths. Chapter 21.7 §8's ordering requirement is a transport-layer one and it reaches all the way down: a reordered TCP flow triggers fast retransmit and collapses. So the fabric's aggregate capacity is structurally unavailable to any individual flow.

Deliberately simplified: flow_hash % PATHS is a modulo where real hardware uses a hash over the five-tuple and a modulo that is not a power of two is expensive. path_load_gbps is an array of 32 registers updated combinationally, which is a model. Flows never end, so the loads only grow and epoch_reset is the only way back — a real measurement uses a decaying average. And flow_gbps is a constant per flow where real flows are bursty and a flow's peak is what collides.

Production implication: the missing flow termination is what makes real imbalance worse than the model's, and the mechanism is worth naming. ECMP assigns a flow to a path when the flow starts and keeps it there for the flow's life — that is what preserves ordering. So a long-lived elephant flow occupies its path for minutes, and the assignment that was random at the start becomes a fixed allocation. Two elephants that collided at 09:00 are still colliding at 09:30. The remedies are flowlet switching — re-hashing at a gap long enough that reordering cannot occur — and load-aware path selection, and both are departures from pure ECMP that exist entirely because of the arithmetic in Section 10.

10. Thirty-Two Paths, Twenty-Eight Per Cent

A fabric with 1:1 oversubscription is non-blocking by construction. This section is why that does not produce non-blocking throughput.

The set-up is exact. A leaf has 32 uplinks, all equal cost, all healthy. n large flows leave that leaf for other leaves. ECMP assigns each flow to an uplink by hashing its five-tuple, which is uniform and independent. The group saturates when its busiest uplink does.

The arithmetic is balls in bins and the results are unintuitive.

Row one is the headline and it is the case that matters most. With as many flows as paths — which is exactly the shape of a distributed training job, a shuffle phase, or a storage rebuild — the busiest uplink carries three and a half times the mean, and the group delivers 28.4% of its 3.2 Tb/s.

A 1:1 fabric, fully healthy, every table correct, behaves like a 3.5:1 fabric when the flow count equals the path count.

And the convergence is slow. Getting to 82.4% needs 3 200 flows — a hundred per path — so a fabric only approaches its nominal capacity when the traffic is finely divided. The workloads that need the most bandwidth are exactly the ones with the fewest, largest flows.

Three consequences, and the third is the one that changes designs.

One — the effective oversubscription is the product of two ratios.

effective ratio = bandwidth ratio ÷ ECMP efficiency

Row three is a sobering number for a cost-optimised fabric: a 3:1 build carrying 32 large flows behaves like 10.6:1.

Two — adding uplinks helps less than it looks. Doubling a leaf's uplinks from 32 to 64 doubles the nominal capacity and halves the flows per path, which moves the efficiency the wrong way — more paths means more bins for the same balls, and the imbalance rises. The effective gain is real but well below 2×.

Three — the fix is not more capacity, it is finer flows or better assignment.

Row one is the commonest and cheapest and it is why distributed-training frameworks open many connections per peer rather than one. Eight connections instead of one takes 32 flows to 256 — efficiency from 28.4% to 55.9%, a factor of 1.97 in throughput, for a configuration change in a library.

And the whole phenomenon is invisible to every counter in the fabric. Each uplink reports its own utilisation correctly; the aggregate utilisation of the group is 28.4% of line rate and looks like underutilisation, not congestion; the drops happen on one uplink whose counter is one of thirty-two. Chapter 21.6's class G, at fabric scale — and Section 20's rejected property is the assertion that cannot see it.

11. RTL 5 — The Failure Impact Model

// ---------------------------------------------------------------------
// failure_impact_model -- what each kind of failure removes, and the
// property that made the topology replace the tree.
//
// In a fabric with no distinguished node, every failure of a given
// kind costs the same. The three kinds are a spine, a leaf, and a
// single uplink, and their impacts differ by two orders of magnitude.
// ---------------------------------------------------------------------
module failure_impact_model
  import dcfabric_pkg::*;
(
  input  logic          clk,
  input  logic          rst_n,
  input  fabric_fail_e  fail_kind,
  input  logic [15:0]   spines_healthy,
  input  logic [15:0]   leaves_present,
  input  logic [15:0]   uplinks_lost_on_one_leaf,

  output logic [15:0]   bisection_lost_pct_x10,
  output logic [15:0]   worst_leaf_uplink_pct,
  output logic [15:0]   servers_disconnected,
  output logic [15:0]   ecmp_paths_remaining,
  output logic          impact_is_uniform,
  output logic          aggregate_hides_it
);
  always_comb begin
    bisection_lost_pct_x10 = 16'd0;
    servers_disconnected   = 16'd0;
    worst_leaf_uplink_pct  = 16'd100;
    ecmp_paths_remaining   = spines_healthy;
    impact_is_uniform      = 1'b1;

    case (fail_kind)
      FAIL_SPINE: begin
        // 1/N of the bisection, for EVERY pair of leaves equally.
        bisection_lost_pct_x10 =
          16'(((32'(SPINES) - 32'(spines_healthy)) * 1000) / 32'(SPINES));
        worst_leaf_uplink_pct  =
          16'((32'(spines_healthy) * 100) / 32'(SPINES));
        impact_is_uniform = 1'b1;
      end

      FAIL_LEAF: begin
        // Every server on it. Nothing else changes at all.
        servers_disconnected   = 16'(DOWN_PORTS);
        bisection_lost_pct_x10 =
          16'((1000) / 32'(LEAVES));
        impact_is_uniform = 1'b1;
      end

      FAIL_UPLINK: begin
        // One leaf loses part of its uplink group. The FABRIC's
        // aggregate barely moves; that leaf's servers lose a
        // proportional share.
        worst_leaf_uplink_pct =
          16'(((32'(UP_PORTS) - 32'(uplinks_lost_on_one_leaf)) * 100)
              / 32'(UP_PORTS));
        bisection_lost_pct_x10 =
          16'((32'(uplinks_lost_on_one_leaf) * 1000)
              / (32'(LEAVES) * 32'(UP_PORTS)));
        // NOT uniform: one leaf carries all of it.
        impact_is_uniform = 1'b0;
      end

      default: ;
    endcase

    // The warning. An aggregate bisection figure is nearly unchanged
    // by a failure that halves one leaf's uplinks, and Section 7's
    // production note is the remedy: publish the MINIMUM across
    // leaves, not the total.
    aggregate_hides_it = !impact_is_uniform &&
                         (bisection_lost_pct_x10 < 16'd50) &&
                         (worst_leaf_uplink_pct < 16'd90);
  end
endmodule

Classification: three failure kinds, and the only interesting output is whether the damage is spread or concentrated.

What it teaches: that impact_is_uniform is true for a spine failure and false for an uplink failure, and the two therefore need different reporting. Losing a spine costs 1/32 of the bisection, identically, for every one of 2 016 leaf pairs — it is a fabric-wide event with a fabric-wide number. Losing 16 uplinks on one leaf costs 0.8% of the fabric's bisection and 50% of that leaf's, which no fabric-wide number represents.

And it teaches that aggregate_hides_it is a real condition rather than a rhetorical one. Sixteen uplinks lost on one leaf of 64: the fabric reports 99.2% of its nominal bisection, which is honest, while 32 servers have half their capacity. The dashboard is green and a rack is crippled.

Deliberately simplified: failures are one kind at a time where real events are correlated — a power domain, an optics batch, a firmware rollout — and correlated failures break the uniformity assumption the whole model rests on. FAIL_LEAF disconnects DOWN_PORTS servers with no notion of dual-homing, which real leaves usually have. And the model has no notion of recovery time, which for a fabric is the quantity operations actually cares about.

Production implication: dual-homing is the arrangement that makes FAIL_LEAF survivable and it changes the topology's arithmetic in a way worth stating. A server connected to two leaves survives either one, so a leaf failure disconnects nobody — but it doubles the leaf count needed for the same server count, halves the servers per leaf, and doubles the number of uplink groups, which by Section 10's arithmetic makes each group smaller and its ECMP efficiency worse. The reliability gain is real and it is paid for in effective bandwidth, and the trade is almost never computed because the two effects live in different teams' spreadsheets.

12. Losing a Spine Costs 1/N, Uniformly

The topology's second reason for existing, and it is an availability argument rather than a bandwidth one.

Compare the two structures under a single element failure.

Two things change between the halves of the table and only one of them is the percentage.

First, the magnitude: 1/N where N is the number of spines, and N is large because the spines are commodity parts. A tree's core count is small because a core is expensive; a fabric's spine count is 32 because a spine is the same switch as a leaf.

Second, and more important: the shape of the impact. A tree's core failure removes capacity from a specific set of paths — the ones that were using that core — so the damage is concentrated and its distribution depends on the routing. A spine failure removes one of 32 equal-cost paths between every pair of leaves, so every pair loses exactly 1/32. There is no unlucky pair.

In a fabric, the answer to "what happens if this switch fails" does not depend on which switch.

That property is what makes capacity planning tractable. A fabric designed to survive two simultaneous spine failures needs 32 ÷ 30 times the nominal bisection — 6.7% of headroom — and the calculation does not depend on which two.

And the same is not true of an uplink failure, which is the asymmetry Section 11's model exists to expose.

Row four is the case that matters and it is nearly invisible in the aggregate. A leaf reduced to one working uplink has lost 1.5% of the fabric's bisection — a rounding error on a dashboard — and its 32 servers now share 100 Gb/s where they had 3 200. The fabric is 98.5% healthy and a rack is at 3.1%.

The remedy is the one Section 7 named: publish the minimum, not the total. A fabric health figure that is the worst leaf's uplink fraction would read 3.1% here, and it costs one comparator across 64 values. The figure that gets published instead is the aggregate, because the aggregate is the number the topology's marketing is built on.

13. RTL 6 — The Oversubscription Monitor

// ---------------------------------------------------------------------
// oversubscription_model -- the ratio a fabric is sold on, the ratio
// it is built at, and the ratio it behaves like.
//
// Three numbers. The first is a port count, the second is bandwidth,
// and the third includes Section 10's ECMP efficiency. Only the third
// predicts throughput and only the first ever appears in a datasheet.
// ---------------------------------------------------------------------
module oversubscription_model
  import dcfabric_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [15:0] down_ports,
  input  logic [15:0] up_ports,
  input  logic [15:0] down_rate_gbps,
  input  logic [15:0] up_rate_gbps,
  input  logic [15:0] ecmp_efficiency_pct,
  input  logic [15:0] leaving_leaf_pct,     // traffic locality

  output logic [15:0] port_ratio_x10,
  output logic [15:0] bandwidth_ratio_x10,
  output logic [15:0] effective_ratio_x10,
  output logic [15:0] offered_uplink_gbps,
  output logic [15:0] available_uplink_gbps,
  output logic        uplinks_saturate,
  output logic        port_ratio_misleads,
  output logic [15:0] misleads_by_x10
);
  logic [31:0] down_bw, up_bw;

  always_comb begin
    down_bw = 32'(down_ports) * 32'(down_rate_gbps);
    up_bw   = 32'(up_ports)   * 32'(up_rate_gbps);

    port_ratio_x10 = (up_ports == 0) ? 16'hFFFF
                   : (down_ports * 16'd10) / up_ports;

    bandwidth_ratio_x10 = (up_bw == 0) ? 16'hFFFF
                        : 16'((down_bw * 10) / up_bw);

    // Section 10: the group saturates at its busiest member, so the
    // usable fraction is the efficiency.
    effective_ratio_x10 = (ecmp_efficiency_pct == 0) ? 16'hFFFF
                        : (bandwidth_ratio_x10 * 16'd100)
                          / ecmp_efficiency_pct;

    // The locality term of Section 4: traffic that stays on the leaf
    // never touches an uplink.
    offered_uplink_gbps   = 16'((down_bw * 32'(leaving_leaf_pct)) / 100);
    available_uplink_gbps = 16'((up_bw * 32'(ecmp_efficiency_pct)) / 100);

    uplinks_saturate = (offered_uplink_gbps > available_uplink_gbps);

    // A mixed-rate leaf makes the port ratio wrong, and wrong in the
    // pessimistic direction: 48 x 25 down and 8 x 100 up is 6:1 by
    // ports and 1.5:1 by bandwidth.
    port_ratio_misleads = (port_ratio_x10 != bandwidth_ratio_x10);
    misleads_by_x10 = (bandwidth_ratio_x10 == 0) ? 16'd10
                    : (port_ratio_x10 * 16'd10) / bandwidth_ratio_x10;
  end
endmodule

Classification: three ratios, computed from the same inputs, spanning a factor of four.

What it teaches: that the three ratios can be 6.0, 1.5 and 5.28 for the same leaf, and only the last predicts anything. A 48×25 down, 8×100 up leaf carrying 32 flows: port ratio 6:1, bandwidth ratio 1.5:1, effective ratio 5.28:1. The port ratio is wrong by 4× in the pessimistic direction and the bandwidth ratio is wrong by 3.5× in the optimistic one — and by coincidence the wrong pessimistic number is closer to the truth than the right optimistic one, which is how a bad habit survives.

And it teaches that uplinks_saturate needs the locality term to be honest. Traffic that stays inside a leaf never touches an uplink, so a fabric's real oversubscription is a property of the workload as much as the wiring. The three-way product — bandwidth ratio, ECMP efficiency, locality fraction — is the smallest honest model of a leaf's uplink pressure.

Deliberately simplified: ecmp_efficiency_pct is an input rather than derived from a flow count, so the block cannot warn that the efficiency it was given is optimistic. leaving_leaf_pct is a single number where locality varies enormously by workload and by time of day. All down ports share one rate and all up ports another, which covers the common mixed-rate leaf and not a leaf with three rates. And there is no notion of direction — a leaf's uplink pressure is usually asymmetric, and a storage tier's is heavily so.

Production implication: the missing directionality is worth adding because the asymmetry is large and predictable. A rack of storage servers sends far more than it receives; a rack of compute receives far more than it sends — so the uplink group's two directions can differ by several times, and a fabric sized on a symmetric model over-provisions one direction and under-provisions the other on every leaf. Model the two directions separately, size each against its own offered load, and the same uplink count often supports a noticeably larger fabric.

14. What a Fabric Must Never Promise

Five prohibitions, each derived earlier in the chapter.

One — never promise that a 1:1 fabric is non-blocking in throughput.

Non-blocking is a statement about the wiring. Throughput is a statement about where a hash put the flows, and Section 10 is the arithmetic. The honest promise is: the capacity exists and its reachability depends on your flow count.

Two — never quote a bisection figure without the division by two.

A "bisection" quoted as leaves × uplinks × rate is the aggregate tier-to-tier capacity, which is twice the bisection. Both are real numbers, they are routinely interchanged, and the factor of two is exactly the size of the error that matters most in a capacity argument.

Three — never compute oversubscription from port counts.

Four — never publish a fabric-wide health figure alone.

The minimum across leaves costs one comparator over 64 values and is the figure that corresponds to what a user experiences. The aggregate is the figure the topology's marketing is built on, and Section 12 shows it is nearly unmoved by a rack being crippled.

Five — never assume a fabric bounds latency.

It does not. Section 8's 1.635 µs is Chapter 17.1 §4's bounded subtotal; the two unbounded terms — interference and same-class queueing — are present, unmanaged, and can be milliseconds. A fabric has no scheduling mechanism at all, by deliberate choice: Chapter 22.2 showed what one costs, and 2 048 servers' traffic cannot be scheduled. What a fabric provides is a uniform and small bounded part, and everything above it is congestion control's problem.

A fabric's promises are: every pair is equidistant, every failure is uniform, and the capacity exists. It does not promise that the capacity is reachable, that the aggregate describes any particular leaf, or that latency has a tail bound.

15. RTL 7 — Fabric Telemetry

// ---------------------------------------------------------------------
// fabric_telemetry -- what a leaf knows about itself, plus the two
// fields that make a fabric-wide figure honest.
//
// Those two are the ECMP imbalance of this leaf's uplink group, and
// the MINIMUM healthy-uplink fraction across leaves -- which no single
// leaf can compute, and which Section 12 showed is the only figure
// that corresponds to what a user experiences.
// ---------------------------------------------------------------------
module fabric_telemetry
  import dcfabric_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        snapshot,

  input  logic [15:0] i_uplinks_healthy,
  input  logic [15:0] i_spines_reachable,
  input  logic [15:0] i_busiest_path_gbps,
  input  logic [15:0] i_mean_path_gbps,
  input  logic [31:0] i_flows,
  input  logic [31:0] i_drops_on_busiest,
  input  logic [31:0] i_drops_total,
  input  logic [15:0] i_leaving_leaf_pct,
  input  logic [15:0] i_fabric_min_uplink_pct,   // IMPORTED, not local
  input  logic        i_fabric_min_valid,

  output logic [15:0] o_uplinks_healthy,
  output logic [15:0] o_spines_reachable,
  output logic [15:0] o_imbalance_x10,
  output logic [15:0] o_ecmp_efficiency_pct,
  output logic [31:0] o_flows,
  output logic [15:0] o_drop_concentration_pct,
  output logic [15:0] o_leaving_leaf_pct,
  output logic [15:0] o_fabric_min_uplink_pct,
  output logic [2:0]  o_caveats,
  output logic        o_capacity_claim_valid
);
  localparam int C_IMBALANCE = 0;   // ECMP is not spreading the load
  localparam int C_DEGRADED  = 1;   // this leaf has lost uplinks
  localparam int C_NOFABRIC  = 2;   // no fabric-wide minimum available

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      o_uplinks_healthy <= '0; o_spines_reachable <= '0;
      o_imbalance_x10 <= 16'd10; o_ecmp_efficiency_pct <= 16'd100;
      o_flows <= '0; o_drop_concentration_pct <= '0;
      o_leaving_leaf_pct <= '0; o_fabric_min_uplink_pct <= 16'd100;
      o_caveats <= '0;
    end else if (snapshot) begin
      o_uplinks_healthy  <= i_uplinks_healthy;
      o_spines_reachable <= i_spines_reachable;
      o_flows            <= i_flows;
      o_leaving_leaf_pct <= i_leaving_leaf_pct;

      if (i_mean_path_gbps != 0)
        o_imbalance_x10 <= (i_busiest_path_gbps * 16'd10)
                           / i_mean_path_gbps;

      if (i_busiest_path_gbps != 0)
        o_ecmp_efficiency_pct <= (i_mean_path_gbps * 16'd100)
                                 / i_busiest_path_gbps;

      // Section 10's invisible failure: drops concentrated on one
      // uplink of thirty-two, while the group looks underutilised.
      if (i_drops_total != 0)
        o_drop_concentration_pct <= 16'((i_drops_on_busiest * 100)
                                        / i_drops_total);

      o_fabric_min_uplink_pct <= i_fabric_min_valid
                               ? i_fabric_min_uplink_pct : 16'd100;

      o_caveats[C_IMBALANCE] <= (i_busiest_path_gbps >
                                 ((i_mean_path_gbps * 16'd15) / 16'd10));
      o_caveats[C_DEGRADED]  <= (i_uplinks_healthy < 16'(UP_PORTS));
      o_caveats[C_NOFABRIC]  <= !i_fabric_min_valid;
    end
  end

  assign o_capacity_claim_valid = (o_caveats == 3'b0);
endmodule

Classification: a leaf's self-report, with one imported field and a verdict that requires it.

What it teaches: that i_fabric_min_uplink_pct is an input rather than a computation, and that is structural. No leaf can know the fabric's worst leaf; the minimum across leaves is a fabric-wide quantity and every switch in the fabric is a node. Chapter 22.2 §20's twelfth taxonomy group, arriving in a data centre: a guarantee that spans nodes cannot be evaluated at one, and the fix is to import the number rather than to compute it more cleverly.

And it teaches that o_drop_concentration_pct is the signature of Section 10's failure. When ECMP has piled flows onto one uplink, that uplink drops and the other 31 do not — so the concentration reads near 100% while the group's aggregate utilisation reads 28%. Drops with low utilisation is a contradiction on any per-link dashboard; it is the expected signature here, and the two fields together name it in one glance.

Deliberately simplified: no read-clear, so counters need external differencing. o_imbalance_x10 uses instantaneous loads where a decaying average is the useful form — a single measurement of a bursty group is noise. The 1.5× threshold for C_IMBALANCE is arbitrary and should scale with the flow count, since Section 10 shows the expected imbalance falls as flows rise. And there is no per-uplink breakdown, only the busiest and the mean, so a group with two hot uplinks looks like one with one.

Production implication: the flow-count-dependent threshold is the refinement that makes this field usable rather than noisy. An imbalance of 3.5× is entirely expected with 32 flows on 32 paths and is a genuine anomaly with 3 200 — so a fixed threshold either cries wolf on every small job or misses a real hash failure on a large one. Compare against the expected imbalance for the observed flow count — Section 10's table is the model — and alarm on the ratio of observed to expected, which is scale-free and needs one division.

16. RTL 8 — The Fabric Conformance Monitor

// ---------------------------------------------------------------------
// fabric_conformance_monitor -- properties for a leaf in a spine-leaf
// fabric.
//
// Properties 1 to 10 are local and every one of them holds while
// Section 10's failure is happening. Properties 11 and 12 are the two
// that do not, and both require a number from outside the switch.
// ---------------------------------------------------------------------
module fabric_conformance_monitor
  import dcfabric_pkg::*;
(
  input logic        clk,
  input logic        rst_n,

  input logic [15:0] uplinks_healthy,
  input logic [15:0] spines_reachable,
  input logic [15:0] ecmp_paths,
  input logic [7:0]  chosen_path,
  input logic        flow_arrives,
  input logic [31:0] flow_hash,
  input logic [3:0]  hops,
  input logic        same_leaf,
  input logic [15:0] busiest_path_gbps,
  input logic [15:0] mean_path_gbps,
  input logic [15:0] ecmp_efficiency_pct,
  input logic [31:0] healthy_bisect_gbps,
  input logic [31:0] nominal_bisect_gbps,
  input logic        capacity_claim_valid,
  input logic [15:0] fabric_min_uplink_pct,
  input logic        fabric_min_valid,
  input logic        spine_has_servers
);
  // 1. A chosen path is always inside the path set.
  p_fb_path_in_set: assert property (@(posedge clk) disable iff (!rst_n)
    flow_arrives |-> (chosen_path < 8'(ecmp_paths)));

  // 2. The path set is the reachable spines.
  p_fb_paths_are_spines: assert property (@(posedge clk) disable iff (!rst_n)
    !same_leaf |-> (ecmp_paths == spines_reachable));

  // 3. Same leaf means one hop; otherwise three. There is no third value.
  p_fb_hops: assert property (@(posedge clk) disable iff (!rst_n)
    hops == (same_leaf ? 4'(HOPS_SAME_LEAF) : 4'(HOPS_CROSS)));

  // 4. The same flow always takes the same path -- ordering depends on it.
  p_fb_flow_sticky: assert property (@(posedge clk) disable iff (!rst_n)
    (flow_arrives && (flow_hash == $past(flow_hash))) |->
      (chosen_path == $past(chosen_path)));

  // 5. Healthy uplinks never exceed the port count.
  p_fb_uplinks_bounded: assert property (@(posedge clk) disable iff (!rst_n)
    uplinks_healthy <= 16'(UP_PORTS));

  // 6. Reachable spines never exceed the spine count.
  p_fb_spines_bounded: assert property (@(posedge clk) disable iff (!rst_n)
    spines_reachable <= 16'(SPINES));

  // 7. Healthy bisection never exceeds nominal.
  p_fb_bisect_bounded: assert property (@(posedge clk) disable iff (!rst_n)
    healthy_bisect_gbps <= nominal_bisect_gbps);

  // 8. The busiest path is at least the mean.
  p_fb_busiest_ge_mean: assert property (@(posedge clk) disable iff (!rst_n)
    busiest_path_gbps >= mean_path_gbps);

  // 9. Efficiency is mean over max, and never above 100%.
  p_fb_efficiency: assert property (@(posedge clk) disable iff (!rst_n)
    ecmp_efficiency_pct <= 16'd100);

  // 10. A spine never has servers on it -- a cabling check.
  p_fb_spine_clean: assert property (@(posedge clk) disable iff (!rst_n)
    !spine_has_servers);

  // 11. A capacity claim requires a fabric-wide minimum, which this
  //     switch cannot compute.
  p_fb_claim_needs_fabric: assert property (@(posedge clk) disable iff (!rst_n)
    capacity_claim_valid |-> fabric_min_valid);

  // 12. And it requires the imported minimum to be healthy.
  p_fb_claim_needs_min: assert property (@(posedge clk) disable iff (!rst_n)
    capacity_claim_valid |-> (fabric_min_uplink_pct >= 16'd90));
endmodule

Classification: ten local properties and two that cannot be discharged without a number from another switch.

What it teaches: that property 4 — a flow always takes the same path — is simultaneously a correctness requirement and the cause of Section 10's problem. It exists because striping a flow across paths reorders it; it means a flow that collided at the start of its life collides for the whole of it. The property is right, it must be kept, and every mitigation in Section 10 is a careful, bounded relaxation of it.

And it teaches that properties 11 and 12 are the whole reason the monitor is not just bookkeeping. Everything else is a range check or an identity. The two that constrain a claim both need fabric_min_valid, which arrives from outside, and a switch that does not receive it may not assert that its fabric has capacity — which is the correct behaviour and is almost never implemented.

Deliberately simplified: property 4 compares against $past(flow_hash) and therefore only catches back-to-back repeats of the same flow — a real check needs a per-flow record, which is exactly the state ECMP is designed not to keep. Property 2 assumes symmetric reachability, which Section 5's production note showed is the assumption that fails silently. Property 12's 90% threshold is arbitrary. And there is no property at all about the imbalance, because Section 15 showed the right threshold depends on the flow count and the monitor does not have it.

Production implication: the missing symmetric-reachability assumption is the one to test for in a real fabric, and the test is cheap. A spine reachable from leaf A and not from leaf B makes A's path set wrong, so A hashes flows onto a path that leads nowhere useful and A's own view reports full health. The check is to compare each leaf's reachable-spine set with every other leaf's, which is 64 bitmaps of 32 bits — 256 bytes of state for the whole fabric — and it converts a silent black hole into a one-line difference. No switch can do it; a controller can, trivially, and most do not.

17. What a Fabric Looks Like From Module 21

Module 21 built a diagnostic method for one link. This section runs it against a fabric and three of its nine chapters change character completely.

Row eight first, because it is the chapter's contribution to Module 21's framework. Chapter 21.8 enumerated three ceilings below the line rate — framing efficiency, the design's frame rate, and the outstanding-request limit — and found the lowest one at 65.5 Gb/s on a 100 Gb/s port. A fabric adds a fourth: the ECMP efficiency of the uplink group, which at 32 flows on 32 paths is 28.4% and therefore the binding ceiling by a wide margin. It is computable in advance from a flow count and a path count, it belongs in Chapter 21.8 §5's table, and it is the one nobody computes.

Row six is transformed rather than unchanged. Chapter 21.6's class G is frames that arrived nowhere and no counter names; in a fabric the drops have a distinctive and diagnostic shape.

Drops at low utilisation is a contradiction on a per-link dashboard and it is the expected signature here, which makes it one of the few genuinely easy fabric diagnoses — provided the drop concentration is published, which Section 15's o_drop_concentration_pct is for.

Row nine is the one where Chapter 21.9's arithmetic gets much worse.

Row four is the new difficulty and it is circular in a familiar way. A capture on the wrong uplink of the group contains none of the flow in question, and knowing which uplink to tap requires computing the switch's hash — which needs the five-tuple, the hash function and its seed. Chapter 21.9 §12's filter argument arrives with a twist: in a fabric the filter has to select a location as well as a predicate, and the location is a hash's output.

Row four's row one is the practical difference. Chapter 21.4's link faults become the fabric's commonest event purely by count: 4 096 optical links, each with a failure rate, means a fabric of this size sees link events routinely rather than exceptionally. The per-link diagnosis is unchanged and the operational posture is completely different — a fabric is designed to run continuously with some links down, which is Section 12's whole argument.

18. What the Fabric Assumes

Seven premises, and three of them have no local evidence.

Row one deserves emphasis because it is usually stated as though it were a design goal rather than a statistical fact. A uniform, independent hash does spread flows evenly in the sense of equal expectation; what it does not do is produce equal loads, and the gap between those two statements is the entire content of Section 10. The hash is not broken at 28.4% — it is behaving exactly as specified.

Row two has no local evidence at all. A leaf's view of the fabric is its own uplinks and what they reach; a spine that is reachable from this leaf and not from another looks perfectly healthy here. The check is a comparison across leaves — 64 bitmaps of 32 bits, 256 bytes for the whole fabric — and no switch can do it.

Row three is the same shape. The minimum uplink health across leaves is a fabric-wide quantity; each leaf knows only its own. Section 15 imports it as a field for exactly this reason.

And rows two and three together are Chapter 22.2 §20's twelfth taxonomy group in its second setting. That chapter named it for a vehicle's schedule — a guarantee that is a relation between nodes, evaluated at one — and predicted more members as the track moved to fabrics. Here are two, in one chapter, and the remedy is identical in both settings: import a number rather than compute a cleverer property.

Row seven is worth a sentence because it is a physical assumption. A fabric's uniformity is a property of how it is cabled, and the cabling is done by people, at scale, in a building. A server patched into a spine port works — traffic flows, nothing complains — and that server is now one hop from other leaves while everyone else is three, with a spine as its failure domain. Section 3's spine_has_servers is one comparison and it is the only thing that will ever find it.

19. The Cost, Accounted

Priced against Chapter 19.7 §19's 14 166-flop MAC receive datapath, for the fifth chapter running.

Six of the eight blocks are purely combinational and that is characteristic of the subject. A topology is arithmetic over constants; what carries state is the measurement — the per-path load array of ecmp_hash_distribution, which is 32 × 16 bits = 512 flops, 70.8% of the chapter's total.

And that array is the interesting cost, because it scales with the path count.

A measurement whose size is the number of things being measured — the fourth time in five chapters a mechanism's cost has been set by data rather than by design. Chapter 21.9 §19's capture buffer, Chapter 22.1 §19's delay line, Chapter 22.3 §19's video encoder, and now a per-path load array. In each case the flop unit degrades from informative to merely arithmetic.

The comparison that does work here is the one the chapter opened with.

Row five against row four is the chapter's practical conclusion. Getting from 28.4% to 55.9% efficiency — a factor of 1.97 in delivered throughput — is a library configuration change, and the alternative is buying twice the fabric. That is the single largest return available anywhere in Modules 19 to 23, and it is invisible to every counter in the network.

20. Properties Worth Asserting, and One Worth Refusing

Six groups. The refused property is the one every fabric validation suite contains, and it is true throughout the failure that costs 73.3 Tb/s.

Group A — topology.

// A1. Two hop counts exist and no others.
p_tp_hops: assert property (@(posedge clk) disable iff (!rst_n)
  hops == (same_leaf ? 4'(HOPS_SAME_LEAF) : 4'(HOPS_CROSS)));

// A2. Path length variance is zero in a two-tier fabric.
p_tp_variance: assert property (@(posedge clk) disable iff (!rst_n)
  path_length_variance == 16'd0);

// A3. The equal-cost path set is the reachable spines.
p_tp_paths: assert property (@(posedge clk) disable iff (!rst_n)
  !same_leaf |-> (ecmp_paths == spines_healthy));

// A4. Servers supported is ports times down ports.
p_tp_scale: assert property (@(posedge clk) disable iff (!rst_n)
  servers_supported == 32'(MAX_SERVERS_TWO_TIER));

// A5. Exceeding it demands a third tier.
p_tp_third: assert property (@(posedge clk) disable iff (!rst_n)
  needs_third_tier == (servers_requested > servers_supported));

// A6. Nothing in the switch model depends on its role.
p_tp_role_cabling: assert property (@(posedge clk) disable iff (!rst_n)
  role_is_cabling_only);

Group B — the ratios.

// B1. The bandwidth ratio is bandwidth over bandwidth.
p_rt_bandwidth: assert property (@(posedge clk) disable iff (!rst_n)
  (up_gbps != 0) |->
    (bandwidth_ratio_x10 == ((down_gbps * 16'd10) / up_gbps)));

// B2. A mixed-rate leaf makes the port ratio disagree.
p_rt_misleads: assert property (@(posedge clk) disable iff (!rst_n)
  (down_rate_gbps != up_rate_gbps) |-> port_ratio_misleads);

// B3. The effective ratio includes the ECMP efficiency.
p_rt_effective: assert property (@(posedge clk) disable iff (!rst_n)
  (ecmp_efficiency_pct != 0) |->
    (effective_ratio_x10 ==
      ((bandwidth_ratio_x10 * 16'd100) / ecmp_efficiency_pct)));

// B4. Bisection is half the tier-to-tier capacity.
p_rt_bisection: assert property (@(posedge clk) disable iff (!rst_n)
  nominal_bisect_gbps ==
    ((32'(LEAVES) * 32'(UP_PORTS) * 32'(RATE_GBPS)) / 2));

// B5. Healthy bisection never exceeds nominal.
p_rt_bisect_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  healthy_bisect_gbps <= nominal_bisect_gbps);

// B6. Effective never exceeds healthy.
p_rt_effective_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  effective_bisect_gbps <= healthy_bisect_gbps);

Group C — ECMP.

// C1. A chosen path is inside the set.
p_ec_in_set: assert property (@(posedge clk) disable iff (!rst_n)
  flow_arrives |-> (chosen_path < 8'(PATHS)));

// C2. The same flow always takes the same path.
p_ec_sticky: assert property (@(posedge clk) disable iff (!rst_n)
  (flow_arrives && (flow_hash == $past(flow_hash))) |->
    (chosen_path == $past(chosen_path)));

// C3. A flow uses exactly one path.
p_ec_one_path: assert property (@(posedge clk) disable iff (!rst_n)
  one_flow_one_path);

// C4. The busiest path is at least the mean.
p_ec_busiest: assert property (@(posedge clk) disable iff (!rst_n)
  busiest_path_gbps >= mean_path_gbps);

// C5. Efficiency is mean over max and is capped at 100%.
p_ec_efficiency: assert property (@(posedge clk) disable iff (!rst_n)
  ecmp_efficiency_pct <= 16'd100);

// C6. Imbalance is at least 1.0.
p_ec_imbalance: assert property (@(posedge clk) disable iff (!rst_n)
  imbalance_x10 >= 16'd10);

Group D — failures.

// D1. A spine failure is uniform.
p_fl_spine_uniform: assert property (@(posedge clk) disable iff (!rst_n)
  (fail_kind == FAIL_SPINE) |-> impact_is_uniform);

// D2. An uplink failure is not.
p_fl_uplink_local: assert property (@(posedge clk) disable iff (!rst_n)
  (fail_kind == FAIL_UPLINK) |-> !impact_is_uniform);

// D3. Losing a spine of N costs 1/N of the bisection.
p_fl_spine_cost: assert property (@(posedge clk) disable iff (!rst_n)
  (fail_kind == FAIL_SPINE) |->
    (bisection_lost_pct_x10 ==
      16'((((32'(SPINES) - 32'(spines_healthy)) * 1000)) / 32'(SPINES))));

// D4. A leaf failure disconnects its servers and nothing else.
p_fl_leaf: assert property (@(posedge clk) disable iff (!rst_n)
  (fail_kind == FAIL_LEAF) |-> (servers_disconnected == 16'(DOWN_PORTS)));

// D5. The aggregate-hides-it condition means a small fabric loss and a
//     large leaf loss together.
p_fl_hidden: assert property (@(posedge clk) disable iff (!rst_n)
  aggregate_hides_it |-> ((bisection_lost_pct_x10 < 16'd50) &&
                          (worst_leaf_uplink_pct < 16'd90)));

// D6. No failure increases capacity.
p_fl_monotone: assert property (@(posedge clk) disable iff (!rst_n)
  (fail_kind != FAIL_NONE) |-> (bisection_lost_pct_x10 >= 16'd0));

Group E — telemetry.

// E1. A capacity claim requires all caveats clear.
p_tm_claim: assert property (@(posedge clk) disable iff (!rst_n)
  o_capacity_claim_valid |-> (o_caveats == 3'b0));

// E2. And it requires the imported fabric minimum.
p_tm_needs_import: assert property (@(posedge clk) disable iff (!rst_n)
  o_capacity_claim_valid |-> !o_caveats[C_NOFABRIC]);

// E3. Drop concentration is a percentage.
p_tm_conc: assert property (@(posedge clk) disable iff (!rst_n)
  o_drop_concentration_pct <= 16'd100);

// E4. A degraded leaf sets its caveat.
p_tm_degraded: assert property (@(posedge clk) disable iff (!rst_n)
  snapshot |=> (o_caveats[C_DEGRADED] ==
                ($past(i_uplinks_healthy) < 16'(UP_PORTS))));

// E5. Efficiency and imbalance are reciprocal.
p_tm_reciprocal: assert property (@(posedge clk) disable iff (!rst_n)
  (o_imbalance_x10 != 0) |->
    (o_ecmp_efficiency_pct == (16'd1000 / o_imbalance_x10)));

Group F — cabling.

// F1. A spine never has servers on it.
p_cb_spine_clean: assert property (@(posedge clk) disable iff (!rst_n)
  !spine_has_servers);

// F2. Uplinks plus down ports never exceed the port count.
p_cb_ports: assert property (@(posedge clk) disable iff (!rst_n)
  (ports_down_count + ports_uplink_count) <= 8'(PORTS));

// F3. Ports up is the sum of the two categories.
p_cb_census: assert property (@(posedge clk) disable iff (!rst_n)
  ports_up == (ports_down_count + ports_uplink_count));

// F4. Reachable spines never exceed the spine count.
p_cb_spines: assert property (@(posedge clk) disable iff (!rst_n)
  spines_reachable <= 16'(SPINES));

// F5. Healthy uplinks never exceed the uplink port count.
p_cb_uplinks: assert property (@(posedge clk) disable iff (!rst_n)
  uplinks_healthy <= 16'(UP_PORTS));

Thirty-four properties. Every one of them holds while a fabric delivers 28.4% of its bisection, and the one that is supposed to cover that case is the one to refuse.

21. Verification Scenarios

Fifty-eight scenarios. Group 3's are the ones that need a flow generator rather than a packet generator, and group 5's need more than one switch.

Group 1 — topology arithmetic (10).

Group 2 — the ratios (10).

Group 3 — ECMP, which needs flows rather than packets (12).

Group 4 — failures (10).

Group 5 — multi-switch, which one testbench cannot do (8).

Group 6 — latency and Module 21's method (8).

Scenario 39 is the one to show an operations team and scenario 43 is the one no switch can run.

The directed test random stimulus will not produce

This test requires a fabric in perfect health delivering a quarter of its capacity, with every property passing and every dashboard green.

The arrangement:

Random stimulus will not produce this because a randomised traffic generator produces many flows. Coverage-driven stimulus is built to explore state, and exploring a fabric's state means varying the five-tuples, which means hundreds or thousands of distinct flows — and Section 10's table shows that 3 200 flows reach 82.4% efficiency, where the problem is invisible. The interesting case is the smallest flow count that can saturate the group, which is exactly the count a generator has no reason to choose, and it is also exactly what a distributed training job or a storage rebuild produces.

The oracle is in four parts and two of them are expected passes.

Part 3 is the assertion worth writing and it is an assertion about a shortfall. Require the delivered aggregate to be within 10% of the value Section 10's model predicts for 32 flows — not within 10% of nominal. A test that expects nominal fails and gets "fixed" by raising the tolerance; a test that expects the model passes, and the model is the thing worth having.

Part 4 is what separates this from congestion. A congested group is at 80%+ utilisation with drops spread; this group is at 28% with drops on one uplink of 32. Those two states are trivially distinguishable once the concentration is published, and indistinguishable without it.

Run the test a second time with each server opening eight connections instead of one. Flows go from 32 to 256, efficiency from 28.4% to 55.9%, delivered throughput from 29.1 to 57.2 Tb/s — a factor of 1.97, for a library configuration change, on identical hardware carrying identical work. That contrast is the chapter's whole practical content, and it is the demonstration to put in front of anyone proposing to buy more fabric.

22. Debugging a Fabric

A procedure, ordered so that the cheapest and most diagnostic reads come first.

Step 7 being seventh inverts the instinct, because a fabric of 4 096 optical links has link events routinely and most of them are not the problem being investigated. A fabric is designed to run with some links down — Section 12 — so a failed link is a normal state, not a diagnosis.

Four signatures.

Row one's third clause is the number to memorise. Throughput that lands near 28% of nominal on a healthy 1:1 fabric is not a coincidence — it is the expected value for a flow count equal to the path count, and recognising it turns a week of investigation into a question about how many connections the application opens.

Row four is the hardest and it is the one with no local evidence. A flow hashed onto a spine that its source leaf believes is reachable and its destination leaf cannot reach is black-holed; the source leaf reports full health, the destination reports full health, and only a comparison across leaves shows it. The comparison is 64 bitmaps of 32 bits — 256 bytes for the whole fabric — and it belongs in a controller, because no switch can perform it.

23. Misconceptions

Six, in wrong-model / what-it-costs / corrected-model form.

Misconception 1 — "a 1:1 fabric is non-blocking."

Wrong model: the wiring provides as much uplink capacity as access capacity, so nothing blocks.

What it costs: a factor of 3.5 in throughput, and a wrong diagnosis when it shows up. Section 10: 32 large flows across 32 equal-cost uplinks put 3.52× the mean on the busiest one, and a group saturates at its busiest member — so the group delivers 28.4% of its 3.2 Tb/s.

Corrected model: non-blocking is a statement about the wiring; throughput is a statement about where a hash put the flows. The effective ratio is bandwidth ratio ÷ ECMP efficiency, and at 32 flows a 1:1 fabric behaves like 3.52:1. It converges: 256 flows give 55.9%, 3 200 give 82.4% — so the honest promise is that the capacity exists and its reachability depends on your flow count.

Misconception 2 — "oversubscription is the ratio of down ports to up ports."

Wrong model: count the ports on each side and divide.

What it costs: a factor of four on the commonest leaf in the industry, in the pessimistic direction. A 48×25 Gb/s down, 8×100 Gb/s up leaf is 1 200 Gb/s down and 800 up — 1.5:1 — and its port ratio is 6:1.

Corrected model: the ratio is bandwidth over bandwidth, always. And there are three ratios worth computing rather than one: port (6.0), bandwidth (1.5), and effective (5.28 at 32 flows) — only the third predicts throughput, and by an unhelpful coincidence the wrong port ratio is closer to it than the right bandwidth ratio is, which is how the bad habit survives.

Misconception 3 — "the fabric's bisection bandwidth is leaves × uplinks × rate."

Wrong model: add up the leaf-to-spine capacity and that is the bisection.

What it costs: a factor of two, in every capacity argument. Bisection is the capacity across a cut that splits the fabric in half, and the leaf-to-spine links are bidirectional — only half of them cross a given cut.

Corrected model: bisection = leaves × uplinks × rate ÷ 2. For 64 leaves, 32 uplinks and 100 Gb/s that is 102.4 Tb/s, not 204.8. Both numbers are real and they mean different things; the undivided one is the aggregate tier-to-tier capacity, and the two are used interchangeably often enough that the figure should always be labelled.

Misconception 4 — "the fabric dashboard is green, so the fabric is fine."

Wrong model: an aggregate health figure describes the fabric's users' experience.

What it costs: an entire rack, invisibly. Section 12: a leaf that has lost 31 of its 32 uplinks costs the fabric 1.5% of its bisection — a rounding error — and leaves its 32 servers sharing 100 Gb/s where they had 3 200. The fabric reports 98.5% healthy and a rack is at 3.1%.

Corrected model: publish the minimum across leaves, not the total. It costs one comparator over 64 values, it is the figure that corresponds to what somebody experiences, and the aggregate is the figure the topology's marketing is built on. Both belong on the dashboard and only one of them belongs alone.

Misconception 5 — "adding uplinks fixes an imbalanced group."

Wrong model: the group is short of capacity, so widen it.

What it costs: money, for less than half the expected return. Doubling a leaf's uplinks from 32 to 64 doubles the nominal capacity and halves the flows per path — more bins for the same balls — so the imbalance gets worse and the efficiency falls. The net gain is real and well under 2×.

Corrected model: the constraint is flows per path, not paths. Eight connections per peer instead of one takes 32 flows to 256, efficiency from 28.4% to 55.9%, a factor of 1.97 in delivered throughput for a library configuration change on identical hardware. The hardware alternative — buying the 73.3 Tb/s gap — is 3.5× the fabric.

Misconception 6 — "a spine-leaf fabric bounds latency."

Wrong model: every path is the same length, so latency is predictable.

What it costs: any design that depends on a tail bound. Section 8's 1.635 µs is Chapter 17.1 §4's bounded subtotal — serialisation, store-and-forward, blocking, propagation and lookup. The two unbounded terms, interference and same-class queueing, are present and entirely unmanaged, and a fabric has no scheduling mechanism at all.

Corrected model: a fabric makes the bounded part uniform and small and leaves the rest to congestion control. That is a deliberate choice and the alternative was priced in Chapter 22.2: a time-aware schedule costs 19.48% of the link and cannot be built for 2 048 servers. Uniform latency is not bounded latency, and the fabric promises the first.

24. Interview Questions

Six, with what a strong answer contains.

1. Derive a spine-leaf fabric's size from one switch's port count.

P = D + U; spines = U; leaves = P; servers = P × D; bisection = leaves × U × rate ÷ 2. For a 64-port 100 Gb/s switch split 32:32 that is 32 spines, 64 leaves, 2 048 servers and 102.4 Tb/s. A strong answer names the division by two and why it is there — bisection is a cut, and only half the bidirectional links cross it — and names the scaling wall: P × D, beyond which a third tier appears and the uniform three-hop property is lost.

2. Why is every switch in the fabric the same part?

Because a spine and a leaf differ only in what is plugged into them. A spine's forwarding table is smaller — one entry per leaf rather than one per server — so a part sized for the leaf role is sufficient for the spine role. A strong answer gives the operational consequences: one part number, one spare pool, one qualification campaign, one firmware image, and scaling by adding commodity parts rather than replacing a tier with larger ones. It is an economic argument, not a packet-forwarding one, and it is why the topology displaced the tree.

3. Your 1:1 fabric is delivering 28% of its bisection with every link up. What is happening?

ECMP has hashed a small number of large flows onto a small number of paths. With 32 flows on 32 uplinks the busiest carries 3.52× the mean, and a group saturates at its busiest member — so 28.4% is the expected value, not a fault. A strong answer names the signature — low group utilisation with drops concentrated on one uplink, which contradicts every per-link dashboard's intuition — and the fix: more, smaller flows. Eight connections instead of one gives 55.9%, a factor of 1.97.

4. What does losing one spine cost, and why does the answer not name the spine?

1/N of the bisection — 3.1% for 32 spines — uniformly, for every one of 2 016 leaf pairs. A strong answer contrasts with a tree: a core failure removes capacity from the specific paths that used it, so the damage is concentrated and depends on routing. In a fabric there is no unlucky pair, which is what makes capacity planning tractable: surviving two spine failures needs 6.7% of headroom and the calculation does not depend on which two.

5. A fabric reports 98.5% healthy and one rack is unusably slow. Explain.

A leaf has lost most of its uplinks. Thirty-one of 32 uplinks on one leaf of 64 is 1.5% of the fabric's bisection — arithmetically honest — and 3.1% of that leaf's uplink capacity, shared among its 32 servers. A strong answer names the remedy: publish the minimum across leaves, not the total — one comparator over 64 values — and observes that the aggregate is exactly the figure the topology's marketing is built on, which is why it is the one that gets published.

6. Why can a single switch not verify that the fabric has its capacity?

Because two of the premises are fabric-wide. Whether every spine is reachable from every leaf is a comparison across leaves; the minimum uplink health is a minimum over leaves. A leaf's own view reports full health in both failure cases. A strong answer names the remedy — import the number: a reachability bitmap per leaf is 256 bytes for the whole fabric, and the minimum is one comparison — and connects it to Chapter 22.2 §20's taxonomy group: a guarantee that spans nodes cannot be evaluated at one.

25. Questions and Answers

26. What's Next

The fabric's shape is established. What is inside it, what runs on it, and what it is built from are the rest of Module 23.

The chapter's one number to carry forward is 28.4%. A fabric in perfect health, with every property in Section 20 passing, delivers 29.1 Tb/s of a 102.4 Tb/s bisection to the flow pattern that a distributed job actually produces — and the 73.3 Tb/s gap is closed by opening eight connections instead of one, not by buying 3.5× the hardware.

Chapter 23.2 is where that gap stops being an inefficiency and becomes the design constraint. An AI training cluster's traffic is the worst possible case for everything this chapter derived: a small number of enormous, simultaneous, synchronised flows, arriving in a collective pattern where the slowest path sets the pace of every participant, and where a single imbalanced uplink stalls a job across thousands of accelerators. The questions are what that traffic pattern actually is, why ECMP's statistics fail hardest exactly there, and what a fabric has to add — scheduling, telemetry, or a different path-selection rule — to carry it.