Fabric Scaling

16.4 found the ceilings on one switch and said the way past them is a second switch. 18.1 charged 60 ns for each switch hop and 18.2 found the ceiling on one link.

This chapter is what happens when the second switch becomes four, and the four become a tier.

1. The Engineering Problem — Growth Is Not Uniform

Six things change as a fabric scales.

Hop count is a property of the topology, not of the endpoint count. Direct attach is zero hops, one switch is one, a two-tier fabric is three and a three-tier fabric is five — and the endpoint count does not appear in that list. Section 5.

Every flow that crosses the fabric passes through a cut, and the cut is a fixed number of links. Sixteen flows wanting 400 Gbps each across a 3200 Gbps bisection get 200 each, whatever their links are rated at. Section 6.

A tier's oversubscription ratio is edge capacity over uplink capacity, and it is the single number that says how much of the offered load can leave. Section 7.

Consolidation grows the blast radius. The same endpoints over half the switches doubles what one failure takes with it. Section 9.

Path diversity is what survives a failure, and a fabric with one path between a pair has a single point of failure regardless of how many switches it has. Section 10.

And a switch spends some of its own ports reaching the rest of the fabric. Those ports are cost, not capacity, and a port count that ignores them overstates the fabric by exactly the uplink fraction. Section 12.

This chapter against 16.4, stated precisely. That one owns the ceilings on one switch and the decision to add a second. This one owns what a fabric of many switches costs and delivers.

2. The One-Sentence Model

A fabric's scale is not its port count — it is its port count minus what the fabric spends on itself, divided by what the topology costs per hop — and every defect below is a claim that multiplies switches by ports and stops.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Four topologies, four spines, a single cut and integer port counts are far coarser than a real fabric analysis. They are sized so every boundary is reachable and every result recomputable on paper.

What is not simplified is the structure: hops selected by topology, a cut compared against crossing demand, a ratio of edge to uplink, endpoints divided across failure domains, disjoint paths counted and failed, and usable ports computed after uplinks are deducted.

Three things are absent by design. There is no routing-algorithm model — section 10 counts disjoint paths and says nothing about whether the routing will use them, which is 16.2's ground. There is no adaptive load balancing: flows are assumed to spread evenly across the cut. And failure correlation is out of scope — section 9 treats each switch as an independent domain, where shared power or cooling makes real domains larger than the topology suggests.

5. RTL 1 — Hops Come From The Topology

// Hop count is a property of the topology, not of the endpoint count.
module hop_count #(parameter int ASSUME_ONE_HOP = 0) (
  input  logic clk, rst_n,
  input  logic        route,
  input  logic [1:0]  topology,     // 0 direct, 1 single switch, 2 two-tier, 3 three-tier
  input  logic [15:0] per_hop_ns, base_ns,
  output logic [3:0]  hops,
  output logic [31:0] path_ns,
  output logic [15:0] fabric_ns,
  output logic [7:0]  n_routes,
  output logic        hop_blind_err
);
  logic [3:0] true_hops;
  always_comb begin
    case (topology)
      2'd0:    true_hops = 4'd0;   // direct attach
      2'd1:    true_hops = 4'd1;   // one switch
      2'd2:    true_hops = 4'd3;   // leaf, spine, leaf
      default: true_hops = 4'd5;   // leaf, spine, super-spine, spine, leaf
    endcase
  end
  // The assuming build charges one hop whatever the topology is.
  assign hops = (ASSUME_ONE_HOP != 0) ? 4'd1 : true_hops;
  assign fabric_ns = per_hop_ns * {12'd0, hops};
  assign path_ns   = {16'd0, base_ns} + {16'd0, fabric_ns};
  // Charging fewer hops than the topology has.
  assign hop_blind_err = route && (hops < true_hops);
  // ... route counter omitted for length
endmodule

A 150 ns base path at 60 ns per hop:

  hops: topology=0 hops=0 fabric=0ns path=150ns | assuming path=210ns blind=2

Three hops, not two. A two-tier fabric costs a leaf, a spine and a leaf again — the return through the second leaf is a hop and it is the one most often forgotten. The three-tier count of five follows the same rule and is why a super-spine is expensive in latency terms as well as in ports.

The assuming build charges one hop and is exactly right for the single-switch case, which is the configuration everybody benchmarks. It is 120 ns low on a two-tier fabric and 240 ns low on a three-tier one, and it reports the same 210 ns path for all three.

6. RTL 2 — Bisection Bandwidth

// Bisection bandwidth: the cut that every crossing flow must fit through.
module bisection #(parameter int IGNORE_CUT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] cut_links, per_link_gbps,
  input  logic [15:0] crossing_flows, per_flow_gbps,
  output logic [31:0] bisection_gbps, demand_gbps,
  output logic [15:0] per_flow_actual,
  output logic [7:0]  utilisation_pct,
  output logic        oversubscribed, cut_blind_err
);
  logic [31:0] ut_q, pf_q;
  assign bisection_gbps = {16'd0, cut_links} * {16'd0, per_link_gbps};
  assign demand_gbps    = {16'd0, crossing_flows} * {16'd0, per_flow_gbps};
  assign oversubscribed = (demand_gbps > bisection_gbps);
  // The ignoring build hands every flow what it asked for, which is what a model
  // does when it has no concept of a cut at all.
  assign pf_q = (IGNORE_CUT != 0) ? {16'd0, per_flow_gbps}
              : ((crossing_flows == 16'd0) ? 32'd0
                 : (oversubscribed ? (bisection_gbps / {16'd0, crossing_flows})
                                   : {16'd0, per_flow_gbps}));
  // ... utilisation and the blindness check omitted for length
endmodule

Eight cut links of 400 Gbps against sixteen flows wanting 400 each:

  bisection: cut=3200 demand=6400 util=200% per_flow=200 | ignoring per_flow=400 blind=2

Each flow gets 200 Gbps on a 400 Gbps link. Every link in the path is rated 400, every endpoint is capable of 400, and the achieved rate is half that — because sixteen flows are sharing eight links' worth of cut.

This is 18.2's min-of-ceilings argument at fabric scale, and it adds a ceiling that chapter did not have: the cut is a property of the topology, not of any link either endpoint can see. Neither end of a flow has any visibility into how many other flows are crossing with it.

The oversubscription test is strict and driven exactly: eight flows demand exactly the cut and are not oversubscribed; nine flows are, and each gets 355.

7. RTL 3 — Oversubscription Ratio

// Oversubscription: edge capacity against uplink capacity, per tier.
module oversubscription (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] edge_ports, uplink_ports, port_gbps,
  output logic [31:0] edge_gbps, uplink_gbps,
  output logic [15:0] ratio_x100,
  output logic        oversubscribed, one_to_one,
  output logic [7:0]  n_eval, n_over,
  output logic        no_uplink_err
);
  logic [31:0] r_q;
  assign edge_gbps   = {16'd0, edge_ports}   * {16'd0, port_gbps};
  assign uplink_gbps = {16'd0, uplink_ports} * {16'd0, port_gbps};
  // The ratio of what can arrive to what can leave, times a HUNDRED. Times ten
  // is not enough: 32 edge ports against 31 uplinks is 1.03, which floors to 1.0
  // and reads as one-to-one on a tier that is oversubscribed.
  assign r_q = (uplink_gbps == 32'd0) ? 32'hFFFF
                                      : ((edge_gbps * 32'd100) / uplink_gbps);
  assign ratio_x100 = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  assign oversubscribed = (ratio_x100 > 16'd100);
  assign one_to_one     = (ratio_x100 == 16'd100);
  // A tier with edge ports and no uplink cannot forward anything at all.
  assign no_uplink_err = evaluate && (edge_ports != 16'd0) && (uplink_ports == 16'd0);
  // ... evaluation counters omitted for length
endmodule

  oversub: edge=12800 uplink=3200 ratio=4.00 over=4 of 6

Six configurations:

The last two rows matter more than they look. A tier with edge ports and no uplink cannot forward anything, and reporting an infinite ratio is more useful than reporting an error. A tier with no ports at all is not a missing uplink — it is not a tier, and a checker that fires there fires on every unpopulated slot in the fabric.

8. Waveform — A Path Deepening By Tier

Transcribed from the printed trace. One stimulus stream, both builds.

9. RTL 4 — Consolidation Grows The Blast Radius

// The blast radius grows as the fabric consolidates.
module blast_radius #(parameter int IGNORE_DOMAIN = 0) (
  input  logic clk, rst_n,
  input  logic        analyse,
  input  logic [15:0] endpoints, switches,
  input  logic [15:0] split_limit,
  output logic [15:0] per_switch, worst_domain,
  output logic [7:0]  blast_pct,
  output logic        over_limit, unbounded_err
);
  logic [31:0] bp_q;
  // Endpoints spread evenly, rounded up: the last switch carries the remainder.
  assign per_switch = (switches == 16'd0) ? 16'd0
                    : ((endpoints + switches - 16'd1) / switches);
  // The ignoring build reports the fabric as one domain's worth of loss, which
  // is what a model does when it has no concept of a failure boundary.
  assign worst_domain = (IGNORE_DOMAIN != 0) ? 16'd0 : per_switch;
  // ... blast share omitted for length
  assign over_limit = (per_switch > split_limit);
  // A fabric whose worst domain is unreported has an unbounded blast radius.
  assign unbounded_err = analyse && (endpoints != 16'd0) && (worst_domain == 16'd0);
endmodule

  blast: per_switch=16 worst=16 pct=25% | ignoring unbounded=6

64 endpoints across a shrinking switch count:

Halving the switch count doubles what one failure takes. The endpoints did not change, the topology did not change, and the fabric became half as resilient — which is the trade every consolidation makes and the one that is easiest to make accidentally.

The division rounds up, because the remainder lands on some switch and that switch is the worst domain. Rounding down reports 12 where the truth is 13, and a split limit checked against the wrong number is not checked.

The split-limit comparison is strict and constructed exactly: 60 endpoints over 3 switches is precisely 20 each, which is at a 20-endpoint limit and inside it. 63 over 3 is 21 and is not.

Two degenerate cases are driven. A fabric with no switches carries nobody and is correctly reported as unbounded — there is no domain to bound. A fabric with no endpoints has nothing to lose and is correctly reported as fine.

10. RTL 5 — Path Diversity

// Path diversity: how many disjoint paths a pair has, and what one failure does
// to the rest.
module path_diversity #(parameter int SINGLE_PATH = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [3:0]  spines, failed,
  input  logic [15:0] per_path_gbps,
  output logic [3:0]  paths, surviving,
  output logic [31:0] capacity_gbps, surviving_gbps,
  output logic [7:0]  loss_pct,
  output logic        connected, spof_err
);
  logic [31:0] lp_q;
  // One disjoint path per spine, unless the design has only one.
  assign paths     = (SINGLE_PATH != 0) ? 4'd1 : spines;
  assign surviving = (failed >= paths) ? 4'd0 : (paths - failed);
  assign capacity_gbps  = {16'd0, per_path_gbps} * {28'd0, paths};
  assign surviving_gbps = {16'd0, per_path_gbps} * {28'd0, surviving};
  assign connected = (surviving != 4'd0);
  // ... loss share omitted for length
  // A fabric where one failure disconnects a pair has a single point of failure.
  assign spof_err = evaluate && (paths == 4'd1);
endmodule

  paths: paths=4 surviving=4 loss=0% | single-path spof=4

Four spines, failing one at a time:

The single-path build is disconnected by the same one failure that costs the diverse build 25%. Both fabrics have four spines; only one of them routes across all four.

The surviving guard is failed >= paths, and the last row is why. An unsigned paths - failed at four paths and six failures produces 14 — a fabric reporting more surviving paths after a total failure than it had before it.

spof_err fires on every evaluation of the single-path build and never on the diverse one. It is a property of the design, not of the current failure state: a fabric with one path between a pair has a single point of failure whether or not anything has failed yet.

11. RTL 6 — Incast

// Congestion at scale: many senders onto one receiver, and what each gets.
module incast #(parameter int ASSUME_FAIR_FULL = 0) (
  input  logic clk, rst_n,
  input  logic        send,
  input  logic [15:0] senders, want_gbps, receiver_gbps,
  output logic [31:0] demand_gbps,
  output logic [15:0] each_gets_gbps,
  output logic [7:0]  overrun_x,
  output logic        congested, starve_err
);
  logic [31:0] eg_q, ov_q;
  assign demand_gbps = {16'd0, senders} * {16'd0, want_gbps};
  assign congested   = (demand_gbps > {16'd0, receiver_gbps});
  // Under congestion each sender gets the receiver's rate divided by the sender
  // count. The assuming build hands every sender what it asked for.
  assign eg_q = (ASSUME_FAIR_FULL != 0) ? {16'd0, want_gbps}
              : ((senders == 16'd0) ? 32'd0
                 : (congested ? ({16'd0, receiver_gbps} / {16'd0, senders})
                              : {16'd0, want_gbps}));
  // ... overrun and starvation check omitted for length
endmodule

Eight senders of 400 Gbps onto one 400 Gbps receiver:

  incast: senders=8 demand=3200 each=50 overrun=8x | assuming starved=2

Each sender gets 50 Gbps — an eighth of what its own link can carry. Nothing is broken, nothing is oversubscribed at any individual link, and the aggregate demand is 8× what the destination can absorb.

Incast is the failure mode that scale creates for free: the more endpoints a fabric has, the more of them can address the same destination at once, and the per-sender rate falls as 1/n with no warning from any individual link's statistics.

The congestion test is strict and driven exactly: two senders of 200 demand precisely the receiver's 400 and are not congested; 201 each is.

12. RTL 7 — What A Switch Actually Buys

// What each added switch actually buys, once its own uplinks are paid for.
module scaling_efficiency #(parameter int LINEAR_ASSUMED = 0) (
  input  logic clk, rst_n,
  input  logic        add,
  input  logic [15:0] switch_ports, uplink_ports, switches,
  output logic [15:0] usable_per_switch,
  output logic [31:0] total_endpoints, ideal_endpoints,
  output logic [7:0]  efficiency_pct,
  output logic        efficient, overcount_err
);
  logic [31:0] ef_q;
  // A switch spends some of its ports reaching the rest of the fabric. Only the
  // remainder can hold endpoints.
  assign usable_per_switch = (uplink_ports >= switch_ports) ? 16'd0
                                                            : (switch_ports - uplink_ports);
  assign total_endpoints = (LINEAR_ASSUMED != 0)
                         ? ({16'd0, switch_ports} * {16'd0, switches})
                         : ({16'd0, usable_per_switch} * {16'd0, switches});
  assign ideal_endpoints = {16'd0, switch_ports} * {16'd0, switches};
  // ... efficiency share omitted for length
  // Counting uplink ports as endpoint ports.
  assign overcount_err = add && (uplink_ports != 16'd0)
                         && (total_endpoints == ideal_endpoints);
endmodule

Four 32-port switches:

  scaling: usable=24 total=96 ideal=128 eff=75% | linear overcounts=4

Four 32-port switches do not hold 128 endpoints. They hold 96, because 32 of their 128 ports are reaching each other. The linear model reports 128 and is correct only for the zero-uplink case — which is a single switch, which is the configuration this chapter exists to move beyond.

The usable_per_switch guard exists for the last row. Forty uplinks on a 32-port switch is a configuration error, and an unguarded subtraction reports 65,528 usable ports on a switch that has 32.

The no-uplink case is driven and is not an overcount: a fabric with no uplinks genuinely holds every port's worth of endpoints, and a checker that fired there would fire on every single-switch deployment.

13. RTL 8 — Mean Hops Across A Population

// Mean hops across a population of pairs, which is what a fabric-wide latency
// figure actually averages.
module mean_hops (
  input  logic clk, rst_n,
  input  logic        sample,
  input  logic [15:0] n_local, n_intra, n_inter,
  input  logic [3:0]  h_local, h_intra, h_inter,
  input  logic [15:0] per_hop_ns,
  output logic [31:0] total_hops, n_pairs, total_ns,
  output logic [15:0] mean_hops_x10, mean_ns,
  output logic [3:0]  worst_hops,
  output logic        mean_hides_worst_err
);
  // ... arithmetic omitted for length
  // The worst hop count that any pair in the population actually uses.
  assign worst_hops = (n_inter != 16'd0) ? h_inter
                    : ((n_intra != 16'd0) ? h_intra : h_local);
  // A mean under half the worst is describing a distribution it does not cover.
  assign mean_hides_worst_err = sample && (n_pairs != 32'd0)
                                && ({28'd0, worst_hops} * 32'd10
                                    > {16'd0, mean_hops_x10} * 32'd2);
endmodule

A thousand pairs, mostly local:

  hops: pairs=1000 total=1220 mean=1.2 worst=5 mean_ns=73

A mean of 1.2 hops on a fabric whose worst path is 5. The number is correct and it describes a fabric where one pair in a hundred takes four times the mean — which is 18.1 section 7's tail argument arriving as topology rather than as queueing.

worst_hops is the deepest path in use, not the deepest the topology supports. The third row proves it: with no inter-group pairs the worst any pair experiences is three hops, and a fabric reporting five because a super-spine exists somewhere is describing paths nobody is taking.

14. RTL 9 — What The Fabric Costs Itself

// What a fabric costs per endpoint, in switches and in ports.
module fabric_cost (
  input  logic clk, rst_n,
  input  logic        price,
  input  logic [15:0] endpoints, switches, ports_per_switch, uplink_ports,
  output logic [31:0] total_ports, endpoint_ports, overhead_ports,
  output logic [7:0]  overhead_pct, switches_per_100,
  output logic        overhead_dominates, free_fabric_err
);
  logic [31:0] op_q, sp_q;
  assign total_ports    = {16'd0, ports_per_switch} * {16'd0, switches};
  assign overhead_ports = {16'd0, uplink_ports}     * {16'd0, switches};
  assign endpoint_ports = total_ports - overhead_ports;
  // ... shares omitted for length
  // A fabric with uplinks reporting no overhead is not counting its own cost.
  assign free_fabric_err = price && (uplink_ports != 16'd0) && (overhead_pct == 8'd0);
endmodule

  cost: ports=128 overhead=32 (25%) endpoints=96 switches_per_100=4

The last row is the interesting one and it is a real reporting failure rather than a contrived one. A very wide switch with a handful of uplinks has an overhead that rounds to zero percent, and a fabric reporting zero overhead while spending five ports on uplinks is reporting a cost it does not show. free_fabric_err exists for exactly that case, and it was unreachable until the bench drove a wide enough switch — which made it a dead checker until then.

switches_per_100 is the number a bill of materials is actually built from: four switches per hundred endpoints, at this port count and uplink ratio. Change either and the number changes, which is why it is derived rather than assumed.

15. RTL 10 — The Scaling Model Assembled

// The fabric-scaling model assembled: every term a scale claim needs.
module fabric_scaling_model #(parameter int PORT_COUNT_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       hops_counted,      // the topology's hop count is charged
  input  logic       bisection_known,   // the cut is modelled
  input  logic       oversub_stated,    // the edge-to-uplink ratio is stated
  input  logic       blast_bounded,     // the worst failure domain is bounded
  input  logic       overhead_counted,  // uplink ports are not counted as endpoints
  output logic       credible,
  output logic [4:0] fail_mask,
  output logic [7:0] n_eval, n_credible,
  output logic       port_count_err
);
  assign fail_mask[0] = ~hops_counted;
  assign fail_mask[1] = ~bisection_known;
  assign fail_mask[2] = ~oversub_stated;
  assign fail_mask[3] = ~blast_bounded;
  assign fail_mask[4] = ~overhead_counted;
  // The port-counting build multiplies switches by ports and stops there.
  assign credible = (PORT_COUNT_ONLY != 0) ? hops_counted : (fail_mask == 5'd0);
  assign port_count_err = evaluate && credible && (fail_mask != 5'd0);
  // ... evaluation counters omitted for length
endmodule

  model: evaluated=6 credible=1 | port-count-only credible=5

One credible scale claim out of six, and the port-counting build found five. Its four extra are the bisection, the oversubscription ratio, the blast radius and the port overhead — every term that is a property of the topology rather than of the switch:

This is the widest assembled-model gap in either batch, and it has the same shape as 18.3 section 15: a model built from the component specification sees the component and nothing about the system it is in.

16. Quantitative Reasoning

Every number is from a printed line above. None describes any product.

Hops. Direct 0, one switch 1, two-tier 3, three-tier 5. At 60 ns per hop on a 150 ns base: 150, 210, 330, 450 — the deepest path is three times the base.

Bisection. 8 × 400 = 3200 Gbps cut. 16 × 400 = 6400 demanded, 200% utilisation, 200 Gbps per flow on 400 Gbps links.

Oversubscription. 32 edge over 8 uplink is 4.00 : 1. Over 31 uplinks it is 1.03 : 1 — which times-ten precision rounds to 1.0 and calls balanced.

Blast radius. 64 endpoints: over 5 switches 13 each, over 4 16, over 2 32. Halving the switches doubles the blast radius to 50% of the fabric.

Path diversity. Four spines, one failure: 25% capacity lost, still connected. One path, one failure: 100% lost, disconnected.

Incast. 8 senders × 400 = 3200 onto 400 — an 8× overrun, 50 Gbps each.

Scaling efficiency. 4 × 32 = 128 ports, 8 uplinks each, 96 endpoints — 75%. At 16 uplinks, 50%. At 32, zero.

Mean hops. 900/90/10 gives 1220 hops over 1000 pairs — mean 1.2, worst in use 5, a tail 4.2× the mean.

Fabric cost. 32 of 128 ports are uplinks — 25% overhead, 4 switches per 100 endpoints. A 1000-port switch with 5 uplinks reports 0%.

17. Assertions

Every property is an immediate check written as cond !== 1'b1, sampled after a settle. 188 assertion sites across two testbenches.

18. Mutation Testing

78 mutations, one at a time, each required to make the baseline print RESULT: FAIL.

78 of 78 were killed.

The first run killed 69 and left 9 survivors — the largest first-run survivor count of the batch, and the classes are instructive:

Four of the nine were degenerate cases — a fabric with no switches, one with no endpoints, a tier with no ports, and a switch with more uplinks than ports. Each is a configuration nobody intends and each is reachable from a typo in a topology file, and none of them was in the stimulus until a mutation pointed at it.

The provably-equivalent survivor was the survival guard at failed == paths: both >= and > yield zero survivors there, so the mutation cannot change any output. Replaced with one that reports full survival after a total failure.

A representative sample:

19. Verification Strategy

Two builds, one stimulus. The parameter is the only difference.

Drive every degenerate case. No switches, no endpoints, no ports, more uplinks than a switch has. Four mutations survived on configurations nobody would deploy and a typo would produce.

Guard every unsigned subtraction, then drive past the guard. Six failures of four paths; forty uplinks on a thirty-two-port switch. Both produce a maximum rather than a zero without the guard.

Construct the boundary. 60 endpoints over 3 switches for exactly 20; 16 uplinks of 32 for exactly 50%; 64 uplinks against 32 edge ports for the under-subscribed case.

Check the precision, not only the value. The oversubscription ratio at times-ten rounds 1.03 to 1.0, which reads as balanced on an oversubscribed tier. Times a hundred distinguishes them, and only a boundary case reveals the difference.

A checker that never fires is not verified. free_fabric_err was unreachable until the bench drove a 1000-port switch. Look for the configuration that reaches it before concluding it is correct.

Drive the case where the shortcut is right. The single-switch topology, where the one-hop assumption is exact. A fabric with no uplinks, where the linear count is exact. In both, a checker that fired would fire on the common case.

20. Synthesis and Implementation Reality

None of this is hardware. These are executable specifications for a topology analysis, and their value is that they force numbers a spreadsheet lets you skip.

Bisection is not always a clean cut. Section 6 assumes a topology with an identifiable bisection and flows that spread evenly across it. Real fabrics have irregular topologies, and the achieved per-flow rate depends on the routing spreading load — which is 16.2's ground and is deliberately assumed here.

The blast radius in section 9 is a lower bound. It treats each switch as an independent failure domain. Shared power, shared cooling and shared management make real domains larger than the topology suggests, and a rack-level failure takes every switch in the rack regardless of how the endpoints are distributed.

Uplink count is not free to choose. Section 12 treats uplinks as a parameter. In practice it is constrained by the spine port count, the cabling, and the oversubscription target — and changing it changes section 7's ratio and section 14's cost simultaneously.

Hop latency is not constant across hops. Section 5 charges a uniform 60 ns. A leaf-to-spine hop and a spine-to-super-spine hop may differ, and 18.1 section 9 established that any hop's cost rises with its utilisation — so a deep path through a loaded spine costs more than the hop count suggests.

21. Silicon Observability

The last row is the one that is hardest and most valuable. Incast does not appear in any individual link's statistics — every link is within its rate, the receiver's link is saturated, and only a count of concurrent senders per destination explains why each of them is getting an eighth of what it asked for.

22. Debug Lab

Symptom: a fabric performs well at small scale and badly once it is populated.

Count the hops on the slow paths. Section 5: a two-tier path is three hops, not two, and a three-tier path is five. If the benchmark ran within one leaf, it measured a different topology.

Compute the bisection and compare it against the crossing demand. Section 6: sixteen flows across an eight-link cut get half rate each, and no individual link reports a problem.

Derive the oversubscription ratio per tier. Section 7: it is two multiplications and a division, and it is the single number that says how much offered load can leave.

Count concurrent senders per destination. Section 11: an 8× incast gives each sender an eighth, and every link involved looks healthy.

Check whether the endpoints are spread or consolidated. Section 9: the same endpoints over half the switches double the blast radius, which is not a performance problem until it is the only problem.

Recompute the endpoint capacity after uplinks. Section 12: four 32-port switches hold 96, not 128, and a capacity plan built on 128 is 33% over.

23. Design Review

How many hops is the worst path in this fabric, and how many pairs take it?

What is the bisection bandwidth, and what is the crossing demand at peak?

What is the oversubscription ratio at each tier? If nobody can state it, the fabric's ability to carry offered load is unknown.

How many endpoints does one switch failure take with it, and is that inside the limit?

How many disjoint paths does a pair have? One is a single point of failure whatever the switch count is.

How many ports does the fabric spend on itself? And is the endpoint capacity in the plan the gross or the net number?

24. How This Appears In Real Engineering

Fabric-scaling problems arrive as a fabric that behaved well in a pilot and badly in production, with nothing identifiably broken.

The characteristic case is a pilot inside one leaf. Every path was one hop, no traffic crossed the bisection, and the measured latency was section 5's second row. Production spans leaves, every path is three hops, and the latency rose 57% with no change to any component.

The second is a capacity plan built on gross port count. Section 12: 128 ports of switch hold 96 endpoints, and the missing 32 are discovered during deployment.

The third is incast. A workload where many nodes read from one, every link healthy, and each reader getting a fraction of its rate that tracks the reader count exactly.

The fourth is a consolidation that quietly doubled the blast radius. Nothing about performance changed; the first switch failure took twice what the previous topology would have.

25. Common Misconceptions

"A two-tier fabric is two hops." It is three: leaf, spine, leaf. The return leaf is the one that is forgotten, and it is 60 ns.

"Every link is 400 Gbps so every flow gets 400." Not across a cut. Sixteen flows over an eight-link bisection get 200 each, and no link is at fault.

"We are not oversubscribed — all the links are the same speed." Oversubscription is a ratio of port counts, not of link speeds. Thirty-two edge ports over eight uplinks is 4:1 whatever the speed.

"Adding switches adds capacity linearly." It adds capacity minus the uplinks each new switch spends reaching the others. At eight uplinks of thirty-two, that is 75% efficiency.

"Four spines means four times the bandwidth." It means four disjoint paths — which is four times the bandwidth and the ability to lose one and keep 75%. The second property is the one that justifies the cost.

"Consolidating onto fewer switches is cheaper." It is, and it doubles the blast radius per halving. That is a trade, not a saving.

"The mean hop count is 1.2 so the fabric is flat." The worst path in use is five. The mean is correct and one pair in a hundred is four times it.

"Every link is under 50% so there is no congestion." Incast does not show in link utilisation on the sender side. Eight senders at full rate onto one receiver leaves every sender's link busy and each getting an eighth.

26. Interview Reasoning

Q1. How many hops is a leaf-to-leaf path in a two-tier fabric? Three — leaf, spine, leaf. The return leaf is a hop and it is the one most often missed.

Q2. Does scaling from 100 to 1000 endpoints change the hop count? Only if it changes the topology. Adding leaves under the same spine does not; adding a tier changes every path in the fabric.

Q3. Sixteen flows of 400Gbps across an eight-link, 400Gbps-per-link cut. What does each get? 200. The cut is 3200 and the demand is 6400, so each flow gets half what its own link could carry.

Q4. How would either endpoint of that flow know? It would not. The cut is a property of the topology, and neither end has visibility into how many other flows are crossing with it.

Q5. What is the oversubscription ratio of a switch with 32 edge ports and 8 uplinks? 4:1. It is a ratio of port counts, not of link speeds.

Q6. Why carry that ratio at two decimal places rather than one? Because 32 over 31 is 1.03, which floors to 1.0 at one decimal and reads as balanced on a tier that is oversubscribed.

Q7. You consolidate 64 endpoints from four switches onto two. What changed? The blast radius doubled — from 16 endpoints to 32, from a quarter of the fabric to half. Nothing about performance moved.

Q8. Four spines, one fails. What do you lose? 25 percent of the capacity, and you stay connected. With one path you lose everything and disconnect.

Q9. Eight senders at 400Gbps onto one 400Gbps receiver. What does each get? 50. And every link in the fabric is within its rate, which is why incast is invisible in per-link statistics.

Q10. How many endpoints do four 32-port switches hold? 96, at eight uplinks each. Not 128 — 32 ports are spent reaching each other.

Q11. What does the efficiency of that fabric depend on? The uplink fraction. Zero uplinks is 100 percent and a single switch; half the ports as uplinks is 50 percent; all of them is zero.

Q12. Mean hop count 1.2, worst path 5. Is the fabric flat? On average. One pair in a hundred takes four times the mean, and if those are the pairs that matter the mean is describing somebody else's traffic.

Q13. Why is the worst hop count "in use" rather than the deepest the topology supports? Because a path nobody takes produces no latency anybody experiences. With no inter-group traffic the worst in use is three, not five.

Q14. A fabric reports zero port overhead and has uplinks. What is happening? The overhead is rounding away — a wide switch with a few uplinks is well under one percent. The ports are still spent; the report is not showing them.

Q15. What can a switch datasheet tell you about a fabric's scale? The hop cost of one switch. Bisection, oversubscription, blast radius and port overhead are all properties of the topology, which is why the port-counting model in section 15 passes one gate in five.

Q16. A pilot inside one leaf performs well and production does not. First hypothesis? The pilot was one hop and production is three. A 57 percent latency increase with no component change, before any congestion is considered.

27. Exercises

1. Extend RTL 1 with a topology where leaf-to-spine and spine-to-super-spine hops cost differently. Which assertions become per-hop rather than per-path?

2. In RTL 2, make the flows spread unevenly across the cut. At what imbalance does the worst flow's rate fall below half the fair share?

3. RTL 3 computes a ratio per tier. Compose two tiers and derive the end-to-end ratio. Is it the product, the maximum, or neither?

4. Add correlated failures to RTL 4 — a rack containing several switches. At what rack size does the rack become the failure domain rather than the switch?

5. In RTL 5, make paths non-disjoint, sharing some links. How does the surviving capacity after one failure change, and does spof_err still mean what it meant?

6. RTL 6 divides the receiver's rate evenly. Model a scheme that gives a share proportional to demand and find the sender count at which the smallest sender is starved.

7. Using RTL 7 and RTL 9 together, find the uplink count that maximises endpoints per switch subject to a 2:1 oversubscription target.

8. Add a sixth term to RTL 10 for routing convergence time after a failure. Is it knowable from the topology, and where does it belong in the mask?

28. Summary

A fabric that grows does not grow uniformly, and this chapter builds every term that changes.

Hops come from the topology. Zero, one, three and five — a 450 ns path against a 150 ns base, and the endpoint count appears nowhere in that list.

Every crossing flow passes through a cut. Sixteen flows over a 3200 Gbps bisection get 200 Gbps each on 400 Gbps links, with no link at fault.

Oversubscription is a ratio of port counts. 4.00 : 1 at eight uplinks — and 1.03 : 1 at thirty-one, which one decimal place rounds to balanced.

Consolidation doubles the blast radius per halving. 64 endpoints over four switches is 16 each; over two it is 32, half the fabric.

Diversity is what survives. Four spines lose 25% to one failure; one path loses everything.

Incast is invisible per link. Eight senders onto one receiver: 50 Gbps each, an 8× overrun, every link healthy.

And a fabric spends ports on itself. Four 32-port switches hold 96 endpoints, not 128 — 75% efficiency, and a plan built on the gross number is a third over.

18.5 — Performance-Analysis Discipline takes the three performance chapters and asks the question they all defer: how to turn any of this into a model somebody can act on.