Switch Scalability

16.1 built the switch. 16.2 decided where flits go. 16.3 decided who gets the slot.

All three assumed one switch, of some size. This chapter asks what stops it growing, and what the second one costs.

1. The Engineering Problem — Growth Stops Somewhere, And It Matters Where

Six things decide how a switched fabric scales.

A switch has four independent ceilings — ports, routing table, buffering and pins — reached at different sizes. Which one binds first is the only one worth engineering around; after two are reached they are indistinguishable and they need different work. Section 5.

The crossbar is quadratic and the buffering is linear, so the shape of the cost changes as a switch grows. That crossover is what says when a second switch is cheaper than a bigger one. Section 6.

The second switch adds a hop to every pair that crosses between them, and what that costs depends entirely on how much traffic actually crosses — a placement property, not a switch one. Section 8.

The link between them is a hard ceiling. It is 15.2's bisection seen from inside a two-switch design. Section 9.

Credits do not cross a switch boundary. Each hop has its own flow control, and a design that lets one hop's credit authorise a slot on the next has promised something it does not own. Section 10.

And every switch added is another failure domain and another element the fabric manager must reach, configure and quiesce. Sections 11 and 12.

This chapter against the rest of Module 16, stated precisely. 16.1, 16.2 and 16.3 own how a switch works. This one owns how big it can be and what happens past that. If a section here could be moved into any of them without loss, it is in the wrong chapter.

2. The One-Sentence Model

A switch grows until one of four ceilings binds, and past it the fabric grows by adding switches — which costs a hop, an uplink, a failure domain and a managed element — and every defect below is one of those four costs uncounted.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Sixteen ports, a 64-entry table, a three-wide uplink, two switches of five endpoints each. A real switch is larger in every dimension and a real fabric has more than two.

What is faithful: four independent ceilings with a first-binding record, the quadratic-versus-linear cost shape, the crossing-share latency argument, the uplink as a hard ceiling, per-hop flow control, the failure domain per switch, and management cost scaling with switch count.

What is not: every ceiling value, every latency, and the two-switch limit.

Every model is parameterised so the correct behaviour and a specific plausible failure are the same source under a different parameter, driven by one stimulus stream.

5. RTL 1 — What Stops A Switch Growing

module single_ceiling #(parameter int NO_CEILING_CHECK = 0) (
  input  logic clk, rst_n,
  input  logic       add_port,
  input  logic [7:0] ports, table_entries, buffer_kb, pins,
  output logic [3:0] limit_mask,
  output logic       accept,
  output logic       over_ceiling_err,
  output logic [7:0] binding_limit, n_added, n_refused
);
  localparam logic [7:0] MAX_PORTS = 8'd16, MAX_TABLE = 8'd64;
  localparam logic [7:0] MAX_BUF   = 8'd48, MAX_PINS  = 8'd40;
  // Four independent ceilings, each a different physical quantity.
  assign limit_mask = {pins >= MAX_PINS, buffer_kb >= MAX_BUF,
                       table_entries >= MAX_TABLE, ports >= MAX_PORTS};
  assign accept = add_port && ((limit_mask == 4'd0) || (NO_CEILING_CHECK != 0));
  assign over_ceiling_err = accept && (limit_mask != 4'd0);
  ...
      // Which ceiling was reached FIRST is the one that limits the design.
      if ((bl_q == 8'd0) && (limit_mask != 4'd0)) begin
        if      (limit_mask[0]) bl_q <= 8'd1;   // ports
        else if (limit_mask[1]) bl_q <= 8'd2;   // routing table
        else if (limit_mask[2]) bl_q <= 8'd3;   // buffering
        else                    bl_q <= 8'd4;   // pins
      end
    end
  end
endmodule

  ceiling : first limit=4 (pins), refused=2 | no-check build grew past it=1

Four ceilings and four different physical quantities: package pins, SRAM for the routing table, SRAM for the buffers, and the port logic itself. They are reached at different sizes and they are relieved by completely different work — a bigger package, a smaller table, shallower buffers, or nothing at all.

binding_limit is latched at the first ceiling reached and never rewritten. The testbench then drives every other ceiling and asserts it is still the pins, because a "first limit" that later ceilings can overwrite reports whichever constraint appears first in the priority chain rather than whichever one actually stopped the design.

The mask bit order matters and is a mutation: transposing two fields reports the wrong resource with complete confidence.

6. RTL 2 — What The Crossbar Costs As It Grows

module xbar_cost (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [7:0]  n_ports, per_port_buf,
  output logic [15:0] xbar_cells, buffer_cells, total_cells,
  output logic [7:0]  xbar_share_pct, cells_per_port
);
  logic [15:0] np;
  logic [31:0] w_share;
  assign np      = {8'd0, n_ports};
  assign w_share = {16'd0, xbar_cells} * 32'd100;
  // The crossbar is ports squared; the buffering is ports times depth.
  assign xbar_cells   = np * np;
  assign buffer_cells = np * {8'd0, per_port_buf};
  assign total_cells  = xbar_cells + buffer_cells;
  assign xbar_share_pct = (total_cells == 16'd0) ? 8'd0
                        : (w_share / {16'd0, total_cells});
  // Cost per port is what actually rises: the quadratic divided by the linear.
  assign cells_per_port = (n_ports == 8'd0) ? 8'd0 : (total_cells / np);

  crossbar: 4 ports=48 cells (33% xbar, 12/port) | 16 ports=384 cells (66% xbar, 24/port)

Four times the ports, eight times the cost, and the cost per port doubles. That is the entire scaling argument in one line: the crossbar's share rises from a third to two thirds, and past some size the marginal port is mostly crossbar.

The per-port figure is what makes the comparison with a second switch possible. A 16-port switch costs 24 cells per port; two 8-port switches cost 16 each plus an uplink — and section 13 is where that comparison is netted out.

Both guards are driven: a switch with no ports costs nothing and its cost per port is zero rather than a division, and the crossbar share is measured against the total, not against the crossbar, which would be 100 percent always.

7. Waveform — Eight Cycles At A Two-Switch Uplink

Transcribed from the printed trace. Both uplink widths see one stimulus stream.

Read xfer against in_flt at cycle 0. Local traffic leaves the uplink completely empty — which is the property section 9's monitor exists to check, and the one a design that charges every transfer to the uplink gets wrong.

8. RTL 3 — What The Second Switch Costs A Transfer

module multi_switch #(parameter int IGNORE_CROSSING = 0) (
  input  logic clk, rst_n,
  input  logic        xfer,
  input  logic [2:0]  src_sw, dst_sw,
  input  logic [7:0]  local_ns, hop_ns,
  output logic        crosses,
  output logic [15:0] latency_ns, n_local, n_cross, total_ns, max_ns,
  output logic [7:0]  cross_pct, mean_ns
);
  logic [15:0] n_all;
  logic [31:0] w_cross;
  assign n_all   = n_local + n_cross;
  assign w_cross = {16'd0, n_cross} * 32'd100;
  assign crosses = xfer && (src_sw != dst_sw);
  // IGNORE_CROSSING prices every transfer as local, which is how a two-switch
  // design gets signed off on a one-switch latency model.
  assign latency_ns = (IGNORE_CROSSING != 0) ? {8'd0, local_ns}
                    : (crosses ? ({8'd0, local_ns} + {8'd0, hop_ns})
                               : {8'd0, local_ns});
  assign cross_pct = (n_all == 16'd0) ? 8'd0 : (w_cross / {16'd0, n_all});
  assign mean_ns   = (n_all == 16'd0) ? 8'd0 : (total_ns / n_all);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_local <= 16'd0; n_cross <= 16'd0; total_ns <= 16'd0; max_ns <= 16'd0;
    end else if (xfer) begin
      if (crosses) n_cross <= n_cross + 16'd1;
      else         n_local <= n_local + 16'd1;
      total_ns <= total_ns + latency_ns;
      if (latency_ns > max_ns) max_ns <= latency_ns;
    end
  end
endmodule

At 120ns local and a 60ns hop:

  two-sw  : 10% crossed, mean=126ns worst=180ns | ignore-crossing model reports 120ns

Ten percent crossing costs 6ns on the mean. The same model driven to a mostly-crossing workload costs over 45ns. Same hardware, same hop; the entire difference is where the endpoints were placed.

That is the honest form of the multi-switch latency argument, and it is the same shape as 15.5 section 10's pooled-memory argument: the hardware sets a per-crossing cost and the workload sets how often it is paid.

IGNORE_CROSSING reports 120ns for both workloads, and its worst case never rises above the local one. A two-switch design signed off on a one-switch latency model is not wrong about a particular workload — it cannot be wrong about any.

9. RTL 4 — The Link Between Two Switches

module uplink_load #(parameter int NARROW_UPLINK = 0) (
  input  logic clk, rst_n,
  input  logic       xfer, crosses, uplink_free,
  output logic [3:0] uplink_width, in_flight,
  output logic       accept, uplink_full_err,
  output logic [7:0] n_crossing, n_stalled, peak_uplink, stall_pct
);
  logic [3:0]  inf_q;
  logic [15:0] weighted;
  assign in_flight = inf_q;
  assign weighted  = {8'd0, n_stalled} * 16'd100;
  assign uplink_width = (NARROW_UPLINK != 0) ? 4'd1 : 4'd3;
  assign accept = xfer && (!crosses || (inf_q < uplink_width));
  // A crossing transfer refused because the uplink is full.
  assign uplink_full_err = xfer && crosses && (inf_q >= uplink_width);
  assign stall_pct = (n_crossing == 8'd0) ? 8'd0
                   : (weighted / {8'd0, n_crossing});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      inf_q <= 4'd0; n_crossing <= 8'd0; n_stalled <= 8'd0; peak_uplink <= 8'd0;
    end else begin
      if (xfer && crosses) begin
        n_crossing <= n_crossing + 8'd1;
        if (inf_q < uplink_width) begin
          inf_q <= inf_q + 4'd1;
          if ({4'd0, inf_q} + 8'd1 > peak_uplink)
            peak_uplink <= {4'd0, inf_q} + 8'd1;
        end else n_stalled <= n_stalled + 8'd1;
      // Guarded: freeing an empty uplink must not wrap the count.
      end else if (uplink_free && inf_q != 4'd0) inf_q <= inf_q - 4'd1;
    end
  end
endmodule

  uplink  : width=3 peak=3 stalled=1 (20%) | narrow build stalled=3

One stall against three, on the identical stream. The uplink is a count of transfers, not a bandwidth — a crossing transfer occupies one slot until it completes, and no amount of switch throughput on either side changes how many fit.

The !crosses term is what makes local traffic free, and the testbench drives it with the uplink completely full: five local transfers with three crossing transfers in flight, all accepted. A design that charges every transfer to the uplink works perfectly until the uplink fills, at which point it stalls traffic that never needed it.

The stall share is measured against crossing transfers, not against stalls — twenty percent of the crossing traffic was stalled, which is the number that sizes the uplink.

10. RTL 5 — Credits Do Not Cross A Switch Boundary

module cascade_credit #(parameter int END_TO_END = 0) (
  input  logic clk, rst_n,
  input  logic       send,
  input  logic [3:0] hop1_credits, hop2_credits,
  input  logic       hop1_free, hop2_free,
  output logic       go_hop1, go_hop2,
  output logic       foreign_credit_err,
  output logic [7:0] n_hop1, n_hop2, n_blocked_hop2, max_block2
);
  logic [7:0] b2_q;
  // Each hop is authorised by its own credits. END_TO_END lets the first
  // hop's credit carry the flit through the second as well.
  assign go_hop1 = send && (hop1_credits != 4'd0);
  assign go_hop2 = go_hop1 && ((END_TO_END != 0) || (hop2_credits != 4'd0));
  // The second hop moved on a credit that belongs to the first.
  assign foreign_credit_err = go_hop2 && (hop2_credits == 4'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      b2_q <= 8'd0; n_hop1 <= 8'd0; n_hop2 <= 8'd0;
      n_blocked_hop2 <= 8'd0; max_block2 <= 8'd0;
    end else begin
      if (go_hop1) n_hop1 <= n_hop1 + 8'd1;
      if (go_hop2) n_hop2 <= n_hop2 + 8'd1;
      if (go_hop1 && !go_hop2) begin
        n_blocked_hop2 <= n_blocked_hop2 + 8'd1;
        b2_q <= b2_q + 8'd1;
        // How long a flit sat between two switches: it has left the first
        // and has not entered the second.
        if (b2_q + 8'd1 > max_block2) max_block2 <= b2_q + 8'd1;
      end else b2_q <= 8'd0;
    end
  end
endmodule

  cascade : hop2 moved=5 blocked=4 for 3 cycles | end-to-end build borrowed a credit=1

This is 16.3 section 5's credit invariant extended across a switch boundary, and the answer is that it does not extend. Each hop's credits belong to that hop's receiver. A flit that has left the first switch and cannot enter the second sits between them, and max_block2 measures exactly that interval.

END_TO_END moves the flit through the second hop on the first hop's credit — which is a slot on a switch the sender does not own and has not been promised. The consequence is 16.3's overflow, one hop away from the accounting error that caused it.

The monitor's second term is what makes it a defect: the first hop being unable to move because it has no credits is ordinary back pressure, and the testbench drives that case and asserts the error stays low in both builds.

11. RTL 6 — What One Switch Failure Takes Down

module switch_blast #(parameter int NO_SPLIT_LIMIT = 0) (...);
  localparam logic [3:0] MAX_PER_SW = 4'd5;
  assign worst_domain = (s0_q >= s1_q) ? s0_q : s1_q;
  // A switch may not hold more than its share of the endpoints.
  assign accept = attach &&
    ((NO_SPLIT_LIMIT != 0)
     || ((which_switch == 2'd0) ? (s0_q < MAX_PER_SW) : (s1_q < MAX_PER_SW)));
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      s0_q <= 4'd0; s1_q <= 4'd0;
      n_attached <= 8'd0; n_refused <= 8'd0; n_lost <= 8'd0; max_domain <= 8'd0;
    end else begin
      if (accept) begin
        if (which_switch == 2'd0) s0_q <= s0_q + 4'd1;
        else                      s1_q <= s1_q + 4'd1;
        n_attached <= n_attached + 8'd1;
      end else if (attach) n_refused <= n_refused + 8'd1;
      // A switch failure takes down everything attached to it -- and nothing
      // attached to the other.
      if (fail_ev) begin
        n_lost <= n_lost + {4'd0, worst_domain};
        if (which_switch == 2'd0) s0_q <= 4'd0;
        else                      s1_q <= 4'd0;
      end
      if ({4'd0, worst_domain} > max_domain) max_domain <= {4'd0, worst_domain};
    end
  end
endmodule

  blast   : worst domain=5 lost=5 refused=3 | no-split-limit build worst domain=8

Adding a second switch adds endpoints and adds a failure domain. Five against eight: the split limit refused three attachments to hold the worst domain at five, and the build without it put every endpoint behind one switch — so one switch failure is the entire fabric.

This is 15.5 section 8's blast radius at switch granularity, and it has the same property: nothing about normal operation makes the number visible. Every individual attachment is legitimate, the fabric behaves identically either way, and the number only matters once.

The failure is driven and its consequence measured — five endpoints lost, and the other switch's four unaffected, which is the isolation 16.1 section 10 built at port granularity now visible at switch granularity.

12. RTL 7 — Reaching Every Switch As The Fabric Grows

module mgmt_scale #(parameter int QUIESCE_ALL = 0) (
  input  logic clk, rst_n,
  input  logic       change_req,
  input  logic [3:0] switches_affected, switches_total,
  input  logic       done,
  output logic [3:0] switches_stopped,
  output logic       changing, overquiesce_err,
  output logic [7:0] n_changes, stopped_cycles, max_stopped, overstop_total
);
  logic       ch_q;
  logic [3:0] stop_q;
  logic [7:0] sc_q;
  assign changing         = ch_q;
  assign switches_stopped = stop_q;
  assign stopped_cycles   = sc_q;
  assign overquiesce_err = ch_q && (stop_q > switches_affected);
  ...
      if (change_req && !ch_q) begin
        ch_q <= 1'b1; sc_q <= 8'd0;
        // QUIESCE_ALL stops the whole fabric for a change that touches part
        // of it, which is 15.1's quiesce applied at the wrong granularity.
        stop_q <= (QUIESCE_ALL != 0) ? switches_total : switches_affected;
      end else if (ch_q && done) begin
        ch_q <= 1'b0; stop_q <= 4'd0;
        n_changes <= n_changes + 8'd1;
      end else if (ch_q) begin
        sc_q <= sc_q + 8'd1;
        if (sc_q + 8'd1 > max_stopped) max_stopped <= sc_q + 8'd1;
        if (stop_q > switches_affected)
          overstop_total <= overstop_total + {4'd0, (stop_q - switches_affected)};
      end
    end
  end
endmodule

  mgmt    : stopped=1 of 4 for 4 cycles | quiesce-all build over-stopped 10 switch-cycles

The management cost is linear in switches — one more element to reach, configure and quiesce. The quiesce cost is not, and that is the point: a change touching one switch of four should stop one, and QUIESCE_ALL stops four.

overstop_total counts switch-cycles unnecessarily stopped: ten against zero, for a change that took four cycles. At fabric scale that is the difference between a maintenance window measured in switches and one measured in fabrics.

A change spanning two switches legitimately stops two, and the testbench drives that to prove the monitor is comparing against what the change actually touches rather than against one.

A second change request while one is open must not restart it — otherwise the interval never accumulates and the change never completes. That is the same guard as 15.3's outstanding-probe rule, in a different place.

13. RTL 8 — What The Second Switch Costs, Net

module scale_cost (
  input  logic clk, rst_n,
  input  logic        tick,
  input  logic [7:0]  ports_one_sw, ports_two_sw,
  input  logic [7:0]  crossing_ns, mgmt_cycles,
  input  logic        served_cycle,
  output logic [15:0] t_one, t_two, t_cost, t_served,
  output logic [7:0]  gain_pct, cost_pct, net_pct
);
  logic [15:0] gained, net_gained;
  logic [31:0] w_gain, w_cost, w_net;
  assign gained     = (t_two <= t_one) ? 16'd0 : (t_two - t_one);
  // The crossing latency and the management overhead come out of the gain.
  assign net_gained = (gained <= t_cost) ? 16'd0 : (gained - t_cost);
  assign gain_pct = (t_one == 16'd0) ? 8'd0 : (w_gain / {16'd0, t_one});
  assign cost_pct = (t_one == 16'd0) ? 8'd0 : (w_cost / {16'd0, t_one});
  assign net_pct  = (t_one == 16'd0) ? 8'd0 : (w_net  / {16'd0, t_one});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      t_one <= 16'd0; t_two <= 16'd0; t_cost <= 16'd0; t_served <= 16'd0;
    end else if (tick) begin
      t_one  <= t_one + {8'd0, ports_one_sw};
      t_two  <= t_two + {8'd0, ports_two_sw};
      // Both halves of the cost, or the gain is a headline.
      t_cost <= t_cost + {8'd0, crossing_ns} + {8'd0, mgmt_cycles};
      if (served_cycle) t_served <= t_served + 16'd1;
    end
  end
endmodule

  cost    : gain=75% cost=18% net=56% of the one-switch baseline

75 percent gross, 56 percent net. The nineteen points are the crossing latency from section 8 and the management overhead from section 12, charged against the ports the second switch adds.

The cost includes both halves, and each is a separate mutation. Charging the crossing and not the management, or the reverse, understates what cascading costs by roughly half.

Both subtractions saturate and both are driven to saturation: a second switch that adds fewer ports than one reports a gain of zero, not a large positive number from an underflow. The testbench also drives a run where the gain per port added is falling, which is what actually happens as switches are added — each one adds ports and adds crossing traffic to all the others.

14. RTL 9 — Auditing The Growth Plan

module growth_audit (
  input  logic clk, rst_n,
  input  logic       propose,
  input  logic [7:0] want_ports, have_ports,
  input  logic [3:0] want_uplink, have_uplink,
  input  logic [3:0] want_domain, max_domain_allowed,
  output logic       ports_ok, uplink_ok, domain_ok, approve,
  output logic       unplanned_growth_err,
  output logic [7:0] n_proposed, n_approved, n_refused, worst_gap
);
  logic [7:0] gap;
  // Saturating: a plan that fits has no gap, not a negative one.
  assign gap = (want_ports <= have_ports) ? 8'd0 : (want_ports - have_ports);
  assign ports_ok  = (want_ports  <= have_ports);
  assign uplink_ok = (want_uplink <= have_uplink);
  assign domain_ok = (want_domain <= max_domain_allowed);
  assign approve   = propose && ports_ok && uplink_ok && domain_ok;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_proposed <= 8'd0; n_approved <= 8'd0;
      n_refused <= 8'd0; worst_gap <= 8'd0;
    end else if (propose) begin
      n_proposed <= n_proposed + 8'd1;
      if (approve) n_approved <= n_approved + 8'd1;
      else         n_refused  <= n_refused + 8'd1;
      if (gap > worst_gap) worst_gap <= gap;
    end
  end
endmodule

  audit   : 2 of 5 approved, worst port gap=4

Five plans: one inside every ceiling, three each exceeding exactly one, and one that exactly fills every ceiling — which is approved, because exactly fitting is fitting. All three boundaries are <= and all three are driven at equality.

worst_gap records the largest port shortfall across every plan proposed. It is the number that says how much a growth plan was out by rather than merely that it was refused, and it saturates when the plan fits — four ports where sixteen are available is not a negative gap.

Each of the three ceilings is driven alone, with the other two satisfied, because a mutation dropping any one leaves the others passing.

15. RTL 10 — Scalability Assembled

module scale_top #(parameter int GROW_ANYWAY = 0) (
  input  logic clk, rst_n,
  input  logic       grow_req,
  input  logic       ceiling_ok, uplink_ok, domain_ok, mgmt_ok,
  output logic       grow,
  output logic [3:0] blocked_by,
  output logic       unbounded_growth_err,
  output logic [7:0] n_requests, n_grown, n_blocked, n_unbounded
);
  assign blocked_by = {~mgmt_ok, ~domain_ok, ~uplink_ok, ~ceiling_ok};
  assign grow = grow_req && ((blocked_by == 4'd0) || (GROW_ANYWAY != 0));
  // Grown past a constraint the fabric cannot carry.
  assign unbounded_growth_err = grow && (blocked_by != 4'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_grown <= 8'd0;
      n_blocked <= 8'd0; n_unbounded <= 8'd0;
    end else if (grow_req) begin
      n_requests <= n_requests + 8'd1;
      if (grow) n_grown   <= n_grown + 8'd1;
      else      n_blocked <= n_blocked + 8'd1;
      if (unbounded_growth_err) n_unbounded <= n_unbounded + 8'd1;
    end
  end
endmodule

  assembled: 1 of 5 grown, mask for mgmt=8 | grow-anyway build grew 5 past 4 constraints

Four constraints, and the assembled model is the point of the chapter: growth is allowed only when the switch has ceiling headroom, the uplink can carry the new crossing traffic, the failure domain stays bounded, and the manager can still reach everything.

The constraint with no natural advocate is the third. Ceiling headroom is visible to whoever designs the switch; uplink capacity is visible to whoever cables it; management reach is visible to whoever operates it. Nothing about normal operation makes the failure domain visible to anybody — which is why it is a counter here rather than a review item, exactly as in 15.5 section 15.

16. Quantitative Reasoning

Three worth a sentence.

Four times the ports, double the cost per port. That is the crossbar's quadratic term showing up as a per-unit price, and it is the number that makes a second switch worth considering at all.

126ns against over 165ns. Same two switches, same 60ns hop. The entire difference is the crossing share, which is a placement decision, and the untiered model reports 120ns for both.

5 endpoints against 8. One switch failure. The split limit refused three attachments to hold that number, and nothing about the fabric's operation would have shown the difference until the failure.

17. Assertions

Every property is an immediate check written as cond !== 1'b1, sampled after a settle.

18. Mutation Testing

88 mutations, one at a time, each required to make the baseline print RESULT: FAIL.

88 of 88 were killed.

The first run killed 83 and left 5 survivors — the lowest first-run survivor count in this batch, which the earlier chapters' lessons explain:

The first row is the one worth recording. accept = xfer && (!crosses || inf_q < width) — the !crosses term makes local traffic free of the uplink, and it was only ever driven with the uplink empty, where both expressions agree. Driving five local transfers with the uplink completely full is what separates them, and it is the case a real fabric hits constantly: local traffic continuing while the inter-switch link is saturated.

A representative sample:

19. Verification Strategy

Parameterised twin builds. NO_CEILING_CHECK, IGNORE_CROSSING, NARROW_UPLINK, END_TO_END, NO_SPLIT_LIMIT, QUIESCE_ALL, GROW_ANYWAY. Every comparison in section 16 is one source under one parameter, driven by one stimulus stream.

Independent oracles. The crossbar cost and the three growth ceilings are checked against plain-integer functions with no reference to the design's expressions.

Drive the exempt case while the resource is exhausted. Local traffic with the uplink full; a first hop with credits while the second has none; a change spanning two switches where only one is over-quiesced. Every exemption term agrees with its absence until the constraint binds.

Latch, clear, and re-enter. The between-hops wait, the stopped interval and the worst failure domain are each driven, cleared, and driven again shorter — because a counter that never resets is invisible on one occurrence.

Boundaries in both directions. A plan that exactly fills every ceiling. A port number exactly at the pin limit. An uplink at exactly its width. A switch with exactly zero ports.

Saturating arithmetic driven to saturation. A second switch that adds fewer ports than one. A growth plan that fits, so the gap is zero rather than a wrapped maximum.

Assert the value, not the range. Every ratio in sections 6, 8, 9 and 13 is asserted to an exact number, following 16.1's five bound-satisfied survivors.

Delta discipline. Every combinational sample follows a settle, and every latched interval is sampled a cycle after the condition that clears it.

20. Synthesis and Implementation Reality

The four ceilings are four different engineering problems and they are owned by different people. The port count is RTL, the routing table and buffers are SRAM compiler instances, and the pin budget is package and board. Section 5's binding_limit matters because the answer decides which of those four teams is asked to do work.

The crossbar's quadratic term is real and it is why switches are not arbitrarily wide. Section 6's cost model is a proxy — a real crossbar's cost is wires and their routing congestion more than cells — but the shape is exactly right, and the shape is what decides between one large switch and two smaller ones.

Hierarchical and partial crossbars are the standard answer and they are outside this chapter. What section 6 establishes is the pressure that produces them: a full crossbar past some port count is mostly crossbar, and every structure that replaces it is trading connectivity for area.

The uplink is one or more physical links and its width in section 9 is a transfer count, not a bandwidth. In silicon the equivalent is credits on the inter-switch link, which is 16.3 section 5 applied across a boundary — and section 10 is what happens when those credits are confused with the endpoint's.

Per-hop flow control is not optional and it is not free. Each hop needs its own credit counters, its own return path, and its own buffer. That per-hop cost is a real part of what a second switch charges, and it does not appear in a port count.

The management cost is linear and the quiesce cost is a policy. Section 12's switches_affected is the granularity a fabric manager chooses to work at, and choosing "all" is much easier to implement. overstop_total is what it costs.

The divisions are firmware. Crossbar share, crossing share, stall share, gain and cost are computed from raw counters by the manager.

21. Silicon Observability

Three to alarm on.

foreign_credit_err non-zero at all means a switch moved a flit on a credit belonging to a different hop. The overflow it causes will appear one hop away from the accounting error, with nothing connecting them.

peak_uplink at the width means the inter-switch link is at its ceiling. Like 16.1's buffer occupancy, it is a predictive signal — every per-switch measurement looks comfortable while the link between them saturates.

max_domain above its designed value means the failure domain has grown past what anyone planned for. It has no operational symptom: the fabric behaves identically, and the number is knowable the entire time.

22. Debug Lab

22.1 A switch cannot be made wider and nobody agrees why

Symptom. A port count increase is refused or fails timing. Different teams give different explanations.

The reading. binding_limit, not limit_mask.

The diagnosis. After more than one ceiling is reached, the mask says several things are full and none of them is the answer. The first ceiling reached is the one that stopped the design, and it names which team has work to do — package, memory compiler, or RTL.

Why the mask alone misleads. Adding a port raises the pin count and consumes table entries and consumes buffer memory. By the time growth is refused, several ceilings are usually near, and relieving the wrong one buys nothing.

22.2 Cross-rack latency is worse than the model predicted

Symptom. A two-switch fabric's measured latency exceeds the projection. Per-switch latency is exactly as designed.

The reading. cross_pct measured, against whatever the projection assumed.

The diagnosis, in two sub-cases. If the measured crossing share is higher than assumed, the hardware is fine and the placement is not — section 8's two runs differ by over 40ns on identical switches, and the entire difference is where the endpoints were put.

If the share matches and the latency still exceeds the projection, check whether the projection modelled crossing at all. IGNORE_CROSSING reports the local latency for every workload, and a two-switch design signed off on it is not wrong about this workload — it cannot be wrong about any.

22.3 Local traffic is fine and cross-switch traffic has stopped

Symptom. Transfers within each switch are healthy. Transfers between them have stalled or slowed sharply.

The reading. peak_uplink against the uplink width, and stall_pct.

The diagnosis. The inter-switch link is at its ceiling. This is invisible to every per-switch metric — both switches report normal occupancy and normal utilisation, because the congestion is on neither of them.

The tell that it is the uplink and not the switches. Local traffic is unaffected. Section 9 drives exactly that: five local transfers accepted with the uplink completely full. If local traffic is also stalling, the uplink is not the problem — or the design is charging local traffic to the uplink, which is one of the mutations.

22.4 A buffer overflowed on a switch two hops from the sender

Symptom. A switch's receive buffer overran. Its own credit accounting is consistent. The sender is two hops away.

The reading. foreign_credit_err on every switch in the path.

The diagnosis. A switch forwarded a flit into the next hop on a credit that belonged to the previous one. Each switch's local accounting is correct — the error is that a credit authorised a slot on a switch that never issued it.

Why it surfaces so far from the cause. The switch that borrowed the credit is fine; the switch that received the flit is the one that overflows. foreign_credit_err is the only signal that names the switch that actually did it, and section 10 is why it must exist per hop rather than end to end.

23. Design Review

1. Which of the four ceilings binds first, and is that recorded? After two are reached the mask cannot say, and relieving the wrong one buys nothing.

2. At what port count is the crossbar the majority of the switch? That is where a second switch starts being cheaper than a wider one.

3. What fraction of traffic is expected to cross between switches, and was it measured or assumed? It is the only input to the latency question and it varies more than the hardware does.

4. Does the latency model distinguish crossing from local? A model that cannot report a hop will not.

5. How wide is the uplink, and in what units? Transfers, not bandwidth — and no amount of switch throughput changes how many fit.

6. Does local traffic consume any inter-switch resource? It must not, and the only way to test that is with the uplink full.

7. Does each hop have its own flow control? A credit that authorises a slot on a switch that did not issue it will overflow that switch, one hop from the cause.

8. How many endpoints does one switch failure take down, and who chose that number? "However many were cabled to it" is a choice, just not a considered one.

9. What granularity does the fabric manager quiesce at? Whole-fabric is much easier to implement and the cost is switch-cycles stopped for changes that did not touch them.

10. What is the net gain of the second switch — ports added, minus crossing latency and management overhead? The gross number is what gets funded.

24. How This Appears In Real Engineering

The binding ceiling is argued about rather than recorded. Every team believes their resource is the limit, and after two are reached they are all partly right. One latched register settles it.

The crossbar's quadratic term is understood and its per-port consequence is not. "Sixteen ports costs eight times four ports" is intuitive; "and each port costs twice as much" is the form that decides between one switch and two.

The crossing share is assumed at design time and measured never. It is a placement property owned by a different team, it changes when the placement policy changes, and it is the only input that matters to the latency question.

The uplink is sized from a bandwidth number and behaves like a transfer count. Section 9's ceiling is occupancy, and a link sized on aggregate bandwidth can be at its transfer ceiling while showing plenty of headroom.

End-to-end credits are attempted because they look simpler. One accounting scheme instead of one per hop, and the overflow it causes appears on a switch that did nothing wrong.

Whole-fabric quiesce ships first because per-switch quiesce needs the manager to know what a change touches, which is 15.1 and 15.3's ground. The cost is measured in switch-cycles nobody counts.

25. Common Misconceptions

"The switch ran out of ports." It ran out of something, and which one decides who does the work. Section 5's fabric ran out of pins first, with ports, table and buffers all still having headroom.

"Doubling the ports doubles the cost." It quadruples the crossbar. Four times the ports was eight times the total and double the cost per port.

"Adding a switch adds a hop." It adds a hop to the traffic that crosses. Ten percent crossing cost 6ns on the mean; the same hardware with a mostly-crossing workload cost over 45ns.

"The uplink has plenty of bandwidth." The ceiling in section 9 is a transfer count. A link can be at its occupancy ceiling with bandwidth to spare, and every per-switch metric looks healthy while it is.

"Local traffic is unaffected by inter-switch congestion." Only if nothing charges it to the uplink. That term was the one mutation in this chapter that survived every test until the uplink was driven full.

"Credits work end to end." They authorise a slot on a specific receiver. A credit carried across a switch boundary promises a slot on a switch that never issued it, and the overflow lands one hop from the cause.

"Two switches is more available than one." It is two failure domains. Whether that is better depends entirely on how the endpoints were split, and without a limit they are usually not split at all.

"The second switch doubles the ports." It adds ports and costs crossing latency and management overhead: 75 percent gross, 56 percent net.

26. Interview Reasoning

Q1. What stops a switch growing? One of four ceilings — ports, routing table, buffering, or package pins — and they are reached at different sizes. Which one binds first is the only one worth relieving.

Q2. Why record the first ceiling rather than reading the mask? Because adding a port consumes several resources at once, so by the time growth is refused several ceilings are near. The mask says everything is full; the first-reached record says which one stopped it.

Q3. Why is the crossbar's cost per port the interesting number rather than its total? Because the total is expected to grow. Doubling the ports quadrupling the crossbar means each port costs more than the last, and that per-unit price is what a second switch competes against.

Q4. At four ports the crossbar is a third of the cost, at sixteen it is two thirds. What does that mean? Past some size the marginal port is mostly crossbar, and adding connectivity stops being the thing you are paying for.

Q5. What does the second switch cost a transfer? A hop, on the traffic that crosses. Nothing at all on the traffic that does not, which is why the crossing share is the only input that matters.

Q6. Ten percent crossing costs 6ns on a 120ns mean. Is that acceptable? It depends on nothing in the hardware. The same two switches with a mostly-crossing workload cost over 45ns, and the decision is a placement one.

Q7. What is wrong with a latency model that prices every transfer as local? It reports the same number for both of those workloads. It is not wrong about a particular case — it is structurally incapable of being wrong, which makes every projection built on it meaningless.

Q8. The uplink is three wide. Three what? Transfers in flight. It is an occupancy ceiling, not a bandwidth, and a link with bandwidth to spare can be at it.

Q9. Should local traffic consume uplink capacity? No, and testing that requires the uplink to be full. With it empty, a design that charges local traffic and one that does not behave identically.

Q10. That term survived every mutation until you drove the uplink full. What is the general lesson? A term that exempts something from a constraint can only be tested while the constraint binds. The exemption and its absence agree until the resource runs out.

Q11. Do credits work end to end across two switches? No. A credit authorises a slot on a specific receiver. Carried across a boundary it promises a slot on a switch that never issued one.

Q12. Where does that failure surface? On the switch that receives the flit, whose own accounting is perfectly consistent — one hop from the switch that borrowed the credit, with nothing connecting them except a per-hop counter.

Q13. A flit has left one switch and cannot enter the next. Where is it? Between them, and max_block2 measures how long. That interval is real state in a real design and it needs somewhere to sit.

Q14. What does a second switch do to availability? It creates a second failure domain. Whether that improves anything depends on how the endpoints are split, and without a limit they are usually all on the first switch.

Q15. Five endpoints against eight behind one switch. What made the difference? A split limit that refused three attachments. Every one of those attachments was individually legitimate, and nothing about the fabric's operation would have shown the difference.

Q16. Why is the failure domain the constraint with no advocate? Ceiling headroom is visible to the switch designer, uplink capacity to whoever cables it, management reach to whoever operates it. Nothing in normal operation makes the failure domain visible to anybody.

Q17. A change touches one switch of four. How many should stop? One. Stopping four is the same quiesce mechanism applied at the wrong granularity, and it cost ten switch-cycles on a four-cycle change.

Q18. Why must a second change request not restart an open one? Because the stopped interval never accumulates and the change never completes. It is the same guard as an outstanding probe that must not be re-issued.

Q19. What is the net gain of the second switch? Ports added, minus the crossing latency and the management overhead. 75 percent gross, 18 percent cost, 56 percent net — and only the last number decides anything.

Q20. Why must the cost include both halves? They are separate costs from separate mechanisms — a hop on crossing traffic and another element to manage. Charging one understates cascading by about half.

Q21. A second switch adds fewer ports than the first. What is the gain? Zero. Not a large positive number from an unsaturated subtraction, which is what an unguarded difference reports on exactly the deployment where it matters.

Q22. What does a falling gain-per-switch tell you? That each switch added contributes ports and adds crossing traffic to every other one. The gain is not linear and the cost is not either.

Q23. A growth plan asks for exactly the ports available. Approved? Yes. Exactly fitting is fitting, and all three ceilings are <= with the equality driven explicitly.

Q24. Why record the worst port gap rather than just refusing? Because "refused" says nothing about scale. A plan out by four ports and one out by forty need completely different responses.

Q25. What are the four constraints on growth in the assembled model? The switch has ceiling headroom, the uplink can carry the new crossing traffic, the failure domain stays bounded, and the manager can still reach everything.

Q26. Which of those has no operational symptom? The failure domain. The fabric behaves identically whether it is bounded or not, right up until the failure.

Q27. A switch cannot be widened and three teams disagree about why. What do you read? binding_limit. After two ceilings are reached the mask says several things are full, and relieving the wrong one buys nothing.

Q28. Cross-switch traffic has stalled and per-switch metrics are healthy. First reading? peak_uplink against the width. The congestion is on neither switch, so no per-switch metric can see it — and if local traffic is also stalling, it is not the uplink.

Q29. Which is the more dangerous failure in this chapter? The borrowed credit, because it surfaces on a switch that did nothing wrong, one hop away, with every local accounting check passing.

Q30. If you could expose one counter from a multi-switch fabric, which? peak_uplink. Every per-switch metric looks comfortable while the link between them saturates, and it is the only signal that sees the resource neither switch owns.

27. Exercises

1. Add a fifth ceiling to single_ceiling — power — and show that binding_limit still records the first one reached whatever order they arrive in.

2. Replace xbar_cost's full crossbar with a two-stage hierarchical one. Derive its cost shape and find the port count at which it beats the full crossbar.

3. Extend multi_switch to three switches with different hop counts between pairs. Show that mean_ns becomes a weighted sum and identify what cross_pct must become.

4. Give uplink_load two uplinks with independent occupancy. Show what changes in the stall share, and whether it is the same as one uplink of twice the width.

5. Make cascade_credit return the second hop's credit to the first hop's sender. Show precisely which invariant that breaks and where the overflow lands.

6. Add a third switch to switch_blast and derive the per-switch split limit that bounds the failure domain to a third of the endpoints.

7. Give mgmt_scale a change that touches a set of switches rather than a count. Show what overquiesce_err must compare against, and why a count is not enough at three switches.

8. Take the 88-mutation suite and drive local traffic only while the uplink is empty. Confirm the exemption mutation returns, and find every other exemption term in Module 16 tested the same way.

28. Summary

A switch grows until one of four ceilings binds, and past it the fabric grows by adding switches.

• Which ceiling: pins, in this design — with ports, table and buffers all still having headroom. Four different physical quantities, four different teams, and one latched register that says which.
• What the crossbar costs: four times the ports, eight times the total, and double the cost per port. Its share rises from a third to two thirds, and that is the pressure that produces a second switch.
• What the second switch costs a transfer: 126ns against 120 at a ten percent crossing share, and over 165ns when most of the traffic crosses. Same hardware; the difference is placement.
• What it costs the fabric: an uplink that is an occupancy ceiling neither switch can see, per-hop credits that do not extend, a second failure domain, and another element to quiesce.

And the number that decides: 75 percent gross gain, 18 percent cost, 56 percent net.

88 mutations, 88 killed — the fewest first-run survivors in this batch, because the earlier chapters' lessons were applied from the start. The one that did survive is worth carrying: !crosses exempts local traffic from the uplink, and it can only be tested with the uplink full. An exemption and its absence agree until the constraint binds.

Module 16 is complete: a switch's structure, its routing, its resource sharing, and its limits.