Fabric Scaling — When One Switch Becomes a Tree

Chapters 21.1 and 21.2 described one switch: how it picks an egress port, and how a packet crosses it.

Real systems have several. A server has switches behind switches; an accelerator chassis fans one uplink out to a dozen cards; a storage shelf aggregates many drives into one host connection.

And the striking thing is how little has to change. The routing rules from Chapter 21.1 compose recursively — no switch needs a table of every endpoint in the system, at any depth.

What does not compose is bandwidth. Eight downstream Links behind one uplink do not deliver eight uplinks' worth of throughput, and no amount of correct routing changes that.

So what scales, what does not, and what breaks when a hierarchy is configured inconsistently?

1. Sources and Scope

2. The Tree

PCIe hierarchy is a rooted tree, not a general mesh. One Root Complex, switches as interior nodes, endpoints as leaves — and exactly one path between any two points.

                Root Complex
                     │
                  Switch A          upstream port + downstream ports
              ┌──────┴──────┐
          Endpoint 0     Switch B
                       ┌───┴───┐
                   Endpoint 1  Endpoint 2

Every interior node is the same thing (Chapter 21.1 §2): a set of virtual PCI-to-PCI bridges around an internal bus. Switch B is not a special case of Switch A — it is the same structure, one level down.

Which is why this chapter is short on new mechanism and long on consequence. The rules do not change; what changes is that they now have to agree with each other across levels (§5).

3. Ranges Are Subtree Summaries

Three things to read out of the figure.

The Root Complex has no entry for Endpoint 1 or Endpoint 2. It knows only that a range lives behind Switch A's upstream port — the subtree is compressed into that range (§3).

Switch B is structurally identical to Switch A, one level down, which is the recursion §2 described and §11 Model 1 verified over 180,000 lookups.

And the edge from Switch A's port 1 to Switch B is where the invariant lives. That port's range must contain everything Switch B advertises. When it does not, §5's failure appears — addresses that Switch B can route and Switch A will never deliver to it.

4. Enumeration Builds the Nesting

The ranges are not designed; they are assigned during enumeration, walking the tree and giving each bridge a Secondary and Subordinate bus number and address windows.

The pattern is depth-first and inherently recursive:

assign this bridge's Secondary Bus
  descend and enumerate everything below it
  the highest bus number used down there becomes this bridge's Subordinate

Which produces the nesting property automatically — a child's range is a sub-interval of its parent's, because the parent's Subordinate is computed after its children have taken their numbers.

And the same holds for address windows: a parent's Base/Limit must span the union of its children's allocations, because those allocations were made from within the parent's span.

Chapter 21.4 owns enumeration at scale. What matters here is the invariant it is supposed to establish — and §5 is about what happens when something disturbs it.

5. Nesting Is an Invariant, and It Can Break

6. Depth Costs Latency and Adds Stall Points

Each additional switch is another forwarding stage (Chapter 21.2), so end-to-end latency decomposes as a derived model:

T_total  ≈  Σ (per-hop forwarding) + Σ (Link and service delays) + endpoint time

This chapter publishes no per-hop figure (§1). What generalizes is the shape: latency grows with depth, roughly linearly in the number of hops.

And the second effect matters more than the first. Every hop is an independent place to stall (Chapter 21.2 §8): its own credits, its own arbitration, its own head-of-line behaviour, its own Link state.

So depth multiplies the ways a transaction can be delayed, and a packet's worst-case latency is not the sum of typical hop times but the sum of worst cases — which is why deep trees are used for aggregation and fan-out rather than for latency-critical paths.

Non-posted traffic feels this twice. A Memory Read traverses the depth outbound and its Completions traverse it back (Chapter 20.3 §4), so the round trip scales with roughly twice the depth — which directly raises the bandwidth-delay product a DMA engine must cover (Chapter 20.5 §5).

7. Bandwidth Does Not Add

8. Fair Share Is a Policy, Not a Guarantee

When demand exceeds the uplink, something decides who gets what. §7's table used an equal-share model:

share = min(per_endpoint_demand, uplink / N)

That formula is a derived illustration of one scheduler. PCIe does not guarantee it — arbitration policy is implementation and configuration, and Chapter 21.2 §1's source exposes both a Port Arbitration Table and a Source Queue Weight, so real switches make it programmable and weighted.

Two ways the equal-share model misleads if taken literally.

Fixed priority does not share at all. A high-priority port that is continuously busy can starve a lower one indefinitely — harmless with two ports, serious with sixteen (§9).

And equal packets is not equal bytes — Chapter 21.4 measures that directly, and it is why "round-robin" alone does not specify a fairness outcome.

9. Starvation Scales Badly

A fixed-priority arbiter that seems fine with two downstream ports becomes dangerous with sixteen.

With two, the lower-priority port gets whatever the higher one leaves — and unless the higher port is saturated, that is usually adequate.

With sixteen, a port near the bottom is behind fifteen others. If several above it are persistently busy, it may never be granted — and its endpoint appears functional but unusably slow.

Round-robin bounds the wait (§10's arbiter): with N requesters, a port waits at most N−1 grants. Bounded is the property that matters, not equality — and it is why Chapter 21.2 §11 chose round-robin as its default policy while labelling it policy.

This chapter asserts no PCIe fairness requirement (§8). What it asserts is that fairness must be a deliberate choice at scale, because the failure mode is silent: a starved port reports no error.

10. RTL — Subtree Ranges, Lookup, Containment and Uplink Arbitration

// SYNTHESIZABLE. Hierarchical subtree summaries (§3).
// The register semantics are sourced (Chapter 21.1 §1); the widths and the
// struct layout are internal normalization.
package fabric_pkg;
 
  parameter int N_DOWN = 4;
  parameter int PORT_W = (N_DOWN <= 1) ? 1 : $clog2(N_DOWN);
  parameter int BUS_W  = 8;
  parameter int MAX_HOPS = 8;                 // §12 -- LOCAL instrumentation
  parameter int HOP_W  = $clog2(MAX_HOPS + 1);
 
  // One downstream port's summary of everything below it.
  typedef struct packed {
    logic              valid;
    logic [BUS_W-1:0]  secondary;   // first bus below this port
    logic [BUS_W-1:0]  subordinate; // highest bus reachable below it
    logic [63:0]       mem_base;
    logic [63:0]       mem_limit;
  } subtree_t;
 
  // Chapter 21.1 §1: a window exists only when Base <= Limit.
  function automatic bit range_enabled(input subtree_t s);
    return s.valid && (s.secondary <= s.subordinate) && (s.mem_base <= s.mem_limit);
  endfunction
 
  function automatic bit bus_in(input subtree_t s, input logic [BUS_W-1:0] b);
    return range_enabled(s) && (b >= s.secondary) && (b <= s.subordinate);
  endfunction
 
endpackage

import fabric_pkg::*;
 
// SYNTHESIZABLE. One hop of the recursive lookup (§3).
// SAME THREE OUTCOMES as Chapter 21.1 §8, because the rule is identical at
// every level -- that identity is what makes the tree compose.
module subtree_lookup (
  input  subtree_t [N_DOWN-1:0] child,
  input  logic [BUS_W-1:0]      target_bus,
 
  output logic [N_DOWN-1:0] match_vec,
  output logic              hit_down,
  output logic              go_upstream,
  output logic              ambiguous,
  output logic [PORT_W-1:0] port
);
  always_comb
    for (int i = 0; i < N_DOWN; i++) match_vec[i] = bus_in(child[i], target_bus);
 
  assign hit_down    = $onehot(match_vec);
  assign go_upstream = (match_vec == '0);      // 21.1 §4's default
  assign ambiguous   = !$onehot0(match_vec);   // §11: 17.5% with overlap
 
  always_comb begin
    port = '0;
    if (hit_down) for (int i = 0; i < N_DOWN; i++) if (match_vec[i]) port = PORT_W'(i);
  end
endmodule

import fabric_pkg::*;
 
// SYNTHESIZABLE (configuration-time / formal). THE INVARIANT CHECKER (§5).
// A hierarchy is only correct if every child's summary is contained by its
// parent's. Section 11: 92.9% of random range pairs violate containment,
// and every violation is detectable by this comparison.
module range_containment_check (
  input  subtree_t parent,
  input  subtree_t child,
 
  output logic bus_contained,
  output logic mem_contained,
  output logic ordering_ok,
  output logic config_valid,
  output logic err_child_escapes
);
  // Both ends must be ordered, or the range is disabled rather than wrapped.
  assign ordering_ok   = (parent.secondary <= parent.subordinate)
                      && (child.secondary  <= child.subordinate)
                      && (parent.mem_base  <= parent.mem_limit)
                      && (child.mem_base   <= child.mem_limit);
 
  // ==================================================================
  // CONTAINMENT, both range kinds. A child bus range escaping its parent
  // creates buses that are locally routable and globally unreachable
  // (§5's worked case: parent 8-20, child 18-30, buses 21-30 lost).
  // ==================================================================
  assign bus_contained = (child.secondary >= parent.secondary)
                      && (child.subordinate <= parent.subordinate);
  assign mem_contained = (child.mem_base  >= parent.mem_base)
                      && (child.mem_limit <= parent.mem_limit);
 
  assign config_valid      = ordering_ok && bus_contained && mem_contained;
  assign err_child_escapes = ordering_ok && !(bus_contained && mem_contained);
endmodule

import fabric_pkg::*;
 
// SYNTHESIZABLE. Several downstream subtrees competing for one uplink (§7).
// SAME ARBITER CONTRACT as Chapter 21.2 §11 -- held under stall, released
// only on transfer. Reused deliberately rather than reimplemented.
module uplink_arbiter #(
  parameter int N = N_DOWN,
  parameter int IW = (N <= 1) ? 1 : $clog2(N)
) (
  input  logic clk,
  input  logic rst_n,
  input  logic [N-1:0]  request,
  input  logic [N-1:0]  eop,
  input  logic          uplink_ready,
  input  logic          uplink_operational,
  input  logic          fc_grant,
 
  output logic          out_valid,
  output logic [IW-1:0] owner,
  output logic          owner_valid,
  output logic [31:0]   grants_per_port [N]     // §15's fairness diagnostic
);
  logic [IW-1:0] own_q, rr_q; logic held_q;
  logic [31:0]   cnt_q [N];
 
  assign owner       = own_q;
  assign owner_valid = held_q;
  assign out_valid   = held_q && request[own_q] && uplink_operational && fc_grant;
  always_comb for (int i=0;i<N;i++) grants_per_port[i] = cnt_q[i];
 
  // Round-robin bounds the wait at N-1 grants (§9). Fixed priority does
  // not bound it at all, which is fine at N=2 and dangerous at N=16.
  logic [IW-1:0] pick; logic found;
  always_comb begin
    pick='0; found=1'b0;
    for (int k = N-1; k >= 0; k--) begin
      int idx = (int'(rr_q) + k) % N;
      if (request[idx]) begin pick = IW'(idx); found = 1'b1; end
    end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin held_q<=1'b0; own_q<='0; rr_q<='0; for (int i=0;i<N;i++) cnt_q[i]<='0; end
    else begin
      if (!held_q) begin
        if (found) begin own_q <= pick; held_q <= 1'b1; end
      end else if (out_valid && uplink_ready && eop[own_q]) begin
        held_q <= 1'b0;
        if (!(&cnt_q[own_q])) cnt_q[own_q] <= cnt_q[own_q] + 32'd1;
        rr_q <= (own_q == IW'(N-1)) ? '0 : (own_q + IW'(1));
      end
    end
  end
endmodule

import fabric_pkg::*;
 
// SYNTHESIZABLE / VERIFICATION instrumentation. Hop guard (§12).
// PCIe TLPs CARRY NO HOP COUNT OR TTL. This is LOCAL modelling state used
// to detect a misconfigured hierarchy in simulation or in a debug build --
// it is not a protocol field and is not transmitted.
module hop_guard (
  input  logic clk,
  input  logic rst_n,
  input  logic pkt_enters_hop,
  input  logic pkt_retires,
  output logic [HOP_W-1:0] hops,
  output logic err_hop_limit
);
  logic [HOP_W-1:0] h_q; logic e_q;
  assign hops = h_q; assign err_hop_limit = e_q;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin h_q <= '0; e_q <= 1'b0; end
    else if (pkt_retires) h_q <= '0;
    else if (pkt_enters_hop) begin
      if (h_q >= HOP_W'(MAX_HOPS)) e_q <= 1'b1;   // bounded, and REPORTED
      else h_q <= h_q + HOP_W'(1);
    end
  end
endmodule

Classification: all synthesizable; the containment checker is configuration-time and the hop guard is instrumentation.

The lookup is deliberately identical to Chapter 21.1 §10's — that sameness at every level is what makes the recursion work (§3), and §11 verified it: 0 disagreements across 180,000 lookups.

Failure — five. Overlapping child ranges priority-encoded (17.5% ambiguous). A disabled range treated as wrapped. Skipping containment validation (§5). Fixed priority at high port counts (§9). And claiming the hop guard is a PCIe TTL (§12).

11. Measured Behaviour

12. The Hop Guard — and Why PCIe Has No TTL

13. Assertions

// ==================================================================
// LOOKUP -- the recursion (§3). Same shape as Chapter 21.1's decoder,
// because the rule must be identical at every level.
// ==================================================================
 
// P1: exactly one outcome, always. No packet is undecided.
property p_lookup_total;
  @(posedge clk) disable iff (!rst_n)
    $onehot({hit_down, go_upstream, ambiguous});
endproperty
 
// P2: an unrecognized bus goes upstream (21.1 §4's default).
property p_unmatched_upstream;
  @(posedge clk) disable iff (!rst_n)
    (match_vec == '0) |-> go_upstream;
endproperty
 
// P3: ambiguity is REPORTED, never resolved by priority (§11 Model 2:
// 17.5% of lookups under overlap).
property p_overlap_flagged;
  @(posedge clk) disable iff (!rst_n)
    (!$onehot0(match_vec)) |-> (ambiguous && !hit_down);
endproperty
 
// P4: a downstream hit names a port whose range actually contains the bus.
property p_hit_consistent;
  @(posedge clk) disable iff (!rst_n)
    hit_down |-> bus_in(child[port], target_bus);
endproperty
 
// P5: a disabled range (Base > Limit) never matches. It is disabled,
// not wrapped -- Chapter 21.1 §1.
property p_disabled_never_matches;
  @(posedge clk) disable iff (!rst_n)
    (child[0].secondary > child[0].subordinate) |-> !match_vec[0];
endproperty
 
// P6: an invalid child entry never matches.
property p_invalid_never_matches;
  @(posedge clk) disable iff (!rst_n)
    (!child[0].valid) |-> !match_vec[0];
endproperty
 
// ==================================================================
// CONTAINMENT -- the invariant that only exists above one level (§5).
// ==================================================================
 
// P7: a valid configuration means the child's bus range is inside the
// parent's. This is the whole of §5.
property p_bus_containment;
  @(posedge clk) disable iff (!rst_n)
    config_valid |-> (child.secondary >= parent.secondary)
                  && (child.subordinate <= parent.subordinate);
endproperty
 
// P8: the same for address windows.
property p_mem_containment;
  @(posedge clk) disable iff (!rst_n)
    config_valid |-> (child.mem_base  >= parent.mem_base)
                  && (child.mem_limit <= parent.mem_limit);
endproperty
 
// P9: an escaping child is FLAGGED, not tolerated. §11 Model 3 found
// 92.9% of random pairs violate containment and 100% were detectable.
property p_escape_flagged;
  @(posedge clk) disable iff (!rst_n)
    (ordering_ok && !(bus_contained && mem_contained)) |-> err_child_escapes;
endproperty
 
// P10: valid and escaping are mutually exclusive.
property p_valid_excludes_escape;
  @(posedge clk) disable iff (!rst_n)
    !(config_valid && err_child_escapes);
endproperty
 
// P11: containment is checked on BOTH range kinds. A hierarchy correct in
// buses and wrong in addresses is still wrong.
property p_both_kinds_required;
  @(posedge clk) disable iff (!rst_n)
    config_valid |-> (bus_contained && mem_contained);
endproperty
 
// ==================================================================
// UPLINK ARBITRATION -- Chapter 21.2 §11's contract, at the tree level.
// ==================================================================
 
// P12: ownership survives a stalled uplink. Availability is not permission.
property p_owner_held_under_stall;
  @(posedge clk) disable iff (!rst_n)
    (owner_valid && !uplink_ready) |=> (owner_valid && $stable(owner));
endproperty
 
// P13: ownership is released only on an ACCEPTED end-of-packet.
property p_release_on_transfer_only;
  @(posedge clk) disable iff (!rst_n)
    ($fell(owner_valid)) |-> $past(out_valid && uplink_ready && eop[owner]);
endproperty
 
// P14: a down Link never transmits. Chapter 21.2 §5's rule, unchanged.
property p_no_tx_when_link_down;
  @(posedge clk) disable iff (!rst_n)
    (!uplink_operational) |-> !out_valid;
endproperty
 
// P15: no credit, no transmission (Chapter 21.2 §3).
property p_no_tx_without_credit;
  @(posedge clk) disable iff (!rst_n)
    (!fc_grant) |-> !out_valid;
endproperty
 
// P16: round-robin bounds the wait -- the pointer advances past the port
// that just finished, so no port is skipped twice (§9).
property p_rr_pointer_advances;
  @(posedge clk) disable iff (!rst_n)
    (out_valid && uplink_ready && eop[owner]) |=> (rr_q != $past(owner));
endproperty
 
// P17: a continuously requesting port is eventually granted. Bounded
// waiting is the property that matters, not equal shares.
property p_no_starvation;
  @(posedge clk) disable iff (!rst_n)
    (request[0] && uplink_operational && fc_grant)
      |-> s_eventually (owner_valid && (owner == '0));
endproperty
 
// ==================================================================
// HOP GUARD -- LOCAL instrumentation only (§12). Asserting on it says
// nothing about PCIe, which has no TTL for ordinary TLP forwarding.
// ==================================================================
 
// P18: the counter is bounded. §11 Model 5: 100% non-terminating without it.
property p_hop_bounded;
  @(posedge clk) disable iff (!rst_n)
    (hops <= HOP_W'(MAX_HOPS));
endproperty
 
// P19: exceeding the bound REPORTS rather than silently wrapping.
property p_hop_limit_reported;
  @(posedge clk) disable iff (!rst_n)
    (pkt_enters_hop && (hops >= HOP_W'(MAX_HOPS))) |=> err_hop_limit;
endproperty
 
// P20: retirement clears the count -- the guard is per-packet, not global.
property p_hop_cleared_on_retire;
  @(posedge clk) disable iff (!rst_n)
    pkt_retires |=> (hops == '0);
endproperty

Twenty properties, and the load-bearing ones are P7–P11. P1–P6 restate Chapter 21.1's decoder contract because §3's recursion depends on it holding identically at every level. P7–P11 are new to this chapter — they cannot even be stated with one switch. P12–P17 are Chapter 21.2's arbitration contract, and P18–P20 constrain instrumentation, not protocol.

14. Verification — Mutations

Each mutation is a plausible implementation, not a typo. The assertion column names the property from §13 that fires.

Two counterexamples worth stating explicitly.

Mutation 8 is the one that survives review. Checking child.secondary >= parent.secondary looks like containment and passes every test where children start above their parent — which is most of them. The failing case needs a child whose top exceeds its parent's, exactly §5's 8–20 / 18–30 pair. P7's conjunction is what catches it, and it is why the property is written with both bounds rather than as one comparison.

Mutation 25 is a design-review failure, not an assertion failure. No property can fire, because the RTL would be internally consistent. It fails because it describes a protocol that does not exist (§12) — a reminder that assertions verify an implementation against its specification and cannot verify the specification.

15. Debugging a Fabric

Symptom — a device works when attached directly and disappears behind a switch. Suspect containment (§5). Read the parent bridge's Secondary/Subordinate and Base/Limit, then the child's, and compare them numerically rather than checking each for plausibility. Any child value outside the parent's span is the answer; the device is reachable within its own switch and invisible through the parent.

Symptom — Configuration cycles reach the device but Memory accesses do not. That separation is diagnostic. Bus ranges are contained and address windows are not — mutation 10 exactly. Both range kinds must nest independently (P11).

Symptom — every endpoint benchmarks correctly alone; together they underperform. This is §7, and nothing is broken. Compute the aggregate demand and compare it to the uplink. If the ratio explains the shortfall, the fabric is oversubscribed by design and the fix is topology or expectations, not debugging. The tell is that the shortfall scales with the number of active endpoints.

Symptom — one endpoint behind a busy switch is far slower than its siblings. Suspect arbitration (§9), not the endpoint. Read the per-port grant counters (§10's grants_per_port): a starved port shows grants far below its siblings while its request line is continuously asserted. A port with no grants and constant requests is starvation, not a link problem — and note that equal grant counts do not mean equal bandwidth, because grants count packets rather than bytes (Chapter 21.4).

Symptom — traffic hangs and no error is reported anywhere. Check for a routing cycle from a malformed configuration. Without a bound this never resolves (§11 Model 5: 100.0%). §10's guard converts it into a reported error — and remember it is local instrumentation, so this signature only exists if someone built it (§12).

Symptom — an endpoint appears at an unexpected bus number after hot-add. Enumeration re-ran and re-assigned. The failure to look for is a parent whose Subordinate was not re-expanded to cover the newly added subtree — §5's invariant broken by a change rather than by an initial mistake.

A general rule for hierarchies: check relationships, not devices. Every device in a broken fabric is usually individually correct (§5). The bug lives between two configurations, so a debugging method that examines one register set at a time will not find it.

16. Misconceptions

"Each switch needs a routing table of all endpoints below it." No — it needs one entry per downstream port (§3). Ranges summarize subtrees, which is precisely why hierarchy scales.

"A deeper tree needs a bigger table." No. Depth is absorbed by the ranges; the table width is set by port count, not by how much is below.

"Eight 8 GB/s downstream Links behind one 16 GB/s uplink give 64 GB/s." No — 16 GB/s at best, 4× oversubscribed (§7).

"Oversubscription is a bug." No. It is the intended use of a switch — the sourced device is "principally aimed at fan-in/out or aggregation" (§1).

"PCIe TLPs have a TTL / hop limit." No. PCIe has no hop count for ordinary TLP forwarding (§12). The tree property and correct enumeration replace it; a hop guard is local instrumentation.

"A hop counter in my RTL is part of the protocol." No — it is never transmitted and no other device sees it (§12).

"Ranges just need to look reasonable." No — they must nest (§5). 92.9% of random pairs fail containment (§11).

"Enumeration guarantees correctness forever." It establishes the invariant; hot-add, firmware/OS conflicts and partial reconfiguration can break it afterwards (§5, §15).

"If a range is wrong, the switch reports an error." No. An address outside the parent's window is simply routed upstream — a correct action on incorrect configuration (§5), which is why it produces silence rather than an error.

"Overlapping sibling ranges are resolved by priority, so they are harmless." They are resolved consistently, which is worse: 17.5% of lookups take a fixed wrong path and one port's devices are unreachable with no error (§11).

"Round-robin means fair bandwidth." It bounds waiting (§9). Equal packets is not equal bytes — Chapter 21.4 quantifies the gap.

"Fixed priority is fine, it works in my two-port design." Two ports hides it; sixteen exposes it (§9), and starvation reports no error.

"Latency is just the sum of per-hop times." Each hop is also an independent stall point (§6), so worst-case latency compounds rather than adds.

"Depth only costs on the way out." Non-posted traffic pays it twice — request down, Completion back (§6).

"A device that works standalone and fails behind a switch has a device problem." Usually the opposite: the device is correct and the relationship between two bridge configurations is not (§15).

17. Understanding Check

Q1. Why does a switch three levels down need no knowledge of the endpoints above it? Because it only ever makes a local decision: in one of my downstream ranges, or upstream (Chapter 21.1 §4). The upstream default absorbs everything it does not know, so ignorance of the rest of the tree is not a defect — it is the mechanism.

Q2. A parent bridge claims buses 8–20; a child claims 18–30. What breaks, and what does not? Buses 18–20 work. Buses 21–30 are unreachable through the parent — it does not recognize them and routes them upstream, away from the child that owns them. Both devices are individually consistent (§5).

Q3. Eight endpoints, each capable of 8 GB/s, behind a 16 GB/s uplink. What is the aggregate, and is anything wrong? 16 GB/s, 4× oversubscribed, roughly 2 GB/s each (§7, §11 Model 4). Nothing is wrong — that is the aggregation the switch exists to provide. It becomes a problem only if the workload activates all eight at once.

Q4. Should a switch drop a TLP that has been forwarded too many times? The question contains a false premise. PCIe TLPs carry no hop count and no TTL for ordinary forwarding (§12), so there is nothing to test. The tree topology plus correct enumeration is what prevents endless forwarding; a bounded counter in an implementation is local debug instrumentation and is never transmitted.

Q5. Why does §10 report ambiguous instead of picking the lower-numbered port? Because picking is silent. §11 Model 2 measured 17.5% of lookups matching two overlapping ranges — a priority encoder sends all of them the same wrong way, permanently, with no error. Reporting turns a permanent mis-route into a diagnosable configuration fault.

Q6. Fixed priority passed every test on a 2-port switch. Why is it a defect on a 16-port switch? Because it bounds nothing. At two ports the lower-priority port gets the gaps; at sixteen it may be behind many persistently busy ports and receive nothing (§9). The endpoint reports no error — it is merely never served, which is why §13's P17 asserts eventual grant rather than any bandwidth share.

18. What's Next

This chapter scaled the topology. Chapter 21.4 scales the traffic: many endpoints active at once behind the tree you just built.

And that raises questions this chapter deliberately did not touch. Two endpoints under different switches both issue a Read with Tag 5 — how does anything tell the Completions apart? Several DMA engines contend for the same uplink — how is that measured per port? One endpoint misbehaves — how far does the damage reach?

One correction to carry forward. A Tag is unique per Requester, not across the fabric. The identity that separates two outstanding requests is the pair (Requester ID, Tag) — Chapter 20.3 §3 established it for one device, and 21.4 measures what happens when a fabric forgets it.

Module 22 then owns performance itself — throughput, latency, credit bottlenecks, payload-size effects, link efficiency and benchmark interpretation. §7's oversubscription arithmetic here is a topology consequence, not a performance model, and this chapter published no per-hop latency or link-rate figures (§1) precisely because that material belongs there.