Multi-Level Fabrics — Depth, Convergence, and Failure Domains

Chapter 4.3 examined one Switch: ports, forwarding, and the convergence points inside it. Everything there assumed a downstream port leads to a device.

It need not. A downstream port can lead to another Switch, and that single fact changes the reasoning enough to warrant its own chapter:

What changes when PCIe Switches are cascaded and a hierarchy becomes several levels deep?

1. A Fabric to Reason About

The rest of this chapter refers to one topology. It is worth studying before reading on, because every subsequent argument is a claim about it.

Some vocabulary, marked as conceptual. These are ordinary topology and graph terms used here for clarity; they are not normative PCIe terminology, and you should not expect a specification to define them this way:

• Branch — a Switch and everything below it. Switch B's branch contains EP2, Switch C, EP3, EP4.
• Subtree — the same idea, used when the emphasis is on the set of leaves.
• Depth — how many Links separate a device from the root.
• Sibling — two devices sharing an immediate parent. EP3 and EP4 are siblings.
• Cousin — two devices in different branches sharing a more distant ancestor. EP1 and EP3 are cousins.
• Common ancestor — the nearest Switch through which two devices' paths both pass. §3 makes this the chapter's central tool.

2. Path Tracing

Work through Figure 1 explicitly. Each path is the sequence of Links from device to root.

EP1 → root: its own Link, then L0. Two Links, one forwarding hop.

EP2 → root: its own Link, then L1, then L0. Three Links, two forwarding hops.

EP3 → root: its own Link, then L2, then L1, then L0. Four Links, three forwarding hops.

EP4 → root: its own Link, then L2, L1, L0. Identical to EP3 except for the first hop.

Now the questions that matter — where do pairs of paths first meet?

Two structural observations, and they drive everything that follows.

Paths nest. EP3's path is a superset of EP2's, which is a superset of EP1's. Every segment a shallow device uses is also used by every device below it. L0 carries everything in this fabric.

Depth predicts sharing. The deeper a device, the more segments it shares with others and the more places its traffic can be limited. EP3 has four segments where something can go wrong or become congested; EP1 has two.

3. The Common-Ancestor Method

Now the reusable tool, and the most valuable thing in this chapter.

For any two devices, find the nearest Switch through which both paths pass. That Switch — and specifically its upstream Link — is where their traffic first interacts.

For EP3 and EP4, the common ancestor is Switch C, so L2 is where they first meet. For EP2 and EP3 it is Switch B, so L1. For EP1 and EP3 it is Switch A, so L0.

This one construction predicts four different things, which is why it is worth internalising rather than re-deriving each time:

Where contention starts. Two devices cannot contend for a segment neither traverses. Their first shared segment is the earliest point their demands can sum.

Which failures affect both. A fault at or above the common ancestor takes out both. A fault below it — on either device's own branch — takes out one.

How much they can interfere. Deeper common ancestors mean the devices share more segments and interfere more thoroughly. EP3 and EP4 share three segments; EP1 and EP3 share one.

Whether traffic can be separated. If two workloads must not interfere, placing them so their common ancestor is as close to the root as possible minimises shared path. That is a placement decision derived from topology, which is what Chapter 4.1 meant by placement being a design decision.

4. Latency Through Depth

Each forwarding hop adds delay, and it is worth being precise about what kind.

A transaction crossing a Switch experiences:

• Processing and pipeline delay through the ingress path, the route decision, and the egress path.
• Queueing delay if the selected egress is busy — zero when idle, unbounded in principle when congested.
• Arbitration delay when other ingresses are competing for the same egress.

So EP3's path involves three of these, EP1's one. But the useful lesson is not an arithmetic one:

Depth adds places where latency can accumulate and, more importantly, where it can vary.

The variance matters more than the mean for most workloads. A deep path under light load may be only slightly slower than a shallow one; a deep path under load can be substantially and unpredictably slower, because each hop contributes queueing and arbitration delay that depends on other traffic.

Two things this chapter deliberately does not claim: any particular number of nanoseconds per hop, or that every Switch adds the same delay. Both vary by implementation, and a tutorial that invented figures would be teaching something false.

5. Three Different Bandwidth Numbers

The distinction that most often produces wrong system-level expectations. Three quantities are routinely conflated:

Local Link capability. What a device's own connection can carry. A property of one hop.

Branch upstream capacity. What the segment above a Switch can carry — L2 for Switch C's branch, L1 for Switch B's, L0 for Switch A's. Shared by everything below.

Root-facing aggregate capacity. What ultimately reaches the host side, and beyond it what the memory subsystem can absorb.

These are independent numbers, and the smallest one along a path bounds what a device achieves:

Local Link capability ≠ branch upstream capacity ≠ root-facing aggregate capacity.

A conceptual illustration, kept deliberately abstract because tying it to real figures would import generation-specific material that belongs to Module 5 and Module 6:

Suppose EP3 and EP4 each have a local Link able to carry demand D, and Switch C's upstream L2 can carry C₂. If both devices offer D simultaneously, the offered load at L2 is 2D. When 2D > C₂, contention is inevitable — not because anything is broken, but because arithmetic. Add EP2's demand at L1, and everything from Switch B's branch at L0, and the concentration compounds at each level.

The relevant capacity for a device is the minimum along its whole path, and in a deep fabric that minimum is frequently several hops away from the device.

6. Oversubscription as a Design Choice

The natural reaction to §5 is that a fabric where aggregate downstream demand exceeds upstream capacity is badly designed. That reaction is wrong often enough to be worth correcting carefully.

Oversubscription — provisioning less upstream capacity than the sum of downstream demands — is frequently a rational choice, and it rests on a workload assumption:

Peak demands do not coincide.

If eight devices each occasionally burst but rarely burst together, provisioning upstream capacity for all eight simultaneously wastes resources that are almost never used. Sizing for realistic concurrent demand rather than theoretical maximum is normal engineering, and it is how most shared infrastructure is provisioned.

The design becomes a problem when the assumption is wrong:

• Workloads that were expected to be uncorrelated turn out to be synchronised — several devices busy in the same phase.
• The workload mix changes after deployment.
• A single device's demand grows to consume the shared segment alone.

7. Failure Domains Nest Too

Depth stratifies failures exactly as it stratifies convergence. Using Figure 1:

A device's own Link fails. EP3's Link down affects EP3 alone. EP4, sharing Switch C but not that Link, is unaffected.

A Switch fails, or its upstream Link fails. Switch C or L2 down affects EP3 and EP4 — its entire subtree — and nothing else. EP1 and EP2 are unaffected.

A higher Switch or its upstream Link fails. Switch B or L1 down affects EP2, EP3, EP4 — everything in its branch, including the whole Switch C subtree beneath.

The root-facing segment fails. L0 or Switch A down removes every device in the fabric.

The nesting is exact and it is the inverse of the path structure: a fault at any point affects precisely the leaves whose paths traverse it. That is why §3's backward reading works — the affected set is the subtree below the fault.

8. Every Hop Is Its Own Reliability Boundary

One concise connection back to Module 3, because depth multiplies something easy to overlook.

Chapter 3.2 established that hop-local delivery covers one Link, with each Switch terminating one relationship and originating another. In Figure 1, EP3's path to the root involves four Links and therefore four independent delivery relationships:

EP3 ↔ Switch C, Switch C ↔ Switch B, Switch B ↔ Switch A, Switch A ↔ Root Complex.

They share no state. Consequences that matter in a deep fabric:

• Successful delivery on hop one says nothing about hops two, three, or four. EP3's Data Link Layer confirms its packet reached Switch C, and that is the entirety of what it confirms.
• A retry on L1 is invisible to EP3. Switch B handles it within its own relationship; EP3 released its copy long before.
• Four hops means four places a delivery problem can occur, each detectable only by the two components sharing that Link.

The practical upshot for debugging: "the transaction was delivered" is meaningless without naming the hop, and in a four-hop path it is meaningful for exactly one of them at a time.

9. Next-Hop Selection in RTL

A component in a multi-level fabric must decide which port a transaction should leave by. Modelling that honestly means modelling the decision structure, not inventing PCIe's rules.

// Illustrative internal representation — NOT a PCIe routing-table format.
typedef struct packed {
  logic                  valid;      // this entry is populated
  logic [PORT_W-1:0]     next_port;  // port to forward by
  logic                  local;      // destination is this component itself
} route_entry_t;

// Illustrative synthesizable RTL — next-hop selection with enable checking.
// NOT a PCIe routing engine. Models: look up a next-hop port for an internal
// destination class, confirm the port is usable, and report a defined error
// when no valid route exists.
module next_hop_select #(
  parameter int unsigned NUM_PORTS  = 4,
  parameter int unsigned NUM_DESTS  = 16,
  localparam int unsigned PORT_W    = $clog2(NUM_PORTS),
  localparam int unsigned DEST_W    = $clog2(NUM_DESTS)
) (
  input  logic                clk,
  input  logic                rst_n,
 
  // Route table programming (by local configuration logic)
  input  logic                tbl_we,
  input  logic [DEST_W-1:0]   tbl_addr,
  input  route_entry_t        tbl_wdata,
 
  // Which ports are currently usable
  input  logic [NUM_PORTS-1:0] port_enabled,
 
  // Lookup request
  input  logic                lk_valid,
  input  logic [DEST_W-1:0]   lk_dest,
 
  // Result — combinational, valid when lk_valid
  output logic                res_hit,        // a usable route exists
  output logic [PORT_W-1:0]   res_port,
  output logic                res_local,      // terminate here rather than forward
  output logic                res_no_route    // no valid or no usable route
);
 
  route_entry_t table_q [NUM_DESTS];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int unsigned i = 0; i < NUM_DESTS; i++) begin
        table_q[i] <= '0;   // valid = 0: every destination unroutable at reset
      end
    end else if (tbl_we) begin
      table_q[tbl_addr] <= tbl_wdata;
    end
  end
 
  wire route_entry_t e = table_q[lk_dest];
 
  // A route is usable only if the entry is populated AND the target port is
  // currently enabled. Forwarding to a disabled port would strand the
  // transaction; reporting no-route lets the caller handle it explicitly.
  wire port_ok = e.valid && !e.local && port_enabled[e.next_port];
 
  assign res_local    = lk_valid && e.valid && e.local;
  assign res_hit      = lk_valid && (res_local || port_ok);
  assign res_port     = e.next_port;
  assign res_no_route = lk_valid && !res_hit;
endmodule

Classification: synthesizable.

What it models: the decision structure of forwarding — destination to next-hop port, with usability validated at lookup time and an explicit no-route outcome.

Deliberately simplified: a flat direct-mapped table rather than the range- or identity-based matching a real forwarding component performs; no multicast; no per-class routing; single-cycle combinational lookup with no pipelining; and no representation of what PCIe actually uses to determine a destination.

What to notice:

• Port enablement is checked at lookup, not assumed at programming time. A port can become unusable after the table was written, and a design that validates only at programming time will forward into a hole. Checking at lookup makes the result reflect current reality.
• res_no_route is an explicit output, not an implicit default. A forwarding component that silently drops unroutable transactions produces the hardest class of failure to debug — a transaction that vanishes with no indication. An explicit signal lets the caller take a defined action.
• Reset clears valid on every entry, so nothing is routable until deliberately programmed. An uninitialised table producing plausible-looking ports would forward traffic to arbitrary destinations.

Production RTL would additionally require: the normative matching PCIe specifies, per-class handling, pipelining for timing closure at realistic table sizes, and defined handling of the no-route case consistent with the specification.

10. Observability: Finding the Bottleneck Level

In a deep fabric, knowing which level is congested is most of the diagnosis. Hardware can make that visible cheaply.

// Illustrative performance-observability RTL — NOT PCIe-mandated counters.
// Per-port statistics that let firmware or debug software identify which
// topology level is saturated, rather than inferring it from endpoint symptoms.
module port_observability #(
  parameter int unsigned CNT_W   = 32,
  parameter int unsigned DEPTH_W = 10     // width of the occupancy input
) (
  input  logic               clk,
  input  logic               rst_n,
 
  input  logic               clear,        // software-initiated reset of stats
 
  // Per-port activity
  input  logic               accepted,     // a transaction was accepted
  input  logic               stalled,      // wanted to send, egress not ready
  input  logic [DEPTH_W-1:0] occupancy,    // current queue occupancy
 
  output logic [CNT_W-1:0]   n_accepted,
  output logic [CNT_W-1:0]   n_stall_cycles,
  output logic [DEPTH_W-1:0] occupancy_watermark
);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      n_accepted          <= '0;
      n_stall_cycles      <= '0;
      occupancy_watermark <= '0;
    end else begin
      // Saturating counters: wrapping would silently understate a busy port,
      // and an understated count is worse than a pinned maximum because it
      // looks plausible.
      if (accepted && !(&n_accepted))
        n_accepted <= n_accepted + 1'b1;
 
      if (stalled && !(&n_stall_cycles))
        n_stall_cycles <= n_stall_cycles + 1'b1;
 
      // High-water mark: peak occupancy is what indicates pressure. An
      // average would hide bursts entirely.
      if (occupancy > occupancy_watermark)
        occupancy_watermark <= occupancy;
    end
  end
endmodule

Classification: synthesizable.

What it models: per-port statistics sufficient to identify a congested level — how much traffic a port carried, how long it spent unable to send, and how full its queue got.

Deliberately simplified: no time windowing (a single cumulative count since clear); no per-class breakdown; no bandwidth calculation, which needs a time base this module does not have.

What to notice:

• Counters saturate rather than wrap. A wrapped count looks like a small number and silently understates a heavily-loaded port — worse than a pinned maximum, which is obviously "at least this much."
• Occupancy is tracked as a high-water mark. Peak occupancy indicates pressure; an average smooths away exactly the bursts you are hunting.
• clear is software-driven, so measurements can be scoped to a window of interest rather than reflecting everything since power-on.

How this is actually used: reading stall cycles and occupancy watermarks across every port at every level turns "the fabric is slow" into "L1 is the saturated segment." That is the difference between a topology-level diagnosis and guesswork, and it is why real switches expose statistics of this kind — though nothing in PCIe mandates these specific ones.

11. Assertions for Forwarding

// SVA over the illustrative routing module. Implementation invariants for
// THIS design — not PCIe protocol requirements.
 
// P1 — a hit never selects a disabled port. Forwarding into a disabled port
// strands the transaction with no indication anywhere.
property p_no_route_to_disabled;
  @(posedge clk) disable iff (!rst_n)
  (res_hit && !res_local) |-> port_enabled[res_port];
endproperty
a_no_disabled : assert property (p_no_route_to_disabled);
 
// P2 — every lookup produces exactly one outcome. Neither both nor neither.
property p_exactly_one_outcome;
  @(posedge clk) disable iff (!rst_n)
  lk_valid |-> $onehot({res_hit, res_no_route});
endproperty
a_one_outcome : assert property (p_exactly_one_outcome);
 
// P3 — local and forward are mutually exclusive. A transaction is either
// terminated here or sent onward, never both.
property p_local_xor_forward;
  @(posedge clk) disable iff (!rst_n)
  res_local |-> !(res_hit && !res_local);
endproperty
a_local_xor_fwd : assert property (p_local_xor_forward);
 
// P4 — nothing is routable until programmed. Catches a table whose reset
// leaves plausible-looking entries.
property p_unprogrammed_no_route;
  @(posedge clk) disable iff (!rst_n)
  (lk_valid && !table_q[lk_dest].valid) |-> res_no_route;
endproperty
a_unprogrammed : assert property (p_unprogrammed_no_route);

P1 catches the stranding case. Simulation misses it because it requires a port to become disabled after the table was programmed — a sequence a directed test rarely produces, and precisely what happens when a Link goes down in a running system.

P2 catches the silent-drop failure mode: a lookup producing neither a hit nor an explicit no-route leaves the caller with no defined action, and the transaction disappears. This is the hardest fabric failure to debug because there is no evidence anywhere.

P4 catches uninitialised-table routing. A table whose reset leaves entries with valid set forwards traffic to arbitrary ports during early bring-up, producing chaos that looks like a routing bug and is actually a reset bug.

12. Verification: The Topology Matrix

Deep fabrics need verification organised along two independent axes — depth and traffic pattern — because bugs live in their combination.

Depth axis: direct-attached · one Switch · two Switches · three Switches where the environment supports it.

Traffic axis:

• Single device active — baseline.
• Siblings — devices sharing an immediate parent (EP3 + EP4). Exercises the nearest convergence point.
• Cousins — devices in different branches (EP1 + EP3). Exercises a distant common ancestor and should show less interference; if it shows the same, something is coupling paths that should be independent.
• All leaves simultaneously — the oversubscription case from §6, and the one that must be run deliberately.
• One hot device plus light background — the realistic mixed case, and the one that reveals whether a heavy device starves light ones.

The matrix matters because a bug can require both a specific depth and a specific pattern. A convergence bug at Switch B needs traffic from EP2 and from the Switch C subtree simultaneously — neither depth alone nor pattern alone produces it.

A verification-only topology model makes this tractable:

// Verification-only. NOT synthesizable. A testbench model of the fabric,
// used to compute expected paths, common ancestors, and failure domains.
typedef struct {
  int    node_id;
  int    parent_id;      // -1 for the root
  int    ingress_port;   // port on the parent that reaches this node
  int    depth;          // Links from the root
  string name;
} topo_node_t;

From this the testbench can compute, for any pair of devices, their path, their common ancestor, the segments they share, and the set of leaves affected by a fault at any element. That makes several checks mechanical:

• Expected path — confirm a transaction traverses the segments the model predicts and no others.
• Predicted contention — confirm that interference between two devices appears at their computed common ancestor and not before.
• Predicted failure domain — inject a fault at an element and confirm exactly the predicted leaf set is affected. A device outside the predicted set being affected means paths are coupled somewhere the model does not capture, which is itself a significant finding.
• Performance correlation — attribute measured shortfalls to specific segments rather than to devices.

Failure injection worth running: a leaf's own Link down; a downstream Switch's upstream Link down; a mid-level branch unavailable; one egress blocked while others are active; and the root-facing segment down.

Coverage worth defining: every depth exercised; sibling, cousin, and all-leaves patterns at each depth; every convergence segment driven to saturation; every failure-domain injection; and occupancy watermarks reaching maximum at each level.

13. Debugging by Affected Set

The method from §3, applied. In each case the reasoning is the same: find the smallest topology element containing every affected device.

Scenario 1 — EP3 and EP4 slow, EP2 and EP1 fine.

The smallest element containing exactly EP3 and EP4 is Switch C. Suspects: Switch C itself, or L2, its upstream segment. EP2 being unaffected rules out L1 and everything above — a problem there would take EP2 with it. Next step: read Switch C's port statistics. High stall cycles and a pinned occupancy watermark on its upstream port confirm L2 saturation.

Scenario 2 — EP2, EP3, EP4 degrade together; EP1 unaffected.

The smallest element containing all three is Switch B. Suspects: Switch B or L1. EP1 being fine rules out L0, Switch A, and the Root Complex. Note this is consistent with the Switch C subtree being fine internally — its traffic is simply being throttled one level up.

Scenario 3 — every device disappears.

The smallest element containing all leaves is Switch A. Suspects: Switch A, L0, the Root Complex, or something beyond PCIe entirely — power, or the host side. "Everything" is the least specific evidence, so start by determining whether the fabric is unreachable or unresponsive, which distinguishes a connection failure from a host-side one.

Scenario 4 — only EP3 retries heavily; everyone else normal.

Retries are hop-local (Chapter 3.2), and the affected set is one device, so this implicates EP3's own Link — its physical connection, its port on Switch C, or the device itself. Crucially it does not implicate anything shared: the retries are confined to one relationship, and the fact that EP4 (sharing Switch C) is unaffected rules out Switch C's shared resources.

14. Common Misconceptions

15. Understanding Check

16. Summary

Cascading Switches adds four things per level: another forwarding hop, another convergence point, another local reliability boundary, and another place topology affects behaviour.

Paths nest: every segment a shallow device uses is also used by every device below it, so the root-facing segment carries everything. The common ancestor of two devices — the nearest Switch both paths pass through — predicts where their traffic first interacts, which faults affect both, and how thoroughly they can interfere. Read backward, it localises faults: the smallest topology element containing every affected device is the primary suspect.

Depth adds places where latency can accumulate and vary, with variance mattering more than mean for most workloads. Three bandwidth quantities must be kept distinct — local Link capability, branch upstream capacity, and root-facing aggregate capacity — and what a device achieves is bounded by the minimum along its whole path, often several hops away.

Oversubscription is a workload assumption, not a defect, and it must be tested by saturating the shared segments — distinguishing graceful degradation from functional failure. Failure domains nest exactly as paths do: a fault affects precisely the leaves whose paths traverse it. And every hop is its own delivery relationship, so a four-hop path has four, sharing no state.

Hold the model: every additional level creates another hop, another convergence point, another reliability boundary, and another place topology shapes what you observe.

17. What Comes Next

Module 4 has now built the reasoning: where hierarchies are anchored, how leaves attach, how fanout works, and how depth changes everything.

Chapter 4.5 — Real System Examples applies it. Concrete desktop and server topologies, walked with these tools: tracing actual paths, identifying actual convergence points, and reading real system layouts the way this module has taught you to read diagrams.

Revisit Switches in Topology for the single-level case this chapter cascades, or The PCIe Fabric for the path reasoning underneath it all. Browse the full path on the PCIe tutorials index.