Real System Examples — Reading a Topology You Did Not Draw

Module 4 built the reasoning on clean diagrams: anchors, leaves, fanout, depth. Real systems are messier — they arrive as a block diagram in a datasheet, a lspci tree, or nothing at all.

How do we apply topology reasoning to realistic systems rather than clean textbook trees?

1. The Procedure

Applied identically to every system in this chapter. Worth internalising as a checklist, because doing it in order prevents the most common error — reasoning about a device before establishing what its path actually is.

1. Identify the device and confirm it is a leaf rather than a forwarding element.
2. Trace Links upstream, hop by hop, until you reach the Root Complex.
3. Record every Switch on the way, in order.
4. Identify the Root Port the path terminates at.
5. Mark every shared segment — any Link carrying traffic for more than this device.
6. Find the common ancestor with each other device of interest.
7. Identify likely contention points — the shared segments, in order of how much traffic converges on them.
8. Identify the failure domain — which other devices go down if each element on the path fails.

Steps 1–4 are mechanical. Steps 5–8 are where the reasoning happens, and they are only reliable if 1–4 were done honestly.

2. Example A — A Desktop or Workstation

Run the procedure on three devices:

What this predicts. The accelerator and NVMe have no shared PCIe segment — their paths meet only at the Root Complex. Heavy accelerator traffic should not throttle NVMe through the fabric. Meanwhile the network device, storage controller, and other peripherals all share one segment, so they can throttle each other.

And the crucial qualification. "No shared PCIe segment" is not "independent." Both the accelerator and the NVMe device converge at the Root Complex and target system memory, shared with the processor. If both saturate simultaneously, they can interact there — and that interaction is invisible on this diagram because it happens beyond PCIe. §6 develops when this matters.

3. Example B — A Server Behind One Switch

Every one of the four devices has the same path shape: own Link → Switch upstream segment → Root Complex. Their common ancestor is the Switch, so the upstream segment is where all four interact — the first and only shared PCIe segment.

Three inferences follow directly:

Any device can affect any other. Unlike Example A, where the accelerator and NVMe were fabric-independent, here all four converge one hop out. A single heavy device consumes shared capacity that the other three then cannot use.

The failure domain is the whole group. The Switch or its upstream Link failing removes all four simultaneously. An individual device's Link failing removes one.

Oversubscription is likely and probably intentional. If each device could individually demand close to what the upstream segment carries, four of them cannot all be satisfied at once. As Chapter 4.4 argued, that is a rational design when peak demands are not expected to coincide — and a problem when that assumption is wrong.

4. Example C — A Multi-Level Fabric

The common-ancestor table is where the structure becomes actionable:

Read the design intent from this. Storage devices are grouped together and network devices are grouped together, so intra-group traffic contends within its own branch while inter-group traffic contends only at the top. That is a deliberate arrangement: it isolates the two workloads from each other except at the segment everything must cross.

L0 carries everything. It is the single most loaded segment in the fabric and the single largest failure domain. In any multi-level fabric, the root-facing segment deserves attention first — for capacity and for reliability.

5. Performance Exercises

Apply the procedure to questions an engineer actually gets asked.

"The accelerator and NVMe are both busy in Example A. Do they interfere?"

Trace both: each has its own Link straight to a Root Port. No shared PCIe segment — their paths meet only at the Root Complex. Through the fabric they are independent. But both target system memory, so they can interact in the memory subsystem. The honest answer: not in the PCIe fabric; possibly at the host side, which requires different measurements to establish.

"All four devices in Example B drive sustained traffic toward the host. What happens?"

All four paths traverse the Switch's upstream segment. Offered load there is the sum of four devices' demand; if that exceeds the segment's capacity, all four are limited and each gets a share determined by the Switch's arbitration. Expect reduced throughput for everyone, roughly together. This is not a fault — it is the oversubscription assumption being tested.

"Storage 1 and Network 1 in Example C both go busy. Where do they first meet?"

Their common ancestor is Switch A, so L0 — the topmost segment. They share nothing before that. Storage 1 contends with Storage 2 on LB long before it contends with Network 1 anywhere.

"Switch B fails in Example C. What disappears?"

Everything below it: Storage 1 and Storage 2. The accelerator and both network devices are unaffected, because their paths do not traverse Switch B. The failure domain is exactly the subtree.

6. When Topology Stops Explaining It

The method has a boundary, and knowing where it lies is as valuable as the method itself.

Topology explains behaviour when the coupling is through shared PCIe segments. It stops explaining when the shared resource is somewhere else:

System memory. Every device targeting host memory converges there, regardless of path. Two devices with entirely disjoint PCIe paths can still contend for memory bandwidth.

Host-side resources within or beyond the Root Complex. Structures serving multiple Root Ports can couple traffic the fabric keeps separate.

Software. Driver behaviour, interrupt handling, and submission/completion processing can limit a device regardless of what its Link can carry — and can serialise devices that share a CPU core or a lock.

Platform-level effects. Power delivery, thermal management, and clocking are shared in ways no topology diagram shows.

7. A Verification Topology Model

Topology reasoning is mechanical enough to automate, and a testbench that knows the fabric can check things a topology-blind one cannot.

// Verification-only. NOT synthesizable. NOT PCIe protocol state.
// A testbench model of fabric structure, sufficient to compute paths,
// common ancestors, and failure domains.
typedef struct {
  int    node_id;        // unique within this model
  int    parent_id;      // -1 for the Root Complex
  int    upstream_port;  // port on the parent that reaches this node
  int    depth;          // Links from the root
  string name;
} pcie_topo_node_t;
 
class pcie_topology;
  pcie_topo_node_t nodes[int];   // node_id -> node
 
  function void add(int id, int parent, int port, string name);
    pcie_topo_node_t n;
    n.node_id       = id;
    n.parent_id     = parent;
    n.upstream_port = port;
    n.name          = name;
    n.depth         = (parent < 0) ? 0 : nodes[parent].depth + 1;
    nodes[id]       = n;
  endfunction
 
  // Path from a node up to the root, as a list of node ids (leaf first).
  function void path_to_root(int id, ref int path[$]);
    int cur = id;
    path.delete();
    while (cur >= 0) begin
      path.push_back(cur);
      cur = nodes[cur].parent_id;
    end
  endfunction
 
  // Nearest node through which both paths pass. Returns -1 if the model is
  // inconsistent (no shared ancestor), which itself indicates a model bug.
  function int common_ancestor(int a, int b);
    int pa[$], pb[$];
    path_to_root(a, pa);
    path_to_root(b, pb);
    foreach (pa[i]) begin
      foreach (pb[j]) begin
        if (pa[i] == pb[j]) return pa[i];
      end
    end
    return -1;
  endfunction
 
  // Every node whose path traverses `element` — i.e. the failure domain if
  // that element (or its upstream Link) becomes unusable.
  function void affected_by(int element, ref int victims[$]);
    int p[$];
    victims.delete();
    foreach (nodes[id]) begin
      path_to_root(id, p);
      foreach (p[k]) begin
        if (p[k] == element) begin
          victims.push_back(id);
          break;
        end
      end
    end
  endfunction
endclass

Classification: verification-only (SystemVerilog class, not synthesizable).

What it enables. Three checks that are otherwise manual and error-prone:

• Expected path. Confirm a transaction traverses exactly the segments the model predicts — and no others. A transaction appearing where the model says it should not is either a forwarding bug or a wrong model, and both are worth knowing.
• Predicted contention. Confirm interference between two devices appears at their computed common ancestor and not before. Interference earlier than predicted means paths are coupled somewhere the model does not capture.
• Predicted failure domain. Inject a fault at an element and confirm exactly affected_by(element) degrades. A device outside that set being affected is a significant finding — it means something couples paths the topology says are separate.

Deliberately simplified: a pure tree with one parent per node, so it cannot represent alternate paths; no capacity or capability per Link, so it predicts where contention occurs but not how much; and common_ancestor is O(n²) in path length, which is irrelevant at realistic depths.

What a production environment would add: per-Link capability so predictions become quantitative, per-node device class, and integration with the correlation-ID tracing from Chapter 3.5 so measured paths can be compared against predicted ones automatically.

8. Debugging: Affected Set to Suspect

The method from Chapter 4.4, applied to realistic systems. In each case: find the smallest topology element containing every affected device.

One device affected, neighbours fine. Something local: its own Link, its port on the parent, or the device itself. The neighbours' health is what rules out everything shared.

All devices behind one Switch affected. That Switch or its upstream segment. In Example B this means all four devices — and it is not a device problem, so investigating any individual one wastes time.

One subtree affected in a multi-level fabric. The Switch heading that subtree, or its upstream Link. In Example C, both storage devices affected while everything else is fine points at Switch B or LB.

Everything under one Root Port affected. That Root Port, its Link, or the top Switch. In Example C that is Switch A or L0 — and note this is indistinguishable from "the whole fabric" if there is only one populated Root Port.

Devices from unrelated branches affected together. The signature from §6. If their only common element is the Root Complex, look above PCIe: memory, host-side resources, software, or platform.

Load-dependent versus always. Cutting across all of the above: a fault that appears only under load points at capacity at the shared element; one that appears regardless points at function — the element or its Link.

9. Common Misconceptions

10. Understanding Check

11. Summary — and the Close of Module 4

Reading a real PCIe system is a procedure: identify the device, trace its Links upstream, record the Switches, identify the Root Port, mark shared segments, find common ancestors, identify contention points, identify failure domains. Steps one to four are mechanical; the reasoning depends entirely on having done them honestly.

Three shapes recur. Independent Root Ports give devices short unshared paths that meet only at the Root Complex. One Switch with fanout puts every device on a common upstream segment, making them mutually coupled and forming one failure domain — the canonical oversubscription arrangement. Multi-level fabrics make the common-ancestor question interesting, since the answer differs per pair and grouping is usually deliberate.

The method's boundary matters as much as its use. Topology explains coupling through shared PCIe segments. When devices with no shared segment degrade together, the model is exhausted and the suspects move above PCIe — memory, host-side resources, software, platform. And devices sharing a segment degrading together is consistent with a fabric cause without proving one.

Module 4 as a whole: 4.1 established where hierarchies are anchored, 4.2 what placement does to a leaf, 4.3 how fanout concentrates traffic, 4.4 what depth adds, and this chapter applies all of it to systems you did not draw.

Hold the model: trace the path, identify what is shared along it, and use path overlap to predict coupling and failure domains.

12. What Comes Next

Module 4 answered where things are and what shares what. Every conclusion it produced was relative — this segment carries more than that one, these devices interfere, that branch is a failure domain. Nothing so far has attached a number to a Link.

Module 5 does. It introduces the generational ladder and, more importantly, the quantitative discipline that goes with it: what a signalling rate actually means, how encoding transforms it, and why the number on a datasheet is not the throughput an application sees. Chapter 5.1 starts at the beginning, with Gen1.

Revisit Multi-Level Fabrics for the common-ancestor method this chapter applies. Browse the full path on the PCIe tutorials index.