PCIe · Module 2

The PCIe Fabric — Paths, Not a Medium

Why a PCIe system is better understood as a fabric of routed paths than as a bus: how the components form one connected structure, why every transaction has a traceable path, and why dedicated local Links do not imply dedicated end-to-end resources.

Module 2 has introduced the components one at a time. Chapter 2.1 drew the map, 2.2 defined the host-side root, 2.3 defined path termination, 2.4 defined intermediate forwarding, 2.5 defined crossing into another architecture, and 2.6 defined how positions in the result are identified.

Knowing the parts is not the same as understanding the system. This chapter assembles them:

Why is a PCIe system better understood as a fabric of routed paths than as a bus?

The question is not terminological fussiness. Calling PCIe "a bus" imports a set of intuitions — that traffic is globally visible, that bandwidth is one pool, that every device competes with every other — and every one of those intuitions produces wrong predictions. Replacing them with the right ones is what this chapter is for.

1. What the Components Become Together

Each component introduced in this module does something local. A Root Complex bounds the architecture at the host. A Switch forwards between its connections. An Endpoint terminates. A Link joins two adjacent components.

Compose them and something appears that none of them has individually: connectivity between components that are not adjacent. An Endpoint two levels down has no connection to the Root Complex, yet it can reach system memory. That reachability is not a property of any single component — it emerges from the arrangement.

That emergent property is what "fabric" names. The word is not decoration for "network of switches"; it points at the thing worth reasoning about, which is end-to-end connectivity through a structure, as distinct from the local behaviour of any part of it.

This is why the fabric deserves its own chapter after the components have theirs. Understanding a Switch tells you what happens at one hop. Understanding the fabric tells you what happens to a transaction that traverses several — and those are different questions with different answers.

2. Fabric Versus Bus, Precisely

The contrast is worth stating property by property, because loose versions of it are the source of most PCIe misconceptions.

A shared bus is one physical medium that every participant attaches to. Traffic on it is globally visible — every device sees the same wires. Transport is one common resource, so its capacity is divided among all attached devices. Access is granted by arbitration, and contention is universal: any device's activity is potentially any other device's problem, regardless of whether their traffic is related.

A PCIe fabric is many dedicated Links joined by forwarding components. Traffic is not globally visible — a component on one Link cannot observe another. Transport is many separate resources rather than one, and each transaction consumes only the specific Links along its route. Access is not granted; transactions are forwarded. Contention is not universal but path-dependent: two devices interfere only if their routes share something.

3. Every Transaction Has a Path

The central discipline of fabric reasoning is that a transaction is not an event on a medium — it is a journey, and it has four properties worth naming explicitly:

An origin — where it was generated. The Root Complex for host-directed traffic; an Endpoint for device-originated traffic.
A destination — where it is going. An Endpoint's resources, or system memory, or something else reachable through the structure.
A route — the specific sequence of Links it traverses.
Forwarding points — the Switches along that route that had to decide where to send it next.

Two transactions with the same origin and destination take the same route. Two with different endpoints may take routes that are completely disjoint, partially overlapping, or nearly identical — and which of those it is is determined entirely by topology.

Figure 1 — one fabric, several paths. Endpoint A is one Link from the Root Complex. Endpoint B is two. Endpoints C and D sit deeper, behind a second Switch. Traffic from C and D toward the host shares Switch 2's upstream Link, then shares Switch 1's upstream Link, then reaches the Root Complex — so the deeper a path runs, the more segments it shares with its neighbours. Endpoint A's path shares nothing with the others until the root.

Trace three routes in Figure 1 and the reasoning becomes concrete:

Endpoint A to the host traverses one Link.
Endpoint B to the host traverses two: its own Link, then Switch 1's upstream Link.
Endpoint D to the host traverses three: its own Link, Switch 2's upstream Link, then Switch 1's upstream Link.

Now ask about pairs. A and B share nothing until the Root Complex. C and D share Switch 2's upstream Link and Switch 1's upstream Link. B and D share Switch 1's upstream Link but not Switch 2's. Every one of those answers is read directly off the topology, and none of them is available from a device's identity alone.

This section is the practical heart of fabric reasoning, because shared segments are where system behaviour stops being intuitive.

The key move is to stop asking "is this device fast?" and start asking "what does this device's path share, and with whom?" Three kinds of convergence appear in real fabrics:

A Switch's upstream Link. Everything below a Switch traverses it. In Figure 1, Switch 1's upstream Link carries traffic for Endpoints B, C, and D combined. Their aggregate demand can exceed what that one segment provides, and when it does, all three are affected regardless of how capable their individual Links are.

Nested convergence. Deeper structures converge more than once. Endpoints C and D first meet at Switch 2's upstream Link, then meet Endpoint B's traffic at Switch 1's upstream Link. Each level of depth adds a shared segment, which is why fabric depth is a performance consideration and not only a connectivity one.

Shared destinations. Even paths that share no Link converge if they target the same thing. Traffic from every Endpoint in Figure 1 heading for system memory arrives at the Root Complex and continues into a memory subsystem also serving the processor — the point Chapter 2.2 made about the root being a convergence point, and the point Chapter 1.4 made about AGP long before that.

Local bandwidth is not end-to-end bandwidth

If one idea from this chapter survives, make it this one, because it is the single most common source of wrong expectations about PCIe systems.

An Endpoint's Link is genuinely dedicated. Nothing else attaches to it, and its capability is a real property of that connection. It is therefore tempting to treat that capability as what the device will achieve.

It is not. What the device achieves depends on every segment along its route, and those segments are shared with whatever else sits behind them. An Endpoint with a highly capable Link, sitting behind a Switch whose upstream Link is modest and busy, will not achieve what its own Link suggests — and nothing is wrong with either connection.

A dedicated Link guarantees something about one hop. It guarantees nothing about the path.

Chapter 2.8 develops what a Link does and does not promise; the fabric-level consequence is here.

5. Why Position in the Fabric Matters

Put the previous two sections together and a conclusion follows that has no analogue in a shared-bus system:

Two identical devices in different positions in the same fabric can behave differently.

On a shared bus this could not happen in the same way. Every device attached to the medium faced the same conditions — the same contention from the same population, the same aggregate pool. Position was not a meaningful variable, because there was only one place to be.

In a fabric, position determines what a device's path traverses, which determines what it shares, which determines how it behaves under load. Endpoint A and Endpoint D in Figure 1 could be the same part with the same capabilities and still show different throughput and latency characteristics, purely because one is a single hop from the root and the other is three hops behind two convergence points.

This has a direct consequence for how you describe a system: naming a device is not enough; you must name its position. It is why Chapter 2.6 argued that hierarchical identity exists for structural reasons rather than bookkeeping ones, and why a bug report that says "it fails on this device" is weaker than one that says "it fails at this position in this topology."

6. Common Misconceptions

Myths to unlearn

"Fabric means a full mesh." It does not. A PCIe fabric is a rooted hierarchy — a tree — in which every component has exactly one path back toward the root. "Fabric" here refers to structured multi-hop connectivity, not to arbitrary any-to-any wiring. Importing mesh intuitions will make you expect routes that do not exist.
"Every endpoint has a dedicated path all the way to memory." Each Endpoint has a dedicated Link. The rest of its route is shared with everything else behind the same Switches, and its destination is shared with everything targeting system memory. Dedicated at one hop, shared beyond it.
"A fabric has no contention." It has no globally imposed contention. Contention still occurs wherever paths converge — upstream Links, nested convergence points, shared destinations. What changed is that contention became a consequence of topology rather than of mere attachment.
"Fabric is just another word for switch." A Switch is one component that forwards at one hop. The fabric is the whole connected structure and the end-to-end connectivity it provides. Reasoning about a Switch answers questions about one hop; reasoning about the fabric answers questions about routes.
"If two devices have dedicated Links, their traffic is independent." Only if their paths do not overlap. Two Endpoints under the same Switch each have dedicated Links and still contend on the segment above them. Dedicated local connections say nothing about whether routes intersect.
"A fabric guarantees arbitrary peer-to-peer communication." The structure is a rooted hierarchy, and host-directed and device-to-memory traffic are the ordinary patterns. Whether direct transfers between two Endpoints occur depends on system configuration and platform support, as Chapter 2.1 noted — it is not a default capability to rely on.

7. Real Engineering Relevance

Fabric thinking converts vague performance complaints into structural questions with checkable answers. Four are worth internalising, because they cover most of what goes wrong.

What path did this transaction take? Before analysing why something was slow or failed, establish the route: which Links, which forwarding points, how many hops. A great deal of confusion comes from reasoning about a device without reasoning about its path.

Where do two traffic streams first share a resource? When one workload degrades another, the interference has a location. Find the first shared segment between the two routes, and you have found the candidate. If the routes share nothing until the root, the interference is at the root or beyond it — in the memory subsystem, most likely — and no amount of investigating the Links will help.

Is this bottleneck link-local or fabric-wide? A problem confined to one connection presents differently from a saturated shared segment. The diagnostic signature differs too: link-local problems affect one device; convergence problems affect everything behind the convergence point simultaneously, which is a much more distinctive pattern once you know to look for it.

Does the failure reproduce only through one branch? If a problem appears for a device on one branch and not for an equivalent device elsewhere, that asymmetry is information. It usually means something about the shared segments differs — different depth, different neighbours, different aggregate load.

For verification specifically, this reframes what a meaningful test is. Exercising an Endpoint in isolation tells you about the Endpoint. Exercising it while its path is loaded tells you about the system — and those are different tests that find different bugs. A testbench that can only produce the first will report clean while the real system misbehaves under exactly the conditions that matter.

8. Understanding Check

9. Summary

A PCIe system is a fabric: the Root Complex paths, Switches, Links, and Endpoints that together provide end-to-end connectivity through a structured hierarchy. That connectivity is emergent — no single component provides it — which is why the fabric deserves reasoning distinct from reasoning about any part.

The contrast with a bus is precise. A bus is one medium with globally visible traffic, one divided transport pool, and universal contention. A fabric is many Links, traffic invisible outside the connection it traverses, transport consumed only along a route, and path-dependent contention. The correct inference is not that a fabric is unlimited but that sharing follows topology instead of being imposed by attachment.

Every transaction has an origin, a destination, a route of specific Links, and forwarding points along the way. Paths converge in three ways: at a Switch's upstream Link, at nested convergence points in deeper structures, and at shared destinations such as system memory. From this follows the chapter's most consequential claim: a dedicated Link guarantees something about one hop and nothing about the path, and therefore two identical devices at different positions in the same fabric can behave differently.

Hold the model: traffic follows paths through the fabric rather than appearing on one shared medium.

10. What Comes Next

This chapter reasoned about whole routes. Chapter 2.8 — Point-to-Point Links goes the other direction and examines a single edge in depth: what it means that a Link joins exactly two components, what is genuinely local to that connection, and — sharpening a distinction this chapter has been leaning on — why a Link and a path must never be used interchangeably.

Chapter 2.9 then closes Module 2 by comparing the complete switched system against the shared-bus architectures it replaced, with every component now defined.

Revisit Switch for the forwarding component this fabric is built around, or Hierarchy Domains for why positions in it need structured identity. Browse the full path on the PCIe tutorials index.