PCIe · Module 2

Switched Architecture — How the Finished System Behaves

Module 2's capstone: how a switched hierarchy of dedicated Links behaves structurally against the shared bus it replaced — where concurrency comes from, where arbitration and bottlenecks moved to, and what still limits scaling.

Module 2 has defined every component of a PCIe system and both ways of reasoning about it — whole routes and single edges. This chapter closes the module by asking what the finished system actually does:

How does a switched hierarchy of point-to-point Links differ structurally from the shared-bus architectures it replaced?

A word on what this chapter is not. Chapter 1.8 derived PCIe from constraints — it argued why the architecture had to change, working forward from problems to required properties. That argument is complete and is not repeated here. This chapter runs in the opposite direction: every component is now defined, so the question is how the assembled system behaves, which is a structural question rather than a historical one.

1. The Structural Comparison

Set the two architectures side by side property by property. Module 1 explained why the left column stopped working; the point here is what the right column is.

Figure 1 — the shared bus. One medium, and every device attached to it. Bandwidth is one pool divided among all attached devices, ownership must be arbitrated because only one transfer can occupy the medium at a time, and every attachment is part of the same electrical environment. Contention here is universal: it follows from attachment, not from what any device is doing.

Figure 2 — the switched hierarchy serving the same three devices. Every edge is now a separate Link with exactly two attached components. Device A reaches the root directly; B and C reach it through a Switch that forwards between connections. Transfers on non-overlapping Links proceed concurrently, and contention no longer follows from attachment — it appears only where paths converge, here on the Switch's upstream Link.

The differences that matter are these:

	Shared bus	Switched hierarchy
Transport	One resource for everyone	Many Links, consumed only along a route
Contention	Universal — follows from attachment	Path-dependent — follows from overlap
Concurrency	None; one transfer occupies the medium	Available where paths do not overlap
Capability	Global property of the medium	Local property of each connection
Growth	Adds a claimant to a fixed pool	Adds a Link, and can add a forwarding component
Electrical scope	Every attachment affects every device	Determined per connection

Three devices are served in both figures. In Figure 1 they are three claimants on one resource. In Figure 2 they occupy four Links, of which two are shared by exactly two of them.

2. Contention Became Path-Dependent

The single most consequential difference is why two devices interfere.

On a shared bus, the answer is because they are both attached. Device A and Device C in Figure 1 contend even if their traffic is entirely unrelated, targets different destinations, and never occurs at the same moment for any application reason. Attachment alone is sufficient. There is no arrangement of devices, no assignment of workloads, and no scheduling policy that removes it.

In a switched hierarchy, the answer is because their paths overlap. Endpoints B and C in Figure 2 contend, because both routes traverse the Switch's upstream Link. Endpoint A does not contend with either until the root, because its path shares no Link with theirs.

That converts an unconditional property into a structural one:

Interference is no longer a consequence of being in the system. It is a consequence of where in the system.

Everything else in this chapter follows from that sentence. Concurrency exists because some paths do not overlap. Bottlenecks have locations because overlap has locations. Topology matters because topology determines overlap.

3. Arbitration Moved — It Did Not Vanish

A precise claim is needed here, because the loose version is wrong in a way that will cost you.

What disappeared is global physical-bus ownership arbitration: the mechanism by which one device is granted exclusive use of a shared medium while all others wait. That mechanism is gone because the medium it arbitrated is gone. Nothing in a PCIe hierarchy grants exclusive use of a common set of wires, because there is no common set of wires.

What did not disappear is the need to resolve competition for a finite resource. Wherever traffic converges, something must decide what proceeds and in what order. Those places are identifiable:

A Switch's outgoing connection, when traffic from several sources needs to leave by the same one.
The upstream Link, which carries everything from below toward the root.
The Root Complex and system memory, where traffic from the whole hierarchy converges — the point Chapter 2.2 made about the root being a convergence point.
Internal resources within components, which are implementation matters this chapter makes no claims about.

So the accurate formulation:

PCIe eliminated arbitration for ownership of a shared physical medium. It did not eliminate the need to schedule access to convergence points — it relocated that need to specific, identifiable places.

The difference is not semantic. Global arbitration made every device's performance a function of every other device's activity. Convergence scheduling makes a device's performance a function of what shares its path, which is a much smaller and more predictable set — and one a system designer can choose.

How scheduling decisions are actually made at those points is an implementation matter, and later modules cover the mechanisms PCIe defines for managing traffic. This chapter deliberately stops at the architectural fact that convergence points exist and must be resolved.

4. Concurrency Is Topology-Dependent

This is the property that has no equivalent in the older model, and the most useful way to understand it is through cases.

Case A — non-overlapping paths. Endpoint A is moving data toward system memory. Simultaneously, Endpoint B is doing the same. Their routes are: A → its Link → root; and B → its Link → Switch → upstream Link → root. Those routes share no Link. Both transfers therefore occupy entirely separate transport resources and proceed concurrently within the fabric.

Case B — overlapping paths. Endpoints B and C are both moving data toward system memory. Their routes are: B → its Link → Switch → upstream Link → root; and C → its Link → Switch → upstream Link → root. Those routes share the upstream Link. Their own Links remain dedicated and uncontended, but on the shared segment their demands meet, and together they can want more than it provides.

Two conclusions, and the second is the one people miss:

Concurrency is real. Case A describes something a shared bus could not do at all. Two transfers genuinely in progress at once is the structural payoff of the architecture.

Concurrency is conditional. Case B is equally real, and no property of PCIe prevents it. Whether two flows proceed independently is not a fact about PCIe — it is a fact about their positions in a particular topology.

The question that replaces “how fast is this bus?”

On a shared bus there was one meaningful performance question, because there was one resource: how much can the bus carry, and how is it divided?

In a switched hierarchy that question has no answer, because there is no single resource to ask about. The question that replaces it is:

For the flows I care about, where do their paths first overlap — and how much capacity is there at that point?

This is a better question in every respect. It has a definite answer, the answer is derivable from topology, and the answer tells you what to change. It is also harder, because it requires knowing the structure rather than reading one number off a datasheet.

That trade — a harder question with an actionable answer, in place of an easy question with a useless one — is a fair summary of what the architecture did to performance reasoning.

5. Where Bottlenecks Live Now

Bottlenecks did not disappear; they migrated, and knowing where they went is most of the practical value of this chapter.

Under a shared bus the bottleneck was singular and obvious: the bus. Every performance problem eventually reduced to the same resource, which made diagnosis simple and improvement nearly impossible — the only responses available were to reduce demand or replace the bus for everyone.

In a switched hierarchy bottlenecks appear in several distinguishable places:

A shared upstream Link, carrying the aggregate of everything behind it. The most common convergence point, and the one that grows more significant with fabric depth, since deeper structures converge more than once.
A Switch's outgoing connection, when several sources target the same downstream direction.
Root-side convergence, where traffic from the entire hierarchy meets.
System memory, shared with the processor and with every device targeting it.
The device or its own Link, when the constraint is genuinely local.

This is a harder diagnostic situation than "the bus is full," and it is a far better engineering situation. Each of those locations can be sized independently: a deeper branch can be given more upstream capacity, a demanding device moved closer to the root, a fabric restructured to separate flows that were interfering. None of those moves was available when there was one resource.

The trade is explicit: better scalability in exchange for topology mattering. You gain the ability to put capacity where it is needed; you accept that reasoning about performance now requires knowing the structure.

6. How It Scales, and What Still Limits It

The scaling story is genuinely strong, and it is worth stating without overstating.

What improved. A new device attaches through a new Link rather than as another claimant on a fixed pool, so connectivity and transport grow together — the property Chapter 1.7 identified as missing and required. Fan-out is added by adding forwarding components. Capability is provisioned per connection, so demanding and undemanding devices coexist without one constraining the other. Non-conflicting traffic overlaps, so aggregate throughput is not capped by a single resource.

What still limits it. Several things, and none of them is a defect:

Shared upstream capacity. Everything behind a Switch converges above it. Adding devices below a Switch does not increase what that Switch's upstream Link carries.
Depth. Each level adds a forwarding step and another convergence point. Depth is not free in latency or in shared segments.
Shared destinations. System memory remains common to everything targeting it, no matter how well the fabric separates paths.
Implementation limits. Components have finite resources, and this chapter makes no claims about specific ones.
Topology quality. A poorly-arranged hierarchy can perform worse than a well-arranged smaller one. The architecture permits good scaling; it does not confer it.

That last point deserves emphasis because it is where the architecture's promise is most often misread. Switching does not make a system fast. It makes it possible to build a system whose capacity is where its demand is — and whether a given system achieves that depends on decisions its designers made.

7. Common Misconceptions

Myths to unlearn

"PCIe eliminated arbitration." It eliminated arbitration for ownership of a shared physical medium, because that medium no longer exists. Competition for finite resources at convergence points — a Switch's outgoing connection, the upstream Link, the root, system memory — still has to be resolved. What changed is that this is now local and identifiable rather than global and unconditional.
"Switched means every device has a private route to memory." Every device has a private Link. Beyond that hop the route is shared with everything behind the same Switches, and system memory is shared with everything targeting it. Private at one hop, shared beyond it.
"Switches guarantee non-interference." They guarantee that devices do not contend merely for being attached. Devices whose paths overlap still interfere, and every device below a Switch shares its upstream Link. Non-interference is a property of disjoint paths, not of switching.
"A switched architecture is always faster regardless of topology." The architecture makes good outcomes achievable, not automatic. A badly-arranged hierarchy — excessive depth, demanding devices crowded behind one modest upstream connection — can underperform. Topology is now a design variable, which means it can be got wrong as well as right.
"A PCIe Switch behaves like an Ethernet switch in every respect." Both forward between connections, and the analogy is useful exactly that far. PCIe's structure is a rooted hierarchy with a host-side boundary, not an arbitrary network of peers, and its rules for how transactions are handled are its own. Carrying networking assumptions across will produce wrong expectations about what traffic patterns are ordinary.
"More switches always increase bandwidth." A Switch adds connectivity and fan-out. It does not add capacity to the Link above it — everything it serves still converges there. Adding depth without adding upstream capacity increases the number of devices sharing the same segment, which can make performance worse.
"Point-to-point plus switching means a full mesh." The structure is a tree with a root. Every component has exactly one path back toward the root, and there are no alternate routes between arbitrary components. Mesh intuitions predict routes that do not exist.

8. Real Engineering Relevance

This section is the capstone skill of Module 2, because it is what all nine chapters were building toward: reading a system structurally instead of anecdotally.

Given a PCIe topology and a performance or correctness symptom, an engineer who has this module should be able to work through the following without hesitating:

Identify where traffic is forwarded. Which components make forwarding decisions on the routes in question, and how many hops each route involves. This establishes what the traffic actually traverses.

Identify where paths overlap. For any two flows, find the first shared segment. If there is none before the root, they are independent within the fabric and any interaction between them is happening at the root or beyond — in the memory subsystem, most often.

Predict likely contention points. Before measuring, the topology already suggests where trouble will appear: upstream Links serving many devices, deep branches with several convergence levels, and any segment where a demanding device sits behind a modest connection.

Distinguish link-level from fabric-level bottlenecks. A link-local problem affects one device. A convergence problem affects everything behind that point simultaneously — a distinctive signature once you know to look for it, and one that immediately narrows where to investigate.

Explain why endpoint behaviour depends on topology. Two identical devices at different positions can behave differently, and this is expected rather than anomalous. Being able to say why prevents a great deal of wasted investigation into devices that are working correctly.

Read symptoms structurally. "Several devices behind one Switch all degraded when a new workload started" is not a mystery — it is a convergence signature, and it names its own suspect. "One device degraded while its neighbours did not" points somewhere entirely different.

For verification work, the consequence is that topology is a test variable. An Endpoint exercised in isolation is being tested under conditions its real deployment will not reproduce. Meaningful system-level verification exercises devices while their paths are loaded, because that is when convergence behaviour appears — and convergence behaviour is precisely what the switched architecture introduced and what a shared-bus mental model cannot predict.

9. Understanding Check

10. Summary — and the Close of Module 2

A switched PCIe architecture replaces global shared-medium contention with path-based forwarding over independent Links. Three devices served by one bus are three claimants on one resource; served by a hierarchy, they occupy several Links of which only some are shared.

Contention became path-dependent. Devices interfere because their routes overlap, not because both are attached — converting an unconditional property into a structural one. Arbitration moved rather than vanished: ownership of a shared medium is gone because the medium is gone, but competition at convergence points still has to be resolved. Concurrency is real and conditional — transfers on non-overlapping paths genuinely proceed at once, and whether two flows overlap is a fact about topology rather than about PCIe. Bottlenecks migrated from one obvious resource to several identifiable ones, which is harder to diagnose and far better to engineer, since each can be sized independently. Scaling improved because connectivity and transport grow together and capability is provisioned per connection — bounded still by shared upstream capacity, depth, shared destinations, and the quality of the topology itself.

Module 2 as a whole moved from a system map through each component to two complementary ways of reasoning about the assembled result: the map, the root, the leaves, the branching element, the architectural boundary, structural identity, whole routes, single edges, and finally the behaviour of the whole.

Hold the model: resources and bottlenecks are localized to specific connections and convergence points, rather than imposed on every device by one common bus.

11. What Comes Next

Modules 1 and 2 together answer why PCIe is shaped this way and what it is made of. Both have stayed deliberately at the level of components, connections, and topology — with one recurring deferral. Every chapter that needed to describe what actually travels across a Link has said "a transaction" and moved on.

Module 3 takes that up. PCIe organises the work of moving a transaction across a Link into layers, each with distinct responsibilities: producing and consuming transactions, ensuring they arrive reliably across a connection, and putting them on the physical medium. That layering is why a Switch can forward without concerning itself with a payload's meaning, and why a Link can have its own operational state independent of the traffic it carries.

Everything in this module was structure. Module 3 is the first look inside what moves through it.

Revisit Point-to-Point Links for the local view or The PCIe Fabric for the whole-route view. Browse the full path on the PCIe tutorials index.