Evolution of System Interconnects

Before you learn a single PCIe packet format, lane count, or link speed, answer a more basic question: why did the industry abandon the shared parallel bus? For roughly fifteen years a PC connected its graphics, network, and storage cards to the CPU across a shared parallel bus — ISA, then PCI, with AGP bolted on as a graphics-only exception. Each ran into limits that were physical as much as architectural, and PCIe is the architecture that answered them.

This chapter is the map of that arc: shared expansion buses → rising bandwidth demand → electrical and architectural scaling limits → point-to-point serial links → a switched fabric. The chapters right after this one walk the territory one stop at a time; read this first so you can see why the road bends before you walk it.

1. What a System Interconnect Actually Does

Start with the job, not the technology. A computer is a host — a CPU and its memory — surrounded by devices that are useless unless they can move data to and from it: a GPU, an SSD, a network controller, an accelerator. The system interconnect is the wiring plus the rules that carry that traffic. It is the off-chip counterpart of the on-chip bus: where AMBA AXI governs how blocks inside a chip talk, PCI or PCIe governs how whole chips and cards talk across a board.

The job has three parts: address a device, move data to or from it, and do both while many devices share one host. That last part — sharing — is what this chapter is about, and a shared bus and a switched fabric are two different answers to it.

2. The Parallel Bus Era — One Medium, Every Device On It

The early answer was simple and, for its time, correct: put every device on one shared, parallel bus. Parallel means address and data ride many wires at once — a 32-bit bus moves 32 data bits simultaneously, sampled against a shared bus clock. Shared means every device electrically taps those same wires, and only one may drive a transfer at a time.

Three generations mark the arc, each with its own chapter ahead:

• ISA — the original PC expansion bus: narrow and slow, adequate when a peripheral was a serial port or a floppy controller. (Chapter 1.2)
• PCI — wider and faster, with an arbiter granting the shared wires to one device at a time. It carried graphics, networking, and storage for a generation. (Chapters 1.3, 1.5)
• AGP — a telling exception: graphics alone outgrew what shared PCI could supply, so that one slot was given its own point-to-point connection to the chipset while everything else stayed on the shared bus. (Chapter 1.4)

For a mid-1990s PC this was the right design. The difficulty is that all three defining traits — shared, parallel, bus — became constraints once devices got fast and numerous.

3. Where the Shared Parallel Bus Ran Out

Pressure came from two directions at once, and separating them is the most useful distinction in this chapter.

Electrical — what the medium can carry. A shared bus is multi-drop: many devices tap the same wires through connectors and stubs, and every attachment adds capacitance and an impedance discontinuity. That sets a trade rather than a fixed number — for a given signalling rate there is a practical ceiling on how many devices the medium can carry, and raising the rate tightens it. Independently, a synchronous parallel bus needs every bit of a word valid inside one sampling window, but routing and driver differences spread bit arrival times (skew) by an amount the clock period does not shrink along with it.

Architectural — what the model can organise. Even with ideal signal integrity, a shared bus has one pool of bandwidth divided among all devices, and arbitration serialises transfers: while one owns the wires, the rest wait. Adding devices adds waiting, not throughput. Width is costly too — pins on every chip, length-matched traces on every board — so buying bandwidth by widening buys routing constraint with it.

Chapters 1.5–1.7 take these apart properly; what matters here is the shape of the problem.

4. The Shift — From Shared Bus to Switched Serial Fabric

PCIe keeps the job of a system interconnect and changes how it is done. Three linked ideas define the shift, each answering a constraint from §3.

Point-to-point, not multi-drop. A PCIe link connects exactly two devices — nothing else taps it. That removes multi-drop loading at its source: one transmitter, one receiver, a controlled-impedance channel with nothing else to reflect off, which sustains far higher signalling rates than a shared medium.

Serial with embedded timing, not wide synchronous parallel. A PCIe lane is a differential pair in each direction, with timing recovered from the data stream rather than taken from a bus clock accompanying a parallel word. That retires the source-synchronous model in which many bits must land inside one common window, so the per-lane rate is not bounded by bit-to-bit skew across a wide bus. Width returns through lane aggregation — ×1, ×4, ×8, ×16 — with the receiver realigning lanes in logic, making lane-to-lane skew a per-link budget rather than a limit on how fast one wire may run.

Switched, not arbitrated onto one medium. Because links are point-to-point, devices are joined through a switch. A root complex — the host's gateway into the fabric — reaches endpoints directly or through switches that route traffic between links. Transfers to different targets traverse different links simultaneously, so nothing takes turns on a shared medium; contention moves to switch ports, buffers, and shared upstream links, where it can be engineered.

A fourth idea falls out of switching: packets. Once traffic is routed between links instead of asserted onto one shared wire, each transaction must carry its own routing information — so it is wrapped as a packet and forwarded much as a network forwards frames. Packet formats belong to later modules; what matters now is why the model is packet-and-switch at all.

5. A Compact Mental Model

One bridge, then back to the engineering. A shared bus is a shared physical resource — one road, where each additional vehicle both takes capacity from the others and wears the surface everyone drives on. A switched fabric is dedicated links joined by routing — each device has its own connection into a junction that directs traffic between connections. The analogy earns its place only by fixing the direction of scaling: a shared medium scales against itself, a fabric of dedicated links scales with itself.

6. The Counterintuitive Part — How Serial Replaces Parallel

Newcomers stumble here: how can sending one bit at a time be better than sending 32 at once? Serial is not magically faster — the two designs face different limits. Bandwidth is roughly signalling rate × width, and a wide synchronous parallel bus maximises width but pays for it in rate, because it demands two hard things at once: many bits valid in one shared sampling window, on a medium several devices tap. A serial link gives up width per wire and in exchange removes both demands, so the per-wire rate can rise by a large factor. Lane aggregation buys width back, and because lanes are realigned in receive logic rather than latched together off one bus clock edge, width and rate stop competing directly.

Whether one lane out-runs a particular historical parallel bus depends on which generations you compare, and that comparison is not the lesson. The structural point is: point-to-point serial raises the ceiling on rate per wire, and lane aggregation makes width a design choice instead of a physical liability.

7. Common Misconceptions

8. Real Engineering Relevance

This is not history for its own sake — it shapes what you design, verify, and debug.

• Design. Endpoint and root-complex RTL is organised around links and packets, not a shared bus interface — and everything the physical layer does exists because the model is serial and point-to-point.
• Verification. A PCIe testbench thinks in transactions and packets over links, and must model link training and flow control, which have no counterpart on a shared parallel bus. Knowing why the fabric is switched tells you which properties are worth checking.
• Debug. When a link underperforms, the questions are fabric questions: did it train to the expected width and speed, is a switch port or upstream link congested, is flow control throttling the sender?

9. Understanding Check

10. Summary

A system interconnect addresses devices and moves data between them and the host; the hard part is sharing one host among many. The parallel-bus era — ISA, PCI, and the AGP exception — answered sharing with one shared, parallel, multi-drop bus, cheap and correct for its time. It ran out for reasons partly electrical (loading on a multi-drop medium; one sampling window across many wires) and partly architectural (one bandwidth pool divided by arbitration; width paid for in pins and matched traces), and clocking it faster or widening it pushed against those same constraints.

PCIe changed the model rather than the numbers: packetized communication over point-to-point serial links connected as a switched fabric. Point-to-point removes the multi-drop medium, embedded timing removes the shared bus-clock model, lane aggregation restores width as a design choice, and switching removes arbitration for a shared medium from the link topology. Hold that shift and everything ahead reads as machinery for a model you already understand.

11. What Comes Next

You now have the map. The next chapters walk it in detail: ISA, PCI, and AGP as the interconnects themselves; PCI limitations, shared bus problems, and scalability challenges as the forces that ended them; and the emergence of PCIe as the answer. Module 2 onward builds the fabric — root complex, endpoints, switches and bridges, then the layered stack, then enumeration.

Browse the full path on the PCIe tutorials index. For the on-chip counterpart to this story — the same "the shared bus ran out, so decouple and scale" reasoning applied inside a chip — see Why AXI Exists.