"PCIe Is Just a Faster PCI" — What Actually Changed

The myth is almost never stated out loud. It survives as an assumption — in a bandwidth estimate that divides a total by a device count, in a topology diagram drawn as a line with devices hanging off it, in a sentence like "we'll put both cards on the same bus."

PCIe has no bus. The name is inherited and the architecture is not, and the difference is measurable.

1. Where the Myth Comes From

It is not stupid. It is a reasonable inference from three true facts.

The name is continuous. PCI Express was designed as a software-compatible successor, and a great deal of effort went into making existing software work. Configuration space, enumeration and the programming model were deliberately preserved — so from a driver's point of view, much of it genuinely does look like PCI with bigger numbers.

The marketing was about speed. Each generation is announced with a rate. Nobody announces a topology change, so the visible difference between generations is a number, and the visible difference between PCI and PCIe reads as a bigger version of the same number.

And the mental picture is sticky. A bus is easy to draw: one line, several devices. A switched fabric is a tree, and the tree only matters when you ask a question that a line answers wrongly — which is exactly the class of question §4 covers.

So the myth persists because it is harmless until it is expensive. Everything about a single device on a single link behaves as the myth predicts. The error appears the moment there is more than one device, or more than one flow.

2. What a Shared Bus Actually Is

A bus is one medium that many devices take turns using.

        ┌─────┬─────┬─────┬─────┐
device →│     │     │     │     │← device
        └─────┴─────┴─────┴─────┘
              one shared medium
   arbitration decides who transmits this cycle

Three properties follow, and all three are the myth's blind spots:

Total capacity is fixed. The medium carries what it carries. Adding devices does not add capacity; it adds contenders.

Every transfer excludes every other. Two devices cannot transmit simultaneously even if they want to talk to different partners — there is one medium and it is either carrying A's traffic or B's.

And arbitration is a shared cost. Someone must decide who transmits, and that decision is on the critical path of every transfer.

§10 measured all three. The shared total flattens at 200,000 from four devices onward — it saturated at two — and per-device throughput falls as 1/N.

3. What a Switched Fabric Is

Every link is point-to-point and belongs to exactly two endpoints.

              Root Complex
                   │
              ┌────┴────┐
           Switch     Switch
           ┌──┴──┐    ┌──┴──┐
         EP    EP    EP    EP
    every edge is a private link

The three properties invert:

Capacity scales with links. Adding a device adds its link. §10 measured the switched total rising linearly — 8× at 16 devices.

Independent flows do not contend. A→B and C→D share no link and run simultaneously. §10 measured zero contended cycles on the fabric against 35,959 on the bus.

And arbitration is local. A switch arbitrates only where flows actually converge, which is a smaller and more predictable problem than arbitrating one global medium.

What the myth gets right, and it is worth saying: the programming model really is largely preserved. Configuration space, enumeration, BARs — a driver author's mental model transfers substantially. The myth is wrong about the fabric, not about the software interface, and that is precisely why it survives: the part that is visible from software is the part that did not change.

4. The Decisions the Myth Corrupts

Four, and each one is a real engineering mistake with a measurable cost.

Bandwidth estimation by division. "The link is 16 GB/s and we have four devices, so 4 GB/s each." That is bus arithmetic. On a fabric, four endpoints under a root port each have their own link; what they share is whatever converges upstream, which may be a great deal less than four-way division or a great deal more. The right question is where the flows converge, and the topology answers it.

Assuming devices interfere. A design that serialises two unrelated transfers because "the bus is busy" has invented a constraint. §10 measured independent flows running concurrently with zero contention.

Assuming devices don't interfere. The opposite error, and equally common. Flows that converge on a shared upstream link absolutely do contend — that is 21.3's subject, and a fabric is not magic.

And drawing the topology as a line. A diagram with devices hanging off a horizontal bar cannot express where contention happens, so it cannot be used to answer a bandwidth question. The tree can, and this is why every system chapter in Module 26 draws one.

5. What Replaced the Bus's Machinery

A bus provides several things implicitly. A fabric has to provide each one explicitly, and this is the honest accounting of what PCIe added.

Every row is a mechanism that exists because the medium stopped being shared. 28.1 §2 makes the same observation about AXI: much of the PCIe stack exists to reconstruct guarantees that a shared on-chip medium provides for free.

The useful reframing for a beginner is that PCIe did not add speed and keep everything else. It removed the shared medium and rebuilt, as explicit protocol, everything the medium used to do implicitly.

6. The Topology, Drawn

Two readings.

In the upper arrangement every device touches the same box. That box is the capacity, and it does not grow.

In the lower arrangement the only shared edges are the two upstream links. A→B traffic within one switch never touches the other switch's link. Contention exists exactly where the drawing shows convergence — which is why the tree is the diagram that can answer a bandwidth question and the line is not.

7. RTL — Why Arbitration Moved

Block 1 — a shared-medium arbiter. One grant per cycle, globally, for everyone.

// A shared medium has ONE grant per cycle no matter how many devices want
// to transmit and no matter whether their destinations are related.
// This exclusivity IS the bus.
module shared_medium_arbiter #(
  parameter int unsigned NDEV = 8
)(
  input  logic clk,
  input  logic rst_n,
  input  logic [NDEV-1:0]              req,
  input  logic [$clog2(NDEV)-1:0]      dest [NDEV],   // deliberately UNUSED
  output logic [NDEV-1:0]              grant,
  output logic                         medium_busy,
  output logic [31:0]                  contended_cycles
);
 
  logic [$clog2(NDEV)-1:0] rr, sel;
  logic found;
 
  always_comb begin
    sel = rr; found = 1'b0;
    for (int k = 0; k < NDEV; k++) begin
      automatic logic [$clog2(NDEV)-1:0] i = ($clog2(NDEV))'((rr + k) % NDEV);
      if (!found && req[i]) begin sel = i; found = 1'b1; end
    end
    // EXACTLY ONE grant. Note that `dest` is not consulted: two devices with
    // completely unrelated destinations still exclude each other, because
    // there is one medium. That is the property §10 measured as 35,959
    // contended cycles between flows that shared no endpoint.
    grant       = '0;
    if (found) grant[sel] = 1'b1;
    medium_busy = found;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      rr <= '0; contended_cycles <= '0;
    end else begin
      if (found) rr <= (sel == ($clog2(NDEV))'(NDEV-1)) ? '0 : sel + 1'b1;
      // Every requester that did not win this cycle was blocked by a
      // transfer it has no relationship to.
      if ($countones(req) > 1 && contended_cycles != 32'hFFFF_FFFF)
        contended_cycles <= contended_cycles + 32'($countones(req) - 1);
    end
  end
 
endmodule

Block 2 — per-port forwarding. Arbitration only where flows converge.

// A switch port arbitrates only among the flows that actually want THIS
// egress. Two flows heading for different egress ports do not interact at
// all, which is the structural source of §10's linear scaling.
module switch_egress_arbiter #(
  parameter int unsigned NIN = 4
)(
  input  logic clk,
  input  logic rst_n,
  input  logic [NIN-1:0]           in_valid,
  input  logic [NIN-1:0]           in_for_this_egress,  // routing decided upstream
  input  logic                     egress_ready,        // credit-gated (16.1)
  output logic [NIN-1:0]           in_grant,
  output logic                     egress_valid,
  output logic [31:0]              local_contention
);
 
  logic [NIN-1:0] eligible;
  logic [$clog2(NIN)-1:0] rr, sel;
  logic found;
 
  always_comb begin
    // Only requests routed to THIS egress are candidates. A request for a
    // different port is invisible here — it is being arbitrated elsewhere,
    // concurrently. Consulting in_valid alone instead of in_for_this_egress
    // would recreate the bus (mutation 2).
    eligible = in_valid & in_for_this_egress;
    sel = rr; found = 1'b0;
    for (int k = 0; k < NIN; k++) begin
      automatic logic [$clog2(NIN)-1:0] i = ($clog2(NIN))'((rr + k) % NIN);
      if (!found && eligible[i]) begin sel = i; found = 1'b1; end
    end
    // Transfer requires the egress to be ready — a credit question, not an
    // arbitration one. Granting without it advances state on `valid` alone.
    egress_valid = found;
    in_grant     = '0;
    if (found && egress_ready) in_grant[sel] = 1'b1;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      rr <= '0; local_contention <= '0;
    end else begin
      if (found && egress_ready)
        rr <= (sel == ($clog2(NIN))'(NIN-1)) ? '0 : sel + 1'b1;
      // Contention is counted only among flows that genuinely converge.
      if ($countones(eligible) > 1 && local_contention != 32'hFFFF_FFFF)
        local_contention <= local_contention + 32'($countones(eligible) - 1);
    end
  end
 
endmodule

The difference between the two modules is one signal. Block 2 masks its request vector with in_for_this_egress; Block 1 has a dest input and ignores it. That single mask is the architectural change the myth misses — and a switch whose egress arbiter forgets it has rebuilt a bus inside a fabric.

8. Same-Cycle Audit

9. Invariants

// SHARED MEDIUM — the invariant is exclusivity, and it is the definition.
 
// P1 — at most one grant per cycle, regardless of destinations.
// This property IS the bus. A design that violates it is no longer a bus.
property p1_bus_single_grant;
  @(posedge clk) disable iff (!rst_n) $onehot0(grant);
endproperty
 
// P2 — a grant goes only to a requester.
property p2_bus_grant_implies_req;
  @(posedge clk) disable iff (!rst_n) (grant != 0) |-> ((grant & req) == grant);
endproperty
 
// SWITCHED FABRIC — the invariant is independence.
 
// P3 — an egress arbiter grants only requests routed to it. Violating this
// recreates the bus inside the fabric: flows that share no link begin to
// exclude one another (mutation 2).
property p3_grant_only_own_egress;
  @(posedge clk) disable iff (!rst_n)
    (in_grant != 0) |-> ((in_grant & in_for_this_egress) == in_grant);
endproperty
 
// P4 — a transfer requires the egress to be ready. Granting on validity
// alone advances state on `valid`, which is the error 30.5 §11 P1 measures
// as overstated throughput precisely when the sink is stalling.
property p4_grant_requires_ready;
  @(posedge clk) disable iff (!rst_n) (in_grant != 0) |-> egress_ready;
endproperty
 
// P5 — contention is counted only among converging flows. If this counter
// rises for non-converging traffic, the routing mask is wrong and the
// fabric is behaving as a bus.
property p5_contention_only_on_convergence;
  @(posedge clk) disable iff (!rst_n)
    ($countones(eligible) <= 1) |=> $stable(local_contention);
endproperty
 
// P6 — no starvation: a persistent eligible request is eventually granted.
// Assumption: the egress becomes ready infinitely often, which is a credit
// property (16.6) and not an arbitration one.
property p6_no_starvation;
  @(posedge clk) disable iff (!rst_n)
    (in_valid[i] && in_for_this_egress[i]) |-> s_eventually in_grant[i];
endproperty

P3 is the one worth naming in a review. It is the difference between the two architectures expressed as a single assertion, and a switch that fails it has quietly reintroduced the constraint PCIe exists to remove.

10. Measured — Scaling and Independence

Independence, measured separately — two flows, A→B and C→D, sharing no endpoint:

Four readings.

The shared medium saturates at two devices and never delivers more. From four devices onward the total is exactly 200,000 — one transfer per step, which is the medium's capacity. Adding devices adds contenders, not capacity.

Per-device throughput falls as 1/N. 100,004 → 99,687 → 50,000 → 24,999 → 12,499. This is the arithmetic behind the myth's most expensive symptom, §4's bandwidth-estimation-by-division — except that on a fabric it does not apply.

The independence measurement is the mechanism. C→D got 23,927 on the bus and 59,886 on the fabric, from identical offered load. The flow was not slow; it was excluded by a flow it has no relationship to.

And the asymmetry in the bus row is worth noticing. A→B got 60,023 and C→D got 23,927 — the arbitration was not even. A shared medium's fairness is an arbiter property, and an unfair arbiter turns a capacity problem into a starvation problem, which is a second failure mode the fabric simply does not have.

11. Debugging — When the Myth Produces the Bug

12. Misconceptions

"PCIe is a bus." Why it sounds plausible: the software model, the configuration space and the name are all inherited, and buses are how peripheral interconnects worked for two decades. What really happens: every link is point-to-point (§3). §10 measured the structural consequence — linear scaling versus a fixed total. What it causes: bandwidth estimates by division (§4), and topology diagrams that cannot express where contention occurs.

"A faster bus would have been equivalent." Why it sounds plausible: if the problem is capacity, more capacity should solve it. What really happens: §10 measured two unrelated flows serialising on a bus and running concurrently on a fabric. A faster medium serialises the same transfers more quickly; it does not stop them excluding each other (§8 audit A). What it causes: the belief that generation upgrades and topology changes are interchangeable, which prices both wrongly.

"Devices on the same switch contend with each other." Why it sounds plausible: they are visibly attached to the same box. What really happens: they contend only where their flows converge. Two endpoints on one switch talking to different destinations may share no link at all. What it causes: designs that serialise unrelated traffic defensively, giving away the concurrency the fabric provides.

"Devices on different switches never contend." Why it sounds plausible: separate switches sound like separate resources. What really happens: they converge upstream. 21.3 owns the analysis, and the upstream link is frequently the actual limit. What it causes: the opposite error to the previous one — a design that assumes independence it does not have, discovered under load.

"PCIe just made everything faster." Why it sounds plausible: every generation is announced with a rate, and rates did increase. What really happens: removing the shared medium required rebuilding, as explicit protocol, everything the medium provided implicitly (§5) — routing, packet identity, credit flow control, and a replay layer. What it causes: treating credits, headers and the Data Link Layer as overhead to be minimised rather than as the mechanisms that make a fabric possible.

"The programming model changed too." Why it sounds plausible: if the architecture changed this much, surely software did. What really happens: the software-visible model was deliberately preserved — configuration space, enumeration, BARs (§3). This is the part of the myth that is true, and it is exactly why the rest of it survives. What it causes: the reverse error — engineers who learn the fabric and then over-correct, expecting the driver interface to be unfamiliar when much of it is not.

13. Understanding Check

Q1. A colleague estimates that four endpoints under one root port will each get a quarter of the link bandwidth. When is that right, when is it wrong, and what should they have asked?

It is right only if all four flows converge on that one link, and wrong otherwise (§4, §10). If the four endpoints are talking to each other through a switch, some flows may never touch the root port's link at all — §10 measured independent flows achieving 119,909 transfers against a bus's 83,950 from identical offered load. If all four are DMA-ing to host memory, they do converge and division is roughly right. The question they should have asked is where the flows converge, which is a topology question, not a bandwidth one — and it is the question a line-shaped diagram cannot answer.

Q2. §10 shows a shared medium delivering the same total at 4, 8 and 16 devices. Explain the mechanism, and what per-device throughput does.

The medium carries one transfer per step, so the total is the medium's capacity and adding devices adds only contenders (§2). At two devices offering 0.5 each the medium is already saturated; beyond that the total is pinned at 200,000. Per-device throughput falls as 1/N — 100,004, 99,687, 50,000, 24,999, 12,499 across the table. The switched fabric's total rises linearly instead, because a new device brings its own link rather than a claim on a shared one.

Q3. Two flows share no endpoint and interfere anyway. Give three candidate causes in the order you would check them.

Convergence, then a non-link shared resource, then an arbitration bug (§11 cases 2 and 4). First trace both paths and find the first shared link — convergence is by far the most likely and it is a topology read (21.1). If they genuinely share no link, the shared resource is not in the fabric: Tags, completion buffer space, or the host memory path (26.1 §4). Only then suspect an egress arbiter that ignores its routing mask — P3's violation, which recreates a bus inside the fabric and is the one cause of the three that is an RTL bug.

Q4. What single signal distinguishes §7's fabric arbiter from its bus arbiter, and what does removing it do?

The in_for_this_egress mask (§7, P3). Block 2 masks its request vector with it; Block 1 has a destination input and ignores it entirely. Removing the mask makes every request compete with every other regardless of destination — which is the definition of a shared medium, rebuilt inside a switch. §10 measured that difference as 35,959 contended cycles versus zero, between flows that share no endpoint. It is one signal and it is the entire architectural change.

Q5. Why does the myth survive despite being wrong, and which part of it is actually true?

Because the software-visible model was deliberately preserved (§1, §3). Configuration space, enumeration and BARs transfer substantially from PCI, so from a driver author's position much of it genuinely does look like PCI with bigger numbers — that part of the myth is true. What changed is beneath software: the shared medium was removed, and routing, packet identity, credit flow control and a replay layer were added to reconstruct what it used to provide implicitly (§5). The myth is therefore harmless for anyone who only touches the programming model and expensive for anyone who reasons about topology or bandwidth — which is why it survives so long before it costs anything.

14. What Comes Next

This chapter corrected a belief about the fabric. The next corrects one about what travels on it.

The module's method is the same each time, and it is worth naming once here: take a belief that sounds reasonable, find where it comes from, measure the consequence, and name the decision it corrupts. A myth that produces no wrong decision is not worth a chapter; every one in this module produces a specific, reproducible mistake.

31.2 takes the belief that PCIe carries memory reads and writes and nothing else — a myth that is self-refuting in one step, because a device that only handled memory transactions could never have received the Configuration writes that gave it its BARs.