Switches in Topology — Ports, Forwarding, and Convergence

Chapter 2.4 established what a Switch is — an intermediate forwarding element with one upstream connection and several downstream ones, forwarding between Links rather than granting ownership of a shared medium. Chapter 4.2 treated it as a fixed feature of an Endpoint's path.

This chapter opens it:

How does a PCIe Switch organise multiple Links into a working topology, and where do forwarding, convergence, congestion, and verification complexity actually appear?

1. Port Is Not Link

A terminology distinction first, because the two words are routinely swapped and the swap causes real confusion.

A port is logic and interface on a component. A Link is the connection joining two ports on two components.

A Switch with five ports participates in up to five Links, each with a different neighbour. Every one of those Links is a separate connection with its own capability, its own signalling conditions, its own operational state, and its own hop-local delivery relationship (Chapter 3.2).

The practical consequence: a port can be perfectly healthy while its Link is not, and vice versa. "Port 3 is down" and "the Link on port 3 is down" describe different things and lead to different investigations — one at the component, one at the connection between two components.

2. Orientation: Upstream and Downstream Facing

Chapter 2.4 defined upstream as toward the Root Complex and downstream as away from it. In a real Switch that orientation is a property of each port:

• One port faces upstream. Its Link leads toward the root — to a Root Port, or to another Switch closer to the root.
• The others face downstream. Their Links lead away from the root, to Endpoints or to further Switches.

3. The Datapath: Ingress, Decision, Egress

The datapath has three logical stages, and each corresponds to a distinct engineering problem.

Ingress. A transaction arrives on some port. It must be received according to that port's Link relationship — integrity checked, delivery resolved with that specific neighbour — before it can be considered for forwarding at all.

Route decision. Something determines which port the transaction should leave by. PCIe defines normative rules for this based on information the transaction carries, and those rules are Module 11. What matters here is that a decision exists, it is per transaction, and its output is a port selection.

Egress arbitration. Several ingresses may select the same egress. Something must decide the order. This is where a Switch's internal contention lives, and §5 develops it.

4. Fanout Is Not Bandwidth

The most consequential structural fact about a Switch, and the one most often assumed away.

Adding downstream ports adds connectivity. It does not add capacity to the upstream Link. Every transaction from every device below the Switch that is heading toward the root traverses that one upstream connection.

So a Switch with eight downstream ports and one upstream port does not multiply anything. It concentrates: eight sources of demand, one segment to carry their combined toward-root traffic.

That yields two structural claims worth holding precisely:

• Adding a device below a Switch adds a claimant to the upstream segment, not capacity there.
• The upstream segment's capability bounds aggregate toward-root throughput for the entire subtree, regardless of how capable the individual downstream Links are.

Chapter 4.4 develops what happens when this concentration is stacked. The single-level case is already enough to explain most surprising measurements.

5. Where Traffic Actually Converges

Two distinct convergence points exist inside a Switch, and distinguishing them matters because they produce different symptoms.

The upstream-facing egress. Everything below the Switch heading toward the root selects this port. With several downstream devices active simultaneously, their traffic meets here. This is the dominant convergence point in most topologies, and it is the one Chapter 2.7 identified from the fabric side.

Any downstream egress receiving from several sources. Less obvious but real: traffic from the upstream port and from other downstream ports can both target the same downstream device. That egress must then arbitrate between them.

Beyond those, shared internal resources — buffering, datapath structures — can couple traffic that does not share an egress at all. Whether and how much this occurs is implementation-dependent, and it is why a Switch's behaviour under mixed load cannot be fully predicted from its port count and Link capabilities alone.

6. Egress Arbitration in RTL

Egress contention is a genuine hardware problem with a general solution, which makes it the right place for substantive synthesizable RTL.

// Illustrative synthesizable RTL — generic round-robin egress arbitration.
// NOT a PCIe switch. Models N ingress sources contending for one egress,
// with grant held stable across backpressure and rotation only on transfer.
module egress_rr_arbiter #(
  parameter int unsigned N = 4,
  localparam int unsigned IDX_W = (N > 1) ? $clog2(N) : 1
) (
  input  logic             clk,
  input  logic             rst_n,
 
  // Ingress requests: one bit per source, held until granted and transferred
  input  logic [N-1:0]     req,
 
  // Egress side
  output logic             eg_valid,
  input  logic             eg_ready,
  output logic [IDX_W-1:0] eg_select,   // which ingress is currently selected
 
  // Per-ingress grant — exactly one bit set while a selection is active
  output logic [N-1:0]     grant
);
 
  logic [IDX_W-1:0] ptr_q;      // rotation pointer: search starts here
  logic [IDX_W-1:0] sel_q;      // currently selected ingress
  logic             locked_q;   // a selection is active and must be held
 
  // Combinational round-robin search: first requester at or after ptr_q.
  logic [IDX_W-1:0] sel_next;
  logic             found_next;
 
  always_comb begin
    sel_next   = '0;
    found_next = 1'b0;
    for (int unsigned i = 0; i < N; i++) begin
      // Wrap the search so it begins at ptr_q and sweeps all N sources once.
      automatic int unsigned idx = (i + ptr_q) % N;
      if (!found_next && req[idx]) begin
        sel_next   = idx[IDX_W-1:0];
        found_next = 1'b1;
      end
    end
  end
 
  // While locked, the existing selection is presented. Otherwise a new one is
  // chosen combinationally, so a request can be served the cycle it appears.
  wire [IDX_W-1:0] sel_active = locked_q ? sel_q : sel_next;
  wire             have_sel   = locked_q ? 1'b1  : found_next;
 
  assign eg_valid  = have_sel;
  assign eg_select = sel_active;
 
  always_comb begin
    grant = '0;
    if (have_sel) grant[sel_active] = 1'b1;
  end
 
  wire transfer = eg_valid && eg_ready;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      ptr_q    <= '0;
      sel_q    <= '0;
      locked_q <= 1'b0;
    end else begin
      if (transfer) begin
        // Transfer complete: release the lock and rotate past the served
        // source so the next search starts elsewhere. This is what makes the
        // policy fair rather than fixed-priority.
        locked_q <= 1'b0;
        ptr_q    <= (sel_active == (N-1)) ? '0 : (sel_active + 1'b1);
      end else if (have_sel && !locked_q) begin
        // Selected but the egress is not ready: latch the choice so it cannot
        // change while the payload it refers to is still waiting.
        locked_q <= 1'b1;
        sel_q    <= sel_active;
      end
    end
  end
endmodule

Classification: synthesizable.

What it models: several ingress sources contending for one egress, with the two properties any correct arbiter needs — a stable grant while the output is backpressured, and rotation only on successful transfer so no source is starved.

Deliberately simplified: one transfer per cycle with no payload path; requests assumed held until granted; no priority classes or quality-of-service differentiation; no separation of traffic types, which a real switch requires because different transaction classes may not be allowed to block each other.

What to notice — the two subtle points:

1. The grant locks when the egress is not ready. Without locked_q, a change in req while backpressured would move the selection — and the payload presented to the egress would change mid-offer. That is exactly the metadata-stability corruption Chapter 3.1 described: a well-formed transaction assembled from two different sources' data.
2. The pointer advances only on transfer, not on selection. Rotating on selection would let a source that is selected but never served be skipped, which reintroduces starvation through the mechanism intended to prevent it.

Production RTL would additionally require: separation of traffic classes so one class cannot block another, integration with whatever flow-control provisions govern the egress, per-class buffering, and the ordering rules a real design must respect.

7. Arbitration Assertions

// SVA over the illustrative arbiter. Implementation invariants for THIS
// design — not PCIe protocol requirements.
 
// SAFETY — P1: at most one grant. Two simultaneous grants would let two
// sources drive one egress, corrupting whatever is transferred.
property p_onehot_grant;
  @(posedge clk) disable iff (!rst_n)
  $onehot0(grant);
endproperty
a_onehot_grant : assert property (p_onehot_grant);
 
// SAFETY — P2: a grant implies a request. Granting an idle source would
// present an egress transfer with no corresponding payload.
property p_grant_implies_req;
  @(posedge clk) disable iff (!rst_n)
  |grant |-> |(grant & req);
endproperty
a_grant_implies_req : assert property (p_grant_implies_req);
 
// SAFETY — P3: the selection is stable while the egress is stalled. This is
// the property that prevents payload corruption under backpressure.
property p_select_stable_when_stalled;
  @(posedge clk) disable iff (!rst_n)
  (eg_valid && !eg_ready) |=> $stable(eg_select);
endproperty
a_select_stable : assert property (p_select_stable_when_stalled);
 
// SAFETY — P4: no transfer without both sides agreeing.
property p_transfer_handshake;
  @(posedge clk) disable iff (!rst_n)
  (eg_valid && eg_ready) |-> |grant;
endproperty
a_transfer_handshake : assert property (p_transfer_handshake);
 
// LIVENESS — P5: a persistent request is eventually served.
// REQUIRES the environment assumption below; without it the property is
// unprovable, because an egress that never asserts ready legitimately
// starves everyone and that is not an arbiter bug.
//   assume property (@(posedge clk) s_eventually eg_ready);
property p_no_starvation;
  @(posedge clk) disable iff (!rst_n)
  req[0] |-> s_eventually grant[0];
endproperty
// a_no_starvation : assert property (p_no_starvation);

P1 and P2 are the structural safety net. Simulation with a single active requester never exercises either — they need concurrent contention, which a directed single-stream test does not produce.

P3 is the one that catches the most damaging bug. A selection changing under backpressure produces a transfer whose payload came from a different source than the grant indicated. Nothing downstream can detect this: the transaction is well-formed and will be delivered faithfully to whatever it now claims to be for. Simulation misses it because it requires concurrent requests and a stalled egress at the same moment.

P5 is shown commented and with its assumption stated, deliberately. Starvation is a liveness property, and liveness without an environment assumption is unprovable — an egress that never accepts starves every source, and no arbiter can fix that. Asserting the strong form without the assumption produces failures on ordinary backpressure and trains engineers to ignore the suite.

8. Traffic Scenarios and What They Predict

Reasoning about a Switch means reasoning about which port pairs are in play.

Two downstream siblings both sending toward the root. Both select the upstream egress. They contend there. Their own Links are uncontended; the shared upstream segment carries the sum, and if that sum exceeds its capability, both are limited.

The host sending to two different downstream devices. Both transactions arrive at the upstream-facing ingress and select different egresses. They can proceed concurrently — subject to the ingress being able to deliver them at rate and to the switch's internal resources, which is where implementation dependence enters.

Two sources targeting the same downstream device. For instance the host and another downstream device both accessing one Endpoint. They select the same egress and contend there. This case surprises people because the sources are unrelated and neither is at fault.

One heavy device alongside light ones. The heavy device can consume a large share of the shared upstream segment. Whether it also affects traffic that does not share that segment depends on the switch's internal resources — which is implementation-dependent and not derivable from the topology diagram alone.

9. Verifying a Switch

The organising principle: a Switch's responsibility is forwarding, so correctness at one port proves almost nothing. The interesting behaviour appears when several ports are active at once.

Port-level. Each Link-facing interface exercised independently. Establishes basic function and nothing about forwarding.

Forwarding correctness. For every ingress, traffic destined for every reachable egress. This is the core check and requires a model of expected routing:

// Verification-only. NOT synthesizable. NOT PCIe routing state — the
// expected egress is computed by the testbench's topology model.
typedef struct {
  int              ingress_port;
  int              expected_egress_port;
  longint unsigned corr_id;        // testbench correlation handle
  time             t_ingress;
} route_expect_t;

The method: observe at ingress, compute the expected egress from the testbench's own topology model, then confirm the transaction appears at that egress and at no other. The second half matters — a switch that forwards correctly and also leaks a copy elsewhere passes a naive check.

Contention. Several ingresses targeting one egress simultaneously. Verify all are eventually served, none is starved over a long run, and — critically — that payloads are not mixed under backpressure. This is where P3 earns its place.

Concurrent non-conflicting traffic. Different ingresses to different egresses. Verify they proceed independently, and measure how well; a large shortfall against expectation indicates internal sharing worth understanding.

Backpressure. Block one egress while others remain active. Verify unaffected paths continue, the blocked path resumes cleanly when released, and nothing is lost or duplicated across the stall.

Failure isolation. Take down one downstream Link. Verify unrelated paths are unaffected and the switch handles the unavailable port sensibly — no traffic accepted for it, no traffic emitted to it, no corruption of other paths.

Stress. Sustained multi-port traffic with queue occupancy driven to extremes, including all ingresses targeting the upstream egress simultaneously — the worst case for the most common convergence point.

Coverage worth defining: every ingress-egress pair exercised; concurrent contention at each egress with varying numbers of contenders; occupancy extremes per port; backpressure at each egress; and each port disabled in turn.

10. Debugging a Switch

Eight signatures with likely ownership.

Correct ingress, wrong egress. The route decision. The transaction was received fine and sent to the wrong place — check the decision logic and the testbench's expected-egress model, since a mismatch may be either.

One egress permanently blocked. Either the Link on that port is down, or the neighbour is not accepting, or the egress logic is stuck. Check the port's Link state first: an unavailable Link explains it entirely and is not a switch bug.

All downstream devices stall together. Implicates the shared upstream path or the switch itself. Not a per-device problem — investigating any individual Endpoint wastes time. Check whether the upstream Link is healthy and whether toward-root traffic is being forwarded at all.

One downstream branch missing. That port or its Link. The other branches' health is the evidence that rules out everything shared.

Duplicate forwarding. A transaction appearing at more than one egress, or twice at one. Check the route decision for producing multiple selections, and check hop-local retry bookkeeping (Chapter 3.2) for resending something already delivered.

Accepted but never emitted. The transaction entered and did not leave. Follow it through the stages: was a route selected, was it granted at the egress, was the egress ready? A permanently-unready egress explains it; so does a route decision producing no valid selection.

Throughput collapses under multi-port traffic but single-stream is fine. Almost the definition of a convergence or arbitration problem. Single-stream tests do not exercise contention at all, which is why this class of bug survives to system integration.

Works with one port active, fails with several. Same root cause family: arbitration, shared internal resources, or a stability bug that only manifests under simultaneous requests and backpressure. This is the signature P1–P3 exist to catch.

11. Common Misconceptions

12. Understanding Check

13. Summary

A Switch terminates several independent Links and forwards between them. A port is component logic; a Link is the connection between two ports — and either can fail without the other.

Orientation is hierarchical, not directional: one port faces upstream toward the root, the rest downstream, and every one of them both transmits and receives.

The datapath is ingress → route decision → egress arbitration, an illustrative decomposition of the work rather than a mandated structure. Convergence occurs at the upstream-facing egress — the dominant case, where everything toward the root meets — and at any egress receiving from several sources, with shared internal resources potentially coupling more.

Fanout is not bandwidth. More downstream ports concentrate demand onto one upstream segment rather than multiplying capacity, so adding a device adds a claimant rather than capability.

For implementation, egress contention needs an arbiter whose grant is stable under backpressure and which rotates only on transfer — the first prevents payload mixing, the second prevents starvation. Neither policy nor fairness is conferred by the topology; both are design properties.

For verification, the governing fact is that correctness at one port proves almost nothing. Every bug class unique to a Switch requires several ports active simultaneously, which is why a design can pass every single-stream test and collapse in a real system.

Hold the model: each port has its own Link relationship, while shared egress and upstream resources create convergence points inside the topology.

14. What Comes Next

This chapter examined one Switch. Chapter 4.4 — Multi-Level Fabrics cascades them: what changes when a downstream port leads to another Switch, how convergence nests, how failure domains layer, and how to localise a fault from nothing more than the set of devices affected.

Chapter 4.5 then applies all of it to real system topologies.

Revisit Endpoints in Topology for the leaves this component fans out to, or Switch for the architectural role. Browse the full path on the PCIe tutorials index.