Skip to content

PCIe · Module 11

Routing Information — How a TLP Finds Its Destination

PCIe answers "where does this packet go" three different ways depending on what kind of packet it is. Why address routing and ID routing cannot share one lookup, why a Completion travels home by identity rather than by address, and how a Switch port decides in one cycle which way to forward.

Chapter 11.3 decoded the header and Chapter 11.4 moved the data. Neither said where the packet was going.

A packet on a link between two components is not yet at its destination. Something has to look at it and decide which way it leaves — and the answer to "which field do I look at" is not the same for every packet.

How does a TLP carry destination information, and why do different transaction families use fundamentally different routing mechanisms?

1. The Question Every Packet Must Answer

Consider what a Switch port actually has to do. A TLP arrives. In one decision, before it can forward anything, it must determine which of its ports the packet leaves by — or that the packet is for the Switch itself, or that the packet has no legal destination from here.

It has the header. That is all it has. There is no routing table distributed by a protocol, no path stamped into the packet by the sender, no destination port number.

So the destination has to be derived from the header — and the derivation depends on what the packet is, because a memory write and a Completion do not describe their destinations in remotely the same terms.

PacketHow it names its destinationWhat the fabric must know to resolve it
a memory writean addresswhich address ranges live behind which port
a configuration accessa Functionwhich bus numbers live behind which port
a Completionthe identity that made the requestthe same bus-number map
certain messagesa relationship — upstream, or the Root Complex, or everywherethe port's own position in the hierarchy

Four rows, three mechanisms. The last column is the reason: the first row needs an address map and the middle two need a bus-number map, and those maps are built from different configuration registers by different means.

2. The Three Mechanisms

3. Why One Lookup Cannot Serve All Three

The instinct — build one routing table, index it with "the destination" — fails on the first Completion, and it is worth seeing exactly why.

4. Address Routing

The packet carries an address; the forwarding element asks which port owns that address.

This is the mechanism behind everything Module 9 built. Chapter 9.5 assigned each Function's BARs a non-overlapping window; Chapter 9.6 had the Function decode accesses that landed in its window. The fabric between them does the same comparison one level up: a Switch port's Base/Limit registers describe the union of everything behind it, and a packet within that union is forwarded down.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
address-routed TLP arrives
   → compare against each downstream port's window
   → forward down the port that contains it
   → or, if no downstream port contains it, forward upstream

Two properties make this work, and both were established by configuration long before any packet moved:

  • The windows are non-overlapping. Chapter 9.5 §4's discipline. If two ports claimed the same address, the comparison would have two answers and the fabric would have no basis for choosing.
  • Each port's window covers everything behind it. A Switch port's window is the union of the windows of every Function in its subtree — which is why the assignment process works bottom-up, sizing children before parents.

The default direction matters. A packet whose address matches no downstream window is forwarded upstream, toward the Root Complex, because the address may belong to system memory or to another branch of the hierarchy. Upstream is the fallback, not an error — which means an address-routing bug frequently presents as "packets going the wrong way but still going somewhere" rather than as a dropped packet.

Whether a Switch forwards a packet from one downstream port directly to another — peer-to-peer — and under what conditions, is a real and important question that this chapter does not answer normatively.

5. ID Routing

The packet carries a Bus/Device/Function identity; the forwarding element asks which port leads to that Bus Number.

Two very different-looking packet families use it.

Configuration Requests target a Function directly. There is no address involved — the packet names a Bus, a Device and a Function, and the fabric delivers it there. This is how enumeration works at all (Chapters 7.47.6): before any BAR is assigned, before any address window exists, configuration accesses can already reach every Function, because they route by identity rather than by address.

Completions travel home by the identity of whoever asked. §7 is that asymmetry in full.

The resolution uses the Bus Number registers. Each Switch port's Type 1 header carries a Secondary Bus Number — the bus immediately below it — and a Subordinate Bus Number — the highest bus number reachable below it. Together they describe a contiguous range, and a packet whose target Bus Number falls in that range is forwarded down that port.

And a qualification that a design cannot skip. The 16-bit identity's internal division into Bus, Device and Function is subject to ARI, which reallocates the Device and Function portions. A routing element extracting a Bus Number must do so consistently with the configuration in force; the ARI mechanism itself is owned by its own chapter and is not published here.

6. Implicit Routing

Some Messages do not name a destination at all. They name a relationship.

"Toward the Root Complex." "To whichever component receives this, and no further." "Down every path from here." None of these is a lookup — the receiving port resolves them from what it already knows about its own position in the hierarchy.

The Message indicatesThe port resolves it by knowing
a destination defined by hierarchy positionwhich of its links faces upstream
that the receiver is the destinationthat it received the packet
a distribution relationshipwhich of its links face downstream

Why the mechanism exists. Some of what a Message conveys is about the fabric rather than about a location in it — and the sender frequently does not know, and should not need to know, the identity or address of the component that should act on it. A device signalling an event toward the host does not know the host's identity. "Upstream" is the only description it can honestly give.

The specific routing indications, their encodings, and which Message types use which are Chapter 11.7's, along with the observation that not every Message uses implicit routing — some are routed by address or by ID like any other packet, which is precisely why §10's classifier treats the Message family as needing a further decision rather than as automatically implicit.

7. The Completion Asymmetry

A Request goes out by address. Its Completion comes home by identity. The two halves of one operation use two different routing mechanisms, and this is the chapter's most important structural fact.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
Requester issues a memory read
   → routed by ADDRESS to the Completer
   → Completer produces a Completion
   → routed by ID back to the Requester

And the same asymmetry explains a Configuration Request's Completion. The Request routed by ID, the Completion routes by ID, and both use the same mechanism — which makes configuration the one family whose two halves are symmetric, and a useful contrast for seeing why memory traffic is not.

8. A Switch Port's Decision

A TLP arrives at a switch port and is first classified into a routing mode. Address-routed packets go to an address window comparator that consults the base and limit registers. Identity-routed packets go to a bus number range comparator that consults the secondary and subordinate bus number registers. Implicitly routed packets go to a resolver that uses the port's own hierarchy position. All three feed a port selector that chooses an egress port, with an upstream fallback for unmatched addresses.Inbound TLPheaderthe only informationavailableRouting modeclassifierwhich mechanismappliesAddress windowcompareBase and LimitregistersBus number rangecompareSecondary andSubordinateHierarchypositionno lookup —relationship onlyEgress portselectdownstream port, orupstream12
Figure 1 — the routing decision inside a Switch. Classification comes first and selects which resolver runs; the two lookups consult different configuration resources and cannot be merged. An address that matches no downstream window falls back upstream, which is why address-routing faults usually misdeliver rather than drop.

The figure's argument is the three parallel resolvers that never merge. They consume different fields, consult different registers, and produce the same kind of answer — an egress port — by entirely different means. The classifier is what makes them exclusive, and it is the only block through which every packet passes.

9. Microarchitecture — Classify, Then Resolve

StageConsumesProducesWhy separate
1 — classifythe header's classification fieldsthe routing modethe mode selects everything downstream, and a wrong mode makes a correct lookup useless
2 — resolvethe field that mode names, plus that mode's configuration registersan egress porteach resolver is small and independently verifiable
3 — selectthe resolvers' resultsone egress port, or an unroutable indicationexclusivity is checkable in one place

Stage 1 is Chapter 11.3's classification, reused rather than repeated. The routing block consumes the normalized descriptor (Chapter 11.3 §12) rather than re-slicing raw header bits — which is the whole reason that descriptor exists.

Illustrative microarchitecture. PCIe defines routing rules, not a pipeline. A real Switch may resolve all ports in parallel, pipeline the comparison, or apply a different structure entirely; what does not vary is that the mode determines which field is authoritative.

10. RTL — Routing Mode Classifier

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SYNTHESIZABLE. Select a routing mode from an already-classified TLP.
// The three routing mechanisms: NORMATIVE. The family-to-mode mapping is a
// REPRESENTATIVE VERIFIED SUBSET; the Message family requires a further
// decision this model does not make (section 6).
package tlp_route_pkg;
 
  typedef enum logic [1:0] {
    ROUTE_ADDRESS  = 2'd0,
    ROUTE_ID       = 2'd1,
    ROUTE_IMPLICIT = 2'd2,
    ROUTE_UNKNOWN  = 2'd3   // not represented by this model
  } route_mode_e;
 
endpackage
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import tlp_hdr_pkg::*;     // Chapter 11.3's family classification
import tlp_route_pkg::*;
 
module tlp_route_classify (
  input  hdr_meta_t   meta,          // Chapter 11.3 section 12's descriptor
  output route_mode_e mode,
  output logic        unresolved     // this model cannot classify it
);
 
  // The mode follows from the FAMILY, not from any single header field.
  // Note that no address, identity or routing sub-field is examined here —
  // this stage decides WHICH field will be authoritative, and reading one
  // before that decision is section 3's bug.
  always_comb begin
    mode       = ROUTE_UNKNOWN;
    unresolved = 1'b1;
 
    if (meta.valid && !meta.unsupported) begin
      unique case (meta.family)
        FAM_MEM_REQ: begin mode = ROUTE_ADDRESS; unresolved = 1'b0; end
        FAM_CFG_REQ: begin mode = ROUTE_ID;      unresolved = 1'b0; end
        FAM_COMPL:   begin mode = ROUTE_ID;      unresolved = 1'b0; end
        // The Message family routes by an indication carried in the packet,
        // which may select implicit, address or ID routing. Resolving it
        // needs the Message taxonomy (Chapter 11.7), so this model reports
        // rather than assumes.
        default:     begin mode = ROUTE_UNKNOWN; unresolved = 1'b1; end
      endcase
    end
  end
 
endmodule

Classification: synthesizable (package: compile-time).

Architecture. Purely combinational on the descriptor. No packet field is read — deliberately, because reading one would mean having already decided which one matters.

State. None.

Contract. The caller supplies Chapter 11.3's descriptor, already classified. This module does not re-derive the family and must not.

Failure. Mapping the Completion family to address routing — the single most damaging error available here, because a Completion has no destination address and the resolver will therefore compare whatever bits occupy that position. §14's third scenario is what that looks like from the outside.

11. RTL — Switch Downstream Port Decision

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SYNTHESIZABLE. One Switch downstream port's forwarding decision.
// The use of Base/Limit windows for address routing and of Secondary /
// Subordinate Bus Numbers for ID routing: NORMATIVE (section 2).
// The port interface, the single-window simplification and the upstream
// fallback signalling: ILLUSTRATIVE.
import tlp_route_pkg::*;
 
module switch_port_route #(
  parameter int ADDR_W = 64
) (
  input  logic clk,
  input  logic rst_n,
 
  // ---- Configuration, written by enumeration software ------------------
  // One memory window. A real Type 1 header carries several windows of
  // different kinds; this models the mechanism, not the full register set.
  input  logic [ADDR_W-1:0] mem_base,
  input  logic [ADDR_W-1:0] mem_limit,      // inclusive
  input  logic              mem_window_en,
 
  // Bus numbers behind this port, inclusive range.
  input  logic [7:0]        secondary_bus,
  input  logic [7:0]        subordinate_bus,
  input  logic              bus_range_en,
 
  // ---- Inbound packet, arriving from upstream --------------------------
  input  logic              in_valid,
  input  route_mode_e       mode,
  input  logic              mode_unresolved,
  input  logic [ADDR_W-1:0] tlp_address,    // meaningful iff ROUTE_ADDRESS
  input  logic [7:0]        tlp_bus,        // meaningful iff ROUTE_ID
 
  // ---- Decision --------------------------------------------------------
  output logic              claim,          // forward down THIS port
  // No downstream port claimed it and this model cannot decide: reported,
  // not silently dropped and not silently forwarded.
  output logic              undecidable
);
 
  generate
    if (ADDR_W < 32) $error("ADDR_W must be at least 32");
  endgenerate
 
  // ADDRESS routing: inclusive range compare against the configured window.
  // Note the window must be ENABLED — an unconfigured window is all-zero,
  // and an unguarded compare would claim address zero.
  wire addr_hit = mem_window_en
               && (tlp_address >= mem_base)
               && (tlp_address <= mem_limit);
 
  // ID routing: inclusive range compare against the bus numbers behind this
  // port. Contiguity is guaranteed by depth-first enumeration (section 5).
  wire bus_hit = bus_range_en
              && (tlp_bus >= secondary_bus)
              && (tlp_bus <= subordinate_bus);
 
  // PREDICATION. Each comparison is consulted ONLY under the mode that makes
  // its field meaningful. Dropping either mode term is section 3's bug.
  always_comb begin
    claim       = 1'b0;
    undecidable = 1'b0;
 
    if (in_valid) begin
      if (mode_unresolved) begin
        undecidable = 1'b1;
      end else begin
        unique case (mode)
          ROUTE_ADDRESS: claim = addr_hit;
          ROUTE_ID:      claim = bus_hit;
          // Implicit routing is resolved from hierarchy position, not from a
          // window or a range. A downstream port does not decide it with the
          // registers modelled here.
          ROUTE_IMPLICIT: undecidable = 1'b1;
          default:        undecidable = 1'b1;
        endcase
      end
    end
  end
 
endmodule

Classification: synthesizable.

Architecture. Two independent range comparators selected by the routing mode. They share no logic and no registers, because §3's argument is that they cannot.

State. None — the decision is a pure function of the packet and the configuration. Configuration stability during a packet's decision is the caller's obligation.

Cycle behaviour. Combinational; a real Switch would register the decision and pipeline the port arbitration.

Contract. The caller guarantees that tlp_address is meaningful when and only when mode is ROUTE_ADDRESS, and tlp_bus likewise for ROUTE_ID. The module relies on that rather than checking it, which is exactly why the mode terms in the case are load-bearing.

Failure — four, and the first two are the ones that ship. Omitting mem_window_en makes an unconfigured port claim address zero, so a Switch fresh out of reset captures traffic it has no business seeing. An exclusive limit compare (< instead of <=) silently excludes the top location of every window, producing a fault only at range boundaries. Consulting addr_hit regardless of mode routes Completions by whatever occupies the address position. And swapping Secondary and Subordinate produces an empty range for every non-degenerate subtree, so nothing routes down the port at all.

Deliberately simplified: one window rather than the full Type 1 register set; no I/O window; no peer-to-peer; no implicit-routing resolution; no error response for unroutable packets; no upstream-port logic.

Production implication: a real Switch port implements every window kind, resolves implicit routes from its position, arbitrates among ports, and applies the specification's required responses to unroutable packets — none of which changes the two range compares above.

12. Assertions

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SVA over tlp_route_classify and switch_port_route. Local contracts plus
// the normative mechanism selection of section 2 — not claims about the
// complete PCIe routing rule set, which includes Message routing and
// peer-to-peer rules this model does not implement.
 
// CLASSIFY — P1: exactly one outcome. A mode or the unresolved indication.
property p_mode_exclusive;
  @(posedge clk) disable iff (!rst_n)
  meta.valid |-> (unresolved == (mode == ROUTE_UNKNOWN));
endproperty
a_mode_exclusive : assert property (p_mode_exclusive);
 
// CLASSIFY — P2: THE COMPLETION RULE. A Completion is NEVER address-routed.
// The single most damaging misclassification available (section 10).
property p_completion_is_id_routed;
  @(posedge clk) disable iff (!rst_n)
  (meta.valid && !meta.unsupported && (meta.family == FAM_COMPL))
    |-> (mode == ROUTE_ID);
endproperty
a_completion_id : assert property (p_completion_is_id_routed);
 
// CLASSIFY — P3: a configuration request is ID-routed, never address-routed.
// Stated separately because it fails for a different reason than P2 — a
// design that "helpfully" derived an address for config space.
property p_config_is_id_routed;
  @(posedge clk) disable iff (!rst_n)
  (meta.valid && !meta.unsupported && (meta.family == FAM_CFG_REQ))
    |-> (mode == ROUTE_ID);
endproperty
a_config_id : assert property (p_config_is_id_routed);
 
// CLASSIFY — P4: an unsupported family never yields a routing mode.
property p_unsupported_unresolved;
  @(posedge clk) disable iff (!rst_n)
  (meta.valid && meta.unsupported) |-> unresolved;
endproperty
a_unsupported : assert property (p_unsupported_unresolved);
 
// PORT — P5: PREDICATION. A claim is always attributable to the comparison
// belonging to the packet's own routing mode — so the window comparison can
// never influence an ID-routed decision, or the range comparison an
// address-routed one.
property p_claim_source_matches_mode;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && claim)
    |-> ((mode == ROUTE_ADDRESS) && addr_hit)
     || ((mode == ROUTE_ID)      && bus_hit);
endproperty
a_claim_source : assert property (p_claim_source_matches_mode);
 
// PORT — P6: a disabled window never claims. Catches the reset-state bug
// where an all-zero window captures address zero.
property p_disabled_window_never_claims;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && (mode == ROUTE_ADDRESS) && !mem_window_en) |-> !claim;
endproperty
a_window_gate : assert property (p_disabled_window_never_claims);
 
// PORT — P7: a disabled bus range never claims.
property p_disabled_range_never_claims;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && (mode == ROUTE_ID) && !bus_range_en) |-> !claim;
endproperty
a_range_gate : assert property (p_disabled_range_never_claims);
 
// PORT — P8: INCLUSIVE boundaries, both ends, both windows. Stated as four
// implications so an exclusive compare fails on the exact boundary rather
// than only under a sweep that happens to hit it.
property p_window_inclusive;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && (mode == ROUTE_ADDRESS) && mem_window_en
   && ((tlp_address == mem_base) || (tlp_address == mem_limit))) |-> claim;
endproperty
a_window_inclusive : assert property (p_window_inclusive);
 
property p_range_inclusive;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && (mode == ROUTE_ID) && bus_range_en
   && ((tlp_bus == secondary_bus) || (tlp_bus == subordinate_bus))) |-> claim;
endproperty
a_range_inclusive : assert property (p_range_inclusive);
 
// PORT — P9: strictly outside the window never claims. The other half of P8;
// together they pin the boundary exactly.
property p_outside_window_no_claim;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && (mode == ROUTE_ADDRESS)
   && ((tlp_address < mem_base) || (tlp_address > mem_limit))) |-> !claim;
endproperty
a_outside : assert property (p_outside_window_no_claim);
 
// PORT — P10: claim and undecidable are mutually exclusive. A port that both
// forwards and reports it could not decide has duplicated the packet.
property p_claim_xor_undecidable;
  @(posedge clk) disable iff (!rst_n)
  in_valid |-> !(claim && undecidable);
endproperty
a_exclusive_outcome : assert property (p_claim_xor_undecidable);
 
// SYSTEM — P11: at most ONE downstream port claims a given packet. The
// property that catches overlapping windows, and the one that cannot be
// written inside a single port. Bound at the Switch level.
property p_at_most_one_claimer;
  @(posedge clk) disable iff (!rst_n)
  in_valid |-> ($countones(port_claim_vector) <= 1);
endproperty
a_one_claimer : assert property (p_at_most_one_claimer);
 
// SAFETY — P12: the decision is never unknown.
property p_decision_never_unknown;
  @(posedge clk) disable iff (!rst_n)
  in_valid |-> !$isunknown({claim, undecidable});
endproperty
a_no_x : assert property (p_decision_never_unknown);

P2 is the chapter's central property. Everything else checks a comparison; P2 checks that the right comparison was selected. A design that address-routed Completions would pass every window test it was given and destroy every read in the system — and P2 fires on the first Completion.

P11 is the one that cannot live in the module. Window non-overlap is a system invariant established by Chapter 9.5's assignment, and a single port cannot observe it — it only knows whether it claimed. Binding it at the Switch level is what makes an overlapping-window configuration bug visible in RTL rather than as a mysterious duplicate packet.

P8 and P9 are a boundary pair and neither is sufficient alone. P8 alone passes a design whose window is too wide; P9 alone passes one that is too narrow. Together they pin both edges exactly, which is what an inclusive-limit register demands.

P6 and P7 exist because reset state is a real configuration. An unconfigured window is all-zero, and a comparator without an enable term claims address zero — a Switch that captures traffic before software has told it to own anything.

13. Verification

Monitors observe: the descriptor and the classified mode; each port's configuration inputs; and the per-port claim vector.

The scoreboard maintains its own routing model — its own family-to-mode mapping and its own range comparisons — built from §2's mechanisms rather than from the design's tables. It must not import tlp_route_pkg as its expected mode source or reuse addr_hit.

Mode classification

  • Each family in the modelled subset. Verify the mode (P1–P3).
  • A Completion. Verify ROUTE_ID explicitly, as a named test — this is P2's stimulus and it deserves to fail loudly if it regresses.
  • An unsupported family. Verify unresolved and that no port claims (P4).
  • A Message family packet. Verify it is reported unresolved by this model, and that the report says unresolved-by-this-model rather than unroutable-per-PCIe.

Address routing

  • An address strictly inside the window. Claim.
  • An address exactly at the base, and exactly at the limit. Claim, both (P8) — the two tests that catch an exclusive compare, and neither is produced by random stimulus with useful probability.
  • One below the base and one above the limit. No claim (P9).
  • The window disabled. No claim at any address, including zero (P6).
  • A window of exactly one location (base == limit). Claim at that location only.
  • Multiple ports with non-overlapping windows. Verify exactly one claims (P11).
  • Multiple ports with deliberately overlapping windows. Verify P11 fires — a negative test of the assertion itself, confirming the system-level check is live.

ID routing

  • A bus number inside the range, and one at each end (P8's range counterpart).
  • One below Secondary and one above Subordinate. No claim.
  • The range disabled. No claim (P7).
  • A degenerate range where Secondary equals Subordinate. Claim at that bus only.
  • Secondary greater than Subordinate. An invalid configuration; verify no claim and no X.

Cross-mode and negative

  • The same numeric value presented as an address and as a bus number, with windows configured so that one would hit and the other would not. Verify the decision follows the mode — the direct test for P5.
  • A Completion presented with a plausible address on tlp_address. Verify the address path is not consulted and the bus range decides.
  • Configuration changed between packets. Verify each packet is decided against the configuration in force.
  • in_valid low. Verify no claim and no undecidable.

Coverage should include: every routing mode; every family in the subset; window and range hits at both boundaries and both misses; disabled window and disabled range; degenerate and inverted ranges; the multi-port exactly-one-claimer case; and the unresolved path.

14. Debugging

Completions never reach the requester

Check the Requester ID in the Completion first, and check it in the Completion rather than in the Request.

The Completion routes by the identity the Completer copied from the Request (§7). If the Completer returned its own identity, or re-derived one from an internal table, the Completion routes toward the wrong component — usually toward the Completer itself, which then discards it.

Two more candidates. The Requester ID was truncated or the wrong slice was taken, so it names a valid-looking but wrong component. Or a Switch port's Secondary/Subordinate range does not cover the Requester's bus, which happens when the Completion is crossing a boundary that enumeration numbered inconsistently.

The observation that separates them. Capture the Request and its Completion and compare the Requester ID field byte for byte. If they differ, it is the Completer; if they match, it is the fabric's bus-number map — and then compare the Requester's bus number against each port's range.

Note what this is not. Chapter 10.2 §9's correlation failure looks superficially similar — the Requester never resolves the operation — but there the Completion arrived and failed to match an entry. Here it never arrived, and the distinction is one monitor away.

Memory writes land on the wrong device

Address routing sent the packet down a port that should not have claimed it, and the usual cause is not in the routing logic.

Most likely: overlapping windows. Two ports both contain the address, and the arbitration picks one arbitrarily. That is a configuration faultChapter 9.5's assignment produced an overlap — not a comparator bug, and no amount of staring at the compare will find it.

Also likely: a window enable that was never set, or one that was set before the base and limit were written. A window enabled around all-zero registers claims low addresses.

Least likely but most confusing: an exclusive limit compare, which does not misroute — it fails to route the top location of a window, so the symptom is a single address at a range boundary going upstream while everything else works.

The observation: dump every port's base, limit and enable, and check for overlap before looking at any RTL. P11 is the assertion that would have caught it in simulation, and if it is not bound at the Switch level it cannot fire.

A configuration access reaches the wrong function, and address routing looks fine

Because address routing is fine — configuration accesses do not use it (§5).

Look at the bus-number map, not the window map. The two are populated independently and one can be wrong while the other is right — which is precisely why they are separate comparators.

Three candidates. Enumeration assigned a Subordinate Bus Number that does not cover the whole subtree, so accesses to the deepest buses fall out of range. Secondary and Subordinate are swapped, producing an empty range. Or the identity's internal division is being interpreted inconsistently with the ARI configuration in force (§5), so the extracted Bus Number is not the one the packet named.

The clean check: for each port, confirm that Secondary ≤ Subordinate and that the union of all downstream ranges covers every bus behind the Switch, with no gaps and no overlaps.

Every read fails but writes work perfectly

The asymmetry is the diagnosis, and it points at §7 rather than at anything in the write path.

A posted write is one packet in one direction. If address routing works, writes work. A read is two packets using two different routing mechanisms — out by address, back by ID — so a read exercises the ID path that a write never touches.

So "writes work, reads do not" means the outbound direction is healthy and the return path is broken, and the return path is ID routing. Check the Completion's Requester ID and the bus-number map, in that order.

The version of this that is a routing-logic bug rather than a configuration bug is §10's failure: Completions classified as address-routed. The signature there is distinctive — the fabric compares whatever bits occupy the address position in a Completion header, which is not an address, so the packets go to essentially arbitrary ports and the pattern of which reads fail looks random. P2 catches it; nothing in a write test does.

15. Common Misconceptions

  • "All TLPs are routed by address." Three mechanisms exist. Configuration Requests and Completions route by ID; some Messages route implicitly (§2).
  • "A Completion is routed back to the address the request came from." A Requester is not an address. The Completion routes by the Requester ID the Completer copied from the Request (§7).
  • "The Requester ID is only for correlation." It is also the return path. One field, doing routing at the fabric level and correlation at the Requester (§7).
  • "A Completer can regenerate the Requester ID from its own state." It must return the one it received, unmodified. A re-derived identity routes the Completion somewhere else (§7).
  • "Address routing and ID routing are two encodings of one lookup." They consult different configuration registers, populated by different processes, and can be independently wrong (§3).
  • "Implicit routing means the destination is implied by the address." It means no destination is named at all — the route is a relationship the receiving port resolves from its own hierarchy position (§6).
  • "Every Message is implicitly routed." Some Messages route by address or by ID. Resolving which requires the Message taxonomy (Chapter 11.7), which is why §10 reports rather than assumes (§6).
  • "A Switch needs a routing table the protocol distributes." It needs its Base/Limit and Secondary/Subordinate registers, written by enumeration software. No routing protocol runs on PCIe (§2).
  • "Bus numbers behind a port can be any set." They are a contiguous range, because enumeration assigns them depth-first — which is what lets two registers express the map (§5).
  • "An address matching no downstream window is an error." It is forwarded upstream, which is the default direction. That is why address-routing faults usually misdeliver rather than drop (§4).
  • "A single comparator can serve both windows." The address window and the bus range have different widths, different registers and different enables. Sharing one is how a Completion gets compared against a memory window (§11).
  • "Window limits are exclusive." The Base/Limit window is an inclusive range, and an exclusive compare loses exactly the top location — a fault that appears only at boundaries (P8, P9).

16. Understanding Check

17. What's Next

This chapter answered the last of the three questions a packet must answer about itself. Chapter 11.3 asked what am I, Chapter 11.4 asked how much data do I carry, and this one asked where do I go — and found that the third question has three different answers depending on the first.

Chapter 11.6 — Attributes takes what remains of the header's handling context: the Traffic Class, the ordering attributes, and the hints that tell the fabric and the Completer how to treat a packet once its destination is settled. Chapter 11.7 — Packet Types then closes Module 11 with the complete taxonomy that §10's classifier stands in for — including the Message routing indications this chapter deliberately left unresolved.

Module 12 then takes memory transactions end to end, with the alignment and boundary rules that Chapter 11.4 §7 left out.

The idea to carry forward: the destination is derived, not carried — and which field you derive it from is the first thing you have to know.