Bus Number Assignment — Numbering the Regions of a Hierarchy

Chapter 7.3 probed "candidate hierarchy locations" and deliberately never said what one is. Chapter 2.6 established that identity has a bus part, a device part, and a function part, and equally deliberately gave none of them a value.

Both were waiting for this.

Why does a PCIe hierarchy need bus numbers, and how do bridges and switch ports create numbered regions that software can discover and traverse?

1. The Question Topology Forces

Take the hierarchy from Chapter 4.4: a Root Complex, a Root Port, a Switch, and endpoints — with a second switch level below.

Software wants to probe something two levels down. The configuration access leaves the Root Complex and immediately faces a decision at every branching component it meets: does the target lie through this port, or not?

That question has to be answerable locally, by each component, without any component knowing the whole topology. A switch downstream port cannot hold a map of the system; it can only decide whether what it was handed belongs behind it.

Bus numbers exist to make that decision possible. They give every region of the tree a name, and give every branching port enough information to say whether a named region lies behind it.

2. Primary, Secondary, Subordinate

Every hierarchy-expanding port — a Root Port, a switch upstream port, a switch downstream port, a bridge — carries three bus numbers describing its relationship to the tree:

Primary bus number. The bus on the upstream side of the port — the region it sits in, looking back toward the Root Complex.

Secondary bus number. The bus immediately downstream of the port — the first region on the far side.

Subordinate bus number. The highest-numbered bus reachable anywhere in the hierarchy below this port.

Primary and secondary are positional: they name the two regions the port sits between. Subordinate is different in kind — it is a statement about everything beneath.

3. Why a Range, Not a Number

Secondary alone would be enough if every downstream port led to exactly one region. It does not.

A switch downstream port may lead to another switch, which creates its own internal region and its own set of downstream regions beneath that. One port can therefore have an arbitrary number of buses behind it.

So a port needs to answer "is the target behind me?" for a set of regions, and the cheapest representation of a set of consecutively numbered regions is a range:

target is downstream  ⟺  secondary ≤ target_bus ≤ subordinate

Subordinate is the upper bound of that range. That is the whole reason it exists, and it is why a bridge whose subordinate is set too low will forward accesses for regions near itself and silently fail to forward accesses for regions deeper in its own subtree — the failure §11 develops.

4. A Worked Hierarchy

The five regions in this hierarchy:

Bus 2 is worth pausing on. A switch is not one component from the configuration namespace's point of view — it presents an upstream port and several downstream ports, and those downstream ports have to sit somewhere. They sit on an internal region the switch creates. That region is a real, numbered, probeable bus, and it is where Chapter 7.5's device numbers finally become interesting.

Check the ranges against the tree. The Root Port's subordinate is 4 because bus 4 is the highest-numbered region anywhere beneath it. The switch upstream port's subordinate is also 4, for the same reason. Downstream port A's subordinate is 3, because nothing below it goes further. Every range covers exactly its own subtree and no more.

5. Range Reasoning

Take the switch upstream port — primary 1, secondary 2, subordinate 4 — and ask how it classifies a configuration access aimed at each region:

The two boundary rows are where implementations go wrong. A comparison written with < instead of ≤ at either end silently excludes a whole region — and the region excluded at the subordinate end is the deepest part of the subtree, which is exactly the part that is hardest to notice missing.

6. Numbering Happens During Discovery

Bus numbers are not present at reset waiting to be read. They are assigned as the tree is walked, which follows directly from Chapter 7.3: the topology is not known in advance, so the set of regions needing numbers is not known either.

The consequence for the range values is worth being precise about. A port's subordinate cannot be known when the port is first found, because what lies below it has not been explored yet. Only after the subtree beneath a port has been walked is the highest bus number in it known — and only then can that port's subordinate be given its final value.

So numbering is interleaved with discovery rather than preceding it: a port is found, its secondary is established so accesses can be directed through it, the region beyond is explored, and the port's subordinate is settled once the extent of its subtree is known.

7. Microarchitecture — A Port's Range State

From a hierarchy-expanding port's point of view, its participation in all of this is small and specific:

• Hold three configured values.
• Answer, for a given target region, whether it lies downstream.
• Continue to behave sanely while those values are being changed.

The illustrative internal representation:

// Illustrative internal representation — NOT a PCIe configuration-register
// encoding and not a header layout. Field placement in configuration space
// is Module 8's subject.
typedef struct packed {
  logic [BUS_W-1:0] primary;
  logic [BUS_W-1:0] secondary;
  logic [BUS_W-1:0] subordinate;
} bus_range_t;

Classification: conceptual.

What it teaches: that the three values belong together as one coherent unit. §9 is entirely about what happens when they are treated as three independent registers instead.

8. RTL — The Range Decoder

// SYNTHESIZABLE. Classifies a configuration target's bus relative to one
// hierarchy-expanding port's configured range.
// NOT complete PCIe routing logic.
package bus_range_pkg;
  typedef enum logic [1:0] {
    BUS_UNCLASSIFIED = 2'd0,  // port disabled or range not yet valid
    BUS_UPSTREAM     = 2'd1,  // target is this port's primary region
    BUS_DOWNSTREAM   = 2'd2,  // target lies within [secondary, subordinate]
    BUS_OUTSIDE      = 2'd3   // target is neither
  } bus_dir_e;
endpackage
 
module bus_range_decode
  import bus_range_pkg::*;
#(
  parameter int BUS_W = 8      // verified: PCIe bus numbers are 8 bits, 0-255
) (
  input  logic [BUS_W-1:0] target_bus,
  input  logic [BUS_W-1:0] primary,
  input  logic [BUS_W-1:0] secondary,
  input  logic [BUS_W-1:0] subordinate,
  input  logic             range_valid,
  input  logic             port_enabled,
 
  output bus_dir_e         direction,
  output logic             downstream_match
);
 
  // Both comparisons are INCLUSIVE. Using < at either end silently excludes a
  // whole region — and at the subordinate end that region is the deepest part
  // of the subtree, which is the hardest absence to notice.
  wire in_downstream = (target_bus >= secondary) && (target_bus <= subordinate);
 
  always_comb begin
    if (!port_enabled || !range_valid) direction = BUS_UNCLASSIFIED;
    else if (target_bus == primary)    direction = BUS_UPSTREAM;
    else if (in_downstream)            direction = BUS_DOWNSTREAM;
    else                               direction = BUS_OUTSIDE;
  end
 
  // A port must never claim a target while its range is unconfigured. During
  // enumeration the range genuinely is unconfigured for a period, and a port
  // that forwards on a reset-value range forwards into a subtree that may not
  // be the one the access was aimed at.
  assign downstream_match = (direction == BUS_DOWNSTREAM);
 
endmodule

Classification: synthesizable, with a compile-time enumeration.

What it models: the local decision that lets a port forward without global topology knowledge.

What it teaches: that range_valid is not defensive decoration. Bus ranges are written during enumeration, so there is a real interval in which a port's range does not yet describe anything. Forwarding during that interval sends accesses into a subtree chosen by a reset value.

Deliberately simplified: configuration targets only; one range per port; no behaviour defined for BUS_OUTSIDE beyond classification; and no other routing modes.

Production implication: a real port must implement the routing rules for every transaction type it handles, define behaviour for unmatched targets, and integrate this classification with the access-delivery mechanism Chapter 7.7 covers.

9. RTL — Atomic Range Commit

Here is the problem §6 set up.

The three range values are written separately. Between the write that changes secondary and the write that changes subordinate, the port holds a mixture of old and new values — and that mixture can describe a range that never existed and was never intended.

Concretely: a port currently configured [3, 3] is being reconfigured to [5, 7]. After the secondary write and before the subordinate write, it holds [5, 3] — a range whose lower bound exceeds its upper bound. A naive comparator classifies nothing as downstream, so accesses aimed at a legitimately reachable region are not forwarded for as long as that window lasts.

The reverse ordering is worse. Reconfiguring [5, 7] to [3, 4] passes through [5, 4] if subordinate is written first, and through [3, 7] if secondary is written first — and [3, 7] is a wider range than either the old or the new one. A port in that state claims regions belonging to somebody else.

// SYNTHESIZABLE. Staged range configuration with validated atomic commit.
// The staging/commit microarchitecture is an IMPLEMENTATION CHOICE — PCIe
// does not mandate it. It is shown because the transient-range problem is
// real and this is one clean way to remove it.
module bus_range_commit #(
  parameter int BUS_W = 8
) (
  input  logic             clk,
  input  logic             rst_n,
 
  // Staging writes. One field at a time, as configuration writes arrive.
  input  logic             stage_we,
  input  logic [1:0]       stage_sel,     // 0=primary, 1=secondary, 2=subordinate
  input  logic [BUS_W-1:0] stage_data,
 
  input  logic             commit,        // apply the staged set as one unit
  input  logic             clear_error,
 
  output logic [BUS_W-1:0] active_primary,
  output logic [BUS_W-1:0] active_secondary,
  output logic [BUS_W-1:0] active_subordinate,
  output logic             range_valid,
  output logic             config_error   // sticky: an illegal commit attempt
);
 
  logic [BUS_W-1:0] stg_pri, stg_sec, stg_sub;
 
  // A staged set is legal only if its range is non-empty. This is the check
  // that a field-at-a-time design cannot perform, because it never has a
  // complete proposed set to check.
  wire staged_legal = (stg_sec <= stg_sub);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      stg_pri <= '0; stg_sec <= '0; stg_sub <= '0;
      active_primary <= '0; active_secondary <= '0; active_subordinate <= '0;
      range_valid <= 1'b0;      // nothing is claimed until a legal commit
      config_error <= 1'b0;
    end else begin
      if (stage_we) begin
        case (stage_sel)
          2'd0: stg_pri <= stage_data;
          2'd1: stg_sec <= stage_data;
          2'd2: stg_sub <= stage_data;
          default: ;            // no fourth field; ignore rather than corrupt
        endcase
      end
 
      if (commit) begin
        if (staged_legal) begin
          // ATOMIC: all three active fields change in the same cycle, so no
          // observer ever sees a mixture of old and new. This is the entire
          // point of the module.
          active_primary     <= stg_pri;
          active_secondary   <= stg_sec;
          active_subordinate <= stg_sub;
          range_valid        <= 1'b1;
        end else begin
          // REFUSE and RECORD. The active range is left untouched — an
          // illegal proposal must not disturb a working configuration — and
          // the attempt is made visible rather than silently dropped.
          config_error <= 1'b1;
        end
      end else if (clear_error) begin
        config_error <= 1'b0;
      end
    end
  end
 
endmodule

Classification: synthesizable.

What it models: configuration of a coherent multi-field value that is written one field at a time.

What it teaches — three things:

1. Staging makes the legality check possible at all. A design that writes directly to active fields never holds a complete proposed set, so it has nothing to validate. Validation requires a staging area, not merely a desire to validate.
2. Atomic commit removes the transient entirely rather than shrinking it. There is no window, however brief, in which the active range is a mixture — which matters because the window's duration is set by software's write timing, not by anything the hardware controls.
3. Refusing must leave the working configuration alone. An illegal proposal is a software or sequencing bug; destroying a valid range because of it converts a configuration error into a connectivity failure.

Deliberately simplified: no relationship checked between primary and the downstream range; no interlock preventing a commit while accesses are in flight; and stage_sel is an internal encoding rather than any configuration-space addressing.

Production implication: a real port must decide what happens to accesses already in flight when a range changes, coordinate range changes with the readiness gating of Chapter 7.2, and expose enough state for an engineer to tell a not-yet-configured port from a mis-configured one.

10. Assertions

// SVA over bus_range_decode and bus_range_commit. Implementation invariants
// for THESE designs — not PCIe protocol requirements.
 
// COHERENCE — P1: an active range is always non-empty. The property the
// staging design exists to guarantee; it is FALSE in any field-at-a-time
// implementation for at least one cycle per reconfiguration.
property p_active_range_nonempty;
  @(posedge clk) disable iff (!rst_n)
  range_valid |-> (active_secondary <= active_subordinate);
endproperty
a_range_nonempty : assert property (p_active_range_nonempty);
 
// ATOMICITY — P2: the three active fields change together or not at all. A
// mixture of old and new describes a range nobody configured.
property p_atomic_commit;
  @(posedge clk) disable iff (!rst_n)
  ($changed(active_primary) || $changed(active_secondary) || $changed(active_subordinate))
    |-> $past(commit && staged_legal);
endproperty
a_atomic : assert property (p_atomic_commit);
 
// SAFETY — P3: an illegal proposal never becomes active. Catches a commit
// path that validates and then applies anyway.
property p_illegal_never_active;
  @(posedge clk) disable iff (!rst_n)
  (commit && !staged_legal) |=> ($stable(active_secondary) && $stable(active_subordinate)
                                 && $stable(active_primary));
endproperty
a_illegal_refused : assert property (p_illegal_never_active);
 
// SAFETY — P4: no target is classified downstream while the range is invalid.
// During enumeration the range genuinely is invalid for a period, and
// forwarding then sends accesses into a subtree chosen by a reset value.
property p_no_match_when_invalid;
  @(posedge clk) disable iff (!rst_n)
  (!range_valid || !port_enabled) |-> !downstream_match;
endproperty
a_no_match_invalid : assert property (p_no_match_when_invalid);
 
// CORRECTNESS — P5: downstream classification agrees with the range,
// inclusively at both ends. This is the off-by-one property: an implementation
// using < at either boundary passes every test whose targets avoid the edges.
property p_downstream_matches_range;
  @(posedge clk) disable iff (!rst_n)
  (range_valid && port_enabled && target_bus != primary)
    |-> (downstream_match == ((target_bus >= secondary) && (target_bus <= subordinate)));
endproperty
a_range_exact : assert property (p_downstream_matches_range);
 
// EXCLUSIVITY — P6: a target is never classified both upstream and downstream.
// Catches a configuration in which primary falls inside the downstream range,
// which is a topology error that would make the port's behaviour ambiguous.
property p_direction_unambiguous;
  @(posedge clk) disable iff (!rst_n)
  (direction == BUS_UPSTREAM) |-> !downstream_match;
endproperty
a_unambiguous : assert property (p_direction_unambiguous);
 
// SAFETY — P7: a recorded configuration error is sticky until explicitly
// cleared. An illegal commit attempt is a software bug and must stay visible.
property p_error_sticky;
  @(posedge clk) disable iff (!rst_n)
  (config_error && !clear_error) |=> config_error;
endproperty
a_error_sticky : assert property (p_error_sticky);

P5 is the property that catches the off-by-one. An implementation using < instead of ≤ at the subordinate boundary is correct for every target except the single highest region in its subtree — and that region is typically the deepest, least-exercised part of the hierarchy. Directed tests that probe "a device behind the switch" will usually not hit it; a property comparing against the range specification does, on the first randomised target that lands on the edge.

P2 is the reason the staging module exists. In a field-at-a-time implementation this property fails for at least one cycle on every reconfiguration, and the failure is precisely the transient misrouting window. It cannot be fixed by making writes faster — only by making them atomic.

P4 catches a failure with a confusing signature. A port that forwards on its reset-value range during enumeration delivers accesses to somewhere plausible-looking. The symptom is a device appearing under the wrong parent, or two ports both claiming a region, and neither points obviously at "we forwarded before we were configured."

11. Verification

Monitors observe: the staging writes, the commit, the active range fields, range_valid, config_error, and every classification result with its target.

The scoreboard independently models: which buses are reachable below each port, from its own model of the topology — not by evaluating the design's comparator. This matters more here than usual: a scoreboard that computes the expected classification with the same >=/<= expression the design uses will agree with the design about an off-by-one, and P5 becomes the only thing standing between the bug and silicon.

Scenarios:

• One-level hierarchy. Root Port with a single endpoint beneath. Secondary equals subordinate — the narrowest legal range.
• Multi-level hierarchy. The §4 topology. Verify each port's range covers exactly its subtree.
• Narrow range (secondary == subordinate) and broad range (spanning many regions).
• Target exactly at secondary. The lower boundary — must classify downstream.
• Target exactly at subordinate. The upper boundary — must classify downstream. This is the P5 scenario and it must be written explicitly.
• Target one below secondary and one above subordinate. Must classify outside.
• Target equal to primary. Must classify upstream, not downstream (P6).
• Illegal range proposed. secondary > subordinate staged, then committed. Verify refusal, sticky error, and that the previously active range survives untouched.
• Reconfiguration while active. Change from [3, 3] to [5, 7] and back. Verify no intermediate classification is ever observable — the atomicity that a field-at-a-time design fails.
• Classification before any commit. Verify nothing is claimed while range_valid is low (P4).
• Reset with a configured range. Verify range_valid clears and nothing is claimed afterwards.

Coverage should include: target at each boundary and each boundary ± 1; ranges of width 1 and width > 1; every stage_sel value including the unused encoding; commit with legal and illegal staged sets; and reset in both valid and invalid range states.

12. Debugging

Symptom: an entire downstream subtree is missing while upstream devices enumerate

What the upstream success establishes. The Root Complex is issuing accesses, the path down to this port works, and enumeration is running and reaching this depth. That is Chapter 7.3's smallest-shared-element reasoning applied one level up.

What makes bus-range configuration a strong suspect. The subtree beneath a port is reachable only if that port forwards accesses into it, and forwarding depends entirely on the range. A port whose range is unconfigured, too narrow, or refused claims nothing — and everything behind it is silently unreachable while remaining perfectly healthy.

The ladder for this symptom:

1. Was the port itself discovered? If the port was never found, nothing below it was ever probed. Stop; the fault is at the port, not the subtree.
2. Is the downstream Link usable? Chapter 7.2's question. A healthy port with a dead Link forwards into nothing.
3. Was a secondary bus number established? Without it no region exists to direct accesses at.
4. Is range_valid set? If the commit was refused (config_error), the port is holding its previous — possibly reset — range.
5. Is subordinate broad enough? The specific failure §5 warns about. A subordinate set to the secondary value works for the first level and excludes everything deeper.
6. Are accesses actually being forwarded? Observe at the port's downstream side. A classification that says downstream but no access appearing is a different fault from a classification that says outside.

Symptom: the first level below a port appears, deeper levels do not

This is diagnostically sharper than the previous case, and it points almost directly at the range.

The reasoning. If the port forwards accesses to its secondary region successfully, then the port is discovered, its Link works, its range is valid, and forwarding functions. All of §12's first four rungs are eliminated by the first level working.

What is left is the extent of the range. A port whose secondary is correct and whose subordinate is too small forwards to exactly the regions within the narrow range and refuses everything beyond — which presents as "one level works, deeper levels do not."

The distinguishing observation. Compare the port's subordinate against the highest bus number that should exist beneath it. If subordinate is smaller, the diagnosis is settled without any further measurement.

The reason this happens in practice. Subordinate cannot be finalised until the subtree has been explored (§6). Any interruption of that sequence — an error partway through the walk, a refused commit, a discovery failure deeper down that truncates the walk — can leave a subordinate value that reflects a partially explored subtree. The range then causes the very absence that produced it.

13. Common Misconceptions

• "A bus number identifies a physical electrical bus." In PCIe it identifies a region of the hierarchy. The multi-drop shared bus that the term came from is PCI (Chapter 1.3); PCIe kept the numbering model for software continuity while the physical arrangement became point-to-point links.
• "A switch consumes one bus number." A switch creates an internal region for its downstream ports to sit on, and each downstream port creates a further region below it. The §4 example consumes buses 2, 3, and 4 for one switch with two endpoints.
• "Every port has a permanent bus number from reset." Bus numbers are assigned during enumeration, because the topology they describe is not known in advance. A port's range fields are written — often more than once — while the tree is being walked.
• "Subordinate means the next bus." It means the highest-numbered bus reachable anywhere below this port. For a port with a single endpoint beneath it, subordinate happens to equal secondary; for a port leading to another switch it is larger, and that difference is the entire reason the field exists.
• "Bus numbering is bookkeeping for software displays." It determines which accesses a port forwards. A wrong range makes a healthy subtree unreachable, with no error reported anywhere.
• "Primary, secondary, and subordinate are device or function numbers." They are all bus numbers — three values naming regions relative to one port. Device and function are separate coordinates covered in Chapter 7.5 and Chapter 7.6.
• "Bus number is the same as Link number." A Link is a physical connection between two components (Chapter 6.1). A bus number names a configuration region. A Link creates a region, but the two are different kinds of thing and a switch's internal region corresponds to no Link at all.
• "Numbering does not affect discovery." Discovery of a subtree requires forwarding into it, and forwarding requires a range that includes it. Numbering and discovery are interleaved, and a numbering failure presents as a discovery failure.
• "A healthy Link guarantees the downstream region is reachable." The Link carries the access to the port. Whether the port forwards it depends on the range. Chapter 7.2 made the same point for readiness; this is its configuration-layer counterpart.

14. Understanding Check

15. What's Next

Bus numbers name regions. An access that has reached the right region still has to find the right thing within it.

Chapter 7.5 — Device Number Assignment takes up the second coordinate: what a device number identifies, why its meaning is local to its bus rather than globally unique, and why PCIe's point-to-point links make the device-number space behave very differently below a Root Port than on a switch's internal region — the bus 2 in this chapter's example, where the two downstream ports sit at different device positions.