Enumeration Overview — Turning a Physical Hierarchy Into a Configured System

Module 6 ended with a Link that is physically connected, correctly aggregated, and quantified down to the byte. From software's point of view, none of that means anything yet.

The device is there. Nothing knows it is there.

What problem does PCIe enumeration solve, and what sequence turns an unknown hierarchy into a configured system?

1. Why Enumeration Exists

After power-up, the topology exists physically. Components are connected, Links can be established, and traffic could in principle flow.

What does not exist is a machine-readable description of any of it. The host does not know:

• Which devices are present.
• Where they sit in the hierarchy.
• Which functions each one exposes.
• What resources each function requires.
• Where those resources should live in the system's address space.
• How a transaction issued later will be routed to reach them.

None of that can be assumed. A platform's topology is not fixed at design time — devices are added, removed, and replaced, and switches expand the hierarchy in ways that are only discoverable by looking.

Enumeration is how the system builds a description of a topology it cannot know in advance.

Everything that comes later — routing, address decoding, interrupt delivery, driver binding — depends on that description existing and being correct.

2. Six Stages

The lifecycle decomposes into six questions, each distinct.

This chapter maps the process. Each stage has its own chapter, and none of them is taught in depth here:

• Discovery — Chapter 7.3
• Identification — conceptually 7.3, with configuration-space structure owned by Module 8
• Numbering — Chapters 7.4, 7.5, and 7.6, which own the hierarchy identity scheme entirely
• Resource sizing and allocation — Chapter 7.8, with the underlying mechanics in Module 9
• Configuration access — the mechanism used throughout, owned by Chapter 7.7

3. The Lifecycle

Two properties of the figure are worth stating explicitly.

Every outward arrow is a configuration access. Enumeration is conducted almost entirely through one mechanism, applied repeatedly with different targets and different intent. That mechanism is Chapter 7.7's subject; what matters here is that discovery, identification, allocation, and enablement all ride on it.

The last arrow is the point of everything before it. A function that has been discovered, identified, numbered, and given resources still does not participate in normal traffic until it is enabled. Enumeration is not complete when the device is visible; it is complete when the device is usable.

4. Discovery Follows Topology

PCIe hierarchies are trees (Chapter 4.4). Discovery is therefore naturally recursive:

Start at the root. Examine the locations reachable from it. Where something is found that expands the hierarchy — a switch path, a bridge — the locations beyond it become reachable and must be examined too. Continue until nothing further is reachable.

The key idea is that discovery cannot be a flat list. The set of locations worth probing is not known at the start; it grows as the hierarchy is explored. Finding a switch reveals a subtree that was invisible a moment earlier.

This has a direct consequence for failure: a discovery failure high in the tree hides everything beneath it. If the element that expands a subtree is not found, nothing in that subtree is ever probed — not because those devices failed, but because nobody ever looked. That distinction drives the debugging in §9 and in Chapter 7.3.

How the hierarchy positions are named and numbered as this walk proceeds is Chapter 7.4's subject and is not addressed here.

5. Who Does What

This is where treatments of enumeration usually go wrong in one of two directions.

"Enumeration is BIOS software." It is orchestrated by host software — which may be platform firmware, an operating system, or both at different stages, and platforms differ in how they divide the work. But the orchestration only functions because hardware holds up its end.

"Enumeration is a hardware state machine." It is not. No single piece of hardware runs the algorithm; there is no PCIe enumeration FSM in an Endpoint.

The accurate framing is a division of responsibility:

6. RTL — The Configuration Access Front End

Whatever else a function does, it must be able to accept a configuration access, perform it once, and return exactly one response.

// Implementation-defined internal configuration request. NOT a PCIe packet
// format and NOT a standardised configuration-header layout.
typedef struct packed {
  logic                     write;
  logic [CFG_ADDR_W-1:0]    addr;    // internal register-file offset
  logic [CFG_DATA_W-1:0]    wdata;
  logic [CFG_DATA_W/8-1:0]  be;      // byte enables
} cfg_req_t;

// SYNTHESIZABLE. One-outstanding configuration access front end.
// NOT PCIe protocol logic: no packet parsing, no routing, no config-space
// layout. This is the request/response ownership problem in isolation.
module cfg_front_end #(
  parameter int CFG_ADDR_W = 12,
  parameter int CFG_DATA_W = 32
) (
  input  logic                        clk,
  input  logic                        rst_n,
 
  // Inbound configuration request.
  input  logic                        req_valid,
  output logic                        req_ready,
  input  logic                        req_write,
  input  logic [CFG_ADDR_W-1:0]       req_addr,
  input  logic [CFG_DATA_W-1:0]       req_wdata,
  input  logic [CFG_DATA_W/8-1:0]     req_be,
 
  // Internal register-file port. The register file performs the access ONCE,
  // on the cycle it asserts reg_ack, and may take any number of cycles.
  output logic                        reg_sel,
  output logic                        reg_write,
  output logic [CFG_ADDR_W-1:0]       reg_addr,
  output logic [CFG_DATA_W-1:0]       reg_wdata,
  output logic [CFG_DATA_W/8-1:0]     reg_be,
  input  logic                        reg_ack,
  input  logic [CFG_DATA_W-1:0]       reg_rdata,
  input  logic                        reg_unsupported,
 
  // Outbound response.
  output logic                        rsp_valid,
  input  logic                        rsp_ready,
  output logic [CFG_DATA_W-1:0]       rsp_rdata,
  output logic                        rsp_unsupported
);
 
  typedef enum logic [1:0] {
    CFG_IDLE   = 2'b00,   // no access outstanding
    CFG_ACCESS = 2'b01,   // presented to the register file, awaiting ack
    CFG_RESP   = 2'b10    // response held for the requester
  } cfg_state_e;
 
  cfg_state_e                 state_q;
  logic                       write_q;
  logic [CFG_ADDR_W-1:0]      addr_q;
  logic [CFG_DATA_W-1:0]      wdata_q;
  logic [CFG_DATA_W/8-1:0]    be_q;
  logic [CFG_DATA_W-1:0]      rdata_q;
  logic                       unsup_q;
 
  // Ready only when nothing is outstanding. Depends on state, never on
  // req_valid — so there is no combinational path from a requester's valid
  // back to its own ready.
  assign req_ready = (state_q == CFG_IDLE);
  wire   req_accept = req_valid && req_ready;
 
  // Held stable for the whole access. The register-file contract is that the
  // access is performed exactly once, on the reg_ack cycle — which is why a
  // level-held reg_write cannot produce a duplicate side effect here.
  assign reg_sel   = (state_q == CFG_ACCESS);
  assign reg_write = (state_q == CFG_ACCESS) && write_q;
  assign reg_addr  = addr_q;
  assign reg_wdata = wdata_q;
  assign reg_be    = be_q;
 
  // rsp_valid is derived from REGISTERED state and never observes rsp_ready.
  assign rsp_valid       = (state_q == CFG_RESP);
  assign rsp_rdata       = rdata_q;
  assign rsp_unsupported = unsup_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      state_q <= CFG_IDLE;
      write_q <= 1'b0;
      addr_q  <= '0;
      wdata_q <= '0;
      be_q    <= '0;
      rdata_q <= '0;
      unsup_q <= 1'b0;
    end else begin
      case (state_q)
        CFG_IDLE: begin
          if (req_accept) begin
            write_q <= req_write;
            addr_q  <= req_addr;
            wdata_q <= req_wdata;
            be_q    <= req_be;
            state_q <= CFG_ACCESS;
          end
        end
 
        CFG_ACCESS: begin
          if (reg_ack) begin
            rdata_q <= reg_rdata;          // meaningful for reads
            unsup_q <= reg_unsupported;
            state_q <= CFG_RESP;
          end
        end
 
        CFG_RESP: begin
          // Response held until taken. Nothing writes rdata_q or unsup_q in
          // this state, so the response is stable across any stall.
          if (rsp_ready) state_q <= CFG_IDLE;
        end
 
        // The enum uses three of four encodings. A default is provided so an
        // upset state resolves to a safe one rather than being unreachable
        // in simulation and undefined in silicon.
        default: state_q <= CFG_IDLE;
      endcase
    end
  end
 
endmodule

Classification: synthesizable, with a conceptual typedef for the request structure.

What it models: ownership of a single configuration access from acceptance to response — the smallest complete unit of hardware participation in enumeration.

What it teaches — three things:

1. One outstanding access makes duplication impossible by construction. req_ready is low outside CFG_IDLE, so a second request cannot be accepted while the first is unresolved. There is no state in which two accesses could be confused.
2. Exactly one response per accepted request. The only path to CFG_RESP is through CFG_ACCESS, and the only path out is a completed handshake. A request cannot produce two responses, and it cannot produce none.
3. The write side effect happens once. reg_write is level-held while awaiting reg_ack, which would be a duplicate-write hazard under a different register-file contract. The contract is stated in the port comment and asserted in §7 — an unstated contract here is a real and common bug.

Deliberately simplified: one access at a time, so throughput is limited by register-file latency; no distinction between kinds of unsupported access; no error classification; and no representation of how the access arrived or where it will return to.

Production implication: a real configuration path must parse and validate the incoming access, determine whether this function is the target, forward it onward if the component is a hierarchy element, apply byte enables and access-size rules, produce a properly formed response with the correct completion semantics, and handle accesses that arrive before the function is ready — which is Chapter 7.2's subject.

7. Assertions

// SVA over cfg_front_end. Implementation invariants for THIS design —
// not PCIe protocol requirements.
 
// OWNERSHIP — P1: no request is accepted while one is outstanding. The
// property that makes "one access at a time" a checkable claim rather than
// an assumption about how the requester behaves.
property p_single_outstanding;
  @(posedge clk) disable iff (!rst_n)
  (state_q != CFG_IDLE) |-> !req_ready;
endproperty
a_single_outstanding : assert property (p_single_outstanding);
 
// CONSERVATION — P2: every accepted request eventually reaches a response
// state, and reaching one requires an accepted request. Catches a response
// fabricated from nothing, which would return data for an access the
// requester never made.
property p_response_needs_accept;
  @(posedge clk) disable iff (!rst_n)
  (!rsp_valid ##1 rsp_valid) |-> $past(state_q == CFG_ACCESS && reg_ack);
endproperty
a_response_needs_access : assert property (p_response_needs_accept);
 
// STABILITY — P3: the response is stable while the requester stalls. A
// response that changes under stall lets the requester sample data belonging
// to no access it issued.
property p_response_stable_under_stall;
  @(posedge clk) disable iff (!rst_n)
  (rsp_valid && !rsp_ready) |=> (rsp_valid && $stable(rsp_rdata)
                                 && $stable(rsp_unsupported));
endproperty
a_response_stable : assert property (p_response_stable_under_stall);
 
// SAFETY — P4: the register-file selection is asserted only while an access
// is genuinely outstanding. Catches reg_sel leaking into IDLE or RESP, which
// would present a stale address to the register file.
property p_reg_sel_only_in_access;
  @(posedge clk) disable iff (!rst_n)
  reg_sel |-> (state_q == CFG_ACCESS);
endproperty
a_reg_sel_scoped : assert property (p_reg_sel_only_in_access);
 
// SAFETY — P5: the captured access is immutable while it is being performed.
// Catches a write path that updates addr_q or wdata_q mid-access, which would
// apply the write to a different location than the one requested.
property p_access_immutable;
  @(posedge clk) disable iff (!rst_n)
  (state_q == CFG_ACCESS) |=> ($stable(addr_q) && $stable(wdata_q)
                               && $stable(write_q) && $stable(be_q));
endproperty
a_access_immutable : assert property (p_access_immutable);
 
// SAFETY — P6: exactly one acknowledged access per accepted request. This is
// the duplicate-write property: if the register file could acknowledge twice
// while reg_sel is held, a single configuration write would take effect twice.
property p_one_ack_per_access;
  @(posedge clk) disable iff (!rst_n)
  (state_q == CFG_ACCESS && reg_ack) |=> (state_q != CFG_ACCESS);
endproperty
a_one_ack : assert property (p_one_ack_per_access);
 
// SAFETY — P7: reset clears ownership. A front end that comes out of reset
// believing an access is outstanding will never accept another one.
property p_reset_clears_ownership;
  @(posedge clk)
  !rst_n |=> (state_q == CFG_IDLE && !rsp_valid);
endproperty
a_reset_clears : assert property (p_reset_clears_ownership);
 
// LIVENESS — P8: every accepted request eventually produces a response.
// ASSUMPTION, stated explicitly: this holds only if the register file
// eventually acknowledges and the requester eventually accepts. A permanently
// unresponsive register file is not a bug in this module, so the environment
// must constrain reg_ack and rsp_ready to be eventually asserted.
property p_access_completes;
  @(posedge clk) disable iff (!rst_n)
  req_accept |-> s_eventually (rsp_valid && rsp_ready);
endproperty
a_access_completes : assert property (p_access_completes);

P6 is the one that ordinary testing misses. A register file that pulses reg_ack twice while reg_sel is held applies a configuration write twice. For most reads that is harmless and invisible; for a write with a side effect — a counter, a state change, a clear-on-write field — it is a real corruption whose symptom appears far from its cause. The property is cheap and the bug is expensive.

P2 and P3 are a matched pair in the same way Chapter 6.2's were: P2 bounds responses from below (none fabricated), P3 bounds them from above (none mutated). Together they mean the requester sees exactly the response its access produced.

8. Verification

Enumeration verification works at three levels, and each catches a class the others cannot.

Configuration-response level. Does each function respond correctly to an individual access? Monitors observe request and response handshakes and the register-file port; the scoreboard predicts, from its own model of the register map, what each access should return and which writes should take effect. This is where P1–P7's failures show up.

Topology level. Does the environment discover the set of functions it was built to contain? The testbench holds a model of which locations are populated — expanded considerably in Chapter 7.3 — and checks the discovered set against it. This catches functions that never respond and functions that respond when they should not.

Sequence level. Does the host advance only on well-formed responses? This is the subtlest and the most valuable. A host that treats an incomplete, undefined, or mistimed response as a valid one will build a wrong description of the topology and proceed confidently. The check is that every state transition in the enumeration flow is justified by a response that actually satisfied its preconditions.

Scenarios for the front end:

• Single read, single write. Baseline: one response each, correct data, correct side effect.
• Back-to-back accesses. Request presented in the cycle the previous response is taken. Verify the second is accepted and not merged with the first.
• Requester stalls the response. Hold rsp_ready low for varying durations. Verify stability (P3) and that no new request is accepted (P1).
• Register file delays acknowledgement. Vary reg_ack latency from one cycle to long. Verify the access is held stable throughout (P5).
• Double acknowledgement. Deliberately pulse reg_ack twice. Verify P6 fires — a property never observed to fail has not been shown to work.
• Unsupported access. Verify the response carries the indication and that no write side effect occurred.
• Reset mid-access. Reset in CFG_ACCESS and in CFG_RESP. Verify ownership clears (P7) and the next request is accepted normally.
• Request presented during reset. Verify it is not accepted and not silently retained.

9. Debugging — The Enumeration Ladder

Reference scenario: the operating system does not see an Endpoint.

The instinct is to call this an enumeration bug and start reading host code. That is almost always the wrong first move, because "the OS does not see it" is the only symptom that a dozen unrelated faults produce.

The method is the boundary-first approach from Chapter 3.5, applied to a longer path: find the last stage where the expected event is observed, then the first stage where it disappears or becomes invalid. The fault is between them.

The ladder, in order:

1. Is the physical Link available? Not "is the cable in" — is the Link established and usable. If not, nothing above this can work and the investigation belongs to Module 17.
2. Is the upstream path healthy? Every element between the root and this device must be present and forwarding. From Chapter 4.4: a failure upstream hides everything behind it.
3. Is a configuration probe generated at all? If the host never probes this location, no hardware fault explains the absence. This is frequently where the answer is, and it is frequently checked last.
4. Does the probe reach the Endpoint? Forwarding through the hierarchy is hardware's responsibility. A probe that is issued but not delivered points at a hierarchy element, not at the Endpoint.
5. Does the Endpoint receive it? Distinguish "arrived at the component" from "reached the configuration front end." Between them sits the readiness gating of Chapter 7.2, which may legitimately be blocking it.
6. Does the Endpoint produce a valid response? A response that is malformed, incomplete, or carries undefined content is worse than no response, because the host may act on it.
7. Does the response reach the host? The return path is a separate path with separate failure modes. A response generated but not delivered looks identical from the host to a response never generated.
8. Does the host interpret the response as "present"? A well-formed response that the host reads as absence is a real and distinct fault class.
9. Is the hierarchy identity and path correct? Owned by Chapters 7.4–7.6. A function found but recorded at the wrong location may be unreachable later.
10. Did resource assignment succeed? Owned by Chapter 7.8. A discovered, identified, numbered function that never received resources is present in the topology description and unusable — which from a user's perspective is indistinguishable from missing.

10. Common Misconceptions

• "Enumeration is BIOS software." It is orchestrated by host software — firmware, an operating system, or both depending on platform and stage. It only works because hardware becomes accessible at the right time, responds correctly, exposes stable state, and forwards accesses. An Endpoint's RTL can break enumeration completely without containing any enumeration logic.
• "A trained Link means the device is enumerated." A usable Link is a prerequisite, not a result. Link establishment says a path exists; enumeration is everything that happens afterwards to make the function discoverable, addressable, resourced, and enabled.
• "If the device exists physically, software knows about it." Nothing in the hardware announces itself. The host learns of a function only by directing a configuration access at its location and interpreting the response — which is why a device can be perfectly functional and completely invisible.
• "Enumeration assigns transaction semantics." It establishes identity, resources, and enablement. What a transaction means — its type, ordering, and completion rules — is protocol behaviour defined independently and owned by Modules 10 onward.
• "Bus, device, and function numbers exist before discovery." Hierarchy identity is established as the topology is explored, because the topology is not known in advance. How that is done is Chapters 7.4–7.6; the point here is that identity is an output of enumeration, not an input to it.
• "Enumeration and configuration access are the same thing." Configuration access is the mechanism (Chapter 7.7). Enumeration is a process that uses it repeatedly with different targets and intents. Confusing them makes it impossible to say which layer a failure is in.
• "Enumeration is just reading an identifier." Reading identity is one stage of six. A function that has been identified but not numbered, resourced, and enabled is not usable — and the later stages fail in ways that look identical to the earlier ones from a user's perspective.
• "Resource assignment is part of discovery." Discovery determines what exists. Assignment determines what each function gets. They are separated because assignment cannot begin until the full set of requirements is known, which requires discovery to have completed.
• "An OS-visible failure means the Endpoint RTL is wrong." It means one of ten stages failed. The Endpoint's response logic is one of them, and several of the others are far more commonly the cause — starting with whether a probe was ever generated for that location.

11. Understanding Check

12. What's Next

This chapter's ladder starts with "is the Link available" and immediately moves on. That step deserves more than a yes or no.

Chapter 7.2 — Power-Up takes it up: enumeration cannot begin until the relevant path and function are sufficiently initialised to accept configuration accesses, and that is a dependency chain rather than a single reset-deassert event. Power, clocks, several reset domains, the PHY, the Link, and the function's own configuration state each become ready at their own time — and a function that becomes visible before all of them are satisfied fails enumeration in a way that looks like a software problem.