Power-Up — The Dependency Chain Before Enumeration Can Begin

Chapter 7.1's debug ladder began with "is the Link available?" and moved on in a sentence. That step conceals most of the interesting engineering.

A device that answers yes to that question can still fail enumeration completely — and the failure will look like a software problem.

What must be true before the host can meaningfully begin PCIe enumeration?

1. The Dependency Chain

Several things must become true, and they do not become true together:

Power is valid. Supplies are within their operating range and stable.

Reset is in a defined state. Which reset, and in which domain, is §2's subject — and it is where the "single event" intuition first breaks.

Clocks are available. Reference and derived clocks are running and stable enough for the logic that depends on them.

The local controller is initialised. Internal state machines, buffers, and control registers reach their defined post-reset values.

Physical-layer establishment progresses. The PHY reaches a state where signalling is possible.

The Link becomes usable. The two components establish a working connection at some width and rate.

Function configuration state becomes available. The function's configuration registers hold defined values and can be read and written coherently.

The host begins probing. Only now can a configuration access produce a meaningful answer.

2. "Reset Released" Is Not One Event

This is the section that most changes how an engineer reads a power-up problem.

A PCIe function typically has several initialisation domains that are released at different times, by different logic, for different reasons:

Platform or global reset. Applied across the component or the board, generally tied to supply validity.

Controller reset. The PCIe controller's own digital logic.

PHY reset. The physical-layer block, which frequently has its own sequencing tied to clock and supply readiness distinct from the digital side.

Local function reset. The application function behind the PCIe interface — the thing the device actually does.

Software-visible configuration state. Configuration registers reaching their defined values, which may be a distinct step from the controller being out of reset.

"Is it out of reset?" is not a well-formed question about a PCIe function. "Which of its resets, and released by what?" is.

Why this matters concretely. A design that gates configuration visibility on the wrong domain — controller reset released, say, rather than configuration state initialised — will answer configuration accesses correctly in the sense of producing a response, and the content will be whatever the registers happened to contain at that moment. The host reads it, believes it, and builds a wrong description.

That failure is invisible in every place an engineer normally looks. The Link is up. The controller is running. Accesses are answered. Nothing reports an error.

3. Clock and Reset Crossing

The domains in §2 are not only separated in time — they are frequently separated in clock domain, which turns a sequencing question into a physical-design one.

Reset release must be synchronised. A reset deasserting asynchronously with respect to the clock that samples it can leave different flops in a domain seeing different release cycles. The standard response is asynchronous assertion with synchronised release, and the consequence for this chapter is that even a single reset does not release everywhere in the same cycle.

PHY and core clocks are frequently different. Readiness indications crossing between them need proper synchronisation, and a naively sampled multi-bit status can be observed in a state it never actually held.

Nothing may become externally visible before its internal state is coherent. This is the design rule the whole chapter serves. If configuration registers are readable while their initialisation is still propagating, the values read are not the design's intent — they are a snapshot of a transition.

Reset values must be defined and reached. A register whose reset value is undefined, or which is initialised by a sequence that has not completed, has no meaningful content to expose regardless of when it is read.

This is not a clock-domain-crossing tutorial, and the mechanisms above are general digital-design practice rather than PCIe-specific. They appear here because power-up is exactly where they bite, and because a designer who treats readiness as a single Boolean will get them wrong.

4. Microarchitecture — A Local Initialisation Sequencer

The structure:

RESET → WAIT_PHY → WAIT_LINK → INIT_CFG → READY, with a FAULT state reachable from any of them.

Each transition waits on one prerequisite, and — a detail worth designing in deliberately — waits for it to be stable, not merely momentarily asserted.

5. The Sequence in Time

Cycle 3 is the interesting one. The Link is available. By the coarse reading — "the Link is up, so the device is ready" — a probe should be answered here. Configuration initialisation has not completed, so anything read would be a snapshot of a transition.

Cycle 4 is where a bad design fails. The probe arrives. A design that gates on link_available answers it, with whatever the configuration registers contain. A design that gates on function_ready holds it for two cycles and then answers correctly.

The waveform values are illustrative — real intervals between these events are vastly longer and vary by implementation. What the figure shows is the order and the gap, which is the part that generalises.

6. RTL — The Initialisation Sequencer

// SYNTHESIZABLE. Local initialisation sequencer for one PCIe function.
// NOT the LTSSM and NOT a link-training state machine — link_available is an
// abstract INPUT here, produced elsewhere. State names are deliberately
// unlike any protocol state name.
module func_init_seq #(
  // A prerequisite must hold continuously for this many cycles before the
  // sequencer advances. Guards against a momentarily-asserted indication
  // advancing initialisation that then proceeds on a false premise.
  parameter int STABLE_CYCLES = 4
) (
  input  logic clk,
  input  logic rst_n,
 
  // Abstract prerequisites. Not PCIe signals — stand-ins supplied by the
  // surrounding design, already synchronised into this clock domain.
  input  logic phy_ready,
  input  logic link_available,
  input  logic cfg_init_done,
  input  logic fatal_init_error,
 
  output logic cfg_access_enable,
  output logic function_ready,
  output logic init_error
);
 
  typedef enum logic [2:0] {
    INIT_RESET     = 3'd0,
    INIT_WAIT_PHY  = 3'd1,
    INIT_WAIT_LINK = 3'd2,
    INIT_CFG       = 3'd3,
    INIT_READY     = 3'd4,
    INIT_FAULT     = 3'd5
  } init_state_e;
 
  // Width holds STABLE_CYCLES-1. $clog2(1) is 0, which would be a zero-width
  // signal, so the degenerate case is handled explicitly rather than left to
  // produce an elaboration error nobody expected.
  localparam int STB_W = (STABLE_CYCLES <= 1) ? 1 : $clog2(STABLE_CYCLES);
 
  initial begin
    if (STABLE_CYCLES < 1) $fatal(1, "STABLE_CYCLES must be at least 1");
  end
 
  init_state_e      state_q;
  logic [STB_W-1:0] stable_q;
 
  // The prerequisite for the CURRENT state. Note that later states re-test
  // the earlier prerequisites: losing the PHY while waiting for the Link
  // must not let the counter keep accumulating toward an advance.
  logic cond_met;
  always_comb begin
    case (state_q)
      INIT_WAIT_PHY  : cond_met = phy_ready;
      INIT_WAIT_LINK : cond_met = phy_ready && link_available;
      INIT_CFG       : cond_met = phy_ready && link_available && cfg_init_done;
      default        : cond_met = 1'b0;      // assigned on every path: no latch
    endcase
  end
 
  wire stable_reached = (stable_q == STB_W'(STABLE_CYCLES - 1));
  wire advance        = cond_met && stable_reached;
 
  // Outputs are derived from REGISTERED state only. Nothing here observes a
  // prerequisite combinationally, so readiness cannot glitch with an input.
  assign function_ready    = (state_q == INIT_READY);
  assign cfg_access_enable = (state_q == INIT_READY);
  assign init_error        = (state_q == INIT_FAULT);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      state_q  <= INIT_RESET;
      stable_q <= '0;
    end else if (fatal_init_error) begin
      // A fatal error wins from any state and is TERMINAL until reset.
      // Recovering silently would let a function that failed initialisation
      // present itself as healthy on a later attempt.
      state_q  <= INIT_FAULT;
      stable_q <= '0;
    end else begin
      // Stability counter: accumulates while the current prerequisite holds,
      // clears the moment it does not.
      if (!cond_met)            stable_q <= '0;
      else if (!stable_reached) stable_q <= stable_q + 1'b1;
 
      case (state_q)
        INIT_RESET     : begin state_q <= INIT_WAIT_PHY;  stable_q <= '0; end
        INIT_WAIT_PHY  : if (advance) begin state_q <= INIT_WAIT_LINK; stable_q <= '0; end
        INIT_WAIT_LINK : if (advance) begin state_q <= INIT_CFG;       stable_q <= '0; end
        INIT_CFG       : if (advance) begin state_q <= INIT_READY;     stable_q <= '0; end
        INIT_READY     : ;                 // terminal on the success path
        INIT_FAULT     : ;                 // terminal until reset
        default        : state_q <= INIT_FAULT;   // unreachable encoding
      endcase
    end
  end
 
endmodule

Classification: synthesizable.

What it models: the local decision about when a function's configuration state may be exposed, expressed as an ordered dependency chain rather than a single condition.

What it teaches — three things:

1. Prerequisites must be re-tested, not remembered. INIT_WAIT_LINK requires phy_ready and link_available. A sequencer that only checked the newest condition would keep counting toward an advance while the foundation underneath it had gone away.
2. Stability is not the same as assertion. A single-cycle indication is not evidence that a condition holds. STABLE_CYCLES makes the qualification explicit and parameterisable, and the counter clears on any loss rather than pausing.
3. A fatal error is terminal. Silently recovering would let a function that failed initialisation present itself as healthy later, with no record that anything went wrong. Requiring a reset to clear it makes the failure impossible to miss.

Deliberately simplified — and one of these matters a great deal: the sequencer is forward-only, so there is no path for a prerequisite lost after INIT_READY. That is a real gap and it is deliberate: what a design should do when the Link goes away under a ready function is a genuine architectural decision involving in-flight traffic and error reporting, and it depends on behaviour Module 18 owns. Also simplified: prerequisites are assumed already synchronised into this clock domain, and there is no distinction between kinds of fatal error.

Production implication: a real controller must define behaviour when a prerequisite is lost after readiness, synchronise each prerequisite from its own domain, distinguish recoverable from fatal initialisation errors, expose the initialisation state for debug — the difference between "stuck in INIT_WAIT_LINK" and "stuck somewhere" is most of a debug session — and coordinate with whatever platform logic sequences supplies and clocks.

7. RTL — The Configuration Visibility Gate

The sequencer decides when. Something must enforce it.

// SYNTHESIZABLE. Gates configuration accesses until the function is ready.
// The local contract chosen here: accesses are HELD (not accepted), not
// rejected and not answered with placeholder content.
module cfg_visibility_gate #(
  parameter int CFG_ADDR_W = 12,
  parameter int CFG_DATA_W = 32
) (
  input  logic                      function_ready,
 
  // From the fabric-facing side.
  input  logic                      up_valid,
  output logic                      up_ready,
  input  logic                      up_write,
  input  logic [CFG_ADDR_W-1:0]     up_addr,
  input  logic [CFG_DATA_W-1:0]     up_wdata,
 
  // To the configuration front end of Chapter 7.1.
  output logic                      dn_valid,
  input  logic                      dn_ready,
  output logic                      dn_write,
  output logic [CFG_ADDR_W-1:0]     dn_addr,
  output logic [CFG_DATA_W-1:0]     dn_wdata
);
 
  // Before readiness: dn_valid stays low, so the front end never sees the
  // access; up_ready stays low, so the access is not consumed. It waits.
  assign dn_valid = up_valid && function_ready;
  assign up_ready = dn_ready && function_ready;
 
  assign dn_write = up_write;
  assign dn_addr  = up_addr;
  assign dn_wdata = up_wdata;
 
endmodule

Classification: synthesizable.

What it models: the enforcement point between "the access arrived" and "the function may answer it."

Which contract to choose. Holding is one option. A design could instead reject the access, or answer it with a defined indication that the function is unavailable. Each is a different externally visible behaviour and one must be chosen explicitly, because the host's response to each differs. What PCIe requires of a component in this situation is a protocol question this chapter does not answer; the point here is that leaving it implicit is not an option — an ungated design has made the choice "answer with whatever is in the registers," which is the one contract that is definitely wrong.

Deliberately simplified: purely combinational, so it adds no pipelining; one contract, with no configurability; and no counting of accesses arriving early, which a real design would expose because "the host probed before we were ready" is a valuable debug fact.

Production implication: a real gate must implement whatever behaviour the protocol requires for its situation, report early accesses for debug, and handle readiness that can be lost.

8. Assertions

// SVA over func_init_seq and cfg_visibility_gate. Implementation invariants
// for THESE designs — not PCIe protocol requirements.
 
// SAFETY — P1: readiness implies every prerequisite held at the moment it was
// granted. The core claim of the whole chapter, made checkable.
property p_ready_implies_prereqs;
  @(posedge clk) disable iff (!rst_n)
  (state_q != INIT_READY ##1 state_q == INIT_READY)
    |-> $past(phy_ready && link_available && cfg_init_done);
endproperty
a_ready_prereqs : assert property (p_ready_implies_prereqs);
 
// SAFETY — P2: no configuration access is accepted before readiness. The
// property the visibility gate exists to guarantee; without it, a function
// answers with content that is a snapshot of a transition.
property p_no_access_before_ready;
  @(posedge clk) disable iff (!rst_n)
  !function_ready |-> !(up_valid && up_ready);
endproperty
a_no_early_access : assert property (p_no_access_before_ready);
 
// SAFETY — P3: a fatal initialisation error prevents readiness and is
// terminal. Catches a design that recovers silently and then presents a
// function that failed initialisation as healthy.
property p_fault_is_terminal;
  @(posedge clk) disable iff (!rst_n)
  (state_q == INIT_FAULT) |=> (state_q == INIT_FAULT && !function_ready);
endproperty
a_fault_terminal : assert property (p_fault_is_terminal);
 
// SAFETY — P4: readiness is monotonic outside reset. This is the property
// the visibility gate silently depends on. If a future edit adds a path that
// clears readiness, this fires — rather than the gate becoming quietly wrong.
property p_ready_monotonic;
  @(posedge clk) disable iff (!rst_n)
  function_ready |=> function_ready;
endproperty
a_ready_monotonic : assert property (p_ready_monotonic);
 
// LEGALITY — P5: the sequencer only takes forward transitions, or goes to
// FAULT. Catches a next-state expression that skips a stage, which would
// grant readiness without one of its prerequisites ever being tested.
property p_legal_transitions;
  @(posedge clk) disable iff (!rst_n)
  $changed(state_q) |-> (state_q == INIT_FAULT)
                     || (state_q == init_state_e'($past(state_q) + 3'd1));
endproperty
a_legal_transitions : assert property (p_legal_transitions);
 
// CORRECTNESS — P6: an advance requires the prerequisite to have held for the
// full stability window. Catches a counter that is reset in the wrong place,
// which would let a momentary indication advance initialisation.
property p_advance_requires_stability;
  @(posedge clk) disable iff (!rst_n)
  (state_q inside {INIT_WAIT_PHY, INIT_WAIT_LINK, INIT_CFG} && $changed(state_q))
    |-> $past(cond_met) && $past(stable_reached);
endproperty
a_advance_stable : assert property (p_advance_requires_stability);
 
// SAFETY — P7: reset returns the sequencer to its initial state with
// readiness deasserted. A sequencer that comes out of reset believing it is
// ready exposes uninitialised configuration state immediately.
property p_reset_clears_ready;
  @(posedge clk)
  !rst_n |=> (state_q == INIT_RESET && !function_ready && !cfg_access_enable);
endproperty
a_reset_clears : assert property (p_reset_clears_ready);
 
// STABILITY — P8: the gate does not alter an access in flight. Catches a
// gate that lets payload change while the access waits for readiness.
property p_gate_payload_stable;
  @(posedge clk) disable iff (!rst_n)
  (up_valid && !up_ready) |=> (up_valid && $stable(up_addr) && $stable(up_wdata)
                               && $stable(up_write));
endproperty
a_gate_stable : assert property (p_gate_payload_stable);

P4 is the most valuable property here and the least obvious. It asserts something the sequencer already does — readiness never falls — for the benefit of a different module that depends on it. It costs one line and it converts an undocumented cross-module assumption into a failure that appears the moment someone violates it.

P1 catches the failure this chapter exists to prevent. A sequencer that grants readiness on link_available alone passes every functional test in which configuration initialisation happens to complete first. P1 fires the moment the ordering differs, which is exactly the condition that varies between silicon revisions and operating conditions.

P6 catches a counter bug with a nasty signature. If the stability counter is cleared in the wrong place, a prerequisite that asserts for one cycle can advance the sequencer. The design then works whenever the indication happens to be clean and fails intermittently when it is not — the hardest class of bug to reproduce.

9. Verification

Monitors observe: each prerequisite, the sequencer state, readiness, the gate's upstream and downstream handshakes, and the fault indication.

The scoreboard independently predicts: the expected state at each cycle from its own model of the dependency chain and the stability rule, and whether an access presented at a given cycle should have been accepted. It must not derive expected readiness by reading function_ready.

The core of this chapter's verification is sequencing permutation, because the ordering between prerequisites is exactly what varies in real systems and exactly what a naive design gets wrong.

Scenarios:

• Nominal order. PHY, then Link, then configuration initialisation, each well separated. The baseline.
• PHY ready early, everything else late. Verify no premature advance — the case that catches a sequencer keying off the first available indication.
• PHY ready late. Everything else asserted while PHY is still low. Verify the sequencer waits, because §6's cond_met re-tests earlier prerequisites.
• Link delayed well past PHY. A long gap. Verify nothing times out or advances.
• Configuration initialisation delayed past Link. The chapter's signature scenario — the exact window where "the Link is up so the device is ready" produces a wrong answer.
• Prerequisite lost while waiting. Assert a prerequisite, drop it before the stability window completes. Verify the counter clears and the advance does not occur.
• Single-cycle prerequisite glitch. A one-cycle assertion. Verify no advance (P6). If the environment cannot generate sub-cycle behaviour, a one-cycle pulse is the closest available representation and is worth running.
• Fatal error before readiness. In each waiting state. Verify INIT_FAULT and that readiness never asserts.
• Fatal error after readiness. Verify the design enters INIT_FAULT and readiness drops — which, note, violates P4. That is the correct outcome: it demonstrates the monotonicity assumption is conditional on no fatal error, and it is the scenario that would expose the visibility gate's hidden dependency.
• Reset during initialisation. In each state. Verify a clean restart from INIT_RESET.
• Reset after readiness. Verify readiness drops and the full chain is re-walked rather than short-circuited.
• Configuration probe arriving too early. Present an access in each pre-ready state. Verify it is held, not accepted (P2), its payload is preserved (P8), and it completes correctly once readiness arrives.

Coverage should include: every ordering permutation of the three prerequisites; each prerequisite lost at each point in its stability window; fatal error injected in each state; reset in each state; and an early access presented in each pre-ready state.

10. Debugging

Reference scenario: the Link appears up, but enumeration still fails.

This is the chapter's diagnostic moat, because it is the symptom where the most common mental model actively misleads.

What the observation establishes. A usable Link means the physical path works, both PHYs are functioning, and the two components negotiated a connection. That is a genuine and substantial elimination — the entire physical domain is out of scope.

What it does not establish. Anything at all about whether the function behind that Link has coherent configuration state.

Link-up is necessary, not sufficient.

Hypothesis classes, roughly ordered:

Configuration state is not initialised. The most common. The Link came up before the function's configuration registers reached their defined values, and either the accesses are being held indefinitely or — worse — answered with undefined content.

The function is held in reset. A reset domain that was never released, or released by logic that is itself waiting on something. §2's point made concrete: the controller is out of reset and the function is not.

A clock or reset crossing problem. A readiness indication crossing domains without proper synchronisation, observed in a state it never held. Characteristically intermittent and sensitive to conditions — which is the signature that distinguishes it from a plain sequencing bug.

The configuration front end is disabled. The gate is not opening, because the sequencer is stuck. This is entirely diagnosable if the sequencer state is observable, and nearly undiagnosable if it is not — which is why exposing it was a production implication in §6.

The host probes before local readiness and does not retry. A race rather than a fault in either party. Whether the host retries, and for how long, is platform behaviour.

Platform or root sequencing. Something upstream released this device before its own dependencies were satisfied.

The measurement that splits these fastest. Observe the sequencer state and the arrival of the first configuration access together.

• Stuck in a waiting state → the named prerequisite never arrived. The question becomes why, and the state names the answer.
• In INIT_FAULT → initialisation failed, and it failed for a reason the design detected. Find what asserted the error.
• In INIT_READY, no access ever arrives → the fault is upstream. Go to Chapter 7.1's ladder, rung 3.
• In INIT_READY, accesses arrive and are answered, device still absent → this chapter is exonerated. The fault is in the response content, the return path, or interpretation — rungs 6 through 10.

Two observations, four outcomes, each pointing at a different domain. That is what the sequencer state buys, and it is the argument for exposing it.

11. Common Misconceptions

• "Power-up ends when reset deasserts." Reset release is one event in a chain that includes clocks stabilising, the PHY reaching a usable state, the Link being established, and configuration state reaching defined values. Each takes its own time and depends on the ones before it.
• "Link up means the device is ready for all accesses." A usable Link means a path exists between two components. It says nothing about whether the function at the far end has coherent configuration state to expose. This is the misconception the chapter exists to correct.
• "All PCIe logic shares one reset." A function typically has several initialisation domains — platform, controller, PHY, local function, configuration state — released at different times by different logic. "Is it out of reset?" is not a well-formed question about a PCIe function.
• "PHY ready means function ready." The PHY reaching a usable state is one prerequisite among several, and it is one of the earliest. Treating it as permission is the specific error §5's waveform illustrates at cycle 2.
• "Configuration registers can be exposed early because software will wait." Software cannot distinguish a response containing transitional content from a correct one. It will not wait; it will read, believe, and proceed. Exposing early does not defer the problem — it converts a visible failure into a silent misconfiguration.
• "Enumeration begins a fixed time after power-on." Platforms differ in when and how enumeration starts, and the interval between power-on and the first probe is not something a device may rely on. A design whose correctness depends on the host being slow enough is not correct.
• "The LTSSM and the endpoint initialisation FSM are the same thing." The LTSSM is protocol-defined and governs link training and link states (Module 18). A local initialisation sequencer is implementation-defined and decides when this component's configuration state may be exposed. It consumes link availability as an input.
• "A power-up bug must be analog or PHY-related." The chain includes reset domains, clock domains, synchronisation, sequencing logic, and configuration initialisation — all digital, all in RTL, and all capable of producing "the Link is up but the device does not enumerate."
• "Reset sequencing is board-level logic." Board logic sequences supplies and global reset. Everything after that — which internal domains release when, what depends on what, and when configuration state becomes exposable — is inside the component and is the RTL designer's responsibility.

12. Understanding Check

13. What's Next

The function is ready. Configuration accesses will now be accepted and answered correctly. Nothing has yet arrived.

Chapter 7.3 — Device Discovery takes up the other side: how the host determines whether a function exists at a location in the hierarchy at all. It is a directed probe-and-response process rather than a broadcast, it must handle responses that never come, and it introduces a hardware problem this chapter did not have — a late response from a probe that already gave up must never be attributed to the next one.