Skip to content

UCIe · Module 10

Endpoint Connectivity

What connected means when a PCIe endpoint sits behind a UCIe link — two independent readiness domains, the endpoint visibility vector, configuration lifetime versus transport lifetime, outstanding-request ownership, completion matching, exactly-once semantic acceptance under replay, and the integration error class.

Chapter 10.1 drew the ownership boundary. Chapter 10.2 built the mechanism that moves a PCIe transaction across it exactly once. Both were about the path.

This chapter is about the device at the end of it. A PCIe endpoint on a conventional link sits in a slot: the link trains, the root complex sees a device, software enumerates it. Replace the slot with a die-to-die link and the same words stop meaning the same things. "The link is up" is now a statement about a UCIe link, and it says considerably less about the endpoint than a PCIe engineer's instinct expects.

The bug this chapter exists to prevent is one line of RTL. It is assign endpoint_visible = ucie_link_up; and it produces a device that appears to software before it can answer.

1. The One-Sentence Model

Endpoint connectivity is the composition of two independent readiness domains: UCIe must be able to transport packets, and the PCIe endpoint must be in a state where those packets have valid semantic meaning.

Read that twice, because both halves are load-bearing and the word independent is the whole chapter.

A physical link can be up while endpoint logic is not ready. An endpoint can be fully configured internally while the UCIe transport is unavailable. These are different states, they are reached by different sequences, they are lost for different reasons, and neither one implies the other in either direction.

Everything that follows — the readiness vector, the reset-domain question, the outstanding-request table, the failure taxonomy — is a consequence of taking that independence seriously instead of collapsing it into one bit.

2. What "Connected" Stops Meaning

On a conventional PCIe link, several things happen close enough together that engineers learn them as one event:

  • the electrical link trains;
  • the data link layer reaches its operational state;
  • the endpoint's configuration space becomes accessible;
  • the root complex can issue configuration reads and get answers.

They are not the same event even on PCIe, but the coupling is tight enough that "link up" is a usable shorthand.

Over UCIe the coupling is gone. The UCIe link reaching its operational state — Chapter 8.6's ACTIVE, reached through the bring-up flow of Chapter 8.5 — is a statement about a transport that has no idea what a PCIe endpoint is. The Adapter negotiated parameters, the PHY trained lanes, CRC and retry are armed. None of that involved the PCIe function.

3. The Connectivity Stack

Five layers, each with its own notion of "ready":

LayerReady meansOwned by
PCIe endpoint functionconfiguration state valid, function able to answerthe PCIe engine
PCIe/UCIe mapping boundaryobjects can be accepted and represented (Ch 10.2)integration logic
UCIe D2D Adapterflit transfer, CRC/retry, credits armed (Ch 9.4, 9.5)the Adapter
UCIe PHYlanes trained at the negotiated rate (Ch 8.3, 8.4)the PHY
Package linkthe dies are physically joinedthe package

Connectivity exists only when every layer that a given operation depends on has reached a compatible state — and "compatible" is per-operation, not global.

That last qualifier matters more than it looks. A configuration read needs the mapping layer, the Adapter, and a PCIe function able to answer configuration. A memory write to a BAR needs all of that plus the BAR to have been programmed. A message-signalled interrupt needs the endpoint to have been enabled to send one. Treating "connected" as a single global predicate flattens all of these into a bit that is correct for none of them.

4. The Readiness Vector

The first piece of RTL, and the one that determines whether the rest of the design has a chance.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Illustrative integration state — not a UCIe or PCIe normative structure.
// Each bit has a different owner, a different lifetime, and a different
// reason to be lost. That is exactly why they are separate flops.
 
logic ucie_operational_q;        // UCIe transport can move flits
logic tunnel_ready_q;            // mapping layer can accept/emit objects
logic endpoint_cfg_valid_q;      // endpoint config path is answerable
logic endpoint_function_ready_q; // the function itself is out of reset, sane
logic endpoint_visible_q;        // registered composition of the above

Architecture. Five bits rather than one because five different subsystems can independently fail to be ready, and because the diagnosis of a bring-up failure is precisely "which of these was low". A single endpoint_ready erases the only information a debugger needs.

State. Five single-bit flops in the integration block. ucie_operational_q is sampled from the Adapter's link state and belongs to the UCIe link epoch — it goes away when the link goes away. tunnel_ready_q belongs to the same epoch plus its own resource state (Chapter 10.1's queue). endpoint_cfg_valid_q and endpoint_function_ready_q belong to the PCIe function's lifetime, which as §7 develops is a different lifetime. endpoint_visible_q is derived and registered.

Cycle behaviour. Each source bit updates on its own event: link state transitions, queue reset/ready, function reset deassertion, configuration-path initialisation complete. There is deliberately no cycle on which they are all guaranteed to move together.

Contract. The host-facing side of the integration block uses endpoint_visible_q to decide whether this function is presented as reachable. The request-acceptance path of §11 uses it as one of its vetoes. Nothing else may read the raw bits to make a go/no-go decision — they are for composition and diagnosis.

Failure. Collapse these into one bit and every bring-up failure produces the same symptom — a device that does not answer — with no way to tell a PHY problem from an unprogrammed function. Post-silicon debug then costs days instead of minutes.

DV. Cover each bit's rise and fall independently, and cover the orderings: UCIe ready first, endpoint ready first, and each losing readiness while the other holds. §22's coverage model is built on exactly those orderings.

5. Deriving Visibility

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Illustrative. The composition is the point; the registration is the detail.
assign endpoint_visible_d =
    ucie_operational_q        &&
    tunnel_ready_q            &&
    endpoint_cfg_valid_q      &&
    endpoint_function_ready_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) endpoint_visible_q <= 1'b0;
  else        endpoint_visible_q <= endpoint_visible_d;
end

Three things about this that are easy to get wrong.

It is a conjunction, and the direction is asymmetric. Visibility should assert only when everything agrees, but it should deassert the instant any contributor drops. A registered AND does both, which is why the naive combinational form is not obviously wrong — it is the use of the result that needs care.

Real implementations usually qualify it further. A single registered AND will glitch visibility if any contributor is itself glitchy, and a device that appears and disappears within a few cycles is worse than one that never appears, because software may have started something in between. Common qualifications: require the conjunction to hold for a defined interval before asserting, hold visibility through a short transport disturbance if the integration contract says the function survives it, or make the assertion edge a deliberate event rather than a level. Which of these is correct is an integration contract decision, not a fact about UCIe or PCIe, and §7 and §8 are where that decision gets made.

Deassertion is not the inverse of assertion. Coming up carefully and going down carefully are different problems. Assertion can afford to be slow. Deassertion, when it represents a transport that has genuinely gone, must be fast enough that the acceptance path in §11 stops taking new work — but as §17 develops, fast to stop accepting is not the same as fast to forget.

6. The Bug This Chapter Exists To Prevent

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// WRONG — the single most common integration bug in this whole module.
assign endpoint_visible = ucie_link_up;

The UCIe link being operational says the transport can move flits. It says nothing about whether a PCIe function exists at the far end, whether its configuration path has been initialised, whether the mapping layer has resources, or whether the function is out of its own reset.

What the host does with this. It sees a device and begins acting on it: configuration reads to identify it, configuration writes to program it, and eventually memory traffic. Every one of those is now aimed at a target that may be unable to respond correctly.

What the symptoms look like, and why each one is a different debugging morning:

SymptomWhat actually happened
Configuration read times outRequest reached a mapping layer or function with nothing to answer it.
Malformed or default responseSomething answered from an uninitialised configuration path.
Configuration request silently droppedMapping layer had no resources; nothing recorded that it existed.
Device appears, then disappearsVisibility tracked a transport bit that flapped during late training.
Device enumerates with wrong identitySoftware read the configuration path before it held real values.

The fourth is the worst of them, and it is worth naming precisely. Software-visible device identity must not flap. An operating system that has begun enumerating a function, assigned it resources, and possibly bound a driver to it does not gracefully tolerate that function vanishing because a transport bit went low for eight microseconds. The correct engineering response is not to make recovery faster; it is to make sure the transport's momentary state was never what determined identity in the first place.

7. Configuration State and Its Three Lifetimes

Now the question that the naive design never asks: when the UCIe link retrains, what happens to the endpoint's configuration?

Before answering, classify what "configuration" even covers. Three lifetimes, and they are genuinely different:

ClassExamplesLifetimeSet by
Static capabilitywhich functions exist, what they are capable of, fixed identity valuesdesign/strap — outlives everythingthe implementation
Enumerated / programmedaddress-window assignments, enable bits, interrupt configurationfrom software programming until something defined resets itsystem software
Transport-lifetimewhether flits can currently move, credits, replay statethe UCIe link epoch (Ch 9.4, 9.5)the Adapter

The third column is where the design decision lives. Static capability plainly should not care about the link. Transport state plainly must be re-established with the link — Chapter 9.5 §13 already established that credits are re-advertised, and Chapter 9.4 that replay state is re-baselined.

The middle row is the one nobody has decided.

Physical transport lifetime and PCIe function lifetime are separate architectural decisions. Coupling them is a choice, and it must be a deliberate one made against the software-visible behaviour you intend.

8. Reset Domains Are the Real Question

Restate §7 as the question a designer actually has to answer at integration time:

Which reset domain does the PCIe function live in?

Three broad arrangements, each defensible, each with consequences:

The function is in the transport's reset domain. A UCIe recovery resets the function. Simple to build and easy to reason about — there is one lifetime, so nothing can be inconsistent. The cost is that any transport disturbance is a software-visible device reset, with everything that implies for drivers and for whatever the device was doing.

The function is in its own domain, independent of the transport. Configuration survives a transport recovery; the device is briefly unreachable and then answers again with its state intact. This is what a slot-attached device's behaviour most resembles, and it is usually what platform software expects. The cost is that the two domains can now disagree, and §17's freeze logic exists precisely to manage the window where they do.

A hybrid, defined per state class. Transport-coupled state re-initialises; programmed state persists; a defined subset is invalidated because it cannot meaningfully survive. Most real systems land here, and the engineering work is writing down which class each register is in rather than discovering it during bring-up.

The point is not which one is right. It is that the answer must be written down before RTL is written, because every mechanism downstream — the freeze logic, the outstanding-request policy, the recovery sequence, the scoreboard — is a consequence of it.

9. Wrong Design — Clear Everything on Retrain

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// WRONG (as an unconsidered default) — the transport's recovery is being
// treated as the function's reset, without that being an intended contract.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n || ucie_recovery_event)
    endpoint_cfg_q <= '0;          // all programmed configuration discarded
end

Architecture. Nothing here is inherently illegal — arrangement one in §8 is a legitimate choice. What makes this wrong is that it is a side effect rather than a decision: the designer wired a convenient event to a convenient reset and did not ask what software would observe.

Cycle behaviour. One pulse on a transport recovery event and every programmed register in the function returns to its reset value, in the same cycle, with no announcement to anything above.

Failure. A temporary, fully recovered transport disturbance — the kind Chapter 9.4's retry mechanism exists to make invisible — becomes a host-visible device reconfiguration. The device is still present and answering, but its programmed state is gone. Software's model of the device and the device's actual state have diverged, and nothing reported an error, because at the transport layer nothing went wrong. Subsequent memory traffic aimed at a previously programmed address window now reaches a function that no longer claims that window.

The severity comes from the silence. A device that disappears is at least a visible event. A device that quietly forgets what it was told is a device that will misbehave later, far from the cause.

DV. Inject a transport recovery in the middle of an otherwise clean traffic run, after full programming, and check the configuration model against the device afterwards. If the intended contract is retention, this fires immediately. If the intended contract is reset, the scoreboard must model that too — the test is not "nothing changed", it is "what changed is what the contract says changes".

10. Outstanding Requests: Who Owns the Obligation

The endpoint has accepted PCIe requests. The UCIe link fails. What happens?

This is where integration stops being about bits and starts being about obligations. A non-posted PCIe request creates an expectation: something is owed back. That expectation exists in the requester's model of the world, and it does not evaporate because a die-to-die link had a bad moment.

Four questions have to have answers, and they are the design, not the protocol:

Where does the request state live? In the endpoint function, in the mapping layer, or split between them. If it is split, both halves need the same lifetime rules, or a recovery will clear one and leave the other.

Who owes the completion? The endpoint function has committed to producing one. If the transport cannot carry it, the obligation still exists — it is merely unfulfillable at this instant.

Is the request retained across recovery, or abandoned? Retention means the function continues and completes when transport returns. Abandonment means something must eventually resolve the requester's expectation. Both are viable; silence is not.

What resolves the requester if the request is abandoned? PCIe defines mechanisms for a requester whose completion never arrives. This chapter does not restate their timing, thresholds, or error semantics, because doing so accurately requires the Base Specification text and getting it approximately right would be worse than not stating it.

11. The Outstanding-Request Table

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Illustrative integration state. This is the mapping/endpoint boundary's
// own record of work in flight — NOT a PCIe normative structure, and the
// tag field here is a local handle, discussed below.
typedef struct packed {
  logic                     valid;
  logic [TAG_W-1:0]         tag;      // see the note on identity below
  logic [REQ_META_W-1:0]    meta;     // what is needed to route the response
} outstanding_req_t;
 
outstanding_req_t outstanding_q [MAX_OUTSTANDING];

Architecture. Accepting a request is a commitment, and a commitment with no record is a commitment that cannot be honoured. This table is the record. It exists because the response arrives later, possibly after a transport recovery, and something must connect it back to what asked.

State. MAX_OUTSTANDING entries, each with per-request lifetime — allocated at acceptance, freed at resolution. Note what that means: this is the only structure in the chapter whose lifetime is neither the link epoch nor the function's configuration lifetime. It is its own thing, and §18's table makes that explicit.

Cycle behaviour. On an accepted request, allocate a free entry and populate it. On a matching response, look up, use the metadata, and free. On an abandonment event, free with the abandonment path taken rather than silently.

Contract. The response path depends absolutely on this table: without a live entry, an arriving response has no meaning and nowhere to go. The acceptance path in §12 depends on it too, since a full table is a reason to stop accepting.

Failure. Undersize it and throughput collapses in a way that looks like a link problem. Fail to free entries and it fills permanently — the endpoint stops accepting and looks hung, with a perfectly healthy link. Free entries on the wrong event and responses arrive for entries that no longer exist.

DV. Cover the table empty, at one entry, at full, and at full-with-a-request-waiting. Cover a recovery occurring at each of those occupancies. The interesting bugs live at full and during recovery, not in the middle of the range.

12. Accepting a Request Is a Three-Way Veto

Chapter 10.2 §9 built a four-way veto on accepting an object into the transport. Here is its counterpart at the endpoint boundary, and it is a different set of vetoes for a different reason.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Illustrative. Three independent conditions, three different owners.
assign request_ready =
    endpoint_visible_q      &&   // semantics are meaningful right now
    outstanding_space       &&   // this side can record the obligation
    tunnel_resources_ready;      // the response has a path back

Architecture. Each term guards a different failure. endpoint_visible_q guards meaning — accepting a request the function cannot semantically service. outstanding_space guards accountability — accepting an obligation with nowhere to record it. tunnel_resources_ready guards the return path — accepting work whose result cannot get home.

Cycle behaviour. Combinational over three registered inputs, sampled at the acceptance handshake. As in Chapter 9.5 §7, if the decision is made on the pre-update value, the assertions that check it must use $past accordingly.

Contract. The upstream requester relies on this as genuine backpressure — not-ready means retry later, not dropped. The outstanding table relies on the second term to guarantee the allocation it is about to perform will succeed.

Failure. Each omission fails differently, and this is why the three-way form earns its keep:

Omitted termSymptomWhere you debug it
endpoint_visible_qRequest accepted into a function that cannot service it — timeout or nonsense responsethe endpoint function
outstanding_spaceRequest accepted with no record; response later unmatchedthe response path, far from the cause
tunnel_resources_readyRequest serviced; response cannot be sent; requester times outthe transport, blaming the link

Three different subsystems, three different engineers, one missing term. That is the argument for writing the conjunction explicitly rather than deriving readiness from whatever bit is nearest.

DV. Cross each term's deassertion with a request attempt. Also cover the case where two are low at once — the design must not depend on the order in which they recover.

13. Wrong RTL — Accept Without Recording

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// WRONG — the request is taken and passed on with no accounting entry.
always_ff @(posedge clk) begin
  if (req_valid && endpoint_visible_q) begin
    fn_req_valid_q <= 1'b1;
    fn_req_data_q  <= req_data;      // straight into function logic
    // no outstanding_q allocation, no metadata retained
  end
end

Architecture. This is Chapter 10.1's ownership bug in its endpoint form. The request has crossed a boundary and the boundary kept no record that it happened.

Cycle behaviour. One cycle, one handoff, no allocation. The RTL is shorter and looks cleaner, which is exactly why it survives review.

Failure. The request enters function logic and is serviced. Then the response appears — and there is nothing to match it to. Whatever routing metadata the response needed to get back to the right requester was in req_data and was never saved. Depending on downstream design, the response is dropped, sent to a default destination, or matched against a stale entry from an unrelated earlier request.

The last case is the dangerous one, and it is worth being explicit: a response matched to the wrong outstanding entry delivers correct-looking data to a requester that asked for something else. No error is signalled, because every field is individually well-formed. This is the endpoint-layer sibling of Chapter 10.2 §7's metadata/payload misalignment, and it has the same character — structurally invalid, individually valid, silent.

DV. The assertion in §15 catches this the first time a response arrives. More importantly, the scoreboard of §19 catches the routing error, which the assertion cannot: an assertion can confirm that some live entry matched, but only a model that knows who asked can confirm it was the right one.

14. Matching a Response

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Illustrative — the lookup, not a PCIe completion decoder.
// PCIe's own rules for what constitutes a valid completion for a given
// request are protocol semantics and are not restated here.
logic                     match_hit;
logic [IDX_W-1:0]         match_idx;
 
always_comb begin
  match_hit = 1'b0;
  match_idx = '0;
  for (int i = 0; i < MAX_OUTSTANDING; i++) begin
    if (outstanding_q[i].valid && (outstanding_q[i].tag == rsp_tag)) begin
      match_hit = 1'b1;
      match_idx = i[IDX_W-1:0];
    end
  end
end

Architecture. The response carries an identifier; the table holds the context. This lookup is the only thing that turns an arriving response back into an answer for a specific requester.

State. Purely combinational over the table. The table entry it selects has per-request lifetime; the match itself has none.

Cycle behaviour. Evaluated when a response arrives; the entry is consumed and freed in the same handshake.

Contract. Two contracts meet here. Upward, the requester relies on receiving its own answer. Downward, the table relies on the response path to free exactly the entry it matched — freeing a different index leaks one entry and prematurely releases another.

Failure. No match means an unmatchable response, which is either a lost request record (§13) or a lifetime error. Multiple matches mean duplicate live identifiers, which is worse: the loop above silently takes the last one, so the design does not fail, it just answers the wrong requester. If your identifier space is meant to be unique among live entries, assert that it is rather than assuming it.

DV. Cover match, no-match, and — deliberately, by injection — the multiple-match case. An implementation that has never seen two live entries with the same identifier has never tested its uniqueness assumption.

15. SVA — A Response Requires a Live Request

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Illustrative. Local invariant, verification-only.
property p_response_matches_live_request;
  @(posedge clk) disable iff (!rst_n)
    rsp_fire |-> match_hit;
endproperty
a_response_matches_live_request: assert property (p_response_matches_live_request);
 
// Uniqueness of live identifiers, if that is the intended contract.
property p_outstanding_ids_unique;
  @(posedge clk) disable iff (!rst_n)
    rsp_fire |-> $onehot0(match_vec);   // match_vec[i] = valid && tag equal
endproperty
a_outstanding_ids_unique: assert property (p_outstanding_ids_unique);

What the first catches. Every mechanism that lets a response exist without a corresponding request record: the §13 acceptance bug, entries freed early, entries cleared by a recovery that should not have cleared them, and responses generated spuriously.

What the second catches. Identifier reuse while an earlier user is still live — a lifetime bug that produces wrong-requester delivery rather than an error.

What neither can catch, and this is the standing rule of the curriculum. These assertions prove that a live entry matched. They cannot prove it was the correct entry, because correctness there is defined by who issued the request, and that information is not present at this interface. Only the scoreboard in §19 holds it.

Assertions prove local invariants. Only a scoreboard proves that the right requester got the right answer.

16. Transport Replay Must Not Become a Second Request

This is Chapter 10.2 §12's exactly-once property, stated at the endpoint where its consequences are concrete.

The transport may deliver the same object twice. Chapter 9.4 established why: a retry happens when corruption is detected or when a confirmation is lost, and in the second case the original arrived perfectly. The receiver therefore genuinely sees the same transport object a second time, and this is correct transport behaviour, not a bug.

The endpoint function must not see a second semantic request.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Illustrative duplicate suppression at the semantic boundary.
// The transport identity space is the transport's own — see 10.2 §11.
logic [XPORT_ID_W-1:0] last_accepted_id_q;
logic                  have_accepted_q;
 
assign is_duplicate = have_accepted_q &&
                      (rx_xport_id == last_accepted_id_q);
 
assign semantic_accept = rx_object_complete && !is_duplicate;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    have_accepted_q    <= 1'b0;
    last_accepted_id_q <= '0;
  end else if (semantic_accept) begin
    have_accepted_q    <= 1'b1;
    last_accepted_id_q <= rx_xport_id;
  end
end

Architecture. A gate between transport arrival and semantic acceptance whose only job is to make retransmission invisible upward. The single-entry form shown is the simplest case that teaches the mechanism; the real width of the suppression window is set by how far the transport's retry can reach back, which is Chapter 9.4's unconfirmed-window question.

State. last_accepted_id_q and have_accepted_q have UCIe link epoch lifetime — the identity space is re-baselined with the link, so carrying a stale value across a re-establishment is itself a bug.

Cycle behaviour. Acceptance is gated in the same cycle the object completes. Nothing downstream of this gate ever observes the duplicate.

Contract. The function relies on this absolutely. Everything below it is free to retransmit; nothing above it may observe that anything happened.

Failure. Omit it and a retransmitted object becomes a second PCIe request at the endpoint. For a memory write that is the same write executed twice. For a non-posted request it is two obligations where the requester expects one, so the outstanding table either allocates twice for one expectation or reports a duplicate identifier. None of it produces a CRC error, because the transport behaved exactly as designed.

DV. The injection that matters is a lost confirmation on an object that arrived intact — the only stimulus that produces a genuine duplicate. A regression that only injects corruption exercises replay but never exercises this gate, and the entire mechanism ships untested.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Illustrative — the property that makes the requirement explicit.
property p_replay_no_duplicate_accept;
  @(posedge clk) disable iff (!rst_n)
    (rx_object_complete && is_duplicate) |-> !semantic_accept;
endproperty
a_replay_no_duplicate_accept: assert property (p_replay_no_duplicate_accept);

17. Bring-Up, As a Sequence

A UCIe link reports operational to the mapping layer. The endpoint function reports its configuration path valid. The mapping layer then reports endpoint visible to the host integration logic, which issues a configuration read that travels through the UCIe link and mapping layer to the endpoint function, whose completion returns by the same path. Normal traffic follows.Endpoint bring-up over UCIe — conceptual orderingHost sideUCIe linkMappingEndpoint fnoperationalcfg path validvisibleconfig readcarriedconfig readcompletioncompletionprogrammednormal traffic
Figure 1 — endpoint bring-up as a composition of two readiness domains rather than one sequence. The UCIe link reaching its operational state is the transport's own event and involves no PCIe logic. Only when the mapping layer and the endpoint's configuration path have also reported ready does the integration layer present the function as visible; only then does host configuration traffic have a valid target. The ordering shown is one architecturally reasonable arrangement, not a normative sequence.

Two properties of this picture are worth stating explicitly.

The first two messages are independent. Neither causes the other. They are drawn in an order because a diagram needs one, but the design must work when they arrive in the opposite order — and §22's coverage requires testing both.

Visibility is an event with consequences. Everything after the third message is only legal because visibility asserted. That is the whole argument for making it a composition rather than a copy of a transport bit.

18. When the Transport Goes: Freeze, Not Amnesia

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Illustrative. The distinction the whole section rests on:
// stop taking new work, without forgetting existing work.
logic endpoint_freeze_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n)                      endpoint_freeze_q <= 1'b0;
  else if (!ucie_operational_q)    endpoint_freeze_q <= 1'b1;
  else if (recovery_complete)      endpoint_freeze_q <= 1'b0;
end
 
// Freeze participates in acceptance; it does NOT clear the table.
assign request_ready =
    endpoint_visible_q && outstanding_space &&
    tunnel_resources_ready && !endpoint_freeze_q;

Architecture. When the transport is unavailable, continuing to accept requests is accumulating obligations that cannot be met. Freezing stops the accumulation. What freezing must not do is discard the obligations already accumulated, because those belong to requesters who are still waiting.

State. endpoint_freeze_q has UCIe link epoch lifetime — set on loss, cleared on defined recovery completion. Note it is deliberately not a pure inversion of ucie_operational_q: clearing requires a recovery-complete event, so the design does not resume mid-recovery on a transient assertion.

Cycle behaviour. Set immediately on transport loss — this is one place where fast deassertion matters. Cleared only on the defined completion event.

Contract. Upstream sees backpressure rather than errors. The outstanding table's contract is that freeze does not touch it. §10's policy decision determines what does eventually resolve those entries.

Failure. Two opposite failures, and both are common. Freeze that also clears the table produces §9's silent amnesia with the additional harm that requesters are left permanently waiting. Freeze that never engages produces an outstanding table that fills during an outage, so recovery arrives to find no capacity and the endpoint appears hung after the link is demonstrably healthy again.

DV. Inject transport loss at several outstanding-table occupancies, including full. Check that the table contents are byte-identical across the outage when the contract says retention, and that every entry reaches its defined resolution when the contract says abandonment.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Illustrative.
property p_no_accept_while_frozen;
  @(posedge clk) disable iff (!rst_n)
    endpoint_freeze_q |-> !req_accept_fire;
endproperty
a_no_accept_while_frozen: assert property (p_no_accept_while_frozen);
 
// The complementary property, and the one people forget to write.
property p_freeze_preserves_outstanding;
  @(posedge clk) disable iff (!rst_n)
    ($rose(endpoint_freeze_q) && outstanding_valid_vec != '0)
      |=> (outstanding_valid_vec == $past(outstanding_valid_vec));
endproperty
a_freeze_preserves_outstanding: assert property (p_freeze_preserves_outstanding);

The second property is the one worth arguing for. It encodes §8's intended contract into the RTL's verification, which means that if someone later wires a convenient reset to a convenient event — §9's bug — the assertion fires at the moment the coupling is introduced rather than during a post-silicon investigation months later.

19. Three Error Domains, Not Two

The organising idea of this chapter's debug methodology.

PCIe semantic errors. The protocol behaviour was wrong: a malformed request, a response that does not satisfy its request, a violation of PCIe's rules. These are diagnosed with PCIe knowledge and are largely independent of what transport is underneath.

UCIe transport errors. The path failed: CRC errors, retry exhaustion, training failure, credit violations. Chapters 8.x and 9.x built the machinery, and these are diagnosed with UCIe knowledge.

Integration errors. Both subsystems are individually correct and their combined state is inconsistent. The link is fine, the endpoint is fine, and the system does not work.

20. Integration Failure Signatures

The table to have open during bring-up. Each row is a class-three failure, and each points somewhere specific.

SignatureMost likely causeFirst thing to check
UCIe operational, endpoint never visibleA readiness contributor never assertedWhich of §4's bits is low
Endpoint enumerates, then disappearsVisibility coupled to a transport bit that flappedWhether visibility has any qualification (§5)
Configuration works, then device forgets its programmingReset-domain coupling (§9)What clears the configuration registers
Request reaches mapping layer, no response everFunction not semantically ready, or table fullendpoint_visible_q and table occupancy at the time
Same request appears to execute twiceReplay boundary error (§16)Whether duplicate suppression exists and is armed
Response arrives, matches nothingRequest accepted without a record (§13), or entry freed earlyTable occupancy trace across the request's lifetime
Response arrives, matches the wrong requesterIdentifier reuse or wrong-index free (§14)Uniqueness assertion, and the index used to free
Endpoint hung after a healthy recoveryTable filled during the outage; freeze absent (§18)Occupancy at the moment of transport loss

The last row deserves a note because it is the most misleading symptom in the list. Everything measurable is healthy — link operational, no errors, endpoint present — and the device does nothing. The cause is in the past, during an outage that has already been recovered from and possibly already forgotten by the logs.

21. State Lifetimes, Side by Side

The table this chapter exists to produce, in the format Chapter 10.2 §14 established.

StateAllocated / set whenLost whenOn transport recoveryOn function reset
Static capabilitydesign/strapneverunaffectedunaffected
ucie_operational_qlink reaches operationallink leaves itre-establishedunaffected
tunnel_ready_qmapping resources readyresources or link lostre-establishedunaffected
endpoint_cfg_valid_qconfig path initialisedfunction resetcontract decision (§8)cleared
Programmed configurationsoftware writesfunction reset — or a recovery, if coupledcontract decision (§8)cleared
Outstanding entryrequest acceptedresolved or abandonedretained, or abandoned by defined path (§10)resolved per contract
endpoint_freeze_qtransport lostrecovery completecleared by the recovery itselfcleared
Duplicate-suppression IDfirst semantic acceptlink epoch endsre-baselined (§16)cleared
Diagnostics / sticky statusfirst eventbroader reset onlysurvive (Ch 8.1 §18)survive

Three rows carry the weight. Two rows say "contract decision" and that is not evasion — it is the chapter's central finding, and a design that has not written those two rows down explicitly has not finished its architecture. Outstanding entries have their own lifetime, belonging to neither the link epoch nor the configuration lifetime. And diagnostics survive everything short of a broad reset, because the whole value of §20's table depends on evidence outliving the event.

22. The Endpoint Scoreboard

Assertions cover §15's local invariants. This is what proves the system.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
host_requests[]        — every semantic request the host issued, with identity
endpoint_accepts[]     — every request the endpoint semantically accepted
outstanding_model      — expected table contents, by allocation and resolution
responses[]            — every response, with the request it claims to answer
transport_events[]     — retries, recoveries, link epochs, from the UCIe model

Five structures because there are five things that can independently be wrong. The checks:

Exactly one semantic accept per host request. This is §16's property at system level, and only a model that holds transport_events[] alongside endpoint_accepts[] can distinguish "the transport retransmitted" (correct) from "the endpoint accepted twice" (corrupt).

Exactly one response per request that requires one, delivered to the requester that issued it. The second clause is the part assertions cannot check.

No response without a request. The system-level form of §15, catching spurious responses that happen to match a live entry by accident.

No outstanding entry silently lost across a transport disturbance, unless the contract's defined abandonment path was taken — in which case the abandonment must be observable, not inferred from the entry's absence.

The outstanding model matches the device's table at every quiescent point. Divergence at a quiescent boundary is the cleanest possible evidence of a lifetime bug, and it usually shows up long before the resulting misbehaviour would.

The reason five structures are needed rather than one is the same reason Chapter 10.2 needed two reference models: the semantic model tells you that something was duplicated, lost, or misrouted; the transport model tells you why. A duplicate accept and a duplicate transport delivery look identical in a semantic-only log, and one of them is correct behaviour.

23. Coverage

Not vanity coverage. The equivalence classes that actually contain the bugs, crossed the way §4's independence demands.

Readiness orderings. UCIe operational first then endpoint ready; endpoint ready first then UCIe operational; both in the same window. The design must not depend on order, and only crossing them proves it.

Recovery position relative to configuration. Recovery before any configuration; recovery mid-configuration; recovery after full configuration. The third is where §9's bug becomes visible; the second is where partially-programmed state produces its own surprises.

Recovery position relative to traffic. Recovery with the outstanding table empty, partially occupied, and full. The full case is where §18's freeze either works or does not, and it is routinely omitted.

Response arrival relative to recovery. Response arrives before the disturbance, during the outage, and after recovery completes. The third is the one that exercises whether the entry survived.

Transport events. Retry with an intact object and a lost confirmation — the §16 injection. Retry with genuine corruption. Multiple retries on the same object.

Reset independence. Function reset without a transport event, and a transport event without a function reset — if §8's contract permits them to be independent. If it does not, cover the coupling instead, and assert it.

The crosses that matter. Bring-up phase × recovery position × table occupancy. That three-way cross is small enough to be achievable and contains, in my experience of this class of design, nearly every integration bug worth finding.

24. Debug Checklist

In order. Each step is cheap, and the order is chosen so that the first failing step localises the problem.

  1. Is the UCIe link operational — Chapter 8.6's state, not an inference?
  2. Is the mapping layer ready, with resources available?
  3. Is the endpoint's configuration path initialised and answerable?
  4. Does endpoint_visible_q reflect the conjunction — and if not, which bit is low?
  5. Was the configuration request accepted at the boundary, or refused?
  6. Did the endpoint allocate outstanding state for it?
  7. Was a UCIe retry involved anywhere in this transaction's life?
  8. Did the semantic request appear at the function exactly once?
  9. Is the outstanding entry still live, and does it hold the metadata expected?
  10. Did a transport recovery occur — and did anything in the endpoint change that should not have?
  11. Which reset domain changed? This single question resolves a large fraction of class-three failures.
  12. Given all of the above: is this a PCIe failure, a UCIe failure, or an integration failure?

Question 11 is the highest-yield one on the list, and it is the one least often asked, because it requires knowing §8's contract — which is why §8 insists that contract be written down before RTL.

25. Common Misconceptions

"UCIe link up means the PCIe endpoint is up." The transport can move flits. That is orthogonal to whether a PCIe function exists, is out of reset, has an initialised configuration path, and has mapping resources. §6 lists five distinct symptoms of believing otherwise.

"Endpoint state and UCIe state are one state machine." They are two, with different transitions, different triggers, and different lifetimes. §4 exists because they are independent; §21 exists because their lifetimes differ.

"A UCIe retrain should reset all PCIe configuration." It may, if that is the intended contract — arrangement one in §8. As an unexamined default it converts a recovered transport disturbance into a silent host-visible reconfiguration, which is §9's bug.

"Transport replay may legitimately be visible as a repeated PCIe request." It may not. Retry is a transport mechanism, and a retransmitted object becoming a second semantic request means a memory write executes twice. §16 is the gate that prevents it.

"A response can be accepted without outstanding-request state." Then it cannot be routed to the requester that asked, and the best case is that it is dropped. The realistic case is that it matches something stale and delivers a correct-looking answer to the wrong requester.

"An endpoint disappearing during UCIe recovery is inevitable." It is a design outcome, not a law. Whether software observes the device vanish is determined by §8's reset-domain decision and §5's qualification of visibility.

"If the packet bits are correct, the integration is correct." Every bug in this chapter occurs with perfectly correct packet bits. That is the definition of §19's third error class.

"There are only PCIe errors and UCIe errors — integration itself cannot fail." This is the misconception that produces the longest debugging sessions, because it means nobody owns the failure that is actually occurring.

26. Understanding Check

27. Summary and What Comes Next

Endpoint connectivity is the composition of two independent readiness domains: UCIe must be able to transport packets, and the PCIe endpoint must be in a state where those packets have valid semantic meaning.

The mechanisms: a readiness vector rather than a bit, because five subsystems fail independently and the first debug question is which one. Visibility as a registered, qualified composition, because software-visible device identity must not flap. Reset domains decided deliberately — the chapter's central finding, since whether configuration survives a transport recovery is an integration contract, not a protocol fact, and a design that has not written it down has not finished its architecture. An outstanding-request table with its own lifetime, belonging to neither the link epoch nor the configuration lifetime. A three-way veto on acceptance, each term failing in a different subsystem. Freeze without amnesia on transport loss. And duplicate suppression at the semantic boundary, because a retransmitted object becoming a second request means a memory write executes twice with nothing reporting an error.

The idea that organises the debugging: there is a third error class. Not PCIe, not UCIe, but integration — both subsystems correct, their composition inconsistent. Every bug here is one, which is why the tools of both specialisms show green while the system does nothing.

An endpoint can now exist correctly behind a UCIe transport. The harder half of the problem is on the other side: the root complex must discover that endpoint, assign it resources, route traffic to it, preserve its configuration across disturbances, and hide the fact that the device is physically on another die:

Browse the full path on the UCIe tutorials index.