UCIe · Module 10
Endpoint Connectivity
What connected means when a PCIe endpoint sits behind a UCIe link — two independent readiness domains, the endpoint visibility vector, configuration lifetime versus transport lifetime, outstanding-request ownership, completion matching, exactly-once semantic acceptance under replay, and the integration error class.
Chapter 10.1 drew the ownership boundary. Chapter 10.2 built the mechanism that moves a PCIe transaction across it exactly once. Both were about the path.
This chapter is about the device at the end of it. A PCIe endpoint on a conventional link sits in a slot: the link trains, the root complex sees a device, software enumerates it. Replace the slot with a die-to-die link and the same words stop meaning the same things. "The link is up" is now a statement about a UCIe link, and it says considerably less about the endpoint than a PCIe engineer's instinct expects.
The bug this chapter exists to prevent is one line of RTL. It is assign endpoint_visible = ucie_link_up; and it produces a device that appears to software before it can answer.
1. The One-Sentence Model
Endpoint connectivity is the composition of two independent readiness domains: UCIe must be able to transport packets, and the PCIe endpoint must be in a state where those packets have valid semantic meaning.
Read that twice, because both halves are load-bearing and the word independent is the whole chapter.
A physical link can be up while endpoint logic is not ready. An endpoint can be fully configured internally while the UCIe transport is unavailable. These are different states, they are reached by different sequences, they are lost for different reasons, and neither one implies the other in either direction.
Everything that follows — the readiness vector, the reset-domain question, the outstanding-request table, the failure taxonomy — is a consequence of taking that independence seriously instead of collapsing it into one bit.
2. What "Connected" Stops Meaning
On a conventional PCIe link, several things happen close enough together that engineers learn them as one event:
- the electrical link trains;
- the data link layer reaches its operational state;
- the endpoint's configuration space becomes accessible;
- the root complex can issue configuration reads and get answers.
They are not the same event even on PCIe, but the coupling is tight enough that "link up" is a usable shorthand.
Over UCIe the coupling is gone. The UCIe link reaching its operational state — Chapter 8.6's ACTIVE, reached through the bring-up flow of Chapter 8.5 — is a statement about a transport that has no idea what a PCIe endpoint is. The Adapter negotiated parameters, the PHY trained lanes, CRC and retry are armed. None of that involved the PCIe function.
3. The Connectivity Stack
Five layers, each with its own notion of "ready":
| Layer | Ready means | Owned by |
|---|---|---|
| PCIe endpoint function | configuration state valid, function able to answer | the PCIe engine |
| PCIe/UCIe mapping boundary | objects can be accepted and represented (Ch 10.2) | integration logic |
| UCIe D2D Adapter | flit transfer, CRC/retry, credits armed (Ch 9.4, 9.5) | the Adapter |
| UCIe PHY | lanes trained at the negotiated rate (Ch 8.3, 8.4) | the PHY |
| Package link | the dies are physically joined | the package |
Connectivity exists only when every layer that a given operation depends on has reached a compatible state — and "compatible" is per-operation, not global.
That last qualifier matters more than it looks. A configuration read needs the mapping layer, the Adapter, and a PCIe function able to answer configuration. A memory write to a BAR needs all of that plus the BAR to have been programmed. A message-signalled interrupt needs the endpoint to have been enabled to send one. Treating "connected" as a single global predicate flattens all of these into a bit that is correct for none of them.
4. The Readiness Vector
The first piece of RTL, and the one that determines whether the rest of the design has a chance.
// Illustrative integration state — not a UCIe or PCIe normative structure.
// Each bit has a different owner, a different lifetime, and a different
// reason to be lost. That is exactly why they are separate flops.
logic ucie_operational_q; // UCIe transport can move flits
logic tunnel_ready_q; // mapping layer can accept/emit objects
logic endpoint_cfg_valid_q; // endpoint config path is answerable
logic endpoint_function_ready_q; // the function itself is out of reset, sane
logic endpoint_visible_q; // registered composition of the aboveArchitecture. Five bits rather than one because five different subsystems can independently fail to be ready, and because the diagnosis of a bring-up failure is precisely "which of these was low". A single endpoint_ready erases the only information a debugger needs.
State. Five single-bit flops in the integration block. ucie_operational_q is sampled from the Adapter's link state and belongs to the UCIe link epoch — it goes away when the link goes away. tunnel_ready_q belongs to the same epoch plus its own resource state (Chapter 10.1's queue). endpoint_cfg_valid_q and endpoint_function_ready_q belong to the PCIe function's lifetime, which as §7 develops is a different lifetime. endpoint_visible_q is derived and registered.
Cycle behaviour. Each source bit updates on its own event: link state transitions, queue reset/ready, function reset deassertion, configuration-path initialisation complete. There is deliberately no cycle on which they are all guaranteed to move together.
Contract. The host-facing side of the integration block uses endpoint_visible_q to decide whether this function is presented as reachable. The request-acceptance path of §11 uses it as one of its vetoes. Nothing else may read the raw bits to make a go/no-go decision — they are for composition and diagnosis.
Failure. Collapse these into one bit and every bring-up failure produces the same symptom — a device that does not answer — with no way to tell a PHY problem from an unprogrammed function. Post-silicon debug then costs days instead of minutes.
DV. Cover each bit's rise and fall independently, and cover the orderings: UCIe ready first, endpoint ready first, and each losing readiness while the other holds. §22's coverage model is built on exactly those orderings.
5. Deriving Visibility
// Illustrative. The composition is the point; the registration is the detail.
assign endpoint_visible_d =
ucie_operational_q &&
tunnel_ready_q &&
endpoint_cfg_valid_q &&
endpoint_function_ready_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) endpoint_visible_q <= 1'b0;
else endpoint_visible_q <= endpoint_visible_d;
endThree things about this that are easy to get wrong.
It is a conjunction, and the direction is asymmetric. Visibility should assert only when everything agrees, but it should deassert the instant any contributor drops. A registered AND does both, which is why the naive combinational form is not obviously wrong — it is the use of the result that needs care.
Real implementations usually qualify it further. A single registered AND will glitch visibility if any contributor is itself glitchy, and a device that appears and disappears within a few cycles is worse than one that never appears, because software may have started something in between. Common qualifications: require the conjunction to hold for a defined interval before asserting, hold visibility through a short transport disturbance if the integration contract says the function survives it, or make the assertion edge a deliberate event rather than a level. Which of these is correct is an integration contract decision, not a fact about UCIe or PCIe, and §7 and §8 are where that decision gets made.
Deassertion is not the inverse of assertion. Coming up carefully and going down carefully are different problems. Assertion can afford to be slow. Deassertion, when it represents a transport that has genuinely gone, must be fast enough that the acceptance path in §11 stops taking new work — but as §17 develops, fast to stop accepting is not the same as fast to forget.
6. The Bug This Chapter Exists To Prevent
// WRONG — the single most common integration bug in this whole module.
assign endpoint_visible = ucie_link_up;The UCIe link being operational says the transport can move flits. It says nothing about whether a PCIe function exists at the far end, whether its configuration path has been initialised, whether the mapping layer has resources, or whether the function is out of its own reset.
What the host does with this. It sees a device and begins acting on it: configuration reads to identify it, configuration writes to program it, and eventually memory traffic. Every one of those is now aimed at a target that may be unable to respond correctly.
What the symptoms look like, and why each one is a different debugging morning:
| Symptom | What actually happened |
|---|---|
| Configuration read times out | Request reached a mapping layer or function with nothing to answer it. |
| Malformed or default response | Something answered from an uninitialised configuration path. |
| Configuration request silently dropped | Mapping layer had no resources; nothing recorded that it existed. |
| Device appears, then disappears | Visibility tracked a transport bit that flapped during late training. |
| Device enumerates with wrong identity | Software read the configuration path before it held real values. |
The fourth is the worst of them, and it is worth naming precisely. Software-visible device identity must not flap. An operating system that has begun enumerating a function, assigned it resources, and possibly bound a driver to it does not gracefully tolerate that function vanishing because a transport bit went low for eight microseconds. The correct engineering response is not to make recovery faster; it is to make sure the transport's momentary state was never what determined identity in the first place.
7. Configuration State and Its Three Lifetimes
Now the question that the naive design never asks: when the UCIe link retrains, what happens to the endpoint's configuration?
Before answering, classify what "configuration" even covers. Three lifetimes, and they are genuinely different:
| Class | Examples | Lifetime | Set by |
|---|---|---|---|
| Static capability | which functions exist, what they are capable of, fixed identity values | design/strap — outlives everything | the implementation |
| Enumerated / programmed | address-window assignments, enable bits, interrupt configuration | from software programming until something defined resets it | system software |
| Transport-lifetime | whether flits can currently move, credits, replay state | the UCIe link epoch (Ch 9.4, 9.5) | the Adapter |
The third column is where the design decision lives. Static capability plainly should not care about the link. Transport state plainly must be re-established with the link — Chapter 9.5 §13 already established that credits are re-advertised, and Chapter 9.4 that replay state is re-baselined.
The middle row is the one nobody has decided.
Physical transport lifetime and PCIe function lifetime are separate architectural decisions. Coupling them is a choice, and it must be a deliberate one made against the software-visible behaviour you intend.
8. Reset Domains Are the Real Question
Restate §7 as the question a designer actually has to answer at integration time:
Which reset domain does the PCIe function live in?
Three broad arrangements, each defensible, each with consequences:
The function is in the transport's reset domain. A UCIe recovery resets the function. Simple to build and easy to reason about — there is one lifetime, so nothing can be inconsistent. The cost is that any transport disturbance is a software-visible device reset, with everything that implies for drivers and for whatever the device was doing.
The function is in its own domain, independent of the transport. Configuration survives a transport recovery; the device is briefly unreachable and then answers again with its state intact. This is what a slot-attached device's behaviour most resembles, and it is usually what platform software expects. The cost is that the two domains can now disagree, and §17's freeze logic exists precisely to manage the window where they do.
A hybrid, defined per state class. Transport-coupled state re-initialises; programmed state persists; a defined subset is invalidated because it cannot meaningfully survive. Most real systems land here, and the engineering work is writing down which class each register is in rather than discovering it during bring-up.
The point is not which one is right. It is that the answer must be written down before RTL is written, because every mechanism downstream — the freeze logic, the outstanding-request policy, the recovery sequence, the scoreboard — is a consequence of it.
9. Wrong Design — Clear Everything on Retrain
// WRONG (as an unconsidered default) — the transport's recovery is being
// treated as the function's reset, without that being an intended contract.
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n || ucie_recovery_event)
endpoint_cfg_q <= '0; // all programmed configuration discarded
endArchitecture. Nothing here is inherently illegal — arrangement one in §8 is a legitimate choice. What makes this wrong is that it is a side effect rather than a decision: the designer wired a convenient event to a convenient reset and did not ask what software would observe.
Cycle behaviour. One pulse on a transport recovery event and every programmed register in the function returns to its reset value, in the same cycle, with no announcement to anything above.
Failure. A temporary, fully recovered transport disturbance — the kind Chapter 9.4's retry mechanism exists to make invisible — becomes a host-visible device reconfiguration. The device is still present and answering, but its programmed state is gone. Software's model of the device and the device's actual state have diverged, and nothing reported an error, because at the transport layer nothing went wrong. Subsequent memory traffic aimed at a previously programmed address window now reaches a function that no longer claims that window.
The severity comes from the silence. A device that disappears is at least a visible event. A device that quietly forgets what it was told is a device that will misbehave later, far from the cause.
DV. Inject a transport recovery in the middle of an otherwise clean traffic run, after full programming, and check the configuration model against the device afterwards. If the intended contract is retention, this fires immediately. If the intended contract is reset, the scoreboard must model that too — the test is not "nothing changed", it is "what changed is what the contract says changes".
10. Outstanding Requests: Who Owns the Obligation
The endpoint has accepted PCIe requests. The UCIe link fails. What happens?
This is where integration stops being about bits and starts being about obligations. A non-posted PCIe request creates an expectation: something is owed back. That expectation exists in the requester's model of the world, and it does not evaporate because a die-to-die link had a bad moment.
Four questions have to have answers, and they are the design, not the protocol:
Where does the request state live? In the endpoint function, in the mapping layer, or split between them. If it is split, both halves need the same lifetime rules, or a recovery will clear one and leave the other.
Who owes the completion? The endpoint function has committed to producing one. If the transport cannot carry it, the obligation still exists — it is merely unfulfillable at this instant.
Is the request retained across recovery, or abandoned? Retention means the function continues and completes when transport returns. Abandonment means something must eventually resolve the requester's expectation. Both are viable; silence is not.
What resolves the requester if the request is abandoned? PCIe defines mechanisms for a requester whose completion never arrives. This chapter does not restate their timing, thresholds, or error semantics, because doing so accurately requires the Base Specification text and getting it approximately right would be worse than not stating it.
11. The Outstanding-Request Table
// Illustrative integration state. This is the mapping/endpoint boundary's
// own record of work in flight — NOT a PCIe normative structure, and the
// tag field here is a local handle, discussed below.
typedef struct packed {
logic valid;
logic [TAG_W-1:0] tag; // see the note on identity below
logic [REQ_META_W-1:0] meta; // what is needed to route the response
} outstanding_req_t;
outstanding_req_t outstanding_q [MAX_OUTSTANDING];Architecture. Accepting a request is a commitment, and a commitment with no record is a commitment that cannot be honoured. This table is the record. It exists because the response arrives later, possibly after a transport recovery, and something must connect it back to what asked.
State. MAX_OUTSTANDING entries, each with per-request lifetime — allocated at acceptance, freed at resolution. Note what that means: this is the only structure in the chapter whose lifetime is neither the link epoch nor the function's configuration lifetime. It is its own thing, and §18's table makes that explicit.
Cycle behaviour. On an accepted request, allocate a free entry and populate it. On a matching response, look up, use the metadata, and free. On an abandonment event, free with the abandonment path taken rather than silently.
Contract. The response path depends absolutely on this table: without a live entry, an arriving response has no meaning and nowhere to go. The acceptance path in §12 depends on it too, since a full table is a reason to stop accepting.
Failure. Undersize it and throughput collapses in a way that looks like a link problem. Fail to free entries and it fills permanently — the endpoint stops accepting and looks hung, with a perfectly healthy link. Free entries on the wrong event and responses arrive for entries that no longer exist.
DV. Cover the table empty, at one entry, at full, and at full-with-a-request-waiting. Cover a recovery occurring at each of those occupancies. The interesting bugs live at full and during recovery, not in the middle of the range.
12. Accepting a Request Is a Three-Way Veto
Chapter 10.2 §9 built a four-way veto on accepting an object into the transport. Here is its counterpart at the endpoint boundary, and it is a different set of vetoes for a different reason.
// Illustrative. Three independent conditions, three different owners.
assign request_ready =
endpoint_visible_q && // semantics are meaningful right now
outstanding_space && // this side can record the obligation
tunnel_resources_ready; // the response has a path backArchitecture. Each term guards a different failure. endpoint_visible_q guards meaning — accepting a request the function cannot semantically service. outstanding_space guards accountability — accepting an obligation with nowhere to record it. tunnel_resources_ready guards the return path — accepting work whose result cannot get home.
Cycle behaviour. Combinational over three registered inputs, sampled at the acceptance handshake. As in Chapter 9.5 §7, if the decision is made on the pre-update value, the assertions that check it must use $past accordingly.
Contract. The upstream requester relies on this as genuine backpressure — not-ready means retry later, not dropped. The outstanding table relies on the second term to guarantee the allocation it is about to perform will succeed.
Failure. Each omission fails differently, and this is why the three-way form earns its keep:
| Omitted term | Symptom | Where you debug it |
|---|---|---|
endpoint_visible_q | Request accepted into a function that cannot service it — timeout or nonsense response | the endpoint function |
outstanding_space | Request accepted with no record; response later unmatched | the response path, far from the cause |
tunnel_resources_ready | Request serviced; response cannot be sent; requester times out | the transport, blaming the link |
Three different subsystems, three different engineers, one missing term. That is the argument for writing the conjunction explicitly rather than deriving readiness from whatever bit is nearest.
DV. Cross each term's deassertion with a request attempt. Also cover the case where two are low at once — the design must not depend on the order in which they recover.
13. Wrong RTL — Accept Without Recording
// WRONG — the request is taken and passed on with no accounting entry.
always_ff @(posedge clk) begin
if (req_valid && endpoint_visible_q) begin
fn_req_valid_q <= 1'b1;
fn_req_data_q <= req_data; // straight into function logic
// no outstanding_q allocation, no metadata retained
end
endArchitecture. This is Chapter 10.1's ownership bug in its endpoint form. The request has crossed a boundary and the boundary kept no record that it happened.
Cycle behaviour. One cycle, one handoff, no allocation. The RTL is shorter and looks cleaner, which is exactly why it survives review.
Failure. The request enters function logic and is serviced. Then the response appears — and there is nothing to match it to. Whatever routing metadata the response needed to get back to the right requester was in req_data and was never saved. Depending on downstream design, the response is dropped, sent to a default destination, or matched against a stale entry from an unrelated earlier request.
The last case is the dangerous one, and it is worth being explicit: a response matched to the wrong outstanding entry delivers correct-looking data to a requester that asked for something else. No error is signalled, because every field is individually well-formed. This is the endpoint-layer sibling of Chapter 10.2 §7's metadata/payload misalignment, and it has the same character — structurally invalid, individually valid, silent.
DV. The assertion in §15 catches this the first time a response arrives. More importantly, the scoreboard of §19 catches the routing error, which the assertion cannot: an assertion can confirm that some live entry matched, but only a model that knows who asked can confirm it was the right one.
14. Matching a Response
// Illustrative — the lookup, not a PCIe completion decoder.
// PCIe's own rules for what constitutes a valid completion for a given
// request are protocol semantics and are not restated here.
logic match_hit;
logic [IDX_W-1:0] match_idx;
always_comb begin
match_hit = 1'b0;
match_idx = '0;
for (int i = 0; i < MAX_OUTSTANDING; i++) begin
if (outstanding_q[i].valid && (outstanding_q[i].tag == rsp_tag)) begin
match_hit = 1'b1;
match_idx = i[IDX_W-1:0];
end
end
endArchitecture. The response carries an identifier; the table holds the context. This lookup is the only thing that turns an arriving response back into an answer for a specific requester.
State. Purely combinational over the table. The table entry it selects has per-request lifetime; the match itself has none.
Cycle behaviour. Evaluated when a response arrives; the entry is consumed and freed in the same handshake.
Contract. Two contracts meet here. Upward, the requester relies on receiving its own answer. Downward, the table relies on the response path to free exactly the entry it matched — freeing a different index leaks one entry and prematurely releases another.
Failure. No match means an unmatchable response, which is either a lost request record (§13) or a lifetime error. Multiple matches mean duplicate live identifiers, which is worse: the loop above silently takes the last one, so the design does not fail, it just answers the wrong requester. If your identifier space is meant to be unique among live entries, assert that it is rather than assuming it.
DV. Cover match, no-match, and — deliberately, by injection — the multiple-match case. An implementation that has never seen two live entries with the same identifier has never tested its uniqueness assumption.
15. SVA — A Response Requires a Live Request
// Illustrative. Local invariant, verification-only.
property p_response_matches_live_request;
@(posedge clk) disable iff (!rst_n)
rsp_fire |-> match_hit;
endproperty
a_response_matches_live_request: assert property (p_response_matches_live_request);
// Uniqueness of live identifiers, if that is the intended contract.
property p_outstanding_ids_unique;
@(posedge clk) disable iff (!rst_n)
rsp_fire |-> $onehot0(match_vec); // match_vec[i] = valid && tag equal
endproperty
a_outstanding_ids_unique: assert property (p_outstanding_ids_unique);What the first catches. Every mechanism that lets a response exist without a corresponding request record: the §13 acceptance bug, entries freed early, entries cleared by a recovery that should not have cleared them, and responses generated spuriously.
What the second catches. Identifier reuse while an earlier user is still live — a lifetime bug that produces wrong-requester delivery rather than an error.
What neither can catch, and this is the standing rule of the curriculum. These assertions prove that a live entry matched. They cannot prove it was the correct entry, because correctness there is defined by who issued the request, and that information is not present at this interface. Only the scoreboard in §19 holds it.
Assertions prove local invariants. Only a scoreboard proves that the right requester got the right answer.
16. Transport Replay Must Not Become a Second Request
This is Chapter 10.2 §12's exactly-once property, stated at the endpoint where its consequences are concrete.
The transport may deliver the same object twice. Chapter 9.4 established why: a retry happens when corruption is detected or when a confirmation is lost, and in the second case the original arrived perfectly. The receiver therefore genuinely sees the same transport object a second time, and this is correct transport behaviour, not a bug.
The endpoint function must not see a second semantic request.
// Illustrative duplicate suppression at the semantic boundary.
// The transport identity space is the transport's own — see 10.2 §11.
logic [XPORT_ID_W-1:0] last_accepted_id_q;
logic have_accepted_q;
assign is_duplicate = have_accepted_q &&
(rx_xport_id == last_accepted_id_q);
assign semantic_accept = rx_object_complete && !is_duplicate;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
have_accepted_q <= 1'b0;
last_accepted_id_q <= '0;
end else if (semantic_accept) begin
have_accepted_q <= 1'b1;
last_accepted_id_q <= rx_xport_id;
end
endArchitecture. A gate between transport arrival and semantic acceptance whose only job is to make retransmission invisible upward. The single-entry form shown is the simplest case that teaches the mechanism; the real width of the suppression window is set by how far the transport's retry can reach back, which is Chapter 9.4's unconfirmed-window question.
State. last_accepted_id_q and have_accepted_q have UCIe link epoch lifetime — the identity space is re-baselined with the link, so carrying a stale value across a re-establishment is itself a bug.
Cycle behaviour. Acceptance is gated in the same cycle the object completes. Nothing downstream of this gate ever observes the duplicate.
Contract. The function relies on this absolutely. Everything below it is free to retransmit; nothing above it may observe that anything happened.
Failure. Omit it and a retransmitted object becomes a second PCIe request at the endpoint. For a memory write that is the same write executed twice. For a non-posted request it is two obligations where the requester expects one, so the outstanding table either allocates twice for one expectation or reports a duplicate identifier. None of it produces a CRC error, because the transport behaved exactly as designed.
DV. The injection that matters is a lost confirmation on an object that arrived intact — the only stimulus that produces a genuine duplicate. A regression that only injects corruption exercises replay but never exercises this gate, and the entire mechanism ships untested.
// Illustrative — the property that makes the requirement explicit.
property p_replay_no_duplicate_accept;
@(posedge clk) disable iff (!rst_n)
(rx_object_complete && is_duplicate) |-> !semantic_accept;
endproperty
a_replay_no_duplicate_accept: assert property (p_replay_no_duplicate_accept);17. Bring-Up, As a Sequence
Two properties of this picture are worth stating explicitly.
The first two messages are independent. Neither causes the other. They are drawn in an order because a diagram needs one, but the design must work when they arrive in the opposite order — and §22's coverage requires testing both.
Visibility is an event with consequences. Everything after the third message is only legal because visibility asserted. That is the whole argument for making it a composition rather than a copy of a transport bit.
18. When the Transport Goes: Freeze, Not Amnesia
// Illustrative. The distinction the whole section rests on:
// stop taking new work, without forgetting existing work.
logic endpoint_freeze_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) endpoint_freeze_q <= 1'b0;
else if (!ucie_operational_q) endpoint_freeze_q <= 1'b1;
else if (recovery_complete) endpoint_freeze_q <= 1'b0;
end
// Freeze participates in acceptance; it does NOT clear the table.
assign request_ready =
endpoint_visible_q && outstanding_space &&
tunnel_resources_ready && !endpoint_freeze_q;Architecture. When the transport is unavailable, continuing to accept requests is accumulating obligations that cannot be met. Freezing stops the accumulation. What freezing must not do is discard the obligations already accumulated, because those belong to requesters who are still waiting.
State. endpoint_freeze_q has UCIe link epoch lifetime — set on loss, cleared on defined recovery completion. Note it is deliberately not a pure inversion of ucie_operational_q: clearing requires a recovery-complete event, so the design does not resume mid-recovery on a transient assertion.
Cycle behaviour. Set immediately on transport loss — this is one place where fast deassertion matters. Cleared only on the defined completion event.
Contract. Upstream sees backpressure rather than errors. The outstanding table's contract is that freeze does not touch it. §10's policy decision determines what does eventually resolve those entries.
Failure. Two opposite failures, and both are common. Freeze that also clears the table produces §9's silent amnesia with the additional harm that requesters are left permanently waiting. Freeze that never engages produces an outstanding table that fills during an outage, so recovery arrives to find no capacity and the endpoint appears hung after the link is demonstrably healthy again.
DV. Inject transport loss at several outstanding-table occupancies, including full. Check that the table contents are byte-identical across the outage when the contract says retention, and that every entry reaches its defined resolution when the contract says abandonment.
// Illustrative.
property p_no_accept_while_frozen;
@(posedge clk) disable iff (!rst_n)
endpoint_freeze_q |-> !req_accept_fire;
endproperty
a_no_accept_while_frozen: assert property (p_no_accept_while_frozen);
// The complementary property, and the one people forget to write.
property p_freeze_preserves_outstanding;
@(posedge clk) disable iff (!rst_n)
($rose(endpoint_freeze_q) && outstanding_valid_vec != '0)
|=> (outstanding_valid_vec == $past(outstanding_valid_vec));
endproperty
a_freeze_preserves_outstanding: assert property (p_freeze_preserves_outstanding);The second property is the one worth arguing for. It encodes §8's intended contract into the RTL's verification, which means that if someone later wires a convenient reset to a convenient event — §9's bug — the assertion fires at the moment the coupling is introduced rather than during a post-silicon investigation months later.
19. Three Error Domains, Not Two
The organising idea of this chapter's debug methodology.
PCIe semantic errors. The protocol behaviour was wrong: a malformed request, a response that does not satisfy its request, a violation of PCIe's rules. These are diagnosed with PCIe knowledge and are largely independent of what transport is underneath.
UCIe transport errors. The path failed: CRC errors, retry exhaustion, training failure, credit violations. Chapters 8.x and 9.x built the machinery, and these are diagnosed with UCIe knowledge.
Integration errors. Both subsystems are individually correct and their combined state is inconsistent. The link is fine, the endpoint is fine, and the system does not work.
20. Integration Failure Signatures
The table to have open during bring-up. Each row is a class-three failure, and each points somewhere specific.
| Signature | Most likely cause | First thing to check |
|---|---|---|
| UCIe operational, endpoint never visible | A readiness contributor never asserted | Which of §4's bits is low |
| Endpoint enumerates, then disappears | Visibility coupled to a transport bit that flapped | Whether visibility has any qualification (§5) |
| Configuration works, then device forgets its programming | Reset-domain coupling (§9) | What clears the configuration registers |
| Request reaches mapping layer, no response ever | Function not semantically ready, or table full | endpoint_visible_q and table occupancy at the time |
| Same request appears to execute twice | Replay boundary error (§16) | Whether duplicate suppression exists and is armed |
| Response arrives, matches nothing | Request accepted without a record (§13), or entry freed early | Table occupancy trace across the request's lifetime |
| Response arrives, matches the wrong requester | Identifier reuse or wrong-index free (§14) | Uniqueness assertion, and the index used to free |
| Endpoint hung after a healthy recovery | Table filled during the outage; freeze absent (§18) | Occupancy at the moment of transport loss |
The last row deserves a note because it is the most misleading symptom in the list. Everything measurable is healthy — link operational, no errors, endpoint present — and the device does nothing. The cause is in the past, during an outage that has already been recovered from and possibly already forgotten by the logs.
21. State Lifetimes, Side by Side
The table this chapter exists to produce, in the format Chapter 10.2 §14 established.
| State | Allocated / set when | Lost when | On transport recovery | On function reset |
|---|---|---|---|---|
| Static capability | design/strap | never | unaffected | unaffected |
ucie_operational_q | link reaches operational | link leaves it | re-established | unaffected |
tunnel_ready_q | mapping resources ready | resources or link lost | re-established | unaffected |
endpoint_cfg_valid_q | config path initialised | function reset | contract decision (§8) | cleared |
| Programmed configuration | software writes | function reset — or a recovery, if coupled | contract decision (§8) | cleared |
| Outstanding entry | request accepted | resolved or abandoned | retained, or abandoned by defined path (§10) | resolved per contract |
endpoint_freeze_q | transport lost | recovery complete | cleared by the recovery itself | cleared |
| Duplicate-suppression ID | first semantic accept | link epoch ends | re-baselined (§16) | cleared |
| Diagnostics / sticky status | first event | broader reset only | survive (Ch 8.1 §18) | survive |
Three rows carry the weight. Two rows say "contract decision" and that is not evasion — it is the chapter's central finding, and a design that has not written those two rows down explicitly has not finished its architecture. Outstanding entries have their own lifetime, belonging to neither the link epoch nor the configuration lifetime. And diagnostics survive everything short of a broad reset, because the whole value of §20's table depends on evidence outliving the event.
22. The Endpoint Scoreboard
Assertions cover §15's local invariants. This is what proves the system.
host_requests[] — every semantic request the host issued, with identity
endpoint_accepts[] — every request the endpoint semantically accepted
outstanding_model — expected table contents, by allocation and resolution
responses[] — every response, with the request it claims to answer
transport_events[] — retries, recoveries, link epochs, from the UCIe modelFive structures because there are five things that can independently be wrong. The checks:
Exactly one semantic accept per host request. This is §16's property at system level, and only a model that holds transport_events[] alongside endpoint_accepts[] can distinguish "the transport retransmitted" (correct) from "the endpoint accepted twice" (corrupt).
Exactly one response per request that requires one, delivered to the requester that issued it. The second clause is the part assertions cannot check.
No response without a request. The system-level form of §15, catching spurious responses that happen to match a live entry by accident.
No outstanding entry silently lost across a transport disturbance, unless the contract's defined abandonment path was taken — in which case the abandonment must be observable, not inferred from the entry's absence.
The outstanding model matches the device's table at every quiescent point. Divergence at a quiescent boundary is the cleanest possible evidence of a lifetime bug, and it usually shows up long before the resulting misbehaviour would.
The reason five structures are needed rather than one is the same reason Chapter 10.2 needed two reference models: the semantic model tells you that something was duplicated, lost, or misrouted; the transport model tells you why. A duplicate accept and a duplicate transport delivery look identical in a semantic-only log, and one of them is correct behaviour.
23. Coverage
Not vanity coverage. The equivalence classes that actually contain the bugs, crossed the way §4's independence demands.
Readiness orderings. UCIe operational first then endpoint ready; endpoint ready first then UCIe operational; both in the same window. The design must not depend on order, and only crossing them proves it.
Recovery position relative to configuration. Recovery before any configuration; recovery mid-configuration; recovery after full configuration. The third is where §9's bug becomes visible; the second is where partially-programmed state produces its own surprises.
Recovery position relative to traffic. Recovery with the outstanding table empty, partially occupied, and full. The full case is where §18's freeze either works or does not, and it is routinely omitted.
Response arrival relative to recovery. Response arrives before the disturbance, during the outage, and after recovery completes. The third is the one that exercises whether the entry survived.
Transport events. Retry with an intact object and a lost confirmation — the §16 injection. Retry with genuine corruption. Multiple retries on the same object.
Reset independence. Function reset without a transport event, and a transport event without a function reset — if §8's contract permits them to be independent. If it does not, cover the coupling instead, and assert it.
The crosses that matter. Bring-up phase × recovery position × table occupancy. That three-way cross is small enough to be achievable and contains, in my experience of this class of design, nearly every integration bug worth finding.
24. Debug Checklist
In order. Each step is cheap, and the order is chosen so that the first failing step localises the problem.
- Is the UCIe link operational — Chapter 8.6's state, not an inference?
- Is the mapping layer ready, with resources available?
- Is the endpoint's configuration path initialised and answerable?
- Does
endpoint_visible_qreflect the conjunction — and if not, which bit is low? - Was the configuration request accepted at the boundary, or refused?
- Did the endpoint allocate outstanding state for it?
- Was a UCIe retry involved anywhere in this transaction's life?
- Did the semantic request appear at the function exactly once?
- Is the outstanding entry still live, and does it hold the metadata expected?
- Did a transport recovery occur — and did anything in the endpoint change that should not have?
- Which reset domain changed? This single question resolves a large fraction of class-three failures.
- Given all of the above: is this a PCIe failure, a UCIe failure, or an integration failure?
Question 11 is the highest-yield one on the list, and it is the one least often asked, because it requires knowing §8's contract — which is why §8 insists that contract be written down before RTL.
25. Common Misconceptions
"UCIe link up means the PCIe endpoint is up." The transport can move flits. That is orthogonal to whether a PCIe function exists, is out of reset, has an initialised configuration path, and has mapping resources. §6 lists five distinct symptoms of believing otherwise.
"Endpoint state and UCIe state are one state machine." They are two, with different transitions, different triggers, and different lifetimes. §4 exists because they are independent; §21 exists because their lifetimes differ.
"A UCIe retrain should reset all PCIe configuration." It may, if that is the intended contract — arrangement one in §8. As an unexamined default it converts a recovered transport disturbance into a silent host-visible reconfiguration, which is §9's bug.
"Transport replay may legitimately be visible as a repeated PCIe request." It may not. Retry is a transport mechanism, and a retransmitted object becoming a second semantic request means a memory write executes twice. §16 is the gate that prevents it.
"A response can be accepted without outstanding-request state." Then it cannot be routed to the requester that asked, and the best case is that it is dropped. The realistic case is that it matches something stale and delivers a correct-looking answer to the wrong requester.
"An endpoint disappearing during UCIe recovery is inevitable." It is a design outcome, not a law. Whether software observes the device vanish is determined by §8's reset-domain decision and §5's qualification of visibility.
"If the packet bits are correct, the integration is correct." Every bug in this chapter occurs with perfectly correct packet bits. That is the definition of §19's third error class.
"There are only PCIe errors and UCIe errors — integration itself cannot fail." This is the misconception that produces the longest debugging sessions, because it means nobody owns the failure that is actually occurring.
26. Understanding Check
27. Summary and What Comes Next
Endpoint connectivity is the composition of two independent readiness domains: UCIe must be able to transport packets, and the PCIe endpoint must be in a state where those packets have valid semantic meaning.
The mechanisms: a readiness vector rather than a bit, because five subsystems fail independently and the first debug question is which one. Visibility as a registered, qualified composition, because software-visible device identity must not flap. Reset domains decided deliberately — the chapter's central finding, since whether configuration survives a transport recovery is an integration contract, not a protocol fact, and a design that has not written it down has not finished its architecture. An outstanding-request table with its own lifetime, belonging to neither the link epoch nor the configuration lifetime. A three-way veto on acceptance, each term failing in a different subsystem. Freeze without amnesia on transport loss. And duplicate suppression at the semantic boundary, because a retransmitted object becoming a second request means a memory write executes twice with nothing reporting an error.
The idea that organises the debugging: there is a third error class. Not PCIe, not UCIe, but integration — both subsystems correct, their composition inconsistent. Every bug here is one, which is why the tools of both specialisms show green while the system does nothing.
An endpoint can now exist correctly behind a UCIe transport. The harder half of the problem is on the other side: the root complex must discover that endpoint, assign it resources, route traffic to it, preserve its configuration across disturbances, and hide the fact that the device is physically on another die:
- 10.4 — Host-Side Integration — how a root complex absorbs a UCIe-attached chiplet into its normal PCIe topology.
Browse the full path on the UCIe tutorials index.