Host-Side Integration

Chapter 10.3 ended with a PCIe function that exists correctly behind a die-to-die link: readiness composed rather than copied, configuration lifetime decided rather than inherited, obligations recorded rather than assumed.

That function is now reachable in principle. Nothing yet gets to it.

Getting to it is the host's problem, and it is the harder half. The root complex must discover that the function is there, decide it is worth presenting to software, give it addresses, route configuration and memory traffic to it, keep track of what it owes and is owed, survive the link underneath it disappearing and coming back — and do all of that while telling software a story in which none of this exists.

This chapter is about maintaining that story.

1. The One-Sentence Model

The host should see PCIe topology; the implementation knows UCIe topology.

That boundary is the entire lesson, and almost every bug in this chapter is a leak across it.

Software reasons in buses, devices and functions; configuration space; address windows; interrupts; requesters and completers. Hardware underneath reasons in on-die fabric ports, UCIe links, package positions, and chiplet instances. Both models are correct. They describe different graphs, and the moment a value from one appears as a value in the other, modularity is gone and the bug that results will not look like a topology bug at all.

2. Two Graphs

Make the abstraction concrete before building anything.

The package graph is physical and small. There are n UCIe links. Link 0 goes to chiplet A. Link 1 goes to chiplet B. Each link is up or down. Each has an epoch, a width, a rate, and a credit state. This graph is fixed at design and assembly time, and it changes only when a link's state changes.

The PCIe graph is logical and software-defined. It is a hierarchy of buses, devices and functions, discovered by enumeration and given meaning by configuration. Bus numbers are assigned. Address windows are allocated. Identity is programmed. This graph does not exist until software builds it, and it can be rebuilt differently on the next boot.

The existing PCIe track covers the logical graph properly, and this chapter does not restate it: PCIe Architecture · Root Complex · Endpoint · Hierarchy Domains · Enumeration Overview · Device Discovery · Configuration Access · Resource Allocation · What BARs Are.

What follows is only what changes because the far end is across a package.

3. The Host Integration Path

The pipeline in words, since the diagram compresses it:

CPU and coherent fabric issue a transaction. The root complex decides it belongs to the PCIe domain. The configuration or address routing engine decides which PCIe function it is aimed at, and therefore which path. The PCIe/UCIe mapping layer of Chapter 10.2 turns it into a transport object. The Adapter and PHY carry it. The remote endpoint of Chapter 10.3 receives it.

Everything this chapter adds lives in the third step.

4. Enumeration: Only the Part That Changed

Generic PCIe enumeration is already covered. The UCIe-specific questions are these four, and none of them has a PCIe answer.

When should a remote function appear? Not when the package contains a chiplet. Not when a link exists in the design. The appearance of a device to software is a commitment, and §6 develops what backs it.

What evidence justifies appearing? Chapter 10.3 §4 built the answer at the far end: a composed readiness vector, not a transport bit. The host needs its own local version of that evidence, because the host cannot see the remote vector directly — it can only see what the transport reports plus what it has itself established.

What happens if the link is not operational when software looks? Software will read configuration space. Something must respond, or not respond, in a defined way. "Whatever happens to happen" is not a design.

How is configuration access routed? A configuration request names a function. Something must turn that name into a path. That is §7's table, and it is the first place the two graphs meet.

5. Presence Is Not Reachability

The distinction that §6's bug violates, and it is worth stating on its own because it is the host-side twin of Chapter 10.3 §6.

Presence is a physical fact: a chiplet is in the package, bonded, powered. It is knowable statically or nearly so.

Reachability is a live property: a transaction issued now would arrive, be understood, and be answered. It depends on the link's current state, on the mapping layer's resources, and on the remote function's readiness.

Presence is a precondition for reachability and nothing more. A design that presents devices to software based on presence has published a claim it cannot back.

6. The Remote Endpoint Table

// Illustrative host integration state — not a PCIe normative structure.
// This is the host's own record of what it believes exists at the far end
// of each link, and how to get there.
typedef struct packed {
  logic                     present;          // physically there (§5)
  logic                     transport_ready;  // link currently usable
  logic [ROUTE_W-1:0]       route;            // how to reach it
} remote_ep_t;
 
remote_ep_t remote_ep_q [NUM_REMOTE_EP];
 
// Reachable is a conjunction, exactly as in Chapter 10.3 §5.
assign ep_reachable[i] = remote_ep_q[i].present &&
                         remote_ep_q[i].transport_ready &&
                         ep_visible_reported[i];

Architecture. The host needs a local model of the far side, because it cannot query it synchronously — asking whether the endpoint is ready is itself a transaction that requires the endpoint to be ready. So the host maintains what it has been told and what it can observe, and derives reachability from the conjunction.

State. One entry per remote function. present has static or platform lifetime. transport_ready has UCIe link-epoch lifetime. route has per-enumeration lifetime — it is meaningful only after the topology has been established, and it is re-derived when the topology is rebuilt. Three lifetimes in one struct, which is fine as long as everyone reading it knows that.

Cycle behaviour. present changes essentially never. transport_ready follows link state. route is written once during the enumeration sequence of §19 and is stable thereafter.

Contract. The configuration routing engine of §7 and the address routing engine of §9 both read route. The acceptance path of §16 reads ep_reachable. Nothing may read present alone to make a routing decision — that is §7's bug in embryo.

Failure. Conflating the three fields produces §7's bug directly. Letting route be read before it is written produces traffic aimed at an uninitialised destination, which is §21's failure. Failing to update transport_ready on link loss means the host keeps routing into a dead link and reports timeouts instead of a link failure.

DV. Cover each field changing independently. Cover the ordering where transport_ready asserts before and after route is programmed — the design must not depend on which comes first, and the assertion in §20 is what enforces it.

7. Wrong RTL — Presence From a Strap

// WRONG — a package strap says a chiplet is fitted, and that is taken
// as sufficient reason to present a device to software.
assign remote_ep_present = pkg_strap_chiplet_fitted;
assign ep_visible_to_sw   = remote_ep_present;

Architecture. The strap is real information — it tells you the die is bonded. It is the wrong kind of information, because it is static and reachability is not.

Cycle behaviour. Constant. That is precisely the problem: the value cannot change when the link fails, so the host's model of the world cannot either.

Failure. Software enumerates a device whose link may be untrained, mid-recovery, or permanently failed. Configuration reads are issued and time out. Depending on the platform's error policy, that either produces a device that reads as all-ones and is discarded — a chiplet that is physically present but silently absent from the system — or a hang while something waits for an answer that cannot come.

The worse variant is a link that is sometimes up. The device enumerates successfully on a lucky boot, gets resources assigned, gets a driver bound, and then fails during operation in a way that looks like a device fault rather than an integration fault.

DV. Boot with the link held down. Boot with the link coming up late, after enumeration has started. Both must produce a defined, intentional outcome — and the test is not "it did not crash", it is "what software observed matches what the platform intended it to observe".

8. Routing Configuration Access

A configuration request names a function. The host must decide: is this a local function, or one behind a link, and if the latter, which link?

// Illustrative. The mapping between the PCIe graph and the package graph
// is a table because the correspondence is arbitrary (§2).
typedef struct packed {
  logic               valid;
  logic [BDF_W-1:0]   bdf;        // the software-visible identity
  logic [LINK_W-1:0]  ucie_link;  // the physical path
} cfg_route_t;
 
cfg_route_t cfg_route_q [NUM_CFG_ROUTES];
 
// Match is over software identity; the result is a physical path.
always_comb begin
  cfg_match_vec = '0;
  for (int i = 0; i < NUM_CFG_ROUTES; i++)
    cfg_match_vec[i] = cfg_route_q[i].valid && (cfg_route_q[i].bdf == req_bdf);
end

Architecture. This table is the boundary in §1. Its input column is a software-visible PCIe identity; its output column is a physical UCIe link. Every row is one arbitrary correspondence that somebody decided.

State. NUM_CFG_ROUTES entries with per-enumeration lifetime — written when the topology is established, invalidated when it is rebuilt. Note that this is a different lifetime from the link epoch: a link going down and coming back does not by itself change which PCIe identity lives on it.

Cycle behaviour. Combinational match on each configuration request. Written by the enumeration sequencer, not by the datapath.

Contract. The configuration path relies on exactly one row matching, or none. Two rows matching is a configuration error, and §11 is where that gets caught.

Failure. A match to an invalid entry sends configuration traffic nowhere. A match to a link that is down produces a timeout that looks like a device fault. No match at all should produce the platform's defined behaviour for an absent function — and "should" is doing work there, because a table that silently routes unmatched requests to entry zero is a table that answers configuration reads for devices that do not exist.

DV. Cover a hit, a miss, a hit on an invalid entry, and a hit on an entry whose link is down. Then cover the pathological case of two entries claiming the same identity, which §11's assertion exists to catch.

9. SVA — No Configuration Route To a Dead Link

// Illustrative.
property p_cfg_route_link_operational;
  @(posedge clk) disable iff (!rst_n)
    (cfg_route_fire && cfg_match_hit)
      |-> link_operational[cfg_sel_link];
endproperty
a_cfg_route_link_operational: assert property (p_cfg_route_link_operational);

What it catches. Configuration traffic dispatched onto a link that is not in a state to carry it. This happens more often than it sounds, for a structural reason: the route table's lifetime is per-enumeration and the link's state has link-epoch lifetime, so a perfectly valid route can point at a link that went down five microseconds ago. Nothing about the table is wrong; it is simply describing a path that is temporarily unusable.

What it does not catch. Whether the route points at the correct link. That is a mapping-correctness question, and only §24's scoreboard — which independently predicts the destination from its own model of the topology — can answer it.

The design question it forces. If a configuration request arrives for a valid route on a down link, what should happen? Refuse at the boundary with a defined status? Hold until recovery? Let it go and rely on a timeout? All three are implementable. The assertion above encodes the first: the request never leaves. Choose deliberately, and make the assertion match the choice.

10. Address Routing: Apertures

Configuration access is the easy half, because it names its target. Memory traffic does not — it carries an address, and the host must decide from the address alone which function, and therefore which link, it belongs to.

Once BARs have been programmed, each function claims one or more address windows. The host's job is to turn an address into a path.

// Illustrative host address-decode state. Generic window matching —
// PCIe BAR semantics are covered in the PCIe track and not restated.
typedef struct packed {
  logic                  valid;
  logic [ADDR_W-1:0]     base;
  logic [ADDR_W-1:0]     limit;
  logic [LINK_W-1:0]     link_id;
  logic [FN_W-1:0]       fn_id;
} aperture_t;
 
aperture_t aperture_q [NUM_APERTURES];
 
logic [NUM_APERTURES-1:0] ap_match_vec;
 
always_comb begin
  for (int i = 0; i < NUM_APERTURES; i++)
    ap_match_vec[i] = aperture_q[i].valid &&
                      (req_addr >= aperture_q[i].base) &&
                      (req_addr <= aperture_q[i].limit);
end

Architecture. This is the second and larger crossing between the two graphs. The input is a host physical address — a purely software-visible concept. The outputs are a physical link and a function identity. Every memory transaction to a remote chiplet passes through this decode.

State. NUM_APERTURES entries with per-resource-window lifetime: written when software programs the corresponding BAR, invalidated when it is reprogrammed or disabled. This lifetime is distinct from both the route table's and the link epoch's, which means there are now three independent lifetimes governing whether a memory transaction can be routed — and a bug in any of them looks the same from the CPU.

Cycle behaviour. Combinational match per request, in the address path, which makes it timing-critical in a way the configuration table is not. Real implementations pipeline it; the pipelining must not let a window update land between the match and the use, which is a stability property worth asserting.

Contract. The requesting fabric relies on a routed transaction reaching exactly one destination. The endpoint relies on receiving only transactions genuinely inside its window. Software relies on the windows it programmed being the windows the hardware decodes.

Failure. Three distinct failures, and they are worth separating. No match means an address in no window — the platform's unclaimed-address behaviour applies. Match on a stale window means traffic delivered to a function that no longer claims that address, which is a silent misdelivery. Multiple matches is §11, and it is the serious one.

DV. Cover base, limit, base−1 and limit+1 for every window — the boundaries are where the comparison operators are wrong, and >=/<= versus >/< errors produce off-by-one windows that work for every address a directed test happens to pick. Cover adjacent windows with no gap, which is where two windows' boundary arithmetic must agree exactly.

11. Wrong RTL — Overlapping Windows

// WRONG — nothing prevents two apertures covering the same address.
// Programmed by software or by firmware, this elaborates and simulates
// happily until an address falls in both.
always_comb begin
  route_link = '0;
  for (int i = 0; i < NUM_APERTURES; i++)
    if (ap_match_vec[i]) route_link = aperture_q[i].link_id;  // last wins
end

Architecture. The loop is not defending an invariant it depends on. It assumes at most one match and silently resolves the case it did not expect by taking the highest index.

Cycle behaviour. For an address in one window, correct. For an address in two, it picks the higher-numbered entry deterministically — which is worse than picking randomly, because it will be consistent and therefore pass every test that does not specifically probe the overlap.

Failure. Two endpoints claim one address. Depending on which entry wins, memory traffic intended for endpoint A arrives at endpoint B. Both are healthy. The link is healthy. The data goes to the wrong die.

The symptom is the one in §23's table: the wrong endpoint receives the request. From the CPU's point of view a write went somewhere and a read returned something; from endpoint B's point of view a perfectly well-formed transaction arrived inside its window and was serviced. Nothing anywhere reports an error. This is the host-side sibling of Chapter 10.2 §7 and Chapter 10.3 §13 — the recurring shape of this entire module, where every field is individually valid and the composition is wrong.

The fix has two halves. At programming time, validate that a new window does not overlap an existing valid one, and refuse or report if it does. At run time, assert that at most one matches. Both are needed: validation catches the error where it is caused, and the assertion catches the case where a window was updated without going through the validation path.

12. SVA — One-Hot Route Match

// Illustrative. The single most valuable system-integration assertion
// in this chapter.
property p_aperture_match_onehot;
  @(posedge clk) disable iff (!rst_n)
    mem_req_fire |-> $onehot0(ap_match_vec);
endproperty
a_aperture_match_onehot: assert property (p_aperture_match_onehot);
 
// The configuration table needs the same discipline.
property p_cfg_match_onehot;
  @(posedge clk) disable iff (!rst_n)
    cfg_req_fire |-> $onehot0(cfg_match_vec);
endproperty
a_cfg_match_onehot: assert property (p_cfg_match_onehot);
 
// And the window must be stable while it is being used, if the decode
// is pipelined across more than one cycle.
property p_aperture_stable_in_flight;
  @(posedge clk) disable iff (!rst_n)
    (decode_in_flight && !decode_done)
      |=> (ap_match_vec == $past(ap_match_vec));
endproperty
a_aperture_stable_in_flight: assert property (p_aperture_stable_in_flight);

$onehot0 rather than $onehot is deliberate: zero matches is legal — an address in no window is a real, defined situation with its own handling. Two matches is not legal under any interpretation, which is exactly what makes this assertion so cheap and so effective.

Why this one earns its place above the others. It is a single expression, it costs nothing, it requires no model of the topology, and it catches a class of bug whose symptom is silent misdelivery to the wrong die. It also catches the bug at the moment the configuration becomes inconsistent, rather than when a transaction later happens to land in the overlap — which may be days of runtime later, or never in simulation and immediately in silicon.

Write it for every table in the design that is supposed to be a decode.

13. Three Identity Spaces

The most persistent conceptual error in host-side integration, and it deserves naming precisely.

Three spaces. Three assigners. Three lifetimes. One of them is a protocol-visible value and two of them are implementation details.

They may be related by tables. They may not be substituted for one another, and this is not a style preference — the middle column is the argument. A value assigned by software during enumeration cannot be equal to a value fixed at design time except by coincidence, and designing to that coincidence means the design breaks when software enumerates differently.

14. Wrong Design — The Link Number As Identity

// WRONG — physical topology promoted into protocol identity.
assign pcie_requester_id = { 8'd0, 3'd0, ucie_link_id };

Architecture. This is attractive because it is free: the link number is already there, it is unique, and it makes the requester identity trivially decodable on the return path. It also destroys the abstraction the entire chapter is built on.

Failure, in escalating order of severity.

Software now depends on package topology. The requester identity is protocol-visible. If it is derived from the link number, then anything that reasons about requester identity — error logs, address translation, access control, virtualisation — is now reasoning about the physical package. The abstraction has not just leaked; it has been published.

The design cannot be re-partitioned. Move a function to a different link — a routine thing between silicon revisions — and its protocol identity changes. Every piece of software and firmware that recorded the old identity is now wrong.

Identity collides where the graphs disagree. Two functions on one link must have distinct requester identities and the formula cannot give them any. So either the formula is extended with a second field that is itself topology-derived, compounding the problem, or the design is silently limited to one function per link.

And the return path breaks first. A completion routed by requester identity now routes by link number, so a completion for a function that has been re-enumerated goes to whatever is on that link now.

The fix is a table, exactly as in §2 and §8:

// Illustrative — an explicit, programmable mapping instead of arithmetic.
logic [BDF_W-1:0] src_to_bdf_q [NUM_SOURCES];

One indirection, written during enumeration, with per-enumeration lifetime. It costs a small memory and it keeps the two graphs separate.

15. Interrupts, At the Right Depth

Interrupt configuration and delivery are PCIe subjects, and this chapter does not teach them. The integration question is narrower and worth stating on its own.

A remote function generates an interrupt with PCIe semantics — the event has a protocol-defined meaning and a protocol-defined identity that ties it to the function that raised it. UCIe's role is to transport whatever information that requires. It has no interrupt concept of its own and should not acquire one.

// Illustrative host-side integration state only.
logic [NUM_REMOTE_EP-1:0] irq_pending_q;   // per remote function, not per link

Architecture. The host tracks pending interrupt state per function, because that is the granularity software reasons about. Indexing this by link would be the §14 bug in a different costume.

State. One bit per remote function, per-enumeration lifetime in the sense that its indexing is meaningful only against the current topology.

Contract. Whatever consumes this must map back to a software-visible function identity, not to a link.

Failure. If interrupt identity is derived from the link, then two functions on one link are indistinguishable as interrupt sources, and a function that moves links appears to software as a different interrupt source. The symptom in §23's table is an interrupt arrives from the wrong device, and the driver that receives it will act on a device that did not raise it.

DV. Cover an interrupt from each remote function, including two functions sharing a link — the case the link-indexed design cannot represent at all.

16. Outstanding Transactions, Per Link

Where host integration stops being tables and becomes real hardware.

The root complex has many transactions in flight. Some are aimed at remote functions across links that can stall, fail, and recover independently. The host needs to know how much work is committed to each link, both to bound its own resources and to reason about what happens when a link goes.

// Illustrative. Per-link accounting of host-issued work awaiting resolution.
logic [OUT_W-1:0] outstanding_q [NUM_UCIE_LINKS];
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int l = 0; l < NUM_UCIE_LINKS; l++) outstanding_q[l] <= '0;
  end else begin
    unique case ({req_accept_fire, rsp_match_fire})
      2'b10: outstanding_q[req_link]  <= outstanding_q[req_link]  + 1'b1;
      2'b01: outstanding_q[rsp_link]  <= outstanding_q[rsp_link]  - 1'b1;
      2'b11: begin
        // Different links in the same cycle must both move; the same link
        // nets to zero. Writing this as two independent if-statements is
        // the Chapter 9.5 §6 bug, and it silently loses one of the updates.
        if (req_link == rsp_link) begin
          outstanding_q[req_link] <= outstanding_q[req_link];
        end else begin
          outstanding_q[req_link] <= outstanding_q[req_link] + 1'b1;
          outstanding_q[rsp_link] <= outstanding_q[rsp_link] - 1'b1;
        end
      end
      default: ;
    endcase
  end
end

Architecture. Per-link rather than global, because links fail independently. A global count cannot answer the only question that matters during a link failure: how much of what I am owed was owed by the link that just died?

State. NUM_UCIE_LINKS counters with per-outstanding-transaction lifetime in aggregate — each increment is matched by exactly one decrement, and the counter is the difference. Note this is a resource counter in the Chapter 9.5 §8 sense: it must be asserted on, never saturated, because a count that clamps has already lost the information that would explain the failure.

Cycle behaviour. The unique case structure is the point. Chapter 9.5 §6 established why two independent if statements assigning the same register lose one update; here the additional subtlety is that simultaneous increment and decrement may target different counters, in which case both must move, and only the same-link case nets to zero.

Contract. The acceptance path uses these counts for backpressure. The recovery logic of §17 reads them to know what is at risk. The scoreboard of §24 predicts them.

Failure. Undercount and the host over-commits a link. Overcount and it throttles a healthy link, producing a performance bug that looks like a link problem. Lose an update through the two-if bug and the count drifts monotonically, so the link appears progressively more congested until it stops accepting entirely — with the cause thousands of transactions in the past.

DV. Cover each counter empty, mid-range, and at its limit. Cover simultaneous accept and match on the same link and on different links — the two branches of the 2'b11 case, and the one that gets skipped. Assert the bound rather than clamping it.

17. A Link Fails With Work Outstanding

The scenario that separates a host integration design from a diagram.

The host issued a memory read to a remote function. The read is outstanding. The UCIe link fails.

Software is waiting for data. The root complex has a record saying it is owed a response. The transport that would carry that response no longer exists. Every subsequent moment, that record is either being preserved for a recovery that may come, or it is stale.

The architecture must define, in advance:

Whether recovery is attempted, and for how long. Chapter 8.x's recovery machinery may bring the link back. If it does, and if the endpoint retained its outstanding state per Chapter 10.3 §18, the transaction can complete normally and software never learns anything happened. This is the good case and it is worth designing for.

What resolves the request if recovery does not come. The requester's expectation must end somehow. PCIe defines mechanisms for a requester whose completion does not arrive, and platforms define error-reporting behaviour for unrecoverable conditions. This chapter does not restate their thresholds, encodings, or escalation rules, because doing so accurately requires the relevant specification text and approximating them would be worse than omitting them.

Whether the host abandons or waits. These are different designs with different failure modes: abandoning too early breaks transactions that would have completed after a normal recovery; waiting indefinitely converts a link failure into a system hang.

18. Wrong RTL — Clear Outstanding State on Link Reset

// WRONG — the link's lifetime is applied to state that does not share it.
always_ff @(posedge clk) begin
  if (ucie_link_reset[l])
    outstanding_q[l] <= '0;      // "clean slate"
end

Architecture. The reasoning is superficially sound: the link is being re-established, so its state should be re-initialised. Chapter 9.5 §13 said exactly that about credits, and Chapter 9.4 about replay state. The error is applying a rule that is correct for link-epoch state to state that has per-transaction lifetime and an owner above the link.

Cycle behaviour. One pulse, and the count is zero. The RTL is short and reads as tidy housekeeping.

Failure. The host now believes it is owed nothing. Software, which knows nothing about UCIe link epochs, is still waiting for a read to return. The two models have diverged, and the divergence is silent — no error was signalled, because from the transport's point of view a reset is a normal part of recovery.

Worse follows. If the link recovers and the endpoint retained its outstanding entries per Chapter 10.3 §18, responses will arrive for transactions the host has forgotten. They match nothing. Depending on the response path, they are dropped — leaving software waiting forever — or they match a later transaction that has since reused the same tracking slot, delivering one requester's data to another. That last case is the same silent-misdelivery family as §11 and Chapter 10.2 §7.

And the counter is now permanently wrong in the other direction: transactions that were counted and cleared will never decrement again, so the count underflows on the next legitimate response. If the counter saturates rather than asserting, that underflow becomes a large positive value and the link appears permanently congested.

The right shape. Link-epoch state re-initialises with the link. Per-transaction state persists and is resolved by the §17 policy — completed after recovery, or abandoned through an explicit path that tells the requester. The two must be separated in the RTL, not just in the designer's head.

DV. Inject link loss with a non-zero outstanding count on that link and a non-zero count on a different link. Check that the other link's count is untouched — a surprisingly common secondary bug — and that every transaction on the failed link reaches a defined resolution that the scoreboard can observe.

// Illustrative — encode the contract so the coupling cannot be added later.
property p_outstanding_survives_link_reset;
  @(posedge clk) disable iff (!rst_n)
    (ucie_link_reset[l] && outstanding_q[l] != '0)
      |=> (outstanding_q[l] == $past(outstanding_q[l]));
endproperty
 
property p_outstanding_bounded;
  @(posedge clk) disable iff (!rst_n)
    outstanding_q[l] <= MAX_OUTSTANDING_PER_LINK;
endproperty
 
// Independence: one link's reset must not disturb another's accounting.
property p_link_reset_isolated;
  @(posedge clk) disable iff (!rst_n)
    (ucie_link_reset[l] && (m != l))
      |=> (outstanding_q[m] == $past(outstanding_q[m]));
endproperty

19. Root-Complex Readiness and the Enumeration Sequence

Remote routing must not be enabled until everything it depends on is in place. That is a conjunction, and by now the shape is familiar.

// Illustrative — the host-side counterpart of Chapter 10.3 §5.
assign remote_route_ready[i] =
    route_table_valid[i]      &&   // §8: the path is programmed
    link_operational[route_link[i]] && // the path is usable
    ep_reachable[i]           &&   // §6: the far end will answer
    windows_programmed[i];         // §10: addresses have meaning

And the sequencing that establishes those preconditions:

// Illustrative host integration FSM. This is NOT a PCIe LTSSM and not a
// UCIe link state machine — it is the host's own bring-up sequencer, and
// it exists at a layer above both.
typedef enum logic [2:0] {
  HOST_DISCOVER_LINK,   // wait for the UCIe link to report operational
  HOST_PROBE_ENDPOINT,  // establish that a function is there and answering
  HOST_ASSIGN_RES,      // program identity routes and address windows
  HOST_ENABLE_ROUTE,    // only now may traffic be routed to it
  HOST_ACTIVE
} host_enum_state_t;
 
host_enum_state_t host_enum_q;

Architecture. The states exist because each represents a set of preconditions that the next state's actions require. Probing before the link is operational produces timeouts. Assigning resources before probing assigns them to something that may not be there. Enabling routes before assigning resources is §21's bug.

State. One register per remote function or per link depending on the design's granularity, with per-enumeration lifetime.

Cycle behaviour. Advances on defined evidence, not on timers where evidence is available. The transition out of HOST_DISCOVER_LINK is the one place where a timeout is genuinely appropriate, because absence of a link is not an event.

Contract. HOST_ACTIVE is what the rest of the system reads as "this function is usable". Nothing downstream should be inspecting the intermediate states to make routing decisions.

Failure. Skipping or reordering states produces the failures in §21 and §7 respectively. A state machine that can reach HOST_ACTIVE without passing through HOST_ASSIGN_RES — for instance via a recovery shortcut added later — reintroduces §21's bug through a path nobody reviewed.

DV. Cover every state. Cover link loss in each state, which is where recovery paths are added carelessly. Assert that HOST_ACTIVE is only reachable through the full sequence.

20. Wrong RTL — Routing Enabled Before Resources

// WRONG — routing turned on as soon as the far end answers.
assign remote_route_enable[i] = ep_reachable[i];

Architecture. Reachability was established in §6 and it is genuine. It is also insufficient: it says the function will answer, not that anyone has told the hardware where its addresses are.

Failure. CPU memory traffic is routed toward a function whose address windows are not yet programmed, or are partially programmed. The aperture table either has no matching entry — so the transaction takes the unclaimed-address path — or, during a partial programming sequence, has an entry with a base written and a limit not yet written, which is an aperture describing a range nobody intended. In the worst arrangement that half-written window overlaps another function's, which is §11 arriving by a different route.

The general form is worth stating: any table that is read while it is being written must be written atomically or held out of service until complete. Chapter 8.5 §12 made this argument for link configuration with shadow registers and a commit at a quiesced boundary; the same structure applies here, and for the same reason.

DV. Program windows in a deliberately slow, interleaved order while traffic is offered. Cover the window between the first write and the last. This is not an exotic scenario — it is what happens on every boot, and it is only safe because something holds routing off.

21. A Word About Hotplug

Some behaviours in this chapter resemble hot-plug: a device appearing after boot, a device becoming unreachable and returning, resources needing to be assigned to something that was not there a moment ago.

The resemblance is architectural, not protocol-level. A UCIe-attached chiplet is not automatically a PCIe hot-plug device, and using the term loosely causes two concrete problems.

It implies protocol machinery that may not be present. PCIe hot-plug is a defined capability with defined behaviour and software support. Saying "it's like hotplug" invites the assumption that the corresponding mechanisms exist.

And it points debugging in the wrong direction. An engineer told a symptom is "a hotplug issue" will look at hot-plug infrastructure. If the actual cause is §7's presence-from-strap or §18's cleared outstanding state, that search finds nothing.

Use the precise language instead: a function became reachable, a function became unreachable, the transport recovered. Those are the events that actually occurred.

22. Software Transparency Is the Product

Worth stating plainly, because it is the reason all of this machinery exists.

Host software should not need to know that a PCIe function sits on another die — unless platform policy deliberately chooses to expose that topology.

The qualification matters. There are real reasons a platform might want topology visible: NUMA-style locality decisions, thermal or power management, fault isolation and reporting, or serviceability. Those are policy choices, made once, exposed through defined mechanisms.

What §14's bug does is different. It exposes topology accidentally, through a protocol field that software was never told to interpret that way, with no mechanism, no documentation, and no way to turn it off. That is not a policy decision; it is a leak that later becomes a compatibility constraint because something started depending on it.

The value delivered by getting this right is concrete: the chiplet can be re-partitioned between revisions, functions can move between links, and the same software runs. That is what modularity buys, and it is only worth what the abstraction's integrity is worth.

23. State Lifetimes, Side by Side

The row that carries the chapter is the outstanding count, because it is the one whose lifetime is shorter than the tables around it and independent of the link beneath it. Everything else is either static, per-enumeration, or per-link-epoch — three tidy categories that tempt a designer into thinking there are only three. The outstanding count belongs to none of them, which is exactly why §18's bug is written so easily.

24. Host-Side Debug Taxonomy

The last row deserves its own note. Boot-to-boot variability in this layer is almost always an ordering assumption: something worked because the link happened to come up before software probed, and failed when it did not. Those bugs are not flaky hardware; they are missing preconditions, and §19's conjunction is what makes them impossible rather than unlikely.

25. The Host Integration Scoreboard

The reference model, and it must model both graphs or it cannot check the mapping between them.

pcie_topology         — functions, identities, hierarchy, as software built it
resource_map          — every programmed address window, with its owner
expected_routes       — for each identity and each address: the predicted link
package_topology      — links, their epochs, their current states
outstanding_model     — per-link expected counts and per-transaction records

The checks that matter:

On every request, predict the destination independently. From pcie_topology and resource_map, the model computes which function the transaction belongs to; from expected_routes, which link it should take. Comparing that against the hardware's actual choice is the only check that catches a wrong route rather than an invalid one — the thing §9's and §12's assertions structurally cannot do.

On every completion, match it to its request and its path. A completion that returns on a different link than its request departed on is either a routing bug or evidence that the topology model is wrong; both are worth knowing.

Predict per-link occupancy continuously. Divergence between outstanding_model and the hardware counters is the earliest possible detection of §18's bug, and it appears at the moment the state is cleared rather than when a response later fails to match.

Check the two graphs never mix. This is the check unique to this chapter: assert that no protocol-visible identity in pcie_topology is a function of any value in package_topology. In practice this is done by running the same test with a deliberately different link assignment and confirming that every software-visible value is unchanged. That single experiment catches §14's bug and every variant of it, and no amount of single-configuration testing will.

That last technique is worth adopting as standard practice. If the abstraction holds, permuting the physical topology changes nothing software can see. If it does not hold, the diff tells you exactly which value leaked.

26. Coverage

Topology shape. One remote function; several functions on one link; several links each with one function; several links each with several functions. The fourth is the configuration that breaks link-indexed identity, and it is the one most often left untested because the first is enough to make a demo work.

Aperture geometry. A single window; two windows with a gap; two windows exactly adjacent with no gap; and — by deliberate injection, since it should be prevented — two overlapping. Adjacent-with-no-gap is where boundary arithmetic must agree exactly, and it is the realistic case that finds >= versus > errors.

Address positions. For each window: base, base+1, limit−1, limit, base−1, limit+1. Six points per window, cheap, and they contain every off-by-one this decode can have.

Link events against enumeration position. Link down before discovery; during probe; between resource assignment and route enable; after active. The third is §20's window and is the one a directed test never lands in by accident.

Link events against traffic. Loss with zero, some, and maximum outstanding transactions on that link — crossed with non-zero outstanding on a different link, which is what proves isolation.

Recovery outcomes. Recovery that completes and allows the transaction to finish; recovery that fails and forces the defined abandonment path. Both must be exercised, because they are different code.

Interrupts. One from each remote function, including two functions sharing a link.

Rejected configurations. An overlapping window offered at programming time; a route to a non-existent link; a duplicate identity. Each must be refused or reported, and testing only valid configurations means the validation logic ships unexercised.

The cross that matters. Endpoint count × window arrangement × link-event position. Small enough to close, and it contains the overwhelming majority of the bugs in this chapter.

27. Debug Checklist

1. Did the UCIe link reach its operational state — Chapter 8.6's state, not an inference?
2. Did the remote function become reachable, and by §6's conjunction rather than a strap?
3. Was a configuration route installed, and is it valid?
4. Does that route point at a link that is operational now?
5. Were address windows programmed, completely, before routing was enabled?
6. Does the address hit exactly one aperture?
7. Does that aperture point at the correct link — checked against a model, not by inspection?
8. Was the request accepted, or refused by a precondition, or silently dropped?
9. Was per-link outstanding state allocated?
10. Did the completion return by the expected path, and match the right request?
11. Did a UCIe recovery occur at any point during this transaction's life?
12. Was outstanding state cleared by that recovery when it should not have been?
13. Did interrupt identity stay tied to the function rather than the link?
14. Is a software-visible PCIe value being derived from a package-physical value anywhere?

Question 14 is the one that resolves the whole class of bug this chapter names, and §25's permuted-topology experiment answers it definitively rather than by inspection.

28. Common Misconceptions

"Host software should know the UCIe link number." Only if platform policy deliberately exposes it through a defined mechanism. Leaking it through a protocol field, as in §14, is not a policy decision — it is an accident that becomes a compatibility constraint.

"PCIe topology and package topology are the same graph." They are two graphs related by a table. One is software-assigned per enumeration; the other is fixed at design time. §2 develops why the relationship cannot be a formula.

"Endpoint presence can come from a package strap." A strap says a die is fitted. Reachability is a live property depending on link state, mapping resources, and remote readiness. §5 separates them and §7 shows what conflating them costs.

"BAR routing is independent of link readiness." The aperture table says which link; whether that link can carry anything is separate state with a different lifetime. A valid aperture pointing at a dead link is a normal, transient situation that must have defined behaviour.

"A UCIe link ID can serve as a requester ID." Three identity spaces, three assigners, three lifetimes. §13's table is the argument, and §14 lists four distinct failures that follow.

"Clearing outstanding transactions on link retrain is safe." It is the strongest integration bug in this chapter. Link-epoch state re-initialises; per-transaction state does not, because its owner is above the link and is still waiting.

"A UCIe-attached endpoint is automatically hot-plug." The resemblance is architectural. The protocol machinery is a separate question, and the loose terminology sends debugging to the wrong place.

"Enumeration success proves MMIO routing." They use different tables with different lifetimes. Configuration routing matches on identity; address routing matches on address. Either can be correct while the other is wrong — which is exactly the second row of §24.

"One root-complex scoreboard is enough." Not without modelling UCIe route state. A model that knows only the PCIe graph can tell you a completion is missing; it cannot tell you the request went to the wrong die, because in its world there is only one die.

29. Understanding Check

30. Summary — and What Module 10 Built

The host should see PCIe topology; the implementation knows UCIe topology.

The mechanisms: presence separated from reachability, because a strap cannot back a claim about a live link. Two routing tables with different lifetimes — configuration matched on identity, memory matched on address — both of them tables rather than formulas, because the two graphs correspond arbitrarily. One-hot validation on every decode, the cheapest high-value assertion in the module, because two matching windows deliver a well-formed transaction to the wrong die in silence. Three identity spaces kept apart, with the link-as-requester-identity shortcut failing in four distinct ways. Per-link outstanding accounting that survives link recovery, because its owner is above the link and is still waiting. And a readiness conjunction gating route enable, so the boot-ordering bug becomes impossible rather than unlikely.

The verification idea worth taking away: run the test again with the links permuted. If the abstraction holds, nothing software can see changes. If it does not, the diff names the leak.

What Module 10 established

10.1 — PCIe's transaction semantics survive UCIe tunnelling; the Adapter takes over the data-link role; an accepted object becomes the transport's responsibility.

10.2 — PCIe packet objects map through UCIe natively, with flit definitions aligned rather than a container invented; the transport may retry freely, and the transaction must still be delivered exactly once.

10.3 — a remote function becomes a valid semantic PCIe endpoint, through composed readiness, deliberate reset domains, and recorded obligations.

10.4 — the host absorbs that function into normal PCIe topology and resources, without letting package physics become protocol identity.

PCIe can now discover devices across a package boundary and move transactions to them correctly. Which raises the question that the next module exists to answer:

What if the remote chiplet is not merely an I/O endpoint? What if it contributes memory that CPUs should load and store coherently, or hosts an accelerator whose cache participates in the host's memory hierarchy?

PCIe's transaction semantics are excellent for moving data to and from devices. They were not designed to make a device's memory part of the host's coherent address space, or to let a device cache host memory and stay coherent while doing it:

• 11.1 — Why CXL Matters — why PCIe semantics alone are insufficient for coherent memory and accelerator attach, and what changes when CXL rides a UCIe link.

Browse the full path on the UCIe tutorials index.