UCIe · Module 24
Future Architectures
What multiple packages, multiple suppliers and multiple transport scopes do to identity, routing and configuration — the double-buffered route table that survives an update under live traffic, why software-defined does not mean software in the datapath, retry composed across two transports, and a distributed deadlock where every component is locally correct.
Chapter 24.3 kept one package and two dies. This chapter removes that limit — and the hard problem stops being connection and becomes composition.
1. The One-Sentence Model
A future system is not "more chiplets." It is more independent ownership domains connected through explicit contracts. Every domain added brings identity, routing, configuration, failure, recovery, trust, locality and observability with it — and those compose far less gracefully than links do.
Connecting two more dies is an engineering exercise. Composing two more ownership domains is a state-lifetime problem — which is why the three flagship failures in this chapter (§13, §17, §19) all involve state that outlived the thing that created it.
2. What This Chapter Owns
| Question | Where |
|---|---|
| Ecosystem roles; what suppliers publish | 24.1 — Open Chiplet Ecosystems |
| Manifest, compatibility predicate, atomic commit, identity scopes | 24.2 — The Chiplet Marketplace Vision |
| Partitioning, contract freeze, microarchitecture leaks | 24.3 — Modular Semiconductor Design |
| Transport scope; package-local vs scale-up; retry at two layers | 23.4 — UCIe vs NVLink |
| Identity, generation, ordering across a die boundary | 22.4 — Data-Centre Processors |
| Credit conservation; throughput classes; protocol violations; silicon evidence | 21.4 · 21.5 · 21.6 · 21.7 |
| Interview preparation from the ground up | Module 25 (next) |
Four things are new here:
The control-plane / data-plane split (§8–§9) — software-defined means software sets policy, and the datapath reads only committed state. Never per-transaction firmware.
The double-buffered route table (§10–§13) — this chapter's centrepiece RTL, and the flagship failure it prevents: a route updated under live traffic where the request takes the old path and the response is mapped under the new one.
Retry composed across scopes (§16–§17), extending 23.4 §13 to three transports and a bridge.
And a distributed deadlock (§19) where every component is locally correct and the wait cycle spans three ownership domains — which no component-level verification can find.
3. Sourcing
4. Three Transport Scopes
23.4 §11 established two. A multi-package system has three, and they differ in every dimension that matters.
| LOCAL_DIE | LOCAL_PACKAGE | REMOTE_PACKAGE | |
|---|---|---|---|
| transport | none — on-die fabric | die-to-die link | system-scale interconnect |
| reach | microns | millimetres | chassis / rack |
| topology | fixed | fixed at design time | switched, reconfigurable |
| addressing | local | a small die graph | discoverable space |
| failure model | fails with the die | dies fail together | a peer can fail alone |
| error rate | negligible | low | budgeted, non-trivial |
| latency class | 1 | ~10× | ~100×+ |
| may reroute? | no | rarely | yes — §13 |
Three readings.
The last row is what makes this chapter necessary. A reroutable scope means a destination's path can change while a transaction is outstanding — which no single-package design has to handle.
Latency classes differ by orders of magnitude, so a scheduler or allocator that treats them identically will make systematically bad placements (§20).
And the failure models are qualitatively different, not merely quantitatively: a remote peer that fails alone must be isolated, whereas a local die failing with its package needs no isolation mechanism at all (§18).
5. A Conceptual Multi-Package System
Three things to read.
The whole topology is CONCEPTUAL (§3) and is not attributed to any product.
Software reaches the control plane and never the data plane. §8's discipline drawn as an absent edge — there is no path from sw to dp.
And the bridges are where scope changes. 23.4 §15's t_bridge term is real work, paid twice on a round trip — and §16 is what a bridge must preserve.
6. Routing by Scope
// ILLUSTRATIVE SYSTEM ARCHITECTURE (§3). Scope is a property of the
// DESTINATION, resolved once and carried — never re-derived downstream.
typedef enum logic [1:0] {
SCOPE_LOCAL_DIE = 2'd0,
SCOPE_LOCAL_PACKAGE = 2'd1,
SCOPE_REMOTE_PACKAGE= 2'd2
} route_scope_e;
typedef struct packed {
logic [SEM_W-1:0] sem_id; // the OPERATION — stable everywhere
logic [GEN_W-1:0] generation; // which use of that id (22.4 §12)
logic [15:0] dst_logical_id; // NOT a link, NOT a port (24.2 §15)
route_scope_e scope; // resolved at issue
logic [EPOCH_W-1:0] route_epoch; // WHICH route table produced it (§11)
logic [EPOCH_W-1:0] cfg_epoch; // which contract it runs under (24.2 §11)
} sys_txn_t;
// Transport selection is a pure function of scope. There is no silent default:
// an unresolvable scope is OBSERVABLE, because a dropped transaction with no
// record is 21.6 §31's causeless event seen from the sending side.
always_comb begin
local_valid = 1'b0; d2d_valid = 1'b0; fabric_valid = 1'b0; scope_error = 1'b0;
unique case (txn.scope)
SCOPE_LOCAL_DIE: local_valid = txn_valid;
SCOPE_LOCAL_PACKAGE: d2d_valid = txn_valid;
SCOPE_REMOTE_PACKAGE: fabric_valid = txn_valid;
default: scope_error = txn_valid;
endcase
endArchitecture. One resolution, carried on the transaction, driving a three-way selection with three genuinely different transport contracts behind it (§4).
State. None here — the transaction carries it.
Event. Scope is resolved at issue, from the destination's logical identity. Downstream blocks read txn.scope and never recompute it (22.3 §11's captured-not-recomputed rule).
Contract. The transaction carries two epochs. cfg_epoch says which contract it runs under (24.2 §11); route_epoch says which route table produced its path. They change independently, and §13 is what happens when the second is missing.
Failure. A silent default arm routes an unresolvable destination somewhere or nowhere, producing an orphan with no record (21.6 §31). scope_error makes it a finding instead.
DV/debug. scope and both epochs belong in the trace event (21.7 §16). A SCOPE_LOCAL_PACKAGE transaction observed on the fabric interface is a one-line finding rather than a latency mystery (23.4 §12).
7. Wrong RTL — the Link Number Is the Destination
// WRONG — the destination field IS a physical port index.
assign egress_port = txn.dst; // dst is a port number
assign resp_match = (resp_port == pending_port[resp_slot]); // correlate on PORTWhy this is worse in a multi-package system than in 24.2 §14's single package:
| Single package | Multi-package | |
|---|---|---|
| how often does the path change? | rarely | routinely — failover, rebalancing, policy |
| what changes with it? | a link | a whole scope transition |
| outstanding work at the time | some | more, and longer-lived (22.4 §8) |
| correlation on a port | breaks on failover | breaks on every reroute |
And the fix is 24.2 §15's, now load-bearing: correlate on sem_id + generation, route on a separate table, and stamp the transaction with the route_epoch it was issued under — because in a reroutable scope, "which table produced this path?" is a question that will be asked.
8. Software-Defined Does Not Mean Software in the Datapath
Software sets policy. Hardware executes it. Software-defined means the topology, mappings, partitions and capabilities are configurable — not that firmware participates in forwarding a transaction.
| Software may decide | Hardware must do |
|---|---|
| which logical resource maps where | forward every transaction deterministically |
| routing policy and preferences | select a path from committed state |
| partitioning and isolation | enforce it per transaction |
| power and QoS policy | apply it at line rate |
| when a change takes effect | commit it atomically (§10) |
And the anti-pattern is per-transaction firmware. A datapath that consults software for a forwarding decision has latency and jitter set by an interrupt path — which for a coherent or latency-sensitive operation (22.4 §9) is not a performance choice but a correctness one.
9. Control Plane and Data Plane
ILLUSTRATIVE architecture (§3).
SOFTWARE / CONTROL PLANE
writes REQUESTED policy (never active, never the datapath)
|
HARDWARE CONTROL PLANE
validate -> is the request self-consistent and within capability?
prepare -> stage it in a shadow structure (§10)
quiesce -> are the affected transactions drained or trackable?
commit -> atomic swap + epoch increment
|
DATA PLANE
reads ACTIVE state ONLY. It never sees requested, never sees shadow,
and never observes a partially-updated structure (§12).Two properties.
Validation happens before staging, and quiescence before commit. Committing a request that was never validated puts an inconsistent table into the datapath; committing without quiescence produces §13.
And the data plane's read set is deliberately narrow. It reads active structures and nothing else — which is what makes §12's assertion writable at all.
10. The Double-Buffered Route Table
// ILLUSTRATIVE SYSTEM ARCHITECTURE (§3). The centrepiece of this chapter:
// a route table that can be updated under live traffic without any transaction
// ever observing a partially-written state.
module route_table_2bank #(
parameter int N_DEST = 64,
parameter int PORT_W = 4,
parameter int EPOCH_W = 8
) (
input logic clk,
input logic rst_n,
// ---- control plane (§9) ----
input logic shadow_we, // software writes shadow ONLY
input logic [$clog2(N_DEST)-1:0] shadow_addr,
input logic [PORT_W-1:0] shadow_port,
input route_scope_e shadow_scope,
input logic shadow_valid_in,
input logic validate_req, // "check the shadow"
input logic commit_req, // "make it active"
input logic quiesced, // no affected txn in flight
output logic shadow_validated_q,
output logic commit_done,
output logic [EPOCH_W-1:0] route_epoch_q,
// ---- data plane ----
input logic [$clog2(N_DEST)-1:0] lookup_addr,
output logic [PORT_W-1:0] lookup_port,
output route_scope_e lookup_scope,
output logic lookup_valid
);
typedef struct packed {
logic [PORT_W-1:0] port;
route_scope_e scope;
logic valid;
} route_e_t;
route_e_t bank_q [2][N_DEST];
logic active_sel_q; // which bank the DATA PLANE reads
// ---------------- DATA PLANE ----------------
// Reads the ACTIVE bank only. It cannot observe the shadow, ever.
assign lookup_port = bank_q[active_sel_q][lookup_addr].port;
assign lookup_scope = bank_q[active_sel_q][lookup_addr].scope;
assign lookup_valid = bank_q[active_sel_q][lookup_addr].valid;
// ---------------- CONTROL PLANE ----------------
logic shadow_sel;
assign shadow_sel = ~active_sel_q; // always the other bank
// Validation: every entry must name a legal port for its scope, and a valid
// entry must not point at a port that is down. Deliberately a SEPARATE step
// from commit, so an inconsistent table can never become active (§9).
logic shadow_ok;
always_comb begin
shadow_ok = 1'b1;
for (int i = 0; i < N_DEST; i++) begin
if (bank_q[shadow_sel][i].valid) begin
if (!port_legal_for_scope(bank_q[shadow_sel][i].port,
bank_q[shadow_sel][i].scope)) shadow_ok = 1'b0;
if (!port_up(bank_q[shadow_sel][i].port)) shadow_ok = 1'b0;
end
end
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
active_sel_q <= 1'b0;
route_epoch_q <= '0;
shadow_validated_q <= 1'b0;
commit_done <= 1'b0;
for (int b = 0; b < 2; b++)
for (int i = 0; i < N_DEST; i++) bank_q[b][i] <= '{default: '0};
end else begin
commit_done <= 1'b0;
// Any write to the shadow INVALIDATES a previous validation. Otherwise a
// late write could ride in on an earlier approval.
if (shadow_we) begin
bank_q[shadow_sel][shadow_addr] <= '{ port: shadow_port,
scope: shadow_scope,
valid: shadow_valid_in };
shadow_validated_q <= 1'b0;
end else if (validate_req) begin
shadow_validated_q <= shadow_ok;
end else if (commit_req && shadow_validated_q && quiesced) begin
// THE ATOMIC SWAP. One bit changes which bank the data plane reads,
// so every lookup sees either wholly-old or wholly-new — never a mix.
active_sel_q <= shadow_sel;
route_epoch_q <= route_epoch_q + 1'b1; // exactly once
shadow_validated_q <= 1'b0; // the new shadow is stale
commit_done <= 1'b1;
end
end
end
endmoduleArchitecture. Two banks, a one-bit active selector, and a validate → commit sequence. The datapath's entire coupling to configuration is one multiplexer select.
State. 2 × N_DEST entries, the active-bank bit, a validation flag, and the route epoch.
Event. Software writes the shadow only. validate_req checks it. commit_req swaps only if validated and quiesced — three conditions, deliberately separate.
Contract. Three properties do the work. A shadow write clears shadow_validated_q, so a late write cannot ride in on an earlier approval. The swap is one bit, so no lookup can see a mixture. The epoch increments exactly once per commit, which is what lets a response be attributed to the table that produced its request (§11).
Failure. Committing without quiesced produces §13 — the request takes the old route and the response is mapped under the new one. Validating and committing in the same cycle removes the ability to reject an inconsistent table. Letting the datapath read shadow_sel defeats the entire structure.
DV/debug. route_epoch_q on every transaction (§6) is what makes a cross-update trace interpretable. A response whose route_epoch predates the current one is a straggler, not a violation — and telling those apart requires the field.
11. Route Epoch and Transaction Lifetime
| Quantity | Stable for | Changes on |
|---|---|---|
sem_id + generation | the operation's whole life | never, while live (22.4 §12) |
dst_logical_id | the operation's whole life | never, while live |
route_epoch | stamped at issue, carried | a route commit (§10) |
cfg_epoch | stamped at issue, carried | a contract commit (24.2 §11) |
| the path taken | one attempt | any reroute |
Two properties.
A live transaction keeps the epoch it was issued under. New transactions use the new table; old obligations remain correlatable — which is what makes an update-under-load safe rather than merely fast.
And the two epochs are independent. A route change need not change the contract, and a contract change need not change routes. Merging them into one counter makes every route change look like a contract change, invalidating traffic that was perfectly fine.
12. Assertions — Route Table Atomicity
// MANDATORY. Illustrative architectural properties (§3) — not normative.
// (1) THE ACTIVE BANK CHANGES ONLY ON A VALID COMMIT.
// English: the data plane's view changes only when a validated shadow is
// committed at quiescence. Fires on §13's premature or unvalidated commit,
// in simulation, at the cycle of the commit.
a_active_only_on_valid_commit: assert property (
@(posedge clk) disable iff (!rst_n)
$changed(active_sel_q)
|-> ($past(commit_req) && $past(shadow_validated_q) && $past(quiesced))
);
// (2) AN UNVALIDATED SHADOW CANNOT BECOME ACTIVE.
// English: validation is a precondition, not advice.
a_no_unvalidated_activation: assert property (
@(posedge clk) disable iff (!rst_n)
$changed(active_sel_q) |-> $past(shadow_validated_q)
);
// (3) THE EPOCH INCREMENTS EXACTLY ONCE PER COMMIT.
// English: one commit, one epoch. A double increment breaks straggler
// attribution; no increment makes an old response indistinguishable from new.
a_epoch_once_per_commit: assert property (
@(posedge clk) disable iff (!rst_n)
commit_done |-> (route_epoch_q == $past(route_epoch_q) + 1)
);
a_epoch_stable_without_commit: assert property (
@(posedge clk) disable iff (!rst_n)
!commit_done |=> $stable(route_epoch_q)
);
// (4) A LIVE TRANSACTION'S DESTINATION AND EPOCH ARE STABLE.
// English: an in-flight operation is never re-stamped by a route change.
// This is what makes §13's correct version work.
a_live_txn_immutable: assert property (
@(posedge clk) disable iff (!rst_n)
txn_live |-> ($stable(txn_dst_logical_id) && $stable(txn_route_epoch))
);
// (5) THE DATA PLANE NEVER READS THE SHADOW.
// English: lookups are served from the active bank only. Structural
// non-interference — discharge formally (21.7 §17).
a_lookup_from_active_only: assert property (
@(posedge clk) disable iff (!rst_n)
lookup_valid |-> (lookup_port == bank_q[active_sel_q][lookup_addr].port)
);Architecture. Five properties: two on commit preconditions, two on epoch behaviour, one structural.
State. The bank selector, validation flag and epoch.
Sampled timing. Property (1) uses $past on all three preconditions because they must have held in the cycle the commit was accepted, not when its effect appears. Property (3) is written as a pair — increments-once and stable-otherwise — because either alone permits the other failure.
Contract. Property (4) requires txn_live to be true for the whole transaction lifetime, including across a reroute. A liveness signal that clears on a route change makes the property vacuous exactly when it matters.
Failure if omitted. Without (1), §13 ships. Without (3)'s second half, an epoch that silently fails to increment makes every straggler look current — which is worse than no epoch, because the field is trusted.
DV/debug. Property (5) belongs in a formal flow: non-interference is a proof obligation, and simulation shows only that it held for the stimulus you ran.
13. Flagship — Route Update Under Live Traffic
The wrong implementation writes the live table entry by entry.
// WRONG — software updates the active table directly, one entry per write.
always_ff @(posedge clk)
if (sw_we) route_tbl[sw_addr] <= '{ sw_port, sw_scope, 1'b1 }; // LIVE tableTwenty events, illustrative:
| # | Cycle | Event |
|---|---|---|
| 1 | 1,000 | txn A issued to dst=7; table says port 2; request leaves on port 2 |
| 2 | 1,001 | txn B issued to dst=9; table says port 2 |
| 3 | 1,004 | software begins remapping: writes dst=7 → port 5 |
| 4 | 1,005 | txn C issued to dst=7; reads the table → port 5 |
| 5 | 1,006 | software writes dst=9 → port 5 |
| 6 | 1,008 | A's response arrives on port 2 |
| 7 | 1,008 | the receiver correlates by looking up dst=7 → now port 5 |
| 8 | 1,008 | mismatch: response port 2, expected port 5 → A is dropped as spurious |
| 9 | 1,012 | B's response arrives on port 2; same mismatch; B dropped |
| 10 | 1,015 | C's response arrives on port 5; matches; C completes |
| 11 | 1,200 | A's timeout fires; A's identity is freed |
| 12 | 1,201 | a new transaction A′ allocates the freed identity |
| 13 | 1,260 | a late retransmission of A's response arrives (transport retry) |
| 14 | 1,260 | it now matches A′ — same identity, no generation check |
| 15 | 1,260 | A′ completes with A's data — silent corruption (22.4 §11) |
| 16 | — | no error was logged at any point |
| 17 | — | link counters clean; CRC clean; conservation closes |
| 18 | — | the symptom surfaces as application data corruption |
| 19 | — | the trigger is "a routine routing policy change" |
| 20 | — | it reproduces only when a response is in flight during the update |
Five readings.
Steps 6–9 are the primary bug, and they are a correlation failure, not a routing failure: the response went to the right place and was matched against a table that had moved.
Steps 11–15 are the escalation, and they are why this is a corruption rather than a stall — 22.4 §12's generation tagging is the second half of the fix.
Step 17 is what makes it hard. Every transport-level instrument is clean, so 21.4's ledger and 21.5's counters all say the link is healthy.
Step 20 is the reproduction condition, and it is why this survives testing: an update with no traffic in flight works perfectly.
And the correct version fixes it twice over. §10's atomic swap with quiescence means no transaction is issued under one table and correlated under another; §11's route_epoch on the transaction means that even a straggler arriving after a legitimate later change is attributable rather than spurious.
14. Readiness Is a Hierarchy
A resource becomes usable in stages, and publishing it early produces §15.
| Stage | Means | Publishing here is |
|---|---|---|
| physically present | the die is in the package | far too early |
| link up | the transport trained (21.1) | still too early |
| capability exchanged | the manifest was read (24.2 §6) | too early |
| contract committed | both sides agreed and committed (24.2 §11) | nearly |
| routes committed | the path exists in the active table (§10) | nearly |
| logically activated | the resource accepts work | correct |
| software-published | the OS sees it | after the above |
15. Wrong — Publish on Physical Detection
// WRONG — one bit, set on presence.
assign resource_ready = chiplet_present;| Step | Event |
|---|---|
| 1 | the chiplet is detected as present |
| 2 | resource_ready asserts; software enumerates it |
| 3 | software issues the first access |
| 4 | the contract has not committed — the peer interprets under defaults |
| 5 | no route exists — the lookup returns valid = 0 |
| 6 | the access fails, or worse, is misinterpreted |
| 7 | software marks the resource failed and does not retry |
| 8 | the resource becomes usable milliseconds later — and is never used |
And the fix is a readiness vector, not a bit:
// CORRECTED. Readiness is a conjunction, and each term is independently
// observable so a stuck stage is diagnosable (21.1 §17's waiting-reason).
typedef struct packed {
logic present;
logic link_up;
logic caps_exchanged;
logic contract_committed;
logic routes_committed;
logic logically_active;
} readiness_t;
readiness_t rdy_q;
assign resource_ready = rdy_q.present && rdy_q.link_up && rdy_q.caps_exchanged
&& rdy_q.contract_committed && rdy_q.routes_committed
&& rdy_q.logically_active;
// MANDATORY. English: no traffic is issued to a resource before every
// readiness term holds. Fires at the premature access of step 3.
a_no_traffic_before_ready: assert property (
@(posedge clk) disable iff (!rst_n)
txn_issue_fire |-> resource_ready
);Architecture. Six independent terms and one conjunction, so which stage is missing is directly readable.
State. Six bits per resource.
Event. Each term is set by its own completion event; none is inferred from another.
Contract. Software is published the conjunction, never an individual term — and step 7 above is why: a premature publication is not merely early, it can be permanent, because software concludes the resource is broken.
Failure. Setting logically_active on link_up collapses the hierarchy back to §15. Inferring routes_committed from contract_committed couples two independent commits and produces a window where a resource is contractually agreed and unreachable.
DV/debug. The vector is 21.1 §17's waiting-reason encoder for resources: a resource stuck below readiness reports which term is missing, instead of a timeout with no explanation.
16. Identity Across Three Transports
23.4 §14 established two attempt scopes. A multi-package path has three, plus bridges.
// ILLUSTRATIVE SYSTEM ARCHITECTURE (§3). One semantic identity; an INDEPENDENT
// attempt identity per transport boundary.
typedef struct packed {
// SEMANTIC — one per operation, stable across every hop and every retry
logic [SEM_W-1:0] sem_id;
logic [GEN_W-1:0] generation;
// PER-TRANSPORT attempt counters — none of these is semantic
logic [ATT_W-1:0] local_attempt; // source package D2D hop
logic [ATT_W-1:0] fabric_attempt; // system-scale hop
logic [ATT_W-1:0] remote_attempt; // destination package D2D hop
// CONTEXT
logic [EPOCH_W-1:0] route_epoch;
logic [EPOCH_W-1:0] cfg_epoch;
} multi_txn_t;
// Each transport increments ONLY its own field. No branch writes sem_id or
// generation — and that absence is the correctness argument.
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
local_attempt_q <= '0; fabric_attempt_q <= '0; remote_attempt_q <= '0;
end else begin
if (local_retx_fire) local_attempt_q <= local_attempt_q + ATT_W'(1);
if (fabric_retx_fire) fabric_attempt_q <= fabric_attempt_q + ATT_W'(1);
if (remote_retx_fire) remote_attempt_q <= remote_attempt_q + ATT_W'(1);
end
end
// De-duplication keys on the SEMANTIC pair ONLY. Including an attempt field
// here defeats the mechanism entirely — every retry presents a new key and
// executes (23.4 §14).
assign already_done = delivered_set_contains(rx.sem_id, rx.generation);
assign do_execute = rx_valid && !already_done;Architecture. A four-level identity — operation, plus one attempt counter per transport boundary.
State. Three attempt counters plus the receiver's delivered set.
Event. Each transport increments only its own field. A bridge forwards the semantic identity unchanged and does not touch any attempt counter that is not its own.
Contract. De-duplication keys on sem_id + generation only. This is the single most easily-broken rule in the chapter: including an attempt field looks implemented and disables the mechanism completely.
Failure. §17.
DV/debug. All three attempt fields belong in the trace. A trace showing three non-zero attempt counters, one sem_id, and one delivery is the system working — and being able to see that at a glance is what stops an engineer blaming a transport (23.4 §14).
17. Wrong Retry Composition
// WRONG — each transport allocates a NEW semantic id for its retry, because
// "a retransmission is a new request from this layer's point of view."
always_ff @(posedge clk)
if (any_retx_fire) sem_id_q <= next_free_sem_id; // <-- new semantic identity| Step | Event |
|---|---|
| 1 | operation X issued, sem_id = 100 |
| 2 | local D2D hop retransmits → sem_id becomes 101 |
| 3 | fabric hop retransmits → sem_id becomes 102 |
| 4 | remote D2D hop retransmits → sem_id becomes 103 |
| 5 | the destination sees four distinct semantic operations |
| 6 | de-duplication finds no match for any of them |
| 7 | X is executed up to four times |
| 8 | exactly-once is broken, and each transport behaved correctly |
Four readings.
Every retry was correct (21.6 §19). The defect is that a transport event was allowed to write a semantic field.
It scales with the number of scopes, so a multi-package path is strictly worse than 23.4 §13's two-transport case — and the failure grows as systems compose further.
The diagnosis points at the wrong place. The destination reports duplicate operations; the transports each report a normal retry — and the evidence exonerates every component.
And the fix is §16's absence: no branch may write sem_id or generation on a transport event, and the assertion below is a non-interference property rather than a behavioural one.
// MANDATORY. English: a transport retry never alters the semantic identity.
a_retry_preserves_identity: assert property (
@(posedge clk) disable iff (!rst_n)
(local_retx_fire || fabric_retx_fire || remote_retx_fire)
|=> ($stable(sem_id_q) && $stable(generation_q))
);18. Failure Domains Across Packages
| Domain | Scope | A failure here should |
|---|---|---|
| link | one D2D connection | retrain; obligations survive (23.3 §14) |
| chiplet | one die's function | isolate that function |
| package | one package | not reset the system |
| fabric path | one route through the system fabric | reroute (§10), retain obligations |
| system | everything | last resort |
Two properties.
A route failure and a resource failure need different responses. A route failure means the destination is fine and the path is not — reroute and keep the obligations. A resource failure means the destination is gone, and the obligations must be explicitly failed rather than retried forever.
And distinguishing them requires the readiness vector (§15): a resource whose logically_active still holds while its route is being recommitted is a routing event, not a resource event.
19. Flagship — Distributed Deadlock, Everything Locally Correct
Three suppliers, three components, three local correctness proofs, and a system that stops.
| Component | What it does | Locally correct? |
|---|---|---|
| A (compute) | issues requests to B; holds a completion slot until its response returns | yes |
| B (memory-side) | to serve A, must first fetch from C; reserves an entry for the fetch | yes |
| C (bridge to remote) | forwards to the remote package; needs a shared progress credit to return | yes |
The cycle:
| Step | State |
|---|---|
| 1 | A issues many requests; each holds a completion slot |
| 2 | B reserves entries for its fetches from C |
| 3 | C's returns need the shared progress credit pool |
| 4 | A's outstanding requests have consumed that same pool |
| 5 | C cannot return → B cannot complete → A cannot free its slots |
| 6 | A cannot free the credit C needs → cyclic wait across three ownership domains |
Five readings.
Every component satisfies its own contract, and each supplier's verification proves exactly that — 24.3 §17's pattern, extended to a three-party cycle that no pairwise test can construct.
The resource that closes the cycle is shared — the progress credit pool — which is 19.5 §8's physical-pool test failing across three organisations rather than two blocks.
It cannot be found by component conformance (20.7) or by pairwise interoperability, because the cycle needs all three.
And no specification I could reach defines a system-level progress contract for this — so the honest statement is that this requires a system-level dependency analysis that someone must own, and 24.1 §4 says that owner is the integrator.
The structural preventions are two, both derived: a per-component declaration of what it reserves and what it waits on (extending 24.2 §6's manifest), and an acyclicity check over the resulting wait-for graph — which is 13.3's analysis lifted to the composition level.
20. Locality and Telemetry
A composed system must expose enough structure for a scheduler to make sensible placements, and enough evidence for a failure to be attributed.
| Exposed | Why |
|---|---|
| locality class per resource (§4's three scopes) | a remote package is not a local chiplet |
| capacity and service class | placement needs both |
| identity, epoch pair | 21.7 §9's snapshot, per domain |
| error and recovery counters | attribution (24.1 §16) |
| useful-throughput counters | 21.5 §22's minimum set |
| outstanding and readiness | §15 |
21. The Interoperability Matrix Does Not Close
N suppliers × M chiplets × K profiles × revisions is not testable, and pretending otherwise is the ecosystem's most expensive optimism.
| Strategy | Buys |
|---|---|
| component conformance (20.7) | each component against a reference |
| formal component invariants (§22) | properties that hold against any peer |
| declared compatibility (24.2 §8) | eliminates pairings before test |
| targeted pairwise testing | the pairings actually shipped (23.2 §10) |
| system-level dependency analysis | §19's three-party cycles |
And the load-bearing row is the second. A property proved to hold regardless of the peer's implementation — no duplicate semantic completion, configuration atomicity, route stability per epoch, resource conservation — converts an untestable product into a testable one, because composition then only has to cover what the invariants do not.
22. Formal Component Invariants
| Invariant | Holds against | Why formal |
|---|---|---|
| no duplicate semantic completion | any peer, any retry pattern (§16) | the retry space is unbounded |
| configuration atomicity (24.2 §11) | any commit timing | the window is one cycle |
| route stability per epoch (§12) | any update sequence | update orders are combinatorial |
| resource conservation (21.4) | any traffic mix | soak tests find it late |
| no traffic before readiness (§15) | any bring-up order | orderings are permutations |
| non-interference (§12 property 5) | any stimulus | simulation proves only what it ran |
And the last row is the general argument. A simulation shows a property held for the stimulus exercised; a proof shows it holds for every stimulus — which is the only form strong enough to survive an unknown peer.
23. Common Misconceptions
"Future systems are current SoCs with more chiplets." §1: each addition brings identity, routing, configuration, failure, recovery, trust, locality and observability — and those compose worse than links do.
"One routing abstraction can cover every scope." §4, 23.4 §12: three scopes differ in reach, topology, failure model, error budget and latency class by orders of magnitude.
"Software-defined means software routes each request." §8: software sets policy; a datapath consulting firmware per transaction has latency set by an interrupt path.
"Changing a route table is just a register write." §13: twenty events from a routine policy change to silent data corruption, with every transport counter clean.
"Failover can clear old outstanding transactions." §18, 23.3 §14: a route failure means the destination is fine and the path is not. Obligations survive.
"Standard interfaces eliminate multi-vendor deadlock." §19: three locally correct components, a shared resource, and a cycle no pairwise test can construct.
"A common telemetry format proves interoperability." §20: it makes evidence comparable, which changes attribution from a commercial argument to a technical one. That is valuable and it is not proof.
"All resources should share one global address space." §4: three scopes with different failure and reachability models do not flatten without losing the distinctions a scheduler needs.
"More composability means less verification." §21: the matrix does not close, and the answer is formal component invariants plus targeted pairwise testing — more discipline, not less.
24. Understanding Check
25. Summary
Five things.
Composition, not connection, is the hard problem (§1). Every ownership domain brings identity, routing, configuration, failure, recovery, trust, locality and observability.
Scope is a property of the destination (§4, §6), resolved once and carried — three scopes differing by orders of magnitude in reach, latency and failure model.
Software-defined means software sets policy (§8). The datapath reads committed state only, and never a structure being written.
A route table must be double-buffered, validated and committed atomically (§10–§13). Otherwise a routine policy change becomes silent data corruption, with every transport counter clean.
And no transport event may write a semantic identity (§16–§17). Retries compose across scopes; identities must not — and the rule is an absence, checkable as non-interference.
On evidence: the UCIe facts are Level B and the scope distinction is Level C; everything this chapter builds is derived Level-E architecture (§3). No prediction, no timeline, no claim that any product implements any of it.
26. Module 24 Complete
| Ch | What it established |
|---|---|
| 24.1 | Roles and artefacts — one of eleven is standardised; the integrator owns the whole and sees the least; failure attribution has no framework |
| 24.2 | The contract as data — compatibility is a predicate, not a version; four configuration states; five identity scopes |
| 24.3 | Methodology — contracts freeze before implementations; partition quality is measurable; interfaces must not leak microarchitecture |
| 24.4 | Composition — three scopes, atomic route commits, identity across transports, and deadlocks no pairwise test can find |
And the module's through-line is one sentence. A standardised link makes a boundary implementable by strangers; everything else — what is exchanged, when it freezes, and how it composes — is engineering that someone still has to own. Module 24 is what that work consists of, derived from what composition requires rather than predicted from a roadmap.