Skip to content

UCIe · Module 24

Future Architectures

What multiple packages, multiple suppliers and multiple transport scopes do to identity, routing and configuration — the double-buffered route table that survives an update under live traffic, why software-defined does not mean software in the datapath, retry composed across two transports, and a distributed deadlock where every component is locally correct.

Chapter 24.3 kept one package and two dies. This chapter removes that limit — and the hard problem stops being connection and becomes composition.

1. The One-Sentence Model

A future system is not "more chiplets." It is more independent ownership domains connected through explicit contracts. Every domain added brings identity, routing, configuration, failure, recovery, trust, locality and observability with it — and those compose far less gracefully than links do.

Connecting two more dies is an engineering exercise. Composing two more ownership domains is a state-lifetime problem — which is why the three flagship failures in this chapter (§13, §17, §19) all involve state that outlived the thing that created it.

2. What This Chapter Owns

QuestionWhere
Ecosystem roles; what suppliers publish24.1 — Open Chiplet Ecosystems
Manifest, compatibility predicate, atomic commit, identity scopes24.2 — The Chiplet Marketplace Vision
Partitioning, contract freeze, microarchitecture leaks24.3 — Modular Semiconductor Design
Transport scope; package-local vs scale-up; retry at two layers23.4 — UCIe vs NVLink
Identity, generation, ordering across a die boundary22.4 — Data-Centre Processors
Credit conservation; throughput classes; protocol violations; silicon evidence21.4 · 21.5 · 21.6 · 21.7
Interview preparation from the ground upModule 25 (next)

Four things are new here:

The control-plane / data-plane split (§8–§9) — software-defined means software sets policy, and the datapath reads only committed state. Never per-transaction firmware.

The double-buffered route table (§10–§13) — this chapter's centrepiece RTL, and the flagship failure it prevents: a route updated under live traffic where the request takes the old path and the response is mapped under the new one.

Retry composed across scopes (§16–§17), extending 23.4 §13 to three transports and a bridge.

And a distributed deadlock (§19) where every component is locally correct and the wait cycle spans three ownership domains — which no component-level verification can find.

3. Sourcing

4. Three Transport Scopes

23.4 §11 established two. A multi-package system has three, and they differ in every dimension that matters.

LOCAL_DIELOCAL_PACKAGEREMOTE_PACKAGE
transportnone — on-die fabricdie-to-die linksystem-scale interconnect
reachmicronsmillimetreschassis / rack
topologyfixedfixed at design timeswitched, reconfigurable
addressinglocala small die graphdiscoverable space
failure modelfails with the diedies fail togethera peer can fail alone
error ratenegligiblelowbudgeted, non-trivial
latency class1~10×~100×+
may reroute?norarelyyes — §13

Three readings.

The last row is what makes this chapter necessary. A reroutable scope means a destination's path can change while a transaction is outstanding — which no single-package design has to handle.

Latency classes differ by orders of magnitude, so a scheduler or allocator that treats them identically will make systematically bad placements (§20).

And the failure models are qualitatively different, not merely quantitatively: a remote peer that fails alone must be isolated, whereas a local die failing with its package needs no isolation mechanism at all (§18).

5. A Conceptual Multi-Package System

A conceptual block diagram of a multi-package system. At the top, software sets policy, which feeds a control plane that validates and commits configuration. The control plane publishes committed state to the data plane. Package zero contains compute chiplets, a memory-side chiplet and a bridge chiplet connected by die-to-die links. Package one has the same structure. A system-scale fabric connects the two bridge chiplets. A label notes that the data plane reads only committed state and never a table being written, and the whole topology is marked conceptual.Software policyrequests, never routesControl planevalidate + commit (§9)Pkg 0 computeLOCAL_PACKAGEPkg 0 bridgescope transitionSystem fabricREMOTE_PACKAGE (§4)Pkg 1 bridgescope transitionPkg 1 computeLOCAL_PACKAGEData planereads COMMITTED only12
A conceptual multi-package system. Software sets policy; the control plane validates and commits it; the data plane reads only committed state. Within each package, chiplets communicate over die-to-die links; between packages, a system-scale fabric carries traffic through a bridge. Every topology here is conceptual and is not attributed to any product.

Three things to read.

The whole topology is CONCEPTUAL (§3) and is not attributed to any product.

Software reaches the control plane and never the data plane. §8's discipline drawn as an absent edge — there is no path from sw to dp.

And the bridges are where scope changes. 23.4 §15's t_bridge term is real work, paid twice on a round trip — and §16 is what a bridge must preserve.

6. Routing by Scope

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ILLUSTRATIVE SYSTEM ARCHITECTURE (§3). Scope is a property of the
// DESTINATION, resolved once and carried — never re-derived downstream.
typedef enum logic [1:0] {
  SCOPE_LOCAL_DIE     = 2'd0,
  SCOPE_LOCAL_PACKAGE = 2'd1,
  SCOPE_REMOTE_PACKAGE= 2'd2
} route_scope_e;
 
typedef struct packed {
  logic [SEM_W-1:0]   sem_id;          // the OPERATION — stable everywhere
  logic [GEN_W-1:0]   generation;      // which use of that id (22.4 §12)
  logic [15:0]        dst_logical_id;  // NOT a link, NOT a port (24.2 §15)
  route_scope_e       scope;           // resolved at issue
  logic [EPOCH_W-1:0] route_epoch;     // WHICH route table produced it (§11)
  logic [EPOCH_W-1:0] cfg_epoch;       // which contract it runs under (24.2 §11)
} sys_txn_t;
 
// Transport selection is a pure function of scope. There is no silent default:
// an unresolvable scope is OBSERVABLE, because a dropped transaction with no
// record is 21.6 §31's causeless event seen from the sending side.
always_comb begin
  local_valid = 1'b0; d2d_valid = 1'b0; fabric_valid = 1'b0; scope_error = 1'b0;
  unique case (txn.scope)
    SCOPE_LOCAL_DIE:      local_valid  = txn_valid;
    SCOPE_LOCAL_PACKAGE:  d2d_valid    = txn_valid;
    SCOPE_REMOTE_PACKAGE: fabric_valid = txn_valid;
    default:              scope_error  = txn_valid;
  endcase
end

Architecture. One resolution, carried on the transaction, driving a three-way selection with three genuinely different transport contracts behind it (§4).

State. None here — the transaction carries it.

Event. Scope is resolved at issue, from the destination's logical identity. Downstream blocks read txn.scope and never recompute it (22.3 §11's captured-not-recomputed rule).

Contract. The transaction carries two epochs. cfg_epoch says which contract it runs under (24.2 §11); route_epoch says which route table produced its path. They change independently, and §13 is what happens when the second is missing.

Failure. A silent default arm routes an unresolvable destination somewhere or nowhere, producing an orphan with no record (21.6 §31). scope_error makes it a finding instead.

DV/debug. scope and both epochs belong in the trace event (21.7 §16). A SCOPE_LOCAL_PACKAGE transaction observed on the fabric interface is a one-line finding rather than a latency mystery (23.4 §12).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// WRONG — the destination field IS a physical port index.
assign egress_port = txn.dst;          // dst is a port number
assign resp_match  = (resp_port == pending_port[resp_slot]);  // correlate on PORT

Why this is worse in a multi-package system than in 24.2 §14's single package:

Single packageMulti-package
how often does the path change?rarelyroutinely — failover, rebalancing, policy
what changes with it?a linka whole scope transition
outstanding work at the timesomemore, and longer-lived (22.4 §8)
correlation on a portbreaks on failoverbreaks on every reroute

And the fix is 24.2 §15's, now load-bearing: correlate on sem_id + generation, route on a separate table, and stamp the transaction with the route_epoch it was issued under — because in a reroutable scope, "which table produced this path?" is a question that will be asked.

8. Software-Defined Does Not Mean Software in the Datapath

Software sets policy. Hardware executes it. Software-defined means the topology, mappings, partitions and capabilities are configurable — not that firmware participates in forwarding a transaction.

Software may decideHardware must do
which logical resource maps whereforward every transaction deterministically
routing policy and preferencesselect a path from committed state
partitioning and isolationenforce it per transaction
power and QoS policyapply it at line rate
when a change takes effectcommit it atomically (§10)

And the anti-pattern is per-transaction firmware. A datapath that consults software for a forwarding decision has latency and jitter set by an interrupt path — which for a coherent or latency-sensitive operation (22.4 §9) is not a performance choice but a correctness one.

9. Control Plane and Data Plane

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
ILLUSTRATIVE architecture (§3).
 
  SOFTWARE / CONTROL PLANE
      writes REQUESTED policy            (never active, never the datapath)
            |
  HARDWARE CONTROL PLANE
      validate  -> is the request self-consistent and within capability?
      prepare   -> stage it in a shadow structure (§10)
      quiesce   -> are the affected transactions drained or trackable?
      commit    -> atomic swap + epoch increment
            |
  DATA PLANE
      reads ACTIVE state ONLY. It never sees requested, never sees shadow,
      and never observes a partially-updated structure (§12).

Two properties.

Validation happens before staging, and quiescence before commit. Committing a request that was never validated puts an inconsistent table into the datapath; committing without quiescence produces §13.

And the data plane's read set is deliberately narrow. It reads active structures and nothing else — which is what makes §12's assertion writable at all.

10. The Double-Buffered Route Table

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ILLUSTRATIVE SYSTEM ARCHITECTURE (§3). The centrepiece of this chapter:
// a route table that can be updated under live traffic without any transaction
// ever observing a partially-written state.
module route_table_2bank #(
  parameter int N_DEST   = 64,
  parameter int PORT_W   = 4,
  parameter int EPOCH_W  = 8
) (
  input  logic                 clk,
  input  logic                 rst_n,
 
  // ---- control plane (§9) ----
  input  logic                 shadow_we,        // software writes shadow ONLY
  input  logic [$clog2(N_DEST)-1:0] shadow_addr,
  input  logic [PORT_W-1:0]    shadow_port,
  input  route_scope_e         shadow_scope,
  input  logic                 shadow_valid_in,
  input  logic                 validate_req,     // "check the shadow"
  input  logic                 commit_req,       // "make it active"
  input  logic                 quiesced,         // no affected txn in flight
  output logic                 shadow_validated_q,
  output logic                 commit_done,
  output logic [EPOCH_W-1:0]   route_epoch_q,
 
  // ---- data plane ----
  input  logic [$clog2(N_DEST)-1:0] lookup_addr,
  output logic [PORT_W-1:0]    lookup_port,
  output route_scope_e         lookup_scope,
  output logic                 lookup_valid
);
 
  typedef struct packed {
    logic [PORT_W-1:0] port;
    route_scope_e      scope;
    logic              valid;
  } route_e_t;
 
  route_e_t bank_q [2][N_DEST];
  logic     active_sel_q;                 // which bank the DATA PLANE reads
 
  // ---------------- DATA PLANE ----------------
  // Reads the ACTIVE bank only. It cannot observe the shadow, ever.
  assign lookup_port  = bank_q[active_sel_q][lookup_addr].port;
  assign lookup_scope = bank_q[active_sel_q][lookup_addr].scope;
  assign lookup_valid = bank_q[active_sel_q][lookup_addr].valid;
 
  // ---------------- CONTROL PLANE ----------------
  logic shadow_sel;
  assign shadow_sel = ~active_sel_q;      // always the other bank
 
  // Validation: every entry must name a legal port for its scope, and a valid
  // entry must not point at a port that is down. Deliberately a SEPARATE step
  // from commit, so an inconsistent table can never become active (§9).
  logic shadow_ok;
  always_comb begin
    shadow_ok = 1'b1;
    for (int i = 0; i < N_DEST; i++) begin
      if (bank_q[shadow_sel][i].valid) begin
        if (!port_legal_for_scope(bank_q[shadow_sel][i].port,
                                  bank_q[shadow_sel][i].scope)) shadow_ok = 1'b0;
        if (!port_up(bank_q[shadow_sel][i].port))               shadow_ok = 1'b0;
      end
    end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      active_sel_q       <= 1'b0;
      route_epoch_q      <= '0;
      shadow_validated_q <= 1'b0;
      commit_done        <= 1'b0;
      for (int b = 0; b < 2; b++)
        for (int i = 0; i < N_DEST; i++) bank_q[b][i] <= '{default: '0};
    end else begin
      commit_done <= 1'b0;
 
      // Any write to the shadow INVALIDATES a previous validation. Otherwise a
      // late write could ride in on an earlier approval.
      if (shadow_we) begin
        bank_q[shadow_sel][shadow_addr] <= '{ port:  shadow_port,
                                              scope: shadow_scope,
                                              valid: shadow_valid_in };
        shadow_validated_q <= 1'b0;
      end else if (validate_req) begin
        shadow_validated_q <= shadow_ok;
      end else if (commit_req && shadow_validated_q && quiesced) begin
        // THE ATOMIC SWAP. One bit changes which bank the data plane reads,
        // so every lookup sees either wholly-old or wholly-new — never a mix.
        active_sel_q       <= shadow_sel;
        route_epoch_q      <= route_epoch_q + 1'b1;   // exactly once
        shadow_validated_q <= 1'b0;                   // the new shadow is stale
        commit_done        <= 1'b1;
      end
    end
  end
endmodule

Architecture. Two banks, a one-bit active selector, and a validate → commit sequence. The datapath's entire coupling to configuration is one multiplexer select.

State. 2 × N_DEST entries, the active-bank bit, a validation flag, and the route epoch.

Event. Software writes the shadow only. validate_req checks it. commit_req swaps only if validated and quiesced — three conditions, deliberately separate.

Contract. Three properties do the work. A shadow write clears shadow_validated_q, so a late write cannot ride in on an earlier approval. The swap is one bit, so no lookup can see a mixture. The epoch increments exactly once per commit, which is what lets a response be attributed to the table that produced its request (§11).

Failure. Committing without quiesced produces §13 — the request takes the old route and the response is mapped under the new one. Validating and committing in the same cycle removes the ability to reject an inconsistent table. Letting the datapath read shadow_sel defeats the entire structure.

DV/debug. route_epoch_q on every transaction (§6) is what makes a cross-update trace interpretable. A response whose route_epoch predates the current one is a straggler, not a violation — and telling those apart requires the field.

11. Route Epoch and Transaction Lifetime

QuantityStable forChanges on
sem_id + generationthe operation's whole lifenever, while live (22.4 §12)
dst_logical_idthe operation's whole lifenever, while live
route_epochstamped at issue, carrieda route commit (§10)
cfg_epochstamped at issue, carrieda contract commit (24.2 §11)
the path takenone attemptany reroute

Two properties.

A live transaction keeps the epoch it was issued under. New transactions use the new table; old obligations remain correlatable — which is what makes an update-under-load safe rather than merely fast.

And the two epochs are independent. A route change need not change the contract, and a contract change need not change routes. Merging them into one counter makes every route change look like a contract change, invalidating traffic that was perfectly fine.

12. Assertions — Route Table Atomicity

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// MANDATORY. Illustrative architectural properties (§3) — not normative.
 
// (1) THE ACTIVE BANK CHANGES ONLY ON A VALID COMMIT.
// English: the data plane's view changes only when a validated shadow is
// committed at quiescence. Fires on §13's premature or unvalidated commit,
// in simulation, at the cycle of the commit.
a_active_only_on_valid_commit: assert property (
  @(posedge clk) disable iff (!rst_n)
    $changed(active_sel_q)
      |-> ($past(commit_req) && $past(shadow_validated_q) && $past(quiesced))
);
 
// (2) AN UNVALIDATED SHADOW CANNOT BECOME ACTIVE.
// English: validation is a precondition, not advice.
a_no_unvalidated_activation: assert property (
  @(posedge clk) disable iff (!rst_n)
    $changed(active_sel_q) |-> $past(shadow_validated_q)
);
 
// (3) THE EPOCH INCREMENTS EXACTLY ONCE PER COMMIT.
// English: one commit, one epoch. A double increment breaks straggler
// attribution; no increment makes an old response indistinguishable from new.
a_epoch_once_per_commit: assert property (
  @(posedge clk) disable iff (!rst_n)
    commit_done |-> (route_epoch_q == $past(route_epoch_q) + 1)
);
a_epoch_stable_without_commit: assert property (
  @(posedge clk) disable iff (!rst_n)
    !commit_done |=> $stable(route_epoch_q)
);
 
// (4) A LIVE TRANSACTION'S DESTINATION AND EPOCH ARE STABLE.
// English: an in-flight operation is never re-stamped by a route change.
// This is what makes §13's correct version work.
a_live_txn_immutable: assert property (
  @(posedge clk) disable iff (!rst_n)
    txn_live |-> ($stable(txn_dst_logical_id) && $stable(txn_route_epoch))
);
 
// (5) THE DATA PLANE NEVER READS THE SHADOW.
// English: lookups are served from the active bank only. Structural
// non-interference — discharge formally (21.7 §17).
a_lookup_from_active_only: assert property (
  @(posedge clk) disable iff (!rst_n)
    lookup_valid |-> (lookup_port == bank_q[active_sel_q][lookup_addr].port)
);

Architecture. Five properties: two on commit preconditions, two on epoch behaviour, one structural.

State. The bank selector, validation flag and epoch.

Sampled timing. Property (1) uses $past on all three preconditions because they must have held in the cycle the commit was accepted, not when its effect appears. Property (3) is written as a pair — increments-once and stable-otherwise — because either alone permits the other failure.

Contract. Property (4) requires txn_live to be true for the whole transaction lifetime, including across a reroute. A liveness signal that clears on a route change makes the property vacuous exactly when it matters.

Failure if omitted. Without (1), §13 ships. Without (3)'s second half, an epoch that silently fails to increment makes every straggler look current — which is worse than no epoch, because the field is trusted.

DV/debug. Property (5) belongs in a formal flow: non-interference is a proof obligation, and simulation shows only that it held for the stimulus you ran.

13. Flagship — Route Update Under Live Traffic

The wrong implementation writes the live table entry by entry.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// WRONG — software updates the active table directly, one entry per write.
always_ff @(posedge clk)
  if (sw_we) route_tbl[sw_addr] <= '{ sw_port, sw_scope, 1'b1 };  // LIVE table

Twenty events, illustrative:

#CycleEvent
11,000txn A issued to dst=7; table says port 2; request leaves on port 2
21,001txn B issued to dst=9; table says port 2
31,004software begins remapping: writes dst=7 → port 5
41,005txn C issued to dst=7; reads the table → port 5
51,006software writes dst=9 → port 5
61,008A's response arrives on port 2
71,008the receiver correlates by looking up dst=7now port 5
81,008mismatch: response port 2, expected port 5 → A is dropped as spurious
91,012B's response arrives on port 2; same mismatch; B dropped
101,015C's response arrives on port 5; matches; C completes
111,200A's timeout fires; A's identity is freed
121,201a new transaction A′ allocates the freed identity
131,260a late retransmission of A's response arrives (transport retry)
141,260it now matches A′ — same identity, no generation check
151,260A′ completes with A's data — silent corruption (22.4 §11)
16no error was logged at any point
17link counters clean; CRC clean; conservation closes
18the symptom surfaces as application data corruption
19the trigger is "a routine routing policy change"
20it reproduces only when a response is in flight during the update

Five readings.

Steps 6–9 are the primary bug, and they are a correlation failure, not a routing failure: the response went to the right place and was matched against a table that had moved.

Steps 11–15 are the escalation, and they are why this is a corruption rather than a stall — 22.4 §12's generation tagging is the second half of the fix.

Step 17 is what makes it hard. Every transport-level instrument is clean, so 21.4's ledger and 21.5's counters all say the link is healthy.

Step 20 is the reproduction condition, and it is why this survives testing: an update with no traffic in flight works perfectly.

And the correct version fixes it twice over. §10's atomic swap with quiescence means no transaction is issued under one table and correlated under another; §11's route_epoch on the transaction means that even a straggler arriving after a legitimate later change is attributable rather than spurious.

14. Readiness Is a Hierarchy

A resource becomes usable in stages, and publishing it early produces §15.

StageMeansPublishing here is
physically presentthe die is in the packagefar too early
link upthe transport trained (21.1)still too early
capability exchangedthe manifest was read (24.2 §6)too early
contract committedboth sides agreed and committed (24.2 §11)nearly
routes committedthe path exists in the active table (§10)nearly
logically activatedthe resource accepts workcorrect
software-publishedthe OS sees itafter the above

15. Wrong — Publish on Physical Detection

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// WRONG — one bit, set on presence.
assign resource_ready = chiplet_present;
StepEvent
1the chiplet is detected as present
2resource_ready asserts; software enumerates it
3software issues the first access
4the contract has not committed — the peer interprets under defaults
5no route exists — the lookup returns valid = 0
6the access fails, or worse, is misinterpreted
7software marks the resource failed and does not retry
8the resource becomes usable milliseconds later — and is never used

And the fix is a readiness vector, not a bit:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// CORRECTED. Readiness is a conjunction, and each term is independently
// observable so a stuck stage is diagnosable (21.1 §17's waiting-reason).
typedef struct packed {
  logic present;
  logic link_up;
  logic caps_exchanged;
  logic contract_committed;
  logic routes_committed;
  logic logically_active;
} readiness_t;
 
readiness_t rdy_q;
assign resource_ready = rdy_q.present && rdy_q.link_up && rdy_q.caps_exchanged
                     && rdy_q.contract_committed && rdy_q.routes_committed
                     && rdy_q.logically_active;
 
// MANDATORY. English: no traffic is issued to a resource before every
// readiness term holds. Fires at the premature access of step 3.
a_no_traffic_before_ready: assert property (
  @(posedge clk) disable iff (!rst_n)
    txn_issue_fire |-> resource_ready
);

Architecture. Six independent terms and one conjunction, so which stage is missing is directly readable.

State. Six bits per resource.

Event. Each term is set by its own completion event; none is inferred from another.

Contract. Software is published the conjunction, never an individual term — and step 7 above is why: a premature publication is not merely early, it can be permanent, because software concludes the resource is broken.

Failure. Setting logically_active on link_up collapses the hierarchy back to §15. Inferring routes_committed from contract_committed couples two independent commits and produces a window where a resource is contractually agreed and unreachable.

DV/debug. The vector is 21.1 §17's waiting-reason encoder for resources: a resource stuck below readiness reports which term is missing, instead of a timeout with no explanation.

16. Identity Across Three Transports

23.4 §14 established two attempt scopes. A multi-package path has three, plus bridges.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ILLUSTRATIVE SYSTEM ARCHITECTURE (§3). One semantic identity; an INDEPENDENT
// attempt identity per transport boundary.
typedef struct packed {
  // SEMANTIC — one per operation, stable across every hop and every retry
  logic [SEM_W-1:0] sem_id;
  logic [GEN_W-1:0] generation;
  // PER-TRANSPORT attempt counters — none of these is semantic
  logic [ATT_W-1:0] local_attempt;     // source package D2D hop
  logic [ATT_W-1:0] fabric_attempt;    // system-scale hop
  logic [ATT_W-1:0] remote_attempt;    // destination package D2D hop
  // CONTEXT
  logic [EPOCH_W-1:0] route_epoch;
  logic [EPOCH_W-1:0] cfg_epoch;
} multi_txn_t;
 
// Each transport increments ONLY its own field. No branch writes sem_id or
// generation — and that absence is the correctness argument.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    local_attempt_q  <= '0; fabric_attempt_q <= '0; remote_attempt_q <= '0;
  end else begin
    if (local_retx_fire)  local_attempt_q  <= local_attempt_q  + ATT_W'(1);
    if (fabric_retx_fire) fabric_attempt_q <= fabric_attempt_q + ATT_W'(1);
    if (remote_retx_fire) remote_attempt_q <= remote_attempt_q + ATT_W'(1);
  end
end
 
// De-duplication keys on the SEMANTIC pair ONLY. Including an attempt field
// here defeats the mechanism entirely — every retry presents a new key and
// executes (23.4 §14).
assign already_done = delivered_set_contains(rx.sem_id, rx.generation);
assign do_execute   = rx_valid && !already_done;

Architecture. A four-level identity — operation, plus one attempt counter per transport boundary.

State. Three attempt counters plus the receiver's delivered set.

Event. Each transport increments only its own field. A bridge forwards the semantic identity unchanged and does not touch any attempt counter that is not its own.

Contract. De-duplication keys on sem_id + generation only. This is the single most easily-broken rule in the chapter: including an attempt field looks implemented and disables the mechanism completely.

Failure. §17.

DV/debug. All three attempt fields belong in the trace. A trace showing three non-zero attempt counters, one sem_id, and one delivery is the system working — and being able to see that at a glance is what stops an engineer blaming a transport (23.4 §14).

17. Wrong Retry Composition

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// WRONG — each transport allocates a NEW semantic id for its retry, because
// "a retransmission is a new request from this layer's point of view."
always_ff @(posedge clk)
  if (any_retx_fire) sem_id_q <= next_free_sem_id;   // <-- new semantic identity
StepEvent
1operation X issued, sem_id = 100
2local D2D hop retransmits → sem_id becomes 101
3fabric hop retransmits → sem_id becomes 102
4remote D2D hop retransmits → sem_id becomes 103
5the destination sees four distinct semantic operations
6de-duplication finds no match for any of them
7X is executed up to four times
8exactly-once is broken, and each transport behaved correctly

Four readings.

Every retry was correct (21.6 §19). The defect is that a transport event was allowed to write a semantic field.

It scales with the number of scopes, so a multi-package path is strictly worse than 23.4 §13's two-transport case — and the failure grows as systems compose further.

The diagnosis points at the wrong place. The destination reports duplicate operations; the transports each report a normal retry — and the evidence exonerates every component.

And the fix is §16's absence: no branch may write sem_id or generation on a transport event, and the assertion below is a non-interference property rather than a behavioural one.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// MANDATORY. English: a transport retry never alters the semantic identity.
a_retry_preserves_identity: assert property (
  @(posedge clk) disable iff (!rst_n)
    (local_retx_fire || fabric_retx_fire || remote_retx_fire)
      |=> ($stable(sem_id_q) && $stable(generation_q))
);

18. Failure Domains Across Packages

DomainScopeA failure here should
linkone D2D connectionretrain; obligations survive (23.3 §14)
chipletone die's functionisolate that function
packageone packagenot reset the system
fabric pathone route through the system fabricreroute (§10), retain obligations
systemeverythinglast resort

Two properties.

A route failure and a resource failure need different responses. A route failure means the destination is fine and the path is not — reroute and keep the obligations. A resource failure means the destination is gone, and the obligations must be explicitly failed rather than retried forever.

And distinguishing them requires the readiness vector (§15): a resource whose logically_active still holds while its route is being recommitted is a routing event, not a resource event.

19. Flagship — Distributed Deadlock, Everything Locally Correct

Three suppliers, three components, three local correctness proofs, and a system that stops.

ComponentWhat it doesLocally correct?
A (compute)issues requests to B; holds a completion slot until its response returnsyes
B (memory-side)to serve A, must first fetch from C; reserves an entry for the fetchyes
C (bridge to remote)forwards to the remote package; needs a shared progress credit to returnyes

The cycle:

StepState
1A issues many requests; each holds a completion slot
2B reserves entries for its fetches from C
3C's returns need the shared progress credit pool
4A's outstanding requests have consumed that same pool
5C cannot return → B cannot complete → A cannot free its slots
6A cannot free the credit C needs → cyclic wait across three ownership domains

Five readings.

Every component satisfies its own contract, and each supplier's verification proves exactly that — 24.3 §17's pattern, extended to a three-party cycle that no pairwise test can construct.

The resource that closes the cycle is shared — the progress credit pool — which is 19.5 §8's physical-pool test failing across three organisations rather than two blocks.

It cannot be found by component conformance (20.7) or by pairwise interoperability, because the cycle needs all three.

And no specification I could reach defines a system-level progress contract for this — so the honest statement is that this requires a system-level dependency analysis that someone must own, and 24.1 §4 says that owner is the integrator.

The structural preventions are two, both derived: a per-component declaration of what it reserves and what it waits on (extending 24.2 §6's manifest), and an acyclicity check over the resulting wait-for graph — which is 13.3's analysis lifted to the composition level.

20. Locality and Telemetry

A composed system must expose enough structure for a scheduler to make sensible placements, and enough evidence for a failure to be attributed.

ExposedWhy
locality class per resource (§4's three scopes)a remote package is not a local chiplet
capacity and service classplacement needs both
identity, epoch pair21.7 §9's snapshot, per domain
error and recovery countersattribution (24.1 §16)
useful-throughput counters21.5 §22's minimum set
outstanding and readiness§15

21. The Interoperability Matrix Does Not Close

N suppliers × M chiplets × K profiles × revisions is not testable, and pretending otherwise is the ecosystem's most expensive optimism.

StrategyBuys
component conformance (20.7)each component against a reference
formal component invariants (§22)properties that hold against any peer
declared compatibility (24.2 §8)eliminates pairings before test
targeted pairwise testingthe pairings actually shipped (23.2 §10)
system-level dependency analysis§19's three-party cycles

And the load-bearing row is the second. A property proved to hold regardless of the peer's implementation — no duplicate semantic completion, configuration atomicity, route stability per epoch, resource conservation — converts an untestable product into a testable one, because composition then only has to cover what the invariants do not.

22. Formal Component Invariants

InvariantHolds againstWhy formal
no duplicate semantic completionany peer, any retry pattern (§16)the retry space is unbounded
configuration atomicity (24.2 §11)any commit timingthe window is one cycle
route stability per epoch (§12)any update sequenceupdate orders are combinatorial
resource conservation (21.4)any traffic mixsoak tests find it late
no traffic before readiness (§15)any bring-up orderorderings are permutations
non-interference (§12 property 5)any stimulussimulation proves only what it ran

And the last row is the general argument. A simulation shows a property held for the stimulus exercised; a proof shows it holds for every stimulus — which is the only form strong enough to survive an unknown peer.

23. Common Misconceptions

"Future systems are current SoCs with more chiplets." §1: each addition brings identity, routing, configuration, failure, recovery, trust, locality and observability — and those compose worse than links do.

"One routing abstraction can cover every scope." §4, 23.4 §12: three scopes differ in reach, topology, failure model, error budget and latency class by orders of magnitude.

"Software-defined means software routes each request." §8: software sets policy; a datapath consulting firmware per transaction has latency set by an interrupt path.

"Changing a route table is just a register write." §13: twenty events from a routine policy change to silent data corruption, with every transport counter clean.

"Failover can clear old outstanding transactions." §18, 23.3 §14: a route failure means the destination is fine and the path is not. Obligations survive.

"Standard interfaces eliminate multi-vendor deadlock." §19: three locally correct components, a shared resource, and a cycle no pairwise test can construct.

"A common telemetry format proves interoperability." §20: it makes evidence comparable, which changes attribution from a commercial argument to a technical one. That is valuable and it is not proof.

"All resources should share one global address space." §4: three scopes with different failure and reachability models do not flatten without losing the distinctions a scheduler needs.

"More composability means less verification." §21: the matrix does not close, and the answer is formal component invariants plus targeted pairwise testing — more discipline, not less.

24. Understanding Check

25. Summary

Five things.

Composition, not connection, is the hard problem (§1). Every ownership domain brings identity, routing, configuration, failure, recovery, trust, locality and observability.

Scope is a property of the destination (§4, §6), resolved once and carried — three scopes differing by orders of magnitude in reach, latency and failure model.

Software-defined means software sets policy (§8). The datapath reads committed state only, and never a structure being written.

A route table must be double-buffered, validated and committed atomically (§10–§13). Otherwise a routine policy change becomes silent data corruption, with every transport counter clean.

And no transport event may write a semantic identity (§16–§17). Retries compose across scopes; identities must not — and the rule is an absence, checkable as non-interference.

On evidence: the UCIe facts are Level B and the scope distinction is Level C; everything this chapter builds is derived Level-E architecture (§3). No prediction, no timeline, no claim that any product implements any of it.

26. Module 24 Complete

ChWhat it established
24.1Roles and artefacts — one of eleven is standardised; the integrator owns the whole and sees the least; failure attribution has no framework
24.2The contract as data — compatibility is a predicate, not a version; four configuration states; five identity scopes
24.3Methodology — contracts freeze before implementations; partition quality is measurable; interfaces must not leak microarchitecture
24.4Composition — three scopes, atomic route commits, identity across transports, and deadlocks no pairwise test can find

And the module's through-line is one sentence. A standardised link makes a boundary implementable by strangers; everything else — what is exchanged, when it freezes, and how it composes — is engineering that someone still has to own. Module 24 is what that work consists of, derived from what composition requires rather than predicted from a roadmap.