Multi-Die SoCs

26.3 and 26.4 treated the die boundary as a performance boundary. This chapter treats it as a boundary for state, time and ownership — and those are harder, because bandwidth problems are slow while ownership problems are wrong.

1. The One-Sentence Model

Splitting an SoC does not just distribute logic — it creates new boundaries for state, time and ownership, and every assumption that used to be guaranteed by being on one die now has to become an explicit contract.

On a monolithic SoC, a great deal is free. One reset. One clock tree, or at least clock domains you control end to end. One power-up. One configuration authority. One place where a transaction's identity lives. A single-cycle pulse is a perfectly good way to signal an event.

Split the die and every one of those becomes a question with a wrong answer available — and the wrong answers are the subject of §7 through §20. Not one of them is a bandwidth problem, and not one of them is caught by testing either die on its own (§21).

2. What This Chapter Owns

25.10 owned integration as a process — the contract matrix, named owners, versioning, sign-off. This chapter owns the mechanisms, and it owns five things nothing before it did:

Reset as five distinct scopes (§7–§9), not one signal with a fan-out.

Time as a per-die property (§10–§12) — why a single-cycle pulse is not a way to tell another die anything.

Readiness as a hierarchy (§13–§15): a link reaching ACTIVE and a function being usable are separated by four more conditions.

Configuration as an owned, committed, epoched object (§16–§17), and identity as something that survives a route change (§18–§19).

And the case that justifies Module 26 existing (§21): two dies, each fully compliant with its own specification, each passing its own sign-off, integrated into a product that hangs — because they hold different, internally consistent beliefs about what a recovery means.

3. Sourcing

4. Claim-vs-Evidence

Three readings.

Rows 3 and 5 are the pair that carries §20, and together they are the most valuable evidence in this chapter. Two shipping multi-die products, same era, opposite answers to the question "should software see the seam?" — one presents two dies as one CUDA device, the other lets software select one, two or eight logical GPUs. That is a documented architectural decision, not a marketing difference.

Row 4 is architecturally loud. A fully coherent shared L2 across a die boundary means semantic state spans the boundary — which is exactly the condition that makes §21's flagship failure possible in principle. It says nothing about NVIDIA having that bug, and everything about why the class of bug exists.

And row 9 is the honest centre of this chapter. The mechanisms are Class G — engineering inference. They are defensible, they follow from Modules 14/21/25, and no source establishes that any product implements them. Saying so is what separates this from a teardown fantasy.

5. The Boundaries a Split Creates

Three readings.

Every row's left column is free and its right column is engineered. That asymmetry is the chapter: a monolithic design gets eight guarantees at no cost, and a multi-die design has to build all eight.

The five seam boundaries do not coincide, and assuming they do is the single most productive source of bugs here. A link reset is not a die reset (§9). A configuration authority is not a die (§16). A physical route is not an identity (§19). §9, §15 and §19 are three different bugs with one root: two boundaries treated as one.

And none of these is a bandwidth problem. 26.3 and 26.4 could be measured with counters and fixed with tuning. These produce hangs and corruption, and they are found by contracts and assertions rather than by profiling.

6. Why This Is Not "Chiplets Are Modular"

Two readings.

Every benefit in the left column creates an obligation in the right. Independent development is the one that costs most, and it costs most precisely when it succeeds — two teams working efficiently in isolation produce two internally consistent designs that disagree (§21).

And "chiplets allow different process nodes" has been the answer in 26.1, 26.2 and 26.3. It is true, it is established, and it is not this chapter's lesson. The lesson here is that splitting a die splits its state, its time and its authority, and those have to be rebuilt by hand.

7. Reset Is No Longer One Signal

Three readings.

The rows are ordered by scope, and the ordering is the design. A narrower reset must never have wider effects than its row allows. The moment a link reset clears semantic state, rows 3 and 5 have been merged — which is §9's bug.

Row 6 exists because of a real and common mistake. If clearing sticky diagnostics also perturbs functional state, you cannot re-arm your error capture on a live system — so the first fault after any clear is unobservable, exactly when you most need it (21.7 §14).

And the last column is the part that gets skipped. Every row needs a named owner (25.10 §6). A reset with two owners will be asserted by both, at different times, for different reasons — and the resulting interaction is not in either die's specification.

8. What Must Survive Each Reset

Three readings.

The two central columns are the whole table. A link reset and a transport recovery are transport events. Semantic obligations, identities and configuration are not transport state and must be untouched by them (25.3 §14).

Row 7 is the one violated most often and noticed least. If a link reset clears first-fault capture, the reset destroys the evidence for the fault that caused it. You then observe only the second failure, forever — and every debug session starts one step behind (21.7 §14).

And row 3's "depends" for die-local reset is not vagueness — it is the contract that must be written down. A die resetting locally while its peer holds obligations against it is exactly §21's scenario. Whether those obligations survive is a decision; the failure is not making it.

9. Wrong RTL — Link Reset Clears the Transaction Table

// WRONG. ILLUSTRATIVE. The outstanding-transaction table is cleared by the same
// reset that clears the link layer. Written by someone porting a monolithic
// design where "reset" had exactly one meaning, and where clearing everything
// together was not merely correct but obviously correct.
always_ff @(posedge clk or negedge rst_n) begin
  // BUG 1: link_rst_n is folded into the SAME async reset as the semantic
  //        transaction table. Two scopes (§7 rows 3 and 5) merged into one.
  if (!rst_n || !link_rst_n) begin
    for (int e = 0; e < N_ENTRY; e++) begin
      txn_live_q[e] <= 1'b0;
      txn_gen_q[e]  <= '0;          // BUG 2: generation cleared, so ids restart
    end
    next_id_q <= '0;                // BUG 3: id allocation restarts at zero
  end else begin
    if (accept_fire) begin
      txn_live_q[alloc_idx] <= 1'b1;
      txn_gen_q[alloc_idx]  <= gen_ctr_q;
      next_id_q             <= next_id_q + 1'b1;
    end
    if (cmpl_fire) txn_live_q[cmpl_idx] <= 1'b0;
  end
end

Architecture. One reset expression covering link state and semantic state. On a monolithic die this is not just acceptable, it is tidy.

State. The live bits, the generation map and the ID allocator — all three are semantic state, and all three are being cleared by a transport event.

Event. Any link reset.

Contract. It violates §8 rows 3, 4 and 7 simultaneously, and it unilaterally violates them — the peer die was not consulted and does not know.

Failure — the timeline. Die A requests; die B holds the obligation.

Root cause. A transport event was allowed to clear semantic state, and identity allocation was restarted while the peer still held the old identities. The reuse window is not exotic — it is every link reset, and the corruption needs only one outstanding transaction.

Why it survives verification. Standalone testing resets the link with the pipeline drained, because that is how a directed recovery test is naturally written. The bug requires a reset with obligations outstanding at the peer — a state no test constructs unless someone thought of it (25.8 §17).

DV/debug. The signature is wrong data with every integrity check passing (21.6 §14) — no CRC error, no protocol violation, no dropped packet. The discriminator is a peer-side count of obligations outstanding at the moment of any link reset, which should be reportable and, in this design, is not.

10. Time Becomes a Per-Die Property

A die boundary usually means independent clock ownership. That turns a set of guarantees into a set of contracts.

Three readings, at contract level rather than as a CDC tutorial.

Only levels cross safely. A level held until acknowledged is robust to any clock ratio. A pulse whose width is defined in the source's cycles is a bet on the destination's clock period, and §11 is that bet being lost.

Data must be stable, and the protocol must be what makes it stable — not a timing assumption. The standard shape is: source holds the payload constant while the request level is asserted; destination samples only after it has synchronised the request. The payload itself is then never synchronised, only the control signal is.

And relative timing is not knowable across the boundary. "B will have seen it within three cycles" is a monolithic statement. Across a boundary you get ordering, not latency — which is why every cross-die event needs an acknowledgement rather than a delay.

11. Wrong RTL — a Single-Cycle Pulse Across Die Clocks

// WRONG. ILLUSTRATIVE. Die A signals a configuration event with a one-cycle
// pulse. Correct and idiomatic within a clock domain. Across a boundary where
// die B may be running slower, it is a coin flip.
always_ff @(posedge clk_a or negedge rst_a_n) begin
  if (!rst_a_n) cfg_event_pulse_q <= 1'b0;
  else          cfg_event_pulse_q <= cfg_event_req;   // BUG: one clk_a cycle
end
 
// Die B, in a DIFFERENT and slower clock domain.
always_ff @(posedge clk_b or negedge rst_b_n) begin
  if (!rst_b_n) cfg_applied_q <= 1'b0;
  // BUG 2: two-flop synchroniser on a pulse narrower than clk_b's period.
  //        The synchroniser is correct; there is simply nothing to sample.
  // BUG 3: no acknowledgement, so die A never learns the event was lost.
  else if (sync2_q && !sync3_q) cfg_applied_q <= 1'b1;
end

Architecture. A pulse generator and a synchroniser. The synchroniser is textbook-correct, which is why review passes it.

State. One pulse flop; two synchroniser flops. The missing state is any record on either side of whether the event arrived.

Event. One clk_a cycle wide.

Contract. It requires the pulse to be at least one full clk_b period wide, plus setup margin. That requirement is nowhere written down, is not checked, and is not even visible in either die's source.

Failure — the arithmetic. clk_a at 2 GHz (500 ps period); clk_b at 800 MHz (1250 ps period). A 500 ps pulse against a 1250 ps sampling period is missed whenever it falls between two clk_b edges — the majority of the time, and dependent on the phase relationship, which drifts.

Failure — the behaviour.

Root cause. A pulse's meaning is defined in the source's time base and consumed in the destination's. Nothing in the design converts between them. The synchroniser handles metastability, which is a different problem — it does not create a sampling opportunity that never existed.

And the symptom is the worst kind: occasionally a configuration event is silently lost. The system mostly works. Failures are intermittent, unreproducible, correlate with thermal and frequency conditions, and disappear whenever anyone attaches instrumentation that changes the phase relationship.

DV/debug. RTL simulation with a single clock ratio can pass forever. The test that finds it sweeps the ratio and the phase, and the assertion worth binding is "every request event is acknowledged within N destination cycles" — which §12 makes possible by adding the acknowledgement this design lacks.

12. Corrected — a Cross-Die Event Handshake

// CORRECT. ILLUSTRATIVE (§11). A four-phase level handshake. The source HOLDS
// the request until acknowledged, so no clock ratio can lose it. Payload is
// held stable alongside the level and is never synchronised — only the control
// signal crosses through synchronisers.
// NOTE: this is a generic CDC structure, NOT a UCIe interface (§3).
 
// ---- Source domain (clk_a) ----
typedef enum logic [1:0] { S_IDLE, S_REQ, S_WAIT_DEASSERT, S_DONE } src_e;
src_e        src_q;
logic [31:0] payload_q;
logic [15:0] ack_timer_q;
logic        ack_sync1_q, ack_sync2_q;
 
always_ff @(posedge clk_a or negedge rst_a_n) begin
  if (!rst_a_n) begin
    src_q <= S_IDLE; req_level_q <= 1'b0; ack_timer_q <= '0;
    {ack_sync2_q, ack_sync1_q} <= 2'b00;
  end else begin
    {ack_sync2_q, ack_sync1_q} <= {ack_sync1_q, ack_level_b};   // 2-flop sync
 
    unique case (src_q)
      S_IDLE: if (new_event) begin
                payload_q   <= event_payload;   // held stable for the whole txn
                req_level_q <= 1'b1;            // LEVEL, not a pulse
                ack_timer_q <= '0;
                src_q       <= S_REQ;
              end
      // Bounded: a lost peer produces a NAMED timeout, not a silent hang.
      S_REQ:  if (ack_sync2_q) begin
                req_level_q <= 1'b0;
                src_q       <= S_WAIT_DEASSERT;
              end else if (ack_timer_q == ACK_TIMEOUT) begin
                src_q <= S_DONE;                // with cfg_event_timeout raised
              end else ack_timer_q <= ack_timer_q + 16'd1;
 
      // Full four-phase: wait for ack to DROP before another event may start.
      // Without this, back-to-back events merge into one at the destination.
      S_WAIT_DEASSERT: if (!ack_sync2_q) src_q <= S_IDLE;
      default:                            src_q <= S_IDLE;
    endcase
  end
end
 
// ---- Destination domain (clk_b) ----
logic req_sync1_q, req_sync2_q, req_sync3_q;
 
always_ff @(posedge clk_b or negedge rst_b_n) begin
  if (!rst_b_n) begin
    {req_sync3_q, req_sync2_q, req_sync1_q} <= 3'b000;
    ack_level_b <= 1'b0;
    applied_q   <= 1'b0;
  end else begin
    {req_sync3_q, req_sync2_q, req_sync1_q}
      <= {req_sync2_q, req_sync1_q, req_level_q};
 
    // Apply on the RISING EDGE of the synchronised level — exactly once per
    // event. req_sync3_q is the extra stage that makes edge detection safe.
    if (req_sync2_q && !req_sync3_q) begin
      applied_payload_q <= payload_q;   // stable by contract, never synchronised
      applied_q         <= 1'b1;
    end
    ack_level_b <= req_sync2_q;         // ack tracks the request level
  end
end
 
// MANDATORY. English: the source holds the request level stable until it has
// observed an acknowledgement. Catches anyone "optimising" the level back into
// a pulse — the §11 regression, which passes at one clock ratio.
a_req_held_until_ack: assert property (
  @(posedge clk_a) disable iff (!rst_a_n)
    (req_level_q && !ack_sync2_q) |=> req_level_q
);
 
// MANDATORY. English: the payload never changes while a request is pending.
// Catches a source that starts preparing the next event before this one lands.
a_payload_stable_while_req: assert property (
  @(posedge clk_a) disable iff (!rst_a_n)
    req_level_q |-> $stable(payload_q)
);
 
// MANDATORY, destination domain. English: each request level produces exactly
// one application. Catches double-apply from unstable edge detection.
a_apply_once_per_req: assert property (
  @(posedge clk_b) disable iff (!rst_b_n)
    (req_sync2_q && !req_sync3_q) |=> !(req_sync2_q && !req_sync3_q)
);

Architecture. Four-phase level handshake, three synchroniser stages on the destination side, two on the return, a bounded timeout, and a payload held by contract rather than synchronised.

State. A source FSM, a timer, and synchroniser chains. S_WAIT_DEASSERT is the state most implementations omit — and without it, two back-to-back events merge into one at a slow destination, which is the same lost-event bug wearing different clothes.

Event. Apply on the rising edge of the synchronised level. req_sync3_q exists purely so that edge detection operates on a stable signal; detecting an edge on req_sync1_q would sample a potentially metastable value.

Contract. The payload crosses without synchronisation, and that is correct — because it is stable by protocol for the whole request. Synchronising a 32-bit bus flop-by-flop is the actual error: independent synchronisers can resolve on different cycles, producing a value that was never sent.

Failure. ACK_TIMEOUT must come from the peer's stated worst-case response, not from measurement (26.2 §12). And it must raise a named error, not silently return to idle — otherwise this design also loses events, just more slowly.

DV/debug. The three assertions catch the three realistic regressions: level reverted to a pulse, payload changed mid-flight, and double application. The verification that matters is a clock-ratio and phase sweep (§11 DV lens) — a single-ratio simulation passes with §11's broken version too.

13. Readiness Is a Hierarchy

"The link is up" is one condition out of six, and it is not the last one.

Three readings.

Levels 5 and 7 are the two most often conflated, and the gap between them is where §15's bug lives. A trained link proves the transport works. It proves nothing about whether the function behind it has been configured (25.9 §5).

The hierarchy is strictly ordered, and each level is necessary and insufficient. Publishing at level n means the first user exercises levels n+1 onward without any of them being established — and the failure appears at first access, in whatever code happens to touch it first.

And every level needs a positive indication rather than an absence of failure. "No error reported" is not readiness. Level 7 in particular requires the function to have completed something, not merely to have accepted it — because acceptance is level 5 behaviour and completion is level 7 behaviour.

14. Illustrative RTL — the Readiness Vector

// ILLUSTRATIVE (§13). Readiness as an explicit ordered conjunction, with each
// level RECORDED so that a failure names WHICH level was never reached.
// The recording is the debug feature: "not ready" is useless, "reached level 5,
// never reached level 6" is a five-minute diagnosis.
typedef struct packed {
  logic present;
  logic power_good;
  logic clk_seen;        // clock detector + PLL lock, in the peer's domain
  logic reset_done;
  logic link_ready;      // link reached ACTIVE
  logic cfg_valid;       // configuration committed — §17
  logic semantic_ready;  // the function has completed at least one operation
} readiness_t;
 
readiness_t rdy_q [N_DIE];
logic [2:0] highest_level_q [N_DIE];   // sticky high-water — the debug field
 
// Strictly ordered conjunction. Each term implies all the terms before it, so
// a single bit cannot be set out of order by a rogue status register.
function automatic logic logical_ready(readiness_t r);
  logical_ready = r.present & r.power_good & r.clk_seen & r.reset_done
                & r.link_ready & r.cfg_valid & r.semantic_ready;
endfunction
 
always_ff @(posedge clk or negedge rst_n) begin
  for (int d = 0; d < N_DIE; d++) begin
    if (!rst_n) begin
      rdy_q[d] <= '0;
      highest_level_q[d] <= 3'd0;
    end else begin
      rdy_q[d] <= sample_readiness(d);
 
      // Sticky HIGH-WATER, not current level: a die that reached level 5 and
      // fell back to 3 must still report that it once reached 5. Otherwise a
      // late regression is indistinguishable from never having started.
      if (rdy_q[d].semantic_ready      && highest_level_q[d] < 3'd7) highest_level_q[d] <= 3'd7;
      else if (rdy_q[d].cfg_valid      && highest_level_q[d] < 3'd6) highest_level_q[d] <= 3'd6;
      else if (rdy_q[d].link_ready     && highest_level_q[d] < 3'd5) highest_level_q[d] <= 3'd5;
      else if (rdy_q[d].reset_done     && highest_level_q[d] < 3'd4) highest_level_q[d] <= 3'd4;
      else if (rdy_q[d].clk_seen       && highest_level_q[d] < 3'd3) highest_level_q[d] <= 3'd3;
      else if (rdy_q[d].power_good     && highest_level_q[d] < 3'd2) highest_level_q[d] <= 3'd2;
      else if (rdy_q[d].present        && highest_level_q[d] < 3'd1) highest_level_q[d] <= 3'd1;
    end
  end
end
 
// MANDATORY. English: a resource is never published before it is semantically
// ready. This is §15's bug expressed as a property.
a_no_publish_before_ready: assert property (
  @(posedge clk) disable iff (!rst_n)
    publish_fire[chk_die] |-> logical_ready(rdy_q[chk_die])
);
 
// MANDATORY. English: readiness levels are established in order — cfg_valid
// never asserts before link_ready. Catches a status register written by
// firmware out of sequence, which would defeat the conjunction above.
a_readiness_ordered: assert property (
  @(posedge clk) disable iff (!rst_n)
    rdy_q[chk_die].cfg_valid |-> rdy_q[chk_die].link_ready
);

Architecture. Seven bits per die, one ordered conjunction, and a sticky high-water level whose only job is to make a bring-up failure name itself.

State. rdy_q is live; highest_level_q is sticky. The stickiness is the design decision — a die that reached level 5 and regressed to 3 must still report having reached 5, or a late regression looks identical to a die that never started.

Event. Sampled continuously. semantic_ready requires a completed operation, not an accepted one (§13).

Contract. The conjunction must be ordered and total. A resource manager reading only link_ready is §15's bug, and the assertion is what stops that reading from being written in the first place.

Failure. The realistic mistake is a single "ready" bit exported to firmware. It carries no information about which condition failed — so every bring-up failure becomes a bisection instead of a register read (25.9 §20).

DV/debug. highest_level_q is the entire bring-up debug story in one register per die. "Die 3 reached level 5, never reached 6" points directly at configuration commit (§17) and eliminates power, clocks, reset and link training in a single read.

15. Wrong RTL — Link ACTIVE Treated as Function Ready

// WRONG. ILLUSTRATIVE. The resource manager publishes the function as soon as
// the link reports ACTIVE. Written by someone who reasoned — not unreasonably —
// that the link becoming ACTIVE is the last thing that has to happen.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    function_published_q <= 1'b0;
  end else begin
    // BUG 1: link ACTIVE is level 5 of 7 (§13). Configuration and semantic
    //        readiness are simply not consulted.
    // BUG 2: publication is not withdrawn if the link later leaves ACTIVE.
    if (link_state == LINK_ACTIVE)
      function_published_q <= 1'b1;
  end
end

Architecture. One condition, one bit.

State. function_published_q, which is set and never cleared — a second bug that only shows up after the first is fixed.

Event. The link entering ACTIVE.

Contract. It violates §13's ordering by publishing at level 5 and asserting level 7. The first consumer then exercises levels 6 and 7 with neither established.

Failure — the timeline.

Root cause. Two different boundaries treated as one (§5): the transport is usable and the function is usable. They are separated by configuration commit, which is a different die's responsibility on a different timeline.

Why it is so hard to reproduce. It is a race between configuration and the first access. Add tracing and the timing shifts; boot slowly and it passes; boot on a faster part and it fails. The classic report is "it fails only on the first access after a cold boot, and only sometimes" — and the classic non-fix is a delay in the driver, which hides it until a faster die is substituted.

DV/debug. §14's highest_level_q names it immediately: link_ready set, cfg_valid clear, and a publish event recorded. a_no_publish_before_ready fires on the exact cycle, which turns a multi-week intermittent-boot investigation into an assertion failure in the first simulation that exercises the race.

16. Configuration Ownership

Multi-die configuration needs three distinct states and exactly one authority.

And one documented invariant worth reusing. AMD's partitioning documentation states that "the number of memory partitions must be less than or equal to the number of compute partitions" (Class A, 26.3 §11). That is an ownership rule expressed as a configuration constraint — a memory pool with no compute partition to own it would be a published resource with nobody responsible for it. Real configuration spaces have real validity rules, and the "prepared" phase is where they are checked.

Three readings.

Self-commit is the failure mode this structure exists to prevent. If each die applies a change when it is locally ready, there is a window in which different dies are operating under different configurations — and any transaction crossing during that window is interpreted two ways (25.10 §11).

"Prepared" must be a positive confirmation from every affected die, not a timeout. A die that did not confirm has not staged the change, and committing anyway is self-commit with extra steps.

And there must be exactly one owner of the transition to "active" (25.10 §6). Two configuration owners is not a race condition; it is an unspecified system — and the resulting behaviour is not in any die's specification, so nobody's sign-off covers it.

17. Illustrative RTL — Two-Phase Configuration Commit

// ILLUSTRATIVE (§16). Shadow → prepare → all-peers-acked → atomic swap →
// epoch increment. The epoch is what lets a stale request be DETECTED rather
// than silently interpreted under the wrong configuration.
typedef enum logic [2:0] {
  C_IDLE, C_SHADOW, C_PREPARE, C_WAIT_ACK, C_COMMIT, C_ABORT
} cfg_e;
 
cfg_e               cfg_q;
config_t            active_cfg_q, shadow_cfg_q;
logic [EPOCH_W-1:0] cfg_epoch_q;
logic [N_DIE-1:0]   peer_prepared_q;
logic [15:0]        prepare_timer_q;
logic [31:0]        stale_rejected_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    cfg_q <= C_IDLE; cfg_epoch_q <= '0;
    peer_prepared_q <= '0; stale_rejected_q <= '0;
  end else begin
    unique case (cfg_q)
      C_IDLE:    if (cfg_change_req) begin
                   shadow_cfg_q <= proposed_cfg;   // stage; do NOT apply
                   cfg_q        <= C_SHADOW;
                 end
 
      // Validity is checked BEFORE anyone is asked to prepare (§16).
      C_SHADOW:  cfg_q <= cfg_is_valid(shadow_cfg_q) ? C_PREPARE : C_ABORT;
 
      C_PREPARE: begin
                   peer_prepared_q <= '0;
                   prepare_timer_q <= '0;
                   cfg_q           <= C_WAIT_ACK;   // broadcast prepare
                 end
 
      // Every affected die must POSITIVELY confirm. A timeout ABORTS —
      // it never commits, because a silent peer has not staged anything.
      C_WAIT_ACK: if (&peer_prepared_q)                   cfg_q <= C_COMMIT;
                  else if (prepare_timer_q == PREP_TIMEOUT) cfg_q <= C_ABORT;
                  else prepare_timer_q <= prepare_timer_q + 16'd1;
 
      // Atomic: the swap and the epoch bump happen in ONE cycle, and only
      // while nothing is outstanding under the old epoch.
      C_COMMIT:  if (outstanding_total == '0) begin
                   active_cfg_q <= shadow_cfg_q;
                   cfg_epoch_q  <= cfg_epoch_q + 1'b1;
                   cfg_q        <= C_IDLE;
                 end
 
      C_ABORT:   cfg_q <= C_IDLE;                  // shadow discarded, no effect
      default:   cfg_q <= C_ABORT;
    endcase
 
    // Defence in depth: a request stamped with an old epoch is REJECTED and
    // COUNTED, never interpreted under the new configuration.
    if (req_fire && (req_epoch != cfg_epoch_q))
      stale_rejected_q <= stale_rejected_q + 32'd1;
  end
end
 
// MANDATORY. English: configuration becomes active only after every affected
// die has confirmed preparation. Catches self-commit (§16) directly.
a_commit_requires_all_prepared: assert property (
  @(posedge clk) disable iff (!rst_n)
    $changed(cfg_epoch_q) |-> $past(&peer_prepared_q)
);
 
// MANDATORY. English: the epoch never advances while work is outstanding.
// This is what makes epoch comparison an assertion rather than a repair
// mechanism, and it is the load-bearing property of the whole scheme.
a_commit_quiesced: assert property (
  @(posedge clk) disable iff (!rst_n)
    $changed(cfg_epoch_q) |-> ($past(outstanding_total) == '0)
);
 
// MANDATORY. English: a request is never serviced under an epoch other than
// the one it was issued under. Catches a request that survives a commit.
a_no_stale_epoch_service: assert property (
  @(posedge clk) disable iff (!rst_n)
    service_fire |-> (svc_epoch == cfg_epoch_q)
);

Architecture. Six states, a shadow copy, a per-die prepared vector, a bounded timer, an epoch, and a counter for the case that should never occur.

State. active_cfg_q is what the design uses; shadow_cfg_q is never consulted until commit. Keeping them physically separate is what makes the swap atomic — a single register updated field by field is the bug this structure exists to prevent.

Event. Commit happens on one cycle, and only when outstanding_total == 0.

Contract. A timeout must abort, never commit. Committing on timeout is the most tempting shortcut here — it makes bring-up smoother — and it reintroduces exactly the split-configuration window the two phases exist to eliminate.

Failure. The residual risk is a commit that can never quiesce because something is permanently outstanding, turning a configuration change into a hang. outstanding_total non-zero with cfg_q == C_COMMIT is the diagnostic, and it names the blocking work rather than the configuration.

DV/debug. stale_rejected_q should be zero forever. A non-zero value is not a performance signal — it is proof that a_commit_quiesced was violated somewhere, and it converts a class of silent misconfiguration into a single register read.

18. Identity Versus Route

A logical function's identity and the physical path used to reach it are different facts. They coincide until a failover, a remap, or a partition change separates them.

Two readings.

Rows 1–2 must survive everything in row 3. That is the entire content of §19's bug and of 26.1 §15's — the same error appearing at two different layers, which is how you know it is structural rather than incidental.

And row 4 is what makes row 3 safe. Without an epoch, a route change is silent: entries created under the old mapping are indistinguishable from entries created under the new one. The epoch is what turns a silent misroute into a detectable, countable, assertable event (§17).

19. Wrong RTL — Link Number as Function ID

// WRONG. ILLUSTRATIVE. Response correlation keyed on the physical link the
// request went out on. True at design time — function 0 is reached over link 0.
// Written by someone who verified that mapping and then depended on it.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int e = 0; e < N_ENTRY; e++) fn_link_q[e] <= '0;
  end else begin
    // BUG 1: the correlation key is the PHYSICAL link index.
    // BUG 2: no epoch, so a failover cannot invalidate a live entry.
    if (req_fire)  fn_link_q[alloc_idx] <= out_link_idx;
    if (resp_fire) resp_match_ok <= (fn_link_q[resp_idx] == in_link_idx);
  end
end

Architecture. One stored field: which wire it went out on.

State. A physical link index, captured at request time and never revalidated.

Event. Captured on request, compared on response.

Contract. It assumes the function-to-link mapping is immutable. A multi-die SoC with any failover or reconfiguration capability breaks that assumption by design — the capability exists precisely to change the mapping.

Failure — the timeline. Function 4 is reached over link 0; link 0 degrades and the platform fails the function over to link 1 with two requests outstanding.

Root cause. A physical attribute was used as a logical identity. The failover mechanism — the thing designed to improve availability — becomes the trigger for a hang, which is the most damaging possible form of this bug.

Corrected architecture. Key on the logical function ID plus the semantic transaction ID and generation; keep the physical route in a separate, epoched record (§20); resolve route at use time, not at request time; quiesce before changing the epoch (§17).

DV/debug. The signature is valid responses being rejected, which reads as a peer fault and sends the investigation to the wrong die entirely. The discriminator is a rejection count that becomes non-zero within a few cycles of a route change — obvious once correlated, invisible without a route-epoch field in the trace (§20).

20. Failure Domains, Software View, and Observability

Three consequences of the split that shape the product rather than the RTL.

Failure domains. A fault domain, a recovery domain and a semantic domain are three different extents, and conflating them produces either over-reaction or corruption.

The recovery domain is determined by what state crosses the boundary. If a die holds no obligations against another, it can reset alone. If obligations cross, resetting one die alone leaves the other holding state for a peer that has forgotten it — §9's bug arriving through the reset-scope door rather than the link-reset door.

Software view — and here there is real documented evidence. Physical partition does not automatically mean software-visible partition, and two shipping products chose oppositely.

Two readings on that table. Exposing the seam gives isolation, explicit locality and the ability to right-size a device to a job — at the cost of software now owning placement. Hiding it gives one simple programming model and full resources to any single job — at the cost of software being unable to exploit locality even when it would help. Neither is more advanced; they are answers to different customer problems, and 26.3 §11 shows one product supporting both at different times.

Observability. One die's trace is insufficient, because the interesting events are the ones that cross.

And the requirement that makes all six work is a common time base. Without a correlated timestamp or a shared milestone counter, two dies' traces cannot be interleaved, and every cross-die failure in this chapter is a cross-die correlation problem. This is a design-time decision — it cannot be added when the first silicon bug appears.

21. Flagship — Both Dies Pass, the Product Fails

The case that justifies Module 26, and the sharpest statement of why local compliance is not integration.

Setup. Die A is a requester; die B is a completer. They were developed by different teams, to the same interface specification, and each passed its own sign-off against its own VIP.

Neither belief is wrong. The specification did not say.

Failure — the timeline.

And the second, worse variant. If A re-issues on timeout while B had in fact retained the original — the mirror-image mismatch — B completes both, and A receives two responses for one semantic operation. If A's entry was reallocated in between, the second response retires an unrelated transaction: corruption, with every integrity check passing.

Four readings.

Every component is behaving exactly as specified and verified. There is no RTL bug. Both dies would pass their sign-off again today. The defect is in the space between the two specifications, and no amount of testing either die alone can find it — each VIP encodes its own die's assumption, so each die passes by construction.

The root cause is a missing contract line, not a missing test. "What happens to outstanding semantic obligations across a transport recovery?" is a question with two defensible answers, and the specification's job was to pick one (25.3 §14, 25.10 §6). This is the same shape as 25.10 §18's integration failure — and it is here again because the mechanism is different: that chapter's was a process failure about ownership, this one is a state-model disagreement about retention.

The discriminator is two counters, and they must exist on both sides. obligations_outstanding_at_recovery on each die, plus reissued_after_recovery and completed_after_recovery. After the hang: A shows 1 retained, 0 re-issued; B shows 1 discarded, 0 received. One read from each die and the disagreement is explicit — without them it is a multi-week investigation across two teams who each have proof their die is correct.

And the prevention is testing the pair, with the specific stimulus. 25.8 §17's recovery-under-live-traffic test — inject a recovery with obligations outstanding, and check that exactly one completion occurs per semantic operation. A drained-pipeline recovery test, which is what both teams naturally wrote, passes on both dies and proves nothing about the pair.

22. What Public Sources Do Not Disclose

Mandatory, and unusually long here, because this chapter's subject is almost entirely undocumented for every product.

Two readings.

The right column is the entire chapter. Public material reviewed does not establish any of it for any product discussed here — which is why §3 marks the mechanisms Class G and why none of §9–§19 claims to describe anyone's implementation.

And the last row is the one worth ending on. §21's failure hinges on what survives a recovery — a question no product's public documentation answers, and one that any two-die integration must answer explicitly. The framework's value is not that it tells you the answer. It is that it tells you the question is load-bearing, and that leaving it unanswered is a decision with a hang at the end of it.

23. Common Misconceptions

"Reset is one signal." §7: five distinct scopes plus a diagnostic clear, each with its own owner and its own must-not-disturb list.

"A link reset just resets the link." §8–§9: if it clears semantic obligations or restarts ID allocation, the peer still holds the old identities — and the result is wrong data with every integrity check passing.

"A synchroniser makes a signal safe to cross." §11: it handles metastability. It does not create a sampling opportunity for a pulse narrower than the destination's clock period.

"Synchronise the data bus too." §12 Contract: independent synchronisers can resolve on different cycles, producing a value that was never sent. Hold the payload stable by protocol and synchronise only the control signal.

"The link is ACTIVE, so the function is ready." §13, §15: ACTIVE is level 5 of 7. Configuration and semantic readiness are two further conditions, and publishing early produces the "fails only on the first access after boot" bug.

"Each die can apply configuration when it is ready." §16: that is self-commit, and it opens a window in which dies interpret the same transaction differently.

"Commit on timeout so bring-up is smoother." §17 Contract: a timeout means a die has not staged the change. Committing anyway is self-commit with extra steps.

"The link index identifies the function." §19: it does until a failover, after which valid responses are rejected and the availability feature causes the hang.

"A physically partitioned package must look partitioned to software." §20: MI300X exposes the partition and lets software select 1, 2 or 8 logical GPUs; Blackwell presents two dies as one CUDA device. Both ship.

"If both dies pass sign-off, the product works." §21: two internally consistent, individually verified dies can hold opposite beliefs about what a recovery does — and hang.

24. Understanding Check

25. Summary

Six things.

Splitting an SoC creates boundaries for state, time and ownership (§1, §5). Eight guarantees that were free on one die must now be engineered — and the five boundaries along the seam do not coincide.

Reset is five scopes plus a diagnostic clear (§7–§8). Semantic obligations, identities, configuration and first-fault capture must survive a link reset — and §9 is the corruption that follows when they do not.

Time is per-die (§10–§12). Only levels cross safely; a pulse is a bet on the destination's clock period. Hold the request until acknowledged, hold the payload stable by protocol, bound the wait, and name the timeout.

Readiness is seven ordered conditions (§13–§15). Link ACTIVE is level 5 — publishing there produces the "fails only on the first access after boot" race, and a sticky high-water level turns it into one register read.

Configuration is owned, prepared by all, and committed atomically with an epoch (§16–§17); identity must survive a route change (§18–§19), or the failover mechanism becomes the cause of the hang.

And two dies can each be correct and still hang together (§21). Opposite, defensible beliefs about what a recovery retains — found in two counters, prevented by one contract line and one paired test.