Large-Package Systems

Chapter 18.2 gave several accelerators one fabric; 18.3 made them unlike. Both assumed the package was flat — one switching structure, one clock, one reset, one configuration. This chapter removes all four assumptions at once.

1. The One-Sentence Model

A large package is a hierarchy of local fabrics connected by constrained cuts. Everything new about it follows from that sentence: routing gains levels, bandwidth gains bottlenecks that no edge link widens, clocks and resets gain domains, configuration gains a coordination problem, and failure gains a containment problem.

2. What This Chapter Owns

15.4 already owns the package budget: the resource graph, the flow matrix, utilisation per resource, and allocation policy. This chapter does not redo that arithmetic. What it adds is everything that appears when the package acquires levels:

Routing gains a hierarchy (§7–§11), and a flat table that is functionally correct fails timing at scale.

Clocks and resets gain domains (§17–§26). A one-cycle credit-return pulse crossing unsafely becomes a permanent credit leak that stalls a link after a long run — and a single global reset tree turns a recoverable link fault into a package outage.

Configuration gains a coordination problem (§27–§30). Two clusters holding different route tables is a mis-route or a loop, and neither is detectable locally.

Congestion and failure gain a containment problem (§31–§39), including a deadlock whose wait-for cycle spans two levels and therefore cannot be seen from either.

3. Sourcing

4. What "Large" Actually Changes

Five problems that a flat package does not have.

None of these is a bigger version of a flat-package problem. Each is a new kind of problem introduced by the existence of a level boundary — which is why 18.2's chapter, thorough as it is, does not cover any of them.

5. The Hierarchy

The bridge is the chapter. It is a bandwidth cut (§12), a clock boundary (§17), a reset boundary (§23), a configuration boundary (§27), a failure boundary (§31) and a deadlock participant (§38) — all at once, because it is the level boundary.

6. Three Levels

7. The Hierarchical Route

// ILLUSTRATIVE hierarchical route. NOT a UCIe format (Section 3).
typedef struct packed {
  logic [CLUSTER_W-1:0] cluster;      // which cluster owns the destination
  logic [PORT_W-1:0]    local_port;   // which port within that cluster
  logic [EPOCH_W-1:0]   route_epoch;  // which topology configuration decided it
} hierarchical_route_t;

Architecture. Two levels in one record, plus the epoch that produced it. The split is what makes the decode hierarchical (§9) — a packet leaving its cluster needs only the cluster field until it arrives, and the local port is consumed at the far end.

State. Carried with the packet, captured once at acceptance and never recomputed (§10).

Cycle behaviour. The cluster field is consumed by the source cluster's bridge selection; the local port by the destination cluster's fabric.

Contract. Every stage assumes the route is stable for the packet's lifetime. That is not a property of the routing table; it is a property of the capture (18.2 §10).

Failure. Omitting route_epoch, which leaves a packet unable to prove which topology configuration routed it — and §28's mixed configuration then produces a mis-route that nothing attributes.

DV. §11's properties.

8. Wrong Architecture — a Flat Routing Table

// WRONG at scale — every destination decoded centrally, in one level.
logic [PORT_W-1:0] flat_route [NUM_DESTS];        // NUM_DESTS in the hundreds
 
always_comb begin
  out_port = PORT_NONE;
  for (int d = 0; d < NUM_DESTS; d++)
    if (pkt.dst == d[DST_W-1:0]) out_port = flat_route[d];
end

It is functionally correct and it does not close timing.

Four properties.

Every functional test passes. Routing is correct for every destination. The failure is entirely in synthesis and timing closure, which is a class of failure functional verification never reports.

And the fix is not pipelining alone. Adding a stage to the flat decode helps the timing and does nothing about the fanout of a table whose every entry must reach one selector — which is an area and congestion problem on the die, not just a delay.

The hierarchical fix changes the shape, not the depth. A cluster select over NUM_CLUSTERS entries followed by a local select over PORTS_PER_CLUSTER is two small decodes instead of one enormous one. For 512 destinations as 16 clusters of 32, that is a 16-way and a 32-way decode rather than a 512-way one.

And it costs a cycle. §9 is that trade, stated explicitly rather than discovered.

9. Hierarchical Decode

// ILLUSTRATIVE two-stage decode. Two small tables instead of one large one.
localparam int CLUSTER_W = $clog2(NUM_CLUSTERS);
localparam int PORT_W    = $clog2(PORTS_PER_CLUSTER);
 
// Stage 1 — which cluster, and if it is not ours, which bridge reaches it.
logic [BRIDGE_W-1:0] bridge_sel_q;
logic                is_local_q;
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) begin
    is_local_q   <= 1'b0;
    bridge_sel_q <= '0;
  end else if (stage1_en) begin
    is_local_q   <= (pkt_in.route.cluster == MY_CLUSTER);
    bridge_sel_q <= cluster_to_bridge[pkt_in.route.cluster];   // small table
  end
 
// Stage 2 — within our cluster, which local port.
logic [PORT_W-1:0] local_port_q;
always_ff @(posedge clk)
  if (stage2_en)
    local_port_q <= pkt_s1.route.local_port;                   // already carried
 
assign out_port = is_local_q ? local_port_q : bridge_port(bridge_sel_q);

Architecture. Two registered stages. Stage 1 answers "is this leaving the cluster, and by which bridge"; stage 2 answers "which local port" — and a packet leaving the cluster never evaluates stage 2 locally at all, because the destination cluster does it.

State. Two small tables plus two pipeline registers. cluster_to_bridge has NUM_CLUSTERS entries; the local decode has PORTS_PER_CLUSTER. Both are small enough to meet timing at scale, which is the entire point.

Cycle behaviour. One extra cycle of latency versus §8's flat decode. That is the trade and it should be stated in the design document: a cycle of latency bought Fmax and die area.

Contract. The destination cluster's fabric relies on local_port being the one captured at acceptance. A design where the destination cluster re-derives it from its own table has re-introduced a recompute across a configuration change (§10).

Failure. Placing the pipeline register after the port select rather than between the stages, which keeps the whole path combinational and buys nothing.

DV. Cover local and cross-cluster routes for every cluster; assert bridge_sel_q names a bridge that is up (§32).

10. The Route Is Captured, Not Recomputed

// ILLUSTRATIVE. The route decision — BOTH levels and the epoch — is made once
// at acceptance and carried. 18.2 Section 10's rule, at hierarchy scope.
typedef struct packed {
  logic                    valid;
  logic [TXN_W-1:0]        txn_id;
  hierarchical_route_t     route;        // captured at acceptance
  logic [CLUSTER_W-1:0]    src_cluster;
} pkg_txn_t;
 
pkg_txn_t txn_q [MAX_INFLIGHT];

Architecture. One record per in-flight packet holding both levels of the route and the epoch. The hierarchy makes this more important, not less: with two levels there are two opportunities to recompute, and the second one is on a different die whose configuration may have committed at a different time (§28).

Contract. The bridge, the destination cluster's fabric and the completion path all read this. None of them may derive a route from a live table.

Failure. Capturing the cluster and re-deriving the local port at the destination — which is exactly the split-configuration hazard §28 describes, and it produces a packet delivered to the wrong port of the right cluster.

DV. §11.

11. SVA — Route and Epoch Stable for the Packet's Lifetime

// MANDATORY. Both levels and the epoch are immutable while the packet lives.
property p_route_stable;
  @(posedge clk) disable iff (!rst_n)
    txn_q[IDX].valid |-> ($stable(txn_q[IDX].route.cluster)
                       && $stable(txn_q[IDX].route.local_port)
                       && $stable(txn_q[IDX].route.route_epoch));
endproperty
a_route_stable: assert property (p_route_stable);
 
// The port actually taken matches the captured route.
property p_port_matches_capture;
  @(posedge clk) disable iff (!rst_n)
    xfer_fire |-> (out_port == expected_port(txn_q[xfer_txn].route));
endproperty
a_port_matches_capture: assert property (p_port_matches_capture);
 
// A packet's epoch matches the epoch that was active when it was accepted.
property p_epoch_matches_acceptance;
  @(posedge clk) disable iff (!rst_n)
    accept_fire |-> (txn_q[alloc_idx].route.route_epoch == active_topo_epoch_q);
endproperty
a_epoch_matches_acceptance: assert property (p_epoch_matches_acceptance);
 
// A packet is never forwarded onto a bridge that is not usable (Section 32).
property p_no_forward_to_down_bridge;
  @(posedge clk) disable iff (!rst_n)
    (xfer_fire && !is_local_q) |-> (bridge_state_q[bridge_sel_q] == BRIDGE_UP);
endproperty
a_no_forward_to_down_bridge: assert property (p_no_forward_to_down_bridge);

Architecture. Four properties: immutability, effect, acceptance consistency, and bridge liveness.

Why the second exists given the first. The first proves the record is stable; the second proves the datapath reads it — a design can capture correctly and still have a stage that consults the live table.

DV. All four always-on. The third catches a packet accepted during a partially-committed configuration (§28).

12. Shared Cuts

15.4 established min-cut reasoning at package scope. What the hierarchy adds is that the cut is now structural rather than incidental — every cross-cluster packet must traverse a bridge, by construction.

Aggregate bandwidth inside the clusters is irrelevant to cross-cluster throughput. A package with enormous local fabrics and one modest bridge has exactly one modest bridge's worth of cross-cluster capacity — and every local link can be idle while that bridge saturates.

13. A Worked Cut

Illustrative numbers throughout.

Cluster A: 4 accelerators, each capable of offering 40 units/cycle
Cluster B: the destination for all of it
Bridge A->B capacity: 100 units/cycle
 
peak aggregate offered   = 4 × 40 = 160 units/cycle
bridge capacity          = 100 units/cycle
oversubscription         = 160 / 100 = 1.6×
 
sustained cross-cluster throughput = min(160, 100) = 100 units/cycle
per-accelerator share              = 100 / 4 = 25 units/cycle
fraction of each accelerator's
  offered rate achieved            = 25 / 40 = 62.5%

Three readings.

Each accelerator achieves 62.5% of what it could offer, and no amount of local fabric bandwidth changes it. The constraint is one link.

Now make it worse in the way real workloads do. If the same four accelerators instead communicate within cluster A, the bridge is unused and the local fabric — illustratively 400 units/cycle — carries all of it. The same hardware delivers 160 or 100 depending only on where the destinations are (§14).

And adding accelerators to cluster A does not help. Six accelerators offering 240 still get 100 across the bridge, and each now achieves 41.7%. Scaling the compute without scaling the cut lowers per-agent efficiency, which is 15.3 §11's oversubscription argument with the cut made structural.

14. Locality Is a Placement Decision

Two consequences.

Placement is a performance decision made before any traffic flows, and it is made by whoever assigns work to chiplets — which in 18.3 §38's terms is the scheduler. The hardware's job is to expose the information the decision needs (§16, §43).

And the cost is asymmetric in an important way. Local traffic that stays local costs the bridge nothing; cross-cluster traffic costs the bridge and both local fabrics, because it traverses all three. So a cross-cluster placement is not "one extra hop" — it consumes capacity at three levels.

15. Wrong Architecture — Uniform Cost for Local and Cross-Cluster

// WRONG — the placement logic treats every destination as equally reachable.
assign placement_cost[agent] = compute_est_q[agent];      // no path term at all

1. Two agents that communicate heavily are placed in different clusters,
   because both clusters had idle capacity.
2. All of their traffic now crosses the bridge.
3. The bridge saturates at 100 units/cycle while both local fabrics run
   at a small fraction of their 400.
4. -> the package delivers a quarter of what its components could,
     with every component healthy and every link functioning.

Three properties.

It is correctness-clean. Every packet arrives, every result is right. Only a performance model detects it (18.3 §41's Layer 4 argument at package scope).

The diagnostic signature is distinctive and easy to miss. Local link utilisation low, bridge utilisation at 100% — which reads as "we need a bigger bridge" when the actual answer is "these two agents should be in the same cluster".

And it compounds with 18.3's locality mistake. That chapter's scheduler ignored where the data was; this one ignores where the peer is. Both are missing path terms in a cost model, at two different scopes.

16. Locality State the Hardware Must Expose

// ILLUSTRATIVE. The hardware cannot make the placement decision, but it must
// make it decidable. These are diagnostic and hint values, not policy.
logic [CLUSTER_W-1:0]  agent_cluster_q [NUM_AGENTS];    // where each agent lives
logic [63:0]           pair_traffic_q  [NUM_AGENTS][NUM_AGENTS];  // who talks to whom
logic [63:0]           cross_cluster_bytes_q [NUM_CLUSTERS];
logic [63:0]           local_bytes_q         [NUM_CLUSTERS];

Architecture. A cluster map plus a communication matrix. pair_traffic_q is the structure that makes §15's mistake visible: a heavily-communicating pair in different clusters is immediately apparent from it and from nothing else.

State. NUM_AGENTS² counters is expensive; a sampled or hashed approximation is the usual compromise, and it is sufficient because the decision it informs is coarse.

Contract. Placement policy reads it. Nothing in the datapath does — this is diagnostic state, and it must survive the events that reset everything else (14.5).

Failure. Aggregating to per-agent totals rather than per-pair, which loses exactly the information the decision needs — an agent can be busy without any single peer dominating.

DV. Not a correctness structure; verify the counters increment on actual transfers (15.3 §14).

17. Clock Domains Appear at the Level Boundary

A large package may run its clusters on different clocks — different frequencies, different sources, or the same nominal frequency with no phase relationship.

Row 3 is where §20 lives, and row 4 is where a mis-designed pointer crossing lives. Both produce intermittent, load-dependent, extremely hard failures.

18. An Asynchronous FIFO, Sketched

// ILLUSTRATIVE async FIFO across a cluster boundary. Standard structure; the
// point here is WHERE it belongs, not a novel implementation.
logic [PTR_W:0] wr_ptr_bin_q,  wr_ptr_gray_q;    // write domain
logic [PTR_W:0] rd_ptr_bin_q,  rd_ptr_gray_q;    // read domain
logic [PTR_W:0] wr_ptr_gray_sync_q [2];          // synchronised INTO read domain
logic [PTR_W:0] rd_ptr_gray_sync_q [2];          // synchronised INTO write domain
 
// Payload RAM — written in one domain, read in the other, NOT reset.
pkg_flit_t fifo_mem [FIFO_DEPTH];
 
// Gray conversion: exactly one bit changes per increment, so a pointer
// sampled mid-transition is either the old or the new value — never garbage.
function automatic logic [PTR_W:0] bin2gray(input logic [PTR_W:0] b);
  return b ^ (b >> 1);
endfunction
 
always_ff @(posedge wr_clk or negedge wr_rst_n)
  if (!wr_rst_n) begin
    wr_ptr_bin_q  <= '0;
    wr_ptr_gray_q <= '0;
  end else if (wr_en && !wr_full) begin
    wr_ptr_bin_q  <= wr_ptr_bin_q + 1'b1;
    wr_ptr_gray_q <= bin2gray(wr_ptr_bin_q + 1'b1);
  end
 
// Two-flop synchronisers, one per direction.
always_ff @(posedge rd_clk or negedge rd_rst_n)
  if (!rd_rst_n) begin
    wr_ptr_gray_sync_q[0] <= '0;
    wr_ptr_gray_sync_q[1] <= '0;
  end else begin
    wr_ptr_gray_sync_q[0] <= wr_ptr_gray_q;
    wr_ptr_gray_sync_q[1] <= wr_ptr_gray_sync_q[0];
  end

Architecture. Gray-coded pointers crossed through two-flop synchronisers, with the payload RAM written in one domain and read in the other. Gray coding is the essential part: exactly one bit changes per increment, so a pointer sampled mid-transition resolves to either the old value or the new one and never to a garbage combination.

State. Two binary pointers, two Gray pointers, four synchroniser flops, and FIFO_DEPTH payload entries. The payload is not reset — the pointers gate every read and start equal (17.4 §14).

Cycle behaviour. Full and empty are computed conservatively: each domain compares its own pointer against a synchronised, therefore stale copy of the other's. Full may be asserted when the FIFO is not quite full, and empty when it is not quite empty — both are safe directions, and both are unavoidable.

Contract. The producer relies on wr_full; the consumer on rd_empty. Both are conservative, which is why depth must be sized with the synchroniser latency included — a two-cycle-deep FIFO across a slow crossing spends most of its time apparently full.

Failure. Binary pointers instead of Gray, which lets a multi-bit transition be sampled as a value that was never written. Or a single synchroniser flop, which does not adequately reduce metastability propagation.

DV. Formal or constrained-random with the two clocks at unrelated frequencies and phases; assert no overflow, no underflow, and no read of an unwritten slot.

19. Wrong RTL — a Pulse Crossed Directly

// WRONG — a one-cycle event pulse crossed with a plain synchroniser.
always_ff @(posedge dst_clk or negedge dst_rst_n)
  if (!dst_rst_n) begin
    pulse_sync_q[0] <= 1'b0;
    pulse_sync_q[1] <= 1'b0;
  end else begin
    pulse_sync_q[0] <= src_pulse;      // src_pulse is ONE src_clk cycle wide
    pulse_sync_q[1] <= pulse_sync_q[0];
  end
 
assign dst_event = pulse_sync_q[1];

If the destination clock is slower than the source, the pulse can fall entirely between two destination edges and is never sampled.

Four properties.

It works at some clock ratios and not others, so a design verified at one configuration fails at another — and clock ratios are exactly what a large package varies per cluster.

The loss is silent. There is no error, no flag, and nothing downstream knows an event was supposed to arrive.

The correct bridge is a toggle or a handshake. A toggle changes level on each event and the destination detects the edge, so the information is held until sampled. A handshake additionally provides backpressure, which is required if events can arrive faster than the destination can consume them.

And §20 is what makes this catastrophic rather than merely lossy.

20. Wrong CDC — a Credit Return Pulse

// WRONG, and this is the version that kills a system slowly.
// A credit return is a one-cycle pulse crossed with a plain synchroniser.
assign credit_return_dst = pulse_sync_q[1];      // Section 19's bridge

1. Under light traffic, credit returns are sparse and the destination
   happens to sample nearly all of them. Everything works.
2. Under sustained traffic, credit returns are frequent, and a fraction
   of them fall between destination edges and are LOST.
3. Every lost pulse is a credit that will never be returned.
4. The transmit side's credit count decreases monotonically.
5. -> after a long run, the credit count reaches zero and the link
     stops permanently. No error is reported anywhere.

Four properties, and this is the chapter's best bug.

It is a leak, not a corruption. Nothing is ever wrong; there is simply less and less capacity. The symptom is a link that works perfectly and then stops, and the time to failure depends on traffic rate.

It is load-dependent in the direction that hides it. Light traffic loses few pulses; the failure appears only after sustained load, which a functional regression rarely applies.

And the credit-conservation assertion catches it immediately (18.2 §24): available plus outstanding stops equalling the advertised total the moment a pulse is lost. Bounds checks do not — the count simply drifts downward within its range.

The fix is an event-count crossing rather than a pulse crossing:

// ILLUSTRATIVE. Cross a Gray-coded COUNT, not an event. The destination
// computes the difference, so a missed sample is recovered by the next one.
logic [CNT_W-1:0] credit_ret_count_q;              // source domain, monotonic
logic [CNT_W-1:0] credit_ret_gray_q;
logic [CNT_W-1:0] credit_ret_gray_sync_q [2];      // destination domain
logic [CNT_W-1:0] credit_ret_seen_q;               // destination's last value
 
always_ff @(posedge src_clk)
  if (credit_returned) begin
    credit_ret_count_q <= credit_ret_count_q + 1'b1;
    credit_ret_gray_q  <= bin2gray(credit_ret_count_q + 1'b1);
  end
 
// Destination: the DIFFERENCE is the number of credits to add.
wire [CNT_W-1:0] ret_bin = gray2bin(credit_ret_gray_sync_q[1]);
wire [CNT_W-1:0] ret_delta = ret_bin - credit_ret_seen_q;

A missed sample is not a lost credit — the count keeps rising, and the next successful sample delivers the difference. That is the structural difference between crossing an event and crossing a state.

21. SVA — the CDC Contract

// MANDATORY. Credit conservation across the crossing — this is what catches
// Section 20 immediately (18.2 Section 24's property, at a clock boundary).
property p_credit_conserved_across_cdc;
  @(posedge dst_clk) disable iff (!dst_rst_n)
    (credit_available_q + tb_outstanding_at_source() == CREDIT_INIT);
endproperty
a_credit_conserved_across_cdc:
  assert property (p_credit_conserved_across_cdc);
 
// The returned count is monotonic — a decrease means a sampling error.
property p_return_count_monotonic;
  @(posedge dst_clk) disable iff (!dst_rst_n)
    (ret_bin >= $past(ret_bin)) || wrapped(ret_bin, $past(ret_bin));
endproperty
a_return_count_monotonic: assert property (p_return_count_monotonic);
 
// The async FIFO never overflows or underflows (Section 18).
property p_fifo_no_overflow;
  @(posedge wr_clk) disable iff (!wr_rst_n)
    wr_en |-> !wr_full;
endproperty
a_fifo_no_overflow: assert property (p_fifo_no_overflow);
 
property p_fifo_no_underflow;
  @(posedge rd_clk) disable iff (!rd_rst_n)
    rd_en |-> !rd_empty;
endproperty
a_fifo_no_underflow: assert property (p_fifo_no_underflow);

Architecture. Four properties: conservation across the crossing, count monotonicity, and FIFO bounds in each domain.

Why conservation is the one that matters. §20's leak stays within every bound indefinitely. Only the sum stops matching, and it stops matching on the very first lost pulse.

Note the clock each property is written on. A CDC property written on the wrong clock samples a signal that is not stable in that domain — which produces false failures and, worse, false passes. Each property above is on the domain that owns the signals it reads.

DV. Run with the two clocks at several unrelated ratios, including the destination substantially slower. A single 1:1 configuration cannot reach §19's failure at all.

22. Reset Domains

A large package has more than one reason to reset, and they have different scopes.

These are nested, not alternatives, and the most common architectural error is collapsing the last two into the third — which turns a routine link retrain into a cluster outage (§23).

23. Wrong Architecture — One Global Reset Tree

// WRONG — every recoverable event drives the package reset.
assign package_rst_n = por_n && !(|link_recovery) && !(|cluster_fault);

Illustrative, a package with 16 links each entering recovery an illustrative 0.5% of the time, independently:

P(no link recovering) = 0.995^16 ≈ 0.9229
 
fraction of time the package is held in reset ≈ 7.7%
 
and each reset costs a full re-initialisation:
  illustratively 50,000 cycles of bring-up per event
  events per unit time scale with the link count

Four properties.

It is functionally correct. After every reset the package comes back and works. Nothing is corrupted, and a functional regression passes.

The availability cost is severe and scales the wrong way. More links means more recovery events means more package resets — so the parameter meant to provide bandwidth reduces availability. This is the same anti-pattern as a global fabric pause (18.2 §47) and a global maintenance stall (17.4 §29), now with a reset's much larger cost.

And it destroys evidence. A package reset clears the first-fault record, the performance counters and the diagnostic state — so the event that caused it is unattributable afterwards (14.5). Every subsequent investigation starts from nothing.

The correct structure is separate, nested reset enables:

// ILLUSTRATIVE. Each scope has its own control, and a link event drives only
// the narrowest one that can resolve it.
logic por_n;                                  // power-on — everything
logic pkg_soft_rst;                           // deliberate, system-level
logic cluster_rst      [NUM_CLUSTERS];        // per cluster
logic link_recover     [NUM_LINKS];           // transport only — NOT a reset
logic block_rst        [NUM_BLOCKS];          // local
 
// A link recovery drives the link's own recovery sequence and nothing else.
assign cluster_rst[c] = pkg_soft_rst || cluster_fault_fatal[c];

24. What Survives Each Reset Scope

The table this chapter exists to produce, and the one 19.1 builds on.

Three readings.

Column 1 is almost entirely "survives", which is the whole argument of 14.2 §4 — a link recovery rebuilds a link and changes nothing semantic. §23's design turns that column into "lost" for every row.

Rows 9 and 10 say "should survive" even for a cluster or package reset. Diagnostic state that is cleared by the event you are diagnosing has no evidentiary value, so it belongs in a reset domain that deliberate resets do not reach — which is an architectural decision, made once, and easy to get wrong by omission.

And row 4 is the one that constrains everything else. Memory contents survive resets that clear the state describing them, so a cluster reset can leave memory holding values that no surviving structure knows about — which is why bring-up after a cluster reset is a re-discovery problem, not just an initialisation one.

25. SVA — Reset-Domain Survival

// MANDATORY. A link recovery does not clear semantic or diagnostic state.
property p_link_recovery_preserves_semantics;
  @(posedge clk) disable iff (!por_n)
    link_recover[LNK_UT] |=> ($stable(txn_q[IDX].valid)
                           && $stable(txn_q[IDX].route)
                           && $stable(first_fault_q)
                           && $stable(perf_counters_q));
endproperty
a_link_recovery_preserves_semantics:
  assert property (p_link_recovery_preserves_semantics);
 
// A link recovery never asserts a cluster or package reset.
property p_link_recovery_is_not_a_reset;
  @(posedge clk) disable iff (!por_n)
    link_recover[LNK_UT] |-> (!(|cluster_rst) && !pkg_soft_rst);
endproperty
a_link_recovery_is_not_a_reset:
  assert property (p_link_recovery_is_not_a_reset);
 
// One cluster's reset does not disturb another cluster's state.
property p_cluster_reset_is_contained;
  @(posedge clk) disable iff (!por_n)
    (cluster_rst[CL_A] && !cluster_rst[CL_B])
      |=> $stable(cluster_state_q[CL_B]);
endproperty
a_cluster_reset_is_contained: assert property (p_cluster_reset_is_contained);
 
// The first-fault record survives every reset except power-on.
property p_first_fault_survives;
  @(posedge clk) disable iff (!por_n)
    (|cluster_rst || pkg_soft_rst) |=> $stable(first_fault_q);
endproperty
a_first_fault_survives: assert property (p_first_fault_survives);

Architecture. Four properties: semantic and diagnostic survival, reset-scope separation, containment, and evidence preservation.

Why the second is written as a negative. It forbids a specific wiring — a link event reaching a reset tree — which is exactly §23's line of code, and it fires at elaboration-adjacent scope rather than waiting for a symptom.

Why the fourth is disable iff (!por_n) rather than the local reset. The property is about surviving those resets, so it must not be disabled by them. Getting the disable condition wrong here makes the property vacuous, which is the most common way a survival property fails to do its job.

DV. Inject each reset scope separately with live traffic and outstanding work.

26. Configuration Hierarchy

A large package has configuration at several levels, and they must be consistent with one another.

A packet's route is computed from two of these levels (§7), which means a configuration change that is committed at one level and not the other is a mis-route that neither level can detect locally. §27 is that failure.

27. Wrong Configuration Update — a Partial Commit

// WRONG — each cluster commits its own copy when it receives it.
always_ff @(posedge clk)
  if (config_message_received)
    active_route_q <= received_route;      // ← whenever it arrives, per cluster

1. The root distributes a new topology. Cluster A receives it at cycle 100
   and commits. Cluster B receives it at cycle 140 and commits.
2. Between cycles 100 and 140, A routes by the NEW topology and B by the OLD.
3. A packet from A to a destination that MOVED is sent to B by the new map.
4. B's old map says that destination is not local and forwards it back to A.
5. -> the packet loops, or is dropped, or lands at the wrong port.

Four properties.

Both clusters are individually correct. Each committed a valid configuration atomically. The inconsistency is between them and is invisible from either.

The window is small and the consequences are not. A looping packet consumes bridge bandwidth indefinitely and holds resources; a dropped packet is an accepted object that vanished (10.1 §10).

The failure requires a topology change under traffic, which is rare and inevitable — link repair, degradation, rebalancing and partition changes all cause it.

And it is exactly 17.2 §11's overlapping-region failure at a new scope: two structures disagreeing about who owns an address, resolved silently by whichever one the packet reaches first.

28. Coordinated Commit

// ILLUSTRATIVE two-phase topology commit. The clusters PREPARE independently
// and COMMIT together, so no window of disagreement exists.
typedef enum logic [1:0] {
  CFG_STABLE   = 2'd0,
  CFG_PREPARED = 2'd1,   // staged, validated, NOT yet in the datapath
  CFG_COMMIT   = 2'd2,   // the coordinated instant
  CFG_ABORT    = 2'd3
} cfg_phase_e;
 
cfg_phase_e            cfg_phase_q;
hierarchical_route_t   requested_route_q [NUM_DESTS];
hierarchical_route_t   active_route_q    [NUM_DESTS];
logic [EPOCH_W-1:0]    active_topo_epoch_q;
logic [NUM_CLUSTERS-1:0] cluster_prepared_q;     // every cluster must be ready
 
// The root commits only when EVERY cluster reports prepared and drained.
assign global_commit_allowed =
      (cluster_prepared_q == all_clusters_mask)
   && (cross_cluster_inflight == '0);
 
always_ff @(posedge clk or negedge por_n)
  if (!por_n) begin
    cfg_phase_q         <= CFG_STABLE;
    active_topo_epoch_q <= '0;
  end else begin
    unique case (cfg_phase_q)
      CFG_STABLE:   if (cfg_start)              cfg_phase_q <= CFG_PREPARED;
      CFG_PREPARED: if (cfg_abort)              cfg_phase_q <= CFG_ABORT;
                    else if (global_commit_allowed) cfg_phase_q <= CFG_COMMIT;
      CFG_COMMIT:   begin
                      active_route_q      <= requested_route_q;   // atomic
                      active_topo_epoch_q <= active_topo_epoch_q + 1'b1;
                      cfg_phase_q         <= CFG_STABLE;
                    end
      CFG_ABORT:    if (abort_complete)         cfg_phase_q <= CFG_STABLE;
      default:                                  cfg_phase_q <= CFG_ABORT;
    endcase
  end

Architecture. Prepare, then commit together. The cluster_prepared_q bitmap is the coordination: no cluster's new table enters its datapath until every cluster has staged and validated one, and until cross-cluster traffic has drained.

State. Two tables per cluster, a phase register, a prepared bitmap, and an epoch counter.

Cycle behaviour. CFG_COMMIT is a single cycle in which the table transfers and the epoch advances together — the same atomicity as 17.2 §12, extended across dies.

Contract. Every cluster sees a complete, validated configuration or the previous one, and never a mixture with another cluster.

Failure. §27. Also committing without the drain term, which leaves packets in flight under the old topology while the new one is active — §27's window in a different form.

DV. §29's properties; cover a commit attempted with cross-cluster traffic in flight, and an abort from CFG_PREPARED.

29. SVA — No Mixed Configuration

// MANDATORY. All clusters share one active topology epoch.
property p_no_mixed_epoch;
  @(posedge clk) disable iff (!por_n)
    (cluster_active_epoch_q[CL_A] == cluster_active_epoch_q[CL_B]);
endproperty
a_no_mixed_epoch: assert property (p_no_mixed_epoch);
 
// The active table changes only in the commit phase.
property p_commit_only_in_commit_phase;
  @(posedge clk) disable iff (!por_n)
    $changed(active_route_q) |-> ($past(cfg_phase_q) == CFG_COMMIT);
endproperty
a_commit_only_in_commit_phase:
  assert property (p_commit_only_in_commit_phase);
 
// The commit requires every cluster prepared and cross-cluster traffic drained.
property p_commit_requires_all_prepared;
  @(posedge clk) disable iff (!por_n)
    (cfg_phase_q == CFG_COMMIT) |-> ($past(cluster_prepared_q) == all_clusters_mask)
                                 && ($past(cross_cluster_inflight) == '0);
endproperty
a_commit_requires_all_prepared:
  assert property (p_commit_requires_all_prepared);
 
// A packet never traverses a boundary under a different epoch than its capture.
property p_packet_epoch_consistent;
  @(posedge clk) disable iff (!por_n)
    bridge_xfer_fire |-> (txn_q[xfer_txn].route.route_epoch == active_topo_epoch_q);
endproperty
a_packet_epoch_consistent: assert property (p_packet_epoch_consistent);

Architecture. Four properties: epoch agreement, commit-phase-only changes, the commit precondition, and per-packet consistency.

Why the first is the most valuable. It is a single comparison across dies that makes §27's entire failure class impossible to miss — the epochs disagree for exactly the window in which the bug is live.

Why the fourth is separate. With the drain term, no packet should straddle an epoch. The fourth catches the case where the drain accounting was wrong, which is a different bug from the commit being wrong.

DV. Force a commit with the drain term disabled and confirm the fourth fires.

30. Fault Containment

// ILLUSTRATIVE health state at two levels, because two scopes of failure exist.
typedef enum logic [1:0] {
  BRIDGE_UP         = 2'd0,
  BRIDGE_RECOVERING = 2'd1,   // temporarily unusable — HOLD
  BRIDGE_DEGRADED   = 2'd2,   // usable, reduced capacity  <- Section 41
  BRIDGE_DOWN       = 2'd3    // unusable — explicit policy
} bridge_state_e;
 
typedef enum logic [1:0] {
  CLUSTER_UP        = 2'd0,
  CLUSTER_DEGRADED  = 2'd1,   // some resources lost, still serving
  CLUSTER_ISOLATED  = 2'd2,   // unreachable, but its state is intact
  CLUSTER_FAILED    = 2'd3    // explicit failure policy
} cluster_state_e;
 
bridge_state_e  bridge_state_q  [NUM_BRIDGES];
cluster_state_e cluster_state_q [NUM_CLUSTERS];

Architecture. Two independent health hierarchies. A bridge can be down while both clusters it connects are perfectly healthy — which is CLUSTER_ISOLATED, and it is a state a single health variable cannot express.

State. Two small arrays.

Contract. Routing reads bridge_state_q; placement and admission read cluster_state_q; the failure policy reads both. CLUSTER_ISOLATED is the state that matters most: the cluster's memory contents, coherence state and in-flight work are all intact and simply unreachable, so the correct response is to wait or to reconfigure — never to discard.

Failure. Merging bridge health into cluster health, which makes an unreachable-but-healthy cluster indistinguishable from a failed one — and the two demand opposite responses.

DV. Cover all four states at each level; cover a cluster isolated by a bridge failure while itself healthy.

31. Wrong Rerouting — Changing a Route Mid-Transmission

// WRONG — a bridge failure triggers an immediate reroute of everything.
always_ff @(posedge clk)
  if (bridge_failed[b])
    for (int d = 0; d < NUM_DESTS; d++)
      if (active_route_q[d].uses_bridge == b) active_route_q[d] <= alt_route[d];

A packet partly transmitted over the failed bridge is now also routed over the alternate one.

Three properties.

It is 18.2 §9's split packet, arrived at through a failure path instead of a configuration path — and the failure path is far more likely to be under-tested.

Ordering is also destroyed. Packets already in flight on the old bridge and new packets on the alternate arrive interleaved, so any ordering domain spanning them is violated (18.2 §39).

And a packet may be delivered twice if the failure was transient and the original bridge later delivers what it held. Rerouting must therefore be transaction-aware: it applies to packets not yet started, and packets already in flight are either completed on the original path or explicitly failed — never silently moved.

32. Congestion Has a Hierarchy Too

Backpressure should propagate exactly as far as the dependency and no further. §33 is what happens when it propagates globally, and it is the same anti-pattern as a global pause and a global reset — a local condition given a global scope.

33. Wrong Design — Global Backpressure

// WRONG — one congested destination stops the whole package.
assign package_accept = ready_terms && !(|any_destination_full);

Illustrative, 8 destinations, each full an illustrative 3% of the time independently:

P(none full) = 0.97^8 ≈ 0.784
 
fraction of time the package accepts NOTHING ≈ 21.6%
fraction lost with per-destination backpressure ≈ 3% (that destination only)

And it worsens with scale: at 16 destinations, 0.97^16 ≈ 0.614 — nearly 39% lost.

Three properties.

It is functionally correct. Nothing overflows and everything eventually completes. A functional regression passes.

The scaling direction is wrong again — more destinations means more time stopped, so the parameter meant to provide capacity reduces throughput. This is the third appearance of the pattern in three chapters (17.4 §29, 18.2 §47, and here), which is a measure of how natural the mistake is.

And the diagnostic signature is uniform low utilisation everywhere, which reads as a source shortage rather than a scoping error.

34. Per-Cluster Admission

// ILLUSTRATIVE. Admission consults the resources THIS packet needs, at the
// levels it will traverse — and nothing else.
assign pkt_admit =
      local_queue_space[pkt.route.local_port]        // the local level
   && (is_local || bridge_credit_q[sel_bridge] != '0) // the bridge, IF crossing
   && (is_local || (bridge_state_q[sel_bridge] == BRIDGE_UP))
   && (cluster_state_q[pkt.route.cluster] != CLUSTER_FAILED)
   && cfg_stable;                                    // not mid-commit (Section 28)

Architecture. Five terms, and three of them apply only to cross-cluster packets. A local packet is admitted on local resources alone — which is what makes local traffic continue while a bridge is saturated (§32).

State. None of its own; a conjunction over per-level state.

Cycle behaviour. Evaluated at acceptance, with is_local from the captured route (§10) rather than a live lookup.

Contract. Accepting a packet commits every level it will traverse. A design that checks only the local level accepts a cross-cluster packet with no bridge credit, which then blocks the local queue behind it — §33's global stall arrived at from the opposite direction.

Failure. Using an aggregate bridge credit across all bridges rather than the selected one (17.4 §17's bug at bridge scope).

DV. Force each term false alone; cover a local packet admitted while the bridge is saturated.

35. Deadlock Across the Hierarchy

18.2 §35 built a cycle within one fabric. Here the cycle spans two levels, which is why neither level can see it.

Setup: cluster A's local queue holds packets for cluster B.
       The bridge A->B has no credit.
       Bridge credit returns from B, and B's return path is its own
       local fabric, which is full of packets destined for A.
       Those packets need bridge B->A, which has no credit.
       Credit for B->A returns from A, whose local fabric is full.
 
WAIT-FOR GRAPH:
 
   A's local queue  --waits for-->  bridge A->B credit
   bridge A->B credit --waits for-> B's return path
   B's return path  --waits for-->  bridge B->A credit
   bridge B->A credit --waits for-> A's return path
   A's return path  --waits for-->  A's local queue    (it is full)
 
   -> a cycle of length 5, spanning BOTH clusters and BOTH bridges.

Four properties, and the hierarchy is what makes it hard.

Every local assertion in both clusters passes. Each fabric grants at most one requester per output, only to requesters, only with credit. Every credit counter is bounded and conserved. 18.2 §19 and §24 are all satisfied, in both clusters.

Neither cluster can see the cycle, because half of it is on the other die. A per-cluster progress model detects a stall; only a package-level one identifies the cycle.

The channel dependency graph must therefore span levels (18.2 §36's method, applied to the package). Resource classes now include bridge credits and cross-cluster return paths, and the edges between them cross dies.

And the fix is the same shape at a larger scope: separate the dependency classes. Cross-cluster requests and cross-cluster returns must not share bridge credits, exactly as requests and responses must not share a virtual channel (18.2 §38).

36. Progress Reservation at the Bridge

// ILLUSTRATIVE. Separate credit pools at the bridge for traffic that CONSUMES
// resources and traffic that RELEASES them. Generic — no standard requires it.
logic [CREDIT_W-1:0] bridge_bulk_credit_q     [NUM_BRIDGES];
logic [CREDIT_W-1:0] bridge_progress_credit_q [NUM_BRIDGES];   // reserved, non-zero
logic [AGE_W-1:0]    bridge_progress_age_q    [NUM_BRIDGES];   // saturating
 
// Bulk may use only the bulk pool; progress traffic may use either.
assign bridge_bulk_may_send[b]     = (bridge_bulk_credit_q[b] != '0);
assign bridge_progress_may_send[b] = (bridge_progress_credit_q[b] != '0)
                                  || (bridge_bulk_credit_q[b] != '0);
 
assign bridge_progress_override[b] = (bridge_progress_age_q[b] >= BRIDGE_PROG_BOUND);

Architecture. Split pools at each bridge plus a bounded override. The escape path must exist at the cut, because the cut is where the cycle closes (§35).

State. Two credit counters and one saturating age per bridge.

Contract. §37's liveness property depends on bridge_progress_credit_q being initialised non-zero. Setting it to zero silently removes the guarantee while leaving all the code that implements it — which is why it deserves an elaboration check (18.2 §34).

Failure. Reserving at the local fabrics and not at the bridge, which protects against 18.2's cycle and not against §35's — the two cycles close at different resources.

DV. Saturate cross-cluster bulk traffic with returns trickling, and confirm bridge_progress_age_q never reaches AGE_MAX.

37. SVA — Bridge Resources and Progress

// MANDATORY. Bridge credits conserve, per bridge and per class.
property p_bridge_credit_conserved(int b, int cls);
  @(posedge clk) disable iff (!por_n)
    (bridge_credit_q[b][cls] + tb_outstanding_at_bridge(b, cls) == BRIDGE_CREDIT_INIT);
endproperty
a_bridge_credit_conserved:
  assert property (p_bridge_credit_conserved(BR_UT, CLS_UT));
 
// Bulk never consumes the reserved progress pool.
property p_bulk_never_uses_progress_pool;
  @(posedge clk) disable iff (!por_n)
    (bridge_xfer_fire && (xfer_class == CLASS_BULK))
      |=> $stable(bridge_progress_credit_q[$past(sel_bridge)]);
endproperty
a_bulk_never_uses_progress_pool:
  assert property (p_bulk_never_uses_progress_pool);
 
// LIVENESS, bounded, with assumptions.
//   A1: a bridge that is UP eventually transfers a granted packet
//   A2: the peer cluster eventually returns credit
//   A3: recovery terminates
property p_cross_cluster_progress;
  @(posedge clk) disable iff (!por_n)
    (cross_cluster_req[BR_UT] && (bridge_state_q[BR_UT] == BRIDGE_UP))
      |-> ##[1:BRIDGE_SERVICE_BOUND] bridge_xfer_fire_for(BR_UT);
endproperty
a_cross_cluster_progress: assert property (p_cross_cluster_progress);
 
// Local traffic is never blocked by a saturated bridge (Section 33).
property p_local_traffic_independent;
  @(posedge clk) disable iff (!por_n)
    (local_req && local_queue_space[local_port] && (bridge_credit_q[ANY] == '0))
      |-> ##[1:LOCAL_BOUND] local_xfer_fire;
endproperty
a_local_traffic_independent: assert property (p_local_traffic_independent);

Architecture. Conservation, reservation integrity, cross-cluster liveness, and local independence.

Why the fourth is the containment property. It states, as an assertion, that a saturated bridge does not stop local traffic — which is §33's design failing. It is the only property here that is about a non-effect, and those are the ones that get omitted.

DV. Prove all four. Then zero the progress reservation and confirm §35's deadlock becomes reachable, which validates that the reservation is load-bearing.

38. Power and Thermal Change Service Rates

Two architectural consequences, and the second is a correctness issue rather than a performance one.

Measurement windows must be long enough to include throttling (15.5 §17). A short burst measures the unthrottled rate; reporting that as the system's capability describes a rate the package cannot hold.

And every bound derived from a service rate becomes wrong when the rate changes — which is §39.

39. Wrong Design — Fixed Bounds Under Throttling

// WRONG — a cross-cluster timeout measured at full service rate.
localparam int XCLUSTER_TIMEOUT = 8192;      // measured with no throttling
 
always_ff @(posedge clk)
  if (xcluster_outstanding[id] && (age_q[id] >= XCLUSTER_TIMEOUT))
    declare_failed(id);                      // ← fires on a healthy, throttled package

Three consequences.

A capacity change is reported as a fault. The transaction would have completed; the timeout manufactures a failure.

And the response is worse than the fault. Declaring failure may trigger a recovery, a reroute (§31) or an escalation — on a package that was merely running slower, potentially producing a loop.

The fix is that bounds derived from a service rate are re-derived when the rate changes — the same rule as 16.5 §27's link-width case and 17.5 §10's general statement. What is new here is that the rate can change for a thermal reason with no configuration event at all, so a design that only re-derives on configuration commits still gets it wrong.

40. Topology Instrumentation

// Diagnostic only. PER CLUSTER and PER BRIDGE, because the hierarchy is what
// the counters exist to expose.
logic [63:0] local_bytes_q        [NUM_CLUSTERS];
logic [63:0] cross_cluster_bytes_q[NUM_CLUSTERS];   // Section 14
logic [63:0] bridge_bytes_q       [NUM_BRIDGES];
logic [63:0] bridge_stall_q       [NUM_BRIDGES];    // requested, no credit
logic [63:0] bridge_recovery_q    [NUM_BRIDGES];
logic [63:0] bridge_progress_stall_q [NUM_BRIDGES]; // Section 36 precursor
logic [63:0] cluster_isolated_q   [NUM_CLUSTERS];   // Section 30
logic [63:0] cfg_commit_stall_q;                    // drain cost (Section 28)
logic [63:0] cdc_fifo_full_q      [NUM_CROSSINGS];  // Section 18
logic [63:0] throttle_cycles_q    [NUM_CLUSTERS];   // Section 38

Two properties.

The local-against-cross-cluster ratio is the single most valuable number in a large package. It answers the placement question in one comparison, and no aggregate counter can produce it.

And bridge_progress_stall_q rising is the warning that §35 is approaching — visible long before anything stops.

41. Fault and Performance Correlate Across Levels

A worked causal chain, because in a hierarchy the symptom and the cause are usually at different levels.

1. A bridge link's error rate rises.  ->  retries increase (14.3)
2. Effective bridge bandwidth falls.  ->  bridge_stall_q rises
3. Cross-cluster latency rises.       ->  outstanding entries accumulate
4. Local queues back up behind cross-cluster packets.
5. The local scheduler sees full queues and slows dispatch.
6. -> the SYMPTOM is "the compute cluster is slow".
   -> the CAUSE is a transport error rate on one bridge link.

Three readings.

Five levels separate the cause from the symptom, and each step is a different subsystem with a different owner. Without per-bridge instrumentation, step 1 is invisible and the investigation starts at step 6.

The correlation is what makes it diagnosable. bridge_recovery_q and bridge_stall_q rising together, followed by local queue occupancy rising, is a signature — and it is a different signature from a genuine compute-side slowdown, which shows no bridge activity change at all.

And this is why diagnostic counters must survive recovery (14.5). A design that clears them on each link recovery erases step 1 exactly when it matters.

42. The Package Scoreboard

// Verification-only. FIVE models, because a hierarchy has five kinds of state
// that can independently be wrong.
class large_package_scoreboard;
 
  // ---- Layer 1: TOPOLOGY model — destination to cluster/path, per epoch.
  typedef struct { int cluster; int local_port; int bridge; } topo_entry_t;
  topo_entry_t topo [int][int];        // [epoch][dst] — every epoch retained
 
  // ---- Layer 2: PACKET model.
  typedef struct {
    int  src_cluster, dst_cluster, dst_port;
    int  captured_bridge;
    int  predicted_bridge;             // from Layer 1, at the ACCEPT epoch
    int  accept_epoch;
    int  flits_sent_on [int];          // [bridge] -> count. MUST be one bridge.
    int  deliveries;                   // MUST be <= 1
    bit  retired;
  } pkt_model_t;
  pkt_model_t pkts [int];
 
  // ---- Layer 3: RESOURCE model, per bridge and class.
  typedef struct {
    int credits_available;
    int credits_outstanding;
    int queue_occupancy;
  } bridge_res_t;
  bridge_res_t bridge_res [int][int];
 
  // ---- Layer 4: HEALTH model.
  typedef struct {
    int  bridge_state [int];
    int  cluster_state [int];
    int  cluster_epoch [int];          // MUST all be equal (Section 29)
  } health_model_t;
  health_model_t health;
 
  // ---- Layer 5: PERFORMANCE model.
  typedef struct {
    longint local_bytes, cross_bytes;
    int     throttled_cycles;
  } perf_model_t;
  perf_model_t perf [int];             // per cluster
 
  // ---- Catches Section 27 — and only a cross-die model can.
  function void check_epoch_agreement();
    int ref_epoch = health.cluster_epoch[0];
    foreach (health.cluster_epoch[c])
      if (health.cluster_epoch[c] != ref_epoch)
        $error("MIXED CONFIGURATION: cluster %0d at epoch %0d, cluster 0 at %0d "
             , "(Section 27)", c, health.cluster_epoch[c], ref_epoch);
  endfunction
 
  // ---- Catches Section 31 — a packet split across two bridges.
  function void check_single_bridge(int id);
    int used = 0;
    foreach (pkts[id].flits_sent_on[b]) if (pkts[id].flits_sent_on[b] > 0) used++;
    if (used > 1)
      $error("PACKET %0d traversed %0d bridges — split (Section 31)", id, used);
  endfunction
 
  // ---- Catches Section 20 — the credit leak, before it stalls the link.
  function void check_bridge_conservation(int b, int cls);
    if (bridge_res[b][cls].credits_available + bridge_res[b][cls].credits_outstanding
        != BRIDGE_CREDIT_INIT)
      $error("BRIDGE %0d class %0d credits not conserved: %0d + %0d != %0d",
             b, cls, bridge_res[b][cls].credits_available,
             bridge_res[b][cls].credits_outstanding, BRIDGE_CREDIT_INIT);
  endfunction
 
  // ---- Catches Section 15 — a PERFORMANCE fault no correctness model sees.
  function void note_placement(int cluster);
    if (perf[cluster].cross_bytes > (perf[cluster].local_bytes * CROSS_RATIO_LIMIT))
      $display("NOTE: cluster %0d moved %0d cross-cluster bytes against %0d local "
             , "— placement may be wrong (Section 15)",
               cluster, perf[cluster].cross_bytes, perf[cluster].local_bytes);
  endfunction
 
endclass

Architecture. Five models keyed by epoch-and-destination, by packet, by bridge-and-class, by component, and by cluster.

Layer 4's cluster_epoch array is the one no single-cluster model can have. §27's failure is an inconsistency between dies, and a verification environment built per cluster reproduces the design's blindness exactly.

Layer 2's flits_sent_on is a per-bridge histogram rather than a single field, because §31's split packet is precisely a packet with non-zero counts on two bridges — which a single "which bridge" field cannot express.

And Layer 5 exists because §15 is correctness-clean. Every packet arrives; the package delivers a quarter of its capability; only a local-against-cross-cluster ratio detects it.

43. Coverage

covergroup cg_large_package @(posedge clk);
  option.per_instance = 1;
 
  // --- Topology and routing (Sections 6-11).
  cp_traffic_scope : coverpoint packet_scope {
    bins local = {0}; bins cross_cluster = {1};
  }
  cp_cluster_pair : cross_coverage_src_dst_cluster;   // every ordered pair
  cp_bridge : coverpoint sel_bridge { bins each[] = {[0:NUM_BRIDGES-1]}; }
  cp_route_depth : coverpoint decode_stages_used { bins one = {1}; bins two = {2}; }
 
  // --- Cuts and locality (Sections 12-16).
  cp_bridge_util : coverpoint bridge_utilisation_class {
    bins idle = {0}; bins moderate = {1}; bins saturated = {2};
  }
  cp_locality : coverpoint traffic_locality_ratio {
    bins mostly_local = {0}; bins balanced = {1}; bins mostly_cross = {2};  // §15
  }
 
  // --- Clock domains (Sections 17-21).
  cp_clock_ratio : coverpoint clock_ratio_class {
    bins same = {0}; bins dst_slower = {1}; bins dst_faster = {2}; bins unrelated = {3};
  }
  cp_cdc_fifo : coverpoint cdc_fifo_occupancy {
    bins empty = {0}; bins mid = {[1:FIFO_DEPTH-1]}; bins full = {FIFO_DEPTH};
  }
  cp_credit_leak_injected : coverpoint credit_return_pulse_dropped;   // Section 20
 
  // --- Reset domains (Sections 22-25).
  cp_reset_scope : coverpoint reset_event {
    bins none = {0}; bins link_recovery = {1}; bins cluster = {2};
    bins package_soft = {3}; bins por = {4};
  }
  cp_reset_with_work : coverpoint reset_with_outstanding_work;
 
  // --- Configuration (Sections 26-29).
  cp_cfg_phase : coverpoint cfg_phase_q { bins each[] = {[0:3]}; }
  cp_cfg_context : coverpoint config_commit_context {
    bins idle = {0}; bins blocked_by_inflight = {1}; bins forced_with_inflight = {2};
  }
  cp_epoch_mismatch : coverpoint cluster_epoch_mismatch_observed;      // Section 27
 
  // --- Health and containment (Sections 30-34).
  cp_bridge_state : coverpoint bridge_state_q_ut { bins each[] = {[0:3]}; }
  cp_cluster_state : coverpoint cluster_state_q_ut { bins each[] = {[0:3]}; }
  cp_isolated : coverpoint healthy_cluster_isolated;                   // Section 30
  cp_reroute : coverpoint reroute_context {
    bins none = {0}; bins before_start = {1}; bins mid_packet = {2};   // Section 31
  }
 
  // --- Congestion and deadlock (Sections 32-37).
  cp_backpressure_scope : coverpoint backpressure_reach {
    bins local_only = {0}; bins cluster = {1}; bins global = {2};      // Section 33
  }
  cp_progress_wait : coverpoint bridge_progress_age_ut {
    bins none = {0}; bins some = {[1:BRIDGE_PROG_BOUND-1]};
    bins at_bound = {BRIDGE_PROG_BOUND};
  }
  cp_deadlock_injected : coverpoint hierarchical_deadlock_injected;    // Section 35
 
  // --- Throttling (Sections 38, 39).
  cp_throttle : coverpoint throttle_state {
    bins none = {0}; bins light = {1}; bins heavy = {2};
  }
  cp_bound_rederived : coverpoint bounds_rederived_after_throttle;
 
  // --- Crosses that carry the information.
  x_scope_util    : cross cp_traffic_scope, cp_bridge_util;     // Section 13
  x_clock_cdc     : cross cp_clock_ratio, cp_cdc_fifo;          // Section 19
  x_reset_work    : cross cp_reset_scope, cp_reset_with_work;   // Section 24
  x_cfg_inflight  : cross cp_cfg_context, cp_traffic_scope;     // Section 27
  x_throttle_bound: cross cp_throttle, cp_bound_rederived;      // Section 39
endcovergroup

Eight bins worth calling out:

cp_clock_ratio.dst_slower crossed with the CDC bins. §19's failure is unreachable at a 1:1 ratio, so a single-configuration environment cannot find it.

cp_credit_leak_injected. §20 — a deliberately dropped credit-return pulse, to prove the conservation assertion catches it.

cp_epoch_mismatch. §27's window, deliberately produced by delaying one cluster's commit.

cp_reset_scope — all five, crossed with cp_reset_with_work. §24's table verified: each reset scope injected with outstanding work, to check exactly what survives.

cp_isolated. A healthy cluster made unreachable by a bridge failure — the state a merged health variable cannot express (§30).

cp_reroute.mid_packet. §31 — a reroute attempted while a packet is partly transmitted, which must be refused.

cp_backpressure_scope.global. §33 — if this bin is ever hit in a correct design, the containment is broken.

And cp_deadlock_injected with cp_progress_wait.at_bound. §35's cycle, reachable only when the progress reservation is disabled.

44. Flagship Trace 1 — Local Against Cross-Cluster

Illustrative. Two packets accepted on the same cycle: one local to cluster A, one to cluster B.

Five readings.

Cycle 2 is where the paths diverge, on is_local from the captured route — not from a live lookup (§10).

Cycle 10: cluster B uses the local_port captured at acceptance in cluster A, not its own table. §10's failure is B re-deriving it, which under §27's window sends it to the wrong port.

The cross-cluster packet is 2.4× the local one in this illustration, and that ratio is what §15's placement decision is trading against.

Cycle 5 consumes a bridge credit; cycle 18 returns it — thirteen cycles later, across the clock boundary. The round trip determines how many credits are needed to keep the bridge busy (17.2 §22's formula at bridge scope).

And the local packet was never affected by the bridge at all (§34), which is the containment property §37 asserts.

45. Flagship Trace 2 — One Bridge Fails

And §23's design, same event:

Four readings.

In the correct design, both clusters serve local traffic throughout. Only cross-cluster packets to B wait, and only their sources are backpressured (§34).

No reset is asserted at any point — a link recovery is not a reset (§25's second property).

In the wrong design, both clusters stop entirely at cycle 105 and spend an illustrative 50,000 cycles re-initialising afterwards. A 55-cycle recovery became a 50,000-cycle outage.

And the first-fault record survives in the correct design and is destroyed in the wrong one (§24), so the error that caused the recovery is diagnosable in one case and not the other.

46. Flagship Trace 3 — a Coordinated Configuration Commit

And §27's design:

Four readings.

Cycle 241 is a single cycle in which both clusters change together. §29's first property — the epochs are never unequal — holds at every cycle of the correct trace and is violated for eight cycles in the wrong one.

Cycles 219 to 240 are the drain cost: 21 cycles waiting for three cross-cluster packets. Measurable as cfg_commit_stall_q (§40), and routinely mistaken for a fabric slowdown.

The prepare phase is not synchronised and does not need to be — A prepares at 210 and B at 218. Only the commit is coordinated, which is what makes the protocol practical.

And the abort path exists (§28). If B could not prepare, A's staged table is discarded and epoch 7 remains active — with no window at all, because nothing entered a datapath.

47. Flagship Trace 4 — the Hierarchical Deadlock

§35's cycle, with the progress reservation disabled.

At cycle 20,000: every assertion in both clusters passes. One grant per output; grants only to requesters; credits bounded and conserved in every pool; every packet well-formed and correctly routed; both configurations consistent. Nothing is wrong and nothing moves.

With the progress reservation enabled (§36):

Four readings.

The bulk pool still exhausts at cycle 340. The reservation does not prevent congestion — it prevents the congestion from becoming circular.

Only a package-level progress model detects the first case. Each cluster's own model sees a stall; neither can see the cycle, because half of it is on the other die (§35).

Adding bridge credits would not fix it — it would deadlock at cycle 500 instead of 360, which is 18.2 §62's result at hierarchy scope. Deadlock is structural, not a capacity shortfall.

And the reservation must be at the bridge. A reservation inside each local fabric protects against 18.2's cycle and leaves this one entirely open, because this cycle closes at the cut (§36).

48. Debug Taxonomy

Row 5 is the best bug in this chapter. Works perfectly, then dies after a long run, with no error anywhere is a credit leak across a clock boundary — and the conservation assertion catches it on the very first lost pulse, while every bounds check stays silent to the end.

49. Debug Checklist

1. Is this packet local or cross-cluster? (§7)
2. Which cluster and which local port were captured at acceptance? (§10)
3. Under which topology epoch? (§11)
4. Do all clusters currently agree on the epoch? (§29)
5. Was a configuration commit in progress when the packet was accepted? (§28)
6. Which bridge did it take, and did every flit take the same one? (§31, §42)
7. What is that bridge's health state? (§30)
8. What are its credits, per class, and do they conserve? (§37)
9. Is the progress pool non-zero, and what is its age? (§36)
10. What is the local-against-cross-cluster byte ratio for this cluster? (§40)
11. Are the communicating agents in the same cluster? (§15, §16)
12. Which clock domains does this path cross? (§17)
13. What is the clock ratio, and is any event crossed as a pulse? (§19)
14. Do the CDC FIFOs ever report full? (§40)
15. Which reset scope was asserted, if any? (§22)
16. Did a link recovery reach a reset tree? (§25)
17. What survived that reset, against the §24 table?
18. Is the first-fault record intact? (§24)
19. Is any cluster isolated but healthy? (§30)
20. Was a reroute attempted, and was any packet mid-transmission? (§31)
21. How far did backpressure propagate? (§32, §33)
22. Is there a wait-for cycle spanning clusters? (§35)
23. Is the package throttled, and were bounds re-derived? (§38, §39)
24. Which of the five scoreboard layers diverged first? (§42)

50. Common Misconceptions

"A large package is just a larger fabric." It is a hierarchy, and the level boundary is simultaneously a bandwidth cut, a clock boundary, a reset boundary, a configuration boundary, a failure boundary and a deadlock participant. None of those exists in a flat package at any size (§4, §5).

"Flat routing scales if synthesis can compile it." A 512-destination flat decode is functionally perfect and fails timing, with a comparator depth and a table fanout that pipelining alone does not fix. The hierarchical version is two small decodes instead of one enormous one, at the cost of a cycle (§8, §9).

"All chiplets can share one reset." They cannot, because there are at least five reset scopes with different triggers and different survival requirements — and collapsing a link recovery into a package reset turns a 55-cycle event into a 50,000-cycle outage that also destroys the evidence of what caused it (§22, §23, §24).

"A UCIe recovery should pause the package." It should pause traffic destined across the recovering link and nothing else. A global scope loses an illustrative 7.7% of availability at 16 links to avoid 0.5% of unavailability, and it worsens with the link count (§23, §45).

"More links eliminate shared cuts." They move them. Cross-cluster throughput is bounded by the bridge regardless of how much bandwidth exists inside either cluster, and adding accelerators to a cluster lowers each one's achieved fraction rather than raising the total (§12, §13).

"Local and cross-cluster traffic have equal cost." Cross-cluster traffic consumes capacity at three levels — both local fabrics and the bridge — while local traffic consumes one. A placement that ignores this delivers a quarter of the package's capability with every component healthy (§14, §15).

"Backpressure should propagate globally for safety." It should propagate exactly as far as the dependency. A global scope loses an illustrative 21.6% of accept cycles at 8 destinations and 39% at 16, to avoid 3% of local congestion (§32, §33).

"A route can be recomputed after reconfiguration." It cannot, and the hierarchy gives it two chances to be recomputed — the second on a different die whose configuration may have committed at a different time (§10, §27).

"Large-package deadlock is just local NoC deadlock." The cycle spans clusters and bridges, so every local assertion in both clusters passes while nothing moves, and neither cluster's progress model can see it. The escape reservation must be at the cut, because that is where the cycle closes (§35, §36, §47).

"Peak package bandwidth is the sum of edge links." It is bounded by the narrowest cut on each path, and in a hierarchy the cuts are structural rather than incidental — every cross-cluster packet must traverse one by construction (§12).

"CDC is a physical implementation detail." A one-cycle credit-return pulse crossed with a synchroniser is lost at some clock ratios, and every lost pulse is a credit that never returns — so the link works perfectly and then stops permanently, with no error anywhere (§19, §20).

51. Understanding Check

52. Summary and What Comes Next

A large package is a hierarchy of local fabrics connected by constrained cuts, and the level boundary is simultaneously a bandwidth cut, a clock boundary, a reset boundary, a configuration boundary, a failure boundary and a deadlock participant.

Routing gains levels, and a flat table that passes every functional test fails timing at scale — the hierarchical version is two small decodes at the cost of one cycle.

The bridge bounds cross-cluster throughput regardless of local bandwidth, and cross-cluster traffic consumes capacity at three levels rather than one — so placement, not provisioning, determines the outcome.

Clock boundaries need the right bridge per crossing. A one-cycle credit-return pulse crossed with a synchroniser is lost at some clock ratios, and every lost pulse is a permanent credit leak that stalls the link after a long run with no error anywhere.

Reset has five scopes, and a link recovery is not one of them. Collapsing them turns a 55-cycle retrain into a 50,000-cycle outage and destroys the evidence of what caused it.

Configuration must commit together. Two clusters holding different topologies is a loop or a mis-route that neither can detect, and a two-phase prepare-then-commit with a drain guard is what closes the window.

Containment is per level. A saturated bridge must not stop local traffic; a bridge failure must not reset a cluster; and a healthy cluster made unreachable is a state a single health variable cannot express.

And the deadlock spans levels, so every local assertion in both clusters passes while nothing moves — which means the escape reservation must be at the cut, because that is where the cycle closes.

Modules 15 through 18 have built a complete picture of what a UCIe-based system must do: move data, preserve meaning, recover, scale, compose unlike agents, and hold together at package scale. The next module turns all of it into RTL. The first chapter asks the question every one of these ideas eventually becomes: if you had to implement one side of a UCIe link, what blocks exist, what state does each own, and what crosses each boundary?

• 19.1 — Link Architecture — the top-level RTL architecture of a UCIe link.

Browse the full path on the UCIe tutorials index.