Chiplet-Level Coherency

Chapter 16.2 built the state that makes coherence correct: ownership, sharers, transient transitions, response aggregation. Every mechanism in it assumed messages arrive, once, in a usable order.

Across a chiplet boundary none of that is free.

1. The One-Sentence Model

UCIe transports coherence information; it does not implement the coherence protocol. The semantic state of an ownership transition and the transport state carrying its messages are separate lifetimes, and every bug in this chapter is a design that merged them.

2. What This Chapter Owns

Specifically new here: the three-object hierarchy and why one semantic operation can become many messages and many more attempts; the mode question, which decides whether a coherent fabric gets UCIe's retry at all; semantic against transport identity; the delivery gate that makes a replayed probe idempotent; probe fan-out across independent links with independent congestion and retry; arrival order against ownership order, which is the most consequential architectural error in the chapter; what survives a recovery mid-transition; coherence-control starvation deadlocking a system while bulk data flows; and three scoreboard layers that each catch what the others cannot.

3. Sourcing

4. The Mode Question Comes First

Before any of this chapter's mechanisms, one architectural question decides which of them a design needs.

On that reading, a coherent fabric in Raw Mode gets a wire, not a reliable transport. Detection, retransmission and duplicate suppression are the fabric's problem — and §11's duplicate hazard exists whether or not UCIe is the thing producing the duplicate.

Three consequences, and the third is a design decision that must be made early.

Half this chapter is conditional. §11's replayed-probe case, §12's delivery gate and §19's retry interaction all assume a transport that retries. In Raw Mode there is no Adapter retry to reuse (§4), so those sections describe whatever reliability the fabric provides instead — the structure is identical, the provider is not.

And the other half is unconditional. Fan-out across links, arrival order against ownership order, recovery mid-transition, and control-traffic starvation are consequences of crossing a die boundary at all, not of any reliability mechanism. Those apply in both modes.

The design decision is which mode to use, and the UCIe 1.1 change reframes it: reusing the Adapter's CRC and retry means not building them, at the cost of the Adapter in the path. A fabric that already has its own end-to-end reliability may reasonably choose Raw Mode; one that does not should think hard before doing so.

5. Three Objects, Not One

The hierarchy the chapter rests on.

Worked, for one upgrade with three sharers on two links:

1 semantic operation
  → 1 request message      (requester → home)
  → 3 probe messages       (home → A, B, C)
  → 3 response messages    (A, B, C → home)
  → 1 completion message   (home → requester)
  = 8 coherence messages
 
each of which may be transmitted once, or more under retry:
  → 8 or more transport attempts

Three consequences, and each is a section below.

A count at one level says nothing about another. Eight messages delivered does not mean one operation completed; ten attempts does not mean ten messages. §28's three scoreboard layers exist because no single count spans the hierarchy.

Duplicates at the bottom must not propagate upward. A retried attempt is the same message, and a re-delivered message must not become a second semantic action. §11 and §12 are that boundary.

And identity is different at every level. The transaction identity that tracks the operation, the identity that names a message, and the transport identity that tracks retransmission are three different things (§8).

6. The Layered Path

Read the figure top to bottom as one semantic transaction. It is allocated on the first arrow and retired on the last, and everything between — every message, every attempt, the replay — happens inside that one lifetime.

Two arrows carry most of the chapter. The third, gated delivery ×1, is §11: the home is reached by semantic delivery, never by physical arrival. The sixth, replay — no 2nd delivery, is that gate earning its keep — the transport did exactly what 14.3 specifies, and the coherence engine correctly saw nothing. §7's table is this picture expressed as state.

7. Lifetimes, Side by Side

The column that matters is the last-but-one. Everything above the transport rows survives a retry unchanged, and everything at or below it is transport state that the reliability mechanism resolves. §13 is the bug where a semantic row is cleared by a transport event; §21 is the bug where it is cleared by a recovery.

8. Three Identities

// ILLUSTRATIVE identity mapping. These are THREE different namespaces and
// conflating any two is Section 10's bug.
typedef struct packed {
  logic                    valid;
  logic [COH_ID_W-1:0]     coh_id;        // the SEMANTIC operation
  logic [MSG_ID_W-1:0]     msg_id;        // this particular message
  logic [TR_ID_W-1:0]      transport_id;  // UCIe replay/history identity (14.3)
  logic [LINE_W-1:0]       line;
} coh_map_t;
 
coh_map_t coh_map_q [NUM_INFLIGHT];

Architecture. A mapping table, needed only where the architecture cannot derive one identity from another. If a design can carry the coherence identity inside the message and recover it on arrival, this table is unnecessary — and that is the simpler design.

Contract, and it is the section's point. The transport identity is allocated and recycled by the reliability layer on its own schedule, which has nothing to do with coherence. 14.3 §24 established that a retransmission reuses the same transport identity, while 12.4 established that identities are recycled after retirement. So a transport identity means "this object" only within its own epoch, and using it to name a coherence operation inherits a recycling policy the coherence layer did not choose.

Failure. §10.

DV. Assert that a coherence transaction's identity is stable across every transport attempt of every one of its messages — which is the property that distinguishes the levels.

9. The Coherence Envelope

// ILLUSTRATIVE internal envelope — NOT a normative UCIe, CXL or CHI message.
// No mapping of coherence messages onto UCIe flit contents is published in any
// source I could reach (Section 3), so this is internal structure only.
typedef struct packed {
  logic [COH_ID_W-1:0]  coh_id;     // which SEMANTIC operation this belongs to
  logic [LINE_W-1:0]    line_addr;  // which line
  logic [CLASS_W-1:0]   msg_class;  // request / probe / response / completion
  logic [AGENT_W-1:0]   src_agent;  // WHO sent it — Section 16 needs this
  logic                 has_data;
  logic [DATA_W-1:0]    data;
} coh_transport_t;

Architecture. Identity, line, class, source and optional data bundled together. The bundling is the design: each field answers a question the receiver must ask before acting, and a field that arrives separately from the others can be associated with the wrong message (14.1 §15's alignment argument, at the coherence layer).

Why src_agent is not optional. 16.2 §20's pending bitmap clears by agent identity. A response that does not carry its source cannot clear the right bit — and a design that infers the source from the arrival link fails the moment an agent is reachable by more than one path, or when a link carries traffic for several agents.

Why coh_id and not just line_addr. With same-line serialisation (16.2 §16) there is at most one transition per line, so the line address nearly identifies the operation. Nearly is not enough across a recovery: a message from a previous transition for the same line, arriving late, would be indistinguishable. The explicit identity plus an epoch check (§21) is what separates them.

Failure. Omitting src_agent (above). Or omitting has_data and inferring it from the message class, which couples the data path to a class encoding that may change.

DV. Drive a response whose src_agent names an agent not in the pending set and confirm nothing clears (16.2 §22).

10. Wrong RTL — Coherence State Indexed by Transport Attempt

// WRONG — the coherence transaction is allocated per arriving transport object.
always_ff @(posedge clk)
  if (transport_object_arrived) begin
    coh_txn_q[transport_id].valid     <= 1'b1;      // ← indexed by TRANSPORT id
    coh_txn_q[transport_id].line      <= arrived_line;
    coh_txn_q[transport_id].requester <= arrived_src;
    issue_probes(arrived_line);                      // ← and it acts immediately
  end

Two independent defects.

Indexing by transport identity. The transport layer allocates and recycles those identities on its own schedule. Two coherence operations can legitimately reuse one transport identity at different times, and one coherence operation can span several — so the table is keyed by something that does not identify what it is storing.

Acting on physical arrival. A retransmitted object arrives again and this code allocates again and issues probes again. One semantic operation becomes two transitions for one line — which is 16.2 §18's race, produced by the reliability mechanism working correctly.

What the duplicate does downstream:

All three are coherence corruptions with a transport cause, and none of them is a transport defect. The retry did exactly what 14.3 specifies.

11. The Semantic Delivery Gate

// ILLUSTRATIVE. A coherence message reaches the coherence engine only after
// transport has resolved validity, duplication and ordering. This is
// 14.1 Section 19's delivery gate, at the coherence layer.
assign coh_deliver = object_complete        // the whole message arrived
                  && transport_valid        // integrity checks passed (14.1)
                  && !duplicate_attempt     // history says this is not a repeat
                  && epoch_current          // not from a dead link epoch (Section 21)
                  && ordering_ok;           // semantic dependencies respected (Section 17)

Architecture. A conjunction of named terms between the transport and the coherence engine. The engine is driven by semantic delivery, never by physical arrival — that single separation prevents all three of §10's corruptions.

Contract, and the mode caveat from §4 applies here. !duplicate_attempt is satisfied by UCIe's own history machinery when the Adapter is in the path (14.1 §26). In Raw Mode there is no Adapter history to lean on (§4), so the fabric must provide the equivalent — the gate's shape is unchanged and its implementation is not.

Cycle behaviour. Combinational into the coherence engine's accept. A registered gate delivers one message before it decides not to, which is 14.1 §20's argument and has the same consequence here: the engine acts and cannot un-act.

Failure. §10's act-on-arrival. Also omitting epoch_current, which admits a message from before a recovery into a transition allocated after it (§21).

DV. For each term, force it false in isolation and confirm the coherence engine does not act. Five directed tests, and they catch the "counted but not gated" failure that no random regression finds.

12. SVA — One Semantic Action Per Message

// MANDATORY. The property that separates the transport level from the
// semantic level. Uses a verification reference ID, because the mapping from
// transport attempts to semantic messages is the testbench's knowledge.
int unsigned tb_semantic_deliveries [int];   // reference msg id -> count
 
always @(posedge clk) if (coh_deliver)
  tb_semantic_deliveries[tb_ref_id_of(delivered_msg)]++;
 
property p_semantic_delivery_at_most_once(int unsigned mid);
  @(posedge clk) disable iff (!rst_n)
    (tb_semantic_deliveries[mid] <= 1);
endproperty
 
// And the level-crossing claim: many attempts, one semantic operation.
property p_replay_does_not_reallocate;
  @(posedge clk) disable iff (!rst_n)
    (transport_object_arrived && duplicate_attempt) |=> $stable(coh_txn_alloc_count);
endproperty
a_replay_does_not_reallocate: assert property (p_replay_does_not_reallocate);
 
// One semantic operation allocates exactly one transition.
property p_one_allocation_per_operation(int unsigned oid);
  @(posedge clk) disable iff (!rst_n)
    (tb_allocations_for_operation[oid] <= 1);
endproperty

Architecture. Three properties spanning §5's levels: a message is delivered semantically at most once, a duplicate attempt allocates nothing, and an operation allocates exactly one transition.

Why the reference ID must be verification-only. The wire carries transport identity and message identity. The knowledge that attempts 1, 2 and 3 are one message — and that this message belongs to that operation — is the testbench's, because the testbench generated the traffic. Synthesising it to assert against would be building the scoreboard in RTL, where it would share the design's bugs.

DV. These need a retry that actually occurs during a live transition, which is §33's coverage cross and does not happen spontaneously.

13. Wrong RTL — Semantic Pending Cleared on Transmit

// WRONG — the pending bit is cleared when the probe is TRANSMITTED.
always_ff @(posedge clk)
  if (probe_transmitted)
    txn_q[t].pending_resp[probe_target] <= 1'b0;

Transmission is not response. The probe has left the die; the target has not seen it, has not invalidated, and has not agreed to anything.

What happens. The pending set empties as the last probe is sent, so ownership commits while every probed agent still holds its copy. That is 16.2 §23's eager grant, reached through a transport event rather than a policy decision — and across dies the window is the entire probe round trip.

Three properties.

It is the same lifetime error this curriculum has now seen four times. 14.3 §15 freed a replay entry on tx_fire; 15.2 §15 retired a latency entry on response arrival; 11.5 §14 retired semantic state at a transport event. Each time, a semantic obligation was released by a transport signal, and each time the transport signal was the clean, local, tempting one.

The tempting signal is always local and the correct one is always remote. probe_transmitted is available in this module this cycle; the response is a round trip away. That asymmetry is why the bug is written.

And it is invisible without contention. With no other agent reading the line during the window, nothing observes the early commit. The bug needs a concurrent reader at the probed agent to manifest, which is a coverage requirement rather than a lucky test.

14. Probe Fan-Out Across Links

One request can produce probes to several agents on several links, and this is where chiplet coherence differs most from monolithic.

// ILLUSTRATIVE. Two independent structures answering two different questions.
logic [NUM_AGENTS-1:0] pending_semantic_resp_q;  // WHO still owes a response
logic [NUM_LINKS-1:0]  transport_busy_q;         // WHICH LINKS are congested

They must not be merged, and the reason is that an agent can owe a response while its link is idle, and a link can be busy while every agent has already responded. A design with one structure conflates "who owes me" with "what can I send" and will either wait for links or commit on transport progress — the second being §13.

Three consequences of fan-out across independent links.

Response timing is uncorrelated. Each link has its own congestion, its own arbitration and its own retry behaviour (15.3 §32). Responses arrive out of order relative to probe dispatch order, so aggregation must be by agent identity and never by arrival position (§16).

Transient lifetime is set by the slowest link. A transition completes when the last response arrives, so one congested link extends every transition that probes an agent behind it — which couples coherence latency to an unrelated link's traffic.

And a single link failing affects a subset of transitions in a way that is hard to predict. Recovery on one link stalls the transitions probing agents behind it and leaves others untouched (§21). The blast radius is determined by the sharer distribution, not by the topology alone.

15. SVA — Responses Belong to Live Operations

// MANDATORY. Every response must name a live transaction, the right line, and
// an agent that actually owes a response.
property p_response_belongs_to_live_operation;
  @(posedge clk) disable iff (!rst_n)
    coh_response_delivered |-> (txn_q[resp_txn].valid
                             && (txn_q[resp_txn].line == resp_line)
                             && txn_q[resp_txn].pending_resp[resp_agent]);
endproperty
a_response_belongs_to_live_operation: assert property (p_response_belongs_to_live_operation);
 
// A duplicate responder cannot clear twice — 16.2 Section 21, at the
// transport boundary where the duplicate originates.
property p_duplicate_responder_no_effect;
  @(posedge clk) disable iff (!rst_n)
    (coh_response_delivered && !txn_q[resp_txn].pending_resp[resp_agent])
      |=> $stable(txn_q[resp_txn].pending_resp);
endproperty
a_duplicate_responder_no_effect: assert property (p_duplicate_responder_no_effect);
 
// Ownership commits only when the semantic pending set is empty — never on
// transport progress (Section 13).
property p_commit_on_semantic_not_transport;
  @(posedge clk) disable iff (!rst_n)
    ownership_commit |-> (txn_q[commit_txn].pending_resp == '0);
endproperty
a_commit_on_semantic_not_transport: assert property (p_commit_on_semantic_not_transport);

Why the first property checks three things at once. A response can be wrong in three independent ways — dead transaction, wrong line, unexpected agent — and across links all three are reachable: a late response after a retirement, a response misrouted between concurrent transitions, and a duplicate from a retry. On a single link with one responder, none of the three is reachable, which is why these properties belong here.

DV. Inject each of the three violations separately. The third — a response from an agent already cleared — is §10's replay case and is the one that occurs naturally.

16. Arrival Order Must Not Decide Ownership Order

The most consequential architectural error in the chapter.

// WRONG — whichever request's transport object arrives first wins the line.
if (transport_object_arrived && !line_busy[arrived_line])
  grant_ownership(arrived_src, arrived_line);

Why it looks reasonable. Something must break the tie between two same-line requests, and arrival order is a total order that is free.

Why it is catastrophic. Arrival order across chiplets is determined by:

Cache-line ownership would then be decided by transport congestion and by whether a random bit error occurred. The same program, run twice, resolves the same race differently — and one of those runs may be the one where a bit flipped.

Three consequences.

It makes coherence nondeterministic in a way that has nothing to do with the program. Two agents racing for a line is a legitimate program behaviour with a legitimate set of outcomes; letting a retry decide which one occurs adds a source of variation the architecture never sanctioned.

It is not reproducible, which makes it undebuggable. A failure that depends on a bit error at a particular instant cannot be re-run.

And it interacts with §19. If a request is retried, its second attempt arrives later than it would have — so a transport retry can change which agent wins ownership, and the retry mechanism exists precisely to be semantically invisible.

The correct structure is that the home decides, using its own arbitration over requests that have been semantically delivered:

// ILLUSTRATIVE. The home arbitrates among delivered requests. Arrival order
// is an INPUT to that arbitration at most, never the decision itself.
if (!dir_q[line].busy) begin
  winner = home_arbitrate(delivered_requests_for_line);   // documented policy
  allocate_transition(winner, line);
end else begin
  queue_or_defer(delivered_requests_for_line);            // Section 18
end

And the arbitration policy must be documented, because it is now an architectural property rather than an accident. Round-robin, requester priority, or oldest-first are all defensible; "whichever packet arrived first" is not, because it is not a policy at all.

17. Ordering Dependencies, Generically

Transport arbitration must not violate a semantic dependency the coherence protocol requires.

The dependency every protocol has, and the clearest example: two ownership transitions for the same line are ordered with respect to each other. Whatever order the protocol says they resolve in, the transport must not produce a different one — which §16 is the failure of.

Three practical implications.

Same-line messages have a dependency; different-line messages generally do not. So a transport may reorder freely across lines and must respect order within one — which is what makes 16.2 §16's line-granular serialisation the right granularity.

A response and the request it answers are ordered by causality, not by the transport. A design that lets a response overtake its own request's delivery has a structural error, and it is reachable when they travel different paths.

And the ordering must be enforced where the semantics live. Enforcing it in the transport — by forbidding reordering entirely — is correct and expensive: it serialises unrelated traffic to preserve a dependency that applies to a fraction of it. Enforcing it at the home, over delivered messages, costs nothing for unrelated lines.

18. Home Arbitration and Deferral

// ILLUSTRATIVE per-line arbitration at the home. The state a real design needs
// is more than "is the line busy" — deferred requests must be REMEMBERED.
typedef struct packed {
  logic                     valid;
  logic [AGENT_W-1:0]       requester;
  coh_state_t               target_state;
  logic [COH_ID_W-1:0]      coh_id;
} deferred_req_t;
 
deferred_req_t deferred_q [NUM_LINES][MAX_DEFERRED];
logic [$clog2(MAX_DEFERRED+1)-1:0] deferred_count_q [NUM_LINES];
 
always_comb begin
  if (!dir_q[line].busy && (deferred_count_q[line] != '0))
    next_winner = arbitrate_deferred(line);        // documented policy (Section 16)
  else
    next_winner = NO_WINNER;
end

Architecture. A small per-line queue of requests that arrived while the line was transitioning. Deferral rather than rejection, because rejecting forces the requester to retry and turns a local queueing question into a network-level retry storm.

State. Per line, per deferral. MAX_DEFERRED bounds it, and the bound matters: with N agents at most N−1 can be deferred for one line, so sizing it at NUM_AGENTS − 1 makes overflow structurally impossible.

Cycle behaviour. A winner is selected only when the line becomes free. The selection must be a documented policy (§16) and must guarantee that a deferred request is eventually selected (§27).

Contract. A deferred request is a semantic obligation the home has accepted. It must not be dropped on a recovery (§21), must not be lost when the table is written, and must be counted in any liveness argument.

Failure. Dropping deferred requests when the line becomes free and only re-examining the newest. Or under-sizing MAX_DEFERRED and silently discarding on overflow — which loses a request that a requester is waiting on indefinitely.

DV. Drive NUM_AGENTS − 1 concurrent same-line requests and confirm all are eventually served. Then confirm none is lost across a recovery.

19. Coherence and Transport Retry

Where 14.3 meets this chapter, and the distinction is between two kinds of duplicate.

The canonical case, with a probe:

1. Home sends a probe to agent B.
2. B receives it, invalidates the line, sends a response.
3. B's response is lost — or its transport acknowledgement is lost.
4. The transport replays. B receives the SAME probe again.
5. B must NOT treat this as a new invalidation request that arrives when the
   line is already invalid — and it MUST still respond, or the home waits forever.

Step 5 is the whole difficulty, and it has two halves that pull in opposite directions.

The invalidation must not be re-executed as a new semantic action. The line is already invalid; a naive handler might treat the probe as spurious, or might respond in a way that assumes the line was still present.

And the response must be re-sent. 14.3 §22's rule applies exactly: suppress the effect, not the reply. A probed agent that recognises the duplicate and stays silent leaves the home's pending bit set forever — a livelock built from two individually-correct behaviours.

Idempotence is the property that makes this tractable: applying the same probe twice must reach the same state and produce the same response. 16.2 §20's guarded bitmap clear has that property by construction, and a design whose probe handling has it too is robust to replay without special cases.

20. Lost Response, and Why It Is Harder Than a Lost Request

A lost request is straightforward: the home never saw it, nothing changed, and a retransmission is a first delivery.

A lost response is not, because the responder's state has already changed.

1. Home probes B.
2. B invalidates its copy — its state is now I.
3. B sends its response.
4. The response is lost in transport.
5. The home times out, or a recovery occurs.
6. The home cannot assume B is still a sharer.  B is NOT.

The distributed state has already moved, and the home's model of it is stale in a specific direction: the home believes B holds a copy that B has already given up.

Three consequences.

Re-probing B is safe if probes are idempotent (§19) — B is already invalid, re-invalidates to no effect, and responds. That is the argument for idempotence being a design requirement rather than a nicety.

Rolling back is not safe. The home cannot restore its "B is a sharer" belief and proceed as if nothing happened, because B will not behave like a sharer — it has no copy. 14.2 §22's timeout ambiguity applies: a timeout tells you no response arrived, not that the remote did nothing.

And the pending bit must stay set. The home's bitmap still shows B pending, which is correct — it has not received a response. The resolution is to re-probe and collect a response, not to assume one.

21. Recovery During an Ownership Transition

The hardest case in the chapter, and the one that composes the most modules.

Setup: A requests write ownership of line L. Sharers are B and C, on
       different links. B has responded; C has not. C's link enters recovery.

The rule, and it is 14.2 §4's rule specialised:

A link recovery re-establishes a link. It does not un-invalidate B, it does not release A's request, and it does not change what C owes. Every semantic row above survives untouched, and only the transport rows are rebuilt.

Three failures this prevents.

Clearing the transition and returning L to a stable state. The home would then believe L is shared by B and C — but B has already invalidated, so the home's model would be wrong in a way no subsequent operation corrects.

Releasing A's transaction and reporting failure. A would retry, and the retry would be a new semantic operation for a line that already has a transition's worth of work done. §18's deferral or 16.2 §16's same-line rule would then block it against its own predecessor.

And clearing the pending bitmap. Committing with C unprobed is 16.2 §23's eager grant with a recovery as its cause.

What the recovery does change is the epoch (§11's epoch_current). A response from C generated before the recovery, arriving after it, must be validated against the current epoch — otherwise a stale response could clear a pending bit for a transition that has since been re-driven.

22. SVA — Semantic State Survives Transport Recovery

// MANDATORY. The property that makes Section 21's failures impossible.
property p_transition_survives_recovery;
  @(posedge clk) disable iff (!rst_n)
    link_recovery_event |=> ($stable(txn_q[TXN_UT].valid)
                          && $stable(txn_q[TXN_UT].line)
                          && $stable(txn_q[TXN_UT].requester)
                          && $stable(txn_q[TXN_UT].pending_resp));
endproperty
a_transition_survives_recovery: assert property (p_transition_survives_recovery);
 
// The line's transient state is not returned to a stable state by a recovery.
property p_line_stays_transient_across_recovery;
  @(posedge clk) disable iff (!rst_n)
    (link_recovery_event && $past(is_transient(line_state)))
      |=> is_transient(line_state);
endproperty
a_line_stays_transient_across_recovery: assert property (p_line_stays_transient_across_recovery);
 
// Deferred requests are not dropped by a recovery.
property p_deferred_survives_recovery;
  @(posedge clk) disable iff (!rst_n)
    link_recovery_event |=> $stable(deferred_count_q[LINE_UT]);
endproperty
a_deferred_survives_recovery: assert property (p_deferred_survives_recovery);
 
// A message from a previous epoch cannot clear a pending bit.
property p_stale_epoch_response_rejected;
  @(posedge clk) disable iff (!rst_n)
    (coh_response_arrived && (resp_epoch != current_epoch))
      |=> $stable(txn_q[resp_txn].pending_resp);
endproperty
a_stale_epoch_response_rejected: assert property (p_stale_epoch_response_rejected);

Architecture. Four properties: the transaction survives, the line stays transient, deferred work survives, and stale-epoch messages are rejected.

Why the fourth is not redundant with the first three. The first three protect against a recovery destroying state. The fourth protects against a recovery admitting state that should have died with the previous epoch — the opposite direction, and it is the one that produces a premature commit rather than a hang.

DV. All four need a recovery while a transition is mid-flight with at least one response outstanding, which never occurs spontaneously and must be injected.

23. Coherence-Control Starvation

A deadlock that has nothing to do with coherence logic and everything to do with sharing a link.

Coherence probes and responses share transport resources with bulk data. If the arbitration policy always favours data:

1. A large streaming workload saturates the link.
2. Probe responses queue behind bulk data and are never scheduled.
3. Ownership transitions never complete.
4. Agents waiting on those transitions cannot proceed.
5. Eventually the streaming workload itself needs a line whose ownership
   is stuck in step 3.
6. → the whole system deadlocks, having been "optimised" for data throughput.

Three properties.

Steps 1 and 2 look like correct prioritisation. Bulk data is the workload; probe responses are overhead. A strict-priority arbiter favouring data is a defensible-sounding decision that is fatal, and it is 13.4 §16's strict-priority starvation with the starved class being the one that unblocks everything else.

The deadlock is self-inflicted and delayed. For a while the system runs beautifully — high data throughput, coherence quietly falling behind. The failure arrives when the data path itself needs a coherence transition, which may be far into a run.

And the diagnosis is misleading. The symptom is a hang in the middle of a high-throughput workload, with the link busy and no errors. 15.5 §19's classifier would show high useful cycles right up to the hang.

The generic remedy is to make progress-critical traffic non-starvable:

24. Wrong Policy — Strict Data Priority

// WRONG at system scope. Coherence control can be starved indefinitely.
assign tx_select = data_pending ? TX_DATA : TX_COHERENCE;

Under sustained data load, data_pending is permanently true and TX_COHERENCE is never selected.

And unlike 13.4 §16's strict-priority arbiter, this one does not self-resolve. There, a full buffer eventually stopped new admissions and let the starved class through. Here the starved message is what would unblock the data path, so the data path keeps generating work and never yields — until it blocks on the transition it starved.

The minimal fix is not a full QoS scheme. A single reserved slot, or a rule that a control message waiting longer than a bound wins, is sufficient to break the cycle. The requirement is a bound, not fairness.

25. Per-Line Liveness

// LIVENESS, bounded, with assumptions stated (15.2 Section 36's discipline).
//
//   A1: probed agents eventually deliver a response, given transport progress
//   A2: links eventually recover or fail explicitly (14.2 Section 30)
//   A3: control traffic is not starved indefinitely (Section 23)
assume property (@(posedge clk) disable iff (!rst_n)
  coherence_control_pending |-> ##[1:CONTROL_SERVICE_BOUND] coherence_control_sent);
 
property p_transition_resolves_or_fails;
  @(posedge clk) disable iff (!rst_n)
    txn_alloc_fire |-> ##[1:TXN_BOUND] (ownership_commit || txn_explicit_fail);
endproperty
a_transition_resolves_or_fails: assert property (p_transition_resolves_or_fails);
 
// Every deferred request eventually becomes an active transition (Section 18).
property p_deferred_eventually_served;
  @(posedge clk) disable iff (!rst_n)
    deferred_enqueued |-> ##[1:DEFER_BOUND] deferred_dequeued;
endproperty
a_deferred_eventually_served: assert property (p_deferred_eventually_served);

Architecture. Two bounded liveness properties with three assumptions, the third of which is §23's requirement stated as an obligation on the environment.

Why A3 must be an assumption rather than a hope. Without it, p_transition_resolves_or_fails fails on a design that is correct and merely starved. Making it explicit turns "this hangs" into "this hangs because control traffic was starved", which is a diagnosis rather than a symptom.

And the consequent admits explicit failure, for the same reason 14.2 §30 and 16.2 §29 do: the requirement is termination, not success.

DV. Deliberately starve control traffic with the assumption disabled and confirm the design hangs in the predicted way — which validates that the assumption is describing the real dependency.

26. Deadlock Across Links

§23's starvation is one cycle; here is one that needs no unfairness at all.

1. Home H awaits a probe response from agent B, across link 1.
2. B has the response but link 1's outbound path has no credit (13.1)
   — or no replay space (14.3 Section 28).
3. Credit returns only when H drains its inbound queue on link 1.
4. H's inbound head is a coherence request for the SAME line, from agent C
   on link 2.
5. H cannot process it: the line is busy with the transition at step 1
   (16.2 Section 16).
6. → cycle.

Every participant is correct and nothing progresses.

Two additions over 16.2 §28's version.

The cycle now crosses two links, so no single link's flow control can see it. Adding credit on link 1 does not help — the blockage is at H's processing, not at B's transmission.

And §18's deferral is part of the fix. If H can defer C's request rather than blocking its inbound queue on it, step 5 disappears: the head is consumed, the queue drains, credit returns, B's response arrives, the transition completes, and C's deferred request is then served. Deferral is not merely a convenience — it breaks a dependency cycle, which is the strongest argument for building it.

27. Three Scoreboard Layers

// Verification-only. THREE models, because no one of them can detect the
// others' failures (Section 28 explains why).
class chiplet_coherence_scoreboard;
 
  // ---- Layer 1: TRANSPORT. What physically crossed the boundary.
  typedef struct {
    int  attempts;              // physical transmissions
    bit  retired;               // replay entry released
    int  transport_id;
  } transport_obj_t;
  transport_obj_t transport[int];        // keyed by transport object
 
  // ---- Layer 2: COHERENCE MESSAGE / TRANSACTION.
  typedef struct {
    int                    requester;
    bit [LINE_W-1:0]       line;
    bit [NUM_AGENTS-1:0]   pending;
    int                    semantic_deliveries;   // per message: MUST be <= 1
    int                    allocations;           // per operation: MUST be 1
    bit                    committed;
  } coh_txn_model_t;
  coh_txn_model_t txn[int];              // keyed by SEMANTIC operation id
 
  // ---- Layer 3: LINE SEMANTICS. What the system actually believes.
  typedef struct {
    bit [DATA_W-1:0]       latest_value;
    int                    owner;
    bit [NUM_AGENTS-1:0]   sharers;
    bit                    dirty;
  } line_model_t;
  line_model_t line[bit [LINE_W-1:0]];
 
  // ---- Cross-layer conservation (Section 29).
  function void check_conservation(int oid);
    if (txn[oid].allocations != 1)
      $error("OPERATION %0d allocated %0d times (must be 1)", oid, txn[oid].allocations);
    foreach (msg_of_operation[oid][m]) begin
      if (transport[m].attempts < 1)
        $error("MESSAGE %0d never attempted", m);
      if (txn[oid].semantic_deliveries > 1)
        $error("MESSAGE %0d delivered semantically %0d times (must be <= 1)",
               m, txn[oid].semantic_deliveries);
    end
  endfunction
 
  // ---- Layer 3 check: no stale read, using OBSERVED agent state.
  function void check_no_stale(bit [LINE_W-1:0] l, int agent, bit [DATA_W-1:0] got);
    if (got !== line[l].latest_value)
      $error("STALE READ line %0h agent %0d: got %0h latest %0h owner %0d",
             l, agent, got, line[l].latest_value, line[l].owner);
  endfunction
 
  // ---- Cross-layer: transport perfection does NOT imply coherence (Section 28).
  function void check_transport_vs_coherence();
    bit transport_clean = 1;
    foreach (transport[t]) if (!transport[t].retired) transport_clean = 0;
    if (transport_clean && coherence_violation_seen)
      $display("NOTE: transport layer clean while coherence violated — "
             + "this is expected and is why three layers exist");
  endfunction
 
endclass

Architecture. Three models keyed by three different identities — transport object, semantic operation, and line. That is §5's hierarchy as verification structure.

Layer 1 catches replay-buffer leaks, unretired objects, and attempts that never happened.

Layer 2 catches double allocation (§10), semantic delivery more than once (§12), and premature commit (§15).

Layer 3 catches stale reads, wrong ownership, and lost dirty data — and only Layer 3 can, because layers 1 and 2 can both be perfect while the value is wrong (16.2 §31).

28. Why One Layer Cannot Substitute for Another

Read the "blind" cells in the top-left triangle. A transport model reports success for every coherence failure below the first row. That is 16.2 §31's result, generalised: transport correctness is a weaker claim than coherence correctness, and no amount of the former implies the latter.

And read the bottom-left cells. Layer 3 catches the consequences of layer 2's failures, but often much later and with a symptom far from the cause. Layer 2 exists so a double allocation is reported when it happens rather than when its effect surfaces.

Three layers, three identities, three failure classes. A verification plan with one model is verifying one third of the system and reporting on all of it.

29. The Conservation Model

For one semantic coherence operation, the invariants that tie the layers together:

semantic_allocations(operation)          == 1
messages(operation)                       >= 1
attempts(message)                         >= 1
semantic_deliveries(message)              <= 1
pending_responses(operation)
    == required_responses − unique_responses_accepted
ownership_commits(operation)              <= 1

Assumptions, stated because the invariants are only meaningful under them. One operation resolves at most one line transition; required responses are fixed at allocation from the directory's sharer set (16.2 §20); and a response is unique per agent per operation.

The most useful of the six is the fourth, because it is the one every mechanism in this chapter exists to preserve: many attempts, at most one semantic delivery. §12's property is its executable form, and §11's gate is its implementation.

And the fifth is what §15's third property checks. pending == required − unique_accepted fails immediately if a duplicate response was counted, which is 16.2 §21's bug detected by arithmetic rather than by consequence.

30. Error Injection

What a chiplet coherence environment must inject, grouped by the mechanism each targets:

Three notes on using the list.

Rows 1 to 3 are the same physical event at three points, and they produce three different coherence corruptions. Injecting retry only on requests tests a third of the boundary.

Row 13 is the one most environments never build, because it requires controlling relative arrival order across links. It is the only test of §16, and §16 is the chapter's most consequential architectural error.

And row 8 is the hardest and the most valuable. It requires a recovery to be triggered at a precise point in a multi-agent transition. Every property in §22 depends on it.

31. Flagship Trace — Ownership Across a Boundary

Illustrative. A (requester) and B (sharer) are on different dies; the home is on a third. A requests write ownership of L.

Six readings, and the lifetimes are the point.

Cycle 1 to 15: one semantic transaction spans fourteen cycles and outlives every transport object inside it. The replay entry allocated at cycle 2 was retired at cycle 8 — seven cycles before the operation completed. §13's bug is retiring the semantic state at cycle 8 too.

Cycle 6 is the semantic delivery, one cycle after physical arrival. That gap is §11's gate doing its work: integrity checked, duplicate checked, epoch checked, and only then does the coherence engine see it.

Cycle 8: the replay entry retires while the transition continues. Transport's obligation is discharged; coherence's is not. Two lifetimes, correctly decoupled.

Cycle 10 to 12: B invalidates at 10 and its bit clears at 12. Two cycles of die crossing. A design clearing on probe transmission would have cleared at cycle 7 — three cycles before B even invalidated (§13).

Cycle 13: ownership commits only when pending is empty.

And cycle 15 retires the semantic transaction — after the completion is consumed, not when it was sent. The same retirement-point discipline as 15.2 §15.

32. Failure Trace — Replayed Probe

The same scenario, with B's response lost and the probe replayed.

Three readings.

Cycle 22: the effect is suppressed. B does not invalidate twice — the line is already invalid and a second invalidation would be a no-op anyway, but a handler that assumed the line was present could mis-respond.

Cycle 23: the reply is not suppressed, and this is the half that is easy to get wrong. A B that recognises the duplicate and stays silent leaves the pending bit set forever — §19's livelock, built from a correct duplicate-detection and an incorrect silence.

And the semantic action count for B is exactly one across two probe deliveries. That is the conservation invariant of §29 holding under retry, which is the whole point of the level separation.

33. Coverage

covergroup cg_chiplet_coherence @(posedge clk);
  option.per_instance = 1;
 
  // --- Fan-out dimension. The chiplet-specific axis.
  cp_probe_targets : coverpoint num_probe_targets {
    bins one   = {1};
    bins two   = {2};
    bins many  = {[3:$]};
  }
  cp_target_links : coverpoint num_distinct_links_probed {
    bins one_link  = {1};
    bins two_links = {2};
    bins many_links= {[3:$]};        // responses genuinely uncorrelated
  }
 
  // --- Transport interaction (Sections 10, 19).
  cp_retry_point : coverpoint retry_occurred_on {
    bins none     = {0};
    bins request  = {1};
    bins probe    = {2};
    bins response = {3};             // three DIFFERENT corruptions
  }
  cp_duplicate_delivered : coverpoint duplicate_reached_transport;
 
  // --- Recovery (Section 21).
  cp_recovery_point : coverpoint recovery_during {
    bins none            = {0};
    bins request_inflight= {1};
    bins probe_inflight  = {2};
    bins partial_responses = {3};    // THE case — some responded, some not
    bins dirty_owner_probed = {4};
  }
  cp_stale_epoch_msg : coverpoint stale_epoch_message_arrived;
 
  // --- Ordering (Section 16).
  cp_arrival_vs_order : coverpoint same_line_arrival_order {
    bins matches_intent   = {0};
    bins reversed         = {1};     // transport order != request order
  }
 
  // --- Deferral and liveness (Sections 18, 25).
  cp_deferred_count : coverpoint deferred_requests_for_line {
    bins none = {0}; bins one = {1}; bins many = {[2:$]};
  }
  cp_control_starved : coverpoint coherence_control_starved;
 
  // --- Mode (Section 4).
  cp_mode : coverpoint transport_mode {
    bins raw     = {0};              // no UCIe retry — fabric provides it
    bins adapter = {1};
  }
 
  // --- Crosses that carry the information.
  x_retry_targets  : cross cp_retry_point, cp_probe_targets;   // Sections 10, 19
  x_recovery_links : cross cp_recovery_point, cp_target_links; // Section 21
  x_order_targets  : cross cp_arrival_vs_order, cp_probe_targets;
  x_mode_retry     : cross cp_mode, cp_retry_point;            // Section 4
endcovergroup

Six bins whose value is being non-zero:

cp_retry_point — all three of request, probe and response. They produce three different corruptions (§30), and injecting retry only on requests tests a third of the boundary.

cp_recovery_point.partial_responses. §21's case: a recovery with some agents responded and some not. Every property in §22 depends on reaching it, and it never occurs spontaneously.

cp_arrival_vs_order.reversed. §16's test — same-line requests whose transport arrival order is the reverse of their issue order. This requires controlling cross-link timing and is the bin most environments lack.

cp_target_links.many_links. Probes to agents behind three or more different links, where response timing is genuinely uncorrelated (§14).

cp_control_starved. §23's deadlock, which must be induced.

And cp_mode — both bins. §4's architectural fork. A design that supports both modes and tests one has verified half its configurations, and the Raw Mode half is the one with no Adapter reliability underneath it.

34. Debug Taxonomy

The last row is worth its own note. A design tested only through the Adapter inherits CRC and retry; the same design in Raw Mode has neither, and every mechanism that assumed them silently loses its foundation.

35. Debug Checklist

1. What semantic coherence operation is in question? And which line and requester?
2. Which home is coordinating it?
3. When was the semantic transaction allocated — and exactly once? (§12).
4. Which coherence messages does it comprise? Request, probes, responses, completion (§5).
5. How many transport attempts did each message take? (§29).
6. How many semantic deliveries did each message produce? Must be at most one.
7. Which probes were issued, to which agents, over which links? (§14).
8. Which responses are still pending, by agent identity?
9. Did any agent respond twice — and was it a physical or a semantic duplicate? (§19).
10. Is dirty data involved, and where is the obligation? (16.2 §25).
11. Did a transport retry occur, and at which of the three points? (§30).
12. Did a link recovery occur mid-transition, and did every semantic row survive? (§21, §22).
13. Was any message from a previous epoch accepted? (§22).
14. Was control traffic starved? (§23).
15. Did transport arrival order determine ownership order anywhere? (§16).
16. When did ownership commit, and was the pending set empty? (§15).
17. When did the semantic transaction retire — at completion consumption or earlier? (§31).
18. Which transport mode is this? Raw Mode has no Adapter retry beneath it (§4).
19. Do all three scoreboard layers agree? (§28).

36. Common Misconceptions

"UCIe implements coherency." UCIe carries CXL, and carries CHI and other coherency protocols as Streaming Protocols. The coherence semantics belong to those protocols; UCIe provides a transport whose reliability guarantees depend on the mode and the revision (§3, §4).

"A coherence operation is one packet." One operation becomes a request, several probes, several responses and a completion — and each of those may take several transport attempts. A count at one level says nothing about another (§5).

"Transport sequence ID and coherence transaction ID are the same." They are different namespaces, allocated and recycled by different layers on different schedules. Indexing coherence state by a transport identity inherits a recycling policy the coherence layer never chose (§8, §10).

"A replayed coherence request should allocate again." A replay is the same message delivered twice. Allocating again produces two transitions for one line, and both may commit ownership — a coherence corruption whose cause is the reliability mechanism working correctly (§10).

"First packet arrival should win ownership." Arrival order across chiplets depends on congestion, arbitration, queue history, routing, and whether a bit error triggered a retry. Letting it decide ownership makes coherence nondeterministic for reasons the architecture never sanctioned (§16).

"Probe transmission means probe completion." The probe has left the die; the target has not seen it, invalidated, or agreed. Clearing a pending bit on transmit commits ownership while every probed agent still holds its copy (§13).

"One counter is enough to track responses." Duplicates arrive from transport retries, and a counter cannot tell a new responder from a repeat — so it reaches zero with an agent still holding a copy (16.2 §21, §19).

"Recovery can restore the previous stable cache state." Agents that have already invalidated will not behave like sharers again. A timeout says no response arrived, not that the remote did nothing, and the correct action is to re-probe rather than to roll back (§20, §21).

"Bulk-data fairness automatically guarantees coherence progress." A strict-priority arbiter favouring data starves the very messages that would unblock the data path, and the resulting deadlock arrives long after the system appeared to be running well (§23, §24).

"Clean CRC and clean replay prove coherent correctness." Every flit can arrive once with clean CRC, every replay entry can retire, and two agents can still hold conflicting permissions. Transport correctness is a strictly weaker claim (§28).

"One scoreboard can verify transport and coherence." Three identities, three failure classes, and a table of blind spots where each layer cannot see the others' failures. A single model verifies one third and reports on all of it (§27, §28).

"A duplicate probe should be answered with silence." Suppress the effect, not the reply. An agent that recognises a duplicate and stays silent leaves the home's pending bit set forever — a livelock from two individually-correct behaviours (§19, §32).

37. Understanding Check

38. Summary and What Comes Next

UCIe transports coherence information; it does not implement the coherence protocol. Semantic state and transport state are separate lifetimes, and every bug here is a design that merged them.

The mode question comes first. Official material states CHI and SMP coherency protocols were supported only in Raw Mode in UCIe 1.0 — where UCIe provides no CRC and no retry — and that UCIe 1.1 lets Streaming Protocols use the Adapter and reuse them. Half this chapter's mechanisms are conditional on which of those a design chose.

Three objects, not one. One semantic operation becomes several messages and more attempts. Many attempts, at most one semantic delivery is the invariant the delivery gate implements and the conservation model checks.

Identity differs at every level, and indexing coherence state by a transport identity inherits a recycling policy the coherence layer never chose.

Transmission is not response. Clearing a pending bit when a probe is sent commits ownership while every probed agent still holds its copy — the fourth appearance in this curriculum of a semantic obligation released by a transport signal, and every time the transport signal was the local, tempting one.

Arrival order must not decide ownership order. Across chiplets that order depends on congestion, arbitration, routing and whether a bit flipped — so the home must arbitrate over delivered requests using a documented policy, and deferral rather than rejection is what also breaks the two-link deadlock cycle.

A recovery rebuilds a link and changes nothing semantic. The transaction, the transient state, the pending bitmap, the collected responses, the dirty obligation and the deferred requests all survive — and a message from the dead epoch must be rejected.

And control traffic must not be starvable. A strict-priority arbiter favouring bulk data starves the messages that would unblock the data path, and the deadlock arrives long after the system appeared to be performing beautifully.

Three scoreboard layers, three identities, three failure classes — and transport correctness is a strictly weaker claim than coherence correctness, so a single model verifies one third of the system and reports on all of it.

We now understand the generic state and transport problems of coherent chiplets. The next chapter grounds that reasoning in a concrete coherent-fabric relationship: where an Arm CHI domain meets the UCIe boundary and which responsibilities stay with CHI versus UCIe.

• 16.4 — UCIe and CHI — where the CHI fabric meets UCIe.

Browse the full path on the UCIe tutorials index.