Cache Coherency in Chiplets

Chapter 16.1 ended with a system whose addressing was perfect and whose answer was wrong: memory held 0, a cache held 1, and the reader got 0.

This chapter builds the state that makes that impossible.

1. The One-Sentence Model

Coherence is distributed ownership plus distributed invalidation of obsolete permissions. For every line the system must be able to answer who may read it, who may write it, who holds a copy, who owns the newest data, and whether any of those is currently changing.

The last clause is the one that produces most of the hardware. Ownership does not change instantaneously across dies, and the interval during which it is changing is where every race in this chapter lives.

2. What This Chapter Owns

Chapter 11.3 — Cache Coherency Over CXL already built a coherence state machine in depth, and this chapter is not that chapter rewritten.

The N-agent case is not "11.3 with a bigger number". Three mechanisms simply do not exist with one device: a sharer vector, response aggregation, and same-line arbitration among peers where no participant is privileged. Each of those is a structure with its own failure modes, and §21's counter-versus-bitmap bug is meaningless with a single responder.

Specifically new here: the three kinds of state and why collapsing them breaks; the directory entry with owner and sharer vector; the owner-against-sharers invariant; response bitmaps and the duplicate-responder bug a counter cannot catch; same-line serialisation among N peers; eager grant; the probe-against-local race and the double-assignment bug; dirty-owner reads; cross-die transient lifetime, which is the chiplet-specific insight; and a line-level reference model that detects corruption a transport scoreboard reports as perfect.

3. Sourcing

4. Three Kinds of State

Mandatory, and collapsing any two of them is the source of several bugs below.

Why three and not two.

Stable and transient must be distinguishable because a line in transition is not usable even though it physically holds data (§13). A design with only stable states must either grant permission too early or block on something it cannot express.

Transaction state must be separate from line state because they have different cardinalities and different lifetimes. One line has one state; a transition may involve several outstanding probes to several agents, and that set does not fit in a line's metadata. §17 is that table, and §20's bitmap lives in it.

A design that keeps only line state cannot represent "waiting for three of five agents to acknowledge". A design that keeps only transaction state cannot answer "may this local read proceed?" in one cycle. Both questions are asked constantly.

5. The Directory

Something must know, per line, enough to coordinate. Architecturally there are several ways to provide it:

Not a UCIe decision. No official source I could reach states that UCIe mandates or prefers any of these (§3) — UCIe carries the protocol, and the protocol chooses. What this chapter assumes is only that some component can answer the five questions of §1, and it uses a home-based directory as the illustrative vehicle.

The chiplet-specific pressure is on the first row. Snoop broadcast is attractive on a monolithic die where a snoop is cheap. Across dies every snoop is a UCIe traversal with serialisation, queueing and possible retry (16.3 §14), so broadcast cost scales with both agent count and die crossings. That is a real argument for directories in chiplet systems and it is a consequence of the transport, not of the protocol.

6. The Coherence Structure

Two structural properties.

Probes and requests are separate flows in opposite directions, and both cross the same UCIe boundary. That shared boundary is where 16.3 §26's starvation deadlock comes from — if bulk requests can crowd out probe responses, ownership transitions never complete.

And the transaction table hangs off the directory, not off the line. One line, one directory entry; one transition, one transaction entry with a bitmap sized by agent count. §4's separation, drawn.

7. Illustrative States

// ILLUSTRATIVE coherence states — NOT CHI, NOT CXL, and not claimed to
// correspond to either protocol's encodings or semantics (Section 3).
// Neutral names are used deliberately so the MECHANISM can be taught without
// asserting a protocol's rules.
typedef enum logic [3:0] {
  // --- stable: permissions are exactly what the name says
  COH_I        = 4'd0,   // no copy, no permission
  COH_S        = 4'd1,   // shared: may read; others may also hold S
  COH_E        = 4'd2,   // exclusive clean: may read; sole copy; may upgrade silently
  COH_M        = 4'd3,   // modified: may read and write; dirty; sole copy
  // --- transient: a transition is IN PROGRESS. Permissions are NEITHER
  //     the source state's nor the destination's (Section 12).
  COH_I_TO_S   = 4'd4,
  COH_I_TO_M   = 4'd5,
  COH_S_TO_M   = 4'd6,
  COH_M_TO_I   = 4'd7,
  COH_S_TO_I   = 4'd8
} coh_state_t;

Architecture. Four stable states and five transient ones. The stable four are a conventional teaching set; the transient five are the chapter's substance, and §12 is why they are mandatory rather than an optimisation.

State. Per cache line, per agent. Note there is no state meaning "I think I might have permission" — every state is a definite statement about what this agent may do, which is what makes §10's permission functions total.

Contract. Every consumer of the line's state must treat a transient value as denying both read and write unless the design explicitly defines otherwise. §11's functions encode that in one place rather than at every use.

Failure. Treating a transient state as equivalent to its source state — which grants permission that is being given away — or as equivalent to its destination — which is §23's eager grant.

8. Cache-Line Metadata

// ILLUSTRATIVE per-line metadata at a caching agent.
typedef struct packed {
  logic                valid;         // the tag is meaningful
  logic [TAG_W-1:0]    tag;
  coh_state_t          state;         // Section 7
  logic                dirty;         // this agent holds the only newest copy
  logic                probe_pending; // a probe is being processed for this line
} line_meta_t;
 
line_meta_t line_q [NUM_SETS][NUM_WAYS];

Architecture. Structural presence (valid, tag) is kept separate from semantic permission (state). That separation is the point, and §9 is what happens without it.

State. Per line, per agent. Lifetimes differ within the struct: valid and tag change on allocation and eviction; state changes on every transition; dirty follows the data; probe_pending is per probe.

Cycle behaviour. state is written by exactly one next-state function (§24). probe_pending exists so a second probe for a line already being probed can be detected rather than silently merged — which matters because probes for one line can arrive from a directory that has retried.

Contract. Every local access path reads state, never valid alone. dirty is a data obligation and not a permission — an agent that is dirty owes the system the newest value, and losing that obligation loses data (16.3 §22).

Failure. §9.

DV. Assert dirty implies a state that permits writing. Assert state != COH_I implies valid. Both are cheap and both catch metadata that has drifted out of self-consistency.

9. Wrong RTL — Tag Hit Means Usable

// WRONG — structural presence used as semantic permission.
assign cache_hit = valid && (tag == req_tag);
assign read_data = cache_hit ? line_data : miss_data;

What cache_hit is actually true for, in this design:

Three of the five are wrong, and the last three rows are the dangerous ones because the line physically contains plausible data.

Why this survives review. The expression is the textbook definition of a cache hit — in a non-coherent cache it is correct. Coherence adds a second question that the expression does not ask: not "is the data here?" but "am I permitted to use it?"

And the COH_S_TO_I row is the worst. The agent has been told to invalidate, is in the process of doing so, and serves a read from the copy it is relinquishing. The requester that caused the invalidation may already have been granted write permission — so two agents disagree about the line's value with no error anywhere.

A tag hit is a statement about storage. Permission is a statement about the protocol. They are different questions and only one of them makes data safe to use.

10. Permission Functions

// ILLUSTRATIVE. One place where "may I?" is decided, so the answer cannot
// drift between call sites.
function automatic logic read_permitted(coh_state_t s);
  return (s == COH_S) || (s == COH_E) || (s == COH_M);
endfunction
 
function automatic logic write_permitted(coh_state_t s);
  return (s == COH_E) || (s == COH_M);      // E may upgrade silently to M
endfunction
 
// Transient states permit NOTHING. Stated as an executable claim rather than
// left implicit in the two functions above.
function automatic logic is_transient(coh_state_t s);
  return (s == COH_I_TO_S) || (s == COH_I_TO_M)
      || (s == COH_S_TO_M) || (s == COH_M_TO_I) || (s == COH_S_TO_I);
endfunction
 
// The access gate the datapath actually uses.
assign local_read_ok  = valid && tag_match && read_permitted(line_state)  && !is_transient(line_state);
assign local_write_ok = valid && tag_match && write_permitted(line_state) && !is_transient(line_state);

Architecture. Three total functions plus one gate. Centralising the decision is what prevents §9 from reappearing in the third place someone checks a state, which is where it usually reappears.

Contract. read_permitted and write_permitted must be total over the enum — every state maps to a definite answer. A function with a default: return 1'b1 is a permission grant for every state added later, which is how a new transient state silently becomes readable.

Why is_transient is separate rather than folded in. The two permission functions describe stable semantics; is_transient is a structural guard. Keeping them separate means adding a transient state requires updating exactly one function, and forgetting to update the permission functions is then harmless rather than dangerous.

Failure. A non-total function; or checking permission at one site and forgetting is_transient at another.

DV. Exhaustively evaluate all three functions over every enum value and compare against a table — a small, complete, directed test that fully verifies them.

11. SVA — Access Requires Permission

// MANDATORY. The two properties that make Section 9 impossible.
property p_read_requires_permission;
  @(posedge clk) disable iff (!rst_n)
    local_read_fire |-> (read_permitted(line_state) && !is_transient(line_state));
endproperty
a_read_requires_permission: assert property (p_read_requires_permission);
 
property p_write_requires_permission;
  @(posedge clk) disable iff (!rst_n)
    local_write_fire |-> (write_permitted(line_state) && !is_transient(line_state));
endproperty
a_write_requires_permission: assert property (p_write_requires_permission);
 
// Dirty data implies the permission that produced it.
property p_dirty_implies_write_permission;
  @(posedge clk) disable iff (!rst_n)
    (line_meta.dirty && !is_transient(line_state)) |-> write_permitted(line_state);
endproperty
a_dirty_implies_write_permission: assert property (p_dirty_implies_write_permission);

Architecture. Two access properties and one metadata self-consistency property.

Why the third earns its place. A dirty line in a state that cannot write is a contradiction — it means the line was written and then downgraded without the data being written back or forwarded. That is silent data loss, and it is caught here by a property with no reference model at all.

Contract. These are per-agent and need no directory knowledge. They are the cheapest coherence assertions available and should be permanently enabled.

DV. They need transient states actually occupied while local accesses are attempted — which requires concurrent local traffic and remote transitions, not a quiescent test.

12. Why Transient States Are Mandatory

A worked case, with N agents. A holds COH_S. B and C also hold COH_S. A wants to write.

1. A requests write permission from the directory.
2. Directory sends probes to B and C.
3. B responds. C has not yet.
4. ...what state is A in?

A is not COH_S — it has requested an upgrade and the directory is acting on it. A is not COH_M — B and C may still hold readable copies, and C definitely does. There is no stable state that describes A at step 4, which is the argument in one sentence.

Four things break without a transient state:

The last row is the subtlest and the most chiplet-relevant. During the transition A can receive a probe caused by a different agent's request for the same line. A must respond to it in a way that is consistent with the transition it is itself performing, and only a transient state gives it the information to do so.

13. Physically Present Is Not Accessible

Restating §9's lesson at the transient level, because it is a different instance of it.

A line in COH_S_TO_M contains valid, correct, current data. The agent may read the bytes out of the array and they will be the right bytes. And it may not use them, because:

• the permission it is acquiring has not been granted;
• the permission it held may already have been relinquished at the directory;
• a probe may be in flight that this agent must honour.

The array holds data. The state holds permission. In a transient state the first is valid and the second is unresolved, and only the second decides whether an access is legal.

The design consequence is that transient states must block local access, which costs performance — a local read that would have hit now stalls. That cost is why designs are tempted to allow the access, and §22 is what that temptation produces.

14. The Directory Entry

// ILLUSTRATIVE directory entry. NOT a CHI or CXL directory format — no such
// format is claimed (Section 3).
typedef struct packed {
  logic                    owner_valid;   // is there an exclusive/dirty owner?
  logic [AGENT_W-1:0]      owner;         // meaningful only if owner_valid
  logic [NUM_AGENTS-1:0]   sharers;       // one bit per agent
  logic                    dirty_remote;  // the owner holds data newer than memory
  logic                    busy;          // a transition is in progress (Section 18)
} dir_entry_t;
 
dir_entry_t dir_q [NUM_LINES];

Architecture. Owner and sharers tracked separately, with an explicit owner_valid rather than a magic owner value. The sharer vector is the structure that does not exist in a two-agent system and is the reason this chapter's mechanisms differ from 11.3's.

State. Per line, at the home. busy has transition lifetime; the rest have line lifetime.

Cycle behaviour. Updated at defined points in a transition — never speculatively. §20's rule is that owner and sharers change at the commit, not when probes are sent.

Contract, and owner_valid is doing real work. Using a reserved owner value such as all-ones to mean "no owner" makes NUM_AGENTS and the encoding interdependent, and a design that grows to that agent count silently reinterprets "no owner" as a real agent. An explicit valid bit costs one flip-flop and removes the coupling.

Failure. §15's invariant violated; or updating sharers when probes are sent rather than when responses are received, which is §23's premature removal.

DV. Assert §15's invariant every cycle. Assert owner_valid implies the owner is not also in sharers under the design's chosen convention — and state that convention, because both are defensible and mixing them is a real bug.

15. Owner Against Sharers

The invariant, stated for the simplified model this chapter uses:

If an agent holds write permission for a line, no other agent may simultaneously hold read or write permission for it.

// ILLUSTRATIVE system-level invariant. Verification-only — it needs a view of
// every agent, which no single agent has.
property p_single_writer;
  @(posedge clk) disable iff (!rst_n)
    ($countones(tb_agents_with_write_permission[line]) <= 1);
endproperty
 
property p_no_reader_alongside_writer;
  @(posedge clk) disable iff (!rst_n)
    (tb_agents_with_write_permission[line] != '0)
      |-> ((tb_agents_with_read_permission[line]
            & ~tb_agents_with_write_permission[line]) == '0);
endproperty

Assumptions, stated because the invariant is only true under them. This is the simplified model of §7 — four stable states, one writer, and no protocol feature that permits a writer and readers to coexist under a stronger contract. Real protocols define richer state sets and some permit configurations this simplification forbids, and §3 is explicit that no protocol's semantics are being asserted here.

Why it must be verification-only. No agent knows what the others hold. The directory knows what it believes they hold — and the entire class of bug this chapter is about is the directory believing something false. So checking the invariant against the directory would compare the directory with itself; it must be checked against an independent census of what agents actually hold (§29).

What it catches. §18's two-writer race, §23's eager grant, and its premature sharer removal — three different mechanisms with one signature.

16. Same-Line Serialisation

Two agents requesting the same line concurrently is the defining N-agent hazard.

// ILLUSTRATIVE same-line conflict detection across the active transaction set.
logic [NUM_TXN-1:0] line_conflict;
 
always_comb
  for (int t = 0; t < NUM_TXN; t++)
    line_conflict[t] = txn_q[t].valid && (txn_q[t].line == incoming_line);
 
assign same_line_busy = |line_conflict;
 
// A new transition may only allocate if the line is not already transitioning.
assign may_allocate_txn = incoming_valid && !same_line_busy && txn_slot_free;

Architecture. A line-granular CAM over the active transaction set. Granularity is the design decision: too coarse and unrelated lines serialise; too fine and the comparison is expensive. Line granularity is the natural choice because the coherence unit is the line.

State. None of its own — combinational over txn_q (§17).

Cycle behaviour. Evaluated on every incoming request. may_allocate_txn must be a precondition of allocation, not a check performed afterwards — §18's bug is the version where two allocations both pass a stale check.

Contract. Every path that can start a transition must consult it, including probe-initiated ones and directory-internal ones such as an eviction. A path that bypasses it can start a second transition for a line already transitioning, which is exactly §18.

Failure. §19. Also: comparing the full address rather than the line address, which lets two transactions for different offsets in one line proceed concurrently — and they conflict, because coherence is per line.

DV. Drive two same-line requests in the same cycle and in adjacent cycles, and confirm exactly one allocates. Then drive same-line requests differing only in intra-line offset and confirm they still serialise.

17. The Coherence Transaction Table

// ILLUSTRATIVE. SEPARATE from line metadata (Section 4) — one line has one
// state, but one transition may await several agents.
typedef struct packed {
  logic                    valid;
  logic [LINE_W-1:0]       line;
  logic [AGENT_W-1:0]      requester;
  coh_state_t              target_state;   // what the requester is acquiring
  logic [NUM_AGENTS-1:0]   pending_resp;   // Section 20 — a BITMAP, not a count
  logic                    need_data;      // must a data response be collected?
  logic                    data_seen;      // has authoritative data arrived?
} coh_txn_t;
 
coh_txn_t txn_q [NUM_TXN];

Architecture. One entry per in-progress transition. pending_resp is the field that makes this chapter different from a two-agent one, and §21 is why it is a bitmap.

State. Per coherence transaction, allocated when a transition begins and released when ownership commits. Note this is a different lifetime from the line's transient state, which is why §4 separates them: the line returns to a stable state at the commit, and the transaction entry is released at the same event but for a different reason.

Cycle behaviour. pending_resp is set at allocation from the directory's sharer vector; bits clear as responses arrive (§20); the transition commits when it reaches zero and any required data has been collected (§25).

Contract. NUM_TXN bounds concurrent transitions. When full, new transitions must stall rather than proceed — and that stall is a legitimate backpressure, not an error. A design that drops a request when the table is full loses it silently.

Failure. Merging this into the line metadata, which cannot hold a per-agent pending set. Or sizing NUM_TXN at one, which serialises all coherence activity across all lines and is a throughput disaster with no correctness benefit.

DV. Fill the table and confirm new transitions stall and are not dropped. Confirm entries are released exactly once.

18. Wrong RTL — Two Transitions for One Line

// WRONG — allocation does not consult the same-line check.
always_ff @(posedge clk)
  if (incoming_valid && txn_slot_free) begin
    txn_q[free_slot].valid     <= 1'b1;
    txn_q[free_slot].line      <= incoming_line;
    txn_q[free_slot].requester <= incoming_agent;
  end

The race, with three agents. B and C both request write permission for line L in the same cycle.

Whichever writes last wins, and both requesters believe they won. B and C each transition to a write-permitting state, and §15's single-writer invariant is violated with no error anywhere.

Four properties.

Both transactions are individually well-formed. Each allocated correctly, probed the right agent, collected a response, and committed. There is no defective transaction — the defect is that there are two.

A's single response may satisfy both. With one probe target and two transactions expecting a response from A, one response arriving can clear both pending sets depending on how responses are matched. §22's response-identity property is what forbids that.

The window is one cycle wide in this form and much wider in practice, because across dies the requests can arrive in different cycles and still both find the line apparently free if the check is done against stale state (§16's "check afterwards" version).

And the resulting corruption is silent and durable. Two agents hold write permission and diverge. Neither will discover it; a third agent reading the line later gets whichever value it happens to be routed to.

19. SVA — One Transition Per Line

// MANDATORY. The property that makes Section 18 impossible.
property p_one_transition_per_line;
  @(posedge clk) disable iff (!rst_n)
    ($countones(active_txn_for_line(LINE_UT)) <= 1);
endproperty
a_one_transition_per_line: assert property (p_one_transition_per_line);
 
// Allocation requires the line to be free.
property p_alloc_requires_line_free;
  @(posedge clk) disable iff (!rst_n)
    txn_alloc_fire |-> !$past(same_line_busy_for(alloc_line));
endproperty
a_alloc_requires_line_free: assert property (p_alloc_requires_line_free);
 
// And the directory's busy flag agrees with the transaction table.
property p_dir_busy_matches_txn;
  @(posedge clk) disable iff (!rst_n)
    (dir_q[LINE_UT].busy == (active_txn_for_line(LINE_UT) != '0));
endproperty
a_dir_busy_matches_txn: assert property (p_dir_busy_matches_txn);

Why the third property matters. The directory's busy flag and the transaction table are two representations of the same fact. If they can disagree, one of them is wrong and downstream decisions read whichever they happen to consult — the same two-sources-of-truth argument 15.3 §15 made for aggregate counters, here with a correctness rather than a measurement consequence.

DV. These need genuinely concurrent same-line requests, which random traffic produces rarely. Directed same-line contention is mandatory (§30).

20. Response Bitmaps

// ILLUSTRATIVE response tracking. One bit per agent, NOT a counter — Section 21.
// Set at allocation from the directory's sharer vector.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int t = 0; t < NUM_TXN; t++) txn_q[t].pending_resp <= '0;
  end else begin
    // Allocate: the agents that must respond are exactly today's sharers,
    // excluding the requester itself.
    if (txn_alloc_fire)
      txn_q[alloc_slot].pending_resp <=
          dir_q[alloc_line].sharers & ~agent_onehot(incoming_agent);
 
    // A response clears EXACTLY ONE bit, and only if it was set.
    if (resp_valid && txn_q[resp_txn].valid
                   && txn_q[resp_txn].pending_resp[resp_agent])
      txn_q[resp_txn].pending_resp[resp_agent] <= 1'b0;
  end
end
 
// The transition may commit only when nothing is pending AND any required
// data has actually arrived (Section 25).
assign may_commit = txn_q[t].valid
                 && (txn_q[t].pending_resp == '0)
                 && (!txn_q[t].need_data || txn_q[t].data_seen);

Architecture. A bit per agent, cleared by identity rather than counted. The set is initialised from the directory's sharer vector, which ties the two structures together: the directory says who has copies, and the bitmap tracks who has acknowledged giving them up.

State. Per transaction. Note the requester is excluded at allocation — an agent does not probe itself, and including it produces a bit that never clears.

Cycle behaviour. The clear is guarded by pending_resp[resp_agent] — a response from an agent that was not expected, or a second response from an agent already cleared, changes nothing. That guard is the entire mechanism of §21.

Contract. The commit condition is a conjunction of both the empty bitmap and the data condition. A transition that commits on an empty bitmap while data is still owed transfers ownership without the newest value — which is §25.

Failure. §21's counter. Also: initialising pending_resp from a stale sharer vector read before the directory was updated, which can omit an agent that has since acquired a copy.

DV. Assert §22's properties. Then drive a duplicate response and a response from a non-sharer and confirm neither changes the pending set.

21. Wrong RTL — a Response Counter

// WRONG — counts responses without checking WHO responded.
always_ff @(posedge clk) begin
  if (txn_alloc_fire)  pending_count_q <= $countones(dir_q[alloc_line].sharers);
  else if (resp_valid) pending_count_q <= pending_count_q - 1'b1;
end
 
assign may_commit = (pending_count_q == '0);

The failure, with three sharers A, B and C.

Ownership commits while C still holds a readable copy. The requester gets write permission, writes, and C continues serving reads of the old value. Two agents disagree about the line, and §15's invariant is violated.

Four properties make this the signature N-agent bug.

It is invisible with one sharer. With a single responder, a counter and a bitmap are equivalent. This bug is unreachable in a two-agent system, which is exactly why 11.3 does not need it and why it belongs in this chapter.

Its trigger is a transport event, not a coherence event. A duplicate response arises from a UCIe retry — the same object delivered twice (14.3 §20) — or from an agent responding to a re-sent probe. So a coherence bug is triggered by a reliability mechanism, and 16.3 §19 develops that boundary.

The counter is also vulnerable in the other direction. A response from an agent that was never a sharer decrements it, which is the same failure with a different cause.

And the fix costs almost nothing. NUM_AGENTS bits instead of log2(NUM_AGENTS), plus a guarded clear. For eight agents that is eight bits against three — five flip-flops per transaction entry to make an entire class of silent corruption unrepresentable.

22. SVA — Response Identity and Commit Condition

// MANDATORY. A response only clears a bit that was actually pending.
property p_response_clears_only_pending;
  @(posedge clk) disable iff (!rst_n)
    (resp_valid && !txn_q[resp_txn].pending_resp[resp_agent])
      |=> $stable(txn_q[resp_txn].pending_resp);
endproperty
a_response_clears_only_pending: assert property (p_response_clears_only_pending);
 
// A response clears at most one bit.
property p_response_clears_one_bit;
  @(posedge clk) disable iff (!rst_n)
    resp_valid |=> ($countones($past(txn_q[resp_txn].pending_resp)
                             & ~txn_q[resp_txn].pending_resp) <= 1);
endproperty
a_response_clears_one_bit: assert property (p_response_clears_one_bit);
 
// Ownership commits only when nothing is pending and data obligations are met.
property p_commit_requires_all_responses;
  @(posedge clk) disable iff (!rst_n)
    ownership_commit |-> ((txn_q[commit_txn].pending_resp == '0)
                       && (!txn_q[commit_txn].need_data || txn_q[commit_txn].data_seen));
endproperty
a_commit_requires_all_responses: assert property (p_commit_requires_all_responses);
 
// A response must belong to a live transaction for the line it names.
property p_response_belongs_to_live_txn;
  @(posedge clk) disable iff (!rst_n)
    resp_valid |-> (txn_q[resp_txn].valid && (txn_q[resp_txn].line == resp_line));
endproperty
a_response_belongs_to_live_txn: assert property (p_response_belongs_to_live_txn);

Architecture. Four properties covering the four ways response handling fails: clearing a bit that was not pending, clearing more than one, committing early, and accepting a response for a dead or mismatched transaction.

The first two together are what a counter cannot satisfy. A counter has no notion of which bit, so neither property is even expressible over it. Writing these properties forces the bitmap, which is a useful way to think about assertion design: a property you cannot express is telling you something about the structure.

DV. Inject a duplicate response, a response from a non-sharer, and a response naming a retired transaction — three directed cases, each targeting one property.

23. Wrong RTL — Eager Ownership Grant

// WRONG — probes are sent and ownership is granted in the same breath.
if (txn_alloc_fire) begin
  send_probes(dir_q[alloc_line].sharers);
  dir_q[alloc_line].owner       <= incoming_agent;    // ← too early
  dir_q[alloc_line].owner_valid <= 1'b1;
  dir_q[alloc_line].sharers     <= '0;                // ← also too early
  grant_write_permission(incoming_agent);             // ← far too early
end

The window is the probe round trip, and across dies that is long (§27).

cycle 0    A granted write permission; directory says sharers = {}
cycle 0    probes dispatched to B and C
cycle 0..N B and C still hold COH_S and still serve local reads
cycle k    A writes X = 1
cycle k    B reads X → 0        ← B is a legal sharer per its own state
cycle N    B and C finally invalidate

Between cycle 0 and cycle N, A may write while B and C may read. §15's invariant is violated for the whole probe round trip, and both agents behaved exactly as their own state permitted.

Three properties.

The directory's belief and reality diverge immediately. At cycle 0 the directory records sharers = {} while B and C demonstrably hold copies. The directory is not tracking the system; it is tracking its own intentions.

The severity scales with transport latency, which is the chiplet-specific part. On a monolithic die the window is tens of cycles; across UCIe it is a full round trip including serialisation, queueing and any retry — so the same bug that is a narrow race on a monolithic die is a wide one here.

And it produces no error at any layer. Every probe is delivered, every response returns, every CRC passes. The only evidence is that two agents' values diverge, which requires §29's line-level model to detect.

Ownership commits when the last conflicting permission has been relinquished — not when the request to relinquish it was sent. That is the central safety rule of the chapter, and §22's third property is its executable form.

24. The Probe-Against-Local Race

Same cycle: a probe arrives for line L, and the local agent completes an operation on L. Both want to write line_state.

// WRONG — two independent state assignments in the same always_ff.
always_ff @(posedge clk) begin
  if (probe_received)
    line_q[idx].state <= COH_I;          // honour the invalidation
 
  if (local_txn_complete)
    line_q[idx].state <= COH_M;          // take the ownership we acquired
end

Both conditions true in one cycle. The last assignment wins, so state becomes COH_M and the invalidation is silently lost.

What the system now believes:

Two agents hold write permission and the directory records one. §15's invariant is violated and the directory cannot detect it, because from its point of view the probe was answered.

The correct structure is a single next-state function with explicit priority:

// ILLUSTRATIVE. ONE writer, explicit priority, every combination considered.
coh_state_t state_d;
 
always_comb begin
  state_d = line_q[idx].state;                    // default: hold
  unique case (1'b1)
    // A probe is an obligation from the coherence protocol and is honoured
    // ahead of a locally-initiated completion. That ORDER is a design
    // decision and must be documented — see the note below.
    probe_received && probe_requires_invalidate : state_d = COH_I;
    probe_received && probe_requires_downgrade  : state_d = COH_S;
    local_txn_complete && (txn_target == COH_M)  : state_d = COH_M;
    local_txn_complete && (txn_target == COH_S)  : state_d = COH_S;
    local_evict                                  : state_d = COH_I;
    default                                      : state_d = line_q[idx].state;
  endcase
end
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) line_q[idx].state <= COH_I;
  else        line_q[idx].state <= state_d;

Architecture. One combinational next-state function, one registered assignment, unique case (1'b1) for explicit priority.

Cycle behaviour. Exactly one arm fires. The priority order is a protocol-dependent design decision — whether a probe or a local completion wins, and what the losing event must then do, depends on the coherence protocol's rules. This chapter states that the order must be explicit and documented; it does not assert which order any protocol requires (§3).

Contract. The losing event must not be dropped. If the probe wins and the local completion is deferred, the completion must still be handled — otherwise the transaction never retires and §17's table leaks an entry. A real design either re-evaluates the deferred event or records it.

Failure. The two-assignment form above. Also: unique case without a default, which infers a latch and makes the state hold in unanticipated combinations.

DV. Force both conditions in the same cycle and check the documented priority is honoured and that the losing event is still processed. That second half is the part usually missed.

25. Dirty Data and Where the Newest Value Lives

16.1 §6's semantic model, now with the mechanism that maintains it.

memory[X]  = 0
A holds X in COH_M, value 1, dirty         → latest is at A, not in memory
B requests read permission for X

A read served from memory returns 0, which is wrong. The correct outcome requires the newest value to come from A — which is why §17's transaction entry has need_data and data_seen as fields separate from pending_resp.

Three distinct things a transition may need, and they are not the same event:

Why they must be separate fields. A transition invalidating three clean sharers needs three permission responses and no data — memory is current. A transition taking a line from a dirty owner needs one permission response and data. A design with one counter for "responses" cannot distinguish those, and will either commit without data or wait forever for data that is not coming.

And the dirty obligation must survive. An agent holding dirty data owes the system that value. Losing it — by dropping the line on a recovery, by an eviction that discards, by a transition that forgets — is silent data loss, which is 16.3 §22's recovery case.

26. Transport Retry Is Not a New Coherence Request

The boundary that 16.3 develops fully, stated here because it corrupts this chapter's structures.

Chapter 14.3 established that UCIe replay retransmits the same object under the same identity, and that the receiver must recognise the repeat. If a coherence engine is driven directly by physical arrivals, a replayed probe or request becomes a second semantic action:

So the retry mechanism, working exactly as designed, produces three distinct coherence corruptions in a design that lacks the semantic-delivery boundary.

The defences are already in this chapter, which is worth noticing:

• §16's same-line check blocks the duplicate request from allocating;
• §20's guarded bitmap clear makes a duplicate response a no-op;
• §22's properties catch both.

The bitmap is doing double duty: it exists for multi-sharer aggregation and it happens to be idempotent, which makes it robust to duplicate delivery. A counter is neither.

27. Cross-Die Transient Lifetime

The chiplet-specific insight of the chapter.

On a monolithic die a probe round trip is a fabric traversal — tens of cycles. Across chiplets it is:

Consequences, and the third is the one that changes designs.

Transient states are occupied for far longer, so the probability that any given race occurs is much higher. Every bug in this chapter is a race whose window is the transient lifetime, and chiplets widen that window by an order of magnitude or more.

More transitions are in flight concurrently for the same throughput, by 15.2 §14's in_flight ≈ throughput × latency. So §17's table must be deeper — and a table sized for monolithic latencies is a throughput limit in a chiplet system.

And a broadcast-snoop design becomes untenable where a directory does not. Snoop traffic scales with agents and with die crossings, and each crossing has the full latency above. That is a transport-driven architectural conclusion, not a protocol one, and it is why §5's table lists directories first for chiplet systems.

28. Deadlock

A realistic dependency cycle, built from mechanisms this curriculum has already established.

1. Home H is resolving a transition for line L; it awaits a probe response from B.
2. B has generated the response, but cannot send it: B's outbound path to H
   has no credit (13.1) — or no replay space (14.3 Section 28).
3. Credit returns to B only when H drains B's inbound queue.
4. H's inbound queue head is a NEW coherence request for line L from agent C.
5. H cannot process that request: line L is busy with the transition at step 1
   (Section 16's same-line rule).
6. → H waits for B; B waits for credit; credit waits for H to drain; H's drain
     is blocked by a request that waits for the transition H is trying to finish.

Every participant is behaving correctly and nothing is progressing.

Three properties, and the third is the general lesson.

Every local assertion passes. Occupancies are legal, credits are conserved, the bitmap is consistent, no illegal state exists. 13.3 §19 established why: a deadlocked system generates no transitions, so every safety property holds vacuously and forever.

The cycle crosses layers. It runs through coherence state, queueing, flow control and reliability — so no single layer's designer can see it, and it will not be found by reviewing any one of them.

And the structural fix is to break the dependency, not to add capacity. The classic remedies are to ensure responses can always make progress independently of requests — separate queues, reserved credit, or a rule that responses are never blocked behind requests. 11.3 §28 developed the forward-progress classification that makes this precise for CXL, and the generic principle is that a message type which resolves an obligation must not queue behind one that creates obligations.

29. Safety and Liveness

The two failure modes are completely different and need different verification. A safety failure produces wrong data and keeps running; a liveness failure produces no data and is obvious. A test suite that only checks safety will pass a deadlocked design, because a deadlocked design never does anything wrong.

// LIVENESS, bounded, with assumptions stated. An unconditional property fires
// on legitimate congestion and gets disabled (15.2 Section 36's argument).
//
//   A1: probed agents eventually respond
//   A2: the transport eventually delivers (no permanent link failure)
//   A3: no agent is starved of arbitration indefinitely
assume property (@(posedge clk) disable iff (!rst_n)
  probe_sent |-> ##[1:PROBE_RESP_BOUND] probe_response_seen);
 
property p_transition_eventually_resolves;
  @(posedge clk) disable iff (!rst_n)
    txn_alloc_fire |-> ##[1:TXN_BOUND] (ownership_commit || txn_failed);
endproperty
a_transition_eventually_resolves: assert property (p_transition_eventually_resolves);

The consequent admits txn_failed for the same reason 14.2 §30's recovery property admits an explicit failure state: the requirement is that the transition terminates, not that it succeeds. A transition that fails and reports is a correct outcome; one that hangs is not.

30. The Line-Level Reference Model

// Verification-only. Not synthesisable.
//
// This is the model that detects what a transport scoreboard cannot (Section 31).
class coherence_scoreboard;
 
  typedef struct {
    bit [DATA_W-1:0]        latest_value;
    int                     owner;            // -1 = memory
    bit [NUM_AGENTS-1:0]    sharers;          // independently tracked
    bit                     dirty;
    bit                     pending_transition;
    bit [NUM_AGENTS-1:0]    expected_responses;
    int                     semantic_allocations;   // must be 1 per request
  } line_model_t;
 
  line_model_t line[bit [LINE_W-1:0]];
 
  // Independently observed: what each agent ACTUALLY holds, from monitors at
  // each agent — NOT from the directory (Section 15's argument).
  bit [NUM_AGENTS-1:0] observed_read_perm  [bit [LINE_W-1:0]];
  bit [NUM_AGENTS-1:0] observed_write_perm [bit [LINE_W-1:0]];
 
  // ---- Check 1: single writer, against OBSERVED permissions.
  function void check_single_writer(bit [LINE_W-1:0] l);
    if ($countones(observed_write_perm[l]) > 1)
      $error("TWO WRITERS on line %0h: mask %0b", l, observed_write_perm[l]);
    if (observed_write_perm[l] != 0
        && (observed_read_perm[l] & ~observed_write_perm[l]) != 0)
      $error("READER ALONGSIDE WRITER on line %0h: readers %0b, writer %0b",
             l, observed_read_perm[l], observed_write_perm[l]);
  endfunction
 
  // ---- Check 2: every read returns the latest value. Detects Section 23.
  function void check_read(bit [LINE_W-1:0] l, int agent, bit [DATA_W-1:0] got);
    if (got !== line[l].latest_value)
      $error("STALE READ line %0h by agent %0d: got %0h, latest %0h (owner %0d)",
             l, agent, got, line[l].latest_value, line[l].owner);
  endfunction
 
  // ---- Check 3: the DIRECTORY's belief matches reality. Detects Section 23's
  //      divergence and Section 21's premature commit.
  function void check_directory(bit [LINE_W-1:0] l,
                                bit [NUM_AGENTS-1:0] dut_sharers,
                                int dut_owner);
    bit [NUM_AGENTS-1:0] actual = observed_read_perm[l] | observed_write_perm[l];
    if (dut_sharers != (actual & ~agent_mask(dut_owner)))
      $error("DIRECTORY DIVERGENCE line %0h: believes %0b, agents actually hold %0b",
             l, dut_sharers, actual);
  endfunction
 
  // ---- Check 4: dirty data is never lost.
  function void check_dirty_preserved(bit [LINE_W-1:0] l);
    if (line[l].dirty && line[l].owner == -1)
      $error("DIRTY DATA LOST on line %0h: model says dirty with no owner", l);
  endfunction
 
  // ---- Check 5: one semantic allocation per request (Section 26).
  function void on_semantic_request(bit [LINE_W-1:0] l, int req_id);
    line[l].semantic_allocations++;
    if (line[l].semantic_allocations > 1 && line[l].pending_transition)
      $error("DOUBLE ALLOCATION line %0h for request %0d", l, req_id);
  endfunction
 
endclass

Architecture. Five checks over two models: a semantic line model and an independently observed permission census.

Check 3 is the one this chapter exists for, and its independence is essential. It compares what the directory believes against what agents actually hold, observed at the agents. Comparing the directory against itself proves nothing, and the entire class of bug here — §21's premature commit, §23's eager grant, §24's lost invalidation — is the directory believing something false.

Check 1 uses observed permissions rather than directory state for the same reason. §15's invariant is about reality, not about the directory's model of it.

Check 2 is 16.1 §28's stale-read check carried forward, and it is what catches the consequence when checks 1 and 3 catch the cause.

31. Why a Transport Scoreboard Passes

The most important negative result in the chapter.

Consider a run in which the directory removes agent A from its sharer vector one transition too early (§23's eager grant, or §21's counter reaching zero prematurely).

Every mechanism in Modules 12 through 15 reports success. The packets were perfect. The system is wrong, and only a model of line ownership can say so.

A packet scoreboard verifies that the right bits arrived. A coherence scoreboard verifies that the right permission state resulted. These are independent claims, and the first cannot imply the second.

The practical consequence for a verification plan: a chiplet coherence environment needs at least two independent models — one for transport and one for line semantics — and 16.3 §28 argues it needs three. Reusing one for the other leaves an entire failure class unverified.

32. Coverage

covergroup cg_coherence @(posedge clk);
  option.per_instance = 1;
 
  // --- Stable and transient states must ALL be occupied.
  cp_state : coverpoint line_state {
    bins invalid   = {COH_I};
    bins shared    = {COH_S};
    bins exclusive = {COH_E};
    bins modified  = {COH_M};
    bins t_i_s     = {COH_I_TO_S};
    bins t_i_m     = {COH_I_TO_M};
    bins t_s_m     = {COH_S_TO_M};   // the upgrade — Section 12's case
    bins t_m_i     = {COH_M_TO_I};
    bins t_s_i     = {COH_S_TO_I};
  }
 
  // --- How many sharers were probed. The N-agent dimension.
  cp_sharer_count : coverpoint sharers_at_transition_start {
    bins none  = {0};
    bins one   = {1};                // equivalent to the two-agent case
    bins two   = {2};
    bins many  = {[3:$]};            // where Section 21's bug lives
  }
 
  // --- Response anomalies (Sections 21, 26).
  cp_resp_anomaly : coverpoint response_anomaly {
    bins normal        = {0};
    bins duplicate     = {1};        // must NOT clear twice
    bins from_nonsharer= {2};        // must NOT clear anything
    bins late          = {3};        // after the transaction retired
  }
 
  // --- Concurrency on one line.
  cp_same_line : coverpoint same_line_concurrent_requests {
    bins one = {1}; bins two = {2}; bins many = {[3:$]};
  }
  cp_probe_local_race : coverpoint probe_and_local_same_cycle;   // Section 24
 
  // --- Data obligations (Section 25).
  cp_data_need : coverpoint transition_data_requirement {
    bins none_needed = {0};          // clean sharers only
    bins from_owner  = {1};          // dirty owner must supply
  }
  cp_dirty_probed : coverpoint dirty_owner_was_probed;
 
  // --- Transport interaction (Section 26).
  cp_retry_during : coverpoint transport_retry_during_transition;
  cp_recovery_during : coverpoint recovery_during_transient;
 
  // --- Capacity.
  cp_txn_table : coverpoint txn_table_occupancy {
    bins empty = {0}; bins some = {[1:NUM_TXN-1]}; bins full = {NUM_TXN};
  }
 
  // --- Crosses that carry the information.
  x_sharers_anomaly : cross cp_sharer_count, cp_resp_anomaly;   // Section 21
  x_state_race      : cross cp_state, cp_probe_local_race;      // Section 24
  x_dirty_transition: cross cp_data_need, cp_sharer_count;
  x_retry_sharers   : cross cp_retry_during, cp_sharer_count;
endcovergroup

Six bins whose value is being non-zero:

cp_sharer_count.many crossed with cp_resp_anomaly.duplicate. This is §21 exactly — a duplicate response with three or more sharers. With one sharer the bug is unreachable, so a regression that never has three sharers cannot find it.

cp_state — every transient bin. A transient state never occupied means its transitions were never exercised and §11's guards were never tested.

cp_probe_local_race. §24 requires a probe and a local completion in the same cycle, which random traffic produces rarely and which must be forced.

cp_same_line.two and .many. §18's race needs genuine same-line contention.

cp_data_need.from_owner crossed with sharers. §25's dirty-owner case, where permission responses and data are different obligations.

cp_recovery_during. A recovery while a transition is transient — the case 16.3 §21 develops, and which never occurs spontaneously.

33. Flagship Trace — Upgrade With Two Sharers

Illustrative. A, B and C hold line L in COH_S. A requests write permission.

Seven readings, and the ones about timing matter most.

Cycle 1: pending is {B,C}, not {A,B,C}. A is excluded because an agent does not probe itself — and a bit that never clears would hang the transition permanently.

Cycles 1–8: A sits in S_TO_M for seven cycles. During all of them A may neither read nor write, even though its cache physically holds valid data (§13). That is the performance cost of correctness, and §27 says it is far larger across dies.

Cycles 4–6: B invalidates at cycle 4 and its bit clears at cycle 6. Two cycles apart, because the response crosses a die boundary. A design that cleared the bit when the probe was sent would have committed at cycle 3 with both B and C still readable — §23.

Cycle 6: B's response clears exactly one bit. {B,C} becomes {C}. §21's counter would have gone 2 → 1 identically here — the two are indistinguishable until a duplicate arrives, which is why the bug hides.

Cycle 8: pending is empty, and only now may ownership commit. No data was required because all sharers were clean and memory is current (§25).

Cycle 9: the directory and A update on the same event. Sharers cleared, owner set, A's state to COH_M. Splitting these across cycles is a window in which the directory and the agent disagree.

And the Latest column moves only at cycle 10, when A actually writes. Acquiring permission does not change the value — a distinction §25's separate need_data field exists to preserve.

34. Failure Trace — the Duplicate Response

The same scenario with §21's counter, and B's response replayed by the transport.

The buggy design committed two cycles early and one responder short.

Three properties.

The trigger is a transport event. Nothing in the coherence protocol went wrong — UCIe's retry mechanism did exactly what 14.3 designed it to do, and a coherence structure that is not idempotent turned it into corruption.

The correct design needs no special handling. The bitmap's guarded clear makes the duplicate a no-op by construction. It is not defensive code added for retries; it is the natural form of the structure, and its robustness to duplicates is a consequence.

And the failure is invisible for two cycles and then permanent. Between cycles 8 and 12, C holds a readable copy that the directory believes is gone. After cycle 10, C's copy and A's copy differ and nothing will reconcile them.

35. Debug Taxonomy

The eighth row is the diagnostic signature of this chapter. A bug that appears only with three or more sharers is almost certainly a response-aggregation bug, because that is the dimension that does not exist below three.

36. Debug Checklist

1. Which line? And which agents hold it, observed at the agents rather than from the directory?
2. What does the reference model say the latest value is, and who owns it?
3. What does the directory believe — sharers and owner? A divergence from question 1 is the finding (§30).
4. What state is each agent's line in — stable or transient? (§7).
5. Which transaction owns the transition, and is there exactly one for this line? (§19).
6. Which responses are still pending, and from whom? The bitmap, by agent identity.
7. Did any agent respond twice? Or did a non-sharer respond (§21, §22)?
8. Was the requester accidentally in its own pending set? That never clears.
9. Did a local access race with an incoming probe, and which won? Is the priority documented (§24)?
10. Is dirty data involved, and was a data obligation tracked separately from permission? (§25).
11. Did a transport retry occur during the transition? (§26).
12. Did a recovery occur while a line was transient? (16.3 §21).
13. Did ownership commit before the pending set was empty? (§23).
14. How many sharers were involved? If three or more, suspect aggregation first (§35).
15. Is anything progressing at all? If not, this is liveness, and safety assertions will not help (§29).
16. Do the line model and the transport model disagree? If the transport model passes and the line model fails, the transport is fine (§31).

37. Common Misconceptions

"Coherence is MESI." MESI is one stable-state set. Coherence is ownership tracking, invalidation, transient state, response aggregation, serialisation and data obligations — and the stable states are the smallest part of the hardware (§4, §12).

"A tag hit means the data is usable." A tag hit says the storage is present. Permission says the protocol allows using it, and a line can be present while invalid, being invalidated, or awaiting a fetch — three of five cases in §9's table are unsafe.

"Transient states are optional." Without them an agent mid-upgrade must claim to be in its old state or its new one; the first cannot recognise its own completion and the second grants permission others still hold. There is no stable state that describes an agent waiting for probe responses (§12).

"Memory always contains the latest value." With a dirty owner it does not, and it will not until a protocol action moves the value. A read served from memory in that case is wrong (§25, 16.1 §31).

"A response counter is enough." With one sharer it is. With several, a duplicate response — which a transport retry produces routinely — decrements it twice and ownership commits with an agent still holding a readable copy (§21, §34).

"Two same-line requests can proceed independently." Both probe the same agents, both may consume one response, and both commit ownership. Whichever writes the directory last wins and both requesters believe they won (§18).

"A probe and a local completion can each update the state." Two assignments in one cycle means the last wins and the other is silently lost — and if the invalidation loses, the agent keeps permission the directory believes it surrendered (§24).

"Transport replay means another coherence request." A replay is the same object delivered again. A design that allocates on physical arrival turns a reliability mechanism into duplicate ownership transitions, duplicate probes and double-cleared response sets (§26).

"A packet scoreboard verifies coherence." Every flit can arrive exactly once with clean CRC, every transaction can match, throughput can be nominal — and two agents can hold conflicting permissions. Only a line-ownership model detects it (§31).

"If no two caches are Modified, coherence must be correct." Two agents in Modified is one failure signature. A reader alongside a writer, a stale sharer the directory has forgotten, and lost dirty data are all coherence failures with no two-Modified condition (§15, §30).

"Deadlock will violate some local counter assertion." A deadlocked system produces no transitions, so every safety property holds vacuously forever. Detection requires bounded liveness with assumptions stated, or a watchdog (§29).

"Chiplet coherence is monolithic coherence with more latency." More latency means longer transient lifetimes, which means every race window widens by an order of magnitude and more transitions are concurrently in flight for the same throughput (§27).

38. Understanding Check

39. Summary and What Comes Next

Coherence is distributed ownership plus distributed invalidation of obsolete permissions, and the system must be able to say, per line, who may read, who may write, who holds a copy, who owns the newest data, and whether any of that is changing.

Three kinds of state, kept separate. Stable permissions, transient transitions, and the transaction resolving them. A design with only line state cannot represent "waiting for three of five agents"; a design with only transaction state cannot answer "may I read?" in one cycle.

A tag hit is storage; permission is protocol. Three of five cases in §9's table are physically present and unsafe to use, and the worst is a line being invalidated that still serves reads.

The sharer vector and the response bitmap are the N-agent structures. Neither exists with one caching agent, and a response counter commits ownership early the first time a duplicate arrives — which a transport retry produces routinely, making a reliability mechanism the trigger for a coherence corruption.

Ownership commits when the last conflicting permission is relinquished, not when the request to relinquish it was sent. Eager grant opens a window as wide as the probe round trip, and across dies that is an order of magnitude wider than on a monolithic die.

One next-state function with explicit, documented priority — two assignments in a cycle silently lose an invalidation, and the losing event must still be processed rather than dropped.

Permission and data are separate obligations. Invalidating clean sharers needs responses and no data; taking a line from a dirty owner needs both. One counter for "responses" cannot express the difference.

And a packet scoreboard passes while the system is corrupt. Every flit delivered once, every CRC clean, every transaction matched, throughput nominal — and two agents holding conflicting permissions. Only a line-ownership model checked against an independent census of what agents actually hold can see it.

We now know how ownership and data correctness are represented. The next problem is transport composition: how does a coherent operation cross a chiplet boundary through UCIe without transport retries, queueing, or recovery changing its semantic meaning?

• 16.3 — Chiplet-Level Coherency — semantic operations against coherence messages against transport attempts, probe fan-out across links, and what survives a recovery mid-transition.

Browse the full path on the UCIe tutorials index.