“UCIe Automatically Provides Coherency”

The belief is usually acquired from a picture. A layer diagram shows a coherent protocol sitting on top of a die-to-die interconnect, the word coherent appears near the top, and the eye assigns the property to the stack. Nothing in the diagram is wrong. The inference is — and it is the most expensive inference in this module.

1. The Belief

"UCIe carries CXL, and CXL is coherent. So if I connect two dies over UCIe, they're coherent."

The failure mode is not a hang or a slowdown. It is silent data corruption under concurrent access, on a link with no errors, with every integrity check passing — and it is intermittent, workload-dependent, and reproducible only under contention.

The replacement model (§6):

Coherence is a protocol between agents with a defined point of serialisation. It requires caching agents that track line state, a home agent that orders and resolves conflicting requests, a mechanism to find and invalidate other copies, and a stated memory model. A link carries that protocol's messages. Carrying a coherent protocol is necessary and nowhere near sufficient — you must also instantiate the agents.

2. Why a Competent Engineer Believes It

Four reasons, and the first is a genuine property of how the material is drawn.

Layer diagrams assign properties upward by visual convention. A stack showing a coherent protocol above a transport reads as "this stack is coherent" — and for a complete system that is a fair reading. The error is applying it to a link drawn in the same picture. The diagram is describing what can be carried, not what is provided.

The word attaches to link names in industry usage. Phrases like "coherent interconnect" are common and not wrong — they describe an interconnect used within a coherent system. But the adjective migrates, and once it attaches to the wire it stops being a description of context.

Protocol support is real and is genuinely a prerequisite. The UCIe Consortium's public material lists "Protocol: CXL/PCIe for near-term volume attach" among its focus areas (§5). Something real is being provided — and the step from "it carries the coherent protocol" to "it provides coherence" is one word wide.

And the monolithic mental model has no separate concept for it. On a single SoC, coherence is already there — a fabric with a home agent and a snoop filter exists before you start, and it is nobody's explicit design decision. An engineer who has never had to instantiate coherence has no slot in their model for "who provides it."

3. The Hidden Assumption

That coherence is a property of a link — something a connection can possess and confer — rather than a property of a graph of agents with a serialisation point.

Two sub-assumptions, both individually reasonable.

A — that reliable, ordered delivery is the hard part of coherence. It is not even the interesting part. A perfect link delivers exactly what it was given, in order, every time — and two caches connected by it are still incoherent (§7), because nothing decided who may write.

B — that "supports" and "provides" are interchangeable. They are not, and the distinction is the same one 28.2 drew about reliability: the ability to carry a protocol says nothing about whether the agents that protocol requires have been built.

4. The Root-Cause Tree

The second architecture decision deserves its own attention because it survives even when the first is fixed. Link ordering is per-link and per-direction; coherence ordering is a property of a serialisation point across all requesters. A design can have a home agent and still be wrong if it assumes messages that arrived in order were ordered in the coherence sense.

5. Sourcing and Scope

6. What Coherence Actually Requires

Three readings.

Five of six rows require state about other agents, which is definitionally not something a point-to-point link possesses. A link knows about the two ends it joins. Coherence is a property of the whole set of caching agents, and no pairwise connection can hold that.

Row 2 is the decisive one and the one most often absent. Conflicting requests to the same line must be ordered somewhere — and that somewhere is a component with state, arbitration and a queue. 26.1 §9 made exactly this argument from a shipping product: a die owns coherency because coherence is a property of the whole graph, and a link connects two dies.

And the corroborating evidence points the same way (§5). NVIDIA's description of its two-die GPU attributes full coherence to a shared L2 cache, not to the NV-HBI link. In both documented cases, the component named as providing coherence is a piece of logic with state.

7. The Minimal Counterexample

The tightest counterexample in this module, and it needs no product, no protocol and no numbers.

Build it. Two dies. Each has a cache. They are joined by a perfect link: zero errors, infinite bandwidth, in-order delivery, zero latency if you like. No coherence protocol between them — the link carries reads and writes to memory.

Two agents, three different values for one address, at cycle 3. The link delivered everything perfectly and never made an error.

Four readings.

The link was flawless, so the link was never the mechanism. This is the counterexample's entire force: you cannot fix cycle 3 by improving the link, because the link did not fail. Improve it to infinity and the corruption is unchanged.

What is missing is not delivery — it is a decision. Nothing decided that die A's write at cycle 1 must invalidate die B's copy. That decision requires knowing die B has a copy (a directory or a snoop), and a component empowered to order A's write against B's (a home agent). Neither is a transport function.

Adding a coherent protocol to the wire does not fix it either, which is the step people miss. A protocol is a set of messages and rules; it does nothing unless agents implement its state machines and a serialisation point resolves conflicts. Carrying Invalidate on a link that nobody sends or acts on is carrying nothing.

And this is why the belief's failure is silent. No error is possible here — every operation succeeded. The corruption is the absence of an interaction that was never designed, and absences raise no flags (27.6 §6).

8. When the Belief Is Locally Useful

Three readings.

Row 1 is why this belief is nearly universal among strong SoC engineers. On a monolithic part, coherence is infrastructure — present, correct, and not anyone's design decision. An engineer can have twenty years of correct experience in which coherence was simply a service the fabric provided, and nothing in that experience creates a slot for "who provides it?"

Row 3 is the most precise version of the kernel of truth, and it is why the CXL association is so persuasive. In the host-plus-device case the belief is nearly right: attach a device to a host processor, and the host's home agent orchestrates coherency across the caching agents (§5, Class C attributed). The device designer really does get coherence by participating in a protocol. The belief breaks when it is transplanted to a case where both sides are being designed and neither is a host — there is no existing home agent to join.

And that transplant is exactly §4's shape — a locally true idea at the wrong architectural scope.

9. Architectural Consequence

Three readings.

Row 7 is what makes this misconception more expensive than the others in this module. 26.1 §10 established that coherent traffic turns transport properties into correctness properties: a dropped bulk read is a retry; a dropped coherent request is a stalled core. An architect who does not know coherent traffic is crossing has not derived any of the completion, ordering or retention requirements that boundary needs.

Row 3's absence has a performance consequence that surprises people (§13). A snoop filter is not merely a correctness device — it is what prevents every write from broadcasting to every agent. With no filter and no directory, a correct implementation is a broadcast one, and its cost grows with agent count.

And row 2 is a genuinely major architectural decision that the belief skips entirely. Where the point of coherence lives determines latency for every coherent operation in the system — and 26.1 §9 shows a shipping product putting it at the hub, on the path to memory, for exactly that reason.

10. RTL Consequence — a Bridge That Grants a Line Twice

The bridge an engineer writes when the link is assumed to provide coherence: a forwarder.

// WRONG. ILLUSTRATIVE. A "coherent bridge" that grants a line to whoever asks.
// Written by someone who reasoned that the link delivers reliably and in order,
// so requests are naturally serialised and each requester gets what it asked
// for. Both halves of that reasoning are true. Neither is coherence.
typedef enum logic [1:0] { L_INVALID, L_SHARED, L_MODIFIED } lstate_e;
 
lstate_e line_state_q [N_LINE];        // the BRIDGE's idea of the line's state
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int i = 0; i < N_LINE; i++) line_state_q[i] <= L_INVALID;
  end else begin
    if (req_valid && req_ready) begin
      case (req_type)
        REQ_READ: begin
          // BUG 1: grants a shared copy without recording WHICH agent got it.
          //        There is no per-agent state anywhere in this design.
          line_state_q[req_line] <= L_SHARED;
          rsp_data               <= mem_data;
          rsp_grant              <= GRANT_SHARED;
        end
        REQ_WRITE: begin
          // BUG 2: grants write permission WITHOUT invalidating other copies.
          //        Nothing here knows other copies exist, because BUG 1 never
          //        recorded them.
          line_state_q[req_line] <= L_MODIFIED;
          rsp_grant              <= GRANT_EXCLUSIVE;
        end
      endcase
    end
    // BUG 3: no ordering point. Two requests for the same line from different
    //        agents are handled independently in the cycles they arrive.
  end
end

Architecture. A request forwarder with a single global state per line. It has no concept of which agent holds a line, so it cannot invalidate anything — and the absence is invisible because the code never needs the information it does not have.

State. line_state_q — one state per line, not per agent per line. That dimensionality error is the defect in one word. A coherence directory is [line][agent]; this is [line].

Event. Requests handled on arrival. GRANT_EXCLUSIVE is issued with no invalidation exchange preceding it, so the grant means nothing.

Contract. A requester receiving GRANT_EXCLUSIVE believes it is the only agent with write permission. The bridge cannot make that true and does not check it. The strongest promise in the protocol is the one nothing enforces.

Failure — the timeline. Two agents write the same line. The link is perfect throughout.

First divergence: cycle 15 — the moment a second exclusive grant was issued while the first was outstanding. Everything after is consequence, and cycle 60's lost write is the first thing anyone will notice, 45 cycles later.

Root cause. The bridge grants a promise it has no mechanism to keep. GRANT_EXCLUSIVE asserts a global property — no other agent may write this line — from a component holding no information about other agents. BUG 1 is why BUG 2 is possible: without per-agent state there is nothing to invalidate, so the invalidation step does not merely fail, it is unrepresentable.

Why it survives everything. Single-agent tests pass perfectly. Multi-agent tests pass unless two agents contend for the same line in a window of a few cycles. Random stimulus across a large address space almost never generates that contention — so the bug's exposure probability falls as the address space grows, and a bigger, more realistic test is less likely to find it.

DV/debug. The symptom is a lost update discovered far downstream, and the misconception supplies the wrong first hypothesis: "the interconnect dropped a writeback." The link's counters are clean because the link did its job. The discriminator is §11's invariant — were two agents ever simultaneously in a writable state for one line? — which is a single check the design has no state to answer.

11. Corrected — a Serialisation Point and a Single-Writer Invariant

// CORRECT. ILLUSTRATIVE (§10). Three changes, and the first is the one that
// makes the others possible:
//   (a) state is PER LINE PER AGENT — a directory, not a single state;
//   (b) a conflicting request is SERIALISED, not handled on arrival;
//   (c) exclusive access is granted only AFTER other copies are invalidated.
typedef enum logic [2:0] {
  H_IDLE, H_SNOOP, H_WAIT_ACK, H_GRANT
} hstate_e;
 
// The directory: which agents hold this line, and is it writable anywhere.
logic [N_AGENT-1:0] dir_shared_q   [N_LINE];   // agents holding a readable copy
logic [N_AGENT-1:0] dir_modified_q [N_LINE];   // agents holding write permission
hstate_e            home_state_q;
logic [LINE_W-1:0]  active_line_q;
logic [AGENT_W-1:0] active_req_q;
logic [N_AGENT-1:0] snoop_pending_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    home_state_q <= H_IDLE;
    for (int i = 0; i < N_LINE; i++) begin
      dir_shared_q[i]   <= '0;
      dir_modified_q[i] <= '0;
    end
  end else begin
    unique case (home_state_q)
      // SERIALISATION. One conflicting request for a given line is in flight at
      // a time. This is the ordering point the wrong version has nowhere.
      H_IDLE: if (req_valid && !line_busy(req_line)) begin
                active_line_q <= req_line;
                active_req_q  <= req_agent;
                if (req_type == REQ_WRITE) begin
                  // Invalidate every OTHER holder before granting anything.
                  snoop_pending_q <= (dir_shared_q[req_line] |
                                      dir_modified_q[req_line]) &
                                     ~agent_onehot(req_agent);
                  home_state_q    <= (|((dir_shared_q[req_line] |
                                         dir_modified_q[req_line]) &
                                        ~agent_onehot(req_agent)))
                                     ? H_SNOOP : H_GRANT;
                end else begin
                  home_state_q <= H_GRANT;      // shared read: no invalidation
                end
              end
 
      H_SNOOP: home_state_q <= H_WAIT_ACK;      // snoops issued to holders
 
      // Grant only when EVERY snoop has been acknowledged. Granting on a
      // timeout here would reintroduce §10's bug with extra steps.
      H_WAIT_ACK: begin
                snoop_pending_q <= snoop_pending_q & ~snoop_ack_vec;
                if ((snoop_pending_q & ~snoop_ack_vec) == '0)
                  home_state_q <= H_GRANT;
              end
 
      H_GRANT: begin
                if (req_type_q == REQ_WRITE) begin
                  // Exactly one agent is now modified; all others are invalid.
                  dir_modified_q[active_line_q] <= agent_onehot(active_req_q);
                  dir_shared_q[active_line_q]   <= '0;
                end else begin
                  dir_shared_q[active_line_q]   <=
                    dir_shared_q[active_line_q] | agent_onehot(active_req_q);
                end
                home_state_q <= H_IDLE;
              end
 
      default: home_state_q <= H_IDLE;
    endcase
  end
end
 
// MANDATORY. English: at most one agent holds write permission for a line at
// any time. This is THE coherence invariant — it is what §10's cycle 15
// violates, and it is checkable with one $countones.
a_single_writer: assert property (
  @(posedge clk) disable iff (!rst_n)
    $countones(dir_modified_q[chk_line]) <= 1
);
 
// MANDATORY. English: a line is never simultaneously modified at one agent and
// shared at another. Catches an invalidation path that clears the wrong vector
// — a plausible edit that leaves stale readable copies behind a writer.
a_no_shared_with_modified: assert property (
  @(posedge clk) disable iff (!rst_n)
    (|dir_modified_q[chk_line]) |-> (dir_shared_q[chk_line] == '0)
);
 
// MANDATORY. English: write permission is granted only after every outstanding
// snoop has been acknowledged. Catches the "grant on timeout" shortcut, which
// looks like robustness and is exactly §10's failure.
a_grant_after_snoops: assert property (
  @(posedge clk) disable iff (!rst_n)
    (home_state_q == H_GRANT) |-> (snoop_pending_q == '0)
);

Architecture. A directory ([line][agent]), a serialisation point (one conflicting request per line in flight), and an invalidate-before-grant sequence. Those three are the minimum, and none of them is transport.

State. dir_shared_q and dir_modified_q are N_LINE × N_AGENT bits. That is the storage cost the belief assumes away entirely — and sizing it is a real architectural decision, which is why §9 row 3 lists it.

Event. A request is admitted only when its line is not already busy. Grant happens on snoop-acknowledgement completion, not on a timer.

Contract. Requesters must acknowledge snoops and must not write without a grant. The protocol is bidirectional obligation — the wrong version's requesters had no obligations at all, which is another way of saying there was no protocol.

Failure. Two realistic residual defects. Granting on a snoop timeout reintroduces §10 exactly — a_grant_after_snoops exists for that edit. And a line_busy check that is too coarse serialises unrelated lines and destroys throughput, which is the performance half of §13.

DV/debug. a_single_writer is the property, and it is an excellent formal target: small state, one line at a time, and it is the exact invariant. Bind it to a symbolic line index rather than all lines (27.3 §19). In simulation it must be paired with stimulus that actually contends — §12 — or it passes vacuously forever.

12. DV Consequence

Three readings.

Row 3 is a genuinely counter-intuitive property worth stating plainly. Making the test more realistic makes this bug less likely to appear. Uniform random addresses across a large space almost never produce two agents contending for one line within a few cycles — so the environment's realism actively hides it, and a directed contention test is mandatory rather than supplementary.

Row 1 is the misconception infecting the testbench. If the link provides coherence, checking the link is checking coherence — so the scoreboard verifies that every message arrived intact and in order, which it did, at cycle 60 when memory received the wrong value.

And row 4 is the fix that costs almost nothing. $countones(dir_modified) <= 1 is one line and it fires at §10's cycle 15 rather than at the lost write 45 cycles later. An invariant on the state is worth more than any amount of transaction checking, because it catches the illegal condition rather than its eventual consequence.

13. Performance and Integration Consequence

Coherence is not free, and the belief budgets zero for it.

And the integration consequence follows 27.5 §8's pattern precisely. Two dies, each designed by a team that believed the interconnect provided coherence, each therefore containing no home agent. Both pass their own verification — there is nothing locally wrong with either — and the assembled product corrupts data under contention. The contract row that prevents it is one line: where is the point of coherence, and who implements it?

Three readings.

Row 2 puts coherence in the critical path in a way the belief cannot anticipate. A write that must wait for invalidation acknowledgements from remote agents has a latency set by the slowest remote responder — which is a 26.3 §17-style maximum over participants, not an average.

Row 5 is why a directory is a performance structure as much as a correctness one (§9 row 3). Without one, a correct implementation broadcasts, and broadcast cost grows with agent count — so the naive-correct design also does not scale.

And the integration case is the most likely real-world form of this failure, because it requires no individual mistake. Two reasonable teams, one shared misconception, no home agent between them — and the review question that catches it is 27.1 §6 A3, asked once.

14. Debug Consequence

Three readings.

The reproduction difficulty compounds the wrong hypothesis. The bug needs same-line contention in a narrow window, so it appears once per hours of running, and every attempt to instrument it changes the timing that produces it.

Clean link counters plus corrupted data is the signature to recognise (21.6 §14): delivered perfectly, decided incorrectly. It is the same shape as 28.1 §13's duplicate — the transport is exonerated by its own clean evidence, and the misconception refuses to accept the exoneration.

And the observability requirement is a 27.6 item that must exist before tapeout: a sticky record of any cycle in which more than one agent held write permission. One bit, set once, and it converts a weeks-long investigation into a register read — but only if someone believed the condition was possible enough to instrument it.

15. Which Review Gate This Corrupts

And A3 is the cheapest question in Module 27 relative to what it prevents. 27.1 §6 marks it as the question that changes the difficulty class of everything downstream — and an architect holding this belief answers "the interconnect handles that", which is 27.1 §4's "it's handled" in its most costly form.

16. Whiteboard Question

"Two dies, each with a cache, joined by a perfect error-free link. Are they coherent? Draw what you'd need to add."

A weak answer hesitates on perfect and looks for a trick.

A strong answer answers immediately and shows why (§7): "No. Both read line X and cache it; A writes 9 locally; B writes 4 locally; now there are three values for one address and the link never made an error. So the link was never the mechanism — improving it to infinity changes nothing."

Then draw the four things that are missing, not one: per-agent line state at each cache; a home agent that serialises conflicting requests to a line; a directory or snoop mechanism so the home agent knows who holds copies; and an invalidate-before-grant exchange. Say that only the last of these involves the link at all, and only to carry messages.

Then the invariant, because it is what makes the design checkable: "and the property I'd assert is single-writer — $countones(modified_by_agent) <= 1 per line. That's the thing cycle 15 violates, and it's a good formal target."

And the closing observation that shows you have read real systems (26.1 §4): "which matches what shipping products document — in AMD's MI300 documentation the I/O die maintains cache coherency. A die does it, not the interconnect between dies."

17. Interview Ladder

Weak candidate answer. "UCIe supports CXL, so it gives you coherency between chiplets."

Why the interviewer probes. The answer confuses carries with provides, and that distinction is the entire competence being tested. An engineer who has actually built a coherent multi-agent system cannot make this error, because they had to instantiate the agents.

Follow-up. "If I connect two dies over a perfect link, are they coherent?"

Strong revised answer. "No — and the perfect link is the point. Coherence is a protocol between agents with a serialisation point. You need caching agents tracking per-line state, a home agent that orders conflicting requests, a directory or snoop mechanism to find other copies, and invalidation before granting write permission. The link carries those messages. Two caches on a flawless link with no protocol diverge in one cycle, and nothing errors."

The senior addition: "and the failure mode is why it matters — it's silent. No error, no drop, clean link counters, a lost update discovered far downstream under contention. The invariant to assert is single-writer per line, and it's a good formal target."

The addition that shows product literacy (§5): "the documented practice agrees — AMD's MI300 documentation says the I/O die maintains cache coherency, and NVIDIA attributes full coherence across its two dies to a shared L2. In both cases a piece of logic with state provides it, not the interconnect."

Three weak answers and what is incomplete about each.

18. How to Remember the Correct Model

Coherence is a decision, not a delivery.

The decision is who may write this line now, and it is made by a component with state about every agent. A link has state about two endpoints, which is definitionally not enough.

And the operational form for a review: name the point of coherence. If the answer is a wire, a protocol name, or "the interconnect", it has not been designed. 26.1 §4's documented answer is a die, and that is the shape a correct answer has.

The memory hook for the failure: a perfect link and two caches diverge in one cycle. If you remember only the counterexample, you can rebuild the rest.

19. Understanding Check

20. Summary

Six things.

Coherence is a decision, not a delivery (§18). The decision is who may write this line now, made by a component with state about every agent — and a link has state about two endpoints.

The counterexample needs no product and no numbers (§7): two caches, a perfect link, no protocol, three values for one address in three cycles. The link was flawless, so it was never the mechanism.

Five of six requirements are not transport (§6): per-agent line state, a home agent, a directory or snoop mechanism, invalidation, and a stated memory model. Only message-carrying is the link's job.

In RTL it is a dimensionality error (§10): [line] state where coherence needs [line][agent]. Without per-agent state there is nothing to invalidate, so a second exclusive grant is issued at cycle 15 and a write is lost at cycle 60, with clean link counters throughout.

The invariant is one line (§11): $countones(dir_modified[line]) <= 1 — and realistic random stimulus actively hides the bug (§12), so directed same-line contention is mandatory.

And the documented practice agrees (§5): in shipping products, a die maintains coherency and a shared cache provides it. Name the point of coherence — if the answer is a wire, it has not been designed.