Shared Memory Across CXL

13.1 built the permission model on an assumption it never stated: that a snoop and its response are adjacent in time.

Across a link they are not, and every mechanism in that chapter has to survive the gap.

1. The Engineering Problem — The Same Rules, Much Further Apart

Nothing in the previous chapter is repealed here. SWMR still holds. A read still returns the latest write in the coherence order. Dirty data must still survive eviction. Every invariant carries over unchanged.

What changes is time, and time changes three things that were invisible on one die.

Every protocol step becomes a window. Revoking a remote agent's permission is no longer a signal that settles within a cycle — it is a message that crosses a link, is processed, and is acknowledged back. Measured on a four-cycle link, that round trip leaves the old permission alive for four cycles, and everything the protocol does during that window has to be safe.

Responses stop arriving in order. Once several requests are outstanding across a link, the order in which answers come back is not the order in which questions were asked. A design that matches an answer to the oldest outstanding question is right only while it is lucky.

And the protocol starts depending on something it does not provide. Coherency has no retransmission of its own. A dropped invalidation is not a slow invalidation — it is a protocol that stops. The timeout and the retry are somebody else's obligation, and the coherency layer has to be explicit about depending on them.

2. The One-Sentence Model

Distance turns a step into a window. The rules are unchanged; what changes is how long the old world persists after you have decided to replace it, and everything that can go wrong across a link goes wrong inside that window.

Call it same rules, longer windows. Every defect in this chapter is a window that was not waited out, or an identity that was not checked inside one.

3. What This Chapter Owns

Deferred:

This chapter is about consequences, not mechanisms. It introduces no new invariant. Every model below takes something 13.1 treated as instantaneous and gives it a duration, then asks what has to be true for the invariant to survive.

4. Teaching-Model Boundary

The risk in this chapter is inventing protocol. There are no opcodes, no message names, no channel structure and no ordering rules attributed to CXL below. Where a mechanism is needed — a tag, a timeout, a bias flip — it is built here and labelled.

What is transferable is the shape of the problem. Any coherency protocol crossing any link faces the same four obligations: wait out the window, match by identity, serialise conflicts at one point, and depend explicitly on reliable delivery. Those hold whether the link is CXL, a coherent fabric, or two chiplets.

5. RTL 1 — The Link Is A Delay

// Everything in 13.1 assumed a snoop and its response were adjacent in time.
// A link makes them N cycles apart, and every one of those cycles is a window
// in which the old permission still exists.
module link_delay #(parameter int LAT = 4) (
  input  logic       in_valid, input logic [2:0] in_tag, input logic [7:0] in_data,
  output logic       out_valid, output logic [2:0] out_tag, output logic [7:0] out_data,
  output logic [3:0] in_flight,
  output logic [7:0] n_sent, n_arrived
);

Measured: a message sent with tag 1 and payload 0xAA shows in_flight=1 immediately, is absent from the output for three cycles, and arrives on the fourth carrying its tag and payload intact.

That is the entire model, and it is deliberately trivial, because the interesting part is not the link — it is what every other mechanism has to do about it.

The tag matters more than the payload. A link that delivers data without preserving which request the data belongs to has not delivered anything usable, which is section 7.

6. Waveform — The Exposure Window

Transcribed from the printed cycle trace of the cross-link model in section 11.

The exposed row is the whole chapter. It is high from the moment the local agent decides it wants exclusive access until the moment the remote acknowledgement returns — and throughout that time the remote agent holds a copy it believes is valid and will serve reads from.

Nothing here is wrong. The design waits, the invariant holds, and the grant count rises only at cycle 6. The point is that the window exists at all, and that its length is a property of the link rather than of the protocol.

7. RTL 2 — Responses Arrive Out Of Order

// Once responses can be in flight for several cycles they can also arrive in a
// different order from the requests. Matching by ARRIVAL ORDER is the bug;
// matching by IDENTITY is the design.
//
// MATCH_BY_ORDER is a verification-only hook, default 0, so the wrong design
// can be demonstrated rather than described.
  logic [2:0] sel_tag;
  assign sel_tag = (MATCH_BY_ORDER != 0) ? order_q[head_q[2:0]] : resp_tag;
  assign mismatch = resp_en && (sel_tag != resp_tag);

Four requests are issued with tags 0, 1, 2 and 3. The response for tag 3 comes back first. Measured, side by side, on two instances of the same module:

The order-matching instance retires the oldest outstanding request — which was never the one the response named. It is not merely inefficient; it completes the wrong transaction and leaves the right one outstanding forever.

Two rejection cases are also distinguished, because they are found in different places:

Measured at 1 each, and a rejected response completes nothing: the completion count stays at 4.

Tag 0 is used deliberately in that test. A count or a scan that skips index zero is invisible if index zero is never occupied, and a mutation doing exactly that survived the first run until the stimulus was changed.

8. RTL 3 — Measuring The Exposure Window

      if (revoke && !open_q) begin open_q <= 1'b1; cnt_q <= 8'd0; end
      else if (open_q) begin
        if (ack) begin
          open_q <= 1'b0;
          // The maximum is LATCHED. A window that has closed is exactly the
          // one that explains a stale read nobody can reproduce.
          if (cnt_q > window_max) window_max <= cnt_q;
        end else begin
          cnt_q <= cnt_q + 8'd1;
          // A window far longer than the link round trip means the
          // acknowledgement is not coming, not that the link is slow.
          if (cnt_q >= (LAT[7:0] * 8'd4)) still_exposed_err <= 1'b1;
        end
      end

Measured: a window held open for 6 cycles closes on the acknowledgement, and the maximum latches at 6. A subsequent shorter window does not reduce it.

The distinction between a slow window and a stuck one is a threshold, and the threshold is derived from the link. A window of four cycles on a four-cycle link is expected. A window of twenty is not slow — it means the acknowledgement is never coming, and it is reported as still_exposed_err rather than waited on indefinitely. Measured at 1 with the line still marked exposed.

9. RTL 4 — Two Directions, One Manager

  // A device reaching host memory is the caching direction.
  assign use_cache_proto = acc_en &&  initiator && !target_is_device;
  // A host reaching device-attached memory is the memory direction.
  assign use_mem_proto   = acc_en && !initiator &&  target_is_device;
  // Host to host memory, or device to its own memory, needs no cross-link
  // coherency protocol at all.
  assign local_access    = acc_en && (initiator == target_is_device);
  // The host manages coherency in BOTH cross-link directions -- including for
  // memory that is physically attached to the device.
  assign host_manages    = 1'b1;

The four combinations, measured, with exactly one classification per access:

The row people find surprising is the second one. A host reaching memory that is physically attached to a device still has the host managing coherency for it. Physical attachment and coherency management are different questions, and conflating them is the most common misreading of the CXL protocol split.

Measured: 1 caching-direction access, 1 memory-direction access, 2 local, with host_manages asserted in both cross-link cases.

10. RTL 5 — Bias: Whose Memory Is It Anyway

  // In device bias the device reaches its own memory without crossing the link.
  assign dev_direct   = dev_acc && bias_q && !flip_q;
  // The host may only reach it in host bias -- otherwise the device's private
  // view and the host's view could diverge.
  assign host_allowed = host_acc && !bias_q && !flip_q;

Measured behaviours:

The flip is the interesting part. Requested while two host accesses are still outstanding, it does not take effect — measured, the bias stays in host mode for three further cycles and completes only when the outstanding count reaches zero. Flipping early would leave the device operating on its own private view while host traffic that assumed the coherent view was still in flight.

A flip requested again while traffic is outstanding raises unsafe_flip_err, measured at 1.

This is a concept model. It captures the idea that device-attached memory can be operated coherently or privately, and that switching between those modes is a drain-and-swap rather than a mode bit. No CXL mechanism, encoding or timing is claimed.

11. RTL 6 — SWMR Across A Link

  assign inv_sent    = want_write && remote_holds_read && !wait_q;
  // The grant waits for the acknowledgement to come BACK across the link.
  assign grant_write = wait_q && inv_ack;
  // The invariant: a write grant while the remote agent still holds read.
  assign swmr_broken = grant_write && remote_q;

The invariant is character-for-character the one from 13.1. What changed is the wait: measured, the write request waits 8 cycles for the invalidation to cross, be processed, and be acknowledged back, and swmr_broken never fires.

12. RTL 7 — Coherency Assumes Reliable Delivery

// Coherency has no retransmission of its own: a dropped invalidation does not
// degrade performance, it stalls the protocol forever. The timeout and retry
// are a LINK-layer obligation.
  assign timed_out = busy_q && (age_q >= TIMEOUT[7:0]);

An invalidation is sent and dropped. The acknowledgement is offered every cycle and cannot help, because the message never arrived. Measured: 12 cycles of waiting, then timed_out=1, a retry, and the retry is acknowledged — with stalled_err recorded.

This is the dependency the coherency layer must state out loud. Nothing in the permission model retransmits anything. Without a timeout the protocol does not run slowly — it stops, holding a window open forever, which is precisely the still_exposed_err condition from section 8 seen from the other side.

The measured design also refuses to acknowledge a dropped message: the ack_in && !dropped_q term means an acknowledgement offered for a message that never landed does nothing, which is what makes the timeout the only exit.

13. RTL 8 — What To Count Across A Link

      if (sample_en) begin
        n_samples <= n_samples + 8'd1;
        // Both peaks are latched. Across a link the interesting events are
        // transient and a current-value counter reports none of them.
        if (flight_now > peak_flight) peak_flight <= flight_now;
        if (window_now > peak_window) peak_window <= window_now;
      end

Measured: in-flight counts of 2, 7 and 1 sampled in that order leave peak_flight=7; exposure windows of 5, 11 and 2 leave peak_window=11. Idle cycles do not count as samples.

Across a link, every number worth knowing is a peak. The average in-flight count tells you the link is comfortable; the peak tells you whether the outstanding-request table ever ran out of entries. The average window tells you the link is fast; the peak tells you how long a stale copy could have survived.

14. RTL 9 — The Serialisation Point

// Once requests cross a link they can arrive in any order, so "who asked first"
// is not a well-defined question. ONE place must decide the order for a line,
// and both requesters must accept that decision.
  assign conflict = (req_a && req_b) || (req_a && (own_q == 2'd2)) || (req_b && (own_q == 2'd1));
  assign both_granted_err = grant_a && grant_b;

Two agents request exclusive access to the same line in the same cycle. Measured:

A conflict is a normal operating condition, not an error. What would be an error is two agents each believing they won, which is the cross-link form of the two-writer failure from 13.1 — and both_granted_err exists to catch it, with its own fault-injection hook because the single-owner encoding makes it unreachable otherwise.

The tie-break rule itself does not matter. Any deterministic rule works. What does not work is having no rule, or having a different rule at each end of the link — which is exactly what "who asked first" degenerates into once messages can overtake each other.

15. RTL 10 — Did It Actually Work?

Every other model in this chapter verifies a mechanism. This one verifies the outcome.

  // A read issued after a write committed must not return an older value.
  assign stale_read = rd_return && rd_after_write && (rd_val != last_q);

Measured: a read returning the committed value is clean; a read returning an older value after a commit is stale, with the gap recorded. A larger gap latches a new worst case of 8, and a smaller later gap does not reduce it.

The rd_after_write qualifier is doing real work. A read that was not ordered after the write is not stale no matter what it returns — it simply happened earlier in the coherence order. Removing that term turns every ordinary read into a violation, and it is a mutation that survived until the testbench included a read with the qualifier low.

16. Quantitative Reasoning

The exposure window is the link round trip, and it is the number that changes.

window = send + remote processing + acknowledge

Measured on a four-cycle link: 4 cycles in the waveform, 8 cycles in the cross-link model with a longer remote turnaround. On one die the same step is a cycle. Nothing about the protocol changed; the duration did.

Outstanding requests needed to hide the window:

requests_in_flight >= window / issue_interval

Measured peak in-flight of 7 against a peak window of 11 — a design sized for the average would have run out of tags at the peak, and the peak is the only number that reveals it.

Invalidation cost across a link. The messages are the same as 13.1 — one per sharer — but each now carries a round trip, and the grant waits for the slowest of them:

upgrade latency = max over sharers of (round trip to that sharer)

That is a maximum, not a sum and not a mean, which is why a single distant sharer costs as much as many near ones.

Timeout threshold. The measured design uses a threshold derived from the link rather than a constant: a window beyond four times the round trip is treated as stuck rather than slow. Measured, a dropped message times out after 12 cycles and is retried once.

The direction split has no latency in it at all. Two of the four access combinations need no cross-link protocol — measured, 2 of 4 local. The cheapest coherency traffic is the traffic that does not cross.

17. Assertions

Presented as SystemVerilog and executed as procedural checkers — see section 19.

No write grant while a remote copy exists.

property p_cross_link_swmr;
  @(posedge clk) disable iff (!rst_n)  grant_write |-> !remote_holds_read;
endproperty

A response retires the request it names.

property p_identity_match;
  @(posedge clk) disable iff (!rst_n)  complete |-> (complete_tag == resp_tag);
endproperty

Exactly one classification per access.

property p_one_direction;
  @(posedge clk) disable iff (!rst_n)
    acc_en |-> ($countones({use_cache_proto, use_mem_proto, local_access}) == 1);
endproperty

The host manages coherency in both cross-link directions.

property p_host_manages;
  @(posedge clk) disable iff (!rst_n)
    (use_cache_proto || use_mem_proto) |-> host_manages;
endproperty

The host never reaches device memory in device bias.

property p_bias_exclusion;
  @(posedge clk) disable iff (!rst_n)  host_allowed |-> !device_bias;
endproperty

A bias flip completes only with nothing outstanding.

property p_flip_drains;
  @(posedge clk) disable iff (!rst_n)
    $rose(device_bias) |-> $past(outstanding == 0);
endproperty

Never two owners of one line.

property p_single_owner;
  @(posedge clk) disable iff (!rst_n)  !(grant_a && grant_b);
endproperty

A read ordered after a commit returns the committed value.

property p_no_stale;
  @(posedge clk) disable iff (!rst_n)
    (rd_return && rd_after_write) |-> (rd_val == last_committed);
endproperty

A window beyond the link budget is reported, not waited on.

property p_window_bounded;
  @(posedge clk) disable iff (!rst_n)
    (window_now >= LAT*4) |-> still_exposed_err;
endproperty

109 assertion sites across the ten models and the waveform trace. All pass.

18. Mutation Testing

57 mutations injected, 57 killed, 0 surviving.

Representative kills:

Ten mutations did not die on the first run.

Two are worth naming. The slot-zero count is the 12.4 lesson recurring in a new shape: a scan that skips index zero is invisible if index zero is never occupied, and the fix was to use tag 0 deliberately. And one mutation — a reissued tag keeping its done marking — turned out to be provably equivalent, because that bit only matters while the slot is not live and a reissued-then-completed tag ends marked either way. It was replaced with a mutation that changes behaviour rather than recorded as a survivor.

19. Verification Strategy

Tool reality. Icarus Verilog 13.0 only. No concurrent SVA, so section 17's properties are executed procedurally. Two Icarus-specific constraints bit during authoring: a part-select of a function call's result is illegal and needed an explicit temporary, and a zero-argument function had to become a task.

A signal-name collision between two DUTs in one testbench — n_blocked and n_retry appearing in both the bias model and the telemetry bank — was caught at elaboration and fixed by naming the ports explicitly at the instance.

Independent oracles.

• resp_matcher — design: a live-bit array indexed by tag. Oracle: an explicit tag set, with order deliberately not represented.
• direction_split — design: boolean expressions. Oracle: a literal four-row truth table.
• cross_link_swmr — design: a wait flag and counter. Oracle: expected wait length stated per case.
• link_reliability — design: age and drop flags. Oracle: expected timeout and retry counts.
• conflict_order — design: an owner register and queue bits. Oracle: expected winner and grant order per sequence.

The response-matcher oracle is the important one: it holds outstanding requests as a set, with no notion of order at all, because order is exactly what must not matter. An oracle that kept a queue would encode the bug.

Coverage. The points that matter are link latency, outstanding depth, response order, direction, bias state, and conflict shape. The crosses worth driving are outstanding depth crossed with response order — the only way to reach the out-of-order case with more than one candidate — and bias state crossed with initiator, which is what separates the four access classifications. A cross of tag value against latency is noise.

20. Synthesis and Implementation Reality

The outstanding-request table is the new structure, and it is sized by the window.

entries >= peak in flight
tag width = log2(entries)

Measured peak in-flight of 7 on a four-cycle link. Under-sizing it does not degrade throughput gracefully — it stalls issue entirely, because a request with no tag cannot be sent.

Every message carries identity, and identity is width on every wire. The tag travels out with the request and back with the response, so it costs bits in both directions plus a comparator at the return path. That is the price of not matching by arrival order, and it is unavoidable.

Timers scale with outstanding requests, not with the link. One age counter per outstanding request, not one for the link, because each request times out independently. That is entries × log2(timeout) bits of storage doing nothing most of the time and preventing an indefinite stall the rest of it.

The bias flip is a drain, and drains need a counter that cannot be wrong. The outstanding count gating the flip is the same class of derived counter 12.1 warned about: if it can drift low, the flip completes early and the two views diverge. It should be derived from the same structure that issues the requests, not maintained separately.

The serialisation point is a single-owner register and a queue, and its cost is trivial. What is not trivial is guaranteeing there is only one of it per line — that is a topology property, not an RTL one, and it is where Modules 15 and 16 pick up.

No gate counts are offered. The structural claims — table sized by peak in-flight, tag width on both directions, one timer per outstanding request — hold regardless of process.

21. Silicon Observability

n_stale sits at the bottom deliberately. Every other counter measures a mechanism; that one measures whether the mechanisms achieved anything. A system with clean mechanism counters and a non-zero stale count has a gap nobody has modelled.

22. Debug Lab

  out-of-order response : identity match -> tag 3, mismatch=0
  order match (broken)  : retired tag 0 instead, mismatch=1

assign sel_tag  = resp_tag;                       // the response names its request
assign mismatch = resp_en && (sel_tag != resp_tag);

  exposure window : now=6 max=6 (latched)

if (cnt_q > window_max) window_max <= cnt_q;   // latch the peak
assign grant_write = wait_q && inv_ack;        // and never grant on elapsed time

  dropped invalidation : timed_out=1 retries=1

assign timed_out = busy_q && (age_q >= TIMEOUT[7:0]);
// and the acknowledgement for a message that never landed must do nothing:
if (ack_in && !dropped_q) begin ... end

  direction: device->host mem = cache proto (1), host->device mem = mem proto (1)

assign host_manages = 1'b1;    // in BOTH cross-link directions

  bias : flips=1 device_direct=1 host_blocked=1 unsafe_flip=1

else if (flip_q && (outstanding == 4'd0)) begin
  bias_q <= ~bias_q; flip_q <= 1'b0;
end

  simultaneous request : conflict=1 winner=1 both_granted=0 B queued=1

assign both_granted_err = grant_a && grant_b;   // and a fixed, deterministic tie-break

  unissued response : unknown_resp=1 ; duplicate: dup_resp=1

assign complete     = resp_en && live_q[sel_tag];   // only a LIVE slot completes
assign dup_resp     = resp_en && !live_q[resp_tag] &&  done_q[resp_tag];
assign unknown_resp = resp_en && !live_q[resp_tag] && !done_q[resp_tag];

  end-to-end staleness : stale reads=1 worst gap=1 reads=7

assign stale_read = rd_return && rd_after_write && (rd_val != last_q);

23. Design Review

What a reviewer should attack first.

Whether every message carries identity. If a response can be matched to a request by arrival order anywhere in the design, it will complete the wrong transaction the first time the link reorders — and the link will.

Whether the grant waits on the acknowledgement or on time. A grant gated on elapsed cycles is a guess about the remote agent's speed, and it will be wrong under load.

Whether the window is latched. A stale read is a past-tense event. Without a latched peak, the evidence is gone before anyone looks.

Where the timeout lives. Coherency has no retransmission. Ask which layer provides it and what happens if that layer does not.

How many serialisation points exist per line. One is correct. Two is two agents each believing they won.

Whether the bias flip drains. A mode bit that flips on request rather than on a drained count produces two views of the same memory.

Whether anything measures the outcome. Mechanism counters can all be clean while the data is wrong.

What is deliberately not here. No message names, opcodes, channels or credits — none is citable. No transient state storage: a line waiting on a remote acknowledgement is neither S nor M, and 13.4 is where that becomes a table. No ownership transfer between host and device — 13.3. No topology, no switch, no fabric manager — Modules 15 and 16.

24. How This Appears In Real Engineering

In architecture review, the number that decides the design is the peak outstanding count, and it is almost always estimated from the average. The measured contrast — peak in-flight 7 against a peak window of 11 — is the shape of that mistake: a table sized for the mean stalls issue at the peak, and issue stalling looks like a link bandwidth problem.

In RTL design, matching by arrival order is the defect that survives the longest, because it works perfectly until the link reorders something. Directed tests almost never reorder, and the first reordering in production completes the wrong transaction.

In verification, the trap is that the interesting cases all require concurrency plus latency. A test with one outstanding request cannot reorder, cannot conflict, and cannot expose a window. The stimulus has to keep several requests alive at once and vary their return order deliberately.

In bring-up, the lost-message hang is the one that looks like a hardware fault and is a missing timeout. One line becomes permanently unusable, no alarm fires, and the system otherwise runs.

In system integration, the direction confusion is the recurring planning error: memory physically attached to a device is not therefore managed by the device, and capacity plans that assume otherwise mispredict both latency and coherency traffic.

25. Common Misconceptions

"A link just makes coherency slower." It makes every protocol step a window, and windows are where the bugs live. The measured exposure window is time during which the old permission is still real.

"Responses come back in order." Measured false: the response for tag 3 arrived before those for tags 0, 1 and 2, and a design matching by order retired tag 0 for it.

"Coherency handles lost messages." It has no retransmission at all. A dropped invalidation stalls the protocol permanently — measured, it took a 12-cycle timeout and a retry to recover.

"Memory attached to the device is managed by the device." Measured false in the direction model: the host manages coherency in both cross-link directions, including for device-attached memory.

"Bias is a mode bit." It is a drain and a swap. Flipping with traffic outstanding leaves two views of the same memory, and the model refuses to complete the flip until the outstanding count reaches zero.

"Whoever asked first wins." Across a link there is no shared notion of first. One serialisation point decides, and both ends accept it.

"The average window is what matters." The peak is what matters. Measured peak_window=11 against samples of 5 and 2 — the average describes a system that never had a problem.

"Clean mechanism counters mean correct behaviour." Every mechanism counter can be clean while an end-to-end stale read is happening. That is what the outcome check is for.

26. Interview Reasoning

27. Exercises

1. Calculation. A link has a 40-cycle round trip and a request can be issued every 4 cycles. Compute the outstanding-request table depth needed to keep the link busy, the tag width that implies, and the total tag storage if the tag travels in both directions.

2. Analysis. A design reports a mean exposure window of 6 cycles and a latched peak of 45 on a 10-cycle-round-trip link. State what the mean suggests, what the peak suggests, and which of the two would appear in a stale-read investigation.

3. RTL task. Extend resp_matcher so a tag cannot be reissued until its transaction has fully drained. State what state that requires, and the failure that becomes possible if a tag is reused one cycle early.

4. Assertion task. Write the property proving a response retires the request it names. Then explain why this property passes trivially on a design that matches by order when responses happen to return in order, and what stimulus is required to make it meaningful.

5. Design task. Add a second serialisation point so that two lines can be arbitrated in parallel. State what must be true about the mapping from line to serialisation point, and the failure that occurs if one line can reach both.

6. Testbench design. Design the stimulus that distinguishes identity matching from order matching. Explain why any test with a single outstanding request passes on both, and state the minimum number of concurrent requests required.

7. Debug task. A line becomes permanently inaccessible and no alarm fires. Give your investigation order, name the counter that separates a lost message from a slow one, and explain which layer owns the fix.

8. Design review. A colleague argues that because the host manages coherency for device-attached memory, that memory should be treated as ordinary host memory for capacity planning. Give the strongest version of that argument, then the measurement that shows what it misses.

28. Summary

Distance turns a step into a window.

• The rules are unchanged. SWMR, the coherence order and the writeback obligation carry over from 13.1 intact. Only the durations change.
• Every revocation is a window. Measured 4 cycles in the trace and 8 in the cross-link model, with the remote agent holding a readable copy throughout.
• Responses arrive out of order. With four requests outstanding and tag 3 returning first, identity matching retired tag 3 and order matching retired tag 0 — the wrong transaction, silently.
• Two rejections, two causes. unknown_resp for a tag never issued and dup_resp for one already completed, measured at 1 each, with a rejected response completing nothing.
• The window must be latched. Peak 6, unreduced by a later shorter window; a window beyond four round trips raises still_exposed_err rather than waiting forever.
• Two of four access directions cross the link, and the host manages coherency in both of them — including for memory physically attached to the device.
• Bias is a drain and a swap. Measured: the flip held while two accesses were outstanding, completed at zero, blocked the host in device bias, and raised unsafe_flip_err on a second request mid-flip.
• Coherency assumes reliable delivery it does not provide. A dropped invalidation needed a 12-cycle timeout and a retry; without one the protocol stops rather than slows.
• One serialisation point per line. A simultaneous request produced one winner, the loser queued, and 2 grants in a defined order.
• One check measures the outcome. Stale reads with a worst gap latched at 8, because every mechanism counter can be clean while the data is wrong.
• Verification: 109 assertion sites, 57 of 57 mutations killed, zero surviving. Ten first-run escapes were four stimulus gaps, three unobserved outputs, two unreachable checkers needing fault-injection hooks, and one provably equivalent mutation that was replaced rather than recorded.

Next: 13.3 Ownership in CXL, which takes the windows measured here and asks the question they exist to protect: who owns this line right now, and what has to happen for that answer to change.