CXL-over-UCIe Integration

Chapter 11.2 built address ownership. Chapter 11.3 built distributed coherence. Chapter 11.4 built the transport that carries both without changing their meaning.

Each of those chapters verified its own layer in isolation, and each ended with a working mechanism. This chapter is about the thing none of them could see on their own: a system in which all three mechanisms are individually correct and the composition is still wrong.

That is not a hypothetical. It is the normal way coherent chiplets fail, and the reason is structural. Each plane has its own state, its own lifetime, and its own idea of what "ready" and "done" mean — and nothing in any one plane forces it to agree with the others.

1. The One-Sentence Model

A coherent chiplet is a stack of contracts whose state must agree, and no plane can prove the agreement on its own.

Three contracts. Memory mapping: which addresses belong to which device. Coherence: who owns the newest copy of a line. Transport: can a semantic request or response cross UCIe exactly once. The system is correct only when all three hold simultaneously and consistently, and every debugging session in this chapter starts by asking which one broke first.

2. Sourcing, and a Distinction This Chapter Must Make Precisely

3. What Composition Adds That the Parts Did Not Have

It is worth being concrete about why this is a chapter rather than a summary, because "put the three together" sounds like an assembly step.

Each plane has a different notion of "ready". A memory window is ready when software has configured it. A coherence agent is ready when its per-line state has been initialised to a defined start. Transport is ready when the link is operational and the mapping is negotiated. These become true at different times, in an order nobody centrally controls, and admitting traffic on the wrong one is §8.

Each plane has a different notion of "done". Transport is done when delivery is confirmed. A memory transaction is done when its data returns and its entry retires. A coherence transaction is done when ownership has actually transferred — which may be later than either. §14 is what happens when a design picks the earliest of the three.

Each plane has a different notion of "reset". Transport state is per-link-epoch and should be re-baselined on recovery. Memory windows are per-configuration-epoch and should not be. Coherence line state is per-cache-line and outlives both. §17 is why a single reset signal across all three is a data-loss bug.

And no plane can validate another. This is the load-bearing observation, and §5 is a table of it.

4. The Three State Planes

The table this chapter is organised around.

Read the third column, not the second. The state itself is unremarkable — tables, counters, registers. The lifetimes are what make composition hard, because they are not merely different lengths; they are indexed by different things. A window is indexed by address range. A coherence entry is indexed by address. A replay entry is indexed by a transport sequence. A credit is indexed by nothing at all — it is a scalar count.

Nothing that is indexed differently can be reset together correctly, and that single sentence generates §17, §14, and most of §19's taxonomy.

5. Cross-Plane Validity — the Table Worth Memorising

The claim is that each plane's validity is independent. Here it is exhaustively, because the entries that surprise people are the ones that ship.

Two rows carry the whole section.

The fifth row is the one designs get wrong by being defensive. A cache line in a transient state while the transport is retrying is a normal, correct state of a working system. A design that treats "transient plus retry" as an error condition and tears something down converts a recoverable moment into an unrecoverable one.

The fourth row is the one that has no local symptom at all. Coherence is working perfectly — over the wrong memory. Transport is working perfectly — to the wrong die. The only detector is a model that predicts the target independently, which is §20.

One plane being valid does not prove another plane is valid. Every "the link is fine so the problem must be the device" conversation is this sentence being forgotten.

6. The Integrated Stack

One edge in that figure is worth pointing at. The arrow from the remote chiplet back to the local coherence agent is not a response — it is labelled probes, and it is the structural reason coherence cannot be reasoned about as a request/response pipeline. Events arrive at the coherence plane that no local transaction caused, which is Chapter 11.3's whole subject, and it means the coherence plane is never idle just because the local side is.

7. The Integration Readiness Vector

// ILLUSTRATIVE integration readiness. NOT a UCIe or CXL register. Each term
// is owned by a different plane and becomes true at a different time; writing
// them out is what makes "which one is low" a one-look debug question.
logic memory_map_valid_q;   // plane 1 — windows configured, targets resolved
logic coherence_ready_q;    // plane 2 — per-line state initialised to a defined start
logic transport_ready_q;    // plane 3 — link operational AND mapping negotiated
logic device_ready_q;       // the far side — capable, configured, media able to serve
 
assign coherent_path_ready =
    memory_map_valid_q  &&
    coherence_ready_q   &&
    transport_ready_q   &&
    device_ready_q;

Architecture. Four terms, four owners, four independent ways to be false. The value of the conjunction is not that it is clever — it is that each term names a mechanism you can go and inspect, so "why is the path not ready" becomes a lookup instead of an investigation.

State. Four bits with three different lifetimes: memory_map_valid_q is per-configuration-epoch, coherence_ready_q is per-coherence-epoch (the distributed one — Chapter 11.1 §25), transport_ready_q is per-link-epoch, device_ready_q is per-device. Three lifetimes in one four-bit conjunction is not sloppiness; it is the actual situation, and §13 is where it is laid out.

Cycle behaviour. Each term is set by its owner at its own point in bring-up. The conjunction is combinational. It may fall as well as rise — a link recovery clears transport_ready_q — and §9 is about what that must and must not do to traffic already accepted.

Contract. No CXL semantic transaction may be accepted unless this is asserted. Note the word: accepted, not completed. A transaction accepted while all four were true must survive one of them going false, which is exactly Chapter 11.2 §17's rule generalised across planes.

Failure. Any missing term admits traffic the system cannot serve. §8 is the specific case, and it is the fifth time this curriculum has met the shape.

DV. Each term false with the other three true — four directed cases. Then each term falling with traffic in flight, which is a different and harder set, because it tests §9 rather than §7.

8. Wrong RTL — Transport Ready Means CXL Ready

// WRONG — and this is the fifth appearance of this substitution.
assign cxl_ready = ucie_link_active;

Architecture. Count them: Chapter 10.3 §6 derived endpoint visibility from the link; Chapter 10.4 §7 derived host presence from a strap; Chapter 11.1 §18 derived CXL memory readiness from link-active; Chapter 11.2 §14 did it with a CPU load as the client. The shape recurs because the available signal is always more available than the correct one.

Cycle behaviour. One assignment, and in a bring-up where the link happens to come up last it even works.

Failure — and here it is worse than in any previous appearance, because three planes can be wrong at once.

The memory window may be invalid. A load into the range is routed nowhere, or to a stale target. Chapter 11.2 §10's failure.

The coherence agent may be uninitialised. Per-line metadata holding whatever the array powered up with reports lines valid that were never filled — so the very first coherent access returns uninitialised contents, and Chapter 11.1 §13's stale-read failure occurs immediately rather than after a race.

The far side's media may be unavailable. Chapter 11.2 §14.

And the failures compose. A store issued in this window may be routed to the wrong device, into memory that is not initialised, with coherence metadata that claims a copy nobody has. There is no single symptom — which is precisely why the readiness vector's value is diagnostic rather than protective.

DV. Hold each term low in turn while asserting the link, and verify no semantic transaction is accepted. Then assert all four and verify the same traffic completes. The pair is what proves the gate exists rather than that it happened to be true.

9. Readiness Must Be Sampled at Acceptance, Not Polled Continuously

A subtle point with a real bug behind it, and it follows from §7's observation that terms can fall.

// WRONG — readiness consulted continuously, including for work already accepted.
assign semantic_progress_allowed = coherent_path_ready;

Why it looks right. Gating everything on readiness seems maximally safe. If the path is not ready, do nothing.

Why it is wrong. A transaction accepted while the path was ready has created obligations on both dies. If transport_ready_q falls during a link recovery and the design stops making progress on that transaction including stopping its bookkeeping, then the transaction is neither completing nor failing — it is suspended in a way nothing will resolve. And if the design goes further and treats not-ready as a reason to discard, it is Chapter 11.2 §18's bug arriving through a readiness signal instead of a reset.

The right shape is two different uses of the same information:

// Illustrative — readiness gates ACCEPTANCE; a captured epoch governs
// already-accepted work. The distinction is the whole point.
assign semantic_accept_allowed = coherent_path_ready;      // gate new work
 
// Each accepted transaction records which epochs it was accepted under, so a
// later epoch change can be recognised rather than silently ignored.
typedef struct packed {
  logic [EPOCH_W-1:0] link_epoch;    // plane 3 — changes on recovery
  logic [EPOCH_W-1:0] cfg_epoch;     // plane 1 — changes on remap
} accept_epochs_t;
 
accept_epochs_t txn_epochs_q [MAX_SEMANTIC_TXN];
 
// A transaction whose link epoch no longer matches is not automatically
// broken — but it is a transaction whose outcome must be REASONED about
// rather than assumed (§17).
assign txn_epoch_stale[t] = txn_valid_q[t]
                         && (txn_epochs_q[t].link_epoch != link_epoch_q);

Architecture. An epoch counter per plane, captured per transaction at acceptance. It costs a few bits and it converts an invisible condition — "this transaction was accepted under a link that no longer exists" — into a readable one.

State. Per-semantic-transaction, allocated with the transaction. The epoch value itself is per-plane-epoch, so this structure deliberately joins two lifetimes, which is its purpose.

Cycle behaviour. Captured once at acceptance, compared continuously, never rewritten.

Contract. Recovery logic relies on being able to distinguish transactions that predate the current epoch from those that do not, because those two sets need different handling.

Failure. Without it, recovery must treat every outstanding transaction identically — which means either abandoning transactions that would have completed, or waiting on transactions that cannot.

DV. Force a link-epoch change with transactions outstanding from before and after the change, and verify each is classified correctly. Then do it for a configuration epoch, which is the case people forget: a memory window remapped under an outstanding request is a transaction whose target may no longer be its target.

10. One Memory Request, All Three Planes

Now the composition, traced. A CPU load to a CXL-attached address, with the plane owning each step named.

The coherence plane appears nowhere in that table, and that is the point. A CXL.mem access to a host-only-coherent range is a memory operation whose coherency flows the host manages — Chapter 11.1 §4's asymmetry. So one of the three planes is uninvolved in the most common transaction in the system, and a design that couples them will stall memory traffic on coherence state that has nothing to do with it.

Steps 3 and 6 are both "allocate a tracking entry", and they are not the same entry. §12 is that observation developed, and it is the most commonly collapsed distinction in integration RTL.

11. One Coherence Request, All Three Planes

The same trace for an ownership change initiated by the remote accelerator. Generic action names; no CXL message names are asserted.

Two differences from §10 are worth naming.

Step 5 has no counterpart in the memory trace. Memory and coherence traffic compete, and Chapter 11.4 §22 showed that starving the coherence class converts into a coherence timeout with no safe recovery. So the arbitration policy in plane 3 is a correctness property of plane 2.

The memory plane appears nowhere. A coherence request names a cache line, not a device window — the address routing question was already answered when the line was cached. So each of the two semantic planes can be exercised without the other, which is exactly why a testbench that only runs memory traffic verifies two of the three planes and calls it integration.

12. One Transaction, Four Tracking Entries — and They Are Not Duplicates

The section this chapter exists for.

At a single instant, one semantic transaction may be simultaneously represented in four structures:

These look redundant and are not. Each answers a question the others cannot, and the questions have different owners:

The semantic entry is the only one that knows there is an obligation. Delete it and a returning response matches nothing.

The mapping entry is the only one that knows the object exists but has not been framed. Delete it and an accepted object vanishes — Chapter 10.1 §10's rule.

The replay entry is the only one that can recover a corrupted transmission. Delete it early and a recoverable error becomes unrecoverable.

The PHY state is the only one that knows the physical transfer is incomplete. And notably it can be finished while all three above are still live.

A design with one "in flight" bit per transaction has collapsed four questions into one and can answer none of them.

13. State Lifetimes Across the Whole Stack

The composite table, and the one to keep.

Look down the "On UCIe recovery" column. Three answers appear — unaffected, re-baselined, and must survive — and they are distributed across planes in a way no single reset signal can express. That column is the argument against a global soft_reset, and §17 is what happens when one exists anyway.

And look at the "must survive — but its target may now be wrong" cell. A memory request outstanding across a remap is a genuinely awkward case: the entry must not be discarded, and its recorded target may no longer own the address. The resolution is architectural — usually to quiesce the range before remapping it — but the recognition comes from §9's epoch record, which is the only thing that makes the situation visible.

14. Wrong RTL — Semantic State Retired at a Transport Event

// WRONG — semantic bookkeeping driven by a transport milestone.
always_ff @(posedge clk) begin
  if (ucie_tx_object_sent) begin
    semantic_txn_valid_q[sent_txn_id] <= 1'b0;   // "it's gone, we're done"
  end
end

Architecture. The plane that knows the object was transmitted is being allowed to decide the plane that knows the obligation exists. Those are §4's rows 3 and 1, and they have different lifetimes by construction.

Cycle behaviour. One pulse on transmission and the semantic entry is free — available for reuse by the next transaction.

Failure, in escalating order.

A returning response matches nothing. For a memory read, the data arrives with no entry to route it to. Dropped, the core waits forever; matched against a later transaction that reused the identity, one requester receives another's data. That second outcome is the silent-misdelivery family, and Chapter 12.2 is largely about preventing it.

For a coherence transaction it is worse. The line is in a transient state waiting for a grant, and the transaction that would resolve it has been forgotten. The grant arrives and matches nothing, so the line stays transient until a timeout — and Chapter 11.3 §27 established that a coherence timeout cannot be recovered from by restoring the previous state.

And the retry case makes it non-deterministic. If the transport retries, ucie_tx_object_sent may pulse more than once for one transaction. The first pulse frees the entry; the second frees whatever now occupies that slot. A transport retry has now corrupted an unrelated transaction.

Why it survives review. The condition reads as a natural completion event, the code is three lines, and in a testbench where responses return quickly and identities are never reused, it works.

// Illustrative — the contract, encoded so the coupling cannot be reintroduced.
property p_semantic_state_survives_transport_send;
  @(posedge clk) disable iff (!rst_n)
    (ucie_tx_object_sent && semantic_txn_valid_q[sent_txn_id]
                         && !semantic_completion_fire[sent_txn_id])
      |=> semantic_txn_valid_q[sent_txn_id];
endproperty
a_semantic_state_survives_transport_send:
  assert property (p_semantic_state_survives_transport_send);
 
// And the one that catches the retry variant specifically.
property p_retry_does_not_free_semantic_state;
  @(posedge clk) disable iff (!rst_n)
    transport_retry_event |=> $stable(semantic_txn_valid_q);
endproperty

15. Wrong RTL — One Shared Progress Bit Across Planes

A second collapse, less obvious than §14 and more common.

// WRONG — one bit standing for "this transaction is being worked on",
// written by whichever plane happens to act.
logic txn_busy_q [MAX_TXN];
 
always_ff @(posedge clk) begin
  if (semantic_issue_fire)  txn_busy_q[issue_id]  <= 1'b1;
  if (transport_done_fire)  txn_busy_q[done_id]   <= 1'b0;   // plane 3 clears
  if (coherence_resolve)    txn_busy_q[resolve_id] <= 1'b0;  // plane 2 clears
end

Architecture. Three writers of one bit from two planes, with no arbitration and no statement of what happens when two fire together. This is Chapter 11.3 §13's bug lifted to the integration level.

Cycle behaviour. Whichever assignment executes last wins, which depends on the order the statements appear.

Failure. transport_done_fire and coherence_resolve are different events at different times, and the bit cannot represent both. If transport completes first — the normal case — the bit clears while coherence is still resolving, so the coherence plane is now operating on a transaction the rest of the design considers finished. If a same-cycle collision occurs, the outcome depends on source-code order, which is the least defensible form of a race.

And the diagnostic damage is worse than the functional damage. With one bit, txn_busy_q cannot tell you which plane is still working, so §22's debug checklist has nothing to read. The collapse destroys the observability that would have found it.

The right shape is one bit per plane per transaction, which costs a few flops and makes every question in §22 answerable:

// Illustrative — one progress bit per plane. Not elegant; correct and readable.
logic semantic_open_q  [MAX_TXN];   // plane 1 or 2 — the obligation
logic mapping_open_q   [MAX_TXN];   // plane 3 — queued, not yet framed
logic transport_open_q [MAX_TXN];   // plane 3 — framed, not yet confirmed
 
// The ordering invariant between them, which is now expressible.
property p_transport_open_implies_semantic_open;
  @(posedge clk) disable iff (!rst_n)
    transport_open_q[t] |-> semantic_open_q[t];
endproperty
a_transport_open_implies_semantic_open:
  assert property (p_transport_open_implies_semantic_open);

That property is the compact statement of §12. Transport state for a transaction may only exist while the semantic obligation exists — never the other way round, and never independently.

16. Maintaining the Three Planes Without Letting Them Cross

Declaring three bits is easy; the value is in the update logic, because that is where the ordering invariant is either enforced or quietly broken.

// ILLUSTRATIVE per-plane progress maintenance. NOT a UCIe or CXL structure.
// The structure is the lesson: each bit has exactly ONE writer, and the
// SEMANTIC bit is the outermost — nothing below it may outlive it.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int t = 0; t < MAX_TXN; t++) begin
      semantic_open_q[t]  <= 1'b0;
      mapping_open_q[t]   <= 1'b0;
      transport_open_q[t] <= 1'b0;
    end
  end else begin
    // ---- PLANE 1/2: the obligation. Set at acceptance, cleared ONLY by
    //      semantic completion or an explicitly reported abandonment.
    //      Note what is absent: no transport event appears here at all (§14).
    if (semantic_accept_fire)
      semantic_open_q[accept_id]  <= 1'b1;
    else if (semantic_completion_fire || semantic_abandon_reported)
      semantic_open_q[resolve_id] <= 1'b0;
 
    // ---- PLANE 3a: queued but not framed. Bounded by the semantic bit.
    if (mapping_enqueue_fire)
      mapping_open_q[enq_id] <= 1'b1;
    else if (mapping_dequeue_fire)
      mapping_open_q[deq_id] <= 1'b0;
 
    // ---- PLANE 3b: framed but not confirmed. A RETRY does not touch this
    //      bit — the object is still unconfirmed either way (Ch 9.4 §8).
    if (transport_frame_fire)
      transport_open_q[frame_id] <= 1'b1;
    else if (transport_confirm_fire)
      transport_open_q[confirm_id] <= 1'b0;
 
    // ---- THE CROSS-PLANE GUARD. A recovery may clear plane-3 state for
    //      transactions it genuinely owns, and must NOT touch plane 1/2.
    //      Written as a separate, explicitly-scoped block so that the scope
    //      is reviewable rather than implied.
    if (ucie_recovery_event) begin
      for (int t = 0; t < MAX_TXN; t++) begin
        mapping_open_q[t]   <= 1'b0;   // per-link-epoch — correct to clear
        transport_open_q[t] <= 1'b0;   // per-link-epoch — correct to clear
        // semantic_open_q is DELIBERATELY absent. Adding it here is §14's
        // bug, and it is the single most likely edit a later engineer makes.
      end
    end
  end
end

Classification: synthesizable, illustrative.

Architecture. Four blocks, one per bit plus one for recovery, and the recovery block is deliberately separate rather than folded into the three. Folding it in makes the scope of a recovery a property you have to derive by reading three conditions; separating it makes the scope a list you can review, and the absent line is visible as an absence.

State. Three bits per transaction. semantic_open_q has per-transaction lifetime owned above the link; the other two have per-link-epoch lifetime. §13's table is this distinction, and this block is where it is implemented.

Cycle behaviour. Each bit is written from exactly one always_ff with a stated priority. Note that transport_open_q is not touched by a retry: an object being retransmitted is still framed and still unconfirmed, so the bit does not move — which is Chapter 9.4 §8's invariant expressed at this level.

Contract. §15's ordering property relies on this: transport and mapping bits may only be set while the semantic bit is set. The recovery logic relies on the plane-3 bits being safe to clear wholesale.

Failure. Adding semantic_open_q[t] <= 1'b0; to the recovery loop is §14's bug, and it is the most likely future edit — which is exactly why the comment naming it sits on the line where it would go. Without the comment, the omission looks like an oversight to the next reader and gets "fixed".

DV. Force a recovery with transactions at every combination of the three bits and check the resulting pattern against §13's policy column. Then check the pattern that must never occur — a transport or mapping bit set with the semantic bit clear — which §18's p_no_orphan_transport_state asserts continuously.

17. Recovery With All Three Planes Live

The hardest scenario in Module 11, and the honest answer is a required architectural decision rather than an algorithm.

A UCIe link enters recovery. At that instant:

• plane 1 holds valid windows that have nothing to do with the link;
• plane 2 holds several lines in transient states, one of them the only holder of modified data;
• plane 3 holds queued objects, unconfirmed replay entries, credits, and a format epoch.

What is straightforwardly correct. Plane 3's link-epoch state is re-baselined — credits re-advertised, replay state re-synchronised with the peer, format epoch re-established. Chapters 9.4 and 9.5 built exactly this, and it is right.

What is straightforwardly wrong. Applying that same rule to planes 1 and 2. A window is not link state. A cache line's ownership is not link state. Clearing either because the link recovered is Chapter 11.2 §18's category error at system scale.

What is genuinely hard. Plane 2's transient states, because whether the transaction took effect is not knowable from this side:

• the far side may have observed the request and already invalidated another agent;
• a grant may have been sent and lost, in which case ownership transferred and only one party knows;
• a probe response may have been lost, and by the specification's own rules the host cannot issue another snoop to that line;
• dirty data may be in a buffer that a reset is about to clear.

Transport recovery is local. Semantic state may be distributed. A locally-complete recovery does not restore a distributed agreement, and nothing local can tell you whether the agreement survived.

What the architecture must therefore define in advance, and this chapter names the requirements rather than inventing flows:

A per-plane recovery policy, in a stated order. Plane 3 re-baselines first, because nothing above it can make progress until transport exists. Plane 1 is untouched. Plane 2's stable lines are untouched, and its transient lines enter a defined resolution path.

An explicit resolution for each transient transaction, which is completion after recovery where that is provably safe, or containment and escalation where it is not. The CXL specification provides containment concepts — a poison indication for data that must not be used, and a viral error concept — and their existence is the clue: the intended answer to "coherence may have broken" is contain and report, not guess.

A rule that dirty data is never silently lost. If a line's only current copy is held locally and the transaction transferring it has failed, either the value survives to be transferred later or the failure is escalated as a data-integrity error.

// Illustrative — recovery must not silently delete dirty coherence state.
// This is the single most valuable cross-plane property in the chapter.
property p_recovery_preserves_dirty_or_reports;
  @(posedge clk) disable iff (!rst_n)
    (ucie_recovery_event && line_dirty_q[l])
      |=> (line_dirty_q[l] || dirty_loss_reported[l]);
endproperty
a_recovery_preserves_dirty_or_reports:
  assert property (p_recovery_preserves_dirty_or_reports);
 
// Illustrative — plane 1 is not link state.
property p_recovery_preserves_memory_map;
  @(posedge clk) disable iff (!rst_n)
    ucie_recovery_event |=> ($stable(mem_window_q) && $stable(interleave_cfg_q));
endproperty
 
// Illustrative — plane 3 SHOULD re-baseline, and asserting that it does is
// as valuable as asserting that the others do not.
property p_recovery_rebaselines_credits;
  @(posedge clk) disable iff (!rst_n)
    ucie_recovery_complete |-> credits_readvertised;
endproperty

Note the third property. It is easy to write only the "must not change" assertions and forget that a plane which should be re-initialised and is not produces its own failure — stale credits after a recovery let the sender overrun a receiver that has forgotten the allocation. Both directions need asserting.

18. Cross-Plane Assertions

// Illustrative cross-plane properties. LOCAL invariants for this integration
// model, not UCIe or CXL requirements. Each one is the executable form of a
// sentence earlier in the chapter.
 
// ADMISSION — no semantic transaction is accepted unless all four terms hold.
property p_accept_requires_full_readiness;
  @(posedge clk) disable iff (!rst_n)
    semantic_accept_fire |-> coherent_path_ready;
endproperty
a_accept_requires_full_readiness: assert property (p_accept_requires_full_readiness);
 
// STRUCTURE — §15's ordering: transport state only exists under a live obligation.
property p_no_orphan_transport_state;
  @(posedge clk) disable iff (!rst_n)
    (transport_open_q[t] || mapping_open_q[t]) |-> semantic_open_q[t];
endproperty
 
// LIFETIME — §14: the semantic obligation outlives every transport milestone.
property p_semantic_outlives_transport;
  @(posedge clk) disable iff (!rst_n)
    (transport_confirm_fire[t] && !semantic_completion_fire[t])
      |=> semantic_open_q[t];
endproperty
 
// EXACTLY ONCE — a transport retry never creates a second semantic allocation.
// Uses a VERIFICATION-ONLY monitor tag; the protocol identities are reused.
property p_retry_creates_no_second_allocation;
  @(posedge clk) disable iff (!rst_n)
    semantic_alloc_fire |-> !alloc_seen_mon[alloc_mon_tag];
endproperty
 
// STABILITY — the routing decision for an accepted transaction does not move
// under it. Catches a remap landing between decode and transmission.
property p_target_stable_after_accept;
  @(posedge clk) disable iff (!rst_n)
    (semantic_open_q[t] && !semantic_completion_fire[t])
      |=> $stable(txn_target_dev_q[t]);
endproperty
 
// PLANE ISOLATION — a coherence event must not disturb a memory window, and a
// window update must not disturb a line's state. Cheap, and it catches a
// shared-array or shared-index mistake between the two semantic planes.
property p_planes_isolated;
  @(posedge clk) disable iff (!rst_n)
    coherence_state_update |=> ($stable(mem_window_q) && $stable(interleave_cfg_q));
endproperty

On p_target_stable_after_accept, because it is the least obvious. Chapter 11.2 §9 asserted that the window match is stable while a decode is in flight. This is the longer-horizon version: the decision must be stable for the whole transaction, because the transaction carries its target rather than recomputing it, and a remap in between produces a request delivered to a device that no longer owns the address. Same bug, two very different time constants, and only the longer one catches a software-initiated remap.

19. Failure Taxonomy — Attributing a Failure to a Plane

The last two rows are the ones that get misattributed most often, and in the same direction: both look like coherence bugs and neither is. The coherence engine is the most complex block, so it is the default suspect — and in an integration failure it is usually the victim.

20. The Three-Model Integration Scoreboard

Three planes need three models, joined by one identity.

JOIN KEY — a verification-only reference transaction ID, allocated by the
testbench. NOT a protocol field. Required because CXL identities are reused
and transport identities are invisible above the mapping layer.
 
ADDRESS MODEL (plane 1)          — from Chapter 11.2 §30
  windows[]      : base, limit, target, valid   -- INDEPENDENTLY maintained
  interleave[]   : granularity, ways            -- INDEPENDENTLY computed
  predict_target(addr) -> device
 
COHERENCE MODEL (plane 2)        — from Chapter 11.3 §30
  per line: latest_value, owner, sharers[], dirty_at, pending_txn, pending_probe
 
TRANSPORT MODEL (plane 3)        — from Chapter 11.4 §25
  per object: class, queued/framed/unconfirmed/retired, retry_count,
              format_epoch_when_framed, semantic_deliveries (must be 1)
 
INTEGRATION LEDGER (the join)
  per ref_id: plane_of_origin, addr, predicted_target, observed_target,
              accept_epochs, semantic_open, mapping_open, transport_open,
              resolution (completed | abandoned | none)

The four checks only the join can make, and each corresponds to a bug in this chapter.

Predicted target equals observed target, for every transaction. Computed from the model's own copy of the window and interleave configuration — never by reading the design's decode. This is the only check that catches §19's second row, and it is also what catches §18's stability failure: a transaction whose observed target changes mid-flight.

Every transaction has exactly one semantic allocation and exactly one resolution. The ledger's semantic_open plus resolution fields. This catches §14 — a transaction retired at transmission shows up as a resolution with no completion, or as an allocation freed twice.

Transport state never exists without a semantic obligation. §15's ordering invariant, checked across models rather than within one. A transport_open with no semantic_open for the same ref_id is the orphan case.

No plane's state was cleared by another plane's event. Snapshot all three models before and after each recovery and each remap, and diff them against §13's policy column. This is the check that finds §17's failure, and it fires at the recovery rather than when the missing data is eventually noticed.

On what the join must not become. It must not compare occupancies across planes — §12's callout explains why those numbers are unrelated. The join is by identity, never by count.

21. Coverage

// Illustrative integration coverage. Not UCIe- or CXL-defined. Every bin
// exists to reach a cross-plane situation named in this chapter.
covergroup cg_cxl_integration @(posedge clk iff integration_event);
 
  cp_plane     : coverpoint txn_plane_of_origin {
    bins mem = {PLANE_MEM}; bins cache = {PLANE_CACHE};
  }
  cp_ready     : coverpoint readiness_term_low {     // §7 — each of four, alone
    bins map = {0}; bins coh = {1}; bins xport = {2}; bins dev = {3}; bins none = {4};
  }
  cp_open      : coverpoint plane_open_pattern;       // §15 — which bits are set
  cp_recovery  : coverpoint recovery_with_outstanding;
  cp_dirty     : coverpoint dirty_line_present;
  cp_transient : coverpoint transient_line_present;
  cp_remap     : coverpoint remap_with_outstanding;   // §9 — configuration epoch
  cp_epoch     : coverpoint txn_epoch_stale;
  cp_pressure  : coverpoint mapping_queue_pressure {
    bins empty = {0}; bins some = {[1:$-1]}; bins full = {MAP_Q_DEPTH};
  }
  cp_devdown   : coverpoint device_temporarily_unavailable;
 
  // Memory and coherence traffic concurrently — §11 step 5, and the
  // configuration where the two semantic planes first compete.
  x_both_planes    : cross cp_plane, cp_pressure;
  // Recovery with a dirty transient line — the worst case in Module 11.
  x_recovery_dirty : cross cp_recovery, cp_dirty, cp_transient;
  // A remap while a request to that range is outstanding — §13's awkward cell.
  x_remap_out      : cross cp_remap, cp_plane;
  // Each readiness term low with traffic pending, which tests §9 not §7.
  x_ready_pressure : cross cp_ready, cp_pressure;
 
endgroup

Why x_recovery_dirty is the batch's hardest bin. It requires a link recovery, a line in a transient state, and that line holding the only current copy of modified data — three conditions that must be constructed, because random stimulus over a large address space produces the conjunction essentially never. It is also the only bin that reaches §17's dirty-data hazard, which is the failure with the worst consequence in the module.

And x_both_planes is the bin most regressions skip entirely. A suite that runs memory tests and coherence tests as separate scenarios has verified two planes and has never once exercised the arbitration between them.

22. Debug Checklist

1. Which semantic operation failed — a memory access or a coherence transaction? §10 versus §11, and the two have almost disjoint state.
2. What host physical address? Everything downstream depends on getting this right.
3. Which target device did the design choose, and which does the model predict? §20's first check. A mismatch ends the investigation here.
4. What coherence state was the line in — stable, transient, or probe-shadowed?
5. Which mapping-queue entry held the object, and for how long?
6. Which replay entry, and was it allocated exactly once?
7. Was coherent_path_ready asserted at acceptance — and which term was low if not?
8. Did the transport retry, and did the retry touch any semantic state? §14's second property.
9. Did the semantic transaction allocate exactly once? §20's second check.
10. Did a recovery occur while ownership was changing? §17 — and check what it cleared against §13.
11. Did a remap occur while the request was outstanding? §9's epoch record makes this a lookup.
12. Was the target stable for the whole transaction? §18.
13. Which plane's progress bit is still set? §15 — and if there is only one bit, that is the finding.
14. Which plane's model diverged first from the reference? The question that routes everything.

Step 14 is the highest-yield one, and step 13 is what makes it answerable. A design with per-plane progress bits turns a multi-day investigation into a single observation; a design with one shared bit cannot answer step 13 at all, which is the real cost of §15.

23. Common Misconceptions

"CXL integration is just enabling CXL.mem and CXL.cache." It is making three state planes with three different lifetimes agree. Two of them are indexed by address and one is not, and nothing in any plane forces consistency with the others (§4, §5).

"One ready bit represents the whole coherent path." Four terms, four owners, three lifetimes — and each becomes true at a different point in bring-up (§7). Substituting link-active for the conjunction is the fifth appearance of that shape in this curriculum (§8).

"Transport retry and semantic retry are the same." A transport retry re-sends bytes. A semantic retry re-issues an obligation. Allowing the first to trigger anything in the semantic plane produces a duplicated action, and for coherence there is no address to inspect afterwards (§14, §18).

"A UCIe reset can safely clear all coherence state." Link-epoch state re-baselines correctly. Coherence line state is per-address and outlives the link; a line's dirty data may be the only current copy in the system (§13, §17).

"Memory mapping and coherence are independent." They are independent planes and they are not independent concerns: a coherence request names a line whose data lives at an address that the memory plane routes. Get the routing wrong and coherence operates correctly over the wrong memory, with no local symptom (§5, §19).

"A clean transport scoreboard proves coherent correctness." It proves the bytes crossed the package. Both of this module's silent failures — a misrouted store and a stale coherent read — produce perfect transport telemetry (§20).

"One transaction needs only one tracking entry." Four, at the same instant, answering four different questions with four different owners and four different free conditions. Collapsing them destroys both the correctness and the observability (§12, §15).

"If the response arrived, the request must have been routed correctly." No. A request routed to the wrong device is served by that device and produces a well-formed response. Arrival proves delivery, not correctness of destination (§19, and Chapter 12.2 develops the response-side half).

"Readiness can be consulted continuously." Gating already-accepted work on current readiness suspends transactions that neither complete nor fail. Readiness gates acceptance; a captured epoch governs work in flight (§9).

"A transient line during a retry is an error." It is a normal state of a working system. Treating it as a fault converts a recoverable moment into an unrecoverable one (§5, row five).

24. Understanding Check

25. Module 11 Complete

Five chapters, from a motivation to a system.

11.1 Why CXL Matters — what coherent attach provides that device-attach does not, what it costs, and why a clean transport proves nothing about coherence.

11.2 Memory Expansion Over CXL — HDM versus private device memory, and expansion as an address-ownership problem with a routing decode at its centre.

11.3 Cache Coherency Over CXL — distributed ownership, and why the state that matters is mostly the state that exists while ownership is changing.

11.4 CXL Transport on UCIe — a division-of-ownership problem rather than a substitution problem, and exactly-once semantic delivery under replay.

11.5 CXL-over-UCIe Integration — three planes whose state must agree, four tracking entries per transaction, and a recovery policy that cannot be one signal.

The through-line: every layer in this module is individually verifiable and none of them can verify the composition. Module 11's real contribution is the habit of asking, for every piece of state, which plane owns this, what is it indexed by, and which events are allowed to clear it.

26. What's Next

Module 11 has been about what state must exist. It has said comparatively little about when, because a chapter that had to hold three planes in view could not also be cycle-accurate.

Module 12 stops looking at blocks and follows individual transactions through the datapath, cycle by cycle, across the two named interfaces this chapter introduced — FDI between Protocol Layer and Adapter, and RDI between Adapter and Physical Layer:

• 12.1 — Request Flow — the initiator-side path from the Protocol Layer through the link, and what it means for a request to be owned at each boundary rather than copied through it.

Browse the full path on the UCIe tutorials index.