CXL Over UCIe

Chapter 25.4 carried a protocol whose semantics were yours. This one carries a protocol whose semantics belong to another specification entirely — and the interview failure is a boundary error, not a knowledge gap.

1. What They Ask

"How does CXL operate over UCIe?"

Or: "If CXL runs over UCIe, who handles coherence?" · "What does UCIe add to CXL?" · "What happens to a CXL transaction if the UCIe link retrains?"

The question sounds like it is testing CXL knowledge. It is testing whether you keep two layers separate under pressure.

2. The One-Sentence Model

CXL provides the protocol semantics. UCIe provides the die-to-die transport those semantics travel over inside a package. Neither absorbs the other — and the fastest way to lose this question is to say "so UCIe handles coherence now."

The follow-on sentence worth having ready: "UCIe lists CXL as a natively mapped protocol, which means the semantics arrive intact — a CXL transaction crossing a die boundary is still a CXL transaction."

3. What They Are Really Testing

And the third row is the one that separates advanced from intermediate. "A transport retry is not a reissued memory operation" is a sentence that demonstrates you have thought about what a duplicated CXL.mem write would actually do.

4. What You Can Safely Assert

5. The Answer Ladder

15 SECONDS — identification, and the boundary.
 
  "CXL is a protocol — it defines coherent access between a host and a
   device. UCIe is a die-to-die link. UCIe lists CXL as natively mapped,
   so CXL semantics ride over the UCIe transport inside a package. CXL
   still owns the semantics; UCIe just moves them."

60 SECONDS — adds what CXL is and what UCIe supplies.
 
  "CXL has three sub-protocols. CXL.io is PCIe-based and does discovery,
   enumeration and register access — it's mandatory. CXL.cache lets a
   device cache host memory. CXL.mem lets the host use device-attached
   memory as cacheable memory. Coherence is hardware-maintained, at
   cacheline granularity.
 
   At board level those multiplex onto a PCIe physical layer. Over UCIe,
   the same semantics ride the die-to-die transport instead — UCIe lists
   CXL as a natively mapped protocol.
 
   What UCIe supplies underneath is framing, integrity, optionally
   link-level retry, and arbitration if more than one protocol shares the
   link. What it doesn't supply is any of the coherence semantics — those
   stay with CXL.
 
   And the subtle part is that a transport retry underneath must not turn
   into a memory operation issued twice."
 
  [STOP — that last clause is the hook, §20.]

Three properties.

The 60-second answer names all three sub-protocols with a direction each — that is the CXL knowledge check, and it costs about fifteen seconds.

"What it doesn't supply" is the boundary statement, and it is the sentence the whole question is built around (§8).

And the closing clause is deliberately the sharpest thing in the answer (§10). "A transport retry must not become a memory operation issued twice" invites "how do you prevent that?" — which is the strongest ground available on this topic.

6. What CXL Actually Is

Enough to answer the question. This is not a CXL tutorial — the protocol itself belongs to the CXL curriculum; what follows is only what this boundary question needs.

Three properties worth saying.

CXL.io being mandatory matters — every CXL device speaks it, so discovery and configuration exist regardless of which coherent sub-protocols are used.

The two coherent sub-protocols run in opposite directions, and confusing them is a common slip. .cache is the device reaching into host memory; .mem is the host reaching into device memory.

And coherence is hardware-maintained at cacheline granularity, with a defined point of Global Observation — which is the fact that makes §10's duplicate-write failure serious rather than merely wasteful.

7. The Stack, Mapped

CONCEPTUAL layering. Terminology from §4; no encoding is claimed.
 
   ┌──────────────────────────────────────────┐
   │  CXL SEMANTICS                           │   CXL owns:
   │  .io / .cache / .mem                     │   - what a request means
   │  coherence, Global Observation,          │   - ordering rules
   │  ordering, completion                    │   - completion semantics
   └──────────────────────────────────────────┘   - coherence state
                     │  natively mapped
   ┌──────────────────────────────────────────┐
   │  UCIe PROTOCOL LAYER                     │   carries CXL
   ├──────────────────────────────────────────┤   <- FDI
   │  UCIe D2D ADAPTER                        │   UCIe owns:
   │  framing, 2B header + 2B CRC,            │   - transport integrity
   │  optional link-level retry, arbitration  │   - link-level retry
   ├──────────────────────────────────────────┤   <- RDI
   │  UCIe PHYSICAL LAYER                     │   - signalling, training
   ├──────────────────────────────────────────┤
   │  PACKAGE CHANNEL                         │   in-package
   └──────────────────────────────────────────┘
 
   THE LINE THAT MATTERS: everything above the first boundary is CXL's.
   Everything below it is UCIe's. Nothing crosses.

Two readings.

Compare this with board-level CXL, where the three sub-protocols multiplex onto a PCIe physical layer (§4). The top box is unchanged; only what is underneath it moved — which is exactly 23.1 §8's protocol-versus-electrical-link distinction, now with CXL as the example.

And the annotation on the right is the answer to §8. Coherence, ordering, completion and Global Observation are all in the top box.

8. The Major Trap

INTERVIEWER:  "So if CXL runs over UCIe, does UCIe now own coherence?"
 
WRONG:        "Yes — UCIe handles the coherent traffic."
WRONG:        "Effectively, since it's carrying it."
 
STRONG:       "No. UCIe transports the messages; CXL owns the semantics.
               Which agent holds a line, what ordering is required, when a
               write becomes globally observable — all of that is defined
               by CXL and maintained by the CXL agents at each end.
 
               UCIe's job underneath is to move those messages reliably
               across the package. If you swapped the transport, the
               coherence protocol would be unchanged — that's the test of
               whether something belongs to the transport or not."

Four properties.

The last sentence is a test, not an assertion — "if you swapped the transport, would this change?" — and offering a test rather than a claim is a senior move.

Naming Global Observation is precise and cheap (§4), and it demonstrates you know coherence has a defined visibility point rather than being a vague property.

"Effectively, since it's carrying it" is the tempting wrong answer, because it feels like nuance. It is not — carrying a message and defining its meaning are different things (23.3 §10).

And this is 25.1 §8's senior sentence with a specific protocol attached, which makes it stronger: "it transports coherence, it doesn't define it" is good; naming CXL as the owner and GO as the visibility point is better.

9. RTL — A Semantic Request Above a Transport

// ILLUSTRATIVE ONLY. A generic coherent-protocol request — the shape of what
// lives ABOVE the transport. Note what it does NOT contain: no lane count, no
// retry counter, no link state (25.3 §12).
typedef struct packed {
  logic [REQ_W-1:0]   req_id;         // SEMANTIC identity — stable across retry
  logic [GEN_W-1:0]   generation;     // which USE of that id
  logic [OP_W-1:0]    op_class;       // read / write / snoop / completion
  logic [ADDR_W-1:0]  addr_token;     // OPAQUE — not an architectural address
  logic [ORD_W-1:0]   ordering_domain;// ordering scope — protocol-defined
  logic [NODE_W-1:0]  home_agent;     // who owns the coherence decision
  logic               needs_completion;
  logic [EPOCH_W-1:0] cfg_epoch;      // which mode/config this ran under (§13)
} coh_semantic_req_t;
 
// And what the TRANSPORT carries — it REFERS to the semantic request rather
// than containing its meaning.
typedef struct packed {
  logic [OBJ_W-1:0]   obj_id;         // transport object
  logic [REQ_W-1:0]   req_id;         // the semantic request it serves
  logic [GEN_W-1:0]   generation;
  logic [ATT_W-1:0]   attempt;        // 1, 2, 3 ... — TRANSPORT's business only
  logic [EPOCH_W-1:0] link_epoch;
} transport_obj_t;

Architecture. Two types because there are two owners. The semantic request belongs to the coherence protocol and outlives every transport event; the transport object belongs to the Adapter and may be retried.

State. The coherence agent holds an outstanding table keyed by req_id + generation; the Adapter holds one keyed by obj_id.

Event. attempt increments on retransmission. No branch anywhere writes req_id or generation on a transport event — §10 is what happens when one does.

Contract. home_agent and ordering_domain are protocol-layer fields. If the transport derived either from link state, the coherence protocol's correctness would depend on which physical path a message took (23.3 §13).

Failure. §10. And separately: if addr_token were interpreted by the transport, a transport change would alter address handling — which is why it is opaque here.

DV/debug. Both identities in a trace (21.7 §16) is what lets you say "three attempts, one request, one completion — the system is working" rather than reporting a duplicate.

10. Wrong RTL — a Transport Retry Becomes a Reissued Operation

// WRONG — the retry path allocates a NEW semantic request id, because "from
// this layer's point of view it's a new send." Plausible, and it corrupts
// memory.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) req_id_q <= '0;
  else if (transport_retx_fire)
    req_id_q <= next_free_req_id;         // <-- a transport event writing a
                                          //     SEMANTIC field
end

Worked, a CXL.mem write:

Five properties.

Step 3 was correct and step 4 was not. The retry mechanism did exactly its job; the defect is that a transport event was allowed to write a semantic field (25.3 §10).

Step 10 is why this is the flagship. Transport-level integrity is perfect — transport correctness is not protocol correctness (23.3 §14), and no CRC or link counter can see it.

Step 9 is what makes it worse than a duplicate. Duplicated coherent traffic does not merely waste bandwidth; it can leave two agents with different views of a cacheline, which is the exact failure coherence exists to prevent.

And the interview version is one sentence: "A physical retry is a retransmission of the same operation. If it becomes a new semantic request, you've issued the memory operation twice, and CRC will tell you everything is fine."

The fix is §11's absence — and being able to say "the fix is that no branch writes the semantic id" is a stronger answer than describing a mechanism.

11. Corrected — Identity Correlation Across Attempts

// CORRECTED. One semantic request; the transport owns only its own attempt
// counter. The correctness argument is an ABSENCE.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    attempt_q <= '0;
  end else begin
    if (sem_issue_fire)       attempt_q <= ATT_W'(1);   // first attempt
    else if (transport_retx_fire) attempt_q <= attempt_q + ATT_W'(1);
    // NOTE: no branch here writes req_id_q or generation_q.
  end
end
 
// The receiver de-duplicates on the SEMANTIC pair only. Including `attempt`
// in this key would give every retry a new key and disable the mechanism
// entirely — a bug that looks implemented.
logic already_applied;
assign already_applied = applied_set_contains(rx.req_id, rx.generation);
assign do_apply = rx_valid && !already_applied;
 
// MANDATORY. English: a transport retry never alters the semantic identity.
// Structural non-interference — best discharged formally, because simulation
// shows only what the stimulus explored (21.7 §17). Catches §10 directly.
a_retry_preserves_semantic_identity: assert property (
  @(posedge clk) disable iff (!rst_n)
    transport_retx_fire |=> ($stable(req_id_q) && $stable(generation_q))
);
 
// MANDATORY. English: one semantic request produces at most one application.
// Bounded by an OBSERVED retirement rather than an invented window (21.6 §29).
property p_apply_once(int unsigned r, int unsigned g);
  @(posedge clk) disable iff (!rst_n)
    (apply_fire && (rx.req_id == r) && (rx.generation == g))
      |=> !(apply_fire && (rx.req_id == r) && (rx.generation == g))
          throughout (1'b1 [*1:$] ##0 (retire_fire && (retire_id == r)));
endproperty

Architecture. An attempt counter owned by the transport, and a de-duplication key owned by the semantics.

State. The attempt counter, and the receiver's applied set keyed by req_id + generation.

Event. attempt_q loads 1 on first issue and increments on retransmission. Nothing writes the semantic identity.

Contract. The de-duplication key is req_id + generation only. Adding attempt to it is the most easily-made mistake here — it produces a key that changes per retry, so nothing matches and every attempt is applied. The mechanism exists, is consulted, and does nothing.

Failure. §10. And omitting generation from the key means an identity reused after retirement collides with a straggler (25.4 §12).

DV/debug. In silicon this becomes two counters: semantic requests issued and applications performed. A divergence between them is §10 caught from a register read, before it becomes a coherence inconsistency.

12. Ordering Belongs Above the Transport

The transport may retry and may take different physical paths. What it must not do is change the order in which semantics complete, where the protocol declares an ordering requirement.

And the interview-safe framing: "I wouldn't quote CXL's ordering rules from memory. Architecturally, the transport must preserve whatever ordering contract the protocol above it declares — and the protocol declares it per ordering domain, not globally" (21.6 §14).

Asserting a global order is its own bug (22.4 §14): it serialises the link to satisfy a rule nobody made, converting a latency problem into a throughput collapse.

13. Configuration and Mode Context

A trace must be interpreted under the configuration that was active when the event was issued.

And this is a specific interview answer worth having. "If a checker fires, my first question is whether the rule applied — which revision, which mode, which configuration was active when the event was issued. That's why the config epoch is on the request in §9."

14. The Recovery Question

INTERVIEWER:  "What happens to live CXL transactions if the UCIe link
               recovers underneath them?"
 
WEAK:         "They're lost, you have to restart."
WEAK:         "They're fine, recovery is transparent."
 
STRONG:       "It depends on the recovery's scope, and that distinction is
               the whole answer.
 
               A link recovery is a transport event. It doesn't by itself
               mean the CXL requests above it stopped being valid — a
               request that was accepted is still outstanding, and a
               coherent request in particular is blocking something. So
               the semantic state has to survive, or be explicitly
               reconciled.
 
               What the design must decide is whether the transport
               retains outstanding objects across recovery or discards
               them and requires replay. Both are workable; both ends have
               to agree.
 
               What isn't workable is silently dropping them. For coherent
               traffic that's not a lost packet — it's a request that
               never completes, so an agent waits forever and the two ends
               end up disagreeing about a line."

Four readings.

The shape is "it depends, and here is on what", which is honest — the retention rule is a scope question this chapter does not assert (§4).

"A coherent request in particular is blocking something" is the sentence that shows you understand why this matters more for CXL than for bulk traffic (22.4 §9): a lost coherent request is a hang, not a retry.

"Both ends have to agree" is the interoperability bridge (22.5 §4) if they want to go there.

And §18 is the RTL-and-debug follow-up — "show me what going wrong looks like."

15. Assertion Inventory

Two things to say about them in an interview.

The first is a formal property. "Non-interference is a statement about all stimulus, so I'd want it proved rather than simulated" — the same sentence 25.3 §17 recommends.

And the last one samples on the recovery exit edge, because retention is only observable once recovery completes; sampling during it compares against a transient.

16. False Answers

And rows 6 and 8 are the CXL-knowledge checks. They are cheap to get right and conspicuous to get wrong — an interviewer who works on CXL will notice immediately.

17. "How Would You Verify CXL Over UCIe?"

The senior follow-up, and the answer is a structure.

Three properties, and the third is the point of the question.

Two checkers, not one. A single scoreboard conflates "the protocol did the wrong thing" with "the transport corrupted it" — and those go to different teams.

Both models must be independent of the DUT (21.6 §24). A checker that calls the design's own function to decide legality agrees with the design's bugs by construction.

And the correlation layer is where §10 is caught. Neither checker alone sees it: the transport checker sees a valid retry, the semantic checker sees two valid requests. Only the join notices that one request became two.

18. Flagship Case — the Orphan Completion

Five readings, and this is the case to walk when asked about recovery.

Step 3 is a reasonable-sounding decision, which is what makes it a real bug rather than a careless one. It is correct for transport state and catastrophic for semantic correlation (23.3 §14).

Step 6's symptom points at the peer. The natural reading is "the peer sent a completion for something we never asked for" — and the peer is innocent.

Step 8 is why every transport instrument is useless here. The link is healthy by every measure it has.

The first divergence is step 3, not step 6 — and finding it requires knowing that a transport event touched semantic state (21.6 §30).

And the fix is §14's contract made explicit: the recovery selects retain or replay, never clear — with a_recovery_preserves_semantic_set firing at step 3, in simulation, before anything is orphaned.

19. Twelve Follow-Ups

Q1 — "Why run CXL over UCIe rather than something else?" Because the semantics are already what the system speaks. A device or die that already talks CXL can cross a package boundary without changing its protocol layer — and CXL is natively mapped (§4).

Q2 — "Which layer owns coherence?" CXL, and its agents at each end. UCIe transports the messages (§8).

Q3 — "What does UCIe add?" Framing, a header and CRC, optionally link-level retry, and arbitration if several protocols share the link (§4). Say "optionally."

Q4 — "What state does a retry preserve?" The semantic request identity and generation. Only the attempt counter changes (§11).

Q5 — "How would you verify it?" §17 — two independent checkers plus a correlation layer; the join is where the interesting bugs live.

Q6 — "How do you separate semantic and attempt identity?" §9's two structs: the transport object refers to the semantic request rather than containing its meaning.

Q7 — "Could a transport retry reorder semantic operations?" It can reorder attempts; it must not change the order in which semantics complete where the protocol declares an ordering requirement (§12). And ordering is per domain, not global.

Q8 — "What does the config epoch protect?" Against interpreting a straggler under rules that were not active when it was issued (§13) — and against a checker firing on a mode where the rule does not apply.

Q9 — "Does UCIe define host discovery?" No. CXL.io does — discovery, enumeration and register access, and it is mandatory (§4, §6).

Q10 — "What happens on recovery?" §14 — a scope question; retain or replay, agreed by both ends; silent drop is a hang with a healthy link.

Q11 — "Why not share a function between the DUT and the checker?" Because a shared misunderstanding becomes invisible (21.6 §24) — the model agrees with the design's bug by construction.

Q12 — "How do you tell a CXL semantic failure from a UCIe transport failure?" By which checker fires, and by the correlation layer. A transport failure shows integrity or retry symptoms; a semantic failure shows a legal-looking transport with a wrong protocol outcome — §18 is the case where the transport is clean and the protocol is broken.

20. Controlling the Next Question

And the first row is the best hook in the chapter, because the follow-up lands on a failure you can walk in ten steps (§10) with a one-line fix that is an absence — which is memorable and correct.

21. Understanding Check

22. Summary

Five things.

CXL owns semantics; UCIe owns transport (§2, §8). The test to offer is "if you swapped the transport, would this change?"

Know the three sub-protocols and their directions (§6). CXL.io is PCIe-based and mandatory; CXL.cache is device caching host memory; CXL.mem is host using device memory — and reversing those two is a conspicuous slip.

A transport retry is not a reissued operation (§10). One semantic identity, many attempts — and the fix is that no branch writes the semantic id.

Ordering is per domain and belongs above the transport (§12). Asserting a global order is its own bug.

And recovery is a scope question (§14, §18). Retain or replay, agreed by both ends; clearing semantic correlation on a transport event produces an orphan completion with a perfectly healthy link.