Why CXL Matters

Module 10 ended with a working system. A PCIe function lives on another die, the root complex has absorbed it into its topology, transactions route correctly, and software cannot tell the difference.

For an I/O device that is the end of the story. A network controller, a storage controller, an accelerator that works on buffers handed to it — all of these are well served by what Module 10 built.

This module exists because some chiplets are not I/O devices. A memory chiplet whose capacity should become part of the host's address space, and an accelerator whose cache should participate in the host's memory hierarchy, both need something PCIe's transaction semantics were never designed to provide. This chapter is about what that something is, what it costs, and why it changes the verification problem more than it changes the transport.

1. The One-Sentence Model

PCIe connects devices; CXL can integrate memory and caches into a shared coherent system architecture.

Note the word can, and note what the sentence does not say.

It does not say CXL replaces PCIe. The CXL Consortium's own description of the technology places it directly on PCIe infrastructure — CXL.io is described as being based on PCIe, and the three CXL protocols are dynamically multiplexed over that foundation. Everything Module 10 built about carrying PCIe over UCIe remains load-bearing here.

It does not say coherent is better. §14 argues at length that it is not, and that the entire engineering judgement is about whether a given workload needs the semantics enough to pay for them.

What it says is that there is a class of system capability — memory that the CPU can load and store as memory, and device caches that stay coherent with the host — that PCIe's device-attach model does not provide, and that CXL does.

2. The Limitation, Concretely

Take the system Module 10 leaves you with, and make it three chiplets in one package:

• a CPU chiplet with cores, caches, and local memory controllers;
• an accelerator chiplet doing work on data the CPU also touches;
• a memory chiplet with substantial capacity.

Over a purely I/O-style attach, the shape of this system is fixed by the attach model rather than by what the workload wants.

The accelerator works through explicit data movement. The CPU prepares a buffer. Something initiates a transfer. The accelerator operates on its own copy. Results come back the same way. Software owns the choreography, and ownership of each buffer moves explicitly between the two.

The memory chiplet's capacity is device-owned. It is a resource the device manages and exposes through its own mechanisms. The CPU does not simply issue loads and stores to it as it does to memory behind its own controllers.

Sharing is by copy and by protocol, not by address. Two agents working on the same data structure do so by agreeing, in software, whose turn it is.

None of that is a defect. It is a superb model for I/O, and for accelerators with large, coarse, batchable work it is often the right model — §14 returns to this. But it constrains what you can build, and the constraints are exactly:

The last row is the sharpest. Attaching a device with a lot of memory does not increase the amount of memory the system has, in the sense the operating system means. That is the gap.

3. The Explicit-Movement Model, and What It Costs

Worth a paragraph, because the alternative only makes sense against it. The PCIe track covers transaction mechanics; this is only the system-level shape.

Data lives in host memory. The device has its own memory. Moving between them is an explicit operation, initiated by software or by an engine software programmed. While a transfer is in flight, both copies exist and only one is authoritative. Software knows which — because software arranged it.

What this costs. Latency, on every ownership transition, whether or not the data was actually shared. Bandwidth, because data moves whether or not the consumer will touch all of it. Capacity, because two copies exist. And software complexity, because correctness now depends on a discipline the hardware does not enforce.

What it buys. Simplicity where it matters most: no distributed state, no protocol traffic beyond the transfers themselves, no ordering subtleties between agents, and a failure model in which a bug is a missing synchronisation rather than a silent stale read. Those are real benefits and §14 will not let you forget them.

The question CXL answers is what to do when the ratio goes the wrong way — when the sharing is fine-grained enough, or unpredictable enough, that explicit movement costs more than maintaining coherence would.

4. What CXL Actually Is

CXL is described by the Consortium as an industry-supported cache-coherent interconnect for processors, memory expansion and accelerators, which maintains memory coherency between the CPU memory space and memory on attached devices — allowing resource sharing for higher performance, reduced software-stack complexity and lower system cost.

It is built from three protocols, dynamically multiplexed:

5. Device Types

The Consortium's device taxonomy follows directly from which protocols a device implements:

Three observations that matter for a chiplet designer.

CXL.io is in every one of them. The device-attach path never goes away; discovery, configuration and interrupts still happen the PCIe way. Everything Module 10 taught remains directly applicable — which is a large part of why CXL was designed on PCIe in the first place.

Type 3 is the memory chiplet and it is not a coherent caching agent. It is a CXL.mem target. This is the concrete form of §4's asymmetry, and it is the case §17's capability check exists to protect.

Type 2 is the interesting one for chiplets, because it is the accelerator that both caches host memory and contributes memory of its own. It is also the one whose verification cost is highest, for reasons §21 develops.

The Consortium has also extended beyond the strict taxonomy — CXL 3.1 introduced direct peer-to-peer CXL.mem, allowing an accelerator to generate memory accesses toward a Type 3 expander. Worth knowing the taxonomy is a starting point rather than a boundary.

6. Why UCIe Is the Relevant Transport

CXL was designed to ride PCIe. Why is a chiplet interconnect part of this conversation at all?

Because UCIe's protocol layer already carries it. As established in Chapter 10.2, UCIe maps PCIe and CXL natively — the protocol layer hosts PCIe and CXL, with UCIe's flit formats aligned to the carried protocol's own flit definitions, including CXL 2.0 at 68B flit and CXL 3.0 at 256B flit. The D2D Adapter's machinery — CRC, flit retry, link state management, parameter negotiation — is protocol-agnostic and serves CXL exactly as it serves PCIe.

So the die-to-die transport problem is, from CXL's point of view, already solved. The chiplet in your package can be a CXL device.

7. The Layer Picture

8. The System Picture

The two relationships in the figure are worth naming precisely, because they are §4's asymmetry made physical. The memory chiplet is a target: capacity the host reaches, with the host managing the coherency flows. The accelerator is a participant: it holds copies of host memory, and holding a copy creates obligations.

9. Coherence Changes the Contract

Here is the architectural heart of the chapter.

A PCIe endpoint can, to a first approximation, reason about itself. It receives requests, it services them from its own state, it responds. Its correctness is local. Two endpoints on the same fabric need know nothing about each other.

A coherent participant cannot reason only about its own state.

Holding a cached copy of a line that belongs to host memory means:

• some other agent may hold a copy of the same line;
• some other agent may want to modify it;
• your copy may become stale as a consequence of something you did not initiate;
• and you may owe a response, or an action, to an event that originates elsewhere.

That is a categorical change in what "correct" means. Correctness is no longer a property of the device; it is a property of the system, distributed across agents that must agree.

A coherent endpoint cannot reason only about its own local memory state.

Everything in the rest of this chapter — the per-line metadata, the cost analysis, the verification split, the failure taxonomy — is downstream of that one sentence.

The mechanics of how agreement is reached are the substance of a coherence protocol, and CXL's are the specification's to define. Chapter 11.3 is where the CXL.cache flows belong. What this chapter establishes is why there must be state, and what happens when there is not.

10. Why Coherence Needs Per-Line State

Concretely: what does a device need to know before it can answer a local read from a cached copy of host memory?

Whether the copy is valid at all. Something may have happened since it was fetched.

Whether anyone else may hold a copy. Because if you are about to modify it, that matters to them.

Whether your copy is modified relative to memory. Because if it is, you hold the only current value, and the system depends on you to produce it.

Three questions, and none of them can be answered from the data. They can only be answered from metadata that is maintained as a consequence of protocol events — events that arrive from elsewhere, asynchronously, about lines the device may not currently be using.

That is what makes coherence expensive and what makes it distributed: the state is spread across every agent holding a copy, and the system is only correct if all of it agrees.

11. Coherence State — Illustrative

// ILLUSTRATIVE educational coherence state — NOT a CXL protocol encoding,
// not a CXL state set, and not a claim about how any CXL agent represents
// line state. CXL's coherence states, transitions, and messages are the
// specification's to define and are not reproduced or approximated here.
typedef enum logic [1:0] {
  LINE_INVALID,     // no usable copy
  LINE_SHARED,      // readable; others may also hold it
  LINE_EXCLUSIVE,   // readable/writable; no other copy expected
  LINE_MODIFIED     // written locally; this is the only current value
} cache_state_t;

Architecture. The four states are the classical teaching set, and they exist here to make one point that transfers to any real protocol: the states differ in what they permit and what they oblige. Invalid permits nothing. Shared permits reading and obliges you to give it up if someone wants to write. Exclusive permits writing without asking. Modified obliges you to supply the data, because memory does not have it.

State. Two bits per cache line, per-cache-line lifetime, which is a lifetime unlike any in Modules 9 or 10 — it is not per-request, not per-link-epoch, not per-enumeration. It is per address, and it persists across everything else.

Cycle behaviour. Transitions occur on local accesses and on protocol events arriving from elsewhere. That second source is what makes it a coherence state rather than a validity bit.

Contract. The local access path relies on the state to decide whether a hit is legitimate. The protocol engine relies on it to know what it owes when an external event arrives.

Failure. A missing state distinction removes an obligation. If a design cannot distinguish "I have the only current copy" from "I have a copy that matches memory", it will fail to supply data that only it has — and the requester will read stale memory contents with no error anywhere.

DV. Cover every state, every transition, and — most importantly — every state crossed with every incoming external event. That cross is the state space, and it is where a real protocol's complexity lives.

12. The Metadata Array

// Illustrative. Simplified to three flags to make the storage cost visible;
// a real design's metadata is richer and protocol-specific.
typedef struct packed {
  logic valid;    // is there a usable copy here
  logic dirty;    // does this differ from memory
  logic shared;   // may another agent hold a copy
} line_meta_t;
 
line_meta_t meta_q [NUM_LINES];

Architecture. This array is the physical cost of coherence, and it is worth looking at as a number rather than a concept. Metadata is per line, so its size scales with cached capacity divided by line size — independent of how much sharing actually occurs. A device that caches host memory pays for this whether or not any line is ever contended.

State. NUM_LINES entries, per-cache-line lifetime. It must be initialised to a defined state — every line invalid — at a defined point, and §18 is about what that point is.

Cycle behaviour. Read on every local access to determine whether a hit is legitimate. Written on fills, evictions, local writes, and on external protocol events. That last class is the one that makes the array a multi-ported problem, because external events arrive independently of local accesses.

Contract. The local datapath relies on it for hit determination. The protocol engine relies on it to compute responses. Both must see a consistent view, which is a genuine design problem when they can update it in the same cycle.

Failure. Metadata and data going out of step is the coherence-layer form of the misalignment bug this curriculum keeps returning to — Chapter 10.2 §7 and Chapter 10.4 §11 are the same shape. Metadata that says valid over data that was never filled produces a hit returning uninitialised contents. Metadata cleared without the data being invalidated in whatever structure actually holds it produces the reverse.

DV. Assert the pairing invariants directly: a line that is dirty must be valid; a line that is invalid must not be dirty. These are cheap and they catch the update-ordering bugs.

// Illustrative pairing invariants — the cheapest coherence assertions
// that exist, and the ones that catch metadata/data update ordering bugs.
property p_dirty_implies_valid;
  @(posedge clk) disable iff (!rst_n)
    meta_q[i].dirty |-> meta_q[i].valid;
endproperty
 
property p_invalid_implies_clean;
  @(posedge clk) disable iff (!rst_n)
    !meta_q[i].valid |-> !meta_q[i].dirty;
endproperty

13. Wrong RTL — Caching Without Coherence State

// WRONG — a device caches a line of host memory and tracks only presence.
// There is no mechanism by which an external event can invalidate it.
always_ff @(posedge clk) begin
  if (fill_valid) begin
    line_data_q[fill_idx] <= fill_data;
    line_present_q[fill_idx] <= 1'b1;      // presence only
  end
end
 
assign read_hit  = line_present_q[req_idx];
assign read_data = line_data_q[req_idx];

Architecture. This is a perfectly good private cache. It is a correct design for data nobody else can change. It becomes catastrophic the moment the data belongs to host memory that other agents also access, because the design has no representation for "someone else changed this".

Cycle behaviour. Fill sets presence. Read checks presence. Nothing ever clears it except a local eviction. The RTL is short, fast, and passes every single-agent test.

Failure — and this is the fundamental reason coherence exists.

The CPU writes the line. The device's copy is now stale. The device reads and hits. It gets the old value.

No error is signalled anywhere. The UCIe link is healthy — no CRC error, no retry, no credit violation. The transport delivered every byte it was asked to deliver, correctly. The PCIe or CXL.io path is healthy. Every mechanism Modules 7 through 10 built is working exactly as designed.

The device is simply computing on data that is no longer true.

Why it is so hard to find. The symptom is wrong results, not a failure. It is timing-dependent, so it reproduces intermittently. It is data-dependent, so it may only matter for some inputs. And it is invisible to every layer with error reporting, so every diagnostic you have says the system is healthy. Post-silicon, this class of bug is typically found by a workload producing wrong answers, which is the most expensive way to find anything.

DV. No assertion on the device catches this, because locally nothing is wrong — the line is present and it is being read. Only a system model that knows the true current value of each address catches it, which is §22's coherence scoreboard. This is the single clearest example in the whole curriculum of a bug that assertions structurally cannot reach.

14. SVA — What Local Properties Can Do

// Illustrative. Local invariants only — see the limits stated below.
property p_hit_requires_valid;
  @(posedge clk) disable iff (!rst_n)
    cache_hit |-> meta_q[req_idx].valid;
endproperty
a_hit_requires_valid: assert property (p_hit_requires_valid);
 
// A line may not be silently dropped while it holds the only current value.
property p_modified_not_silently_dropped;
  @(posedge clk) disable iff (!rst_n)
    (line_evict && meta_q[evict_idx].dirty) |-> writeback_issued;
endproperty
a_modified_not_silently_dropped: assert property (p_modified_not_silently_dropped);

What the first catches. Any path that returns data from a line the metadata says is unusable — a hit computed from the wrong index, a hit that races an invalidation, a state machine that clears validity without gating the datapath.

What the second catches. The evict-without-writeback bug, which loses the only current copy of a line. The requester subsequently reads memory and gets a stale value, and this too produces no error anywhere.

What neither catches, and §23 makes the general argument: they cannot prove that the metadata is right. p_hit_requires_valid confirms the design believes the line is valid. Whether that belief is correct — whether some other agent has since modified the line — is not knowable from any signal at this interface. The §13 bug satisfies both properties trivially and comfortably.

15. Memory Expansion, At the Right Depth

The other half of CXL's value, and Chapter 11.2 owns the flows. What belongs here is the system consequence.

A CXL memory device exposes capacity into the host's memory architecture. The Consortium's description of CXL.mem is the operative one: the host processor accesses memory attached to the device, as memory load and store transactions running downstream from the host, with the host taking care of the associated coherency flows.

The architectural questions this raises are the ones a chiplet integrator must answer:

Who owns the address range? It occupies host physical address space. Something must allocate it, and something must know it is different from the address space behind local memory controllers.

How is it routed? A load or store to an address in that range must reach the right device. This is exactly Chapter 10.4 §10's aperture decode, at a different layer and with higher stakes, and §16 is the same table with a different name.

What are its performance characteristics? Addressed in §20, and it is the question most often skipped.

16. The Memory Window

// Illustrative host-side integration state — architecture teaching, not a
// CXL register definition. Compare Chapter 10.4 §10: same decode problem,
// different layer, and now the traffic is CPU loads and stores.
typedef struct packed {
  logic              valid;
  logic [ADDR_W-1:0] base;
  logic [ADDR_W-1:0] limit;
  logic [DEV_W-1:0]  target;
} cxl_mem_window_t;
 
cxl_mem_window_t mem_window_q [NUM_MEM_WINDOWS];
 
logic [NUM_MEM_WINDOWS-1:0] mw_match_vec;
 
always_comb begin
  for (int i = 0; i < NUM_MEM_WINDOWS; i++)
    mw_match_vec[i] = mem_window_q[i].valid &&
                      (req_addr >= mem_window_q[i].base) &&
                      (req_addr <= mem_window_q[i].limit);
end

Architecture. The host's memory-side decode must now distinguish local memory from device-attached memory and route accordingly. This is the same structure as the aperture table in Chapter 10.4, and deliberately so — but the consequences of getting it wrong are more severe, because this traffic is CPU loads and stores rather than device MMIO.

State. NUM_MEM_WINDOWS entries with per-memory-window lifetime, established when the memory is configured into the system and invalidated when it is removed or reconfigured.

Cycle behaviour. Combinational match in the memory path, which is the most timing-critical path in the system. Real implementations pipeline and often pre-decode; whatever the structure, the window must be stable while a decode is in flight.

Contract. The coherent fabric relies on exactly one destination per address. Software relies on the range it configured being the range decoded.

Failure. Overlapping windows send a load or a store to the wrong memory device. In Chapter 10.4 that misdelivered MMIO to the wrong function; here it misdirects a CPU memory access, which means a store lands in the wrong device's memory and a load returns another device's data. The blast radius is anything the operating system placed in that range.

DV. Exactly Chapter 10.4 §26's boundary discipline — base, base±1, limit, limit±1 for every window, plus adjacent windows with no gap.

// Illustrative — the same one-hot invariant, and it is if anything more
// valuable here than in Chapter 10.4 §12.
property p_mem_window_onehot;
  @(posedge clk) disable iff (!rst_n)
    mem_req_fire |-> $onehot0(mw_match_vec);
endproperty
a_mem_window_onehot: assert property (p_mem_window_onehot);

17. Capability Is State, and Routing Depends On It

§4 and §5 established that memory-capable and cache-coherent are different capabilities held by different device types. The design consequence is that capability must be represented and checked.

// Illustrative capability state. NOT CXL capability-register bit positions —
// how a real device advertises support is a specification matter and no
// encoding is invented here.
logic cxl_io_supported_q;
logic cxl_mem_supported_q;
logic cxl_cache_supported_q;

Architecture. These bits exist because the host's routing and request-generation logic must not issue a request a device cannot service. They are the CXL form of Chapter 10.4 §6's reachability information: local knowledge about a remote party, established once and consulted continuously.

State. Three bits per device, per-device-capability lifetime — established when the device's capabilities become known during bring-up, and stable thereafter for that device instance.

Cycle behaviour. Written once per device during bring-up; read on every routing decision that depends on protocol class.

Contract. The transaction-class router relies on these absolutely. A request must be routable to a device that can service it, and this is the only local evidence of that.

Failure. §18's bug, in detail.

DV. Cover each capability combination that the system can actually contain, and — importantly — cover a request of each class against a device lacking that capability, to prove the check exists rather than assuming it.

The combinations, illustratively:

The rows are illustrative combinations, not a mandate about what any product must be. The point of the table is the second and third columns: they vary independently, and a design that treats "supports CXL" as one bit cannot represent the difference between a memory expander and a caching accelerator — which is precisely the difference that determines whether a coherent request can be sent.

18. Two Wrong Derivations

First — routing a coherent request by capability nobody checked.

// WRONG — the request class is not checked against what the target supports.
assign cache_req_route = addr_target_device;   // no capability check at all

The request is dispatched to a device with no cache-coherency participation. What happens next depends on the device and is uniformly bad: it may reject the request, producing a protocol error at an unexpected layer; it may not recognise it at all, producing a timeout; or in the most confusing case it may partially handle something it should not, producing behaviour nobody has a model for. The diagnosis is difficult because the link is fine, the address decode is fine, and the request was well formed — it was simply addressed to a party that does not speak that part of the protocol.

// Illustrative — the check that must exist.
property p_coherent_req_targets_capable_device;
  @(posedge clk) disable iff (!rst_n)
    cache_req_fire |-> cxl_cache_supported_q[target_dev];
endproperty
a_coherent_req_targets_capable_device:
  assert property (p_coherent_req_targets_capable_device);

Second — readiness from transport again.

// WRONG — the Chapter 10.3 §6 bug, in its CXL form.
assign cxl_mem_ready = ucie_link_active;

This is the same error the curriculum has now met three times, and it is worth counting them: Chapter 10.3 §6 derived endpoint visibility from the link; Chapter 10.4 §7 derived host-visible presence from a strap; and here memory readiness is derived from the link again. The shape is always identical — an easily available signal substituted for a composed condition nobody wanted to compose.

What CXL memory readiness actually requires, illustratively:

// Illustrative readiness composition, in the shape of Chapter 10.3 §5.
assign cxl_mem_ready =
    ucie_operational_q      &&   // transport can carry flits
    cxl_mapping_ready_q     &&   // the CXL protocol mapping is up
    dev_capability_known_q  &&   // §17: we know what this device supports
    cxl_mem_supported_q     &&   // and it supports memory exposure
    mem_windows_configured_q &&  // §16: its addresses have meaning
    coherent_state_init_q;       // §12: per-line state is at a defined start

Six terms, six owners, six ways to be not-ready. The last is the one with no analogue in Module 10 and it is worth dwelling on: coherence state must be initialised to a defined starting point before any coherent traffic flows. Metadata left at whatever the array powered up holding will report lines valid that were never filled, and §13's failure follows immediately — but from the very first access rather than after a race.

19. Coherence Is Not Free

The section that keeps this chapter from being marketing.

Storage. Per-line metadata, scaling with cached capacity, paid whether or not any line is ever shared (§12).

Protocol traffic. Coherence actions consume link bandwidth that carries no data. This traffic is generated by other agents' activity, so it is not proportional to your own workload.

Latency. An access that must resolve ownership before completing is longer than one that need not. The variance is often the greater cost — occasional long tails are harder to design around than a uniformly higher mean.

Ordering constraints. Coherence imposes ordering, and ordering limits reordering and buffering optimisations that a non-coherent path can use freely.

Incoming-event handling. A coherent participant must respond to events it did not initiate, promptly, which means resources reserved and a path that cannot be blocked by local work.

Deadlock and liveness. Multiple agents, multiple message classes, and dependencies between them. This is a well-known source of the hardest bugs in the field, and the analysis is genuinely difficult rather than merely tedious.

Verification state space. The multiplicative one. Every line state crossed with every incoming event crossed with every concurrent local access, across multiple agents. This is why §21 argues that coherence changes the verification problem more than it changes the transport.

20. Expansion Changes the Hierarchy, Not Just the Capacity

A point that gets skipped, and it has real consequences for system software.

Memory reached across a package link — through a protocol layer, an adapter, a physical link, and a remote memory controller — does not have the same latency and bandwidth characteristics as memory behind a local controller. It cannot; there is more between the core and the DRAM.

Adding CXL-attached capacity changes the shape of the memory hierarchy, not merely its size.

The system now has memory with more than one performance class. That is not a defect — it is the basis of memory tiering, and it can be exploited deliberately. But it means placement matters: the operating system, or the application, or something, must decide what goes where, and a system that treats all physical addresses as interchangeable will place hot data in the slow tier some fraction of the time.

No numbers are given here deliberately. Latency and bandwidth for a given attach depend on the device, the media, the link configuration, the topology, and the generation, and quoting a figure without those qualifiers would be worse than quoting none. The architectural fact — more than one performance class in one address space — is what transfers.

21. Verification: Three Layers, Three Contracts

The idea that reorganises verification for anyone arriving from Modules 9 and 10.

The transport layer. UCIe delivered every flit correctly: CRC passed or retry recovered, credits conserved, ordering preserved where required. Chapters 9.4 and 9.5 built the models. This layer's scoreboard is complete and it is insufficient.

The protocol layer. CXL's own rules were followed: legal transitions, legal responses, obligations discharged. This is the specification's territory and Chapter 11.3's subject.

The system-semantic layer. Every agent observed a value that was legitimate given the true history of the address. This is the only layer at which §13's bug is visible.

22. The Coherence Scoreboard

What a system-semantic model must hold. Not a full coherence simulator — that is a different and much larger undertaking — but the minimum that makes coherence failures visible.

per address:
  true_value        — the authoritative current value, from the global history
  holders[]         — which agents are believed to hold a copy, and how
  last_writer       — which agent most recently modified it
  outstanding[]     — coherence actions in flight for this address
 
per agent:
  observed_reads[]  — every value each agent actually read
  issued_writes[]   — every value each agent actually wrote

The checks:

Every read returned a legitimate value. Given the true history of the address and the ordering rules in force, was the value the agent observed one it was permitted to observe? This is the check that catches §13, and it is the only one that does.

No agent holds a copy it should have relinquished. The model's holders[] versus the devices' actual metadata, compared at quiescent points. Divergence there is a coherence-state bug caught at the moment the state diverges, rather than when a stale read later happens.

Every obligation was discharged. If an agent held the only current value and gave up the line, the value must have been supplied somewhere. §14's second assertion catches the local case; only the model catches the case where the writeback was issued and then lost.

No coherence action was lost across a transport event. This is where the transport model earns its place alongside the coherence model rather than beneath it. A coherence action lost during a UCIe recovery leaves the system's distributed state permanently inconsistent, and — the point that ties this chapter back to Module 10 — a transport-only model records that recovery as a success.

Which is why §21's three layers are three models and not one, and why the injection that matters most is a transport disturbance during an in-flight coherence exchange.

23. What Assertions Can and Cannot Prove

Stated plainly, because overclaiming here is common and expensive.

Assertions can prove local invariants. A hit requires a valid line. A dirty line is a valid line. A coherent request targets a capable device. A memory address matches exactly one window. A modified line is not dropped without a writeback. These are cheap, they are strong, and they catch real bugs early — write all of them.

Assertions cannot prove coherence. Coherence is a property of a distributed history across multiple agents, and no property expressed over the signals at one agent's interface can express it. The §13 bug satisfies every local assertion in this chapter perfectly, because from the device's own point of view nothing whatsoever is wrong: the line is present, it is being read, the data returned is the data stored.

Assertions prove that each agent is internally consistent. Only a system model — or a formal model of the whole protocol — proves that the agents agree.

This is the same rule the curriculum has applied throughout, and coherence is where the gap between the two is widest.

24. Failure Signatures

Diagnostic separation is the practical value of §21's three layers.

The first and second rows are the pair to internalise. They are the two most common serious failures, they have nothing to do with each other, and the first one produces no evidence at the layer where engineers habitually look.

25. State Lifetimes

Two rows are new to the curriculum and both matter. Per-cache-line state is indexed by address rather than by request, link, or enumeration — a genuinely different kind of lifetime, and the reason coherence state cannot simply be swept on a link event. And the coherent epoch is the first lifetime in this entire curriculum that is not local to one die: it is the period over which multiple agents' distributed state is mutually consistent, and it ends when any of them stops agreeing.

26. Common Misconceptions

"CXL replaces PCIe." It is built on it. CXL.io is described by the Consortium as based on PCIe and is present in every device type; the three protocols are multiplexed over that foundation. Module 10 is a prerequisite, not an alternative.

"CXL is just faster PCIe." Rate is not the difference. The difference is a class of semantics — coherent caching and host-accessible device memory — that no amount of bandwidth provides.

"Coherent memory means no metadata is required." The opposite: coherence is made of metadata. §10 lists the three questions that cannot be answered from data, and §12 is what answering them costs.

"A UCIe link automatically makes a device CXL-capable." UCIe transports; CXL semantics are implemented and negotiated by the protocol layers at each end. §6 is explicit, and §18 shows the RTL form of believing otherwise.

"CXL.mem and CXL.cache are the same thing." They are not, and the direction is what people invert. CXL.cache is the device caching host memory; CXL.mem is the host accessing device-attached memory, with the host managing the associated coherency flows.

"A memory-capable device must also support cache coherency." A Type 3 memory expander is a CXL.mem target and not a CXL.cache source. This is exactly why capability is checked rather than assumed.

"More coherence is always better." §19 lists seven distinct costs. For batched, coarse-grained work the explicit-movement model is very likely faster, simpler, and dramatically cheaper to verify.

"A clean UCIe transport proves coherent correctness." It proves the transport. The §13 bug produces perfect transport telemetry and wrong data, which is the single most important sentence in this chapter.

"Memory expansion only changes capacity." It changes the shape of the hierarchy. More than one performance class now exists in one address space, and placement becomes a decision someone has to make.

"CXL readiness equals UCIe ACTIVE." Six independent conditions in §18, of which the transport is one — and the one with no Module 10 analogue is that per-line coherence state must be initialised before any coherent traffic flows.

27. Understanding Check

28. Summary and What Comes Next

PCIe connects devices; CXL can integrate memory and caches into a shared coherent system architecture.

The architecture: three protocols on a PCIe foundation — CXL.io for discovery, configuration, register access and interrupts; CXL.cache as an agent coherency protocol supporting device caching of host memory; CXL.mem allowing the host to access device-attached memory with the host managing the coherency flows. Three device types following directly from which protocols a device implements, with CXL.io in all of them. And UCIe carries CXL natively — but a UCIe link does not make a device CXL-capable, because transport is not semantics.

The mechanisms: per-line state encoding permission and obligation, because a coherent participant cannot answer a local read from data alone. Metadata as the physical cost of coherence, paid per line whether or not sharing occurs. Memory windows decoded one-hot, the same structure as Chapter 10.4 with higher stakes, because the traffic is now CPU loads and stores. Capability represented and checked, because memory-capable and cache-coherent vary independently. And readiness composed from six conditions, one of which — initialised coherence state — has no analogue anywhere in Module 10.

The sentence to carry: a clean UCIe transport is entirely consistent with total coherence failure. The stale-line bug produces perfect telemetry at every layer that reports anything, and wrong data. Assertions prove each agent is internally consistent; only a system model proves the agents agree.

And the judgement that keeps this honest: coherence is not free and not universally better. Seven costs, and for coarse batched work explicit movement wins on every one of them. CXL exists because some systems need the semantics, not because coherent attach is superior.

The most tangible case is the one this chapter has deliberately left at the level of why: a memory-expansion chiplet whose capacity becomes part of the host memory map, with the flows, the address translation, and the integration that requires:

• 11.2 — Memory Expansion Over CXL — CXL.mem and the coherent memory pool.

Browse the full path on the UCIe tutorials index.