Memory Expansion Over CXL

Chapter 11.1 argued that adding a device with a lot of memory does not increase the amount of memory a system has. That was the gap. This chapter is about the mechanism that closes it.

The mechanism is less glamorous than it sounds. Almost none of the difficulty is in moving bytes across the package — Modules 7 through 9 already did that, and did it well. The difficulty is that the host must now answer, for every single load and store the cores issue, a question it never had to ask before: does this address belong to memory behind my own controllers, or to memory on some other die, and if the latter, which one?

Getting that answer wrong does not produce a link error. It produces a store landing in the wrong device's memory.

1. The One-Sentence Model

Memory expansion is an address-ownership and routing problem before it is a bandwidth problem.

Capacity is the outcome. Ownership of host physical address ranges is the mechanism, and every serious failure in this chapter is a failure of ownership: two windows claiming one address, a window claimed before its media can serve it, a request whose ownership record was discarded while software still waited for it.

2. Sourcing, and Where the Line Is

3. HDM and PDM — the Distinction That Makes Expansion Possible

Before any routing, the vocabulary. The CXL specification draws a line between two kinds of memory that can sit on a device, and the whole chapter lives on one side of it.

Host-managed Device Memory (HDM) is defined in the specification as device-attached memory mapped to system coherent address space and accessible to the Host using standard write-back semantics.

Private Device Memory (PDM) is defined as device-attached memory not mapped to system address space or directly accessible to the Host as cacheable memory — and the specification names memory on PCIe devices as being of this type.

Read those two definitions next to each other and the entire argument of Chapter 11.1 §2 becomes a one-line distinction:

The specification is explicit that memory on a CXL device can be mapped as either. A GPU with attached graphics memory treating that memory as private is the PDM case, and it is a perfectly good design. HDM is what makes the memory chiplet in your package into memory rather than into a device that contains some.

4. What the Host Must Actually Know

The mental model, stated as the set of facts the host's memory path must hold before it can serve a single load:

• which host physical addresses belong to local memory controllers;
• which belong to CXL-attached memory;
• for each of those, which device owns the range;
• whether a range is interleaved across several devices, and by what rule;
• whether the owning device is present, configured, and able to serve the range;
• and what happens when that memory becomes unavailable with work outstanding.

Six facts, and only the second and third are what people picture when they hear "memory expansion". The fourth is §22. The fifth is §12. The sixth is §17, and it is the one that produces silent system hangs.

5. Two Kinds of HDM, and Why the Revision Matters

A digression that is not optional, because a chapter that said "HDM" without it would flatten a distinction that changes what the hardware must do.

CXL Consortium material describes two use cases for CXL.mem sharing the HDM term with different protocol requirements, and distinguishes them by suffix: a host memory expander's memory is host-only coherent — written HDM-H — while accelerator memory exposed to the host is device-managed coherent, written HDM-D.

This chapter is about the HDM-H shape. A memory chiplet exposes capacity; the host manages the coherency flows for accesses to it; the device is a CXL.mem target and not a coherent caching agent. That is Chapter 11.1 §4's asymmetry, and it is what makes a routing-and-lifetime chapter possible without a coherence protocol.

HDM-D is where Chapter 11.3 lives, and the revision nuance sits here too. In CXL 1.0 the coherency model for device-attached memory was Bias Based coherency, with the specification describing memory as being in host bias when it is expected to be accessed mainly by the Host and device bias when mainly by the device. The CXL 3.0 white paper states that CXL 3.0 introduced the ability to back-invalidate the host's caches, that this model is called enhanced coherency, and that it replaces Bias Based coherency introduced in previous generations — with a Type 2 device then able to implement a snoop filter for HDM address ranges. The Consortium's feature summary places enhanced coherency on the 256-byte flit format.

So "how coherency works for device-attached memory" has a different answer in CXL 1.1 than in CXL 3.x, and a design or a testbench written against one is not automatically right against the other. This chapter deliberately avoids depending on either answer, because routing and lifetime are the same problem in both.

6. The Path an Address Takes

Read the figure as one decision with three outcomes and two very different downstream costs. The local branch is short and synchronous by the standards of the system. The CXL branch crosses a protocol mapping, an adapter, a physical link, a remote controller, and media — and every one of those is a place where state must be held on behalf of a request that has not finished.

The unmapped branch is not a rounding error. An address in no window is a real, defined situation with platform-defined handling, and §9's assertion depends on it being legal.

7. The Memory Window

// ILLUSTRATIVE host-side integration state — architecture teaching, NOT the
// CXL HDM Decoder register format. No CXL register layout, bit position or
// field width is asserted here. Compare Chapter 10.4 §10: the same decode
// structure, but the traffic is now CPU loads and stores rather than MMIO.
typedef struct packed {
  logic              valid;    // this window currently owns its range
  logic [ADDR_W-1:0] base;     // first host physical address in the window
  logic [ADDR_W-1:0] limit;    // LAST address in the window — inclusive, see §8
  logic [DEV_W-1:0]  target;   // which CXL memory device owns it
} cxl_mem_window_t;
 
cxl_mem_window_t mem_window_q [NUM_WINDOWS];
 
logic [NUM_WINDOWS-1:0] window_match;
 
always_comb begin
  for (int i = 0; i < NUM_WINDOWS; i++)
    window_match[i] = mem_window_q[i].valid &&
                      (addr >= mem_window_q[i].base) &&
                      (addr <= mem_window_q[i].limit);
end

Architecture. This is the crossing between two graphs, in the sense of Chapter 10.4 §2. The input is a host physical address, which is a software-visible concept the package knows nothing about. The output is a device on the other side of a link, which software knows nothing about. Every CPU access to expanded memory passes through here.

State. NUM_WINDOWS entries with per-memory-window lifetime — established when a range is configured into the system, invalidated when it is reconfigured or the memory is removed. This lifetime is distinct from the UCIe link epoch and from any individual request, and §29 is the table that keeps the three apart.

Cycle behaviour. A combinational match per request, in the memory path — which is the most timing-critical path in the system, far more so than Chapter 10.4's configuration decode. Real implementations pipeline this, often pre-decode a coarse region first, and sometimes carry the decision alongside the request rather than recomputing it. Whatever the structure, the window must be stable while a decode is in flight, and that is an assertion rather than a hope.

Contract. The fabric relies on exactly one destination per address. The device relies on receiving only addresses inside the range it was told it owns. Software relies on the range it configured being the range the hardware decodes.

Failure. Three of them, and separating them is most of the debug value. No match is the unmapped path — defined, and not this decode's problem. Match on a stale window delivers a CPU access to a device that no longer owns the address, which is silent misdelivery. Multiple matches is §10, and it is the serious one.

DV. Base, base−1, limit, limit+1 for every window. The comparison operators are where the bugs are, and a directed test that picks addresses in the middle of a window will never see them.

8. Inclusive, Exclusive, and the Off-by-One Nobody Notices

A short section that prevents a long bug, and it is worth being exact rather than idiomatic.

The CXL window definition is half-open. The CXL Consortium's errata to the CXL 2.0 specification states the condition for the set of host physical addresses decoded by an interleave entry as:

Base HPA  <=  HPA  <  Base HPA + Window Size

Base included, top excluded. The RTL in §7 stores an inclusive limit instead, and matches with <=. Those two descriptions are equivalent only if the conversion is done exactly once and in the right direction:

// Illustrative conversion, done ONCE at configuration time — not per access.
// A window of SIZE bytes based at BASE occupies BASE .. BASE+SIZE-1 inclusive.
mem_window_q[i].limit <= base_in + size_in - 1'b1;   // note the -1

Why store the inclusive limit rather than the size. Because a per-access addr < base + size recomputes an addition in the timing-critical path and reintroduces the overflow hazard that a window ending at the top of the address space triggers: base + size wraps to zero, the comparison inverts, and the window matches nothing or everything. Precomputing the inclusive limit does the addition once, at configuration time, where it can be checked.

The three ways this goes wrong, and each has a distinct signature:

All three pass any test whose addresses are not at a boundary. That is the entire reason §7's DV note lists four addresses per window rather than "some addresses in the window".

9. SVA — Exactly One Target, or None

// Illustrative. The single most valuable assertion in this chapter, and it
// costs one expression. Chapter 10.4 §12 made the same argument for MMIO
// apertures; here the traffic being misrouted is a CPU load or store.
property p_window_match_onehot;
  @(posedge clk) disable iff (!rst_n)
    mem_req_fire |-> $onehot0(window_match);
endproperty
a_window_match_onehot: assert property (p_window_match_onehot);
 
// If the decode spans more than one cycle, the windows must not move under it.
property p_window_stable_in_flight;
  @(posedge clk) disable iff (!rst_n)
    (decode_in_flight && !decode_done) |=> (window_match == $past(window_match));
endproperty
a_window_stable_in_flight: assert property (p_window_stable_in_flight);
 
// A window that is not valid must never contribute a match. Catches a decode
// that compares base and limit before checking validity — which matches the
// reset-value range 0..0 and therefore claims address zero.
property p_invalid_window_never_matches;
  @(posedge clk) disable iff (!rst_n)
    !mem_window_q[i].valid |-> !window_match[i];
endproperty
a_invalid_window_never_matches: assert property (p_invalid_window_never_matches);

$onehot0, not $onehot, and the choice is the interesting part.

Zero matches is legal. An address in no CXL window is either local memory or unmapped, and both are real situations with defined handling. Asserting $onehot would fire on every local access, which is most of them.

Two matches is not legal under any interpretation. There is no reading of the architecture in which one host physical address is owned by two memory devices. That asymmetry is what makes this assertion both cheap and unambiguous — it has no false-positive mode to tune.

What it catches, and when. It fires at the moment the configuration becomes inconsistent and an access lands in the overlap — not when the corruption is eventually noticed. And it requires no model of the topology, which means it works in a unit testbench with no scoreboard at all.

What it does not catch. Whether the single matching window points at the correct device. That is a mapping-correctness question and only §30's reference model can answer it.

10. Wrong RTL — Overlapping Memory Windows

// WRONG — the loop assumes at most one match and silently resolves the case
// it did not expect. Nothing here prevents two windows covering one address.
always_comb begin
  route_dev = '0;
  for (int i = 0; i < NUM_WINDOWS; i++)
    if (window_match[i]) route_dev = mem_window_q[i].target;   // last wins
end

Architecture. The code is not defending the invariant it depends on. It is a priority resolution written as though it were an unambiguous selection.

Cycle behaviour. For an address in one window, correct. For an address in two, it takes the highest-numbered entry — deterministically, which is worse than randomly, because it will be consistent and therefore pass every test that does not specifically probe the overlap.

Failure, and this is the most severe failure in the chapter. Two devices claim one host physical address. Depending on which entry wins:

• A store lands in the wrong device's memory. The data is now in a location no software will look at, and the location software did intend still holds its old value.
• A load returns another device's data. The value is well-formed. It is simply from the wrong place.
• The blast radius is whatever the operating system placed in that range — which could be page tables, a file cache, or an application's heap. Nothing about the symptom points at the memory map.

No error is reported anywhere. The UCIe link is healthy: no CRC error, no retry, no credit violation. The CXL request was well formed and carried a legal address. The receiving device saw an address inside the range it believes it owns and served it correctly. Every party behaved correctly given what it was told, and what one of them was told was wrong.

This is the same shape as Chapter 10.2 §7 and Chapter 10.4 §11 — every field individually valid, the composition wrong — and it is the shape this curriculum keeps returning to because it is the family of bug that survives every mechanism designed to catch errors.

The fix has two halves and needs both. At configuration time, validate that a new window does not overlap an existing valid one, and refuse or report if it does:

// Illustrative configuration-time check. Two inclusive ranges overlap unless
// one ends strictly before the other begins.
function automatic logic ranges_overlap(
  input logic [ADDR_W-1:0] a_base, a_limit, b_base, b_limit
);
  return !((a_limit < b_base) || (b_limit < a_base));
endfunction

At run time, assert §9's one-hot property. Validation catches the error where it is caused; the assertion catches the window that was updated without going through the validation path. Designs acquire the second path — a debug backdoor, a firmware patch, a reset sequence that writes the array directly — more often than anyone plans to.

11. The Route Is Architectural State, Not a Boolean

// Illustrative internal classification. NOT a CXL encoding. The value exists
// because "is this remote" is not one bit: unmapped is a third outcome with
// its own handling, and collapsing it into either of the others hides a bug.
typedef enum logic [1:0] {
  MEM_LOCAL    = 2'd0,   // behind a local memory controller
  MEM_CXL      = 2'd1,   // an HDM range on a CXL device, reached over UCIe
  MEM_UNMAPPED = 2'd2    // in no window — platform-defined handling
} mem_route_t;
 
mem_route_t route;
 
always_comb begin
  if (window_match != '0)        route = MEM_CXL;
  else if (local_range_match)    route = MEM_LOCAL;
  else                           route = MEM_UNMAPPED;
end

Architecture. Three outcomes rather than two, because the third is a real case that needs a defined response. A design that computes is_remote as one bit has silently merged unmapped with local, which means an address in no window at all is quietly sent to a local memory controller.

State. Per-request, derived — it holds no state of its own but must be carried with the request rather than recomputed downstream. Recomputing it after the window array may have been reprogrammed is the same class of error as §9's stability property.

Cycle behaviour. Combinational, and the priority is deliberate: a CXL window match wins over a local range match. That ordering encodes an assumption — that the two are mutually exclusive by construction — which is worth asserting rather than relying on:

// Illustrative — local and CXL ranges must not both claim an address.
property p_local_and_cxl_disjoint;
  @(posedge clk) disable iff (!rst_n)
    mem_req_fire |-> !(local_range_match && (window_match != '0));
endproperty
a_local_and_cxl_disjoint: assert property (p_local_and_cxl_disjoint);

Failure. Merging unmapped into local sends a stray address at a local controller, which may absorb it, or report it, or hang — and none of those outcomes is attributable to the memory map, because the map is not in the path of the symptom.

DV. Cover all three route values, and cover the transition of a single address from MEM_CXL to MEM_UNMAPPED when its window is invalidated. That transition is where a stale carried route decision shows up.

12. Present Is Not Usable

The section that recurs in every integration chapter of this curriculum, in its memory form — and the consequences are worse here than anywhere before it, because the client is a CPU load.

A device being physically present does not make its memory usable. There is a chain of independent facts, each with a different owner and a different way of being false:

Six facts, six owners, six independent ways to be not-ready. The last one has no analogue in Module 10 at all, and it is the one that produces the most confusing failures: a link that is up, a window that is configured, a device that is present and capable — and memory that cannot yet answer.

Note also that capacity discovery uses a different protocol from the memory itself. The device is discovered, configured, and interrogated through the CXL.io path — the PCIe-shaped mechanism Chapter 11.1 §5 established is present in every device type. The memory is then reached through CXL.mem. So the readiness chain crosses two protocols, and a failure in the first is visible as an absence in the second.

This chapter deliberately does not invent the register names, capability offsets, or reporting formats through which capacity becomes known. Those are specification detail, and approximating them would be worse than naming the category.

13. The Readiness Vector

// Illustrative readiness composition, in the shape of Chapter 10.3 §5 and
// Chapter 11.1 §18. NOT a CXL register — each term stands for a fact the
// integration must establish, and each has a different owner.
logic ucie_operational_q;    // transport can carry flits          (Ch 8, 9)
logic cxl_mapping_ready_q;   // CXL over UCIe negotiated and up     (Ch 11.4)
logic dev_mem_capable_q;     // this device exposes HDM             (Ch 11.1 §17)
logic dev_capacity_known_q;  // we know how much, via CXL.io
logic mem_window_valid_q;    // §7 — its addresses have meaning
logic media_ready_q;         // the far side can actually serve
 
assign cxl_mem_ready =
    ucie_operational_q   &&
    cxl_mapping_ready_q  &&
    dev_mem_capable_q    &&
    dev_capacity_known_q &&
    mem_window_valid_q   &&
    media_ready_q;

Architecture. A conjunction, not a signal. The value of writing it out is that each term names a mechanism you can go and look at when readiness is unexpectedly low, and the debug question becomes which term rather than why.

State. Six bits with per-device lifetime, except ucie_operational_q which has per-link-epoch lifetime and mem_window_valid_q which has per-window lifetime. Three different lifetimes in one conjunction is not sloppiness — it is the actual situation, and §29 is where it is laid out.

Cycle behaviour. Each term is set by its own owner at its own point in bring-up. The conjunction is combinational and must be stable before routing is enabled, which is §19's ordering point.

Contract. The memory path may only route a request to a device whose cxl_mem_ready is asserted. Software, which sees none of this, relies on the range being absent from the memory map until it is genuinely usable.

Failure. Any missing term admits requests the far side cannot serve. §14 is the specific case.

DV. Cover each term false with the others true — six directed cases — and cover the transition of each term from false to true with traffic pending. The second set is where an enable that was sampled once rather than held shows up.

14. Wrong RTL — ACTIVE Means Memory Is Ready

// WRONG — and this is the fourth time this curriculum has met this shape.
assign cxl_mem_ready = ucie_link_active;

Architecture. Count the previous appearances, because the count is the lesson. Chapter 10.3 §6 derived endpoint visibility from the link. Chapter 10.4 §7 derived host-visible presence from a strap. Chapter 11.1 §18 derived CXL memory readiness from link-active. And here it is again, with a CPU load as the client. The shape is always identical: an easily available signal substituted for a composed condition nobody wanted to compose.

Cycle behaviour. One assignment. It reads as obviously correct, and in a bring-up where the link comes up last it even works.

Failure. The window is configured, the link is active, and a core issues a load into the range before the far side's media controller is initialised. What happens next is device-specific and uniformly unhelpful:

• The request is answered with something meaningless. If the design is badly broken, uninitialised media contents are returned as data, and the CPU has a value it will use.
• The request is answered with an error indication, which at least surfaces — the CXL.mem request field set includes a Poison bit that the specification describes as indicating the data contains an error, with handling described as device-specific. That the mechanism exists does not mean the design is using it correctly.
• The request is not answered at all, and §17's timeout path is what eventually resolves it. From software's point of view a load has stalled for an unbounded time.

No specific CXL response is asserted here, deliberately. What the far side must return for an access to memory that is not yet ready is a specification and implementation matter, and inventing it would be exactly the fabrication §2 forbids.

Why it is worse than the Module 10 version. A misdirected MMIO access fails a device driver. A load into unready memory fails in the CPU's data path, in the middle of whatever code happened to touch the range, and the value may propagate arbitrarily far before anything notices.

DV. Configure the window with media_ready_q low and issue traffic. Verify no request leaves the host. Then assert media_ready_q and verify the same traffic completes — the pair is what proves the gate exists rather than that it happened to be true.

15. The Request Object

// Illustrative internal representation of a host memory request crossing the
// integration boundary. NOT a CXL.mem message format: no opcode encoding,
// field width, or header layout is asserted. Symbolic widths throughout.
typedef struct packed {
  logic [ADDR_W-1:0] addr;        // host physical address
  logic [DATA_W-1:0] data;        // write data; ignored for reads
  logic [BE_W-1:0]   byte_valid;  // which bytes a partial write touches
  logic              write;
  logic [ID_W-1:0]   id;          // host-side identity — see §16
  logic [DEV_W-1:0]  target;      // §7's decode result, carried not recomputed
  mem_route_t        route;       // §11, carried for the same reason
} mem_req_t;

Architecture. Address, data, extent, direction, identity, destination. The five that must travel together and one that must not be recomputed.

Note byte_valid explicitly. Partial writes are not an edge case in a CPU memory path, and the CXL specification's own treatment is instructive: the CXL.cache/CXL.mem link layer optimises for the common case by not transmitting byte-enable bits when all bytes are enabled, clearing a field in the flit header instead, with the receiver required to regenerate the all-ones value. A design that carries byte_valid but ignores it somewhere in the middle produces a full-line write where a partial write was intended — which overwrites bytes nobody asked to change, and is undetectable by any check on the request itself.

Note that target and route are carried. Chapter 10.2 §7's rule — fields consumed together travel together, in one object advanced by one enable — applies with a memory-specific consequence. Pair address N with the decode result of address N−1 and you have constructed a well-formed memory request aimed at the wrong device, which is §10's failure produced by a pipeline-depth bug instead of by a configuration bug.

// Illustrative — one object, one enable, misalignment impossible by construction.
mem_req_t req_q, req_q2;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    req_q  <= '0;
    req_q2 <= '0;
  end else if (pipe_en) begin
    req_q  <= req_in;
    req_q2 <= req_q;
  end
end

// Illustrative — the whole object holds still under backpressure.
property p_req_stable_under_stall;
  @(posedge clk) disable iff (!rst_n)
    (req_valid && !req_ready) |=> ($stable(req_q) && $stable(req_valid));
endproperty
a_req_stable_under_stall: assert property (p_req_stable_under_stall);

16. Outstanding Requests, and the Identity That Tracks Them

A read to remote memory does not complete when it is sent. It completes when data returns, an unknown number of cycles later, and the host must hold enough state to recognise that data when it arrives.

// Illustrative host-side outstanding-request tracking. The IDENTITY REFLECTION
// contract is verified CXL.mem behaviour (§16 callout); the table structure,
// widths and allocation policy are illustrative internal architecture.
typedef struct packed {
  logic              valid;
  logic [ID_W-1:0]   id;        // reflected by the far side in its response
  logic [ADDR_W-1:0] addr;      // kept for checking and for diagnostics
  logic [DEV_W-1:0]  target;    // which device owes this
  logic              write;
} outstanding_mem_t;
 
outstanding_mem_t outstanding_q [MAX_OUTSTANDING];
 
// Per-device accounting, because devices and their links fail independently.
// Chapter 10.4 §16's argument, and the same unique-case discipline.
logic [OUT_W-1:0] out_count_q [NUM_CXL_DEVICES];
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int d = 0; d < NUM_CXL_DEVICES; d++) out_count_q[d] <= '0;
  end else begin
    unique case ({req_alloc_fire, rsp_retire_fire})
      2'b10: out_count_q[req_dev] <= out_count_q[req_dev] + 1'b1;
      2'b01: out_count_q[rsp_dev] <= out_count_q[rsp_dev] - 1'b1;
      2'b11: begin
        // Simultaneous allocate and retire. Same device nets to zero;
        // DIFFERENT devices must both move. Writing this as two independent
        // if-statements is Chapter 9.5 §6's bug and loses one update.
        if (req_dev == rsp_dev) begin
          out_count_q[req_dev] <= out_count_q[req_dev];
        end else begin
          out_count_q[req_dev] <= out_count_q[req_dev] + 1'b1;
          out_count_q[rsp_dev] <= out_count_q[rsp_dev] - 1'b1;
        end
      end
      default: ;
    endcase
  end
end

Architecture. An entry table for matching, plus per-device counters for resource control and for answering the only question that matters when a link fails: how much of what I am owed was owed by the device behind the link that just went?

State. MAX_OUTSTANDING entries with per-request lifetime, allocated before transmission and freed exactly once on resolution. The counters are the aggregate, and they are a resource count in the Chapter 9.5 §8 sense — assert the bound, never saturate it, because a count that clamps has already discarded the information that would explain the failure.

Cycle behaviour. Allocation gates acceptance (§27). Retirement is driven by a matched response. The unique case exists because both can happen in one cycle, on the same device or on different ones, and only the same-device case nets to zero.

Contract. Allocation before transmission. Exactly one retirement per entry. The far side reflects the identity.

Failure. Undercount over-commits a device. Overcount throttles a healthy one, producing a performance bug that looks like a link problem. Losing an update through the two-if bug makes the count drift monotonically, so the device appears progressively more congested until it stops accepting entirely — with the cause thousands of transactions in the past.

DV. Each counter empty, mid-range, and at its limit. Simultaneous allocate and retire on the same device and on different devices — the two branches of the 2'b11 case, and the second is the one regressions skip.

17. A Device Becomes Unreachable With Reads Outstanding

The scenario that separates a memory-expansion design from a block diagram, and it is worse than Chapter 10.4 §17's version because of who is waiting.

A core issued a load. The load is outstanding to a CXL memory device. The UCIe link enters recovery, or fails.

A core is stalled on that load. Not a driver waiting on a completion — a core, in the middle of executing something, with an instruction that cannot retire. The host has a record saying it is owed data. The transport that would carry that data is, for the moment, not there.

The architecture must define, in advance:

Whether recovery is attempted, and for how long. UCIe's own recovery machinery may bring the link back. If it does, and if both sides retained their outstanding state, the transaction completes normally and nothing above ever learns that anything happened. This is the good case and it is worth designing for — it is the entire reason §18 is a bug rather than a simplification.

What resolves the request if recovery does not come. The expectation must end somehow. CXL and the platform define error-reporting and recovery behaviour for unrecoverable conditions; this chapter does not restate their thresholds, encodings, or escalation rules, because doing so accurately requires the specification text and approximating them would be worse than omitting them.

Whether the host waits or abandons. Two different designs with two different failure modes: abandoning early breaks transactions that would have completed after a normal recovery; waiting indefinitely converts a link event into a machine that has stopped.

18. Wrong RTL — Clear Outstanding State on UCIe Recovery

// WRONG — the link's lifetime applied to state that does not share it.
always_ff @(posedge clk) begin
  if (ucie_link_reset[l]) begin
    for (int e = 0; e < MAX_OUTSTANDING; e++)
      if (outstanding_q[e].target == link_dev[l]) outstanding_q[e].valid <= 1'b0;
    out_count_q[link_dev[l]] <= '0;         // "clean slate"
  end
end

Architecture. The reasoning is superficially sound and has correct precedents. Chapter 9.5 §13 said exactly this about credits. Chapter 9.4 said it about replay state. Both were right, because credits and replay entries are link-epoch state. An outstanding memory request is per-transaction state with an owner above the link, and applying a link-epoch rule to it is a category error.

Cycle behaviour. One pulse and the table is clear. The RTL reads as tidy housekeeping, and it is short enough to survive review.

Failure, in escalating order.

The host now believes it is owed nothing. A core is still stalled. The two models have diverged, and the divergence is silent, because from the transport's point of view a reset is a normal part of recovery.

If the link recovers and the far side retained its state, data arrives for a request the host has forgotten. It matches nothing. Depending on the response path it is dropped — leaving the core stalled forever — or, far worse, it matches a later request that has since reused the same identity, delivering one requester's data to another. That is §10's silent-misdelivery family arriving by a completely different route.

And the accounting is now permanently wrong in the other direction. Entries that were counted and cleared will never decrement again, so the counter underflows on the next legitimate response. If it saturates rather than asserting, the underflow becomes a large positive value and the device appears permanently congested — a performance failure whose cause was a link event minutes earlier.

The right shape. Link-epoch state re-initialises with the link. Per-transaction state persists and is resolved by §17's policy — completed after recovery, or abandoned through an explicit path that tells the requester. The two must be separated in the RTL, not only in the designer's head, because the next person to read the file will apply whichever rule the code appears to be following.

// Illustrative — encode the contract so the coupling cannot be added later.
property p_outstanding_survives_link_reset;
  @(posedge clk) disable iff (!rst_n)
    (ucie_link_reset[l] && out_count_q[link_dev[l]] != '0)
      |=> (out_count_q[link_dev[l]] == $past(out_count_q[link_dev[l]]));
endproperty
 
property p_outstanding_bounded;
  @(posedge clk) disable iff (!rst_n)
    out_count_q[d] <= MAX_OUTSTANDING_PER_DEV;
endproperty
 
// One device's link event must not disturb another's accounting — a common
// and easily missed secondary bug when the clear loop is written by hand.
property p_link_reset_isolated;
  @(posedge clk) disable iff (!rst_n)
    (ucie_link_reset[l] && (m != l))
      |=> (out_count_q[link_dev[m]] == $past(out_count_q[link_dev[m]]));
endproperty

DV. Inject link loss with a non-zero outstanding count on the affected device and a non-zero count on a different one. Check the other device is untouched, and that every request on the affected device reaches a resolution the scoreboard can observe.

19. Matching a Response, Exactly Once

// Illustrative response matching. The reflected-identity contract is verified
// CXL.mem behaviour (§16); the table lookup and the retire pulse are not.
logic                  rsp_hit;
logic [ID_W-1:0]       rsp_id;
logic [OUT_IDX_W-1:0]  rsp_idx;
 
always_comb begin
  rsp_hit = 1'b0;
  rsp_idx = '0;
  for (int e = 0; e < MAX_OUTSTANDING; e++)
    if (outstanding_q[e].valid && (outstanding_q[e].id == rsp_id)) begin
      rsp_hit = 1'b1;
      rsp_idx = e[OUT_IDX_W-1:0];
    end
end
 
assign rsp_retire_fire = rsp_valid && rsp_hit;
assign rsp_orphan      = rsp_valid && !rsp_hit;   // report it; never absorb it

Architecture. Three outcomes, and the third is the one designs forget. A response either matches a live entry, or it matches nothing. A response matching nothing is a real event that must be reported rather than dropped, because it is the observable symptom of §18's bug and of an identity-reuse error, and dropping it removes the only evidence.

State. None of its own; a lookup over §16's table.

Cycle behaviour. One retire pulse per matched response. The entry is freed exactly once.

Contract. The requester relies on receiving data for its own request and no other. §16's reflection contract is what makes that possible.

Failure. Retiring twice frees an entry that a second response then matches — which is Chapter 10.2 §9's duplicate problem in memory form. Failing to retire leaks the entry, and the table fills until the device stops accepting.

// Illustrative — a response requires a live request. The memory form of
// Chapter 10.3 §15, and it catches §18's bug from the other side.
property p_response_requires_live_request;
  @(posedge clk) disable iff (!rst_n)
    rsp_valid |-> rsp_hit;
endproperty
a_response_requires_live_request: assert property (p_response_requires_live_request);
 
// Illustrative — exactly one retirement per entry. Uses a VERIFICATION-ONLY
// monitor tag, allocated by the testbench, so that a transport replay cannot
// be mistaken for a second semantic completion (Ch 10.2 §9).
property p_retire_at_most_once;
  @(posedge clk) disable iff (!rst_n)
    rsp_retire_fire |-> !retired_mon_tag_seen[rsp_mon_tag];
endproperty
a_retire_at_most_once: assert property (p_retire_at_most_once);
 
// Illustrative — the address a response resolves must be the address that
// was requested. Catches an index/identity mismatch that returns the right
// data to the wrong entry.
property p_retire_matches_address;
  @(posedge clk) disable iff (!rst_n)
    rsp_retire_fire |-> (outstanding_q[rsp_idx].addr == rsp_expected_addr);
endproperty

On p_response_requires_live_request and its honest limit. It holds only if the design's contract is that orphaned responses cannot legally occur. If the architecture permits them after an abandonment (§17), then this must be written as a coverage item and a reporting requirement rather than an assertion — and the choice must be deliberate, because an assertion that encodes the wrong contract gets waived, and a waived assertion protects nothing.

20. Ordering, At the Right Depth

A warning against reusing Chapter 9.3's per-stream FIFO model, and a statement of what can be said exactly.

Streaming ordering was built for a protocol UCIe does not interpret, where the implementer defined the domain. CXL.mem's ordering is a specification's, and it is not that. Reproducing it here would require specification text this chapter has not inspected in full, and approximating an ordering model fails in both directions: a model stricter than the protocol reports violations that are not real, and a looser one silently accepts reorderings that break software.

Two things are verified and worth stating precisely, because they are the rules an integration most often violates.

Request classes must drain independently. The CXL 1.0 specification's CXL.mem forward-progress rules state that the request-without-data and request-with-data message classes each need to be credited independently between each hop in a multi-hop fabric, that back pressure due to lack of resources at the destination is allowed, but that these must eventually drain without dependency on any other traffic type.

That is a direct constraint on your buffering. A shared queue in which a blocked write can prevent a read from making progress creates exactly the cross-class dependency the rule forbids, and the failure it produces is not a wrong value — it is a deadlock, which safety assertions pass right through (Chapter 5.5 §12).

Some ordering is defined at cacheline granularity. The same rules state that no transaction should pass a MemRdFwd or a MemWrFwd if the transaction and that command are to the same cacheline address — the specification's stated reason being that those opcodes, although sent on the request class, are in fact responses.

The general form of that is the rule to carry: a scheduler's freedom to reorder is bounded by relationships it may not itself be able to see. A mapping layer that does not model the protocol's ordering must therefore preserve the order in which requests were accepted, per address at minimum, and treat any reordering as a decision requiring justification.

The transport must not introduce reorderings the protocol layer does not expect, and whatever order the mapping preserves must be at least as strong as what CXL.mem requires between any two requests.

21. Interleaving, and Why It Is a Routing Change Not a Performance Knob

So far one address range has had one owner. Real systems spread a range across several devices, and that changes the shape of the decode rather than just its parameters.

The consequence is worth stating as a sentence, because it reframes the problem:

Interleaving converts one routing decision into a deterministic address function. The window still says which set of devices; a field of address bits then says which one.

And the engineering consequence: the host and every device must compute that function identically. There is no negotiation per access and no error detection on the result. A mismatch is not caught by anything.

22. Illustrative Interleave Selector

// ILLUSTRATIVE interleave target selection. NOT the CXL HDM decoder format:
// no register layout, field encoding, or Interleave Arithmetic variant is
// asserted. What it models is the VERIFIED structural fact of §21 — the
// target index is a contiguous field of host-physical-address bits taken
// from the granularity boundary upward, and the boundary is never below
// HPA[8] because 256 B is the finest interleave granularity CXL defines.
module cxl_interleave_select #(
  parameter int ADDR_W  = 52,
  parameter int WAYS_W  = 3,     // ways = 2**WAYS_W; CXL material: 2, 4 or 8
  parameter int DEV_W   = 4
) (
  // Per-window interleave configuration, per-window lifetime.
  input  logic [3:0]              ig_shift,     // granularity = 2**(8+ig_shift)
  input  logic [WAYS_W:0]         ways_log2,    // 1, 2 or 3 for 2, 4, 8 ways
  input  logic [DEV_W-1:0]        target_list [2**WAYS_W],
 
  input  logic [ADDR_W-1:0]       addr,
 
  output logic [WAYS_W-1:0]       way_index,
  output logic [DEV_W-1:0]        target_dev,
  output logic                    config_legal
);
 
  // The granularity boundary. 8 is NORMATIVE (§21): the finest CXL interleave
  // granularity is 2**8 bytes, so no bit below HPA[8] selects a device.
  localparam int IG_BASE_BIT = 8;
 
  // Compile-time-ish legality, checked rather than assumed. A configuration
  // that would index past the address width, or select more ways than the
  // target list holds, must not decode at all (§9's discipline applied to
  // interleave configuration rather than to window overlap).
  wire [5:0] lsb = IG_BASE_BIT[5:0] + {2'b00, ig_shift};
  assign config_legal = (ways_log2 != '0)
                     && (ways_log2 <= WAYS_W[WAYS_W:0])
                     && ((lsb + ways_log2) <= ADDR_W[5:0]);
 
  // Extract the way index: ways_log2 bits starting at the granularity
  // boundary. Written as a shift and a mask rather than a variable part-select
  // so the widths are explicit and no unsized literal decides them for us.
  wire [ADDR_W-1:0] shifted   = addr >> lsb;
  wire [ADDR_W-1:0] way_mask  = (ADDR_W'(1) << ways_log2) - ADDR_W'(1);
  wire [ADDR_W-1:0] way_full  = shifted & way_mask;
 
  assign way_index  = config_legal ? way_full[WAYS_W-1:0] : '0;
  assign target_dev = config_legal ? target_list[way_index] : '0;
 
endmodule

Classification: synthesizable, illustrative architecture. The IG_BASE_BIT value of 8 and the 2/4/8 way range come from verified CXL material; everything else — the port list, the legality check, the target list as an array — is internal design.

Architecture. The selector sits after §7's window match and before the request leaves. Window match answers "is this range interleaved, and across which set"; this answers "which member".

State. None. The configuration inputs have per-window lifetime; the outputs are per-request.

Cycle behaviour. Purely combinational, in the timing-critical path, which is why real implementations often fold the extraction into the window match rather than cascading them.

Contract. Every agent that must reach the same line must compute the same index. That is not enforceable locally, which is §23.

Failure. §23, in detail.

On the arithmetic, because this is where width bugs live. lsb is computed in an explicitly-sized 6-bit space rather than letting an unsized literal choose. way_mask uses ADDR_W'(1) rather than 1 so the shift happens at address width instead of at 32 bits — the classic silent truncation, and on a 52-bit address it produces a mask that is correct for small shifts and wrong for large ones, which is the worst possible failure profile. And config_legal checks that the extracted field fits inside the address, because a configuration that shifts past the top yields zero, which is a valid-looking way index that always selects device 0.

23. Wrong Interleave Mapping — the Best Debug Signature in the Chapter

Suppose the host computes the way index from HPA[10:9] and a device believes the granularity is one step finer, so it expects HPA[9:8].

Both sides are internally consistent. Both compute a legal index. Neither detects anything.

What the system does. Consecutive 256-byte blocks are routed to devices in an order the two sides disagree about. For some addresses they agree by coincidence; for others they do not. The result is that a deterministic subset of addresses is served by the wrong device.

Why this is the most recognisable failure in the chapter. The corruption has periodic structure. Walk a buffer and the errors appear at a fixed stride. That stride is a direct measurement of the disagreement:

An engineer who knows to look for the stride diagnoses this in one experiment. An engineer who does not sees intermittent memory corruption and starts looking at media, at the link, and at the transport — none of which is involved.

Why no error is reported. The link is clean. The request carries a legal address. The device that receives it believes the address is inside the range it owns, because under its interleave arithmetic it is. Every party is behaving correctly.

DV. Sweep a contiguous region larger than ways × granularity and check every location against a reference model that computes the target independently. A directed test touching a handful of addresses has roughly a 1/ways chance of hitting the disagreement, which is exactly the probability profile that lets this ship.

24. Backpressure Is a Conjunction

// WRONG — a decoded route says nothing about whether the request can be held.
assign mem_req_ready = route_valid;

Between the decode and the far side sit a request queue, an outstanding-entry table, the mapping layer, and whatever transport resources the adapter needs. A valid route says none of them has room.

// Illustrative — every stage that must hold the request gets a veto.
assign mem_req_ready =
    route_valid              // §11 — the address has exactly one owner
 && cxl_mem_ready            // §13 — that owner can actually serve it
 && req_queue_space          // local buffering available
 && outstanding_space        // §16 — an entry can be PRE-ALLOCATED
 && cxl_transport_ready;     // Ch 11.4 — mapping and transport can take it

Architecture. Accepting a memory request commits every stage between here and the media. Each is independently able to be the constraint, and none is the others' proxy.

State. None of its own; a conjunction over five registered facts owned by five mechanisms.

Failure, and the diagnostic value is in the differences. Without outstanding_space the request is sent with no record, so its response is an orphan (§19) and a core waits forever. Without req_queue_space it is overwritten locally. Without cxl_transport_ready it is accepted into a mapping layer that cannot represent it, and Chapter 10.1 §10's rule — an accepted object cannot vanish — is violated. Without cxl_mem_ready it is §14.

Note that route_valid is necessary and nowhere near sufficient, which is the whole point of the section: the most locally available signal is the one most likely to be mistaken for the answer.

25. One Host Read, Cycle by Cycle

Illustrative latency throughout; the point is which state exists when, not how long any step takes.

Five things to read off it.

Cycle 3: the entry exists before the request leaves. §16's "pre-allocated" made concrete. Reverse cycles 3 and 4 and cycle 15's data can arrive with nothing to match.

Cycles 9 to 13: the outstanding entry survives the link event. This is the whole of §18 in one row. Transport state was re-baselined — correctly, it is link-epoch state — and the per-request entry was not touched.

Cycle 13: the far side's read was never interrupted. The media access does not know a link recovered. A design that cancelled the transaction on the host side would have produced a completed read on the device with nobody to give it to.

Cycle 15: the identity is what makes the data mean something. Data arriving on its own is a value with no destination.

Cycle 16: retirement is once. Not once per arrival — Chapter 10.2 §9's rule, and §19's assertion.

And the row that never appears: no cycle in which the core learns that a recovery happened. That invisibility is the product.

26. Capacity Is Not the Only Thing That Changes

Chapter 11.1 §20 made the architectural point; here is the part that has RTL consequences.

Memory reached through a protocol mapping, an adapter, a physical link, and a remote controller cannot have the same latency and bandwidth characteristics as memory behind a local controller. There is more between the core and the media. So the system now has more than one performance class inside one address space, which has four consequences a designer must plan for:

Occupancy is higher for the same throughput. Longer round-trip means more requests in flight for the same bandwidth — Chapter 9.5 §11's bandwidth-delay product, now sizing MAX_OUTSTANDING rather than a credit pool. Undersize it and the link is idle while the table is full.

The variance may matter more than the mean. A tail latency that occasionally spikes is harder to design around than a uniformly higher average, and a link event (§17) is an enormous tail.

Congestion is shared. Several address ranges behind one UCIe link contend for it. A bandwidth-hungry access pattern to one range degrades another, and nothing in the memory map suggests they are related.

Placement becomes someone's decision. This is the tiering problem, and it is a system-software one. Hardware's contribution is measurement.

No latency or bandwidth figures are given here, deliberately. They depend on the device, the media, the link configuration, the topology, and the generation, and a number without those qualifiers is worse than none.

27. Counters That Answer the Question You Will Actually Ask

// Illustrative diagnostic state. Not a CXL or UCIe register. Sized to be
// affordable, not comprehensive — the goal is to distinguish four causes
// of "remote memory is slow", not to build a monitoring product.
logic [31:0] local_req_count_q;
logic [31:0] cxl_req_count_q  [NUM_CXL_DEVICES];
logic [31:0] cxl_lat_bucket_q [NUM_CXL_DEVICES][NUM_LAT_BUCKETS];
logic [15:0] out_high_water_q [NUM_CXL_DEVICES];   // peak occupancy
logic [31:0] stall_no_entry_q [NUM_CXL_DEVICES];   // §24 vetoed: table full
logic [31:0] stall_no_xport_q;                     // §24 vetoed: transport

Architecture. Each counter exists to eliminate one hypothesis. The pair that matters most is the last two: stall_no_entry_q rising says the outstanding table is the limit — undersized for the round-trip, per §26 — while stall_no_xport_q rising says the link or its resources are, which is a different investigation on a different die.

State. Diagnostic lifetime: these must survive link recovery and device reset and clear only on a broad, deliberate reset. A counter cleared by the event you are trying to diagnose is worse than no counter, because it looks like evidence.

Cycle behaviour. Increment on the event. Latency buckets need a timestamp in the outstanding entry, which is verification-and-diagnostic state rather than functional state, and should be labelled so nobody removes it as dead logic.

Failure. Saturating silently. A counter at its maximum is indistinguishable from a counter that stopped, and both look like a quiet system.

28. State Lifetimes, Side by Side

The table this chapter exists to produce.

Three rows carry the weight. Outstanding requests must survive a link recovery — the single most consequential row, and §18's bug. Transport state must not — re-baselining credits and replay state is right, and the error is generalising it. And media readiness is device-dependent, which means it is the row you must go and find the answer to rather than assume, because both answers are plausible and they produce opposite bugs.

29. Failure Taxonomy

The second and third rows are the pair to internalise. Both are silent, both look like memory corruption, and they have completely different causes. The stride is what separates them: window overlap corrupts a contiguous region; an interleave mismatch corrupts a periodic subset.

30. The Semantic Memory Scoreboard

A transport model is not enough here, for exactly the reason Chapter 11.1 §21 gave: every flit can be delivered correctly while the value a core reads is wrong.

per modelled address:
  owner_dev        — which device the model says owns this address
  latest_value     — the value the model believes memory holds
  last_writer      — which requester wrote it, for diagnosis
 
per outstanding request:
  id, addr, dev, write, issue_time, mon_tag
 
address map model:
  windows[]        — base, limit, target, valid, INDEPENDENTLY maintained
  interleave[]     — granularity and ways, INDEPENDENTLY computed

The checks, and what each one uniquely catches.

The predicted target matches the observed target. The model computes the destination from its own copy of the window and interleave configuration — never by reading the design's window_match or calling its selector. A model that reuses the design's arithmetic agrees with it about §22's width bug and §23's granularity mismatch, which are precisely the bugs worth catching. This is the check that catches misrouting, and nothing else does.

Every read returned the value the model last wrote to that address. The end-to-end check. It catches §10 and §23 as data errors even if the target prediction were somehow also wrong.

Every request was retired exactly once, and only against a live entry. §19's assertions have local scope; the model catches the case where a request was retired against the wrong entry and both entries later resolved plausibly.

Every accepted request eventually reached a defined resolution. Completion or explicit abandonment. This is the check that fires on §18, and note it fires at the end of the test rather than at the moment of the bug — which is why §19's orphan reporting matters as an earlier signal.

Writes commit at a defined point. The model must pick one: at acceptance, or at completion. Either is defensible; not choosing produces a scoreboard that reports failures whose only cause is its own ambiguity, and those get waived, and then the real ones get waived with them.

What this scoreboard deliberately does not do. It holds one value per address with one owner. It does not model multiple cached copies, ownership transfer, or sharers — that is Chapter 11.3's coherence model, and building it here would be building the wrong chapter's checker.

31. Coverage

// Illustrative coverage for host-side memory expansion. Not CXL- or
// UCIe-defined. Every bin exists to reach a bug named in this chapter.
covergroup cg_cxl_mem_expansion @(posedge clk iff mem_req_event);
 
  cp_route     : coverpoint route {
    bins local = {MEM_LOCAL}; bins cxl = {MEM_CXL}; bins unmapped = {MEM_UNMAPPED};
  }
  cp_boundary  : coverpoint addr_position_in_window {
    bins first = {0}; bins last = {1}; bins inside = {2};
    bins below = {3}; bins above = {4};        // base-1 and limit+1 (§8)
  }
  cp_way       : coverpoint way_index;          // every interleave way (§22)
  cp_ways_cfg  : coverpoint ways_log2 { bins two = {1}; bins four = {2}; bins eight = {3}; }
  cp_ig        : coverpoint ig_shift;           // every granularity (§21)
  cp_out_occ   : coverpoint out_count_q[req_dev] {
    bins empty = {0}; bins some = {[1:$-1]}; bins full = {MAX_OUTSTANDING_PER_DEV};
  }
  cp_veto      : coverpoint which_veto_blocked;  // §24 — each term, separately
  cp_recovery  : coverpoint recovery_with_outstanding;   // §18
  cp_ready     : coverpoint readiness_term_low;  // §13 — each of six, separately
 
  // A link recovery while reads are outstanding, at every occupancy — the
  // configuration §18's bug needs, and the one regressions omit.
  x_recovery_occ  : cross cp_recovery, cp_out_occ;
  // Every way exercised at every configured granularity — §23's mismatch is
  // invisible unless both axes move.
  x_way_by_ig     : cross cp_way, cp_ig;
  // Boundary addresses in an interleaved window: §8's off-by-one and §22's
  // extraction interact exactly here.
  x_boundary_way  : cross cp_boundary, cp_way;
 
endgroup

Why x_boundary_way is the valuable cross. A boundary address in a non-interleaved window tests §8. A middle address in an interleaved window tests §22. The first address of an interleaved window tests both at once — and it is where an inclusive/exclusive error and a granularity error can cancel for that one address and disagree everywhere else, which is the hardest version of both bugs to find.

32. Debug Checklist

1. Which route did the request take? §11 — local, CXL, or unmapped. A surprising answer ends the investigation immediately.
2. Did exactly one window match? §9 — check the assertion, not the intent.
3. Which window, and does its base/limit match what software configured? §8 — including the −1.
4. Is the window still valid, or was it reprogrammed? A stale match is silent misdelivery.
5. Is the range interleaved? If so, compute the target by hand from the address bits.
6. Does the failing address set have a stride? §23 — the stride names the mismatch.
7. Which device was targeted, and was it the right one? Only an independent model answers this (§30).
8. Was cxl_mem_ready asserted for that device? §13 — and which of the six terms was low.
9. Was an outstanding entry pre-allocated before the request left? §16.
10. Did a UCIe recovery occur during the request's lifetime? §18 — and check the entry survived it.
11. Did a response arrive matching nothing? §19 — orphans are evidence, not noise.
12. Was the request retired exactly once? Compare issue and retire counts per device.
13. Which veto is limiting throughput? §27 — outstanding table or transport, and they live on different dies.
14. Do the design's counters agree with the model's outstanding set? §30.
15. Did the transport model or the semantic model diverge first? The single question that routes the whole investigation: a clean transport model with a diverged semantic model puts the bug in the map, the interleave, or the lifetime — never in the link.

33. Common Misconceptions

"CXL memory is just remote DRAM with a different cable." It is memory-mapped into system coherent address space and reached by loads and stores — which is exactly what distinguishes HDM from the PDM on a PCIe device. The mechanism that makes that possible is address ownership and routing, and it is where the bugs are.

"UCIe ACTIVE means CXL memory is ready." Six independent facts, six owners (§13), and the one with no Module 10 analogue is that the far side's media must be able to serve. This is the fourth appearance of this exact substitution in the curriculum (§14).

"A memory-capable device automatically owns an address range." Capability, capacity discovery, and window configuration are three separate steps with three owners. A device can be present, capable, and own nothing.

"Overlapping memory windows are harmless because software would not do that." They are the most severe failure in the chapter: a CPU store lands in the wrong device's memory, no error is reported anywhere, and the blast radius is whatever the operating system placed in the range (§10). Software does do this — usually through a path that bypassed the validation.

"Interleaving only affects performance." It converts one routing decision into a deterministic address function that every agent must compute identically, with no negotiation and no error detection (§21). A granularity mismatch corrupts a periodic subset of memory silently (§23).

"Outstanding memory requests can be cleared when the link retrains." That applies to credits and replay state, which are link-epoch state. An outstanding request is per-transaction state owned above the link, and clearing it leaves a core waiting forever — or worse, delivers its data to a later requester that reused the identity (§18).

"A transport retry can be visible as a second load or store." Then a write executes twice. The rule is Chapter 10.2 §9's and it does not weaken here: transport retirement and memory-transaction retirement are different events.

"Correct UCIe transport proves the memory map is correct." It proves the bytes crossed the package. §10 and §23 both produce perfect transport telemetry and wrong data.

"A valid route means the request can be accepted." Five vetoes (§24), each owned by a different mechanism, each failing differently and often on a different die.

"More CXL memory only changes capacity." It adds a second performance class inside one address space, which changes occupancy sizing, latency variance, congestion coupling, and who decides placement (§26).

"The window definition is inclusive, like most RTL ranges." CXL's is half-open — Base HPA <= HPA < Base HPA + Window Size. Store an inclusive limit if you like, but convert exactly once and remember the −1 (§8).

34. Understanding Check

35. Summary and What Comes Next

Memory expansion is an address-ownership and routing problem before it is a bandwidth problem.

The architecture: HDM versus PDM is the distinction that makes the whole thing possible — device-attached memory mapped into system coherent address space and reached with loads and stores, rather than memory a device happens to contain. CXL.mem carries host physical addresses and is media-independent, with the device translating internally. And HDM comes in two shapes — host-only coherent and device-managed coherent — whose coherency requirements differ, and whose answer changed between CXL revisions when enhanced coherency replaced bias-based coherency.

The mechanisms: one window match with exactly one target, asserted with $onehot0 because zero matches is legal and two never is. A half-open window definition converted to an inclusive limit exactly once, with the −1 that three separate bugs come from omitting. Three route outcomes, not two, because unmapped is real. Six readiness facts with six owners, of which the far side's media has no Module 10 analogue. Outstanding entries pre-allocated before transmission and retired exactly once against a reflected identity. And interleaving as a deterministic function of address bits above the 256-byte granularity floor, computed identically by every agent, with no negotiation and no error detection.

The lifetime rule that everything else hangs from: an outstanding memory request must survive a UCIe recovery. Transport state is re-baselined and that is correct; generalising the rule to per-transaction state leaves a core waiting forever, or delivers its data to whoever reused the identity.

The two silent failures to recognise on sight: overlapping windows corrupt a contiguous region, and an interleave mismatch corrupts a periodic one. The stride is what tells them apart, and neither produces a single error at any layer that reports errors.

Memory expansion lets the host address remote capacity, with the host managing the coherency flows for those accesses. The next problem is harder: what if the remote accelerator itself caches host memory and must stay coherent with CPU caches? Then correctness is no longer a property of either die, and a device holding a copy owes obligations to events it did not initiate:

• 11.3 — Cache Coherency Over CXL — CXL.cache flows mapped onto a UCIe link.

Browse the full path on the UCIe tutorials index.