CXL.mem Overview

Module 8 was about a device that borrows host memory. This module reverses the direction entirely: the device is the memory, and the host is the one reaching across the link.

1. The Engineering Problem — "Remote RAM" Is the Wrong Model

The tempting one-line summary of CXL.mem is that a CPU reads and writes memory attached to a device. That sentence is true and it is almost useless, because it hides every property that makes the thing work.

If device memory were simply remote RAM, then:

The last row is the sharpest. A DRAM controller that cannot produce a value has a defined way to say so. A device that answers a broken location with plausible bytes has corrupted a program that had no way to know.

So the real question of this module is not "can the CPU reach it". It is: what must a device build to be trusted with a range of the host's own physical address space?

2. The One-Sentence Model

Device memory becomes host memory by contract, not by wiring. The device agrees to answer for a window of host physical addresses — every address inside it, no address outside it — and in exchange it must behave like memory: never lose a request, never exceed the concurrency it promised, never answer a failed read with data, and never make the host wait without saying why.

Call it the window contract. Everything in Module 9 is a consequence of it.

3. What This Chapter Owns

Deliberately not repeated: 3.1 already builds the host's address decoder, outstanding-transaction table and home agent. This module is the other end of that link — what the responder must build — and it assumes 3.1's vocabulary rather than re-deriving it. That mirrors how Module 8 sat opposite 3.4.

4. Where CXL.mem Sits

Read the top band carefully: software does nothing special. There is no driver call and no copy. That is the entire value proposition, and it is also why the lower bands carry so much weight — every guarantee ordinary memory provides must now be provided across a link by a device.

The band that surprises people is the bottom one. Protocol work is bounded and fast; the memory controller and the media are where the time goes, and §13 measures it.

5. Teaching-model boundary

6. RTL 1 — The Window Contract, in Logic

The device does not own an address space. It is handed a range of the host's physical addresses and must answer for exactly that range.

  logic [31:0] top;
  assign top       = base + size;
  assign below     = req_valid && (req_addr <  base);
  // The top address belongs to the NEXT target. Including it is the classic
  // off-by-one that makes two devices claim one line.
  assign above     = req_valid && (INCLUSIVE_TOP ? (req_addr >  top)
                                                 : (req_addr >= top));
  assign in_window = req_valid && !below && !above;
  assign dev_offset = in_window ? (req_addr - base) : 32'd0;

=== EXP1: the device answers for a window, not for everything ===
  just below base : in_window=0 below=1
  an address below the window is not ours              : ok
  exactly base    : in_window=1 offset=0x00000000
  base maps to device offset zero                      : ok
  last address    : in_window=1 offset=0x0FFFFFFF
  the last address in the window is ours               : ok
  base+size       : correct in_window=0 | inclusive-top in_window=1
  base+size belongs to the NEXT target, not us         : ok
  the off-by-one variant claimed another target's line : ok

Three things worth separating.

Both boundary errors are real, and they are not symmetric. Refusing an address inside the window makes part of the host's RAM vanish — bad, and loud. Claiming an address outside it means two targets answer for one line, which is silent corruption of a neighbour's memory. The second is far worse and far harder to find.

base + size belongs to the next target. The window is half-open by construction. INCLUSIVE_TOP is the classic off-by-one, and it only ever misbehaves on exactly one address out of hundreds of millions — which is why it survives random testing and fails in the field.

The device works in offsets, the host works in addresses. dev_offset = req_addr - base is the translation, and it is the only place the two coordinate systems meet.

7. RTL 2 — Identity Belongs to the Requester

A memory device is the responder. It does not choose when work arrives and it does not choose the transaction identity — the requester does. The device's job is to validate identity, not allocate it.

  // A tag maps directly to an entry -- the requester owns identity, so the
  // device does not allocate, it VALIDATES.
  assign tag_busy  = live_q[req_tag[2:0]];
  assign full      = (occupancy_q == NENT[3:0]);
  assign req_ready = !full && !tag_busy;

=== EXP2: identity is owned by the requester, validated by the device ===
  tag 3 accepted : ready=1 occupancy=1
  two distinct tags are two live transactions          : ok
  tag 3 offered again while live : ready=0
  a live tag is refused, not silently overwritten      : ok
  and the requester's early reuse was counted          : ok
  the device never accepted a duplicate tag            : ok

A checker I had pointed at the wrong party

The first version raised an error flag when a requester offered a tag that was still live. Simulation immediately failed the correct device — because refusing that request is exactly right, and the error belonged to the requester rather than the device.

The fix separates the two, and it is the same distinction 8.2 needed:

        // Offered a tag that is already live. Refusing is CORRECT behaviour;
        // this counts how often the requester did it.
        if (tag_busy) n_early_reuse_q <= n_early_reuse_q + 16'd1;
...
        // The device fault would be accepting a tag that is already live.
        if (live_q[req_tag[2:0]]) accepted_dup_err <= 1'b1;

n_early_reuse_q is telemetry about the partner — genuinely useful, because a rising count means the host is running out of tags and the device is the visible symptom of a problem it did not cause. accepted_dup_err is the device's own safety property. Assert on what you control; count what you observe.

A slot frees at retire, not at the response

=== EXP3: a slot frees at RETIRE, not at the response ===
  after response only : correct occupancy=2 | reuse-variant occupancy=1
  the slot survives the response                       : ok
  the broken variant released it early                 : ok
  retire released it                                   : ok

This is the same release-at-completion rule Modules 7 and 8 kept arriving at, and here it protects the requester's tag space: releasing early tells the requester a tag is free while the device is still finishing with it.

8. RTL 3 — Bounded Media Concurrency

This is the block that makes a memory device not a wire.

  assign issue_ok = issue && (UNBOUNDED || (inflight_q < MAXO[3:0]));
  ...
      // Exhaustive: issue and completion in one cycle must hold the count.
      case ({issue_ok, done})
        2'b10: begin
          inflight_q <= inflight_q + 4'd1;
          if (inflight_q + 4'd1 > peak_q) peak_q <= inflight_q + 4'd1;
        end
        2'b01: inflight_q <= (inflight_q == 4'd0) ? 4'd0 : inflight_q - 4'd1;
        default: ;
      endcase

=== EXP4: media concurrency is bounded, and blocking is counted ===
  8 issues into MAXO=4 : inflight=4 peak=4 issued=4 blocked=4
  four accepted, four blocked, none lost              : ok
  peak concurrency was exactly the bound, 4            : ok
  unbounded variant : inflight=8 issued=8
  the unbounded variant accepted all eight            : ok
  and then exceeded its own stated capacity           : ok
=== EXP5: media latency is what the outstanding budget covers ===
  after the pipeline drains : done=4 inflight=0
  every accepted request completed exactly once        : ok
  and the media pipeline drained to empty              : ok

Note the case on the concatenation. This is the fifth appearance in this track of the counter defect where independent increment and decrement paths are written as two non-blocking assignments — the second wins, so issue-and-complete together decrements. It survives fill-then-drain testing and fails under exactly the steady-state traffic a memory device lives in.

A diagnostic that disabled itself

The overflow checker was originally written if (!UNBOUNDED && inflight > MAXO). That guard disables the check on the only configuration that can trip it — the unbounded variant. It is a checker that cannot fire by construction, and mutation testing found it:

      // Observe the STATED capacity regardless of policy -- gating this on
      // !UNBOUNDED would disable the check on the only design that can trip it.
      if (inflight_q > MAXO[3:0]) overflow_err <= 1'b1;

9. RTL 4 — Refuse, Never Drop

  assign room      = table_has_room && media_has_room;
  assign req_ready = DROP_WHEN_FULL ? 1'b1 : room;
  assign accepted  = req_valid && req_ready && room;

=== EXP6: refuse, never drop ===
  room available : ready=1 accepted=1
  table full     : correct ready=0 accepted=0 | drop-variant ready=1
  with no room the device deasserts ready              : ok
  the broken variant said yes with no room             : ok
  every offer was accepted or stalled, none lost       : ok
  the drop variant flagged a lost request              : ok

ready is the entire safety contract of a responder. Deassert it and the requester keeps the request; assert it and the device has taken ownership. A device that says yes and forgets produces a host that waits forever for a load that no longer exists anywhere.

Note that room requires both the transaction table and the media to have space. Two independent resources, either of which can be the constraint — and §13 shows why counting them separately is what makes the device diagnosable.

10. The Device, as Structures

The block with no analogue in an ordinary DRAM controller is the window check. A local controller owns its address space by construction; a CXL device is lent part of someone else's and has to prove it stays inside the loan.

11. RTL 5 — A Failed Read Is Not Data

Architectural. This models the obligation, not any CXL-defined encoding.

  // The whole point: a failed read is an ERROR response, never data.
  assign rsp_is_error = done && media_bad && !SILENT_ERROR;

=== EXP7: an unreadable location returns an ERROR, not data ===
  media bad : correct is_error=1 | silent variant is_error=0
  good=1 error=1
  the failed read was reported as an error             : ok
  the silent variant answered with data anyway         : ok

The silent variant is the worst failure mode in the module, and it is worth being precise about why. It does not hang, it does not slow down, it does not set a flag. It returns bytes. Software that reads a failed location gets a value it will happily compute on, and nothing anywhere in the system distinguishes that value from a real one.

This is the memory-device analogue of 8.2's stale-read: a correctness alarm whose only symptom is the absence of an alarm.

12. RTL 6 — Four Bottlenecks, Four Counters

=== EXP8: four stall causes, four different fixes ===
  offered=40 accepted=32 reads=20 writes=10
  stalls: link=8 table=6 media=4
  acceptance matched an independent oracle             : ok
  read/write split matched the oracle                  : ok
  completions never exceeded acceptances               : ok
  link stalls counted exactly (8 of 40)               : ok
  table stalls counted exactly (6 of 40)              : ok
  media stalls counted exactly (4 of 40)              : ok
  mean latency 14 cycles, worst 18

A memory path can be limited in four distinct places, and they need different fixes:

One "busy" counter cannot distinguish these, and a team reading only an aggregate utilisation figure will size the wrong resource. The oracle here is a testbench-side model with no connection to the design's expressions, which is what makes the acceptance and read/write numbers worth anything.

13. Quantitative Reasoning — How Much Outstanding Is Enough

Illustrative. The single most useful calculation for a memory device is the bandwidth-delay product.

bytes in flight = offered bandwidth × round-trip latency
outstanding     = bytes in flight ÷ transaction size

Take a device offered 32 GB/s with a 250 ns round trip and 64-byte accesses:

32e9 × 250e-9 = 8000 bytes in flight
8000 ÷ 64     = 125 outstanding requests

125. Not 8, not 32. Now read what happens if the design guessed lower:

Two conclusions an architect should carry away.

Under-provisioning outstanding capacity caps bandwidth linearly, and no amount of link speed fixes it — a 32-deep device on a 32 GB/s link delivers a quarter of the link.

Over-provisioning buys nothing. Past 125 the link is the constraint, and the extra entries are area and CAM depth spent for no throughput.

Where the 250 ns goes

Illustrative decomposition, and the shape matters more than the numbers:

The protocol is not the problem. Link and protocol together are 16% of the round trip; the device's own controller and media are 64%. That is the number that reframes most CXL memory discussions — the interesting engineering is inside the device, which is why this module spends its time there and defers link-level performance modelling to Module 18.

14. The Sizing Picture

Both inputs at the top are outside the device's control — the link's speed and the round trip's length. The only thing the designer chooses is the bottom row, and choosing it without doing the multiplication is how a device ends up delivering a quarter of its link.

15. Assertions

Icarus Verilog 13.0 is the simulator available here and does not support concurrent SystemVerilog assertions, so every property below is synthesisable checker logic verified in simulation. The assert property form states the intent.

Safety

Liveness

Each is stated with its assumption, because none is true unconditionally — a device whose media never completes satisfies every safety property above perfectly.

Performance goals

16. Mutation Testing

Eighteen mutations. Eighteen killed.

The first run scored 16 of 18, and both survivors were shapes this track has now seen repeatedly.

"Link stalls not counted" survived a bound. The test displayed the three stall counts and asserted none of them. The stimulus is deterministic — link stalls on i%5==4 over 40 offers is exactly 8 — so the fix is an equality, not an inspection:

  link stalls counted exactly (8 of 40)               : ok
  table stalls counted exactly (6 of 40)              : ok
  media stalls counted exactly (4 of 40)              : ok

"Unknown response not flagged" needed illegal stimulus. A response naming a transaction that was never opened cannot be driven into the instance under test — that instance's own checker would fire and be scored a device failure. It got a dedicated abuse instance, asserting both that the stray response is flagged and that it retires nothing:

  abuse instance flagged a response with no request    : ok
  and it retired nothing: an unknown tag is void       : ok

17. Verification Plan

The coverage cross is address region × operation × resource state: below / at-base / interior / at-top / above, crossed with read and write, crossed with room-available / table-full / media-full. The at-top column is the one random address streams essentially never hit, and it is where the off-by-one lives.

18. Silicon Observability

Two of these are alarms rather than telemetry: outside_answered and bad_data_returned. Both must read zero for the life of the product, and both describe failures with no other symptom. Everything else is for tuning.

The most diagnostic pair is transaction peak against media peak. If the transaction table is saturated while media concurrency is not, the device is protocol-limited and needs more entries; if media is saturated first, more entries buy nothing.

19. Debug Lab

assign above = req_valid && (req_addr > base + size);   // inclusive top

  base+size : correct in_window=0 | inclusive-top in_window=1

assign above = req_valid && (req_addr >= base + size);
if (in_window && (req_addr >= base + size)) outside_answered_err <= 1'b1;

assign req_ready = 1'b1;                    // always accept
if (req_valid && room) store_request();     // ...but only store when there is room

  table full : correct ready=0 accepted=0 | drop-variant ready=1
  the drop variant flagged a lost request              : ok

assign req_ready = table_has_room && media_has_room;
assign accepted  = req_valid && req_ready;
if (req_ready && !room) dropped_err <= 1'b1;

// Media reports the location is unreadable.
rsp_data  <= media_data;      // return whatever came back
rsp_error <= 1'b0;

  media bad : correct is_error=1 | silent variant is_error=0
  the silent variant answered with data anyway         : ok

assign rsp_is_error = done && media_bad;
if (media_bad && !rsp_is_error) bad_data_returned_err <= 1'b1;

mem_txn_table #(.NENT(32)) u_tbl (...);   // sized by intuition

  offered=40 accepted=32 stalls: link=8 table=6 media=4

outstanding = bandwidth × round-trip latency ÷ transaction size
            = 32e9 × 250e-9 ÷ 64
            = 125

if (issue_ok) inflight_q <= inflight_q + 1;
if (done)     inflight_q <= inflight_q - 1;

  8 issues into MAXO=4 : inflight=4 peak=4 issued=4 blocked=4

case ({issue_ok, done})
  2'b10: inflight_q <= inflight_q + 4'd1;
  2'b01: inflight_q <= inflight_q - 4'd1;
  default: ;                                 // both or neither: hold
endcase

if (!UNBOUNDED && (inflight_q > MAXO)) overflow_err <= 1'b1;

  the unbounded variant accepted all eight            : ok
  and then exceeded its own stated capacity           : ok

// Observe the STATED capacity regardless of policy.
if (inflight_q > MAXO[3:0]) overflow_err <= 1'b1;

if (tag_busy) reuse_too_early_err <= 1'b1;   // scored as a device fault

  a live tag is refused, not silently overwritten      : ok
  and the requester's early reuse was counted          : ok

if (tag_busy)             n_early_reuse_q <= n_early_reuse_q + 16'd1;  // observe
if (live_q[req_tag])      accepted_dup_err <= 1'b1;                    // assert

20. Design Review

1. Is the window half-open, and what asserts it? base+size belongs to the next target.
2. Who owns transaction identity, and what happens if a live tag is re-offered?
3. When does a slot free — at the response or at retire?
4. What bounds media concurrency, and is the bound asserted exactly?
5. Can the device ever assert ready without room?
6. What does a read of unreadable media return?
7. How many outstanding requests does bandwidth × latency require, and how many does the design have?
8. Which counters distinguish link-limited from table-limited from media-limited?
9. Which two counters must be zero for the life of the product?
10. What happens on reset with requests outstanding?

21. How This Appears in Real Engineering

Architecture. The outstanding-capacity calculation is the single most consequential number in a memory-device design, and it is frequently guessed. Getting it wrong caps bandwidth linearly and cannot be recovered by any other change.

RTL. The window comparison and the inflight counter are each a handful of gates and each has a classic defect — an off-by-one and a two-assignment counter — that survives ordinary testing.

DV. The four boundary addresses and the issue-and-complete-together cycle are directed tests. Neither is reachable by random stimulus with any useful probability.

Post-silicon. Stall attribution is what turns "the memory is slow" into a specific undersized resource. Without it the discussion is opinion.

System software. The early-reuse counter is often the first place a host-side tag shortage becomes visible, which makes a memory device a surprisingly good instrument for diagnosing its partner.

22. Common Misconceptions

23. Interview Reasoning

24. Exercises

1. Calculation. A device is offered 64 GB/s with a 400 ns round trip and 64-byte accesses. Compute the outstanding requirement. Then compute the bandwidth actually achievable if the design provides 128 entries, and state which resource to change.

2. Analysis. A device reports window_hits=10^9, outside_answered=0, table_stalls=0, media_stalls=10^8, and bandwidth well below the link. Name the constraint and the one change that would help.

3. RTL task. Extend hpa_window to support two disjoint windows. State the new invariant, and explain what makes overlapping windows harder to detect than an out-of-range access.

4. Assertion task. Write the property that catches a device releasing a transaction slot before retire, and explain why it must be expressed across two cycles rather than one.

5. Debug task. A CPU core hangs on a load to one specific address while all neighbouring addresses work. Give your investigation order and the single counter that distinguishes a decode bug from a dropped request.

6. Design review. A colleague proposes removing the media concurrency bound "because the media never actually stalls in our workload". Give the strongest version of that argument, then name what it costs and the first workload that breaks it.

25. Summary

Device memory becomes host memory by contract, not by wiring.

• CXL.mem is sourced as a "memory access protocol, host manages (coherency) device attached memory similar to host memory". Software issues an ordinary load or store.
• The device answers for a half-open window of host physical addresses. base+size belongs to the next target, and claiming it is silent corruption of a neighbour.
• The requester owns transaction identity; the device validates it. Refusing a live tag is correct and belongs in a counter; accepting one is the device's own violation.
• A slot frees at retire, not at the response — here it protects the requester's tag space.
• Media concurrency must be bounded, and the bound asserted exactly. The two-assignment counter defect appeared for the fifth time in this track.
• Refuse, never drop. The waiting party is a CPU core executing a load, with no software-visible retry.
• A failed read is an error, never data — the module's most dangerous failure, because its only symptom is the absence of one.
• Measured: 125 outstanding requests are needed for 32 GB/s at 250 ns. A 32-deep design delivers 26% of the link, and no link upgrade fixes it.
• Link and protocol were 16% of the round trip; the device's controller and media were 64%.
• Verification lesson: a checker written if (!UNBOUNDED && …) disabled itself on the only configuration that could trip it, and displaying three stall counters without asserting them let a counter mutation through.

Chapter 9.2 takes the other end: what the host must do to turn a load instruction into a request this device will answer.