CXL.mem Read Flows

Chapter 9.3 established what the access must guarantee. This chapter and the next walk the two flows that deliver those guarantees. The read is the simpler of the two — and it is still four stages, three possible sources and two ways to be wrong about identity.

1. The Engineering Problem — A Read Is a Question

A read asks: what is the value at this address? The device owes exactly one answer, and the answer must be

Each of those is a separate way to be wrong, and three of the four fail silently — an unanswered read looks like a hang, but a doubly-answered read, a wrongly-sourced read and a misdelivered read all return plausible values.

The read path is where a device's internal parallelism becomes visible, and every optimisation it makes — merging, reordering, forwarding — is an opportunity to break one of the four.

2. The One-Sentence Model

One question, one answer, one source, one identity. A read passes through ordered stages exactly once, draws its value from exactly one of three places, and returns carrying the identity of the request that asked — and every performance optimisation on the read path is a chance to violate one of those without producing an error.

Call it the single answer. Everything below is a way of keeping it true while going fast.

3. What This Chapter Owns

4. The Read, End to End

Architectural. The arrows are obligations, not named messages.

The two final arrows are the point. Merging the media access is the optimisation; merging the responses would lose a requester. §8 builds exactly that mistake, and it strands an agent waiting forever for an answer that was folded into someone else's.

5. Teaching-model boundary

6. RTL 1 — Four Stages, Each Exactly Once

      if (data_ready) begin
        if (stage_q[data_id] == 3'd3)      double_data_err <= 1'b1;
        else if (stage_q[data_id] != 3'd2) out_of_order_stage_err <= 1'b1;
        else stage_q[data_id] <= 3'd3;
      end
      if (retire) begin
        // Retiring before data is ready completes nothing.
        if ((stage_q[retire_id] != 3'd3) && !ALLOW_SKIP)
          retire_without_data_err <= 1'b1;

=== EXP1: a read passes four stages, each exactly once ===
  accepted                                            : ok
  looked up                                           : ok
  data ready                                          : ok
  retired, and the slot is free                        : ok
  no stage was skipped or repeated                    : ok

Teaching values, but the ordering is architectural: a read cannot be looked up before it is accepted, cannot produce data before it is looked up, and cannot retire before it has data.

Three distinct failures live here, and they are worth separating because they present completely differently:

=== EXP1b: retiring before the data is ready ===
  abuse instance flagged a retire with no data       : ok
  the allow-skip variant permits it silently          : ok

That check needed an instance reserved for illegal stimulus — a correct device never retires early, so driving it into the instance under test would trip that instance's own checker and be scored as a device fault.

7. RTL 2 — Exactly One of Three Sources

  // Priority is not arbitrary: an error beats everything (there is no value),
  // and a pending write beats media (media is stale by definition).
  assign src_error   = rd_valid && media_bad;
  assign src_forward = rd_valid && !media_bad && has_pending_write;
  assign src_media   = rd_valid && !media_bad &&
                       (BOTH_SOURCES ? 1'b1 : !has_pending_write);

=== EXP2: exactly one source per read ===
  clean media : media=1 fwd=0 err=0 data=0xDEAD0000 hot=1
  a clean read comes from media                       : ok
  pending write : media=0 fwd=1 data=0xFEED0001 | both-sources hot=2
  a pending write beats stale media                   : ok
  and the forwarded value was returned                : ok
  the both-sources variant selected two               : ok
  media bad : err=1 fwd=0 hot=1
  an unreadable location beats everything             : ok

The priority is derived, not chosen. An error wins because there is no value to return at all. A pending write wins over media because 9.3 established that media is stale by definition while an uncommitted write exists. Media is the default because it is the only source when neither special case applies.

A design that can select two sources has no defined answer, and hot — the count of asserted sources — is what makes that checkable rather than assumed.

8. RTL 3 — Merge the Access, Not the Answer

This is the chapter's most important distinction, and it is easy to get half right.

  // EVERY waiter on this line gets its own response. Merging the media access
  // is the optimisation; merging the responses would lose a requester.
  assign respond_mask = media_done ? (MERGE_RESPONSES ? (same_line & -same_line)
                                                      : same_line) : '0;

=== EXP3: two readers, one media access, two responses ===
  two readers of one line : media issues=1 merges=1
  one media access served both readers                : ok
  media done : correct mask=00100010 | merge-responses mask=00000010
  BOTH readers are responded to separately            : ok
  the broken variant answered only one                : ok
  two requests, two responses, one media access       : ok
  the broken variant stranded a waiting reader        : ok

Read the mask: 00100010 sets bits 1 and 5 — both waiters. The broken variant sets only the lowest, so agent 5 waits forever for an answer that was folded into agent 1's.

This is the device-side counterpart to 8.4's MSHR merging, and the constraint is different. An accelerator merging misses wakes several lanes from one fill and that is the whole point. A memory device is a responder: each request arrived separately with its own identity, and each is owed its own response. Merging the expensive part — the media access — is right; merging the cheap part is a lost requester.

A counter defect in my own RTL, for the eighth time

The response counter was originally written inside the wake loop:

        for (k = 0; k < NENT; k = k + 1)
          if (respond_mask[k]) begin
            wait_q[k]   <= 1'b0;
            n_respond_q <= n_respond_q + 16'd1;    // WRONG
          end

Four iterations, four non-blocking assignments to one variable, all computing n_respond_q + 1 from the same pre-edge value — so the counter advanced by one no matter how many readers were woken. The baseline caught it immediately (responses=1 expected 2). The fix accumulates combinationally and adds once:

    resp_this_cycle = 16'd0;
    for (k = 0; k < NENT; k = k + 1)
      if (media_done && respond_mask[k]) resp_this_cycle = resp_this_cycle + 16'd1;
...
        n_respond_q <= n_respond_q + resp_this_cycle;

Eighth appearance of this defect family in this track, and the second time I wrote it myself. The bit-vector-wake shape is a reliable place for it: whenever a loop clears several bits, any counter in that loop is wrong.

9. Waveform — One Media Access, Two Responses

Read media_issues and responses at the right edge: 1 and 2. That ratio is the entire value of merging, and it is also the invariant — responses must exceed or equal media accesses, never the reverse.

10. RTL 4 — Out of Order, With Identity Intact

  assign rsp_id    = REUSE_ID ? oldest_q : deq_sel;
  ...
        // The response must carry the id of the entry it drained.
        if (rsp_id != deq_sel) wrong_id_err <= 1'b1;

=== EXP4: responses may leave in any order, with their own id ===
  3 queued : level=3 peak=3
  drain id 2 first : rsp_id=2 data=0x33330002 | reuse-id rsp_id=0
  the response carried its own id                     : ok
  and its own data                                    : ok
  the reuse-id variant used the oldest id instead     : ok
  reordering was recorded, not prevented             : ok
  every response carried the id it drained           : ok

The device is free to reorder. 9.2 established that the host must tolerate it, so the device's only obligation is that the pairing survives — a response drained from entry 2 must say it is entry 2.

n_reorder_q records rather than prevents. That is deliberate: a device that never reorders is not more correct, it is less parallel, and the counter exists so the amount of reordering is visible to whoever is sizing the host's tolerance for it.

11. RTL 5 — A Partial Read Fails Only on Bytes It Asked For

Media works in lines; requesters work in bytes.

  logic [7:0] touched_bad;
  assign touched_bad = byte_en & bad_bytes;
  // The read fails only if a byte it actually asked for is unreadable.
  assign rd_error    = rd_valid && (WHOLE_LINE_ERROR ? (bad_bytes != 8'd0)
                                                     : (touched_bad != 8'd0));

=== EXP5: a partial read fails only on bytes it asked for ===
  request low half, bad high half : correct error=0 | whole-line error=1
  bad bytes it did not ask for did not fail it        : ok
  the whole-line variant failed it anyway            : ok
  request the bad half : error=1
  a read of unreadable bytes does fail               : ok
  the spared read was counted separately             : ok

Over-reporting an error is a real defect, not a conservative choice. A line with one unreadable byte can still serve every request that does not touch it, and failing those requests turns a small media defect into a large functional outage — the difference between losing eight bytes and losing a whole page.

n_spared_q counts reads that succeeded despite the line having a bad byte elsewhere. On ageing media that number is the difference between a device that degrades gracefully and one that falls over.

12. RTL 6 — Where a Read's Time Goes

=== EXP6: where a read's time goes ===
  20 reads : queue=60 lookup=40 media=192 return=60 max_total=20
  forwarded=4 merged=2
  forwarded reads counted exactly (4 of 20)          : ok
  and forwarded reads charged zero media time        : ok
  stage accounting held                              : ok
  media is 54% of total read time

Media is 54% of read time even in a model with generous queue and return costs — consistent with 9.1's finding that the device's own internals dominate, not the protocol.

The stage checker is the interesting part:

      // A forwarded read must not have spent media time.
      if (was_forwarded && (t_media != 8'd0)) stage_sum_err <= 1'b1;

A forwarded read never touched the media, so charging it media time is an accounting contradiction — and one that would make forwarding look useless in exactly the telemetry meant to justify it.

13. RTL 7 — Everything Accepted Is Answered Once

=== EXP7: everything accepted is answered exactly once ===
  accepted=32 media issues=21 answered=30 errors=2 peak=2
  answers matched an independent oracle              : ok
  error count matched the oracle                     : ok
  nothing was answered that was not accepted         : ok
  peak in-flight was exactly 2                       : ok
  media issues (21) below acceptances (32): merging works : ok

Media issues below acceptances is the signature of a working merge path — 21 media accesses served 32 reads. If those two numbers are equal, merging is not happening, and on a workload with line sharing that is a third of the media bandwidth wasted.

14. Quantitative Reasoning — What the Read Path Saves

Illustrative, from the measured runs.

Merging

  accepted=32  media issues=21

A third of reads were served without a media access. Using 9.1's 100 ns media latency and 4-deep media concurrency, eliminating 11 of 32 media accesses returns roughly 11 × 100 ns of media occupancy to other work — on a device whose media is the bottleneck, that is directly throughput.

Forwarding

  20 reads : media=192 cycles total
  forwarded=4, each charging zero media time

Four forwarded reads at 12 cycles of media time each is 48 cycles avoided in 20 reads, and the checker guarantees the saving is real rather than an accounting artefact.

The stage split

Media dominates, which is why merging and forwarding — the two optimisations that avoid media entirely — matter more than anything done to the other three stages.

15. The Read Path, as Structures

Every box carries its own way of being silently wrong, and the captions name them. That is the shape of this chapter: the read path is not complicated, but each of its five optimisations has a failure that returns a plausible value.

16. Assertions

Icarus Verilog 13.0 does not support concurrent SVA here, so every property is synthesisable checker logic verified in simulation.

Safety

Liveness

The second is the one merging creates: a merged reader has no media access of its own, so its liveness depends entirely on someone else's. If the leading access is lost, every merged waiter is lost with it — which is why missing_response_err is a safety property and not merely telemetry.

Performance goals

17. Mutation Testing

Twenty-one mutations. Twenty-one killed.

The first run scored 18 of 21, and all three escapes were the same shape: illegal stimulus that a correct device never produces — a lookup for an unaccepted read, data produced twice, and a forwarded read charging media time. Each got a dedicated abuse instance.

Notably, the most valuable defect this chapter found was not found by mutation testing. The baseline caught it: the response counter written inside the wake loop reported responses=1 where two readers had been answered. Mutation testing verifies the checkers; the baseline verifies the design, and here the design was mine and wrong.

18. Verification Plan

The coverage cross is line sharing × source × byte coverage: unique / shared line, crossed with media / forward / error, crossed with full-line / partial. The shared-line column is what exercises merging, and a testbench generating distinct addresses per request never enters it — the same blind spot 8.4 had.

19. Silicon Observability

Two alarms, eight tuning counters. A stranded reader and a misdelivered response both produce silent failures — one hangs a requester, the other corrupts it — and both must read zero for the life of the product.

The most useful ratio is media issues over acceptances. Near 1.0 means no merging is happening; well below means line sharing is being exploited. On a media-bound device that single ratio predicts throughput better than any latency number.

20. Debug Lab

// Wake the waiter for this line.
respond_mask = same_line & -same_line;      // lowest set bit only

  media done : correct mask=00100010 | merge-responses mask=00000010
  the broken variant stranded a waiting reader        : ok

assign respond_mask = media_done ? same_line : '0;
if ((same_line & ~respond_mask) != '0) missing_response_err <= 1'b1;

for (k = 0; k < NENT; k = k + 1)
  if (respond_mask[k]) begin
    wait_q[k]   <= 1'b0;
    n_respond_q <= n_respond_q + 1;      // inside the loop
  end

  two requests, two responses, one media access       : ok

resp_this_cycle = 16'd0;
for (k = 0; k < NENT; k = k + 1)
  if (media_done && respond_mask[k]) resp_this_cycle = resp_this_cycle + 16'd1;
...
n_respond_q <= n_respond_q + resp_this_cycle;

assign rsp_id = oldest_q;      // assume responses drain in order

  drain id 2 first : rsp_id=2 | reuse-id rsp_id=0
  reuse-id variant flagged a wrong response id       : ok

assign rsp_id = deq_sel;
if (rsp_id != deq_sel) wrong_id_err <= 1'b1;

assign rd_error = (bad_bytes != 0);      // any bad byte fails the read

  request low half, bad high half : correct error=0 | whole-line error=1
  bad bytes it did not ask for did not fail it        : ok

assign touched_bad = byte_en & bad_bytes;
assign rd_error    = rd_valid && (touched_bad != 0);
if (rd_error && (touched_bad == 0)) overreported_err <= 1'b1;

if (data_ready) stage_q[data_id] <= DATA_READY;   // no repeat check

  abuse instance flagged duplicate data             : ok

if (stage_q[data_id] == DATA_READY)      double_data_err <= 1'b1;
else if (stage_q[data_id] != LOOKED_UP)  out_of_order_stage_err <= 1'b1;
else stage_q[data_id] <= DATA_READY;

sum_media_q <= sum_media_q + t_media;   // charged even for forwarded reads

  and forwarded reads charged zero media time        : ok

if (was_forwarded && (t_media != 8'd0)) stage_sum_err <= 1'b1;

assign issue_media = rd_valid;      // every read issues its own access

  media issues (21) below acceptances (32): merging works : ok

for (k = 0; k < NENT; k = k + 1)
  if (wait_q[k] && (line_m[k] == rd_line)) line_busy = 1'b1;
assign issue_media = rd_valid && !line_busy;

21. Design Review

1. What stages does a read pass, and what enforces the order?
2. Can data be signalled twice for one read?
3. How many sources can a read draw from simultaneously?
4. When two reads share a line, how many media accesses and how many responses?
5. Does a response carry its own identity or a positional one?
6. Does a partial read fail on bytes it did not request?
7. Is a forwarded read charged media time?
8. What is the ratio of media issues to acceptances on a sharing workload?
9. Which two counters must be zero for the life of the product?
10. What happens to merged readers if the leading media access is lost?

22. How This Appears in Real Engineering

Architecture. The media-issues-to-acceptances ratio is the read path's headline number on a media-bound device, and merging is the cheapest way to move it.

RTL. The wake loop is a small structure with two classic defects — a merged response mask and a counter inside the loop — and this chapter hit both.

DV. Shared lines, out-of-order drains and partial reads on partially bad lines are all directed scenarios. None arises from a random address generator.

Post-silicon. Spared reads are the counter that distinguishes media ageing from media failure, and it only exists if someone decided partial reads should not over-report.

Firmware. The reorder count tells a driver how much response reordering to expect, which sizes the host-side tolerance 9.2 requires.

23. Common Misconceptions

24. Interview Reasoning

25. Exercises

1. Calculation. A workload issues 1000 reads with 40% of them sharing lines with an in-flight read. Compute the media accesses with and without merging, and the media time saved at 100 ns per access.

2. Analysis. A device reports accepted=10000, media_issues=10000, merges=0, and high media occupancy on a workload known to share data. Name the defect and the counter that confirms it.

3. RTL task. Extend read_merge so a read arriving after the media access has returned but before the responses are sent does not merge. State the new race and the assertion that catches it.

4. Assertion task. Write the property that catches a read retiring before its data is ready, and explain why it must be expressed over the stage rather than over the retire signal alone.

5. Debug task. One requester hangs on a read while all others succeed, correlated with data sharing rather than load. Give your investigation order and the single counter that identifies the cause.

6. Design review. A colleague proposes failing any read to a line with a bad byte, "because partial errors are complicated". Give the strongest version of that argument, then quantify what it costs using line and byte sizes.

26. Summary

One question, one answer, one source, one identity.

• A read passes ordered stages, each exactly once. Skipping, repeating and early retirement are three different failures with three different symptoms.
• Its value comes from exactly one of three sources — error, forward, or media — and the priority is derived rather than chosen.
• Merge the media access, not the responses. Two readers of one line means one access and two answers; merging the answers strands a requester forever.
• Reordering is normal; losing identity is the defect. A response must carry the id of the entry it drained.
• A partial read fails only on bytes it requested. Over-reporting turns a byte-granular defect into a line-granular outage.
• Latency accounting is logic. A forwarded read charging media time is a contradiction, and it appears in the counter meant to justify forwarding.
• Measured: 21 media accesses served 32 reads — a third saved by merging; media was 54% of read time.
• Verification lesson: mutation testing found three escapes, all illegal stimulus needing abuse instances — but the most valuable defect, a non-blocking counter inside a wake loop, was found by the baseline, not by mutation. The two techniques answer different questions.

Chapter 9.5 takes the write path, which is harder for the reason 9.3 gave: a read is done when it has a value, and a write is not done until others can see it.