Host Cache Access

Chapter 8.1 established that a cached line is borrowed and that the host coordinates. This chapter takes the first consequence, and it is sharper than it sounds: when the device wants a line, the value in memory may simply be wrong.

1. The Engineering Problem — The Obvious Read Is the Wrong Read

A device needs a line of host memory. The obvious implementation is a read of that address.

That implementation is broken, and not in a corner case. If any host cache holds a modified copy of the line, then memory has not been updated and the bytes at that address are a previous version. The read succeeds. The data is well-formed. It is stale, and nothing anywhere reports an error.

This is the worst failure shape in the module:

A device that reads memory directly is not slightly wrong; it is silently wrong exactly when another agent is doing something interesting.

2. The One-Sentence Model

Ask the coherence authority, not memory blindly. The location of a line's bytes and the location of its current value are different questions — so a device access must resolve who holds the authoritative copy before it decides where to read, and must hold that decision until the data actually lands.

Call it source authority. The address tells you where the line lives; only the directory tells you where its value is.

3. What This Chapter Owns

Deliberately not repeated: 3.4 already builds the host-side directory, conflict serialisation and snoop-wait as a generic coherence point. This chapter is the same problem seen from the device's side of the link — what the device must track, decide and bound while it waits — and it uses 3.4's vocabulary rather than re-deriving it.

4. The Exchange

Architectural. The arrows are decisions, not messages — this repository's source policy does not permit naming CXL.cache messages, and inventing them would be worse than omitting them.

Three things are worth reading off it.

Memory is never consulted in the first exchange. Not "consulted and discarded" — not read at all. Reading it would produce a stale value that looks exactly like a good one.

The device does not choose the source. It asks for a line; the coordinator resolves authority. The device's job is to track the request and accept whatever arrives, which is why its side of this chapter is mostly tracking rather than deciding.

The two exchanges cost different amounts, and §11 measures the difference.

5. Teaching-model boundary

6. RTL 1 — What the Directory Knows

  // Memory is authoritative exactly when no cache holds a modified copy.
  // This is the whole chapter in one expression.
  assign memory_authoritative = !dirty_m[rd_idx];

=== EXP1: memory is not automatically authoritative ===
  uncached line   : host_cached=0 dirty_owner=0 memory_authoritative=1
  nothing cached: memory holds the value             : ok
  cached, clean   : host_cached=1 dirty_owner=0 memory_authoritative=1
  a clean copy does not make memory stale           : ok
  cached, DIRTY   : host_cached=1 dirty_owner=1 owner=2 memory_authoritative=0
  a modified copy makes memory stale                : ok

Cached is not the same as stale. A clean cached copy is identical to memory, so memory remains a perfectly good source. Only a modified copy displaces it. Designs that treat "someone has it cached" as "memory is untrustworthy" are correct but needlessly slow — they consult an owner on every shared line.

The directory also refuses to represent a contradiction:

      // A modified owner that is not cached anywhere is a contradiction.
      if (upd_dirty_owner && !upd_host_cached) owner_without_cache_err <= 1'b1;

7. RTL 2 — One Source, Chosen by Authority

  // Exactly one source, chosen by who holds the newest value.
  assign src_owner  = req && !TRUST_MEMORY && dirty_owner;
  assign src_memory = req && (TRUST_MEMORY ? 1'b1 : (BOTH_SOURCES ? 1'b1 : !dirty_owner));
  ...
      // THE invariant of this chapter.
      if (src_memory && dirty_owner) stale_read_err <= 1'b1;

=== EXP2: choosing the source ===
  dirty line  : correct owner=1 memory=0 | trust-memory memory=1
  the owner's copy was chosen                       : ok
  the broken variant read stale memory              : ok
  clean line  : owner=0 memory=1
  with no dirty owner, memory is used               : ok
  sources used: from memory=1 from owner=1
  exactly one source per access                     : ok

The TRUST_MEMORY variant is the bug in Debug Lab 1, and it is worth noticing that it produces no error of its own — the stale-read flag is a checker, not a consequence. In silicon that flag does not exist unless someone built it, which is exactly why the failure is silent.

8. RTL 3 — One Line, One Access, and Two Reasons to Refuse

      end else if (alloc) begin
        // Two distinct reasons to refuse, and they need different fixes.
        if (!found)     n_full_q     <= n_full_q     + 8'd1;
        else            n_conflict_q <= n_conflict_q + 8'd1;
      end

=== EXP3: one line, one access in flight ===
  two lines   : occupancy=2 allocated=2
  different lines proceed together                  : ok
  same line   : correct allocated=2 conflicts=1 | allow-same-line allocated=3
  the same-line request was refused                 : ok
  the permissive variant allowed both               : ok
  and it then held two live accesses on one line    : ok
=== EXP4: the table fills; refusals have two reasons ===
  4 entries   : occupancy=4 peak=4 full-refusals=1 conflict-refusals=1
  full and conflict refusals counted separately     : ok

"Refused" is not a diagnosis. A device refusing because its table is full needs a bigger table; a device refusing because the line is already busy needs nothing at all — that is correct serialisation, and a high count simply means the workload is hammering one address. One counter cannot separate them, and a team that sees only a total refusal rate will size the wrong resource.

9. RTL 4 — The Value Must Reach the Requester That Asked

        // Identity: did the right requester get the right bytes?
        if (ret_data != exp_m[eff_id]) mismatch_err <= 1'b1;
        // Authority: did the value come from where the directory said?
        if (ret_from_owner != exp_from_owner) wrong_source_err <= 1'b1;

=== EXP5: the value must reach the requester that asked ===
  return tag 2 : correct delivers id=2 | by-order delivers id=0
  the tag chose the requester, not arrival order    : ok
  delivered=3 mismatches on correct=0 | by-order mismatched=1
  every value reached the requester that asked      : ok
  matching by arrival order delivered wrongly       : ok

Two independent properties, and a design can satisfy one while violating the other. Identity asks whether the right requester got the right bytes. Authority asks whether those bytes came from the source the directory chose — a value that is correct by luck, fetched from memory that happened not to be stale yet, still indicates the decision path is broken and will produce corruption on the next line.

Checking only identity misses an entire class of bug, because most of the time memory and the owner agree.

10. RTL 5 — Waiting for an Owner, With a Bound That Names Names

=== EXP6: whose owner response is late ===
  id 3 hung   : correct timeouts=1 reported id=3 max_wait=11
  the per-access timer named the access that hung   : ok
  one shared timer blamed a healthy access          : ok

The same argument 7.5 made about completions applies unchanged: a shared timer can say something is late but not which thing, and recovery needs identity. Here the consequence is sharper, because the access that hung is holding a line locked — so blaming the wrong one leaves the real blockage in place.

max_wait_q is the counter worth exposing. It is the observed worst case, and it is what tells you whether the ten-cycle limit is generous or marginal for a real workload.

11. Waveform — A Read That Needs an Owner

Read src_memory across the whole trace: it is never asserted. That flat line is the chapter. The device asked for a line, the directory knew a cache held a newer copy, and memory was not merely deprioritised — it was never a candidate.

The measured tail confirms it:

stale_read_err=0 mismatch=0 wrong_source=0 from_owner=1 from_mem=0

12. RTL 6 — The Decision Must Hold Until the Data Lands

This is the subtlest module in the chapter, and it is why per-line serialisation is a correctness requirement rather than a scheduling nicety.

A source decision is taken at one instant. The data arrives later. If the directory can change in between, the access can be authorised to read memory and then deliver a value that became stale while it was in flight.

  // While a decision is outstanding the line is locked: a conflicting change
  // is deferred, not applied. NO_LOCK removes exactly that protection.
  assign change_blocked = dirty_change && line_locked_q && !NO_LOCK;

=== EXP7: the decision must hold until the data lands ===
  decided from memory, line locked=1
  line dirtied mid-flight : correct blocked=1 | no-lock blocked=0
  the conflicting change was deferred, not applied  : ok
  the locked design delivered the value it chose    : ok
  without the lock a stale value was delivered      : ok

The lock is what makes the decision meaningful. Without it, "memory is authoritative" is a statement about the past — true when it was evaluated, and possibly false by the time it is acted on. This is the same shape as the read-passing-a-pending-write hazard in 7.5: a fact that was true when sampled, acted on after it stopped being true.

13. The Device-Side Structures

The line lock is the block that does not appear in a non-coherent read path at all, and it is the one §12 showed is load-bearing. Everything else has a plain-memory analogue: a table of outstanding reads, a return matcher, a timeout. The lock exists because the answer to a question has a shelf life here, which is not true when the only source is memory.

14. Quantitative Reasoning — Where a Device Read's Time Goes

Illustrative, using this chapter's teaching latencies. Decompose a device read:

T_read ≈ table allocation
       + authority resolution
       + (owner interaction, if any)
       + data path

The measured split, with 15 memory-sourced reads at 6 cycles and 5 owner-sourced reads at 18:

=== EXP8: where the time goes ===
  from memory: n=15 sum=90 max=6
  from owner : n=5 sum=90 max=18
  the source split matched an independent oracle    : ok
  15 memory reads and 5 owner reads cost the same    : ok
  but the worst owner read was 3x the memory read   : ok

Five owner reads cost as much total time as fifteen memory reads. That is the number worth carrying: at a 25% owner-source rate, half the access time is spent on a quarter of the accesses.

The design consequence is that the owner-source rate is a first-class performance metric, not a curiosity — and it is a property of the workload's sharing pattern, not of the device. A device cannot make it smaller by being faster; it can only be sized for it. That is why n_from_owner_q against n_from_mem_q belongs in silicon.

15. Assertions

Icarus Verilog 13.0 does not support concurrent SVA here, so each property is synthesisable checker logic verified in simulation.

Safety

Liveness

Performance goals

16. Mutation Testing

Seventeen mutations. Seventeen killed.

The first run scored 15 of 17, and both survivors were the same two shapes 8.1 hit.

The one-source check was unreachable by construction — src_owner requires dirty_owner and src_memory requires !dirty_owner, so they could never both assert. A BOTH_SOURCES variant makes the checker earn its keep.

The wrong-source check needed illegal stimulus. A value arriving from a source the directory did not choose cannot be driven into the instance under test, so it got a dedicated DUT:

  both-sources variant flagged multi_src           : ok
  abuse instance flagged the wrong data source      : ok

Three chapters in a row have now needed an instance reserved for illegal stimulus. It should be the default structure for any checker whose condition a correct design cannot reach, not a repair applied after a mutation escapes.

17. Verification Plan

The coverage model is a cross of directory state against access outcome: uncached / cached-clean / cached-dirty, crossed with returns-promptly / returns-late / never-returns / changes-mid-flight. The last column is the one that produced RTL 6, and no random stimulus generates it — the change has to be injected deliberately during a live decision.

18. Silicon Observability

The last row deserves emphasis: it is the one counter here that should be identically zero for the life of the product. A non-zero value is not a performance signal, it is silent data corruption already in flight — and without the counter, nothing else in the system will tell you.

19. Debug Lab

// Device needs a line — read it.
assign src_memory = req;

  dirty line  : correct owner=1 memory=0 | trust-memory memory=1

assign src_owner  = req && dirty_owner;
assign src_memory = req && !dirty_owner;
if (src_memory && dirty_owner) stale_read_err <= 1'b1;   // and prove it never fires

assign src_owner  = req && dirty_owner;
assign src_memory = req;                 // not mutually exclusive

  both-sources variant flagged multi_src           : ok

assign src_owner  = req && dirty_owner;
assign src_memory = req && !dirty_owner;   // a partition, by construction
assign hot = src_memory + src_owner;
if (hot > 1) multi_src_err <= 1'b1;

assign alloc_ok = alloc && found;    // no per-line check

  same line   : correct allocated=2 conflicts=1 | allow-same-line allocated=3
  and it then held two live accesses on one line    : ok

for (k = 0; k < NENT; k = k + 1)
  if (live_q[k] && (line_m[k] == alloc_line)) line_busy = 1'b1;
assign alloc_ok = alloc && found && !line_busy;

// Deliver to the oldest outstanding access.
deliver_id <= oldest_q;
oldest_q   <= oldest_q + 1;

  return tag 2 : correct delivers id=2 | by-order delivers id=0

assign eff_id  = ret_id;                       // the id decides
assign deliver = ret_valid && live_q[eff_id];
if (ret_data != exp_m[eff_id]) mismatch_err <= 1'b1;

if (|wait_q) gtimer_q <= gtimer_q + 1;
if (gtimer_q >= LIMIT) begin timeout_valid <= 1; timeout_id <= any_waiting; end

  id 3 hung   : correct timeouts=1 reported id=3 max_wait=11
  one shared timer blamed a healthy access          : ok

for (k = 0; k < NENT; k = k + 1) if (wait_q[k]) age_m[k] <= age_m[k] + 1;
if (wait_q[k] && (age_m[k] >= LIMIT)) begin hit = 1; pick = k; end
if (hit && age_m[pick] < LIMIT) wrong_id_err <= 1;

// Resolve the source, then fetch.
decided_memory <= !dirty_now;      // nothing locks the line afterwards

  line dirtied mid-flight : correct blocked=1 | no-lock blocked=0
  without the lock a stale value was delivered      : ok

assign change_blocked = dirty_change && line_locked_q;   // defer, do not apply
// line_locked_q is set at the decision and cleared when the data lands

if (alloc && !alloc_ok) n_refused_q <= n_refused_q + 1;   // one counter

  4 entries   : occupancy=4 peak=4 full-refusals=1 conflict-refusals=1

if (!found) n_full_q     <= n_full_q     + 1;   // undersized
else        n_conflict_q <= n_conflict_q + 1;   // hot line — expected

20. Design Review

1. What decides where a read is sourced from? If it is the address, the design is broken.
2. Does a clean cached copy force an owner interaction? It should not.
3. Is the source selection a partition? Assert at most one and at least one.
4. Is the line locked for the lifetime of the decision? Consulting the directory is not sufficient.
5. Do returns match on identity or on order? Order fails as soon as sources differ.
6. Is the source of a returned value checked against the decision? Identity alone misses it.
7. Is the owner wait bounded, and does the bound name the access?
8. Are the two refusal reasons counted separately?
9. Which counter would be non-zero if stale data were being delivered right now?

21. How This Appears in Real Engineering

Bring-up. The stale-read failure is the one that costs weeks, because it is silent, load-dependent and looks like an accelerator bug rather than a coherence bug. Teams look at the compute pipeline first.

Performance. The owner-source rate is a workload property the device cannot improve by being faster. Measuring it early changes sizing decisions; discovering it late changes schedules.

DV. The modified-owner directed case and the mid-flight directory change are the two stimuli that random traffic will not produce, and they are the two that find the worst bugs.

Post-silicon. The stale-read flag is a correctness alarm, not telemetry. It belongs in a register that is checked, not merely logged.

Firmware. Placement matters: buffers that a CPU thread actively writes are the ones that generate owner interactions, and moving them is often cheaper than any hardware change.

22. Common Misconceptions

23. Interview Reasoning

24. Exercises

1. Analysis. A device reports from_memory=10000, from_owner=0, and intermittent stale data on a workload where a CPU thread writes the same buffers. Nothing else is abnormal. Name the two implementation faults consistent with this and the one directed test that separates them.

2. Design. Extend the directory so a line may have several clean sharers and at most one modified owner. State the new invariant, and say which existing check becomes stronger and which becomes unnecessary.

3. RTL task. Add a second decision to decision_race so two different lines can have live decisions simultaneously. State what must be replicated per line, and which property stops being true if the lock is made global instead of per line.

4. DV task. Write the coverage cross for the device read path, then explain why two of its points can only be reached by directed injection.

5. Debug task. Owner-sourced reads are 4% of accesses but 60% of total read latency. Give your investigation order and the single ratio that decides whether this is a hardware or a data-placement problem.

6. Design review. A colleague proposes skipping the directory lookup for a memory region marked read-only by software. Give the strongest version of that argument, then name what enforces the promise and what happens the first time it is violated.

25. Summary

The address tells you where a line lives; only the directory tells you where its value is.

• Memory is authoritative exactly when no cache holds a modified copy — one expression, and the whole chapter.
• Cached is not stale. Only a modified copy displaces memory; a clean one is identical to it.
• The device asks; the coordinator resolves authority. The device's job is tracking, not deciding.
• One line, one access. Different lines proceed concurrently; the same line serialises, for correctness rather than fairness.
• Returns must match on identity, and the source must be checked against the decision — data that is correct by luck still means the decision path is broken.
• The decision must hold until the data lands. Consulting the directory is necessary and not sufficient; the line is locked for the decision's lifetime.
• Bound the owner wait, and name the access that hung — a shared timer blames a healthy one and leaves the real blockage in place.
• Two refusal reasons, two counters: table-full means undersized, line-busy means correct.
• Measured: five owner-sourced reads cost as much as fifteen memory-sourced ones. The owner-source rate is a workload property, not a device one.
• One counter here is a correctness alarm rather than telemetry: memory selected while a modified owner exists must be zero forever.

Chapter 8.3 turns the direction around: what the device owes when host coherence activity reaches a line the device is holding.