Host Access to Device Memory

Chapter 9.1 built the device's half of the window contract. This chapter is the other half: what the host must do so that an ordinary load instruction — one that knows nothing about CXL — arrives at that device and comes back correctly.

1. The Engineering Problem — The Instruction Does Not Know

A core executes a load. That instruction contains an address and nothing else. It does not name a device, it does not know a link exists, and it cannot be told to wait longer because the data is far away.

Everything that makes CXL memory reachable therefore happens after the instruction and before the data comes back:

Miss any of those and the failure is not a slow load. It is a core that stalls forever, a value delivered to the wrong register, or two targets answering one address.

And the host cannot retry in software. There is no driver in this path. Whatever the hardware does is what happens.

2. The One-Sentence Model

The host turns an address into an obligation it must track. Decode picks exactly one target, a tag makes the request identifiable, credits make it sendable, and a deadline makes its failure detectable — and the tag is not free again until the core has the value, not merely until the data arrived.

Call it the tracked obligation. The device in 9.1 validates identity; the host allocates it, and that difference drives everything here.

3. What This Chapter Owns

Deliberately not repeated: 3.1 already builds a host address decoder and outstanding-transaction table as generic structures. This chapter asks what changes when one of the targets is across a link — which turns out to be the tag pool, response reordering, credits and timeouts, none of which local DRAM needs.

4. The Exchange

Architectural. The arrows are obligations, not named messages.

The two loads at the top and bottom are the same instruction with different addresses. Everything that distinguishes them happens in the host, and the bottom path has no tag, no credit and no deadline because none is needed for a target that cannot disappear.

5. Teaching-model boundary

6. RTL 1 — Exactly One Target

  // Exactly one target must claim an address. Two is a configuration error
  // that produces silent corruption; zero is an abort, which is loud.
  assign n_targets = 2'd0 + in_dram + in_cxl;
  assign to_dram   = in_dram && (OVERLAP_ALLOWED || !in_cxl);
  assign to_cxl    = in_cxl  && (OVERLAP_ALLOWED || !in_dram);
  assign unclaimed = req_valid && (n_targets == 2'd0);

=== EXP1: one instruction, one target ===
  local DRAM address : dram=1 cxl=0 targets=1
  a local address goes to local memory                : ok
  CXL address        : dram=0 cxl=1 targets=1
  a CXL address goes to the CXL target                : ok
  the SAME load instruction reached both                : ok
  unowned address    : unclaimed=1 routed=0
  an address nobody owns is refused, not guessed      : ok
  every probe classified into exactly one target      : ok

Zero targets and two targets fail in opposite directions. Zero is an abort — loud, immediate, and the system tells you. Two is silent: writes land in one place and reads come from another, and no error is raised anywhere.

So the design refuses rather than guesses:

=== EXP1b: when two ranges overlap, refuse — do not guess ===
  overlapped address : targets=2 dram=0 cxl=0
  both ranges claimed it: a configuration error       : ok
  and it was routed nowhere rather than guessed       : ok
  overlap was reported, not silently resolved         : ok

Note that this experiment needed a router deliberately configured wrong. Overlapping ranges are a system misconfiguration; driving them into the instance under test would trip that instance's own checker and be scored a routing defect. That distinction — my fault versus the configuration's fault — has now come up in every chapter of this module.

7. RTL 2 — The Tag Pool Gates Issue

The host allocates transaction identity, so the tag pool is a hard limit on how many accesses can exist at once — and running out is a core stall that never reaches the link.

      end else if (issue) begin
        // No free tag. The core stalls; nothing reaches the link.
        n_tagstall_q <= n_tagstall_q + 16'd1;
      end

=== EXP2: the host owns the tag pool ===
  8 issues into 8 tags : outstanding=8 can_issue=0 issued=8
  with every tag live the host cannot issue           : ok
  9th issue attempt : tag stalls=1
  the core stalled; nothing reached the link          : ok
  peak outstanding was exactly the tag count, 8       : ok

A tag stall is invisible from the device. The link goes quiet, the device sees no traffic, and its own counters show it idle — while the host is stalled. This is the single most misleading performance situation in the module, and the only counter that identifies it lives on the host side.

8. RTL 3 — A Tag Frees When the Core Has the Value

=== EXP3: a tag frees when the CORE has the value ===
  data back only : correct outstanding=8 | free-on-data outstanding=7
  data arriving does not free the tag                 : ok
  the broken variant reclaimed it early               : ok
  the core consuming the value freed it               : ok
  and the broken variant then freed a dead tag        : ok

Data arriving at the host is not the end of the transaction — the value still has to reach the instruction that asked for it. Freeing at data-arrival lets the next load claim a tag whose value is still in flight to a register.

This is the third distinct place in Modules 8–9 where the same rule appears: release the resource at completion, not at the answer. Here the resource is the tag and the completion is the core consuming the value.

9. Waveform — Tag Exhaustion and an Out-of-Order Answer

Read issued against tag_stalls at cycle 6: the counter that moved is on the host. Nothing appeared on the link, so a device-side trace of this moment shows a perfectly idle device. That is why the tag-stall counter exists.

Then read data_tag at cycle 7 — it is 2, not 0. Tag 0 was issued first and has not answered. The host must be built for that.

10. RTL 4 — Responses Return Out of Order

  assign eff_tag     = MATCH_BY_ORDER ? oldest_q : rsp_tag;
  ...
        // The value must reach the request that asked for it.
        if (rsp_data != exp_m[eff_tag[2:0]]) mismatch_err <= 1'b1;

=== EXP4: responses come back out of order ===
  tag 2 answers first : correct delivers=2 | by-order delivers=0
  the tag chose the destination, not arrival order    : ok
  every value reached the request that asked          : ok
  the host observed and tolerated reordering          : ok
  matching by arrival order delivered wrongly         : ok

Reordering is not an exceptional case here. Two requests can land in different banks, hit different queue depths, or one can find the media busy — and the second issued is routinely the first answered. A host that assumes order works perfectly against local DRAM and corrupts data against a CXL device.

n_reorder_q counts how often a response was not the oldest outstanding. It is worth exposing: a rising reorder rate means the device's internal parallelism is being exercised, which is usually good news, and a rate of zero on a busy link suggests the device is serialising internally.

11. RTL 5 — Credits Carry Back-Pressure Across a Link

A combinational ready signal cannot cross a link — by the time it arrives, the sender has already sent. Credits solve that by telling the sender in advance how much room exists.

  assign send = want_send && (NO_CREDIT_CHECK || (credits_q != 4'd0));
      case ({send, credit_return})
        2'b10: credits_q <= credits_q - 4'd1;
        2'b01: credits_q <= credits_q + 4'd1;
        default: ;                          // both or neither: hold
      endcase

=== EXP5: credits carry back-pressure across a link ===
  12 sends, 8 credits : sent=8 blocked=4 credits=0 min=0
  the host sent exactly its credit budget, then stopped: ok
  credit low-water mark reached exactly zero          : ok
  no-credit-check variant : sent=12
  the unchecked variant sent past the device's room   : ok
  and flagged an overrun the device cannot absorb     : ok
  returned credits restored send capacity             : ok

min_credits_q is the counter worth having in silicon. Credits sitting near their maximum means the link is over-provisioned for the workload; credits pinned at zero means the device is the constraint and the host is throttled. The low-water mark distinguishes those, and an average does not.

Note the exhaustive case again — the sixth appearance in this track of the two-assignment counter defect, this time on credits, where the consequence is over-sending into a device with no room.

12. RTL 6 — Which Access Is Late

=== EXP6: which access is late ===
  tag 5 hung : timeouts=1 reported=5 max_age=21
  the per-access timer named the access that hung     : ok
  and it named tag 5 specifically                     : ok
  shared timer reported tag 2
  one shared timer blamed a healthy access            : ok

The shared timer reported tag 2 — an access issued long after tag 5 and progressing normally. The per-access timer reported tag 5, which is the one that never answered.

This matters more on the host than anywhere else in the module, because the recovery action is to fail a load that a core is waiting on. Failing the wrong one takes down healthy work and leaves the real hang in place.

13. RTL 7 — A Memory Access Is Not a Register Access

Both are loads from software's point of view, and they behave completely differently.

  // Register access serialises; memory access pipelines. That is the whole
  // performance difference, and it is invisible to a functional test.
  assign may_pipeline = use_mem_path;

=== EXP7: a memory access is not a register access ===
  device memory : correct mem_path=1 pipeline=1 | mmio-variant mem_path=0 pipeline=0
  a memory access pipelines                           : ok
  routed as a register access it serialises          : ok
  a genuine register access does not pipeline         : ok
  the mis-routing variant flagged a serialised access : ok

This is a functionally invisible catastrophe. A device-memory access routed down the register path returns the right value every time. Every functional test passes. What changes is that accesses stop overlapping — throughput collapses to one access per round trip, which at 250 ns is roughly 4 million accesses per second instead of tens of millions.

The distinction is 7.1's: CXL.io reaches device registers, CXL.mem reaches device memory. Getting the routing wrong does not break anything a test can see.

14. Quantitative Reasoning — Tags, Credits and the Two Limits

Illustrative. 9.1 established that filling a 32 GB/s link at 250 ns needs 125 outstanding requests. The host must supply that concurrency, and it has two independent ways to fall short:

Both look identical on the link — quiet — and they have different owners. A tag shortage is the host's problem; a credit shortage means the device is full. The counters that separate them are n_tagstall_q on the host and min_credits_q on the link.

Measured in this chapter's teaching configuration:

  8 issues into 8 tags : outstanding=8 can_issue=0
  9th issue attempt : tag stalls=1
  12 sends, 8 credits : sent=8 blocked=4 credits=0 min=0

Near and far, measured apart

=== EXP8: near memory and far memory, measured apart ===
  local : n=30 sum=240 max=8
  CXL   : n=10 sum=250 max=25
  the target split matched an independent oracle      : ok
  far memory measured slower than near, per access    : ok
  10 CXL accesses cost 250 cycles; 30 local cost 240
  tag stalls counted exactly (5 of 40)               : ok

Ten CXL accesses cost slightly more total time than thirty local ones. At roughly 3× the per-access latency, a quarter of the traffic consumed just over half the time.

That ratio is the number software people need, and it is why the split must be measured per target. An aggregate mean over this mix reports 12.25 cycles — a number that describes no access that actually happened, and that hides both the 8-cycle local case and the 25-cycle far case.

15. The Two Ceilings

All three must independently cover the bandwidth-delay product, and sizing two of them correctly buys nothing if the third is short. The design value of giving each its own counter is that the binding one becomes identifiable instead of inferred — and two of the three live on the host, which is the half teams instrument least.

16. Assertions

Icarus Verilog 13.0 does not support concurrent SVA here, so every property is synthesisable checker logic verified in simulation.

Safety

Liveness

Each names its assumption. The tag-stall property in particular is only true if something does complete — a host whose device never answers stalls permanently and violates no safety property at all.

Performance goals

17. Mutation Testing

Nineteen mutations. Nineteen killed.

The first run scored 16 of 19, and the two escapes were instructive in different ways.

"Overlapping ranges both route" survived because the good instance never saw an overlap. The mutation removes the !in_cxl guard, which is unobservable while the configured ranges are disjoint — and mine were. The repair was not more stimulus but a differently configured instance, which is a distinct idea from the abuse instances used elsewhere: the stimulus was legal, the configuration was the thing that had to be wrong.

"Data for an unknown tag" needed illegal stimulus, and got a dedicated instance asserting both that the stray data is flagged and that it completes nothing.

There is also a repair worth recording that mutation testing did not find — my own baseline caught it. Driving the overlapping configuration into the instance under test made that instance's own checker fire, and the testbench scored it as a routing failure. The correct device was failing its own test. That is now the fourth chapter in which the fix was to separate "a fault I am responsible for" from "a fault I am correctly reporting".

18. Verification Plan

The coverage cross is target × response order × resource state: local / CXL / unowned, crossed with in-order / out-of-order / never, crossed with tags-available / tags-exhausted / credits-exhausted. The tags-exhausted column is the one a testbench with a generous tag pool never reaches, and it is where the most misleading performance bug lives.

19. Silicon Observability

The pair that resolves the most common "CXL is slow" escalation is tag stalls against minimum credits. Both produce a quiet link and low throughput; one means the host cannot track more accesses and the other means the device cannot hold more. They have different owners and different fixes, and an aggregate utilisation number distinguishes neither.

20. Debug Lab

assign to_dram = in_dram;      // no exclusivity check
assign to_cxl  = in_cxl;

  overlapped address : targets=2 dram=0 cxl=0
  both ranges claimed it: a configuration error       : ok

assign n_targets = in_dram + in_cxl;
assign to_dram   = in_dram && !in_cxl;
assign to_cxl    = in_cxl  && !in_dram;
if (n_targets > 1) overlap_err <= 1'b1;

host_outstanding #(.NTAG(8)) u_track (...);   // sized for local DRAM

  8 issues into 8 tags : outstanding=8 can_issue=0
  9th issue attempt : tag stalls=1

deliver_tag <= oldest_q;      // assume responses return in order
oldest_q    <= oldest_q + 1;

  tag 2 answers first : correct delivers=2 | by-order delivers=0

assign eff_tag = rsp_tag;                       // identity, not order
if (rsp_data != exp_m[eff_tag]) mismatch_err <= 1'b1;

if (data_back) busy_q[data_tag] <= 1'b0;   // freed when data arrives

  data back only : correct outstanding=8 | free-on-data outstanding=7
  and the broken variant then freed a dead tag        : ok

if (data_back) arrived_q[data_tag] <= 1'b1;   // answered
if (consumed)  busy_q[consumed_tag] <= 1'b0;  // finished — only now

assign send = want_send;      // credits ignored

  12 sends, 8 credits : sent=8 blocked=4
  the unchecked variant sent past the device's room   : ok

assign send = want_send && (credits_q != 0);
if (send && (credits_q == 0)) overrun_err <= 1'b1;

if (|wait_q) gtimer_q <= gtimer_q + 1;
if (gtimer_q >= LIMIT) begin timeout <= 1; timeout_tag <= any_waiting; end

  tag 5 hung : timeouts=1 reported=5 max_age=21
  shared timer reported tag 2
  one shared timer blamed a healthy access            : ok

for (k = 0; k < NTAG; k++) if (wait_q[k]) age_m[k] <= age_m[k] + 1;
if (wait_q[k] && (age_m[k] >= LIMIT)) begin hit = 1; pick = k; end
if (hit && age_m[pick] < LIMIT) wrong_tag_err <= 1'b1;

// Device address — use the register path.
assign use_reg_path = req_valid && addr_is_device;

  device memory : correct mem_path=1 pipeline=1 | mmio-variant mem_path=0 pipeline=0
  routed as a register access it serialises          : ok

assign use_mem_path = req_valid && addr_is_mem;
assign may_pipeline = use_mem_path;
if (addr_is_mem && !may_pipeline) mem_serialised_err <= 1'b1;

21. Design Review

1. Can two decode ranges claim one address, and what happens if they do?
2. Is the tag pool sized from bandwidth × latency, or inherited from the local-DRAM design?
3. When does a tag free — on data arrival or on consumption?
4. Does response matching use identity or arrival order?
5. What stops the host sending into a device with no room?
6. Is the timeout per-access, and does it name the tag?
7. Does device memory pipeline, and what asserts that it does?
8. Which counter distinguishes a tag shortage from a credit shortage?
9. Is latency measured separately for near and far targets?
10. What happens on reset with loads outstanding?

22. How This Appears in Real Engineering

Architecture. The tag pool is the host-side twin of the device's outstanding table, and both must cover the same bandwidth-delay product. Sizing one and not the other caps throughput just as effectively.

RTL. Response matching by tag rather than order is a small change that is easy to get wrong and impossible to catch functionally against local memory.

DV. Out-of-order responses, an exhausted tag pool and a deliberately slow device are all directed scenarios. None arises naturally from a well-behaved model.

Post-silicon. Tag stalls versus minimum credits is the first diagnostic split on any "CXL is slow" report, and both counters must exist before tape-out.

Software and placement. The near/far latency split is what tells a runtime whether a hot data structure has landed in far memory — a placement decision, not a hardware one, and usually the cheapest thing to change.

23. Common Misconceptions

24. Interview Reasoning

25. Exercises

1. Calculation. A host has 16 tags and issues 64-byte accesses to a device with a 300 ns round trip. Compute the maximum achievable bandwidth. Then state how many tags are needed to reach 24 GB/s, and which counter would confirm the shortage in silicon.

2. Analysis. A system reports tag_stalls=0, min_credits=0, reorder_count=0, and low CXL throughput. Explain what each of the three readings rules out, and name the remaining candidate.

3. RTL task. Extend target_route to three targets with a programmable priority for ambiguous addresses. State why priority is a worse answer than refusal, and what you would have to add to make it defensible.

4. Assertion task. Write the property that catches a tag freed before the core consumed the value, and explain why it must span two events rather than one cycle.

5. Debug task. Bandwidth to CXL memory is two orders of magnitude below expectation and every functional test passes. Give your investigation order and the single property that identifies the cause.

6. Design review. A colleague proposes matching responses by arrival order "because the device answers in order in practice". Give the strongest version of that argument, then name what makes it false and the one workload that exposes it fastest.

26. Summary

The host turns an address into an obligation it must track.

• A load instruction carries an address and nothing else. Everything else is hardware.
• Decode must yield exactly one target. Zero aborts loudly; two corrupts silently, so ambiguity must be refused rather than resolved.
• The host allocates transaction identity; the device validates it. The tag pool is therefore a hard ceiling on concurrency.
• A tag shortage is invisible from the device — nothing reaches the link, and every device counter reports idle.
• A tag frees when the core consumes the value, not when data arrives.
• Responses return out of order as normal operation. Matching by arrival order works against local DRAM and corrupts against CXL.
• Credits carry back-pressure across a link, because a ready signal cannot. The two-assignment counter defect appeared for the sixth time in this track.
• Timeouts must be per-access. The shared timer blamed tag 2 while tag 5 was the one that hung.
• Routing device memory down the register path is functionally invisible and costs roughly two orders of magnitude of throughput.
• Measured: 10 CXL accesses cost more total time than 30 local ones — which is why latency must be split by target rather than averaged.
• Verification lesson: one mutation survived because the configuration, not the stimulus, was too well-behaved — configuration space needs coverage too.

Chapter 9.3 asks what the access actually guarantees: ordering, atomicity, and the coherence contract that makes device memory usable as memory rather than merely reachable.