Type 2 Devices

Chapter 10.1 built a device that borrows and lends nothing, and its defining behaviour was a refusal: a Type 1 has no host-visible memory, so an inbound memory access has no correct answer.

Remove that refusal and you have a Type 2. It sounds like an addition. It is an interaction.

1. The Engineering Problem — A Device That Must Ask Permission to Use Its Own Memory

An accelerator has memory on its own board — high-bandwidth memory sitting centimetres from its compute, wired directly to it. It also wants that memory visible to the host, so software can fill it, read results out of it, and treat it as part of the machine's address space rather than something reachable only by explicit copies.

Both of those are reasonable and each is easy alone. Together they produce a situation with no counterpart in a Type 1 or a Type 3 device:

The host may be holding a line of the device's own memory when the device wants to use it.

The device is physically attached to that DRAM. It owns the wires. And it cannot read the location, because the current value is in a host cache and the copy in its own DRAM is stale. It has to ask for it back.

That is the Type 2 problem, and everything else in this chapter is a consequence of it.

2. The One-Sentence Model

A Type 2 device is its own neighbour. It caches memory belonging to someone else and owns memory that someone else caches, so it must ask permission to use storage it physically owns — and it is simultaneously a requester in one direction and a responder in the other.

Call it the neighbour. The word to hold on to is permission: physical ownership is not permission, and a Type 2 device that assumes otherwise reads stale data from its own DRAM.

3. What This Chapter Owns

3.3 flagged this chapter's subject precisely: "Type 2 is not 'Type 1 plus Type 3'. It is a device that both caches host memory and exposes its own — and those two relationships interact... That interaction is why Type 2 is the architecturally hardest class and why its treatment belongs in Module 10."

6.5 already priced the interaction — its cost model predicted 41 additively and measured 50, a 9-unit interaction term. This chapter builds what that 9 units is.

4. Two Relationships, One Device

Architectural. The arrows are relationships, not messages.

Follow the bottom row. Device compute wants a line of attached memory. That memory is physically on the device's own board, and the path to it goes through a permission check, because a host cache may hold the current copy.

A Type 1 has only the top-left path. A Type 3 has only the bottom-right one. A Type 2 has both, and they meet at the attached memory — which is why the interaction is not additive and why 6.5's model needed a cross term.

5. Teaching-model boundary

6. RTL 1 — Every Exposed Region Carries an Attribute

The device now has memory the host can reach, so the first hardware question is which memory, and under what coherence terms.

  assign mapped      = hit;
  assign dev_managed = NO_ATTR ? 1'b0 : (hit && attr);
  assign accept      = acc_valid && (ALLOW_GAP || hit);

=== EXP1: a Type 2 exposes memory, and every region has an attribute ===
  addr 10 : mapped=1 device-managed=0 accept=1
  a host-only-coherent region is mapped and needs no check : ok
  addr 70 : mapped=1 device-managed=1 accept=1
  a device-managed region is mapped and carries coherence state : ok
  addr 200: mapped=0 accept=0 | allow-gap variant accept=1
  an unmapped address is not accepted                    : ok
  the gap variant was caught inventing memory            : ok
  last address of each region : 31=1 63=1 95=1
  every region includes its own last address             : ok

The published distinction is between memory that "does not include any coherence protocol assumptions" and memory that "includes cache state and cache snooping attributes". In hardware that is one bit per region, and it decides whether §7's permission check applies at all.

That bit is the cheapest optimisation available to a Type 2 device. Memory the host will never cache needs no check and no recall path; memory the host may cache needs both. Marking everything as the stricter kind is safe and costs the device a permission check on every access to its own DRAM.

The boundary test that had to be added

The mutation that changed acc_addr <= limit[k] to < survived the first run, because the test used addresses 10, 70 and 200 — comfortably inside regions and comfortably outside. A region whose last address falls outside itself loses exactly one line per region, and every access to it is refused as unmapped. It is a one-character edit that produces a sparse, address-dependent failure, and only a boundary test finds it.

7. RTL 2 — Physical Ownership Is Not Permission

This is the chapter, in fourteen lines.

  // A host-only-coherent region makes no coherence assumptions, so the device
  // may use it directly. A device-managed region must be recalled first.
  wire blocked = dev_managed && host_holds;
 
  assign proceed     = dev_acc && (ASSUME_OWNED || !blocked);
  assign need_recall = dev_acc && blocked && !ASSUME_OWNED;

=== EXP2: physical ownership is not permission ===
  host-only region, host not holding : proceed=1 recall=0
  the device used its own memory directly                : ok
  managed region, host not holding   : proceed=1 recall=0
  a managed line the host does not hold is usable directly : ok
  managed region, HOST HOLDS IT      : correct proceed=0 recall=1 | assume-owned proceed=1
  the device could NOT use memory it physically owns     : ok
  the assume-owned variant used it regardless            : ok
  host-only region while host holds  : proceed=1
  a host-only-coherent region needs no recall            : ok
  direct=3 recalls=1 stalls=1 | assume-owned stale-use flag=1
  the assume-owned variant was caught reading a stale line : ok

Read the third case. The region is device-managed, the host holds the line, and the device — which is physically wired to that DRAM — cannot read it. The data in its own memory is stale; the current value is in a cache on the other side of the link.

Read the fourth case too, because it is the design lever. Same host-holds condition, but a host-only-coherent region: the device proceeds immediately. The attribute is what makes the difference, and choosing it well is most of what a Type 2 device can do about its own performance.

The ASSUME_OWNED variant is the mistake this chapter exists to prevent, and it is an extremely natural one to make. Every instinct a designer has says the device owns the memory attached to it. In a Type 2 that instinct silently reads stale data.

8. Waveform — The Device Locked Out of Its Own DRAM

Teaching-model timing derived from the simplified RTL in this chapter. Not CXL wire timing.

direct_uses is frozen at 3 across cycles 4 to 7. Those are four cycles in which an accelerator sat idle in front of memory bolted to its own board. No component failed, nothing is slow, and the link is not the problem — the current copy is simply elsewhere.

Compare the first two cycles with cycles 4 to 7: identical host_holds behaviour would have changed nothing there, because the region was host-only coherent. The attribute bit is the difference between those two outcomes, which is why §6's one bit per region is the most consequential configuration decision a Type 2 device exposes.

9. RTL 3 — Changing Owner Is a Barrier, Not a Bit

If the device is going to work on a region for a while, and then the host is, something has to hand it over. That handover is not a flag write.

  wire       drained = (outstanding == 8'd0);
  assign flip_done = flipping_q && (FLIP_WITHOUT_DRAIN || drained);

=== EXP4: changing owner is a barrier, not a bit ===
  flip requested, 3 still outstanding : done=0 owner=0 wait=4 | no-drain done=0
  the flip has not completed and the owner has not changed : ok
  it has been waiting exactly 4 cycles                   : ok
  the no-drain variant already handed the region over    : ok
  and was caught flipping with work still in flight      : ok
  drained : done=0 owner=1 flips=1 max wait=4
  the owner changed only after the drain                 : ok
  one flip, worst wait 4 cycles                          : ok

The shape is 9.6's barrier, in a new place. Work outstanding for the old owner must drain before the new owner may proceed, so the cost of a handover is the depth of the work in flight — not the cost of the handover logic, which is trivial.

That has the same counter-intuitive consequence: the more work you allow in flight, the more expensive each handover becomes. A Type 2 device tuned for deep pipelining pays for it at every ownership change, and a workload that alternates fine-grained between device and host phases can spend more time handing over than computing.

The no-drain variant handed the region over immediately and was caught with work still in flight. It is fast for the same reason it is wrong.

10. RTL 4 — Two Roles, Two Identity Spaces

A Type 1 is a requester. A Type 3 is a responder. A Type 2 is both at the same time, and the identities it uses for each must not be confused.

        if (SHARED_SPACE) begin
          // Retiring from the outbound pool: if that bit belonged to a borrow,
          // the borrow has just been silently cancelled.
          if (out_live_q[in_rsp_id] && !in_live_q[in_rsp_id]) cross_role_err <= 1'b1;
          out_live_q[in_rsp_id] <= 1'b0;

=== EXP3: a Type 2 is a requester AND a responder ===
  outbound tag 3 and inbound id 3 both live : out=00001000 in=00001000
  the same number is live in both roles at once          : ok
  inbound 3 retired : out=00001000 in=00000000 cross-role=0
  the outbound borrow survived                           : ok
  shared-space variant : out=00000000 cross-role flag=1
  the shared-space variant cancelled a borrow with an inbound id : ok
  borrows 3 and 5 live : out=00101000
  borrow 3 returned    : out=00100000
  one response retired exactly one borrow                : ok

The number 3 is live in both directions simultaneously and means two unrelated transactions. One is a line this device borrowed from host memory; the other is a host request being served from this device's memory. They have nothing to do with each other.

The shared-space variant uses one pool for both, so an inbound response retires an outbound borrow. The borrow is silently cancelled — no error, no timeout — and when its real response eventually arrives it matches nothing, or worse, matches a reused tag.

This is 10.1's identity discipline with a second axis: not only "is this tag live?" but "live in which role?" A Type 1 never has to ask the second question, which is a concrete example of the interaction costing more than the sum.

11. RTL 5 — The Cycle Both Directions Can Close

The most serious structural hazard in a Type 2 device, and the one that is invisible until both directions are busy at once.

  // The responder path must be drainable on its own. Coupling it to outbound
  // progress closes the cycle.
  assign serve_in = in_pending && in_can_serve &&
                    (COUPLE_PATHS ? !out_pending : 1'b1);

=== EXP5: two directions can close a cycle ===
  both directions pending, 15 cycles : correct served=15 | coupled served=0 wait=12
  the independent responder path served every cycle      : ok
  the coupled variant served nothing at all              : ok
  the coupled variant was caught deadlocking             : ok
  wait grew to 5, one service, then one stalled cycle : wait=1
  after a service the wait restarted from zero           : ok

Fifteen cycles, fifteen served versus zero served. The coupled variant is not slow — it is stopped, permanently, and only while both directions have work.

The cycle is easy to build by accident. The device is waiting on the host to return a borrowed line. The host is waiting on the device to serve a request from device memory. If the device's inbound service path shares a resource with, or waits on, its outbound path — a buffer, a credit pool, an arbiter that prioritises outbound, an ordering rule — neither side can move.

The rule is a one-liner and it is the most important sentence in this chapter's design review: inbound service must never require an outbound completion. The responder path has to be drainable on its own. That is not a performance guideline; it is what stops the two roles from forming a cycle.

It is also invisible in single-direction testing, which is how most bring-up proceeds — one path at a time, both working perfectly.

Rows three and four are the deadlock, written out. Neither party is faulty and neither is slow — each is waiting for something entirely reasonable, and the cycle exists only because one path was made to depend on the other. Rows five and six are the same scenario with that dependency removed.

12. RTL 6 — Contains, Negotiated, Enabled

  // The type the device is BEHAVING as, which is not the type it IS.
  assign effective_type = ( cache_en &&  mem_en) ? 2'd2 :
                          ( cache_en && !mem_en) ? 2'd1 :
                          (!cache_en &&  mem_en) ? 2'd3 : 2'd0;

=== EXP6: contains, negotiated, enabled ===
  Type 2 silicon, both enabled  : cache=1 mem=1 effective type=2
  a Type 2 with both enabled behaves as Type 2           : ok
  Type 2 silicon, .mem disabled : cache=1 mem=0 effective type=1
  with .mem disabled it BEHAVES as Type 1                : ok
  Type 1 silicon, .mem requested: cache=1 mem=0 effective=1 | no-gate mem=1
  a Type 1 cannot be enabled into Type 2                 : ok
  the ungated variant enabled memory it does not have    : ok
  and was caught enabling an absent protocol             : ok

6.5 established three states that must stay distinct — what silicon contains, what the link negotiated, what the platform enabled. This is the hardware that keeps them apart.

A Type 2 with .mem disabled behaves as a Type 1 and is not one. The cache engine is still present, still holding host lines, still carrying 10.1's eviction obligations — and its own memory is still physically there, merely unexposed. Treating that configuration as a Type 3, or as a Type 1, means testing the wrong device.

The ungated variant enables memory the silicon does not have. That is 10.1's declared-type defect one level up: not lying about what it is, but being told to be something it cannot be, and agreeing.

13. RTL 7 — The Interaction Is What Must Be Observable

=== EXP7: the interaction is what must be observable ===
  outbound=14 inbound=10 both-directions=4 direct=17 recalls=4 flips=4
  oracle: outbound=14 inbound=10 both=4 direct=17 recalls=4
  both roles matched an independent oracle               : ok
  the both-directions counter matched the oracle         : ok
  the device really did have traffic in both directions at once : ok
  abuse: recalls with no local accesses : flag=1

both_directions = 4 is the counter that does not exist on a Type 1 or a Type 3, and it is the one that matters most here. Four cycles in forty had traffic in both directions simultaneously — which is exactly the condition under which §11's cycle can close and §7's permission check is most likely to block.

A device that reports outbound traffic and inbound traffic separately, and never their coincidence, cannot see the state in which its hardest bugs live. The merged-roles mutation is the opposite failure: one counter for both directions, which loses the distinction entirely.

The recalls counter is the direct measure of §7's cost: 4 recalls against 17 direct uses means 19% of the device's accesses to its own memory required asking the host first.

14. Quantitative Reasoning

Illustrative, with assumptions stated.

What a recall costs

From §13: 21 local accesses, 17 direct, 4 recalls. Suppose a direct local access costs 20 ns (on-board memory) and a recall costs a full round trip of 250 ns plus the access:

direct only        : 20 ns
recall then access : 250 + 20 = 270 ns
mean at 19% recalls: 0.81 x 20 + 0.19 x 270 = 67.5 ns

A 3.4× increase in the mean cost of touching the device's own memory, caused entirely by the host holding lines. And the sensitivity is brutal:

This is the number that decides whether a Type 2 device is worth building. The whole premise is memory close to compute; at a 50% recall rate that memory behaves like memory across a link, and the device has paid for on-board DRAM to get remote-DRAM latency.

What the attribute bit is worth

Every access in a host-only-coherent region skips the check entirely. If a workload can place 80% of its data in such regions:

mean = 0.80 x 20 + 0.20 x 67.5 = 29.5 ns

Down from 67.5 ns — a 2.3× improvement from a region attribute, with no change to the compute, the memory, or the link. That is why §6 calls it the cheapest optimisation available to a Type 2.

What a handover costs

From §9, the flip waited 4 cycles for 3 outstanding items. Scaled: a handover costs the drain of everything in flight. With 9.6's 125-outstanding budget and a 250 ns round trip, a handover at full occupancy costs roughly one full round trip during which neither side computes.

A workload alternating every 1000 accesses spends about 0.25% of its time handing over; one alternating every 10 spends 25%. Fine-grained alternation between device and host phases is the pathological case for a Type 2, and it is a software structure rather than a hardware one.

What the interaction costs in verification

6.5 measured a 9-unit interaction term on top of an additive 41. In DV terms this chapter names what those units are:

Four pairwise crosses that exist only because both engines are present. A Type 1 plan and a Type 3 plan, run back to back, cover neither engine's interaction with the other — which is exactly why the cost is not additive.

15. Assertions

Icarus Verilog 13.0 does not support concurrent SVA here, so every property is synthesisable checker logic verified in simulation. 52 assertions.

Safety

Liveness

The third is the deadlock property, and it is the reason cycle_guard has a timeout at all. A liveness failure here is not slowness; it is a permanent stop that only occurs when both directions are busy.

Performance goals

16. Mutation Testing

Twenty-five mutations. Twenty-five killed.

The first run scored 22 of 25, and all three escapes were missing stimulus rather than missing checks — a pattern now consistent enough across this track to be worth naming.

A region boundary. The test used addresses comfortably inside and outside regions, so a limit comparator that excludes its own last address survived. It loses one line per region and produces a sparse, address-dependent failure.

A second concurrent borrow. With only one outbound borrow live, a mutation that retires all borrows on any response is indistinguishable from one that retires the right one. Two live borrows separate them immediately.

A wait counter across a service. The correct guard served every cycle, so the inbound wait never accumulated and clearing it was never tested. The fix was a directed sequence — stall, accumulate to 5, serve once, stall again — asserting the wait restarted from zero rather than resuming.

Two wrong expectations of mine, both caught by the baseline: I predicted flip waits of 5 and 6 cycles and the measured values were 4 and 4. The design was right; the assertions now carry the measured values, because a prediction rounded toward what the theory suggested is how a model stops being checkable.

17. Verification Plan

The coverage cross that defines this chapter is region attribute × host-holds × direction concurrency. The point where a device-managed region is held by the host while both directions have work is where the permission check, the recall path and the cycle hazard all interact — and it is unreachable by any single-engine test.

18. Silicon Observability

The single most valuable counter here is the recall rate, because it directly answers whether the device's premise holds. On-board memory that requires a round trip 50% of the time is not on-board memory in any sense that matters — the design has paid for local DRAM and is getting remote latency.

The second is both-directions cycles, because it is the only counter that reports the state in which the cycle hazard exists, and it is not present on either simpler device type.

19. Debug Lab

assign proceed = dev_acc;   // it's our memory, it's on our board

  managed region, HOST HOLDS IT : correct proceed=0 recall=1 | assume-owned proceed=1
  the assume-owned variant was caught reading a stale line : ok

wire blocked   = dev_managed && host_holds;
assign proceed = dev_acc && !blocked;
if (proceed && blocked) used_while_host_held_err <= 1'b1;

assign serve_in = in_pending && in_can_serve && !out_pending;

  both directions pending, 15 cycles : correct served=15 | coupled served=0 wait=12
  the coupled variant served nothing at all              : ok
  the coupled variant was caught deadlocking             : ok

assign serve_in = in_pending && in_can_serve;

reg [7:0] live_q;   // one pool for both directions

  inbound 3 retired : out=00001000 in=00000000 cross-role=0
  the outbound borrow survived                           : ok
  shared-space variant : out=00000000 cross-role flag=1

reg [7:0] out_live_q, in_live_q;   // separate pools, separate meanings

if (flip_req) dev_owned_q <= ~dev_owned_q;   // ownership is a bit

  flip requested, 3 still outstanding : done=0 owner=0 wait=4 | no-drain done=0
  the no-drain variant already handed the region over    : ok
  and was caught flipping with work still in flight      : ok

wire drained     = (outstanding == 8'd0);
assign flip_done = flipping_q && drained;
if (flip_done && !drained) flipped_dirty_err <= 1'b1;

if ((acc_addr >= base[k]) && (acc_addr < limit[k]))   // was <=

  last address of each region : 31=1 63=1 95=1
  every region includes its own last address             : ok

if ((acc_addr >= base[k]) && (acc_addr <= limit[k]))

assign mem_en = req_mem_en;   // whatever the platform asks for

  Type 1 silicon, .mem requested: cache=1 mem=0 effective=1 | no-gate mem=1
  the ungated variant enabled memory it does not have    : ok
  and was caught enabling an absent protocol             : ok

assign mem_en = req_mem_en && HAS_MEM;
if ((cache_en && !HAS_CACHE) || (mem_en && !HAS_MEM))
  enabled_absent_err <= 1'b1;

// .mem disabled -> treat as a memory-less device, skip the cache tests

  Type 2 silicon, .mem disabled : cache=1 mem=0 effective type=1
  with .mem disabled it BEHAVES as Type 1                : ok

assign effective_type = ( cache_en &&  mem_en) ? 2'd2 :
                        ( cache_en && !mem_en) ? 2'd1 : ...

wire counted = out_acc || in_acc;   // "traffic is traffic"

  outbound=14 inbound=10 both-directions=4 direct=17 recalls=4
  both roles matched an independent oracle               : ok
  the device really did have traffic in both directions at once : ok

wire both_now = out_acc && in_acc;
if (both_now) n_both_q <= n_both_q + 16'd1;

20. Design Review

1. Which exposed regions are device-managed and which make no coherence assumptions?
2. What happens when the device accesses a device-managed line the host holds?
3. Can the device ever read its own memory without checking?
4. Are the outbound and inbound identity spaces separate?
5. Can inbound service ever be blocked by outbound progress?
6. What is the maximum inbound wait, and what reports it?
7. What must drain before a region changes hands, and what proves it did?
8. What is the recall rate under real load?
9. Which enable combinations are reachable, and what is the effective type of each?
10. Does any counter report cycles with traffic in both directions?
11. What survives reset — held host lines, or exposed regions, or both?

21. How This Appears in Real Engineering

Architecture. The recall rate decides whether the device's premise survives. On-board memory needing a round trip half the time is remote memory with extra cost.

RTL. Two identity spaces, two independent drain paths, and one permission check on the local memory path. The cycle hazard is a structural review item, not a performance one.

DV. The four pairwise crosses in §14 exist only because both engines are present, and neither single-engine plan contains them.

Performance. The region attribute is the cheapest lever available — 2.3× in the model — and it is a placement decision rather than a hardware one.

Firmware. Every reachable enable combination is a distinct device with a distinct effective type, and the degraded modes are the ones that get mis-tested.

Post-silicon. Recall rate and both-directions cycles are the two counters no simpler device type has, and every hard Type 2 bug lives where they are both non-zero.

22. Common Misconceptions

23. Interview Reasoning

24. Exercises

1. Calculation. A Type 2 device has 20 ns local memory and a 250 ns recall. Compute the mean local access time at recall rates of 0%, 10%, 30% and 60%. Then compute the recall rate at which local memory offers no advantage over simply reading host memory at 250 ns, and state what that threshold implies for region placement.

2. Analysis. A device reports healthy outbound throughput, healthy inbound throughput, zero errors, and hangs under a mixed workload. Name the hazard, the counter whose absence made it invisible, and the single regression change that would have caught it.

3. RTL task. Extend local_access_gate to track, per line, whether a recall has been requested but not yet satisfied. State the new invariant that creates, and the failure mode if a second access to the same line is allowed to issue a duplicate recall.

4. Assertion task. Write the property that proves inbound service is independent of outbound progress. Explain why it cannot be verified with stimulus that drives one direction at a time, and what the stimulus must look like instead.

5. Waveform analysis. Using §8's trace, compute the fraction of cycles the device was locked out of its own memory, and the direct-use rate before and after. Then state what the trace would look like if the region had been host-only coherent throughout, and which counter would distinguish the two runs.

6. Design review. A colleague proposes a single shared transaction-identity pool for both directions, arguing it halves the tracking state and simplifies the arbiter. Give the strongest version of that argument, then state precisely what breaks, why random testing will not find it, and the directed test that will.

7. Architecture tradeoff. Your accelerator needs 512 GB of memory visible to the host and coherent access to a 4 GB host-resident index. Compare building it as one Type 2 device against a Type 3 expander plus a separate Type 1 accelerator. Cover state, verification crosses, the caching-device fanout limit, and the failure modes each option removes.

25. Summary

A Type 2 device is its own neighbour.

• It caches host memory and exposes memory the host caches, so physical ownership is not permission — it must ask for its own lines back.
• Every exposed region carries a coherence attribute. Host-only-coherent regions make no coherence assumptions and need no check; device-managed regions carry cache state and do. That one bit is worth 2.3× in the model.
• Measured: 19% of local accesses required a recall, turning a 20 ns access into a 67.5 ns mean — 3.4×. At 50% the device has bought local DRAM and is getting remote latency.
• Changing owner is a barrier, not a bit. The handover costs the drain, so deeper pipelining makes every handover more expensive.
• Two roles, two identity spaces. The same number can be live in both directions meaning unrelated transactions; one shared pool silently cancels borrows.
• The two directions can close a cycle — 15 served versus 0 in the measured contrast. Inbound service must never require an outbound completion, and the hazard is invisible to single-direction testing.
• Contains, negotiated, enabled must stay distinct. A Type 2 with .mem disabled behaves as a Type 1 and is not one — its cache engine still holds host lines.
• The interaction must be observable: cycles with traffic in both directions is the counter no simpler device type has, and every hard bug here lives where it is non-zero.
• Verification: 25 of 25 mutations killed, 52 assertions. All three first-run escapes were missing stimulus rather than missing checks — an untested region boundary, a single live borrow where two were needed, and a wait counter never exercised across a service.

Next: 10.3 — Type 3 Devices, the device that only lends — where everything in this chapter disappears at once, and that simplicity is exactly why it is the class that scales.