Type 3 Devices

Chapter 10.2 built the hardest device in the taxonomy — one that borrows and lends at the same time, and must ask permission to use memory it physically owns.

Take away the borrowing and almost all of it disappears.

1. The Engineering Problem — A Server Runs Out of Memory Slots Before It Runs Out of Work

A socket has a fixed number of DDR channels. Adding memory beyond them is not a matter of cost or willingness — the pins do not exist. Meanwhile working sets grow, and a server with idle cores and full memory is a machine that has stopped scaling for a reason that has nothing to do with compute.

The fix is memory that arrives over a link rather than over dedicated channels. And the moment you propose that, the design question becomes narrow and specific:

What is the least a device can be, and still be memory?

Not "how do we build an accelerator that also has memory" — that was 10.2, and it was hard. This is the opposite exercise: strip away everything that is not required to serve host memory accesses, and see what remains.

What remains is a Type 3 device, and its simplicity is the entire product.

2. The One-Sentence Model

A Type 3 device only answers. It exposes memory to the host and never originates a coherent request of its own, so it has no cache, no coherence agent tracking its own copies, no second identity space, no permission check on its own memory, no handover, and no cycle between two directions — and that is precisely why you can put many of them in a system.

Call it the responder. Everything else follows: with one relationship instead of two, the device's hard problems become memory problems — interleaving, capacity, media health — rather than coherence problems.

3. What This Chapter Owns

Module 9 taught the protocol a Type 3 speaks. This chapter is about the device: the media behind the protocol, the decode in front of it, the capacity contract, and the one structural fact — that it never asks — from which its scalability follows.

4. What Is Left After Removing One Engine

Architectural. The arrows are dependencies, not signals.

The left column is what a Type 3 does not have, and it is a chain rather than a list. Removing the cache engine removes the coherence agent that tracked its contents, which removes the permission check on local memory, which removes the second identity space and the handover and the cycle between directions. One deletion, six consequences.

The right column is what that buys: many of these devices per host, because a memory device adds no per-device coherence tracking state. §12 measures it.

5. Teaching-model boundary

6. RTL 1 — Consecutive Addresses Must Reach Different Channels

The first thing a memory expander does with an address is decide where it lives, and that decision determines whether the device has bandwidth at all.

  // Interleave on the bits just above the granularity, so consecutive
  // granules land on different channels.
  wire [1:0] low_sel  = acc_addr[GRAN_LOG2+1 -: 2];
  assign channel   = NO_INTERLEAVE ? 2'd0 : (HIGH_BITS ? high_sel : low_sel);
  // Whatever selects the channel must be removed from the per-bank address,
  // or two different addresses land on the same bank location.
  assign bank_addr = ... {acc_addr[11:GRAN_LOG2+2], acc_addr[GRAN_LOG2-1:0]};

=== EXP1: consecutive addresses must reach different channels ===
  16 consecutive granules : ch0=4 ch1=4 ch2=4 ch3=4
  channel distribution matched an independent oracle     : ok
  the load was spread exactly evenly, 4 per channel      : ok
  no-interleave variant   : ch0=16 ch1=0 ch2=0 ch3=0
  the no-interleave variant sent everything to one channel : ok
  exact decode : addr0->ch0/bank0 addr4->ch1/bank0 addr8->ch2 addr16->ch0/bank4
  consecutive granules landed on channels 0, 1, 2        : ok
  two different channels reuse bank address 0            : ok
  the wrapped granule advanced the bank address to 4     : ok

Interleaving is what turns four channels into four times the bandwidth. A sequential stream is the common case, and without interleaving every access in it queues on one channel while three sit idle — 16 versus 0, 0, 0 in the measured contrast.

The second half is the part that gets designed wrong. Whatever bits select the channel must be removed from the per-bank address. Note addr0 and addr4 both map to bank address 0 on different channels: that is correct, and it is what lets each channel use its full address space. Leave the channel bits in and every channel uses only a quarter of its own banks — the device advertises its capacity and can physically reach a fraction of it.

An even distribution proved nothing

The mutation that shifted the channel-select bits by one survived the first run, and the reason is instructive. Over a strided stream, selecting addr[4:3] instead of addr[3:2] still produces a perfectly even 4/4/4/4 distribution — it just assigns the wrong addresses to each channel.

Evenness is a statistic; a decode is a function. The fix was to assert the exact channel and the exact bank address for specific addresses, which is the same "assert the value, not the property" lesson that 10.1's victim-rotation check taught in a different disguise.

7. RTL 2 — Throughput Comes From Banks Busy At Once

  wire bank_free = (timer_q[bank] == 8'd0);
  assign accept  = issue && (IGNORE_BUSY || bank_free);

=== EXP2: throughput comes from banks busy at once ===
  one access per bank : issued=4 busy=4 peak=4
  four banks were busy simultaneously                    : ok
  four accesses to ONE bank : issued=5 blocked=3 | ignore-busy issued=8 overlap=1
  the busy bank pushed back                              : ok
  the ignore-busy variant overlapped two media accesses  : ok

This is 9.6's concurrency argument arriving at the media. A bank serves one access at a time, so the device's throughput is the number of banks it can keep busy — and §6's decode is what determines whether traffic reaches them all.

The ignore-busy variant is the media version of every "faster because wrong" defect in this track: it issues 8 where the correct design issues 5, by overlapping two accesses on media that can physically serve one.

8. Waveform — Spread Across Banks, Then Hammering One

Teaching-model timing derived from the simplified RTL in this chapter. Not CXL or DRAM timing.

The demand is identical in both halves — issue is high for all ten cycles. The first half accepts four of four; the second accepts one of six.

Nothing changed except which bank the addresses landed on, and that is a property of the decode, not of the media. A memory expander's bandwidth is mostly an address-mapping decision, which is why §6 comes before §7 and why the interleave granularity is one of the few genuinely consequential knobs on this class of device.

9. RTL 3 — Advertised Capacity Must Be Backed

  assign in_advertised = (acc_addr < ADVERTISED[7:0]);
  assign backed        = (acc_addr < PHYSICAL[7:0]);
  assign serve         = acc_valid && in_advertised && (SERVE_GAP || backed);

=== EXP3: advertised capacity must be backed by media ===
  addr 10 (backed)    : advertised=1 backed=1 serve=1
  an address inside real media is served                 : ok
  addr 80, honest device : advertised=0 serve=0
  an honest device never advertised it                   : ok
  addr 80, over-advertised : advertised=1 backed=0 serve=0 | serve-gap serve=1
  the over-advertising device refused what it cannot back : ok
  the serve-gap variant answered for absent media        : ok
  unbacked accesses=1 | serve-gap phantom flag=1
  the serve-gap variant was caught inventing memory      : ok

This is a Type 3's version of 10.1's negative obligation, and it is sharper here because a memory device's entire purpose is to have the memory it claims.

Three states, not two. An address can be outside the advertised range, inside the advertised range and backed by media, or inside the advertised range and not backed. The third only exists if the device advertised more than it has, and it is the interesting one: the host will use that memory, because the device said it was there.

The serve-gap variant is the dangerous behaviour — it answers, with whatever its buffers held. A refusal is recoverable and diagnosable; a fabricated response corrupts something far away and is attributed elsewhere. Note that even the over-advertising device does the right thing at access time by refusing; the defect is in what it advertised, and the counter n_unbacked is what makes that visible before anything corrupts.

10. RTL 4 — It Answers, and Never Asks

The structural fact from which everything in §4 follows.

  assign out_req = ORIGINATES && want_own_access;
  ...
      // The buffer is transit storage: it must be released at completion.
      // Holding data past completion turns it into an untracked cached copy.
      if (in_complete) buf_holds_q <= BUFFER_AS_CACHE ? 1'b1 : 1'b0;

=== EXP4: a Type 3 answers and never asks ===
  internal agent wants a line : correct out_req=0 | originating variant out_req=1
  the Type 3 device originated nothing                   : ok
  and was caught originating a coherent request          : ok
  request staged : buffer holds=1
  completed : correct buffer=0 | buffer-as-cache buffer=1 stale flag=1
  the buffer released at completion                      : ok
  the buffer-as-cache variant kept an untracked copy     : ok

A Type 3 has storage that is not a cache, and the distinction is one of the easiest to lose. A staging buffer holds data in transit and is released when the transaction completes. A cache holds a copy for reuse — and a copy held for reuse in a device with no coherence agent is a copy nobody is tracking.

The BUFFER_AS_CACHE variant is a natural optimisation: the data is already there, why throw it away? Because the host has no idea the device still holds it, so a subsequent write goes unnoticed and the retained copy is silently stale. Turning a buffer into a cache turns a Type 3 into a broken Type 1.

The ORIGINATES variant is the same boundary from the other side. A Type 3 that issues its own coherent request is a device the host is not tracking as a caching device, participating in coherence anyway.

11. RTL 5 — Report, Never Return

  // Corrected data is still good data. Uncorrected data is not data at all.
  assign data_valid   = read_valid && (SILENT_UE || !uncorrectable);
  assign report_error = read_valid && uncorrectable && !SILENT_UE;

=== EXP5: a memory device must never return data it could not reconstruct ===
  5 correctable reads : CEs=5 degraded=1 data_valid seen=0
  the device flagged itself degraded past the threshold  : ok
  uncorrectable read  : correct data_valid=0 report=1 | silent variant data_valid=1 report=0
  the correct device reported instead of returning data  : ok
  the silent variant returned the corrupt data           : ok
  the silent variant was caught returning unreconstructable data : ok

A memory device is the last line of defence for its own media, and it is the only party that knows whether the bits it read were reconstructable. Nothing downstream can tell.

The two error classes have opposite handling and the distinction is the whole design:

The degraded flag is the useful one operationally. Five correctable errors crossed the threshold and the device declared itself degraded — before any uncorrectable error occurred. A rising correctable rate predicts an uncorrectable one, which makes it the difference between scheduled replacement and an outage.

The silent variant returns the corrupt data with report_error low. No party downstream can distinguish that from a successful read, which puts it in the same family as 9.5's silent write failure: the device is the only one who could have said, and it did not.

Rows three and five are the distinction the whole error path exists to make. Corrected bits are data; unreconstructable bits are not, and only the device can tell them apart — which is why returning the second as though it were the first is the one failure nothing downstream can catch.

12. RTL 6 — Why This Class Scales

  assign admit_caching = attach_caching &&
                         ((track_used_q + CACHE_TRACK_COST[15:0]) <= TRACK_BUDGET[15:0]);
  // Memory devices are bounded by decode capacity, not by tracking state.
  assign admit_memory  = attach_memory && (n_memory_q < ADDR_SLOTS[7:0]) && ...

=== EXP6: why memory devices scale and caching devices do not ===
  6 caching attach attempts : admitted=4 tracking used=32 refused=2
  only four caching devices fitted the tracking budget   : ok
  they consumed the entire budget of 32                  : ok
  10 memory attach attempts : admitted=8 tracking used=32
  exactly eight memory devices fitted the decode capacity : ok
  and they added nothing at all to the tracking budget   : ok
  charge-memory variant : caching=4 memory=0
  no-budget-check abuse : caching=14 tracking used=112 budget=32 flag=1

Four caching devices exhausted the tracking budget; eight memory devices added nothing to it. That is the whole scalability argument in two lines.

The reason is 10.1's asymmetry read backwards. A caching device holds copies of host memory, so the host must track which device holds what — per-device state, and published material puts CXL 3.0's caching fanout at up to 16 devices. A memory device holds no copies of anything, so there is nothing to track. Its bound is address space and decode capacity, which are much larger and much cheaper to extend.

This is why memory pooling is a Type 3 story. CXL 2.0 introduced pooling and CXL 3.0 added sharing, and both are practical for the device class that costs the host nothing per device.

13. RTL 7 — Even Use, and Media Health

=== EXP7: is the media being used evenly, and is it healthy? ===
  requests=40 served=34 blocked=6 channels=8/9/9/8 imbalance=2
  requests, services and blocks matched an independent oracle : ok
  after 12 accesses all to channel 0 : ch=20/9/9/8 imbalance=12 (oracle 12)
  the imbalance matched an independently computed spread : ok
  a skewed stream produced a large, visible imbalance    : ok

Imbalance is this chapter's signature counter, and it is the one a Type 1 or Type 2 has no equivalent of. A striding stream produced a spread of 2 across four channels; a skewed stream produced 12.

That number distinguishes two failures that look identical from outside — the device is slow because the media is slow, versus the device is slow because three of its four channels are idle. The second is an address-mapping problem with a software or configuration fix; the first is not. Without the counter, both present as low bandwidth with blocked accesses.

The mutation that pinned imbalance to zero survived the first run, because the original assertion was a bound — imbalance stays small — and a permanently-zero counter satisfies it. Asserting a bound on a counter that is supposed to move cannot detect a counter that never moves.

14. Quantitative Reasoning

Illustrative, with assumptions stated.

What interleaving is worth

From §8: identical demand, 4 accepted spread across banks versus 1 accepted on a single bank over six cycles.

banks busy, spread   : 4 concurrent -> 4 accesses per BUSY_TIME
banks busy, one bank : 1 concurrent -> 1 access  per BUSY_TIME

A 4× throughput difference from address mapping alone, with the same media, the same link and the same demand. Generalising to C channels, a sequential stream achieves C× the single-channel bandwidth with interleaving and 1× without — which makes the decode the highest-leverage design decision in the device.

Choosing the granularity

The granularity trades two effects:

The useful rule: the granularity should be at least one access wide and no wider than the shortest run you expect to see. Below the first, single accesses split across channels; above the second, sequential streams stop spreading. Both failures show up as low bandwidth with high imbalance.

The cost of over-advertising

A device advertising 96 lines with 64 present has 32 lines of phantom capacity — 33% of its advertised memory does not exist. If the device refuses those accesses, the system sees errors on a third of its address range. If it answers them (§9's variant), it silently corrupts a third of its address range.

Neither is a graceful degradation, which is why advertised capacity is a hard contract rather than a hint, and why n_unbacked belongs in silicon telemetry rather than only in bring-up.

Scaling contrast

Using §12's teaching units: a tracking budget of 32 at 8 units per caching device admits 4 caching devices. The same budget admits as many memory devices as decode capacity allows — 8 in the model, and the bound moves with address space rather than with host storage.

caching devices : bounded by host tracking state
memory devices  : bounded by address space and decode

Those two bounds grow differently, which is the structural reason capacity expansion and pooling are Type 3 stories and coherent acceleration is not.

15. Assertions

Icarus Verilog 13.0 does not support concurrent SVA here, so every property is synthesisable checker logic verified in simulation. 50 assertions.

Safety

Liveness

Performance goals

16. Mutation Testing

Twenty-six mutations. Twenty-six killed.

The first run scored 21 of 26 — the largest number of escapes in this batch — and they sort cleanly into three lessons already established in earlier chapters, arriving in new disguises.

Two were assertions that were true and useless. The channel distribution was asserted as even, and a decode selecting the wrong address bits is still perfectly even — it merely sends the wrong addresses to each channel. The imbalance counter was asserted with a bound, and a counter pinned to zero satisfies any upper bound. Both were fixed by asserting exact values: the specific channel and bank address for specific addresses, and the imbalance against an independently computed spread.

Two were untested capacity paths — offering more memory devices than decode slots, and the per-bank address never being checked at all.

One needed an abuse instance. The tracking-budget checker is unreachable on a design that honours its budget, so a NO_BUDGET_CHK instance was added that admits past its own limit, exceeding a declared budget of 32 with 112 units and tripping the checker. Note the checker is deliberately written without a guard on that parameter — the self-disabling-checker defect from 9.1 and 9.6, whose mutation re-adds the guard and is killed by exactly that instance.

The recurring pattern across this batch is now unambiguous. Nearly every mutation escape has been a stimulus or assertion-precision problem rather than a missing check. The instinct on an escape is to add an assertion; the correct move is usually to ask what state was never reached, or what value was asserted as a property when it should have been asserted exactly.

17. Verification Plan

The coverage cross is access pattern × channel distribution × media state: sequential / strided / hot-spot, crossed with balanced / skewed, crossed with idle / busy / degraded. The hot-spot × skewed × busy point is where the decode, the bank concurrency and the imbalance counter all interact, and it is the operating point a badly-mapped workload actually occupies.

18. Silicon Observability

The imbalance counter is the one to fight for. It separates two failures that are indistinguishable from outside: slow media, and a decode that is using a quarter of the media it has. One needs different hardware; the other needs a configuration change, and without the counter a team will buy the first to fix the second.

The degraded flag is the one operations will care about most, because it is predictive rather than diagnostic. Everything else on the list explains a failure that already happened.

19. Debug Lab

assign channel = acc_addr[11:10];   // interleave on the high bits

  no-interleave variant   : ch0=16 ch1=0 ch2=0 ch3=0
  the no-interleave variant sent everything to one channel : ok

wire [1:0] low_sel = acc_addr[GRAN_LOG2+1 -: 2];
assign channel = low_sel;

assign bank_addr = acc_addr[9:0];   // channel bits left in

  exact decode : addr0->ch0/bank0 addr4->ch1/bank0 addr16->ch0/bank4
  two different channels reuse bank address 0            : ok
  the wrapped granule advanced the bank address to 4     : ok

assign bank_addr = {acc_addr[11:GRAN_LOG2+2], acc_addr[GRAN_LOG2-1:0]};

wire [1:0] low_sel = acc_addr[GRAN_LOG2+2 -: 2];   // off by one bit

  16 consecutive granules : ch0=4 ch1=4 ch2=4 ch3=4
  exact decode : addr0->ch0/bank0 addr4->ch1/bank0 addr8->ch2
  consecutive granules landed on channels 0, 1, 2        : ok

wire [1:0] low_sel = acc_addr[GRAN_LOG2+1 -: 2];

localparam ADVERTISED = 96;   // PHYSICAL is 64

  addr 80, over-advertised : advertised=1 backed=0 serve=0 | serve-gap serve=1
  the serve-gap variant was caught inventing memory      : ok

if (serve && !backed) phantom_serve_err <= 1'b1;

// data's already here, keep it for the next access
if (in_complete) buf_holds_q <= 1'b1;

  completed : correct buffer=0 | buffer-as-cache buffer=1 stale flag=1
  the buffer released at completion                      : ok
  the buffer-as-cache variant kept an untracked copy     : ok

if (in_complete) buf_holds_q <= 1'b0;

assign data_valid = read_valid;   // always return something

  uncorrectable read : correct data_valid=0 report=1 | silent variant data_valid=1 report=0
  the silent variant was caught returning unreconstructable data : ok

assign data_valid   = read_valid && !uncorrectable;
assign report_error = read_valid && uncorrectable;

imbalance_q <= 16'd0;   // reduction dropped in a refactor

  after 12 accesses all to channel 0 : ch=20/9/9/8 imbalance=12 (oracle 12)
  the imbalance matched an independently computed spread : ok

chk(t_im == orc_spread[15:0], "imbalance matched an independently computed spread");

assign accept = issue;   // media is fast, just send it

  four accesses to ONE bank : issued=5 blocked=3 | ignore-busy issued=8 overlap=1
  the ignore-busy variant overlapped two media accesses  : ok

wire bank_free = (timer_q[bank] == 8'd0);
assign accept  = issue && bank_free;
if (accept && !bank_free) overlap_err <= 1'b1;

20. Design Review

1. Which address bits select the channel, and are they removed from the bank address?
2. What is the interleave granularity, and against which access pattern was it chosen?
3. Does a sequential stream reach every channel?
4. Does advertised capacity come from a constant or from the populated media?
5. What happens to an access inside the advertised range with no media behind it?
6. Can the device originate a coherent request under any condition?
7. Is staging storage released at completion, and what asserts it?
8. What happens on an uncorrectable read — report, or return?
9. What is the correctable error rate, and what threshold escalates it?
10. How many of these devices can one host take, and what is the actual bound?
11. Does any counter distinguish slow media from a skewed decode?

21. How This Appears in Real Engineering

Architecture. The decode is the highest-leverage decision — 4× bandwidth in the model, from address mapping alone.

RTL. Small and disciplined: decode, bank timers, capacity check, error path. The temptation to retain the staging buffer is where a Type 3 design goes wrong.

DV. Sequential streams and repeat-address tests, both of which a random memory sweep omits. Every range check needs its boundaries.

Performance. Imbalance is the counter that separates a media problem from a mapping problem, and the two have entirely different fixes.

Firmware. Advertised capacity must be derived from populated media, and the degraded flag is the interface to predictive replacement.

Post-silicon. The correctable error rate is the only leading indicator in this chapter; everything else explains a failure that already happened.

22. Common Misconceptions

23. Interview Reasoning

24. Exercises

1. Calculation. A device has 8 channels, a 256-byte interleave granularity and a 64-byte access size. Compute how many consecutive accesses are needed to touch every channel, and the achieved bandwidth of a sequential stream as a fraction of peak. Then recompute both for a 4 KB granularity and state which workload each choice favours.

2. Analysis. A device reports even per-channel counts, an imbalance of zero, and half of expected sequential bandwidth. Explain why the first two facts do not exonerate the decode, and give the single directed test that settles it.

3. RTL task. Extend interleave_decode to support a configurable channel count that is not a power of two. State what breaks in the bit-slicing approach, what the bank address must become, and the new invariant that must be asserted.

4. Assertion task. Write the property that proves the decode is a bijection over the advertised range. Explain why a per-address check is insufficient on its own, and what additional structure the testbench needs.

5. Waveform analysis. Using §8's trace, compute the accept rate in each half and the effective bank concurrency. Then state what the second half would look like with an 8-cycle bank busy time, and which counter would reveal the difference first.

6. Debug task. A memory expander returns correct data for the first two thirds of its advertised range and aliased data above that. Give your investigation order, the two defects in this chapter that both produce that symptom, and the single measurement that distinguishes them.

7. Design review. A colleague proposes retaining the staging buffer across transactions as a small read cache, arguing it is a few lines of storage and cannot affect correctness because the device never modifies data. Give the strongest version of that argument, then state exactly what breaks and what the device would have to become for the optimisation to be legal.

25. Summary

A Type 3 device only answers.

• It exposes memory and never originates a coherent request, which removes — in one chain — the cache, the coherence agent, the permission check, the second identity space, the handover and the cycle between directions.
• Interleaving is worth 4× on sequential access in the model: same media, same link, same demand, spread across banks versus queued on one.
• The channel bits must be removed from the bank address, or every channel reaches a quarter of its own media while the device advertises all of it.
• Advertised capacity is a hard contract. An address inside the advertised range with no media behind it is either an error across a contiguous region or silent corruption of one.
• A buffer is not a cache. With no coherence agent, staging data retained past completion is a copy nobody is tracking.
• Report, never return. Correctable errors are good data to be counted; uncorrectable ones are not data at all, and the device is the only party that can tell. The degraded flag is the chapter's only leading indicator.
• This class scales because it costs the host nothing per device — four caching devices exhausted the tracking budget while eight memory devices added nothing to it. Published fanout limits bound caching devices.
• Imbalance separates a media problem from a mapping problem, and nothing else does.
• Verification: 26 of 26 mutations killed, 50 assertions. Five first-run escapes, and the lesson is now unmistakable across this batch: assertion precision and unreached states, not missing checks. An even distribution proved nothing about a decode, and a bound proved nothing about a counter pinned to zero.

Next: 10.4 — Device-Type Selection Discipline, which closes Module 10 with the question these three chapters have been setting up — not which type is best, but what a device becomes when part of it is turned off, and how to keep the answer from surprising you.