AMBA CHI · Module 20 · Real Industry Case Studies

Cache-Coherent DMA

A DMA engine needs coherence without a cache, and cache-coherent DMA is that case. An I/O device accesses the memory the CPUs cache, but holds no cache itself. On CHI it is an I/O-coherent request node — an RN-I, or an RN-D if it needs distributed-virtual-memory support — issuing coherent reads and writes to the fully-coherent home. The home snoops the CPUs on its behalf, so the device sees their latest data and its writes invalidate stale copies. But because it holds no cache, the CPUs never snoop it: coherence is one-way. Software no longer flushes caches around every DMA transfer. The pitfall is conflating an I/O-coherent request node with a fully-coherent one. Representative case study, not a product specification.

Intermediate13 min readAMBA CHIDMARN-II/O CoherentCache Maintenance

Module 20 · Chapter 20.4 · Real Industry Case Studies

Project thread — 20.3 was fully-coherent CPU clusters. 20.4 is a device coherent without a cache; 20.5 is composable subsystems.

1. The Domain

Cache-coherent DMA — also called I/O-coherent DMA — is a device or DMA engine that accesses the same memory the CPUs cache, staying coherent with them, without holding a cache of its own. It is one of the most common real CHI configurations: every accelerator, network controller, or storage engine that moves data in and out of coherent memory does it this way. The distinctive property is one-way coherence, and the payoff is the end of software cache flushing around DMA.

2. Why CHI Here

A DMA engine uses CHI's I/O-coherent request nodes because it needs to participate in coherence without the cost of being a full cache. CHI defines request-node types (Chapter 5): a fully-coherent node (RN-F) holds a cache and answers snoops; an I/O-coherent node (RN-I, or RN-D with DVM support) issues coherent requests but holds no cache and is never snooped. A DMA engine is exactly the second kind — it wants its reads to see CPU-cached data and its writes to invalidate CPU copies, but it has no cache for anyone to snoop. CHI's typed request nodes make this precise, which is why the protocol supports coherent I/O natively.

3. Key Terms

4. Previous Chapter Connection

This chapter rests on the node-type taxonomy (Chapter 5) — RN-F vs RN-I/RN-D, HN-F vs HN-I — and on the idea that the home snoops on a requester's behalf (Chapters 8, 15.6). The coherent transactions the device issues (ReadOnce, WriteUnique-style, Chapter 7) are the ones that let the home fetch CPU-modified data or push through to memory. DVM (Chapter 12) is why an RN-D exists.

Where the CPU cluster (20.3) was all RN-F (every node caches and is snooped), the DMA engine is the complement: an RN-I/RN-D that caches nothing. Putting them together — RN-F CPUs and an RN-I DMA on the same coherent home — is the canonical coherent-I/O picture, and it is where the difference between the node types stops being abstract and becomes a concrete architectural choice with a software payoff.

5. The CHI Configuration

The cache-coherent DMA configuration:

The DMA engine is an I/O-coherent request node. It is an RN-I — or an RN-D if it needs DVM to walk page tables — that issues coherent reads and writes but holds no cache.
Coherent data path → HN-F. The device's data transfers go to the fully-coherent home (HN-F), which snoops the CPUs (RN-F) on the device's behalf, so the device sees their latest data and its writes invalidate stale copies.
Config path → HN-I. The device's MMIO / configuration registers are reached through a non-coherent I/O home (HN-I) — a separate, non-coherent path.
No device cache to snoop. Because the device holds no cache, the CPUs never snoop it — coherence is one-way.

The synthesis:

Cache-coherent DMA models the engine as an I/O-coherent request node (RN-I, or RN-D for DVM) that issues coherent transactions to the fully-coherent home (HN-F) — which snoops the CPUs for it — while holding no cache of its own, so the CPUs never snoop the device. Config goes via HN-I. Coherence is one-way, and software cache maintenance disappears.

6. The Domain Constraints

What shapes a cache-coherent DMA design.

Coherence needed, but no cache wanted. The device must see CPU-modified data and make its writes visible — so it needs coherence — but a DMA engine streams data and has no reuse, so a cache would be wasted area and add snoop burden. Hence I/O-coherent, not fully coherent.
One-way is sufficient. The device is coherent with the CPUs (reads current data, writes invalidate them), but nothing the device holds needs protecting, so the CPUs need not snoop it — one-way coherence is exactly enough.
Kill software cache maintenance. Non-coherent DMA forces software to flush caches before a device read and invalidate after a device write — slow and bug-prone. I/O-coherent DMA makes the hardware handle it — no flushes.
Config is separate. MMIO/config is non-coherent and goes via HN-I — don't route it through the coherent home.

The point to carry:

The DMA engine wants coherence without a cache, and one-way coherence gives exactly that: it participates in coherence as a requester (RN-I/RN-D → HN-F, home snoops the CPUs) but not as a snoop target (no cache to snoop). The reward is the elimination of software cache maintenance — the single biggest reason to choose I/O-coherent over non-coherent DMA. The node type (RN-I vs RN-F) is the design decision.

7. Engineering Diagram — the cache-coherent DMA system

Figure 1 — cache-coherent (I/O-coherent) DMA on CHI. The DMA engine is an I/O-coherent request node (RN-I, or RN-D with DVM) that issues coherent reads and writes to the fully-coherent home, which snoops the CPUs on its behalf. The device holds no cache, so the CPUs never snoop it — coherence is one-way. The device's configuration registers reach a separate non-coherent I/O home. Data path is coherent; config path is not.

The DMA engine (RN-I/RN-D) and the CPUs (RN-F) share the coherent home (HN-F), which snoops the CPUs on the device's behalf. No arrow snoops the device — it holds no cache, so coherence is one-way. Config goes to the separate HN-I. Data coherent, config not.

8. The Distinctive Design Decisions

What cache-coherent DMA does — and why.

Decision	DMA choice	Why
Request-node type	RN-I (or RN-D for DVM)	coherent requester, no cache
Cache	none	streaming, no reuse; avoid snoop burden
Coherence direction	one-way (device→CPUs)	nothing on the device to snoop
Data path	coherent, to HN-F	home snoops CPUs for the device
Config path	non-coherent, to HN-I	MMIO is not coherent
Software flushes	eliminated	hardware keeps DMA coherent

The rule to carry: the node-type choice (RN-I/RN-D, not RN-F) is the whole design, and one-way coherence is its consequence. Choosing an I/O-coherent node gives coherent access without a cache, which means the device is a coherent requester but never a snoop target — and that is precisely what lets software drop cache maintenance around DMA. Everything else (HN-F for data, HN-I for config, DVM only if needed) follows.

9. The Trade-offs

The tensions a cache-coherent DMA design balances.

I/O-coherent vs non-coherent. I/O-coherent DMA costs a snoop on the home per coherent access (the home checks the CPUs) but eliminates software flushes; non-coherent DMA has no snoops but forces software cache maintenance — slow and error-prone. For anything sharing live data with CPUs, I/O-coherent wins.
No cache vs a cache. No device cache saves area and snoop burden and fits streaming; a cache would only help a device with reuse — which a DMA engine does not have. So no cache (RN-I), not RN-F.
RN-D vs RN-I. RN-D adds DVM support (TLB/page-table maintenance) — needed only if the device does its own virtual-address translation; otherwise plain RN-I suffices.
Coherent data vs non-coherent config. Keep the data path coherent (HN-F) and the config path non-coherent (HN-I) — mixing them wastes coherence on registers or loses it on data.

The point to carry:

The central trade is I/O-coherent vs non-coherent DMA: a snoop per access on the home in exchange for no software cache maintenance. For a device that shares live data with the CPUs, that trade is overwhelmingly worth it — which is why cache-coherent DMA is the default for modern accelerators and I/O. The no-cache (RN-I) and one-way properties are what make it cheap enough to be the default.

10. A Concrete Walkthrough

A DMA write into memory the CPUs are caching — the scenario that shows one-way coherence.

CPU holds a dirty line. A CPU (RN-F) has a buffer line Modified in its cache — not yet in memory.
DMA writes the buffer. The DMA engine (RN-I) issues a coherent write (WriteUnique-style) for that address to the coherent home (HN-F).
Home snoops the CPU. The HN-F snoops the CPU (Chapter 15.6), invalidating its stale copy — so the CPU will not later write old data over the DMA's.
Write completes coherently. The DMA's data lands, the CPU's copy is gone; a later CPU read misses and gets the new data. No software flush was needed.
Reverse — DMA reads CPU data. A DMA read likewise makes the home snoop the CPU to source its Modified data — the device sees the latest value.

At no point did anyone snoop the DMA engine — it holds no cache. The home did all the coherence work for it. That is one-way coherence, and it is why software never flushed a cache. The pitfall is modelling this device as an RN-F (which would need cache state and snoop responses it does not have) — see section 14.

11. Representative Configuration

A cache-coherent DMA CHI configuration. Representative.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

// Representative cache-coherent (I/O-coherent) DMA configuration (educational, not a spec).
// The DMA engine is an I/O-coherent REQUEST NODE -- coherent, but holds NO cache.
localparam bit  IO_COHERENT      = 1;      // coherent with CPUs (vs non-coherent DMA)
localparam bit  HAS_CACHE        = 0;      // NO device cache -> RN-I/RN-D, NOT RN-F
localparam bit  NEEDS_DVM        = 0;      // 1 => RN-D (TLB/page-walk); 0 => plain RN-I
// Node type follows from the two flags above:
//   IO_COHERENT & !HAS_CACHE & !NEEDS_DVM -> RN-I
//   IO_COHERENT & !HAS_CACHE &  NEEDS_DVM -> RN-D
//   HAS_CACHE (a real cache, snooped)     -> RN-F   (NOT this device)
 
// Coherent DATA path: coherent transactions to the fully-coherent home.
localparam int  COHERENT_HOME    = 1;      // HN-F: snoops the CPUs (RN-F) for the device
// Non-coherent CONFIG path: MMIO/registers to the I/O home.
localparam int  CONFIG_HOME      = 1;      // HN-I: non-coherent MMIO, separate path
 
// ONE-WAY coherence: device is coherent WITH the CPUs (home snoops them for it),
// but the CPUs NEVER snoop the device (no cache) -> software needs NO flush/invalidate.

The flags make the design decision explicit: I/O-coherent + no cache yields RN-I (or RN-D with DVM), never RN-F. The data path is coherent (HN-F), the config path is not (HN-I). The comment states the one-way property that eliminates software cache maintenance.

12. Analysis View — cost and benefit of I/O coherence

What I/O-coherent DMA costs and saves.

Cost: a snoop per coherent access. Each coherent DMA read/write makes the home snoop the CPUs (Chapter 15.6) — a small latency/traffic cost on the home, bounded because snoops are targeted.
Saving: no software cache maintenance. The flush-before / invalidate-after software cost of non-coherent DMA — often many cache-line operations per transfer — disappears. This is usually a large net win.
No device-cache overhead. With no device cache, there is no area for it and no snoop traffic to it — the device is a light coherent participant.
Config stays cheap. MMIO via HN-I avoids putting register traffic on the coherent home.

The point to carry:

The analysis is a trade: a targeted snoop per coherent access (small, on the home) buys the elimination of software cache maintenance (often large, and bug-prone). For a device sharing live data with CPUs, the trade is strongly favourable — which is why I/O-coherent DMA is the default. The one-way, no-cache structure keeps the device's coherence footprint minimal.

13. Design Considerations

The deeper checklist for a cache-coherent DMA design.

Choose the node type from the requirements — I/O-coherent + no cache ⇒ RN-I; add DVM ⇒ RN-D; never RN-F for a cacheless DMA.
Route the data path to HN-F — coherent transactions, home snoops the CPUs.
Route config to HN-I — non-coherent MMIO, separate path.
Rely on one-way coherence — the device is a requester, never a snoop target.
Drop software cache maintenance — the hardware keeps DMA coherent.
Add DVM only if the device translates its own virtual addresses.

14. The Pitfall

Pitfall: conflating an I/O-coherent request node (RN-I/RN-D) with a fully-coherent one (RN-F)

The classic cache-coherent-DMA mistake is conflating RN-I/RN-D with RN-F — treating the DMA engine as a fully-coherent node. Two failure shapes follow. First, modelling the cacheless DMA as an RN-F: an RN-F is expected to hold cache state and answer snoops, so the design adds snoop-response logic and cache-state tracking the device does not have and does not need — wasted area and complexity, and a functional mismatch (the "cache" is empty, so its snoop responses are meaningless). Second, assuming coherence is two-way: expecting the CPUs to snoop the device. But the device holds no cache — there is nothing to snoop. Coherence is one-way: the device is coherent with the CPUs (the home snoops them for it), not the reverse. Wire up CPU-snoops-device paths and you have built logic that can never do anything useful, and possibly a deadlock waiting for a snoop response from a node with no cache. The inverse error is just as bad — treating an actually-caching device (RN-F) as an RN-I, which loses coherence on the data it caches.

The fix: derive the node type from the device's real behaviour. Coherent + no cache ⇒ RN-I (or RN-D if it needs DVM); the coherence is one-way (home snoops the CPUs, never the device). Only a device that genuinely holds a cache with reuse is an RN-F.

15. Common Mistakes

DMA modelled as RN-F. Assumption: coherent means fully-coherent. Bug: needless cache/snoop logic (the Pitfall). Fix: RN-I/RN-D.
Expecting two-way coherence. Assumption: CPUs snoop the device. Bug: dead paths / deadlock. Fix: one-way — nothing to snoop.
Config on the coherent home. Assumption: one home for all. Bug: register traffic on HN-F. Fix: MMIO via HN-I.
Keeping software flushes. Assumption: still flush around DMA. Bug: wasted, error-prone maintenance. Fix: hardware keeps it coherent.
RN-D when RN-I suffices. Assumption: always add DVM. Bug: needless DVM logic. Fix: RN-D only if the device translates.
Caching device as RN-I. Assumption: I/O node is fine. Bug: lost coherence on cached data. Fix: RN-F if it really caches.

16. Engineering Checklist

Model a coherent, cacheless DMA as an RN-I (or RN-D for DVM), not RN-F.
Route the coherent data path to the fully-coherent home (HN-F).
Route MMIO / config to the non-coherent I/O home (HN-I).
Rely on one-way coherence — the home snoops the CPUs, never the device.
Remove software cache maintenance around DMA — hardware keeps it coherent.
Add DVM (RN-D) only if the device does its own address translation.

17. Key Takeaways

Cache-coherent (I/O-coherent) DMA is coherent with the CPUs without holding a cache.
The engine is an RN-I — or RN-D with DVM — issuing coherent requests to HN-F.
Coherence is one-way: the home snoops the CPUs for the device; nobody snoops the device.
It eliminates software cache maintenance around DMA.
Config goes via a separate non-coherent home (HN-I).
The pitfall is conflating RN-I/RN-D with RN-F; the model here is representative.

18. Quick Revision

Cache-coherent DMA. A device or DMA engine that accesses the same memory the CPUs cache, staying coherent with them, without holding a cache of its own. On CHI it is an I/O-coherent request node — an RN-I, or an RN-D if it also needs DVM (TLB/page-table maintenance) — that issues coherent reads and writes to the fully-coherent home (HN-F). Because the requests are coherent, the home snoops the CPUs (RN-F) on the device's behalf, so the device always sees the CPUs' latest data and its writes invalidate their stale copies. Because the device holds no cache, the CPUs never snoop it — coherence is one-way (device coherent with the CPUs, not the reverse). The device's MMIO / config registers go through a separate, non-coherent I/O home (HN-I). The payoff is the elimination of software cache maintenance — no flush-before / invalidate-after around every DMA transfer — at the cost of a targeted snoop per coherent access on the home, a trade that is strongly favourable for any device sharing live data with CPUs, which is why it is the default. The pitfall: conflating RN-I/RN-D with RN-F — giving the cacheless DMA needless cache-state and snoop-response logic, or assuming two-way coherence (the CPUs snooping a device with nothing to snoop, risking dead paths or deadlock). Representative case study; 20.5 covers composable multi-core subsystems.

Coming Next

Chapter 20.5 — Multi-Core Subsystems. Cache-coherent DMA was one device on a coherent home; the next case composes coherent subsystems. Chapter 20.5 covers multi-core subsystems — how coherent subsystem IP is built to compose (hierarchical coherence, coherence boundaries between subsystems, and reusable coherent building blocks), and the boundary-leak pitfall that composition introduces.