AMBA CHI · Module 20 · Real Industry Case Studies

AI Accelerators

AI accelerators push CHI toward bandwidth. A neural accelerator moves enormous data and shares it with the CPUs, so CHI lets it join as a coherent request node, reading and writing shared data without any DMA copy or flush. That zero-copy sharing is the value: a non-coherent design pays a memory copy plus cache-maintenance on every hand-off. The design is dominated by bandwidth — the accelerator saturates the data channel, so it must be wide and keep many requests outstanding — and directory coverage for the large working set. The trade-off is coherence overhead against copy elimination, a win when sharing is frequent. The traps: under-provisioning data-channel bandwidth, and under-sizing the directory. Representative case study, not a product specification.

Advanced14 min readAMBA CHIAI AcceleratorBandwidthCoherent SharingZero-Copy

Module 20 · Chapter 20.2 · Real Industry Case Studies

Project thread — 20.1 was mobile (power). 20.2 is AI accelerators (bandwidth); 20.3 is CPU clusters.

1. The Domain

An AI accelerator — a neural-processing unit (NPU), tensor engine, or ML accelerator — is a specialized datapath that performs the massive matrix and convolution math of neural networks. It moves enormous amounts of data — weights, activations, intermediate tensors — and, increasingly, it shares that data with the CPUs of the same system rather than operating on a private, isolated memory. The domain's defining feature is bandwidth: the accelerator is a data-movement machine, and keeping its datapath fed is the whole game.

2. Why CHI Here

AI accelerators use CHI so they can share coherent memory with the CPUs — a zero-copy hand-off. A vision pipeline where a CPU pre-processes an image and an NPU runs inference on it, or an LLM where CPU and accelerator cooperate on a KV-cache, involves constant data exchange. Doing that non-coherently means copying data through memory and issuing software cache-maintenance on every hand-off — pure overhead. As a coherent request node on CHI, the accelerator reads CPU-produced data (even if still dirty in a CPU cache) and writes accelerator results the CPU can immediately consume — without copies or flushes. CHI turns a copy-heavy hand-off into a coherent share.

3. Key Terms

4. Previous Chapter Connection

This chapter applies the throughput levers of Chapter 14.6 (outstanding requests, bandwidth-delay product), the channel-bottleneck analysis of Chapter 15.2 (the data channel is the limit), and the directory-efficiency material of Chapter 15.5 (coverage) to a bandwidth-hungry accelerator. The accelerator is a coherent request node (Chapter 5), and its zero-copy sharing rests on the cache-coherence guarantees of Modules 9–13.

Where mobile (Chapter 20.1) optimized for energy, AI accelerators optimize for bandwidth — the opposite extreme, using the same protocol. The technical concepts you've built — Little's Law (outstanding = throughput × latency), the bottleneck channel, directory coverage — become the dominant design considerations here, because an accelerator that can't be fed is useless. This case study is where CHI's throughput and coherence features are pushed together: maximum bandwidth and zero-copy sharing at once.

5. The CHI Configuration

The typical AI-accelerator CHI setup — bandwidth-first, coherent:

Accelerator as a coherent RN. The NPU joins the coherence domain as a request node — RN-F (with its own cache) if it benefits from local caching, or an uncached coherent agent — so it shares the CPUs' coherent memory space.
Wide data path. The DAT channel is wide (many bytes per cycle) and the accelerator keeps many requests outstanding — because the accelerator's demand is data, and the data channel is the bottleneck (Chapter 15.2).
CPUs as coherent peers. The CPU cores are RN-F; the accelerator and CPUs share data coherently — no copies.
High-bandwidth memory (SN) + directory coverage. Multiple memory controllers feed the bandwidth, and the directory is sized to cover the accelerator's large working set (Chapter 15.5).

The synthesis:

An AI accelerator joins CHI as a coherent request node, sharing the CPUs' memory space so data moves without copies. The configuration is bandwidth-first: a wide data channel, many outstanding requests (to cover latency, Chapter 14.6), multiple memory controllers, and a directory sized to cover the accelerator's large footprint (Chapter 15.5). It uses CHI's coherence for zero-copy sharing and CHI's throughput levers to stay fed.

6. The Domain Constraints

What shapes an AI-accelerator CHI design — bandwidth above all.

The data channel is the bottleneck. The accelerator's demand is overwhelmingly data (tensors), so the DAT channel saturates first (Chapter 15.2) — it must be wide, and widening the control channels buys nothing.
Latency must be hidden by concurrency. To keep a wide data channel full, the accelerator must keep many requests outstanding — enough to cover the bandwidth-delay product (Chapter 14.6). Too few, and the pipe runs dry.
The working set is large. Neural-network weights and activations are big, so the accelerator touches many lines — stressing directory coverage (Chapter 15.5). An under-covered directory degrades to broadcasts and back-invalidations, killing bandwidth.
Sharing is frequent and fine-grained. CPU-accelerator hand-offs happen often, so the coherence overhead is amortized by the copies avoided — a net win precisely because sharing is dense.

The point to carry:

AI accelerators are bandwidth-bound and coherence-heavy at once. Feeding the datapath requires the throughput levers (wide data channel, many outstanding, multiple memory controllers), and sharing with the CPUs requires the coherence machinery (the accelerator as a coherent RN, a directory that covers its footprint). A design that nails one and neglects the other fails: raw bandwidth with no coherence forces copies; coherence with insufficient bandwidth starves the accelerator. The distinctive AI challenge is doing both at scale.

7. Engineering Diagram — the coherent AI-accelerator system

Figure 1 — a representative AI-accelerator subsystem on CHI. The CPU cores and the neural accelerator are both coherent request nodes sharing one coherent memory space, so data moves between them without copies. A wide, high-bandwidth CHI fabric with a directory sized to cover the accelerator's large working set connects them to multiple memory controllers. The accelerator keeps many requests outstanding to keep the wide data channel full.

The NPU and CPUs share one coherent memory space (no copies), connected by a wide CHI fabric with a covered directory to multiple, high-bandwidth memory controllers. The accelerator keeps many requests outstanding to keep the wide data channel full. The picture is bandwidth-first — the opposite of mobile's compact fabric.

8. The Distinctive Design Decisions

What AI accelerators do differently — and why.

Decision	AI-accelerator choice	Why
Accelerator role	coherent RN (RN-F / uncached)	zero-copy sharing with CPUs
Data channel	wide, high-bandwidth	the accelerator's demand is data
Outstanding requests	many (fill the pipe)	cover the bandwidth-delay product
Memory	multiple high-BW controllers (HBM)	feed the datapath
Directory	sized to cover the large footprint	avoid broadcasts/back-invalidations
Sharing model	frequent, fine-grained, coherent	amortizes coherence over copies avoided

The rule to carry: every AI decision trades toward bandwidth and zero-copy sharing. A wide data channel, many outstanding, multiple memory controllers, and HBM all serve bandwidth; the accelerator-as-coherent-RN and the covered directory serve zero-copy sharing. The accelerator is a data-movement engine that shares coherently — so both the throughput levers and the coherence levers are turned up. CHI is the substrate that lets a bandwidth-hungry accelerator be a coherent peer of the CPUs.

9. The Trade-offs

The tensions an AI-accelerator CHI design balances.

Coherence overhead vs copy elimination. Making the accelerator coherent adds snoop/coherence traffic but eliminates copies and flushes. A win when sharing is frequent and fine-grained (as in fused CPU-NPU pipelines); a loss if the accelerator rarely shares (then non-coherent DMA is simpler — Chapter 20.4).
Local cache vs uncached. An accelerator cache (RN-F) reduces memory traffic for reused data but stresses the directory (more tracked lines). An uncached coherent agent is simpler but re-fetches. The choice depends on data reuse.
Bandwidth vs area/power. A wide data channel and many memory controllers cost area and power — provisioned for the accelerator's peak demand, unlike mobile's common-case sizing.
Directory coverage vs storage. Covering the accelerator's large footprint needs directory storage — under-cover and coherence degrades to broadcasts (Chapter 15.5).

The point to carry:

The defining AI trade-off is coherence overhead for copy elimination, and it is favorable exactly when the accelerator and CPUs share data densely — which modern fused AI pipelines do. The secondary trade-offs (cache vs uncached, bandwidth vs area) all serve the primary goal: a fed accelerator that shares coherently. Unlike mobile, area and power are spent freely to buy bandwidth — because a starved accelerator is worthless.

10. A Concrete Walkthrough

A fused CPU-preprocessing / NPU-inference pipeline — the scenario that shows the coherent-accelerator value.

CPU preprocesses. A CPU core normalizes an input tensor and writes it to memory — the data is in its cache (dirty) and the coherent space.
NPU reads it, coherently. The NPU (coherent RN) reads the tensor. The fabric snoops the CPU and delivers the latest data — including the parts still dirty in the CPU's cache. No copy, no flush.
NPU runs inference at bandwidth. The NPU streams weights and activations from HBM, keeping many requests outstanding to keep the wide data channel full — the datapath stays fed.
NPU writes results, CPU consumes. The NPU writes its output coherently; a CPU core reads it immediately — again no copy, no flush.
The directory keeps up. The NPU's large footprint is covered by the directory, so snoops stay targeted — no broadcast storm.

The pipeline ran with zero copies (coherent sharing) and a full data channel (bandwidth provisioning + outstanding requests) — the two AI-accelerator pillars. The pitfall is starving the datapath (too-narrow data channel or too-few outstanding) or overflowing the directory (footprint exceeds coverage).

11. Representative Configuration

A bandwidth-first, coherent AI-accelerator CHI configuration. Representative.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

// Representative AI-accelerator CHI configuration (educational, not a product spec).
// Bandwidth-first and coherence-heavy -- the opposite of mobile's compact/gated part.
localparam int  N_CPU_CORES     = 8;       // coherent CPU peers (RN-F)
localparam bit  ACCEL_IS_RN_F   = 1'b1;    // accelerator is a COHERENT request node
localparam bit  ACCEL_HAS_CACHE = 1'b1;    // RN-F with a cache (data reuse) vs uncached
localparam int  DAT_WIDTH_BYTES = 64;      // WIDE data channel -- the accelerator bottleneck
localparam int  MAX_OUTSTANDING = 256;     // MANY in-flight to cover the bandwidth-delay product
localparam int  N_MEM_CTRL      = 4;       // multiple high-BW controllers (HBM)
localparam int  DIR_ENTRIES     = 1 << 20; // directory sized to COVER the large footprint (15.5)
 
// Sizing leans to the PEAK (bandwidth) case, not the common case:
//   wide data path, deep outstanding window, many memory channels, large directory.
// The value: zero-copy CPU<->accelerator sharing (no DMA copies, no software flushes).
// Contrast mobile (20.1): compact, gated, sized for the common low-power case.

The configuration is wide and deep: a wide data channel, a deep outstanding window (Little's Law), multiple memory controllers, and a large directory to cover the footprint. Every knob is turned toward bandwidth and coverage — the opposite of mobile's compact, gated sizing, because the constraint (bandwidth vs battery) is opposite.

12. Analysis View — bandwidth and coverage

The two numbers that govern an AI-accelerator design.

Sustained bandwidth = min over channels of BW ÷ demand. For the accelerator, the data channel demand dominates, so the design must widen the data channel and provision memory bandwidth (Chapter 15.2) — the accelerator's throughput is DAT-bound.
Outstanding = throughput × latency (Little's Law). To sustain that bandwidth over the memory latency, the accelerator needs enough outstanding requests to cover the bandwidth-delay product (Chapter 14.6) — too few, and it's latency-bound at a fraction of the bandwidth.
Directory coverage ≥ footprint. The accelerator's large working set must be covered (directory entries ≥ cached lines, Chapter 15.5), or coherence degrades to broadcasts and back-invalidations — which destroy the bandwidth the rest of the design worked to provide.
Coherence overhead vs copies avoided. The energy/traffic of coherence must be less than the copies + flushes it eliminates — true when sharing is dense.

The point to carry:

An AI-accelerator design is a bandwidth pipeline with a coherence backbone: the pipeline (wide data channel, deep outstanding, HBM) keeps the datapath fed, and the backbone (coherent RN, covered directory) enables zero-copy sharing. The two interact — an under-covered directory triggers broadcasts that consume the bandwidth the pipeline provides, so coverage and bandwidth must be provisioned together. This coupling is the distinctive AI-accelerator analysis: you cannot maximize bandwidth while starving the directory.

13. Design Considerations

The deeper checklist for an AI-accelerator CHI design.

Widen the data channel to the accelerator's data demand — the bottleneck is DAT.
Deepen the outstanding window to cover the bandwidth-delay product — keep the pipe full.
Provision memory bandwidth with multiple high-BW controllers (HBM).
Size the directory to cover the accelerator's large footprint — avoid broadcasts.
Choose cache vs uncached by the accelerator's data reuse — cache stresses the directory.
Confirm coherence pays — sharing must be frequent enough to amortize the overhead over copies avoided.

14. The Pitfall

Pitfall: starving the datapath (bandwidth or directory coverage)

The classic AI-accelerator mistake is to starve the datapath — either by under-provisioning the data-channel bandwidth (a too-narrow DAT channel or too-few outstanding requests) or by under-sizing the directory for the accelerator's large footprint. Both cap the accelerator's throughput, often severely. A narrow data channel makes the accelerator bandwidth-bound below its compute rate; too-few outstanding requests make it latency-bound, running the pipe dry between completions; and an under-covered directory degrades coherence to broadcast snoops and back-invalidations — which flood the fabric and consume the very bandwidth the accelerator needs, a doubly painful failure because the coherence machinery turns against the bandwidth goal. The insidious part is that each of these can look fine in isolation (the compute is fast, the coherence is correct) while the system runs at a fraction of its potential.

The fix: provision bandwidth (wide data channel, deep outstanding window sized to the bandwidth-delay product, multiple memory controllers) and directory coverage (entries ≥ the accelerator's footprint) together — because an under-covered directory's broadcasts consume the bandwidth the rest of the design provides. Feed the datapath from both ends.

15. Common Mistakes

Narrow data channel. Assumption: standard width suffices. Bug: bandwidth-bound (the Pitfall). Fix: widen DAT to the demand.
Too-few outstanding. Assumption: a few in-flight is enough. Bug: latency-bound, pipe runs dry. Fix: cover the bandwidth-delay product.
Under-covered directory. Assumption: standard directory size. Bug: broadcasts/back-invalidations. Fix: cover the footprint.
Non-coherent when sharing is dense. Assumption: DMA is simpler. Bug: copy/flush overhead. Fix: coherent RN for frequent sharing.
Cache without coverage. Assumption: a cache is free. Bug: stresses the directory. Fix: size coverage for the cache footprint.
Provisioning bandwidth but not coverage (or vice versa). Assumption: one is enough. Bug: broadcasts eat the bandwidth. Fix: provision both together.

16. Engineering Checklist

Join the accelerator as a coherent RN for zero-copy CPU sharing.
Widen the data channel to the accelerator's data demand.
Deepen the outstanding window to the bandwidth-delay product.
Provision memory bandwidth with multiple high-BW controllers.
Size the directory to cover the accelerator's large footprint.
Confirm coherence pays — sharing frequent enough to amortize the overhead.

17. Key Takeaways

AI accelerators use CHI to share coherent memory with CPUs — zero-copy.
The design is bandwidth-first — wide data channel, many outstanding, HBM.
The data channel is the bottleneck; outstanding requests keep it full.
The accelerator's large footprint must be covered by the directory.
The trade-off is coherence overhead for copy elimination — a win when sharing is dense.
Provision bandwidth and coverage together; the model here is representative.

18. Quick Revision

AI accelerators. An AI accelerator (NPU) joins CHI as a coherent request node — with its own cache (RN-F) or uncached — so it shares the CPUs' coherent memory space and moves data without copies or software flushes. That zero-copy sharing is the central value: a non-coherent design pays a memory copy + cache-maintenance on every CPU-accelerator hand-off, and coherence eliminates both. The design is bandwidth-first: the data channel is the bottleneck (Chapter 15.2), so it must be wide; the accelerator must keep many requests outstanding to cover the bandwidth-delay product (Chapter 14.6) and keep the pipe full; multiple high-BW memory controllers (HBM) feed it; and the directory must be sized to cover the accelerator's large working set (Chapter 15.5). The trade-off is coherence overhead for copy elimination — favorable exactly when sharing is frequent and fine-grained, as in fused CPU-NPU pipelines. The pitfall: starving the datapath — an under-provisioned data channel or outstanding window (bandwidth/latency-bound), or an under-covered directory (whose broadcasts and back-invalidations consume the very bandwidth the accelerator needs). Provision bandwidth and coverage together. Unlike mobile, area and power are spent freely for bandwidth, because a starved accelerator is worthless. Representative case study; 20.3 covers CPU clusters.

Coming Next

Chapter 20.3 — CPU Clusters. Mobile and AI were single subsystems; CPU clusters are the canonical Arm coherent-CPU architecture. Chapter 20.3 covers CHI in DynamIQ / CMN-style CPU clusters — the two-level hierarchy of intra-cluster shared units and an inter-cluster coherent mesh network, the distributed home nodes with a system-level cache, and why this is the reference pattern for a scalable coherent CPU.