Accelerator Chiplets

Specialised compute is the most tempting thing to turn into a chiplet. The pitch writes itself: build an engine that does one class of work far better than general-purpose cores, make it a die, drop it into products that need it, omit it from those that do not. Much of that pitch is sound. But accelerators fail as chiplets for a reason that has nothing to do with how good the engine is, and everything to do with what it costs to keep it fed. This chapter is about that constraint, because it decides the outcome more often than the arithmetic does.

1. The Engine Is Not the System

An accelerator chiplet is a chiplet specialised for one or more classes of work — a machine-learning engine, a DSP, media or video processing, cryptography, compression, packet processing, or any other domain-specific compute.

Where a general-purpose core optimises flexibility — run anything reasonably well — an accelerator optimises around a specific computation: its dataflow, its arithmetic structure, and its memory-access pattern. Giving up generality is what buys the advantage, which can appear as higher throughput, better energy efficiency, lower latency for that work, or some combination.

Now the constraint. An accelerator only produces value while it is computing, and it can only compute while it has data. Every operand must arrive and every result must leave — and once the accelerator is a separate die, all of that crosses a die-to-die boundary with finite bandwidth and real latency.

An accelerator's value is set not only by how fast it computes, but by how efficiently data reaches it and leaves it.

This is why the interesting question about an accelerator chiplet is never "how fast is the engine?" It is: how much work does it do per byte that crosses the boundary?

2. What Crosses, and How Often

Before deciding whether a function should become an accelerator chiplet, characterise its traffic. The useful questions are concrete:

• What data crosses the boundary — operands in, results out, and any state or configuration.
• How many bytes per unit of work? This is the decisive ratio: work performed per byte moved, sometimes called arithmetic intensity. High ratios survive a boundary; low ratios do not.
• How often? A steady stream is very different from occasional bursts, and different again from fine-grained request–response.
• How much reuse happens locally? If the accelerator reads a value once and discards it, every byte must cross. If it revisits data many times, most accesses can be served locally.
• Is it latency-sensitive? Streaming work tolerates latency if bandwidth suffices; work where the host waits on each result does not.
• Does it touch shared memory, and must that memory be coherent?
• Is output volume comparable to input volume? Compression and decompression are asymmetric in opposite directions; a classifier may consume a great deal and emit almost nothing.

The pattern these questions expose is what matters. Work with high local reuse and a favourable work-per-byte ratio survives being moved to another die. Work that streams large volumes for little computation, or that requires fine-grained interaction with the host, generally does not — the boundary becomes the system, and the engine idles waiting for it.

That framing also explains why the same accelerator function can be a good chiplet in one system and a bad one in another. Nothing about the engine changed; the traffic it generates relative to the available boundary bandwidth did.

3. Local Storage Buys Back the Boundary

The main architectural lever against data-movement cost is giving the accelerator its own memory — SRAM, a cache, a scratchpad, or local memory — so a working set can be loaded once and reused many times without recrossing the boundary.

The trade runs both ways, and neither end is correct in general:

More local storage. Higher local reuse, so fewer bytes cross per unit of work, and less sensitivity to boundary bandwidth and latency. Costs: a larger accelerator die (memory arrays consume real area), more power, and a working-set size baked into the silicon — if the workload exceeds it, the benefit degrades, sometimes sharply.

Less local storage. A smaller, cheaper die that adapts to varying working-set sizes, at the price of depending heavily on shared-memory bandwidth and on the boundary being fast enough to keep the engine fed.

Notice what local storage really purchases: it converts a bandwidth problem into a capacity problem. That is usually a good trade, because capacity is a design-time decision while boundary bandwidth is a system-wide contended resource. It is a bad trade when the working set is unpredictable or much larger than anything reasonable to integrate. Sizing local storage is therefore a workload question first and an area question second.

4. Two Ways to Attach an Accelerator

How the accelerator sees memory is the other decisive architectural choice, and the two ends of the spectrum have genuinely different consequences.

Coherent attachment. The accelerator participates in the system's shared memory semantics — it sees the same coherent view of memory the compute dies do. The benefit is a much easier programming model: data structures can be shared without explicit copying, and the software does not orchestrate movement. The costs land on the hardware. Coherence traffic is fine-grained, latency-sensitive, and amplifying (as Chapter 2.2 noted for compute dies), and now it crosses a die boundary. Ordering guarantees must hold across that boundary, and the verification burden grows considerably because correctness now depends on cross-die coherence behaviour under contention.

Explicit-transfer (non-coherent) attachment. Software or a runtime explicitly moves data into the accelerator's memory, the accelerator works on it, and results are moved back. The hardware boundary is dramatically simpler: bulk transfers, no cross-die coherence, no shared-memory ordering to maintain. The costs move into software — explicit copies must be scheduled and managed, buffers sized, transfers overlapped with computation to avoid stalls, and the programming model is more demanding. Copies also cost time and energy that the coherent model does not always pay.

The choice tends to follow the traffic pattern from §2. Coarse-grained work on large blocks favours explicit transfer, because a few bulk movements amortise well and the hardware stays simple. Fine-grained work interleaved with host computation on shared structures favours coherence, because explicit copies at that granularity would cost more than the coherence traffic does. Many real systems support both and let the workload choose — which is a design decision, not a hedge.

5. Why Accelerators Suit the Chiplet Model

Given all the caution above, the case for accelerator chiplets is still strong — and it is a different case from the one made for compute in Chapter 2.2, which is worth being precise about.

Optionality. An accelerator is frequently not needed by every product in a family. As a chiplet it can simply be absent — omitted from the package rather than occupying dark silicon on every die shipped. That is a benefit compute rarely has, since every product needs compute.

Divergent evolution rates. Domain-specific architectures can change much faster than general-purpose compute microarchitecture, because the workloads they target are themselves moving. Separating them lets the accelerator be revised without touching the host design — genuinely valuable when a field is changing quickly, and worth little when it is not.

Variable count. A product line can populate zero, one, or several accelerator dies according to what the market segment needs.

Independent technology choice. An accelerator with unusual requirements can be built where those requirements are best met.

These are conditional. Plenty of specialised functions belong on the compute die — particularly ones that are small, universally needed, or tightly coupled to the core pipeline, where a boundary would cost more than modularity returns. Optionality is the argument that most often decides it, because it is the one that saves area on every unit shipped rather than only on some.

6. Replication Scales Engines, Not Necessarily Throughput

Chapter 2.2 made this point for compute dies. It applies here with additional force, and for partly different reasons.

Adding accelerator dies adds arithmetic capability. Whether it adds delivered performance depends on:

• Memory bandwidth, which is usually shared. Two accelerators can consume more than the memory subsystem supplies, at which point they take turns.
• Die-to-die bandwidth on the paths feeding them.
• Host scheduling. Something must dispatch work; if the host cannot issue fast enough, engines starve.
• Synchronisation, where results must be combined or ordered.
• Workload parallelism. The work must actually decompose across engines.
• Package routing, since each die needs its links physically carried.
• Thermal and power limits, which apply to the package as a whole rather than per die.

Replication scales the engines available. It does not necessarily scale useful system throughput.

The accelerator-specific twist is that accelerators are usually more bandwidth-hungry per unit of area than general-purpose compute — that is often exactly why they are fast. So they tend to hit the shared-bandwidth ceiling sooner than adding compute dies would. Scaling accelerators frequently means scaling the memory and interconnect that feed them, and a system that adds engines without adding supply produces a smaller improvement than the arithmetic suggests.

7. Reuse Requires a Stable Contract

A reusable accelerator die can evolve independently of the host — but only within a contract that stays fixed, and the contract here is broader than the electrical interface.

Compatibility must hold across the interface behaviour itself, the memory model the accelerator assumes (coherent or not, what ordering it relies on), discovery and configuration so the host can find it and set it up, management including telemetry and power states, error semantics — what it reports, how, and what the host must do about it — and software expectations, since drivers and runtimes encode assumptions about all of the above.

That last one is the one hardware teams underestimate. An accelerator whose hardware interface is unchanged but whose behaviour differs in ways the runtime notices is not, in practice, a drop-in replacement. Reusable silicon does not imply reusable system integration, and the software contract is usually the harder one to hold stable across generations.

8. What This Means for Verification

An accelerator boundary generates verification work that is distinct from the compute and I/O cases, because the accelerator is both a target and an independent initiator of memory traffic:

• Are work requests routed to the correct accelerator, and do results return to the correct requester?
• Are work descriptors and their parameters interpreted correctly, including malformed or boundary cases?
• Are ordering requirements preserved between work submission, the accelerator's memory accesses, and completion signalling?
• Does the accelerator's memory access behave correctly — under the coherent model, or with correct explicit-transfer semantics?
• What happens if the accelerator is reset while work is in flight, and can the host recover without a system-level reset?
• Does backpressure propagate correctly when the accelerator cannot accept more work, or when its output cannot drain?
• Does the system behave correctly when the accelerator is absent — a configuration that must work, given optionality is a main reason for the partition?
• Does it behave correctly with several instances, including uneven load and partial population?
• Are errors detected, reported, and attributed to the right work item, and can the system recover?

The absent-accelerator case deserves emphasis. Optionality means "no accelerator present" is a supported configuration rather than a degraded one, and it must be verified as carefully as the populated cases.

9. Common Misconceptions

10. Understanding Check

11. Summary and What Comes Next

An accelerator chiplet is a die specialised for a class of work, trading generality for throughput, energy efficiency, or latency on that work. Its value, though, is decided at the boundary: every operand and result crosses a die-to-die link, so the ratio of work performed to bytes moved predicts real behaviour far better than peak arithmetic rate. Functions with high local reuse and coarse-grained interaction survive the boundary; those that stream large volumes for little computation, or interact with the host at fine granularity, generally do not.

Local storage is the main lever, converting a contended bandwidth problem into a design-time capacity problem — until the working set outgrows it. Attachment style is the other: coherence buys an easier programming model and costs fine-grained cross-die traffic and verification burden, while explicit transfer buys a simpler hardware boundary and costs software-managed movement, with the traffic pattern deciding. The case for accelerators as chiplets rests mostly on optionality — they can simply be absent from products that do not need them — and on independent evolution, not on the replication argument that motivated compute. And replication here scales engines rather than throughput, hitting shared-bandwidth limits sooner than compute does because accelerators consume more bandwidth per unit of area.

Module 2 has now separated the system into roles: compute, connectivity and system control, memory, and specialised compute. Every one of those chapters ran into the same wall from a different direction — the boundary, the links, the routing, the thermal envelope, all properties of the thing physically holding the dies together:

• 2.6 — Packaging Fundamentals — the substrate, interposer, and bridge structures that carry these connections, and what each makes possible or impossible.

Browse the full path on the UCIe tutorials index.