Memory Chiplets

Memory has appeared in every chapter of this module as a destination — something compute reaches through a link, through a controller, through the I/O die. This chapter turns and looks at the destination itself. The engineering question is narrower than "how does DRAM work", and more useful: what changes when the memory a system depends on is supplied by separately manufactured dies sitting inside the same package? The answer turns on a distinction that sounds trivial and is responsible for a great deal of disappointing silicon: adding memory and adding memory bandwidth are not the same act.

1. Memory Was Always a Separate Technology

Begin with something that predates chiplets entirely: memory and logic have never wanted the same manufacturing process.

A logic process is optimised for fast, dense switching — transistors that change state quickly and pack tightly. A DRAM process is optimised for something else: storage cells that hold charge reliably and can be built at extreme density with acceptable leakage and retention behaviour. These are different structures with different priorities, and a process tuned for one is not simply a better or worse version of a process tuned for the other.

Which is why bulk main memory has overwhelmingly lived on its own dies, made by suppliers specialising in it, connected to logic across some interface. The separation is not absolute — logic dies carry SRAM caches, and embedded DRAM has been integrated onto logic processes in some products — but those are compromises made for specific reasons, not the default for bulk capacity. So the chiplet question is not whether to separate memory from logic; that separation is largely a given. The question is how close the two can be, and what changes when the boundary moves from the board into the package.

Memory and logic do not need to share a manufacturing process to function as one system. Packaging changes how far apart they have to be.

2. Terminology, With Its Edges Showing

Industry usage here is genuinely loose. These are architecture-oriented definitions, offered for reasoning rather than as claims about how everyone speaks.

• Memory die — one fabricated die of memory. A physical object with its own yield and test.
• Memory stack — several memory dies assembled into one vertically integrated component, presented to the system as a single higher-capacity, higher-connectivity unit.
• On-package memory — memory placed inside the package alongside the logic dies rather than on the board. A statement about location, nothing more.
• Memory subsystem — the whole path: controllers, interfaces, the memory devices, and the logic managing them. What actually determines behaviour.
• Memory chiplet — an on-package memory component treated as a composable building block of the system.

That last one deserves care. By Chapter 2.1's definition, "chiplet" implies modular intent — a component designed to be composed. A memory die placed in a package because the design needed memory there is on-package memory; whether it is a chiplet depends on whether the system is built to compose it, vary it, and substitute it. Not every DRAM die in a package earns the term, and the distinction matters because it predicts whether capacity is actually a product variable or just a number fixed at design time.

3. Capacity and Bandwidth Are Different Products

This is the chapter's load-bearing distinction.

Capacity is how much information the system can hold. Bandwidth is how quickly information can move between memory and the logic that needs it. They are supplied by different things:

Capacity comes from the amount of storage present — more dies, denser dies, taller stacks. Bandwidth comes from the interface: how many independent channels exist, how wide each is, how fast it signals, how many independent memory resources can be worked in parallel, how the controllers schedule and interleave requests, and whether the package can physically route all those connections.

So the failure mode is easy to state and common to hit: you can add memory dies and gain capacity while gaining little or no bandwidth, if those dies attach to the same channels, share the same controllers, or are limited by the same package routing. Conversely a system can have ample bandwidth and insufficient capacity, which fails differently and just as hard.

Capacity is how much you can hold. Bandwidth is how fast you can reach it. Adding storage does not automatically add either channels or controllers.

The architectural payoff of on-package memory is that these become separable product dimensions: a family can offer more capacity with the same compute, or more bandwidth with the same capacity, by changing what is populated in the package rather than by redesigning silicon. That is real — provided the interface and controller resources actually scale with what is added, which is a design property rather than a consequence of adding dies.

4. Closer Is Not Cache

On-package memory can be physically closer and far more densely connected to logic than board-level memory. That is the point of putting it there. It is also where an intuitive over-extrapolation begins.

On-package memory is not another level of cache, and treating it as one produces bad architecture. Cache is small, integrated into the logic die, managed automatically by hardware, and reachable in a small number of cycles. On-package memory is large, on separate dies built in a different technology, reached through a controller and an interface, and sits in a fundamentally different latency class. The gap between an on-die cache hit and any main-memory access — however well packaged — is not a matter of degree.

Three things keep on-package memory in the memory latency class regardless of proximity:

• The controller is in the path. Scheduling, arbitration, and the memory protocol's own timing constraints dominate access time far more than the physical distance does.
• Die-to-die hops may still be involved. If the requesting compute die reaches memory through an I/O die, the access crosses at least one link before it reaches a controller at all.
• The memory technology's own timing. Access latency is largely set by how the storage works, and proximity does not change that.

Proximity genuinely helps — it enables much wider, denser connection than a board channel can, which is mostly a bandwidth and energy story rather than a latency one. Keep that framing and the architecture stays honest.

5. Where the Controller Sits, Again — With a Third Option

Chapters 2.2 and 2.3 examined controller placement from the compute and I/O sides. On-package memory adds a third possibility, and comparing all three in one place is what makes the decision legible.

On the compute die. Memory is local to the requester; no link in the path for its own accesses; bandwidth scales as compute dies are added. Costs: replicated controller area in every compute die, a memory personality that reduces compute-die reuse, and non-uniform latency when reaching another die's memory.

On the I/O / system die. Controllers exist once; every compute die sees memory uniformly; the compute die stays clean and reusable. Costs: every access crosses a link, and the I/O die becomes a bandwidth convergence point for the whole system.

Near the memory itself. Some architectures place controller or interface logic in a die adjacent to (or integrated with) the memory component. This can shorten and specialise the memory-facing interface and let that logic be built in a process suited to it. Costs: another die in the path, another boundary to define and verify, and another component in the failure domain.

What placement changes, in every case: die-to-die traffic volume, access latency and its uniformity, achievable bandwidth, the path coherence traffic must take, which die is reusable, what a failure takes down, and where the verification boundary falls. There is no universal answer — but there is a universal method, which is to ask what fraction of memory traffic crosses how many boundaries.

6. Bandwidth Only Helps If the System Can Reach It

Here is the failure that on-package memory invites, and it is a locality failure rather than a provisioning one.

Suppose a system provisions substantial memory bandwidth on-package. Now suppose a compute die's access path runs: compute die → die-to-die link → I/O die → fabric and arbitration → controller → memory. The memory subsystem can deliver more than that path can carry. The realised bandwidth is then set by the narrowest stage, and every intermediate stage adds latency, contends with traffic from other compute dies, and can throttle under load.

Memory bandwidth only helps if the rest of the system can reach it.

Which reframes the design problem usefully. Provisioning memory bandwidth is a purchasing decision — buy more channels, more components. Making it reachable is an architecture decision: how many boundaries sit in the access path, how much arbitration and contention it passes through, and whether the traffic that needs bandwidth most is close to the memory that supplies it. A system where every compute die reaches all memory through one shared hub has a different bandwidth profile from one where memory is associated with specific compute dies — even with identical memory components installed.

This is also why memory locality reappears as an architectural concept rather than an optimisation. If the physical arrangement makes some memory closer to some compute, the system either exposes that (and asks software to care) or hides it (and pays for it uniformly). Both are legitimate; pretending the asymmetry does not exist is not.

7. Memory Architecture and Package Architecture Are the Same Problem

On-package memory is where the package stops being a constraint on architecture and becomes part of it.

• Heat couples. Compute dies produce a great deal of heat; memory sits next to them. Memory devices have thermal limits and can behave differently as they warm, so a thermally aggressive compute die can affect the memory beside it. Placement is therefore a thermal decision as much as an electrical one.
• Routing density caps connectivity. Wide memory interfaces need many connections between the logic and memory components. How many are achievable, and over what distance, is a package capability — which means the package sets an upper bound on bandwidth before any controller is designed.
• Placement and reach interact. Memory must sit close enough to the logic that reaches it for the interface to work at the intended rate; that competes with other placement pressures, including thermal separation.
• Power delivery must serve everything. Every die in the package needs current delivered through the same structure.

In a multi-die system, memory architecture and package architecture cannot be separated cleanly.

The practical consequence is sequencing: a memory architecture must be checked against package feasibility while it is being chosen, not after. "How much bandwidth do we want?" and "how much can the package route to where we want to put it?" are one question.

8. Yield, Test, and Repair

Briefly, because Module 1 covered the principles and only the specifics change.

Each memory die or stack is separately manufactured with its own yield and test outcome, so known-good die applies here as it does anywhere — committing an untested memory component to an expensive package risks the good dies beside it. Stacked components add their own assembly step with its own yield. Memory devices commonly include redundancy and repair mechanisms that allow a device with certain defects to be made fully functional, which changes what "good" means for a memory die relative to a logic die. And the assembled package still requires validation, because a set of individually good components is not automatically a working memory subsystem.

9. Common Misconceptions

10. Understanding Check

11. Summary and What Comes Next

Memory and logic have always been separate silicon because their manufacturing processes optimise different things; packaging does not change that, it changes how close the two can be. What on-package integration adds is the ability to treat memory as a composable part of the system — provided it is genuinely designed to be composed, which is what distinguishes a memory chiplet from a memory die that happens to be in the package.

The distinction to carry away is that capacity and bandwidth are different dimensions supplied by different things: storage supplies capacity, while channels, width, signalling rate, controller organisation, and package routing supply bandwidth. Proximity keeps memory in the memory latency class — it improves bandwidth and energy, not cache-like access — and provisioned bandwidth only helps if the access path can carry it, which makes locality and the number of boundaries in that path architectural rather than incidental. Controller placement (on compute, on the I/O die, or near the memory) shifts traffic, latency uniformity, reuse, failure domain, and verification scope in one decision. And because reach, routing density, thermal coupling, and power delivery are all package properties, memory architecture and package architecture are one problem.

Compute has been modularised, connectivity has been modularised, and now state and bandwidth have been. The remaining question is whether specialised compute can become a building block too — and that turns out to depend less on the specialisation than on what it costs to feed it:

• 2.5 — Accelerator Chiplets — when moving specialised compute onto its own die pays, and when the cost of moving data across the boundary erases the advantage.

Browse the full path on the UCIe tutorials index.