The Memory Wall

Every mechanism CXL defines is an answer to a constraint that existed before CXL did. Read the protocol first and its features look arbitrary — three sub-protocols, three device types, a coherence model, a pooling architecture. Read the constraints first and the same features read as consequences.

This chapter builds the constraints. No CXL mechanism appears in it.

1. The One-Sentence Model

The memory wall is not one gap. It is the accumulated distance between what a system can compute and what its memory subsystem can deliver — measured in bandwidth, in latency, in capacity, in the energy spent moving bytes, and in where memory physically sits relative to the compute that needs it.

Most incomplete answers about the memory wall collapse those five axes into one. They are not interchangeable, they bind at different times, and a fix for one routinely does nothing for the others. Sections 4 through 11 take them one at a time.

2. What This Chapter Owns

The term "memory wall" entered wide use through Wulf and McKee's 1995 note Hitting the Memory Wall: Implications of the Obvious, which argued that processor performance and DRAM performance were improving at different rates and that the average memory access time would eventually dominate everything else. That argument was about latency, and it was correct about the direction.

Three decades of engineering later, the shape of the problem is broader than the original note described. Caches, prefetching, out-of-order execution and multithreading substantially blunted the pure-latency argument for many workloads. What they did not blunt — and in several cases made worse — is everything else on the list: the bandwidth required to keep a large number of parallel execution units supplied, the capacity a working set demands, the energy spent moving bytes, and the fact that memory is attached to one socket while the work may be somewhere else.

This chapter therefore treats the memory wall as multidimensional and says explicitly which axis each argument is about. That discipline matters later: CXL does not address all five axes, and a reader who holds them separately will be able to say which ones it does address and which it does not.

3. Two Curves That Stopped Tracking Each Other

The intuition is easy to state and easy to state badly.

The correct version: the number of operations a socket can perform per second has grown faster than the number of bytes its memory subsystem can supply per second, and faster still than the capacity attached to it. The reason is structural rather than a failure of DRAM engineering.

Compute capability grows along axes that are largely internal to the die. More cores fit as density improves. Vector units widen. Specialised units — matrix engines in particular — increase operations per cycle by a large factor without a proportional increase in area or pin count. None of that growth requires anything of the package boundary.

Memory bandwidth and capacity grow along axes that are largely external. Bandwidth is the product of how many channels a socket has, how wide each is, and how fast each runs. Channel count costs package pins, board routing and power-delivery area. Width costs the same. Transfer rate is bounded by signal integrity over a channel that must reach a socketed module. Capacity costs DIMM slots, which cost board area and add electrical load. Every one of those is a physical, packaging or economic constraint rather than a lithography one — so they do not improve at the rate transistor density does.

4. Bandwidth Is a Sustained Rate, Not a Frequency

Memory bandwidth is the rate at which the memory subsystem can sustainably deliver useful bytes to the requesters that need them. Three words in that sentence carry the weight: sustainably, useful, and deliver. A specification number satisfies none of them by itself.

Start with the arithmetic, because it is the part everyone quotes.

Theoretical peak

Peak bandwidth is a product of three independent factors:

per-channel bytes/s = transfer rate × bus width
 
    a DDR5 device rated 6400 MT/s, on a 64-bit-wide channel:
 
    = 6.4e9 transfers/s × 8 bytes/transfer
    = 51.2e9 bytes/s                        →  51.2 GB/s
 
socket peak = channels × per-channel bytes/s
 
    with eight such channels:
 
    = 8 × 51.2 GB/s
    = 409.6 GB/s

Two unit traps live in those four lines, and both appear in interviews.

MT/s is not MHz. DDR memory transfers data on both edges of its I/O clock — that is what "double data rate" names. A device rated 6400 MT/s runs a 3200 MHz I/O clock. Quoting the transfer rate as a clock frequency doubles the answer; quoting the clock as a transfer rate halves it.

Transfers are not bytes. A transfer moves one bus-width of data. The conversion factor is the width, and it must be written down. A DDR5 DIMM presents 64 data bits in total — organised as two independent 32-bit sub-channels rather than one 64-bit channel, which changes concurrency behaviour but not the bytes-per-transfer arithmetic at the module level. Eight bytes per transfer is the figure that belongs in the calculation.

Throughout this chapter, GB/s means 10⁹ bytes per second, which is the convention transfer-rate arithmetic uses. Capacity figures conventionally use binary multiples. Mixing the two silently is a third unit trap.

Sustained bandwidth

Peak is an upper bound the subsystem approaches and never reaches. Between peak and what an application observes sit several distinct losses:

• Refresh. DRAM cells hold charge that decays, so rows must be refreshed periodically. A bank being refreshed is not serving requests. The cost grows with device density.
• Row activation. A DRAM bank exposes one open row at a time through its row buffer. A request that hits the open row is served quickly; one that needs a different row in the same bank must close the current row and open the new one first. Access patterns that jump between rows in the same bank pay that penalty repeatedly.
• Bus turnaround. Switching a channel between reads and writes costs idle cycles on the bus. A read-heavy stream and a write-heavy stream each sustain more than a finely interleaved mix of both.
• Controller scheduling. The memory controller reorders queued requests to increase row hits and reduce turnarounds. It can only do that if it has requests queued to choose between — a shallow queue leaves it nothing to optimise.
• Bank and channel imbalance. Bandwidth is aggregate only if the traffic spreads across banks and channels. An access pattern whose stride maps every request onto the same channel gets one channel's bandwidth regardless of how many are installed.

Write η for the fraction of peak a given workload actually achieves. Then:

sustained = η × peak
 
    illustratively, at η = 0.7 against the 409.6 GB/s above:
 
    ≈ 287 GB/s

That 0.7 is an illustration, not a measurement. Achievable efficiency depends on the access pattern, the read/write mix, the request concurrency, the controller, and the DRAM timing parameters in use. The reason to write the equation anyway is that it makes the right question askable: not "how much bandwidth does this platform have" but "what is η for this access pattern, and which of the five losses above is setting it".

The structural point in Figure 1 is the asymmetry between its two ends. The left side grows with every generation that fits more cores or wider vector units on the die. The right side does not: n was decided when the socket was designed, and no amount of demand changes it.

5. Arithmetic Intensity — When Bandwidth Caps Compute

The connection between "the memory subsystem supplies B bytes per second" and "the application runs at R operations per second" is one ratio.

Arithmetic intensity is the number of operations a computation performs per byte it moves between memory and the processor. It is a property of the algorithm and its data layout, not of the machine.

Work an example all the way through. Take the vector update y[i] = a * x[i] + y[i], in double precision, over arrays far larger than the last-level cache:

bytes moved per element
    read  x[i]                8 B
    read  y[i]                8 B   ← see note below
    write y[i]                8 B
                             ────
                             24 B
 
flops per element             2     (one multiply, one add)
 
arithmetic intensity  =  2 / 24  =  0.083 flop/byte

The read of y[i] is the line people miss. y[i] is written, not read, by the algorithm — but a write to a location not present in cache normally triggers a read-for-ownership: the line is fetched so the partial write can be merged into it. The traffic is 24 bytes per element, not 16, and skipping that gives an answer 50% too optimistic.

Now put the machine's number next to it:

achievable rate = bandwidth × arithmetic intensity
                = 409.6e9 B/s × 0.083 flop/B
                ≈ 34 GFLOP/s
 
machine balance = peak flop rate ÷ peak bandwidth
                = 2.0e12 flop/s ÷ 409.6e9 B/s
                ≈ 4.9 flop/B

The peak flop rate above is an illustrative round number for a large server socket, chosen so the arithmetic is reproducible; substitute the figure for whatever part you actually care about and the structure of the conclusion does not change.

Machine balance is the break-even intensity. A computation whose arithmetic intensity is below it is memory-bound: its rate is set by bandwidth, and its execution units idle waiting for data. Above it, the computation is compute-bound: its rate is set by the execution units, and spare bandwidth goes unused.

The vector update sits at 0.083 flop/byte against a balance of 4.9 — memory-bound by roughly a factor of sixty. It will reach about 34 GFLOP/s on a socket rated at 2 TFLOP/s, which is under 2% of the peak figure in the marketing material. Nothing is broken. The kernel is simply asking for bytes faster than the machine can supply them, and the flop rate is a derived quantity, not an achievable one.

Two refinements worth carrying, because interviewers probe them:

Arithmetic intensity is not fixed by the mathematics — it is fixed by the data movement the implementation causes. Blocking a matrix multiplication so a tile is reused many times while resident in cache raises its intensity by a large factor without changing a single arithmetic operation. That is why dense linear algebra can be compute-bound while a vector update over the same hardware is not: one implementation was restructured to move fewer bytes per operation, and one has no reuse to exploit.

The relevant byte count is traffic across the boundary you are analysing. Intensity measured against DRAM traffic answers a different question from intensity measured against last-level-cache traffic. Name the boundary before quoting the ratio.

6. Latency Is a Different Constraint

Bandwidth is a rate. Latency is a delay. Increasing one does not automatically improve the other, and the reason is worth deriving rather than asserting.

Access latency is the interval from a requester issuing a memory request to the first byte of the response being usable. It decomposes into queueing delay in the controller, the DRAM device timings for activating a row and reading a column, transport over the channel, and return through the cache hierarchy. Only some of those parts shrink when the transfer rate rises: raising the data rate shortens the time to move a burst across the wires, but does not shorten the bank's activation timing, and it does not shorten a queue.

The relationship between the two is Little's Law, and it is the most useful equation in this chapter:

bytes in flight = bandwidth × latency
 
    to sustain 409.6 GB/s when each access takes 100 ns:
 
    = 409.6e9 B/s × 100e-9 s
    ≈ 41,000 bytes in flight
    ≈ 640 cache lines of 64 B, outstanding simultaneously

Read that result carefully, because it reframes both quantities. The bandwidth figure is only reachable if the workload can keep roughly 640 independent memory requests outstanding at once. Bandwidth is not something the memory subsystem hands over; it is something a requester earns by generating enough concurrency to fill the pipe. The name for that concurrency is memory-level parallelism.

Now the opposite case. A traversal in which each address is computed from the data just returned — a linked list, a tree walk, a chased index — can have exactly one request outstanding, because the next address is unknown until the current one returns:

// Dependent accesses. The address of the next load is the result of the
// previous load, so memory-level parallelism is pinned at one.
while (node) {
    sum += node->value;
    node = node->next;
}

// Independent accesses. Every address is computable in advance, so many
// requests can be in flight before the first result arrives.
for (int i = 0; i < n; i++) {
    sum += a[i];
}

For the dependent version, throughput is one cache line per access latency:

64 B ÷ 100 ns = 0.64 GB/s
 
    against a 409.6 GB/s peak, that is about 0.16%

Both loops run on the same hardware, at the same clock, with the same DRAM. One reaches a fraction of one percent of peak bandwidth; the other can approach it. The difference is a property of the dependency structure, not of the memory subsystem — which is precisely why bandwidth and latency have to be reasoned about separately.

This is also the sharp answer to "why does a latency-bound workload not improve when I install faster memory": faster memory raises peak bandwidth, but a workload with one request in flight was never limited by bandwidth. It was limited by the round trip, and the round trip barely moved.

7. Capacity Is a Third Constraint

Bandwidth and latency describe how well memory serves the working set. Capacity decides whether the working set fits at all — and it fails in a completely different way. A bandwidth shortfall makes a program slower. A capacity shortfall makes it spill to storage, or refuse to run.

Capacity per socket is bounded by the number of memory interfaces the socket provides and the density of the modules those interfaces can carry. Both are physical. Slots occupy board area near the socket; each additional module loads the channel electrically, which interacts with the transfer rate the channel can sustain. This is why capacity and speed are frequently traded against each other in platform configuration, and why "just install more" has a hard ceiling rather than a soft one.

Put a working set next to that ceiling:

model weights at 2 bytes per parameter, 70e9 parameters
 
    = 140e9 bytes                     ≈ 140 GB   (weights alone)
 
    before activations, key/value cache, optimiser state,
    framework overhead, or a second concurrent job
 
socket capacity, 8 slots × 64 GB modules
 
    = 512 GB

The weights fit. The point is not that they do not — it is what happens to the ratio when the workload grows. Working-set sizes in machine learning, in-memory databases, graph analytics and large-scale simulation have grown by orders of magnitude over a period in which per-socket capacity grew by a factor that is not close to that.

And here is the structural problem, which is independent of any particular number:

Adding compute does not add memory capacity. A second socket brings its own channels and its own slots, so aggregate capacity rises — but it brings cores in the same proportion. Capacity per core does not improve; it is roughly preserved. A workload that needs more memory per unit of compute cannot buy that ratio by scaling out. It has to buy whole machines it does not need the compute of, and then leave those cores idle.

That is the first appearance of a theme this chapter closes on: the ratio of memory to compute is fixed by the platform, while the ratio a workload needs varies enormously between workloads.

8. Moving Bytes Costs Energy

The three constraints so far are about time. This one is about power, and in large systems it frequently becomes the binding one, because a socket, a board and a rack all have a power budget that is fixed before any workload arrives.

The engineering fact underneath is that the energy required to move a piece of data grows with the distance it travels and the interface it crosses, while the energy required to operate on it once it has arrived does not. The ordering, from cheapest to most expensive, is stable across designs: a register access, an access to a nearby on-die SRAM, a wire crossing the die, a transfer across the package boundary, an access to external DRAM.

Published energy-per-operation studies — Horowitz's 2014 ISSCC paper Computing's Energy Problem (and what we can do about it) is the standard reference — place an off-chip DRAM access two to three orders of magnitude above an on-die arithmetic operation of the same width. The absolute numbers are process- and design-specific and should not be quoted as universal, but the ratio is the durable part, and it has not been reversed by any subsequent technology.

Two consequences follow directly.

Arithmetic is no longer the expensive part. For a low-intensity computation, the energy bill is dominated by the movement of operands rather than by the operations performed on them. Optimising the operation count of such a kernel is close to pointless; optimising its data movement is the whole problem.

A copy that changes nothing still costs full price. Staging a buffer from host memory into an accelerator's memory performs no arithmetic and produces no result, but it pays DRAM read energy, link energy, and DRAM write energy for every byte. When the same data is copied back afterwards, it pays again. In a pipeline where copies are not overlapped with computation, that movement can consume more time and more energy than the computation it exists to serve.

9. Memory Is Not Uniformly Close

"Memory" in a modern system is not one pool with one set of characteristics. It is a hierarchy of places, and the place matters as much as the technology.

Within a socket, the cache hierarchy already makes this visible: an access served from a local cache and one served from DRAM differ by more than an order of magnitude in both latency and energy. Across sockets, it appears again. In a multi-socket server, each socket owns memory channels, and a core reaching memory owned by another socket must traverse an inter-socket interconnect first. That access is slower, and the aggregate bandwidth available to remote memory is bounded by that interconnect rather than by the remote socket's channels. This is what NUMA — non-uniform memory access — names.

Enough NUMA to support the model, and no more: placement is a first-class performance property, and the system cannot always choose the placement the workload wants. A thread runs where the scheduler put it; a page lives where it was first touched; a device sits on the link it was plugged into. When those three disagree, the workload pays on every access, and no amount of additional bandwidth on the local channels helps, because the traffic is not going there.

10. Accelerators Press on All Five Axes

Accelerators — GPUs, FPGAs, NPUs, domain-specific engines — did not create the memory wall. They made every axis of it bind harder at once, for reasons that follow from what an accelerator is.

They have far more parallelism to feed. An accelerator's value proposition is a large number of execution lanes operating concurrently. From Section 5, execution lanes only produce results if bytes arrive to occupy them, so multiplying lanes multiplies the required byte rate. An accelerator sized for a compute-bound kernel is a memory-bound machine for anything below its own (typically high) machine balance.

Their local memory is fast and small. Memory placed close to an accelerator — on-package, on a short high-bandwidth interface — buys bandwidth precisely by being close, which is also what bounds how much of it there can be. That is a genuine engineering trade, not a shortcoming: proximity is what makes the bandwidth possible, and proximity limits capacity.

They are a separate memory domain. Host memory and device memory are distinct address spaces backed by distinct DRAM. Data the device needs and the host produced must be moved, and Section 8 has already priced that movement.

Utilisation is lost while waiting. An accelerator stalled on a transfer is not doing less work per second — it is doing none. Where copy and compute cannot be overlapped, the effective throughput is the sum of both phases rather than the larger of the two, and the expensive silicon idles for the copy phase. (The PCIe track works this arithmetic through in detail in AI Accelerators — When the Link Is Not the Bottleneck and Performance Implications.)

Figure 3 is deliberately drawn as two domains rather than one system, because that is the property later chapters return to. The accelerator cannot spend the host's spare capacity, the host cannot spend the device's spare bandwidth, and the only traffic between them is copying.

11. Stranded Memory — The Utilisation Axis

The fifth axis is the least discussed and, at fleet scale, often the most expensive.

Memory is stranded when it is installed, powered and healthy, but cannot be used by the work that needs it. It is not a performance problem in the usual sense — the memory is not slow, it is simply unreachable by the workload that would benefit.

Stranding is a direct consequence of the fixed memory-to-compute ratio from Section 7. A machine is configured with some amount of memory per socket, and then real workloads arrive with wildly different ratios. A memory-heavy job exhausts capacity while leaving cores idle. A compute-heavy job saturates cores while leaving most of the memory untouched. Both are running on the same platform configuration, and there is no way to move capacity from the second machine to the first — the DRAM is soldered into a topology that was decided at purchase.

Scale that across a fleet and the arithmetic gets uncomfortable: a large fraction of installed capacity can be simultaneously unused and unavailable. It was paid for, it draws power, and no scheduler can allocate it because it is behind the wrong socket.

This axis is why the memory wall is not purely a chip-design problem. Bandwidth, latency and capacity are properties a socket can be designed for. Stranding is a property of the topology — of the fact that memory is bound to one host at build time — and no improvement in DRAM technology addresses it.

12. Which Wall Are You Actually Hitting?

Five constraints means five different diagnoses, and treating the wrong one is a common and expensive mistake. The table below is a starting point for reasoning, not a substitute for measurement.

Two habits behind that table are worth stating explicitly, because they distinguish an engineer who reasons about memory from one who guesses.

Compare against a bound, not against a previous run. "Bandwidth is 180 GB/s" means nothing alone. Against a 409.6 GB/s peak it means η ≈ 0.44, and the next question is which of Section 4's losses is responsible.

Separate the rate from the concurrency. A low observed bandwidth has two entirely different causes — the subsystem is saturated, or the requester is not asking for enough at once. Little's Law tells them apart, and the counters that distinguish them are outstanding-request occupancy, not bytes per second.

13. What the Industry Has Already Tried

Every mitigation below is real, deployed, and useful. Each also has a boundary, and knowing where each stops is what makes the case for a different kind of answer.

Read the right-hand column as a set. Caches need reuse. Prefetch needs predictability. Channels and rates are fixed by the package. Memory placed near the die trades capacity for proximity. Scaling out buys the wrong resource.

What none of them changes is the binding assumption underneath all of them: that memory is attached to exactly one host, in a quantity chosen before the workload existed. Every technique above optimises within that assumption. That observation is where this chapter is heading.

14. Real Engineering Relevance

The memory wall is not a topic that lives only in architecture papers. It shows up as concrete daily questions, and the question differs by role.

CPU and SoC architect

Sizing decisions are memory decisions. How many memory channels does this part need to keep its core count useful? At what core count does additional throughput stop appearing because bandwidth is already saturated? Is the target workload latency-sensitive or throughput-sensitive, because the two argue for different microarchitecture? Is capacity, rather than bandwidth, becoming the limit that decides how the part is deployed? Every one of these is answered with the arithmetic in Sections 4 to 7, not with a preference.

Accelerator architect

The dominant question is keeping execution lanes supplied. What is the machine balance of the part being designed, and what is the arithmetic intensity of the workloads it is for? How much local memory does the target working set need, and what does the proximity-versus-capacity trade cost? How much host traffic does a unit of work generate, and can that traffic overlap with computation? An accelerator whose lanes idle waiting for data has converted a compute problem into a memory problem.

RTL and microarchitecture engineer

At this level the constraints become structures. Sustaining bandwidth means supporting many outstanding transactions, which means request-tracking state, buffering to hold data that has been requested and not yet consumed, and enough queue depth for the scheduler to have choices. Latency hiding means allowing subsequent requests to proceed past an outstanding one. Backpressure has to propagate correctly when a downstream resource fills, and arbitration has to be fair enough that no requester starves under load. Every one of these is a direct consequence of Little's Law: to sustain bandwidth × latency bytes in flight, something has to track them all.

Verification engineer

The failures that matter here are load-dependent, so the interesting stimulus is not a correct single transaction. It is the conditions under which the structures above break: many requesters bursting simultaneously; sustained backpressure held long enough to fill every queue; the maximum supported number of outstanding requests, and one beyond it; responses returning out of order, or in the least convenient order; ordering-sensitive sequences interleaved with unrelated traffic; and long-running fairness scenarios that expose a requester being starved by a more aggressive neighbour. A memory path that passes directed tests and fails at saturation is the normal outcome of verifying only the former.

Performance engineer

The work is interpretation. Bandwidth counters answer "how many bytes", occupancy counters answer "how many requests were in flight", and only together do they say whether a low number means saturation or starvation. Cache-miss rates locate where traffic originates. The habit that matters is the one from Section 12: always compare a measurement to a computed bound, and always separate the rate from the concurrency before proposing a fix.

15. Common Misconceptions

16. Interview Reasoning

Memory-wall questions are used as a discriminator because they separate a candidate who has memorised a definition from one who can reason about a system. The questions below progress from foundational to architectural.

17. Summary

The memory wall is the accumulated distance between what a system can compute and what its memory subsystem can deliver, and it has five distinguishable axes.

Bandwidth is a sustained rate, computed as channels × transfer rate × bus width and then reduced by refresh, row activation, bus turnaround, scheduling and channel imbalance. Arithmetic intensity — operations per byte moved — multiplied by achievable bandwidth bounds the operation rate, and comparing it against machine balance says whether adding compute will help at all. Below balance it will not.

Latency is a separate constraint related to bandwidth by Little's Law: sustaining bandwidth requires bandwidth × latency bytes in flight. A workload that cannot generate that concurrency cannot use the bandwidth, which is why a dependent chain reaches a fraction of a percent of peak on hardware that would happily deliver hundreds of gigabytes per second to a streaming loop.

Capacity fails differently from both — it decides whether a working set fits, not how fast it runs — and it does not improve when compute is added, because a second socket brings cores in the same proportion as memory. Data movement costs energy that grows with distance and with each boundary crossed, so for low-intensity work the operands cost more than the operations, and a copy that computes nothing still pays full price. Placement decides whether any of the above is even reachable: memory behind another socket, inside a device's private domain, or in a machine the workload is not running on is capacity that cannot be spent.

Every established mitigation optimises within one assumption — that memory is attached to exactly one host, in a quantity chosen before the workload existed. Caches need reuse, prefetch needs predictability, channels and rates are fixed by the package, high-bandwidth memory trades capacity for proximity, and scaling out buys compute nobody asked for.

Hold the durable form: conventional socket-local memory architecture is increasingly insufficient for heterogeneous, memory-hungry systems — not because DRAM stopped improving, but because memory is bound to one host in a fixed quantity while workloads need wildly different ratios of memory to compute.

18. What Comes Next

The five axes are now separable, which makes the next question precise rather than vague.

Sections 12 and 13 established that bandwidth, latency and capacity are things a socket can be designed for, while stranding and cross-domain data movement are properties of the topology — of what is attached to what. No improvement in DRAM technology addresses a topology problem. That leaves one direction: change what memory can be attached to, and how.

Doing that raises requirements immediately. Memory reached over a link must still behave like memory to software, or the fix becomes a new programming model rather than an architectural improvement. A device that caches host memory must not be able to observe stale data. Existing device attach must keep working, because no ecosystem replaces its I/O in one step. The question the rest of this track answers is therefore:

What architectural mechanism lets compute and memory resources be connected more flexibly — expansion beyond a socket's channels, pooling across hosts, accelerators that share memory rather than copy it — without abandoning coherence or compatibility?

Module 1 continues by working through the remaining forces one at a time: where the CPU-as-master model runs out of headroom, what data movement actually costs, how accelerator proliferation created a device-attach problem, why discrete accelerator memory cannot grow with working sets, what conventional device attach does and does not provide, and finally why coherent attach is the property that ties the answers together.

For the link-side view of several arguments here, the PCIe track covers latency accounting, why a fast link can still be slow, and host–device transfers in detail. The full path is on the CXL tutorials index.