The Memory Wall Problem

Chapter 1.7 justified the tier. DRAM holds main memory because it is the only technology satisfying all of the tier's requirements at once — and the requirements it satisfies come packaged with three facts established earlier in this module. It is far from compute, because capacity at that scale cannot sit on the compute die. Its access is a sequence with mandatory ordering and minimum intervals. And the cost of an access depends on the device's state, so identical requests can take very different amounts of time.

Meanwhile the thing being served has become extraordinarily capable. This chapter is about what happens when those two facts meet, and the question is:

Why does increasing compute capability eventually stop translating into proportional system performance?

The name for the phenomenon is the memory wall, and the name is unfortunate, because it suggests a hard barrier and invites the summary that everyone reaches for: DRAM is slow. That summary is wrong in a way that actively prevents engineering. It is wrong because the same device delivers excellent results on some workloads and poor ones on others, so the device cannot be the explanation. It is wrong because it conflates two independent problems — latency and bandwidth — that have different mechanisms and different solutions. And it is wrong because it suggests nothing to do.

What this chapter builds instead is a decomposition: an account of where a workload's time actually goes, which variables control each part, and what an engineer can measure to find out. By the end, "the memory is slow" should feel like the unhelpful non-answer it is.

1. Start From the Only Thing That Matters

An execution unit does useful work only when its operands are available. That is the whole foundation, and everything else in this chapter is a consequence.

So the performance of a machine is not determined by how fast it can compute. It is determined by how often its computation has operands to work on — which makes the supply of operands, not the capability to consume them, the interesting variable. Chapter 1.2 built the fastest possible operand supply and showed it cannot scale. Chapter 1.1 built the hierarchy that follows from that. This chapter asks what happens at the bottom of it.

The two paths differ in kind, not just in length. The short path is a lookup that either succeeds or does not. The long path has a queue in it, so its duration depends on what else is queued; a scheduler, so its duration depends on decisions; and a sequenced device access, so its duration depends on the device's state. Three sources of variability, none of which exists on the short path.

That is the structural reason a single latency figure cannot describe main memory, and it is why the rest of this chapter is about distributions and mechanisms rather than numbers.

2. What the Memory Wall Actually Is

State it carefully, because the careless statement is the one that spreads.

The memory wall is the system-level consequence of three things together: compute capability and main-memory access latency having improved at very different rates; workloads having only as much locality as they have; and the memory system having finite latency and finite bandwidth. Any one of those alone is manageable. Together they mean that beyond some point, adding compute capability produces less and less additional performance, because the added capability spends its time waiting.

On the scaling, precisely and without invented figures. Across generations of the technology, the headline figure that has improved most is the transfer rate — how much data the interface can move per unit time. The latency of an individual access has improved far more slowly. That asymmetry is directional and well established, and its mechanism is exactly Chapter 1.6 §5's coupling: transfer rate is improved by signalling faster over the channel, while the latency of an access is dominated by the physical operations inside a dense array — charge sharing, amplification, restoration — which do not get faster simply because the interface does. The two halves of "memory performance" have been improving at different rates for structural reasons.

And on what the wall is not. It is not a hard barrier; performance does not stop, it stops scaling proportionally. It is not a DRAM defect; a device that answers one workload well and another badly is not the variable that changed. It is not solved by a faster interface alone, because a faster interface primarily improves the half that was already improving. And it is not universal: a workload whose working set fits in cache barely encounters it, which is why benchmark results diverge so widely.

3. Latency and Bandwidth Are Different Problems

This distinction does more work than any other idea in memory-system engineering, and it is routinely collapsed.

Latency is how long one access takes, measured from a defined observation point. Its unit is time. It is what a computation waiting on a specific value experiences.

Bandwidth is how much data can be transferred per unit time under defined conditions. Its unit is data per time. It is what a stream of independent work experiences.

They are not two views of one quantity, and high bandwidth does not imply low latency. The clearest way to see why is to look at a single request's timeline and ask which parts bandwidth affects.

   issue        accepted         device work         transfer      available
     |             |                  |                 |             |
     v             v                  v                 v             v
     |--queueing---|---scheduling-----|---sequence------|--data move--|
     |                                                                |
     |<------------------- one request's latency -------------------->|
                                                       |<---------->|
                                                    this part is what
                                                    bandwidth governs

Only the last segment is bandwidth. Raising the transfer rate shortens the time spent moving the data and leaves queueing, scheduling and the device's internal sequence essentially untouched. For a request whose latency is dominated by those earlier segments, a large bandwidth improvement produces a small latency improvement — which is exactly why generational transfer-rate increases do not deliver proportional gains on latency-sensitive workloads.

And the converse matters just as much. A workload consisting of many independent accesses does not care much about any individual request's latency, because it never has to wait for one before issuing the next. It cares about how many bytes per second arrive in total. For that workload, bandwidth is nearly everything and latency is nearly irrelevant.

Which gives the single most useful diagnostic question in memory-system engineering: is this workload waiting for a particular value, or waiting for enough values? The first is a latency problem and more bandwidth will not fix it. The second is a bandwidth problem and lower latency will not fix it. Answering that question wrongly is how teams spend a generation of hardware improvement on the wrong axis.

4. RTL — Making Waiting Visible

To reason about waiting, it helps to have something that waits. This model is the smallest thing that makes memory latency a cycle-level, observable fact.

What this is and is not. It is an educational latency abstraction: a request interface, a configurable delay, a response, and an outstanding-request limit. It is not a DRAM model, not a memory controller, and not a memory — it holds no storage at all, because latency is the only thing it exists to teach. It has no banks, no rows, no commands, no refresh, and no state-dependent cost. Chapter 1.4 §6 is the model with state-dependent cost; this one deliberately has a fixed latency so that the effect of concurrency can be isolated from everything else.

What it does. It accepts a request when it has capacity, returns a response LATENCY_CYCLES later, and allows up to MAX_OUTSTANDING requests to be in flight simultaneously. That last parameter is the knob this chapter turns: it is the difference between §6's picture and §7's.

How to simulate it. vlog memory_latency_model.sv tb_memory_latency_model.sv then vsim -c tb_memory_latency_model -do "run -all"; with VCS vcs -sverilog memory_latency_model.sv tb_memory_latency_model.sv && ./simv; with Xcelium xrun -sv memory_latency_model.sv tb_memory_latency_model.sv.

// EDUCATIONAL LATENCY ABSTRACTION. Not a DRAM model, not a controller, and
// not a memory -- it holds no storage. Its only job is to make waiting
// visible at cycle level, and to let MAX_OUTSTANDING be varied so the
// effect of concurrency can be seen in isolation (§7).
module memory_latency_model #(
  parameter int DATA_WIDTH      = 32,
  parameter int ADDR_WIDTH      = 16,
  // Fixed, deliberately: a real access cost depends on device state
  // (Chapter 1.4), and holding it constant here isolates the variable this
  // chapter is about. Must be >= 1.
  parameter int LATENCY_CYCLES  = 8,
  // How many requests may be in flight at once. 1 exposes every latency in
  // full; higher values overlap them. Must be >= 1.
  parameter int MAX_OUTSTANDING = 1
) (
  input  logic                  clk,
  input  logic                  rst_n,

  // Request channel. Accepted when req_valid && req_ready.
  input  logic                  req_valid,
  input  logic [ADDR_WIDTH-1:0] req_addr,
  output logic                  req_ready,

  // Response channel. Unconditional -- the requester must be able to take it.
  output logic                  rsp_valid,
  output logic [DATA_WIDTH-1:0] rsp_data,

  // Instrumentation. Not decoration: §14 and §16 are built on counters like
  // these, and a design that cannot report its own occupancy cannot be
  // diagnosed from outside.
  output logic [31:0]           outstanding
);

  // ── The delay line. One stage per cycle of latency; a request shifts
  //    along it and emerges as a response. This is what "waiting" is. ──────
  logic                  pipe_valid [LATENCY_CYCLES];
  logic [DATA_WIDTH-1:0] pipe_data  [LATENCY_CYCLES];

  // Occupancy, used only to enforce the outstanding limit.
  localparam int CNT_W = (MAX_OUTSTANDING <= 1) ? 1
                                                : $clog2(MAX_OUTSTANDING + 1);
  logic [CNT_W-1:0] occ;

  // Backpressure: refuse a request when the limit is reached. THIS is the
  // mechanism that turns MAX_OUTSTANDING into observable behaviour.
  assign req_ready = (occ < CNT_W'(MAX_OUTSTANDING));

  logic accept;
  assign accept = req_valid && req_ready;

  // The response is a deterministic function of the address, so a testbench
  // can check that a response matched its request without the model needing
  // to store anything. The cast handles either width relationship.
  logic [DATA_WIDTH-1:0] echo;
  assign echo = DATA_WIDTH'(req_addr);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < LATENCY_CYCLES; i++) begin
        pipe_valid[i] <= 1'b0;
      end
      occ <= '0;
    end else begin
      // Shift the delay line. Deepest stage first so each stage reads the
      // PREVIOUS value of the one before it -- nonblocking assignment makes
      // this correct regardless of loop order, but writing it this way keeps
      // the intent obvious to a reader.
      for (int i = LATENCY_CYCLES - 1; i > 0; i--) begin
        pipe_valid[i] <= pipe_valid[i-1];
        pipe_data[i]  <= pipe_data[i-1];
      end
      pipe_valid[0] <= accept;
      pipe_data[0]  <= echo;

      // Occupancy: up on an accepted request, down on a response. Both can
      // happen in one cycle, which is why this is written as a single
      // expression rather than two conditionals.
      occ <= occ + CNT_W'(accept) - CNT_W'(rsp_valid);
    end
  end

  // The response emerges from the deepest stage of the delay line.
  assign rsp_valid   = pipe_valid[LATENCY_CYCLES-1];
  assign rsp_data    = pipe_data [LATENCY_CYCLES-1];
  assign outstanding = 32'(occ);

endmodule

Interface and cycle behaviour. A request is accepted on a cycle where req_valid and req_ready are both high. LATENCY_CYCLES later, rsp_valid rises with the corresponding data. With MAX_OUTSTANDING at one, req_ready falls immediately after an accept and stays low until the response arrives — so the requester cannot issue again while waiting, which is the single-outstanding case §6 is about. Raise the parameter and req_ready stays high for several accepts, which is §7.

Design decisions worth naming. The delay line is a shift register rather than a counter because it naturally supports several requests in flight — a counter would model one. Occupancy is updated as a single expression including both the accept and the response terms, because both can occur in the same cycle and writing them as separate conditionals is how an off-by-one enters. The response channel has no ready signal: the requester must accept its response when it arrives. That is a simplification worth declaring rather than glossing, because a real interface usually has backpressure in both directions and a model without it cannot exhibit response-path congestion.

Expected behaviour. With LATENCY_CYCLES = 4 and MAX_OUTSTANDING = 1, issue one request and the response arrives four cycles later, with req_ready low throughout. With MAX_OUTSTANDING = 3, issue three back-to-back and all three are accepted in consecutive cycles, with the three responses arriving in consecutive cycles starting four cycles after the first accept — three requests served in a little over one request's latency.

Expected waveform. §6 and §7 are exactly those two experiments.

Synthesis implication. The shift register and counter are synthesizable; at a large LATENCY_CYCLES and DATA_WIDTH the delay line is a great many flip-flops, which is acceptable for a model and would be an absurd way to build a real delay. Nothing here describes how real memory latency arises — it is a stand-in for it.

Limitations, stated because they matter for what conclusions the model supports. Fixed latency, so no state-dependent cost. No bandwidth limit, so §8's saturation cannot be observed with this model as written. No response backpressure. No reordering — responses emerge in request order, whereas real memory systems may return out of order. Each of those absences is a later module, and each is a reason not to draw a quantitative conclusion from this model.

Debugging observations. If req_ready never falls, check that MAX_OUTSTANDING is what you think and that occ is actually being incremented. If responses appear one cycle early or late, count the delay-line stages rather than adjusting the parameter. If occ drifts over a long run, the accept-and-response-in-one-cycle case is the first suspect — which is precisely what P3 in §9 exists to catch.

5. RTL — Latency Becomes Control State

The model waits. Now the other side: what waiting does to the requester, which is where memory latency turns into something an RTL engineer can see in their own block.

What this is. A minimal request/consume state machine — the shape of any block that fetches an operand and then uses it. It is not a CPU pipeline, has no instruction concept, no speculation, no reordering and no prefetch. It is the smallest structure that shows memory latency appearing as cycles spent in a wait state, and it carries the instrumentation that §16's debugging depends on.

// SYNTHESIZABLE. The smallest requester that shows latency as control state.
// NOT a CPU pipeline: no instructions, no speculation, no reordering.
module simple_requester #(
  parameter int ADDR_WIDTH = 16,
  parameter int DATA_WIDTH = 32
) (
  input  logic                  clk,
  input  logic                  rst_n,

  input  logic                  start,
  input  logic [ADDR_WIDTH-1:0] start_addr,

  // To the memory model of §4.
  output logic                  req_valid,
  output logic [ADDR_WIDTH-1:0] req_addr,
  input  logic                  req_ready,
  input  logic                  rsp_valid,
  input  logic [DATA_WIDTH-1:0] rsp_data,

  output logic                  work_done,
  // THE MEASUREMENT. Cycles this requester spent unable to proceed because
  // it was waiting on memory. Without it, "the memory is slow" is an
  // opinion; with it, it is a number (§14).
  output logic [31:0]           stall_cycles
);

  typedef enum logic [1:0] {
    S_IDLE,     // nothing to do
    S_ISSUE,    // presenting a request, waiting for it to be ACCEPTED
    S_WAIT,     // request accepted, waiting for the RESPONSE
    S_CONSUME   // response in hand, doing the dependent work
  } state_e;

  state_e state;
  logic [DATA_WIDTH-1:0] operand;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      state        <= S_IDLE;
      req_valid    <= 1'b0;
      req_addr     <= '0;
      work_done    <= 1'b0;
      stall_cycles <= '0;
    end else begin
      work_done <= 1'b0;

      unique case (state)
        S_IDLE: begin
          if (start) begin
            req_valid <= 1'b1;
            req_addr  <= start_addr;
            state     <= S_ISSUE;
          end
        end

        // Two distinct kinds of waiting, deliberately separated: waiting to
        // be ACCEPTED is contention (something else has the resource), and
        // waiting for a RESPONSE is latency. Collapsing them into one state
        // is why so many designs cannot tell the two apart in a trace.
        S_ISSUE: begin
          stall_cycles <= stall_cycles + 1'b1;
          if (req_ready) begin
            req_valid <= 1'b0;
            state     <= S_WAIT;
          end
        end

        S_WAIT: begin
          stall_cycles <= stall_cycles + 1'b1;
          if (rsp_valid) begin
            operand <= rsp_data;
            state   <= S_CONSUME;
          end
        end

        S_CONSUME: begin
          // The dependent work. One cycle here stands in for whatever the
          // operand was needed FOR -- the point is that it could not start
          // until the operand arrived.
          work_done <= 1'b1;
          state     <= S_IDLE;
        end

        default: state <= S_IDLE;
      endcase
    end
  end

endmodule

The design decision that carries the lesson. S_ISSUE and S_WAIT are separate states because they represent different problems with different fixes. Cycles in S_ISSUE mean the memory system would not accept the request — contention, or an outstanding limit already reached — and the fix is on the capacity or arbitration side. Cycles in S_WAIT mean the request was accepted and the answer has not come back — latency — and the fix is concurrency, locality, or a shorter path. A design that lumps both into one "waiting" state destroys the distinction its own performance analysis needs, and §16 shows what that costs during a real investigation.

Expected behaviour. With the §4 model at LATENCY_CYCLES = 4 and MAX_OUTSTANDING = 1, a single start produces one accept, four cycles in S_WAIT, one cycle of work_done, and stall_cycles advancing by five. Crucially, issuing a second request immediately after the first yields the same five-cycle cost again — because this requester has one request in flight at a time, so nothing overlaps.

Synthesis implication. A four-state machine, a data register and a counter — a few tens of flip-flops plus the counter. The stall_cycles counter is the sort of instrumentation that is worth keeping in production silicon rather than being a simulation-only convenience, because it is what makes a system's memory behaviour observable in the field.

Limitations. One outstanding request by construction. No prefetch, so nothing is fetched before it is needed. No way to do other work while waiting — which is the single biggest simplification and the reason this requester is maximally exposed to latency. Real machines are built specifically to avoid that, and §7 is the first mechanism for doing so.

6. The Waveform — Latency, Fully Exposed

Cycle 1 — accepted. req_valid and req_ready are both high, so the request is taken. From here the requester's own behaviour is out of its hands.

Cycles 2 to 5 — the exposed latency. Four cycles in which this requester does nothing. Note req_ready is low throughout: with one outstanding request allowed, the requester cannot even ask for anything else. The waiting is not merely idle time; it is idle time during which no progress of any kind can be initiated.

Cycle 5 into 6 — the response, and the work. The operand arrives and the dependent computation runs, one cycle after the response. The total cost of one operand was five cycles of which one was useful.

And the conclusion to draw. Making the compute twice as fast changes the CONSUME cycle and nothing else. Four of the five cycles are unaffected. This single figure is the memory wall: a requester structured this way cannot benefit from compute improvements, because compute is not what it is spending its time on. Everything else in this chapter is a way out of this picture.

7. Memory-Level Parallelism — Overlapping the Waiting

The first way out is not to make memory faster. It is to wait for several things at once.

If a workload has accesses that do not depend on each other, there is no reason to serialise them. Issue the second before the first has returned, and the second request's waiting happens during the first's. The waiting does not disappear; it overlaps. This is memory-level parallelism, and it is one of the most important ideas in memory-system performance.

Cycles 0 to 2 — three accepts. Each request is taken on consecutive cycles, because the model now permits three in flight. outstanding climbs.

Cycle 3 — the limit. req_ready falls: the requester has reached its concurrency limit and must wait to issue a fourth. The limit, not the latency, is now what stops it — a different bottleneck with a different fix.

Cycles 4 to 6 — three responses, back to back. Three requests completed in a little over the latency of one. Compare with §6: a requester issuing these three serially would have paid three full latencies.

What this does and does not buy. It converts latency into throughput. The elapsed time for any single request is unchanged — request A still took exactly as long as it would have alone. What changed is that the machine got three answers for roughly the price of one wait. Which leads directly to the limitation that people miss.

8. Why Overlap Does Not Remove a Dependency

Memory-level parallelism requires independent requests. When one request's address depends on a previous request's data, there is nothing to overlap: the second address does not exist until the first response arrives.

This is the structural difference between two kinds of memory-bound workload, and it is the most useful classification in the chapter.

Independent accesses — striding through an array, processing a batch, streaming a buffer. Addresses are computable in advance, so many requests can be in flight, latency overlaps into throughput, and the workload's limit becomes bandwidth or the concurrency limit. These workloads respond well to more outstanding requests, wider interfaces and higher transfer rates.

Dependent accesses — following a chain of references, where each step's address comes from the previous step's data. Concurrency is unavailable in principle, not by implementation: there is exactly one request that can be in flight, and the workload pays the full latency for every step. These workloads respond to lower latency and to nothing else. More bandwidth does not help them at all, and neither does a larger outstanding limit.

Which is why bandwidth and latency have separate solutions, restated with the mechanism visible. A dependent chain is a latency problem and cannot be parallelised away. An independent stream is a bandwidth problem and is barely affected by latency. A real workload is a mixture, and its position in that mixture is what determines which hardware improvement will help it. Ask which kind of waiting dominates before choosing what to improve.

9. Assertions for the Latency Model

The model has a contract, and three properties pin the parts a later change could quietly break.

// VERIFICATION-ONLY, inside memory_latency_model.

// P1 -- no response without an outstanding request. The safety property of
// the whole interface: a response the requester did not ask for would be
// consumed as if it were an answer, corrupting whatever it was matched to.
property p_rsp_requires_outstanding;
  @(posedge clk) disable iff (!rst_n)
    rsp_valid |-> (outstanding != 0);
endproperty
assert property (p_rsp_requires_outstanding);

// P2 -- the outstanding limit is actually enforced. This is the contract
// MAX_OUTSTANDING claims, and it is what the requester relies on when it
// stops issuing. A model that accepted beyond its limit would silently
// change the experiment §7 is trying to run.
property p_limit_respected;
  @(posedge clk) disable iff (!rst_n)
    (outstanding == MAX_OUTSTANDING) |-> !req_ready;
endproperty
assert property (p_limit_respected);

// P3 -- accounting integrity: occupancy changes ONLY by an accept or a
// response, and by exactly one each. This catches the accept-and-respond-
// in-one-cycle case, which is the bug that makes a long run drift and is
// invisible in a short one.
property p_occupancy_accounting;
  @(posedge clk) disable iff (!rst_n)
    ##1 (outstanding ==
         $past(outstanding) + $past(req_valid && req_ready) - $past(rsp_valid));
endproperty
assert property (p_occupancy_accounting);

What each buys. P1 is the safety half of the request/response contract — the one whose violation corrupts data rather than merely slowing things down. P2 asserts that the model's headline parameter means what it says, which matters because this chapter's conclusions are drawn by varying it; an unenforced limit would make §7's figure a fiction. P3 is the quietest and most valuable: occupancy accounting bugs do not fail fast, they drift, and a drifted occupancy either throttles a requester that should be free to issue or permits more in flight than the design intended. It is written as an equality over one cycle rather than as a pair of conditionals precisely because the simultaneous case is where the bug lives.

What they do not claim. Nothing here proves the latency is LATENCY_CYCLES — that needs a property tracking each request's issue cycle against its response cycle, which is straightforward with local variables but only meaningful for an in-order model like this one. And nothing proves responses match their requests, which needs either tagging or the address-echo check a testbench can do directly. Both gaps are worth naming, because a real memory interface reorders responses and the matching question becomes central rather than trivial.

10. Bandwidth Saturation and Queueing

Overlap has a second limit, and it is the one that produces the most confusing performance behaviour in real systems.

Suppose the workload has abundant independent accesses and the requester can keep many in flight. Then requests arrive at the memory system faster and faster. But the memory system can only serve so many per unit time — that capacity is its bandwidth, and it is finite.

What happens as offered traffic approaches that capacity. Requests begin to wait in queues for service rather than being served on arrival. As the queues lengthen, the time each request spends waiting grows — so the observed latency rises, even though nothing about the device changed. Push further and the queues grow further, and the latency observed by every requester degrades together.

The counterintuitive consequence, and the reason this section exists: adding concurrency improves throughput right up until the system saturates, and then it makes latency worse without improving throughput, because the extra requests simply queue. A design tuned by increasing outstanding requests until throughput stops improving has usually gone past the point where latency began degrading badly — and if anything in the system is latency-sensitive, that is a poor operating point.

Stated at the level this chapter can support honestly. The mechanism above — utilisation rising, queues growing, waiting time growing with them — is a general property of queueing systems and it is directionally reliable. The shape of the relationship, and any quantitative prediction of latency against utilisation, requires queueing theory with stated assumptions about arrival and service distributions that this chapter has not developed and that real memory systems only partially satisfy. So: learn the mechanism and measure the system. Do not carry a formula from here.

And notice that latency now has three separate contributors, all of which a naive measurement lumps together: the device's own access cost, the queueing delay caused by other traffic, and the scheduling decisions that chose an order. Only the first is a property of the memory. The repository's AXI track develops the interconnect side of this in Bandwidth and Throughput and Latency Analysis, and Why Outstanding Transactions Exist is the concurrency mechanism seen from the bus.

11. Locality Decides How Exposed a Workload Is

Chapter 1.1 established locality and the working set; this is the one consequence that belongs here, without re-teaching cache architecture.

The memory wall is only reachable by requests that escape the cache hierarchy. So locality is not a separate topic from the memory wall — it is the variable that decides how much of the workload ever encounters it.

Good locality means most accesses are satisfied near compute, few requests reach main memory, and the long path of Figure 1 is travelled rarely. Such a workload can run on a system with mediocre memory performance and barely notice.

Poor locality means the opposite: requests escape constantly, the long path dominates, and the memory subsystem's latency, bandwidth, and scheduling become first-order determinants of performance.

Two workloads on identical hardware can therefore be at completely different points on the same curve, which is the mechanism behind the single most common confusion in performance work: two teams measuring "the memory" on the same machine and reaching opposite conclusions. They were measuring their workloads.

And the engineering lever this exposes is data layout, not hardware. How data is arranged and traversed changes how much of it escapes the caches, and therefore changes exposure to everything in this chapter — often by more than a hardware generation would. That is why the same computation, expressed with a different memory access pattern, can differ enormously in performance with no change to the machine.

12. Verification Perspective

A DV engineer's stake in this chapter is concrete, and it is mostly about stimulus realism.

Stimulus must span the concurrency dimension. A testbench issuing one request at a time verifies the functional path and nothing about behaviour under load. Tests should cover one outstanding request, the design's limit, and the limit being reached — because backpressure at the limit is a distinct code path and a common source of protocol errors.

Latency must be varied, including adversarially. A memory model with fixed latency hides an entire bug class. Vary it, make it non-uniform, and include the extremes: immediate responses, very long waits, and — crucially — latency that changes while requests are in flight, which is what real state-dependent memory does.

Queue boundaries are the highest-yield targets. Full, empty, one entry from full, and the transitions between them. Combined with backpressure, these are where the real defects concentrate.

Starvation and fairness need explicit tests. With several requesters and an arbiter, does every requester eventually progress? A latency histogram per requester is the measurement; a maximum-wait assertion is the check. A design that is functionally correct and starves one requester will pass a functional suite completely.

Ordering must be checked, not assumed. If the interface permits out-of-order responses, the scoreboard must match by tag rather than by arrival order, and the tests must actually produce reordering. A scoreboard that implicitly assumes order will pass against an in-order model and fail in the field.

Performance properties belong in the regression. If the design claims a bounded latency or a sustained throughput, something in the regression must fail when it is not met. Bounded-latency assertions like P4 in Chapter 1.4 §9 are the mechanism; a counter-based throughput check over a window is the other. A performance requirement nobody checks is a comment.

13. Performance Reasoning — Separating the Variables

Here is the decomposition the chapter has been building. Seven variables, each independently measurable, each with a different fix.

Execution throughput. What the compute could do with operands always available. The theoretical ceiling, and the number most often quoted and least often achieved.

Locality / hit rate. What fraction of accesses are satisfied without reaching main memory. Controlled by data layout, traversal order and cache capacity.

Memory latency. How long one access takes once it escapes. Controlled by the device, the path, and the device's state.

Memory bandwidth. How much data per unit time the memory system can sustain. Controlled by the interface and the number of channels.

Outstanding concurrency. How many requests can be in flight at once. Controlled by the requester's design and the interconnect's capability — and capped in principle by whether the workload's accesses are independent at all.

Queueing. Additional waiting caused by other traffic. Controlled by utilisation, which is to say by all of the above together.

Access pattern. How the workload's addresses are distributed, which determines both locality and how much the device is made to pay for its access sequence.

Now the question the chapter opened with can be answered properly. Why does increasing compute capability stop translating into performance? Because raising execution throughput improves only the first variable. If a workload spends most of its time waiting — because locality is poor, because its accesses are dependent, because the memory system is saturated, or because its pattern forces expensive device behaviour — then the first variable was not the constraint, and improving it changes almost nothing.

And the corresponding discipline: before improving anything, measure which variable is binding. The seven are separable and each has its own observable. Guessing among them is what produces a generation of engineering effort spent on the wrong axis.

14. Common Misconceptions

"A faster CPU always means a faster application." Wrong model: performance is compute capability. Consequence: effort and cost spent on execution throughput that a memory-bound workload cannot use. §6's figure is the proof: four of five cycles were unaffected by how fast the compute was. Corrected model: performance is compute capability multiplied by the fraction of time operands are available. Raising the first term while the second is small is nearly free of benefit — and the second term is what §13's variables determine.

"A high DDR data rate means low memory latency." Wrong model: the headline number describes the memory's speed. Consequence: an expectation of latency improvement from a generational upgrade, and puzzlement when a latency-sensitive workload barely moves. Corrected model: the data rate governs the transfer segment of §3's timeline. Queueing, scheduling and the device's internal sequence are largely unaffected by it. Transfer rate and access latency are different quantities that have historically improved at very different rates.

"More bandwidth fixes every memory bottleneck." Wrong model: memory performance is one quantity. Consequence: additional channels or a faster interface bought for a workload that is limited by dependent-access latency, where the improvement is close to zero. Corrected model: bandwidth helps a workload waiting for enough values. It does nothing for one waiting for a particular value. §8's classification decides which you have, and it is decided by the workload's dependence structure, not by the hardware.

"The memory wall is a DRAM problem." Wrong model: the device is the deficiency. Consequence: the actual binding variable — locality, concurrency, queueing, access pattern, controller scheduling — goes uninvestigated, and the same disappointing result recurs on faster hardware. Corrected model: it is a system phenomenon arising from differential scaling, workload locality and finite memory-system capability. The device is one of seven variables, and frequently not the binding one. The decisive evidence is that the same device performs well on other workloads.

"Caches eliminate main-memory latency." Wrong model: a cache removes the problem rather than reducing how often it is met. Consequence: architectures sized on the assumption that misses are rare, which fail on workloads whose working set does not fit or whose access pattern has no reuse to exploit. Corrected model: caches reduce the frequency of long-path accesses. What remains is exactly the traffic with the worst locality, so the accesses that do reach memory are the ones a cache could not help with — and their cost is fully exposed.

"One benchmark's memory behaviour represents every workload." Wrong model: memory performance is a property of a machine. Consequence: a hardware or configuration decision made on a measurement that does not represent the intended use — often a benchmark with excellent locality, which makes almost any memory system look adequate. Corrected model: exposure to the memory wall is a property of the workload and the system together. A benchmark is a measurement of one point. Characterise the intended workload's locality, dependence structure and access pattern before generalising anything.

15. Debugging — An Accelerator That Cannot Reach Its Peak

Symptom. A compute accelerator with high theoretical arithmetic throughput achieves a small fraction of it on the real workload. No functional errors. Synthetic benchmarks look much better than the application.

The wrong first move is to conclude the memory is too slow. It explains nothing — the synthetic benchmark ran on the same memory — and it suggests nothing actionable. Work the variables instead.

Hypothesis 1 — poor locality: most accesses are reaching main memory. Inspect: the miss rate or the count of requests leaving the cache hierarchy per unit of work. Expected evidence if true: a miss rate far above the synthetic benchmark's. Discriminates because: it is the top of the causal chain — if few requests escape, nothing below matters, and if many do, this is the first thing to fix and often the cheapest, since data layout is software.

Hypothesis 2 — dependent accesses: the workload cannot keep requests in flight. Inspect: the average outstanding count while the accelerator is stalled. Expected evidence if true: outstanding is low — often one — while the engine waits. Discriminates because: low occupancy with a waiting requester is unambiguous, and it points somewhere completely different from every other hypothesis: the limit is the workload's dependence structure, and no amount of extra bandwidth or extra concurrency capability will help.

Hypothesis 3 — insufficient concurrency capability: the workload could keep more in flight but the design cannot. Inspect: cycles where a request was ready to issue but req_ready was low, and whether the outstanding count is pinned at the design's limit. Expected evidence if true: occupancy sitting exactly at the limit with requests waiting to issue. Discriminates from hypothesis 2 by: the reason occupancy is low. Pinned at the limit means the design is the constraint; sitting at one with capacity to spare means the workload is. These two look identical in a throughput number and have opposite fixes — which is precisely why §5 separated S_ISSUE from S_WAIT.

Hypothesis 4 — queueing or backpressure elsewhere in the path. Inspect: queue occupancies along the path, and where backpressure originates. Expected evidence if true: a queue persistently near full with the stall propagating upstream from it. Discriminates because: it localises the bottleneck to a specific structure rather than to "memory", and the fix — resizing, rebalancing, or arbitration — is local.

Hypothesis 5 — bandwidth saturation. Inspect: achieved data rate against the interface's sustainable capability, and whether observed latency rises with offered load. Expected evidence if true: near-capacity utilisation and latency that grows as load grows — the §10 signature. Discriminates because: rising latency under rising load distinguishes saturation from a fixed-latency problem, and it means adding concurrency will make latency worse rather than better.

Hypothesis 6 — the access pattern is making the device pay repeatedly. Inspect: the hit/miss classification of Chapter 1.4 at the memory controller, and whether independent streams are interleaving. Expected evidence if true: a high row-change rate, especially with the streams-interleaving signature where each stream alone is fast. Discriminates because: it is the only hypothesis fixed by changing address mapping or request ordering rather than by changing capacity or concurrency.

Root-cause discrimination in three counters. Requests escaping the cache hierarchy, average outstanding count while stalled, and achieved bandwidth against capability. High escapes points at 1. Low outstanding with capacity spare points at 2. Outstanding pinned at the limit points at 3. High bandwidth utilisation with rising latency points at 5. Path queues full points at 4. A high row-change rate at the controller points at 6.

What this scenario is really teaching. Not the hypotheses — the method. Name the variables, find the observable that separates them, measure before changing anything. The alternative is to improve something plausible and hope, which on a memory-bound workload is usually improving execution throughput that was never the constraint.

16. Interview Reasoning

"What is the memory wall?" The system-level consequence of three things together: compute capability and main-memory access latency having improved at very different rates, workloads having limited locality, and the memory system having finite latency and finite bandwidth. Beyond some point, added compute capability produces diminishing performance because it spends its time waiting for operands. A strong answer immediately rejects "DRAM is slow": the same device serves other workloads well, so the device is not the explanation, and the useful statement names which of the variables is binding.

"Why does high bandwidth not imply low latency?" Because they govern different parts of a request. Latency is the whole interval a requester waits — queueing, scheduling, the device's internal sequence, and the data transfer. Bandwidth governs only the transfer segment. Raising the transfer rate barely shortens a request dominated by the earlier segments. Conversely, a workload with many independent accesses cares almost nothing about individual latency and almost entirely about aggregate data rate. The diagnostic question is whether the workload waits for a particular value or for enough values.

"How can a workload with a long memory latency still achieve high throughput?" By having independent accesses and keeping many in flight, so the waits overlap. Latency becomes throughput: several requests complete in a little more than the time one takes. The essential qualification is that this requires independence — a chain of dependent accesses cannot be overlapped in principle, because the next address does not exist until the previous response arrives. That distinction is what separates a workload that responds to bandwidth from one that responds only to latency.

"You increase outstanding requests and throughput improves, then latency gets much worse. What happened?" The memory system approached saturation. While there was spare capability, extra concurrency converted waiting into throughput. Once offered traffic approached what the system can sustain, additional requests began queueing rather than being served, so waiting time — and therefore observed latency — grew while throughput stopped improving. If anything in the system is latency-sensitive, the throughput-maximising operating point is the wrong one, and the tuning target should be latency at an acceptable utilisation rather than peak throughput.

"Would doubling the clock frequency of a memory-bound processor double its performance?" No, and the reasoning is the point. It would halve the time spent computing and leave the time spent waiting for memory essentially unchanged — the waiting is set by the memory path, not by the core's clock. Worse, at the higher frequency the same wait is a larger number of cycles, so the core appears to stall more. The benefit depends entirely on what fraction of time was spent computing rather than waiting, which is a property of the workload and must be measured.

"An accelerator hits a fraction of its peak. What do you measure first?" Three counters: how many requests escape the cache hierarchy per unit of work, the average outstanding count while the engine is stalled, and achieved bandwidth against the interface's capability. Those three separate poor locality from a dependence-limited workload, from a concurrency-limited design, from saturation. The instinct to check the memory's data rate first is exactly the wrong order — it is the last thing that distinguishes anything.

17. Engineering Check

System A — high compute throughput, small working set, strong locality. System B — identical compute throughput, large irregular working set, frequent cache misses whose addresses depend on previously fetched data.

1. Which is more exposed to main-memory latency, and why? B, decisively, for two independent reasons that compound. Its working set does not fit, so a large fraction of its accesses reach main memory in the first place. And its misses are dependent, so it cannot overlap them — each access must complete before the next address is known. A's accesses mostly never leave the cache hierarchy, so it barely meets the long path at all.

2. Would doubling arithmetic throughput double performance? For A, plausibly close to it, because A spends most of its time computing — although it will run into some other limit eventually. For B, almost certainly not: doubling the rate at which it could consume operands does nothing about the rate at which operands arrive, and arrival is what it is waiting on. This is §6's figure applied to a system-level question.

3. Would additional outstanding requests help? For B, no — and this is the subtle part worth getting right. Its misses are dependent, so there is no second independent request available to issue. Concurrency capability cannot be used by a workload that has no independence to exploit. If B's misses were independent, the answer would flip completely and concurrency would be the single most effective change. The workload's dependence structure, not the hardware's capability, decides whether concurrency is available at all.

4. When would bandwidth become the next limitation? Once enough independent requests are in flight to keep the memory system busy — which for B means restructuring the algorithm or data layout so its accesses stop being dependent. Only after concurrency is being exploited does aggregate data rate become the binding constraint. Buying bandwidth before that point buys nothing.

5. What would you change in B to improve it, in order of expected effect? First, data layout and traversal order, to convert dependent irregular accesses into independent regular ones — this attacks both locality and dependence at once and is usually software. Second, exploit whatever independence then exists by allowing more requests in flight. Third, and only then, consider bandwidth. Notice that the most effective change is not a hardware change, which is the honest conclusion of this chapter.

6. What signals or counters would you inspect in RTL or simulation? Requests escaping the cache hierarchy per unit of work; the outstanding count while the requester is stalled; cycles spent waiting to be accepted versus waiting for a response — the S_ISSUE/S_WAIT split of §5, which separates contention from latency; achieved bandwidth against capability; queue occupancies along the path; and the row hit/miss classification at the memory controller. Six observables, and between them they identify which of §13's variables is binding.

18. Module 1 in One Chain

This closes the module, so the right ending is the argument rather than eight summaries.

The hierarchy exists because the properties wanted from memory — speed, capacity, cost per bit, power, persistence, closeness to compute — cannot be maximised together. A system stops trying to build one memory and builds several tiers, moving data between them.

Registers are storage relocated into the execution path and named by the instruction encoding. They are the fastest thing a machine has, and the properties that make them fast — per-entry access, per-bit generality, adjacency to compute — are exactly the ones that cannot scale.

SRAM makes the first density trade: a smaller bistable cell, and a periphery shared across many cells so decoding and sensing stop being paid per bit. The price of sharing is exclusivity, which is why access becomes scheduled, can be made to wait, and needs backpressure.

DRAM makes the trade again, maximally: one transistor and one capacitor, a cell that does no work. Three consequences follow — charge leaks so refresh is mandatory, reading destroys so a restore is part of reading, and the access becomes a sequence whose cost depends on which row is already resolved.

Flash buys persistence and pays in a write that is not the inverse of a read, an erase granularity far larger than a word, and finite endurance — which is why it is a storage tier and not a memory tier.

Cost vs density explains why no technology wins everything: cost per bit is a chain from cell area through array efficiency, yield and repair, test, packaging and volume; density is bought by moving costs out of the per-bit term, which means sharing, which means exclusivity; and several axes are coupled by physics so no effort separates them.

DRAM dominates main memory not by winning an axis but by satisfying a conjunction: word-granular access, unlimited symmetric rewriting, capacity at a price, tolerable latency, sufficient bandwidth, and a standardised interface an industry can build both sides of. The cell won the physics; standardisation won the tier.

And the memory wall is the bill. The tier that satisfies those requirements is necessarily far from compute, sequenced, and state-dependent — so a machine's performance comes to depend less on what it can compute than on how often it has operands. That is not a defect in DRAM. It is the cost of the capacity, and managing it is the discipline the rest of this curriculum teaches.

19. What Comes Next

Module 1 answered why DRAM exists. Every mechanism that answer depended on was named and then deliberately left unopened.

Charge leaks — but how much, how fast, and what determines the interval? Reading destroys — but what exactly happens on the bitline, how does a sense amplifier resolve a signal that small, and what does restoring actually involve? The access is a sequence — but what are its steps called, what are the minimum times between them, and where do those times come from?

Module 2 opens the cell. Capacitor storage, charge and leakage, the refresh requirement, the 1T1C cell itself, the destructive-read property, and restore operations — each as its own chapter, with the physics this module was careful not to fabricate. Everything after it rides on those foundations: the array in Module 3, the generations in Module 4, the device organisation in Module 5, and then signals, commands, addressing and timing.

You now know why the tier has to exist, what it costs, and why the standard is as large as it is. The rest is learning how it works.

Browse the full path on the DDR tutorials index. For the interconnect side of latency and concurrency, see Latency Analysis, Bandwidth and Throughput and Why Outstanding Transactions Exist. For the tier above, Cache Hierarchy Review.