DDR3

Chapter 4.3 left two debts. On-die termination existed but was uncalibrated, so its realised value drifted with process, voltage and temperature. And the shared bus topology was already the limiting factor at DDR2's rates, managed by careful routing rather than solved.

DDR3 deepens the prefetch to 8n, which puts the interface clock at four times the array's access rate — and both debts come due at once. The responses are a calibration engine on the die and a change of bus topology, and the topology change has a consequence so large that it defines the next fifteen years of memory-interface engineering.

Because DDR3's new routing makes the clock arrive at a deliberately different time at every device. A synchronous interface's most basic assumption is that the clock arrives when you think it does. DDR3 breaks that assumption on purpose, and the response — the controller must measure its own connection to each device before it can use it — is why training exists at all.

1. The First Debt — Termination That Does Not Know Its Own Value

An on-die termination resistance is built from transistors, and a transistor's conductance depends on the process it was manufactured on, the voltage it is supplied with and the temperature it is operating at. So a device asked for a particular termination value delivers approximately that value, with a spread.

At DDR2's rates the spread fits in the margin. At DDR3's — where the transfer interval has halved again — it does not. Termination that is off by a significant fraction reflects more than it should, and the residue lands in a budget that has no room for it.

The response is to calibrate the on-die impedance against something that does not drift.

DDR3 devices have a dedicated ZQ pad connected through an external precision resistor — nominally 240 Ω with a 1% tolerance — to a supply reference. A calibration engine on the die adjusts its output driver and termination impedance until they match a defined ratio to that external reference. The controller issues a calibration command, the device performs the adjustment, and the result is an on-die impedance traceable to a component whose value does not move with the die's process, voltage or temperature.

2. The Second Debt — A Bus Topology That Cannot Go Faster

Chapter 4.3 §3 established that every device attached to a shared bus is a discontinuity. The question DDR3 faced is what to do when careful termination is no longer sufficient.

The problem is worst on the command, address and clock signals, and for a structural reason: those signals go to every device, while data signals can be organised so that each device owns its own subset. A signal that must reach every device on a module is the most heavily loaded signal in the system, and it is the one whose integrity fails first.

The previous approach routed such a signal to a branch point and then to each device — short stubs off a common trunk. Every stub is an unterminated open end that reflects, and stubs that were electrically negligible become electrically significant as the rate rises.

DDR3's response is to eliminate the stubs by routing the command, address and clock signals as a single daisy chain that passes each device in turn and terminates once at the far end. This is called fly-by topology. Data and strobe signals stay point-to-point, because they do not need to reach every device.

Fly-by solves the integrity problem cleanly. One trace, one termination, no stubs, and each device presents a load in series along a controlled line rather than at the end of an open branch. The command and clock bus can run at DDR3's rates.

And it creates a problem that is not electrical at all.

3. The Consequence — The Clock No Longer Arrives When You Think It Does

A signal traversing a daisy chain reaches the first device before the second, the second before the third, and so on. So the clock arrives at each device at a different time, by an amount set by the board's routing rather than by anything either endpoint knows.

Now recall Chapter 4.2 §3: data is timed by DQS, which is point-to-point, while commands are timed by CK, which travels the chain. Those two signals now have a relationship that differs at every device, and that relationship is exactly what a write needs in order to land correctly: the device captures write data using DQS but interprets the write command against CK.

The controller cannot compute this offset. It depends on trace lengths, on the board's construction, on how many devices are populated and where. It is a property of the assembled system, not of the parts.

So the controller must measure it. DDR3 defines a mode in which the device reports back the relationship it observes between the strobe the controller is sending and the clock the device is receiving. The controller sweeps the strobe's timing, watches the reported response, finds the alignment point, and then applies a per-device delay so that each device's write data arrives correctly relative to the clock that device sees. This is write leveling.

The read direction needs its own procedure for a related reason: on a read, the device drives DQS and DQ, and the round-trip timing back to the controller's receivers again depends on the assembled board and on which device is responding. The controller sweeps its own capture timing against known data and finds the centre of the valid window — read leveling, with per-device and eventually per-bit adjustment.

One clarification that prevents a common error. Write leveling exists because fly-by deliberately skews the clock. It is not a fix for a routing mistake, and it is not compensating for manufacturing variation in the parts. The skew is the intended result of a topology chosen for signal integrity, and leveling is the mechanism that makes that choice affordable. Describing write leveling as "correcting board skew" gets the causality backwards and makes DDR3's design look like a patch rather than a trade.

4. The Generation, With Layers Labelled

Four generations in, the last row still reads "unchanged". The array has not moved. Everything in this module so far is interface, topology and device-architecture work wrapped around substantially the same kind of DRAM array — and that is the single most useful fact to carry out of Module 4.

5. Burst Chop — The First Admission That Granularity Costs

Follow Chapter 4.3 §8's number to this generation:

   minimum granularity  =  prefetch_depth  ×  interface_width

   DDR3:  8n × 64 bits  =  64 bytes  on a 64-bit module
                        =  EXACTLY one typical cache line

The prefetch depth has grown into the size of the thing the requester actually wants. That is not yet a disaster — a cache line is a very common access size, so a full burst is usually exactly right. But it means there is now no smaller access available, and some requesters genuinely want less.

DDR3's response is burst chop: a mode in which a command transfers half the full burst length rather than all of it. The controller can request a chopped burst when the full quantity is not wanted.

And the honest part is what burst chop does not save. The array still performed the full-width read — Module 3 established that a whole row participates regardless, and the prefetch register is filled by the array's internal access, not by the burst length. Burst chop saves interface time and interface switching energy. It does not save array work.

   full burst:     8 beats transferred,  8 beats useful   -> 100% efficient
   chopped burst:  4 beats transferred,  4 beats useful   -> 100% efficient
                   ... but the array read 8 words either way.

   full burst for a 2-beat request:   8 transferred, 2 useful  ->  25%
   chopped for a 2-beat request:      4 transferred, 2 useful  ->  50%

So burst chop is mitigation, not a solution. It halves the interface waste on small accesses and leaves the array waste untouched. That distinction is why deepening prefetch further was not the answer to DDR4's bandwidth problem — chop can hide granularity on the interface, but the array's minimum work only grows.

6. RTL — Choosing a Burst Length and Measuring the Waste

Problem

A controller receiving requests of varying size must choose, per request, between a full burst and a chopped burst, transfer no fewer beats than were asked for, and report how much of what it transferred was actually wanted.

The reporting half is the part worth building. Access efficiency is a measured quantity, not an assumed one, and a controller that does not instrument it cannot tell a bandwidth problem caused by the access pattern from one caused by the interface — which is precisely Chapter 1.8's separation, implemented.

Classification

SYNTHESIZABLE RTL. Burst-length selection with saturating performance counters — exactly the kind of logic a real memory controller contains, including the telemetry.

What it does not model. No array, no prefetch register, no beats — this block decides and counts; the transfer itself is 4.3 §5's serialiser. And nothing here touches fly-by, leveling, calibration or termination, because §1 to §3's mechanisms are analog and topological and no register represents them.

Interface

req_valid with req_beats presents a request measured in useful beats. sel_beats is what will actually be transferred, waste_beats the difference, sel_chop the mode chosen. window_done pulses with useful_acc and xfer_acc valid, giving efficiency over a window. req_illegal reports a request the burst structure cannot express.

How to simulate it

vlog burst_length_select.sv tb_burst_length_select.sv then vsim -c tb_burst_length_select -do "run -all"; VCS vcs -sverilog burst_length_select.sv tb_burst_length_select.sv && ./simv; Xcelium xrun -sv burst_length_select.sv tb_burst_length_select.sv.

// ─────────────────────────────────────────────────────────────────────────
// BURST LENGTH SELECT.  Classification: SYNTHESIZABLE RTL.
//
// Chooses a full or chopped burst per request, and MEASURES the resulting
// access efficiency over a window. The measurement is the point: burst chop
// mitigates the granularity cost of a deep prefetch on the interface, and
// whether it is helping is an empirical question about the access pattern
// that only instrumentation can answer.
//
// WHAT THIS DOES NOT MODEL: fly-by topology, clock skew across devices,
// write or read leveling, ZQ calibration, termination, driver impedance.
// Those are ANALOG AND TOPOLOGICAL properties of the board and the PHY, and
// Sections 1 to 3 treat them in prose and diagrams because no register
// represents them.
//
// It also does not transfer anything -- prefetch_serializer (Chapter 4.3)
// does that. This block decides and counts.
// ─────────────────────────────────────────────────────────────────────────
module burst_length_select #(
  // Full burst length in beats. 8 is DDR3's BL8.
  parameter int  FULL_BEATS = 8,
  // Chopped burst length. DDR3's chop is half the full length.
  parameter int  CHOP_BEATS = FULL_BEATS / 2,
  // 0 models a generation with no chop available (DDR1, DDR2), which makes
  // the cost of a deep prefetch WITHOUT mitigation directly observable in
  // the same testbench.
  parameter bit  ALLOW_CHOP = 1'b1,
  // Requests per telemetry window.
  parameter int  WINDOW     = 16,
  // Width of the accumulators. Saturating, so a long run cannot silently
  // wrap and report a flattering efficiency.
  parameter int  ACC_W      = 16,
  // DERIVED. Wide enough to hold FULL_BEATS itself, hence the +1.
  parameter int  BEAT_W     = $clog2(FULL_BEATS + 1),
  parameter int  WIN_W      = (WINDOW <= 1) ? 1 : $clog2(WINDOW)
) (
  input  logic              clk,
  input  logic              rst_n,

  input  logic              req_valid,
  // Beats the requester actually wants. Legal range 1..FULL_BEATS.
  input  logic [BEAT_W-1:0] req_beats,

  output logic              sel_valid,
  // Beats that will actually be transferred: never fewer than req_beats.
  output logic [BEAT_W-1:0] sel_beats,
  // sel_beats - req_beats: transferred and discarded.
  output logic [BEAT_W-1:0] waste_beats,
  output logic              sel_chop,
  output logic              req_illegal,

  // Telemetry. Valid on the window_done pulse; see the sampling contract
  // note in the sequential block.
  output logic              window_done,
  output logic [ACC_W-1:0]  useful_acc,
  output logic [ACC_W-1:0]  xfer_acc
);

  // ── COMPILE-TIME legality.
  if (FULL_BEATS < 2) begin : g_full_min
    initial $fatal(1, "burst_length_select: FULL_BEATS must be >= 2");
  end
  if (CHOP_BEATS < 1) begin : g_chop_min
    initial $fatal(1, "burst_length_select: CHOP_BEATS must be >= 1");
  end
  if (CHOP_BEATS >= FULL_BEATS) begin : g_chop_max
    initial $fatal(1, "burst_length_select: CHOP_BEATS must be < FULL_BEATS");
  end
  if (WINDOW < 1) begin : g_win_min
    initial $fatal(1, "burst_length_select: WINDOW must be >= 1");
  end

  // ── The selection decision. Chop whenever the request fits inside a
  //    chopped burst; otherwise a full burst. Never fewer beats than asked
  //    for, which is why the test is <= and not <.
  logic req_ok, use_chop;
  logic [BEAT_W-1:0] beats_now;

  assign req_ok = req_valid
               && (req_beats != '0)
               && (req_beats <= BEAT_W'(FULL_BEATS));

  assign use_chop  = ALLOW_CHOP && (req_beats <= BEAT_W'(CHOP_BEATS));
  assign beats_now = use_chop ? BEAT_W'(CHOP_BEATS) : BEAT_W'(FULL_BEATS);

  // ── Saturating accumulation. Computed one bit wide than the accumulator
  //    so the carry out IS the overflow flag, then clamped.
  //
  //    WHY SATURATE RATHER THAN WRAP: a wrapped transferred-beat total can
  //    make efficiency look BETTER than it is, and a performance counter
  //    that lies optimistically is worse than one that is obviously pinned.
  logic [ACC_W:0]   u_sum, x_sum;
  logic [ACC_W-1:0] u_sat, x_sat;

  always_comb begin
    u_sum = {1'b0, useful_acc} + (ACC_W+1)'(req_beats);
    x_sum = {1'b0, xfer_acc}   + (ACC_W+1)'(beats_now);
    u_sat = u_sum[ACC_W] ? {ACC_W{1'b1}} : u_sum[ACC_W-1:0];
    x_sat = x_sum[ACC_W] ? {ACC_W{1'b1}} : x_sum[ACC_W-1:0];
  end

  logic [WIN_W-1:0] win_cnt;
  logic             win_last;
  assign win_last = (win_cnt == WIN_W'(WINDOW - 1));

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      sel_valid   <= 1'b0;
      sel_beats   <= '0;
      waste_beats <= '0;
      sel_chop    <= 1'b0;
      req_illegal <= 1'b0;
      window_done <= 1'b0;
      useful_acc  <= '0;
      xfer_acc    <= '0;
      win_cnt     <= '0;
    end else begin
      sel_valid   <= 1'b0;
      req_illegal <= 1'b0;

      // ── SAMPLING CONTRACT: the accumulators hold the completed window's
      //    totals for exactly the cycle window_done is high, and are
      //    cleared on the following cycle. A consumer must sample them on
      //    that pulse. Clearing a cycle late rather than in the same cycle
      //    is what makes the totals observable at all -- clearing them
      //    together with the pulse would publish zeroes.
      if (window_done) begin
        window_done <= 1'b0;
        useful_acc  <= '0;
        xfer_acc    <= '0;
      end

      if (req_ok) begin
        sel_valid   <= 1'b1;
        sel_beats   <= beats_now;
        waste_beats <= beats_now - req_beats;
        sel_chop    <= use_chop;

        // Accumulate on top of whatever the window holds. When the previous
        // cycle ended a window the clear above wins, because it is the
        // earlier assignment to the same registers -- so this request would
        // be lost. The counter therefore never lets a window end on a cycle
        // that also accepts a request: win_last closes the window with THIS
        // request included, and the clear lands on the next cycle when no
        // accumulation is pending.
        useful_acc <= u_sat;
        xfer_acc   <= x_sat;

        if (win_last) begin
          win_cnt     <= '0;
          window_done <= 1'b1;
        end else begin
          win_cnt <= win_cnt + 1'b1;
        end
      end

      if (req_valid && !req_ok) begin
        req_illegal <= 1'b1;
      end
    end
  end

endmodule

Combinational logic

use_chop is the whole selection policy in one comparison, and <= rather than < is what guarantees never transferring fewer beats than requested. The saturating adders compute one bit wide so the carry out is the overflow flag.

Sequential logic

Registered outputs, saturating accumulators, and a window counter. The interesting part is the sampling contract, and the hazard above.

Cycle trace

FULL_BEATS = 8, CHOP_BEATS = 4, ALLOW_CHOP = 1:

Totals: 20 useful beats, 28 transferred — 71% interface efficiency. Re-run with ALLOW_CHOP = 0 and the same requests transfer 40 beats for 50% efficiency. That difference is exactly what burst chop buys, and having it come out of a counter rather than an argument is the point of building the telemetry.

Simulation

A directed test should check sel_beats >= req_beats always; chop chosen exactly when the request fits a chopped burst; waste_beats equal to the difference; a zero or over-length request rejected with no selection; the window pulse arriving every WINDOW accepted requests with totals matching an independently computed sum; and the accumulators pinning rather than wrapping when driven past ACC_W. Then re-run the identical stimulus with ALLOW_CHOP = 0 and compare — that comparison is the block's most informative test.

Synthesis

Two comparators, a multiplexer, a subtractor, two saturating adders and two counters. Small. The saturating adders are the widest logic, and ACC_W is the parameter to tune against the reporting interval you actually need.

Corner cases

req_beats == CHOP_BEATS chooses chop with zero waste — the boundary the <= gets right and a < would get wrong by always promoting it to a full burst. req_beats == CHOP_BEATS + 1 forces a full burst with FULL_BEATS - CHOP_BEATS - 1 wasted beats, the worst case for chop, and worth testing explicitly because it is where a naive policy looks worst. ALLOW_CHOP = 0 disables the policy entirely. WINDOW == 1 pulses every request. Accumulators pin at all-ones rather than wrapping.

Verification

What DV must prove: sel_beats >= req_beats unconditionally — the safety property, since violating it means silently returning less data than asked for; the chop decision matching the boundary exactly; waste arithmetic; illegal requests reported and never selected; window totals matching an independent model; saturation rather than wrap; and the counters not dropping a request at a window boundary, which is the hazard the callout named.

Debugging

If efficiency reads better than the access pattern can justify, suspect the accumulators: either a wrap being read as a small number, or the window-boundary hazard dropping transferred beats. Check xfer_acc against an independently computed total before believing any efficiency figure — a performance counter is a design under test like any other. If chop is never chosen, check ALLOW_CHOP and then the comparison direction. If small requests still transfer full bursts, check that req_beats carries useful beats and not an already-rounded burst length, which is the most common integration error: rounding upstream makes the policy unable to see that anything was small.

Limitations

No knowledge of the array, so it cannot report the array-side waste §5 said chop does not save — the efficiency this block measures is interface efficiency only, and reading it as system efficiency would overstate the benefit. No per-bank or per-row awareness, so it cannot distinguish waste from granularity from waste from row conflicts. No chop beyond a single half-length option. And the window-boundary hazard above is present by construction.

7. Four Assertions Worth Writing

// VERIFICATION-ONLY, inside burst_length_select.

// P1 -- THE SAFETY PROPERTY. Never transfer fewer beats than requested.
// Everything else here is efficiency; this one is correctness, because
// under-transferring silently returns less data than the requester asked
// for and nothing downstream would necessarily notice.
property p_never_short;
  @(posedge clk) disable iff (!rst_n)
    sel_valid |-> (sel_beats >= $past(req_beats));
endproperty
assert property (p_never_short);

// P2 -- the chop boundary is exact. Written as a biconditional so BOTH
// errors are caught: failing to chop when it was possible (lost
// efficiency), and chopping when the request did not fit (lost data, which
// P1 also catches, but from the policy side rather than the outcome side).
property p_chop_boundary;
  @(posedge clk) disable iff (!rst_n)
    sel_valid |-> (sel_chop == (ALLOW_CHOP
                                && ($past(req_beats) <= BEAT_W'(CHOP_BEATS))));
endproperty
assert property (p_chop_boundary);

// P3 -- waste is exactly the difference, never inferred.
property p_waste_exact;
  @(posedge clk) disable iff (!rst_n)
    sel_valid |-> (waste_beats == (sel_beats - $past(req_beats)));
endproperty
assert property (p_waste_exact);

// P4 -- THE TELEMETRY-INTEGRITY PROPERTY. Every accepted request
// contributes to the transferred accumulator, unless the accumulator has
// saturated. This is the assertion that catches the window-boundary hazard
// the Section 6 callout named: a dropped request leaves the data path
// perfectly correct and the counters quietly wrong.
property p_acc_advances;
  @(posedge clk) disable iff (!rst_n)
    (sel_valid && !$past(window_done) && (xfer_acc != {ACC_W{1'b1}}))
      |-> (xfer_acc == ($past(xfer_acc) + ACC_W'(sel_beats)));
endproperty
assert property (p_acc_advances);

// P5 -- an illegal request selects nothing. Report, never repair.
property p_illegal_selects_nothing;
  @(posedge clk) disable iff (!rst_n)
    req_illegal |-> !sel_valid;
endproperty
assert property (p_illegal_selects_nothing);

P1 versus the rest is the lesson. P2, P3 and P4 are about efficiency and measurement; P1 is about correctness. Mixing the two classes in one assertion list without noticing is how a reviewer ends up weighting a lost-efficiency property as heavily as a lost-data property. Know which of your assertions guard correctness and which guard performance, because when one has to be relaxed for a schedule, the answer must not be a coin toss.

P4 asserts a property of the instrument. A performance counter is a design under test. Because its failure mode is silent and misleading rather than broken, it needs assertions more than obviously-visible logic does — and P4's saturation exclusion is deliberate: a pinned counter must not be reported as a bug.

What none of them prove — and this is the chapter's central verification boundary. Nothing here says anything about fly-by skew, write leveling, read leveling, ZQ calibration, termination or margin. §1 to §3 are the substance of DDR3, and no assertion in any simulation can reach them. They are verified by analog and mixed-signal simulation, by measurement on real hardware, and by the training procedure's own convergence checks. Treating a green simulation regression as evidence that a DDR3 interface will work is the most expensive mistake available in this domain, and the size of the gap between §5's RTL and §1's subject matter is the honest measure of it.

8. Efficiency in Cycles

The request for 5 beats is the instructive one. Five does not fit a chopped burst, so a full 8-beat burst is transferred and 3 beats are discarded. One beat more than the chop threshold costs three wasted beats — a sharply non-linear penalty, and exactly the shape of cost that makes access-size distributions matter more than access-size averages.

The request for 1 beat is the other instructive one. Chop helps — 4 transferred instead of 8 — and the result is still only 25% efficient. Chop halves the waste on tiny accesses; it does not make them efficient. Nothing in DDR3 can, because the granularity is set by the prefetch depth and the prefetch depth is what bought the bandwidth.

And xfer_acc reaching 28 for 20 useful beats is the number to take away: 71% interface efficiency on a plausible mixed workload, with chop enabled. Without it, 50%.

9. The New Costs, and the Next Bottleneck

Cost 1 — the interface must be trained before it can be used. A mandatory measurement procedure, per device, per direction, with results that are state to be maintained and re-established. Everything in Modules 19 to 21 descends from this.

Cost 2 — a fault class with no functional signature. Calibration or leveling that completed but converged badly produces a working system with no margin. No simulation finds it, no functional test fails. It is the dominant source of difficulty in real memory bring-up.

Cost 3 — granularity has reached the size of a cache line. Burst chop mitigates the interface half and nothing mitigates the array half.

And the next bottleneck is a wall rather than a pressure. Bandwidth demand keeps rising. The lever that worked for three generations was prefetch depth. Apply it once more:

   16n prefetch on a 64-bit module:
     minimum granularity  =  16 × 64 bits  =  128 bytes
                          =  TWICE a typical cache line

Every access would transfer a line that was wanted and a line that was not. The peak bandwidth number rises and the useful bandwidth does not — and chop cannot fix it, because chopping a 16n prefetch back to 8 beats means the array read 16 words to deliver 8, which is the array waste chop never addressed.

So the prefetch lever is finished. Transfers-per-cycle was exhausted at DDR1. Prefetch depth is exhausted here. Width is still expensive. The clock is still bounded by what the interface can settle and what the array can supply.

Chapter 4.5 is the chapter about what is left. Its answer is the one remaining place to find rate: not by making a single access wider or faster, but by arranging for independent accesses inside the device to overlap — parallelism instead of granularity. That is what bank groups are, and it is why DDR4 is the generation where the prefetch number stops changing.

10. Common Misconceptions

"Write leveling corrects a board design error." Wrong mental model: leveling compensates for skew that should not be there. Engineering action: treating leveling as a workaround; expecting a well-routed board not to need it; trying to eliminate the skew in layout. Observable failure / bad conclusion: attempting to length-match a fly-by bus, which reintroduces the stubs and the loading that fly-by exists to eliminate — a board that is worse in exactly the dimension the topology was chosen for. And an inability to explain why leveling is mandatory rather than conditional. Correct model: fly-by deliberately staggers the clock at each device, as the price of a stub-free, singly terminated command and clock bus. The skew is the intended outcome of a signal-integrity trade, and leveling is what makes it affordable. Prevention: ask what the skew buys. If the answer is "a clean CA bus at rates a stubbed trunk could not reach", it is a trade and not a defect.

"ZQ calibration sets the termination to 240 Ω." Wrong mental model: the external resistor's value is the termination value. Engineering action: expecting a measured on-die impedance of 240 Ω; substituting a differently valued resistor on the assumption it directly sets termination. Observable failure / bad conclusion: a mis-terminated bus with no functional symptom, and confusion when measured impedance does not match the resistor. A wrong reference resistor is a particularly nasty fault because everything still works. Correct model: the external precision resistor is a reference. The device calibrates its driver and termination impedance to a defined ratio against it. The reference's job is to be stable and accurate — not to be the answer. Prevention: ask what the die contributes and what the reference contributes. The die contributes adjustability; the reference contributes accuracy.

"Burst chop makes small accesses efficient." Wrong mental model: chop solves the granularity problem. Engineering action: assuming fine-grained access patterns are fine on DDR3; not measuring interface efficiency; expecting chop to reduce array work. Observable failure / bad conclusion: a system whose measured bandwidth is far below peak on scattered small accesses, with the shortfall attributed to the interface rather than to granularity. And on DDR4-or-later reasoning, the wrong conclusion that deeper prefetch would have been fine because chop exists. Correct model: chop halves interface waste on accesses smaller than a chopped burst. It saves no array work — the array read the full prefetch width regardless, because a whole row participates and the prefetch register is filled by the internal access. A 1-beat request on DDR3 with chop is still 25% efficient. Prevention: compute useful beats over transferred beats, and separately ask what the array did. Chop moves the first number and never the second.

"Training happens once at boot, so it is not an ongoing concern." Wrong mental model: trained values are static configuration. Engineering action: not scheduling periodic re-calibration; treating low-power state exit as free; assuming a system trained at one temperature is trained at all temperatures. Observable failure / bad conclusion: an interface that is stable after boot and becomes marginal hours later or under thermal load, with no code or configuration change to explain it. And power-management designs that enter deep states aggressively and pay an unmodelled cost to leave them. Correct model: trained values track conditions that drift — temperature, voltage — so they need periodic refresh, and anything that resets the device invalidates them. Exiting a deep low-power state can mean re-establishing measured state, not merely re-enabling a clock. Prevention: ask what invalidates the training, and what re-establishes it, and how long that takes. Chapter 4.7 is where those answers dominate the design.

11. Debugging — The Interface Works After Boot and Fails Under Load

Symptom. A memory interface passes initialisation and functional tests. Under sustained load — particularly after the system has been running long enough to warm up — errors begin to appear. Restarting the system clears the problem for a while.

Warm-up-dependent, load-dependent, cleared-by-restart is a very specific signature, and it points hard at measured state that has drifted away from the conditions it was measured under.

Mechanism 1 — training converged at a temperature unlike the operating temperature. Inspect: the die temperature at initialisation versus under load, and whether the failure threshold correlates with temperature rather than with access rate. Expected evidence: onset at a repeatable temperature, independent of the traffic pattern at that temperature. Discriminator: restarting after warm-up either fixes it or does not. If re-training while hot produces a stable system, the original training was the problem. This is the single most diagnostic experiment available, and it is cheap.

Mechanism 2 — periodic re-calibration is not happening. Inspect: whether the controller issues calibration commands on an ongoing schedule, and whether the schedule survives low-power transitions. Expected evidence: a configuration with calibration performed only at initialisation. Discriminator: read the controller's calibration schedule. Distinguished from mechanism 1 by whether re-training helps permanently — if a hot re-train is stable indefinitely, training was bad; if it degrades again, maintenance is missing.

Mechanism 3 — training converged on an edge rather than a centre. Inspect: the margin reported by the training procedure, not merely whether it reported success. Expected evidence: a pass with a narrow window, or a window whose reported centre sits near one extreme of the swept range. Discriminator: the reported margin, which many bring-up flows never look at. A procedure that reports "passed" while having found a 1-step-wide window has told you the system will fail; reading only the pass/fail bit throws that away.

Mechanism 4 — low-power state exit is not re-establishing state. Inspect: whether failures correlate with idle periods preceding them rather than with sustained load. Expected evidence: errors clustering immediately after a quiet period. Discriminator: the temporal pattern relative to idleness. This mechanism produces failures after idle, which is the opposite correlation from load-driven drift — and that opposition is what makes it cleanly separable.

Mechanism 5 — not training at all: thermal throttling or refresh behaviour changing with temperature. Inspect: whether the device's refresh rate requirement changes with temperature and whether the controller is honouring it. Expected evidence: data loss rather than timing-marginal corruption, correlated with temperature. Discriminator: is data lost, or is it mis-sampled? Chapter 2.3 established that retention time falls as temperature rises, so a controller that does not increase its refresh rate when hot loses data — a completely different mechanism with the same temperature correlation, and one that timing work will never fix.

Discrimination, cheapest first. Read the training procedure's reported margins, not just its pass bit — free, and it resolves mechanism 3 outright. Then correlate failures against temperature versus against preceding idleness, which separates drift from state loss. Then re-train while hot and observe whether stability is permanent or temporary, which separates bad initial training from missing maintenance. Then check the refresh rate against temperature, which rules the non-timing mechanism in or out.

The reasoning lesson. "It worked and then it did not, with nothing having changed" means something that is not in your source code is carrying state. On a DDR3-or-later interface that something is the trained and calibrated values, and the framing that unlocks the whole problem is to stop treating the interface as configuration and start treating it as a measurement that has an expiry date and an assumed set of conditions. Engineers who hold that framing check margins and schedules first; engineers who do not, re-read RTL that was never wrong.

12. Interview Reasoning

"Why does DDR3 use fly-by routing, and what did it cost?" Because the command, address and clock signals must reach every device, making them the most heavily loaded and the first to fail as rates rise. The previous trunk-with-stubs routing left every stub as an unterminated open end that reflects, and stubs that were negligible at DDR2 rates were not at DDR3's. Fly-by daisy-chains those signals past each device with a single termination at the far end, eliminating the stubs. The cost is that the clock arrives at each device at a different time, by an amount set by board routing that neither endpoint knows — so the write path has to be aligned per device by measurement, which is write leveling.

"Why can't the controller just calculate the write leveling delay?" Because the quantity depends on the assembled system, not on the parts: trace lengths, board construction, how many devices are populated and where. It is not in any datasheet because it is not a property of any component. The controller therefore has to measure it — it sweeps the strobe timing, the device reports the relationship it observes between that strobe and the clock it is receiving, and the controller finds the alignment point and applies a per-device delay. That is the general pattern from DDR3 onward: interface timing stopped being designed and started being measured.

"What does ZQ calibration actually calibrate against, and why is an external resistor needed?" Against an external precision resistor, nominally 240 Ω at 1% tolerance, connected to the ZQ pad. It is needed because on-die impedance is built from transistors whose conductance varies with process, voltage and temperature, so a die cannot know its own impedance accurately. The die contributes adjustability and the external component contributes accuracy: the calibration engine tunes the driver and termination to a defined ratio against the reference. The reference's value is not the termination value, which is a common and consequential confusion — substituting a differently valued resistor mis-terminates the bus with no functional symptom.

"Does burst chop reduce the work the DRAM array does?" No. The array reads the full prefetch width regardless — a whole row participates in every access and the prefetch register is filled by the internal access, not by the burst length. Chop reduces the number of beats put on the interface, so it saves interface time and switching energy. On a 1-beat request with an 8n prefetch, chop takes the transfer from 8 beats to 4, which is a real saving and still only 25% efficient. That distinction is why deepening prefetch further was not viable for DDR4: chop can hide granularity on the interface but the array's minimum work only grows.

"A memory interface passes all functional tests and is unstable in the field. What class of fault is that and how do you approach it?" It is almost certainly trained or calibrated state that is wrong or stale, because that is the state a functional test cannot see — the interface works at the point of measurement and the margin is what is missing. The first thing to read is the training procedure's reported margins rather than its pass bit, since a procedure can report success having found a one-step-wide window. Then correlate the failures: onset with temperature points at training done under unrepresentative conditions or at missing periodic re-calibration, and the discriminator between those two is whether re-training while hot is permanently stable. Failures clustering after idle periods instead point at low-power state exit not re-establishing measured state. And it is worth confirming the failures are mis-sampled data rather than lost data, because retention time falls with temperature and a controller that does not raise its refresh rate when hot produces the same temperature correlation for an entirely different reason.

13. Engineering Check

An educational DDR3-style module: eight 8-bit devices forming a 64-bit interface, 8n prefetch, array access rate 100 × 10⁶ accesses/s. Educational figures for clean arithmetic, not any real part.

1. Interface clock, transfer rate, module bandwidth? The array supplies 100 × 10⁶ × 8 = 800 × 10⁶ words/s per device. At 2 transfers per interface clock cycle the interface clock is 400 MHz and the transfer rate is 800 MT/s. Module bandwidth is 800 × 10⁶ × 64 bits = 51.2 × 10⁹ bits/s = 6.4 GB/s. Four numbers — 100, 400, 800, 6400 — and naming which is which remains the skill.

2. Full-burst granularity? 8n × 64 bits = 64 bytes, one typical cache line.

3. A workload issues only 8-byte requests. Efficiency with and without burst chop? Without chop: 64 bytes transferred per 8 wanted = 12.5%. With chop: 32 bytes transferred per 8 wanted = 25%. Chop doubles it and it remains dreadful — which is the honest answer, and the reason Module 17's controllers work so hard to coalesce small accesses before they reach the device.

4. Effective useful bandwidth for question 3, with chop? 6.4 GB/s × 25% = 1.6 GB/s. This is Chapter 1.8's point in numbers: the interface is 100% busy and the system receives a quarter of the peak. Quoting 6.4 GB/s to describe this workload would be wrong by a factor of four, and nothing about the interface is malfunctioning.

5. Now 16n prefetch, same array access rate. Bandwidth and granularity? Transfer rate 1600 MT/s, module bandwidth 12.8 GB/s, granularity 16n × 64 bits = 128 bytes. The peak doubled.

6. Re-run question 3's 8-byte workload on the question 5 device, with chop to 8 beats. Is it better? Transferred is 8 beats × 64 bits = 64 bytes per 8 wanted = 12.5% interface efficiency, and useful bandwidth is 12.8 GB/s × 12.5% = 1.6 GB/s. Identical to question 4. The peak doubled and the delivered bandwidth did not move at all — and the array now reads 16 words to deliver 8, so it is doing twice the work for the same result. That is the wall. Deepening prefetch has stopped converting into useful bandwidth for this workload, which is why DDR4 had to find rate somewhere other than granularity.

14. Summary

DDR3 takes the prefetch to 8n, putting the interface clock at four times the array's access rate, and pays two debts DDR2 left.

Termination had to learn its own value. On-die impedance is built from transistors and varies with process, voltage and temperature, and at DDR3's rates that variation no longer fits the margin. A ZQ pad and an external precision resistor — nominally 240 Ω at 1% — give a calibration engine on the die a stable reference to tune against. The reference supplies accuracy; the die supplies adjustability. The reference's value is not the termination value.

The bus topology had to change. Command, address and clock reach every device, so they are the most loaded signals and the first to fail. Fly-by routing daisy-chains them past each device with one termination at the end, eliminating the reflective stubs — and thereby making the clock arrive at each device at a deliberately different time.

That is the consequence that defines everything after. The offset between the clock a device receives and the strobe the controller sends depends on the assembled board, so the controller must measure its connection to each device before it can use it: write leveling for the write path, read leveling for the read path. Before DDR3, interface timing was designed. From DDR3 onward it is measured.

Three costs carry forward. A mandatory, per-device, per-direction measurement procedure whose results are drifting state that must be maintained — all of Modules 19 to 21. A fault class with no functional signature: training that completed badly leaves a working system with no margin, which no simulation can find. And granularity that has reached a full cache line, mitigated on the interface by burst chop — which halves interface waste on small accesses and saves no array work at all, because the array reads the full prefetch width regardless.

And the prefetch lever is finished. Transfers-per-cycle ended at DDR1; prefetch depth ends here. One more doubling puts granularity at twice a cache line, where the peak bandwidth rises and the useful bandwidth does not.

15. What Comes Next

Chapter 4.5 is the generation where the prefetch number stops changing — the first break in the pattern this module has followed for three chapters, and the most informative thing about it.

With granularity blocking further depth, DDR4 finds rate in the only place left: overlap. Instead of making one access wider, it arranges the device's banks into groups that can be accessed more independently of one another, so that consecutive accesses to different groups can proceed closer together than consecutive accesses to the same one. The interface rate rises because accesses interleave, not because any single access got bigger.

DDR4 also changes how its signals are driven and how commands are encoded, adds integrity protection to the data and command paths, and — following DDR3's pattern — makes its receiver's reference voltage another trained quantity rather than a designed one.

Return to DDR2 for prefetch as a frequency ratio and for on-die termination, DDR1 for the source-synchronous strobe that leveling aligns, or The Refresh Requirement for the temperature dependence §11's last mechanism turns on. The full path is on the DDR tutorials index.