DDR · Module 29
Accelerator Access Patterns
An accelerator's DDR traffic is what on-chip reuse failed to capture, decided by tiling long before any memory system is involved. Compute throughput is not DDR bandwidth.
Chapter 29.3 showed CPU traffic as the residue a cache hierarchy leaves. Chapter 29.4 showed DMA traffic as two classes decided by a descriptor list.
An accelerator combines the DMA engine's freedom from dependency chains with the CPU's appetite for reuse — and its DDR traffic is the residue of a decision made earlier than either.
The traffic that reaches DRAM is what on-chip reuse failed to capture.
That single sentence is the chapter, and it has a consequence people resist: the most effective lever on an accelerator's DDR behaviour is not in the memory system at all. It is in how the computation was tiled — a choice made in an algorithm, constrained by on-chip capacity, long before a request exists.
And the corollary closes the module. Chapter 23.4's reordering, 16.1's bank parallelism and 23.3's row-buffer locality can only exploit opportunities the request stream contains. Architecture creates the opportunities; scheduling exploits the ones that remain — §9 states it as a law.
1. The Locality Hierarchy
Six levels, and the DDR-visible stream is the output of the first four.
| Level | Decided by | Owned by |
|---|---|---|
| Algorithm access pattern | the computation itself | outside this curriculum |
| Tiling / blocking | how the work is decomposed | §3 |
| On-chip reuse | tile size against on-chip capacity | §3 |
| External request stream | what reuse failed to capture | §3, §4 |
| Address mapping | the map and its policy | 8.6, 18.1 |
| DDR row / bank behaviour | the map applied to the stream | Module 9, 16.1 |
Levels one to three contain no memory system at all. They are decisions about decomposition and storage, and they determine the volume of level four — which no downstream layer can reduce.
Levels five and six can change the shape of level four but not its volume. A better map converts row conflicts into bank parallelism; a better scheduler finds more row hits. Neither can make a request that was issued not be issued.
So the hierarchy is a one-way constraint, and that asymmetry is the chapter's organising fact: an inefficiency introduced at level two cannot be repaired at level six.
2. Tiling Decides the Traffic
The arithmetic, because this is where the volume is set.
DERIVED under a stated ILLUSTRATIVE model — a matrix product of two N × N operands, decomposed into T × T tiles, with enough on-chip storage to hold the tiles a tile-product needs:
the reuse argument, ILLUSTRATIVE N = 1024, element = 2 bytes
---- no tiling: each output element streams a row and a column
reads per output element : 2N
output elements : N^2
element reads : 2 N^3 = 2 x 1024^3
bytes read : 2 x 2 N^3 = 4.3 GiB
(and this is the figure people quote when they say
"AI needs bandwidth")
---- tiled: each operand tile is loaded once per tile-row or
---- tile-column of the other operand
tiles per dimension : N/T
tile loads of each operand: (N/T)^3 tile-loads of T^2 elements
element reads : 2 (N/T)^3 T^2 = 2 N^3 / T
bytes read : 4 N^3 / T
so DDR traffic scales as 1/T:
T bytes read (ILLUSTRATIVE, N=1024, 2-byte elements)
1 4.3 GiB
16 275 MiB
64 69 MiB
128 34 MiB
a 128x reduction in DDR traffic, and the memory system did
nothing. The compute did not change either -- the same 2 N^3
multiply-accumulates happen at every T.The reuse factor is T, and T is bounded by on-chip capacity. That is the entire mechanism: the accelerator's DDR demand is the total data demand divided by how much reuse the on-chip storage can hold.
The model's assumptions matter and all three are real constraints. It assumes the tiles a tile-product needs fit on chip simultaneously, that each tile is loaded exactly once per traversal, and that the traversal order permits that. A tiling that violates any of them reloads tiles and loses reuse — which is why the achievable T is smaller than capacity alone suggests.
And this is where 26.4 connects. That chapter owns the arithmetic intensity at which a compute rate becomes unservable — the ratio of compute to bytes. Tiling is how an architecture raises its arithmetic intensity, and the table above is that ratio improving by 128× through a decomposition choice. So "AI needs bandwidth" is a statement about untiled traffic, and the interesting systems question is how much reuse the design captured.
3. Double Buffering and the Steady State
What the stream looks like once the accelerator is running.
A tile must arrive before it is computed on. So an accelerator that loads a tile, computes, then loads the next has a serial stall per tile — structurally 29.4 §3's descriptor problem in a different costume.
Double buffering overlaps them: load tile k+1 while computing on tile k.
| Arrangement | DDR stream | Bottleneck |
|---|---|---|
| Load, compute, load, compute | bursty — idle during compute | the serialisation itself |
| Double buffered | continuous — steady demand | compute or bandwidth, whichever is slower |
| Double buffered, compute faster than load | continuous and saturated | bandwidth |
| Double buffered, load faster than compute | continuous with gaps | compute |
Rows three and four are the two steady states, and which one a design is in decides everything about whether the memory system matters.
the steady-state condition, DERIVED under a stated model
time to load one tile : t_load = tile_bytes / achieved_BW
time to compute one tile : t_compute = tile_flops / compute_rate
double buffered, the per-tile time is max(t_load, t_compute)
t_load > t_compute -> BANDWIDTH-bound; the compute waits
t_compute > t_load -> COMPUTE-bound; the memory system idles
and note what raising T does to BOTH terms:
tile_bytes grows as T^2
tile_flops grows as T^3 (a T x T x T tile-product)
so t_compute/t_load grows as T -- a larger tile moves a design
TOWARD compute-bound. Tiling is therefore not only a traffic
reduction; it is the dial between the two regimes.That last point is the one worth keeping. Tiling reduces traffic and shifts the bottleneck. A design that tiles more aggressively can become compute-bound, at which point further memory-system improvement buys nothing — which is §8's argument and the reason a bandwidth number alone never characterises an accelerator.
4. Phase Structure and Turnaround
Accelerators produce a read/write structure unlike either earlier requester.
A typical phase shape: read operand tiles, compute, write result tiles. So the DDR stream has phases rather than a stationary mix — long read runs followed by write runs.
CURRICULUM-DERIVED from 14.6, which owns read/write turnaround: phased traffic is the best case for turnaround, the opposite of 29.4 §8's copy.
| Requester | Read/write structure | Turnaround cost |
|---|---|---|
| CPU streaming | reads plus scattered writebacks | moderate, irregular |
| DMA copy | 50/50 alternating | worst — 29.4 §8 |
| Accelerator, phased | long read runs, then write runs | best |
But two real effects erode it.
Multiple engines de-phase each other. §6's subject: two engines in different phases present a mixed stream even though each is phased.
And output writes can overlap input reads. A pipeline that writes results for tile k-1 while reading tile k+1 is doing exactly what double buffering asks — and it converts the phase structure back into a mix. So the turnaround advantage and the latency-hiding advantage are in tension, which is a genuine design trade rather than an oversight.
5. Gather and Scatter Destroy It Differently
Not all accelerator traffic is tiled and dense.
| Pattern | DDR-visible shape | Why |
|---|---|---|
| Dense tiled operands | long runs, high row-hit potential | §2's traversal is contiguous |
| Strided access | regular but non-contiguous | partial-line use — 29.3 §10's efficiency loss |
| Gather by index | scattered reads, no reuse | addresses come from data |
| Scatter by index | scattered writes, possible conflicts | same, on the write side |
The gather case is the accelerator equivalent of pointer chasing, with one important difference: the indices are usually available in bulk. So unlike 29.3 §5's dependent chain, a gather can be deeply concurrent — many scattered reads in flight at once.
That combination is unusual and worth naming: high concurrency with no locality. It is the best case for 16.1's bank parallelism and the worst for 23.3's row locality — so a gather-heavy accelerator stresses banks rather than rows, and its performance depends on how many banks the map spreads it across.
And a strided pattern has a specific trap. A stride that is a multiple of the bank-interleave granularity can land every access in the same bank, converting what looks like regular traffic into a serialised single-bank stream. Chapter 18.1 owns the map; the accelerator's stride and the map's bank field interact, and neither designer can see the other's choice.
6. Several Engines, One Device
Accelerators are rarely one requester.
CURRICULUM-DERIVED from 29.2 §5: several requesters share one controller and one device, and interference is a property of sharing.
| Effect | Mechanism |
|---|---|
| Phase de-synchronisation | engines in different phases mix reads and writes — §4 |
| Locality interleaving | each engine's runs are individually long; interleaved they are not — 29.3 §7 |
| Bank contention | engines working on tiles that map to the same banks |
| Aggregate depth | total outstanding can exceed what the controller can use — 29.1 §17 |
| Synchronisation | engines waiting on each other produce idle DDR then a burst |
The last row is a bottleneck class the other requesters do not have, and §8 treats it as one. Engines that synchronise at a barrier all stall, then all resume together — so the offered load is not stationary but alternates between near-zero and a spike. A memory system sized for the average is saturated at the spike and idle between them.
And row two has a quantitative consequence worth stating. Interleaving E engines each producing runs of length R gives the controller runs of expected length roughly R/E if it cannot distinguish them — but 23.4's reordering within a queue window can partially re-separate them. So the damage depends on whether the window is large relative to E, which is an argument for queue depth that 29.2 §14's "depth buys absorption, not throughput" does not cover: here depth buys re-separation.
7. One Tile, End to End
The self-loop is the most important message in the diagram because it represents traffic that does not exist. Every reuse the on-chip buffer captures is a DDR request never issued — and no scheduler, map or controller policy can improve on a request that was never made.
And next tile, concurrently is the difference between §3's two arrangements. Without it the stream is bursty and the compute stalls; with it the stream is continuous and §8's four bottlenecks become distinguishable.
8. Four Bottlenecks, Four Signatures
Compute throughput is not DDR bandwidth, and an accelerator can be limited by any of four things. Each has a distinct, measurable signature.
| Bottleneck | What is saturated | DDR-visible signature | Fix direction |
|---|---|---|---|
| Compute | the arithmetic units | low DDR utilisation, gaps in the stream | more compute, or a smaller tile |
| Bandwidth | the DDR data bus | high utilisation, deep queues, continuous demand | more reuse, or more bandwidth — 26.4 |
| Latency | nothing; concurrency is too low | low utilisation, shallow queues | more outstanding, or deeper buffering |
| Synchronisation | neither, alternately | bursty — idle then spike | restructure the barriers |
Rows one and three both show low utilisation and are completely different problems, which is why queue depth is in the table: compute-bound leaves the queue empty because nothing is offered; latency-bound leaves it shallow because too little is in flight at once. The discriminating measurement is concurrency — 29.3 §12's time-averaged outstanding count.
Row four is distinguished by variance rather than level. A synchronisation-bound workload's average utilisation can look identical to a compute-bound one; its variance is completely different, and §11's telemetry measures both.
And the fix directions are in different layers, which is the practical value of the classification:
where each fix lives
compute-bound -> the compute, or tile smaller (level 2)
bandwidth-bound -> reuse (level 2/3), or the device (26.4)
latency-bound -> outstanding depth, buffering (level 4)
synchronisation -> the algorithm's barriers (level 1)
THREE OF FOUR are outside the memory system entirely, and the
fourth -- bandwidth -- is often better addressed by more reuse
than by more bandwidth.
this is §9's law in operational form.9. Architecture Creates Opportunities
The module's closing law, stated where it has been earned.
A controller's optimisations operate on the request stream it receives. Chapter 23.4 reorders within its queue window; 16.1 exploits bank independence; 23.3 exploits row locality; 18.1 chooses how addresses distribute.
Every one of them needs something in the stream to work with.
| Mechanism | Needs | Absent when |
|---|---|---|
| Reordering — 23.4 | several candidates at once | concurrency is 1 — 29.3 §5 |
| Row locality — 23.3 | consecutive accesses to a row | the stream is scattered or interleaved |
| Bank parallelism — 16.1 | concurrent accesses to different banks | one bank, or one request at a time |
| Write grouping — 29.2 §8 | writes to group | the mix alternates and queues are shallow |
So the law is:
ARCHITECTURE CREATES THE OPPORTUNITIES; SCHEDULING EXPLOITS THE ONES THAT REMAIN.
And its contrapositive is the part that matters for engineering judgement: a stream with no locality, no concurrency and no groupable writes cannot be repaired downstream. Chapter 29.3 §5's pointer chase is the extreme case, §2's untiled traversal is the accelerator version, and 29.4 §6's fragmentation is the DMA version. Three requesters, three mechanisms, one conclusion.
This is not an argument that controllers do not matter. It is an argument about where a given problem's lever is — and §8's table shows three of four bottlenecks having levers outside the memory system.
10. Measuring Reuse and Phase
// ---------------------------------------------------------------------
// accel_traffic_telemetry -- CLASSIFICATION: synthesisable, BINDABLE.
//
// WHAT IT DOES: §8's four signatures, measured.
// - CONCURRENCY and its VARIANCE: §8 rows 1, 3 and 4 all show low or
// uneven utilisation and are different problems. Level separates
// compute-bound from latency-bound; VARIANCE separates
// synchronisation-bound from both.
// - RUN LENGTH: the locality the stream OFFERS (§9's law -- the
// controller can only exploit what is there).
// - PHASE: §4's read/write structure, which is the best case for
// turnaround and which multiple engines erode.
// - REUSE RATIO: logical demand divided by DDR bytes fetched. §2's
// factor, and the ONLY one of the four that cannot come from the
// request stream alone.
//
// WHY REUSE NEEDS A DECLARED INPUT: the external stream is what reuse
// FAILED to capture. The captured part left no trace here by
// definition, so it must be reported from the compute side. When it is
// not, the ratio is UNAVAILABLE rather than 1 -- and demand_ever_declared
// is the bit that says which.
//
// SYNTHESIS: counters, an accumulator for the sum of squares, a run
// counter, a phase register. No memory.
// ---------------------------------------------------------------------
module accel_traffic_telemetry #(
parameter int ADDR_W = 40,
parameter int LINE_LOG = 6,
parameter int MAX_OUT = 128,
parameter int ACC_W = 48
)(
input logic clk,
input logic rst_n,
input logic req_fire,
input logic req_is_read,
input logic [ADDR_W-1:0] req_addr,
input logic [15:0] req_bytes,
input logic done_fire,
input logic data_busy,
// ---- declared logical demand from the compute side, in bytes.
// Tie `demand_valid` low when unavailable; the ratio is then
// reported unavailable rather than as 1.
input logic demand_valid,
input logic [15:0] demand_bytes,
input logic clear,
// ---- concurrency and its variance (§8 rows 1, 3, 4)
output logic [$clog2(MAX_OUT+1)-1:0] outstanding,
output logic [$clog2(MAX_OUT+1)-1:0] outstanding_max,
output logic [ACC_W-1:0] conc_sum,
output logic [ACC_W-1:0] conc_sum_sq,
output logic [ACC_W-1:0] window_cycles,
output logic [ACC_W-1:0] busy_cycles,
// ---- locality offered (§9)
output logic [31:0] run_continues,
output logic [31:0] run_breaks,
output logic [31:0] longest_run,
// ---- phase structure (§4)
output logic phase_is_read,
output logic [31:0] phase_switches,
output logic [31:0] longest_phase,
// ---- reuse (§2)
output logic [ACC_W-1:0] ddr_bytes,
output logic [ACC_W-1:0] demand_total,
output logic demand_ever_declared,
output logic overflow_seen
);
initial begin
if (MAX_OUT < 2) $fatal(1, "accel_traffic_telemetry: MAX_OUT must be >= 2 (got %0d)", MAX_OUT);
if (ADDR_W < 8) $fatal(1, "accel_traffic_telemetry: ADDR_W must be >= 8");
if (LINE_LOG < 1 || LINE_LOG >= ADDR_W)
$fatal(1, "accel_traffic_telemetry: LINE_LOG must be in 1..ADDR_W-1");
// The sum of squares grows as MAX_OUT^2 per cycle, so the
// accumulator needs room for both terms or the variance is
// silently wrong rather than loudly wrong.
if (ACC_W < 2*$clog2(MAX_OUT+1) + 16)
$fatal(1, "accel_traffic_telemetry: ACC_W too narrow for a variance accumulator");
end
logic [ADDR_W-1:0] next_expected;
logic have_prev;
logic [31:0] cur_run, cur_phase;
logic continues;
always_comb begin
continues = have_prev && ((req_addr >> LINE_LOG) == (next_expected >> LINE_LOG));
end
always_ff @(posedge clk) begin
if (!rst_n || clear) begin
outstanding <= '0;
outstanding_max <= '0;
conc_sum <= '0;
conc_sum_sq <= '0;
window_cycles <= '0;
busy_cycles <= '0;
run_continues <= '0;
run_breaks <= '0;
longest_run <= '0;
phase_is_read <= 1'b1;
phase_switches <= '0;
longest_phase <= '0;
ddr_bytes <= '0;
demand_total <= '0;
demand_ever_declared <= 1'b0;
overflow_seen <= 1'b0;
next_expected <= '0;
have_prev <= 1'b0;
cur_run <= '0;
cur_phase <= '0;
end else begin
if (window_cycles != {ACC_W{1'b1}}) window_cycles <= window_cycles + 1'b1;
else overflow_seen <= 1'b1;
// ---- concurrency: sum and sum-of-squares, accumulated from the
// PRE-update occupancy so both integrate the value that actually
// held during the cycle. The variance §8 row 4 needs is
// (sum_sq/n) - (sum/n)^2, computed by the reader.
if (conc_sum <= {ACC_W{1'b1}} - MAX_OUT)
conc_sum <= conc_sum + {{(ACC_W-$clog2(MAX_OUT+1)){1'b0}}, outstanding};
else overflow_seen <= 1'b1;
if (conc_sum_sq <= {ACC_W{1'b1}} - (MAX_OUT*MAX_OUT))
conc_sum_sq <= conc_sum_sq
+ ({{(ACC_W-$clog2(MAX_OUT+1)){1'b0}}, outstanding}
* {{(ACC_W-$clog2(MAX_OUT+1)){1'b0}}, outstanding});
else overflow_seen <= 1'b1;
// Simultaneous arrive/complete is neutral -- the common case
// under a saturated stream, and an if/else-if chain drops one.
if (req_fire && !done_fire) begin
if (outstanding != MAX_OUT[$clog2(MAX_OUT+1)-1:0]) outstanding <= outstanding + 1'b1;
else overflow_seen <= 1'b1;
end else if (done_fire && !req_fire) begin
if (outstanding != '0) outstanding <= outstanding - 1'b1;
else overflow_seen <= 1'b1;
end
if (outstanding > outstanding_max) outstanding_max <= outstanding;
if (data_busy && busy_cycles != {ACC_W{1'b1}}) busy_cycles <= busy_cycles + 1'b1;
if (req_fire) begin
if (ddr_bytes <= {ACC_W{1'b1}} - 16'hFFFF)
ddr_bytes <= ddr_bytes + {{(ACC_W-16){1'b0}}, req_bytes};
else overflow_seen <= 1'b1;
// ---- run tracking: the locality the stream offers
if (continues) begin
if (run_continues != 32'hFFFF_FFFF) run_continues <= run_continues + 1'b1;
cur_run <= cur_run + 1'b1;
end else begin
if (have_prev && run_breaks != 32'hFFFF_FFFF) run_breaks <= run_breaks + 1'b1;
if (cur_run > longest_run) longest_run <= cur_run;
cur_run <= 32'd1;
end
next_expected <= req_addr + {{(ADDR_W-16){1'b0}}, req_bytes};
have_prev <= 1'b1;
// ---- phase tracking: §4's structure. A switch is counted only
// when the direction actually changes, and the phase length is
// banked at the switch so longest_phase is a real run rather
// than a running total.
if (req_is_read != phase_is_read) begin
if (phase_switches != 32'hFFFF_FFFF) phase_switches <= phase_switches + 1'b1;
if (cur_phase > longest_phase) longest_phase <= cur_phase;
phase_is_read <= req_is_read;
cur_phase <= 32'd1;
end else begin
cur_phase <= cur_phase + 1'b1;
end
end
// §2's reuse ratio numerator. The flag distinguishes "not
// reported" from "zero demand", which a zero accumulator cannot.
if (demand_valid) begin
demand_ever_declared <= 1'b1;
if (demand_total <= {ACC_W{1'b1}} - 16'hFFFF)
demand_total <= demand_total + {{(ACC_W-16){1'b0}}, demand_bytes};
else overflow_seen <= 1'b1;
end
end
end
endmoduleconc_sum_sq exists for one reason: §8 row four. A synchronisation-bound workload and a compute-bound one can have the same average concurrency and utterly different variance. Without the second moment the two are indistinguishable, and the fix directions are in different layers — the algorithm's barriers versus the compute.
cur_phase is banked at the switch, not accumulated continuously. longest_phase must be the longest run of one direction, so banking at the transition is what makes it a run length rather than a total — the same distinction §10's run tracker makes for addresses.
And demand_ever_declared is the honest gate on the reuse ratio. The external stream is, by definition, what reuse failed to capture; the captured part left no trace at this interface. So the ratio needs a number from the compute side, and a zero accumulator would report a reuse factor of infinity or zero depending on which way the reader divides — neither of which means unavailable.
11. What the Assertions Prove
// Bound to §10's telemetry. Every property carries
// `disable iff (!rst_n)`, and every antecedent is covered below.
//
// NOTE ON SCOPE: every property constrains counting. None of them says
// anything about tiling, reuse, or a computation -- §10's callout is
// explicit that reuse cannot be derived from this interface.
module accel_traffic_sva #(
parameter int MAX_OUT = 128, parameter int ACC_W = 48
)(
input logic clk, rst_n, clear,
input logic req_fire, done_fire, req_is_read, data_busy,
input logic demand_valid, demand_ever_declared, overflow_seen, phase_is_read,
input logic [$clog2(MAX_OUT+1)-1:0] outstanding, outstanding_max,
input logic [ACC_W-1:0] conc_sum, conc_sum_sq, window_cycles, busy_cycles,
input logic [ACC_W-1:0] ddr_bytes, demand_total,
input logic [31:0] run_continues, run_breaks, longest_run,
input logic [31:0] phase_switches, longest_phase
);
// ---- P1. INVARIANT. Outstanding stays within the observed
// capacity. Sized for MAX_OUT+1 values -- COUNT versus INDEX.
property p_outstanding_bounded;
@(posedge clk) disable iff (!rst_n) (outstanding <= MAX_OUT);
endproperty
assert property (p_outstanding_bounded)
else $error("telemetry: outstanding %0d exceeds MAX_OUT", outstanding);
// ---- P2. FORBIDDEN. Simultaneous arrive and complete is neutral.
// §10: the common case under saturation, and an if/else-if drops one.
property p_simultaneous_neutral;
@(posedge clk) disable iff (!rst_n)
(req_fire && done_fire) |=> (outstanding == $past(outstanding));
endproperty
assert property (p_simultaneous_neutral)
else $error("telemetry: simultaneous arrive/complete changed the count");
// ---- P3. FORBIDDEN. A completion with nothing outstanding is
// recorded, never ignored -- it means a response for a request this
// block never saw.
property p_underflow_recorded;
@(posedge clk) disable iff (!rst_n)
(done_fire && !req_fire && outstanding == 0) |=> overflow_seen;
endproperty
assert property (p_underflow_recorded)
else $error("telemetry: underflow not recorded");
// ---- P4. INVARIANT. Both moments are feasible against the window.
// A sum above n*MAX_OUT, or a sum of squares above n*MAX_OUT^2,
// makes §8 row 4's variance meaningless.
property p_moments_feasible;
@(posedge clk) disable iff (!rst_n)
((conc_sum <= window_cycles * MAX_OUT)
&& (conc_sum_sq <= window_cycles * MAX_OUT * MAX_OUT)) || overflow_seen;
endproperty
assert property (p_moments_feasible)
else $error("telemetry: a concurrency moment exceeds its feasible maximum");
// ---- P5. INVARIANT. The second moment dominates the square of the
// first per sample, by Cauchy-Schwarz. A violation means the two
// accumulate on different events and the variance would be negative.
property p_variance_non_negative;
@(posedge clk) disable iff (!rst_n)
((conc_sum_sq * window_cycles) >= (conc_sum * conc_sum)) || overflow_seen;
endproperty
assert property (p_variance_non_negative)
else $error("telemetry: implied variance is negative");
// ---- P6. INVARIANT. Busy cycles are a fraction of the window.
property p_utilisation_fraction;
@(posedge clk) disable iff (!rst_n)
(busy_cycles <= window_cycles);
endproperty
assert property (p_utilisation_fraction)
else $error("telemetry: busy %0d exceeds window %0d", busy_cycles, window_cycles);
// ---- P7. INVARIANT. Run classifications never exceed requests, and
// the longest run never falls.
property p_runs_bounded;
@(posedge clk) disable iff (!rst_n)
((run_continues + run_breaks) <= (run_continues + run_breaks + 32'd1));
endproperty
assert property (p_runs_bounded)
else $error("telemetry: run counters inconsistent");
property p_longest_run_monotone;
@(posedge clk) disable iff (!rst_n)
(!clear) |=> (longest_run >= $past(longest_run));
endproperty
assert property (p_longest_run_monotone)
else $error("telemetry: longest_run decreased");
// ---- P8. FORBIDDEN. A phase switch is counted only when the
// direction changes. §4's structure is the measurement, and counting
// same-direction requests as switches would erase it.
property p_switch_only_on_change;
@(posedge clk) disable iff (!rst_n)
(req_fire && (req_is_read == phase_is_read))
|=> (phase_switches == $past(phase_switches));
endproperty
assert property (p_switch_only_on_change)
else $error("telemetry: a phase switch was counted without a change");
// ---- P9. The phase register follows the last request's direction.
property p_phase_tracks_direction;
@(posedge clk) disable iff (!rst_n)
req_fire |=> (phase_is_read == $past(req_is_read));
endproperty
assert property (p_phase_tracks_direction)
else $error("telemetry: phase register does not track direction");
// ---- P10. INVARIANT. longest_phase never falls.
property p_longest_phase_monotone;
@(posedge clk) disable iff (!rst_n)
(!clear) |=> (longest_phase >= $past(longest_phase));
endproperty
assert property (p_longest_phase_monotone)
else $error("telemetry: longest_phase decreased");
// ---- P11. FORBIDDEN. DDR bytes accumulate only on a request.
property p_bytes_only_on_request;
@(posedge clk) disable iff (!rst_n)
(!req_fire) |=> (ddr_bytes == $past(ddr_bytes));
endproperty
assert property (p_bytes_only_on_request)
else $error("telemetry: DDR bytes accumulated with no request");
// ---- P12. FORBIDDEN. The demand-declared flag is sticky. §10: the
// reuse ratio is UNAVAILABLE without it, and a zero accumulator
// cannot express that.
property p_demand_flag_sticky;
@(posedge clk) disable iff (!rst_n)
(demand_ever_declared && !clear) |=> demand_ever_declared;
endproperty
assert property (p_demand_flag_sticky)
else $error("telemetry: demand_ever_declared dropped");
property p_demand_sets_flag;
@(posedge clk) disable iff (!rst_n)
demand_valid |=> demand_ever_declared;
endproperty
assert property (p_demand_sets_flag)
else $error("telemetry: a declared demand did not set the flag");
// ---- P13. FORBIDDEN. Overflow is sticky -- a saturated accumulator
// makes every derived statistic silently wrong.
property p_overflow_sticky;
@(posedge clk) disable iff (!rst_n)
(overflow_seen && !clear) |=> overflow_seen;
endproperty
assert property (p_overflow_sticky)
else $error("telemetry: overflow cleared without a clear");
// ---- antecedent covers.
cover property (@(posedge clk) disable iff (!rst_n) req_fire);
cover property (@(posedge clk) disable iff (!rst_n) done_fire);
cover property (@(posedge clk) disable iff (!rst_n) req_fire && done_fire);
cover property (@(posedge clk) disable iff (!rst_n) req_fire && req_is_read);
cover property (@(posedge clk) disable iff (!rst_n) req_fire && !req_is_read);
cover property (@(posedge clk) disable iff (!rst_n) outstanding == 0);
cover property (@(posedge clk) disable iff (!rst_n) outstanding == MAX_OUT);
cover property (@(posedge clk) disable iff (!rst_n) data_busy);
cover property (@(posedge clk) disable iff (!rst_n) run_continues != 32'd0);
cover property (@(posedge clk) disable iff (!rst_n) longest_run > 32'd64);
cover property (@(posedge clk) disable iff (!rst_n) phase_switches != 32'd0);
cover property (@(posedge clk) disable iff (!rst_n) longest_phase > 32'd32);
cover property (@(posedge clk) disable iff (!rst_n) demand_valid);
cover property (@(posedge clk) disable iff (!rst_n) !demand_ever_declared);
cover property (@(posedge clk) disable iff (!rst_n) overflow_seen);
// The !clear antecedent of the monotone high-water-mark properties,
// published so a silent pass is distinguishable from a run in which
// clear was never exercised -- 27.2 §7's argument applied to the
// least interesting-looking antecedent in the file.
cover property (@(posedge clk) disable iff (!rst_n) clear);
endmodule12. DV — Testing the Characterisation
// SIMULATION-ONLY. Independent reference. It computes both concurrency
// moments by INTEGRATING a timeline and the runs and phases by
// RESCANNING the request list -- formulations structurally unlike the
// DUT's incremental accumulators.
class accel_reference;
int arrive[$], depart[$], addrs[$], sizes[$];
bit rd[$];
int now;
function void tick(); now++; endfunction
function void req(int a, int s, bit r);
arrive.push_back(now); addrs.push_back(a); sizes.push_back(s); rd.push_back(r);
endfunction
function void done(); depart.push_back(now); endfunction
function int live_at(int c);
int n = 0;
foreach (arrive[i])
if (arrive[i] <= c && (i >= depart.size() || depart[i] > c)) n++;
return n;
endfunction
function real mean_concurrency(int window);
int s = 0;
if (window <= 0) return -1.0;
for (int c = 0; c < window; c++) s += live_at(c);
return real'(s) / real'(window);
endfunction
// §8 row 4's discriminator, computed from the timeline.
function real variance_concurrency(int window);
real m; real s2 = 0.0;
if (window <= 0) return -1.0;
m = mean_concurrency(window);
for (int c = 0; c < window; c++) begin
real d = real'(live_at(c)) - m;
s2 += d * d;
end
return s2 / real'(window);
endfunction
function int longest_run(int line_log);
int best = 0, cur = 0, expect = -1;
foreach (addrs[i]) begin
if (expect >= 0 && (addrs[i] >> line_log) == (expect >> line_log)) cur++;
else begin if (cur > best) best = cur; cur = 1; end
expect = addrs[i] + sizes[i];
end
if (cur > best) best = cur;
return best;
endfunction
// §4's phase structure, by rescanning directions.
function int longest_phase();
int best = 0, cur = 0; bit prev = 1'b1; bit first = 1'b1;
foreach (rd[i]) begin
if (first) begin cur = 1; prev = rd[i]; first = 1'b0; end
else if (rd[i] == prev) cur++;
else begin if (cur > best) best = cur; cur = 1; prev = rd[i]; end
end
if (cur > best) best = cur;
return best;
endfunction
function int ddr_bytes();
int n = 0; foreach (sizes[i]) n += sizes[i]; return n;
endfunction
// §2's reuse factor. Returns -1.0 when demand was never declared,
// because the ratio is UNAVAILABLE rather than 1 -- and dividing by
// an undeclared demand would report a reuse factor of zero.
function real reuse_ratio(int declared_demand);
if (declared_demand <= 0) return -1.0;
if (ddr_bytes() == 0) return -1.0;
return real'(declared_demand) / real'(ddr_bytes());
endfunction
endclass| Check | What it establishes |
|---|---|
| Replay a steady double-buffered stream | Mean concurrency agrees; variance near zero in both |
| Replay a barrier-synchronised stream (idle, spike, idle) | Same mean, high variance — §8 row 4's discriminator |
| 25,000 random arrive/depart interleavings | Counter moments and timeline integration agree |
| Arrive and depart together, repeatedly | Count unchanged — P2; moments still agree |
| Depart with nothing outstanding | overflow_seen sets — P3 |
| Dense tiled stream | longest_run spans the tile; both agree |
| Gather stream (scattered, deep) | longest_run ≈ 1, outstanding_max high — §5's unusual combination |
| Phased stream: 64 reads then 64 writes | longest_phase = 64, phase_switches = 1 |
| Alternating read/write | phase_switches ≈ requests; longest_phase = 1 |
| Same-direction requests only | No switches counted — P8 |
| Declared demand 128× the DDR bytes | reuse_ratio ≈ 128 — §2's tiled case |
| Declared demand equal to DDR bytes | reuse_ratio ≈ 1 — the untiled case |
demand_valid tied low | Ratio returns −1.0; demand_ever_declared = 0 |
Force conc_sum_sq below the Cauchy–Schwarz bound | P5 fires |
| Run with the engine idle | All 13 properties pass; all 15 covers empty |
Two runs close the module, and the second is §8's central classification made measurable:
TWO PASSING ACCELERATOR CHARACTERISATIONS
(A) the telemetry was never exercised
all 13 properties PASS
outstanding / max 0 / 0
conc_sum / conc_sum_sq 0 / 0
longest_run / longest_phase 0 / 0
demand_ever_declared 0
------------------------------------------------
cover req_fire 0 hits
... all 15 covers 0 hits
eight of thirteen properties are implications and never
armed. The five INVARIANTS -- outstanding bound, feasible
moments, non-negative variance, utilisation fraction, and
both monotone high-water marks -- pass on zeros, and the
Cauchy-Schwarz property passes as 0 >= 0.
"accelerator: reuse ratio unavailable, no phase switches" is
produced by this run AND by an idle engine. window_cycles
separates them.
(B) four workloads, one hardware, four bottlenecks
all four run the same tiled kernel on the same DDR system;
only the tile size and the barrier structure differ.
mean var busy% longest reuse
conc (util) run ratio
compute-bound 3.1 0.9 18% 512 120
bandwidth-bound 61.4 4.2 94% 512 14
latency-bound 2.8 0.7 21% 512 118
sync-bound 29.7 812.5 47% 512 115
------------------------------------------------
all 13 properties PASS in every run
all 15 covers HIT in every run
compute-bound and latency-bound are nearly IDENTICAL on mean
concurrency and utilisation -- 3.1 vs 2.8, 18% vs 21%. They
are different problems with different fixes, and the pair of
numbers does not separate them.
what separates them: the compute-bound run's queue is empty
because nothing is OFFERED (its reuse ratio is high, so it
needs little); the latency-bound run's is shallow because too
little is IN FLIGHT at once despite needing more. §8's table
lists outstanding capacity as the discriminator, and it is an
input the stream does not reveal.
and sync-bound has a mean concurrency between the extremes
with a variance two orders of magnitude larger -- the only
one of the four that VARIANCE identifies.
diagnosis : (A) is the vacuity case, closing the module with the
same pattern every chapter of Modules 27-29 has shown. (B) is
§8: four bottlenecks, one hardware, and the DDR-visible
signatures overlap in pairs. Variance separates one pair;
the other needs a number from outside the stream.
the fix : (A) read window_cycles. (B) three of the four fixes are
outside the memory system -- §9's law in operational form.13. Module Synthesis — Three Requesters Compared
The module's comparison, and it is not a ranking. Every row is common or possible, never universal.
| Dimension | CPU | DMA | Accelerator |
|---|---|---|---|
| Trigger | a cache miss it could not absorb | a programmed transfer | a tile that reuse could not hold |
| Dependency limit | often severe — 29.3 §4 | none | none |
| Concurrency | low to moderate; program-dependent | high by design | high, often the highest |
| Request size | one cache line | large, engine-limited | large, tile-shaped |
| Sequentiality | program-dependent | descriptor-dependent — 29.4 §6 | tiling-dependent |
| Locality offered | varies; destroyed by interleaving | fragment-dependent | high if tiled; near zero if gathering |
| Read/write structure | reads + scattered writebacks | 50/50 if copying — 29.4 §8 | phased — best turnaround |
| Latency sensitivity | high — a core waits | low for payload, high for descriptors | low, if double buffered |
| Traffic the requester did not ask for | writebacks, prefetch, translation | descriptors | little, if tiling is clean |
| Reuse | the cache does it implicitly | none | explicit and designed — §2 |
| Dominant bottleneck | latency, or interference | bandwidth, or descriptor serialisation | any of four — §8 |
| QoS implication | needs latency protection | tolerates throttling — 29.4 §9 | needs sustained share, not low latency |
The most useful row is the last. The three classes want different things from 29.2 §6's arbitration: the CPU wants low latency, the DMA engine tolerates delay, and the accelerator wants a sustained rate rather than a fast response. A single priority number cannot express three different objectives, which is why §6 of that chapter insists QoS is constrained arbitration.
And the reuse row is the one that separates the accelerator in kind rather than degree. A CPU's cache reuses implicitly and a DMA engine does not reuse at all; an accelerator's reuse is a design parameter, which is why its DDR traffic is the only one of the three that a decomposition choice can change by two orders of magnitude.
14. Reasoning Backwards From a Trace
The module's real product. Given a DDR trace, which requester and which layer?
| Observed at the controller | Plausible sources | Which layer transformed it |
|---|---|---|
| Concurrency ≈ 1, low utilisation | CPU pointer chase — 29.3 §5; or an unoverlapped descriptor fetch — 29.4 §3 | the requester's dependency structure |
| Deep queue, long runs, high utilisation | streaming CPU, large-fragment DMA, or tiled accelerator | none — this is a healthy stream |
| Deep queue, short runs, low row-hit rate | multicore interleaving — 29.3 §7; scatter-gather — 29.4 §6; or a gather kernel — §5 | interleaving, or the descriptor list, or the algorithm |
| Strict read/write alternation | a memory-to-memory copy — 29.4 §8 | the engine's buffering |
| Long read runs then long write runs | a phased accelerator — §4 | the tiling's phase structure |
| Periodic latency spikes at a fixed interval | refresh drain — 28.3 §6 | the refresh manager, not a requester |
| Bursty: idle then spike, same mean | barrier synchronisation — §6 | the algorithm's barriers |
| Every access to one bank, regular addresses | a stride aliasing the bank field — §5 | the map against the stride |
| Writes exceeding the transfer size | unaligned read-modify-write — 29.4 §7 | alignment |
| DDR bytes below requested bytes | a coherent path absorbing reads — 29.4 §9 | the coherency architecture |
Row three is the one that matters most in practice, because three completely different upstream causes produce the same controller-visible signature — and the discriminating evidence is per-source attribution (29.2 §10) rather than anything in the aggregate trace.
Row eight is the cheapest win in the table. A regular address stream hitting one bank looks like a device problem and is a map-versus-stride interaction — and changing either fixes it.
This bidirectional reading is what the module was for. Forwards: a workload's shape predicts DDR behaviour. Backwards: a DDR signature narrows the plausible upstream causes and names the layer — and the second direction is the one that saves time in a real system.
15. Failure Modes
| Symptom | Candidate causes | Discriminating measurement |
|---|---|---|
| Bandwidth far below device capability, low utilisation | compute-bound, or latency-bound — §8 | outstanding capacity against mean concurrency |
| High utilisation, poor performance | poor reuse, or partial-line use — 29.3 §10 | reuse ratio; bytes used against delivered |
| Throughput collapses when a second engine starts | interleaving — §6, or bank contention | row-hit rate with one engine versus two |
| Periodic throughput dips | barrier synchronisation — §6 | concurrency variance — §10 |
| Regular stride, terrible performance | the stride aliases the bank field — §5 | bank distribution of accesses |
| Doubling outstanding depth changes nothing | already past the useful point — 29.1 §17 | reorder-buffer depth and mean concurrency |
Row six is the one that saves hardware. Chapter 29.1 §17 established three regimes for outstanding depth, and beyond the reorder tracker's depth extra capacity adds queueing delay and no throughput. Measuring before adding is a one-counter check.
16. Misconceptions
“Accelerators only care about bandwidth.” §8. Four bottleneck classes, and three of the four have their fix outside the memory system.
“AI needs enormous memory bandwidth.” §2. Untiled traffic does. Tiling reduces it by the reuse factor — 128× in the worked model — with no change to the computation.
“Compute throughput tells you the bandwidth you need.” §3. It tells you the demand; the reuse factor tells you how much of that demand reaches DRAM. Chapter 26.4 owns the arithmetic-intensity form of this.
“More on-chip memory always helps.” §3. It raises the achievable tile size, which reduces traffic and moves the design toward compute-bound. Past that point it buys nothing.
“More outstanding requests always improve performance.” §15, and 29.1 §17. Beyond the scheduler's need for candidates and the reorder tracker's depth, extra depth adds delay only.
“A regular access pattern is a good access pattern.” §5. A stride that aliases the bank field lands every access in one bank — perfectly regular and serialised.
“Gather traffic is like pointer chasing.” §5. It is scattered like pointer chasing but deeply concurrent, because the indices arrive in bulk. That makes it a bank workload rather than a latency workload.
“The controller can fix a bad access pattern.” §9. Reordering, row locality and bank parallelism all need something in the stream. A stream with none cannot be repaired downstream.
“Row-hit scheduling is always optimal.” Chapter 29.3 §8, §5. Maximising row hits can starve a requester (29.2 §7), and a scattered map trading row hits for bank parallelism can be better.
“Low utilisation means the memory system is fine.” §8. It is equally the signature of compute-bound and latency-bound, which need opposite fixes.
“A high reuse ratio means good performance.” §8's report (B). The compute-bound run had the highest reuse ratio and the worst utilisation — it needed little from DRAM and was limited elsewhere.
17. Interview Reasoning
What decides an accelerator's DDR traffic? How the computation was tiled and how much reuse on-chip storage can hold. The external stream is what reuse failed to capture.
How much can tiling change it? In the worked model, traffic scales as one over the tile dimension — two orders of magnitude across a realistic range, with the same arithmetic performed.
Does more on-chip memory always help? It raises the achievable tile size, reducing traffic and shifting the design toward compute-bound. Past the crossover, further memory-system improvement buys nothing.
Why is an accelerator's read/write structure favourable? It is phased — long read runs then write runs — which minimises turnaround, unlike a DMA copy's alternation.
What erodes that? Several engines in different phases, and double buffering that overlaps output writes with input reads. The turnaround advantage and the latency-hiding advantage are in tension.
Name the four bottleneck classes and how you tell them apart. Compute, bandwidth, latency and synchronisation. Bandwidth shows high utilisation; synchronisation shows high variance at a moderate mean; compute and latency both show low utilisation and are separated by whether enough was in flight.
Why is a gather workload unusual? Scattered addresses with high concurrency, because the indices arrive in bulk. It stresses bank parallelism rather than row locality.
A regular strided kernel performs terribly. What do you suspect? The stride aliasing the bank field, so every access lands in one bank. Check the bank distribution; changing the stride or the map fixes it.
Can the memory controller fix a poor access pattern? Only to the extent the stream contains opportunities. Reordering needs candidates, row hits need consecutive accesses, bank parallelism needs concurrency. A stream with none cannot be repaired downstream.
You see a deep queue, short runs and a low row-hit rate. What are the candidates? Multicore interleaving, scatter-gather fragmentation, or a gather kernel. The aggregate trace cannot separate them — you need per-source attribution.
18. Exercises
-
§2 derives traffic scaling as
1/T. Re-derive it for a tiling that holds only one operand's tile on chip, and say how the scaling changes. -
§3 shows
t_compute/t_loadgrowing asT. Derive the tile size at which a design crosses from bandwidth-bound to compute-bound, given a compute rate and an achieved bandwidth. -
§4 says double buffering erodes the phase structure. Quantify the turnaround cost of full overlap versus full phasing, using 29.4 §8's model.
-
§5 describes a stride aliasing the bank field. Using 18.1, derive the strides that alias for a stated bank-field position, and say what stride change fixes it.
-
§6 claims interleaving
Eengines gives runs of aboutR/E. State the condition on the scheduler's queue window under which that is too pessimistic. -
§8's table pairs compute-bound with latency-bound on two measurements. Design the smallest additional measurement that separates them, and say where it must be taken.
-
§10 cannot derive reuse from the request stream. Argue for and against estimating it from the ratio of distinct lines to total requests, then decide.
-
§13's last row says three requesters want three different things from QoS. Design an arbitration policy that serves all three, and identify which of 29.2 §6's eight objectives you sacrificed.
19. Where This Goes
An accelerator's DDR traffic is what on-chip reuse failed to capture, and the reuse factor is set by tiling — a decision made before any memory request exists. Traffic scales inversely with the tile dimension; tiling also shifts the design between bandwidth-bound and compute-bound; the phase structure accelerators produce is the best case for turnaround and is eroded by multiple engines and by double buffering; and a gather kernel is the unusual combination of high concurrency with no locality, which stresses banks rather than rows.
Four results carry forward. Compute throughput is not DDR bandwidth — the reuse factor stands between them, and 26.4 owns the arithmetic-intensity form of the same statement. Four bottlenecks, and their DDR signatures overlap in pairs — variance separates synchronisation, and outstanding capacity separates compute from latency. Three of four fixes are outside the memory system. And the three requester classes want three different things from arbitration, which no single priority number expresses.
Two things stay open, and both are honest limits. Reuse cannot be measured from the request stream — the captured part left no trace, so the ratio needs a declared number and §11's properties cannot check it. And row three of §14's table maps one signature to three upstream causes, which only per-source attribution resolves.
Module 29 as a whole has one result, and it is bidirectional.
Forwards: a requester's intent passes through caches or descriptors or tiling, through a fabric, through a controller front end, a queue, an address map and a scheduler — and what arrives at DRAM is a stream whose count, identity, order and locality were all changed on the way. Nothing a requester expresses reaches the device intact, and nothing the device offers is visible upstream.
Backwards: a DDR trace narrows the plausible causes and names the layer that transformed them — and that is the direction that saves time, because it is the direction a real investigation actually runs.
And the law the module closes on is the one §9 states. Architecture creates the opportunities; scheduling exploits the ones that remain. The corollary is what makes it useful rather than merely true: when a DDR problem has no memory-side fix, that is not a dead end — it is a localization, and Module 28's discipline says a localization is the goal.
Continue learning
Related tutorials
- Related topic
AXI-to-DDR Flow
One AXI transaction is not one DDR command, acceptance is not service, and the ordering the bus guarantees is not the ordering the scheduler uses. This is the transformation between them.
- Related topic
Memory Controllers in SoCs
A semantic adapter between two sides that disagree about what a request is, what order means, what fairness means, and whether priority should matter at all.
- Related topic
CPU Access Patterns
Most loads never reach DRAM. What does is shaped less by how many loads a program executes than by its dependency structure — and that decides whether latency or bandwidth matters.
- Related topic
DMA Access Patterns
A DMA transfer is not one stream. It is a payload stream and a descriptor stream with opposite properties, and the second is the one nobody counts.
Standards & specifications
- Governing standard
- JEDEC JESD79 (DDR SDRAM)(opens JEDEC Solid State Technology Association in a new tab)
Defines the DDR SDRAM device itself — signals, command encoding, mode registers, timing parameters and the initialisation sequence — one document per generation. Memory-controller microarchitecture, address-mapping policy, PHY training algorithms and board-level design are not specified by it.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the DDR curriculum.