DDR · Module 29
DMA Access Patterns
A DMA transfer is not one stream. It is a payload stream and a descriptor stream with opposite properties, and the second is the one nobody counts.
Chapter 29.3 established that CPU DDR traffic is the residue a cache hierarchy leaves, shaped by dependency structure. A DMA engine has neither a cache hierarchy filtering its accesses nor a dependency chain limiting its concurrency, so almost every property from that chapter inverts.
But the interesting question is not why DMA traffic is better for DRAM, because it is not always better. It is why it is differently shaped — and the two most widely repeated beliefs about it are both architecture-dependent rather than true:
| Common belief | Reality |
|---|---|
| “DMA bypasses caches” | architecture-dependent — §10 |
| “DMA traffic is sequential” | descriptor-dependent — §6 |
And there is a third thing, which is this chapter's own contribution: a DMA transfer is not one stream. It is a payload stream and a descriptor stream with opposite properties, and the second is routinely left out of every bandwidth calculation.
1. What Inverts, and What Does Not
Against 29.3's CPU baseline.
| Property | CPU miss traffic | DMA traffic |
|---|---|---|
| Filtered by a cache hierarchy | yes — most accesses never leave | usually not, but see §10 |
| Concurrency limited by dependencies | yes — §4 there | no — programmed, not dependent |
| Request size | one cache line | as large as the engine and bus allow |
| Traffic the requester did not ask for | writebacks, prefetch, translation | descriptors — §3 |
| Latency sensitivity | high; a core waits | low for payload, high for descriptors — §3 |
| Locality | depends on the program | depends on the descriptors — §6 |
Row two is the big inversion and it is why DMA is a throughput requester. Chapter AXI 19.2 owns the design consequence — deep outstanding, long bursts — and from the DDR side the consequence is that the controller gets many candidates at once, which is exactly the raw material 23.4's reordering needs and 29.3 §5's pointer-chase denied it.
Row five is where the chapter's contribution starts. DMA is not latency-sensitive is true of the payload and false of the descriptors, and treating the engine as one homogeneous requester loses that.
2. What a DMA Engine Actually Issues
A programmed transfer produces at least two kinds of memory access, and often three.
| Class | What it is | Typical shape |
|---|---|---|
| Descriptor reads | fetching the instructions that say what to move | small, scattered, serialising |
| Payload reads | reading the source data | large, sequential within a fragment |
| Payload writes | writing the destination | large, sequential within a fragment |
| Status/completion writes | reporting progress | small, occasional |
CURRICULUM-DERIVED from AXI 11.7 and 19.2: the engine is told move N bytes from A to B, and in a descriptor-driven design that instruction itself lives in memory.
So the DDR-visible stream is an interleaving of classes with opposite properties, and that is the shape the controller actually schedules:
what the controller sees from one descriptor-driven transfer
[descriptor read] small, one address, and NOTHING can proceed
until it returns
[payload read] x N large, sequential
[payload write] x N large, sequential, to a DIFFERENT region
[status write] small
[descriptor read] the NEXT descriptor -- small, scattered again
[payload ...] ...
two facts follow immediately:
1. the stream alternates between two localities -- the payload
regions and the descriptor region -- so even a perfectly
sequential payload arrives with periodic excursions
elsewhere.
2. the read and write payload streams target DIFFERENT regions,
so a copy is TWO sequential streams, not one.Point two is easy to miss and it matters for banks. A memory-to-memory copy reads from one region and writes to another. If the map places those two regions in the same bank but different rows, every read/write alternation is a row conflict — 29.3 §7's interleaving problem, arising inside a single requester rather than between two.
3. The Descriptor Class Is the Interesting One
The traffic nobody counts, with three properties that all cut against the payload's.
It is small. A descriptor is a few tens of bytes against a payload of kilobytes. So it contributes almost nothing to bandwidth — which is precisely why it is left out of bandwidth calculations, and why leaving it out is safe for bandwidth and wrong for latency.
It is serialising. The engine cannot issue payload requests for a fragment until it knows the fragment's address and length. So a descriptor read is a dependent access — structurally the same hazard as 29.3 §11's translation miss, and with the same effect: it converts a high-concurrency requester into a momentarily serial one.
And it is scattered. Descriptors typically live in a region unrelated to the payload, so each fetch is an excursion to a different row and often a different bank.
DERIVED under a stated ILLUSTRATIVE model.
transfer : 4 KiB per fragment
descriptor : 32 bytes
payload burst : 256 bytes per request
descriptor latency, row conflict : 1 row-conflict access
payload latency, pipelined, 16 outstanding : effectively hidden
bandwidth accounting
bytes of descriptor per fragment : 32
bytes of payload per fragment : 4096
descriptor share of bytes : 32/4128 = 0.78%
-> negligible, and correctly ignored for bandwidth
time accounting, ILLUSTRATIVE latencies
descriptor fetch (serial, row conflict) : 1 x L_conflict
payload, 16 requests deep : ~ (4096/256)/16 x L
= 1 x L, pipelined
-> the descriptor fetch costs roughly as much TIME as the
entire 4 KiB payload it describes
so a class contributing 0.78% of the BYTES can contribute ~50% of
the TIME. And halving the fragment size doubles the descriptor
count while halving the payload per fetch -- so the ratio gets
worse, fast.The model is crude and its assumptions matter: it assumes the descriptor fetch is not overlapped with anything, and a well-designed engine prefetches the next descriptor while the current payload transfers, which hides most of this. That is exactly the point. Descriptor prefetching is not an optimisation detail — it is the mechanism that keeps a descriptor-driven engine from being latency-bound by a class carrying under 1% of its bytes.
And it predicts a real symptom. An engine whose descriptor prefetch is disabled, or whose descriptor chain is too short to prefetch from, loses throughput in a way no bandwidth analysis explains — because the bandwidth analysis correctly ignored the class responsible.
4. One Transfer, Two Classes
The self-loop is the chapter's central point drawn as a message that accomplishes nothing. Between the descriptor returning and the payload issuing, the engine's entire concurrency advantage is unavailable — it has one access in flight, exactly like 29.3 §5's pointer chase.
And next descriptor, concurrently is drawn as a separate message deliberately. It is the mechanism that hides the serialisation, and an engine without it has a periodic serial stall its bandwidth model does not predict.
5. Long Sequential Transfers — the DDR Consequence
What a large sequential payload actually buys at the device.
CURRICULUM-DERIVED from 23.3 and Module 9: consecutive addresses within one row become row hits, and a row hit costs one command where a conflict costs three.
DERIVED under a stated ILLUSTRATIVE model — 1 KiB rows, 64-byte lines, 16 lines per row:
a 64 KiB sequential payload
lines : 65536 / 64 = 1024
rows spanned : 1024 / 16 = 64
row openings : 64 (one ACT per row)
accesses : 1024
row-hit rate : (1024 - 64) / 1024 = 93.75%
commands, close-page policy (23.5):
ACT 64 + RD 1024 + PRE 64 = 1152 commands
commands if every access were a conflict:
PRE 1024 + ACT 1024 + RD 1024 = 3072 commands
so sequentiality reduces command count by ~2.7x on this model --
and command bandwidth is a real, separate resource from data
bandwidth (23.2 owns the composition).The 93.75% figure is the same number 29.3 §8 derived for a streaming CPU workload, and that is the honest conclusion: a sequential DMA payload and a sequential CPU stream present the same shape to DRAM. The difference is not the shape — it is that the DMA engine can sustain it with high concurrency while a CPU needs prefetch and enough miss capacity to do so.
And it does not depend on burst length the way people expect. Chapter AXI 13.4 owns bus burst efficiency; from the DDR side what produces the row hits is address consecutiveness, not the size of the bus transaction that delivered those addresses. Sixteen 64-byte requests to consecutive lines and one 1 KiB request give the controller the same column sequence within a row.
6. Scatter-Gather Destroys the Locality
The correction to DMA traffic is sequential.
A scatter-gather transfer moves data described by a list of fragments, and the fragments need not be adjacent, ordered, or large.
| Fragment size | Payload shape at DRAM | Descriptor overhead |
|---|---|---|
| Large — many KiB | sequential within each; excursions between | negligible |
| Medium — a few KiB | mostly sequential; frequent row changes | noticeable |
| Small — below a row | essentially scattered | dominant — §3 |
DERIVED, extending §3's model to small fragments:
move 64 KiB, ILLUSTRATIVE, 32-byte descriptors, 1 KiB rows
as ONE 64 KiB fragment
descriptors : 1 -> 32 bytes
rows spanned: 64 -> 64 row openings
descriptor share of bytes : 0.05%
as 1024 fragments of 64 bytes each
descriptors : 1024 -> 32,768 bytes
rows: each 64-byte fragment may be in a different row
-> up to 1024 row openings
descriptor share of bytes : 32768/98304 = 33%
the same 64 KiB of useful payload:
16x more row openings
descriptors from 0.05% to 33% of delivered bytes
and 1024 SERIALISING descriptor fetches instead of oneSo "DMA" spans a range from the best-case DRAM workload to one worse than scattered CPU misses, and the descriptor list decides which. That is why the belief is architecture-dependent rather than false: a DMA engine can produce ideal traffic, and a fragmented buffer guarantees it will not.
And the origin of the fragmentation is usually far upstream. A buffer assembled from small allocations, a network packet chain, or a page-granular mapping of a large logical buffer all produce fragment lists the DMA engine merely executes. The locality was destroyed before the engine saw the work — this module's recurring point, and the reason 29.5 §8 states it as a law.
7. Alignment and Boundary Effects
Two mechanisms that split a transfer the programmer thought was contiguous.
Unaligned starts. Chapter AXI 7.7 owns narrow and unaligned transfers. From the DDR side, a payload that does not start on a line boundary means the first and last accesses touch partial lines — and a partial write requires masking (6.11 owns the data mask) or a read-modify-write, which turns one write into a read and a write.
Boundary crossings. Chapter 29.1 §4 owns splitting. A fragment crossing a row boundary needs two rows; one crossing a channel boundary becomes two controller requests.
| Effect | Cost | Visible where |
|---|---|---|
| Unaligned start or end | partial-line masking, or read-modify-write | write traffic higher than payload size |
| Row-boundary crossing | an extra row opening per crossing | row-hit rate below the §5 model |
| Channel-boundary crossing | one transfer becomes two requests | request count above expectation |
The read-modify-write case is the one worth naming, because it converts a write-only transfer into mixed traffic. CURRICULUM-DERIVED from 14.6, which owns read/write turnaround: mixed traffic costs turnarounds that a pure write stream does not. So an unaligned write transfer pays twice — once for the extra read and once for the turnaround it creates.
8. A Copy Is the Worst Case for Turnaround
§2 noted that a memory-to-memory copy is two streams. This section is the cost of that, because a copy has a property no other workload in this module shares: its read/write mix is 50/50 by construction.
CURRICULUM-DERIVED from 14.6, which owns column-to-column spacing and the read/write turnaround cases that share the same structure: switching the data bus between reading and writing costs time, and that cost is paid per switch.
| Workload | Read/write mix | Switches per N accesses |
|---|---|---|
| CPU streaming read — 29.3 §6 | read-dominated plus writebacks | few |
| DMA device-to-memory | writes only, after the source read | few |
| DMA memory-to-memory copy | 50/50 | up to N, if unbuffered |
DERIVED under a stated ILLUSTRATIVE model — a turnaround costing T cycles, a burst occupying B cycles of data bus:
copy 64 KiB in 256-byte units: 256 reads + 256 writes = 512 accesses
worst case: strict alternation R,W,R,W,...
switches : 511
data-bus cycles : 512 x B
turnaround cycles : 511 x T
efficiency : 512B / (512B + 511T)
with ILLUSTRATIVE B = 8, T = 6:
4096 / (4096 + 3066) = 57%
grouped in runs of 16: R x16, W x16, R x16, ...
switches : 511/16 ~= 32
turnaround cycles : 32 x 6 = 192
efficiency : 4096 / (4096 + 192) = 96%
the SAME copy, the same total bytes, 57% versus 96% of the data
bus doing useful work -- and the only difference is how many times
the direction changed.So a copy's efficiency is decided by buffering, not by bandwidth. The engine must hold enough read data to issue a run of writes, and 29.2 §8's separate read and write queues exist to let the controller group what the engine interleaved. Either side can do the grouping; if neither does, the bus alternates.
Two consequences worth stating.
A small buffer in the engine is a large DRAM cost. An engine with room for one burst must alternate. The buffering that fixes it is in the requester, but the symptom appears at DRAM as a turnaround-dominated bus — which is the module's recurring shape: a DDR-visible inefficiency whose lever is upstream.
And the controller's write queue is what absorbs the mismatch. A deep write queue lets the controller delay writes and group them, converting an alternating stream into runs. But 29.2 §8 also established that a full write queue eventually blocks acceptance — so the grouping is bounded by the queue depth, and beyond it the alternation returns.
This is also the one case where a copy is worse than a scatter-gather read. §6 showed fragmentation destroying locality; here a perfectly sequential copy destroys efficiency through direction changes instead — a different resource, the same lesson that sequential is not sufficient.
9. Backpressure Reaches a DMA Engine Differently
A saturated memory system eventually affects its requesters, and DMA is where the mechanism is clearest — because a DMA engine's response to pressure is visible as throughput rather than hidden as latency.
The path is a chain of finite buffers, not a combinational ready signal. That distinction matters and is worth stating carefully.
how pressure propagates -- and where it stops
DRAM busy / bank conflict
| the scheduler finds fewer legal candidates
v
controller queues fill (17.2)
| the front end deasserts ready (17.5)
v
fabric buffers absorb -- FOR A WHILE (AXI 3.3)
| each buffer decouples the layers around it
v
the fabric deasserts ready to the engine
|
v
the engine's outstanding capacity fills (AXI 8.1)
|
v
the engine stops issuing: THROUGHPUT FALLS
note what is NOT happening: there is no single ready signal from
DRAM to the engine. Every arrow is a separate handshake with a
buffer behind it, so pressure propagates as a FILLING FRONT with
a delay at each stage -- and if any buffer is deep enough, the
pressure never reaches further.So local acceptance does not mean memory service, and this is the module's second law in its backpressure form. Chapter 29.1 §2 established that acceptance is not service; the buffers between the layers are exactly why the two can diverge for a long time.
What each requester class does when the front arrives:
| Requester | Response to backpressure | Visible as |
|---|---|---|
| CPU | a core stalls on a dependent miss | latency — 29.3 §4 |
| DMA | the engine issues less; the transfer takes longer | throughput |
| Real-time engine | a deadline may be missed | a functional failure |
Row two is why DMA throttling is benign and row three is why it is not. A DMA transfer that takes twice as long is usually acceptable; a display engine that misses its deadline produces a visible artefact, which is why 29.2 §6's QoS exists and why an age bound alone is not a deadline guarantee.
And there is a feedback effect worth naming, because it inverts an expectation. Throttling a DMA engine reduces the offered load, which reduces queue occupancy, which can improve the locality the scheduler sees — because fewer interleaved streams compete. So a throttled system can deliver better per-request efficiency than an unthrottled one, which is 29.3 §7's interleaving argument running in reverse.
That is a real design lever rather than a curiosity. Deliberately limiting a bulk requester's outstanding depth can raise total system throughput by preserving another requester's row locality — and it is invisible to any analysis that treats bandwidth as additive.
10. Coherency Is Not Universal
The correction to DMA bypasses caches, stated carefully because the truth depends on the SoC.
Whether a DMA path is coherent is an architectural choice, and both choices are common. The CHI track owns the mechanisms: CHI 20.4 owns cache-coherent DMA, and CHI 5.2 owns the IO-coherent request node — a requester participating in coherency without holding coherent caches of its own.
| Case | What happens | DDR consequence |
|---|---|---|
| Non-coherent | the engine accesses memory directly; software must maintain consistency | DMA traffic reaches DDR in full |
| IO-coherent | the fabric snoops caches on the engine's behalf — CHI 5.2 | some accesses never reach DDR; a snoop may satisfy them |
| Fully coherent | the engine participates as a caching agent | the engine's own reuse can reduce DDR traffic further |
Row two has a DDR consequence people find surprising. A coherent DMA read whose data is dirty in a CPU cache is satisfied from that cache. So making a DMA path coherent can reduce DDR traffic, not just add snoop latency — the same effect 29.2 §3 noted for coherent requesters generally.
And the non-coherent case has a DDR consequence too, in the opposite direction. Software maintaining consistency means cache maintenance operations — and a writeback forced by a maintenance operation is DDR write traffic caused by the DMA setup rather than by the transfer. So the non-coherent path can generate DDR traffic the transfer size does not account for.
11. Translation Is Not Universal Either
The same treatment, briefly, because 29.1 §16 owns the address-space distinction.
A DMA engine may issue addresses that require translation, or physical addresses directly. Both are common: a device programmed by a driver with physical addresses needs none; one operating on a process's address space does.
| Case | DDR consequence |
|---|---|
| No translation on the path | the engine's addresses are the ones the mapper sees |
| Translation present, cached, hitting | no additional DDR traffic |
| Translation present, missing | additional serialising reads — 29.3 §11's mechanism |
The third row compounds with §6's fragmentation in a specific way. A scatter-gather list of many small fragments touches many pages, so it stresses translation caching exactly when it is also stressing row locality — and both effects push the same direction. A fragmented transfer can therefore be worse than the fragment arithmetic alone predicts, and that interaction is the honest reason to prefer larger fragments beyond the descriptor-overhead argument.
This chapter does not teach translation, and the architectural claim is only that where the layer exists it adds serialising traffic with locality unrelated to the payload.
12. Separating the Two Classes
// ---------------------------------------------------------------------
// dma_class_separator -- CLASSIFICATION: synthesisable, BINDABLE.
//
// WHAT IT DOES: §2's two classes, measured separately. The key output
// is a PAIR of shares -- descriptor share of BYTES and descriptor
// share of STALL TIME -- because §3 shows a class carrying under 1% of
// the bytes can cost ~50% of the time.
//
// WHY RUN LENGTH AND NOT JUST SIZE (§6): a scatter-gather transfer can
// issue large requests to scattered addresses. What predicts row hits
// is CONSECUTIVENESS -- how many requests in a row continue the
// previous address -- and §5 is explicit that request size does not
// determine it.
//
// WHY THE SERIAL GAP IS MEASURED (§3): the descriptor fetch is
// SERIALISING. An engine that prefetches descriptors hides it; one
// that does not has a periodic stall its bandwidth model does not
// predict. The gap is the direct evidence.
//
// WHAT IT IS NOT: a DMA engine or a descriptor parser. Classification
// comes from a platform-supplied region hint, because descriptor
// FORMATS are engine-specific and none is reproduced here.
//
// SYNTHESIS: two counter sets, one address comparator, a run counter.
// No memory, no CAM.
// ---------------------------------------------------------------------
module dma_class_separator #(
parameter int ADDR_W = 40,
parameter int LINE_LOG = 6,
parameter int ACC_W = 40
)(
input logic clk,
input logic rst_n,
input logic req_fire,
input logic req_is_read,
input logic req_is_descriptor, // platform region hint
input logic [ADDR_W-1:0] req_addr,
input logic [15:0] req_bytes,
input logic desc_done, // a descriptor completed
input logic payload_active, // any payload in flight
input logic clear,
// ---- per-class byte and request accounting
output logic [ACC_W-1:0] desc_bytes,
output logic [ACC_W-1:0] payload_bytes,
output logic [31:0] desc_reqs,
output logic [31:0] payload_reqs,
output logic [31:0] payload_reads,
output logic [31:0] payload_writes,
// ---- §3's time accounting: the serialising gap
output logic in_serial_gap,
output logic [ACC_W-1:0] desc_stall_cycles,
output logic [ACC_W-1:0] payload_active_cycles,
output logic [ACC_W-1:0] window_cycles,
// ---- §5/§6's consecutiveness
output logic [31:0] run_continues,
output logic [31:0] run_breaks,
output logic [31:0] longest_run,
output logic overflow_seen
);
initial begin
if (ADDR_W < 8) $fatal(1, "dma_class_separator: ADDR_W must be >= 8");
if (LINE_LOG < 1 || LINE_LOG >= ADDR_W)
$fatal(1, "dma_class_separator: LINE_LOG must be in 1..ADDR_W-1");
if (ACC_W < 24) $fatal(1, "dma_class_separator: ACC_W must be >= 24 for a useful window");
end
logic [ADDR_W-1:0] next_expected; // where a continuing run would go
logic have_prev;
logic [31:0] cur_run;
// A request continues the run when it starts exactly where the
// previous one ended. Compared in LINE units so a sub-line
// difference does not break a run that is consecutive in lines.
logic continues;
always_comb begin
continues = have_prev && ((req_addr >> LINE_LOG) == (next_expected >> LINE_LOG));
end
always_ff @(posedge clk) begin
if (!rst_n || clear) begin
desc_bytes <= '0;
payload_bytes <= '0;
desc_reqs <= '0;
payload_reqs <= '0;
payload_reads <= '0;
payload_writes <= '0;
in_serial_gap <= 1'b0;
desc_stall_cycles <= '0;
payload_active_cycles <= '0;
window_cycles <= '0;
run_continues <= '0;
run_breaks <= '0;
longest_run <= '0;
cur_run <= '0;
next_expected <= '0;
have_prev <= 1'b0;
overflow_seen <= 1'b0;
end else begin
if (window_cycles != {ACC_W{1'b1}}) window_cycles <= window_cycles + 1'b1;
else overflow_seen <= 1'b1;
// ---- §3's serial gap. It OPENS when a descriptor completes with
// no payload in flight -- the engine now knows what to do and has
// not yet issued it -- and CLOSES when payload becomes active.
// An engine that prefetches descriptors keeps payload active
// across the descriptor's completion, so the gap never opens.
if (desc_done && !payload_active) in_serial_gap <= 1'b1;
else if (payload_active) in_serial_gap <= 1'b0;
if (in_serial_gap && desc_stall_cycles != {ACC_W{1'b1}})
desc_stall_cycles <= desc_stall_cycles + 1'b1;
if (payload_active && payload_active_cycles != {ACC_W{1'b1}})
payload_active_cycles <= payload_active_cycles + 1'b1;
if (req_fire) begin
if (req_is_descriptor) begin
if (desc_reqs != 32'hFFFF_FFFF) desc_reqs <= desc_reqs + 1'b1;
if (desc_bytes <= {ACC_W{1'b1}} - 16'hFFFF)
desc_bytes <= desc_bytes + {{(ACC_W-16){1'b0}}, req_bytes};
// A descriptor fetch is an excursion, so it BREAKS the
// payload run rather than continuing it -- §2's point that
// the stream alternates between two localities. Counting it
// as a continuation would overstate payload locality.
if (cur_run > longest_run) longest_run <= cur_run;
cur_run <= '0;
have_prev <= 1'b0;
end else begin
if (payload_reqs != 32'hFFFF_FFFF) payload_reqs <= payload_reqs + 1'b1;
if (payload_bytes <= {ACC_W{1'b1}} - 16'hFFFF)
payload_bytes <= payload_bytes + {{(ACC_W-16){1'b0}}, req_bytes};
if (req_is_read) begin
if (payload_reads != 32'hFFFF_FFFF) payload_reads <= payload_reads + 1'b1;
end else begin
if (payload_writes != 32'hFFFF_FFFF) payload_writes <= payload_writes + 1'b1;
end
// ---- run tracking, payload only
if (continues) begin
if (run_continues != 32'hFFFF_FFFF) run_continues <= run_continues + 1'b1;
cur_run <= cur_run + 1'b1;
end else begin
if (have_prev && run_breaks != 32'hFFFF_FFFF)
run_breaks <= run_breaks + 1'b1;
if (cur_run > longest_run) longest_run <= cur_run;
cur_run <= 32'd1;
end
next_expected <= req_addr + {{(ADDR_W-16){1'b0}}, req_bytes};
have_prev <= 1'b1;
end
end
end
end
endmoduleA descriptor fetch breaks the payload run rather than continuing it, and that choice encodes §2's argument. The descriptor is an excursion to a different region; counting it as part of the payload run would overstate payload locality and hide the very interleaving this block exists to expose.
in_serial_gap opens only when a descriptor completes with no payload in flight. That condition is precisely the engine is not prefetching — an engine that overlaps descriptor fetch with payload transfer never satisfies it, so the counter distinguishes the two designs directly rather than by inference.
And next_expected is the previous address plus its byte count, not the previous address plus a fixed stride. So a run of differently-sized consecutive requests still counts as one run, which is what §5 requires: consecutiveness, not uniform size, produces row hits.
13. What the Assertions Prove
// Bound to §12's separator. Every property carries
// `disable iff (!rst_n)`, and every antecedent is covered below.
//
// NOTE ON SCOPE: every property constrains counting and
// classification. None says anything about a DMA engine, a descriptor
// format, or whether a path is coherent -- §10 is explicit that the
// coherency case is architecture-dependent and this block cannot see
// it.
module dma_class_sva #(parameter int ACC_W = 40)(
input logic clk, rst_n, clear,
input logic req_fire, req_is_read, req_is_descriptor,
input logic desc_done, payload_active, in_serial_gap, overflow_seen,
input logic [ACC_W-1:0] desc_bytes, payload_bytes, window_cycles,
input logic [ACC_W-1:0] desc_stall_cycles, payload_active_cycles,
input logic [31:0] desc_reqs, payload_reqs, payload_reads, payload_writes,
input logic [31:0] run_continues, run_breaks, longest_run
);
// ---- P1. PARTITION. Every request is exactly one class. A request
// in neither or both means the classification input was never
// resolved, and §3's two shares become meaningless.
property p_class_partition;
@(posedge clk) disable iff (!rst_n)
(req_fire && req_is_descriptor) |=> (desc_reqs == $past(desc_reqs) + 1);
endproperty
assert property (p_class_partition)
else $error("separator: a descriptor request was not counted");
property p_payload_counted;
@(posedge clk) disable iff (!rst_n)
(req_fire && !req_is_descriptor) |=> (payload_reqs == $past(payload_reqs) + 1);
endproperty
assert property (p_payload_counted)
else $error("separator: a payload request was not counted");
// ---- P2. FORBIDDEN. A descriptor request never increments the
// payload counters. §3's byte share depends on the split being
// clean, and a leak would understate the payload.
property p_no_cross_counting;
@(posedge clk) disable iff (!rst_n)
(req_fire && req_is_descriptor)
|=> (payload_reqs == $past(payload_reqs)
&& payload_bytes == $past(payload_bytes));
endproperty
assert property (p_no_cross_counting)
else $error("separator: a descriptor leaked into payload accounting");
// ---- P3. PARTITION. Payload reads plus writes equal payload
// requests. §7's read-modify-write case makes the mix matter.
property p_payload_direction_partition;
@(posedge clk) disable iff (!rst_n)
((payload_reads + payload_writes) == payload_reqs)
|| (payload_reqs == 32'hFFFF_FFFF);
endproperty
assert property (p_payload_direction_partition)
else $error("separator: %0d reads + %0d writes != %0d payload requests",
payload_reads, payload_writes, payload_reqs);
// ---- P4. FORBIDDEN. A descriptor breaks the run. §12: counting it
// as a continuation would overstate payload locality and hide the
// interleaving this block exists to expose.
property p_descriptor_breaks_run;
@(posedge clk) disable iff (!rst_n)
(req_fire && req_is_descriptor)
|=> (run_continues == $past(run_continues));
endproperty
assert property (p_descriptor_breaks_run)
else $error("separator: a descriptor continued the payload run");
// ---- P5. INVARIANT. Run continuations plus breaks never exceed
// payload requests -- each payload request is classified at most
// once.
property p_runs_le_payload;
@(posedge clk) disable iff (!rst_n)
((run_continues + run_breaks) <= payload_reqs);
endproperty
assert property (p_runs_le_payload)
else $error("separator: more run classifications than payload requests");
// ---- P6. INVARIANT. The longest run bounds any run that ended, and
// never decreases -- a high-water mark that fell would erase the
// best locality observed.
property p_longest_run_monotone;
@(posedge clk) disable iff (!rst_n)
(!clear) |=> (longest_run >= $past(longest_run));
endproperty
assert property (p_longest_run_monotone)
else $error("separator: longest_run decreased");
// ---- P7. FORBIDDEN. The serial gap never opens while payload is
// active. §3, §12: the gap IS "the engine is not overlapping", and
// an open gap with active payload would misreport a prefetching
// engine as a serialising one.
property p_no_gap_while_payload_active;
@(posedge clk) disable iff (!rst_n)
payload_active |=> !in_serial_gap;
endproperty
assert property (p_no_gap_while_payload_active)
else $error("separator: serial gap asserted while payload was active");
// ---- P8. The gap opens on a descriptor completing with no payload.
property p_gap_opens_on_idle_descriptor;
@(posedge clk) disable iff (!rst_n)
(desc_done && !payload_active) |=> in_serial_gap;
endproperty
assert property (p_gap_opens_on_idle_descriptor)
else $error("separator: gap did not open on an unoverlapped descriptor");
// ---- P9. INVARIANT. Stall and active cycles each fit the window.
// §3's time share is a fraction, and a value above the window would
// make it meaningless.
property p_time_shares_are_fractions;
@(posedge clk) disable iff (!rst_n)
((desc_stall_cycles <= window_cycles)
&& (payload_active_cycles <= window_cycles)) || overflow_seen;
endproperty
assert property (p_time_shares_are_fractions)
else $error("separator: a time share exceeds the window");
// ---- P10. INVARIANT. Stall and active are mutually exclusive by
// construction, so their sum also fits the window.
property p_stall_and_active_disjoint;
@(posedge clk) disable iff (!rst_n)
!(in_serial_gap && payload_active);
endproperty
assert property (p_stall_and_active_disjoint)
else $error("separator: gap and payload-active overlapped");
// ---- P11. FORBIDDEN. Bytes accumulate only on a request.
property p_bytes_only_on_request;
@(posedge clk) disable iff (!rst_n)
(!req_fire) |=> ((desc_bytes == $past(desc_bytes))
&& (payload_bytes == $past(payload_bytes)));
endproperty
assert property (p_bytes_only_on_request)
else $error("separator: bytes accumulated with no request");
// ---- P12. FORBIDDEN. Overflow is sticky. A saturated accumulator
// makes both of §3's shares silently wrong.
property p_overflow_sticky;
@(posedge clk) disable iff (!rst_n)
(overflow_seen && !clear) |=> overflow_seen;
endproperty
assert property (p_overflow_sticky)
else $error("separator: overflow flag cleared without a clear");
// ---- antecedent covers.
cover property (@(posedge clk) disable iff (!rst_n) req_fire);
cover property (@(posedge clk) disable iff (!rst_n) req_fire && req_is_descriptor);
cover property (@(posedge clk) disable iff (!rst_n) req_fire && !req_is_descriptor);
cover property (@(posedge clk) disable iff (!rst_n) req_fire && !req_is_descriptor && req_is_read);
cover property (@(posedge clk) disable iff (!rst_n) req_fire && !req_is_descriptor && !req_is_read);
cover property (@(posedge clk) disable iff (!rst_n) desc_done);
cover property (@(posedge clk) disable iff (!rst_n) desc_done && payload_active);
cover property (@(posedge clk) disable iff (!rst_n) desc_done && !payload_active);
cover property (@(posedge clk) disable iff (!rst_n) in_serial_gap);
cover property (@(posedge clk) disable iff (!rst_n) payload_active);
cover property (@(posedge clk) disable iff (!rst_n) run_continues != 32'd0);
cover property (@(posedge clk) disable iff (!rst_n) run_breaks != 32'd0);
cover property (@(posedge clk) disable iff (!rst_n) longest_run > 32'd16);
cover property (@(posedge clk) disable iff (!rst_n) overflow_seen);
// The !clear antecedent of the monotone high-water-mark properties,
// published so a silent pass is distinguishable from a run in which
// clear was never exercised -- 27.2 §7's argument applied to the
// least interesting-looking antecedent in the file.
cover property (@(posedge clk) disable iff (!rst_n) clear);
endmodule14. DV — Testing the Separation
// SIMULATION-ONLY. Independent reference. It stores the request stream
// in QUEUES and derives runs and shares by rescanning, rather than by
// the DUT's incremental next-expected comparison.
class dma_reference;
int addrs[$];
int sizes[$];
bit is_desc[$];
bit is_read[$];
function void req(int a, int s, bit d, bit r);
addrs.push_back(a); sizes.push_back(s);
is_desc.push_back(d); is_read.push_back(r);
endfunction
function int desc_bytes();
int n = 0;
foreach (sizes[i]) if (is_desc[i]) n += sizes[i];
return n;
endfunction
function int payload_bytes();
int n = 0;
foreach (sizes[i]) if (!is_desc[i]) n += sizes[i];
return n;
endfunction
// §3's byte share. Returns -1.0 when nothing was observed, because
// 0.0 would read as "descriptors are free" rather than "no data".
function real desc_byte_share();
int tot = desc_bytes() + payload_bytes();
if (tot == 0) return -1.0;
return real'(desc_bytes()) / real'(tot);
endfunction
// Runs by rescanning: a payload request continues when it starts
// where the previous PAYLOAD request ended, and a descriptor breaks
// the chain -- §12's rule, implemented differently.
function int longest_payload_run(int line_log);
int best = 0, cur = 0, expect = -1;
foreach (addrs[i]) begin
if (is_desc[i]) begin
if (cur > best) best = cur;
cur = 0; expect = -1;
end else begin
if (expect >= 0 && (addrs[i] >> line_log) == (expect >> line_log)) cur++;
else begin
if (cur > best) best = cur;
cur = 1;
end
expect = addrs[i] + sizes[i];
end
end
if (cur > best) best = cur;
return best;
endfunction
// §6's fragmentation arithmetic, independently: rows touched by the
// payload under a stated row size.
function int rows_touched(int row_log);
int seen[int];
foreach (addrs[i]) if (!is_desc[i]) seen[addrs[i] >> row_log] = 1;
return seen.size();
endfunction
endclass| Check | What it establishes |
|---|---|
| Replay a single-fragment 64 KiB transfer | desc_byte_share ≈ 0.0005; longest_run spans the fragment |
| Replay 1024 × 64-byte fragments | desc_byte_share ≈ 0.33 — §6's arithmetic, independently |
Compare rows_touched for both | 64 versus up to 1024 — §6's 16× claim |
| 25,000 random streams through both | Incremental and rescanning run tracking agree |
| Descriptor mid-payload | Run breaks in both models — P4 |
| Consecutive requests of differing sizes | Counted as one run by both — §12's next_expected rule |
| Payload with a one-line gap | Run breaks; run_breaks rises |
desc_done with payload_active high | Gap never opens — the prefetching engine — P7 |
desc_done with payload_active low | Gap opens and accumulates — P8 |
Both in_serial_gap and payload_active forced | P10 fires |
| No requests at all | desc_byte_share returns −1.0, not 0.0 |
Accumulate past ACC_W | overflow_seen sets; P9's escape holds |
| Unaligned start producing a read-modify-write | payload_reads rises on a write-only transfer — §7 |
| Run with the engine idle | All 12 properties pass; all 14 covers empty |
Two runs are worth publishing, and the second is §3's claim measured:
TWO PASSING DMA CHARACTERISATIONS
(A) the separator was never exercised
all 12 properties PASS
desc_reqs / payload_reqs 0 / 0
desc_bytes / payload_bytes 0 / 0
in_serial_gap never asserted
longest_run 0
------------------------------------------------
cover req_fire 0 hits
... all 14 covers 0 hits
seven of twelve properties are implications and never armed.
The five INVARIANTS -- runs bounded, longest-run monotone,
time shares feasible, disjointness, overflow sticky -- pass
on zeros.
"DMA: descriptor overhead 0%" is produced by this run AND by
a transfer with no descriptors. window_cycles separates them.
(B) the same transfer, descriptor prefetch on and off
workload : 16 fragments of 4 KiB, 32-byte descriptors
PREFETCH ON PREFETCH OFF
desc_reqs 16 desc_reqs 16
desc_bytes 512 desc_bytes 512
payload_bytes 65,536 payload_bytes 65,536
desc_byte_share 0.78% desc_byte_share 0.78%
longest_run 64 longest_run 64
------------------------------------------------
desc_stall_cycles 0 desc_stall_cycles 3,184
payload_active 3,290 payload_active 3,290
total cycles 3,340 total cycles 6,530
------------------------------------------------
all 12 properties PASS all 12 properties PASS
all 14 covers HIT all 14 covers HIT
IDENTICAL byte accounting. Identical descriptor share of
bytes: 0.78% in both. Identical payload run length.
and a 1.95x difference in completion time, entirely in
desc_stall_cycles -- a class carrying 0.78% of the bytes.
diagnosis : (A) is the vacuity case. (B) is §3: a bandwidth
analysis of these two runs is IDENTICAL and correct, and it
explains none of the 1.95x. The time share is a separate
measurement and it is the one that found the difference.
the fix : (A) read window_cycles. (B) nothing in the memory
system -- the lever is descriptor prefetch in the engine, and
§3 says so.Report (B) is the chapter's central claim as numbers. Two runs with identical bandwidth accounting and a factor of two in completion time, and the byte-share metric everyone computes is identical in both.
15. Failure Modes
| Symptom | Candidate causes | Discriminating measurement |
|---|---|---|
| Throughput far below bus capability, bandwidth accounting looks fine | descriptor serialisation — §3 | desc_stall_cycles against payload_active_cycles |
| Row-hit rate far below the §5 model | fragmentation — §6; or an unfavourable map | longest_run and rows_touched |
| Write traffic exceeds the transfer size | unaligned read-modify-write — §7 | payload_reads non-zero on a write-only transfer |
| DDR bytes below transfer bytes | a coherent path satisfying reads from caches — §10 | DDR bytes against transfer bytes |
| DDR bytes above transfer bytes | maintenance traffic, or read-modify-write | same comparison, opposite sign |
| Throughput collapses with many small fragments | §6 and §11 compounding | fragment size distribution plus translation-miss rate |
Rows four and five are the same measurement reading in opposite directions, and together they are the evidence §10 says distinguishes a coherent path from a non-coherent one — without asserting any protocol mechanism.
16. Misconceptions
“DMA traffic is one stream.” §2. It is at least two — payload and descriptors — with opposite size, locality and latency sensitivity.
“DMA is not latency-sensitive.” §3. The payload is not; the descriptor fetch is serialising, and it can cost as much time as the payload it describes.
“Descriptor overhead is negligible.” §3. Negligible in bytes, potentially half the time. Those are two different measurements and only one of them is usually taken.
“DMA traffic is sequential.” §6. Sequential within a fragment. A scatter-gather list of sub-row fragments produces essentially scattered traffic with dominant descriptor overhead.
“DMA bypasses caches.” §10. Architecture-dependent. A coherent path can be satisfied from a cache and deliver fewer DDR bytes than the transfer size.
“A non-coherent path generates exactly the transfer's traffic.” §10. Software maintenance can force writebacks, so it can generate more.
“Longer bus bursts give better row locality.” §5. What produces row hits is address consecutiveness. Sixteen consecutive small requests and one large request give the controller the same column sequence.
“A copy is one sequential stream.” §2. It is two — a read region and a write region — and if the map puts them in the same bank on different rows, every alternation is a row conflict.
“Unaligned transfers cost a little extra.” §7. A partial write can become a read-modify-write, which adds a read and a read/write turnaround.
“Smaller fragments just mean more descriptors.” §6, §11. They also mean more row openings, more serialising fetches, and more translation pressure — all pushing the same direction.
17. Interview Reasoning
Why does DMA traffic look different from CPU miss traffic? No cache filtering it, no dependency chain limiting concurrency, and much larger requests — so the controller receives many candidates at once instead of one.
What traffic does a DMA engine generate besides payload? Descriptor reads, and often status writes. The descriptors are small, scattered and serialising.
Why do descriptors matter if they are under 1% of the bytes? Because the engine cannot issue payload for a fragment until its descriptor returns. A class negligible in bandwidth can dominate time.
How would you detect that? Measure the stall between a descriptor completing and payload becoming active. An engine that prefetches descriptors never has that gap.
Is DMA traffic sequential? Within a fragment. A scatter-gather list of fragments smaller than a row produces scattered traffic and dominant descriptor overhead.
Does DMA bypass caches? It depends on the SoC. A coherent or IO-coherent path may be satisfied from a cache, so it can deliver fewer DDR bytes than the transfer size.
How would you tell which case you have, without reading the manual? Compare DDR bytes delivered against transfer bytes requested. Below means some accesses were absorbed; above means extra traffic — maintenance or read-modify-write.
Why is a memory-to-memory copy potentially bad for banks? It is two streams in different regions. If the map places them in the same bank on different rows, every read/write alternation becomes a row conflict.
What does an unaligned transfer cost? A partial line at each end, which may become a read-modify-write — adding a read to a write-only transfer and creating a turnaround.
Would longer bus bursts improve row-hit rate? Not by themselves. Row hits come from address consecutiveness, which many small consecutive requests provide just as well.
18. Exercises
-
§3's model gives descriptors ~50% of the time at 0.78% of the bytes. Recompute for 1 KiB fragments and for 64 KiB fragments, and identify the fragment size at which descriptors stop mattering for time.
-
§2 notes a copy is two streams. Using 8.6, construct a source/destination placement that makes every alternation a row conflict, and one that makes them bank-parallel.
-
§6 shows 16× more row openings at 64-byte fragments. Derive the fragment size at which row openings stop scaling with fragment count, for a stated row size.
-
§7 says an unaligned write can become a read-modify-write. Derive the extra bytes delivered for a transfer of
Nbytes startingkbytes into a line. -
§10 gives one measurement distinguishing coherent from non-coherent. Construct the case where it is ambiguous, and say what second measurement resolves it.
-
§12's separator uses a region hint. Design the failure that a wrong region produces, and say which of §13's properties would still pass.
-
Report (B) shows identical byte accounting and a 1.95× time difference. Write the one-line metric you would add to a bandwidth dashboard to make that visible.
-
A colleague proposes fixing a fragmented-transfer throughput problem by increasing the DMA engine's outstanding depth. Explain when that helps and when it cannot, referring to §3 and §6.
19. Where This Goes
A DMA transfer is two traffic classes wearing one name. The payload is large, sequential within a fragment and latency-tolerant; the descriptors are small, scattered and serialising, and a class carrying under 1% of the bytes can cost half the time. Scatter-gather turns the best-case DRAM workload into one worse than scattered CPU misses, and both of the common beliefs about DMA — that it is sequential and that it bypasses caches — are architecture- and descriptor-dependent rather than true.
Four results carry forward. Bandwidth accounting and time accounting are different measurements, and report (B) shows two runs identical in the first and a factor of two apart in the second. Fragmentation compounds — more descriptors, more row openings and more translation pressure all push the same way. A copy is two streams, and their relative placement decides whether the alternation is a conflict or parallelism. And DDR bytes against transfer bytes is the measurement that distinguishes a coherent path, without asserting any protocol mechanism.
Two things stay open. The class split depends on a platform-supplied region hint, so a wrong region yields a self-consistent split of the wrong traffic that no property detects. And a measured run predicts row hits only under a favourable map — 29.3 §8 shows the opposite map converting the same run into bank spread instead, which is better for different reasons.
Chapter 29.5 takes the last requester class and the module's synthesis. An accelerator has the DMA engine's freedom from dependency chains and the CPU's appetite for reuse, and its DDR traffic is the residue of a decision made much earlier than any memory system — in how the computation was tiled. The traffic that reaches DRAM is what on-chip reuse failed to capture, which makes it the clearest case in the module of a DDR problem whose cause is architectural rather than memory-side, and the reason the module closes with a law rather than a technique.
Continue learning
Related tutorials
- Related topic
DQ — The Data Bus
Saying DQ is bidirectional says almost nothing. What matters is who drives these wires right now, how ownership changes, and what guarantees both sides never drive at once — which is a state machine, not a property.
- Related topic
DDR Bandwidth
Chapter 12.4 named four efficiency measures and built three. This builds the fourth: every bus cycle charged to exactly one named cause, with the categories provably summing to the window.
- Related topic
AXI-to-DDR Flow
One AXI transaction is not one DDR command, acceptance is not service, and the ordering the bus guarantees is not the ordering the scheduler uses. This is the transformation between them.
- Related topic
Memory Controllers in SoCs
A semantic adapter between two sides that disagree about what a request is, what order means, what fairness means, and whether priority should matter at all.
Standards & specifications
- Governing standard
- JEDEC JESD79 (DDR SDRAM)(opens JEDEC Solid State Technology Association in a new tab)
Defines the DDR SDRAM device itself — signals, command encoding, mode registers, timing parameters and the initialisation sequence — one document per generation. Memory-controller microarchitecture, address-mapping policy, PHY training algorithms and board-level design are not specified by it.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the DDR curriculum.