Data-Movement Costs

Chapter 1.1 established that memory cannot supply what compute can consume. Chapter 1.2 established that one agent owning memory, coherence and translation forces work through a centre that does not scale with the number of peers.

Both chapters kept reaching for the same unpriced quantity: the copy. This chapter prices it.

1. The One-Sentence Model

Moving a byte is not bookkeeping around the real work — it is work. It consumes read bandwidth at the source, capacity on every interconnect it crosses, write bandwidth at the destination, buffering and queue entries along the way, energy proportional to distance and to the number of interfaces crossed, latency the consumer waits through, and engineering effort in software and verification. In a large class of modern workloads, moving the operands costs more than operating on them.

The chapter's whole argument follows from one observation an engineer can check on any system: an N-byte copy does not consume N bytes of anything. It consumes N bytes of several different things at once, and the total is what determines whether the machine goes faster.

2. What This Chapter Owns

Not here: why accelerators proliferated (Ch 1.4), discrete accelerator memory limits (Ch 1.5), what conventional device attach provides (Ch 1.6), or coherent attach as a mechanism (Ch 1.7). No CXL protocol mechanism appears in this chapter.

3. The Three Responses to Displaced Data

Start from a statement that is almost tautological and turns out to organise everything:

A compute engine consumes data where the engine is. If the data is somewhere else, the system has exactly three options: move the data to the compute, move the compute to the data, or let the compute reach across and access the data where it lies.

Every real system does all three, in different places, for different reasons.

CPU needs a result the accelerator produced
    → move the data back toward the CPU
 
accelerator needs a tensor that lives in host memory
    → move the data toward the accelerator
 
two engines work on one dataset
    → duplicate it, migrate ownership of it, or share access to it

The third option is the one conventional architecture makes hardest, and it is the one Chapter 1.2's ownership structure explains: two memory domains that cannot address each other leave only the first option available. Placement decides movement. A system that placed the data correctly does not need a clever transfer engine, and a system that placed it badly cannot be rescued by one.

Moving compute is a genuine option and worth naming, because engineers forget it exists. Running a reduction on the device that already holds the data — rather than shipping the array to the host to reduce it — is moving the computation, and it is frequently the largest single win available. Near-memory and in-network processing are the same idea taken further.

4. A Taxonomy of Movement

"Data movement" is not one mechanism, and the costs differ enough that the distinctions matter. memcpy() is one row in this table and not the interesting one.

Two rows deserve emphasis because they are the ones people leave out of a cost model. Cache fills are data movement — a program that never calls a copy routine still moves its entire working set through the memory system. And implicit remote access is data movement — reaching a line on another socket moves that line, whether or not any software named the transfer.

5. Control Path and Data Path Are Different Costs

Chapter 1.2 introduced this split for the CPU. Here it becomes the frame for pricing a transfer, because the two paths consume different resources and saturate independently.

The two paths have different scaling laws, and confusing them is the most common error in a movement cost model:

• Control-plane cost scales with the number of transfers. Ten thousand 4 KiB transfers cost ten thousand descriptors, doorbells and completions, whatever the payload.
• Data-plane cost scales with the number of bytes. One 40 MiB transfer costs one descriptor and 40 MiB of bandwidth.

Which is why "the CPU copied the data" is usually false and always imprecise. With a DMA engine the CPU built a descriptor, rang a doorbell, and later read a completion — the payload went nowhere near a core. That the CPU can still be the bottleneck in this arrangement is exactly Chapter 1.2's argument, and the PCIe track works the mechanism in DMA over PCIe and DMA concepts.

6. Where the Time Goes: the Latency Stack

End-to-end movement latency is almost never dominated by the thing people name first — propagation across the wire. It is a stack, and most of the stack is not the wire.

A usable model separates the part that depends on size from the part that does not:

T_move  ≈  T_setup            (steps 1-2: size-independent)
         + T_queue            (step 3: depends on contention, not size)
         + bytes / B_eff      (steps 4-6: the only size-dependent term)
         + T_complete         (step 7: size-independent)

Everything except the middle term is fixed cost per transfer. That single structural fact produces the batching argument in Section 12, and it explains a result that surprises people the first time they measure it: for small transfers, making the link twice as fast changes almost nothing, because the size-dependent term was never the majority of the time.

B_eff deserves its own caution. It is not the link's rated bandwidth. It is the rate the slowest stage in steps 4 to 6 can sustain — which may be the source memory, the link, the destination memory, or the engine itself. A transfer runs at the speed of its narrowest point, and the narrowest point is frequently a memory controller rather than the interconnect everyone was watching.

7. Where the Bandwidth Goes: Occupancy and Amplification

Now the central quantitative result of the chapter.

Consider a memory-to-memory copy of N bytes from one memory to another. Count what the system actually services:

The payload-side accounting, stated precisely so it can be checked:

source memory       reads   N bytes
destination memory  writes  N bytes
                            ───────
memory subsystem services   2N bytes of payload traffic
 
interconnect carries        N bytes  (once per hop the copy crosses)

A copy of N bytes costs at least 2N bytes of memory traffic. That is the read-plus-write amplification, and it is the number to carry.

Two honest qualifications, because this is exactly where cost models get overstated:

"At least" is doing real work. Beyond the payload there can be descriptor and completion traffic, translation-structure accesses, coherence or snoop traffic where the buffers are cached, protocol headers and framing on the link, retried transfers, and — where the destination is cached and written partially — a read-for-ownership that fetches a line before it is overwritten (Chapter 1.1 §5 works that case). Each is real and each is implementation-dependent. The 2N figure is a floor, not a total.

If both endpoints are the same physical memory, the traffic is still 2N — a host-to-host copy reads and writes the same DRAM, so it consumes twice the payload of that one memory's bandwidth and no link capacity at all. People are often surprised that a "local" copy is the most memory-bandwidth-expensive kind.

8. Where the Energy Goes

Latency and bandwidth are the costs engineers instrument. Energy is the one that decides what can be built, because a socket, a board and a rack all have a fixed power budget before any workload arrives.

The physical intuition is short. Moving a bit means charging and discharging a capacitance. That capacitance grows with wire length and with the number of interfaces the signal crosses, so:

• an operand read from a register file travels micrometres inside the same block;
• an on-die SRAM access travels across a macro and its peripheral circuitry;
• a cross-die wire drives a long, capacitive route;
• crossing a package boundary drives a pad, a bump, a trace and a receiver;
• a serial link additionally runs a SerDes, whose transmitter, receiver, clock recovery and equalisation all draw power whether or not the data is useful;
• a DRAM access activates a row, moves charge on bitlines, and drives an off-chip bus.

Hence the model, which is deliberately shallow because a deeper one would be specific to a process and a design:

E_move  ≈  bytes_moved  ×  energy_per_byte(path)

The interesting variable is not bytes_moved, which everyone can measure, but energy_per_byte(path) — a function of where the bytes went, not of how many there were. The same byte costs radically different amounts depending on whether it crossed a register file port or a serial link.

No universal picojoules-per-bit number exists, and any tutorial that offers one is overreaching. The figure depends on process node, voltage, link rate, encoding, distance, and how much of the interface's fixed power is amortised across the traffic. What is durable, and is supported by published energy-per-operation studies — Horowitz's ISSCC 2014 paper Computing's Energy Problem (and what we can do about it) is the standard citation — is the ordering and the magnitude of the gaps: an off-chip DRAM access sits two to three orders of magnitude above an on-die arithmetic operation of similar width, with on-die SRAM and wire traversal in between.

Two consequences follow, and they are the reason this section exists:

For a low-intensity computation, the energy bill is the operands, not the operations. Optimising the arithmetic of such a kernel changes almost nothing; optimising its data movement changes the whole number.

A copy that computes nothing still pays full price, twice. Staging a buffer across a link pays source DRAM read energy, link energy, and destination DRAM write energy — and pays all three again on the way back. Section 11 puts numbers on the time; the energy tracks the same byte count.

9. Intensity and Reuse: Reducing Bytes Beats Adding FLOPs

Chapter 1.1 defined arithmetic intensity as operations performed per byte moved, and used it to decide whether a workload is compute-bound or memory-bound. Here it does different work: it is the lever that converts a movement problem into a design change.

arithmetic intensity  =  operations / bytes moved

The ratio has a numerator and a denominator, and engineering attention is distributed between them very unevenly. Adding execution units raises the numerator's ceiling and does nothing for a workload below machine balance. Reducing the denominator raises the ratio directly, and it is almost always the cheaper move.

The mechanism that reduces the denominator is reuse: fetch a byte once, and do as much work on it as possible before letting it go.

fetch once, use 100 times     →  intensity ×100, one traversal of the memory path
fetch 100 times, use once     →  intensity ×1,   one hundred traversals

Reuse is not a software abstraction; it is the reason specific hardware exists. Each of the following is storage placed close to compute for exactly this purpose:

And the software techniques that exploit them — blocking a loop nest so a tile stays resident, fusing two passes so an intermediate never reaches memory, batching so weights are fetched once and applied to many inputs — are all the same idea: change the schedule so the same byte is used more times per traversal. None of them adds arithmetic capability. All of them can produce large speedups on a movement-bound workload.

10. Placement Is Multidimensional

It is tempting to draw a strict ladder from registers down to storage and read latency off it. Resist that: the ordering is not universal across architectures, and latency is only one of the properties that matters.

register  →  local cache  →  shared cache  →  local DRAM  →  remote-socket DRAM
          →  accelerator memory  →  fabric-attached memory  →  storage

Treat that as a rough proximity gradient, then evaluate any actual placement on six axes at once:

• Proximity — how far the bytes travel per access, which sets latency and energy.
• Bandwidth — what the path can sustain, which may not correlate with proximity: accelerator-local memory is farther from the host than host DRAM and delivers far more bandwidth to the accelerator.
• Capacity — how much fits, which usually trades against proximity.
• Ownership — who may address it, and whether anything else can.
• Coherence — whether hardware keeps copies consistent or software must.
• Access semantics — load/store, or an explicit transfer with a completion.

The last three are the ones a latency table hides, and they are the ones Chapters 1.1 and 1.2 spent their length on. Memory that is fast, plentiful and unreachable by the engine that needs it is worth nothing to that engine.

11. Worked Example — the Host ↔ Accelerator Round Trip

Everything so far assembles into one calculation. The numbers below are illustrative: the payload size is chosen for round arithmetic and the bandwidth is a plausible sustained figure, not a measurement of any product. Substitute your own and the structure of the conclusion does not change.

GIVEN
  working set              S = 8 GiB = 8 × 2^30 = 8.590e9 bytes
  sustained link rate      B_eff = 20 GB/s  (illustrative)
  fixed cost per transfer  T_setup + T_complete = 20 µs  (illustrative)
  kernel execution time    T_compute = 0.150 s  (illustrative)
 
LINK TRAFFIC
  host → device                   8.590e9 bytes
  device → host                   8.590e9 bytes
                                 ─────────────
  total payload across the link  17.18e9 bytes
 
TIME ON THE LINK
  T_transfer = 17.18e9 / 20e9  =  0.859 s
  T_fixed    = 2 × 20 µs       =  0.000040 s   (negligible at this size)
 
MEMORY TRAFFIC (Section 7 applied to both directions)
  host DRAM     read  8.590e9  +  written  8.590e9  =  17.18e9 bytes
  device memory written 8.590e9 + read     8.590e9  =  17.18e9 bytes
                                                      ─────────────
  total memory traffic caused by the movement          34.36e9 bytes

Four things are worth extracting from that block, and the last is the one that changes decisions.

The device is busy 15% of the wall time. 0.150 / (0.859 + 0.150) = 14.9%. Nothing is broken and the accelerator is not slow — it is idle, waiting for its operands.

Overlap helps and does not rescue it. Double-buffer the transfer against the compute (Section 15) and the time becomes max(0.859, 0.150) = 0.859 s, for a device utilisation of about 17.5%. Overlap hides the smaller phase behind the larger one; it never removes the larger one.

Fixed cost is invisible here and dominant elsewhere. At 8 GiB, 40 µs of setup is 0.005% of the transfer. Section 12 is what happens when the same fixed cost is paid on a 4 KiB transfer instead.

The memory systems moved four times the working set. 34.36e9 bytes of DRAM traffic so that 8.590e9 bytes could be computed on once. That 4× is Section 7's amplification applied twice, and it is the figure that belongs in a power and bandwidth budget — not the link number everyone quotes.

12. Batching: Amortising Setup, and What It Costs

From Section 6, a transfer costs a fixed amount plus a size-dependent amount:

T_total(n transfers of size s)  =  n × T_fixed  +  (n × s) / B_eff

Hold the total payload constant and vary how it is divided. Using the same illustrative T_fixed = 20 µs and B_eff = 20 GB/s, for a total payload of 64 MiB (67.1e6 bytes):

                                       fixed        transfer      total     efficiency
16384 transfers × 4 KiB   16384×20µs = 0.328 s  +  0.00336 s  =  0.331 s      1.0%
 1024 transfers × 64 KiB   1024×20µs = 0.0205 s +  0.00336 s  =  0.0238 s    14.1%
   64 transfers × 1 MiB      64×20µs = 0.00128 s+  0.00336 s  =  0.00464 s   72.4%
    1 transfer  × 64 MiB       1×20µs= 0.00002 s+  0.00336 s  =  0.00336 s   99.4%

Efficiency here is transfer time / total time — the fraction of the elapsed time that actually moved bytes. At 4 KiB per transfer, 99% of the time is overhead, and the link is idle for essentially all of it. That is why every mature data-movement stack works hard to make transfers larger: descriptor chaining, scatter-gather lists, coalesced submissions, and hardware queues that consume many descriptors per doorbell.

But "always batch" is wrong, and the reason is the one thing this table does not show.

latency of the FIRST result  ≈  time to accumulate a batch
                              + T_fixed
                              + batch_bytes / B_eff

A batch does not start until it is full or a timeout fires. A request that arrives first waits for the rest of the batch, so batching converts a throughput problem into a latency problem — the same trade Chapter 1.2 found in interrupt coalescing, appearing again one layer down. The signature is identical too: throughput improves, CPU cost falls, dashboards look better, and the latency distribution grows a second peak at exactly the batching timeout.

The engineering position is therefore per-workload, not universal:

13. Copy or Reach Across? A Crossover Model

Given data that is somewhere else, the choice from Section 3 is between moving it and reaching for it. This is the decision architects actually make, and it has a crossover rather than an answer.

Build the simplest model that captures the trade honestly:

COPY, then access locally
  T_copy   =  T_fixed  +  S / B_bulk  +  A × t_local
 
REACH ACROSS, access remotely
  T_remote =  A × t_remote
 
where
  S        = bytes that would be copied
  A        = number of accesses the computation performs
  B_bulk   = sustained bandwidth of a bulk transfer
  t_local  = average latency of an access to local memory
  t_remote = average latency of an access to remote memory

Copying wins when T_copy < T_remote. Ignoring the fixed cost, which is small for a large S:

S / B_bulk  <  A × (t_remote − t_local)

Read the inequality rather than solving it. The left side is paid once and scales with the size of the data. The right side is paid per access and scales with how many times the computation touches it. So:

• Touch the data many times → copy. The one-time transfer amortises across every access, and every access afterwards is local.
• Touch it once, or sparsely → reach across. Copying S bytes to read a small fraction of them is pure waste.

Put illustrative numbers on it to see how sharp the boundary is. Take S = 1 GiB = 1.074e9 bytes, B_bulk = 20 GB/s, t_remote − t_local = 200 ns, and 64-byte accesses:

copy time            = 1.074e9 / 20e9              = 0.0537 s
 
break-even accesses  = 0.0537 / 200e-9             = 268,000 accesses
break-even bytes     = 268,000 × 64 B              = 17.2e6 bytes
break-even fraction  = 17.2e6 / 1.074e9            = 1.6% of the data

Touch more than about 1.6% of that buffer and copying it wins. The threshold is low because bulk bandwidth is enormous relative to the per-access penalty — which is the quantitative reason bulk staging has been the default in accelerator programming for two decades, and the reason it stops being obviously right for genuinely sparse access patterns like graph traversal, large hash-table probes, or a model whose parameters are mostly cold.

Three refinements the model deliberately omits, each of which moves the crossover:

• Reuse at the far end. If remote accesses hit a device-side cache, the effective t_remote falls and reaching across improves. Chapter 1.1's locality argument applies unchanged.
• Concurrency. t_remote as a latency only bounds throughput when the requester cannot keep enough accesses in flight. With adequate memory-level parallelism, remote access is bandwidth-limited rather than latency-limited, and the comparison becomes a bandwidth one.
• Capacity. If S does not fit in local memory, copying is not on the menu at any price. That is not a performance question; it is the feasibility question from Chapter 1.1.

14. Zero-Copy Does Not Mean Zero Movement

"Zero-copy" is one of the most frequently misread terms in systems engineering, and it is a favourite interview probe for exactly that reason.

What zero-copy actually means: eliminating software-visible duplication between buffers. In a naive path, data arriving from a device is written into a kernel buffer and then copied by a core into a user buffer, so the payload lands in memory twice and a core executes the second copy. A zero-copy path arranges for the data to be placed once, where the consumer can already reach it.

What zero-copy does not mean: that no bytes moved. The payload still crossed a link, still traversed a memory controller, still occupied buffers in the device and the root complex, still consumed DRAM write bandwidth at its destination, and may still have been fetched into a cache when the consumer touched it.

naive path       device → host DRAM → core-executed copy → host DRAM
                 = 1 link traversal + 3N bytes of DRAM traffic
 
zero-copy path   device → host DRAM (consumer reads it there)
                 = 1 link traversal + N bytes of DRAM traffic

The saving is real and often large — a whole core-executed copy and 2N bytes of memory traffic. It is a saving of duplicate movement, not of movement.

The same distinction applies one level up. Peer-to-peer device transfers — the general class that GPUDirect-style mechanisms and NVMe peer-to-peer belong to — remove the host memory staging hop, so data goes device-to-device rather than device-to-host-to-device. That is a genuine reduction in movement, and it is not zero movement: the bytes still cross the fabric, and they still consume write bandwidth at the destination.

15. Double Buffering: Overlapping Movement with Work

If the transfer cannot be removed, the next question is whether the consumer must wait for it. Double buffering is the standard answer, and it is the bridge from this chapter's architecture into its RTL.

The performance model is one line, and it is the reason the technique exists:

serialised   T = T_move + T_compute
overlapped   T = max(T_move, T_compute)   +  one buffer-fill of startup

The gain is min(T_move, T_compute) — bounded by the smaller phase. Which produces the rule that stops teams over-investing in it: double buffering is worth most when the two phases are balanced, and worth almost nothing when one dominates. Section 11's example is the pathological case: T_move = 0.859 s against T_compute = 0.150 s, so overlap recovers 0.150 s of a 1.009 s runtime and utilisation moves from 14.9% to 17.5%. Real, and not the fix.

The costs are hardware costs and they are not free:

• Storage doubles. Two buffers instead of one, in whatever memory is scarcest — usually the fast, small, close one.
• Ownership must be tracked. Somebody has to know which buffer belongs to which agent, and be right every cycle.
• The two agents must synchronise. A fill that completes must become visible to the consumer, and a drain that completes must release the buffer.
• Deeper pipelining multiplies both. Triple buffering absorbs more jitter and costs another buffer plus more state.

Section 19 builds the ownership tracker in RTL, and the reason it is only four bits of state is worth previewing: the invariant "a buffer is never owned by both agents" turns out to be structural rather than something the design has to check.

16. The Hardware of Movement — and How These Models Were Checked

Sections 16 to 19 build four small SystemVerilog modules. Together they are a complete, working data mover: an engine that copies a length of beats from a source to a destination, an elastic buffer that decouples the two sides, an ownership tracker for double buffering, and the instrumentation that measures what actually moved.

17. RTL 1 — A Copy Engine

The smallest complete model of a data mover: take a length, fetch that many beats from a source, deliver them to a destination, report completion.

// A teaching model of a copy engine. It fetches `length_beats` beats from a
// source interface and delivers them to a destination interface, honouring
// backpressure in both directions.
//
// The two counters are the architectural content. A copy engine cannot use one
// counter, because fetching and delivering are decoupled by the holding
// register: at any moment some beats have been fetched but not yet accepted.
module copy_engine #(
  parameter int unsigned DATA_W = 64,
  parameter int unsigned LEN_W  = 16
) (
  input  logic              clk,
  input  logic              rst_n,
  input  logic              start,
  input  logic [LEN_W-1:0]  length_beats,
  // source side — the engine asks, the source answers
  output logic              src_req,
  input  logic              src_valid,
  input  logic [DATA_W-1:0] src_data,
  // destination side — the engine offers, the destination accepts
  output logic              dst_valid,
  output logic [DATA_W-1:0] dst_data,
  input  logic              dst_ready,
  output logic              busy,
  output logic              done
);
 
  logic              busy_q;
  logic              hold_valid_q;   // the holding register is occupied
  logic [DATA_W-1:0] hold_data_q;
  logic [LEN_W-1:0]  to_fetch_q;     // beats not yet taken from the source
  logic [LEN_W-1:0]  remaining_q;    // beats not yet ACCEPTED by the destination
 
  logic src_fire;
  logic dst_fire;
 
  // Ask the source only when there is somewhere to put the answer. The
  // `dst_fire` term lets a new beat arrive in the same cycle the previous one
  // leaves — without it the engine idles every other cycle.
  assign src_req  = busy_q && (to_fetch_q != '0) && (!hold_valid_q || dst_fire);
  assign src_fire = src_req && src_valid;
 
  assign dst_valid = hold_valid_q;
  assign dst_data  = hold_data_q;
  assign dst_fire  = dst_valid && dst_ready;
 
  assign busy = busy_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      busy_q       <= 1'b0;
      hold_valid_q <= 1'b0;
      hold_data_q  <= '0;
      to_fetch_q   <= '0;
      remaining_q  <= '0;
      done         <= 1'b0;
    end else begin
      done <= 1'b0;                       // single-cycle pulse unless set below
 
      if (!busy_q) begin
        if (start) begin
          if (length_beats == '0) begin
            done <= 1'b1;                 // zero-length is a legal request
          end else begin
            busy_q      <= 1'b1;
            to_fetch_q  <= length_beats;
            remaining_q <= length_beats;
          end
        end
      end else begin
        if (src_fire) begin
          hold_valid_q <= 1'b1;
          hold_data_q  <= src_data;
          to_fetch_q   <= to_fetch_q - 1'b1;
        end else if (dst_fire) begin
          hold_valid_q <= 1'b0;
        end
 
        // Completion accounting advances ONLY on an accepted beat. Advancing
        // it when the beat is fetched is the bug in Debug Lab 3.
        if (dst_fire) begin
          remaining_q <= remaining_q - 1'b1;
          if (remaining_q == 1) begin
            busy_q <= 1'b0;
            done   <= 1'b1;
          end
        end
      end
    end
  end
endmodule

What it does. Moves length_beats × DATA_W/8 bytes from a source to a destination and pulses done when the last byte has been accepted.

Architecture. A one-beat holding register between two independent handshakes. That single register is what makes the engine work at all: without storage between the two sides, the source and destination handshakes would have to agree every cycle.

State. busy_q, hold_valid_q, hold_data_q, and two length counters. Note that src_req is combinational from dst_ready — a stall at the destination is visible at the source in the same cycle. Section 18 is the module that removes that path, and the simulation below shows the difference.

Contract. dst_data must remain stable while dst_valid is asserted and dst_ready is low. done is a one-cycle pulse, asserted in the cycle after the final accepted beat, and never coincident with busy.

Cycle behaviour. With both sides ready the engine sustains one beat per cycle. When the destination stalls, the holding register keeps its beat, src_req deasserts, and the engine consumes no source bandwidth for the duration of the stall.

Expected simulation output. From the directed testbench: 8 beats of 64 bits, source always valid, destination stalled for four cycles mid-transfer. This is copied verbatim from the Icarus run.

=== SCENARIO A: 8 beats, destination backpressure ===
cyc=3  src_req=1 src_valid=1 dst_valid=0 dst_ready=1 transfer=0 busy=1 done=0 bytes=0
cyc=4  src_req=1 src_valid=1 dst_valid=1 dst_ready=1 transfer=1 busy=1 done=0 bytes=0
cyc=5  src_req=1 src_valid=1 dst_valid=1 dst_ready=1 transfer=1 busy=1 done=0 bytes=8
cyc=6  src_req=0 src_valid=1 dst_valid=1 dst_ready=0 transfer=0 busy=1 done=0 bytes=16
cyc=7  src_req=0 src_valid=1 dst_valid=1 dst_ready=0 transfer=0 busy=1 done=0 bytes=16
cyc=8  src_req=0 src_valid=1 dst_valid=1 dst_ready=0 transfer=0 busy=1 done=0 bytes=16
cyc=9  src_req=0 src_valid=1 dst_valid=1 dst_ready=0 transfer=0 busy=1 done=0 bytes=16
cyc=10 src_req=1 src_valid=1 dst_valid=1 dst_ready=1 transfer=1 busy=1 done=0 bytes=16
...
--- A done: accepted_beats=8 bytes_moved=64 bytes_offered=96 busy_cycles=13
            dst_stall=4 src_starve=0 errors=0
RESULT: PASS (0 invariant failures, 9 beats accepted)

Two results in that tail are the whole point of Section 20, so note them now: the engine moved 64 bytes and a counter that watched dst_valid instead of the handshake would have reported 96. And 8 beats took 13 busy cycles, so the engine ran at 61% of its one-beat-per-cycle ceiling — the missing 39% is the four stall cycles plus the pipeline fill.

Synthesis. A small FSM's worth of control (one busy flip-flop), two LEN_W down-counters with comparators, one DATA_W register plus its valid bit, and the handshake logic. No latches: every output is a continuous assignment or a fully-reset flop. The one thing to flag in a timing review is the dst_ready → src_req combinational path, which is exactly what the next section fixes.

Failure modes. Overwriting the holding register while the destination is stalled (Debug Lab 2). Advancing completion accounting on fetch rather than acceptance (Debug Lab 3). Over-fetching past the requested length if src_req forgets the to_fetch_q term.

DV strategy. Scoreboard every accepted beat against an ordered expected sequence; check payload stability across every stall cycle; check done never coincides with busy; run zero-length, one-beat and maximum-length transfers; stall the destination for longer than the whole transfer; starve the source; and reset mid-transfer.

18. RTL 2 — The Skid Buffer

The copy engine has a combinational path from dst_ready to src_req. In a real design that path crosses a module boundary, and it gets longer every time somebody inserts another stage. A skid buffer breaks it, at the cost of one register and one bit.

// Breaks the combinational ready path between a producer and a consumer.
// `in_ready` depends only on registered state, so a consumer stall is felt by
// the producer one cycle later rather than in the same cycle. The extra cycle
// is exactly what the skid slot is for: it holds the beat that was already
// in flight when the consumer stalled.
module skid_buffer #(
  parameter int unsigned W = 64
) (
  input  logic         clk,
  input  logic         rst_n,
  input  logic         in_valid,
  output logic         in_ready,
  input  logic [W-1:0] in_data,
  output logic         out_valid,
  input  logic         out_ready,
  output logic [W-1:0] out_data
);
  logic         out_valid_q;
  logic [W-1:0] out_data_q;
  logic         skid_valid_q;   // the overflow slot
  logic [W-1:0] skid_data_q;
 
  assign out_valid = out_valid_q;
  assign out_data  = out_data_q;
 
  // The whole reason this module exists: no `out_ready` term here.
  assign in_ready  = !skid_valid_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      out_valid_q  <= 1'b0;
      out_data_q   <= '0;
      skid_valid_q <= 1'b0;
      skid_data_q  <= '0;
    end else if (out_ready || !out_valid_q) begin
      // The output register is free this cycle. Drain the skid slot first so
      // ordering is preserved; only take a new input when the slot is empty.
      if (skid_valid_q) begin
        out_valid_q  <= 1'b1;
        out_data_q   <= skid_data_q;
        skid_valid_q <= 1'b0;
      end else begin
        out_valid_q <= in_valid && in_ready;
        out_data_q  <= in_data;
      end
    end else if (in_valid && in_ready) begin
      // The output is stalled and a beat is arriving anyway — catch it.
      skid_valid_q <= 1'b1;
      skid_data_q  <= in_data;
    end
  end
endmodule

What it does. Passes beats through with one cycle of latency, absorbing exactly one beat of backpressure so no data is lost when a stall arrives late.

Architecture. Two storage slots — the output register and the skid slot — and the ordering rule that the skid slot drains first. Two is the minimum depth that both registers the outputs and never drops a beat.

Key invariant. in_ready is a function of registered state only. That is what breaks the timing path, and it is also why the buffer needs a second slot: the producer cannot know about a stall until the following cycle, so one beat is always potentially in flight.

Expected simulation output. Same stimulus shape as Section 17 — consumer stalls mid-stream. Verbatim from the Icarus run:

cyc=4  in_valid=1 in_ready=1 out_valid=1 out_ready=1 xfer=1
cyc=5  in_valid=1 in_ready=1 out_valid=1 out_ready=0 xfer=0   <-- consumer stalls
cyc=6  in_valid=1 in_ready=0 out_valid=1 out_ready=0 xfer=0   <-- producer told, 1 cycle later
cyc=7  in_valid=1 in_ready=0 out_valid=1 out_ready=0 xfer=0
cyc=8  in_valid=1 in_ready=0 out_valid=1 out_ready=1 xfer=1   <-- consumer resumes
cyc=9  in_valid=1 in_ready=1 out_valid=1 out_ready=1 xfer=1
--- skid: beats_out=9 errors=0
RESULT: PASS (skid buffer, 9 beats, no loss/reorder)

Compare this against Section 17's trace, because that comparison is the lesson. In the copy engine, dst_ready fell at cycle 6 and src_req fell at cycle 6 — the same cycle, because the path is combinational. Here out_ready falls at cycle 5 and in_ready falls at cycle 6 — one cycle later, because the path is registered. The skid slot holds the beat that was in flight during that extra cycle, which is why beats_out=9 with zero loss or reordering rather than eight beats and a dropped one.

Synthesis. Two W-bit registers, two valid bits, and a small amount of mux and control logic. The area cost is one extra data register over a plain output register; the return is a shorter critical path and a clean module boundary.

Failure modes. Making in_ready depend on out_ready reintroduces the combinational path and defeats the purpose. Draining the input before the skid slot reorders the stream. Forgetting the second slot entirely loses the in-flight beat, which is Debug Lab 2's failure in a different guise.

DV strategy. Random in_valid and out_ready patterns including adjacent single-cycle stalls, an ordered scoreboard on the output, a payload-stability check across every stall, and a count check that beats in equals beats out at the end of the run.

19. RTL 3 — Ping-Pong Ownership

Section 15 argued for double buffering. This is the state that makes it safe, and it is smaller than most engineers expect.

// Tracks which of two buffers each agent may touch. The producer fills a
// buffer that is empty; the consumer drains one that is full. Everything else
// falls out of that rule — including the safety property, which is structural
// rather than checked.
module ping_pong_ctrl (
  input  logic clk,
  input  logic rst_n,
  input  logic fill_done,       // producer finished the buffer it owns
  input  logic consume_done,    // consumer finished the buffer it owns
  output logic fill_sel,        // buffer index the producer may write
  output logic consume_sel,     // buffer index the consumer may read
  output logic fill_en,         // producer may proceed
  output logic consume_en       // consumer may proceed
);
  logic [1:0] full_q;           // full_q[i] = buffer i holds unconsumed data
  logic       fill_sel_q;
  logic       consume_sel_q;
 
  assign fill_sel    = fill_sel_q;
  assign consume_sel = consume_sel_q;
 
  // A producer may only fill an EMPTY buffer; a consumer may only drain a FULL
  // one. Because a buffer cannot be both, the two agents can never be enabled
  // on the same buffer — the mutual-exclusion property needs no extra logic.
  assign fill_en     = !full_q[fill_sel_q];
  assign consume_en  =  full_q[consume_sel_q];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      full_q        <= 2'b00;
      fill_sel_q    <= 1'b0;
      consume_sel_q <= 1'b0;
    end else begin
      if (fill_done && fill_en) begin
        full_q[fill_sel_q] <= 1'b1;
        fill_sel_q         <= ~fill_sel_q;
      end
      if (consume_done && consume_en) begin
        full_q[consume_sel_q] <= 1'b0;
        consume_sel_q         <= ~consume_sel_q;
      end
    end
  end
endmodule

What it does. Grants each agent exclusive access to one of two buffers and swaps the assignment as each finishes.

Architecture. Four bits of state total: two full flags and two selection bits. That is the entire double-buffering mechanism — the buffers themselves are memory the controller never touches.

Key invariant. fill_en && consume_en implies fill_sel != consume_sel. Worth pausing on why: fill_en requires buffer fill_sel to be empty and consume_en requires buffer consume_sel to be full. If the two indices were equal, one buffer would have to be simultaneously empty and full. The property is therefore structural — it cannot be violated without the full flags themselves being wrong, which is a much easier thing to check.

Expected simulation output. Producer takes 2 cycles per buffer, consumer takes 3 — deliberately unequal, which is when ownership bugs surface. Verbatim from the Icarus run:

cyc=19 fill_en=1 fill_sel=0 consume_en=1 consume_sel=1
cyc=20 fill_en=1 fill_sel=0 consume_en=1 consume_sel=1
cyc=21 fill_en=0 fill_sel=1 consume_en=1 consume_sel=1
cyc=22 fill_en=1 fill_sel=1 consume_en=1 consume_sel=0
cyc=23 fill_en=1 fill_sel=1 consume_en=1 consume_sel=0
--- ping-pong: fills=10 consumes=9 overlap_cycles=18 errors=0
RESULT: PASS (ownership never shared; 18 cycles of true overlap)

overlap_cycles=18 is the measurement that matters: for 18 of the 30 simulated cycles both agents were working at once, on different buffers. That is the overlap Section 15's model assumes, demonstrated rather than asserted. And fill_sel never equals consume_sel while both are enabled, across the whole run.

Synthesis. Four flip-flops, two multiplexers and an inverter. Essentially free, which is worth knowing when someone proposes managing ping-pong ownership in software instead.

Failure modes. Toggling a selection bit without checking the corresponding enable lets an agent advance onto a buffer the other still owns. Clearing a full flag on the wrong index desynchronises the two pointers permanently. Both are caught by the ownership check in the testbench.

DV strategy. Drive the producer and consumer at deliberately unequal rates in both directions, hold one agent stalled indefinitely while the other runs, assert the ownership invariant every cycle, and confirm the overlap count is non-zero — a design that is functionally correct but never overlaps has failed at its actual purpose.

20. RTL 4 — Movement Counters, and the Number That Lies

An engine that moves data correctly and reports its throughput incorrectly is worse than useless, because every subsequent decision is made on the wrong number.

// Counts what a movement engine actually did. The critical line is
// `beat_accepted`: a transfer occurs when BOTH sides agree, never when only
// the producer offers. `bytes_offered_q` is deliberately included as the WRONG
// counter, so the gap between the two is visible in the same run.
module move_counters #(
  parameter int unsigned BYTES_PER_BEAT = 8,
  parameter int unsigned CNT_W          = 32
) (
  input  logic clk,
  input  logic rst_n,
  input  logic clear,
  input  logic engine_busy,
  input  logic dst_valid,
  input  logic dst_ready,
  input  logic src_req,
  input  logic src_valid,
  output logic [CNT_W-1:0] bytes_moved_q,
  output logic [CNT_W-1:0] busy_cycles_q,
  output logic [CNT_W-1:0] dst_stall_cycles_q,
  output logic [CNT_W-1:0] src_starve_cycles_q,
  output logic [CNT_W-1:0] bytes_offered_q
);
  logic beat_accepted;
  logic beat_offered;
  logic dst_stalled;
  logic src_starved;
 
  assign beat_accepted = dst_valid && dst_ready;   // THE transfer event
  assign beat_offered  = dst_valid;                // what a naive counter sees
  assign dst_stalled   = dst_valid && !dst_ready;  // offered and refused
  assign src_starved   = engine_busy && src_req && !src_valid;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      bytes_moved_q       <= '0;
      bytes_offered_q     <= '0;
      busy_cycles_q       <= '0;
      dst_stall_cycles_q  <= '0;
      src_starve_cycles_q <= '0;
    end else if (clear) begin
      bytes_moved_q       <= '0;
      bytes_offered_q     <= '0;
      busy_cycles_q       <= '0;
      dst_stall_cycles_q  <= '0;
      src_starve_cycles_q <= '0;
    end else begin
      if (beat_accepted) bytes_moved_q       <= bytes_moved_q   + CNT_W'(BYTES_PER_BEAT);
      if (beat_offered)  bytes_offered_q     <= bytes_offered_q + CNT_W'(BYTES_PER_BEAT);
      if (engine_busy)   busy_cycles_q       <= busy_cycles_q   + 1'b1;
      if (dst_stalled)   dst_stall_cycles_q  <= dst_stall_cycles_q  + 1'b1;
      if (src_starved)   src_starve_cycles_q <= src_starve_cycles_q + 1'b1;
    end
  end
endmodule

The measured result, from the same run as Section 17:

accepted_beats=8   bytes_moved=64   bytes_offered=96
busy_cycles=13     dst_stall=4      src_starve=0

Sixty-four bytes moved. A counter watching dst_valid reports ninety-six — a 50% over-report on a four-cycle stall. The error is not a constant: it is proportional to backpressure, so the counter is most wrong exactly when the system is most congested and someone is looking at it to find out why. A naive counter says the link is running at 1.5× its real rate precisely during the incident it was meant to diagnose.

The derived quantities the counters support, and what each answers:

Note what this pair of counters localises in one shot. Destination stall at 31% and source starvation at 0% says the engine had data and could not deliver it — so the investigation goes downstream, not to the source memory. Reverse the two and the same instrumentation sends you upstream instead. That is the practical value of counting both sides rather than only throughput.

Synthesis. Five counters with enables, three small comparators, and the width of CNT_W. Counters are cheap; the reason instrumentation gets cut in reviews is area anxiety rather than area. A design shipped without them is a design whose performance cannot be explained after tape-out.

What changes, and when. dst_ready falls at t3. dst_valid does not fall — the engine is still offering the same beat, and it must keep offering it until somebody takes it. transfer (dst_valid && dst_ready) goes to zero for the whole stall, and bytes_moved is flat at 16 across t3 to t6.

Why src_req falls too. The engine has nowhere to put a new beat while the holding register is occupied, so it stops asking. Backpressure has propagated from the destination all the way to the source in one cycle — which is efficient in bandwidth terms and is the combinational path Section 18 exists to break.

What a bug would look like. If src_req stayed high across the stall, the engine would fetch beats it cannot hold and overwrite the one waiting — Debug Lab 2, where the simulation shows four beats silently lost. If bytes_moved climbed during t3 to t6, the counter would be watching dst_valid instead of the handshake — Debug Lab 1, worth 96 bytes against a true 64. Both bugs are invisible in a trace that never stalls, which is why a regression without sustained backpressure proves very little about a data mover.

21. Assertions for a Movement Engine

The properties below are the invariants of a data mover, expressed as bind-ready SystemVerilog assertions. They test the transfer contract, not the implementation — bind them to any engine, including one you did not write.

// P1 — the transfer event. A beat counts as moved only when both sides agree.
// This is the property that makes every downstream bandwidth number honest.
property p_transfer_needs_both;
  @(posedge clk) disable iff (!rst_n)
    (dst_valid && dst_ready) |-> (dst_valid && dst_ready);
endproperty
 
// P2 — payload stability under backpressure. An offered beat may not change
// until it is accepted; a producer that reloads under a stall silently
// destroys the beat the consumer was about to take.
property p_data_stable_until_accepted;
  @(posedge clk) disable iff (!rst_n)
    (dst_valid && !dst_ready) |=> (dst_valid && $stable(dst_data));
endproperty
 
// P3 — no accounting without acceptance. The byte counter must not move on a
// cycle where nothing was accepted. Catches the counter bug directly.
property p_bytes_only_on_accept;
  @(posedge clk) disable iff (!rst_n || clear)
    (!(dst_valid && dst_ready)) |=> $stable(bytes_moved_q);
endproperty
 
// P4 — completion follows acceptance, not presentation. `done` may only fire
// when the engine has stopped being busy, which happens on the final accepted
// beat. Catches premature completion.
property p_done_not_while_busy;
  @(posedge clk) disable iff (!rst_n)
    done |-> !busy;
endproperty
 
// P5 — no over-fetch. The engine must not ask the source for a beat it has no
// room to hold, unless the holding register is being emptied this cycle.
property p_no_request_without_room;
  @(posedge clk) disable iff (!rst_n)
    (src_req && dst_valid) |-> dst_ready;
endproperty
 
// P6 — ping-pong ownership. The producer and the consumer may never be
// enabled on the same buffer. Structural in the design of Section 19, and
// worth asserting because a refactor can break structure.
property p_ownership_exclusive;
  @(posedge clk) disable iff (!rst_n)
    (fill_en && consume_en) |-> (fill_sel != consume_sel);
endproperty

Icarus Verilog has no concurrent-assertion support, so these were not executed. They were instead checked as procedural invariants inside the testbench, which is why the run in Section 17 prints errors=0 rather than an assertion summary. The mapping is one to one:

P1 is deliberately trivial as written — it is a definition, not a check. It earns its place in the file as documentation of what the rest of the properties mean by "transfer", because the single most common instrumentation bug in this domain is disagreeing with that definition.

22. Verifying a Data Mover

A transfer engine can have perfect throughput and still be wrong. Correct movement means preserving six things at once, and each has its own failure mode:

The scoreboard model

The mental model is small and it is the same for every data mover:

stimulus generator  →  reference queue  ─┐
                                          ├→  compare on every accepted beat
DUT source port     →  DUT  →  destination ┘

Push the expected payload into a queue as it is offered to the source; pop and compare on every dst_valid && dst_ready. That single comparison catches value corruption, reordering and duplication simultaneously, and an end-of-test check that the queue is empty catches loss. The testbench behind this chapter does exactly that, which is what errors=0 in Section 17 attests.

The stimulus that actually finds bugs

Directed corners first, because they are cheap and they fail often:

• Zero-length transfer. Legal in most descriptor formats and routinely unhandled. Section 17's Scenario B confirms the engine completes without emitting a phantom beat.
• One-beat transfer. The shortest path through the completion logic, where off-by-one errors live.
• Maximum length. Counter width, and the wrap that happens one beat past it.
• Destination stalled for longer than the whole transfer. Nothing may be lost and nothing may time out incorrectly.
• Source starved mid-transfer. Section 17's Scenario C: src_starve=3 and the transfer still completes correctly.
• Simultaneous start and completion. The engine must not accept a new descriptor while retiring the previous one, unless it was designed to.
• Reset during traffic. Counters, pointers and full flags must all return to a consistent state — not some of them.
• Duplicate or missing completion. Design-specific, but if the protocol forbids it, assert it.

Then randomised pressure, because the interesting bugs are interleavings:

• dst_ready toggling every cycle, which hits the simultaneous accept-and-fetch path continuously.
• Randomised gaps on src_valid interleaved with randomised stalls on dst_ready.
• Back-to-back transfers with no idle cycles between them.

Coverage that means something

Coverage bins should name behaviour spaces the design can actually be wrong in, not enumerate values:

The cross that matters most is stall arrival × transfer length, and specifically a stall on the final beat of a one-beat transfer. That single cell is where premature-completion bugs live, and a random test reaches it far less often than its importance warrants — so it is worth writing directly.

23. Debug Lab

Each of the three failures below was produced by injecting the bug into the real module from Sections 17 to 20 and re-running the same testbench under Icarus. The failure output is the actual simulator output, not a reconstruction.

// Instrumentation counts every cycle the engine OFFERS a beat.
assign beat_moved = dst_valid;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n)         bytes_q <= '0;
  else if (beat_moved) bytes_q <= bytes_q + CNT_W'(BYTES_PER_BEAT);
end

--- A done: accepted_beats=8 bytes_moved=64 bytes_offered=96

assign beat_moved = dst_valid && dst_ready;

a_bytes_only_on_accept: assert property (@(posedge clk) disable iff (!rst_n || clear)
  (!(dst_valid && dst_ready)) |=> $stable(bytes_moved_q));

// The engine asks the source for a beat whenever it still owes beats,
// without checking whether it has anywhere to put the answer.
assign src_req = busy_q && (to_fetch_q != '0);

FAIL cycle=7  INV-1 data changed under backpressure
FAIL cycle=8  INV-1 data changed under backpressure
FAIL cycle=9  INV-1 data changed under backpressure
FAIL cycle=10 INV-1 data changed under backpressure
FAIL cycle=10 INV-3 got a000000000000006 exp a000000000000002
FAIL cycle=11 INV-3 got a000000000000007 exp a000000000000003
RESULT: FAIL - timeout

assign src_req = busy_q && (to_fetch_q != '0) && (!hold_valid_q || dst_fire);

// Completion accounting advances when the beat is FETCHED from the source.
if (src_fire) begin
  hold_valid_q <= 1'b1;
  hold_data_q  <= src_data;
  to_fetch_q   <= to_fetch_q - 1'b1;
  remaining_q  <= remaining_q - 1'b1;        // <-- wrong event
  if (remaining_q == 1) begin
    busy_q <= 1'b0;
    done   <= 1'b1;                          // fires one beat early
  end
end

--- A done: accepted_beats=7 bytes_moved=56 bytes_offered=88 busy_cycles=12
            dst_stall=4 src_starve=0 errors=0

if (dst_fire) begin
  remaining_q <= remaining_q - 1'b1;
  if (remaining_q == 1) begin
    busy_q <= 1'b0;
    done   <= 1'b1;
  end
end

a_done_not_while_busy: assert property (@(posedge clk) disable iff (!rst_n)
  done |-> !busy);

24. When Data Movement Is the Bottleneck

Movement bottlenecks are hard to find because nothing reports an error. Every component is healthy, every transfer completes, and the machine is slower than the sum of its parts suggests. What follows is diagnostic reasoning, not a list of vendor commands.

Symptom — accelerator utilisation is low

Order the hypotheses by how cheaply each can be eliminated:

That table is why Section 20 counts both sides. Destination stall at 31% with source starvation at 0% sends the investigation downstream; the reverse sends it upstream; both near zero means the engine is not the problem and the compute is genuinely slow.

Symptom — link utilisation is high but useful throughput is low

The link is busy carrying something that is not useful work. The candidates, in rough order of frequency:

• Small transactions. Protocol overhead is per-transaction, so a stream of tiny transfers can saturate a link while delivering a fraction of it as payload. Check average transfer size — Section 12's diagnostic.
• Duplicate movement. The same bytes crossing twice because a staging hop was not removed. Check total bytes moved against bytes the application logically consumed; a ratio above 1 that nobody can explain is a copy somebody forgot.
• Retries. Bytes carried more than once because the first attempt failed.
• Metadata and descriptors. Control traffic sharing the link with payload.
• Copies that do not feed compute. Data staged, then not used, because the schedule fetched more than the kernel needed.

Symptom — CPU utilisation is high during transfers

This is Chapter 1.2's control plane reappearing. Attribute the CPU time before acting: descriptor construction, interrupt handling, polling loops, and — the one worth checking explicitly — software memory copies that somebody assumed were DMA. A bounce buffer inserted for an alignment or mapping constraint puts a core back on the data plane without announcing it.

Symptom — memory bandwidth is saturated but no single consumer looks large

Amplification. Sum Section 7's three columns across every transfer in the workload, not per transfer. A pipeline that stages data through memory between stages multiplies the working set by the number of stages, and no individual stage looks unreasonable.

Also check the read/write balance: DRAM pays a bus-turnaround penalty for switching direction, so a copy — which is a read stream and a write stream interleaved on the same memory — achieves lower efficiency than either stream alone would.

25. How This Appears in Real Engineering

System and SoC architect

The recurring decision is Section 3's: move the data, move the compute, or reach across. Answering it needs the byte accounting of Section 7 and the crossover of Section 13, applied per dataflow rather than per component. The questions that follow are placement questions: which memory should hold this working set, how many times will it be traversed before the answer is produced, and which of the three columns saturates first. An architecture with a clean block diagram and four staging hops through host memory has already lost, and the block diagram will not show it.

RTL and microarchitecture engineer

Everything in Sections 17 to 20. Handshake discipline — hold valid and payload stable until acceptance. Elastic buffering, and where a registered ready is worth the extra slot. Two counters rather than one wherever fetch and delivery are decoupled. Ownership state for double buffering. Instrumentation designed in rather than added after silicon, because a movement engine whose stall attribution cannot be read is a movement engine nobody can tune. The recurring sizing judgement is the same one Chapter 1.2 raised: depth, credits and buffer counts all come from a rate multiplied by a latency, and all are frozen at design time.

Verification engineer

Section 22 in full. The failures are load-dependent and silent, so a directed test that never stalls proves almost nothing. Scoreboard every beat; assert payload stability across stalls; test zero-length and one-beat and maximum-length; stall the final beat specifically; reset mid-transfer. The insight worth internalising is that a data mover can have perfect throughput and still be wrong — value, order, count, boundaries, ownership and completion semantics are six independent things to preserve, and throughput measures none of them.

Performance engineer

The number that matters is bytes_moved / elapsed where bytes_moved counts accepted movement, and the numbers that explain it are the occupancy counters. Section 20's derived table is the working set: useful throughput, engine efficiency, destination pressure, source pressure, average transfer size. The habit carried from Chapter 1.1 applies unchanged — compare against a computed bound, not against a previous run — and the habit specific to this chapter is to always compute the amplification ratio: bytes moved anywhere, divided by bytes the application logically consumed.

System software and firmware engineer

Descriptor formats and the ordering rules that make a ring safe between two agents. Buffer allocation, pinning and mapping — and the alignment constraints whose violation silently inserts a bounce buffer. Batch size and completion policy, which is Section 12's trade made concrete. Whether a path advertised as zero-copy actually is, which Section 14 says how to check. And the rule that done means the far end accepted the data, so a buffer freed on completion is only safe if completion means what the hardware makes it mean.

Named industry context

Several device classes make these costs visible, and each is a legitimate example because the mechanism is public:

• GPU host-device transfers over PCIe are Section 11's round trip almost exactly: stage in, compute, stage out, with the link and both memory systems paying the full byte count each way.
• NVMe places its submission and completion queues in host memory and rings a doorbell — the control path of Figure 1, with the storage controller performing the payload movement by DMA.
• RDMA removes the remote host's CPU from the data path entirely, so the payload lands in the destination's memory without a core executing the copy. It does not remove the network, the memory controller, or the memory write.
• Peer-to-peer device transfers — the class GPUDirect-style mechanisms and NVMe peer-to-peer belong to — remove the host memory staging hop, changing device→host→device into device→device. That is a genuine reduction in movement, and Section 14's caution applies: it is not zero movement.
• HBM buys bandwidth by placing memory in the package, close to the compute. That is Section 10's proximity axis traded against capacity, exactly as Chapter 1.1 described.
• The IOMMU makes device access to host memory safe, and adds translation-structure accesses and invalidation traffic to the cost of every mapped transfer.

26. Common Misconceptions

27. Interview Reasoning

Data movement is a favourite senior-level topic because almost every question has a confident wrong answer. The follow-up is what discriminates.

28. Summary

Moving a byte is work, and the work is charged to several accounts at once. A copy of N bytes consumes N bytes of read bandwidth at the source, N on every interconnect hop, and N of write bandwidth at the destination — at least 2N bytes of memory traffic before descriptors, coherence, framing or read-for-ownership are counted. Section 11's round trip makes that concrete: an 8 GiB working set staged to a device and back moves 17.2 GB across the link and causes 34.4 GB of DRAM traffic, four times the data being computed on.

Time behaves the same way. End-to-end movement latency is a stack — software preparation, queueing, source read, traversal, destination write, completion — and only one term scales with size. That single fact produces the batching result: at 4 KiB per transfer, 99% of the elapsed time is fixed overhead and the link is idle for nearly all of it. It also produces the caution that batching converts a throughput problem into a latency problem, because a batch does not depart until it is full.

Energy tracks bytes and distance rather than operations, so for low-intensity work the operands cost more than the arithmetic. Which makes reducing bytes moved the highest-leverage optimisation available — through reuse, blocking, fusion and batching — none of which adds any arithmetic capability at all.

Whether to copy or to reach across is a crossover with a threshold you can compute: copying wins once the computation touches the data enough times to amortise S / B_bulk against a per-access remote penalty, which for the illustrative numbers here is about 1.6% of a 1 GiB buffer. Caching at the far end and adequate concurrency both move that threshold, which is why device-side caches and deep outstanding-request support are architectural requirements rather than optimisations.

In hardware all of it reduces to a handshake. The transfer event is valid && ready and nothing else: an offered beat is not a moved beat, a producer must hold its payload stable until acceptance, completion must follow acceptance rather than fetch, and a counter that disagrees with any of those reports a bandwidth the interface cannot physically deliver. The four simulated modules in Sections 17 to 20 are those rules in eighty lines of SystemVerilog, and the three Debug Labs are what each one costs when it is broken: 50% mis-reported bandwidth, four silently lost beats, and a completion that fires while the last beat is still in the engine.

29. What Comes Next

Hold the conclusion precisely, because the imprecise version is a marketing claim.

Data movement is not free and cannot be made free. Reaching a byte over a link consumes link bandwidth, latency, protocol resources, buffering, and energy — whatever the protocol on that link is called. Any architecture that promises otherwise is promising something physics does not supply.

What an architecture can change is which movements are necessary. Chapters 1.1 to 1.3 have located three that are not inherent to the computation:

• Staging copies forced by unaddressable domains. If a device cannot address host memory, the data must be duplicated into memory it can address. That copy exists because of a reachability limit, not because the algorithm needs two copies.
• Duplication forced by fixed attachment. If memory is behind exactly one socket, sharing it means copying it. Chapter 1.1's stranding argument and this chapter's amplification argument are the same fact seen from two sides.
• Software-managed consistency. If two agents share data without hardware coherence, correctness is manufactured with flushes and invalidations that are themselves memory traffic — coarse, expensive, and paid on whole buffers rather than on the lines actually touched.

Each is a consequence of where memory may live and who may address it, which is exactly the assumption Chapter 1.2 identified as the load-bearing one. That is the pressure the rest of this track resolves, and the honest statement of what changes is narrow:

A more flexible attachment model can remove staging copies that exist only because two domains cannot address each other. It does not remove the bytes, the bandwidth, the latency or the energy — and memory reached over a link remains farther away than memory on the local channels.

Module 1 continues by examining the forces this chapter has priced but not explained. Chapter 1.4 covers why accelerators proliferated and what that did to device attach. Chapter 1.5 examines why discrete accelerator memory cannot grow with working sets. Chapter 1.6 is the accounting of what conventional device attach provides and where it stops. Chapter 1.7 makes the case for coherent attach as a mechanism — the first chapter in this track where the answer, rather than the pressure, is the subject.

For the mechanisms behind several arguments here, the PCIe track covers DMA concepts, DMA engines and host memory access; the AXI track builds this chapter's hardware properly in the transfer event, backpressure and stalls and outstanding depth tuning; and UCIe bandwidth decomposes raw against useful throughput on a link in detail. The full path is on the CXL tutorials index.