"DMA Bypasses PCIe Protocol" — It Bypasses the CPU, Not the Rules

The belief: DMA is a separate high-speed path. The device writes straight into memory, and the protocol machinery — headers, credits, tags, ordering — applies to the other kind of traffic.

The word doing the damage is "bypass". DMA bypasses the CPU. It does not bypass the protocol, because there is no other way to move a byte across the link.

1. Why a Competent Engineer Believes It

The myth is assembled from four accurate observations.

DMA genuinely does bypass something, and it is the CPU. The whole point is that the processor is not copying bytes. "Bypass" is the right word attached to the wrong object.

The performance difference is real and large. Moving a megabyte by MMIO stalls a core for the entire transfer; the DMA engine does it without one. A 100× difference invites a structural explanation, and "different path" is the obvious one.

The programming interfaces look nothing alike. MMIO is a pointer dereference. DMA is descriptors, doorbells and completion callbacks. Two interfaces that different suggest two mechanisms.

And the word appears in adjacent, correct contexts. IOMMU bypass is real; cache-coherency bypass is real; peer-to-peer traffic bypassing host memory is real. In a room where "bypass" is used precisely three times, the fourth use slides past unchallenged.

2. The Locally True Kernel

Preserve the intuition; change its object.

DMA bypasses the CPU. For reasoning about who spends time, that simplification is exactly right — the core is free, the transfer proceeds asynchronously, and software's only involvement is at submission and completion.

That model is correct for: capacity planning of CPU time, deciding whether to use MMIO or DMA for a given transfer size, and reasoning about which agent initiates work (28.3 §2).

The scope boundary is the moment you ask about anything on the link. Bandwidth, latency, ordering, resource sizing, failure modes — for every one of those questions, DMA is ordinary traffic, and §8 measures how ordinary.

3. The Hidden Assumption

The myth assumes that "the CPU is not involved" implies "the transport is different".

It does not, and the reason is structural: a PCIe link carries TLPs. There is no second wire, no side channel, no privileged mode. A device that wants to write host memory issues a Memory Write TLP (12.2) — the same transaction type a CPU-initiated write would produce in the opposite direction.

What DMA actually changes is one thing: the initiator.

Exactly one row differs. The replacement model:

DMA changes who initiates a transfer, not what a transfer is. Every DMA byte crosses the link inside an ordinary TLP, subject to every rule that governs any other TLP.

4. The Root-Cause Tree

The wrong diagnosis costs the most. The engineer measures the link, finds it negotiated correctly at the expected width and speed, and concludes PCIe is underperforming — when the engine is stalling on its own tag pool (27.4 §14 measured that exact confusion).

5. The Minimal Counterexample

One transfer, one number, no simulation.

transfer size   1 MiB
MPS             256 bytes   (negotiated, not capability)
 
payload TLPs = 1,048,576 / 256 = 4,096

If DMA bypassed the protocol, MPS would not appear in that calculation. It does, because every 256 bytes needs its own header, and the header is protocol.

Then the follow-up that closes it: those 4,096 writes are Posted, so they consume posted credit and receive no Completions (12.2). The descriptor fetch that started the transfer is Non-Posted, so it consumed a tag and waited for a Completion. Three protocol mechanisms in one "bypass".

6. The Transfer, Step by Step

Three readings.

Every arrow into the Transaction Layer is a TLP. There is no arrow that skips it, because there is nowhere for it to go.

The descriptor fetch is the only Non-Posted step and it is the only one that consumes a tag. The 4,096 payload writes consume credit instead — a different resource with a different exhaustion signature (22.3).

And the last two arrows are ordinary writes. The status record and the interrupt are Memory Writes like the payload — which is 31.5's entire subject.

7. The RTL the Myth Produces

The wrong RTL. An engine written by someone who believes DMA is a fast path.

// WRONG — the engine issues as fast as its datapath allows. There is no tag
// reservation, no credit gate, and no chunking bound, because "DMA is the
// fast path and the link keeps up".
module dma_engine_wrong (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        job_valid,
  input  logic [63:0] job_addr,
  input  logic [31:0] job_bytes,
  output logic        req_valid,
  output logic [63:0] req_addr,
  output logic [31:0] req_len,        // the WHOLE remaining length
  input  logic        req_ready,
  output logic        job_done
);
 
  logic [63:0] cur_addr;
  logic [31:0] remaining;
  logic        active;
 
  assign req_valid = active && (remaining != 0);
  assign req_addr  = cur_addr;
  assign req_len   = remaining;                 // no MPS bound
  assign job_done  = active && (remaining == 0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      active <= 1'b0; cur_addr <= '0; remaining <= '0;
    end else if (job_valid && !active) begin
      active <= 1'b1; cur_addr <= job_addr; remaining <= job_bytes;
    end else if (req_valid) begin               // <-- advances on VALID
      cur_addr  <= cur_addr + 64'(req_len);
      remaining <= '0;
      active    <= 1'b0;
    end
  end
 
endmodule

Why it passes basic tests. In a testbench where req_ready is tied high and the transaction layer is a perfect sink, this works. Every directed test written by the same engineer passes, because those tests model the fast path they believe exists.

Three defects, and each is the myth made structural:

req_len is the whole remaining length. There is no MPS bound, because a bypass path would not have one. A single request for 1 MiB is not a thing the Transaction Layer can express.

State advances on req_valid, not on req_valid && req_ready. The engine believes it cannot be back-pressured. The moment the transaction layer stalls, the engine advances anyway and the transfer is silently truncated.

And there is no resource gate at all — no tag for the descriptor read, no credit check for the writes. The engine assumes the resources are somebody else's problem, which is the myth stated in RTL.

8. Measured — The Same Transfer, Different Resources

And the transaction count itself, for the same bytes at different MPS:

Three readings.

The TLP count is identical in all four resource rows — 4,096. The bytes do not change. What changes by nearly 9× is how long the engine spends unable to issue them, which is the definition of being subject to a resource.

Halving MPS doubles the transaction count. A negotiated MPS the device did not choose changes the shape of every transfer it makes (22.4).

And the fixed cost is three transactions, which matters enormously at 4 KiB — 3 of 19, nearly 16% — and not at all at 1 MiB. A "bypass" would have no fixed cost to amortise.

9. The Failure Timeline

The wrong engine, in a system where the transaction layer occasionally stalls.

cycle    0   job accepted: 1 MiB at 0x1000_0000. remaining = 1,048,576
cycle    1   req_valid asserts with req_len = 1,048,576
             The TL cannot express this; assume it clamps to MPS silently
             or the adapter truncates. Either way the engine does not know.
cycle    1   req_ready is LOW — the TL has no posted credit this cycle
cycle    2   req_valid is still high (correct), but the engine's always_ff
             already saw req_valid last cycle:
                 cur_addr  <= +1,048,576
                 remaining <= 0
                 active    <= 0
             THE ENGINE HAS RETIRED THE JOB. Nothing was transferred.
cycle    3   job_done asserts. The driver is told the transfer completed.
cycle  400   software reads the destination buffer and finds stale data

The first divergence is cycle 2 — state advancing on valid without ready.

Why the symptom points elsewhere. The driver received a completion; the device reported success; the link is healthy; no error was raised anywhere. The buffer simply contains the wrong bytes, and the investigation starts at the consumer of the buffer. This is 25.6 §3's corrupting class, reached by a different route.

And under light load it does not happen. If req_ready is high whenever req_valid is, valid-advance and valid && ready-advance are indistinguishable. The bug appears exactly when the protocol asserts itself — which the myth says it will not.

10. The Corrected RTL

// The same engine written by someone who knows DMA is ordinary traffic.
// Three changes, each undoing one clause of §7's myth.
module dma_engine_correct #(
  parameter int unsigned MAX_PAYLOAD = 256      // NEGOTIATED MPS, not capability
)(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        job_valid,
  output logic        job_ready,
  input  logic [63:0] job_addr,
  input  logic [31:0] job_bytes,
  // resources the engine must respect because it is ordinary traffic
  input  logic        credit_ok,                // posted credit for one TLP
  input  logic        tag_granted,              // for the non-posted descriptor read
  input  logic        need_tag,
  // request interface
  output logic        req_valid,
  output logic [63:0] req_addr,
  output logic [11:0] req_len,
  input  logic        req_ready,
  output logic        job_done,
  output logic [31:0] bytes_issued
);
 
  logic [63:0] cur_addr;
  logic [31:0] remaining;
  logic        active;
  logic [11:0] this_len;
  logic        xfer;
 
  always_comb begin
    // 1. Chunked to the NEGOTIATED payload. A request larger than MPS is not
    //    expressible, and the final chunk is partial whenever the length is
    //    not a multiple — the dropped-tail bug of 25.6 §4 if omitted.
    this_len  = (remaining > MAX_PAYLOAD) ? 12'(MAX_PAYLOAD) : 12'(remaining);
 
    // 2. Gated on the resources this traffic actually consumes. A DMA write
    //    needs posted credit like any other write; the descriptor read needs
    //    a tag like any other non-posted request.
    req_valid = active && (remaining != 0) && credit_ok && (!need_tag || tag_granted);
    req_addr  = cur_addr;
    req_len   = this_len;
 
    // 3. Progress on the TRANSFER, never on valid alone.
    xfer      = req_valid && req_ready;
    job_ready = !active;
    job_done  = active && (remaining == 0);
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      active <= 1'b0; cur_addr <= '0; remaining <= '0; bytes_issued <= '0;
    end else begin
      if (job_valid && job_ready) begin
        active <= 1'b1; cur_addr <= job_addr; remaining <= job_bytes;
        bytes_issued <= '0;
      end else if (xfer) begin
        cur_addr     <= cur_addr + 64'(this_len);
        remaining    <= remaining - 32'(this_len);
        bytes_issued <= bytes_issued + 32'(this_len);
      end else if (job_done) begin
        active <= 1'b0;
      end
    end
  end
 
endmodule

The six lenses.

ARCHITECTURE. The engine exists to originate transfers without the CPU. It does not exist to originate them without the Transaction Layer, and the three gates encode that.

STATE. cur_addr, remaining, active, bytes_issued. remaining is the transfer's ownership: it belongs to the engine from acceptance to job_done.

EVENT. Ownership begins at job_valid && job_ready and ends at remaining == 0. Progress happens only at req_valid && req_ready.

CONTRACT. The Transaction Layer relies on req_len <= MPS and on req_valid being held stable until req_ready. The driver relies on job_done meaning every byte was issued. bytes_issued is what makes that checkable.

FAILURE. Removing the xfer qualifier reproduces §9 exactly — a job that retires having transferred nothing, reported as success.

DV / DEBUG. §12's p1_progress_on_transfer and p5_bytes_conserved catch it; §13's scoreboard must count accepted transfers rather than offered ones.

11. Same-Cycle Audit

12. Assertions

// P1 — progress happens on the TRANSFER, never on valid alone. This single
// property is the RTL expression of §3's replacement model: the engine is
// subject to backpressure like any other requester.
// Catches: §9's entire failure timeline.
property p1_progress_on_transfer;
  @(posedge clk) disable iff (!rst_n)
    (remaining != $past(remaining)) |-> $past(req_valid && req_ready);
endproperty
a_p1: assert property (p1_progress_on_transfer);
 
// P2 — no request exceeds the negotiated payload. A bypass path would have
// no such bound; an ordinary requester does.
property p2_chunk_bounded;
  @(posedge clk) disable iff (!rst_n)
    req_valid |-> (req_len <= MAX_PAYLOAD) && (req_len != 0);
endproperty
a_p2: assert property (p2_chunk_bounded);
 
// P3 — nothing is issued without the resources it consumes. Splitting this
// into two properties would be padding; the invariant is one thing —
// "this traffic pays like all other traffic".
property p3_resources_before_issue;
  @(posedge clk) disable iff (!rst_n)
    req_valid |-> (credit_ok && (!need_tag || tag_granted));
endproperty
a_p3: assert property (p3_resources_before_issue);
 
// P4 — the request is stable while offered and not accepted. Without it the
// address can advance underneath a stalled transaction layer.
property p4_request_stable_under_stall;
  @(posedge clk) disable iff (!rst_n)
    (req_valid && !req_ready) |=> (req_valid && $stable(req_addr) && $stable(req_len));
endproperty
a_p4: assert property (p4_request_stable_under_stall);
 
// P5 — bytes issued equal bytes described, at completion. The two-sided
// form matters: 25.6 §4 measured a one-sided check reporting success on an
// engine that moved MORE than described.
property p5_bytes_conserved;
  @(posedge clk) disable iff (!rst_n)
    job_done |-> (bytes_issued == $past(job_bytes, 1) || active);
endproperty
a_p5: assert property (p5_bytes_conserved);
 
// P6 — a new job is not accepted while one is active. §11 audit A.
property p6_no_job_overlap;
  @(posedge clk) disable iff (!rst_n)
    (job_valid && job_ready) |-> !active;
endproperty
a_p6: assert property (p6_no_job_overlap);
 
// Vacuity guards. P2's partial-chunk case and P4's stall case are both
// unreachable against a testbench with an always-ready sink and
// MPS-multiple lengths — which is exactly the stimulus the myth produces.
c1_partial_chunk: cover property (@(posedge clk) disable iff (!rst_n)
                    req_valid && (req_len < MAX_PAYLOAD));
c2_stalled_req:   cover property (@(posedge clk) disable iff (!rst_n)
                    req_valid && !req_ready);
c3_credit_block:  cover property (@(posedge clk) disable iff (!rst_n)
                    active && (remaining != 0) && !credit_ok);

c2_stalled_req is the cover that matters most. A testbench with an always-ready sink makes P1 and P4 vacuous — and an always-ready sink is precisely what an engineer who believes in the bypass path will write.

13. If DV Believes the Same Myth

The testbench models a fast path, and the fast path never stalls.

The scoreboard row is the fatal one. A scoreboard that counts offered requests matches the wrong RTL and disagrees with the correct one — because the correct engine offers a request repeatedly while stalled, and a naive counter records each offer as a transfer.

What a correct testbench does differently. The BFM applies backpressure and withdraws credit adversarially; the scoreboard counts valid && ready; and the reference model tracks which byte ranges were issued rather than a total. That last change is what makes §9's truncation visible.

14. Debugging

One note on instrument choice. Unlike 31.3, an analyzer is not the right first instrument here. The transactions on the wire are all well-formed; what is wrong is how many there are and how long the engine waited between them. 25.9 §13's measurement applies — this is internal state, and the trace looks healthy.

15. Review and Interview

The review gate this myth corrupts. An architecture review that accepts "the DMA engine sustains 16 GB/s" without asking what that implies has accepted a number with no derivation.

The review question: "How many TLPs is a typical transfer at the negotiated MPS, how many tags does the descriptor path need, and what is the posted-credit requirement for the payload?"

Three numbers, all derivable in the room, all of which the myth says are irrelevant. An engineer who cannot produce them has not designed the engine as ordinary traffic.

The interview exchange.

Weak answer: "DMA lets the device write directly to memory without going through the CPU or the protocol overhead."

Why it sounds plausible: the first half is correct and the second half sounds like a natural extension of it.

Interviewer follow-up: "If it doesn't go through the protocol, what does MPS have to do with a DMA transfer?"

Where the weak model breaks: it cannot explain why a device-chosen transfer size is bounded by a negotiated protocol parameter. The myth has no mechanism for MPS to apply.

Strong answer: "DMA bypasses the CPU, not the protocol. The device masters ordinary Memory Reads and Writes — a 1 MiB transfer at a negotiated 256-byte MPS is 4,096 Memory Write TLPs, each with a header, each gated by posted credit. The descriptor fetch is Non-Posted, so it consumes a tag and waits for a Completion. What DMA changes is who initiates; everything about what travels is unchanged."

Senior follow-up: "Your engine hits 60% of target and the link negotiated correctly. Where do you look?" — outstanding occupancy first, because it splits the space in one read (27.4 §3), then request size on the wire, then the completion round trip.

16. Misconceptions Inside the Misconception

"DMA is faster because it has less overhead per byte." Why it sounds plausible: DMA is genuinely much faster than MMIO for bulk data. What really happens: the per-byte protocol overhead is identical — same headers, same MPS, same credit. DMA is faster because it is pipelined and does not stall a core, not because it carries less. What it causes: bandwidth models that omit header overhead for DMA and include it for MMIO, which is backwards.

"Peer-to-peer DMA bypasses the protocol since it never reaches host memory." Why it sounds plausible: the host really is not involved, and "bypass" is used correctly about host memory here. What really happens: it is still Memory Write TLPs, routed by address through the fabric (21.1). A different destination is not a different mechanism. What it causes: peer-to-peer paths sized without credit or ordering analysis, on the assumption that skipping the host skipped the rules.

"The DMA engine can just be given a big buffer and left alone." Why it sounds plausible: buffers absorb bursts, and the engine is autonomous. What really happens: the engine still must not issue without credit and must not advance without acceptance (§10). A buffer changes when it stalls, not whether — the same result 28.3 §13 measured for a rate mismatch. What it causes: a deeper FIFO proposed as a fix for a resource shortfall it cannot address.

"If the link is healthy, DMA will reach line rate." Why it sounds plausible: the link is the thing that carries the bytes. What really happens: §8 measured the identical transfer stalling for 303,831 cycles on a tag pool with the link untouched. 27.4 §14 measured six causes of low throughput, of which the link is one. What it causes: §14's wrong diagnosis — an investigation that ends at the physical layer with nothing found.

17. Understanding Check

Q1. A "1 MiB DMA transfer" is described in a design document. Restate it as what actually crosses the link, and name every protocol resource it consumes.

At a negotiated 256-byte MPS it is 4,096 Memory Write TLPs, plus a descriptor fetch, a status write and an interrupt message — 4,099 transactions (§5, §8). The resources: posted credit for every one of the 4,096 payload writes and the two trailing writes (16.1); a tag for the descriptor fetch, which is Non-Posted and waits for a Completion (23.5); and ordering between the payload, the status write and the interrupt, which is what makes the status meaningful (13.4). Halving MPS to 128 would double the payload count to 8,192 — a device-external parameter reshaping every transfer the device makes.

Q2. §8 shows 4,096 TLPs in every row and stall cycles ranging from 11,469 to 303,831. What does that spread demonstrate, and what would a genuine bypass path look like in that table?

It demonstrates that DMA is subject to protocol resources (§3, §8). The bytes are constant; what varies by nearly 9× is how long the engine spent unable to issue, purely from changing the tag pool and the credit availability. A genuine bypass path would show a flat column — identical stall counts regardless of tags and credit, because it would not be consuming them. The spread is the measurement that refutes the myth, and it is more convincing than the TLP count because the TLP count could be dismissed as bookkeeping.

Q3. In §7's wrong RTL, name the single line that produces §9's timeline and explain why light-load testing never finds it.

end else if (req_valid) begin — state advances on valid rather than on valid && ready (§7, §10, P1). Under light load req_ready is high whenever req_valid is, so the two conditions are indistinguishable; every directed test passes. The bug appears only when the Transaction Layer applies backpressure, which is exactly the situation the myth says will not arise. c2_stalled_req is the cover that proves a testbench exercised it, and a BFM with req_ready tied high — the natural thing to write if you believe in the fast path — makes P1 and P4 vacuous.

Q4. A scoreboard counts req_valid assertions and reports the correct byte total against the wrong RTL. Explain why, and what it must count instead.

Because the wrong RTL advances on valid, and so does the scoreboard (§13). Both hold the same model, so they agree — and worse, the same scoreboard disagrees with the correct RTL, which holds a request asserted across multiple stalled cycles and would be counted several times. It must count req_valid && req_ready, and the reference model must track which byte ranges were issued rather than a running total, because a total cannot distinguish a truncated transfer from a complete one when the truncation is reported as success (§9 cycle 3).

Q5. Why is a protocol analyzer the wrong first instrument here, when it was the right one for 31.3?

Because the fault is internal and the wire looks healthy (§14). In 31.3 the bug was on the link — a second read that the driver author did not know was being issued — so a trace settles it immediately. Here every TLP is well-formed, correctly sized and correctly ordered; what is wrong is how many cycles the engine waited between them, which is engine state and never becomes a packet. 25.9 §13 measured internal faults producing byte-identical traces. The right first instruments are the offered-versus-accepted counter and outstanding-tag occupancy — both device-side registers.

18. What Comes Next

§6's sequence ended with two Memory Writes: a status record and an interrupt message. This chapter treated them as bookkeeping.

31.5 is about the second one. If an interrupt is a Memory Write TLP — an ordinary Posted write like the 4,096 that preceded it — then it is subject to write ordering, and the question of whether the host sees the data before the interrupt stops being a software question and becomes a transaction one.