PCIe vs USB — Who Is Allowed to Start a Transfer

Chapter 28.2 compared two fabrics that both let a device transmit when it has something to send.

USB does not. In the classic USB model the host schedules every transfer, and a device with data waits to be asked. That single rule changes the endpoint's hardware, its latency distribution, and what it can promise — and it is the cleanest available illustration of why PCIe endpoints contain DMA engines at all.

1. Sources, Scope, and What Is Being Compared

2. The Question That Divides Them

Both are host-centric interconnects with enumeration, addressing, and a tree topology. They diverge on one question:

When a device has data, may it send?

PCIe: yes. An endpoint with bus-master capability issues Memory Writes toward host memory whenever it has something to write. The host is not consulted per transfer.

Classic USB: no. The host schedules transactions; a device transmits when it is asked to. A device with data waits for its turn.

Everything else in this chapter is a consequence.

3. What "The Device May Send" Actually Requires

The permission is easy to state and expensive to implement. A PCIe endpoint that originates traffic needs, at minimum:

Bus-master capability, enabled by the host during configuration. Until it is, the endpoint may respond but not originate — and a device that appears enumerated and moves no data is very often one whose bus-mastering was never enabled (§14 case 1).

Somewhere to write. The endpoint needs host addresses, which means the host must have published buffers to it — a descriptor ring, a queue, an address register. The device does not know where host memory is; it is told.

Address translation on its behalf, where an IOMMU is present. The address the device emits is not necessarily the physical address (26.1 §5).

Outstanding-request machinery, if it reads. Tags, an outstanding table, completion matching, a timeout (23.5, 25.7).

And a notification path, because a Posted write receives no Completion (12.2) — so the device publishes a status record and then signals (19.3).

A polled device needs none of these. It needs a buffer and the ability to answer. That is the actual trade: PCIe moved control to the device and, with it, moved a substantial amount of hardware and a whole class of ownership bugs.

4. What "The Host Asks" Actually Buys

Host scheduling is not a limitation somebody failed to remove. It buys three things.

Determinism. If the host decides when every transfer happens, it can reason about when they happen. Bandwidth can be allocated in advance, and a device cannot flood the bus by deciding to.

Simplicity at the device. No DMA engine, no address translation exposure, no outstanding tracking, no completion timeout. A polled device cannot have a descriptor ownership bug because it has no descriptors.

And containment. A device that cannot originate transactions cannot write to arbitrary host memory. The security properties differ substantially, and that is one reason IOMMUs matter more for bus-mastering devices than for polled ones (26.1 §5).

The cost is in §12's table, and it is not primarily latency — it is throughput ceiling. A polled device is limited by how often it is asked, and a device that becomes ready faster than that accumulates work it can never discharge.

5. The Latency Distribution Is the Interesting Part

Mean latency is the least useful number in §12's table. Compare the shapes:

In the first two polled rows the maximum is a small multiple of the service interval. That is a system in equilibrium: work arrives, waits its turn, and is served.

In the last row the maximum is three orders of magnitude larger than the interval. That is not a longer wait — it is a queue that never drains, and the mean and median converging near half the run length is the signature. A latency figure quoted from that regime describes the measurement window, not the system.

The practical rule, and it is the same discipline 22.2 applies to PCIe: check whether the system is in equilibrium before quoting any latency statistic. If the maximum scales with run length rather than with the service interval, the number is meaningless.

6. Where PCIe Uses the USB Model Anyway

PCIe has a host-initiated path too, and recognising it prevents a common confusion.

MMIO is host-initiated. A CPU load or store to a BAR window is exactly the polled model: the host decides when, the device responds (9.6). It has the same properties — bounded by how often the host asks, simple at the device, and unable to flood.

And it has the same limitation, which is why every chapter in Module 26 draws the control-path/data-path distinction: moving bulk data through MMIO stalls a core for the entire transfer (26.2 §3).

So the honest framing is not "PCIe is device-initiated and USB is host-initiated". It is:

PCIe has both models and uses each where it fits. The classic USB model has one.

A PCIe device uses host-initiated MMIO for control — small, latency-sensitive, infrequent — and device-initiated DMA for bulk. The doorbell pattern that appears in 26.2, 26.3 and 26.4 is precisely the handoff between them: a host-initiated write that announces work the device will then fetch on its own initiative.

7. The Comparison, Drawn

The upper path has no waiting box. That is the entire measured difference in §12, and the price of removing it is every block between dma and ir — which is the hardware §3 enumerated.

8. RTL — The Two Endpoint Shapes

Block 1 — a polled device's response path. It cannot initiate, so its only decision is what to answer with.

// A device that responds when asked. It holds data and waits.
// Note what is ABSENT: no address, no request generation, no outstanding
// tracking, no timeout, no completion matching. That absence is §4's point.
module polled_endpoint #(
  parameter int unsigned DEPTH = 32
)(
  input  logic clk,
  input  logic rst_n,
  // the device's own datapath produces data
  input  logic                            data_valid,
  input  logic [63:0]                     data_in,
  output logic                            data_ready,
  // the host asks
  input  logic                            host_poll,
  output logic                            rsp_valid,
  output logic [63:0]                     rsp_data,
  output logic                            rsp_nothing,     // polled with nothing to give
  output logic [$clog2(DEPTH+1)-1:0]      backlog,
  output logic                            overflow
);
 
  logic [63:0] mem [DEPTH];
  logic [$clog2(DEPTH)-1:0] wr, rd;
 
  // The device applies backpressure to its OWN datapath, because it cannot
  // apply it to the host and cannot send unprompted. When the buffer fills,
  // the only options are to stall production or to lose data — which is the
  // same choice 28.2 §3 described, now inside a single device.
  assign data_ready  = (backlog != DEPTH);
  assign overflow    = data_valid && !data_ready;
  assign rsp_valid   = host_poll && (backlog != 0);
  assign rsp_nothing = host_poll && (backlog == 0);
  assign rsp_data    = mem[rd];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr <= '0; rd <= '0; backlog <= '0;
    end else begin
      if (data_valid && data_ready) begin
        mem[wr] <= data_in;
        wr <= (wr == DEPTH-1) ? '0 : wr + 1'b1;
      end
      if (rsp_valid)
        rd <= (rd == DEPTH-1) ? '0 : rd + 1'b1;
 
      // Production and service are independent; all four combinations.
      case ({data_valid && data_ready, rsp_valid})
        2'b10: backlog <= backlog + 1'b1;
        2'b01: backlog <= backlog - 1'b1;
        default: ;
      endcase
    end
  end
 
endmodule

Block 2 — a bus-mastering endpoint's initiation path. It may send, and everything that permission requires is visible.

// A device that initiates. Compare the port list with Block 1: this module
// needs a host address, an enable it does not control, and a way to know
// whether its write was accepted.
module initiating_endpoint #(
  parameter int unsigned DEPTH = 32
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                            data_valid,
  input  logic [63:0]                     data_in,
  output logic                            data_ready,
  // the host must grant permission and supply a destination
  input  logic                            bus_master_enable,
  input  logic                            dest_valid,
  input  logic [63:0]                     dest_addr,
  output logic                            dest_consume,
  // the device originates a Memory Write
  output logic                            mwr_valid,
  output logic [63:0]                     mwr_addr,
  output logic [63:0]                     mwr_data,
  input  logic                            mwr_ready,
  output logic [$clog2(DEPTH+1)-1:0]      backlog,
  output logic                            stalled_no_dest,
  output logic                            stalled_no_enable
);
 
  logic [63:0] mem [DEPTH];
  logic [$clog2(DEPTH)-1:0] wr, rd;
 
  assign data_ready = (backlog != DEPTH);
 
  // The device may send — but only with permission it did not grant itself
  // and to an address it was given. These two stall reasons are reported
  // SEPARATELY because they have different owners: one is configuration,
  // the other is the driver failing to post buffers (§14 cases 1 and 2).
  assign stalled_no_enable = (backlog != 0) && !bus_master_enable;
  assign stalled_no_dest   = (backlog != 0) &&  bus_master_enable && !dest_valid;
 
  assign mwr_valid    = (backlog != 0) && bus_master_enable && dest_valid;
  assign mwr_addr     = dest_addr;
  assign mwr_data     = mem[rd];
  assign dest_consume = mwr_valid && mwr_ready;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr <= '0; rd <= '0; backlog <= '0;
    end else begin
      if (data_valid && data_ready) begin
        mem[wr] <= data_in;
        wr <= (wr == DEPTH-1) ? '0 : wr + 1'b1;
      end
      // The entry is released on the ACCEPTED transfer, not on mwr_valid.
      // Releasing on valid drops data whenever the link back-pressures —
      // the ready/valid contract of 28.1 §11 A2.
      if (mwr_valid && mwr_ready)
        rd <= (rd == DEPTH-1) ? '0 : rd + 1'b1;
 
      case ({data_valid && data_ready, mwr_valid && mwr_ready})
        2'b10: backlog <= backlog + 1'b1;
        2'b01: backlog <= backlog - 1'b1;
        default: ;
      endcase
    end
  end
 
endmodule

Compare the two port lists. Block 2 has bus_master_enable, dest_addr and a full ready/valid egress; Block 1 has none of them. That difference is the hardware cost of permission, and it is why §14's first two debugging cases have no polled analogue.

9. Same-Cycle Audit

10. Invariants

// POLLED DEVICE — the invariant is that it never speaks unprompted.
 
// U1 — a response occurs only in a cycle the host polled. This is the whole
// contract of the model, and violating it means the device is transmitting
// on a bus it does not own.
property p_response_only_when_polled;
  @(posedge clk) disable iff (!rst_n) rsp_valid |-> host_poll;
endproperty
 
// U2 — a poll always produces exactly one outcome: data or nothing.
// A poll that produces neither leaves the host waiting for a response that
// will not come, which in a scheduled model stalls the whole interval.
property p_poll_answered;
  @(posedge clk) disable iff (!rst_n) host_poll |-> (rsp_valid ^ rsp_nothing);
endproperty
 
// U3 — the device applies backpressure to its OWN datapath, because it has
// no other option. Overflow here is data loss inside the device.
property p_no_internal_overflow;
  @(posedge clk) disable iff (!rst_n) !overflow;
endproperty
 
// INITIATING DEVICE — the invariant is that permission precedes transmission.
 
// P1 — no Memory Write is issued without bus-master enable. A device that
// originates before the host has enabled it is transmitting without
// authorisation, and on a real fabric the transaction is rejected upstream.
property p_no_write_without_enable;
  @(posedge clk) disable iff (!rst_n) mwr_valid |-> bus_master_enable;
endproperty
 
// P2 — no Memory Write is issued without a destination the host supplied.
// The device does not know where host memory is; it is told (§3).
property p_no_write_without_dest;
  @(posedge clk) disable iff (!rst_n) mwr_valid |-> dest_valid;
endproperty
 
// P3 — the address is captured with the transfer, not read live. Catches
// the fault of §9 audit D.
property p_addr_stable_under_stall;
  @(posedge clk) disable iff (!rst_n)
    (mwr_valid && !mwr_ready) |=> (mwr_valid && $stable(mwr_addr) && $stable(mwr_data));
endproperty
 
// P4 — an entry is released only on an ACCEPTED transfer.
property p_release_on_accept;
  @(posedge clk) disable iff (!rst_n)
    (backlog != $past(backlog) - 1) or $past(mwr_valid && mwr_ready);
endproperty
 
// SHARED — conservation across simultaneous production and service.
property p_backlog_conserved;
  @(posedge clk) disable iff (!rst_n)
    (data_valid && data_ready && drained) |=> (backlog == $past(backlog));
endproperty
 
// LIVENESS — the property that distinguishes the two models formally.
// For the initiating device this holds under stated assumptions. For the
// polled device it holds ONLY if the poll rate exceeds the production rate,
// which §12 measured failing at a 128-step interval.
property p_backlog_drains;
  @(posedge clk) disable iff (!rst_n)
    (backlog != 0) |-> s_eventually (backlog == 0);
endproperty

p_backlog_drains is the assertion worth dwelling on. It is a liveness property that is unconditionally true for the initiating device and conditionally true for the polled one — the condition being a service rate the device does not control. §12 measured the condition failing, and the failure is not a bug in either device.

11. Bug Bank

12. Measured — Who Starts, and What It Costs

Three readings, and the first is the one that is easy to miss.

The transfer count collapses in the last two rows. 7,967 → 3,124 → 781. The device is producing faster than it is being served, so most of what it produced never moved at all. This is a throughput failure that a latency table presents as a latency failure.

The maximum wait scales with the run length, not the interval. In the first two polled rows the maximum is roughly two to eight times the interval — a queue in equilibrium. In the last two it is a large fraction of the whole run, which is the signature of a queue that never drains (§5).

And the device-initiated row has a maximum of zero. There is no waiting stage to have a distribution. The entire latency question disappears, and is replaced by the hardware §3 enumerated.

13. Measured — Deeper Buffers Do Not Fix a Rate Mismatch

The production rate is ~0.0200/step; the service rate is 1/128 ≈ 0.0078/step. Production exceeds service by 2.56×, and no buffer depth changes that ratio.

The served column is identical at every depth. The service rate sets it, and a buffer cannot change a service rate.

What the buffer changes is where the excess goes, and the last two rows are the instructive ones. At depth 8,192 nothing is recorded as lost — but 4,843 units are still sitting in the buffer when the run ends. The loss has not been prevented; it has been deferred past the observation window. A longer run reaches the same depth and begins losing.

That is why this is worse than the shallow case rather than better. A device with a 32-entry buffer loses data immediately and visibly. A device with an 8,192-entry buffer passes the test, ships, and loses data in the field after an interval nobody measured.

And for contrast, raising the service rate instead:

Halving the interval roughly doubles throughput until service exceeds production, at which point loss reaches zero and stays there. That is the fix.

The engineering consequence is the same one 25.8 §5 reached about credit pools: a resource increase that defers a structural failure converts a reproducible bug into an intermittent one, which is strictly worse. The fixes that work are raising the service rate, lowering the production rate, or moving more than one unit per service opportunity — and only the last is available to the device designer.

14. Debugging

15. Misconceptions

"PCIe is device-initiated and USB is host-initiated." Why it sounds reasonable: it is the headline difference and it is half true. What actually happens: PCIe has both models — host-initiated MMIO and device-initiated DMA — and uses each where it fits (§6). What it causes: engineers who move bulk data through MMIO because "PCIe is fast", stalling a core for the entire transfer.

"A polled device is simpler because the protocol is simpler." Why it sounds reasonable: the transfer rules genuinely are simpler. What actually happens: the simplicity is in the device, which needs no DMA engine, no address logic, no outstanding tracking and no timeout (§3, §4). The scheduling complexity moved to the host. What it causes: underestimating what bus-mastering costs when specifying a PCIe endpoint — and then discovering that §3's list is not optional.

"Mean latency describes the system." Why it sounds reasonable: it is the statistic everyone quotes. What actually happens: §12's 128-step row has a mean of 122,473 against a 128-step interval, because the queue never drains. The mean describes the observation window (§5). What it causes: published latency figures that change with run length, and capacity plans built on them.

"A bigger buffer will fix the polling latency." Why it sounds reasonable: buffers absorb bursts, and this looks like a burst. What actually happens: §13 measured a 256× buffer increase completing the same 3,124 transfers. A buffer absorbs a burst; it cannot absorb a rate mismatch. What it causes: a structural failure converted into an intermittent one — the same trap 25.8 §5 identified for credit pools.

"Device-initiated is strictly better." Why it sounds reasonable: §12's device-initiated row has zero waiting. What actually happens: it costs bus-master permission, address translation exposure, outstanding-request machinery and a notification path (§3) — plus an entire class of ownership bugs (25.6). Host scheduling buys determinism and containment (§4). What it causes: dismissing a scheduled model for a system that needed its guarantees, and then rebuilding those guarantees in software.

"The device decides when to send, so the host can't control bandwidth." Why it sounds reasonable: removing the host from the per-transfer decision does remove that lever. What actually happens: the host controls it elsewhere — by how many descriptors it posts, by bus-master enable, and by the fabric's own arbitration and flow control. What it causes: surprise that a device stops when the driver stops posting buffers, which is §14 case 2 and is the host exercising exactly that control.

16. Understanding Check

Q1. A polled device is serviced every 128 steps and becomes ready roughly every 50. Predict what §12's table shows and explain the mechanism.

Production exceeds service by about 2.6×, so the backlog never drains (§5, §12). The transfer count collapses — §12 measured 3,124 of 7,967 at exactly this interval — because most produced units never move. The latency statistics become meaningless: the mean and median converge near half the run length (122,473 and 122,783 against a 128-step interval), and the maximum scales with the observation window rather than the service interval. The diagnostic is that ratio: a mean wait roughly a thousand times the service interval is not a latency figure, it is a queue that is still filling.

Q2. Your PCIe device enumerates, its BARs respond to MMIO, and it never writes to host memory. Name the two most likely causes in order and how you distinguish them.

Bus-master enable, then destination availability (§14 cases 1 and 2). Bus-mastering is host-controlled and must be enabled before the endpoint may originate anything; until it is, the device can respond but not initiate (§3, p_no_write_without_enable). If it is enabled, the next candidate is that the driver has posted no buffers, so the device has data and no address to put it at (p_no_write_without_dest). They are distinguished by reading the two stall reasons separately, which is why §8 Block 2 keeps stalled_no_enable and stalled_no_dest as distinct outputs — one is a configuration problem and the other is a driver problem, and merging them into "stalled" destroys the distinction.

Q3. A team proposes fixing a polled device's throughput collapse by increasing its buffer from 32 to 8,192 entries. Evaluate.

It changes when the device fills, not whether (§13). Measured at a 128-step service interval, buffer depths from 32 to 65,536 all completed the same 3,124 transfers — the steady state is set by the ratio of production to service rate, and a buffer is a delay line of finite length. Worse, the deep-buffer configuration records zero loss during the run while holding 4,843 unserved units, so it passes the test and loses data in the field. The proposal converts a reproducible failure into a deferred one, which is strictly worse for debugging, and it is the same error 25.8 §5 identified for credit pools. The fixes that work are raising the service rate, lowering production, or moving more than one unit per service opportunity — and only the last is in the device designer's control.

Q4. Explain why the doorbell pattern used throughout Module 26 is a hybrid of both initiation models, and what each half contributes.

The doorbell is host-initiated; the fetch that follows is device-initiated (§6). The host writes a small MMIO register — the polled model's shape: bounded, latency-sensitive, unable to flood — announcing that work exists. The device then reads the work from memory on its own initiative, which is the bus-mastering model and is what allows bulk movement without stalling a core. Each half contributes the property the other lacks: the host retains control of when work is offered, and the device retains control of when and how fast it is consumed. That is why 26.2 §5 insists a doorbell announces work rather than carrying it — carrying it would put bulk data back on the host-initiated path.

Q5. A device works correctly when polled slowly and loses data when polled quickly. Why is this the opposite of what intuition predicts, and where would you look?

Because faster polling makes production and service coincide more often, and coincidence is what the buggy implementations get wrong (§9 audit B, §14 case 6). The backlog counter must handle all four combinations of produce and drain; an exclusive if (produce) ... else if (drain) ... silently discards the drain whenever both occur, so the counter drifts upward and the device stops accepting from its own datapath. At low poll rates the two events almost never coincide and the bug is invisible — 27.4 §17 measured simultaneous events in 23.3% of cycles at load and essentially none at low rate. Look at the backlog update logic, not at the interface.

17. What Comes Next

The three chapters so far have each removed one assumption — that backpressure can be observed, that delivery is guaranteed, that a device may speak when it wishes — and in every case PCIe's mechanism turned out to be a deliberate trade rather than an inevitability.

28.4 removes the last and largest one. Every chapter in this module has assumed that when a device writes host memory, the host will see the new value. That assumption is not free, and PCIe does not provide it — software does, by invalidating caches at the right moments. CXL is what happens when the hardware takes that obligation back.