PCIe · Module 26
PCIe in GPUs — The Control Path Is Not the Data Path
A GPU is an Endpoint whose PCIe link and local memory are different resources. Doubling PCIe changed nothing when local memory was the limit, and reading a live ring entry corrupted 1,449 commands.
Chapter 26.1 built the host side of the boundary. This chapter crosses it — as a device that mostly initiates rather than responds.
A GPU is a PCIe Endpoint with two host-facing paths that share a Link and share almost nothing else: a small control path of BAR writes and doorbells, and a bulk data path driven by copy engines. Treating them as one resource is the origin of most GPU/PCIe confusion.
1. Sources, Scope, and the 26.6 Boundary
2. The GPU as an Endpoint That Mostly Initiates
The host talks to a GPU in small amounts. The GPU moves data in large amounts. That asymmetry is the chapter's organising fact.
host CPU
↕ PCIe
GPU Endpoint
├─ BAR / register aperture ← host writes control, doorbells
├─ command submission ← work described in memory
├─ copy / DMA engines → these ORIGINATE most of the traffic
├─ memory controller + local memory
├─ compute engines
└─ event path (MSI/MSI-X) → notification back to the hostCount the initiators and the picture inverts from the usual mental model. The host initiates a stream of small MMIO writes. The GPU initiates every bulk transfer — reads that pull host data in, writes that push results out, and the writes that deliver completion status and interrupt messages.
By byte volume, a GPU is overwhelmingly a requester, not a completer. Its BAR aperture answers host accesses; its copy engines generate the traffic that matters. This is why 23.1's requester-side machinery — Tag allocation, outstanding-request tracking, Completion matching (23.5) — is where a GPU's PCIe complexity actually lives, and why §7's RTL is about command ownership rather than about BAR decode.
3. Control Path and Data Path
| control path | data path | |
|---|---|---|
| initiated by | host CPU | GPU copy engine |
| transaction size | a few bytes | up to MPS per TLP, many TLPs |
| what it costs | a stalled core | Link bandwidth and Tags |
| limited by | round-trip latency | the slowest stage (§12) |
| failure looks like | registers read wrong | transfers stall or corrupt |
Two rules follow, and both are frequently broken.
Do not model bulk transfer as repeated MMIO stores. Each MMIO write is a host-initiated Posted write costing a core's time; moving a megabyte that way stalls a core for the entire transfer. Bulk movement belongs on the copy engine, which is the whole reason the engine exists.
And do not conclude anything about one path from the other. §15 case 1 is the standard case: doorbells and register reads work perfectly while large copies stall. That is not a contradiction — it is the expected signature of a data-path fault, and it eliminates the Link, the BAR decode and the address routing in one observation.
4. x16, and What a Width Actually Tells You
The registry's purpose for this chapter is "show GPUs as x16 PCIe endpoints", and x16 is used here as the canonical teaching example — a discrete GPU on a x16 slot.
Three qualifications, all of which matter for reasoning:
Not all GPUs are x16. Integrated, mobile and small-form-factor parts differ. x16 is the teaching case, not a universal fact.
A x16 capability is not a x16 negotiated width. What a port can do and what it trained to are different values (18.1). A card capable of x16 in a slot wired for fewer lanes, or one that trained down after link errors, is running narrower. Read the negotiated width; never assume it from the connector, and §15 case 3 is that check.
And width does not predict delivered throughput. §12 measured a configuration where doubling the Link changed the result not at all, because the limit was elsewhere. Width bounds what the Link can carry. It says nothing about what the system will deliver, which is 22.1's central discipline.
This chapter publishes no generation, lane-count or bandwidth figure for any product (§1). Where §12 needs rates, they are normalized model parameters.
5. What a Doorbell Is
A doorbell is a small BAR write whose meaning is "queue state has advanced". It does not carry the work.
host: writes command descriptors into a queue in memory
host: writes the new producer pointer to a doorbell register ← a few bytes
GPU: reads the queue entries from memory ← the actual workTwo consequences, and §13 measured both.
The command must be visible before the doorbell announces it. This is 25.6 §5's ordering window in a new setting: if the pointer advances before the descriptor lands, the engine can take ownership of a slot whose contents have not been written. §13 measured 126 empty fetches — commands taken from ring slots that were still blank.
And the doorbell is not one command. A single doorbell write may announce one new entry or many; the pointer says how far the queue has advanced. Treating every doorbell write as exactly one descriptor is mutation 8, and it produces a design that silently drops work whenever the host batches submissions.
The exact doorbell register layout, queue entry format and pointer semantics are vendor-specific and are not reproduced here (§1). What is architectural — a pointer write announcing memory-resident work — is what §7's RTL models.
6. The Block Diagram
Two readings.
The doorbell edge is thin and the bulk-data edge is thick. They cross the same Link. The control edge carries a handful of bytes per submission; the data edge carries the transfer.
GPU local memory is behind its own controller, on the far side of the copy engine. It is a different resource from the Link, and §12 is the measurement that treating them as interchangeable produces wrong predictions in both directions.
7. RTL — Command Ownership and the Copy Path
Block 1 — the package. COMPILE-TIME.
package gpu_pkg;
// §8: every DMA request states which address space it targets. Inferring
// it from truncated address bits is mutation 12 — the same fault 25.5 §5
// measured as a decoder that accepts an aliased address.
typedef enum logic [1:0] {
AS_HOST = 2'd0, // host memory, reached over PCIe
AS_LOCAL = 2'd1, // GPU local memory, behind the GPU memory controller
AS_REG = 2'd2 // register / MMIO space
} addr_space_e;
typedef enum logic [1:0] {
DIR_H2D = 2'd0, // host to device
DIR_D2H = 2'd1, // device to host
DIR_D2D = 2'd2 // device local to device local
} copy_dir_e;
function automatic int unsigned gw(input int unsigned n);
return (n <= 1) ? 1 : $clog2(n);
endfunction
// A command as the engine owns it: a SNAPSHOT, never a pointer into the
// host ring. §13 measured a live read producing 1,449 wrong contexts.
typedef struct packed {
logic [31:0] cmd_id; // unique while outstanding
copy_dir_e dir;
addr_space_e src_space;
addr_space_e dst_space;
logic [63:0] src_addr;
logic [63:0] dst_addr;
logic [31:0] bytes;
} copy_cmd_t;
endpackageBlock 2 — doorbell capture. SYNTHESIZABLE. §5's mechanism.
Input owner: the BAR write path, for the cycle the write is accepted.
Output owner: this module, which holds the captured pointer until the queue manager consumes it.
Handshake: capture on an accepted BAR write; db_valid held until db_ready.
Stall: a second doorbell for the same queue while one is pending coalesces — the newest pointer wins, which is correct because a pointer is cumulative.
Reset: clears all pending doorbells; queued work is re-announced by the host.
Beyond this example: real designs have many more queues, per-queue enables, and often a doorbell aperture rather than a register file.
module gpu_doorbell_capture #(
parameter int unsigned NQUEUE = 8
)(
input logic clk,
input logic rst_n,
// accepted BAR write (decode owned by 23.2 / 25.5)
input logic bar_wr_valid,
input logic [gpu_pkg::gw(NQUEUE)-1:0] bar_wr_queue,
input logic [31:0] bar_wr_data, // new producer pointer
output logic bar_wr_ready,
// to the queue manager
output logic db_valid,
output logic [gpu_pkg::gw(NQUEUE)-1:0] db_queue,
output logic [31:0] db_pointer,
input logic db_ready,
output logic q_range_error
);
import gpu_pkg::*;
logic [NQUEUE-1:0] pending;
logic [31:0] ptr [NQUEUE];
logic [gw(NQUEUE)-1:0] sel;
// A doorbell write outside the configured queue range is REPORTED, never
// wrapped into a valid queue index (mutation 5). Wrapping turns a host
// software bug into silent execution on an unrelated queue.
assign q_range_error = bar_wr_valid && (bar_wr_queue >= gw(NQUEUE)'(NQUEUE));
assign bar_wr_ready = 1'b1; // a doorbell is never back-pressured
always_comb begin
sel = '0;
for (int i = NQUEUE-1; i >= 0; i--) if (pending[i]) sel = gw(NQUEUE)'(i);
db_valid = (pending != '0);
db_queue = sel;
db_pointer = ptr[sel];
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
pending <= '0;
for (int i = 0; i < NQUEUE; i++) ptr[i] <= '0;
end else begin
// The pointer is CUMULATIVE, so a newer doorbell supersedes an older
// one for the same queue. Treating each doorbell as one descriptor is
// mutation 8 and it drops work whenever the host batches.
if (bar_wr_valid && !q_range_error) begin
ptr[bar_wr_queue] <= bar_wr_data;
pending[bar_wr_queue] <= 1'b1;
end
if (db_valid && db_ready) begin
// Clear only if no newer doorbell landed for this queue this cycle.
if (!(bar_wr_valid && !q_range_error && (bar_wr_queue == sel)))
pending[sel] <= 1'b0;
end
end
end
endmoduleBlock 3 — the command snapshot. SYNTHESIZABLE. The module §13's live_queue_read mutation attacks.
Input owner: host memory, until the fetch returns. Output owner: this module; the snapshot is immutable for the command's lifetime. Beyond this example: real command processors parse variable-length packets and handle indirect buffers; this one takes a fixed record.
module gpu_cmd_snapshot #(
parameter int unsigned NENG = 4
)(
input logic clk,
input logic rst_n,
input logic fetch_valid, // command data returned from host memory
input logic [31:0] fetch_cmd_id,
input gpu_pkg::copy_dir_e fetch_dir,
input gpu_pkg::addr_space_e fetch_src_space, fetch_dst_space,
input logic [63:0] fetch_src, fetch_dst,
input logic [31:0] fetch_bytes,
output logic fetch_ready,
input logic [gpu_pkg::gw(NENG)-1:0] target_eng,
output gpu_pkg::copy_cmd_t cmd_out [NENG],
output logic [NENG-1:0] cmd_valid,
input logic [NENG-1:0] cmd_done
);
import gpu_pkg::*;
// The command is COPIED into engine-owned storage at the moment ownership
// transfers. Nothing downstream ever re-reads host memory for this command.
//
// §13 measured the alternative — holding a reference into the host ring and
// re-reading at completion — producing 1,449 commands that had changed
// underneath the engine.
assign fetch_ready = !cmd_valid[target_eng];
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
cmd_valid <= '0;
for (int i = 0; i < NENG; i++) cmd_out[i] <= '0;
end else begin
if (fetch_valid && fetch_ready) begin
cmd_out[target_eng] <= '{cmd_id: fetch_cmd_id,
dir: fetch_dir,
src_space: fetch_src_space,
dst_space: fetch_dst_space,
src_addr: fetch_src,
dst_addr: fetch_dst,
bytes: fetch_bytes};
cmd_valid[target_eng] <= 1'b1;
end
for (int i = 0; i < NENG; i++)
if (cmd_done[i]) cmd_valid[i] <= 1'b0;
end
end
endmoduleBlock 4 — the copy-engine request arbiter. SYNTHESIZABLE. Arbitration with explicit fairness and reserve-before-issue.
Input owner: each engine owns its request until granted. Output owner: the PCIe request path. Fairness: explicit round-robin, stated rather than emergent. Beyond this example: real arbiters weight by traffic class and priority; this one shows the ownership and starvation properties only.
module gpu_copy_arbiter #(
parameter int unsigned NENG = 4
)(
input logic clk,
input logic rst_n,
input logic [NENG-1:0] req_valid,
input gpu_pkg::copy_cmd_t req_cmd [NENG],
input logic [NENG-1:0] req_is_read,
output logic [NENG-1:0] req_grant,
input logic tag_available, // 23.5's allocator
input logic credit_available,
input logic cpl_path_ready, // the return path
output logic out_valid,
output gpu_pkg::copy_cmd_t out_cmd,
output logic [gpu_pkg::gw(NENG)-1:0] out_eng,
input logic out_ready
);
import gpu_pkg::*;
logic [gw(NENG)-1:0] rr_ptr, sel;
logic any_req, sel_found;
always_comb begin
any_req = (req_valid != '0);
sel = rr_ptr; sel_found = 1'b0;
// Round robin starting from rr_ptr: the fairness policy is EXPLICIT.
// A fixed-priority arbiter here starves the highest index under load,
// which is mutation 17 and appears as "one engine never progresses".
for (int k = 0; k < NENG; k++) begin
automatic logic [gw(NENG)-1:0] i = gw(NENG)'((rr_ptr + k) % NENG);
if (!sel_found && req_valid[i]) begin sel = i; sel_found = 1'b1; end
end
// A read is issued only if the Completion path can accept its reply.
// This is 23.6 §4's reserve-before-issue: availability is not permission,
// and 25.8 §7 measured what an unreserved issue does to a blocked drain.
out_valid = sel_found && tag_available && credit_available &&
(!req_is_read[sel] || cpl_path_ready);
out_cmd = req_cmd[sel];
out_eng = sel;
req_grant = '0;
if (out_valid && out_ready) req_grant[sel] = 1'b1;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) rr_ptr <= '0;
else if (out_valid && out_ready)
rr_ptr <= (sel == gw(NENG)'(NENG-1)) ? '0 : sel + 1'b1;
end
endmoduleBlock 5 — the address-space tagged request. SYNTHESIZABLE. §8's rule.
module gpu_space_route (
input logic req_valid,
input gpu_pkg::addr_space_e req_space,
input logic [63:0] req_addr,
output logic to_pcie_valid,
output logic to_local_valid,
output logic to_reg_valid,
output logic space_error
);
import gpu_pkg::*;
// The address space is carried as EXPLICIT metadata alongside the request.
// It is never inferred from address bits: host and local address spaces may
// overlap numerically, and a design that guesses from bit 63 or from a
// truncated compare produces a host read that silently reads local memory
// (mutation 12 — 25.5 §5's aliasing fault in a new place).
always_comb begin
to_pcie_valid = req_valid && (req_space == AS_HOST);
to_local_valid = req_valid && (req_space == AS_LOCAL);
to_reg_valid = req_valid && (req_space == AS_REG);
space_error = req_valid && !(to_pcie_valid || to_local_valid || to_reg_valid);
end
endmoduleBlock 6 — the completion record, then the event. SYNTHESIZABLE. §11's ordering, and the module §13's event_before_status mutation attacks.
module gpu_copy_complete #(
parameter int unsigned NENG = 4
)(
input logic clk,
input logic rst_n,
input logic [NENG-1:0] eng_last_beat_done,
input logic [31:0] eng_cmd_id [NENG],
// status write toward host memory
output logic status_valid,
output logic [31:0] status_cmd_id,
input logic status_ready,
input logic status_accepted, // the write has been accepted
// event request, strictly afterwards
output logic event_valid,
output logic [31:0] event_cmd_id,
input logic event_ready
);
import gpu_pkg::*;
typedef enum logic [1:0] {S_IDLE, S_STATUS, S_EVENT} st_e;
st_e st;
logic [31:0] cur_id;
// The order is: durable status record FIRST, notification SECOND.
// 19.x owns the interrupt mechanism; what this module owns is that the
// event cannot be raised until the status write has been ACCEPTED.
// §13 measured the inverted order raising 4,422 events ahead of their
// status — every one an opportunity for the host to read a stale record.
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
st <= S_IDLE; cur_id <= '0;
end else begin
unique case (st)
S_IDLE: for (int i = 0; i < NENG; i++)
if (eng_last_beat_done[i]) begin
cur_id <= eng_cmd_id[i]; st <= S_STATUS;
end
S_STATUS: if (status_valid && status_ready && status_accepted) st <= S_EVENT;
S_EVENT: if (event_valid && event_ready) st <= S_IDLE;
default: st <= S_IDLE;
endcase
end
end
assign status_valid = (st == S_STATUS);
assign status_cmd_id = cur_id;
assign event_valid = (st == S_EVENT);
assign event_cmd_id = cur_id;
endmoduleBlock 7 — copy-engine performance counters. VERIFICATION-ONLY. §12's reasoning, made measurable per stage.
module gpu_copy_counters (
input logic clk,
input logic rst_n,
input logic pcie_beat, input logic [15:0] pcie_bytes,
input logic local_beat, input logic [15:0] local_bytes,
input logic stall_pcie, // engine held for Link credit or Tags
input logic stall_local, // engine held by the GPU memory controller
input logic stall_cmd, // engine idle for lack of work
input logic clear,
output logic [63:0] c_pcie_bytes, c_local_bytes,
output logic [31:0] c_stall_pcie, c_stall_local, c_stall_cmd
);
// The three stall counters are separate because §12's whole result is that
// "the copy is slow" is not a diagnosis. Whether the engine is waiting on
// the Link, on local memory, or for work to arrive determines which change
// is worth making — and §12 measured that improving the wrong stage
// produced exactly zero improvement.
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n || clear) begin
c_pcie_bytes <= '0; c_local_bytes <= '0;
c_stall_pcie <= '0; c_stall_local <= '0; c_stall_cmd <= '0;
end else begin
if (pcie_beat && c_pcie_bytes != '1) c_pcie_bytes <= c_pcie_bytes + 64'(pcie_bytes);
if (local_beat && c_local_bytes != '1) c_local_bytes <= c_local_bytes + 64'(local_bytes);
if (stall_pcie && c_stall_pcie != '1) c_stall_pcie <= c_stall_pcie + 1'b1;
if (stall_local && c_stall_local != '1) c_stall_local <= c_stall_local + 1'b1;
if (stall_cmd && c_stall_cmd != '1) c_stall_cmd <= c_stall_cmd + 1'b1;
end
end
endmodule8. Same-Cycle Audit
9. Assertions
Doorbell properties — §5.
// P1 — a doorbell is captured on an accepted BAR write, and only then.
property p1_doorbell_on_accepted_write;
@(posedge clk) disable iff (!rst_n)
(bar_wr_valid && bar_wr_ready && !q_range_error) |=> pending[$past(bar_wr_queue)];
endproperty
a_p1: assert property (p1_doorbell_on_accepted_write);
// P2 — an out-of-range queue index is reported, never wrapped.
property p2_queue_range_checked;
@(posedge clk) disable iff (!rst_n)
(bar_wr_valid && (bar_wr_queue >= NQUEUE)) |-> q_range_error;
endproperty
a_p2: assert property (p2_queue_range_checked);
// P3 — a newer doorbell for the same queue supersedes the older pointer and
// is never lost to a concurrent accept.
property p3_doorbell_not_lost;
@(posedge clk) disable iff (!rst_n)
(db_valid && db_ready && bar_wr_valid && (bar_wr_queue == db_queue))
|=> pending[$past(db_queue)];
endproperty
a_p3: assert property (p3_doorbell_not_lost);
// P4 — the captured pointer is stable while it waits to be consumed.
property p4_pointer_stable;
@(posedge clk) disable iff (!rst_n)
(db_valid && !db_ready && !bar_wr_valid) |=> (db_valid && $stable(db_pointer));
endproperty
a_p4: assert property (p4_pointer_stable);Command ownership properties — §7 Block 3.
// P5 — a command is snapshotted at ownership; the record never changes.
// §13 measured a live read producing 1,449 wrong contexts.
property p5_command_immutable;
@(posedge clk) disable iff (!rst_n)
(cmd_valid[e] && !cmd_done[e]) |=> (cmd_valid[e] && $stable(cmd_out[e]));
endproperty
a_p5: assert property (p5_command_immutable);
// P6 — an engine owns at most one command at a time.
property p6_one_command_per_engine;
@(posedge clk) disable iff (!rst_n)
(fetch_valid && fetch_ready) |-> !cmd_valid[target_eng];
endproperty
a_p6: assert property (p6_one_command_per_engine);
// P7 — a command id is unique among concurrently owned commands.
property p7_cmd_id_unique;
@(posedge clk) disable iff (!rst_n)
(cmd_valid[i] && cmd_valid[j] && (i != j)) |-> (cmd_out[i].cmd_id != cmd_out[j].cmd_id);
endproperty
a_p7: assert property (p7_cmd_id_unique);
// P8 — a zero-byte command moves no beats. The degenerate parameter case.
property p8_zero_bytes_no_beats;
@(posedge clk) disable iff (!rst_n)
(cmd_valid[e] && (cmd_out[e].bytes == 32'd0)) |-> !pcie_beat;
endproperty
a_p8: assert property (p8_zero_bytes_no_beats);Address-space properties — §8.
// P9 — the address space tag is explicit and total.
property p9_space_total;
@(posedge clk) disable iff (!rst_n)
req_valid |-> $onehot({to_pcie_valid, to_local_valid, to_reg_valid});
endproperty
a_p9: assert property (p9_space_total);
// P10 — the tag is immutable for a command's lifetime.
property p10_space_immutable;
@(posedge clk) disable iff (!rst_n)
(cmd_valid[e] && !cmd_done[e]) |=> ($stable(cmd_out[e].src_space) &&
$stable(cmd_out[e].dst_space));
endproperty
a_p10: assert property (p10_space_immutable);
// P11 — a host-space request goes to PCIe and nowhere else.
property p11_host_goes_to_pcie;
@(posedge clk) disable iff (!rst_n)
(req_valid && (req_space == AS_HOST)) |-> (to_pcie_valid && !to_local_valid);
endproperty
a_p11: assert property (p11_host_goes_to_pcie);
// P12 — an unrecognised space is reported, never defaulted.
property p12_space_error_reported;
@(posedge clk) disable iff (!rst_n)
(req_valid && !(req_space inside {AS_HOST, AS_LOCAL, AS_REG})) |-> space_error;
endproperty
a_p12: assert property (p12_space_error_reported);Arbitration properties — §7 Block 4.
// P13 — at most one engine is granted per cycle.
property p13_grant_onehot0;
@(posedge clk) disable iff (!rst_n)
$onehot0(req_grant);
endproperty
a_p13: assert property (p13_grant_onehot0);
// P14 — a grant goes only to an engine that requested.
property p14_grant_implies_request;
@(posedge clk) disable iff (!rst_n)
req_grant[e] |-> req_valid[e];
endproperty
a_p14: assert property (p14_grant_implies_request);
// P15 — the selected request is stable while the output stalls.
property p15_out_stable;
@(posedge clk) disable iff (!rst_n)
(out_valid && !out_ready) |=> (out_valid && $stable(out_cmd) && $stable(out_eng));
endproperty
a_p15: assert property (p15_out_stable);
// P16 — a read is issued only with completion-path space reserved.
// 23.6 §4's law; 25.8 §7 measured what an unreserved issue costs.
property p16_reserve_before_read;
@(posedge clk) disable iff (!rst_n)
(out_valid && req_is_read[out_eng]) |-> cpl_path_ready;
endproperty
a_p16: assert property (p16_reserve_before_read);
// P17 — no engine is starved: a persistent request is eventually granted.
property p17_no_starvation;
@(posedge clk) disable iff (!rst_n)
req_valid[e] |-> s_eventually req_grant[e];
endproperty
a_p17: assert property (p17_no_starvation);
// P18 — nothing is issued without a Tag and credit.
property p18_resources_before_issue;
@(posedge clk) disable iff (!rst_n)
out_valid |-> (tag_available && credit_available);
endproperty
a_p18: assert property (p18_resources_before_issue);Completion and event properties — §11.
// P19 — the event is never raised before the status write is accepted.
// §13 measured the inversion at 4,422 events ahead of their status.
property p19_event_after_status;
@(posedge clk) disable iff (!rst_n)
event_valid |-> $past(status_accepted);
endproperty
a_p19: assert property (p19_event_after_status);
// P20 — status and event carry the same command id.
property p20_ids_agree;
@(posedge clk) disable iff (!rst_n)
event_valid |-> (event_cmd_id == cur_id);
endproperty
a_p20: assert property (p20_ids_agree);
// P21 — exactly one status per command.
property p21_status_once;
@(posedge clk) disable iff (!rst_n)
(status_valid && status_ready && status_accepted) |=> (st != S_STATUS);
endproperty
a_p21: assert property (p21_status_once);
// P22 — cur_id is not overwritten mid-sequence (§8 audit B).
property p22_cur_id_stable;
@(posedge clk) disable iff (!rst_n)
(st != S_IDLE) |=> $stable(cur_id) or (st == S_IDLE);
endproperty
a_p22: assert property (p22_cur_id_stable);
// P23 — reset clears every owned command and every pending event.
property p23_reset_clears;
@(posedge clk)
!rst_n |=> ((cmd_valid == '0) && (st == S_IDLE) && (pending == '0));
endproperty
a_p23: assert property (p23_reset_clears);Cover — the anti-vacuity set.
// P24 — a command leaves an engine exactly once, by completion. An engine
// that can drop a command without completing it loses work with no record,
// which is the one outcome the status/notification ordering cannot detect.
property p24_command_exits_once;
@(posedge clk) disable iff (!rst_n)
(cmd_valid[e] && cmd_done[e]) |=> !cmd_valid[e];
endproperty
a_p24: assert property (p24_command_exits_once);
// P24's covers — the exceptional states must occur, or P2, P3, P12, P16 and P19 are
// satisfied without ever evaluating.
c1_range_error: cover property (@(posedge clk) disable iff (!rst_n) q_range_error);
c2_db_collision: cover property (@(posedge clk) disable iff (!rst_n)
db_valid && db_ready && bar_wr_valid && (bar_wr_queue == db_queue));
c3_space_error: cover property (@(posedge clk) disable iff (!rst_n) space_error);
c4_cpl_blocked: cover property (@(posedge clk) disable iff (!rst_n)
req_valid[out_eng] && req_is_read[out_eng] && !cpl_path_ready);
c5_all_engines: cover property (@(posedge clk) disable iff (!rst_n) req_valid == '1);
c6_zero_bytes: cover property (@(posedge clk) disable iff (!rst_n)
fetch_valid && (fetch_bytes == 32'd0));
c7_status_stall: cover property (@(posedge clk) disable iff (!rst_n)
status_valid && !status_ready);10. Executable Counterexamples
Counterexample A — the live ring reference (violates P5).
// The engine keeps a pointer into the host command ring and re-reads it at
// completion instead of snapshotting the command at ownership.
module ce_a_live_ring_ref (
input logic clk, rst_n,
input logic take, input logic [31:0] ring_index,
input logic [31:0] host_ring_cmd_id, // reflects host memory, live
output logic [31:0] completed_cmd_id
);
logic [31:0] idx;
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) idx <= '0;
else if (take) idx <= ring_index;
assign completed_cmd_id = host_ring_cmd_id; // <-- re-read at completion
endmodule
// Failing stimulus: the engine takes the command at ring index 5, and the host
// wraps the ring and writes a new command into index 5 while the copy runs.
// Golden: the completion reports the command the engine actually executed.
// This: it reports whatever occupies index 5 NOW.
// P5 fails.
// §13 measured 1,449 commands whose ring entry had changed by completion.
// Observable consequence: a completion status written against the wrong
// command id, so the host retires a command that never ran.Counterexample B — the event that precedes its status (violates P19).
// Notification is raised in the same cycle the last data beat completes,
// without waiting for the status record to be accepted.
module ce_b_event_before_status (
input logic last_beat_done,
output logic event_valid, status_valid
);
assign status_valid = last_beat_done;
assign event_valid = last_beat_done; // <-- concurrent, not sequenced
endmodule
// Failing stimulus: the status write is back-pressured for several cycles
// while the interrupt message is delivered immediately.
// Golden: the event waits for status_accepted.
// This: the host is interrupted, reads the status record, and finds the
// PREVIOUS command's result.
// P19 fails, P21 fails.
// §13 measured 4,422 events raised ahead of their status. This is 19.x's
// rule — a notification is not the authoritative completion state — and the
// observable consequence is a host that acts on a stale record every time
// the status write is slower than the interrupt path.11. Completion, Status, and Notification Are Three Things
A GPU copy involves three distinct "done" concepts and conflating them is mutation 20.
| term | what it is | who owns it |
|---|---|---|
| PCIe Completion | the TLP answering a Non-Posted request | 10.2 |
| copy completion | the engine's last beat has moved | this chapter, §7 Block 6 |
| status record | a durable, host-visible result written to memory | this chapter |
| notification | MSI/MSI-X telling the host to look | 19.2, 19.3 |
The required order is status first, notification second, and the reason is that the notification carries no information. It says "something finished"; the host then reads the status record to learn what. If the notification can overtake the record, the host reads a stale result — and §13 measured 4,422 opportunities for exactly that in a single run.
A GPU's Memory Writes to host memory are Posted and receive no PCIe Completion (12.2). So the engine cannot learn from the Link that its data landed; the status write and the ordering rules that govern it (13.4) are the entire mechanism. This is the same structure 25.6 §7 established for DMA generally, and it is why P19 is written against status_accepted rather than against status_valid.
12. Measured Behaviour — The Bottleneck Is Not the Link
| configuration | PCIe | GPU-local | host | steps |
|---|---|---|---|---|
| baseline — PCIe is the limit | 1000 | 4000 | 2000 | 67,109 |
| PCIe doubled | 2000 | 4000 | 2000 | 33,555 |
| PCIe doubled, host now the limit | 2000 | 4000 | 1500 | 44,740 |
| GPU-local halved | 1000 | 500 | 2000 | 134,218 |
| GPU-local doubled | 1000 | 8000 | 2000 | 67,109 — unchanged |
Four readings.
Doubling the Link halved the time — when the Link was the limit. That is the only condition under which a Link upgrade helps, and it is not knowable without measuring the other stages.
Doubling GPU-local bandwidth changed the result by exactly zero. The stage was never the constraint. An engineer who improves the wrong stage gets no improvement and no error message — the design simply performs identically, which is the least informative possible outcome.
Halving local memory made the same Link deliver four times worse. Nothing about PCIe changed: same width, same generation, same negotiated state. This is why a bandwidth complaint cannot be attributed to PCIe without instrumenting the other stages (§7 Block 7's three separate stall counters).
And row 3 shows the limit moving. After doubling PCIe, the host stage became the constraint — so a second doubling would again yield nothing. The bottleneck is a property of the configuration, not of the design, and it moves as stages change.
13. Measured Behaviour — Command Ownership
| configuration | submitted | started | completed | empty fetch | wrong context | event before status |
|---|---|---|---|---|---|---|
| correct (snapshot at ownership) | 4,548 | 4,424 | 4,422 | 0 | 0 | 0 |
| doorbell rung before the command lands | 4,687 | 4,437 | 4,435 | 126 | 0 | 0 |
| engine reads the live ring entry | 6,568 | 6,444 | 6,442 | 0 | 1,449 | 0 |
| event raised before status accepted | 4,548 | 4,424 | 4,422 | 0 | 0 | 4,422 |
Three readings.
The baseline is clean on all three fault metrics.
126 empty fetches are engines taking ownership of a ring slot whose command word had not yet landed — the doorbell announced work that did not exist (§5).
1,449 wrong contexts are commands whose ring entry had changed by the time the engine re-read it. The engine executed one command and reported another, and no PCIe-level instrument sees anything wrong: the transfers were well-formed and completed normally.
14. Verification — Mutations
Twenty-eight mutations. Every "Caught by" entry names a property from §9.
| # | Mutation | Symptom | Caught by |
|---|---|---|---|
| 1 | Doorbell captured on an unaccepted BAR write | phantom work announced | P1 |
| 2 | Doorbell pointer sampled a cycle late | announces the previous pointer | P4 |
| 3 | Pending cleared unconditionally on accept | a doorbell is lost (§8 audit A) | P3 |
| 4 | Doorbell treated as level, not as a pointer update | repeated refetch of the same work | P4 |
| 5 | Queue index wrapped instead of range-checked | work executed on an unrelated queue | P2 |
| 6 | Range error computed but not driven out | a host bug becomes silent execution | P2 |
| 7 | Doorbell back-pressured | host stalls on a control write | P1 |
| 8 | Each doorbell treated as exactly one descriptor | work dropped whenever the host batches | P4 |
| 9 | Command read live from the host ring | 1,449 wrong contexts (§13) | P5 |
| 10 | Engine allowed two concurrent commands | the second overwrites the first | P6 |
| 11 | Command ids reused while outstanding | completions match the wrong command | P7 |
| 12 | Address space inferred from address bits | a host read silently reads local memory | P9, P11 |
| 13 | Address space tag mutable after ownership | destination changes mid-transfer | P10 |
| 14 | Unknown space defaulted to host | traffic sent over PCIe by accident | P12 |
| 15 | Zero-byte command emits one beat | a write for a command describing none | P8 |
| 16 | Arbiter grants two engines in one cycle | two commands share one Tag | P13 |
| 17 | Fixed priority replaces round robin | the highest index never progresses | P17 |
| 18 | Grant issued to a non-requesting engine | an engine owns work it did not ask for | P14 |
| 19 | Selected request changes while output stalls | the issued TLP does not match the grant | P15 |
| 20 | Read issued without completion-path space | an unbounded outstanding queue (25.8 §7) | P16 |
| 21 | Issue permitted without a Tag | Tag reuse under load | P18 |
| 22 | Event raised with the status write | 4,422 events ahead of status (§13) | P19 |
| 23 | Event id taken from the newest completion | notification names the wrong command | P20, P22 |
| 24 | Status written twice per command | the host retires a command twice | P21 |
| 25 | cur_id resampled in every state | the event names a different command | P22 |
| 26 | Reset leaves commands owned | stale work resumes after reset | P23 |
| 27 | Testbench never collides a doorbell with an accept | P3 vacuous | P24 (c2) |
| 28 | Testbench never stalls the status write | the event ordering path is unreachable | P24 (c7) |
15. Debugging
16. Misconceptions
"PCIe is the GPU's memory bus." It is the host-facing transport (§2). GPU local memory is behind its own controller, and §12 measured the two behaving as completely independent resources.
"x16 capability means x16 negotiated." Capability is what a port can do; the negotiated width is what it did (§4). Only the second belongs in a bandwidth calculation.
"A x16 GPU will deliver x16 bandwidth." §12's baseline was PCIe-limited; one row up, doubling the Link helped, and one row down, halving local memory made the same Link four times slower. Width bounds; it does not predict.
"If the copy is slow, upgrade the Link." §12 measured a configuration where doubling PCIe changed the time by exactly zero, because local memory was the limit. Instrument the stages first (§7 Block 7).
"MMIO works, so data movement will work." They share a Link and almost nothing else (§3). Working control is the expected state during a data-path failure (§15 case 1).
"Move the data with MMIO writes." Each is host-initiated and stalls a core for a round trip (§3). Bulk movement is what the copy engine exists for.
"A doorbell contains the command." It carries a pointer; the work lives in memory (§5). Treating each doorbell as one descriptor drops work whenever the host batches.
"The doorbell means the command is ready." Only if the command was made visible first (§5). §13 measured 126 empty fetches when the pointer advanced ahead of the descriptor.
"Reading the ring entry again at completion is harmless." §13 measured 1,449 commands that had changed by then (counterexample A). The engine executed one command and reported another.
"An interrupt means the result is ready." It means something finished; the status record is the result (§11). §13 measured 4,422 events raised before their record was accepted.
"A copy engine's Memory Writes get Completions." Writes are Posted and receive none (12.2, §11). The status write is the only completion mechanism the engine has.
"GPU local memory bandwidth and PCIe bandwidth are comparable numbers." They are different resources measured in different places under different workloads. §12's model keeps them as separate stages precisely so that no single number stands for both, and 22.1 owns the discipline for comparing any two rates at all.
"More copy engines means more throughput." Only until a shared downstream resource saturates (§15 case 4), and a fixed-priority arbiter can make additional engines actively harmful (mutation 17).
"The GPU is mostly a completer — the host drives it." By byte volume the GPU originates nearly all the traffic (§2). Its PCIe complexity is requester-side.
17. Understanding Check
Q1. BAR writes and doorbells work perfectly, and large host→GPU copies stall while GPU-local compute is unaffected. Why does this separate control-path health from data-path health?
Because the two paths share only the Link (§3). Working doorbells prove the Link is trained, the BAR window decodes, and the host's address routing reaches this device — 26.1 §3's classification is correct. They prove nothing about the copy engines, which are separately resourced: they need Tags, completion-buffer space and posted credit, none of which a doorbell write consumes. That unaffected local compute further localises it — the GPU's memory system is working, so the fault is in the movement path between the Link and local memory (§15 case 1).
Q2. Your GPU copy is slower than expected. Why is "upgrade to a wider link" a guess rather than a fix?
Because a transfer crosses several stages and only the slowest one matters (§12). The model measured doubling PCIe producing a 2× improvement when PCIe was the limit and exactly zero improvement when GPU-local memory was — same change, same Link, opposite results. Worse, improving the wrong stage produces no error and no signal, just identical performance. The three separate stall counters (§7 Block 7) are what turn this into a measurement: whether the engine is waiting on the Link, on local memory, or for work decides which change is worth making.
Q3. A doorbell announces work, and the engine finds an empty ring slot. What ordering rule was broken and where does it belong?
The command must be visible before the pointer that announces it (§5). The host wrote the doorbell before the descriptor landed, so the engine took ownership of a slot that was still blank — §13 measured 126 such fetches. The obligation is the producer's, exactly as 25.6 §5 established for descriptor rings: PCIe's ordering rules govern transactions on the Link and do not substitute for the host writing its own structures in the right order.
Q4. Why must a copy engine snapshot its command rather than keep a reference to the host ring entry?
Because the host owns that ring slot again as soon as the pointer passes it (§7 Block 3, counterexample A). Holding a reference and re-reading at completion means the engine reports whatever occupies the slot then — §13 measured 1,449 commands that had changed. The engine executes one command and reports another, and no PCIe-level instrument sees anything wrong: every transfer was well-formed and completed normally. P5 asserts the snapshot is immutable for the command's lifetime.
Q5. Explain why a notification must not overtake its status record, and what the host observes when it does.
Because the notification carries no information (§11). MSI/MSI-X says "something finished"; the host then reads the status record in memory to learn what finished and whether it succeeded. If the event arrives first, the host reads the previous command's result and acts on it. §13 measured 4,422 events raised ahead of their status in one run. The rule is durable status first, notification second (P19), and it matters because a GPU's Memory Writes are Posted — the status write is the engine's only completion mechanism (12.2).
Q6. Host→device copies work and device→host copies fail. What does the direction tell you?
That the fault is in a resource only one direction consumes (§15 case 2). Host→device is dominated by the engine's Memory Reads of host memory — Non-Posted, consuming Tags and requiring completion-buffer space for the replies (23.5). Device→host is dominated by Memory Writes — Posted, consuming credit but no Tags and receiving no Completions. A direction-specific failure is therefore a resource question, not a Link question, and the Link being healthy is consistent with both.
Q7. Why does §7's arbiter refuse to issue a read when the completion path is not ready, even though a Tag and credit are available?
Because availability is not permission (23.6 §4, P16). Issuing a read commits the completer to returning data that needs space at this device; having a Tag and Link credit says nothing about whether that space exists. 25.8 §7 measured the consequence precisely: an unreserved issue alone does not deadlock — it produces an unbounded queue — but it removes the bound that keeps the queue finite, so any dependency that later blocks the drain becomes permanent instead of temporary.
18. What Comes Next
Three chapters remain, and each one changes what the device fetches for itself.
| Chapter | The device, and what it owns |
|---|---|
| 26.1 CPUs | the host — address ownership and translation |
| 26.2 (this) | a GPU — control path and data path as separate resources |
| 26.3 SSD Controllers | a device that fetches its own work from queues |
| 26.4 Network Adapters | a device driven by traffic it does not control |
| 26.5 FPGA Cards | a device whose DMA engine you write yourself |
One structure from this chapter recurs in all three. A doorbell announces memory-resident work; the device fetches it; the device takes ownership by snapshot; the device moves data; the device publishes a status record; the device notifies. 26.3 makes that structure explicit and standard-defined, and the difference is instructive: an SSD's queues are specified by NVM Express rather than by the vendor, which changes what a debugger can assume and what it must read.
And §12's result is the module's most transferable one. Every remaining chapter has a non-PCIe bottleneck waiting — flash media, network wire rate, on-card DDR — and in each case the same trap applies: improving the wrong stage produces no improvement and no error message.