PCIe · Module 26
PCIe in CPUs — The Root Complex as an Ownership Boundary
A CPU-integrated Root Complex is a translation and ownership boundary, not a port. Address classification measured against an independent oracle, the MMIO and DMA paths kept apart, and why a translation response needs its own identity.
Module 25 finished by asking what a fabric can and cannot show you. Module 26 turns outward — to the systems PCIe actually lives inside, starting with the one that hosts it.
A CPU-integrated Root Complex is a translation and ownership boundary between the processor/memory system and the PCIe hierarchy. It is not a port, not a bus controller, and not something the cores talk to directly. Everything in this chapter follows from taking that sentence literally.
1. Sources, Scope, and What This Chapter Will Not Claim
2. PCIe Is Not Connected to the Core
The mental picture that causes the most trouble is a line drawn from "CPU" to "PCIe". Replace it with a chain in which each link is a different agent with different obligations:
CPU cores
↕ virtual → physical translation happens HERE, in the core's MMU
cache / coherent interconnect
↕
SoC / I/O fabric ──────────────► memory controller ──► DRAM
↕
IOMMU / address translation (device-side translation, a different step)
↕
PCIe Root Complex
↕
Root Ports
↕
PCIe hierarchy — Endpoints, SwitchesThree consequences, and each one is a debugging rule.
A core's load/store does not "go to PCIe". It produces a physical address, and something downstream decides who owns that address — DRAM, an MMIO window, a control region, or nobody. §3 is that decision, and §13 measures what happens when it is written three plausible ways.
Device DMA need not involve a core at all. An Endpoint issuing a Memory Write toward host memory can reach the memory controller through the I/O and coherent fabric without any core executing an instruction. A stalled core does not stop DMA, and a busy DMA path does not appear in any core's execution profile.
And the two directions use different translation machinery. The core's MMU translates virtual to physical for CPU accesses. A device's I/O virtual address is translated by the IOMMU, on the device's behalf, using the device's identity. They are different translations of different address spaces performed by different hardware, and §5 is about why calling both of them "address translation" costs hours.
3. Address Ownership Is a Decision, Not a Property
When a physical address leaves the core side, exactly one agent must claim it. The classification is small, total, and the place where system bugs concentrate:
| class | meaning | consequence of getting it wrong |
|---|---|---|
| local memory | DRAM behind the memory controller | a device access answered by RAM |
| PCIe MMIO | a window routed to a Root Port | a memory access turned into a TLP |
| configuration / control | platform registers, config access | control writes land in the wrong place |
| error | nobody owns this address | a fault the platform must report |
The fourth class is the one designs get wrong, and §13 measured why: the tempting implementation is to test the classes you know and let everything else fall through to PCIe. That single decision produced 50,346 addresses in 200,000 that the platform does not own being turned into real PCIe transactions.
The rule is the same one 25.5 §10 established for BAR decoders and it is worth stating as a law:
Every range must be checked. An unclaimed address is an error, never a default.
And the classification must be at full width. §13's third row is a classifier comparing only the low 32 bits: it misrouted 39,648 MMIO accesses into DRAM — the device is never reached, the read succeeds, and it returns whatever the memory controller had at that offset. No error is raised at any layer, which makes it 25.5 §4's Direction B at platform scale.
4. The Two Paths, Kept Apart
These differ in every property that matters for debugging.
| MMIO | DMA | |
|---|---|---|
| initiated by | a core instruction | the device |
| typical size | 4–8 bytes | kilobytes to megabytes |
| what it costs | latency — the core waits | bandwidth — the fabric carries it |
| blocked by | anything in the round trip | host memory, translation, credits |
| visible in | a core's stall profile | fabric and memory counters |
| translation | core MMU | IOMMU, device identity |
The practical consequence is §16 case 2, and it is the most common system-debug misstep in this chapter: MMIO working proves almost nothing about DMA. They share the Link and share very little else. A device whose registers respond perfectly can be completely unable to move data, and the reverse is equally possible.
A second consequence is about performance advice. Using MMIO reads to move bulk data is a well-known mistake, and the reason is in the table: each one costs a full round trip during which the core is stalled. Bulk movement belongs on the DMA path (20.2), and 22.2 owns the measurement that distinguishes them.
5. The Translation Boundary
Where an IOMMU is present and enabled, the address a device puts in a TLP is not the address that reaches DRAM.
device-visible I/O virtual address (IOVA)
→ translation, using the device's identity
→ permission check
→ host physical addressThis chapter teaches the boundary and refuses to teach the mechanism. Page-table formats, translation-cache behaviour, invalidation protocols and register layouts are platform-specific and are not reproduced here (§1). What matters architecturally is that translation is a request/response transaction with all the properties that implies:
- it has latency, so the requester must hold state while it waits;
- it can apply backpressure, so the path in front of it must stall correctly;
- it can fail with a permission fault, and a fault must not produce a memory access;
- and its response must carry its own identity, because responses can return out of order.
That last property is 21.4 §3's law appearing again, and §14 measured what its absence costs: a design that pairs a translation response with whatever occupies the context slot at that moment misrouted 11,857 responses.
One debugging consequence deserves its own line, because it explains an entire class of "PCIe is broken" reports:
A PCIe address is not a DRAM location. Under translation, a device issuing a perfectly correct Memory Write to IOVA
Xmay be writing to any physical page — or to none, if the translation is absent or unmapped.
A device that works with translation disabled and fails with it enabled has not developed a PCIe fault. The Link is unchanged, the TLPs are unchanged, and the LTSSM is in L0 throughout. §16 case 3 is that case, and its resolution is never in the PCIe layer.
6. Root Complex and Root Port Are Not Synonyms
They are used interchangeably in conversation and they are different things (2.2, 4.1).
The Root Complex is the host-side PCIe architecture as a whole — the logical entity that connects the processor and memory subsystem to the PCIe hierarchy. It is where host-side transaction generation, address ownership and platform integration live.
A Root Port is a port on that Root Complex which originates a hierarchy branch. A Root Complex may have several; each one is the top of its own tree of switches and Endpoints.
The distinction is not pedantry, and three practical statements depend on it.
Devices under different Root Ports are in different hierarchy branches. Their traffic converges only inside the Root Complex and the fabric behind it (2.6). "Two devices on the same Root Complex" and "two devices on the same Root Port" describe very different topologies with different contention.
A Link problem is a Root Port problem. LTSSM state, negotiated width and speed, and training all belong to a port (18.1). "The Root Complex won't train" names the wrong agent and misdirects the investigation.
And address ownership is a Root Complex property. Which addresses are routed toward PCIe at all is decided before any port is selected — §3's classification comes first, then §9's port routing.
7. The Block Diagram
Read two things out of the figure.
The DMA arrow does not pass through the cores. It reaches the memory controller through the fabric. This is the structural reason a core profile shows nothing during heavy device DMA, and the reason a hung core does not stop a DMA engine.
Address ownership is decided in the fabric, before a Root Port is chosen. §3 happens at the fab node; §9's port routing happens at rc. A design that folds them together cannot express "this address belongs to no one", which is §13's first fault.
8. The Instruments, Named
| Instrument | Answers |
|---|---|
host_addr_classify | who owns this physical address? (§3) |
rc_port_route | which Root Port owns this MMIO address? |
rp_request_queue | holds a request stable under downstream backpressure |
dma_return_router | where does this inbound transaction go? |
xlate_req_if | the translation request/response boundary (§5) |
xlate_ctx_table | associates a translation response with its request |
rc_perf_counters | where is time actually going? |
rc_first_fault | which boundary failed first, sticky |
9. RTL — The Host Boundary
Block 1 — the package. COMPILE-TIME. Shared types, and the address-class enumeration that makes §3's classification a value rather than an inference.
package rc_pkg;
// §3's total classification. ERROR is a first-class outcome, never a default
// branch — §13 measured that letting unclaimed addresses fall through to
// MMIO produced 50,346 spurious PCIe transactions in 200,000 probes.
typedef enum logic [2:0] {
ACL_NONE = 3'd0,
ACL_DRAM = 3'd1,
ACL_MMIO = 3'd2,
ACL_CFG = 3'd3,
ACL_ERROR = 3'd4
} addr_class_e;
// Which host-side boundary rejected or failed first (§17's recorder).
typedef enum logic [2:0] {
RCF_NONE = 3'd0,
RCF_CLASSIFY = 3'd1, // no window owns the address
RCF_PORT = 3'd2, // no Root Port claims an MMIO address
RCF_XLATE = 3'd3, // translation faulted
RCF_CTX = 3'd4, // no free translation context
RCF_FABRIC = 3'd5 // downstream fabric refused
} rc_fault_e;
function automatic int unsigned gw(input int unsigned n);
return (n <= 1) ? 1 : $clog2(n); // §19: $clog2(1) is 0 in some tools
endfunction
// A window is [base, limit] INCLUSIVE. Naming the convention once, here,
// is what keeps 25.5 §6's off-by-one out of every comparator below.
typedef struct packed {
logic valid;
logic [63:0] base;
logic [63:0] limit; // last owned byte
} window_t;
function automatic logic in_window(input window_t w, input logic [63:0] a);
return w.valid && (a >= w.base) && (a <= w.limit);
endfunction
endpackageBlock 2 — the host address classifier. SYNTHESIZABLE. The direct implementation of §3.
Input owner: the SoC fabric presents a physical address and a request type.
Output owner: the classifier; its result is stable for the cycle the request is valid.
Handshake: combinational decode qualified by req_valid; no state.
Reset: none required — it is pure decode over registered windows.
Beyond this example: production silicon has many more windows, per-window attributes, and often a hardware-managed hole around reserved regions.
module host_addr_classify #(
parameter int unsigned NMMIO = 4 // MMIO windows the platform routes to PCIe
)(
input logic req_valid,
input logic [63:0] req_addr,
input rc_pkg::window_t dram_win,
input rc_pkg::window_t cfg_win,
input rc_pkg::window_t mmio_win [NMMIO],
output rc_pkg::addr_class_e acl,
output logic [rc_pkg::gw(NMMIO)-1:0] mmio_idx,
output logic ambiguous
);
import rc_pkg::*;
logic hit_dram, hit_cfg;
logic [NMMIO-1:0] hit_mmio;
logic [gw(NMMIO+1)-1:0] nhit;
always_comb begin
// Every window is range-checked at FULL width. §13's third row measured a
// low-32 comparator misrouting 39,648 MMIO accesses into DRAM — answered
// by memory, with no error raised anywhere.
hit_dram = in_window(dram_win, req_addr);
hit_cfg = in_window(cfg_win , req_addr);
for (int i = 0; i < NMMIO; i++) hit_mmio[i] = in_window(mmio_win[i], req_addr);
nhit = gw(NMMIO+1)'(hit_dram) + gw(NMMIO+1)'(hit_cfg);
for (int i = 0; i < NMMIO; i++) nhit += gw(NMMIO+1)'(hit_mmio[i]);
// More than one owner is a configuration error, reported rather than
// resolved — the three-outcome decoder of 23.6 §5 and 25.5 §3.
ambiguous = req_valid && (nhit > 1);
mmio_idx = '0;
for (int i = NMMIO-1; i >= 0; i--) if (hit_mmio[i]) mmio_idx = gw(NMMIO)'(i);
if (!req_valid) acl = ACL_NONE;
else if (ambiguous) acl = ACL_ERROR;
else if (hit_dram) acl = ACL_DRAM;
else if (hit_cfg) acl = ACL_CFG;
else if (|hit_mmio) acl = ACL_MMIO;
// The default is ERROR, not MMIO. This single line is the difference
// between §13's healthy row and its 50,346 spurious transactions.
else acl = ACL_ERROR;
end
endmoduleBlock 3 — Root Port routing. SYNTHESIZABLE. Runs only after §3 has said ACL_MMIO.
Input owner: the classifier. Output owner: this module. Concurrency: one request per cycle; multiple ports are selected, not shared. Beyond this example: real platforms route by more than address, and may apply per-port attributes and ACS-style controls.
module rc_port_route #(
parameter int unsigned NPORT = 4
)(
input logic req_valid,
input rc_pkg::addr_class_e acl,
input logic [63:0] req_addr,
input rc_pkg::window_t port_win [NPORT],
output logic route_valid,
output logic [rc_pkg::gw(NPORT)-1:0] port_idx,
output logic no_port,
output logic multi_port
);
import rc_pkg::*;
logic [NPORT-1:0] hit;
logic [gw(NPORT+1)-1:0] n;
always_comb begin
for (int i = 0; i < NPORT; i++) hit[i] = in_window(port_win[i], req_addr);
n = '0;
for (int i = 0; i < NPORT; i++) n += gw(NPORT+1)'(hit[i]);
// Port routing is attempted ONLY for addresses classified as MMIO.
// A design that routes before classifying cannot distinguish "no port
// owns this" from "the platform does not own this" (mutation 6).
no_port = req_valid && (acl == ACL_MMIO) && (n == 0);
multi_port = req_valid && (acl == ACL_MMIO) && (n > 1);
route_valid = req_valid && (acl == ACL_MMIO) && (n == 1);
port_idx = '0;
if (route_valid)
for (int i = NPORT-1; i >= 0; i--) if (hit[i]) port_idx = gw(NPORT)'(i);
end
endmoduleBlock 4 — the Root Port request queue. SYNTHESIZABLE. The ready/valid contract applied to the host boundary.
Input owner: the fabric, until enq_valid && enq_ready.
Output owner: this queue, until deq_valid && deq_ready.
Stall behaviour: a captured request is immutable; it is never re-derived from live fabric signals.
Reset: synchronous; empties the queue and drops ownership of everything held.
module rp_request_queue #(
parameter int unsigned DEPTH = 8
)(
input logic clk,
input logic rst_n,
input logic enq_valid,
output logic enq_ready,
input logic [2:0] enq_type,
input logic [63:0] enq_addr,
input logic [11:0] enq_len,
input logic [15:0] enq_rid, // requester context
output logic deq_valid,
input logic deq_ready,
output logic [2:0] deq_type,
output logic [63:0] deq_addr,
output logic [11:0] deq_len,
output logic [15:0] deq_rid,
output logic [rc_pkg::gw(DEPTH+1)-1:0] level
);
import rc_pkg::*;
typedef struct packed {
logic [2:0] rtype;
logic [63:0] addr;
logic [11:0] len;
logic [15:0] rid;
} req_t;
req_t mem [DEPTH];
logic [gw(DEPTH)-1:0] wr, rd;
assign enq_ready = (level != gw(DEPTH+1)'(DEPTH));
assign deq_valid = (level != '0);
// The output is driven from STORED state, never from the input port.
// Re-deriving it from live fabric signals is mutation 9, and its symptom
// is a request whose address changes while it waits for a busy Root Port.
assign deq_type = mem[rd].rtype;
assign deq_addr = mem[rd].addr;
assign deq_len = mem[rd].len;
assign deq_rid = mem[rd].rid;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
wr <= '0; rd <= '0; level <= '0;
end else begin
// Transfer is valid && ready. Counting `valid` cycles as transfers is
// the forbidden reading of the transfer contract, and mutation 10.
if (enq_valid && enq_ready) begin
mem[wr] <= '{rtype: enq_type, addr: enq_addr, len: enq_len, rid: enq_rid};
wr <= (wr == gw(DEPTH)'(DEPTH-1)) ? '0 : wr + 1'b1;
end
if (deq_valid && deq_ready)
rd <= (rd == gw(DEPTH)'(DEPTH-1)) ? '0 : rd + 1'b1;
case ({enq_valid && enq_ready, deq_valid && deq_ready})
2'b10: level <= level + 1'b1;
2'b01: level <= level - 1'b1;
default: ; // both or neither: level unchanged
endcase
end
end
endmoduleBlock 5 — the translation request interface. CONCEPTUAL boundary, SYNTHESIZABLE shell. §5's boundary with no page walking.
Input owner: the Root Port side, until req_valid && req_ready.
Output owner: the translation agent, which returns exactly one response per accepted request.
Beyond this example: real translation involves caches, invalidation, multi-level walks and platform-specific fault reporting — none of which is modelled or claimed here (§1).
module xlate_req_if #(
parameter int unsigned NCTX = 16
)(
input logic clk,
input logic rst_n,
// request side
input logic req_valid,
output logic req_ready,
input logic [15:0] req_devid, // requester identity
input logic [63:0] req_iova,
input logic req_is_write,
input logic [11:0] req_len,
input logic [rc_pkg::gw(NCTX)-1:0] req_ctx,
// response side
output logic rsp_valid,
input logic rsp_ready,
output logic [rc_pkg::gw(NCTX)-1:0] rsp_ctx,
output logic [63:0] rsp_paddr,
output logic rsp_permit,
output logic rsp_fault
);
// This module deliberately implements NO page walk. It exists to make four
// properties explicit and assertable: translation has latency, it applies
// backpressure, it can fault, and its response carries a context (§5).
//
// A faulting translation must NOT produce a memory transaction. §14
// measured a design that forwarded anyway: 2,103 accesses proceeded on a
// translation that had been denied.
assign req_ready = 1'b1; // placeholder for the agent's own flow control
endmoduleBlock 6 — the outstanding translation table. SYNTHESIZABLE. §5's identity requirement, and the module §14 measures.
Lifetime: a context is owned from allocation until its response is consumed or it is explicitly cancelled. Beyond this example: production designs add per-device queues and fairness; this one is a single pool.
module xlate_ctx_table #(
parameter int unsigned NCTX = 16,
parameter int unsigned CANCEL_LIMIT = 32'd4096 // IMPLEMENTATION POLICY
)(
input logic clk,
input logic rst_n,
input logic alloc_req,
output logic alloc_gnt,
output logic [rc_pkg::gw(NCTX)-1:0] alloc_ctx,
output logic [15:0] alloc_gen, // generation, §14
input logic [63:0] alloc_iova,
input logic rsp_valid,
input logic [rc_pkg::gw(NCTX)-1:0] rsp_ctx,
input logic [15:0] rsp_gen,
input logic rsp_fault,
output logic rsp_accept,
output logic rsp_stale,
output logic [31:0] stale_count,
output logic [rc_pkg::gw(NCTX+1)-1:0] free_count
);
import rc_pkg::*;
logic [NCTX-1:0] busy;
logic [15:0] gen [NCTX];
logic [31:0] age [NCTX];
always_comb begin
alloc_gnt = alloc_req && (busy != '1);
alloc_ctx = '0;
for (int i = NCTX-1; i >= 0; i--) if (!busy[i]) alloc_ctx = gw(NCTX)'(i);
alloc_gen = gen[alloc_ctx];
free_count = '0;
for (int i = 0; i < NCTX; i++) free_count += gw(NCTX+1)'(!busy[i]);
// A response is accepted only if the context is busy AND the generation
// matches. Pairing a response with whatever occupies the slot NOW is
// mutation 15, and §14 measured it misrouting 11,857 responses.
rsp_stale = rsp_valid && (!busy[rsp_ctx] || (gen[rsp_ctx] != rsp_gen));
rsp_accept = rsp_valid && !rsp_stale;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
busy <= '0; stale_count <= '0;
for (int i = 0; i < NCTX; i++) begin gen[i] <= '0; age[i] <= '0; end
end else begin
if (alloc_req && alloc_gnt) begin
busy[alloc_ctx] <= 1'b1;
age [alloc_ctx] <= '0;
end
for (int i = 0; i < NCTX; i++) if (busy[i]) age[i] <= age[i] + 32'd1;
if (rsp_accept) begin
busy[rsp_ctx] <= 1'b0;
// The generation advances at RELEASE, so a late response for this
// context can never match the next occupant (25.7 §7's law).
gen [rsp_ctx] <= gen[rsp_ctx] + 16'd1;
end
if (rsp_stale && stale_count != 32'hFFFF_FFFF)
stale_count <= stale_count + 32'd1;
// Cancellation: a context that has waited too long is reclaimed, and
// its generation advances for the same reason.
for (int i = 0; i < NCTX; i++)
if (busy[i] && age[i] >= CANCEL_LIMIT) begin
busy[i] <= 1'b0; gen[i] <= gen[i] + 16'd1;
end
end
end
endmoduleBlock 7 — the DMA return router. SYNTHESIZABLE. Ownership transfer from Link-facing logic to the system fabric.
Input owner: the Root Port's inbound path. Output owner: exactly one of the three destinations; never more than one. Beyond this example: real Root Complexes handle many more inbound types, peer-to-peer routing and message classes.
module dma_return_router (
input logic in_valid,
output logic in_ready,
input logic [2:0] in_type,
input logic [63:0] in_addr, // already translated, if applicable
input logic in_translated,
input rc_pkg::addr_class_e in_acl,
input logic in_xlate_fault,
output logic to_mem_valid,
input logic to_mem_ready,
output logic to_ctrl_valid,
input logic to_ctrl_ready,
output logic to_msg_valid,
input logic to_msg_ready,
output logic dropped_fault
);
import rc_pkg::*;
localparam logic [2:0] T_MSG = 3'd4;
always_comb begin
// A faulting translation is dropped and REPORTED. Forwarding it anyway
// is mutation 17 — the permission decision is made and then ignored.
dropped_fault = in_valid && in_xlate_fault;
to_mem_valid = in_valid && !in_xlate_fault && (in_acl == ACL_DRAM) && (in_type != T_MSG);
to_ctrl_valid = in_valid && !in_xlate_fault && (in_acl == ACL_CFG ) && (in_type != T_MSG);
to_msg_valid = in_valid && !in_xlate_fault && (in_type == T_MSG);
// Exactly one destination, or none. Never two (P7).
in_ready = (to_mem_valid && to_mem_ready ) ||
(to_ctrl_valid && to_ctrl_ready) ||
(to_msg_valid && to_msg_ready ) ||
dropped_fault;
end
endmoduleBlock 8 — Root Port performance counters. VERIFICATION-ONLY / diagnostic. 22.1's reasoning made measurable.
module rc_perf_counters (
input logic clk,
input logic rst_n,
input logic mmio_rd, mmio_wr,
input logic dma_in_beat,
input logic [15:0] dma_in_bytes,
input logic xlate_stall, // request held because translation is busy
input logic fabric_stall, // request held because the fabric is busy
input logic credit_stall, // request held for lack of PCIe credit
input logic cpl_wait, // an MMIO read is outstanding
input logic clear,
output logic [31:0] c_mmio_rd, c_mmio_wr,
output logic [63:0] c_dma_bytes,
output logic [31:0] c_xlate_stall, c_fabric_stall, c_credit_stall, c_cpl_wait
);
// Diagnostic ONLY: nothing here feeds a functional path. A counter that
// gates behaviour stops being an instrument and becomes a design
// dependency, which is mutation 27 (and P24 asserts it does not happen).
//
// The three stall counters are separate on purpose. 22.1's point is that
// "PCIe is slow" is never a diagnosis: the stall has to be attributed to
// translation, fabric or credit before any change is worth making.
`define SATCNT(c, en) if (en && (c != '1)) c <= c + 1'b1
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n || clear) begin
c_mmio_rd <= '0; c_mmio_wr <= '0; c_dma_bytes <= '0;
c_xlate_stall <= '0; c_fabric_stall <= '0; c_credit_stall <= '0; c_cpl_wait <= '0;
end else begin
`SATCNT(c_mmio_rd, mmio_rd);
`SATCNT(c_mmio_wr, mmio_wr);
`SATCNT(c_xlate_stall, xlate_stall);
`SATCNT(c_fabric_stall, fabric_stall);
`SATCNT(c_credit_stall, credit_stall);
`SATCNT(c_cpl_wait, cpl_wait);
if (dma_in_beat && (c_dma_bytes != '1)) c_dma_bytes <= c_dma_bytes + 64'(dma_in_bytes);
end
end
`undef SATCNT
endmoduleBlock 9 — the first-fault recorder. SYNTHESIZABLE diagnostic. The pattern 25.5 §10 established, applied to the host boundary.
module rc_first_fault (
input logic clk,
input logic rst_n,
input logic req_valid,
input logic [63:0] req_addr,
input logic f_classify, f_port, f_xlate, f_ctx, f_fabric,
input logic clear,
output rc_pkg::rc_fault_e first_fault,
output logic [63:0] first_addr,
output logic captured,
output logic [31:0] fault_count
);
import rc_pkg::*;
rc_fault_e this_f;
// Priority is the ORDER OF THE PIPELINE, so the recorded fault is the most
// upstream one. A later failure is very often a consequence of an earlier
// one, and reporting the wrong end of the chain sends the investigation
// to the wrong block entirely (25.5 §15 case 7).
always_comb begin
if (f_classify) this_f = RCF_CLASSIFY;
else if (f_port ) this_f = RCF_PORT;
else if (f_ctx ) this_f = RCF_CTX;
else if (f_xlate ) this_f = RCF_XLATE;
else if (f_fabric ) this_f = RCF_FABRIC;
else this_f = RCF_NONE;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n || clear) begin
captured <= 1'b0; first_fault <= RCF_NONE; first_addr <= '0; fault_count <= '0;
end else if (req_valid && (this_f != RCF_NONE)) begin
if (!captured) begin
captured <= 1'b1; first_fault <= this_f; first_addr <= req_addr;
end
if (fault_count != 32'hFFFF_FFFF) fault_count <= fault_count + 32'd1;
end
end
endmodule10. Same-Cycle Audit
11. Assertions
Address classification — §3.
// P1 — the classification is total: some class is always produced.
property p1_class_total;
@(posedge clk) disable iff (!rst_n)
req_valid |-> (acl inside {ACL_DRAM, ACL_MMIO, ACL_CFG, ACL_ERROR});
endproperty
a_p1: assert property (p1_class_total);
// P2 — an address in no window classifies as ERROR, never as MMIO.
// §13 measured the alternative at 50,346 spurious PCIe transactions.
property p2_unclaimed_is_error;
@(posedge clk) disable iff (!rst_n)
(req_valid && !hit_dram && !hit_cfg && (hit_mmio == '0)) |-> (acl == ACL_ERROR);
endproperty
a_p2: assert property (p2_unclaimed_is_error);
// P3 — at most one window claims an address; more is reported.
property p3_no_silent_ambiguity;
@(posedge clk) disable iff (!rst_n)
ambiguous |-> (acl == ACL_ERROR);
endproperty
a_p3: assert property (p3_no_silent_ambiguity);
// P4 — window membership is decided at full width.
property p4_full_width_compare;
@(posedge clk) disable iff (!rst_n)
(acl == ACL_MMIO) |-> ((req_addr >= mmio_win[mmio_idx].base) &&
(req_addr <= mmio_win[mmio_idx].limit));
endproperty
a_p4: assert property (p4_full_width_compare);
// P5 — the last byte of a window is inside it, and the next byte is not.
// 25.5 §6's pair, restated at platform scale.
property p5_window_edges;
@(posedge clk) disable iff (!rst_n)
(req_valid && (req_addr == dram_win.limit)) |-> (acl == ACL_DRAM);
endproperty
a_p5: assert property (p5_window_edges);
property p6_window_edge_exclusive;
@(posedge clk) disable iff (!rst_n)
(req_valid && (req_addr == dram_win.limit + 64'd1) && !hit_cfg && (hit_mmio == '0))
|-> (acl == ACL_ERROR);
endproperty
a_p6: assert property (p6_window_edge_exclusive);Port routing — §6.
// P7 — routing is attempted only for MMIO-classified addresses.
property p7_route_needs_mmio;
@(posedge clk) disable iff (!rst_n)
route_valid |-> (acl == ACL_MMIO);
endproperty
a_p7: assert property (p7_route_needs_mmio);
// P8 — exactly one port, or an explicit no-port/multi-port report.
property p8_route_onehot;
@(posedge clk) disable iff (!rst_n)
(req_valid && (acl == ACL_MMIO)) |-> $onehot({route_valid, no_port, multi_port});
endproperty
a_p8: assert property (p8_route_onehot);
// P9 — an unrouted MMIO address is reported, never silently dropped.
property p9_no_port_reported;
@(posedge clk) disable iff (!rst_n)
(req_valid && (acl == ACL_MMIO) && (port_hit_count == 0)) |-> no_port;
endproperty
a_p9: assert property (p9_no_port_reported);Request queue — the ready/valid transfer contract.
// P10 — a captured request is immutable while it waits.
property p10_request_stable;
@(posedge clk) disable iff (!rst_n)
(deq_valid && !deq_ready) |=> (deq_valid && $stable(deq_addr) &&
$stable(deq_len) && $stable(deq_rid));
endproperty
a_p10: assert property (p10_request_stable);
// P11 — valid does not depend on ready.
property p11_valid_independent;
@(posedge clk) disable iff (!rst_n)
(deq_valid && !deq_ready) |=> deq_valid;
endproperty
a_p11: assert property (p11_valid_independent);
// P12 — level tracks transfers, not valid cycles.
property p12_level_accounting;
@(posedge clk) disable iff (!rst_n)
(enq_valid && enq_ready && !(deq_valid && deq_ready)) |=> (level == $past(level) + 1);
endproperty
a_p12: assert property (p12_level_accounting);
// P13 — the queue never overflows or underflows.
property p13_level_bounded;
@(posedge clk) disable iff (!rst_n)
(level <= DEPTH);
endproperty
a_p13: assert property (p13_level_bounded);
// P14 — no enqueue when full, no dequeue when empty.
property p14_no_overrun;
@(posedge clk) disable iff (!rst_n)
((enq_valid && enq_ready) |-> (level < DEPTH)) and
((deq_valid && deq_ready) |-> (level > 0));
endproperty
a_p14: assert property (p14_no_overrun);Translation contexts — §5, §14.
// P15 — a context is allocated only when free.
property p15_alloc_when_free;
@(posedge clk) disable iff (!rst_n)
(alloc_req && alloc_gnt) |-> !busy[alloc_ctx];
endproperty
a_p15: assert property (p15_alloc_when_free);
// P16 — a context is never allocated twice concurrently.
property p16_ctx_unique;
@(posedge clk) disable iff (!rst_n)
(alloc_req && alloc_gnt) |=> busy[$past(alloc_ctx)];
endproperty
a_p16: assert property (p16_ctx_unique);
// P17 — a response is accepted only for a busy context with a matching
// generation. §14 measured pairing-with-current at 11,857 misroutes.
property p17_response_identity;
@(posedge clk) disable iff (!rst_n)
rsp_accept |-> (busy[rsp_ctx] && (gen[rsp_ctx] == rsp_gen));
endproperty
a_p17: assert property (p17_response_identity);
// P18 — a stale response is discarded AND counted.
property p18_stale_counted;
@(posedge clk) disable iff (!rst_n)
rsp_stale |=> (stale_count > $past(stale_count));
endproperty
a_p18: assert property (p18_stale_counted);
// P19 — the generation advances whenever a context is released, so a late
// response can never match the next occupant.
property p19_gen_advances_on_release;
@(posedge clk) disable iff (!rst_n)
rsp_accept |=> (gen[$past(rsp_ctx)] != $past(gen[$past(rsp_ctx)]));
endproperty
a_p19: assert property (p19_gen_advances_on_release);
// P20 — no double free.
property p20_no_double_free;
@(posedge clk) disable iff (!rst_n)
(rsp_accept && !busy[rsp_ctx]) |-> 1'b0;
endproperty
a_p20: assert property (p20_no_double_free);
// P21 — reset invalidates every context.
property p21_reset_clears_ctx;
@(posedge clk)
!rst_n |=> (busy == '0);
endproperty
a_p21: assert property (p21_reset_clears_ctx);
// P22 — the free count agrees with the busy vector.
property p22_free_count_consistent;
@(posedge clk) disable iff (!rst_n)
(free_count == NCTX - $countones(busy));
endproperty
a_p22: assert property (p22_free_count_consistent);Return routing and fault handling — §7.
// P23 — a faulting translation never produces a memory transaction.
// §14 measured a design that forwarded anyway: 2,103 accesses.
property p23_fault_blocks_memory;
@(posedge clk) disable iff (!rst_n)
in_xlate_fault |-> (!to_mem_valid && !to_ctrl_valid);
endproperty
a_p23: assert property (p23_fault_blocks_memory);
// P24 — exactly one destination, or none.
property p24_router_onehot0;
@(posedge clk) disable iff (!rst_n)
$onehot0({to_mem_valid, to_ctrl_valid, to_msg_valid});
endproperty
a_p24: assert property (p24_router_onehot0);
// P25 — diagnostic counters are non-functional: nothing depends on them.
property p25_counters_nonfunctional;
@(posedge clk) disable iff (!rst_n)
$stable(c_dma_bytes) or (route_valid == $past(route_valid_expected));
endproperty
a_p25: assert property (p25_counters_nonfunctional);Cover — the anti-vacuity set.
// P26 — a request produces exactly one outcome: it is classified, routed
// and forwarded, or it is recorded as a fault. A request that produces
// neither has vanished inside the host boundary, and no downstream
// instrument can tell that it did.
property p26_every_request_resolved;
@(posedge clk) disable iff (!rst_n)
req_valid |-> (route_valid || win_reject || (acl == ACL_ERROR) ||
(acl inside {ACL_DRAM, ACL_CFG}));
endproperty
a_p26: assert property (p26_every_request_resolved);
// P26's covers — the exceptional states must actually occur, or P2, P3, P9, P17 and
// P23 are all satisfied without ever evaluating a consequent.
c1_error_class: cover property (@(posedge clk) disable iff (!rst_n) acl == ACL_ERROR);
c2_ambiguous: cover property (@(posedge clk) disable iff (!rst_n) ambiguous);
c3_no_port: cover property (@(posedge clk) disable iff (!rst_n) no_port);
c4_stale_rsp: cover property (@(posedge clk) disable iff (!rst_n) rsp_stale);
c5_xlate_fault: cover property (@(posedge clk) disable iff (!rst_n) in_xlate_fault);
c6_ctx_exhaust: cover property (@(posedge clk) disable iff (!rst_n) free_count == 0);
c7_queue_full: cover property (@(posedge clk) disable iff (!rst_n) !enq_ready);
c8_window_edge: cover property (@(posedge clk) disable iff (!rst_n)
req_valid && (req_addr == dram_win.limit));12. Executable Counterexamples
Counterexample A — the fall-through classifier (violates P2).
// The classifier tests the ranges it knows and sends everything else to PCIe.
module ce_a_fallthrough_classify (
input logic [63:0] a,
input rc_pkg::window_t dram, cfg,
output rc_pkg::addr_class_e acl
);
import rc_pkg::*;
always_comb begin
if (in_window(dram, a)) acl = ACL_DRAM;
else if (in_window(cfg , a)) acl = ACL_CFG;
else acl = ACL_MMIO; // <-- the default
end
endmodule
// Failing stimulus: an address above every configured window.
// Golden: ACL_ERROR — the platform does not own it.
// This: ACL_MMIO. A real TLP is generated toward a Root Port for an address
// no window claims.
// P2 fails.
// §13 measured 50,346 such addresses in 200,000 probes. The observable
// consequence is an Unsupported Request the software never asked for, and a
// fault reported against the DEVICE for an error made by the host.Counterexample B — the low-32 comparator (violates P4).
// The MMIO window comparison uses only the low 32 bits of the address.
module ce_b_low32_classify (
input logic [63:0] a,
input rc_pkg::window_t dram,
output rc_pkg::addr_class_e acl
);
import rc_pkg::*;
always_comb
acl = (a[31:0] <= dram.limit[31:0]) ? ACL_DRAM : ACL_ERROR; // <-- truncated
endmodule
// Failing stimulus: an MMIO address whose low 32 bits fall inside the DRAM
// window — e.g. DRAM [0, 0xFFFF_FFFF] and MMIO at 0x1_0000_0000.
// Golden: ACL_MMIO, routed to a Root Port.
// This: ACL_DRAM. The access is answered by the memory controller.
// P4 fails.
// §13 measured 39,648 MMIO accesses misrouted into DRAM. The device is never
// reached, the read SUCCEEDS, and it returns whatever DRAM held. No error is
// raised at any layer — 25.5 §4's Direction B, at platform scale.13. Measured Behaviour — Address Classification
| classifier fault | disagree | MMIO→DRAM | CFG→MMIO | CFG→DRAM | valid→ERR | ERR→valid |
|---|---|---|---|---|---|---|
| none (correct) | 0 | 0 | 0 | 0 | 0 | 0 |
| unclaimed defaults to MMIO | 50,346 | 0 | 0 | 0 | 0 | 50,346 |
| MMIO window has no top | 90,254 | 0 | 39,908 | 0 | 0 | 50,346 |
| 32-bit address compare | 129,902 | 39,648 | 0 | 39,908 | 0 | 50,346 |
Three readings.
The healthy row is zero in every column, which is the check that makes the rest meaningful.
ERR→valid and MMIO→DRAM are both fatal and they fail in opposite directions. The first invents a PCIe transaction for an address nobody owns; the second answers a PCIe transaction out of local memory. The first produces a visible error; the second produces a plausible wrong value and no error at all.
And the third row is a single design decision. Comparing at 32 bits instead of 64 does not degrade gracefully — it produces three distinct misroutings at once, because every window above 4 GiB aliases onto something below it.
14. Measured Behaviour — Translation Contexts
| configuration | issued | done | cancelled | late responses | misrouted | fault forwarded |
|---|---|---|---|---|---|---|
| correct (serial-checked) | 56,172 | 52,753 | 3,403 | 3,400 | 0 | 0 |
| response paired with slot occupant | 62,748 | 60,195 | 2,542 | 14,395 | 11,857 | 0 |
| permission fault still forwards | 56,172 | 52,753 | 3,403 | 3,400 | 0 | 2,103 |
| context never reclaimed | 16 | 14 | 2 | 2 | 0 | 0 |
Four readings.
Late responses are normal. The correct configuration saw 3,400 of them and misrouted none. A design that treats a late translation response as impossible has a bug; one that treats it as an error has misunderstood the mechanism.
Pairing a response with the current slot occupant misrouted 11,857 responses, measured by a ground-truth serial no hardware mechanism reads. Every one of those is a memory access performed with a physical address translated for a different request.
Forwarding a faulting translation produced 2,103 memory accesses that a permission check had already denied. The check ran; its result was discarded.
And never reclaiming a context collapses the design to 16 issued requests. As in 25.7 §13, refusing to reclaim is not the conservative option — it is a different failure.
15. Verification — Mutations
Thirty mutations. Every "Caught by" entry names a property from §11.
| # | Mutation | Symptom | Caught by |
|---|---|---|---|
| 1 | Unclaimed address defaults to MMIO | 50,346 spurious PCIe transactions (§13) | P2 |
| 2 | Unclaimed address defaults to DRAM | device access answered by memory | P2 |
| 3 | Window comparison at 32 bits | 39,648 MMIO accesses into DRAM (§13) | P4 |
| 4 | MMIO window has no upper bound | swallows the configuration region (§13) | P4 |
| 5 | Window limit treated as exclusive | last page of every window unreachable | P5 |
| 6 | Port routing performed before classification | cannot distinguish "no port" from "not ours" | P7 |
| 7 | Overlapping windows resolved by priority | a silent winner; no ambiguity reported | P3, P8 |
| 8 | ambiguous computed but not driven out | detected and discarded | P3 |
| 9 | Queue output driven from live input signals | request address changes while it waits | P10 |
| 10 | valid cycles counted as transfers | level diverges from contents | P12 |
| 11 | valid deasserted while ready is low | a request withdrawn after being offered | P11 |
| 12 | Enqueue permitted when full | oldest request overwritten | P13, P14 |
| 13 | Dequeue permitted when empty | a stale request replayed | P14 |
| 14 | Level incremented on simultaneous enq and deq | drifts upward until it wedges | P12 |
| 15 | Translation response paired with slot occupant | 11,857 misroutes (§14) | P17 |
| 16 | Generation not advanced on release | a late response matches the next occupant | P19 |
| 17 | Permission fault still forwards the access | 2,103 denied accesses performed (§14) | P23 |
| 18 | Fault gated on a registered value | forwards for one cycle — one full write | P23 |
| 19 | Context allocated while busy | two requests share one context | P15, P16 |
| 20 | Context freed twice | the pool grows beyond NCTX | P20, P22 |
| 21 | Context never reclaimed | 16 requests issued vs 56,172 (§14) | P22 |
| 22 | Reset leaves contexts busy | the pool never recovers | P21 |
| 23 | Stale response discarded silently | indistinguishable from never arriving | P18 |
| 24 | Free count cached rather than derived | allocator believes contexts exist | P22 |
| 25 | Router asserts two destinations | one inbound write performed twice | P24 |
| 26 | Router drops a transaction with no report | inbound data vanishes | P24 |
| 27 | A performance counter gates a functional path | the instrument becomes a dependency | P25 |
| 28 | First-fault priority reversed | reports the consequence, not the cause | P25 |
| 29 | Testbench never generates an unclaimed address | P2, P3, P9 all vacuous | P26 (c1, c2, c3) |
| 30 | Testbench never delays a translation response | the stale path is unreachable | P26 (c4) |
Mutations 29 and 30 break the testbench, and their symptom is that everything passes.
16. Debugging
17. Misconceptions
"PCIe is connected to the CPU." It is connected to the Root Complex, which is connected to the fabric, which the cores reach through the cache and interconnect (§2). The chain matters because each link fails differently.
"DMA goes through the CPU." It reaches the memory controller through the I/O and coherent fabric, with no core executing an instruction (§2). This is why heavy DMA is invisible in a core profile.
"Root Complex and Root Port are the same thing." The Root Complex is the host-side architecture; a Root Port is one port originating one hierarchy branch (§6). Link training belongs to a port; address ownership belongs to the complex.
"An address that isn't DRAM must be MMIO." That default produced 50,346 spurious PCIe transactions in §13. Unclaimed is an error class, not a fall-through.
"A misrouted address will produce an error." Only in one direction. §13's 32-bit comparator sent 39,648 MMIO accesses to DRAM, which answered them successfully with the wrong data.
"The PCIe address is the DRAM address." Not under translation (§5). A correct Memory Write to IOVA X may reach any physical page, or none.
"If the IOMMU broke it, PCIe is broken." The Link is in L0, the TLPs are well-formed, and the LTSSM never left its state (§16 case 3). Nothing in the PCIe layer is involved.
"MMIO works, so the device is fine." MMIO proves the Link is up and the BAR decodes (§4). It says nothing about the DMA path, which is device-initiated and separately translated.
"Use MMIO reads to move the data." Each one stalls a core for a full round trip (§4). Bulk movement belongs on the DMA path (20.2).
"NUMA effects are a PCIe property." They are a platform placement property (§16 case 4). The negotiated Link width is identical in both placements and can be read to prove it.
"A translation response can be matched by order." Responses can return out of order; matching by position is 21.4 §3's forbidden pattern, and §14 measured it at 11,857 misroutes.
"Late translation responses mean something is wrong." The correct configuration saw 3,400 and misrouted none (§14). They are the normal consequence of cancelling a request whose answer is still in flight.
"Not reclaiming a context is the safe choice." It reduced the design to 16 issued requests (§14). It is a different failure, not a conservative one.
"Capability lane width is what the link negotiated." Capability is what a port can do; the negotiated width is what it did (18.1). Only the second predicts anything, and only the second should appear in a bandwidth calculation.
18. Understanding Check
Q1. A device DMA read works with identity mapping and fails once the IOMMU is enabled. The Link stays in L0 and the Endpoint keeps issuing well-formed Memory Reads. Why is "PCIe is broken" a weak diagnosis, and what should be measured next?
Because nothing in the PCIe layer changed (§5, §16 case 3). The Link state, the TLP contents and the Endpoint's behaviour are identical in both configurations — the only thing that changed is what happens to the address after it crosses the Root Port. The next measurement is at the translation boundary: whether a translation was performed for that device's identity, whether it faulted, and whether faults are being reported at all. §14 measured a design that forwarded denied translations anyway, performing 2,103 accesses a permission check had already refused — so an absence of reported faults is not evidence that none occurred.
Q2. Your host classifier sends any address it does not recognise to the PCIe MMIO window. What does that cost, and what is the correct default?
§13 measured 50,346 spurious PCIe transactions in 200,000 probes (§3, counterexample A). Every one is a real TLP generated for an address the platform does not own, producing an Unsupported Request the software never requested — and the fault is then reported against the device for an error the host made. The correct default is ACL_ERROR: every range must be checked and an unclaimed address is an error, never a fall-through (P2). This is the same law 25.5 §10 established for BAR decoders.
Q3. Which is more dangerous — classifying an MMIO address as DRAM, or classifying an unowned address as MMIO? Why?
MMIO→DRAM (§3, §13). The unowned→MMIO case produces a visible error: a UR comes back and something logs it. MMIO→DRAM produces a successful read of the wrong data — the memory controller answers, the device is never reached, and no error is raised at any layer. §13 measured 39,648 such accesses from a single 32-bit comparator. It is 25.5 §4's Direction B at platform scale, and the only way to detect it is comparing returned values against expected ones.
Q4. Why must a translation response carry a context identity rather than being matched by arrival order?
Because responses can return out of order, so position carries no information (§5, P17). §14 measured a design that paired each response with whatever occupied the context slot at that moment: 11,857 misroutes, each one a memory access performed with a physical address translated for a different request. The correct design saw the same 3,400 late responses and misrouted none. This is 21.4 §3's law — an asynchronous response must carry its own identity — and the generation counter (P19) is what extends it to cover reuse of the identity itself.
Q5. A colleague proposes never reclaiming a translation context, to eliminate late-response misrouting entirely. Evaluate.
It eliminates the misrouting and replaces it with starvation. §14 measured that configuration issuing 16 requests against a healthy 56,172, with zero free contexts at the end — every cancelled translation permanently consumes one. Refusing to reclaim is not the conservative option; it is a different failure (P22), exactly as 25.7 §13 measured for Tags. The correct answer is to reclaim and advance the generation, which is what makes a late response detectable rather than impossible.
Q6. MMIO to a device works perfectly and its DMA never completes. What has the working MMIO actually proved?
That the Link is up, the address classification routes to the right port, and the BAR decodes (§4, §16 case 2). It has proved nothing about DMA, because the two paths differ in initiator, direction, size, translation machinery and what limits them. MMIO is CPU-initiated and latency-bound; DMA is device-initiated, translated on the device's behalf using the device's identity, and bandwidth-bound. The next step is the device's own DMA instrumentation (25.6) and the translation boundary — not more MMIO tests.
Q7. Why does §9's request queue drive its outputs from stored state rather than from the input port?
Because a request that has been accepted is owned by the queue, and its fields must not change while it waits (P10, mutation 9). Driving the output combinationally from the input means a request whose Root Port is busy can have its address or length change underneath it when the fabric presents the next request. The ready/valid contract requires metadata to be stable under stall, and the transfer is valid && ready — counting valid cycles instead is mutation 10, which makes the level counter diverge from the queue's actual contents.
19. What Module 26 Does Next
This chapter built the host side of the boundary. The remaining four cross it from the other direction — as devices.
| Chapter | The system, and what it owns |
|---|---|
| 26.1 (this) | the host — address ownership, translation, two directions |
| 26.2 GPUs | a high-bandwidth Endpoint with its own memory and copy engines |
| 26.3 SSD Controllers | a queue-driven Endpoint that fetches its own work |
| 26.4 Network Adapters | a packet Endpoint with descriptor rings in both directions |
| 26.5 FPGA Cards | a programmable Endpoint where you own the DMA engine |
One idea from this chapter recurs in all four, and it is worth naming now. Every boundary in a system is an ownership transfer with an identity: an address to a window, a request to a context, a response to the request that made it. §14's result — that a response paired with a slot rather than a request misroutes 11,857 times — is the same finding 25.6 made about descriptors and 25.7 made about Tags.
And one measurement discipline carries forward. §13's classifier was checked against an oracle written independently, in a different style, sharing no code. Every chapter in this module does the same, because a systems model that validates itself validates nothing.