"PCIe Is Memory-Mapped Only" — The Traffic You Forgot to Decode

Chapter 31.1 corrected a belief about the fabric. This one corrects a belief about what travels on it — and it is the rare myth that refutes itself in a single step.

A device that handled only memory transactions could never have received the Configuration writes that gave it its BARs. The myth requires the device to already be configured, and configuration is not memory traffic.

1. Where the Myth Comes From

Three true things make it a reasonable inference.

Memory traffic dominates by volume. In a working system almost every TLP is a Memory Read or Write. §10's mix has memory at 74%, and in a bulk-transfer device it is higher still. A designer who watches a trace of a running system sees memory traffic and very little else.

The programming model is memory-shaped. BARs claim address windows; drivers read and write those windows; DMA moves data to and from host memory. From software, the interface genuinely is memory-mapped — and that is the part of the myth that is true.

And the other types are invisible when they work. Configuration traffic happens at boot, before anyone is watching. Messages are infrequent. Completions are so tightly bound to reads that they are often thought of as part of the read rather than as a separate transaction type.

So the myth survives because it describes steady state accurately. It fails at the boundaries — at enumeration, at error time, at interrupt time — which is exactly where the hard bugs are.

2. What Actually Travels

A TLP's type is carried in its header, not inferred from context (11.2, 11.3). The families an endpoint deals with:

Two observations that reframe the myth.

Completions are their own transaction type, not part of a read. A Memory Read and its Completion are two separate TLPs, routed differently — the read is address-routed, the Completion is ID-routed back to the requester (11.5). A device that treats a read as one indivisible operation has no place to put a completion timeout, which is why 25.7 exists as a chapter.

And Configuration traffic is what makes memory traffic possible. Enumeration writes BARs; BARs establish the address windows that memory transactions target (9.5). The dependency runs the wrong way for the myth: memory access is downstream of configuration, not a replacement for it.

3. Why the Myth Is Self-Refuting

Follow the boot sequence and the contradiction appears immediately.

1. the device powers on with no BARs assigned
2. host software reads configuration space to discover it   ← Configuration
3. host software sizes and assigns BARs                     ← Configuration
4. host software enables memory decoding and bus-mastering  ← Configuration
5. only NOW can a memory transaction reach the device

Steps 2 through 4 are not memory transactions. They target configuration space, which is a separate address space with its own transaction type and its own routing (8.1, 11.5).

So a strictly memory-only device is unreachable. It has no BARs because nothing could write them, and therefore no memory window to be addressed by.

This is worth stating plainly because it converts an abstract taxonomy into a concrete impossibility. The myth is not merely incomplete — it describes a device that could not exist, and the reason nobody notices is that the configuration step is performed by firmware before any engineer is looking at a trace.

4. What Actually Goes Wrong

Almost nobody builds a decoder that ignores Configuration outright. The realistic failures are narrower and they are all the same shape: a decoder whose default case is wrong.

The last row is the dangerous one, and it is 25.5 §4's Direction B in a new place: a device that answers a request it did not understand returns a plausible value and no error.

The correct behaviour for a genuinely unsupported request is a defined response — an Unsupported Request status (13.2) — which is a reported failure rather than a silent one. §10 measures the difference between the three dispositions.

5. The Decode Path, Drawn

Two readings.

Type decode precedes address decode. A Completion has no meaningful address to compare against a BAR — it is matched by (Requester ID, Tag) (10.2). A decoder that reaches for the BAR comparator first has already made a category error.

And there are two edges out of "unrecognised", not one. The upper one is correct and reported; the lower one is the bug. They differ by whether anything downstream ever learns the transaction arrived.

6. RTL — Type Decode With an Honest Default

Block 1 — the package and the classification. The default case is the whole point.

package tlp_class_pkg;
 
  // The five dispositions an arriving TLP can have. UNSUPPORTED is a
  // first-class outcome, not a fall-through — §4's central point.
  typedef enum logic [2:0] {
    TC_MEMORY      = 3'd0,
    TC_COMPLETION  = 3'd1,
    TC_CONFIG      = 3'd2,
    TC_MESSAGE     = 3'd3,
    TC_IO          = 3'd4,
    TC_UNSUPPORTED = 3'd5     // recognised as NOT handled — and reported
  } tlp_class_e;
 
  function automatic int unsigned gw(input int unsigned n);
    return (n <= 1) ? 1 : $clog2(n);
  endfunction
 
endpackage

module tlp_type_classify (
  input  logic                  rx_valid,
  input  logic [2:0]            rx_fmt,       // placeholder encoding
  input  logic [4:0]            rx_type,      // placeholder encoding
  // which classes THIS device implements — a design decision, made explicit
  input  logic                  impl_memory,
  input  logic                  impl_config,
  input  logic                  impl_message,
  input  logic                  impl_io,
  output tlp_class_pkg::tlp_class_e rx_class,
  output logic                  route_memory,
  output logic                  route_completion,
  output logic                  route_config,
  output logic                  route_message,
  output logic                  emit_ur,
  output logic                  onehot_ok
);
 
  import tlp_class_pkg::*;
 
  // Placeholder decodes. The real Fmt/Type encodings are SPEC-DEFINED and
  // belong to 11.3; what matters structurally is that the type is READ FROM
  // THE HEADER rather than inferred from the address or from context.
  logic is_mem, is_cpl, is_cfg, is_msg, is_io;
  always_comb begin
    is_mem = (rx_type == 5'h00);
    is_cpl = (rx_type == 5'h0A);
    is_cfg = (rx_type == 5'h04) || (rx_type == 5'h05);
    is_msg = (rx_type[4:3] == 2'b10);
    is_io  = (rx_type == 5'h02);
  end
 
  always_comb begin
    // A Completion is ALWAYS handled if the device issues Non-Posted
    // requests — there is no "implements completions" choice. A requester
    // that cannot receive Completions cannot read (§2), and a decoder that
    // makes this optional has made an impossible configuration expressible.
    if      (!rx_valid)             rx_class = TC_UNSUPPORTED;
    else if (is_cpl)                rx_class = TC_COMPLETION;
    else if (is_mem && impl_memory) rx_class = TC_MEMORY;
    else if (is_cfg && impl_config) rx_class = TC_CONFIG;
    else if (is_msg && impl_message)rx_class = TC_MESSAGE;
    else if (is_io  && impl_io)     rx_class = TC_IO;
    // THE DEFAULT. Everything not recognised AND implemented becomes an
    // explicit UNSUPPORTED, which is reported (§4). Falling through to
    // TC_MEMORY here decodes a Configuration write against a BAR
    // (mutation 1); returning nothing leaves the requester waiting for a
    // Completion that will never come (mutation 2).
    else                            rx_class = TC_UNSUPPORTED;
 
    route_memory     = rx_valid && (rx_class == TC_MEMORY);
    route_completion = rx_valid && (rx_class == TC_COMPLETION);
    route_config     = rx_valid && (rx_class == TC_CONFIG);
    route_message    = rx_valid && (rx_class == TC_MESSAGE);
 
    // A Non-Posted request that is unsupported must receive a defined
    // response. A Posted write or a Message has no Completion to return, so
    // emit_ur is qualified — an unconditional emit would fabricate
    // Completions for transactions that never expected one (mutation 5).
    emit_ur = rx_valid && (rx_class == TC_UNSUPPORTED) && !is_msg && !is_mem_write_placeholder;
 
    // Exactly one route, or none with a disposition. Two routes means a
    // transaction is handled twice by different logic (mutation 3).
    onehot_ok = !rx_valid ||
                $onehot0({route_memory, route_completion, route_config, route_message});
  end
 
  // Placeholder for "this memory TLP is a write" — Posted, so no Completion.
  logic is_mem_write_placeholder;
  assign is_mem_write_placeholder = is_mem && (rx_fmt[1] == 1'b1);
 
endmodule

Block 2 — the disposition counters. §7's measurement, in hardware, because a silent drop is otherwise invisible.

// Every arriving TLP leaves a trace in exactly one counter. This is what
// turns §7's "silently dropped" row from an invisible fault into an
// observable one — 30.6 §2's rule that hypotheses are eliminated by
// positive observation, applied to the receive path.
module tlp_disposition_counters (
  input  logic clk,
  input  logic rst_n,
  input  logic rx_valid,
  input  tlp_class_pkg::tlp_class_e rx_class,
  input  logic emit_ur,
  input  logic clear,
  output logic [31:0] c_memory, c_completion, c_config, c_message, c_io,
  output logic [31:0] c_unsupported, c_ur_emitted,
  output logic        accounting_gap      // arrived but counted nowhere
);
 
  import tlp_class_pkg::*;
 
  // If a TLP arrives and no counter advances, the receive path has a hole.
  // Asserting this is cheaper than discovering it from a requester timeout
  // three layers away.
  assign accounting_gap = rx_valid && !(rx_class inside
      {TC_MEMORY, TC_COMPLETION, TC_CONFIG, TC_MESSAGE, TC_IO, TC_UNSUPPORTED});
 
  `define BUMP(c, cond) if ((cond) && (c != 32'hFFFF_FFFF)) c <= c + 32'd1
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      c_memory <= '0; c_completion <= '0; c_config <= '0; c_message <= '0;
      c_io <= '0; c_unsupported <= '0; c_ur_emitted <= '0;
    end else if (rx_valid) begin
      `BUMP(c_memory,      rx_class == TC_MEMORY);
      `BUMP(c_completion,  rx_class == TC_COMPLETION);
      `BUMP(c_config,      rx_class == TC_CONFIG);
      `BUMP(c_message,     rx_class == TC_MESSAGE);
      `BUMP(c_io,          rx_class == TC_IO);
      `BUMP(c_unsupported, rx_class == TC_UNSUPPORTED);
      `BUMP(c_ur_emitted,  emit_ur);
    end
  end
  `undef BUMP
 
endmodule

7. Same-Cycle Audit

8. Invariants

// P1 — the classification is TOTAL. Every arriving TLP gets a class,
// including the ones the device does not implement.
// Catches: a decoder with an implicit fall-through (mutation 2).
property p1_classification_total;
  @(posedge clk) disable iff (!rst_n) rx_valid |-> !accounting_gap;
endproperty
 
// P2 — at most one handler is routed to. Two routes means one transaction
// is processed twice by logic that each believes it owns it.
property p2_route_onehot0;
  @(posedge clk) disable iff (!rst_n) rx_valid |-> onehot_ok;
endproperty
 
// P3 — an unimplemented type never routes to the memory handler. This is
// the central bug of §4: a Configuration or Message TLP decoded against a
// BAR comparator, which is a category error the BAR logic cannot detect.
property p3_unsupported_never_memory;
  @(posedge clk) disable iff (!rst_n)
    (rx_class == TC_UNSUPPORTED) |-> !route_memory;
endproperty
 
// P4 — a Completion is always routed, regardless of implementation flags.
// A requester that cannot receive Completions cannot read (§6 audit A).
property p4_completion_always_routed;
  @(posedge clk) disable iff (!rst_n)
    (rx_valid && is_cpl) |-> route_completion;
endproperty
 
// P5 — an unsupported Non-Posted request produces a defined response.
// Silence leaves the requester to time out, which reports the wrong cause
// three layers away (25.7 §3 class A).
property p5_unsupported_np_answered;
  @(posedge clk) disable iff (!rst_n)
    (rx_class == TC_UNSUPPORTED && rx_is_nonposted) |-> emit_ur;
endproperty
 
// P6 — an unsupported Posted write or Message produces NO Completion.
// Fabricating one sends an unsolicited Completion with a Tag the requester
// never allocated (§7 audit B).
property p6_no_cpl_for_posted;
  @(posedge clk) disable iff (!rst_n)
    (rx_class == TC_UNSUPPORTED && (is_mem_write_placeholder || is_msg)) |-> !emit_ur;
endproperty
 
// P7 — every arrival advances exactly one counter. Makes a silent drop
// observable rather than inferable (30.6 §2).
property p7_exactly_one_counter;
  @(posedge clk) disable iff (!rst_n)
    rx_valid |=> ($countones({$changed(c_memory), $changed(c_completion),
                              $changed(c_config), $changed(c_message),
                              $changed(c_io), $changed(c_unsupported)}) == 1);
endproperty

P3 and P5 are the pair worth naming. P3 prevents the wrong handler; P5 prevents no handler. A decoder can fail either way, and the two failures produce completely different symptoms — a BAR-decoded configuration write versus a requester timeout.

9. Bug Bank

10. Measured — What Each Decoder Drops

Three readings.

Completions are the largest single category dropped by the memory-only decoder — 27,809, more than half the total. Every one is a read the device issued and will never see answered, which turns into a completion timeout and a misattributed root cause (25.7 §3).

Configuration at 12,150 is the self-refuting one (§3). A device dropping these was never enumerated, so in reality it would not be receiving the memory traffic in row 1 either — the model is being generous to the myth by letting the memory traffic arrive at all.

And the middle row is the realistic bug. Memory plus Completions is what a designer builds when they think about the data path and forget the control plane. It handles 88% of traffic and cannot be configured, which fails at boot rather than under load — the one saving grace of this particular mistake.

11. Debugging

12. Misconceptions

"PCIe only carries memory reads and writes." Why it sounds plausible: memory dominates by volume — §10's mix has it at 74% — and the software interface genuinely is memory-shaped (§1). What really happens: Configuration, Completion, Message and IO traffic all arrive, and §10 measured a memory-only decoder dropping 26%. What it causes: a decoder whose default case sends unrecognised types to the memory handler (§4), decoding a Configuration write against a BAR.

"A read and its Completion are one transaction." Why it sounds plausible: they are one logical operation and are always discussed together. What really happens: they are two TLPs, routed differently — address-routed out, ID-routed back (§2, 11.5). What it causes: a design with no place to put a completion timeout, and no way to represent a request that was answered partially (25.7 §3 class D).

"If the device doesn't support it, dropping it is fine." Why it sounds plausible: the device cannot act on it, so discarding seems harmless. What really happens: a Non-Posted request that receives nothing leaves the requester to time out, and the timeout reports the wrong cause three layers away (§4, P5). What it causes: a completion timeout investigation at the requester for a fault in the completer's decoder — the most expensive possible place to look.

"Answering with a Completion is always the safe default." Why it sounds plausible: returning something feels safer than returning nothing. What really happens: a Posted write expects no Completion (§7 audit B). Emitting one sends an unsolicited Completion carrying a Tag the requester never allocated. What it causes: a stale Completion in the fabric that matches a live request once the Tag is reused (§11 case 4).

"Configuration traffic is a boot-time concern only." Why it sounds plausible: enumeration happens once, at startup, before anyone is watching. What really happens: configuration space is accessed during operation too — for error status, control changes and power management. What it causes: a decode path that is exercised only at boot and never verified under concurrent memory traffic, which is where the fall-through of §4 actually fires.

"The percentage dropped tells you how bad it is." Why it sounds plausible: 26% sounds like a proportionate amount of damage. What really happens: composition dominates (§10). The Configuration 6% makes the device unreachable; the Completion 14% strands every outstanding read. Losing the smallest category can be the fatal one. What it causes: triage that prioritises by volume rather than by dependency, and misses the category whose absence prevents everything else.

13. Understanding Check

Q1. Explain in one step why "PCIe is memory-mapped only" describes a device that cannot exist.

Because BARs are assigned by Configuration writes, and Configuration is not memory traffic (§3). A device that handles only memory transactions has no way to receive the writes that size and assign its BARs, so it has no memory window for a memory transaction to target. The dependency runs the wrong way for the myth: memory access is downstream of configuration, not an alternative to it. §10 is generous to the myth by letting memory traffic arrive at all — in reality row 1's 147,930 handled TLPs could never have been addressed to the device.

Q2. A device's reads all time out. c_completion reads zero and the far side reports sending Completions. What is the fault and why is the timeout misleading?

Completions are arriving and being misclassified (§11 case 2, P4). The timeout fires at the requester, so the investigation naturally starts there or at the completer — and the fault is in the requester's own receive decode. 25.7 §3's classification cannot separate "never answered" from "answered and dropped locally" without a counter at the classification point, which is exactly what §6 Block 2 provides. The disposition counters convert a three-layer-away symptom into a one-read answer.

Q3. Two unsupported transactions arrive: a Posted write and a Non-Posted read. Why do they require opposite responses?

Because only one of them has a requester waiting (§7 audit B, P5, P6). The Non-Posted read allocated a Tag and is holding context for an answer; giving it a defined Unsupported Request status lets it retire with a named cause instead of timing out. The Posted write expects nothing back — emitting a Completion for it sends an unsolicited response carrying a Tag the requester never allocated, which sits in the fabric until the Tag is reused and then matches a live request. The disposition depends on whether an answer was expected, which is a property of the transaction type rather than of the support decision.

Q4. Why is Completion handling in §6 Block 1 deliberately not gated by an implementation flag?

Because a device that issues Non-Posted requests must be able to receive their answers (§6, §7 audit A, P4). Making it conditional creates an expressible configuration in which the device can read but cannot be answered — and a single control-register write would then strand every outstanding request simultaneously. The branch ordering encodes that contract: the Completion test precedes every impl_* test, so no configuration change can affect it. It is the one class where "does this device implement it" is not a meaningful question.

Q5. §10 shows a memory-only decoder dropping 26%. Why is that the least useful number in the table?

Because it depends entirely on the traffic mix, while the composition depends on the architecture (§10, §12). Change the mix and the percentage moves; the dependency structure does not. Configuration at 6% makes the device unreachable — the smallest category except IO, and the fatal one. Completions at 14% strand every outstanding read. Triaging by volume would prioritise the memory path, which is the one part that works. The useful reading is which categories the device cannot function without, and both of those are minorities of the traffic.

14. What Comes Next

These two chapters corrected beliefs about the fabric and about what travels on it. The remaining four narrow further — from what PCIe is to what specific mechanisms do.

And they are ordered by how much damage the myth causes. 31.3 corrects a belief about BARs that produces address-decode bugs; 31.4 one about DMA that produces ownership bugs; 31.5 one about MSI that produces ordering bugs; 31.6 one about the LTSSM that produces the belief that a link which trained successfully will stay trained — which is the myth that survives longest, because it is true right up until it is not.

One method runs through all six, established in 31.1 §14: a myth earns a chapter only if it produces a specific, reproducible wrong decision. This one produces a decoder with a wrong default case, and §10 measured what that costs in each of the three shapes it takes.