PCIe Limitations

Module 1 has accumulated a list. Compute engines are peers that initiate work (1.2, 1.4). They hold local memory that becomes an island (1.5). Sharing across the island boundary means copying, and copying has a price (1.3).

Every one of those devices attaches to its host through something. In the overwhelming majority of systems, that something is PCIe. This chapter asks whether the attachment model is adequate for what Module 1 described — and the honest answer requires being much more careful than the usual one-line dismissal.

1. The One-Sentence Model

PCIe solved moving data between a host and a device, superbly and at enormous scale. What the traditional device-I/O model does not define is a way for a device to hold coherent cached copies of host memory, or for device-attached memory to appear to software as ordinary system memory. Those are missing semantics, not missing bandwidth — which is why thirty years of doubling the data rate has not supplied them.

The distinction between bandwidth and semantics is the whole chapter. A faster link moves the same kind of thing faster. It does not change what kinds of things there are.

2. What This Chapter Owns

The PCIe track on this platform owns PCIe itself — layering, TLPs, ordering rules, flow control, enumeration, LTSSM. This chapter borrows from it and does not restate it.

Deliberately not published here: any CXL protocol detail, message name, device-type definition, or coherence state. Those belong to Module 2 and later, and Chapter 2.1 is where the naming begins.

3. What PCIe Gets Right

This section is not a courtesy. A reader who leaves thinking PCIe is obsolete has learned something false, and will design badly as a result.

Point-to-point links that scale. Replacing a shared parallel bus with dedicated serial links removed both the electrical ceiling and the arbitration ceiling at once. Every device gets its own link, and adding a device does not load anybody else's — the lesson AGP foreshadowed and PCIe generalised.

A switched fabric with real topology. Devices are not on a bus; they are in a hierarchy that can be enumerated, routed and extended. This is what makes a machine with dozens of endpoints tractable.

Standardised discovery and configuration. A device announces what it is and what resources it needs; the system assigns address windows and configures it. That an operating system can boot on hardware it has never seen is a direct consequence, and it is an enormous achievement that is easy to take for granted.

A layered contract with reliability built in. Sequence numbers, CRC, and replay at the data link layer mean a corrupted packet is retransmitted without the transaction layer ever knowing. Credit-based flow control means a transmitter stops because the receiver has nowhere to put the packet, not because the fabric guessed.

MMIO and DMA as a complete control and data model. The host reaches device registers through mapped windows; the device moves bulk payload itself. This pairing is genuinely elegant, and Sections 10 to 12 build it in RTL precisely because it is worth understanding well.

An ecosystem measured in decades. Drivers, operating-system support, test equipment, silicon IP, and engineers who know it. Any successor that discards that is not a successor.

4. The Device-I/O Programming Model

Strip away the layering and the model a device presents to software is small, and it has exactly two relationships.

Relationship one — the host reaches the device. Configuration space and mapped address windows let software read and write device registers. This is the control path, and it is how everything starts.

Relationship two — the device reaches host memory. The device's own engine moves payload into and out of host memory without the CPU executing the transfer. This is the data path, and Chapter 1.2 §7 priced exactly what it does and does not remove.

Those two relationships, plus interrupts to signal completion, are enough to build essentially every device class in a modern computer. That is why the model has lasted.

What the model does not define is a third and fourth relationship: a device holding coherent cached copies of host memory as a participant in the host's coherence protocol, and host software addressing device-attached memory with the same load/store semantics it uses for DRAM. Those are the dashed edges in Figure 1, and they are what Module 1 kept reaching for.

5. Being Precise About Coherence

"PCIe is not coherent" is the most common thing said here, and stated flatly it is wrong often enough to be worth dismantling carefully. Four separate things get conflated.

One — is DMA to host memory coherent with CPU caches? On many platforms, for many regions, yes: the platform's root complex participates in the coherence protocol so that a device write to host memory is made visible to a CPU whose cache holds that line. This is a platform property and it varies. Other platforms and other regions require software to perform explicit cache maintenance around every buffer handoff. The PCIe track's PCIe vs CXL chapter measures what a missing invalidation costs when the platform does not help.

Two — may the device hold a cached copy of host memory? This is the different question, and the traditional device-I/O model does not define a mechanism for it. A device may read host memory and keep the result in a private buffer, but that buffer is not a participant in the host's coherence protocol: nothing invalidates it when a CPU writes the line, and nothing snoops it when a CPU reads. Software must therefore treat any such copy as a snapshot with a bounded lifetime.

Three — may the host address device-attached memory as ordinary memory? Software can map a device's memory window and issue loads and stores to it — that is what MMIO is. But a mapped device window is not ordinary system memory: it is typically uncached, it is not part of the coherence domain in the way DRAM is, and the ordering and access-size rules that apply to it are the device's, not the memory system's.

Four — who is responsible for correctness? In the traditional model, software. Buffer ownership, cache maintenance where required, and the ordering of publication against notification are all the driver's job, and Sections 10 to 13 are what that job looks like in hardware.

The precise statement, then: the traditional PCIe device-I/O model does not define coherent device caching of host memory, nor device-attached memory exposed as ordinary coherent system memory. Platform I/O coherency for DMA is a real and common thing, and it is a different thing.

6. Bandwidth Is Not Semantics

Here is the observation that should make the whole chapter click.

Every PCIe generation has roughly doubled the data rate. Not one of those doublings changed any of the four answers in Section 5. A device that could not participate in host coherence at 2.5 GT/s still could not at 64 GT/s, because participation is not a rate — it is a set of defined transactions and states that either exist in the model or do not.

what more bandwidth changes     bytes per second across the link
what more bandwidth cannot      what the two sides may say to each other

The distinction generalises well beyond PCIe, which is why it is worth stating as a principle:

A protocol defines a vocabulary. Increasing throughput lets you say the same things faster. It does not add words. If the thing you need to express — I am caching this line, invalidate your copy, this memory is part of the system's coherent address space — has no expression in the vocabulary, then no amount of speed supplies it.

This is also why "just use more PCIe bandwidth" is not an answer to any of Module 1's problems. Chapter 1.3 showed that the forced copies exist because two domains cannot address each other; that is a semantic limitation, and a faster link makes the copies cheaper without making them unnecessary.

7. Granularity, Latency, and What the Model Assumes

The device-I/O model is optimised for a particular shape of traffic, and it is worth naming the shape because Module 1's requirements have a different one.

The model assumes bulk. Descriptors, doorbells and completions are per-operation costs, so they amortise across a large transfer and dominate a small one — Chapter 1.3 §12 put numbers on exactly that. A model whose fixed cost is paid per operation is a model that wants operations to be large.

Memory access is fine-grained. A cache line is tens of bytes, and a program that shares a data structure with a device touches it in cache-line units, unpredictably, and often. Expressing that as a sequence of descriptor-driven bulk transfers is possible and absurd: the per-operation cost would exceed the payload by orders of magnitude.

The two are not the same workload. This is the mismatch:

A protocol built for the left column does not become suitable for the right one by getting faster. It becomes a faster protocol for the left column.

8. Module 1's List, and Where Each Item Lands

The honest summary of the whole module, requirement by requirement. The right-hand column names a direction, not a mechanism — the mechanism is Chapter 2.1 onwards.

Read the first three rows before the last four. Half of this table says reuse what exists, and that is a real engineering finding rather than a courtesy: an answer to the bottom rows that discarded the top rows would be far worse than the problem. Chapter 2.1 will show that the actual industry answer was built on exactly that observation.

9. The Hardware of the Device-I/O Model

Sections 10 to 12 build the model in RTL: a command register, a descriptor ring with the ownership bit that is its entire protocol, and the publication-order hazard that every driver author eventually meets.

10. RTL 1 — The MMIO Command Register

Purpose

To make concrete what "the host reaches the device" means, and why a mapped register is not memory.

// A memory-mapped command/status register: the smallest complete example of
// the device-I/O control model. The host writes a command word; the device
// runs; the host polls status until it reports done.
//
// The architectural point is what this interface is NOT. It looks like memory
// to software -- a load and a store -- but a write here is a COMMAND with a
// side effect, and a read is a SAMPLE of device state. Neither is a memory
// access in the sense a cache line is, which is why this region is mapped
// uncached and why ordinary memory reasoning does not apply to it.
module mmio_cmd_reg #(
  parameter int unsigned CMD_W = 16
) (
  input  logic             clk,
  input  logic             rst_n,
  input  logic             host_wr,
  input  logic [CMD_W-1:0] host_wdata,
  input  logic             host_rd,
  output logic [CMD_W-1:0] host_rdata,
  output logic             cmd_start,
  output logic [CMD_W-1:0] cmd_word,
  input  logic             dev_done,
  output logic             wr_while_busy_err
);
  logic [CMD_W-1:0] cmd_q;
  logic             busy_q;
  logic             done_q;
 
  assign cmd_word   = cmd_q;
  assign cmd_start  = host_wr && !busy_q;
  // Status is a SAMPLE: bit0 busy, bit1 done. Reading it has no side effect
  // here -- a read-to-clear status register would be a different contract and
  // a classic source of lost-completion bugs.
  assign host_rdata = { {(CMD_W-2){1'b0}}, done_q, busy_q };
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      cmd_q <= '0; busy_q <= 1'b0; done_q <= 1'b0;
      wr_while_busy_err <= 1'b0;
    end else begin
      // A command write while the device is busy is a software error. Silently
      // accepting it would corrupt an operation already in flight.
      if (host_wr && busy_q) wr_while_busy_err <= 1'b1;
 
      if (host_wr && !busy_q) begin
        cmd_q  <= host_wdata;
        busy_q <= 1'b1;
        done_q <= 1'b0;
      end else if (dev_done && busy_q) begin
        busy_q <= 1'b0;
        done_q <= 1'b1;
      end
    end
  end
endmodule

Architectural role, state, contract

The control path of Figure 1, in its smallest form. State is the latched command plus two status bits. The contract is that a command is accepted only when the device is idle, and that a write while busy is reported rather than absorbed.

Simulation evidence

Verbatim from the Icarus run:

=== EXP1: MMIO command register ===
  host writes command: cmd_start=1 status(read)=0000000000000000
  device running     : status=0000000000000001 (bit0=busy bit1=done) cmd_word=c0de
  write while busy   : wr_while_busy_err=1  <-- rejected, cmd_word still c0de
  after dev_done     : status=0000000000000010

Why this is the whole point about semantics

Look at what software had to do. It performed a store, and that store started a machine. It then performed loads in a loop until a bit changed. Neither operation obeys the rules software uses for memory: the store is not idempotent, the loads must not be cached or reordered by the compiler, and the value read is a moment-in-time sample rather than a stored value.

This is a perfectly good interface — and it is a device interface. Nothing about it generalises to "two agents sharing a data structure", which is what Module 1 kept asking for.

Synthesis and failure shape

One command register, two status flops, an error flop, and decode. The classic failure is a read-to-clear status bit combined with a polling loop that samples twice: the second read clears a completion the first read had not yet acted on, and the operation is reported as never finishing.

11. RTL 2 — The Descriptor Ring and Its Ownership Bit

Purpose

Bulk work is not submitted one register write at a time. It is submitted through a ring, and the ring's ownership bit is the entire protocol between host and device.

// A descriptor ring with an explicit per-slot OWNERSHIP bit -- the structure
// almost every device-I/O programming model is built from.
//
// Ownership is the whole contract: a slot belongs either to the host (which
// may fill it) or to the device (which may consume it), never to both. No
// hardware keeps the two views consistent, so the bit IS the protocol, and
// every bug in Section 14 is a disagreement about it.
module descriptor_ring #(
  parameter int unsigned DEPTH  = 4,
  parameter int unsigned DESC_W = 32
) (
  input  logic              clk,
  input  logic              rst_n,
  input  logic              host_post,
  input  logic [DESC_W-1:0] host_desc,
  output logic              ring_full,
  input  logic              doorbell,
  output logic              dev_valid,
  output logic [DESC_W-1:0] dev_desc,
  input  logic              dev_take,
  input  logic                     cpl_valid,
  input  logic [$clog2(DEPTH)-1:0] cpl_slot,
  output logic [DEPTH-1:0]  dev_owns_q,
  output logic [$clog2(DEPTH+1)-1:0] pending_q,
  output logic              overflow_err,
  output logic              host_wrote_dev_slot_err,
  output logic              cpl_unowned_err
);
  localparam int unsigned PW = $clog2(DEPTH);
  logic [DESC_W-1:0] mem_q [DEPTH];
  logic [PW-1:0]     head_q, tail_q;
 
  assign ring_full = (pending_q == DEPTH[$clog2(DEPTH+1)-1:0]);
  assign dev_valid = (pending_q != '0);
  assign dev_desc  = mem_q[tail_q];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      head_q <= '0; tail_q <= '0; pending_q <= '0; dev_owns_q <= '0;
      overflow_err <= 1'b0; host_wrote_dev_slot_err <= 1'b0; cpl_unowned_err <= 1'b0;
    end else begin
      // The host must not write a slot the device owns: the write silently
      // corrupts a descriptor the device is about to read.
      if (host_post && dev_owns_q[head_q]) host_wrote_dev_slot_err <= 1'b1;
      // A doorbell rung with a full ring announces work that was never posted.
      if (doorbell && ring_full)           overflow_err <= 1'b1;
      // A completion for a slot the device does not own means the two sides
      // disagree about what is outstanding.
      if (cpl_valid && !dev_owns_q[cpl_slot]) cpl_unowned_err <= 1'b1;
 
      if (host_post && !ring_full && !dev_owns_q[head_q]) begin
        mem_q[head_q]      <= host_desc;
        dev_owns_q[head_q] <= 1'b1;
        head_q             <= (head_q == PW'(DEPTH-1)) ? '0 : head_q + 1'b1;
        pending_q          <= pending_q + 1'b1;
      end
 
      if (dev_take && dev_valid)
        tail_q <= (tail_q == PW'(DEPTH-1)) ? '0 : tail_q + 1'b1;
 
      // A slot returns to the host on COMPLETION, never on take. Releasing it
      // at take would let the host refill a descriptor still being executed.
      if (cpl_valid && dev_owns_q[cpl_slot]) begin
        dev_owns_q[cpl_slot] <= 1'b0;
        pending_q            <= pending_q - 1'b1;
      end
    end
  end
endmodule

The ownership bit is the argument

There is no hardware anywhere in this design that keeps the host's view and the device's view consistent. There is a bit, and a convention that both sides honour it. That convention is the protocol, it lives partly in a driver and partly in device logic, and every failure in Section 14 is one side violating it.

Contrast that with what happens inside a coherent memory system, where hardware maintains a consistent view of a line across agents without either agent announcing anything. That is the difference Chapter 1.7 is about, and this module is what the alternative costs.

Simulation evidence

Verbatim:

=== EXP2: descriptor ring, ownership handover ===
  post desc0: owns=0001 pending=1 full=0
  after 4 posts: owns=1111 pending=4 full=1  <-- host must stop
  doorbell on a full ring: overflow_err=1
  after completing slot0: owns=1110 pending=3
  completion for an unowned slot: cpl_unowned_err=1

Three things are visible. The ownership bitmap fills as the host hands slots over. The ring reaches capacity and the host must stop — backpressure expressed as a software convention rather than a wire. And both disagreement cases are detected rather than silently applied.

Contract, synthesis, failure shape

A slot is owned by exactly one side. Completion returns a named slot, so out-of-order completion is expressible. Synthesis is a small memory, two pointers, an occupancy counter and a DEPTH-bit ownership vector. The failure modes are all disagreements: the host refilling a slot the device owns, a doorbell announcing work that was never posted, and a completion for a slot nobody was waiting on.

12. RTL 3 — Publication Order

Purpose

The single most common real bug in this model: the descriptor must be visible to the device before the doorbell that announces it.

// Models the producer-consumer publication hazard at the heart of every
// descriptor-ring driver: the descriptor must be VISIBLE to the device before
// the doorbell that announces it is.
//
// `desc_visible` is the modelled fact that the descriptor write has reached a
// point where the device will read the new value. `doorbell` is the MMIO write
// that tells the device to look. If the doorbell is observed first, the device
// fetches a descriptor slot whose contents are still stale.
//
// This module does NOT implement PCIe ordering. It models the software-visible
// consequence of getting publication order wrong, so the failure has a shape.
module doorbell_sync (
  input  logic clk,
  input  logic rst_n,
  input  logic desc_visible,
  input  logic doorbell,
  output logic fetch_ok,
  output logic fetch_stale,
  output logic stale_fetch_err
);
  logic visible_q;
 
  assign fetch_ok    = doorbell &&  (visible_q || desc_visible);
  assign fetch_stale = doorbell && !(visible_q || desc_visible);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      visible_q <= 1'b0; stale_fetch_err <= 1'b0;
    end else begin
      if (fetch_stale) stale_fetch_err <= 1'b1;
      if (desc_visible) visible_q <= 1'b1;
      if (fetch_ok)     visible_q <= 1'b0;
    end
  end
endmodule

What changes, and why. At t1 desc_visible asserts — the descriptor content has reached a point where the device will read the new value. The doorbell at t2 therefore finds something to fetch and fetch_ok asserts. At t5 the doorbell arrives with nothing published, fetch_stale asserts, and at t6 the sticky error records it.

Why the order can invert in a real system. The descriptor lives in memory and the doorbell is an MMIO write — two different paths. Whether the descriptor is guaranteed visible first depends on transaction ordering rules and on any barrier the driver placed between them. The PCIe track's completion-ordering chapter states the applicable rule with its scope: a Posted Request may not pass a previous Posted Request within a Traffic Class and under default attributes — and an attribute exists that permits passing, where a permission is not a prediction.

What a bug looks like. Not an error. The device fetches a descriptor slot containing the previous contents, executes work nobody asked for, and completes successfully. The symptom appears later and elsewhere: a completion for an operation the software does not recognise, or data written to an address from a stale descriptor. This is why the sticky stale_fetch_err exists — it converts a silent misexecution into something a regression can see.

13. Assertions

Bind-ready properties. Icarus does not support concurrent assertions, so these were not executed; the table gives the procedural check that verified each.

// M1 — a command is accepted only when the device is idle.
a_cmd_only_when_idle: assert property (@(posedge clk) disable iff (!rst_n)
  cmd_start |-> !busy_q);
 
// M2 — busy and done are never both asserted: they are a state, not two flags.
a_status_exclusive: assert property (@(posedge clk) disable iff (!rst_n)
  !(busy_q && done_q));
 
// R1 — occupancy never exceeds the ring.
a_ring_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  pending_q <= DEPTH);
 
// R2 — the host never writes a slot the device owns. This is THE ownership
// invariant; violating it corrupts a descriptor mid-execution.
a_host_respects_ownership: assert property (@(posedge clk) disable iff (!rst_n)
  (host_post && !ring_full) |-> !dev_owns_q[head_q]);
 
// R3 — a completion only returns a slot the device actually owns.
a_cpl_owned: assert property (@(posedge clk) disable iff (!rst_n)
  cpl_valid |-> dev_owns_q[cpl_slot]);
 
// R4 — taking a descriptor does not release its slot. Releasing at take is
// the "descriptor reused too early" bug.
a_take_does_not_release: assert property (@(posedge clk) disable iff (!rst_n)
  (dev_take && !cpl_valid) |=> $stable(dev_owns_q));
 
// D1 — a doorbell is never rung for work that was not published.
a_publish_before_notify: assert property (@(posedge clk) disable iff (!rst_n)
  doorbell |-> (visible_q || desc_visible));

Note what D1 is really asserting. It is a property about software behaviour expressed in hardware. In a real system nothing forces the driver to publish before it notifies — the assertion is the design's way of catching a driver that does not, which is the only defence available when the contract lives half in software.

14. Debug Lab

Each failure was produced by injecting the bug into the real module and re-running the testbench. The output is actual simulator output.

/* Driver submit path. */
ring[head].addr = buf_phys;
ring[head].len  = len;
ring[head].flags = DESC_OWN_DEVICE;
writel(head + 1, dev->doorbell);   /* no barrier between publish and notify */

  visible-then-doorbell   : fetch_ok=1 fetch_stale=0
  doorbell-before-visible : fetch_ok=0 fetch_stale=1  <-- device reads a stale slot
  stale_fetch_err=1

ring[head].flags = DESC_OWN_DEVICE;
wmb();                              /* publish before notify */
writel(head + 1, dev->doorbell);

// The device has the descriptor now, so the slot is free.
if (dev_take && dev_valid) begin
  tail_q             <= (tail_q == PW'(DEPTH-1)) ? '0 : tail_q + 1'b1;
  dev_owns_q[tail_q] <= 1'b0;          // BUG: released at TAKE
  pending_q          <= pending_q - 1'b1;
end

=== BUG: slot released at TAKE instead of at COMPLETION ===
  host posts desc into slot0        : owns=0001 pending=1
  device TAKES it (still executing) : owns=0000 pending=0  <-- slot already free
  host refills the ring             : owns=0010 host_wrote_dev_slot_err=0
  original completion arrives       : cpl_unowned_err=1

if (dev_take && dev_valid)
  tail_q <= (tail_q == PW'(DEPTH-1)) ? '0 : tail_q + 1'b1;
 
if (cpl_valid && dev_owns_q[cpl_slot]) begin
  dev_owns_q[cpl_slot] <= 1'b0;
  pending_q            <= pending_q - 1'b1;
end

/* Submit path with no check that the ring had room. */
ring[head] = desc;
head = (head + 1) % DEPTH;
writel(head, dev->doorbell);

  after 4 posts: owns=1111 pending=4 full=1  <-- host must stop
  doorbell on a full ring: overflow_err=1

if (((head + 1) % DEPTH) == READ_ONCE(tail))
    return -EAGAIN;          /* ring full: do not post, do not ring */

/* Timeout path: give up on the operation and reclaim the slot. */
if (time_after(jiffies, deadline)) {
    ring[slot].flags = DESC_OWN_HOST;    /* reclaim, with the op still live */
    free_dma_buffer(ring[slot].addr);
}

  completion for an unowned slot: cpl_unowned_err=1

15. Verifying a Device-I/O Model

The failures above share a property: the contract lives partly in software, so the design cannot enforce it — only detect violations of it. That changes what verification is for.

Assert the contract from the device's side. Every rule the driver is supposed to follow should have a hardware check that fires when it does not: ownership respected on post, doorbell only after publication, completion only for owned slots, occupancy within depth. These are not functional requirements of the device; they are tripwires for its partner.

Make every tripwire sticky. All four failures in Section 14 have narrow windows and silent consequences. A flag that a regression can read at the end is worth more than a message printed at the moment.

Test the orderings, not the values. The interesting stimulus is doorbell-before-publish, completion-after-reclaim, take-without-completion, post-into-an-owned-slot, and doorbell-on-full. Each is a legal sequence of individually legal operations.

Include reset with work outstanding. Both sides hold indices. A reset that clears one side's view and not the other's produces a ring that is permanently misaligned, and the symptom is a device that appears to work and silently drops every Nth operation.

16. How This Appears in Real Engineering

SoC and system architect

The question is which relationship a given device actually needs. A storage controller or a network device is served extremely well by the model in Figure 1 and gains little from anything richer. A compute peer that shares fine-grained state with the host is a different case, and the trade to evaluate is protocol and verification complexity against the copies and software discipline the I/O model requires. Answering "which of my devices are peers?" honestly is the whole architectural decision.

Device RTL engineer

Sections 10 to 12 are the daily surface: register maps, ring pointer logic, doorbell handling, completion generation. The distinctive discipline is that half of the contract belongs to a driver you do not control, so the design's job includes detecting partner violations. Every sticky error flag in this chapter exists for that reason, and each one costs a flip-flop.

Driver and firmware engineer

This role owns the half of the protocol that is not in hardware: publish before notify, check for room before posting, do not reclaim a resource the device can still act on, and perform cache maintenance where the platform requires it. Three of the four Debug Labs are driver bugs. The habit that prevents most of them is treating every ownership transition as a fence with an explicit direction, rather than as an assignment.

Verification engineer

Section 15 in full, with the emphasis on orderings over values. The subtle point is that a device-I/O DUT can be functionally perfect and still ship a system that fails, because the failure lives in the interaction. Testbenches that model a badly behaved driver — one that notifies early, reclaims aggressively, and over-posts — are more valuable here than testbenches that model a correct one.

Performance engineer

Per-operation cost is the metric that matters, because the model's overheads are per-operation and its efficiency is therefore a function of transfer size — Chapter 1.3 §12 is the arithmetic. Average descriptor size, doorbell rate, interrupt rate and ring occupancy together say whether a workload is being served by this model or fighting it. A workload whose natural access unit is a cache line will show a terrible ratio, and that is a signal about fit rather than about tuning.

17. Common Misconceptions

18. Interview Reasoning

19. Summary

PCIe solved a genuinely hard problem and solved it extremely well. Point-to-point links removed the shared-bus ceilings; a switched fabric made large topologies tractable; standardised discovery is why arbitrary hardware boots; layered reliability makes link errors invisible above the data link layer; and MMIO plus DMA is a complete control-and-data model for essentially every device class. None of that is obsolete, and Section 8's table says explicitly that a successor should keep it.

What the traditional device-I/O model does not define is narrower and more specific than the usual dismissal. Not "coherence" as a single undifferentiated thing — platform I/O coherency for DMA is real and common. What is missing is a device holding coherent cached copies of host memory as a protocol participant, and device-attached memory presented to software as ordinary system memory. Both are missing semantics.

That word is the chapter. A protocol defines a vocabulary; bandwidth lets you say the same things faster and adds no words. Every PCIe generation roughly doubled the rate and none of them changed what the two sides may say to each other, which is why more bandwidth has never been an answer to any of Module 1's problems.

The cost of the gap is visible in the hardware. Sections 10 to 12 built the model — a command register whose write is a command and whose read is a sample, a ring whose entire protocol is one ownership bit per slot, and a publication order that nothing enforces. No hardware in any of it keeps the two sides' views consistent. There is a convention, and both sides must honour it, and three of the four Debug Labs are one side failing to. Each failure is silent: work executed from a stale descriptor, a ring that reports empty while an operation is live, a doorbell for work never posted, a completion for a slot already recycled.

The durable form: the I/O model ends where two agents need a shared, hardware-maintained view of the same state rather than a sequence of explicit transfers between two private ones.

20. What Comes Next

Module 1 has now named the pressure from every side. Compute specialised and multiplied (1.4); local memory became a bounded island (1.5); sharing across the boundary means copying and copying has a price (1.3); and the attachment model that carries all of it defines transfers rather than shared state.

Chapter 1.7 takes the obvious next question seriously rather than rhetorically: if the missing thing is a hardware-maintained shared view, what does providing one actually buy — and what does it cost? Coherence is not free, and a chapter that presents it as an unalloyed good would be as misleading as one that presents PCIe as obsolete. That chapter is the conceptual payoff of the module, and it is where the single-writer invariant, invalidation, and the price of maintaining agreement get built and measured.

Chapter 2.1 then names the architecture the industry built on exactly the reuse-plus-additive-semantics observation in Section 8.

For the material this chapter deliberately did not restate: the PCIe track owns the protocol itself, completion ordering owns the ordering rules with their scopes, PCIe vs CXL compares the two contracts and measures what a missing invalidation costs, and DMA concepts covers the data path. The path is on the CXL tutorials index.