PCIe vs CXL — Who Owns the Data

Every chapter in this module has assumed that when a device writes host memory, the host will subsequently see the new value.

That assumption is not free, and PCIe does not provide it. Software does — by invalidating caches at the right moments — and CXL is what happens when the hardware takes the obligation back.

1. Sources, Scope, and What This Chapter Will Not Claim

2. The Question That Divides Them

PCIe answers "how do I move these bytes?" It is a transport with an addressing model, and it is very good at it.

CXL answers a different question: "who owns this data right now?" That is a coherence question, and it is not one a transport can answer, because the answer depends on state held in caches the transport cannot see.

The last-but-one row is worth pausing on. CXL is not a different wire. It reuses the PCIe physical layer and negotiates at link training which protocols will run over it. A CXL link is physically a PCIe link carrying different traffic — which is why the comparison is about semantics rather than about signalling.

3. What PCIe Actually Guarantees

PCIe guarantees that a Memory Write, once accepted, delivers its bytes to the addressed location, and that ordering rules hold between transaction classes (13.4).

It does not guarantee that a CPU reading that location sees the new bytes. Between the location and the CPU there may be one or more caches, and PCIe has no mechanism to reach into them.

So the visibility obligation belongs to software, and it takes a specific form at buffer handoff:

DEVICE-TO-HOST (the device produced data)
  device DMA-writes the buffer
  driver INVALIDATES the CPU's cached copy       ← the step §12 removes
  CPU reads and sees the device's data
 
HOST-TO-DEVICE (the host produced data)
  CPU writes the buffer
  driver FLUSHES/cleans the cache                ← the mirror-image step
  device DMA-reads and sees the CPU's data

Both directions have a software step, and both are easy to omit — the code works without it whenever the line happens not to be cached, which is most of the time during development.

This is what "PCIe is not coherent" means in practice. Not that data fails to move; it moves exactly as specified. The correctness obligation simply sits somewhere the interconnect cannot enforce it, and §12 measured what a missing step costs.

4. What Coherence Buys, and What It Costs

CXL's coherent access changes who is responsible. For coherent accesses, hardware ensures a reader observes the correct value without software invalidation.

What that buys:

The failure class in §12's third row becomes impossible. No driver step to forget, so no silent stale read.

Fine-grained sharing becomes practical. A DMA model is a bulk-transfer model: produce a buffer, hand it over, invalidate, consume. A coherent model permits an accelerator and a CPU to work on the same data structure without a handoff protocol between them.

And pointer-based structures survive the boundary. A linked structure written by a CPU can be traversed by a coherent device without marshalling it into a flat buffer.

What it costs, and this is the part PCIe engineers under-weight:

Coherence traffic. §12 measured 14,711 invalidations over the run, and in the hardware-coherent row those are snoops the interconnect carries. Coherence is not free bandwidth-wise; it is bandwidth spent on ownership rather than on data.

Protocol and verification complexity. A coherence protocol has states, transitions and races that a transport does not. The verification burden is a different order.

A larger correctness surface in hardware. The obligation moved from a driver — where it is a line of code that can be reviewed and tested — into a protocol engine, where a bug is a silicon respin.

And it is not always the right trade. A device that streams large buffers with a clean handoff, which describes most storage and network traffic, gains very little from coherence and pays for it anyway. §7 is that argument.

5. Why the Missing-Invalidation Bug Is So Dangerous

§12's third row is 2.3% of reads. The rate is the problem.

A fault that occurs on every read is found in the first functional test. A fault that occurs on 2.3% of reads, at a rate that depends on cache pressure, access pattern and timing, is not found by functional testing at all. It is found by a customer, and it is reported as "intermittent data corruption under load" — which is the least actionable bug report in existence.

And nothing reports it. Walk the layers:

Every layer behaved correctly. The bug is in a step that was supposed to happen between them and did not, and no layer's error reporting covers the gap.

This is the same shape as 25.5 §4's Direction B — a fault that returns a Successful Completion with wrong data — and it is detectable only by comparing values against expectations. §13's debugging cases are organised around that constraint.

6. The Model Difference, Drawn

The upper path has a box that software must supply. The lower path does not. That single structural difference is what §12 measured, and everything in §4's cost column is what the lower path charges for it.

7. When Each Is the Right Answer

This is the section that matters most for an engineer choosing.

PCIe's model fits when the data has a clean handoff. A storage controller fills a buffer and hands it over; a NIC receives a frame and hands it over; a GPU copy engine moves a block and signals. In every case there is a well-defined moment when ownership transfers, and one invalidation at that moment is cheap and easy to get right.

Coherence fits when there is no clean handoff. An accelerator and a CPU working on the same structure, an accelerator following pointers into host memory, or fine-grained producer-consumer where a buffer-sized handoff would dominate the work. The distinguishing question is whether you can name the moment ownership transfers. If you can, a DMA model with an invalidation is simpler and cheaper. If you cannot, software cache management becomes a correctness minefield and hardware coherence is worth its cost.

And the honest summary:

CXL does not replace PCIe, and a design that adopts coherence for a workload with a clean handoff has bought protocol complexity and snoop traffic to solve a problem it did not have.

8. RTL — The Ownership Boundary

Block 1 — a non-coherent buffer handoff, with the software step made explicit in hardware. The point is that the step exists and can be observed.

// A device-to-host buffer handoff under the non-coherent contract. The
// device cannot invalidate the CPU's cache; what it CAN do is refuse to
// signal completion until software has acknowledged the handoff protocol,
// which turns a silent omission into a visible stall.
module noncoherent_handoff #(
  parameter int unsigned NBUF = 8
)(
  input  logic clk,
  input  logic rst_n,
  // the device finished writing a buffer
  input  logic                            dma_done,
  input  logic [$clog2(NBUF)-1:0]         dma_buf,
  // software acknowledges it has invalidated its cached copy
  input  logic                            sw_invalidated,
  input  logic [$clog2(NBUF)-1:0]         sw_buf,
  // the buffer may be handed back for reuse only after BOTH
  output logic                            buf_reusable,
  output logic [$clog2(NBUF)-1:0]         reuse_buf,
  output logic [NBUF-1:0]                 awaiting_invalidate,
  output logic                            reuse_before_invalidate
);
 
  logic [NBUF-1:0] written;
 
  // The two events are INDEPENDENT and both required. Treating dma_done
  // alone as "the buffer is free" is the hardware expression of the bug
  // §12 measured: the data is in memory and the reader's cached copy is
  // still the old one.
  assign awaiting_invalidate     = written;
  assign buf_reusable            = sw_invalidated && written[sw_buf];
  assign reuse_buf               = sw_buf;
  assign reuse_before_invalidate = sw_invalidated && !written[sw_buf];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      written <= '0;
    end else begin
      // Set and clear can target the same buffer in one cycle only if the
      // acknowledgement raced the write, which the interface forbids: a
      // buffer cannot be acknowledged before it is written. That contract
      // is asserted, not assumed (P3).
      if (dma_done)                       written[dma_buf] <= 1'b1;
      if (sw_invalidated && written[sw_buf]) written[sw_buf] <= 1'b0;
    end
  end
 
endmodule

Block 2 — an ownership-tracked buffer, which is the shape coherence generalises. No protocol is implemented; the state a coherent scheme must track is made visible.

// What coherence generalises: a line has an OWNER, and a read by a
// non-owner must not proceed until ownership resolves. This module tracks
// that state for a small set of buffers. It is NOT a CXL implementation
// and models no CXL protocol behaviour (§1).
module ownership_tracker #(
  parameter int unsigned NLINE = 16
)(
  input  logic clk,
  input  logic rst_n,
  // an agent requests exclusive ownership in order to write
  input  logic                             req_excl,
  input  logic [$clog2(NLINE)-1:0]         req_line,
  input  logic                             req_is_device,   // 1 device, 0 host
  output logic                             grant_excl,
  // an agent reads
  input  logic                             req_read,
  input  logic [$clog2(NLINE)-1:0]         read_line,
  input  logic                             read_is_device,
  output logic                             read_ok,
  output logic                             read_must_wait,
  output logic                             stale_read_would_occur
);
 
  // Two bits per line: is it exclusively owned, and by whom.
  logic [NLINE-1:0] excl;
  logic [NLINE-1:0] owner_is_device;
 
  always_comb begin
    // Exclusive ownership is granted only if nobody else holds it.
    grant_excl = req_excl && (!excl[req_line] ||
                              (owner_is_device[req_line] == req_is_device));
 
    // A read by a non-owner of an exclusively-held line must WAIT for the
    // owner to release. This is the single behaviour that makes the §12
    // third row impossible: the read cannot proceed against a stale copy,
    // because the protocol will not let it.
    read_must_wait = req_read && excl[read_line] &&
                     (owner_is_device[read_line] != read_is_device);
    read_ok        = req_read && !read_must_wait;
 
    // The instrument: under a NON-coherent contract this same condition is
    // exactly when a stale read happens, silently. Exposing it is what
    // turns §12's invisible fault into an observable one.
    stale_read_would_occur = read_must_wait;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      excl <= '0; owner_is_device <= '0;
    end else begin
      if (grant_excl) begin
        excl[req_line]            <= 1'b1;
        owner_is_device[req_line] <= req_is_device;
      end
      // A read by the non-owner releases exclusivity once it has waited —
      // modelled here as the owner relinquishing on the next cycle.
      if (read_must_wait) excl[read_line] <= 1'b0;
    end
  end
 
endmodule

The contrast between the two blocks is the chapter. Block 1 tracks whether software did its job; Block 2 makes the question unnecessary by tracking ownership directly. Block 1's reuse_before_invalidate output exists because the obligation can be violated; Block 2 has no equivalent because it cannot.

9. Same-Cycle Audit

10. Invariants

// NON-COHERENT CONTRACT — the invariant is that the software step happened.
 
// C1 — a buffer is never declared reusable before it was written. Catches an
// acknowledgement for a buffer the device never filled, which usually means
// the driver and device disagree about buffer indices.
property p_no_ack_before_write;
  @(posedge clk) disable iff (!rst_n) !reuse_before_invalidate;
endproperty
 
// C2 — a buffer is reusable only after BOTH the DMA completed and software
// acknowledged. This is the hardware expression of §3's obligation: the
// device cannot invalidate the reader's cache, so it must not proceed as
// though it had.
property p_reuse_requires_both;
  @(posedge clk) disable iff (!rst_n)
    buf_reusable |-> (sw_invalidated && written[sw_buf]);
endproperty
 
// C3 — a written buffer stays marked until acknowledged. Catches a design
// that clears the flag on a timeout or a reset and silently recycles a
// buffer whose reader has not seen the data.
property p_written_persists;
  @(posedge clk) disable iff (!rst_n)
    (written[b] && !(sw_invalidated && (sw_buf == b))) |=> written[b];
endproperty
 
// OWNERSHIP TRACKER — the invariant is that a stale read cannot proceed.
 
// O1 — exclusivity is exclusive: at most one owner class per line.
property p_single_owner;
  @(posedge clk) disable iff (!rst_n)
    grant_excl |-> (!excl[req_line] || (owner_is_device[req_line] == req_is_device));
endproperty
 
// O2 — a non-owner never reads a line held exclusively by the other agent.
// This is THE property that makes §12's third row impossible.
property p_no_stale_read;
  @(posedge clk) disable iff (!rst_n)
    read_ok |-> !(excl[read_line] && (owner_is_device[read_line] != read_is_device));
endproperty
 
// O3 — a waiting read is eventually served. Without this, the coherence
// mechanism trades silent wrong data for a silent hang — which is a
// different failure, not an improvement.
// Assumption: owners release within bounded time.
property p_waiting_read_progresses;
  @(posedge clk) disable iff (!rst_n)
    read_must_wait |-> s_eventually read_ok;
endproperty
 
// SHARED — the instrument that makes the non-coherent fault visible.
// Under the non-coherent contract this condition is exactly a stale read.
// Exposing it as a counter is what turns §12's invisible 2.3% into evidence.
property p_stale_condition_reported;
  @(posedge clk) disable iff (!rst_n)
    (req_read && excl[read_line] && (owner_is_device[read_line] != read_is_device))
      |-> stale_read_would_occur;
endproperty

p_no_stale_read and p_waiting_read_progresses are a pair and neither is sufficient alone. A coherence mechanism that blocks stale reads and never unblocks them has replaced wrong data with a hang. That is a real risk of moving an obligation into hardware, and it is why §4 lists verification complexity as a cost rather than a footnote.

11. Bug Bank

12. Measured — What a Missing Invalidation Costs

Four readings.

Rows 1 and 2 are equally correct. Coherence is not more correct than software management done properly — it is more robust, because there is no step to omit.

The invalidation counts are identical. 14,711 either way. The work does not disappear when hardware does it; it changes from a driver call into snoop traffic on the interconnect. §4's cost column is this number.

Row 3 is one missing line of driver code. Same hardware as row 2, 2,379 wrong values returned, and no error at any layer (§5).

And the 2.3% is the dangerous part. A fault that occurs on every read is found immediately. One that occurs on one read in forty-three, depending on cache pressure and access pattern, is found by a customer — which is why §13's cases are built around comparing values rather than looking for errors.

13. Debugging

14. Misconceptions

"CXL is faster PCIe." Why it sounds reasonable: it is newer, it runs on the same physical layer, and newer usually means faster. What actually happens: it uses the same PCIe physical layer (§2). What it adds is a coherence and memory semantics layer, not signalling speed. What it causes: engineers expecting a bandwidth improvement from a change that provides an ownership model, and then reporting that CXL "didn't help".

"PCIe DMA is coherent because the data ends up in memory." Why it sounds reasonable: the write genuinely does reach memory, and memory is what the CPU reads. What actually happens: the CPU may read a cached copy that predates the write (§3). PCIe has no mechanism to reach into caches. What it causes: §12's third row — 2,379 stale reads with no error at any layer.

"If the invalidation were missing, everything would be corrupt." Why it sounds reasonable: a missing correctness step sounds like a total failure. What actually happens: §12 measured 2.3%, because a line is only stale if it happens to be cached when the device writes it. What it causes: the fault surviving development and being reported from the field as "intermittent corruption under load".

"Coherence eliminates the work." Why it sounds reasonable: software no longer has to invalidate, so the invalidations must have gone away. What actually happens: §12 measured 14,711 invalidations in both the coherent and the correctly-managed non-coherent row. The work moved from a driver call to snoop traffic (§4). What it causes: bandwidth budgets that omit coherence traffic entirely.

"Coherence is strictly safer." Why it sounds reasonable: it removes a step that can be forgotten. What actually happens: it replaces a wrong-data failure with a wait, and a wait that is never released is a hang (§10, §13 case 5). What it causes: a failure class nobody planned for, in hardware where fixing it is expensive.

"CXL replaces PCIe." Why it sounds reasonable: it is a superset in capability and runs on the same wire. What actually happens: it addresses workloads without a clean handoff (§7). Bulk-transfer devices with well-defined ownership transfer — most storage and networking — gain little and pay the cost anyway. What it causes: coherent attach adopted for streaming workloads, buying protocol complexity to solve a problem that did not exist.

"Our driver works, so our cache management is right." Why it sounds reasonable: it passes every test. What actually happens: the omitted invalidation is invisible whenever the line is not cached (§12), which is most of the time during development. What it causes: correctness that depends on a timing coincidence, discovered when cache pressure changes — a new workload, a faster CPU, a different allocator.

15. Understanding Check

Q1. A device DMA-writes a buffer and the CPU reads stale data 2% of the time. Nothing reports an error. Explain why every layer is behaving correctly.

Because each layer's obligation was met (§5). PCIe delivered a well-formed Memory Write to a valid address — its contract is byte delivery and ordering, and both held. The memory controller wrote memory. The CPU read a cache line it legitimately owned and had no reason to consider stale. The application received a plausible value. The obligation that was violated sits between the layers: after the device writes, somebody must invalidate the CPU's cached copy, and PCIe has no mechanism to do it (§3). The rate is low because a line is only stale if it happens to be cached when the device writes — §12 measured 2,379 of 105,488, and that low rate is what lets the bug survive testing.

Q2. §12 shows the coherent row and the correctly-managed non-coherent row both performing 14,711 invalidations. What does that tell you about what coherence actually provides?

That it provides robustness, not efficiency (§4, §12). The invalidation work is identical — it has to happen either way, because a cached copy of overwritten data must be discarded regardless of who notices. What changes is who is responsible: a driver call becomes snoop traffic on the interconnect. So the argument for coherence is never "fewer operations" — it is that there is no step a human can omit, which §12's third row shows costing 2,379 wrong values. And the argument against is that the same work now consumes interconnect bandwidth and lives in hardware where a bug is a respin.

Q3. Your team wants to move to a coherent attach to fix intermittent corruption. What question decides whether that is the right fix?

Can you name the moment ownership transfers? (§7, §13 case 6.) If the workload is produce-a-buffer, hand-it-over, consume — which describes most storage and network traffic — then there is a well-defined handoff point, one invalidation there is cheap and reviewable, and the correct fix is the missing line of driver code. Adopting coherence would buy protocol complexity and snoop traffic to solve a problem a code review would have caught. If you cannot name that moment — an accelerator and CPU working on a shared structure, or pointer-chasing into host memory — then software cache management is a correctness minefield and coherence earns its cost.

Q4. Coherence removes the stale-read failure. What failure does it introduce, and why is that not obviously an improvement?

A read that waits and is never released (§10, §13 case 5). p_no_stale_read prevents a non-owner from reading an exclusively-held line; if the owner never releases, the reader hangs. The obligation did not vanish — it moved into hardware, where the failure mode changed from wrong data to no data. That is not automatically better: a hang is more visible than silent corruption, which is an improvement, but it is now in silicon where a bug costs a respin rather than a driver patch. This is why p_no_stale_read and p_waiting_read_progresses are a pair, and why §4 lists verification burden as a genuine cost rather than a footnote.

Q5. A corruption bug disappears when you add debug printing. Why is that diagnostic rather than merely frustrating?

Because it points at cache visibility specifically (§13 case 2). §12 measured the stale rate depending on eviction behaviour — a line is only stale if it is cached at the moment the device writes it. Printing perturbs cache state and timing, changing which lines are resident and therefore changing the stale window. Most bug classes do not behave this way: a protocol violation, an RTL ownership bug or a descriptor race is not sensitive to the CPU's cache contents. A fault whose rate tracks memory pressure rather than transfer count is pointing at visibility, which narrows the search enormously (§13 case 4).

16. Module 28 Complete

Four comparisons, four assumptions removed.

Each comparison produced the same shape of answer, and it is the module's transferable result:

Every mechanism PCIe has is a trade somebody made, and the alternative is not absent — it is placed somewhere else.

AXI's observed backpressure is exact and does not survive distance; PCIe's credits survive distance and must be sized for it, and can deadlock. Ethernet's best-effort delivery cannot deadlock and requires a recovery layer; PCIe's lossless delivery needs none and can. A polled device needs almost no hardware and cannot exceed its service rate; a bus-mastering device has no such ceiling and needs 28.3 §3's entire list. And PCIe's non-coherent model is simple and correct provided software does one thing, which §12 measured costing 2,379 wrong values when it does not.

Three measurements from this module are worth carrying forward.

A configuration that sends more is not necessarily doing more work. 28.1 §12 measured a delayed-READY link sending 45% more transfers by overrunning the receiver — the fourth time this module and the last have measured a throughput gain that was actually destroyed work.

A resource increase that defers a structural failure makes it worse. 28.3 §13 measured a 2,048× buffer increase completing the same 3,124 transfers while hiding the loss past the observation window.

And a fault with a low rate is more dangerous than one with a high rate. §12's 2.3% survives testing; 100% would not.

Module 29 turns to real systems — NVMe SSDs, GPU interfaces, FPGA accelerator cards, SmartNICs, AI accelerators and data-centre fabrics — where these trades appear together in shipping designs, and where the question is no longer which mechanism is better but which combination a given product chose and why.