CXL vs PCIe

Module 27 answered questions about CXL on its own terms. Module 28 places it against everything it is confused with, and the first comparison is the one every candidate is asked: what is the difference between CXL and PCIe?

CXL is coherent PCIe. It names both sides, it is roughly true, and it is where most answers stop — no access granularity, no initiator, no latency tier, no ordering model and nothing about what happens when either of them fails.

1. The Engineering Problem — The Same Stack Decides Different Things

They share most of a stack and differ in what it decides. Five layers of six in common and five decisions of six apart is a comparison whose shared half is the half that was never in question. Section 5.

The difference is the granularity of an access. Ten sixty-four-byte reads on a two-hundred-and-fifty-six-byte transfer grain move two and a half thousand bytes for six hundred and forty — three quarters of the traffic wasted. Section 6.

A PCIe device is told; a CXL device asks. Eighty of a hundred operations set up by the host is nine hundred and sixty units of host work the device never sees, and a bus-mastering argument leaves all of it out. Section 7.

They are in different latency tiers, and a tier is not a percentage. Twelve hundred nanoseconds against a hundred and fifty is a factor of eight, which is the difference between something software can hide behind a queue and something it cannot hide at all. Section 8.

A PCIe error is a transaction; a CXL.mem error is a fault. Fifteen of fifty faults reaching software as machine checks is fifteen failures with no driver layer between them and the instruction that caused them. Section 10.

This chapter against 27.2, stated precisely. That one owns how CXL.io relates to PCIe inside the protocol — the flit layer, alternate-protocol negotiation, what is literally reused. This one owns the comparison as an argument: what a reader is actually being asked when somebody says "why not just use PCIe".

2. The One-Sentence Model

A comparison of CXL against PCIe is sound when both are named, when the access granularity is stated, when who initiates a transaction is stated, when the latency tier is stated rather than a percentage, when the ordering model is stated, and when what a failure looks like on each is stated — and "CXL is coherent PCIe" is one of those six.

3. What This Chapter Owns

What is genuinely shared is worth stating precisely, because the comparison is only interesting once the easy half is conceded.

The electricals are the same. Same signalling, same lanes, same connectors, same speeds, same retimers. A CXL device plugs into a PCIe slot because it is electrically a PCIe device.

Link training is the same. A CXL link trains as a PCIe link and then, at a specific point in training, negotiates which protocol will run over it. 27.2 builds that negotiation; here it is the boundary between the shared half and the different half.

Enumeration is the same. A CXL device appears in configuration space, has a vendor and device ID, and is discovered by the same mechanism. Its CXL-ness is advertised in DVSEC structures the host reads afterwards.

And CXL.io is PCIe. Not "like PCIe" — the transaction layer semantics are PCIe's, carried in CXL flits. A CXL device does its configuration, its interrupts and its bulk transfers exactly the way a PCIe device does.

What differs is everything above that, and the rest of this chapter is nine ways of saying what.

4. Teaching-Model Boundary

Every model in this chapter is a teaching model, not a protocol implementation. It computes the one relationship the section is about and nothing else. There is no link layer, no transaction encoding and no device anywhere in this file.

Each model is built twice from one source. A parameter selects between the measured build, which counts what the two protocols actually decide differently, and the same-thing build, which counts the shared stack as settling the question. Every section's headline number is the gap between them.

5. RTL 1 — The Same Stack Decides Different Things

Start with the shape of the confusion, because it is a real observation rather than an error.

Almost all of the stack is shared. Anybody who has looked at both will notice that, and the notice is correct. The mistake is treating a shared stack as evidence of a shared purpose: the layers are the same and what the layers carry is not, and the count that decides whether two things are the same thing is the count of decisions rather than of layers.

// RTL 1 - CXL and PCIe share most of a stack and differ in the part that
// matters. Counting the shared layers makes them look like variants of each
// other; counting what the shared layers decide makes the difference visible.
module shared_stack #(parameter int SAME_STACK_SAME_THING = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] layers_total, layers_shared, decisions_total, decisions_shared,
  output logic [15:0] shared_ok, layers_differ, shared_pct, decisions_differ,
  output logic        same_thing,
  output logic [7:0]  n_evals, n_different,
  output logic        stack_err
);
  logic [31:0] s_q;
  logic [15:0] dec_ok, true_differ;
  logic        truly_different;
  assign shared_ok = (layers_shared > layers_total) ? layers_total : layers_shared;
  assign layers_differ = layers_total - shared_ok;
  // The layers are the cheap count. What the layers decide - the access
  // granularity, the initiator, the ordering model - is the count that says
  // whether two things are the same thing.
  assign dec_ok = (decisions_shared > decisions_total) ? decisions_total : decisions_shared;
  assign true_differ = decisions_total - dec_ok;
  assign decisions_differ = (SAME_STACK_SAME_THING != 0) ? 16'd0 : true_differ;
  assign s_q = (layers_total == 16'd0) ? 32'd100
             : (({16'd0, shared_ok} * 32'd100) / {16'd0, layers_total});
  assign shared_pct = s_q[15:0];
  assign same_thing = (decisions_differ == 16'd0) && (decisions_total != 16'd0);
  assign truly_different = (true_differ != 16'd0);
  assign stack_err = evaluate && truly_different && same_thing;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_different <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_different) n_different <= n_different + 8'd1;
    end
  end
endmodule

Five layers of six shared with five decisions of six apart is eighty-three percent commonality and a comparison that has settled nothing — and the same-stack view reports one protocol.

The last case is the one that makes the point sharpest and it is counter-intuitive. A deeper stack with more sharing settles less, not more — ninety-five percent of twenty layers in common, and twenty-eight of thirty decisions apart. The commonality figure rises while the comparison's content stays entirely in the part the figure excludes, which is why "they share ninety-five percent of the stack" is a sentence that can be true and useless in the same breath.

The fourth case separates the two counts completely. A comparison with no stack named at all still reports five decisions apart, because the decisions are the comparison and the layers are context. That is the model saying the two counts are independent, and it is the argument for asking about decisions first.

The fifth case is the honest boundary. A comparison with no decisions named is not wrong about them — it has not made one, and the model reports the shared stack and declines to conclude anything. That is what an answer consisting of "they share the electricals and the link layer" actually is.

The third case is the clamp and it says something mild. Sharing layers that do not exist earns no credit; commonality is bounded by the stack that was named.

It is worth naming the six decisions the model is counting, because "decisions" is otherwise as unhelpful a word as "layers". What size is an access — section 6. Who issues it — section 7. What latency class it lands in — section 8. What ordering it guarantees and to how many parties — section 9. What happens when it fails — section 10. And how software reaches it — section 11. Six questions, and the two protocols answer all six differently while sharing the wires the answers travel on.

That is not a coincidence, it is the design. CXL was built to reuse PCIe's physical and electrical investment precisely so that it would not have to re-solve signalling, connectors, retimers and a whole manufacturing ecosystem. The sharing is deliberate and is the reason CXL was adoptable at all. Treating the deliberate reuse as evidence that nothing new was added inverts the intent, and it is the most common way this comparison goes wrong in a design review — not through ignorance, but by reading a compatibility decision as a statement about purpose.

6. RTL 2 — The Difference Is The Granularity Of An Access

The second thing, and the one every other difference follows from.

PCIe moves buffers because a small transfer costs too much. A descriptor, a doorbell, a DMA engine, a completion and an interrupt are a fixed overhead, and spending it on sixty-four bytes is absurd — so PCIe workloads are written to move kilobytes, and the protocol is optimised for exactly that.

CXL moves a cache line because a load costs nothing but its latency. There is no descriptor, no doorbell and no completion; there is an instruction, an address and a wait. That makes sixty-four bytes the natural unit, and it is why the two protocols suit different workloads rather than different speeds.

// RTL 2 - the difference is the granularity of an access. PCIe moves buffers
// because a descriptor, a doorbell and a completion cost too much to spend on
// sixty-four bytes; CXL moves a cache line because a load costs nothing but
// the latency. Everything else follows from that one number.
module access_grain #(parameter int A_TRANSFER_IS_A_TRANSFER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] bytes_wanted, accesses, pcie_grain, cxl_grain,
  output logic [15:0] pcie_moved, cxl_moved, waste_bytes, waste_pct,
  output logic        grain_fits,
  output logic [7:0]  n_evals, n_wasteful,
  output logic        grain_err
);
  logic [31:0] p_q, c_q, w_q;
  logic [15:0] true_pcie, true_cxl, true_waste;
  logic        truly_wasteful;
  // Each access drags a whole grain across the link whatever it wanted.
  assign p_q = {16'd0, accesses} * {16'd0, pcie_grain};
  assign true_pcie = (p_q > 32'd9999) ? 16'd9999 : p_q[15:0];
  assign c_q = {16'd0, accesses} * {16'd0, cxl_grain};
  assign true_cxl = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign pcie_moved = true_pcie;
  assign cxl_moved = true_cxl;
  assign true_waste = (true_pcie > bytes_wanted) ? (true_pcie - bytes_wanted) : 16'd0;
  assign waste_bytes = (A_TRANSFER_IS_A_TRANSFER != 0) ? 16'd0 : true_waste;
  assign w_q = (true_pcie == 16'd0) ? 32'd0
             : (({16'd0, true_waste} * 32'd100) / {16'd0, true_pcie});
  assign waste_pct = w_q[15:0];
  assign grain_fits = (waste_bytes == 16'd0) && (bytes_wanted != 16'd0);
  assign truly_wasteful = (true_waste != 16'd0);
  assign grain_err = evaluate && truly_wasteful && grain_fits;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_wasteful <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_wasteful) n_wasteful <= n_wasteful + 8'd1;
    end
  end
endmodule

Ten sixty-four-byte accesses on a two-hundred-and-fifty-six-byte transfer grain move two thousand five hundred and sixty bytes for the six hundred and forty wanted — three quarters of the traffic wasted — and the a-transfer-is-a-transfer view reports a fit.

Figure 1 — the one number the rest of the chapter follows from. The upper path is not a misunderstanding of bandwidth; both protocols can carry the traffic. It is a misunderstanding of what a protocol is optimised to carry, and the lower path makes it arithmetic: the same access pattern costs four times as much on the wrong grain, and a workload built around either grain will look absurd on the other.

The sixth case is where the comparison collapses and it is worth keeping. Both grains the same size makes the two protocols cost identically, and both views agree. That is the configuration the weak claim generalises from — bulk transfer, where CXL.io is PCIe and the comparison genuinely has no content.

The last case is the one the whole chapter is about. A single sixty-four-byte access on a four-kilobyte grain moves ninety-eight percent waste, which is what sparse access to a large structure looks like on a transfer protocol — and it is 27.3 section 6's argument arriving from the other direction.

The eighth case is a limit worth stating rather than hiding. Two saturated totals hide most of the difference between them: once both figures hit the model's ceiling the visible waste understates the real gap. The comparison degrades gracefully into being merely directionally right, which is the honest thing for a bounded model to do.

The fifth case bounds it at the other end: a workload that makes no accesses wastes nothing on either, and the model declines to express a preference.

The overhead that sets the transfer grain is worth counting once, because it is the reason the grain is not a tunable. A descriptor has to be written to memory. A doorbell write has to cross the link. The engine has to fetch the descriptor, which is another crossing. The data moves. A completion is written back. An interrupt is raised, taken, and the handler runs. That is five or six link crossings and a context switch of fixed cost, and it is why no amount of tuning makes a transfer protocol good at sixty-four bytes — the overhead is structural rather than parametric.

A coherent load has none of those. The instruction issues, the coherency protocol resolves the line, the data arrives. There is no descriptor to write, no engine to fetch it, no completion and no interrupt. The cost is the latency of the resolution, which is 27.8 section 10's four terms and nothing else.

That asymmetry is the entire technical content of this comparison. Every other section is a consequence: the device issues its own requests because there is no descriptor telling it what to do; the latency matters because the core is waiting rather than queuing; the ordering model is coherence's because the access is coherent; the failure is a fault because there is no driver in the path; and the software needs no port because a load is a load.

7. RTL 3 — A PCIe Device Is Told; A CXL Device Asks

The third thing, and the one that decides who has to know what.

A PCIe device executes a plan the host made. The host writes a descriptor, rings a doorbell, and the device does what the descriptor says. Bus mastering does not change this: the device moves the data, and it moves the data the host told it to move.

A CXL caching agent issues its own coherent requests. It decides it needs a line, it asks, and the coherency protocol resolves it — which is why a Type 1 or Type 2 device is an agent rather than a peripheral, and why it has obligations (snoop responses, 27.8's whole chapter) that a peripheral does not.

// RTL 3 - a PCIe device is told; a CXL device asks. On PCIe the host sets up
// a descriptor and rings a doorbell and the device executes it; on CXL an
// agent issues its own coherent requests. The difference decides who has to
// know what, and it is invisible in a block diagram.
module who_initiates #(parameter int BOTH_ARE_MASTERS = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] ops, host_setups, dev_requests, setup_cost,
  output logic [15:0] told_ops, peer_ops, setup_total, peer_pct,
  output logic        dev_is_peer,
  output logic [7:0]  n_evals, n_told,
  output logic        initiator_err
);
  logic [31:0] s_q, p_q;
  logic [15:0] setups_ok, true_told, true_peer;
  logic        truly_told;
  assign setups_ok = (host_setups > ops) ? ops : host_setups;
  assign true_told = setups_ok;
  assign true_peer = ops - setups_ok;
  assign told_ops = (BOTH_ARE_MASTERS != 0) ? 16'd0 : true_told;
  assign peer_ops = (BOTH_ARE_MASTERS != 0) ? ops : true_peer;
  // Every operation the host has to set up costs the host work the device
  // never sees, which is the cost a bus-mastering argument leaves out.
  assign s_q = {16'd0, true_told} * {16'd0, setup_cost};
  assign setup_total = (s_q > 32'd9999) ? 16'd9999 : s_q[15:0];
  assign p_q = (ops == 16'd0) ? 32'd0
             : (({16'd0, peer_ops} * 32'd100) / {16'd0, ops});
  assign peer_pct = p_q[15:0];
  assign dev_is_peer = (told_ops == 16'd0) && (ops != 16'd0);
  assign truly_told = (true_told != 16'd0);
  assign initiator_err = evaluate && truly_told && dev_is_peer;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_told <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_told) n_told <= n_told + 8'd1;
    end
  end
endmodule

Eighty of a hundred operations set up by the host is nine hundred and sixty units of host work and a fifth of the traffic peer-initiated — and the both-are-masters view reports a peer.

The last case is the boundary the model insists on. One setup operation in a hundred is enough to make "the device is a peer" false, because the question is not what fraction of traffic the device initiates — it is whether the device can act without being told. A device that needs the host for one thing needs a driver for that thing, and the driver is the cost.

The second case is what a genuine peer looks like: no host setup at all, every operation issued by the device, and the host's involvement reduced to coherence responses it would have made anyway.

The fifth case separates the cost from the structure. Setup that costs nothing to perform is still setup somebody had to write, and the model reports the eighty told operations with a zero cost beside them. The runtime cost of a descriptor is the visible part; the software that constructs it is the part that decides whether a device is usable.

The degenerate case bounds it: a comparison with no operations in it attributes nothing and names no peer.

The structural consequence is worth stating separately from the arithmetic, because it is what a system architect hears. A device that issues its own coherent requests has obligations. It must respond to snoops, it must respond within a bound the host's coherency assumes, it must track what it holds, and it must participate correctly in every ordering rule section 9 counts. A peripheral has none of those: it does what it is told and its worst failure mode is doing nothing.

That is why "both are bus masters" is the wrong frame. Bus mastering is about who moves the data, and both do. Participation is about who the protocol holds responsible, and only one does. A device that gets a snoop response wrong corrupts another agent's memory; a bus-mastering PCIe device that misbehaves corrupts its own buffer and is reset.

The third case is the shape this takes in a real design. Every operation set up by the host, on a part advertised as a coherent accelerator, is a device whose CXL.cache path exists in silicon and is never exercised — which is 27.6 section 9's over-specified Type 2 seen from the software side.

8. RTL 4 — Different Tiers, Not Different Percentages

The fourth thing, and the one where a correct-sounding phrase does real damage.

"CXL is faster" is a category error. The difference between a microsecond-class round trip and a hundred-nanosecond-class load is not a percentage improvement; it is a change in what software can do about it. A microsecond can be hidden behind a queue, which is what every DMA-based design does. A hundred nanoseconds inside a load instruction cannot be hidden at all — the core is waiting, and that is simultaneously the whole benefit and the whole constraint.

// RTL 4 - the two are in different latency tiers, and a tier is not a
// percentage. A PCIe round trip is a microsecond-class event that software
// hides with queues; a CXL load is a hundred-nanosecond-class event that
// software cannot hide because it is a load.
module latency_tier #(parameter int IT_IS_A_BIT_FASTER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] pcie_ns, cxl_ns, budget_ns, accesses,
  output logic [15:0] tier_gap, pcie_over, cxl_over, gap_pct,
  output logic        same_tier,
  output logic [7:0]  n_evals, n_tiered,
  output logic        tier_err
);
  logic [31:0] g_q;
  logic [15:0] true_gap;
  logic        truly_tiered;
  assign true_gap = (pcie_ns > cxl_ns) ? (pcie_ns - cxl_ns) : 16'd0;
  assign tier_gap = (IT_IS_A_BIT_FASTER != 0) ? 16'd0 : true_gap;
  assign pcie_over = (pcie_ns > budget_ns) ? (pcie_ns - budget_ns) : 16'd0;
  assign cxl_over  = (cxl_ns > budget_ns) ? (cxl_ns - budget_ns) : 16'd0;
  // The ratio, not the difference, is what puts them in different tiers.
  assign g_q = (cxl_ns == 16'd0) ? 32'd999
             : (({16'd0, pcie_ns} * 32'd100) / {16'd0, cxl_ns});
  assign gap_pct = (g_q > 32'd999) ? 16'd999 : g_q[15:0];
  assign same_tier = (tier_gap == 16'd0) && (accesses != 16'd0);
  assign truly_tiered = (true_gap != 16'd0) && (accesses != 16'd0);
  assign tier_err = evaluate && truly_tiered && same_tier;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_tiered <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_tiered) n_tiered <= n_tiered + 8'd1;
    end
  end
endmodule

Twelve hundred nanoseconds against a hundred and fifty is a factor of eight and a thousand-and-fifty-nanosecond gap — one of them nine hundred over a three-hundred-nanosecond budget and the other comfortably inside it.

Figure 2 — why a percentage is the wrong unit. Both paths on the left are real measurements of the same system. The upper reading keeps the existing design and makes it incrementally better, which does not justify a coherency agent. The lower reading changes what the software is allowed to do: at a hundred nanoseconds an access can sit inside a load instruction, and the queues, descriptors and completion handling that existed to hide a microsecond stop being necessary. The benefit is not the saved time; it is the deleted machinery.

The sixth case is the direction the enthusiastic answer never goes, and it is real. A coherent path slower than the transfer path — a fabric with three hops, 27.7 section 6 — puts them in the same tier or reverses them, and the model reports no gap. The comparison has a direction and it depends on the topology, not on the protocol names.

The last case is the model's own boundary stated honestly. The tier question does not need a budget. With no budget at all both paths are over it by their full latency and the gap is unchanged, because the tier is a property of the two paths against each other. A budget answers a different question — whether either of them is fast enough — and conflating the two is how "CXL is faster" becomes "CXL is fast enough".

The fifth case is the zero-guard and it is the one place accesses matter. A latency gap nobody traverses is not a tier difference, and the model declines to report one for a comparison with no traffic behind it.

The word "tier" is doing specific work and is worth pinning down. A tier is a range within which the same software structure is appropriate. Microseconds is the DMA tier: you build queues, you batch, you overlap, you take an interrupt at the end. Hundreds of nanoseconds is the load tier: you issue the access and wait, because building a queue around it would cost more than the access. Tens of nanoseconds is the cache tier, where even the load tier's structures are too expensive.

Moving between tiers changes the program; moving within one does not. That is why a forty percent improvement inside the DMA tier is worth very little — the queues stay, the batching stays, the interrupt stays, and the workload runs slightly faster. And it is why a factor of eight that crosses a tier boundary is worth a great deal even though it is "only" eight: the machinery comes out.

The third case shows what happens when the model's own ceiling interferes. A ratio past the ceiling saturates at nine hundred and ninety-nine percent, which understates a gap of six thousand — and the model reports the saturated figure rather than pretending to precision it does not have. The gap in absolute nanoseconds is still exact, which is why both are published.

9. RTL 5 — The Ordering Models Are Not Variants

The fifth thing, and the one that is hardest to notice is missing.

PCIe's ordering rules are about a producer and a consumer. Posted writes do not pass posted writes; a read flushes writes ahead of it; completions have their own rules. The purpose is that a device writing data and then setting a flag can be sure software sees them in that order.

Coherence ordering is about what every agent can observe. It is not a stronger version of producer-consumer; it is a different question — one that only arises when several agents can hold and modify the same line. An answer that carries the PCIe rules across has a rule set that is silent in exactly the places coherence exists to speak.

// RTL 5 - PCIe's ordering rules are about producer and consumer; coherence
// ordering is about what every agent can observe. An answer that carries the
// first across has a rule set with holes in exactly the places the second one
// exists to fill.
module ordering_model #(parameter int SAME_ORDERING = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] coh_rules, carried_over, agents, rule_cost,
  output logic [15:0] carried_ok, rules_missing, rules_pct, closing_cost,
  output logic        ordering_covered,
  output logic [7:0]  n_evals, n_missing,
  output logic        ordering_err
);
  logic [31:0] r_q, c_q;
  logic [15:0] true_missing;
  logic        truly_missing;
  assign carried_ok = (carried_over > coh_rules) ? coh_rules : carried_over;
  assign true_missing = coh_rules - carried_ok;
  assign rules_missing = (SAME_ORDERING != 0) ? 16'd0 : true_missing;
  // Each missing rule has to be established for every pair of agents that
  // could observe the difference.
  assign c_q = ({16'd0, true_missing} * {16'd0, agents}) * {16'd0, rule_cost};
  assign closing_cost = (c_q > 32'd9999) ? 16'd9999 : c_q[15:0];
  assign r_q = (coh_rules == 16'd0) ? 32'd100
             : (({16'd0, carried_ok} * 32'd100) / {16'd0, coh_rules});
  assign rules_pct = r_q[15:0];
  assign ordering_covered = (rules_missing == 16'd0) && (coh_rules != 16'd0);
  assign truly_missing = (true_missing != 16'd0);
  assign ordering_err = evaluate && truly_missing && ordering_covered;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_missing <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_missing) n_missing <= n_missing + 8'd1;
    end
  end
endmodule

Nine coherence rules with three carried over is six with no producer-consumer equivalent, and nine hundred and sixty to establish them across four agents — and the same-ordering view reports a covered model.

The sixth case is the scaling property and it is the reason this is not a fixed cost. Forty agents saturates the cost of establishing the rules, because each rule has to hold for every pair of agents that could observe the difference. Producer-consumer ordering is between two parties by construction; coherence ordering is between all of them, and that is the structural reason the two rule sets are different in kind rather than in strength.

The last case removes the escape route. A single agent still needs the rules — the requirement is about what the protocol guarantees, not about how many agents happen to be attached today, and a design that establishes ordering only for the agent count it shipped with has a bug waiting for the second device.

The fourth case bounds the model: an ordering model with no rules named in it reports full carry-over of an empty set, which is an unwritten model rather than a covered one.

The three rules that do carry over are worth naming, since the model counts them and the section has so far only said they exist. Posted writes not passing posted writes has a coherence analogue in the ordering of a requester's own writes. A read not passing a write to the same address has one in the ordering of a requester's accesses to the same line. And completion ordering has one in the acknowledgement ordering 27.8 section 12 describes. All three are about a single requester's own stream, which is exactly the class producer-consumer ordering was built for.

The six that do not carry over are all about other agents. What a second requester observes while a first is mid-transaction. Whether two agents can see two writes in different orders. When a line's new owner may act on it relative to when the old owner relinquished it. What a snooped agent may do between receiving a snoop and responding. None of those questions exist on a protocol with one initiator per address, which is why PCIe's rule set is silent about them rather than weaker on them.

The fifth case is the honest cost of starting from the wrong model. Nothing carried over at all — fourteen hundred and forty units to establish nine rules across four agents — is not a hypothetical: it is what a team building its first coherent agent from a PCIe background actually spends, and most of it goes on the six that have no analogue to reason from.

10. RTL 6 — A PCIe Error Is A Transaction; A CXL.mem Error Is A Fault

The sixth thing, and the one with the largest consequence for system design.

A failed PCIe transaction has a software layer under it. A completion times out, the driver notices, retries, resets the device, logs something. The failure is contained inside a subsystem that was written to expect failure.

A failed CXL.mem access is a load that did not complete. There is no driver between the instruction and the memory, because removing that layer was the point. The failure surfaces as a machine check, and the blast radius of the same physical fault is a process, a kernel, or a machine rather than a device.

// RTL 6 - a PCIe error is a transaction that failed and a driver can retry
// it. A CXL.mem error is a load that faulted, which surfaces as a machine
// check with no software layer between the failure and the instruction that
// caused it. The blast radius of the same physical fault is different.
module failure_semantics #(parameter int AN_ERROR_IS_AN_ERROR = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] faults, retryable, fatal_paths, handled_paths,
  output logic [15:0] as_txn, as_fault, unhandled, contained_pct,
  output logic        contained,
  output logic [7:0]  n_evals, n_fatal,
  output logic        fault_err
);
  logic [31:0] c_q;
  logic [15:0] retry_ok, true_fault, handled_ok, true_unhandled;
  logic        truly_fatal;
  assign retry_ok = (retryable > faults) ? faults : retryable;
  assign as_txn = retry_ok;
  // Everything that is not retryable surfaces to software as a fault rather
  // than as a failed transaction.
  assign true_fault = faults - retry_ok;
  assign as_fault = (AN_ERROR_IS_AN_ERROR != 0) ? 16'd0 : true_fault;
  assign handled_ok = (handled_paths > fatal_paths) ? fatal_paths : handled_paths;
  assign true_unhandled = fatal_paths - handled_ok;
  assign unhandled = (AN_ERROR_IS_AN_ERROR != 0) ? 16'd0 : true_unhandled;
  assign c_q = (faults == 16'd0) ? 32'd100
             : (({16'd0, retry_ok} * 32'd100) / {16'd0, faults});
  assign contained_pct = c_q[15:0];
  assign contained = (as_fault == 16'd0) && (unhandled == 16'd0) && (faults != 16'd0);
  assign truly_fatal = (true_fault != 16'd0) || (true_unhandled != 16'd0);
  assign fault_err = evaluate && truly_fatal && contained;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_fatal <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_fatal) n_fatal <= n_fatal + 8'd1;
    end
  end
endmodule

Fifty faults with thirty-five retryable and eight fatal paths of which three are handled is fifteen faults reaching software as machine checks and five fatal paths with no handler — seventy percent contained, and the an-error-is-an-error view reports containment.

The sixth case separates the two halves, and it is the one a design review should ask about. Everything retryable and no fatal path handled is a failure surface that looks contained on the first count and has eight unhandled paths on the second. Retryability and handling are independent, and a comparison that reports only the first has reported the easier half.

The last case is what the difference means at scale. Four hundred of five hundred faults surfacing as machine checks is a failure surface that mostly reaches the instruction, which is the shape of a memory-expander deployment where the physical fault rate is a property of DRAM rather than of a link.

The fourth case is the one to notice in a specification. Fatal paths enumerated on a design with no fault list reports containment of an empty set while carrying five unhandled paths — which is what a document that lists error registers and never enumerates what can go wrong actually is.

The system-design consequence is the part worth carrying out of this section, because it changes decisions that have nothing to do with protocols. Memory attached over CXL.mem has DRAM's fault rate and memory's blast radius. A correctable error is corrected; an uncorrectable one on a line a process is reading is a machine check against that process, and on a line the kernel is reading it is worse. That is the same failure model as a DIMM, which is well understood — and it is a much larger commitment than attaching a device.

Which is why poisoning and error containment matter so much more on CXL than on PCIe. A poisoned line lets the failure be delivered to whoever actually consumes the data rather than to whoever happened to be running, and it is the mechanism that keeps a memory fault from becoming a machine fault. A comparison that omits failure semantics omits the reason those mechanisms exist.

The second case is the containment target and it is achievable. Everything retryable and every fatal path handled is the PCIe failure model working as designed, and it is what a CXL deployment has to reconstruct deliberately — because it does not come for free the way it does when a driver sits in the path.

11. RTL 7 — The Software Models Are Not Variants Either

The seventh thing, and the one that decides adoption rather than performance.

A PCIe device is reached through a driver. Somebody writes it, somebody maintains it, it lives in a kernel, and every application that wants the device goes through an API that driver defines.

CXL memory is reached with a load. It appears as a NUMA node; an allocation policy decides what goes there; no driver sits in the path. Software that already uses loads and stores needs no porting at all, which is the whole of CXL.mem's adoption argument and is a genuinely different kind of claim from a performance one.

// RTL 7 - the software models are not variants of each other. A PCIe device
// is reached through a driver; CXL memory is reached with a load. One of
// those is a software project and the other is an allocation policy, and a
// comparison that skips this has skipped the part that decides adoption.
module software_model #(parameter int A_DRIVER_IS_A_DRIVER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] call_sites, driver_ports, load_store_ok, port_cost,
  output logic [15:0] ported_ok, sites_left, port_total, ported_pct,
  output logic        software_ready,
  output logic [7:0]  n_evals, n_unported,
  output logic        software_err
);
  logic [31:0] p_q, t_q;
  logic [15:0] ls_ok, reachable, true_left;
  logic        truly_unported;
  // Call sites that already use loads and stores need no porting at all,
  // which is the whole of CXL.mem's software argument.
  assign ls_ok = (load_store_ok > call_sites) ? call_sites : load_store_ok;
  assign reachable = call_sites - ls_ok;
  assign ported_ok = (driver_ports > reachable) ? reachable : driver_ports;
  assign true_left = reachable - ported_ok;
  assign sites_left = (A_DRIVER_IS_A_DRIVER != 0) ? 16'd0 : true_left;
  assign t_q = {16'd0, ported_ok} * {16'd0, port_cost};
  assign port_total = (t_q > 32'd9999) ? 16'd9999 : t_q[15:0];
  assign p_q = (call_sites == 16'd0) ? 32'd100
             : ((({16'd0, ls_ok} + {16'd0, ported_ok}) * 32'd100) / {16'd0, call_sites});
  assign ported_pct = p_q[15:0];
  assign software_ready = (sites_left == 16'd0) && (call_sites != 16'd0);
  assign truly_unported = (true_left != 16'd0);
  assign software_err = evaluate && truly_unported && software_ready;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_unported <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_unported) n_unported <= n_unported + 8'd1;
    end
  end
endmodule

Two hundred call sites with a hundred and twenty already loads and stores and forty ported is forty sites on neither model — eighty percent of the software reachable — and the a-driver-is-a-driver view reports readiness.

The last case is the argument in its pure form and it is why the section exists. Software that is already all loads and stores needs nothing: no porting, no cost, every call site reachable as it stands. That is not a rhetorical flourish — it is the actual situation for a memory-bound application that wants more capacity, and it is the reason CXL.mem adoption looks nothing like CXL.cache adoption.

The fifth case is the other end. Nothing already using loads and stores leaves a hundred and sixty sites and a fifth of the software reachable, which is what a device-centric model costs when the application was written around a driver.

The degenerate case bounds it: a comparison with no software counted reports everything reachable, which is an uncounted code base rather than a ready one.

There is a caveat that belongs next to the argument, because the argument is strong enough to be over-applied. Needing no porting is not the same as needing no changes. CXL memory arrives as a NUMA node with a different latency, so an application that allocates blindly will place hot data there and run slower — 27.6 section 7's software half. The work is placement policy rather than a driver, which is a much smaller project, and it is not zero.

The distinction that matters is who does the work. A driver is a kernel component with a maintainer, a release cycle and a compatibility surface. A placement policy is a configuration decision, sometimes a library call, occasionally an allocator change. Both are engineering, and only one of them is a product.

The third case is the clamp and it makes a small point precisely. Claims larger than the call-site count are bounded by the sites that exist, because porting a call site that is not there is not readiness — and on a large code base the count of call sites is itself an estimate that nobody has usually made.

12. RTL 8 — The Comparison Has A Direction

The eighth thing, and the one an enthusiastic answer never goes.

A workload that moves whole buffers on a schedule it controls is a PCIe workload. Bulk data movement, a streaming accelerator, a network interface pushing packets — all of them are served exactly as well by a transfer protocol, and putting them on CXL buys a coherency agent they will not use, at a silicon premium that is real.

// RTL 8 - the comparison has a direction the enthusiastic answer never goes.
// A workload that moves whole buffers on a schedule it controls is a PCIe
// workload, and putting it on CXL buys a coherency agent it will not use.
module when_pcie_wins #(parameter int CXL_IS_THE_UPGRADE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] workloads, bulk_workloads, cxl_silicon, pcie_silicon,
  output logic [15:0] bulk_ok, cxl_suited, wasted_silicon, suited_pct,
  output logic        cxl_everywhere,
  output logic [7:0]  n_evals, n_wasted,
  output logic        fit_err
);
  logic [31:0] s_q, w_q;
  logic [15:0] true_suited, true_wasted, extra;
  logic        truly_wasted;
  assign bulk_ok = (bulk_workloads > workloads) ? workloads : bulk_workloads;
  // A bulk workload is not suited to CXL, whatever the datasheet comparison
  // says about bandwidth.
  assign true_suited = workloads - bulk_ok;
  assign cxl_suited = (CXL_IS_THE_UPGRADE != 0) ? workloads : true_suited;
  assign extra = (cxl_silicon > pcie_silicon) ? (cxl_silicon - pcie_silicon) : 16'd0;
  assign w_q = {16'd0, bulk_ok} * {16'd0, extra};
  assign true_wasted = (w_q > 32'd9999) ? 16'd9999 : w_q[15:0];
  assign wasted_silicon = (CXL_IS_THE_UPGRADE != 0) ? 16'd0 : true_wasted;
  assign s_q = (workloads == 16'd0) ? 32'd100
             : (({16'd0, true_suited} * 32'd100) / {16'd0, workloads});
  assign suited_pct = s_q[15:0];
  assign cxl_everywhere = (wasted_silicon == 16'd0) && (workloads != 16'd0);
  assign truly_wasted = (true_wasted != 16'd0);
  assign fit_err = evaluate && truly_wasted && cxl_everywhere;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_wasted <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_wasted) n_wasted <= n_wasted + 8'd1;
    end
  end
endmodule

Twenty workloads with eight of them bulk, at a sixty-unit silicon premium, is four hundred and eighty units of silicon bought and unused — sixty percent of the workloads suited — and the CXL-is-the-upgrade view reports a clean sweep.

The fifth case is the configuration where over-specifying costs nothing, and it does exist. The same silicon either way — because the part is bought rather than built, or the coherency agent was going to be there anyway — makes CXL everywhere free, and both views agree. That is the honest case for not thinking about it.

The last case is the boundary. One bulk workload in twenty is enough to make the sweep false, at sixty units of waste, which is a small number attached to a correct statement: the claim was "everywhere", and everywhere includes the one.

The third case is the reverse of the chapter's thesis and worth reading. More bulk workloads than the set has leaves none suited — a workload population that should have stayed on PCIe entirely — and the CXL-is-the-upgrade view still reports a clean sweep, which is the same failure in the opposite direction.

The workloads that genuinely belong on PCIe are worth listing, because an answer that can name them is immediately more credible than one that cannot. A network interface moves packets in bursts on its own schedule and has no use for coherent access to host structures. A storage controller moves blocks. A streaming accelerator — video encode, compression — consumes a buffer and produces a buffer. A GPU running a graphics workload moves whole textures and framebuffers. Each of those is a transfer workload by construction, and each is served exactly as well by a protocol built for transfers.

The interesting cases are the ones that are both. A GPU running a compute workload with sparse access to host-resident structures wants coherence for part of its traffic and transfer for the rest, which is why Type 2 exists and why 27.6 section 11's phase argument is the hard version of this question. A device is not on one side of this comparison; a workload is, and a device that serves several workloads may need both.

13. RTL 9 — A Migration Is Not A Port

The ninth thing, and the one that shows up in a schedule.

The link layer carries over and the protocol logic does not. A PCIe controller's physical and link layers are most of a CXL controller's; the flit layer, the coherency agent, the memory path and the bias machinery are not. And underneath both, the software model changes, which is a different team's estimate entirely and is the one most often missing from the number.

// RTL 9 - moving a design from one to the other is not a port. The link
// layer carries over, the protocol logic does not, and the software model
// changes underneath both - so a migration estimate built from the shared
// layers is built from the part that did not need estimating.
module migration_cost #(parameter int IT_IS_A_PORT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] blocks, blocks_carried, sw_sites, block_cost,
  output logic [15:0] carried_ok, blocks_fresh, hw_cost, fresh_pct,
  output logic        cheap_move,
  output logic [7:0]  n_evals, n_expensive,
  output logic        migration_err
);
  logic [31:0] h_q, f_q;
  logic [15:0] true_fresh, true_cost, sw_counted;
  logic        truly_expensive;
  assign carried_ok = (blocks_carried > blocks) ? blocks : blocks_carried;
  assign true_fresh = blocks - carried_ok;
  assign blocks_fresh = (IT_IS_A_PORT != 0) ? 16'd0 : true_fresh;
  // The hardware cost is the fresh blocks; the software sites are counted
  // separately because they are a different team's estimate entirely.
  assign h_q = {16'd0, true_fresh} * {16'd0, block_cost};
  assign true_cost = (h_q > 32'd9999) ? 16'd9999 : h_q[15:0];
  assign hw_cost = (IT_IS_A_PORT != 0) ? 16'd0 : true_cost;
  assign f_q = (blocks == 16'd0) ? 32'd0
             : (({16'd0, true_fresh} * 32'd100) / {16'd0, blocks});
  assign fresh_pct = f_q[15:0];
  // The it-is-a-port view counts the hardware and stops, which is where the
  // software model's change disappears from the estimate.
  assign sw_counted = (IT_IS_A_PORT != 0) ? 16'd0 : sw_sites;
  assign cheap_move = (blocks_fresh == 16'd0) && (sw_counted == 16'd0)
                    && (blocks != 16'd0);
  assign truly_expensive = (true_fresh != 16'd0) || (sw_sites != 16'd0);
  assign migration_err = evaluate && truly_expensive && cheap_move;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_expensive <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_expensive) n_expensive <= n_expensive + 8'd1;
    end
  end
endmodule

Thirty blocks with eighteen carried and forty-five software call sites is twelve blocks fresh at nine hundred and sixty units, forty percent of the design new, and a software change nobody has costed — and the it-is-a-port view reports a port.

The fifth case is the one that makes the section necessary rather than obvious. The hardware carries entirely and the software does not — thirty blocks of thirty reused, zero hardware cost, and forty-five software sites still to change — is reported as a port by a view that counts blocks, and it is not one. The cheapest hardware migration in this model still has a software project attached, and the it-is-a-port view cannot see it because it is not looking at that column.

The last case is the shape at the other end. Nothing carried is a fresh design wearing a migration's name, at a hundred percent new, and it is what a first CXL controller actually is for a team whose PCIe controller was licensed rather than built.

The degenerate case bounds it: a migration with no hardware counted is an uncounted one rather than a cheap one, and both views decline.

The blocks that carry and the blocks that do not are worth separating explicitly, because the ratio is the estimate. Carrying over: the PHY, the link layer's training and equalisation, the lane management, the configuration-space model, the register file infrastructure, the DFT and the test harness. Not carrying over: the flit layer, the CXL.cache path, the CXL.mem path, the coherency agent with its snoop and bias machinery, the memory controller interface, and the error and poisoning logic that section 10 argues about.

The second list is shorter and is nearly all of the verification. 27.9 section 12's argument applies directly: the reuse percentage is high and it is high on the components that were never the risk, so a migration schedule built from the block ratio is optimistic by the difference between counting blocks and counting difficulty.

The software column is the one with no owner. The hardware team estimates hardware, the software team is not usually in the room when the migration is proposed, and the call-site count arrives later as a surprise. The model publishes it as a separate output for exactly that reason — it is not part of the hardware number and it is not optional.

14. RTL 10 — A CXL-Against-PCIe Comparison Assembled

Nine sections of inputs. This one puts them in one place and makes the confident answer visible as what it is: one bit of six.

// RTL 10 - a CXL-against-PCIe comparison assembled. Nine sections of inputs,
// one summary. "CXL is coherent PCIe" is bit 0: it names both, which is more
// than nothing, and it is one sixth of a comparison.
module comparison_signoff #(parameter int NAMING_BOTH_IS_COMPARING = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic        both_named, grain_stated, initiator_stated,
  input  logic        tier_stated, ordering_stated, faults_stated,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_evals, n_sound, n_claimed,
  output logic        signoff_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~both_named;
  assign fail_mask[1] = ~grain_stated;
  assign fail_mask[2] = ~initiator_stated;
  assign fail_mask[3] = ~tier_stated;
  assign fail_mask[4] = ~ordering_stated;
  assign fail_mask[5] = ~faults_stated;
  assign conditions_met = {15'd0, both_named} + {15'd0, grain_stated}
                        + {15'd0, initiator_stated} + {15'd0, tier_stated}
                        + {15'd0, ordering_stated} + {15'd0, faults_stated};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp: conditions_met sums six one-bit values, so the quotient cannot
  // exceed a hundred and a ceiling would be unreachable code.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  // The naming view reads bit 0 and stops.
  assign claimed = (NAMING_BOTH_IS_COMPARING != 0) ? both_named : truly_sound;
  assign sound = claimed;
  assign signoff_err = evaluate && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When both protocols have been named and any one of the other five fails, the assembled model reports that the comparison is not sound and the naming view reports a comparison.

Across the eight evaluations, the assembled model calls one comparison sound and the naming view calls six of them a comparison.

The bit order is by how much of the difference each condition carries. Bit 0 is the frame. Bit 1 is the difference everything else follows from. Bit 2 is the structural consequence. Bits 3 and 4 are the two properties that decide what software can do. Bit 5 is last because failure semantics are the consequence of all of them and are the part that reaches a system architect rather than a protocol engineer.

"CXL is coherent PCIe" is bit 0, and it is not a bad sentence. It names both sides, it locates CXL correctly in the landscape, and it is the right opening. That is what makes it the right weak definition: the failure is stopping there, and the reason people stop is that the sentence sounds like a complete answer in a way that "it is a coherent interconnect" does not.

The five other bits fail independently. The granularity can be stated by somebody who thinks the difference is bandwidth. The initiator can be stated with no latency tier. The tier can be given as a percentage, which is bit 3 failing while sounding like it passed. The ordering model is the one most often absent entirely, because it is invisible unless somebody has built a coherent agent. And failure semantics are almost never in the answer, which is the bit that matters most to whoever has to run the machine.

Figure 4 — the mask ordered by how much of the difference each condition carries, which is close to the order in which an answer runs out. The first two are usually reached. The third is the one that fails while sounding like it passed, because a percentage is a number and a tier is a judgement, and a slide prefers the number. The last two are almost never in the answer, and they are the two a system architect is actually listening for.

15. Quantitative Reasoning

Eighty-three percent of the stack shared and five decisions of six apart — and at twenty layers, ninety-five percent shared with twenty-eight of thirty apart.

Two thousand five hundred and sixty bytes moved for six hundred and forty wanted, on a two-hundred-and-fifty-six-byte grain. Ninety-eight percent waste at four kilobytes.

Nine hundred and sixty units of host setup for eighty of a hundred operations, and a fifth of the traffic actually peer-initiated.

A factor of eight between the two latency paths — nine hundred nanoseconds over a three-hundred-nanosecond budget on one, and nothing over on the other.

Six coherence ordering rules of nine with no producer-consumer equivalent, and a closing cost that saturates at forty agents.

Fifteen faults of fifty reaching software as machine checks, plus five fatal paths with no handler — four hundred of five hundred at scale.

Forty call sites of two hundred on neither software model, and zero when the application already uses loads and stores.

Four hundred and eighty units of silicon bought and unused by eight bulk workloads on a sixty-unit premium.

Twelve blocks of thirty fresh and forty-five software sites, on a migration a block count calls a port.

One comparison of eight sound; the naming view counts six. The assembled model's summary, and the chapter's.

16. Assertions

The testbenches carry 539 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported nothing on either testbench, the fourth chapter running.

Both builds are asserted on every degenerate case. A comparison with no stack, no decisions, no accesses, no operations, no budget, no ordering rules, no fault list, no software counted, no workloads and no blocks.

Every clamp that an input can reach is driven past its limit exactly once. More sharing than there is to share, more decisions shared than exist, transfer and cache-line totals that both saturate, more setups than operations, a setup cost that saturates, a latency ratio that saturates, more rules carried than exist, a closing cost that saturates, more retries than faults, more handlers than fatal paths, more porting than call sites, a porting cost that saturates, more bulk workloads than workloads, a waste figure that saturates, more blocks carried than exist, and a hardware cost that saturates.

Every percentage whose numerator is clamped is asserted in the case that over-claims.

Every error output is checked in both directions in every case. Section 5's second, third and fifth cases, section 6's second, fifth and sixth, section 7's second, section 8's second and sixth, section 9's second, third and fourth, section 10's second and third, section 11's second, third and last, section 12's second and fifth, and section 13's second and third exist to assert the quiet half. Each is a case where the same-thing view is right, and a model that alarmed on them would be unusable.

17. Mutation Testing

117 mutations, 117 killed. Fifty-four against the first testbench, sixty-three against the second. The first run killed a hundred and sixteen and left one.

The single survivor was the class this batch has now closed three times: a clamp on the cache-line traffic total that no input in the stimulus could reach, because every access count in the comparison was small enough that only the transfer figure saturated. Adding one case where both totals saturate killed it — and that case turned out to be worth its place in section 6 for a reason unrelated to the mutation, because it is the configuration where the ceiling hides most of the difference the section is measuring and the model degrades into being merely directionally right.

Nothing else survived. Every dominated guard was written out rather than discovered, every clamp mutation removes the clamp rather than raising its ceiling, and every percentage with a clamped numerator is asserted in its over-claim case — three rules that each cost a survivor to learn in 27.7, 27.9 and 27.10, and that cost nothing to apply here.

18. Verification Strategy

Concede the shared stack immediately and move to the decisions. Section 5. The commonality figure is not in dispute and is not the comparison.

State the access granularity first. Section 6. Everything else follows from it.

Say who issues the transaction. Section 7. A device that needs the host for one operation needs a driver for that operation.

Give the latency as a tier, not a percentage. Section 8. The question is what software can hide.

Say what the ordering model guarantees and to how many agents. Section 9.

Say what a failure looks like on each. Section 10. A transaction and a machine check are different system-design problems.

Ask what the software does today. Section 11. Loads and stores need no porting; a driver-based application needs a project.

Name a workload that should stay on PCIe. Section 12. An answer that cannot is an advertisement.

Cost a migration in blocks and in call sites. Section 13. The second column is the one that is missing.

19. Synthesis and Implementation Reality

A CXL controller contains a PCIe controller. The physical and link layers are shared silicon, which is why vendors ship parts that do either and why the incremental area for CXL.io alone is small.

The flit layer is where the divergence starts in hardware. A fixed-size flit carrying interleaved protocol traffic is a different structure from a variable-length TLP stream, and it is the first block that does not carry over.

Alternate-protocol negotiation happens during link training, which means a CXL device that fails negotiation comes up as a working PCIe device — 27.10 section 9's most instructive bring-up failure, because everything downstream looks correct.

CXL.mem's memory appears as a host-managed device memory range described by DVSEC and decoded by the host's HDM decoders, which is 27.7 section 11's finite resource arriving in the comparison.

And the coherency agent is the expensive block, in area and far more so in verification — 27.6 section 9's argument, which is why "just use CXL" has a silicon price attached.

Retimers and reach are shared and worth a line anyway, because they are the reason the electrical compatibility is worth so much. At the signalling rates both protocols use, a cable or a long trace needs retiming, and a retimer is a part with a power budget and a failure mode. CXL inherits all of that infrastructure — the parts, the qualification, the board practice — which is an enormous amount of engineering nobody had to redo, and it is the concrete form of the sharing section 5 counts.

The host side is where the asymmetry is sharpest. A PCIe root port is a well-understood block that every server CPU has had for two decades. A CXL host port additionally needs a home agent, a directory or snoop filter, HDM decoders, and a coherency domain that extends off the package — 27.8 sections 8 and 9's structures. The device side of this comparison gets most of the attention and the host side carries more of the new silicon, which is why CXL support is a CPU generation feature rather than something a board vendor can add.

20. Silicon Observability

Free, and on paper. Which protocols a part implements, from its DVSEC structures. Section 5's shared half.

Free, once it is in a system. Whether a link negotiated CXL or PCIe. The single most valuable bring-up register read on this comparison.

Cheap. The access size distribution of a workload, from a profiler that records addresses. Section 6's whole argument.

Cheap. Whether the application already uses loads and stores. Section 11.

Moderate. Measured latency on each path, which needs the observation points 27.9 section 11 says are cheap early and expensive late. Section 8.

Expensive. The ordering rules a design actually establishes, which is a specification review rather than a measurement. Section 9.

Expensive, and usually estimated. The migration's block and call-site counts. Section 13, and the second number is the one nobody owns.

21. Debug Lab

A device is on a CXL link and performing like a PCIe device.

Step 1 — check whether it negotiated CXL at all. Section 5 and 27.2. A device that trained as PCIe works perfectly and is not a CXL device, and this is one register read.

Step 2 — check the access size distribution. Section 6. Bulk transfers over CXL.io are PCIe traffic by any measure, and will perform exactly like it.

Step 3 — check whether the device is issuing its own requests. Section 7. A coherent agent that is being driven by descriptors is not using the protocol it was bought for.

Step 4 — measure the two paths rather than assuming the tier. Section 8. On a multi-hop fabric the coherent path can be the slower one.

Step 5 — check the bias state if it is a Type 2. 27.8 section 7. Host bias makes a coherent device perform like a remote one.

Step 6 — check what the software is doing. Section 11. An application going through a driver over CXL.mem has the cost of both models and the benefit of neither.

Steps 1 to 3 are reads from existing data, and one of them is the answer most of the time.

22. Design Review

Which layers do the two share, and which decisions do they make differently?

What is the access size distribution of the workload?

Who issues the transactions — the host, the device, or both, and in what proportion?

What is the measured latency on each path, and are they in the same tier?

What does the ordering model guarantee, and to how many agents?

What does a failure look like on each — a retryable transaction or a machine check?

What does the software do today, and how many call sites change?

Which workloads in this deployment should stay on PCIe?

How many hardware blocks carry over, and how many software call sites do not?

23. How This Appears In Real Engineering

The comparison is made once, early, in a slide, and the slide decides a programme.

The most common shape is section 6 stated as bandwidth. "CXL gives us more bandwidth than PCIe" — which for the same lane count and speed is simply false, because it is the same link. What CXL gives is a different access granularity, and a programme that bought it for bandwidth measures no improvement and concludes the technology under-delivered.

The second is section 8 stated as a percentage. "CXL is about forty percent faster" turns a change of tier into an incremental gain, and an incremental gain does not justify a coherency agent. The projects that survive review are the ones where somebody said "this moves a memory access from a microsecond to a hundred nanoseconds, which means we can stop hiding it".

The third is section 11 and it is the happiest failure. A team adopts CXL.mem expecting a porting project and discovers the application needs no changes at all, because it already uses loads and stores. The comparison was made on hardware terms and the software answer was better than anybody costed.

The fourth is section 12 and it is the expensive one. Everything goes on CXL because it is the newer thing, including the bulk-transfer workloads, and the silicon premium is paid across a fleet for a capability most of it never exercises.

The pattern is that the comparison is usually made on the axis the reader already understands — bandwidth, percentage speed, component count — and every one of those axes is the one where the two protocols are most alike.

A fifth shape is worth recording because it is the one that produces a good outcome from a bad comparison. A team adopts CXL for the wrong stated reason and the workload happens to suit it. The slide said bandwidth, the real benefit was granularity, the memory-bound application got its capacity, and everybody concluded the bandwidth argument was correct. That is a success and a mis-learned lesson at the same time, and the next programme reaches for the same argument on a workload where it is not true.

The check that separates the two is section 6's access-size distribution, and it costs a profiler run. A programme that has that number knows which of its reasons was the real one, and a programme that does not will attribute the outcome to whichever reason was on the slide.

24. Common Misconceptions

"CXL is coherent PCIe." A good opening and one condition of six. Section 14.

"They share ninety-five percent of the stack." True, and the comparison is entirely in the other five percent. Section 5.

"CXL has more bandwidth." Same lanes, same speed, same link. Different granularity. Section 6.

"Both are bus masters." One executes a plan; the other issues its own requests. Section 7.

"CXL is faster." By a factor that changes what software can do, which is a tier and not a percentage. Section 8.

"The ordering rules carry over." Six of nine have no equivalent. Section 9.

"An error is an error." One is a retry; one is a machine check. Section 10.

"We will need a driver either way." Not if the application already uses loads and stores. Section 11.

"Everything should be CXL." Name the workload that should not be. Section 12.

"It is a port of our PCIe controller." Twelve blocks of thirty and forty-five call sites. Section 13.

25. Interview Reasoning

"What is the difference between CXL and PCIe?" They share the electricals, the training, the enumeration and — in CXL.io — the transaction semantics. The difference is what runs above that. PCIe moves buffers because a small transfer carries a descriptor, a doorbell and a completion; CXL moves a cache line because a load carries nothing but its latency. Everything else follows: the device issues its own coherent requests rather than executing the host's plan, the latency is in a tier software cannot hide rather than one it queues around, the ordering model is about what every agent observes rather than about a producer and a consumer, and a failure is a machine check rather than a retryable transaction.

"Is CXL faster?" Not in bandwidth — same lanes, same speed. In latency it is a different tier, and the useful way to say that is what it enables: a memory access you no longer have to hide behind a queue.

"Why not use PCIe for everything?" For bulk data movement on a schedule you control, you should. The case for CXL is sparse access to a large structure, or memory the host addresses directly. Both of those are terrible on a transfer protocol and neither is about bandwidth.

"What changes in software?" That depends entirely on what it does now. CXL.mem memory is a NUMA node, so an application using loads and stores needs nothing — which is a much stronger adoption story than the hardware comparison suggests. An application built around a driver and DMA has a project.

"What happens when it fails?" That is the difference I would want a system architect to hear. A PCIe failure is a transaction a driver can retry. A CXL.mem failure is a load that faulted, with no software layer between the fault and the instruction — so the same physical defect takes out a process or a machine rather than a device.

"Would you port your PCIe controller?" The physical and link layers carry; the flit layer, the coherency agent and the memory path do not. And the software model changes underneath both, which is usually the larger number and usually missing from the estimate.

"When would you pick PCIe over CXL for a new accelerator?" Whenever the accelerator consumes and produces whole buffers on a schedule it controls — a network interface, a storage controller, a video encoder. Those have no sparse access to host structures and no memory the host wants, so the coherency agent is silicon that will never be exercised, and it costs area and a verification programme.

"Is a CXL device visible to a PCIe host?" Yes, and that is a deliberate property. Link training negotiates the protocol, and a CXL device on a host that does not support it comes up as a working PCIe device through CXL.io. That is excellent for deployment and is the single most misleading bring-up symptom there is, because everything looks correct and the device simply is not doing what it was bought for.

26. Exercises

1. A comparison names 8 layers with 7 shared and 10 decisions with 2 shared. Compute both fractions. Which would you put on a slide, and which would you defend in a review?

2. A workload makes 250 accesses of 64 bytes. Compute the traffic at grains of 64, 512 and 4,096 bytes and the waste at each. At which grain does the transfer protocol move ten times what is wanted?

3. 500 operations, 350 set up by the host at 15 units each. Compute the host cost and the peer fraction. What does the peer fraction have to be for "the device is a peer" to be true?

4. A transfer path of 900 ns and a coherent path of 120 ns against a 200 ns budget. Compute the gap, the ratio and both overruns. What topology change would put them in the same tier?

5. 14 coherence rules, 5 with a PCIe equivalent, 6 agents at 35 units per rule per pair. Compute the closing cost. What is it at 12 agents?

6. 200 faults, 120 retryable, 20 fatal paths with 12 handled. Compute both counts. Which of the two would a PCIe-shaped error document report?

7. 500 call sites, 300 already loads and stores, 60 ported at 25 units. Compute the remaining sites and the reachable fraction. What if none were loads and stores?

8. 50 workloads, 22 bulk, CXL silicon 240 against PCIe 150. Compute the waste and the suited fraction. At what premium does the waste exceed the silicon of the suited workloads?

9. 45 blocks, 28 carried, 90 software call sites, 60 units per block. Compute the hardware cost and the fresh fraction. What would you need to know to cost the software column?

10. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify its position using the rule that the ordering is by how much of the difference each condition carries.

27. Summary

They share most of a stack and differ in what it decides, and the shared fraction rises as the comparison's content stays entirely outside it.

The difference is the granularity of an access — a buffer against a cache line — and everything else follows from that one number.

A PCIe device is told; a CXL device asks, and one operation of a hundred that the host sets up is enough to make the second claim false.

They are in different latency tiers, which is a change in what software can hide rather than a percentage improvement.

The ordering models are not variants — producer-consumer is between two parties, coherence is between all of them, and the cost scales with the agent count.

A PCIe error is a transaction and a CXL.mem error is a fault, so the same physical defect has a different blast radius.

The software models are not variants either, and an application that already uses loads and stores needs no porting at all.

The comparison has a direction, and an answer that cannot name a workload that should stay on PCIe is an advertisement.

A migration is not a port — twelve blocks of thirty, plus a software column nobody owns.

Six bits, and "CXL is coherent PCIe" is one of them. One comparison of eight is sound; the naming view counts six.