CXL 3.0 Peer-to-Peer

21.1 built a fabric that reaches past one switch level. This chapter uses it for the thing a fabric makes possible and a tree does not: two devices exchanging data without the host in the middle.

A GPU reading from an accelerator's memory on a CXL 2.0 topology does it in two hops through the host — device to host memory, host memory to device — with a copy in between. Both legs cross the fabric, and the data lands in host memory it never needed to touch. Removing that is worth a great deal, and it removes rather more than the copy.

The host was doing work you were not paying attention to. It authorised the access. It held a coherent copy and knew when to invalidate it. It ordered the write against subsequent reads. It saw failures and reported them. Take the host out of the path and every one of those becomes somebody else's job, and this chapter is who.

1. The Engineering Problem — The Host Was Load-Bearing

The bounce is two legs and a copy, and a model that counts one leg reports a third of the real cost. Section 5.

A peer transfer still needs permission, and nothing the host does can grant it, because the host is not in the path to be asked. Section 6.

The host may still hold the line. A peer write to memory the host caches is a value the host will not see unless the fabric carries an invalidate back to it. Section 7.

The benefit is the saving times the share. A peer path three times faster is worth almost nothing on a workload that is 5% peer traffic. Section 8.

A failed peer transfer has no host in the path to report it, so the fabric has to forward the error or nobody hears. Section 9.

And ordering was the host's pipeline doing it for you. A peer write and a host read need something to separate them, and there is no longer a shared pipeline to be that thing. Section 11.

This chapter against 21.1, stated precisely. That one owns what a multi-level fabric requires. This one owns what changes when a transfer no longer passes through the host — and section 15 shows a transfer that is genuinely one leg and unsound in five other ways.

2. The One-Sentence Model

A peer-to-peer transfer is sound when the host really is out of the path, the source is authorised to reach the destination, the host learns what the peer wrote, the write is ordered against subsequent reads, failures reach a host that can report them, and a transfer that cannot go direct still goes — and every defect below is a transfer that is faster and fails one of the other five.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real peer transfer involves two device DMA engines, a fabric with routing state, a host coherence directory and a driver stack, and none of that is reproduced. What is reproduced is the decision or the arithmetic each of them has to get right.

Each model is built twice — a correct build and a broken build selected by a parameter. The broken builds here share a shape worth naming: each one assumes the host is still doing something it no longer does. Permission, coherence, ordering, error reporting — every one was free while the host was in the path, and every one becomes an explicit mechanism the moment it is not.

Figure 1 — The two legs across the top are what peer-to-peer removes. The dashed edge is what it does not: the host may still hold a copy of the line device A just wrote, and something has to tell it. Removing the host from the data path does not remove it from the coherence domain.

5. RTL 1 — The Bounce Is Two Legs And A Copy

// RTL 1 - the bounce. A transfer routed through the host crosses the fabric
// twice and lands in host memory it never needed to touch.
module host_bounce #(parameter int COUNT_ONE_LEG = 0) (
  input  logic clk, rst_n,
  input  logic        transfer, peer_capable,
  input  logic [15:0] bytes_kb, leg_ns, host_copy_ns,
  output logic [15:0] legs, reported_legs, total_ns, host_bytes_kb,
  output logic        direct,
  output logic [7:0]  n_transfers, n_bounced,
  output logic        undercount_err
);
  assign direct = peer_capable;
  // A bounced transfer is device to host and host to device: two legs plus the
  // copy in between. A peer transfer is one leg and no copy.
  assign legs = direct ? 16'd1 : 16'd2;
  // The one-leg model reports every transfer as if the host were not in it.
  assign reported_legs = (COUNT_ONE_LEG != 0) ? 16'd1 : legs;
  assign total_ns = (reported_legs * leg_ns)
                    + ((direct || (COUNT_ONE_LEG != 0)) ? 16'd0 : host_copy_ns);
  assign host_bytes_kb = (direct || (COUNT_ONE_LEG != 0)) ? 16'd0 : bytes_kb;
  // A bounced transfer reported as if it were direct.
  assign undercount_err = transfer && !direct && (reported_legs == 16'd1);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_transfers <= 8'd0; n_bounced <= 8'd0;
    end else if (transfer) begin
      n_transfers <= n_transfers + 8'd1;
      if (!direct) n_bounced <= n_bounced + 8'd1;
    end
  end
endmodule

Four transfers. A 300 ns leg and a 200 ns host copy.

Three bounced transfers, all three undercounted.

800 against 300 is the headline and the host bytes are the quieter cost. A megabyte through the bounce is a megabyte written into host memory and read back out — memory bandwidth the host paid for and a cache the transfer displaced, neither of which appears in a latency figure at all.

Row four separates the two costs. Make the host copy free and the bounce is still 600 ns against 300, because the second fabric leg is irreducible. Peer-to-peer removes a crossing, not just a memcpy, and a model that attributes the whole saving to the copy will conclude that a faster host fixes it.

Why the broken build is not a strawman. A per-transfer latency figure quoted from a device datasheet is a one-leg figure — it is what the device does, measured at the device. Composing a system estimate from device figures produces exactly this undercount, and it is the same error 20.1 §7 made about switch crossings, one level up.

6. RTL 2 — A Peer Transfer Still Needs Permission

// RTL 2 - a peer transfer still needs permission. Two devices talking directly
// bypass the host, so nothing the host does can authorise the access.
module peer_permission #(parameter int TRUST_PEERS = 0) (
  input  logic clk, rst_n,
  input  logic       request,
  input  logic [3:0] src_id, dst_owner, allowed_mask,
  output logic       same_owner, in_allowed, permitted,
  output logic [7:0] n_requests, n_denied,
  output logic       peer_escape_err
);
  logic [3:0] src_bit;
  assign src_bit    = 4'b0001 << src_id[1:0];
  assign same_owner = (src_id == dst_owner);
  // A peer may reach a region it owns, or one whose owner has explicitly
  // admitted it. The trusting build admits any device on the fabric.
  assign in_allowed = |(allowed_mask & src_bit);
  assign permitted = (TRUST_PEERS != 0) ? request
                                        : (request && (same_owner || in_allowed));
  // A peer served a region it neither owns nor was admitted to.
  assign peer_escape_err = permitted && !same_owner && !in_allowed;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_denied <= 8'd0;
    end else if (request) begin
      n_requests <= n_requests + 8'd1;
      if (!permitted) n_denied <= n_denied + 8'd1;
    end
  end
endmodule

Two refusals against none, and two peer escapes.

Two independent routes to a legal access. Owning the region is one; being explicitly admitted by its owner is the other. Row three is the mechanism peer-to-peer actually needs — a device deliberately opening a region to a named peer — and a model checking only ownership refuses every legitimate peer transfer there is.

This is 19.3 §6's argument arriving somewhere new. That chapter established that the device must enforce, because the host that would violate the boundary is the host you are asking to police it. Here the host cannot police it even if you wanted it to — it is not in the path. The enforcement point has to be the destination device or the fabric, and there is no third option.

Why the broken build is not a strawman. Trusting every device on the fabric is correct on a fabric with one tenant, which is what a single-workload accelerator cluster is. It becomes an escape the moment the fabric carries a second tenant, and the fabric is the thing that made a second tenant possible.

The enforcement point moved and nothing announced it. On a bounced transfer the host's IOMMU and page tables stood between device A and device B's memory, doing work nobody itemised as "peer authorisation" because there were no peers. Remove the host and that machinery is simply not on the path — not disabled, not bypassed, just absent — and the destination device inherits a job it was never designed for. Section 20 is what that costs in silicon.

7. RTL 3 — The Host May Still Hold The Line

// RTL 3 - the host still holds a copy. A peer write the host caches is a line
// the host must be told about, or it reads its own stale value.
module peer_coherence #(parameter int PEER_IS_INVISIBLE = 0) (
  input  logic clk, rst_n,
  input  logic       peer_write,
  input  logic       host_cached, bi_delivered,
  output logic       host_told, host_stale,
  output logic [7:0] n_writes, n_stale,
  output logic       silent_write_err
);
  // A peer write is invisible to the host unless the fabric carries an
  // invalidate back to it.
  assign host_told  = (PEER_IS_INVISIBLE != 0) ? 1'b0 : bi_delivered;
  assign host_stale = host_cached && !host_told;
  // A peer changing memory the host holds and never hearing about it.
  assign silent_write_err = peer_write && host_stale;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_writes <= 8'd0; n_stale <= 8'd0;
    end else if (peer_write) begin
      n_writes <= n_writes + 8'd1;
      if (host_stale) n_stale <= n_stale + 8'd1;
    end
  end
endmodule

Four peer writes.

One stale line in the correct build, three in the invisible-peer build.

Peer-to-peer is where back-invalidate stops being an optimisation. 21.1 §12 introduced the channel; this is the traffic pattern that requires it. A host that reads a region a peer has been writing gets its own cached value, indefinitely, with no error and no counter — the data is simply wrong and nothing in the system disagrees.

Row three is what keeps the model honest. An uncached host cannot hold a stale line, so an undelivered invalidate costs nothing. That matters because it is the common case: most memory a peer writes is memory the host is not currently caching, which is exactly why the failure is intermittent and hard to reproduce.

8. RTL 4 — The Benefit Is The Saving Times The Share

// RTL 4 - what peer-to-peer is worth. The saving is the host leg and the copy,
// and it only matters against how much of the workload is peer traffic.
module p2p_benefit #(parameter int ASSUME_ALL_PEER = 0) (
  input  logic clk, rst_n,
  input  logic        eval,
  input  logic [15:0] bounced_ns, peer_ns,
  input  logic [7:0]  peer_share_pct,
  output logic [15:0] saved_ns, blended_ns, gain_pct,
  output logic        worth_building,
  output logic [7:0]  n_evals, n_marginal,
  output logic        overclaim_err
);
  logic [31:0] b_q, g_q;
  logic [7:0] share;
  assign saved_ns = (bounced_ns > peer_ns) ? (bounced_ns - peer_ns) : 16'd0;
  // A parameter that gates the cost must gate the share it applies to.
  assign share = (ASSUME_ALL_PEER != 0) ? 8'd100 : peer_share_pct;
  assign b_q = (({16'd0, bounced_ns} * (32'd100 - {24'd0, share}))
              + ({16'd0, peer_ns} * {24'd0, share})) / 32'd100;
  assign blended_ns = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  // A blend no better than the bounce is no gain. Without the floor the
  // subtraction underflows and reports an enormous improvement.
  assign g_q = ((bounced_ns == 16'd0) || (blended_ns >= bounced_ns)) ? 32'd0
             : ((({16'd0, bounced_ns} - {16'd0, blended_ns}) * 32'd100) / {16'd0, bounced_ns});
  assign gain_pct = (g_q > 32'd65535) ? 16'hFFFF : g_q[15:0];
  assign worth_building = (gain_pct >= 16'd10);
  assign overclaim_err = eval && worth_building && (peer_share_pct < 8'd10);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_marginal <= 8'd0;
    end else if (eval) begin
      n_evals <= n_evals + 8'd1;
      if (!worth_building) n_marginal <= n_marginal + 8'd1;
    end
  end
endmodule

Seven evaluations. 800 ns bounced against 300 ns peer.

Four evaluations not worth building, and the all-peer model overclaimed twice.

This is Amdahl's law wearing a fabric. A peer path 2.7 times faster is worth 3% on a workload that is 5% peer traffic, and no amount of making the peer path faster changes that — the ceiling at 5% share is 5%. The share is the number to measure first, and it is the one nobody has, because on a system with no peer-to-peer there is no peer traffic to measure.

Row six is a design finding the mutation harness produced. A congested fabric can make the peer path slower than the bounce, and without a floor the gain calculation underflows and reports an improvement of 65,535%. Section 18 records how it was found: the survivor asked for a case the testbench had never driven, and the case turned out to break the model rather than merely the test.

The overclaim_err condition is deliberately about the real share, not the modelled one. The all-peer build substitutes 100% for the measured figure, so its own view is internally consistent — the error fires against peer_share_pct, the number that was actually observed. A cost model that replaces its input with an assumption cannot detect that it did.

9. RTL 5 — Nobody Is In The Path To Report A Failure

// RTL 5 - who reports the error. A peer transfer that fails has no host in the
// path, so the failure has to reach one by some other route.
module peer_error_path #(parameter int HOST_IN_PATH = 0) (
  input  logic clk, rst_n,
  input  logic       fault,
  input  logic       host_involved, error_forwarded,
  output logic       host_learns, reportable,
  output logic [7:0] n_faults, n_unreported,
  output logic       lost_error_err
);
  // With the host in the path it sees the failure directly. Without it, the
  // fabric has to forward the error or nobody hears.
  assign host_learns = (HOST_IN_PATH != 0) ? host_involved
                                           : (host_involved || error_forwarded);
  assign reportable = host_learns;
  // A peer failure nothing carried to a host.
  assign lost_error_err = fault && !host_learns;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_faults <= 8'd0; n_unreported <= 8'd0;
    end else if (fault) begin
      n_faults <= n_faults + 8'd1;
      if (!host_learns) n_unreported <= n_unreported + 8'd1;
    end
  end
endmodule

One unreported fault in the correct build, two in the host-in-path build.

The second row is the one that should worry an operator. No host in the transfer and no forwarding: the failure happened, both devices know, and no software anywhere will ever learn. The transfer's very advantage — that no host was involved — is what makes its failures invisible by default.

Forwarding is a mechanism the fabric has to provide, not a property that emerges. It means a peer error takes a route that no data took, to a host that was not part of the transaction, identifying two devices it may not own. That is a real amount of plumbing for an event nobody expects to happen.

10. Waveform — A Peer Transfer Against A Bounce

The bi_owed row is the point of the figure. The peer transfer is finished at cycle 1 — four cycles before the bounce — and it has created an obligation the bounce never had. The bounced transfer went through the host, so the host's own coherence machinery handled the line as a matter of course. The peer transfer skipped that, and something now has to do it explicitly.

11. RTL 6 — Ordering Was The Host's Pipeline

// RTL 6 - ordering between a peer write and a host read. Removing the host from
// the path also removes the ordering the host's own pipeline used to provide.
module peer_host_order #(parameter int NO_FENCE = 0) (
  input  logic clk, rst_n,
  input  logic       observe,
  input  logic       write_landed, fence_done, host_reads_after,
  output logic       ordered, sees_new,
  output logic [7:0] n_observes, n_torn,
  output logic       torn_read_err
);
  // A host read is guaranteed to see a peer write only once the write has
  // landed and a fence has separated them.
  assign ordered  = write_landed && fence_done;
  assign sees_new = (NO_FENCE != 0) ? write_landed : ordered;
  // A read taken after a peer write that is not ordered against it.
  assign torn_read_err = observe && host_reads_after && sees_new && !ordered;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_observes <= 8'd0; n_torn <= 8'd0;
    end else if (observe) begin
      n_observes <= n_observes + 8'd1;
      if (host_reads_after && sees_new && !ordered) n_torn <= n_torn + 8'd1;
    end
  end
endmodule

One torn read.

A bounced transfer was ordered for free. The data went through host memory, so the host's own write pipeline and cache hierarchy sequenced it against whatever the host did next. A peer transfer never enters that pipeline, and "the write has landed at the destination" is not the same statement as "a subsequent host read will observe it".

Row four is why torn_read_err requires host_reads_after. The fenceless build makes an unsound promise on every unfenced write, and it only costs anything when somebody reads. An unsound guarantee nobody depends on is a latent bug, not an active one, and the model distinguishes them because the counters an operator sees should too.

This is the hardest of the four things the host used to do, because the other three have obvious mechanisms — a table, a message, a destination — and this one is a timing relationship between two agents that no longer share anything. Section 21 asks for a count of reads not preceded by a fence for exactly that reason: the defect is an absence, and absences do not raise events.

Figure 3 — The two muted arrows at the top are what peer-to-peer removes; the brand arrow replaces them. The two dashed arrows are what it adds, and they have no counterpart in the bounced path — the host's own coherence machinery handled that line as a matter of course when the data passed through it.

12. RTL 7 — A Device Has To Know Its Peers Exist

// RTL 7 - a device has to know its peers exist. Discovery is what turns a
// fabric of reachable endpoints into a set a device can actually address.
module peer_discovery #(parameter int ASSUME_ALL_VISIBLE = 0) (
  input  logic clk, rst_n,
  input  logic       lookup,
  input  logic [3:0] discovered_mask, target_id,
  input  logic       fabric_reachable,
  output logic       known, addressable,
  output logic [7:0] n_lookups, n_unknown,
  output logic       blind_target_err
);
  logic [3:0] tbit;
  assign tbit = 4'b0001 << target_id[1:0];
  assign known = |(discovered_mask & tbit);
  // A peer must be both reachable by the fabric and known to this device.
  assign addressable = (ASSUME_ALL_VISIBLE != 0) ? fabric_reachable
                                                 : (fabric_reachable && known);
  // A transfer aimed at a peer this device never discovered.
  assign blind_target_err = lookup && addressable && !known;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_lookups <= 8'd0; n_unknown <= 8'd0;
    end else if (lookup) begin
      n_lookups <= n_lookups + 8'd1;
      if (!known) n_unknown <= n_unknown + 8'd1;
    end
  end
endmodule

Two unknown targets, one blind aim.

Reachable and known are different facts about different things. The fabric knows what it can route to; the device knows what it has been told about. Row three is a peer that exists and was discovered and whose path is currently down — not addressable, and correctly so in both builds.

Row two is the one the fabric enables and the device must refuse. A CXL 3.0 fabric routes to every endpoint on it, so fabric_reachable is true for peers this device has no business addressing. Reachability is not authorisation and it is not discovery either — it is the weakest of the three and the only one the fabric provides for free.

13. RTL 8 — Peer Traffic Is Fabric Traffic

// RTL 8 - peer traffic is fabric traffic. Removing the host from the path does
// not remove the bytes from the links they both share.
module peer_bandwidth #(parameter int PEER_IS_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        cycle_en,
  input  logic [15:0] host_gbps, peer_gbps, link_gbps,
  output logic [15:0] offered_gbps, delivered_gbps, host_share_gbps,
  output logic        congested,
  output logic [7:0]  n_cycles, n_congested,
  output logic        free_lunch_err
);
  // Peer traffic occupies the same links as host traffic unless the model
  // pretends it does not.
  assign offered_gbps = (PEER_IS_FREE != 0) ? host_gbps : (host_gbps + peer_gbps);
  assign delivered_gbps = (offered_gbps > link_gbps) ? link_gbps : offered_gbps;
  assign congested = (offered_gbps > link_gbps);
  // What the host gets is the link minus what the peers took.
  assign host_share_gbps = (peer_gbps >= delivered_gbps) ? 16'd0
                                                        : (delivered_gbps - peer_gbps);
  // A model reporting no congestion on a link that peer traffic has filled.
  assign free_lunch_err = cycle_en && !congested
                          && ((host_gbps + peer_gbps) > link_gbps);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_cycles <= 8'd0; n_congested <= 8'd0;
    end else if (cycle_en) begin
      n_cycles <= n_cycles + 8'd1;
      if ((host_gbps + peer_gbps) > link_gbps) n_congested <= n_congested + 8'd1;
    end
  end
endmodule

Four cycles on a 400 Gbps link.

Two congested cycles, and the peer-is-free model called neither.

Peer-to-peer moves bytes off the host and not off the fabric. The saving in section 5 is real — one leg instead of two — and it is a saving in link occupancy precisely because the bytes cross once. What it is not is free capacity. A workload that converts host traffic into peer traffic and then doubles the peer traffic has spent the saving and more.

Row two is the noisy-neighbour problem of 19.4 §7 arriving on a fabric. The host asked for 200 Gbps and got 100, and nothing about the host changed — a peer transfer between two other devices took the link. The host has no visibility into it at all, because the traffic it is competing with never passes through it.

14. RTL 9 — Every Peer Path Needs A Bounce Behind It

// RTL 9 - the fallback. A peer transfer that cannot be made directly must still
// happen, which means every peer path needs a bounce behind it.
module p2p_fallback #(parameter int NO_FALLBACK = 0) (
  input  logic clk, rst_n,
  input  logic       attempt,
  input  logic       peer_path_ok, host_path_ok,
  output logic       uses_peer, uses_host, completes,
  output logic [7:0] n_attempts, n_failed,
  output logic       stranded_err
);
  assign uses_peer = peer_path_ok;
  // Without a peer path the transfer falls back through the host. The
  // no-fallback build simply fails.
  assign uses_host = !peer_path_ok && host_path_ok && (NO_FALLBACK == 0);
  assign completes = uses_peer || uses_host;
  // A transfer that could have gone through the host and did not.
  assign stranded_err = attempt && !completes && host_path_ok;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_attempts <= 8'd0; n_failed <= 8'd0;
    end else if (attempt) begin
      n_attempts <= n_attempts + 8'd1;
      if (!completes) n_failed <= n_failed + 8'd1;
    end
  end
endmodule

One failure in the correct build, two without the fallback.

Peer-to-peer is an optimisation, and an optimisation that cannot be declined is a dependency. Row two is the case: a fabric path that is down, congested past a threshold, or crossing a boundary peer transfers are not permitted over — and the transfer still has to happen, through the host, slowly.

Row three is why stranded_err requires host_path_ok. With no path at all the transfer fails and nothing was stranded; there was nowhere for it to go. Stranding requires a route that existed and was not taken, which is a different failure with a different fix, and lumping them together would tell an operator to fix a fabric that is working.

Figure 4 — Only the third gate has a fallback. Discovery and authorisation failing means the transfer should not happen at all; the path being unavailable means it should happen differently. A design that treats all three the same either strands legitimate transfers or completes ones it should have refused.

15. RTL 10 — Peer-To-Peer Assembled

// RTL 10 - peer-to-peer assembled. Everything that must hold before a device
// may talk to another device without the host in the middle.
module p2p_model #(parameter int LATENCY_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       host_removed,      // the transfer really is one leg
  input  logic       peer_authorised,   // the source may reach the destination
  input  logic       host_coherent,     // the host is told what the peer wrote
  input  logic       order_defined,     // a fence separates the write from the read
  input  logic       errors_reach_host, // a peer failure is reportable
  input  logic       fallback_exists,   // a transfer that cannot go direct still goes
  output logic       sound,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_sound,
  output logic       false_sound_err
);
  assign fail_mask[0] = ~host_removed;
  assign fail_mask[1] = ~peer_authorised;
  assign fail_mask[2] = ~host_coherent;
  assign fail_mask[3] = ~order_defined;
  assign fail_mask[4] = ~errors_reach_host;
  assign fail_mask[5] = ~fallback_exists;
  // The latency-only build measures that the host is out of the path and calls
  // peer-to-peer working, which is what a benchmark shows.
  assign sound = (LATENCY_ONLY != 0) ? host_removed : (fail_mask == 6'd0);
  assign false_sound_err = evaluate && sound && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_sound <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (sound) n_sound <= n_sound + 8'd1;
    end
  end
endmodule

One sound of six, and four false claims.

The latency-only definition is what a benchmark measures, and it is right about exactly one of the six. A benchmark shows a transfer completing in 300 ns instead of 800 — which is true, reproducible, and says nothing about whether the source was allowed to read that memory, whether the host now holds a stale copy, or what happens when the fabric path is down.

16. Quantitative Reasoning

The bounce. 300 ns direct against 800 ns bounced — two 300 ns legs and a 200 ns copy. Make the copy free and the bounce is still 600 against 300, because the second fabric leg is irreducible. A megabyte through the bounce parks 1024 KB in host memory that nothing needed.

Permission. Four peer requests, two refused by the correct device and none by the trusting one — and one of the two legitimate accesses is by explicit admission rather than ownership, which is the mechanism peer-to-peer needs.

Coherence. Four peer writes, one stale line in the correct build and three with no back-invalidate channel — and the failure is silent, with no error and no counter.

Benefit. A peer path 2.7 times faster: 31% gain at 50% peer traffic, 3% at 5%. The all-peer model claims 62% regardless, and says build it on a workload where the ceiling is 5%.

The gain floor. A congested fabric making the peer path slower — 900 ns against an 800 ns bounce — underflowed to a 65,535% improvement before the floor was added. Section 18 records how that was found.

Errors. Four peer faults, one unreported with fabric forwarding and two without it. The second row is a failure both devices know about and no software ever will.

Ordering. Four observations, one torn read by the fenceless build — and a bounced transfer was ordered for free by a pipeline the peer transfer never enters.

Discovery. Four lookups, two targets unknown, one aimed at blind. Reachability is not discovery and neither is authorisation.

Bandwidth. A 400 Gbps link: 200 Gbps of host traffic and 300 of peer traffic offers 500, delivers 400, and the host gets 100 of the 200 it asked for — competing with traffic it cannot see.

Fallback. Four attempts, one failure with a fallback and two without.

The assembled model. Six properties, six configurations, one sound. The latency-only definition reported five.

17. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

The bounce. The legs, the total and the host bytes are asserted separately, and the free-copy case isolates the irreducible second leg.

chk(bTn == 16'd800, "600 ns of legs plus a 200 ns copy");
chk(bHb == 16'd64,  "and 64 KB lands in host memory");
chk(bTn == 16'd600, "with a free copy the bounce is still two legs");

Permission. Both routes to a legal access are asserted separately, and a mask admitting somebody else is asserted to refuse.

Coherence. The uncached case is asserted coherent with no invalidate delivered.

chk(cHs == 1'b0, "an uncached host holds nothing stale");
chk(iSw == 1'b0, "in either build");

Benefit. The threshold is driven exactly, and the slower-peer case is asserted to floor rather than wrap.

chk(nGp == 16'd10,  "exactly a 10 percent gain");
chk(nWb == 1'b1,    "which exactly meets the threshold");
chk(nSn == 16'd0,   "a peer path slower than the bounce saves nothing");
chk(nGp == 16'd0,   "with no gain to report");

Errors. Each route to the host learning is asserted alone.

Ordering. The unsound-promise-with-no-read case is asserted as not a torn read, separating a latent bug from an active one.

chk(nSn == 1'b1, "and the fenceless build still promises");
chk(nTe == 1'b0, "but no read follows, so nothing tears");

Discovery. Discovered-but-unreachable and reachable-but-undiscovered are asserted separately.

Bandwidth. Exactly-at-link is asserted as not oversubscribed, and the host share is asserted as an exact rate under contention.

Fallback. The no-path case is asserted as a failure that is not stranding.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 226 checks across two testbenches, 115 on the front five models and 111 on the back five, all passing on the unmutated sources.

18. Mutation Testing

Forty-four mutations were injected one at a time.

44 injected, 44 killed, after two survivors were diagnosed — and one of them found a defect in the design rather than in the test.

Survivor 1 — a boundary never driven, which turned out to break the model. Removing the floor from saved_ns survived because no evaluation had a peer path slower than the bounce. Adding that case — 900 ns peer against an 800 ns bounce, which a congested fabric produces — killed the mutation and immediately failed three unrelated assertions. The gain_pct calculation had no floor of its own, so bounced − blended underflowed in 32-bit unsigned arithmetic and reported a 65,535% improvement on a peer path that was strictly worse.

That is the useful shape. The survivor pointed at one missing stimulus; the stimulus exposed a second missing guard that no mutation had been written for. The fix was a floor on the gain and a comment saying why — and then a new mutation to cover it, because the guard would otherwise have been added and never tested.

Survivor 2 — a threshold never driven at its value. The worth-building comparison survived becoming exclusive because no evaluation produced exactly a 10% gain. Working out which peer share does — 16%, giving a 720 ns blend against an 800 ns bounce — is the "construct the boundary" discipline: the case did not exist in any natural stimulus and had to be solved for.

19. Verification Strategy

What a testbench for a real peer-to-peer path must cover.

Every threshold at exactly its value, constructed if necessary. The 10% gain threshold required solving for a peer share of 16%. A boundary that no natural stimulus produces still has to be driven, and the arithmetic to find it is part of the test.

Inputs outside the range the model expects. A peer path slower than the bounce. A host share where peers took everything. Each is where an unfloored subtraction wraps, and section 18 shows one of them finding a guard that was missing entirely.

Both routes to every disjunction. The host learns by being involved or by forwarding; a peer is permitted by owning or by admission; a transfer completes by peer path or by fallback. Six routes, each driven alone.

The cases that are correct and look like failures. An uncached host that was never invalidated. An unsound promise with no read after it. A transfer with no path at all, which fails without being stranded. A link exactly at capacity. Each trips a naive checker.

Each mask bit driven false alone. Six properties, six single-bit configurations plus the all-clear.

What a real implementation needs that these models do not have. Concurrency — two peer transfers and a host access contending for the same destination. Partial completion — a peer transfer that moved half its bytes before the path failed. Recovery — what state the destination is in after a forwarded error, and who cleans it up.

20. Synthesis and Implementation Reality

Authorisation has to live at the destination device. With no host in the path there is nowhere else to put it: the source cannot be trusted to check itself, and the fabric would need per-region policy for every endpoint. That means every device that may be a peer destination carries an admission table, which is registers, a programming interface and a fabric-manager relationship it would not otherwise need.

Back-invalidate on a peer write is a second traffic pattern through the same channel. 21.1 §20 noted the reverse path needs its own flow control to avoid a dependency cycle. Peer writes make that path busier and less predictable, because the invalidates are generated by a device rather than by the host whose cache they target.

The fence in section 11 is a real transaction, not a bit. Ordering a peer write against a host read means the write must be acknowledged at the destination and the acknowledgement must have reached whatever the host will synchronise on. That is a round trip across the fabric on the critical path of every synchronisation, which is why the model treats fence_done as a separate input rather than folding it into write_landed.

Error forwarding needs a destination that is configured, not discovered. A peer failure has to reach a host, and which host is a policy question — the owner of the source, the owner of the destination, or a management host. That mapping is state a fabric manager programs, and a fabric that has not been told routes peer errors nowhere.

The fallback path is not free to keep. Section 14's bounce has to remain implemented, tested and maintained in every driver that supports peer transfers, for a case that is rare in production and therefore rarely exercised. A fallback nobody tests is a fallback that will not work the day it is needed.

21. Silicon Observability

"Peer errors with no destination configured" is the counter to insist on. Section 9's failure is that nobody hears, and a fabric that has no error destination programmed produces no error telemetry at all — the absence is indistinguishable from no errors having occurred. A counter of unroutable errors is the only thing that distinguishes a healthy fabric from an unconfigured one.

22. Debug Lab

Symptom. An accelerator cluster is migrated from host-mediated transfers to CXL 3.0 peer-to-peer. Aggregate throughput improves by 4%. The design review predicted 30%. Nothing is broken — every transfer completes, latency per peer transfer is exactly the predicted 300 ns, and no errors are reported anywhere.

Step 1 — are the transfers actually going peer-to-peer? Read transfers by path. 97% peer, 3% bounced. The mechanism is working; almost everything is taking the direct path.

Step 2 — is the per-transfer latency right? 300 ns for peer transfers, 800 ns for the remaining bounced ones. Both match the model exactly. Section 5 is not the chapter.

Step 3 — then why is the aggregate only 4% better? Read peer bandwidth per link against host bandwidth. The links are at 96% utilisation. Peer transfers cross the fabric once instead of twice — but the workload responded to cheaper transfers by issuing many more of them, and the fabric is now the constraint.

Step 4 — what is the host getting? Host bandwidth on the shared links has fallen from 200 Gbps to 90. The host's own memory traffic is being crowded out by peer traffic it has no visibility into, and the host-side slowdown is eating most of the peer-side gain.

Step 5 — is the benefit model wrong? Re-read section 8 with measured numbers. The peer share is 97%, so the latency model predicts 60% — but the model has no bandwidth term at all. The prediction was right about the thing it modelled and silent about the thing that bound.

The finding. Not a defect. Peer-to-peer removed a fabric crossing per transfer and the workload spent the saving on more transfers, until the links saturated. Section 13's argument, at cluster scale: peer-to-peer moves bytes off the host, not off the fabric.

The fix, in order. Rate-limit peer traffic so the host retains a floor — which is 19.4 §7's arbiter argument on a fabric rather than a device. Then re-derive the benefit with a bandwidth term, because the latency-only model will make the same prediction on the next cluster.

What made this hard. Every measurement matched its prediction. The transfers were faster, the mechanism worked, and the aggregate did not move — because the model that justified the work described one resource and the system was bound by another.

23. Design Review

1. What fraction of the workload is peer traffic, measured? The gain is bounded by it, and on a system with no peer-to-peer there is nothing to measure. Section 8.

2. Does the benefit model have a bandwidth term? Section 22 exists because it usually does not.

3. Where is peer authorisation enforced, and what programs it? Not the host — it is not in the path. Section 6.

4. Does a peer write generate a back-invalidate, and is the unacknowledged count visible? Section 7, and the second half is the stale window.

5. What orders a peer write against a subsequent host read? If the answer is "the write completing", section 11.

6. Where does a peer error go, and what happens if no destination is configured? Silence that looks like health. Section 9 and section 21.

7. Is peer bandwidth counted per link alongside host bandwidth? The host cannot see its competition otherwise. Section 13.

8. What happens when the peer path is unavailable? If the answer is "the transfer fails", section 14 — and the fallback must be tested, not merely present.

9. Does a device address peers it discovered, or peers the fabric can reach? Section 12, and the fabric reaches more than it should.

10. Which of the six properties does the team believe "peer-to-peer works" means? Section 15 exists because the answer is the benchmark.

24. How This Appears In Real Engineering

An accelerator platform team does section 8 first and section 22 second, in that order and usually months apart. The latency model justifies the work; the bandwidth model explains the result. The discipline worth adopting is to build both before starting, because the peer-traffic share and the link utilisation are the two numbers that decide the outcome and only one of them is obvious.

A device team implementing a peer destination discovers that authorisation is the whole of the work. The DMA path is straightforward; the admission table, its programming interface and its relationship to a fabric manager are a subsystem, and they exist entirely because the host is no longer available to ask.

A driver team owns section 14 and will be tempted to remove the fallback once peer transfers work. The argument against is that the fallback is exercised precisely when the fabric is degraded — which is when the system most needs to keep running, and when an untested path is most likely to be the second failure.

A support engineer meets section 9. A peer transfer that failed with no error destination configured produces a workload that hangs, two devices that know what happened, and no log line anywhere. The first question worth asking is not what failed but whether anything could have reported it.

25. Common Misconceptions

"Peer-to-peer halves the transfer cost." Two legs and a copy become one leg: 800 ns to 300. And the copy is the smaller half — a free copy still leaves 600 against 300. Section 5.

"The host authorised it before, so it is still authorised." The host is not in the path to authorise anything. Section 6.

"The host is not involved, so coherence is not involved." The host may hold the line the peer just wrote. Section 7, and the failure is silent.

"A faster peer path is a faster system." Bounded by the peer-traffic share. 2.7x faster is 3% at a 5% share. Section 8.

"Peer transfers free up bandwidth." They free up host bandwidth. The bytes still cross the fabric. Sections 13 and 22.

"A peer transfer that fails will be reported." Only if the fabric forwards it to a configured destination. Otherwise both devices know and nothing else does. Section 9.

"The write landed, so the host will see it." Landing and being ordered against a subsequent read are different statements. Section 11.

"The fabric can reach it, so we can address it." Reachable, discovered and authorised are three different facts. Section 12.

"We can drop the bounce path once peer works." The fallback runs exactly when the fabric is degraded. Section 14.

"The benchmark shows it works." The benchmark shows the host is out of the path, which is one property of six. Section 15.

26. Interview Reasoning

Q. What does a device-to-device transfer cost without peer-to-peer?

Two fabric legs and a host copy — 800 ns against 300 in the model — plus the payload written into host memory and read back, which is memory bandwidth and cache displacement that no latency figure shows. The follow-up worth reaching: make the copy free and it is still 600 against 300, because the second crossing is the irreducible part.

Q. The host is no longer in the path. What did you just lose?

Authorisation, coherence, ordering and error reporting — four things the host was doing without anybody itemising them. Each becomes an explicit mechanism: an admission table at the destination, a back-invalidate the fabric carries, a fence transaction, and an error-forwarding destination somebody has to configure.

Q. Your peer path is 2.7 times faster. What is the system gain?

Bounded by the peer-traffic share. At 50% it is 31%; at 5% it is 3%, and making the peer path faster still cannot beat 5%. The share is the number to measure, and the catch is that a system without peer-to-peer generates no peer traffic to measure — so it has to be derived from the transfer pattern rather than observed.

Q. Peer transfers are working and aggregate throughput barely moved. Where do you look?

Link utilisation. Peer-to-peer removes a fabric crossing per transfer, so a workload that responds by issuing more transfers can saturate the fabric — and the host's own traffic is then competing with peer traffic it cannot see. The sharper follow-up: what was wrong with the model? Nothing, except that it had no bandwidth term.

Q. A peer transfer fails. Who finds out?

Nobody, unless the fabric forwards the error to a configured host. The follow-up that finds real gaps: what does an unconfigured fabric look like? Identical to a healthy one — zero peer errors reported — which is why a counter of errors with no destination configured is the one to insist on.

Q. Should a peer-capable driver keep the host-bounce path?

Yes, and it has to be tested. The peer path can be unavailable — a down link, congestion past a threshold, a policy boundary — and the fallback runs exactly when the fabric is degraded, which is when the system most needs to keep working. An untested fallback becomes the second failure.

27. Exercises

1. Add a bandwidth term to RTL 4 and re-derive section 22's prediction. Find the peer-traffic share at which the fabric rather than the latency becomes the constraint.

2. Extend RTL 2 so admission is per-region rather than per-device, and show what that costs in table entries at sixteen peers and sixteen regions each.

3. Give RTL 3 a bounded acknowledgement window and quantify the stale window in cycles as a function of fabric depth.

4. Model partial completion in RTL 9: a peer transfer that moved half its bytes before the path failed. Decide what the fallback must re-send.

5. Combine RTL 6 and RTL 3 and show that a fence which does not wait for the back-invalidate acknowledgement is insufficient.

6. Extend RTL 7 to a discovery protocol with a timeout, and show what a device should assume about a peer that never answers.

7. Give RTL 8 an arbiter that reserves a host floor, as 19.4 §7 does for tenants, and find the reservation that keeps the host at 150 Gbps under peer load.

8. Add an error-destination table to RTL 5 and show that an unconfigured fabric produces identical telemetry to a fault-free one.

9. Make RTL 1's leg latency a function of fabric depth from 21.1 §6, and find the depth at which the bounce and the peer transfer cost the same.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the transfer it catches that the current mask calls sound.

28. Summary

Peer-to-peer removes the host from the data path, and the host was load-bearing.

The bounce is two legs and a copy — 800 ns against 300 — and a free copy still leaves 600, because the second fabric crossing is irreducible. A megabyte through the bounce parks 1024 KB in host memory nothing needed.

Permission cannot come from the host any more, because the host is not there to be asked. Ownership or explicit admission, enforced at the destination — and the trusting build served two accesses of four that neither route allowed.

The host may still hold the line. A peer write to cached memory is a stale value the host will read indefinitely, with no error and no counter, unless the fabric carries an invalidate back.

The benefit is the saving times the share. A path 2.7 times faster is worth 31% at a 50% peer share and 3% at 5% — and the all-peer model claims 62% regardless, on a workload whose ceiling is 5%.

A slower peer path underflowed the gain to 65,535% before a floor was added — a design defect a mutation survivor asked for and a new stimulus exposed, which is the batch's clearest case of the harness finding a bug rather than a gap.

A failed peer transfer reaches nobody unless the fabric forwards it somewhere configured — and an unconfigured fabric reports zero peer errors, which is exactly what a healthy one reports.

Ordering was the host's pipeline doing it for free. A landed write and an ordered write are different statements, and the fenceless build made an unsound promise that cost nothing until somebody read.

Reachable, discovered and authorised are three facts, and the fabric provides only the weakest of them.

Peer traffic is fabric traffic. 200 Gbps of host demand and 300 of peer demand on a 400 Gbps link leaves the host 100 of the 200 it asked for, competing with traffic it cannot see — which is section 22's cluster, where every measurement matched its prediction and the aggregate moved 4%.

Every peer path needs a bounce behind it, exercised exactly when the fabric is degraded.

Being faster is one property of six. The definition a benchmark measures called five of six configurations sound when one was.

21.3 — CXL 3.0 Scalability takes this fabric past the boundary every model so far has assumed: one rack.