CXL with AI Accelerators

Module 21 ended with a fabric that can assemble a machine out of pooled parts. Module 22 asks what the largest consumer of that machinery actually wants, and the answer starts one level down: an accelerator attached to a host, and what changes when the attachment is coherent.

Today a discrete accelerator works by copying. Host data is staged into device memory over a DMA engine, the device computes, and results are staged back. Both crossings are real, both are on the critical path, and neither does any arithmetic.

Coherent attach removes them and charges you a protocol instead. This chapter is the trade, and section 15 shows an accelerator that speaks .cache perfectly and is slower than the one that copies.

1. The Engineering Problem — Coherence Is Not Free

Type 1 and Type 2 are not tiers. They are answers to one question — does the host need to see the device's memory — and getting it wrong means a device whose memory the host cannot address at all. Section 5.

The staging copy is two legs and it dominates. 4 GB in and 4 GB out at 20 GB/s is 400 ms around 500 ms of compute: a 44% overhead doing no arithmetic. Section 6.

Coherence is only worth it at the right granularity. A line that ping-pongs between host and accelerator costs more in protocol traffic than a copy would have cost outright. Section 7.

A Type 2 device's memory has a bias, and the wrong one makes every local access a coherence action against a host that holds nothing. Section 8.

The accelerator's cache is finite. A working set larger than it holds evicts, and every eviction of a coherent line is a protocol action rather than a local one. Section 9.

And a coherent link that cannot feed the compute is a compute unit that idles. Section 11.

This chapter against 21.2, stated precisely. That one owns device-to-device transfers across a fabric. This one owns host-to-accelerator attachment — the simpler topology, where the interesting question is not routing but whether coherence beats copying at all.

2. The One-Sentence Model

A coherently attached accelerator is faster than a staged one when it speaks the cache protocol, is the right device type, biases its memory toward whoever is using it, shares at a granularity coarse enough to beat a copy, sits behind a link that feeds its compute, and has its page faults in the model — and every defect below is an accelerator that is coherently attached and slower.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real accelerator attach is a coherence engine, a device cache, an address translation path, a DMA engine and a driver stack, and none of that is reproduced. What is reproduced is the arithmetic or the decision each of them has to get right.

Each model is built twice — a correct build and a broken build selected by a parameter. The broken builds here are all the accelerator model that was right before coherence existed: everything is a copy, everything is pinned, the cache always fits, the link is always fast enough. Each was a reasonable simplification of a DMA-attached device and each stops being one the moment the device can cache host memory.

Figure 1 — The upper path is two crossings of a DMA engine; the lower path is none. The dashed edge is what replaces them, and whether that is a saving depends entirely on how many lines are shared and how often — which is section 7.

5. RTL 1 — Type 1 Or Type 2 Is One Question

// RTL 1 - Type 1 against Type 2. An accelerator with no memory of its own needs
// only a coherent cache; one with memory needs the host to see it.
module device_type_choice #(parameter int ALWAYS_TYPE1 = 0) (
  input  logic clk, rst_n,
  input  logic        classify,
  input  logic [15:0] device_mem_gb,
  input  logic        needs_host_visible,
  output logic        has_memory, needs_cache, needs_hdm,
  output logic [1:0]  device_type,        // 1 or 2; 0 means unclassified
  output logic [7:0]  n_classified, n_type2,
  output logic        misclassified_err
);
  assign has_memory  = (device_mem_gb != 16'd0);
  assign needs_cache = 1'b1;
  // Host-visible device memory is what a Type 2 device has and a Type 1 does not.
  assign needs_hdm   = has_memory && needs_host_visible;
  assign device_type = (ALWAYS_TYPE1 != 0) ? 2'd1 : (needs_hdm ? 2'd2 : 2'd1);
  // A device with host-visible memory classified as having none.
  assign misclassified_err = classify && needs_hdm && (device_type == 2'd1);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_classified <= 8'd0; n_type2 <= 8'd0;
    end else if (classify) begin
      n_classified <= n_classified + 8'd1;
      if (device_type == 2'd2) n_type2 <= n_type2 + 8'd1;
    end
  end
endmodule

Four classifications.

One Type 2 device, and the always-Type-1 build misclassified it.

Both conditions are required and row three is why. A device with 64 GB of its own memory that the host has no need to address is a Type 1 device — the memory is private, the accelerator manages it, and nothing about it needs an HDM decoder. Adding one costs decoder state, address-map reservation and a host-side memory range for no benefit at all.

Row four is the mirror, and it is the request a system architect makes without noticing: make this memory host-visible on a device that has none. There is nothing to expose, so Type 1 is correct, and a model that took the request at face value would classify on the wrong input.

Why the broken build is not a strawman. Every discrete accelerator before CXL was, in effect, Type 1: it had a cache or a buffer, its memory was private, and the host reached it only through a DMA engine and a driver. Type 1 is the shape the industry already had, and Type 2 is the new thing that has to be argued for per device.

The classification is permanent in a way the rest of this chapter is not. Bias can be retuned, sharing can be restructured, a link can be widened on the next board — but whether a device presents host-managed memory is decoder state, address-map reservation and a fabric-manager relationship, decided when the part is designed. Every other property in this chapter is a knob; this one is silicon.

6. RTL 2 — The Staging Copy Is Two Legs

// RTL 2 - the copy an accelerator makes today. Without coherent attach, host data
// is staged into device memory and staged back, and both crossings are real.
module staging_cost #(parameter int COUNT_COMPUTE_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        job, coherent_attach,
  input  logic [15:0] bytes_mb, dma_mbps, compute_ms,
  output logic [15:0] stage_in_ms, stage_out_ms, total_ms, overhead_pct,
  output logic        coherent,
  output logic [7:0]  n_jobs, n_staged,
  output logic        undercount_err
);
  logic [31:0] o_q, s_q;
  assign coherent = coherent_attach;
  // The product must be computed at 32 bits: megabytes times a thousand
  // overflows a 16-bit multiply long before it overflows a millisecond count.
  assign s_q = (dma_mbps == 16'd0) ? 32'd65535
             : (({16'd0, bytes_mb} * 32'd1000) / {16'd0, dma_mbps});
  assign stage_in_ms  = coherent ? 16'd0
                      : ((s_q > 32'd65535) ? 16'hFFFF : s_q[15:0]);
  assign stage_out_ms = stage_in_ms;
  assign total_ms = (COUNT_COMPUTE_ONLY != 0) ? compute_ms
                                              : (stage_in_ms + compute_ms + stage_out_ms);
  assign o_q = (total_ms == 16'd0) ? 32'd0
             : ((({16'd0, total_ms} - {16'd0, compute_ms}) * 32'd100) / {16'd0, total_ms});
  assign overhead_pct = (o_q > 32'd65535) ? 16'hFFFF : o_q[15:0];
  // A staged job reported as if only its compute counted.
  assign undercount_err = job && !coherent && (total_ms == compute_ms)
                          && (stage_in_ms != 16'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_jobs <= 8'd0; n_staged <= 8'd0;
    end else if (job) begin
      n_jobs <= n_jobs + 8'd1;
      if (!coherent) n_staged <= n_staged + 8'd1;
    end
  end
endmodule

Six jobs. 500 ms of compute.

Five staged jobs, four undercounted.

80% overhead at a 20 GB payload is the number that motivates the whole module. Four fifths of the wall clock is a DMA engine moving bytes that the accelerator could have read where they were, and no amount of making the accelerator faster changes it — the compute is already only a fifth of the time.

Row six is the case the guard exists for. A job with no data to stage really does take exactly its compute time, and reporting that as the runtime is correct rather than an undercount. Section 18 records that the term was untestable until a zero-payload job was driven.

The 32-bit widening in s_q is not a stylistic choice. Megabytes times a thousand overflows a 16-bit multiply at 66 MB — far below any realistic payload — and the first version of this model produced a smaller staging time for a larger payload. It passed every assertion written against the small payload.

That failure has a shape worth naming, because it is not a logic error. Every expression was correct; the intermediate was too narrow to hold a correct value. A model whose inputs span orders of magnitude will fail at width before it fails at logic, and no amount of reasoning about the formula finds it — only driving the large input does. Section 19 makes it a standing rule.

7. RTL 3 — Coherence Loses At The Wrong Granularity

// RTL 3 - coherence is only worth its cost at the right sharing granularity. A
// line ping-ponging between host and accelerator costs more than a copy.
module sharing_granularity #(parameter int ASSUME_NO_PINGPONG = 0) (
  input  logic clk, rst_n,
  input  logic        window,
  input  logic [15:0] shared_lines, transfers_per_line,
  input  logic [15:0] coh_ns_per_transfer, copy_ns_per_line,
  output logic [15:0] coherent_ns, true_coherent_ns, copy_ns,
  output logic        coherence_wins,
  output logic [7:0]  n_windows, n_copy_better,
  output logic        wrong_choice_err
);
  logic [31:0] c_q, p_q, t_q;
  logic [15:0] transfers;
  // Every transfer of a shared line costs a coherence round trip.
  assign transfers = (ASSUME_NO_PINGPONG != 0) ? 16'd1 : transfers_per_line;
  assign c_q = {16'd0, shared_lines} * {16'd0, transfers} * {16'd0, coh_ns_per_transfer};
  assign coherent_ns = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  // What coherence really costs, against what the model chose to compute.
  assign t_q = {16'd0, shared_lines} * {16'd0, transfers_per_line}
             * {16'd0, coh_ns_per_transfer};
  assign true_coherent_ns = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign p_q = {16'd0, shared_lines} * {16'd0, copy_ns_per_line};
  assign copy_ns = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign coherence_wins = (coherent_ns <= copy_ns);
  // Choosing coherence on a sharing pattern a copy would beat.
  assign wrong_choice_err = window && coherence_wins && (true_coherent_ns > copy_ns);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_windows <= 8'd0; n_copy_better <= 8'd0;
    end else if (window) begin
      n_windows <= n_windows + 8'd1;
      if (!coherence_wins) n_copy_better <= n_copy_better + 8'd1;
    end
  end
endmodule

Four windows. 100 shared lines, 200 ns per coherence transfer, 500 ns per line copied.

One window where a copy was better, and the no-ping-pong model chose wrongly on it.

Row two is the case that decides whether coherent attach helps a given workload. Eight transfers per shared line — a line the host writes and the accelerator reads repeatedly — costs eight coherence round trips where a copy costs one transfer. Coherence is cheap per access and expensive per bounce, and a workload whose sharing is fine-grained and bidirectional pays the second price.

Row three is exactly break-even, and it is the boundary a design conversation actually lives on: one transfer per line at a coherence cost equal to the copy cost. Below it coherence wins; above it, copying does — and the deciding variable is the sharing pattern, which is a property of the algorithm rather than of the hardware.

8. RTL 4 — Bias Decides Whether A Local Access Is Local

// RTL 4 - bias. A Type 2 device's own memory can be biased toward the device or
// toward the host, and the wrong bias makes every access a coherence action.
module bias_mode #(parameter int ALWAYS_HOST_BIAS = 0) (
  input  logic clk, rst_n,
  input  logic        access,
  input  logic        from_device, host_may_hold,
  input  logic [15:0] local_ns, coherent_ns,
  output logic        device_bias, needs_resolution,
  output logic [15:0] access_ns,
  output logic [7:0]  n_accesses, n_resolved,
  output logic        needless_coherence_err
);
  // Device bias is correct when the host is not holding the line. The always-host
  // build resolves coherence on every access whether or not anything shares it.
  assign device_bias = (ALWAYS_HOST_BIAS != 0) ? 1'b0 : !host_may_hold;
  assign needs_resolution = from_device && !device_bias;
  assign access_ns = needs_resolution ? coherent_ns : local_ns;
  // A device access resolving coherence against a host that holds nothing.
  assign needless_coherence_err = access && needs_resolution && !host_may_hold;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_accesses <= 8'd0; n_resolved <= 8'd0;
    end else if (access) begin
      n_accesses <= n_accesses + 8'd1;
      if (needs_resolution) n_resolved <= n_resolved + 8'd1;
    end
  end
endmodule

Four accesses. 100 ns local, 500 ns resolved.

One access needed resolution in the correct build, three in the always-host-bias build, and two of those were needless.

Row two is what keeps the model honest. A device access to a line the host really is holding must resolve — 500 ns is the correct cost, and a checker that flagged every resolution would be flagging the protocol working.

The failure in row one is a factor of five on every access to the device's own memory. An accelerator running entirely out of its local HBM, with the host touching none of it, paying a coherence round trip per access because the bias was never switched. That is not a subtle degradation; it is the device running at a fifth of its memory bandwidth, and nothing errors.

9. RTL 5 — The Accelerator's Cache Is Finite

// RTL 5 - the accelerator's cache is finite. A working set larger than it holds
// evicts, and every eviction of a coherent line is a protocol action.
module accel_cache #(parameter int ASSUME_FITS = 0) (
  input  logic clk, rst_n,
  input  logic        access,
  input  logic [15:0] working_set_mb, cache_mb,
  input  logic [15:0] hit_ns, miss_ns,
  output logic [15:0] resident_mb, hit_pct, blended_ns,
  output logic        fits,
  output logic [7:0]  n_accesses, n_thrashing,
  output logic        thrash_err
);
  logic [31:0] h_q, b_q;
  assign fits = (working_set_mb <= cache_mb);
  assign resident_mb = fits ? working_set_mb : cache_mb;
  // The hit rate is the resident fraction of the working set.
  assign h_q = (working_set_mb == 16'd0) ? 32'd100
             : (({16'd0, resident_mb} * 32'd100) / {16'd0, working_set_mb});
  assign hit_pct = (ASSUME_FITS != 0) ? 16'd100
                 : ((h_q > 32'd65535) ? 16'hFFFF : h_q[15:0]);
  assign b_q = (({16'd0, hit_ns} * {16'd0, hit_pct})
              + ({16'd0, miss_ns} * (32'd100 - {16'd0, hit_pct}))) / 32'd100;
  assign blended_ns = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  // A working set larger than the cache, reported as if it fitted.
  assign thrash_err = access && !fits && (hit_pct == 16'd100);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_accesses <= 8'd0; n_thrashing <= 8'd0;
    end else if (access) begin
      n_accesses <= n_accesses + 8'd1;
      if (!fits) n_thrashing <= n_thrashing + 8'd1;
    end
  end
endmodule

Four windows. A 128 MB cache, 10 ns hit, 400 ns miss.

One thrashing window; the assume-fits model reported it as fitting.

205 ns against 10 is a factor of twenty, from a working set twice the cache. The miss cost is what makes it violent: a coherent miss is not a local DRAM access, it is a protocol transaction against the host, and section 7's per-transfer cost is what each one pays.

Row three is the boundary and it is exact. A working set exactly the size of the cache fits entirely; one megabyte more and half of it does not. Cache sizing for a coherent accelerator is not a smooth curve — it is a cliff at the working-set boundary, which is why 22.2 spends a chapter on where that boundary is.

10. Waveform — A Staged Job Against A Coherent One

The coh_traf row is the honest half of the picture. The coherent job is finished four cycles earlier and it generated fourteen coherence transactions the staged job did not. At this sharing pattern that is a good trade; at eight transfers per line it is not, and the waveform looks identical either way — the traffic count is the only thing that changes.

11. RTL 6 — A Link That Cannot Feed The Compute

// RTL 6 - the accelerator's bandwidth need against what the attach provides.
// A coherent link that cannot feed the compute is a compute unit that idles.
module feed_rate #(parameter int ASSUME_LINK_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        run,
  input  logic [15:0] flops_g, bytes_per_flop, link_gbps,
  output logic [15:0] demand_gbps, served_gbps, utilisation_pct,
  output logic        compute_bound,
  output logic [7:0]  n_runs, n_starved,
  output logic        overclaim_err
);
  logic [31:0] d_q, u_q;
  // Bytes per FLOP times FLOPs is the byte rate the compute needs.
  assign d_q = {16'd0, flops_g} * {16'd0, bytes_per_flop} * 32'd8;
  assign demand_gbps = (d_q > 32'd65535) ? 16'hFFFF : d_q[15:0];
  assign served_gbps = (ASSUME_LINK_ENOUGH != 0) ? demand_gbps
                     : ((demand_gbps > link_gbps) ? link_gbps : demand_gbps);
  assign compute_bound = (demand_gbps <= link_gbps);
  assign u_q = (demand_gbps == 16'd0) ? 32'd0
             : (({16'd0, served_gbps} * 32'd100) / {16'd0, demand_gbps});
  assign utilisation_pct = (u_q > 32'd65535) ? 16'hFFFF : u_q[15:0];
  // Reporting a served rate the link cannot carry.
  assign overclaim_err = run && (served_gbps > link_gbps);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_runs <= 8'd0; n_starved <= 8'd0;
    end else if (run) begin
      n_runs <= n_runs + 8'd1;
      if (!compute_bound) n_starved <= n_starved + 8'd1;
    end
  end
endmodule

Four runs against a 400 Gbps link.

One link-starved run, and the assume-enough model claimed a rate the link cannot carry.

Bytes per FLOP is the property that decides everything here, and it is an algorithm property rather than a hardware one. A dense matrix multiply reuses each byte many times and needs very few bytes per FLOP; a sparse or element-wise operation needs one or more. The same accelerator on the same link is compute-bound on one and starved by a factor of two on the other.

Row one is exactly at the link rate — the design point every attach is sized for, and the one that has no margin at all for a workload with a slightly worse ratio.

12. RTL 7 — A Device That Takes A Page Fault

// RTL 7 - an accelerator that faults. A coherent device taking a page fault
// against host memory needs the host to service it, which is not a local event.
module device_fault #(parameter int ASSUME_PINNED = 0) (
  input  logic clk, rst_n,
  input  logic        access,
  input  logic        page_present, pinned, host_reachable,
  input  logic [15:0] local_ns, fault_us,
  output logic [15:0] cost_ns,
  output logic        faults, serviceable,
  output logic [7:0]  n_accesses, n_faults, n_unserviceable,
  output logic        stall_unmodelled_err
);
  logic [31:0] f_q;
  // A pinned page never faults. An unpinned absent one costs a host round trip.
  assign faults = (ASSUME_PINNED != 0) ? 1'b0 : (!pinned && !page_present);
  // Only the host can service a device fault, so a fault it cannot reach is a
  // stall with no end.
  assign serviceable = !faults || host_reachable;
  assign f_q = {16'd0, fault_us} * 32'd1000;
  assign cost_ns = faults ? ((f_q > 32'd65535) ? 16'hFFFF : f_q[15:0]) : local_ns;
  // An access that will fault, costed as if it were local.
  assign stall_unmodelled_err = access && !pinned && !page_present
                                && (cost_ns == local_ns);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_accesses <= 8'd0; n_faults <= 8'd0; n_unserviceable <= 8'd0;
    end else if (access) begin
      n_accesses <= n_accesses + 8'd1;
      if (faults) n_faults <= n_faults + 8'd1;
      if (!serviceable) n_unserviceable <= n_unserviceable + 8'd1;
    end
  end
endmodule

Five accesses. 100 ns local, a 20 µs fault.

Two faults, one unserviceable, and the assume-pinned model undercounted twice.

A device fault is a 200x access. 20 µs against 100 ns, and it is not a local event — the accelerator has stalled waiting for a host to walk page tables and map something. A model that assumes everything is pinned has removed the single largest tail-latency contributor from the picture.

Row four is the failure with no recovery. The device faulted, the host cannot be reached to service it, and the access will never complete. Row five is the case that keeps serviceable honest: an access that does not fault does not need servicing, so the host being unreachable is irrelevant — and section 18 records that this case was missing until a mutation asked for it.

Figure 3 — The first gate is the one that decides the answer and the only one that is a property of the workload rather than of the hardware. The upper-right terminal is not a failure: a copy winning is a correct outcome for a sharing pattern coherence is bad at.

13. RTL 8 — Pinning Is Not Free Either

// RTL 8 - pinning is not free. Memory pinned for an accelerator is memory the
// host cannot reclaim, and pinning everything is pinning the machine.
module pinning_cost #(parameter int PIN_EVERYTHING = 0) (
  input  logic clk, rst_n,
  input  logic        plan,
  input  logic [15:0] working_set_gb, host_gb, pin_limit_gb,
  output logic [15:0] pinned_gb, reclaimable_gb, pinned_pct,
  output logic        within_limit, host_healthy,
  output logic [7:0]  n_plans, n_over,
  output logic        overpin_err
);
  logic [31:0] p_q;
  assign pinned_gb = (PIN_EVERYTHING != 0) ? host_gb : working_set_gb;
  assign reclaimable_gb = (pinned_gb >= host_gb) ? 16'd0 : (host_gb - pinned_gb);
  assign p_q = (host_gb == 16'd0) ? 32'd0
             : (({16'd0, pinned_gb} * 32'd100) / {16'd0, host_gb});
  assign pinned_pct = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign within_limit = (pinned_gb <= pin_limit_gb);
  // A host with nothing left to reclaim cannot page anything.
  assign host_healthy = (reclaimable_gb != 16'd0);
  // Pinning past the limit the platform set.
  assign overpin_err = plan && !within_limit;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_over <= 8'd0;
    end else if (plan) begin
      n_plans <= n_plans + 8'd1;
      if (!within_limit) n_over <= n_over + 8'd1;
    end
  end
endmodule

Five plans. A 512 GB host, a 256 GB pin limit.

Three plans exceeded the limit; the pin-everything build exceeded it on all five.

Section 12's fix is section 13's problem. Pinning removes the fault, and the memory it pins is memory the host can no longer reclaim, page out, or give to anything else. The two models are the same trade seen from opposite ends, and the pin limit is where a platform draws the line between them.

Row four is the pathological case that ships. An accelerator pinning the entire host: no faults, ever, and a host with nothing to page — so the first memory pressure anywhere on the machine has no relief valve. Row five is the mis-sized request, where the reclaimable count floors at zero rather than wrapping to 65,000 GB.

14. RTL 9 — What Coherent Attach Is Worth

// RTL 9 - what coherent attach is worth. The saving is the staging, and it only
// matters against how much of the runtime staging was.
module attach_value #(parameter int ASSUME_ALL_STAGED = 0) (
  input  logic clk, rst_n,
  input  logic        eval,
  input  logic [15:0] staged_ms, coherent_ms,
  input  logic [7:0]  staged_share_pct,
  output logic [15:0] saved_ms, blended_ms, gain_pct,
  output logic        worth_attaching,
  output logic [7:0]  n_evals, n_marginal,
  output logic        overclaim_err
);
  logic [31:0] b_q, g_q;
  logic [7:0] share;
  assign saved_ms = (staged_ms > coherent_ms) ? (staged_ms - coherent_ms) : 16'd0;
  // A parameter that gates the cost must gate the share it applies to.
  assign share = (ASSUME_ALL_STAGED != 0) ? 8'd100 : staged_share_pct;
  assign b_q = (({16'd0, staged_ms} * {24'd0, share})
              + ({16'd0, coherent_ms} * (32'd100 - {24'd0, share}))) / 32'd100;
  assign blended_ms = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  // A blend no better than the staged path is no gain.
  assign g_q = ((staged_ms == 16'd0) || (blended_ms >= staged_ms)) ? 32'd0
             : ((({16'd0, staged_ms} - {16'd0, blended_ms}) * 32'd100) / {16'd0, staged_ms});
  assign gain_pct = (g_q > 32'd65535) ? 16'hFFFF : g_q[15:0];
  assign worth_attaching = (gain_pct >= 16'd10);
  assign overclaim_err = eval && worth_attaching && (staged_share_pct < 8'd10);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_marginal <= 8'd0;
    end else if (eval) begin
      n_evals <= n_evals + 8'd1;
      if (!worth_attaching) n_marginal <= n_marginal + 8'd1;
    end
  end
endmodule

Six evaluations. 900 ms staged against 500 ms coherent.

Three evaluations were not worth attaching.

The share here is the fraction of jobs that stage, and the sense is inverted from the models in 21.2 §8 and 21.4 §13: a higher staged share means more of the runtime is the slow path, so the blend is worse. Row three is the extreme — every job stages, so there is nothing coherent in the blend and no gain to report.

Row two is the overclaim. A 42% gain on the 5% of jobs that stage is arithmetically correct and operationally meaningless, which is why overclaim_err tests the measured share rather than the modelled one.

Row five is the case that exposed a floor: a coherent path slower than staging, which a badly-tuned attach produces, and without the floor the gain calculation underflows.

Figure 4 — Sections 12 and 13 are one decision drawn twice. Every path from the access on the left reaches a cost on the right, and the pin limit is where a platform states which cost it prefers to pay.

15. RTL 10 — Coherent Attach Assembled

// RTL 10 - coherent accelerator attach assembled. Everything that must hold
// before a CXL-attached accelerator is faster than a DMA-staged one.
module accel_attach_model #(parameter int COHERENT_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       coherently_attached, // the device speaks .cache
  input  logic       type_correct,        // Type 2 where memory must be visible
  input  logic       bias_managed,        // device bias where the host holds nothing
  input  logic       granularity_right,   // sharing coarse enough to beat a copy
  input  logic       link_feeds_compute,  // the attach carries the byte rate
  input  logic       faults_budgeted,     // unpinned pages are in the model
  output logic       faster,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_faster,
  output logic       false_speedup_err
);
  assign fail_mask[0] = ~coherently_attached;
  assign fail_mask[1] = ~type_correct;
  assign fail_mask[2] = ~bias_managed;
  assign fail_mask[3] = ~granularity_right;
  assign fail_mask[4] = ~link_feeds_compute;
  assign fail_mask[5] = ~faults_budgeted;
  // The coherent-only build checks that the device speaks the cache protocol and
  // calls it faster, which is what a feature checklist does.
  assign faster = (COHERENT_ONLY != 0) ? coherently_attached : (fail_mask == 6'd0);
  assign false_speedup_err = evaluate && faster && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_faster <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (faster) n_faster <= n_faster + 8'd1;
    end
  end
endmodule

One faster of six, and four false claims.

The coherent-only definition is a feature checklist, and it is right about exactly one of the six. Row four is the one that gets built: a correctly typed, correctly biased, coarsely sharing accelerator behind a link that carries half the bytes its compute wants — coherently attached, and running at 50% utilisation because the attach was sized from a FLOP number rather than a byte-per-FLOP number.

16. Quantitative Reasoning

Device type. Four classifications, one Type 2. A device with 64 GB the host does not need to see is Type 1, and giving it an HDM decoder costs address-map reservation for nothing.

Staging. 4 GB in and out at 20 GB/s around 500 ms of compute: 900 ms total, 44% overhead. At 20 GB it is 2500 ms and 80% — four fifths of the wall clock doing no arithmetic.

Granularity. 100 shared lines at 200 ns per transfer: coherence costs 40,000 ns against a copy's 50,000 at two transfers per line, and saturates the counter at eight. Break-even is one transfer at a coherence cost equal to the copy cost.

Bias. A device access to its own memory with the host holding nothing: 100 ns correctly, 500 ns under always-host bias — a factor of five on every access, with nothing erroring.

Cache. A 256 MB working set in a 128 MB cache: 50% hit rate, 205 ns blended against 10 — a factor of twenty, and the boundary is exact at 128 MB.

Feed rate. 50 GFLOP/s at two bytes per FLOP demands 800 Gbps against a 400 Gbps link: 50% utilisation, and the ratio is an algorithm property.

Faults. An unpinned absent page is 20,000 ns against 100 — a 200x access — and one of five accesses faulted with no host to service it.

Pinning. A 512 GB working set on a 512 GB host pins 100% and leaves the host nothing to reclaim.

Value. 900 ms staged against 500 coherent at a 50% staged share is a 22% gain; at 100% staged it is zero, because nothing coherent is in the blend.

The assembled model. Six properties, six configurations, one faster. The coherent-only definition reported five.

17. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Device type. Both conditions are failed alone, and the memory-without-visibility case is asserted Type 1.

chk(tHm == 1'b1, "the device still has memory");
chk(tNh == 1'b0, "but the host does not need to see it");
chk(tDt == 2'd1, "so Type 1 is correct");

Staging. The zero-payload job is asserted as not an undercount.

chk(sTm == 16'd500, "so the runtime is the compute");
chk(sUe == 1'b0,    "and reporting it as compute is not an undercount");

Granularity. Break-even is asserted exactly, and the saturating case is asserted at the clamp.

Bias. The required resolution is asserted as not needless, which is what separates the protocol working from the protocol being wasted.

chk(bAn == 16'd500, "and the access costs 500");
chk(bNe == 1'b0,    "which is not needless, it is required");

Cache. The exact cache-size boundary is driven.

Feed rate. Exactly at the link rate is asserted compute-bound.

Faults. The non-faulting access with an unreachable host is asserted trivially serviceable.

chk(dFa == 1'b0, "a present page does not fault");
chk(dSe == 1'b1, "so it is trivially serviceable, host or no host");

Pinning. The mis-sized request is asserted to floor rather than wrap.

Value. The threshold is constructed — 82% staged against a 400 ms coherent path gives exactly 10% — and the slower-coherent case is asserted to floor.

chk(vGp == 16'd10,  "the gain is exactly 10 percent");
chk(vWa == 1'b1,    "which exactly meets the threshold");

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 245 checks across two testbenches, 117 on the front five models and 128 on the back five, all passing on the unmutated sources.

18. Mutation Testing

Forty-eight mutations were injected one at a time.

48 injected, 48 killed, after four survivors were diagnosed — and one arithmetic defect was found before the harness ran at all.

A 16-bit multiply that overflowed, found at baseline. The staging model computed bytes_mb * 16'd1000 at 16 bits, which overflows at 66 MB — far below any realistic payload. The testbench passed at a 4 GB payload and failed at 20 GB with a smaller staging time than the smaller payload produced. The fix widens the product to 32 bits, and a mutation was added to cover the widening, because a guard that is introduced and not mutated is a guard nobody tests. Batch 020's rule, applied to a defect the baseline caught rather than a survivor.

Survivor 1 — a term that needed a zero-payload job. The undercount check carried stage_in_ms != 0 alongside the total-equals-compute test, and no job had ever had nothing to stage. A zero-byte job really does take exactly its compute time, so reporting that as the runtime is correct — the guard distinguishes a correct report from an undercount, and without the case it looked redundant.

Survivor 2 — a conjunction where both terms were false together. Serviceability reduced to host_reachable survived because every unreachable-host case in the testbench also faulted. A present page with an unreachable host separates them: nothing faults, so nothing needs servicing. This is batch 020's §4.4 pattern exactly, and it was the less dramatic case that resolved it.

Survivors 3 and 4 — a floor and a constructed threshold. The reclaimable floor needed a working set larger than the host — a mis-sized request, which a resource manager produces. The worth threshold needed a gain of exactly 10%, which required solving for the share: 82% staged against a 400 ms coherent path, since the natural 500 ms path admits no integer solution.

19. Verification Strategy

What a testbench for a real accelerator attach must cover.

Payload sizes that exercise the arithmetic width. The 16-bit overflow in section 18 passed every assertion written at 4 GB and failed at 20. A model whose inputs span orders of magnitude has to be driven across them, because the intermediate widths are where the model breaks rather than the logic.

Both device types, and both reasons a device is Type 1. No memory at all, and memory the host does not need to see — the second is the one that gets classified wrongly.

Every conjunction where the obvious case fails both terms. Serviceability is "did not fault or host reachable", and every unreachable case in the first testbench also faulted.

The cases that are correct and look like failures. A required coherence resolution. A zero-payload job reported as compute. A device with private memory classified Type 1. A break-even sharing pattern.

Thresholds constructed where no natural stimulus reaches them. The 10% gain needed a share solved for against a chosen coherent latency.

What a real attach needs that these models do not have. Concurrency — host and device accessing the same line simultaneously, which is where bias transitions actually happen. Bias switching cost, which this chapter treats as a static mode. Translation caching, which changes the fault model from per-page to per-miss.

20. Synthesis and Implementation Reality

A coherent cache on an accelerator is a real cache with real coherence state. Tags, state bits, a snoop path and a protocol engine — and the snoop path is the part that constrains, because it must be serviceable while the accelerator's own datapath is running at full rate.

Bias is a per-page mode with a transition cost. Section 8 treats it as static; in silicon, flipping a page from device bias to host bias means flushing whatever the device holds for it. A workload that alternates is paying the transition, not the steady-state cost, which is why bias granularity and switching policy are design parameters rather than a bit.

Device-side address translation is what makes section 12's fault possible at all. An accelerator issuing host virtual addresses needs a translation agent, a cache of translations, and a path to ask the host when it misses. That path is the fault, and its latency is a round trip through host software.

The link width is chosen against bytes per FLOP, not against FLOPs. Section 11's number is the sizing input, and it is workload-dependent — which means an attach sized for one class of model is under-provisioned for another on identical silicon.

Type 2 costs an HDM decoder and an address-map reservation. 20.4 §7's map reservation applies here: exposing device memory to the host means the host's map must have room for it, chosen at boot.

21. Silicon Observability

The bias-transition counter is the one the static model needs and does not have. Sections 8 and 21 disagree deliberately: the model treats bias as a mode, and the counter exists because a workload that alternates pays a cost the mode has no term for. A device reporting a low resolved-access ratio and a high transition count is paying for coherence in a way section 8 cannot see.

22. Debug Lab

Symptom. A Type 2 accelerator is moved from a DMA-staged driver to a coherent attach. The staging copy is gone — verified, zero bytes staged — and the job takes 12% longer than it did before. No errors, and the coherence protocol is behaving correctly on every transaction.

Step 1 — is the staging really gone? Bytes staged in and out: zero on every job. Section 6's saving is real and complete.

Step 2 — is it the link? Demand byte rate against link rate: demand 180 Gbps, link 400. Compute-bound with margin. Section 11 is not the chapter.

Step 3 — is it the cache? Device cache hit rate: 94%. The working set fits comfortably. Section 9 is fine.

Step 4 — is it faults? Device page faults: 11 for the whole job. Negligible. Section 12 is not it.

Step 5 — read the coherence counters. Coherence transactions per shared line: 6.2. The workload writes a tile on the host, the accelerator reads it, the host updates it, the accelerator reads it again — six round trips per line where the staged version did one bulk copy. Section 7's row two, in production.

Step 6 — check the bias. Accesses resolved against accesses local: 8%. Bias is set correctly; the device's own memory is local. Section 8 is not contributing.

The finding. Every property held except granularity. The attach removed a 400 ms copy and added 6.2 coherence round trips per line across a large shared region, and the second is larger than the first. Coherent attach was strictly correct and strictly slower, because the workload's sharing pattern is exactly the one a copy is good at.

The fix. Not a hardware change. Restructure the sharing so the host produces a whole tile before the accelerator consumes it — one transfer per line instead of six — which is a data-layout change in the model code. Section 7's break-even is the target, and the counter in section 21 is how you know when you have reached it.

What made this hard. The obvious win was real and complete. Removing the copy was worth 400 ms and it was banked. The regression was in a cost that did not exist before the change, so there was no baseline for it and no counter had ever been read.

23. Design Review

1. Does the host need to see the device's memory? That single question decides Type 1 against Type 2, and answering "yes" by default costs an HDM decoder and a map reservation. Section 5.

2. What fraction of the job is staging today? If it is small, coherent attach is not the lever. Section 6.

3. How many times does a shared line transfer per job? Above break-even a copy wins. Section 7, and section 22 turns entirely on this.

4. Is device memory in device bias when the host is not using it? A five-times penalty with nothing erroring. Section 8.

5. Does the working set fit the device cache, and what is the miss latency? The miss is a protocol transaction, not a DRAM access. Section 9.

6. What is bytes-per-FLOP for this workload, and what link does that need? Not FLOPs. Section 11.

7. Is the accelerator's memory pinned, and how much? Faults or pinning — one or the other, and both have a cost. Sections 12 and 13.

8. Is there a counter for coherence transactions per shared line? Section 22's investigation reduces to reading it.

9. Is there a bias-transition counter? The static model has no term for it. Section 21.

10. Which of the six properties does the team believe "coherently attached" implies? Section 15 exists because the answer is "faster".

24. How This Appears In Real Engineering

An accelerator team specifying a new part answers section 5 first and lives with it for the product's life. Type 2 is strictly more capable and strictly more expensive — decoder state, host address-map reservation, a bias mechanism — and the question is whether any workload will ask the host to address device memory.

A framework team porting to a coherent attach meets section 22. The copy disappears, which is easy to verify and easy to celebrate, and the coherence traffic that replaces it is a cost with no prior baseline. The first port of any workload should measure transfers-per-line before and after, because that number is the whole difference.

A platform team owns the sections 12 and 13 trade. Pinning removes faults and removes the host's ability to page; not pinning keeps the host flexible and puts a 200x tail on device accesses. There is no configuration that avoids both, and the pin limit is where the platform states its preference.

A performance engineer finds section 11 is the sizing question nobody asks in the right units. A link is specified in Gbps and an accelerator in FLOPs, and the number that connects them — bytes per FLOP — is a property of the model being run rather than of either component.

25. Common Misconceptions

"Type 2 is better than Type 1." They answer different questions. A device with private memory the host never addresses is correctly Type 1. Section 5.

"Coherent attach is faster." It removes a copy and adds a protocol. At six transfers per shared line the protocol is larger. Sections 7 and 22.

"The staging copy is a small overhead." 80% of the wall clock at a 20 GB payload. Section 6.

"Bias is a detail." A five-times penalty on every access to the device's own memory. Section 8.

"The accelerator's cache will hold it." A working set twice the cache blends to 205 ns against 10 — a factor of twenty. Section 9.

"Size the link against the accelerator's FLOPs." Against bytes per FLOP. The same part on the same link is compute-bound on one workload and half-starved on another. Section 11.

"Coherent devices do not page-fault." They do, and it costs 20 µs against a 100 ns local access. Section 12.

"So pin everything." Then the host has nothing to reclaim. Sections 12 and 13 are one trade.

"Removing the copy is the whole win." It is, and section 22's job got 12% slower anyway.

"The device speaks .cache, so we are done." One property of six. Section 15.

26. Interview Reasoning

Q. What is the difference between a CXL Type 1 and a Type 2 device?

Type 2 has memory the host can address; Type 1 does not. The follow-up worth reaching: a device with plenty of private memory the host never needs to touch is correctly Type 1 — the classification is about host visibility, not about whether the device has memory, and getting that wrong costs an HDM decoder and an address-map reservation for nothing.

Q. Coherent attach removes the staging copy. When is it still slower?

When the sharing is fine-grained and bidirectional. A line transferred six times between host and accelerator costs six coherence round trips where the copy cost one bulk transfer. The break-even is roughly one transfer per line at a coherence cost equal to the per-line copy cost, and above it copying wins.

Q. What is bias, and what happens if it is wrong?

A Type 2 device's memory is resolved toward the device or toward the host. In device bias, the accelerator reads its own memory locally; in host bias, every access resolves coherence. Get it wrong and the device runs at a fifth of its memory bandwidth with nothing reporting an error — the protocol is working exactly as instructed.

Q. How do you size the link for an accelerator?

Against bytes per FLOP, not FLOPs. 50 GFLOP/s at one byte per FLOP is 400 Gbps; at two it is 800. The ratio is a property of the workload, so the same silicon on the same link is compute-bound on a dense matmul and half-starved on something sparse.

Q. A coherent accelerator takes a page fault. What does that cost?

A host round trip — around 20 µs against a 100 ns local access, a 200x hit, and the device stalls for all of it. The obvious fix is pinning, and the follow-up worth being ready for is what pinning costs: memory the host can never reclaim, so pinning the working set of a large model can leave the host with no relief valve at all.

Q. You moved a workload to coherent attach and it got slower. Where do you look?

Coherence transactions per shared line, first. The copy really is gone — that saving is banked — and the cost that replaced it did not exist before the change, so nothing has a baseline for it. If transfers-per-line is high, the fix is a data-layout change rather than a hardware one.

27. Exercises

1. Extend RTL 1 to Type 3 as well and show which of the three questions — cache, host-visible memory, host-managed memory — separates each pair.

2. Make RTL 2's staging overlap with compute and find the payload at which overlapping stops hiding it.

3. Derive RTL 3's break-even transfers-per-line analytically as a function of the two latencies, and confirm the model against it.

4. Add a bias-transition cost to RTL 4 and find the alternation rate at which transitions dominate both steady states.

5. Give RTL 5 an associativity term and show that a working set that fits by size can still thrash by conflict.

6. Extend RTL 6 to a roofline: plot achievable FLOP/s against bytes per FLOP for a fixed link, and mark the ridge.

7. Add a translation cache to RTL 7 and show that the fault rate falls with locality rather than with pinning.

8. Combine RTL 7 and RTL 8 into one model and find the pin fraction that minimises total cost for a given fault rate.

9. Make RTL 9's staged share a function of job size and show which job-size distribution makes coherent attach worthwhile.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the accelerator it catches that the current mask calls faster.

28. Summary

An accelerator that copies is an accelerator paying for every byte twice. Coherent attach removes that and charges a protocol instead.

Type 1 and Type 2 answer one question — does the host need to see the device's memory — and a device with 64 GB of private memory is correctly Type 1.

The staging copy is two legs and it dominates. 4 GB in and out around 500 ms of compute is 900 ms and a 44% overhead; at 20 GB it is 80%, four fifths of the wall clock doing no arithmetic.

Coherence loses at the wrong granularity. 100 lines at two transfers each beats a copy; at eight transfers it saturates the counter and a copy wins — and section 22 is a production workload at 6.2 transfers per line that got 12% slower after the copy was removed.

Bias decides whether a local access is local. A device access with the host holding nothing costs 100 ns correctly and 500 ns under always-host bias — a factor of five, with nothing erroring.

The device cache is a cliff, not a curve. A 256 MB working set in a 128 MB cache blends to 205 ns against 10, and the boundary is exact.

Size the link against bytes per FLOP. 50 GFLOP/s at two bytes per FLOP demands 800 Gbps on a 400 Gbps link — 50% utilisation, decided by the algorithm rather than the hardware.

A device page fault is a 200x access, and one of five had no host reachable to service it at all.

Pinning removes the fault and removes the host's relief valve. A working set the size of the host leaves nothing reclaimable.

A 16-bit multiply overflowed at 66 MB and passed every assertion written at 4 GB — found at baseline, fixed by widening to 32 bits, and covered by a mutation added afterwards.

Speaking .cache is one property of six. The definition a feature checklist uses called five of six accelerators faster when one was.

22.2 — Memory Expansion for AI takes the working set that section 9's cache could not hold and asks how large it actually is.