CXL.io Overview

Chapter 6.1 placed CXL.io beside .cache and .mem and asked what each is — a contract about who owns a resource and who keeps state about it. CXL.io came out as the one that owns nothing and keeps no coherence state.

Module 7 goes inside it. That description is accurate about coherence and badly misleading about everything else, because CXL.io is where a device becomes identifiable, configurable, mappable, manageable and debuggable — and none of that is optional.

1. The Engineering Problem — The Path Nothing Runs Without

A device is powered, the link is trained, .cache and .mem are negotiated. The accelerator is ready.

Nothing can use it. Software does not know it exists, has not assigned it addresses, cannot read its capabilities, cannot enable anything, and would not be told if it failed. Every one of those is CXL.io's job, and until they are done the coherent protocols are inert.

That gives CXL.io an unusual position: it carries the least data and gates the most. It is also the path you need most in exactly the conditions where it is under most pressure — boot, recovery, and a device that has started reporting errors.

2. The One-Sentence Model

CXL.io is the device's control-plane spine — it makes the device identifiable, configurable, addressable, manageable and observable using PCIe's non-coherent load-store semantics, and it does none of the coherent work that makes the device useful.

Call it the spine: small, load-bearing, and noticed only when it stops working.

3. What This Chapter Owns

Deliberately not repeated from 6.1: the ownership boundary, the state-obligation table, the device-type derivation and the directionality figure. This chapter assumes them.

4. What Is Inside CXL.io

The Consortium list is not one traffic type. It is five jobs with different shapes.

Four of those are control: low volume, latency-sensitive and gating. The fifth, DMA, is data: potentially high volume and not gating at all. Lumping them together is how a design ends up sizing its control path for DMA traffic, or its DMA path for configuration traffic.

The split matters because only the left branch gates. A device with a saturated DMA path is slow. A device with a blocked control path cannot be configured, cannot report why, and cannot be recovered — which is Debug Lab 3's subject.

The block label is the Consortium's own: "PCIe/CXL.io Logic". One block serves both, which is Chapter 5.2's reuse at its most literal — and also why a PCIe regression passing says nothing about the CXL-specific content that block carries.

5. Where CXL.io Stops

This is the discipline the whole module rests on.

CXL.io makes the device manageable and PCIe-compatible. It does not itself provide CXL.cache or CXL.mem semantics.

A device implementing only CXL.io is a PCIe device on a CXL-capable link — Chapter 6.1 showed it has no CXL device type at all, because the three types enumerate the coherent combinations. That is not a deficiency; it is the definition.

The trap is DMA. DMA moves data between host memory and the device, which sounds like .cache. It is not: DMA uses the non-coherent load-store semantics of PCIe and leaves no lasting relationship, while .cache leaves the device holding host lines that both ends must track. RTL 1 encodes the difference as an output that must never assert.

Every arrow before the last one runs over CXL.io. The coherent protocols are the only box that is not CXL.io's work, and they are the last box — which is the whole reason a low-bandwidth path deserves this much attention.

6. Quantitative — Why the Small Path Fails First

Control traffic arrives at rate R and is served at rate S. If R exceeds S the queue grows and latency grows with it — arithmetic that is unremarkable until you notice when it applies.

The steady state is benign: a configured device generates almost no control traffic. The interesting states are the ones where it does.

RTL 2 drives exactly that overload — arrivals every cycle, service every third:

  30 arrivals, service every 3rd cycle:
    level=7 peak=8 accepted=17 dropped=13

Thirteen of thirty control requests were dropped. The path is not slow, it is full, and every dropped request is a configuration or management action that did not happen. A design that sized this queue from steady-state traffic sized it for the case that never matters.

The point generalises: the control plane's bandwidth is irrelevant and its latency under burst is everything. Chapter 7.3 takes the error-storm case apart specifically.

7. Teaching-model boundary

8. RTL 1 — What CXL.io Carries, and Where It Goes

module io_dispatch #(
  parameter bit COHERENT_BY_MISTAKE = 1'b0   // 1 = route .io into a coherent engine
) (
  input  logic       clk, rst_n, req_valid,
  input  logic [2:0] req_kind,        // 0 cfg, 1 mmio, 2 mgmt, 3 dma, 4 unsupported
  input  logic [4:0] kind_supported,
  output logic       to_cfg, to_mmio, to_mgmt, to_dma,
  output logic       to_coherent,     // must ALWAYS be low: .io is not coherent
  output logic       refuse,
  output logic       coherent_leak_err, unsupported_accepted_err, multi_target_err
);
  localparam logic [2:0] K_CFG = 3'd0, K_MMIO = 3'd1, K_MGMT = 3'd2, K_DMA = 3'd3;
  logic legal;
  logic [2:0] hot;
 
  assign legal = req_valid && (req_kind <= K_DMA) && kind_supported[req_kind];
 
  assign to_cfg  = legal && (req_kind == K_CFG);
  assign to_mmio = legal && (req_kind == K_MMIO);
  assign to_mgmt = legal && (req_kind == K_MGMT);
  assign to_dma  = legal && (req_kind == K_DMA);
  // CXL.io uses the NON-COHERENT load-store semantics of PCIe. A .io request
  // reaching a coherent engine is a routing defect, not a performance choice.
  assign to_coherent = COHERENT_BY_MISTAKE && legal && (req_kind == K_DMA);
  assign refuse  = req_valid && !legal;
 
  assign hot = 3'd0 + to_cfg + to_mmio + to_mgmt + to_dma + to_coherent;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      coherent_leak_err <= 1'b0; unsupported_accepted_err <= 1'b0;
      multi_target_err  <= 1'b0;
    end else if (req_valid) begin
      if (to_coherent)                       coherent_leak_err        <= 1'b1;
      if ((to_cfg|to_mmio|to_mgmt|to_dma) && !kind_supported[req_kind])
                                             unsupported_accepted_err <= 1'b1;
      if (hot > 3'd1)                        multi_target_err         <= 1'b1;
    end
  end
endmodule

Purpose. To make "CXL.io is non-coherent" a checkable property rather than a sentence. to_coherent exists only so an assertion can require it to be zero.

Architecture position. At the CXL.io ingress, after 6.2's class dispatch has already decided this is .io traffic at all.

State. None on the datapath; three sticky diagnostics. The decode is combinational because it is a classification, not a transfer.

Synthesis. A 3-to-4 decoder, a supported-mask lookup, a small population count for the one-hot check and three set-reset flops. The population count is off the critical path.

Backpressure. None here — this module classifies. Backpressure lives in RTL 2, where a classification becomes a queued operation.

  kind         cfg mmio mgmt dma coherent refuse
  cfg          1   0    0    0     0       0
  mmio         0   1    0    0     0       0
  mgmt         0   0    1    0     0       0
  dma          0   0    0    1     0       0
  unsup        0   0    0    0     0       1

Every cell in that table is asserted by the testbench, including the coherent column — five separate checks that it is zero. A property this important should not be a comment.

The unsup row is refused rather than dropped, which matters more than it looks: a silently dropped control request presents as a software hang over a healthy link, with nothing anywhere to correlate.

9. RTL 2 — The Control-Plane Queue

module control_plane_queue #(
  parameter int unsigned DEPTH = 8,
  parameter bit          IGNORE_FULL = 1'b0
) (
  input  logic        clk, rst_n, push, pop,
  output logic        accept, full, empty,
  output logic [4:0]  level_q, peak_q,
  output logic [15:0] n_accept_q, n_drop_q,
  output logic        overflow_err, underflow_err
);
  logic do_push, do_pop;
 
  assign full   = (level_q == DEPTH[4:0]);
  assign empty  = (level_q == 5'd0);
  assign accept = push && (IGNORE_FULL ? 1'b1 : !full);
  assign do_push = accept;
  assign do_pop  = pop && !empty;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      level_q <= 5'd0; peak_q <= 5'd0;
      n_accept_q <= '0; n_drop_q <= '0;
      overflow_err <= 1'b0; underflow_err <= 1'b0;
    end else begin
      // Simultaneous arrival and service must leave the level unchanged. Two
      // independent increments is the classic occupancy bug.
      case ({do_push, do_pop})
        2'b10: level_q <= level_q + 5'd1;
        2'b01: level_q <= level_q - 5'd1;
        default: ;
      endcase
      if (level_q > peak_q) peak_q <= level_q;
      if (accept)          n_accept_q <= n_accept_q + 16'd1;
      if (push && !accept) n_drop_q   <= n_drop_q   + 16'd1;
 
      if (do_push && full) overflow_err  <= 1'b1;
      if (pop && empty)    underflow_err <= 1'b1;
    end
  end
endmodule

Cycle behaviour. One arrival and one service per cycle, independently. The case on {do_push, do_pop} is the only correct way to write the counter: the both-and-neither cases must leave the level alone, and mutation M5 implements it as two if statements to show what that costs.

Backpressure. accept is the contract. A push that is not accepted is a drop, and it is counted — because a control request that vanished without a counter moving is a request nobody can prove was made.

    level=7 peak=8 accepted=17 dropped=13
    simultaneous push+pop: level=3 (must be unchanged at 3)

accepted + dropped = 30 is asserted against the arrival count, so a design that silently stops counting drops fails immediately — which is mutation M6.

10. RTL 3 — Who May Reach the Control Plane

The control plane is where a device is identified and configured, so who is asking is a real question. This models the decision only — no CXL or PCIe access-control mechanism is implemented or claimed.

  // Teaching policy:
  //   host      : everything
  //   firmware  : management, and reads of configuration
  //   peer      : MMIO and DMA only -- never configuration, never management
  always_comb begin
    case (requester)
      RQ_HOST : allowed = 1'b1;
      RQ_FW   : allowed = (target_kind == K_MGMT) ||
                          ((target_kind == K_CFG) && !is_write);
      RQ_PEER : allowed = (target_kind == K_MMIO) || (target_kind == K_DMA);
      default : allowed = 1'b0;
    endcase
  end
 
  assign permit = req_valid && (TRUST_REQUESTER ? 1'b1 : allowed);
  assign deny   = req_valid && !permit;

  requester  target   write  permit | trust-requester
     0        1       0      1    |      1
     1        0       1      0    |      1
     2        0       1      0    |      1
  over 24 combinations: permitted=15 denied=9
  both totals matched an independent oracle          : ok

Note that permit and deny have separate error signals. Permitting the illegal is security-adjacent; denying the legal is a functionality failure. Different severity, different owners, and a combined flag would report the same thing for both.

11. RTL 4 and 5 — Slots and Per-Kind Telemetry

Some control operations complete immediately and some wait. Anything that waits occupies a slot.

  assign can_issue = NO_LIMIT ? 1'b1 : (outstanding_q < SLOTS[3:0]);
  assign did_issue = issue && can_issue;
  assign do_issue  = did_issue;
  assign do_retire = retire && (outstanding_q != 4'd0);

  8 issues into 4 slots: outstanding=4 peak=4 issued=4
  no-limit variant: outstanding=8 issued=8
  spurious retire with nothing outstanding: outstanding=0 detected=1

The do_retire guard is the interesting one. A retire arriving with nothing outstanding is an illegal input, not an internal fault — so the correct design detects it and refuses to act, and the testbench requires the detector to fire rather than treating it as a failure. Without the guard, mutation M10 decrements a zero counter and wraps it to fifteen.

Per-kind counters then answer the question §6 raised:

      if (offered) n_offered_q <= n_offered_q + 16'd1;
      // Count COMPLETED work, not attempted work. A counter that increments on
      // arrival reports a healthy control plane that is servicing nothing.
      if (offered && completed) begin
        case (kind)
          3'd0: n_cfg_q  <= n_cfg_q  + 16'd1;
          /* ... */
        endcase
      end

  offered=60 cfg=14 mmio=14 mgmt=14 dma=14 refused=4
  conservation offered == cfg+mmio+mgmt+dma+refused : 60 == 60
  every count matched an independent oracle           : ok

Counting arrivals instead of completions is mutation M11, and it is the specific bug that makes a stalled control plane look busy and healthy. Both the conservation law and an independent oracle are checked, because Chapter 5.5 established that conservation proves the parts consistent and cannot prove any part correct.

12. Assertions

Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog — the only simulator available in this environment. The properties below are bind-ready and were not executed; each maps to a procedural stand-in in the testbench and to a mutation in §13.

// SAFETY -------------------------------------------------------------------
// V1 — CXL.io never reaches a coherent engine. The module's whole point.
a_never_coherent: assert property (@(posedge clk) disable iff (!rst_n)
  !to_coherent);
 
// V2 — a valid request reaches exactly one target, or is refused.
a_one_target: assert property (@(posedge clk) disable iff (!rst_n)
  req_valid |-> $onehot({to_cfg, to_mmio, to_mgmt, to_dma, refuse}));
 
// V3 — an unsupported kind is never dispatched.
a_supported_only: assert property (@(posedge clk) disable iff (!rst_n)
  (to_cfg || to_mmio || to_mgmt || to_dma) |-> kind_supported[req_kind]);
 
// V4 — the control queue never overflows or underflows.
a_queue_bounds: assert property (@(posedge clk) disable iff (!rst_n)
  (level_q <= DEPTH) and (do_pop |-> (level_q > 0)));
 
// V5 — simultaneous arrival and service leave the level unchanged.
a_level_stable: assert property (@(posedge clk) disable iff (!rst_n)
  (do_push && do_pop) |=> $stable(level_q));
 
// V6 — an illegal requester never reaches configuration or management.
a_peer_restricted: assert property (@(posedge clk) disable iff (!rst_n)
  (permit && (requester == RQ_PEER)) |-> ((target_kind == K_MMIO) ||
                                          (target_kind == K_DMA)));
 
// V7 — outstanding never exceeds capacity.
a_slots_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  outstanding_q <= SLOTS);
 
// V8 — a retire with nothing outstanding does not change the count.
a_no_phantom_retire: assert property (@(posedge clk) disable iff (!rst_n)
  (retire && (outstanding_q == 0)) |=> (outstanding_q == 0));
 
// V9 — CONSERVATION: every offered operation lands in one kind or is refused.
a_kinds_conserved: assert property (@(posedge clk) disable iff (!rst_n)
  n_offered_q == n_cfg_q + n_mmio_q + n_mgmt_q + n_dma_q + n_refused_q);
 
// LIVENESS -----------------------------------------------------------------
// V10 — an accepted control operation eventually completes.
//       ENVIRONMENT ASSUMPTION: the service side keeps draining. Under the
//       overload of §6 this is FALSE, and that is the point of §6 rather than
//       a defect — the property names what overload takes away.
a_eventual_service: assert property (@(posedge clk) disable iff (!rst_n)
  accept |-> s_eventually pop);

V10 is the pair to §6. It is liveness with an environment assumption that overload violates — so the property is not "always true", it is "true exactly when the control plane is keeping up", which is the thing worth measuring.

The arrival-accounting invariant (accepted + dropped == arrivals) is deliberately not written as SVA above: it is a counter equality that reads naturally as a procedural check and awkwardly as a cycle-level property. The procedural form is the authoritative one, and mutation M6 dies against it.

13. Mutation Testing

Twelve mutations. Clean code restored after each.

12/12 killed, 0 escaped

One escaped on the first run. M10 removes the guard on do_retire, and the original testbench only ever retired operations that existed — so a design honouring a spurious retire was indistinguishable. Adding one retire with nothing outstanding killed it, and the rule generalises:

Every counter that can decrement needs a test that decrements it at zero. The guard is one term, and its absence wraps a 4-bit counter to fifteen — which then reports capacity the design does not have.

Three mutations — M7, M8 and M12 — died against an independent oracle rather than the design's own checks, which is why §10's callout insists the checker be recomputed from the policy rather than sharing a decode helper.

14. Verification Plan

Rows 6 and 9 came from this chapter's own defects: the first because a drain loop that pops past empty is a stimulus fault the design correctly reports, the second because a decrementing counter needs a zero-boundary test.

15. Debug Lab

// DMA touches host memory, so send it to the coherence engine.
assign to_coherent = legal && (req_kind == K_DMA);

  every kind : to_coherent=0   (correct design)
  coherent-leak variant flagged coherent_leak : ok

assign to_dma      = legal && (req_kind == K_DMA);
assign to_coherent = 1'b0;                      // .io is never coherent
if (to_coherent) coherent_leak_err <= 1'b1;

// Decode the kind and route it.
assign legal = req_valid && (req_kind <= K_DMA);

  DMA unsupported : to_dma=0 refuse=1   (correct)
  mutated design  : to_dma=1, reaching an engine that does not exist

assign legal  = req_valid && (req_kind <= K_DMA) && kind_supported[req_kind];
assign refuse = req_valid && !legal;
if ((to_cfg|to_mmio|to_mgmt|to_dma) && !kind_supported[req_kind])
  unsupported_accepted_err <= 1'b1;

// Size the control queue for steady-state control traffic.
localparam int DEPTH = 2;

  30 arrivals, service every 3rd cycle:
    level=7 peak=8 accepted=17 dropped=13

// Size from the burst, not the average; and count what you could not accept.
localparam int DEPTH = 8;
if (push && !accept) n_drop_q <= n_drop_q + 16'd1;

// The request arrived on a valid path, so serve it.
assign permit = req_valid;

  rq=2 tk=0 wr=1 permit=0   (correct: peer denied configuration write)
  trust-requester variant flagged permitted_illegal : ok

assign permit = req_valid && allowed;   // 'allowed' from an explicit policy
if (permit && !allowed) permitted_illegal_err <= 1'b1;
if (deny   &&  allowed) denied_legal_err      <= 1'b1;

// Count control operations.
if (offered) begin
  case (kind) 3'd0: n_cfg_q <= n_cfg_q + 16'd1; /* ... */ endcase
end

  offered=60 cfg=14 mmio=14 mgmt=14 dma=14 refused=4
  conservation offered == cfg+mmio+mgmt+dma+refused : 60 == 60
  arrival-counting variant: totals exceed offered, conservation breaks

if (offered)              n_offered_q <= n_offered_q + 16'd1;
if (offered && completed) begin /* per-kind increment */ end
if (offered && refused)   n_refused_q <= n_refused_q + 16'd1;
if (n_offered_q != n_cfg_q + n_mmio_q + n_mgmt_q + n_dma_q + n_refused_q)
  accounting_err <= 1'b1;

16. Design Review

• Is there any path by which .io traffic reaches a coherent engine? What asserts that there is not?
• Can a control request reach neither a target nor a refusal?
• What was the control queue sized from — the steady state, or boot and error storms?
• Are dropped control requests counted, or do they merely not happen?
• Who may issue configuration and management, and is that policy checked against an independent oracle?
• Do the control counters increment on arrival or on completion?
• Does any counter that can decrement have a test that decrements it at zero?
• If the control plane saturates, what is the recovery path that does not require the control plane?

17. How This Appears in Real Engineering

The control plane is sized last and fails first. It carries almost no data, so it attracts no attention during architecture, and then boot enumeration or an error storm saturates it. The symptom is a device that cannot be configured or diagnosed at exactly the moment it needs to be.

"CXL.io is basically PCIe" survives until it costs something. It is a useful shorthand for the transaction semantics (7.5) and a bad guide to the device: the CXL DVSEC structures and the CXL-specific capability content are reached through CXL.io and are not PCIe.

DMA-versus-coherent confusion is a Type 2 problem. On a device with no cache engine, Debug Lab 1 fails loudly at the first transfer. On a Type 2 device it appears to work, and the corruption arrives later under concurrency.

Arrival-counting telemetry survives because it is never wrong when things work. A control plane keeping up produces identical numbers either way. The divergence appears only under the load that matters, which is also when nobody trusts the dashboard.

18. Common Misconceptions

19. Interview Reasoning

20. Exercises

1. Explain. For each of CXL.io's five jobs, state whether it gates the device becoming usable and whether it can be high volume. Then say which pair of jobs should not share a queue, and why.

2. Calculate. Control requests arrive at 3 per cycle during enumeration and are served at 1 per cycle. With a depth-8 queue, how many cycles until the first drop, and how many of 100 arrivals are dropped? Then find the depth that drops nothing, and say why that depth is still the wrong design.

3. RTL task. Add a separate high-priority path so management traffic cannot be blocked by bulk configuration traffic. State the new invariant this creates and which existing assertion must be weakened.

4. DV task. Write the coverage cross for the permission policy, and explain why an oracle that calls the design's decode function would leave both mutations M7 and M8 alive.

5. Debug task. A fleet reports healthy control-plane counters and timing-out management operations. Give your investigation order and name the single check that settles arrival-counting versus a stalled service side in one step.

6. Design review. A colleague proposes routing DMA through the coherence engine "so it is coherent for free". Give the strongest version of their argument, then price it using 6.1's state obligations.

21. Summary

CXL.io is the control-plane spine: small, load-bearing, and noticed only when it stops.

• It carries discovery, configuration, initialization, interrupts, DMA and ATS over PCIe's non-coherent load-store semantics — five jobs, not one.
• Four of those gate the device becoming usable; DMA does not. Sizing one path for both is the standard error.
• CXL.io makes the device manageable and PCIe-compatible; it does not provide .cache or .mem semantics. A .io-only device is a PCIe device on a CXL link.
• Its bandwidth is small and irrelevant; its latency under burst is the design parameter. Under overload, 13 of 30 control requests were dropped.
• Refuse, never drop — a dropped control request is a hang with nothing to correlate.
• Count completions, not arrivals, and conserve them against offers, or a stalled control plane reports maximum health.
• Verification lessons: an oracle must be recomputed from the policy rather than sharing the design's decode; and every decrementing counter needs a zero-boundary test.

Chapter 7.2 takes the first of the five jobs apart: what a configuration register actually is in hardware, and why getting a single field's access type wrong can make a device undiscoverable.