The CXL Device

Chapter 3.1 argued that the host owns address decode, request tracking and coherence enforcement because each requires information only a system-wide view provides. That argument has a second half, and this chapter is it.

If responsibilities follow information, then the device owns everything that needs only local knowledge — and that turns out to be a large and interesting set. It also means a device is not defined by being "the far end of the link". It is defined by which protocol engines it actually implements.

1. The Engineering Problem

A vendor says "we are building a CXL device." That sentence does not yet specify a design.

Does it hold cached copies of host memory? Does it expose memory that the host will treat as system memory? Both? Neither? Each answer implies a different set of hardware blocks, a different verification surface, and a different relationship with the host — and the three combinations that matter have names.

Worse, the question has a failure mode. A device can declare a capability it does not implement. Discovery succeeds, the host configures accordingly, and the first request that exercises the missing engine fails inside a device that reported itself ready. Section 8 measures exactly that.

2. The One-Sentence Model

A CXL device is defined by which protocol engines it implements — and its device type is a consequence of that choice, not an independent property — with every remaining responsibility on the device being one that local knowledge alone can discharge: dispatching a request to the engine that owns it, tracking what it has accepted until it responds, decoupling the link from its own internal rate, and reporting what happened.

3. What This Chapter Owns

This chapter introduces the device types precisely enough to reason about hardware, and stops there.

4. The Mental Model — a Device Is a Set of Engines

The picture to carry is not a box labelled "device". It is a box with optional contents.

                CXL DEVICE
   ┌──────────────────────────────────┐
   │ link / interface logic           │  always
   │ CXL.io engine                    │  always
   │ CXL.cache engine               ? │  optional
   │ CXL.mem engine                 ? │  optional
   │ class dispatch                   │  always
   │ request / response queues        │  always
   │ outstanding-transaction table    │  always
   │ local cache                    ? │  only with a cache engine
   │ device-attached memory         ? │  only with a mem engine
   │ accelerator core(s)            ? │  only if it computes
   └──────────────────────────────────┘

The question marks are the chapter. Everything unmarked is present in every CXL device; everything marked is a design decision, and the combination of those decisions is what the device is.

5. The Three Protocols, and What Each Means for the Device

The CXL Consortium's public material defines the three sub-protocols compactly. Quoting the CXL 2.0 specification webinar:

• CXL.io — "PCIe based - discovery, register access, interrupts, initialization, I/O Virtualization, DMA"
• CXL.cache — "supports device caching of host memory with host processor orchestrating the coherency management"
• CXL.mem — "memory access protocol, host manages (coherency) device attached memory similar to host memory"

Read those three definitions as directions, because that is what they are, and the direction determines what hardware the device needs.

CXL.cache and CXL.mem point opposite ways, and that single observation resolves most confusion about device types. A device that wants to read host memory into its own cache needs .cache. A device that wants to offer its memory to the host needs .mem. These are unrelated wishes, so a device may want either, both, or neither — which is exactly why three named combinations exist.

Note also the phrase in the .cache definition: "with host processor orchestrating the coherency management". That asymmetry is the subject of Chapter 3.4 and the reason the device's coherence hardware is smaller than the host's.

6. The Three Device Types

The mapping is not arbitrary — it is the enumeration of useful combinations. From the CXL Consortium's "Representative CXL Usages" material:

The Consortium's example usages are PGAS NICs and NIC atomics for Type 1; GP-GPU and dense computation for Type 2; and memory bandwidth expansion, memory capacity expansion and storage-class memory for Type 3.

Three observations worth making before moving on.

.io is in every row. A device must be discoverable and configurable before it can be anything else — which is Chapter 2.4's inherited enumeration doing its job.

The fourth combination is missing on purpose. .io alone is a PCIe device with a CXL-capable link; it holds no coherent copies and offers no system memory, so it has no CXL device type. The types enumerate the coherent combinations.

Type 2 is not "Type 1 plus Type 3". It is a device that both caches host memory and exposes its own — and those two relationships interact, because the host must manage coherence for device-attached memory that the device itself may also be caching. That interaction is why Type 2 is the architecturally hardest class and why its treatment belongs in Module 10.

7. What the Device Owns, and Why

Apply Chapter 3.1's placement test — does the agent making this decision have the information the decision requires? — and the device side sorts cleanly.

The three "no" rows belong to 3.1, 3.4 and 3.3 respectively.

The four "yes" rows are Sections 8 to 12. This is the same principle producing a different answer, which is the useful thing about a principle.

8. RTL 1 — Capability, and the Type That Follows From It

Purpose

Make the declaration and the implementation the same fact.

// What a device declares it can do -- and what follows from that declaration.
//
// The teaching point: device TYPE is not an independent field. It is a
// CONSEQUENCE of which protocol engines the device actually implements.
// A design that stores type separately can disagree with itself.
//
// GENERIC teaching model. This is NOT a CXL capability register.
module dev_capability #(
  parameter bit HAS_IO_ENGINE    = 1'b1,   // what is actually BUILT
  parameter bit HAS_CACHE_ENGINE = 1'b0,
  parameter bit HAS_MEM_ENGINE   = 1'b0,
  parameter bit DECOUPLED        = 1'b0    // 1 = declare from a separate field
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       cfg_we,
  input  logic [2:0] cfg_claim,        // {mem, cache, io} a careless design writes
  output logic       supports_io,
  output logic       supports_cache,
  output logic       supports_mem,
  output logic [1:0] device_type,      // 0 = none/io-only, else 1/2/3
  output logic       claim_mismatch_err
);
  logic [2:0] claim_q;
  logic [2:0] built;
 
  assign built = {HAS_MEM_ENGINE, HAS_CACHE_ENGINE, HAS_IO_ENGINE};
 
  // DECOUPLED=1 is the bug shape: the declaration is whatever software wrote,
  // with no connection to what exists in the datapath.
  assign supports_io    = DECOUPLED ? claim_q[0] : HAS_IO_ENGINE;
  assign supports_cache = DECOUPLED ? claim_q[1] : HAS_CACHE_ENGINE;
  assign supports_mem   = DECOUPLED ? claim_q[2] : HAS_MEM_ENGINE;
 
  // Type is derived, never stored.
  always_comb begin
    if      (supports_io && supports_cache && supports_mem) device_type = 2'd2;
    else if (supports_io && supports_cache)                 device_type = 2'd1;
    else if (supports_io && supports_mem)                   device_type = 2'd3;
    else                                                    device_type = 2'd0;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      claim_q <= 3'b000;               // silence out of reset, never optimism
      claim_mismatch_err <= 1'b0;
    end else begin
      if (cfg_we) claim_q <= cfg_claim;
      // The check that catches the decoupled design: anything DECLARED that is
      // not BUILT is a promise the datapath cannot keep.
      if (({supports_mem, supports_cache, supports_io} & ~built) != 3'b000)
        claim_mismatch_err <= 1'b1;
    end
  end
endmodule

Reset leaves everything at zero. A device that comes out of reset claiming capability is making a promise about a state it has not reached — the same defect Chapter 2.2 logged as an optimistic reset value. Silence is the safe default.

Synthesis. With DECOUPLED = 0 this is entirely parameter-driven: the support outputs are constants, the type decode collapses to constants, and the only real logic is the mismatch check. Three flops and a comparator. That cost is the point — making the declaration structural is nearly free, and the alternative costs a field, a write path, and a class of bug.

Simulation evidence

Three instances, differing only in which engines are built:

=== EXP1: protocol engines built -> device type derived ===
  io+cache        -> Type 1  (io=1 cache=1 mem=0)
  io+cache+mem    -> Type 2  (io=1 cache=1 mem=1)
  io+mem          -> Type 3  (io=1 cache=0 mem=1)

The type was never written anywhere. It is a decode of what exists.

Now the decoupled variant — a device that builds .io and .mem (a Type 3 memory expander) but declares from a separate field:

=== EXP2: a device that declares more than it built ===
  before config: claims io=0 cache=0 mem=0 type=io-only err=0
  after  config: claims io=1 cache=1 mem=1 type=Type 2 err=1
  -> it now presents as Type 2 and has no cache engine at all

A memory expander is now advertising itself as an accelerator with a coherent cache. Every subsequent decision the host makes about this device is made against a device that does not exist. claim_mismatch_err is the only thing in the system that knows, and it exists only because the module compares the declaration against built.

9. RTL 2 — Class Dispatch

Purpose

Route each request to the engine that owns it, and refuse what this device does not implement.

// Route an inbound request to the protocol engine that owns it, and refuse
// anything this device does not implement.
//
// GENERIC teaching model -- the class input is an abstraction, not CXL header
// decode, and no CXL message format is modelled.
module dev_class_dispatch #(
  parameter bit STRICT = 1'b1        // 0 = the "route it anyway" bug shape
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       req_valid,
  input  logic [1:0] req_class,      // 0=io, 1=cache, 2=mem, 3=reserved
  input  logic       supports_io,
  input  logic       supports_cache,
  input  logic       supports_mem,
  input  logic       io_ready,
  input  logic       cache_ready,
  input  logic       mem_ready,
  output logic       to_io,
  output logic       to_cache,
  output logic       to_mem,
  output logic       refuse,
  output logic [1:0] refuse_reason,   // 1=unsupported class, 2=reserved, 3=engine busy
  output logic       req_ready,
  output logic       unsupported_accepted_err
);
  logic cls_ok, eng_ready;
 
  // Is this class implemented at all on this device?
  always_comb begin
    unique case (req_class)
      2'd0: cls_ok = supports_io;
      2'd1: cls_ok = supports_cache;
      2'd2: cls_ok = supports_mem;
      default: cls_ok = 1'b0;          // reserved encoding is never routable
    endcase
  end
 
  // STRICT=0 drops the capability term -- the request is routed to an engine
  // that may not exist. Everything downstream looks normal.
  assign to_io    = req_valid && (req_class == 2'd0) && (STRICT ? cls_ok : 1'b1) && io_ready;
  assign to_cache = req_valid && (req_class == 2'd1) && (STRICT ? cls_ok : 1'b1) && cache_ready;
  assign to_mem   = req_valid && (req_class == 2'd2) && (STRICT ? cls_ok : 1'b1) && mem_ready;
 
  assign req_ready = to_io || to_cache || to_mem;
  assign refuse    = req_valid && !req_ready;
 
  always_comb begin
    if      (req_valid && (req_class == 2'd3)) refuse_reason = 2'd2;
    else if (req_valid && !cls_ok)             refuse_reason = 2'd1;
    else if (refuse)                           refuse_reason = 2'd3;
    else                                       refuse_reason = 2'd0;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) unsupported_accepted_err <= 1'b0;
    // Accepting work for an engine the device does not have.
    else if (req_ready && !cls_ok) unsupported_accepted_err <= 1'b1;
  end
endmodule

Two refusal causes are deliberately distinguished. "I do not implement this class" is permanent and a configuration problem; "that engine is busy" is transient and a flow-control condition. A device that reports one code for both turns a permanent misconfiguration into something that looks like congestion, which is a debugging trap Chapter 2.5 hit from the other direction.

The reserved encoding gets its own path rather than falling into an else. An unexpected class value is a real event — a link error, a version mismatch, a software bug — and it should be reportable, not silently absorbed.

Simulation evidence

The strict and permissive variants on the Type 3 capability (io + mem, no cache):

=== EXP3: dispatch on a Type 3 device (io + mem, no cache) ===
  io request          strict: io=1 cache=0 mem=0 refuse=0 reason=0 | permissive: cache=0
  mem request         strict: io=0 cache=0 mem=1 refuse=0 reason=0 | permissive: cache=0
  CACHE request       strict: io=0 cache=0 mem=0 refuse=1 reason=1 | permissive: cache=1
  reserved encoding   strict: io=0 cache=0 mem=0 refuse=1 reason=2 | permissive: cache=0

Row three is the defect. The permissive dispatcher asserts to_cache on a device with no cache engine — the request is handed to a block that does not exist, and what happens next depends entirely on what that unconnected interface does in silicon. The strict version refuses with reason 1, which names the cause.

Rows one, two and four are identical between the two designs, which is the usual shape: the bug is invisible on every request the device is actually supposed to receive.

10. RTL 3 — The Outstanding-Transaction Table

Purpose

Remember what was accepted, so a completion can be returned through the right engine to the right requester.

The host needs a table for the reason Chapter 3.1 §9 gave — matching returning data to the request that asked for it. The device needs one for a different reason: a completion must leave through the protocol class it arrived on, and the device is the only place that association is recorded.

// Every request the device has accepted and not yet responded to.
//
// GENERIC teaching model. NOT a CXL tag or transaction-ID mechanism.
module dev_outstanding #(
  parameter int unsigned NSLOT    = 8,
  parameter bit          BY_CLASS = 1'b1   // 0 = free the lowest busy slot instead
) (
  input  logic                     clk,
  input  logic                     rst_n,
  input  logic                     alloc,
  input  logic [1:0]               alloc_class,
  input  logic [3:0]               alloc_requester,
  input  logic                     done,
  input  logic [$clog2(NSLOT)-1:0] done_slot,
  output logic                     alloc_ok,
  output logic [$clog2(NSLOT)-1:0] alloc_slot,
  output logic [1:0]               done_class,
  output logic [3:0]               done_requester,
  output logic                     done_matched,
  output logic                     unknown_done_err,
  output logic                     dup_done_err,
  output logic                     slot_reuse_err,
  output logic [NSLOT-1:0]         busy_vec,
  output logic [7:0]               occupancy_q,
  output logic [7:0]               max_occupancy_q
);
  logic [NSLOT-1:0] busy_q;
  assign busy_vec = busy_q;
  logic [1:0]       cls_q     [NSLOT-1:0];
  logic [3:0]       rqr_q     [NSLOT-1:0];
  logic [NSLOT-1:0] retired_q;          // seen a completion for this slot already
  // ... free-slot search elided ...
 
  // No slot means NO ACCEPT. The device backpressures rather than losing track.
  assign alloc_ok = alloc && found;
 
  assign done_class     = cls_q[done_slot];
  assign done_requester = rqr_q[done_slot];
  assign done_matched   = done && busy_q[done_slot];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      if (alloc_ok) begin
        if (busy_q[alloc_slot]) slot_reuse_err <= 1'b1;
        busy_q[alloc_slot]    <= 1'b1;
        retired_q[alloc_slot] <= 1'b0;
        cls_q[alloc_slot]     <= alloc_class;
        rqr_q[alloc_slot]     <= alloc_requester;
      end
      if (done) begin
        if (!busy_q[done_slot]) begin
          // Either the slot was freed too early, or this completion is a
          // duplicate. Both mean the device is about to route a response using
          // stale class and requester fields.
          unknown_done_err <= 1'b1;
          if (retired_q[done_slot]) dup_done_err <= 1'b1;
        end
        if (BY_CLASS) begin
          busy_q[done_slot]      <= 1'b0;   // free the slot the completion names
          retired_q[done_slot]   <= 1'b1;
        end else begin
          busy_q[lowest_busy]    <= 1'b0;   // BUG shape: free the wrong slot
          retired_q[lowest_busy] <= 1'b1;
        end
      end
      // ... occupancy and maximum ...
    end
  end
endmodule

retired_q exists to separate two failures that look identical. A completion for a free slot could mean the slot was released too early, or it could mean a duplicate arrived. unknown_done_err fires for both; dup_done_err fires only when this slot has already seen a completion. One extra bit per slot turns "something is wrong with the tracking" into a named cause.

Simulation evidence — matching, freeing, and freeing the wrong thing

Three requests with distinct class and requester, so a mismatch is visible:

=== EXP4: slot allocation, completion matching, and freeing ===
  3 allocated. busy: correct=00000111 wrong-slot-free=00000111
 
  a completion arrives naming slot 2:
    correct         : matched=1 class=2 requester=3
    wrong-slot-free : matched=1 class=2 requester=3
 
  busy afterwards:
    correct         : busy=00000011  <-- slot 2 retired, the one that finished
    wrong-slot-free : busy=00000110  <-- slot 0 retired, still in flight

The completion itself looks correct in both. Both return class 2 and requester 3, because both read the fields the completion named. The divergence is in what got released, and it is invisible at this point unless you look at the busy vector.

The duplicate check, run before anything is reallocated:

  a second completion for slot 2 (duplicate):
    correct         : unknown_done=1 duplicate_done=1

Then a reallocation, and the consequence:

  reallocating (class=1 requester=7):
    correct picks slot 2, wrong-slot-free picks slot 0
 
  the original slot-0 request (class=0 requester=5) now completes:
    correct         : class=0 requester=5  <-- as issued
    wrong-slot-free : class=1 requester=7  <-- a DIFFERENT request

That last line is the whole point of the table. The buggy device returns a completion for an .io request through the .cache engine, addressed to requester 7 instead of requester 5. Nothing about the request was corrupted — the request completed correctly inside the device. What was lost was the device's memory of whose it was.

Two failures, one root cause: slot 2 leaked permanently (it will never be reused), and slot 0's identity was overwritten while its request was live.

Simulation evidence — exhaustion

=== EXP5: slot exhaustion ===
  12 allocations into 8 slots: alloc_ok=0 occupancy=8 max=8
  -> the device refuses to accept rather than losing track

Same decision as the host in Chapter 3.1, for the same reason: a stall costs cycles and a lost association costs correctness. max_occupancy_q is the sizing input — if it never approaches NSLOT, the table is oversized; if it pins at NSLOT, the device is issue-limited and more slots would buy throughput.

11. RTL 4 — Decoupling the Link From the Device

Purpose

The link runs at its own rate; the accelerator or memory controller behind it runs at another. Something must absorb the difference — and what that something does when the local side stalls is an architectural decision.

// The decoupling queue between the CXL side and whatever the device actually
// does -- an accelerator core, a memory controller, a NIC datapath.
//
// This queue is strictly in-order, which is the simple and usually correct
// choice. Its consequence is head-of-line blocking: a head whose target is
// stalled holds up every follower, including followers whose target is idle.
// The architectural answer is not a cleverer queue -- it is MORE queues, one
// per target class.
//
// GENERIC teaching model.
module dev_local_queue #(
  parameter int unsigned DEPTH = 4
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        in_valid,
  input  logic [7:0]  in_id,
  input  logic        in_slow,      // this request targets the slow resource
  output logic        in_ready,
  input  logic        fast_ready,
  input  logic        slow_ready,
  output logic        out_valid,
  output logic [7:0]  out_id,
  output logic        out_slow,
  output logic [7:0]  level_q,
  output logic        full,
  output logic        hol_stall,    // head blocked while a follower could go
  output logic [15:0] stall_cycles_q,
  output logic        overflow_err
);
  assign full     = (level_q >= DEPTH[7:0]);
  assign in_ready = !full;
 
  // Strictly in-order: only the head is ever a candidate, and it goes only if
  // ITS target is ready.
  assign out_valid = (level_q != 0) && (slow_q[0] ? slow_ready : fast_ready);
 
  // Diagnostic only -- not a datapath signal. It answers "is anything behind
  // the head being held up for no reason of its own?"
  always_comb begin
    follower_ready = 1'b0;
    for (i = 1; i < DEPTH; i = i + 1)
      if (i < level_q && (slow_q[i] ? slow_ready : fast_ready)) follower_ready = 1'b1;
  end
  assign hol_stall = !out_valid && follower_ready;
 
  // ... shift/push/pop, and stall_cycles_q accumulating while hol_stall ...
endmodule

hol_stall is a diagnostic, not a datapath signal, and the distinction is worth being explicit about. It does not change what the queue does. It measures how much the queue's ordering discipline is costing, which is the number you need before deciding whether to build a second queue.

in_ready = !full is the backpressure edge, and it is the only mechanism by which the device's internal rate reaches the link. A device that accepts requests it cannot place has not gained throughput; it has stopped tracking them.

Simulation evidence

Identical stimulus into one shared queue and into two per-class queues. A slow-target request arrives first, then two fast-target requests, with the slow target stalled:

=== EXP6: a slow local resource behind one queue, then behind two ===
  slow request first, 2 fast behind it, slow target stalled:
    one shared queue : out_valid=0 level=3 hol_stall=1 stalled_cycles=7
    two class queues : fast out_valid=0 level=0 | slow level=1
  -> the fast queue drained; the shared queue is still holding both
 
  slow target becomes ready: shared out_valid=1 level=2

Three requests stuck behind one, for seven measured cycles, with two of them targeting a resource that was ready the whole time. The split configuration drained both fast requests immediately and left one request waiting — which is the correct amount of waiting, because exactly one request actually depended on the stalled resource.

That is the general result: head-of-line blocking is not solved by a smarter queue, it is solved by not sharing one. The cost is real — two queues means two sets of storage and two ready paths — and stall_cycles_q is how you decide whether to pay it.

12. RTL 5 — Counters

Purpose

The device is the only thing that knows why it refused, stalled, or errored.

// Device-side observability. Hardware counts events; software computes rates.
// GENERIC teaching model.
module dev_counters (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        acc_io,
  input  logic        acc_cache,
  input  logic        acc_mem,
  input  logic        refused,
  input  logic        queue_full,
  input  logic        completion,
  input  logic        error,
  output logic [15:0] n_io_q,
  output logic [15:0] n_cache_q,
  output logic [15:0] n_mem_q,
  output logic [15:0] n_refused_q,
  output logic [15:0] full_cycles_q,
  output logic [15:0] max_full_run_q,
  output logic [15:0] n_cmpl_q,
  output logic [15:0] n_err_q
);
  logic [15:0] run_q;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* all zero */ end
    else begin
      if (acc_io)     n_io_q      <= n_io_q      + 16'd1;
      if (acc_cache)  n_cache_q   <= n_cache_q   + 16'd1;
      if (acc_mem)    n_mem_q     <= n_mem_q     + 16'd1;
      if (refused)    n_refused_q <= n_refused_q + 16'd1;
      if (completion) n_cmpl_q    <= n_cmpl_q    + 16'd1;
      if (error)      n_err_q     <= n_err_q     + 16'd1;
      // A total tells you how often; a maximum run tells you how bad.
      if (queue_full) begin
        full_cycles_q <= full_cycles_q + 16'd1;
        run_q         <= run_q + 16'd1;
        if (run_q + 16'd1 > max_full_run_q) max_full_run_q <= run_q + 16'd1;
      end else run_q <= '0;
    end
  end
endmodule

Every input to this module must be a single-cycle pulse, and that is not a documentation detail. The same counter fed a level instead of an event produces a number that looks plausible and means something else entirely — measured in the next block.

Simulation evidence

=== counters ===
  io=1 cache=0 mem=1 refused=2 completions=2
  errors counted from an EDGE   = 1    <-- the number of events
  errors counted from the LEVEL = 27   <-- cycles the flag was high

One error, reported as twenty-seven. Two instances of the identical counter, one fed a rising-edge pulse and one fed the sticky error flag directly. Neither is broken; the second was wired to the wrong kind of signal.

The reason this matters more than it looks: 27 is not an obviously wrong number. It is a plausible error count that would drive a real decision — and it scales with how long the flag stays set, which means the same defect reported differently on a longer run.

13. Assertions

Icarus does not execute concurrent SVA, so these were not run; the table gives the procedural check that stood in for each.

// D1 — a device never declares a protocol it did not build.
a_claim_is_built: assert property (@(posedge clk) disable iff (!rst_n)
  ({supports_mem, supports_cache, supports_io} & ~built) == '0);
 
// D2 — the device type always agrees with the declared engine set.
a_type_derived: assert property (@(posedge clk) disable iff (!rst_n)
  (device_type == 2'd2) == (supports_io && supports_cache && supports_mem));
 
// D3 — nothing out of reset is claimed.
a_reset_silent: assert property (@(posedge clk)
  $fell(rst_n) |=> ({supports_mem, supports_cache, supports_io} == '0) || !DECOUPLED);
 
// D4 — dispatch is one-hot, and only to an implemented engine.
a_dispatch_legal: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({to_io, to_cache, to_mem}) &&
  (to_cache |-> supports_cache) && (to_mem |-> supports_mem));
 
// D5 — every valid request is dispatched or explicitly refused with a cause.
a_req_resolved: assert property (@(posedge clk) disable iff (!rst_n)
  req_valid |-> (req_ready ^ (refuse && refuse_reason != '0)));
 
// D6 — a live slot is never reallocated.
a_no_slot_reuse: assert property (@(posedge clk) disable iff (!rst_n)
  alloc_ok |-> !busy_q[alloc_slot]);
 
// D7 — every completion names a live slot.
a_cmpl_live: assert property (@(posedge clk) disable iff (!rst_n)
  done |-> busy_q[done_slot]);
 
// D8 — the completion frees the slot it named, and no other.
a_frees_named_slot: assert property (@(posedge clk) disable iff (!rst_n)
  (done && busy_q[done_slot]) |=> !busy_q[$past(done_slot)]);
 
// D9 — the queue never exceeds its depth, and never accepts when full.
a_queue_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (level_q <= DEPTH) && (full |-> !in_ready));
 
// D10 — LIVENESS: a queued request eventually leaves once its target is ready.
a_queue_drains: assert property (@(posedge clk) disable iff (!rst_n)
  (level_q != 0 && fast_ready && slow_ready) |-> ##[1:DEPTH] (level_q == 0));

D8 is the property that separates the two tables, and it is worth noticing why: D7 passes on the buggy design. The completion did name a live slot. What went wrong happened afterwards, to a different slot, and only a property that relates the freed slot to the named one can express it.

D10 is the liveness property. Everything else here is safety, and a device whose queue never drains presents as a hung link, not as a protocol error.

14. Debug Lab

// Firmware writes the capability field during initialisation.
assign supports_cache = claim_q[1];
assign device_type    = type_q;        // stored, not derived

  before config: claims io=0 cache=0 mem=0 type=io-only err=0
  after  config: claims io=1 cache=1 mem=1 type=Type 2 err=1
  -> it now presents as Type 2 and has no cache engine at all

assign supports_cache = HAS_CACHE_ENGINE;      // the same parameter that builds it
// type is a decode, never a register
if      (supports_io && supports_cache && supports_mem) device_type = 2'd2;
else if (supports_io && supports_cache)                 device_type = 2'd1;
else if (supports_io && supports_mem)                   device_type = 2'd3;

// Decode the class and send it to the right engine.
assign to_cache = req_valid && (req_class == 2'd1) && cache_ready;

  CACHE request   strict: io=0 cache=0 mem=0 refuse=1 reason=1 | permissive: cache=1

assign to_cache = req_valid && (req_class == 2'd1) && supports_cache && cache_ready;
assign refuse_reason = (!cls_ok) ? UNSUPPORTED : (refuse ? ENGINE_BUSY : NONE);

// A completion arrived, so retire the oldest outstanding entry.
if (done) busy_q[lowest_busy] <= 1'b0;

  busy afterwards:
    correct         : busy=00000011  <-- slot 2 retired, the one that finished
    wrong-slot-free : busy=00000110  <-- slot 0 retired, still in flight
 
  the original slot-0 request (class=0 requester=5) now completes:
    correct         : class=0 requester=5  <-- as issued
    wrong-slot-free : class=1 requester=7  <-- a DIFFERENT request

if (done) begin
  if (!busy_q[done_slot]) unknown_done_err <= 1'b1;
  busy_q[done_slot] <= 1'b0;
end

// One queue between the link and the device. Simple and ordered.
dev_local_queue #(.DEPTH(4)) u_q (.in_valid(any_req), ...);

  slow request first, 2 fast behind it, slow target stalled:
    one shared queue : out_valid=0 level=3 hol_stall=1 stalled_cycles=7
    two class queues : fast out_valid=0 level=0 | slow level=1

dev_local_queue #(.DEPTH(4)) u_fast (.in_valid(req && !targets_slow), ...);
dev_local_queue #(.DEPTH(4)) u_slow (.in_valid(req &&  targets_slow), ...);

// Count errors.
dev_counters u_c (.error(unknown_done_err), ...);   // a STICKY flag, not a pulse

  errors counted from an EDGE   = 1    <-- the number of events
  errors counted from the LEVEL = 27   <-- cycles the flag was high

always_ff @(posedge clk) err_q <= unknown_done_err;
assign err_pulse = unknown_done_err && !err_q;
dev_counters u_c (.error(err_pulse), ...);

15. Design Review — Reading a Device Implementation

What a reviewer asks, in order:

On capability. Is the protocol-support declaration derived from the same parameters that instantiate the engines, or is it a separate field? Is device type stored anywhere? What does the device claim out of reset? Is there a check that anything claimed is built?

On dispatch. Does the routing condition include capability, or only the class? Are "unsupported" and "busy" distinguishable in the refusal? What happens to a reserved class encoding — is it reported, or does it fall into an else?

On the outstanding table. Which slot does a completion free — the one it names, or a positional one? What happens to a completion for a free slot? Can a duplicate be distinguished from a premature release? Does the device stall when the table is full, or wrap?

On queueing. Is there one queue or one per target class? Is head-of-line stalling measured or assumed absent? Does in_ready actually reach the link, or is backpressure absorbed internally?

On counters. Is every counter input a pulse? Are maxima recorded, or only totals?

And the structural question. For every decision this device makes, is the information it needs genuinely local? Anything that needs the system map, the topology, or other agents' caches is in the wrong place — and it will work in a bench with one device in it.

16. Quantitative Reasoning — Sizing the Outstanding Table

The table's depth is not a taste decision. It follows from Little's Law.

If the device sustains R requests per cycle at an average service latency of L cycles, the number of requests in flight is:

in-flight  =  R × L

So a device that must sustain one request every two cycles against a 40-cycle internal latency needs 0.5 × 40 = 20 slots merely to avoid being issue-limited. With eight slots it will stall, and its achievable rate is capped at:

R_max  =  NSLOT / L  =  8 / 40  =  0.2 requests per cycle

— a 2.5× shortfall that no amount of link bandwidth fixes, because the limit is the device's own bookkeeping.

This is why max_occupancy_q earns its area. It answers the sizing question directly: an occupancy that never approaches NSLOT means the table is oversized and the area could go elsewhere; an occupancy pinned at NSLOT means the device is issue-limited and more slots would convert directly into throughput. The measured run pinned at 8 of 8 under a deliberate flood, which is the signature to recognise.

17. How This Appears in Real Engineering

CXL / SoC architect

The first decision is which engines to build, because it determines the device type, the verification surface and the coherence relationship with the host — all three at once. A .cache engine is not an incremental feature: it commits the device to participating in coherence, with snoop-response hardware and the latency obligations that come with it.

RTL engineer

Five blocks, four disciplines. Derive capability from what is instantiated. Include capability in the dispatch condition. Free the slot the completion names. Split queues by target class when head-of-line stalling is measured to matter — and measure it before deciding.

Verification engineer

Three requirements the measured defects justify. Every class against every device type, because the dispatch bug only appears for a class the device does not implement. Out-of-order completion with distinct class and requester per slot, because in-order completion cannot distinguish the two table designs. And a deliberately exhausted table, because the backpressure path is otherwise never exercised.

Coherency engineer

Only Type 1 and Type 2 concern you at all, and what the device owes the coherence protocol is a response to a snoop within a bounded time. Chapter 3.4 covers the semantics; from the device side the obligation is that the cache engine cannot be starved by other classes, which is what the per-class queueing in Section 11 protects.

Performance engineer

Two numbers to instrument first: max_occupancy_q on the outstanding table, which tells you whether the device is issue-limited, and stall_cycles_q on the queue, which tells you whether ordering is costing throughput. Section 16's R_max = NSLOT / L converts the first into a bandwidth ceiling.

Firmware and system software

Read the device's capability rather than assuming it from a product name or a type field, and treat a mismatch between claimed and functional capability as a device fault rather than a configuration to work around. The counters are the device's only account of why it refused something.

System integrator

The refusal reasons are the interface that matters here. "Unsupported class" is a configuration or part-selection problem; "engine busy" is congestion. A device that reports one code for both will send every investigation to the wrong team.

18. Common Misconceptions

19. Interview Reasoning

20. Exercises

1. Derive the fourth combination. A device implements .io only. Which device type is it? What does it mean that the answer is "none"? Now extend dev_capability to report an explicit io_only indication rather than folding it into type 0 — and argue whether that is worth the encoding.

2. Size the table. A Type 3 device has an internal read latency of 120 cycles and must sustain 0.25 requests per cycle. How many slots does it need? What does max_occupancy_q read if you build 16?

3. Break D8 differently. The wrong-slot bug freed lowest_busy. Write a different bug with the same symptom — for example, freeing on a stale registered copy of done_slot — and determine whether property D8 still catches it.

4. Cost the split. Using stall_cycles_q from a workload of your choosing, compute what fraction of cycles the shared queue spent head-of-line stalled. At what fraction would you pay for a second queue, and what would you need to know about the ordering requirements before you could?

5. Find the missing refusal. The dispatcher distinguishes unsupported, reserved and busy. Name a fourth reason a real device might refuse a request that none of these three covers, and decide whether it belongs in this module or somewhere else.

21. Summary

A CXL device is defined by which protocol engines it implements. .io is always present; .cache lets the device hold coherent copies of host memory; .mem lets the host use device-attached memory as system memory. The two coherent protocols point in opposite directions, which is why their combinations enumerate to exactly three named types — Type 1 (.io + .cache), Type 2 (all three), Type 3 (.io + .mem).

The type is derived, not declared. Measured, a device that stored its declaration separately presented as a Type 2 accelerator with no cache engine built, and the only thing in the system that knew was a comparator between claimed and built.

Everything else the device owns passes Chapter 3.1's placement test — it needs only local knowledge. Dispatch must include capability in the routing condition, or a coherent request lands in a block that does not exist. The outstanding table must free the slot the completion names: freeing positionally delivered an .io completion through the .cache engine to a different requester, and leaked a slot permanently, with nothing observable at the moment the completion arrived. The queue must decouple the link from the device's own rate, and its in-order discipline cost seven measured cycles of head-of-line stalling for traffic that never touched the stalled resource. The counters must be fed events: one error reported as 27 is the shape of that mistake.

Two habits carry beyond this chapter. Make the declaration structural — derive what you advertise from what you instantiate, and the entire class of advertise-without-datapath bugs stops being possible. And relate the effect to the cause in your properties: done |-> busy_q[done_slot] passes on a table that frees the wrong entry, because the completion really did name a live slot.

22. What Comes Next

Host and device are both accounted for, and both were reasoned about as a single link with one device on it. Chapter 3.3 removes that assumption: several hosts, several devices, and something in between that has to route, arbitrate and isolate. The measured consequences — oversubscription, head-of-line blocking at scale, and what a stale routing table does — are the fabric's versions of problems this chapter met at device scale.

For adjacent material: The CXL Host is the other half of the placement argument, What Is CXL? has the protocol overview, and Architectural Goals has the obligations a device must satisfy. The path is on the CXL tutorials index.