The Three Protocol Layers

The three protocols have appeared throughout this course — named in Chapter 2.1, used to define device types in Chapter 3.2, and treated in Chapter 4.3 as a source of divergent requirements.

This chapter puts them side by side and asks the questions the earlier treatments deferred: what is each one actually for, who initiates it, and what does running all three at once cost?

1. The Engineering Problem — Three Names, One Link

An engineer new to CXL learns three names quickly and a useful distinction slowly. "CXL has three protocols" is easy; "which one does a NIC need, and why does a memory expander not have a cache" is not.

The confusion has a specific source: the three are usually presented as a list, and a list implies peers. They are not peers. They differ in direction, in who is even allowed to start a transaction, in which device types implement them, and in what they demand from the layers below.

Get the distinction wrong and two concrete things go wrong. A device is specified with the wrong protocol set — which Chapter 3.2 showed makes it the wrong device type entirely. And traffic is accepted from the wrong initiator, which is a security-adjacent failure rather than a performance one.

2. The One-Sentence Model

The three protocols are distinguished by direction before anything else — .io and .mem are host-initiated and .cache is the one the device initiates — and that single asymmetry explains the device types, the latency targets, and why a device implementing all three is doing something categorically harder than one implementing two.

Call it the who-asks model. Everything else follows from it.

3. What This Chapter Owns

A deliberate boundary. Chapter 4.3 asked what the layer must do given that classes differ. This chapter asks what the classes are. Modules 7 to 9 take each one apart individually; this is the comparison that makes those modules navigable.

4. The Three, From the Consortium

The definitions, quoted from CXL Consortium material and already used in Chapter 3.2:

Read the three definitions for direction and the structure appears immediately.

5. Direction Is the Discriminator

The why follows from the direction: the host must set the device up, the device wants to compute on host data, and the host wants to use device memory as system memory.

.cache is the only class where the device is the requester, and that is the whole reason it is the class that makes a device a coherence participant. A device that never initiates anything toward host memory needs no cache, no snoop-response logic, and no coherence obligations — it is a target, not a participant.

Two consequences worth drawing out.

.io and .mem look similar and are not. Both are host-initiated. But .io reaches registers — small, side-effecting, order-sensitive — and .mem reaches memory — large, idempotent per address, latency-critical. Same direction, opposite characters, which is why Chapter 4.3 gave them different ordering rules.

The asymmetry the Consortium names is visible here. "Asymmetric Complexity: eases burdens of cache coherent interface designs" means the device's coherence job is the small half — hold copies, answer snoops — while the host orchestrates. The device initiates .cache requests and does not resolve them, which is Chapter 3.4's model seen from the protocol side.

6. The Comparison Table

Everything this course has established about the three, in one place.

Switch behaviour is per Chapter 3.3: .io is decoded as PCIe, .mem gets address look-up with fan-out and interleave, and .cache is direct routed with a single caching device within a hierarchy in CXL 2.0.

The last row is worth re-reading. Chapter 3.3 established from Consortium material that in CXL 2.0 switching, .io is decoded as PCIe, .mem gets address look-up with fan-out and interleave, and .cache is direct routed with a single caching device within a hierarchy. Three protocols, three different fabric behaviours — which is the strongest evidence available that they are not variations of one mechanism.

The .io row is present in every device, and that is not a formality. A device must be discoverable and configurable before it can be anything else, which is Chapter 2.4's inherited enumeration doing its job at the protocol level.

7. Quantitative Reasoning — Coexistence Is Not Free

Three classes sharing one link raises a question the individual protocols do not: what does each one get?

With N classes all offering traffic and fair rotation:

share per class  =  1/N       worst wait  =  N − 1 cycles

For three classes that is a third each and a two-cycle bound. Section 11 measures exactly that.

Under fixed priority the arithmetic degenerates:

share of the top class  =  1        share of the others  =  0

which is not an approximation — Section 11 measures 150 and 0 and 0.

The interesting part is what each class loses. A third of the link is a real reduction for a class that could have had all of it, and the classes value it differently: .io is bounded in volume so a third is ample, while .cache and .mem have a latency target and care about when they are served rather than how much they get in aggregate. That is why Chapter 4.3's per-class resources matter more than the raw share — the share is about throughput and the wait bound is about latency, and only the second speaks to the Consortium's stated target.

8. Teaching-model boundary

9. RTL 1 — Who Is Allowed to Ask

Purpose

Enforce the discriminator.

// Who is allowed to initiate what.
//
// The three CXL protocols are not symmetric in direction, and that asymmetry
// is the clearest way to tell them apart:
//   .io     host configures the device        (host initiates)
//   .cache  device reaches into host memory   (DEVICE initiates)
//   .mem    host reaches into device memory   (host initiates)
//
// ARCHITECTURAL TEACHING MODEL. No CXL message, opcode or direction rule is
// modelled; the class and initiator encodings are teaching values.
module direction_check #(
  parameter bit CHECK_DIRECTION = 1'b1     // 0 = accept any initiator
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        req_valid,
  input  logic [1:0]  req_class,        // 0=io, 1=cache, 2=mem
  input  logic        from_device,      // 1 = device initiated, 0 = host
  output logic        accept,
  output logic        reject,
  output logic        wrong_direction_err,
  output logic [15:0] n_accept_q,
  output logic [15:0] n_reject_q
);
  logic expect_device, dir_ok;
 
  // Only .cache is device-initiated.
  assign expect_device = (req_class == 2'd1);
  assign dir_ok        = (from_device == expect_device);
 
  assign accept = req_valid && (CHECK_DIRECTION ? dir_ok : 1'b1);
  assign reject = req_valid && !accept;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      if (accept) n_accept_q <= n_accept_q + 16'd1;
      if (reject) n_reject_q <= n_reject_q + 16'd1;
      // Accepted a request whose initiator does not match its class.
      if (accept && !dir_ok) wrong_direction_err <= 1'b1;
    end
  end
endmodule

This is a security-adjacent check, not merely a correctness one. A device that can initiate .mem requests toward its own memory is doing something the model does not define; a host that can initiate .cache requests is claiming a role the architecture assigns to the device. Neither is a performance bug.

Synthesis. One comparison. The cost of enforcing direction is a single equality against a constant derived from the class, which is worth noting because the check is sometimes omitted on the grounds that it costs area.

Simulation evidence

=== EXP1: who initiates each class ===
  .io from the HOST                  strict accept=1 | unchecked accept=1
  .io from the DEVICE                strict accept=0 | unchecked accept=1
  .cache from the DEVICE             strict accept=1 | unchecked accept=1
  .cache from the HOST               strict accept=0 | unchecked accept=1
  .mem from the HOST                 strict accept=1 | unchecked accept=1
  .mem from the DEVICE               strict accept=0 | unchecked accept=1
 
  accepted: strict=3 unchecked=6 | rejected: strict=3
  -> .cache is the only class the DEVICE initiates

Exactly half the combinations are legal, and the pattern is the discriminator: .io and .mem from the host, .cache from the device. The unchecked variant accepted all six — and every one of the three legal cases behaves identically between the designs, which is the familiar shape.

10. RTL 2 — Which Classes a Device Serves

Purpose

Connect the three protocols to the three device types, structurally.

// A device implements a SUBSET of the three protocols, and traffic for a class
// it did not build must never be acted on.
//
//   io + cache        -> Type 1
//   io + cache + mem  -> Type 2
//   io + mem          -> Type 3
//
// ARCHITECTURAL TEACHING MODEL. The type mapping follows CXL Consortium
// published material; the gating logic here is a teaching abstraction and no
// CXL capability mechanism is modelled.
module class_capability_gate #(
  parameter bit HAS_IO    = 1'b1,
  parameter bit HAS_CACHE = 1'b0,
  parameter bit HAS_MEM   = 1'b0
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        req_valid,
  input  logic [1:0]  req_class,
  output logic        serve,
  output logic        refuse,
  output logic [1:0]  device_type,
  output logic        served_unbuilt_err,
  output logic [15:0] n_served_q  [2:0],
  output logic [15:0] n_refused_q
);
  always_comb begin
    unique case (req_class)
      2'd0: have = HAS_IO;
      2'd1: have = HAS_CACHE;
      2'd2: have = HAS_MEM;
      default: have = 1'b0;
    endcase
  end
 
  // Type is a decode of what exists, never a stored field.
  always_comb begin
    if      (HAS_IO && HAS_CACHE && HAS_MEM) device_type = 2'd2;
    else if (HAS_IO && HAS_CACHE)            device_type = 2'd1;
    else if (HAS_IO && HAS_MEM)              device_type = 2'd3;
    else                                     device_type = 2'd0;
  end
 
  assign serve  = req_valid && have;
  assign refuse = req_valid && !have;
  // ... per-class served counts, refused count, and the unbuilt check ...
endmodule

The type decode is a parameter function, so with the parameters fixed it collapses to a constant at elaboration. That is Chapter 3.2's "type is derived, not declared" made structural: there is no register anywhere that could disagree with what was instantiated.

Simulation evidence

Three instances, one per device type, driven with all three classes:

=== EXP2: which classes each device type serves ===
  device types derived: Type1=Type 1 Type2=Type 2 Type3=Type 3
 
  .io      : Type 1 serve=1 | Type 2 serve=1 | Type 3 serve=1
  .cache   : Type 1 serve=1 | Type 2 serve=1 | Type 3 serve=0
  .mem     : Type 1 serve=0 | Type 2 serve=1 | Type 3 serve=1
 
  refused: Type 1=3 Type 2=0 Type 3=3

Read the middle column. Type 2 refused nothing — it is the only type that serves all three, which is precisely why Chapter 3.2 called it the architecturally hardest class. Types 1 and 3 each refused three requests, and their refusals are complementary: Type 1 refuses .mem, Type 3 refuses .cache.

The .io row is all ones. Every type serves it, which is the structural expression of "a device must be discoverable before it can be anything else".

11. RTL 3 — Running All Three at Once

Purpose

Measure what coexistence actually costs on a Type 2 device.

// A Type 2 device runs all three protocols at once. This model checks that
// concurrent traffic in all three classes is served without any class being
// shut out -- the coexistence property, measured rather than assumed.
//
// ARCHITECTURAL TEACHING MODEL. No CXL scheduling or arbitration rule is
// modelled; the rotation below is a teaching abstraction.
module class_coexist #(
  parameter bit FAIR = 1'b1              // 0 = fixed priority
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        req_io, req_cache, req_mem,
  input  logic        ready,
  output logic        gnt_io, gnt_cache, gnt_mem,
  output logic [15:0] n_io_q, n_cache_q, n_mem_q,
  output logic [7:0]  wait_io_q, wait_cache_q, wait_mem_q,
  output logic [7:0]  max_wait_q,
  output logic        multi_grant_err,
  output logic        starved_err
);
  always_comb begin
    gnt = 3'b000; found = 1'b0;
    if (ready) begin
      if (!FAIR) begin
        for (j = 0; j < 3; j = j + 1)
          if (!found && req[j]) begin gnt[j] = 1'b1; found = 1'b1; end
      end else begin
        for (j = 0; j < 3; j = j + 1) begin
          k = (ptr_q + j) % 3;                 // rotate the starting point
          if (!found && req[k]) begin gnt[k] = 1'b1; found = 1'b1; end
        end
      end
    end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    // ... per-class counts, pointer advance, per-class wait counters ...
    // The maximum across ALL THREE, never one of them.
    if (req_io    && !gnt_io    && wait_io_q    + 8'd1 > max_wait_q) max_wait_q <= wait_io_q + 8'd1;
    if (req_cache && !gnt_cache && wait_cache_q + 8'd1 > max_wait_q) max_wait_q <= wait_cache_q + 8'd1;
    if (req_mem   && !gnt_mem   && wait_mem_q   + 8'd1 > max_wait_q) max_wait_q <= wait_mem_q + 8'd1;
  end
endmodule

Three separate wait counters and one maximum over all of them. That structure is the direct response to Chapter 1.2's recorded defect and to Chapter 4.1's measurement of it — a maximum computed from one class is a number about that class, not about the arbiter. Mutation Q4 in Section 13 confirms the checker for this actually fires.

Simulation evidence

150 cycles with all three classes requesting continuously:

=== EXP3: all three classes concurrently on a Type 2 device ===
  fair          : io=50 cache=50 mem=50  worst wait=2
  fixed priority: io=150 cache=0 mem=0   worst wait=150
 
  -> fixed priority served ONLY .io for 150 cycles
  -> fair coexistence served all three

Exactly a third each, and a two-cycle bound — which is N − 1 for three classes and is the design's stated guarantee. Fixed priority gave .io everything.

Notice which classes fixed priority starved: .cache and .mem, the two the Consortium targets at near-CPU-cache latency, starved by the one with no stated latency target. That is the same harm Chapter 4.3 measured at the resource level, appearing again at the scheduling level — and it recurs because .io is naturally first in any list one writes.

The rotation resumes rather than restarts. After the stall at cycle 4, .cache is served next — which is where the pointer was — rather than .io. A pointer reset on stall would give .io an extra turn after every stall, and since stalls are common under load, that is a systematic bias which only appears when the link is busy.

12. Assertions

Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog. Not executed; each maps to its procedural stand-in and mutation.

// SAFETY -------------------------------------------------------------------
// U1 — a request is accepted only from its defined initiator.
a_direction: assert property (@(posedge clk) disable iff (!rst_n)
  accept |-> (from_device == (req_class == CLASS_CACHE)));
 
// U2 — a device never serves a class it did not implement.
a_only_built: assert property (@(posedge clk) disable iff (!rst_n)
  serve |-> ((req_class == CLASS_IO)    ? HAS_IO    :
             (req_class == CLASS_CACHE) ? HAS_CACHE : HAS_MEM));
 
// U3 — device type always agrees with the implemented set.
a_type_derived: assert property (@(posedge clk) disable iff (!rst_n)
  (device_type == 2'd2) == (HAS_IO && HAS_CACHE && HAS_MEM));
 
// U4 — one class is granted at a time, and never without the link.
a_grant_legal: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({gnt_io, gnt_cache, gnt_mem}) &&
  ((gnt_io || gnt_cache || gnt_mem) |-> ready));
 
// U5 — the reported maximum dominates EVERY class's wait. A maximum computed
// from one class is a number about that class, not about the arbiter.
a_max_covers_all: assert property (@(posedge clk) disable iff (!rst_n)
  (max_wait_q >= wait_io_q) && (max_wait_q >= wait_cache_q)
                            && (max_wait_q >= wait_mem_q));
 
// LIVENESS -----------------------------------------------------------------
// U6 — EVERY class is eventually served. Instantiated three times, not once.
// ASSUMPTION: `ready` is not permanently low.
a_live_io:    assert property (req_io    |-> ##[1:BOUND] gnt_io);
a_live_cache: assert property (req_cache |-> ##[1:BOUND] gnt_cache);
a_live_mem:   assert property (req_mem   |-> ##[1:BOUND] gnt_mem);
 
// GOAL ---------------------------------------------------------------------
// U7 — the stated service guarantee for three fair classes.
a_bounded_wait: assert property (@(posedge clk) disable iff (!rst_n)
  (FAIR && ready) |-> (max_wait_q <= 2));

U5 is the property that a plain "maximum" check misses, and mutation Q4 proves it: restricting the maximum to .io left every other check passing while the reported worst-case wait became a number about the winning class.

13. Mutation Testing

Q4 escaped initially, and the reason is worth naming: the testbench checked that max_wait_q did not exceed a starvation threshold, which a maximum restricted to the favoured class satisfies trivially. The fix was to relate the reported maximum to the per-class counters:

if ((fmx < fwi) || (fmx < fwc) || (fmx < fwm)) ... // must dominate all three

This is the third batch in a row where a fairness-adjacent measurement turned out to be checked only against the class that wins. It is worth treating as a standing review question rather than a per-design one.

14. Debug Lab

// A valid request of a supported class; serve it.
assign accept = req_valid;

  .io from the DEVICE    strict accept=0 | unchecked accept=1
  .cache from the HOST   strict accept=0 | unchecked accept=1
  .mem from the DEVICE   strict accept=0 | unchecked accept=1
 
  accepted: strict=3 unchecked=6

assign expect_device = (req_class == CLASS_CACHE);
assign accept        = req_valid && (from_device == expect_device);

// The request is well-formed; serve it.
assign serve = req_valid;

  .cache   : Type 1 serve=1 | Type 2 serve=1 | Type 3 serve=0
  .mem     : Type 1 serve=0 | Type 2 serve=1 | Type 3 serve=1
 
  refused: Type 1=3 Type 2=0 Type 3=3

assign serve = req_valid && have;      // `have` selected from the parameters
if      (HAS_IO && HAS_CACHE && HAS_MEM) device_type = 2'd2;
else if (HAS_IO && HAS_CACHE)            device_type = 2'd1;
else if (HAS_IO && HAS_MEM)              device_type = 2'd3;

// Serve in order.
for (j = 0; j < 3; j++)
  if (!found && req[j]) begin gnt[j] = 1'b1; found = 1'b1; end

  fair          : io=50 cache=50 mem=50  worst wait=2
  fixed priority: io=150 cache=0 mem=0   worst wait=150

k = (ptr_q + j) % 3;
if (!found && req[k]) begin gnt[k] = 1'b1; found = 1'b1; end
...
if (gnt_io) ptr_q <= 2'd1;  // and so on, so the winner goes to the back

// Track the worst wait any class experienced.
if (req_io && !gnt_io && wait_io_q + 1 > max_wait_q) max_wait_q <= wait_io_q + 1;
// ... and nothing for cache or mem

if (req_cache && !gnt_cache && wait_cache_q + 1 > max_wait_q) max_wait_q <= wait_cache_q + 1;
if (req_mem   && !gnt_mem   && wait_mem_q   + 1 > max_wait_q) max_wait_q <= wait_mem_q + 1;
// and in the checker:
assert (max_wait_q >= wait_io_q && max_wait_q >= wait_cache_q && max_wait_q >= wait_mem_q);

15. Verification Plan

Reference model. A per-class expectation model: which initiator is legal, which device types serve it, and — for coexistence — a fairness oracle predicting grant shares from the request pattern. The scoreboard compares per-class grant counts and per-class waits against that oracle rather than against an aggregate.

Directed tests. All six class/initiator combinations. All nine class/device-type combinations. All three classes saturating simultaneously under both policies. A stall during rotation. Reset with requests pending.

Constrained-random dimensions. Per-class request density, initiator correctness rate (mostly legal, occasionally not), ready duty cycle, and device-type parameterisation across the three types.

Functional coverage:

The class × initiator cross is the whole space and only six bins — a case where exhaustive coverage is trivially achievable and is often skipped because the illegal half feels unrealistic. It is the half that matters.

16. Design Review

On direction. Is the initiator checked against the class, or assumed? Are all six combinations in the regression, including the three illegal ones?

On capability. Is the serving condition gated on what was instantiated? Is device type derived or stored? Are all nine class-by-type combinations tested?

On coexistence. Is the rotation pointer advanced on every grant, and does it survive a stall? Is the liveness property instantiated per class or once? Is the worst-case wait computed across all three classes — and is there a check relating it to the per-class counters?

On the comparison itself. Does the design treat the three classes as variations of one mechanism anywhere? Every place it does is a place where one class's characteristics have been assumed for all three, and Sections 9 to 11 each measure a different consequence.

17. How This Appears in Real Engineering

CXL / protocol architect

The protocol set is the product decision, because it fixes the device type, the verification surface and the coherence obligations simultaneously. The question to ask first is which direction the device needs: does it want to read host memory (.cache), or offer its memory to the host (.mem), or both. Everything else follows.

RTL engineer

Three disciplines from the measured runs: check the initiator against the class; gate serving on what was instantiated and derive the type; and rotate between classes with a pointer that survives stalls.

DV engineer

Two small exhaustive spaces worth doing exhaustively: six class/initiator combinations and nine class/device-type combinations. Both are trivially small and both are usually tested only in their legal halves. And instantiate liveness per class — this is the third batch where a one-class fairness check was found to be a tautology.

Performance engineer

Fair rotation gives 1/N share and an N−1 wait bound; measured, a third each and two cycles. The share number matters for .io; the wait bound matters for .cache and .mem, because they have a latency target and aggregate throughput does not speak to it.

Firmware and system software

The protocol set a device implements determines what it can be asked to do. A Type 3 expander will refuse coherent traffic, and correctly — treating that refusal as a fault rather than as a device-class fact is a common bring-up misdiagnosis.

Silicon debug

Per-class grant counts are the single most useful observable here. Two classes at zero with a third saturated is unambiguous, and no aggregate throughput number shows it.

18. Common Misconceptions

19. Interview Reasoning

20. Exercises

1. Explain. .cache is the only device-initiated class. Derive from that fact alone why a Type 3 memory expander needs no snoop-response logic.

2. Calculate. Four classes share a link with fair rotation. What is each one's share and the worst-case wait? Now suppose one class requests only 10% of cycles — what do the other three get, and what happens to the wait bound?

3. RTL modification. Change class_coexist so .cache receives two turns per rotation. Predict the new shares and wait bound, then verify. Which of Section 12's properties needs updating, and which does not?

4. DV task. Write the six-bin coverage group for class × initiator. Then argue whether the three illegal bins should be illegal_bins or ordinary bins that the test must hit.

5. Debug task. A Type 2 device reports .io grants 900, .cache 50, .mem 50 over 1000 cycles, and the maximum wait reads 3. Name two different explanations consistent with those numbers, and the single additional observation that separates them.

21. Summary

The three CXL protocols are distinguished by direction before anything else. .io and .mem are host-initiated; .cache is the one the device initiates — and that single reversed arrow is what separates a coherence participant from a target.

Half the class/initiator combinations are illegal. Measured, a design that did not check direction accepted all six where the correct one accepted three, and the three legal cases behaved identically between them. It is a security-adjacent check that costs one comparison.

The protocol set determines the device type, structurally. Type 1 is .io + .cache, Type 3 is .io + .mem, Type 2 is all three. Measured across all nine class-by-type combinations, Type 2 refused nothing and Types 1 and 3 each refused three — with complementary refusals, which is the type system visible in one table.

Coexistence gives 1/N and an N−1 wait bound. Measured: exactly a third each and two cycles for three classes. Fixed priority gave .io all 150 cycles and starved .cache and .mem — the two the Consortium targets at near CPU cache latency, starved by the one with no stated target, because .io is class 0 and comes first in every list.

And a maximum computed from one class is a number about that class. Mutation Q4 restricted the worst-case wait to .io, and every existing check still passed — including a starvation threshold, which was being applied to a value that was already wrong. A threshold on a derived value tests the threshold, not the derivation.

22. What Comes Next

Module 4 has now worked up the stack — physical, link, transaction, and the three protocols that sit on top. One question remains, and it is the one this module has been circling since Chapter 4.1: given all of that, exactly where does CXL stop being PCIe?

Chapter 4.5 answers it layer by layer, and closes Module 4.

For adjacent material: The CXL Device has the device types this chapter's capability gate implements, Transaction Layer has the per-class divergence, and The CXL Fabric has the three different routing behaviours. The path is on the CXL tutorials index.