Protocol Interactions

Chapter 6.1 established what each protocol is: a contract about who owns a resource and who must keep state about it. Each got its own engines, its own obligations, its own diagram column.

They do not get their own link. This chapter is about what happens when three semantic engines share one physical journey — and it starts by correcting a picture almost everyone carries.

1. The Engineering Problem — Three Names, Two Stacks, One Link

Ask an engineer to sketch CXL protocol coexistence and you will almost always get three boxes feeding one arbiter. It is a natural picture and it is wrong, in a way that changes the answer to every fairness question that follows.

A peer-reviewed survey citing the specification describes the structure directly: the Arb/Mux "performs the arbitration and multiplexing between the two stacks (CXL.io and CXL.cache + mem)". .cache and .mem are not peers of .io at the arbitration point — they are peers of each other, inside a stack that is itself one of two.

That single structural fact has a consequence you can measure, and §9 measures it: round-robin at every level does not produce equal service. It produces 50% for .io and 25% each for .cache and .mem. Nobody chose that split; it falls out of the shape.

2. The One-Sentence Model

Three semantic engines, two stacks, one link — .cache and .mem contend with each other inside the coherent stack, that stack contends with .io at the Arb/Mux, and every fairness, starvation and resource question in a CXL controller has to be asked twice because arbitration happens at two levels.

Call it two-level, not flat. The rest of the chapter is what follows from it.

3. What This Chapter Owns

4. The Structure

Read the two arbitration points as separate questions. Level 1 asks ".cache or .mem?" Level 2 asks "io stack or coherent stack?" A design that asks only one of those has either merged .cache and .mem into one queue or promoted them to peers of .io — and §9 shows the second choice changes every service ratio in the system.

5. What Is Shared, and What Must Not Be

Coexistence is a question about which structures are common and which must stay separate. Getting the split wrong in either direction has a characteristic failure.

The middle rows are where designs go wrong. Sharing bandwidth is unavoidable and sharing a queue is a choice — and a shared ingress queue means a stalled .mem device can block .io configuration traffic that has nothing to do with it. RTL 2 measures exactly that.

The last row is the subtle one. The tag pool can be shared — pooling improves utilisation — but each tag must carry its class for the whole transaction lifetime, because the response has to reach the engine that keeps state for it. RTL 5 and RTL 6 are that requirement split across allocation and completion.

6. Teaching-model boundary

7. Quantitative — Offered Load Above One

The three protocols draw on one link. If each offers a fraction of link capacity, the total offered load is their sum, and nothing about having three protocol names changes the arithmetic.

offered = IO + CACHE + MEM

Take a plausible accelerator profile:

A link that can serve 100% cannot serve 140%. The excess does not vanish; it becomes queue occupancy, then backpressure, then latency — and the sub-200 ns coherent latency target from 6.1 is the first thing it destroys.

Three responses exist and they are genuinely different:

• Contention, unmanaged — whoever the arbiter favours wins. §9 shows what "favours" means once the structure is two-level.
• Partitioning — reserve capacity per class. Guarantees a floor, wastes the reservation when a class is idle. RTL 4.
• QoS — weight the arbitration deliberately rather than accepting the shape's default.

The point of §9 is that the default is not neutral. A design that adds no policy at all has still chosen one, and it is 50/25/25.

8. RTL 1 and 2 — Dispatch to Two Stacks, and Keep the Queues Apart

module class_dispatch #(
  parameter bit FLAT_THREE = 1'b0     // 1 = pretend the three are peers
) (
  input  logic       clk, rst_n, req_valid,
  input  logic [1:0] req_class,       // 0=.io 1=.cache 2=.mem
  input  logic [2:0] class_enabled,   // {mem, cache, io}
  output logic       to_io_stack, to_coh_stack,
  output logic [1:0] coh_sub,
  output logic       refuse,
  output logic       stack_mismatch_err, disabled_class_err
);
  localparam logic [1:0] C_IO = 2'd0, C_CACHE = 2'd1, C_MEM = 2'd2;
  logic legal;
 
  assign legal = req_valid && (req_class <= C_MEM) && class_enabled[req_class];
 
  // .io is its own stack. .cache and .mem share the coherent stack.
  assign to_io_stack  = legal && (req_class == C_IO);
  assign to_coh_stack = legal && (FLAT_THREE ? 1'b0
                                             : ((req_class == C_CACHE) ||
                                                (req_class == C_MEM)));
  assign coh_sub      = req_class;
  assign refuse       = req_valid && !legal;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      stack_mismatch_err <= 1'b0; disabled_class_err <= 1'b0;
    end else if (req_valid) begin
      // A legal request must reach exactly one stack.
      if (legal && (to_io_stack == to_coh_stack)) stack_mismatch_err <= 1'b1;
      if ((to_io_stack || to_coh_stack) && !class_enabled[req_class])
        disabled_class_err <= 1'b1;
    end
  end
endmodule

Purpose. To place each class on its stack and to refuse a class the device does not implement — since .cache and .mem are optional, "not present" is a normal configuration.

Invariant. A legal request reaches exactly one stack. The check is written as to_io_stack == to_coh_stack, which catches both failure directions at once: reaching neither, and reaching both.

The per-class queues then enforce the separation that matters most.

module class_fifo #(
  parameter int unsigned DEPTH = 4,
  parameter bit          IGNORE_FULL = 1'b0    // 1 = accept when full
) (
  input  logic        clk, rst_n, push,
  input  logic [7:0]  push_data,
  input  logic        pop,
  output logic [7:0]  head_data,
  output logic        empty, full, accept,
  output logic [3:0]  count_q, max_count_q,
  output logic        overflow_err, underflow_err
);
  logic [7:0] mem_q [0:DEPTH-1];
  logic [3:0] rd_q, wr_q;
  logic       do_push, do_pop;
 
  assign empty  = (count_q == 4'd0);
  assign full   = (count_q == DEPTH[3:0]);
  assign accept = push && (IGNORE_FULL ? 1'b1 : !full);
  assign do_push = accept;
  assign do_pop  = pop && !empty;
  assign head_data = mem_q[rd_q];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      rd_q <= 4'd0; wr_q <= 4'd0; count_q <= 4'd0; max_count_q <= 4'd0;
      overflow_err <= 1'b0; underflow_err <= 1'b0;
    end else begin
      if (do_push) begin
        mem_q[wr_q] <= push_data;
        wr_q <= (wr_q == DEPTH[3:0]-4'd1) ? 4'd0 : wr_q + 4'd1;
      end
      if (do_pop) rd_q <= (rd_q == DEPTH[3:0]-4'd1) ? 4'd0 : rd_q + 4'd1;
 
      // Simultaneous push and pop must leave the count unchanged. Handling
      // them as two independent increments is the classic occupancy bug.
      case ({do_push, do_pop})
        2'b10: count_q <= count_q + 4'd1;
        2'b01: count_q <= count_q - 4'd1;
        default: ;
      endcase
 
      if (count_q > max_count_q) max_count_q <= count_q;
      if (do_push && full) overflow_err  <= 1'b1;
      if (pop && empty)    underflow_err <= 1'b1;
    end
  end
endmodule

  class 0 : io_stack=1 coh_stack=0 coh_sub=0
  class 1 : io_stack=0 coh_stack=1 coh_sub=1
  class 2 : io_stack=0 coh_stack=1 coh_sub=2
  after 4 pushes to .cache : io=0 cache=4 mem=0 full=010
  a full .cache queue leaves .io and .mem untouched: ok
  simultaneous push+pop : count=3 (must be unchanged at 3)

A full .cache queue leaves .io and .mem at zero. That is the whole argument for separate ingress queues stated as a measurement: with one shared queue, a backed-up coherent device blocks configuration traffic, and the system loses the ability to diagnose the device that is blocking it.

The simultaneous push-and-pop case is worth its own line in the plan. Mutation M4 implements the count as two independent increments, which is correct in every cycle except the one where both happen — and that is the cycle a busy queue spends most of its life in.

9. RTL 3 — Two-Level Arbitration, and the Fairness Result

This is the chapter's centre.

module two_level_arb #(
  parameter bit FIXED_PRIORITY = 1'b0,  // 1 = .io always wins (starvation)
  parameter bit FLAT_RR        = 1'b0   // 1 = flat 3-way RR instead of 2-level
) (
  input  logic       clk, rst_n,
  input  logic       req_io, req_cache, req_mem, downstream_ready,
  output logic       gnt_io, gnt_cache, gnt_mem,
  output logic [1:0] gnt_class,
  output logic       gnt_valid,
  output logic       onehot_err, gnt_without_req_err, gnt_while_stalled_err
);
  logic       coh_req, top_tok_q, coh_tok_q;
  logic [1:0] flat_tok_q;
  logic       pick_io, pick_cache, pick_mem, coh_pick_cache;
 
  assign coh_req = req_cache || req_mem;
 
  // ---- level 1: inside the coherent stack
  assign coh_pick_cache = req_cache && (!req_mem || (coh_tok_q == 1'b0));
 
  // ---- level 2: between the two stacks
  always_comb begin
    pick_io = 1'b0; pick_cache = 1'b0; pick_mem = 1'b0;
    if (FIXED_PRIORITY) begin
      if      (req_io)    pick_io    = 1'b1;
      else if (req_cache) pick_cache = 1'b1;
      else if (req_mem)   pick_mem   = 1'b1;
    end else if (FLAT_RR) begin
      /* a flat three-way round robin over the leaves — see the testbench */
    end else begin
      // Two-level: choose a STACK, then a member of it.
      if (req_io && coh_req) begin
        if (top_tok_q == 1'b0) pick_io = 1'b1;
        else if (coh_pick_cache) pick_cache = 1'b1;
        else                     pick_mem   = 1'b1;
      end else if (req_io) begin
        pick_io = 1'b1;
      end else if (coh_req) begin
        if (coh_pick_cache) pick_cache = 1'b1;
        else                pick_mem   = 1'b1;
      end
    end
  end
 
  assign gnt_io    = pick_io    && downstream_ready;
  assign gnt_cache = pick_cache && downstream_ready;
  assign gnt_mem   = pick_mem   && downstream_ready;
  assign gnt_valid = gnt_io || gnt_cache || gnt_mem;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      top_tok_q <= 1'b0; coh_tok_q <= 1'b0; flat_tok_q <= 2'd0;
      onehot_err <= 1'b0; gnt_without_req_err <= 1'b0; gnt_while_stalled_err <= 1'b0;
    end else begin
      // Rotate a token only when that level actually granted.
      if (gnt_io)                    top_tok_q <= 1'b1;
      else if (gnt_cache || gnt_mem) top_tok_q <= 1'b0;
      if (gnt_cache)    coh_tok_q <= 1'b1;
      else if (gnt_mem) coh_tok_q <= 1'b0;
 
      if ((gnt_io + gnt_cache + gnt_mem) > 1) onehot_err <= 1'b1;
      if (gnt_io && !req_io)                  gnt_without_req_err <= 1'b1;
      if (gnt_cache && !req_cache)            gnt_without_req_err <= 1'b1;
      if (gnt_mem && !req_mem)                gnt_without_req_err <= 1'b1;
      if (gnt_valid && !downstream_ready)     gnt_while_stalled_err <= 1'b1;
    end
  end
endmodule

Backpressure. Every grant is gated by downstream_ready. Mutation M5 removes that term from the .io grant, and §11's waveform is what it violates.

The measurement

All three classes request continuously for 300 cycles. This is the symmetric test — no favoured requester.

  --- two-level (io stack | cache+mem stack) over 300 grants ---
    grants : io=150 cache=75 mem=75
    share  : io=50% cache=25% mem=25%
    max wait: io=1 cache=3 mem=3
  --- flat three-way round robin over 300 grants ---
    grants : io=100 cache=100 mem=100
    share  : io=33% cache=33% mem=33%
    max wait: io=2 cache=2 mem=2
  --- fixed priority (io first) over 300 grants ---
    grants : io=300 cache=0 mem=0
    max wait: io=0 cache=299 mem=299

The fixed-priority row is the control. .cache and .mem receive zero grants in 300 cycles and reach a maximum wait of 299 — complete starvation, and exactly what a naive "control traffic is more important" priority produces under sustained load.

Asymmetric load

  .io idle : io=0 cache=60 mem=60 (cache+mem split evenly)
  .mem bursty (2 of 8) : io=60 cache=46 mem=15

An idle class yields its share rather than wasting it — the coherent stack takes the whole link when .io is silent, and .cache/.mem split it evenly. A bursty class is still served in proportion to its demand. Both are properties worth asserting rather than assuming, because a work-conserving arbiter and a slot-reserving one differ exactly here.

10. Waveform — Arbitration, Stall and Recovery

Read the grant row across cycles 2 to 6: io, mem, io, cache, io. .io takes every second grant because it is one of two stacks; .cache and .mem alternate within the slots the coherent stack wins. That is the 50/25/25 split visible cycle by cycle rather than as a statistic.

Cycles 7 to 9 are the backpressure case. No grant is issued — not a deferred grant, not a grant that is later cancelled. And at cycle 10 service resumes with the coherent stack, because the token did not rotate during the stall: a stalled cycle is not a turn taken. An arbiter that rotates its token on a grant that never happened loses a turn for that requester every time downstream stalls.

11. RTL 4 to 6 — Sharing Without Losing Ownership

Three modules cover the resource questions: a shared pool with per-class floors, a shared tag table that must remember the class, and a demultiplexer that must return the response to the engine that owns it.

module resource_partition #(
  parameter int unsigned POOL       = 12,
  parameter int unsigned RESERVE    = 2,      // guaranteed per class
  parameter bit          NO_RESERVE = 1'b0    // 1 = pure sharing
) (
  input  logic       clk, rst_n, alloc,
  input  logic [1:0] alloc_class,
  input  logic       free,
  input  logic [1:0] free_class,
  output logic       grant,
  output logic [4:0] used_io_q, used_cache_q, used_mem_q, used_total_q,
  output logic       overflow_err, reserve_stolen_err, accounting_err
);
  /* a class may take from the shared remainder, or from its own reserve;
     the shared remainder excludes every OTHER class's untouched reserve */

  .cache allocated hard : io=0 cache=8 mem=0 total=8 (pool 12, reserve 2)
  pure-sharing variant  : io=0 cache=10 mem=0 total=10
  18 allocation attempts never exceeded the pool  : ok
  .io allocates after   : grant=1 io=1 | pure-sharing grant=1
  .io reached its reserve despite a greedy class  : ok

The greedy .cache class takes 8 of 12 under the reserved scheme and 10 of 12 under pure sharing — because the reserved scheme holds back the other classes' floors. .io can still allocate afterwards in both cases here, which is the honest result: with a pool of 12 and 3 classes, pure sharing does not starve .io at this load. The reserve earns its keep at higher pressure, and the invariant reserve_stolen_err is what states the guarantee independently of whether this particular load happens to exercise it.

  // The class must survive the whole transaction lifetime, because the
  // response has to go back to the engine that keeps state for it.
  assign retire_class = LOSE_CLASS ? 2'd0 : cls_q[retire_tag];

  // Inferring the class from anything other than the stored class is how a
  // response reaches the wrong engine.
  assign eff = ROUTE_BY_TAG_PARITY ? rsp_tag[1:0] : rsp_class;

  alloc .cache -> tag 0
  retire tag 0 : class back = 1 (correct) | lose-class = 0
  rsp class 1 : io=0 cache=1 mem=0 | tag-parity io=0 cache=0 mem=1

The class goes into the table and must come back out unchanged. Anything that reconstructs it — from the tag, the address, or the response type — is a guess, and mutation M11 shows a tag-parity guess sending every response to the .mem engine.

11b. The Journey of One Request

Two arbitration hops, and the class carried through both. Every step that discards the class is a bug this chapter has a Debug Lab for — dispatch by address (6.1), merged queues (Lab 1), a tag without its owner (Lab 5).

12. Assertions

Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog. Not executed; each maps to a procedural stand-in and a mutation.

// SAFETY -------------------------------------------------------------------
// V1 — at most one grant per cycle.
a_grant_onehot: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({gnt_io, gnt_cache, gnt_mem}));
 
// V2 — a grant implies a request.
a_grant_implies_req: assert property (@(posedge clk) disable iff (!rst_n)
  (gnt_io |-> req_io) and (gnt_cache |-> req_cache) and (gnt_mem |-> req_mem));
 
// V3 — no grant while downstream is stalled.
a_respect_backpressure: assert property (@(posedge clk) disable iff (!rst_n)
  !downstream_ready |-> !gnt_valid);
 
// V4 — a legal request reaches exactly one stack.
a_one_stack: assert property (@(posedge clk) disable iff (!rst_n)
  legal |-> (to_io_stack ^ to_coh_stack));
 
// V5 — queue occupancy never overflows or underflows.
a_queue_bounds: assert property (@(posedge clk) disable iff (!rst_n)
  (count_q <= DEPTH) and (do_pop |-> (count_q > 0)));
 
// V6 — simultaneous push and pop leave the count unchanged.
a_push_pop_stable: assert property (@(posedge clk) disable iff (!rst_n)
  (do_push && do_pop) |=> $stable(count_q));
 
// V7 — pool usage never exceeds capacity.
a_pool_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  used_total_q <= POOL);
 
// V8 — no tag is handed out while live.
a_no_dup_tag: assert property (@(posedge clk) disable iff (!rst_n)
  alloc_ok |-> !live_q[alloc_tag]);
 
// V9 — the class is preserved end to end.
a_class_preserved: assert property (@(posedge clk) disable iff (!rst_n)
  retire_ok |-> (retire_class == cls_q[retire_tag]));
 
// V10 — a response reaches the engine its class names.
a_response_routed: assert property (@(posedge clk) disable iff (!rst_n)
  to_cache_eng |-> (rsp_class == C_CACHE));
 
// V11 — CONSERVATION: grants partition across the three classes.
a_grants_conserved: assert property (@(posedge clk) disable iff (!rst_n)
  n_gnt_total_q == n_gnt_io_q + n_gnt_cache_q + n_gnt_mem_q);
 
// LIVENESS ------------------------------------------------------------------
// V12 — a continuously requesting class is eventually granted.
//       ENVIRONMENT ASSUMPTIONS, both required and both violated in practice:
//       (a) downstream_ready is not held low forever, and (b) the policy is
//       not fixed-priority. Under FIXED_PRIORITY this property is FALSE and
//       the measurement in §9 shows it: 299 cycles of wait and zero grants.
a_eventual_service: assert property (@(posedge clk) disable iff (!rst_n)
  req_cache |-> s_eventually gnt_cache);
 
// PERFORMANCE GOAL ----------------------------------------------------------
// V13 — bounded wait. NOT a safety property and NOT free: it holds for this
//       teaching round-robin policy and is measured, not assumed. Under
//       sustained contention the observed maxima were io=1, cache=3, mem=3.
a_bounded_wait: assert property (@(posedge clk) disable iff (!rst_n)
  req_cache |-> ##[0:6] gnt_cache);

The V12/V13 pair is the point of the taxonomy. V12 is liveness and is false under fixed priority — the assertion is only meaningful once you state the policy assumption. V13 is a performance goal, not a correctness property: the bound 6 comes from the measurement, not from the specification, and it would be a different number under a different policy or a deeper stall. Writing V13 as if it were safety is how a design ends up with an assertion that fails on a legal configuration.

13. Mutation Testing

Fourteen mutations. Clean code restored after each.

14/14 killed, 0 escaped

Six escaped on the first run, and the most important one is M6.

M6 is the batch-005 carry-forward, on the chapter where it matters most. The mutation stops the top-level token returning to the .io stack, so .io receives one grant and then starves. The testbench printed io=150 cache=75 mem=75 and share: io=50% cache=25% mem=25% — the headline result of the entire chapter — and asserted only that no class received zero grants. .io received one, so the check passed.

The numbers that are the point of a chapter are exactly the numbers most likely to be printed rather than asserted, because they read as results rather than as checks. The fix asserts the shares to within a tolerance band, and it is what kills M6, M12 and M14.

The other five split three ways:

M5 is worth expanding. The mutation removes downstream_ready from the .io grant, and the original stall test kept all three classes requesting — which parked the top-level token on the coherent stack, so .io was never picked during the stall and the missing term was never exercised. Stalling a two-level arbiter tests only the level the token happens to be on, so the stall has to be driven with the requester under test as the only claimant.

M13 is the unreachable-checker category from 5.5's taxonomy: accounting_err can only fire if a counter is already wrong, so disabling it changes nothing on a correct design. The resolution is a deliberately miscounting instance whose accounting_err is required to fire.

14. Debug Lab

// One ingress queue for all classes.
assign push = req_valid;
assign accept = push && !shared_full;

  after 4 pushes to .cache : io=0 cache=4 mem=0 full=010
  a full .cache queue leaves .io and .mem untouched  (correct design)

// One queue per class; independence is the point.
class_fifo u_q_io    (.push(push_io),    .accept(acc_io),    ...);
class_fifo u_q_cache (.push(push_cache), .accept(acc_cache), ...);
class_fifo u_q_mem   (.push(push_mem),   .accept(acc_mem),   ...);

// Size the coherent path for one third of link bandwidth.
localparam int COH_BUFFER = LINK_BW / 3;

  two-level : io=150 cache=75 mem=75  ->  50% / 25% / 25%
  flat RR   : io=100 cache=100 mem=100 -> 33% / 33% / 33%

// Size from the arbitration TREE, not from the leaf count.
localparam int COH_STACK_SHARE = LINK_BW / 2;      // one of two stacks
localparam int COH_BUFFER      = COH_STACK_SHARE / 2;  // one of two members

// Control traffic is more important.
if      (req_io)    pick_io    = 1'b1;
else if (req_cache) pick_cache = 1'b1;
else if (req_mem)   pick_mem   = 1'b1;

  fixed priority over 300 grants:
    grants : io=300 cache=0 mem=0
    max wait: io=0 cache=299 mem=299

// Rotate a token at each level; grant only when downstream can accept.
if (gnt_io)                    top_tok_q <= 1'b1;
else if (gnt_cache || gnt_mem) top_tok_q <= 1'b0;

// Rotate every cycle we made a choice.
if (pick_valid) top_tok_q <= ~top_tok_q;   // pick, not grant

  cycle 7..9  : downstream stalled, no grant issued
  cycle 10    : resumes with the coherent stack -- the token HELD

// Rotate only on a real grant.
if (gnt_io)                    top_tok_q <= 1'b1;
else if (gnt_cache || gnt_mem) top_tok_q <= 1'b0;

// Recover the class from the tag.
assign eff_class = rsp_tag[1:0];

  rsp class 0 : io=1 cache=0 mem=0 | tag-parity io=0 cache=0 mem=1
  rsp class 1 : io=0 cache=1 mem=0 | tag-parity io=0 cache=0 mem=1
  misrouted_err=1

// Store the class at allocation; return it at retire; route on it.
cls_q[pick]  <= alloc_class;
assign retire_class = cls_q[retire_tag];
assign to_cache_eng = rsp_valid && (rsp_class == C_CACHE);

// Shared pool, first come first served.
assign grant = alloc && (used_total < POOL);

  pure sharing  : io=0 cache=10 mem=0 total=10
  with reserve  : io=0 cache=8  mem=0 total=8   (floors held back)
  .io reached its reserve despite a greedy class : ok

// A floor per class, and a shared remainder above the floors.
assign may = (my_used < RESERVE) || (shared_free > other_classes_unmet_reserves);
if ((used[0] < RESERVE) && (tot >= POOL) && !NO_RESERVE) reserve_stolen_err <= 1'b1;

if (do_push) count_q <= count_q + 1;
if (do_pop)  count_q <= count_q - 1;

  simultaneous push+pop : count=3 (must be unchanged at 3)
  correct design holds; two-increment design drifts

case ({do_push, do_pop})
  2'b10: count_q <= count_q + 4'd1;
  2'b01: count_q <= count_q - 4'd1;
  default: ;                       // both or neither: unchanged
endcase

15. Verification Plan

Rows 4, 7 and 12 are this chapter's additions to the standard plan. Row 4 exists because the headline numbers were printed and not asserted; row 7 because stalling a two-level arbiter only tests the level the token is on; row 12 because a conservation check cannot fail on a correct design and therefore needs a broken one to validate it.

16. Design Review

• Is the arbitration one level or two, and does the buffer sizing match the tree?
• What are the measured per-class service shares under symmetric contention? Not the intended ones — the measured ones.
• Does any arbitration state advance on valid rather than valid && ready?
• Are ingress queues per class, and can any class block .io?
• Does the class travel with the transaction, or is it reconstructed at the far end?
• Can one class consume the entire tag pool or resource pool? Is the floor an invariant or a hope?
• Is any counter driven by two independent events without an explicit simultaneous case?
• Which fairness properties are safety, which are liveness with a stated policy assumption, and which are performance goals with measured bounds?

17. How This Appears in Real Engineering

Buffer sizing is where the flat model costs money. Debug Lab 2 is not an RTL bug and will not appear in any regression. It appears as coherent latency missing its budget under mixed load, on silicon, with every functional test passing.

Starvation is found by the workload, not by the testbench. Fixed-priority arbitration passes every directed test — one requester at a time is exactly the pattern that hides it. It fails when a management agent polls continuously in production.

Backpressure-dependent unfairness survives most fairness testing. Fairness is usually measured with a permanently-ready downstream, which is precisely the condition under which a token that rotates on pick rather than grant behaves correctly.

Per-class telemetry is what settles arguments. When coherent latency misses budget, the first question is whether the coherent stack is being starved or is simply oversubscribed. Grant counts and max waits per class answer it in minutes; a merged counter cannot answer it at all.

18. Common Misconceptions

19. Interview Reasoning

20. Exercises

1. Calculate. Under the two-level arbiter with all three classes saturated, compute each class's share. Then compute it if .io is idle half the time, and if a fourth class were added to the coherent stack. State the general rule.

2. RTL task. Modify the arbiter so the coherent stack receives two grants for every one to .io. Give the new predicted shares, then state what you would measure to confirm it and what would indicate you had put the knob at the wrong level.

3. DV task. Write the fairness check that would have caught the token mutation (top-level token never returning to .io). Explain why "no class receives zero grants" passes it, and what tolerance band you would choose and why.

4. Debug task. A device meets its coherent latency target in isolation and misses it by 30% under mixed load, with no functional failures. Give your investigation order and name the measurement that distinguishes arbiter starvation from workload oversubscription.

5. Design. Add a per-class QoS weight to the two-level arbiter. State which level each weight acts at, what invariant must still hold regardless of weighting, and which existing assertion must be relaxed from a fixed bound to a weighted one.

6. Critique. Argue that .cache and .mem should be arbitrated as peers of .io rather than sharing a stack. Give the strongest case, then identify what the two-stack structure buys and which sourced fact your proposal would contradict.

21. Summary

Three semantic engines, two stacks, one link.

• The Arb/Mux arbitrates between CXL.io and CXL.cache + mem — .cache and .mem are peers of each other, not of .io, and they share the flit's slot mechanism.
• Round-robin at every level gives 50/25/25, not 33/33/33. Service divides by the arbitration tree, and a design sized from a flat model under-provisions the coherent path by a third.
• $onehot0() is safety; fairness is a measurement. Fixed priority holds one-hot while starving two classes for 300 cycles.
• Share bandwidth, never the queue. A shared ingress queue couples independent protocols and blocks the .io traffic you would diagnose with.
• Rotate the token on a grant, not on a pick — a stalled cycle is not a turn taken.
• The class travels with the transaction. Reconstructing it at the response sends completions to engines that keep no state for them.
• Verification lessons: assert the numbers that are the point of the design; stall a hierarchical arbiter with the requester under test as the sole claimant; and a conservation check needs a deliberately broken variant to prove it works.

Chapter 6.3 turns from mechanism to selection: given all of this, which combination of protocols does a particular device actually need?