CXL Protocol Engines

Module 23 spent four chapters on data-centre arithmetic — stranding, composition, measurement and deployment. This chapter goes back inside the silicon, and the question it asks is the one every CXL implementation answers first.

A CXL link carries three protocols over one physical layer. .io is PCIe: producer-consumer ordered, transaction-based, and completely familiar. .cache and .mem are coherent: unordered, credit-based, and nothing like it. They share a PHY, a link layer and a flit format — and beyond that they must share as little as possible, because every resource they do share is a path by which one protocol can stall, starve or corrupt another.

Every model in this chapter is a design that shared one thing too many.

1. The Engineering Problem — What Three Protocols Must Not Share

A single queue blocks a flit whose engine was ready. Six flits offered, two stalled with a demultiplexer and five behind one queue. Section 5.

Credits must be per protocol. An 8-credit .io request against a 4-credit pool is refused — and granted from a shared pool by spending .cache's budget. Section 6.

The engines do not want equal shares. .mem weighted five of eight earns five slots and round robin gives it two. Section 7.

Only .io is ordered. A coherent transaction behind four .io and six coherent ones waits for none of them, and for all ten under one order domain. Section 8.

And a link-layer retry replays the whole flit. One bad slot in four costs the other protocols three slots of collateral. Section 10.

This chapter against 21.1, stated precisely. That one owns what a fabric does between devices. This one owns what happens inside one device's link interface — three engines, one flit stream, and the resources they must keep apart.

2. The One-Sentence Model

A CXL link interface is correct when the link trains, each protocol's flits reach its own engine without waiting for the others, each engine has its own credits, only .io carries producer-consumer ordering, a full engine stalls rather than drops, and the cost of a replay is charged to the whole flit — and every defect below is a link that trains and carries three protocols badly.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real CXL link interface is a PHY, a link layer with CRC and replay buffers, three transaction layers and a scheduler, and none of that is reproduced. What is reproduced is the decision each has to get right, and the shape of the mistake when it does not.

Two simplifications are worth stating plainly. Section 5 compresses head-of-line blocking into a single cycle — a real single queue blocks because of what is ahead of the flit, and modelling that requires state this block does not carry, so the per-cycle equivalent is "accept only when nothing could block." The conclusion is the same and the mechanism is abbreviated. Section 8 likewise counts outstanding transactions rather than tracking their identities.

Each model is built twice — a correct build and a broken build selected by a parameter. Every broken build here is a simplification that saves real gates: one queue, one credit pool, one arbiter, one order domain, one engine. Each is cheaper, each works on a link carrying one protocol, and each fails in a way that is invisible until the second protocol is busy.

Figure 1 — Three engines that share a PHY and a demultiplexer and nothing else. The dashed path is the design that shares one thing more, and section 5 is what that costs.

5. RTL 1 — A Single Queue Blocks A Ready Engine

// RTL 1 - flit demultiplex. One link carries three protocols, and each flit has
// to reach the engine that speaks its protocol without waiting for the others.
module flit_demux #(parameter int SINGLE_QUEUE = 0) (
  input  logic clk, rst_n,
  input  logic       flit_valid,
  input  logic [1:0] flit_type,          // 0 = .io, 1 = .cache, 2 = .mem
  input  logic       io_ready, cache_ready, mem_ready,
  output logic       target_ready, accepted, bad_type,
  output logic [2:0] engine_select,
  output logic [7:0] n_flits, n_stalled,
  output logic       head_of_line_err
);
  logic all_ready;
  assign bad_type = flit_valid && (flit_type == 2'd3);
  assign engine_select = bad_type ? 3'b000
                       : {(flit_type == 2'd2), (flit_type == 2'd1),
                          (flit_type == 2'd0)};
  assign target_ready = (engine_select[0] & io_ready)
                      | (engine_select[1] & cache_ready)
                      | (engine_select[2] & mem_ready);
  // A single queue accepts only when nothing ahead of it can block, which one
  // cycle at a time is the same as requiring every engine to be ready.
  assign all_ready = io_ready & cache_ready & mem_ready;
  assign accepted = flit_valid && !bad_type
                    && ((SINGLE_QUEUE != 0) ? all_ready : target_ready);
  // A flit whose own engine was ready and which was not accepted anyway.
  assign head_of_line_err = flit_valid && target_ready && !accepted;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_flits <= 8'd0; n_stalled <= 8'd0;
    end else if (flit_valid) begin
      n_flits <= n_flits + 8'd1;
      if (!accepted) n_stalled <= n_stalled + 8'd1;
    end
  end
endmodule

Six flits. Engine ready signals as marked.

Two stalled on the demultiplexed path; five behind a single queue.

Row three is the stall that is not a defect. A .io flit whose own engine is busy waits, and it should — the engine has nowhere to put it. The demultiplexer's job is not to eliminate stalls; it is to eliminate stalls caused by somebody else's engine.

Rows one, four and six are that failure three ways. In each, the flit's target is ready and the flit does not move, because a queue it shares is held by traffic for a different protocol. Two of the three are the case a single-protocol design never encounters, which is why the defect ships.

The reserved type is worth its own row. A flit type outside the three protocols selects no engine and must be rejected rather than routed anywhere — and it must not count as head-of-line blocking either, because nothing was ready to receive it. head_of_line_err requires target_ready, which a bad type never asserts.

Three cycles of a seven-flit burst, on a link where one engine was busy for half of it. Scale that to a link running at capacity with .io configuration traffic interleaved into a .mem stream, and the shared queue converts an occasional .io stall into a continuous .mem throughput loss.

6. RTL 2 — Credits Must Be Per Protocol

// RTL 2 - credits per protocol. Each engine has its own credit pool, so one
// protocol running dry cannot stop another from making progress.
module engine_credit #(parameter int SHARED_CREDITS = 0) (
  input  logic clk, rst_n,
  input  logic       request,
  input  logic [1:0] target,             // 0 = .io, 1 = .cache, 2 = .mem
  input  logic [7:0] io_credits, cache_credits, mem_credits, size,
  output logic [7:0] own_credits, total_credits, available,
  output logic       granted,
  output logic [7:0] n_requests, n_denied,
  output logic       cross_protocol_err
);
  assign own_credits = (target == 2'd0) ? io_credits
                     : ((target == 2'd1) ? cache_credits
                        : ((target == 2'd2) ? mem_credits : 8'd0));
  assign total_credits = io_credits + cache_credits + mem_credits;
  // Sharing one pool lets a burst on one protocol consume another's budget.
  assign available = (SHARED_CREDITS != 0) ? total_credits : own_credits;
  assign granted = request && (size <= available);
  // A grant made against credits belonging to another protocol.
  assign cross_protocol_err = request && granted && (size > own_credits);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_denied <= 8'd0;
    end else if (request) begin
      n_requests <= n_requests + 8'd1;
      if (!granted) n_denied <= n_denied + 8'd1;
    end
  end
endmodule

Six requests. Four .io credits, eight .cache, sixteen .mem — twenty-eight in total.

Four denied with per-protocol pools; one with a shared pool.

A shared credit pool is a starvation channel with a nice name. A .mem burst that drains twenty-eight credits leaves .io unable to make progress — and .io carries the configuration and interrupt traffic that might be needed to fix whatever the .mem burst is doing. The dependency runs exactly the wrong way.

Row six is the one that matters for security as much as for performance. A request naming a protocol that does not exist owns no credits, and a per-protocol design refuses it. A shared pool grants it, which means a malformed or hostile flit consumes real buffer space against a target that will never return the credit.

Row three is the boundary the grant test defines, driven exactly: a request the pool covers to the last credit is granted, and one credit more is not. That is where a credit-return race shows up in silicon, and it is the case a random test almost never generates.

7. RTL 3 — The Engines Do Not Want Equal Shares

// RTL 3 - weighted arbitration. The three engines do not need equal shares of
// the transmit path, and giving them equal shares starves the one that does.
module arb_weight #(parameter int ROUND_ROBIN_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       arb,
  input  logic [7:0] window_slots, weight_target, weight_total, demand_slots,
  output logic [7:0] true_slots, rr_slots, granted_slots,
  output logic       demand_met,
  output logic [7:0] n_arbitrations, n_starved,
  output logic       weight_ignored_err
);
  logic [15:0] t_q, r_q;
  // The share a weighted arbiter owes this engine, computed unconditionally so
  // the check below does not depend on the arbiter being tested.
  assign t_q = (weight_total == 8'd0) ? 16'd0
             : (({8'd0, window_slots} * {8'd0, weight_target})
                / {8'd0, weight_total});
  assign true_slots = (t_q > 16'd255) ? 8'hFF : t_q[7:0];
  assign r_q = {8'd0, window_slots} / 16'd3;
  assign rr_slots = (r_q > 16'd255) ? 8'hFF : r_q[7:0];
  assign granted_slots = (ROUND_ROBIN_ONLY != 0) ? rr_slots : true_slots;
  assign demand_met = (demand_slots <= granted_slots);
  // An engine granted a share other than the one its weight earns.
  assign weight_ignored_err = arb && (granted_slots != true_slots);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_arbitrations <= 8'd0; n_starved <= 8'd0;
    end else if (arb) begin
      n_arbitrations <= n_arbitrations + 8'd1;
      if (!demand_met) n_starved <= n_starved + 8'd1;
    end
  end
endmodule

Seven arbitrations. An eight-slot window with weights one, two and five.

Two starved under weighted arbitration; three under round robin.

Round robin is fair between requesters and the requesters are not peers. .mem carries bulk data and .io carries configuration; an equal share starves the first and wastes capacity on the second. Row two is the waste half: round robin gives .io twice what its weight earns, and those slots come from somewhere.

Row three is the engine for which round robin happens to be right, and it is the reason the defect survives review. With three engines, an equal share is exactly one third — so any engine weighted at one third is served correctly, and a bench that exercises only that engine finds nothing.

Row seven is a weight table that was never programmed, which is the reset state of every implementation. A weighted arbiter with no weights grants nothing and stalls visibly; round robin grants a share it has no basis for and looks like it is working, which is a much worse bring-up failure. Section 17 records that this case was demanded by a surviving mutation.

8. RTL 4 — Only .io Is Ordered

// RTL 4 - ordering domains. Producer-consumer ordering belongs to .io; the
// coherent protocols complete out of order, and one domain over all three
// serialises transactions that never needed it.
module ordering_domain #(parameter int ONE_ORDER_DOMAIN = 0) (
  input  logic clk, rst_n,
  input  logic       issue,
  input  logic       io_class,           // this transaction is a .io transaction
  input  logic [7:0] io_outstanding, other_outstanding,
  output logic [7:0] blockers, all_outstanding,
  output logic       issue_now,
  output logic [7:0] n_issues, n_blocked,
  output logic       over_ordered_err
);
  assign all_outstanding = io_outstanding + other_outstanding;
  // A .io transaction waits behind the .io transactions ahead of it. A coherent
  // one waits behind nothing.
  assign blockers = (ONE_ORDER_DOMAIN != 0) ? all_outstanding
                  : (io_class ? io_outstanding : 8'd0);
  assign issue_now = (blockers == 8'd0);
  // A coherent transaction made to wait for anything at all.
  assign over_ordered_err = issue && !io_class && (blockers != 8'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_issues <= 8'd0; n_blocked <= 8'd0;
    end else if (issue) begin
      n_issues <= n_issues + 8'd1;
      if (!issue_now) n_blocked <= n_blocked + 8'd1;
    end
  end
endmodule

Six issue attempts.

Two blocked with per-class ordering; five with one order domain.

Producer-consumer ordering is a .io property and it is expensive. A .io write must not pass a .io write, because software depends on it — and a .mem read has no such obligation to anything. One order domain applies the expensive rule to traffic that never asked for it.

Row one is the cost at its clearest. A coherent transaction with ten transactions in flight ahead of it waits for none of them under correct ordering and all ten under one domain — which on a link running deep queues is the difference between a pipelined memory interface and a serialised one.

Rows five and six are where the two designs agree, and they are why the defect is invisible on a quiet link: with nothing or almost nothing outstanding, the order domain has nothing to over-serialise. The failure scales with depth, so it appears exactly when the link starts to matter.

9. RTL 5 — A Flit Carries Slots, So Pack Them

// RTL 5 - flit packing. A flit carries several slots, and a message that does
// not fill one still occupies it unless the packer puts something beside it.
module flit_pack #(parameter int NO_PACKING = 0) (
  input  logic clk, rst_n,
  input  logic       pack,
  input  logic [7:0] messages, slots_per_flit,
  output logic [7:0] flits, slots_used, slots_sent, efficiency_pct,
  output logic       acceptable,
  output logic [7:0] n_packs, n_wasteful,
  output logic       packing_ignored_err
);
  logic [15:0] f_q, s_q, e_q;
  // Packing fits several messages into one flit; without it each message takes
  // a flit of its own.
  assign f_q = (NO_PACKING != 0) ? {8'd0, messages}
             : ((slots_per_flit == 8'd0) ? 16'd0
                : (({8'd0, messages} + {8'd0, slots_per_flit} - 16'd1)
                   / {8'd0, slots_per_flit}));
  assign flits = (f_q > 16'd255) ? 8'hFF : f_q[7:0];
  assign slots_used = messages;
  assign s_q = {8'd0, flits} * {8'd0, slots_per_flit};
  assign slots_sent = (s_q > 16'd255) ? 8'hFF : s_q[7:0];
  assign e_q = (slots_sent == 8'd0) ? 16'd0
             : (({8'd0, slots_used} * 16'd100) / {8'd0, slots_sent});
  assign efficiency_pct = (e_q > 16'd255) ? 8'hFF : e_q[7:0];
  assign acceptable = (efficiency_pct >= 8'd80);
  // Several messages that could have shared a flit and did not.
  assign packing_ignored_err = pack && (slots_per_flit > 8'd1) && (messages > 8'd1)
                               && (flits == messages);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_packs <= 8'd0; n_wasteful <= 8'd0;
    end else if (pack) begin
      n_packs <= n_packs + 8'd1;
      if (!acceptable) n_wasteful <= n_wasteful + 8'd1;
    end
  end
endmodule

Seven packing decisions.

Three wasteful when packed; all seven unpacked.

A flit is the unit the link layer moves and a message is not. Packing several messages into one flit is the difference between 100% and 25% link efficiency on this traffic — a factor of four, from a scheduler decision with no protocol implications at all.

Row six is the sizing problem packing does not solve. Five messages into four-slot flits needs two flits and fills eight slots with five messages: 62%, and no packer does better, because the messages arrived in that quantity. Efficiency is bounded by arrival patterns as well as by the packer.

Row two is the more common case and the more encouraging one. Thirteen messages into four-slot flits is 81% — the loss from a partial final flit shrinks as the batch grows, so a link with any depth of traffic sits near the top of this range.

10. RTL 6 — A Retry Replays The Whole Flit

// RTL 6 - the scope of a retry. A link-layer replay resends the whole flit, so
// a CRC error on one protocol's slot costs every protocol sharing that flit.
module retry_scope #(parameter int BLAME_THE_SLOT = 0) (
  input  logic clk, rst_n,
  input  logic       retry,
  input  logic [7:0] slots_in_flit, erring_slots,
  output logic [7:0] replayed_slots, collateral_slots,
  output logic       acceptable,
  output logic [7:0] n_retries, n_collateral,
  output logic       collateral_ignored_err
);
  assign replayed_slots = slots_in_flit;
  // Slots replayed that had nothing wrong with them belong to other protocols.
  assign collateral_slots = (BLAME_THE_SLOT != 0) ? 8'd0
                          : ((slots_in_flit > erring_slots)
                             ? (slots_in_flit - erring_slots) : 8'd0);
  assign acceptable = (collateral_slots <= 8'd1);
  // A replay charged only to the slot that erred.
  assign collateral_ignored_err = retry && (slots_in_flit > erring_slots)
                                  && (collateral_slots == 8'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_retries <= 8'd0; n_collateral <= 8'd0;
    end else if (retry) begin
      n_retries <= n_retries + 8'd1;
      if (!acceptable) n_collateral <= n_collateral + 8'd1;
    end
  end
endmodule

Seven retries. A four-slot flit.

Three retries with unacceptable collateral when the flit is the scope; none when the slot is.

CRC is computed over the flit and replay resends the flit, so error isolation stops at the flit boundary. One bad slot costs every other protocol in that flit a full round trip — which means a marginal link degrades all three protocols at a rate set by whichever one is unlucky.

Row two is the case that argues for protocol-homogeneous flits, and it is a real design option: a packer that fills a flit from one protocol's queue rather than from all three produces zero collateral at the cost of section 9's efficiency. The two models trade directly against each other.

Row three is the failure everybody forgets. A CRC error caused by the link rather than by any particular slot replays four slots and all four are collateral — there is no erring protocol to charge it to, and a design that attributes replay cost per protocol has nowhere to put it.

Figure 3 — One bad slot, four replayed. The dashed path is the coupling between protocols that the flit format creates and no amount of engine separation removes — which is why section 9's packing decision and this section's isolation are the same decision seen twice.

11. RTL 7 — A Full Engine Stalls, It Does Not Drop

// RTL 7 - backpressure. An engine that cannot accept a flit must say so, and a
// design that pushes anyway loses the flit rather than delaying it.
module engine_backpressure #(parameter int NO_BACKPRESSURE = 0) (
  input  logic clk, rst_n,
  input  logic       push, pop,
  input  logic [7:0] depth, occupancy,
  output logic       ready, accepted, lost, blocked,
  output logic [7:0] n_pushes, n_blocked, n_lost,
  output logic       flit_dropped_err
);
  assign ready = (occupancy < depth);
  // Without backpressure the push is taken regardless and the flit is gone.
  assign accepted = (NO_BACKPRESSURE != 0) ? push : (push && ready);
  assign lost = (NO_BACKPRESSURE != 0) ? (push && !ready) : 1'b0;
  assign blocked = push && !ready && (NO_BACKPRESSURE == 0);
  // A flit accepted into an engine that had no room for it.
  assign flit_dropped_err = push && (lost != 1'b0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_pushes <= 8'd0; n_blocked <= 8'd0; n_lost <= 8'd0;
    end else if (push) begin
      n_pushes <= n_pushes + 8'd1;
      if (blocked) n_blocked <= n_blocked + 8'd1;
      if (lost)    n_lost <= n_lost + 8'd1;
    end
  end
endmodule

Five pushes.

Three blocked with backpressure and nothing lost; none blocked without it and three flits lost.

This is the only model in the chapter where the broken build corrupts rather than degrades. Every other defect makes the link slower; this one makes it wrong — a lost flit is a transaction that never completes, and on a coherent protocol that is a hang rather than a retry.

Row four is the configuration nobody means to build and everybody has built. A zero-depth engine — a queue whose depth parameter was never set, or was set from a register that reset to zero — is never ready, and a design without backpressure loses every flit sent to it while reporting nothing.

Row five is what a real occupancy counter does after a bug. An occupancy above the depth is impossible in a correct design and entirely possible after a double-increment; the < comparison keeps the engine unready rather than wrapping into a ready state, which is the difference between a stalled link and a silently corrupted one.

12. RTL 8 — Pipeline Depth Is Not Latency

// RTL 8 - the latency an engine actually delivers. A pipeline depth is a
// specification; what a transaction waits is that plus everything ahead of it.
module latency_budget #(parameter int IGNORE_QUEUING = 0) (
  input  logic clk, rst_n,
  input  logic       measure,
  input  logic [7:0] pipeline_cycles, occupancy, budget_cycles,
  output logic [7:0] queue_cycles, latency_cycles, headroom_cycles,
  output logic       within_budget,
  output logic [7:0] n_measures, n_over,
  output logic       queuing_ignored_err
);
  logic [15:0] l_q;
  // Each entry ahead costs a cycle of service before this one is looked at.
  assign queue_cycles = (IGNORE_QUEUING != 0) ? 8'd0 : occupancy;
  assign l_q = {8'd0, pipeline_cycles} + {8'd0, queue_cycles};
  assign latency_cycles = (l_q > 16'd255) ? 8'hFF : l_q[7:0];
  assign headroom_cycles = (budget_cycles > latency_cycles)
                           ? (budget_cycles - latency_cycles) : 8'd0;
  assign within_budget = (latency_cycles <= budget_cycles);
  // A latency quoted as the pipeline depth with a queue standing in front of it.
  assign queuing_ignored_err = measure && (occupancy != 8'd0)
                               && (latency_cycles == pipeline_cycles);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_over <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (!within_budget) n_over <= n_over + 8'd1;
    end
  end
endmodule

Six measurements. A twelve-cycle pipeline against a 24-cycle budget.

Three over budget when the queue is counted; one when it is not.

A datasheet's pipeline depth is the latency of an empty engine and no engine is empty when it matters. Twelve cycles becomes thirty-two with twenty entries ahead — and the twenty entries are the normal operating point, not a pathology.

Row three is the budget boundary, and it says something useful about sizing: with a twelve-cycle pipeline and a 24-cycle budget, the queue may hold twelve entries and no more. That is a design constraint on depth derived from a latency requirement, which is the direction the calculation should run.

Row six is the saturation clamp doing its job. A 250-cycle pipeline plus twenty queued exceeds what the counter can report, and it saturates rather than wrapping to 14 — the difference between a report that says "at least 255" and one that says the engine is fast.

13. RTL 9 — Three Engines Or One

// RTL 9 - three engines or one. Unifying them saves area and costs frequency,
// and a proposal that reports only the first is half an argument.
module engine_area #(parameter int ONE_ENGINE_FITS_ALL = 0) (
  input  logic clk, rst_n,
  input  logic       compare,
  input  logic [7:0] io_area, cache_area, mem_area, shared_area,
  input  logic [15:0] spec_fmax, unified_fmax,
  output logic [15:0] spec_area, unified_area,
  output logic [7:0]  area_saving_pct, fmax_loss_pct,
  output logic        worth_unifying,
  output logic [7:0]  n_compares, n_not_worth,
  output logic        fmax_ignored_err
);
  logic [15:0] biggest, s_q, a_q, f_q;
  assign biggest = (io_area > cache_area)
                 ? ((io_area > mem_area) ? {8'd0, io_area} : {8'd0, mem_area})
                 : ((cache_area > mem_area) ? {8'd0, cache_area} : {8'd0, mem_area});
  assign spec_area = {8'd0, io_area} + {8'd0, cache_area} + {8'd0, mem_area};
  assign unified_area = biggest + {8'd0, shared_area};
  assign a_q = (spec_area == 16'd0) ? 16'd0
             : ((((spec_area > unified_area) ? (spec_area - unified_area) : 16'd0)
                 * 16'd100) / spec_area);
  assign area_saving_pct = (a_q > 16'd255) ? 8'hFF : a_q[7:0];
  // Unifying deepens the datapath mux, and a proposal that omits it compares
  // area against nothing.
  assign f_q = (ONE_ENGINE_FITS_ALL != 0) ? 16'd0
             : ((spec_fmax == 16'd0) ? 16'd0
                : ((((spec_fmax > unified_fmax) ? (spec_fmax - unified_fmax)
                     : 16'd0) * 16'd100) / spec_fmax));
  assign fmax_loss_pct = (f_q > 16'd255) ? 8'hFF : f_q[7:0];
  assign worth_unifying = (area_saving_pct >= fmax_loss_pct);
  // A slower unified engine reported as costing no frequency.
  assign fmax_ignored_err = compare && (unified_fmax < spec_fmax)
                            && (fmax_loss_pct == 8'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_compares <= 8'd0; n_not_worth <= 8'd0;
    end else if (compare) begin
      n_compares <= n_compares + 8'd1;
      if (!worth_unifying) n_not_worth <= n_not_worth + 8'd1;
    end
  end
endmodule

Six comparisons. Engines of 10, 25 and 40 units, unified with 15 of shared logic — 75 against 55, a 26% saving.

Three not worth unifying when frequency is priced; none when only area is.

Unifying three engines is a real and frequently correct optimisation, and row one says so: a 26% area saving for a 20% frequency loss is a trade many designs should take. The failure is not unifying; it is a proposal that reports the area and not the timing.

Row four is the break-even and it is the number to compute first. At a 26% frequency loss the two exactly balance, which means any unified engine holding above 740 MHz against a specialised 1000 is worth building — one ratio, available from a synthesis trial before any RTL is restructured.

Row five is the degenerate case that keeps the model honest. Engines with no area to save still pay the frequency cost of unification, so the answer is no — and a model that reported only the saving would report zero and call it neutral.

14. RTL 10 — A CXL Protocol Engine Assembled

// RTL 10 - a CXL protocol engine assembled. Everything that must hold before a
// link that trains is a link that carries three protocols correctly.
module protocol_engine_model #(parameter int LINK_UP = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       link_trained,        // the physical layer is up
  input  logic       flits_demuxed,       // each protocol reaches its own engine
  input  logic       credits_per_protocol,// one protocol cannot drain another
  input  logic       ordering_per_class,  // only .io is producer-consumer ordered
  input  logic       backpressure_honoured,// a full engine stalls, never drops
  input  logic       retry_scoped,        // replay cost is attributed to the flit
  output logic       correct,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_correct,
  output logic       false_correct_err
);
  assign fail_mask[0] = ~link_trained;
  assign fail_mask[1] = ~flits_demuxed;
  assign fail_mask[2] = ~credits_per_protocol;
  assign fail_mask[3] = ~ordering_per_class;
  assign fail_mask[4] = ~backpressure_honoured;
  assign fail_mask[5] = ~retry_scoped;
  // The link-up build is what a bring-up milestone reports.
  assign correct = (LINK_UP != 0) ? link_trained : (fail_mask == 6'd0);
  assign false_correct_err = evaluate && correct && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_correct <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (correct) n_correct <= n_correct + 8'd1;
    end
  end
endmodule

One correct configuration of six, and four false claims.

"Link trained" is the milestone every CXL bring-up celebrates and it is right about one of the six. Row four is the one that will be found last and cost most: the link trains, all three protocols enumerate, traffic flows — and under load an engine drops flits, which presents as a coherence hang with no error bit set anywhere.

Figure 4 — Five exits, of which two are correct behaviour. The first two questions eliminate the cases where nothing is wrong, and only then does the flowchart start naming defects — which is the order a bring-up engineer needs, because "a flit is not moving" is the normal state of a link under backpressure.

15. Quantitative Reasoning

Demultiplex. Six flits offered: two stalled behind a demultiplexer, five behind one queue — and one of the two is a genuine stall both designs share.

Credits. Four .io, eight .cache, sixteen .mem: an 8-credit .io request is refused per-protocol and granted from the shared 28.

Arbitration. An eight-slot window weighted one, two and five: .mem earns five slots and round robin gives it two, while .io earns one and is given two.

Ordering. A coherent transaction behind four .io and six coherent: no blockers correctly, ten under one order domain — five of six issue attempts blocked instead of two.

Packing. Sixteen messages into four-slot flits: four flits at 100% packed, sixteen at 25% unpacked. Five messages give 62% however they are packed.

Retry. A four-slot flit with one bad slot replays four — three slots of collateral, and a link-level error makes all four collateral.

Backpressure. Three pushes into engines with no room: three stalls and nothing lost, against three flits lost silently.

Latency. A twelve-cycle pipeline with twenty queued is 32 cycles against a 24-cycle budget; the queue may hold twelve and no more.

Area. Three engines at 75 against one at 55 — a 26% saving against a 20% frequency loss, break-even at 740 MHz.

The assembled model. Six properties, six configurations, one correct. The bring-up milestone reported five.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Every inclusive threshold is driven at exactly equal, every ceiling both on and off its boundary, and every floor past it — the three rules this batch has accumulated.

Demultiplex. The reserved flit type is driven, and the genuine .io stall is asserted as not head-of-line blocking.

chk(dGa == 1'b0, "so it waits");
chk(dGe == 1'b0, "which is a real stall, not head-of-line blocking");

Credits. A request exactly covering its own pool is driven, and a reserved target is asserted to own no credits.

Arbitration. The engine whose weight equals its round-robin share is asserted as one both arbiters get right, and an unprogrammed weight table is driven.

chk(aGt == 8'd2, ".cache's weight of two earns two slots");
chk(aBe == 1'b0, "for this engine");

Ordering. Both agreement cases are driven — nothing outstanding, and one .io transaction ahead — and a .io transaction blocked by .io traffic is asserted as not over-ordering.

Packing. A message count that is an exact multiple and one that is not are both driven, and an efficiency of exactly 80% is constructed from sixteen messages into five-slot flits.

Retry. Collateral of exactly one slot is driven at the threshold, and a decoder reporting more bad slots than the flit holds is asserted to floor at zero.

Backpressure. A zero-depth engine and an over-filled one are both driven, and the correct build is asserted to lose nothing in every case.

Latency. A latency exactly at the budget is driven, and a pipeline deep enough to saturate the report is asserted to clamp rather than wrap.

Area. A frequency loss exactly equal to the area saving is constructed at 740 MHz.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 277 checks across two testbenches, 144 on the front five models and 133 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Sixty-two mutations were injected one at a time. 62 injected, 62 killed, after three survivors.

Two of the three survivors were the class this batch has now met four times, and one was the older equality rule.

Survivor 1 — an unprogrammed weight table. Section 7's zero-weight guard survived because every case had a programmed weight total. A weight table of all zeros is the reset state of every implementation, and without the guard the division is undefined. This is section 13 of 23.4's class exactly: a guard for a configuration outside the model's own premise.

Survivor 2 — a decoder reporting more bad slots than the flit holds. Section 10's collateral floor survived because no case had erring_slots exceed slots_in_flit. That is an inconsistency a real error decoder can produce, and without the floor the subtraction wraps to 254.

Survivor 3 — an efficiency of exactly 80%. Section 9's acceptance threshold survived >= becoming > because no case landed on it. Constructing it needed solving backwards: sixteen messages into five-slot flits gives four flits, twenty slots and exactly 80% — and neither four-slot nor six-slot flits produce it at any message count near this range.

The complete set was re-run after every stimulus change, per standing discipline, and all sixty-two held.

18. Verification Strategy

What a testbench for a protocol-engine block must cover — and two mechanical findings that cost real debugging time here.

A strobe-gated output must be sampled with its strobe asserted. Section 6's granted and section 5's accepted are both assign x = strobe && ..., and checking them before raising the strobe reads zero every time. Four assertions failed for this reason and none of them was a design bug — the testbench was asking a question the design was not being asked.

Step off the clock edge after reset. Setting a valid signal in the same timestep as a posedge leaves the sampling race unresolved, and one extra flit was counted. A single #1 after the reset sequence made every counter in the chapter exact, and it is a one-line fix for a class of counter discrepancy that is otherwise very hard to attribute.

Drive the configuration outside the model's premise. An unprogrammed weight table. A decoder that contradicts itself. A zero-depth queue. An occupancy past the depth. Four such cases in this chapter, two of them found by mutations rather than by design.

The cases where the cheap design is right. A flit whose own engine is busy. A request its own pool covers. An engine weighted at exactly one third. A link with nothing outstanding. A flit belonging to one protocol. Five cases across five models, each exempted explicitly, and each the case that lets the cheap design pass a bench.

Counters as a second signature. Ten models, ten pairs of totals, differing in all ten — two stalled against five, four denied against one, two blocked against five, nothing lost against three.

What a real link interface needs that these models do not have. State. Head-of-line blocking is about what is ahead of a flit; ordering is about transaction identity; credits are returned asynchronously. Every model here is a snapshot of a decision whose real difficulty is sequential, and section 26 exercises 1 and 4 are the closest this chapter comes.

19. Synthesis and Implementation Reality

Section 5's demultiplexer is a decode and three ready terms — a handful of gates on the critical path between the link layer and the engines, and the cheapest structure in the chapter to get right.

Section 6's per-protocol credits cost three counters instead of one, and the reason designs share them is buffer area rather than counter area: separate credit pools imply separate buffers, and a shared buffer is genuinely smaller. The trade is real; the starvation is the price.

Section 7's weighted arbiter is a deficit or lottery scheme in practice, and the weights are usually programmable — which is why section 17's unprogrammed-table case is not hypothetical.

Section 8's ordering domain is where the most silicon goes. Tracking which .io transactions must complete before which others requires per-transaction state, and the temptation to collapse it into one counter across all three protocols is a large area saving — paid for exactly as row one describes.

Section 9's packer sits between the transaction layers and the link layer, and it competes with section 10: filling a flit from three queues raises efficiency and couples three protocols to one CRC.

Section 11's backpressure is a single ready signal and its absence is a class of bug rather than a design choice. Nobody designs without backpressure; designs lose flits when a ready signal is registered on the wrong side of a pipeline boundary, which the model abbreviates to a parameter.

20. Silicon Observability

"Stalls where the target engine was ready" is the counter this chapter argues hardest for, because it is the only signal that separates a link doing its job from a link with a structural defect. A stall counter alone rises under legitimate backpressure and under head-of-line blocking identically, and the two have opposite responses: one means the engine needs more throughput, the other means the queue needs splitting.

21. Debug Lab

Symptom. A CXL device brings up cleanly. .io configuration works, .cache and .mem enumerate, and a single-protocol benchmark hits the expected bandwidth on each. Under a mixed workload: .mem bandwidth drops 40%, .io completions occasionally take milliseconds, and once in a long soak the device hangs with no error bit set.

Step 1 — which engine is stalling? Flits stalled per protocol: .mem stalls account for most of it, and the .mem engine reports itself ready during 71% of them. Section 5, and the shared-queue signature is exactly this — a stall on a ready engine.

Step 2 — confirm the structure. The link layer has one receive queue feeding a three-way fan-out. .io configuration writes are slow to retire, and every one of them parks at the head and holds the .mem stream behind it. That is the 40%.

Step 3 — the millisecond .io completions. Transactions blocked by ordering, by class: coherent transactions are being counted. Section 8 — the design uses one outstanding-transaction counter across all three protocols, so a .io completion waits behind .mem reads that have no ordering relationship to it at all. The .io latency is a symptom of .mem depth.

Step 4 — the hang. Pushes rejected for want of room: the counter reads zero, always, on all three engines. There is no rejection counter — there is a drop. Section 11, and the ready signal from the .cache engine is registered one cycle late, so a push that arrives in the shadow of a full queue is accepted and lost.

Step 5 — why only in a long soak? The drop needs the queue to fill exactly as a push arrives, which needs sustained mixed traffic. A single-protocol benchmark never fills that queue, which is why every bring-up test passed.

Step 6 — and the credits? Credits outstanding per protocol: three separate pools, correctly. Section 6 is the one thing this design got right, and establishing that took one counter and removed a whole branch of the investigation.

The finding. Three defects, all of them "one thing shared too many": one queue, one ordering counter, one ready signal on the wrong side of a register. The link trained, all three protocols worked alone, and the mixed case failed three separate ways.

The fix. Split the receive queue three ways — section 5, and it costs buffer area. Split the outstanding-transaction counter by class — section 8, and it costs state. Move the ready register — section 11, and it costs nothing but a respin. Two of the three are area the original design deliberately saved.

What made this hard. Every protocol worked in isolation, which is how link bring-up is tested, and all three defects require two protocols to be busy at once. The counter that would have found the first in a day — stalls on a ready engine — did not exist.

22. Design Review

1. Is there one receive queue or three? A stall on a ready engine is the signature. Section 5.

2. Are credits per protocol, and is the buffer split with them? A shared pool is a starvation channel. Section 6.

3. Is transmit arbitration weighted, and are the weights programmed at reset? An unprogrammed table is the reset state. Section 7.

4. Is there one outstanding-transaction counter or one per class? Coherent traffic must not inherit .io ordering. Section 8.

5. Does the packer fill a flit from one protocol's queue or from all three? It trades link efficiency against retry isolation. Sections 9 and 10.

6. What does a replay cost the protocols that did not err? Three slots of four. Section 10.

7. Where is every ready signal registered, relative to the queue it describes? A cycle late is a dropped flit. Section 11.

8. Is the quoted engine latency the pipeline depth or the pipeline depth plus occupancy? Twelve against thirty-two. Section 12.

9. If the engines are unified, what did synthesis say about frequency? Break-even at a 26% loss. Section 13.

10. Which of the six properties does "the link trained" imply? Section 14 exists because the answer is the first one only.

23. How This Appears In Real Engineering

A link-layer designer owns sections 5, 9 and 10 together, and they form one trade: a packer that fills flits from all three queues maximises efficiency and couples all three protocols to one CRC. Protocol-homogeneous flits invert both. There is no free position, and the right one depends on the link's error rate.

A transaction-layer designer owns sections 6, 8 and 11, which are the three places a protocol can be harmed by another's traffic. Section 8 is where the area pressure is greatest — per-class ordering state is expensive, and collapsing it is the single most tempting simplification in the block.

A verification engineer meets section 21's core difficulty: every defect in this chapter requires two protocols to be busy simultaneously, and single-protocol bring-up tests are what exists first and what gets run most. Mixed-traffic stress is the only bench that finds any of it.

An architect deciding on unification owns section 13, and the discipline it asks for is modest: run the synthesis trial before restructuring the RTL. Break-even is one ratio, and it is knowable early.

24. Common Misconceptions

"The link carries three protocols, so the engines share the interface." They share a PHY and a flit format. Everything else they share is a defect. Section 4.

"A stalled flit means the engine is busy." Or that somebody else's engine is. Section 5.

"One credit pool is simpler." It is a starvation channel — a .mem burst stops .io configuration. Section 6.

"Round robin is fair." Between requesters, which are not peers. .mem earns five slots of eight and gets two. Section 7.

"Ordering is ordering." .io is producer-consumer ordered; the coherent protocols are not, and applying .io's rule to them serialises a pipeline for nothing. Section 8.

"A retry costs the protocol that erred." It replays the whole flit — three slots of four belong to somebody else. Section 10.

"A full engine applies backpressure." If the ready signal is registered on the right side. Otherwise it drops. Section 11.

"The engine's latency is twelve cycles." Empty. With twenty queued it is thirty-two. Section 12.

"Unifying the engines saves 26% of the area." And costs 20% of the frequency. Section 13.

"The link trained." One property of six. Section 14.

25. Interview Reasoning

Q. A CXL link carries three protocols. What must the three engines not share?

Everything except the PHY and the flit format. A shared receive queue blocks a flit whose own engine was ready; a shared credit pool lets one protocol starve another; a shared ordering domain applies .io's producer-consumer rule to coherent traffic that never needed it. Each is cheaper, each works with one protocol active, and each fails only when two are busy.

Q. How do you tell head-of-line blocking from a legitimate stall?

By whether the target engine was ready. A stall counter alone rises identically in both cases — the counter that separates them is "stalls where the target engine reported ready," and it is the single most valuable observability point in the block because the two have opposite fixes.

Q. Why is round robin the wrong arbiter here?

Because the three engines are not peers. .mem carries bulk data and .io carries configuration, so an eight-slot window weighted one, two and five gives .mem five slots — and round robin gives it two while over-serving .io at twice its weight. The subtlety: any engine weighted at exactly one third is served correctly by round robin, so a bench exercising only that engine finds nothing.

Q. What does a link-layer retry cost?

The whole flit. A four-slot flit with one bad slot replays four, so three slots belonging to other protocols are resent for nothing — and a CRC error caused by the link rather than by a slot makes all four collateral. It is why a packer that fills flits from one protocol trades efficiency for isolation.

Q. A device brings up cleanly and hangs under mixed traffic with no error bit. Where do you look?

At every ready signal, relative to the queue it describes. A ready registered a cycle late means a push arriving in the shadow of a full queue is accepted and lost, which is a coherence hang rather than a retry — and it needs sustained mixed traffic to fill the queue at the right instant, so single-protocol bring-up never sees it.

Q. Should the three engines be unified into one?

Compute the ratio first. Three engines at 75 units against one at 55 is a 26% area saving; if the unified engine holds above 740 MHz against a specialised 1000, it is worth building. That is a synthesis trial, not an RTL restructure, and doing it in the other order is how the decision gets made on area alone.

26. Exercises

1. Give RTL 1 a real queue with state and show that head-of-line blocking depends on arrival order, not only on readiness.

2. Extend RTL 2 to return credits asynchronously and find the return latency at which a per-protocol pool must grow.

3. Replace RTL 3's proportional split with a deficit round-robin scheme and show it converges to the same shares.

4. Give RTL 4 transaction identities and show which .io orderings actually constrain which.

5. Combine RTL 5 and RTL 6: find the link error rate at which protocol-homogeneous flits beat mixed ones.

6. Model RTL 7's ready signal registered one cycle late and find the traffic pattern that loses a flit.

7. Drive RTL 8 with an occupancy distribution and compare the mean latency against the ninety-ninth percentile.

8. Extend RTL 9 to a partially unified design — two engines merged, one separate — and find whether it beats both extremes.

9. Model section 21 end to end: a shared queue, a shared ordering counter and a late ready signal under mixed traffic.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the design it catches that the current mask calls correct.

27. Summary

Module 23 was arithmetic about data centres. This chapter is arithmetic about gates, and the finding is structurally the same one Module 23 kept producing: a resource that looks shareable, shared, and a failure that only appears when two things need it at once.

A single receive queue blocks a flit whose engine was ready. Six flits offered leave two stalled behind a demultiplexer and five behind one queue — and one of the two is a genuine stall both designs share, which is the control that makes the comparison honest.

Credits must be per protocol. An 8-credit .io request against a four-credit pool is refused, and a shared pool grants it out of .cache's budget — a starvation channel running in exactly the wrong direction, since .io carries the traffic that would fix the problem.

The engines are not peers. .mem weighted five of eight earns five slots and round robin gives it two — while over-serving .io at twice its weight, and being exactly right for any engine weighted at one third.

Only .io is ordered. A coherent transaction behind ten outstanding transactions waits for none of them correctly and all ten under one order domain, and the cost scales with queue depth — so it appears precisely when the link starts to matter.

Pack the flit. Sixteen messages into four-slot flits is four flits at 100% packed and sixteen at 25% unpacked — a factor of four from a scheduling decision with no protocol implications.

And a retry replays the whole flit. One bad slot of four costs the other protocols three slots of collateral, which is the coupling that packing creates and engine separation cannot remove.

A full engine stalls; it must never drop. Three pushes with no room are three stalls and nothing lost, or three flits gone silently — the only defect in the chapter that corrupts rather than degrades.

Pipeline depth is not latency. Twelve cycles with twenty queued is thirty-two against a 24-cycle budget, which turns a latency requirement into a queue-depth constraint.

Unifying the engines saves 26% of the area and costs 20% of the frequency — worth doing, and only knowable from a synthesis trial that break-evens at 740 MHz.

Three mutations survived, two of them on guards for configurations outside the models' premises — an unprogrammed weight table and an error decoder contradicting itself — which is the fourth chapter running that this class has produced survivors, and the rule now catches them on the first reading.

And two mechanical testbench findings worth carrying: a strobe-gated output must be sampled with its strobe asserted, and a single #1 after reset stops a stimulus case straddling two clock edges. Four assertion failures and one counter discrepancy in this chapter were the testbench, not the design.

A link that trains is one property of six. The milestone every bring-up celebrates called five of six designs correct when one was — and section 21 is a device that passed every single-protocol test and failed three separate ways the moment two protocols were busy together.

24.2 — Transaction Processing goes one layer inward: what happens to a request between arriving at an engine and reaching the memory controller, and why the pipeline that does it is where the depth in section 12 comes from.