Protocol Selection Discipline

Chapter 6.3 derived what a workload needs. That is a technical question with a technical answer, and it is the easy half.

This chapter is the hard half: how an architect defends that answer — against the pull toward implementing everything, against a capability nobody can justify, and against the belief that unused hardware is free.

1. The Engineering Problem — Implementing Everything Is a Decision Too

The most common CXL architecture failure is not choosing wrongly. It is not choosing at all.

"Type 2 covers every case, so build Type 2." It sounds like risk reduction, and it is the opposite. Every protocol added commits state on both ends (6.1), takes a share of a two-level arbiter (6.2), doubles the configuration space DV must cover, and needs software enablement that may not exist. §7 measures what that comes to: all three protocols cost roughly 8× an .io-only device in this chapter's model, and the verification space grows from 2 configurations to 11 DV points.

The discipline is not "choose less". It is: every capability must have an owner, a justification, and a measurement that could prove it wrong.

2. The One-Sentence Model

Select the minimum set the requirement demands, price every addition beyond it in state, DV and software, and keep three states distinct — what silicon contains, what the link negotiated, and what the platform chose to run.

Call it minimum, priced, and three-state. The first stops feature creep, the second makes the argument concrete, and the third stops a design conflating "we built it" with "it is on".

3. What This Chapter Owns

4. The Decision Sequence

The order matters, because each step constrains the next and reversing any two lets a conclusion in ahead of its evidence.

workload            what does it actually do?
  -> memory ownership     who owns the data?
  -> access direction     who reaches across?
  -> coherency need       must copies be tracked?
  -> latency / bandwidth  which is the constraint?
  -> software model       what will drivers do?
  -> device capability    which protocols follow?
  -> implementation cost  what does that commit?
  -> verification cost    what state space does it open?
  -> protocol selection   the decision, and its owner

Protocol selection is last, not first. A process that begins "we are building a Type 2 device" has answered the final question before asking the first nine, and every subsequent step becomes a justification exercise rather than an analysis.

Note what the sequence does not contain: the product category, the competitor's feature list, and the specification's device-type table. Those are inputs to a product decision and outputs of this one.

Every arrow points forward, and protocol selection is the last box, not the first. A process that starts at the bottom — "we are building a Type 2 device" — runs every arrow backwards and turns each step into a justification rather than an analysis.

5. The Design-Review Matrix

The questions a senior reviewer should ask, and what each answer commits.

Rows 4 and 5 are the ones that do the work. A capability whose justification cannot be falsified is not an engineering argument, and the tell in a review is a justification that is a noun — "it's an accelerator" — rather than a sentence about what the workload does with data.

6. Teaching-model boundary

7. RTL 1 and 2 — Select the Minimum, Then Price It

module minimal_selector #(
  parameter bit IMPLEMENT_ALL = 1'b0,  // 1 = "Type 2 sounds powerful"
  parameter bit UNDER_SELECT  = 1'b0   // 1 = drop a required protocol
) (
  input  logic       clk, rst_n,
  input  logic [2:0] required_mask,    // {mem, cache, io} from 6.3
  input  logic [2:0] wishlist_mask,    // what someone would like to have
  input  logic       valid,
  output logic [2:0] selected_mask, beyond_requirement,
  output logic       minimal,
  output logic       under_selected_err, silent_widening_err
);
  assign selected_mask = valid
                       ? (IMPLEMENT_ALL ? 3'b111
                                        : (UNDER_SELECT ? (required_mask & 3'b011)
                                                        : required_mask))
                       : 3'b000;
  assign beyond_requirement = valid ? (selected_mask & ~required_mask) : 3'b000;
  assign minimal = valid && (beyond_requirement == 3'b000);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      under_selected_err <= 1'b0; silent_widening_err <= 1'b0;
    end else if (valid) begin
      // Never select less than the requirement.
      if ((required_mask & ~selected_mask) != 3'b000) under_selected_err <= 1'b1;
      // Widening beyond the requirement is a decision, not a default. It is
      // only legitimate if someone asked for it explicitly.
      if ((beyond_requirement & ~wishlist_mask) != 3'b000) silent_widening_err <= 1'b1;
    end
  end
endmodule

The wishlist_mask input is the whole design. Capability beyond the requirement is not forbidden — a product decision may legitimately add it. What is forbidden is adding it silently, with no one having asked. The diagnostic fires on exactly that: something in the selection that neither the requirement nor an explicit wish accounts for.

Then the cost.

  // Per-capability teaching weights: .io is the floor, .cache is the most
  // state-heavy, .mem carries software enablement.
  always_comb begin
    dp = 8'd0; st = 8'd0; dv = 8'd0; sw = 8'd0;
    if (mask[0]) begin dp = dp + 8'd2; st = st + 8'd1; dv = dv + 8'd2; sw = sw + 8'd1; end
    if (mask[1]) begin dp = dp + 8'd3; st = st + 8'd6; dv = dv + 8'd8; sw = sw + 8'd2; end
    if (mask[2]) begin dp = dp + 8'd3; st = st + 8'd3; dv = dv + 8'd6; sw = sw + 8'd4; end
    // The interaction term: two coherent protocols together are more than
    // their sum, because their states interact (Chapter 6.1's Type 2 point).
    if (mask[1] && mask[2]) begin st = st + 8'd3; dv = dv + 8'd6; end
  end

  mask | datapath state  DV  software | total  DV-configs
  001  |    2       1     2     1      |   6       2
  011  |    5       7    10     3      |  25       4
  101  |    5       4     8     5      |  22       4
  111  |    8      13    22     7      |  50       8
 
  io=6  io+cache=25  io+mem=22  all three=50
  additive prediction = 41, actual = 50, interaction = 9
  state cost: io=1  +cache=7  +mem=4

Three things in that table carry the chapter.

Adding .cache to an .io device roughly quadruples the total. Not because the datapath grows — datapath goes 2 → 5 — but because state goes 1 → 7 and DV goes 2 → 10. The cheap part of a coherent protocol is the wires.

Type 2 is superadditive. Adding both coherent protocols costs 50, while the additive prediction from adding each separately is 41. The extra 9 is the interaction: a device holding both a cache and host-managed memory has state that interacts, which is exactly 6.1's point that Type 2 is not Type 1 plus Type 3.

DV is the largest column in every row. At 111 it is 22 of 50 — larger than datapath, state and software combined.

8. RTL 3 — What the Verification Space Actually Does

  always_comb begin
    k = 4'd0;
    if (built_mask[0]) k = k + 4'd1;
    if (built_mask[1]) k = k + 4'd1;
    if (built_mask[2]) k = k + 4'd1;
    cfg = 6'd1 << k;                       // every subset may be enabled
    prs = (k >= 4'd2) ? ((k * (k - 4'd1)) >> 1) : 6'd0;
  end

  built=001 : caps=1 configs=2 pairs=0 DV points=2
  built=011 : caps=2 configs=4 pairs=1 DV points=5
  built=101 : caps=2 configs=4 pairs=1 DV points=5
  built=111 : caps=3 configs=8 pairs=3 DV points=11

The exponent is the point. A built capability can be enabled or disabled, and both states must work — 5.6 showed that a capability present in silicon and off by negotiation or policy is a normal configuration, not an edge case. So k capabilities give 2^k configurations before any pairwise interaction is considered.

Going from one protocol to three takes the configuration count from 2 to 8 and the DV points from 2 to 11 — five and a half times the verification surface for three times the protocols. And this model is a lower bound: it counts enable-combinations and pairs, not the cross of those with traffic patterns, error injection, or the fairness matrix from 6.2.

9. RTL 4 — Over-Provision, Under-Provision, and Dead Capability

A design review has three findings available, and collapsing them loses the one that matters.

  assign under = valid ? (required_mask & ~built_mask) : 3'b000;
  assign over  = valid ? (built_mask & ~required_mask) : 3'b000;
  // Dead capability is the strongest evidence: built, and no traffic at all.
  assign dead  = valid ? (built_mask & ~traffic_seen)  : 3'b000;
 
  // A review passes only if nothing is missing. Over-provision is reported,
  // not fatal -- collapsing them loses which one was found.
  assign pass_review = valid && (ONE_VERDICT ? ((under == 3'b000) && (over == 3'b000))
                                             : (under == 3'b000));

  need=111 built=111 traffic=111 : under=000 over=000 dead=000 pass=1
  need=111 built=011 traffic=011 : under=100 over=000 dead=000 pass=0  <-- gap
  need=101 built=111 traffic=111 : under=000 over=010 dead=000 pass=1  <-- over
  need=111 built=111 traffic=011 : under=000 over=000 dead=100  <-- DEAD .mem

Under-provision fails the review; over-provision is reported and does not. A device that meets every requirement and carries a spare engine still does the job — refusing it is 6.3's XOR bug. The ONE_VERDICT variant collapses them and blocks a perfectly good architecture.

The fourth row is the strongest finding a review can produce. Dead capability is built, negotiated, and carrying zero traffic — not a prediction that it might be unnecessary, but a measurement that it was. That is the evidence 6.3 said must ship in the current part because the decision it informs arrives during the next one's architecture phase.

10. RTL 5 — Capability Is Not Enablement

The distinction that most often collapses in real designs.

module capability_vs_policy #(
  parameter bit CONFLATE = 1'b0        // 1 = capability bit doubles as enable
) (
  input  logic       clk, rst_n,
  input  logic [2:0] built_mask,       // what silicon contains
  input  logic [2:0] negotiated_mask,  // what the link agreed (Chapter 5.6)
  input  logic [2:0] policy_mask,      // what the platform chooses to run
  input  logic       valid,
  output logic [2:0] capability_mask, active_mask, off_by_policy,
  output logic       enabled_unbuilt_err, policy_lost_err
);
  assign capability_mask = valid ? built_mask : 3'b000;
  // Active requires all three: built, negotiated, and chosen.
  assign active_mask = valid
                     ? (CONFLATE ? (built_mask & negotiated_mask)
                                 : (built_mask & negotiated_mask & policy_mask))
                     : 3'b000;
  assign off_by_policy = valid ? (built_mask & negotiated_mask & ~policy_mask) : 3'b000;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      enabled_unbuilt_err <= 1'b0; policy_lost_err <= 1'b0;
    end else if (valid) begin
      if ((active_mask & ~built_mask)  != 3'b000) enabled_unbuilt_err <= 1'b1;
      if ((active_mask & ~policy_mask) != 3'b000) policy_lost_err     <= 1'b1;
    end
  end
endmodule

  built=111 negotiated=111 policy=111 : capability=111 active=111 off-by-policy=000
  policy disables .mem                : capability=111 active=011 off-by-policy=100
  conflated variant active=111  <-- ignores policy entirely
  link cannot do .mem                 : capability=111 active=011

Three independent conditions, and active requires all three. Built, negotiated (5.6), chosen. capability_mask follows only the first — because what silicon contains does not change when a platform turns something off, and software asking "what can this device do" is a different question from "what is running".

Rows 2 and 4 make the point from both sides: policy off with the link capable, and link incapable with policy on. Both yield active = 011, and they are completely different situations — one is a configuration choice, the other a compatibility outcome. off_by_policy is what separates them, and a design with a single enable bit cannot.

This also completes a thread running through the whole module. Chapter 5.6 required that advertised must equal enabled. Here, capability and active are deliberately different — because they answer different questions. The rule is not "one mask", it is that every mask must say which question it answers.

11. RTL 6 — Scoring Reviews

  assign gap_c   = (under != 3'b000);
  assign waste_c = (under == 3'b000) && ((over != 3'b000) || (dead != 3'b000));
  /* every review lands in exactly one of pass / gap / waste */

  reviews=40 pass=10 gap=10 waste=20
  conservation reviews == pass+gap+waste : 40 == 40
  every bucket matched an independent reference   : ok
  10 of 40 architectures had a gap; 20 carried waste

Twice as many architectures carried waste as had a gap. That ratio is the argument for this chapter's existence: gaps get found, because something does not work. Waste does not, because everything works.

12. Assertions

Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog. Not executed; each maps to a procedural stand-in and a mutation.

// SAFETY -------------------------------------------------------------------
// V1 — the selection always covers the requirement.
a_covers_requirement: assert property (@(posedge clk) disable iff (!rst_n)
  valid |-> ((required_mask & ~selected_mask) == '0));
 
// V2 — nothing is selected beyond the requirement without an explicit wish.
a_no_silent_widening: assert property (@(posedge clk) disable iff (!rst_n)
  valid |-> ((selected_mask & ~required_mask & ~wishlist_mask) == '0));
 
// V3 — cost is monotone in capability.
a_cost_monotone: assert property (@(posedge clk) disable iff (!rst_n)
  valid |-> (cost_total >= cost_of(mask & (mask - 1))));
 
// V4 — SUPERADDITIVITY: Type 2 exceeds the sum of its coherent halves.
//      A self-consistency check on the total cannot see the interaction term.
a_type2_superadditive: assert property (@(posedge clk) disable iff (!rst_n)
  (mask == 3'b111) |-> (cost_total > cost_011 + cost_101 - cost_001));
 
// V5 — the configuration space is exponential in the capability count.
a_dv_exponential: assert property (@(posedge clk) disable iff (!rst_n)
  valid |-> (n_configs == (1 << n_caps)));
 
// V6 — a review passes only if nothing is unmet.
a_review_gap: assert property (@(posedge clk) disable iff (!rst_n)
  pass_review |-> (under == '0));
 
// V7 — over-provision alone never fails a review.
//      An INDEPENDENCE property: it states what must NOT block the verdict.
a_over_not_fatal: assert property (@(posedge clk) disable iff (!rst_n)
  ((under == '0) && (over != '0)) |-> pass_review);
 
// V8 — nothing runs that was not built.
a_active_subset_built: assert property (@(posedge clk) disable iff (!rst_n)
  (active_mask & ~built_mask) == '0);
 
// V9 — nothing runs that policy disabled.
a_policy_respected: assert property (@(posedge clk) disable iff (!rst_n)
  (active_mask & ~policy_mask) == '0);
 
// V10 — capability does NOT track policy.
//       The second independence property: capability answers a different
//       question and must not collapse into the enable state.
a_capability_independent: assert property (@(posedge clk) disable iff (!rst_n)
  valid |-> (capability_mask == built_mask));
 
// V11 — CONSERVATION: every review lands in exactly one bucket.
a_scorecard_conserved: assert property (@(posedge clk) disable iff (!rst_n)
  n_reviews_q == n_pass_q + n_gap_q + n_waste_q);
 
// PROCESS PROPERTY — NOT ENFORCEABLE IN RTL --------------------------------
// V12 — every built capability eventually carries traffic.
//       This is the property the whole chapter is about and NO assertion can
//       enforce it: it depends on a workload that arrives after tapeout. It is
//       stated so the verification plan records that the design cannot
//       guarantee it and the telemetry must.
a_no_dead_capability: assert property (@(posedge clk) disable iff (!rst_n)
  built_mask |-> s_eventually (traffic_seen == built_mask));

V7 and V10 are both independence properties, and both catch mutations that positive properties miss — V7 that over-provision must not block a verdict, V10 that capability must not follow policy. Chapters 6.1 and 6.3 needed the same shape, which is now four independence properties across three chapters.

V12 is stated deliberately as unenforceable. It is the property this entire chapter exists to serve, and no RTL can guarantee it — which is precisely why the answer is telemetry rather than an assertion, and why the counters have to ship.

13. Mutation Testing

Twelve mutations. Clean code restored after each.

12/12 killed, 0 escaped

Four escaped on the first run, and they split into three causes.

M3 and M4 were missing checks on the model's claims. The testbench verified that the cost total equalled the sum of its parts and that cost increased with capability — both of which hold under a mutation that deletes the interaction term or prices .cache at .io rates. §7's callout draws the rule: assert what the model claims, not that the model adds up.

M2 was the unreachable-checker category — under_selected_err cannot fire on a design that never under-selects, so disabling it changes nothing. Killing it needed a deliberately under-selecting variant, the same pattern used in 6.2 and 6.3.

The verdict-ordering mutations were equivalent mutants — §11's callout explains why no branch-structure mutation can change behaviour here, and why the fix was to mutate the condition instead.

14. Debug Lab

// Type 2 covers everything, so build Type 2.
assign selected_mask = 3'b111;

  required=001 : selected=001 (correct) | implement-all=111
  silent_widening_err=1
  cost: io=6  all three=50   DV points: 2 -> 11

assign selected_mask      = required_mask;                       // the minimum
assign beyond_requirement = selected_mask & ~required_mask;
// widening is legitimate only if someone explicitly asked:
if ((beyond_requirement & ~wishlist_mask) != 3'b000) silent_widening_err <= 1'b1;

// Review: does the device meet the requirement?
assign pass_review = (required_mask & ~built_mask) == 3'b000;

  need=111 built=111 traffic=011 : under=000 over=000 dead=100  <-- DEAD .mem
  40 reviews: pass=10 gap=10 waste=20

assign dead = built_mask & ~traffic_seen;    // built, and never used
// report it as a distinct finding alongside under and over
assign waste_c = (under == 3'b000) && ((over != 3'b000) || (dead != 3'b000));

// Cost pressure: drop the coherent engine.
assign selected_mask = required_mask & 3'b101;   // io + mem only

  required=111 selected=101 : under=010
  under_selected_err=1

assign selected_mask = required_mask;
if ((required_mask & ~selected_mask) != 3'b000) under_selected_err <= 1'b1;
// and positive-test it with an under-selecting build

// The device has memory and a .mem engine, so advertise it.
assign capability_mask = built_mask;
assign active_mask     = built_mask;      // no negotiation, no policy

  link cannot do .mem : capability=111 active=011   (correct)
  conflated variant   : active=111  <-- ignores negotiation and policy
  enabled_unbuilt_err / policy_lost_err

assign capability_mask = built_mask;                                  // what exists
assign active_mask     = built_mask & negotiated_mask & policy_mask;  // what runs
assign off_by_policy   = built_mask & negotiated_mask & ~policy_mask;

// One mask for everything.
assign capability_mask = built_mask & policy_mask;

  policy disables .mem : capability=111 (correct) active=011
  collapsed variant    : capability=011  <-- hardware "lost" a feature

assign capability_mask = built_mask;    // independent of policy, by construction
assign active_mask     = built_mask & negotiated_mask & policy_mask;

// One configuration to verify: everything on.
localparam int DV_CONFIGS = 1;

  built=111 : caps=3 configs=8 pairs=3 DV points=11
  a plan sized for 1 configuration covers 1 of 8

// The space is exponential in the capability count.
cfg = 6'd1 << k;                              // enable-combinations
prs = (k >= 2) ? ((k * (k - 1)) >> 1) : 0;    // interacting pairs
assign dv_points = cfg + prs;

15. Verification Plan

Rows 4, 7, 8 and 11 are this chapter's additions. Row 4 exists because self-consistency is not a model check; rows 7 and 8 because independence properties catch what positive ones cannot; row 11 because a priority between two conditions is untestable unless some case satisfies both — and here it turned out no case can, which was itself the finding.

16. Design Review

• For each protocol in the design, who asked for it, and what workload sentence justifies it?
• What measurement would prove that justification wrong? Does it ship in this part?
• What does each capability cost in state and DV, not in datapath?
• How many enable-configurations does the DV plan cover, of 2^k?
• Are capability, negotiation and policy three separate masks?
• Can a review report over-provision without failing the design?
• Is there any capability with no traffic in the previous generation's telemetry?
• If the workload changed tomorrow, which capability would become dead, and how would you know?

17. How This Appears in Real Engineering

"Implement everything" is proposed as risk reduction. It converts an uncertain requirement into a certain cost — roughly 8× in this chapter's model, with DV the largest column and the verification space growing from 2 configurations to 11 points.

Waste outnumbers gaps two to one. The scorecard's ratio matches the field: gaps get found because something does not work, and dead capability never announces itself. Only per-class telemetry surfaces it, and only if it shipped.

The capability/policy collapse breaks fleet management. Debug Lab 5 is a tooling failure with a hardware cause: an administrator's reversible choice gets recorded as a permanent hardware fact, and the capability database quietly degrades.

The disabled-capability configurations are where field failures live. A plan that verifies the all-on case covers one of 2^k, and the untested remainder is exactly what a heterogeneous fleet produces — mixed generations negotiating subsets (5.6) and sites disabling features by policy.

18. Common Misconceptions

19. Interview Reasoning

20. Exercises

1. Price. Using §7's model, compute the total and per-dimension cost of a Type 1 versus a Type 3 device, then of Type 2. Show the interaction term explicitly and state which dimension grows fastest.

2. Remove. An architecture specifies all three protocols for a device whose workload streams host memory once and keeps its local memory private. Remove the unnecessary protocols, give the new cost and DV point count, and state what capability the design loses.

3. DV task. Write the two independence properties from §12 and explain, for each, why every positive property in the plan passes on a design that violates it.

4. Design. Extend capability_vs_policy so a capability can be built, negotiated, policy-enabled, and quiesced for maintenance as four states. State which existing diagnostic becomes ambiguous and what new invariant is required.

5. Debug task. Telemetry shows a .cache engine with zero traffic across a fleet. Give your investigation order, and name the two measurements that distinguish "never negotiated", "negotiated but disabled by policy", and "available and genuinely unused".

6. Critique. Argue that protocol selection should be a product decision rather than an architectural one. Give the strongest case, then say what this chapter would require before accepting it.

21. Summary

Selection is a discipline, and its output must be defensible.

• Select the minimum the requirement demands. Widening beyond it is a decision with an owner, never a default — which is what the wishlist models.
• Price every addition in four dimensions, and note that datapath is the smallest. State and DV dominate; DV is the largest column in every configuration.
• Type 2 is superadditive — more than the sum of its coherent halves, because the states interact.
• The verification space is exponential: k capabilities give 2^k enable-combinations, because every capability must be verified off as well as on.
• Under-provision is fatal; over-provision is reported. And dead capability — built, negotiated, zero traffic — is the strongest finding a review can produce.
• Three states, not two: built, negotiated, chosen. Capability must not follow policy, and active must require all three.
• Verification lessons: assert what a model claims, not that it adds up; independence properties catch what positive ones cannot; and a mutation that cannot change behaviour should be replaced, not chased.

22. Module 6 Complete

Module 6 asked what the three CXL protocols are, how they coexist, and how to choose among them.

• 6.1 — three contracts about who owns a resource and who keeps state about it. .cache and .mem depend on disjoint structures on both ends, and the device type is derived, never declared.
• 6.2 — three engines, two stacks, one link. The Arb/Mux arbitrates .io against .cache + .mem, so round-robin at every level gives 50/25/25, not equal service.
• 6.3 — five questions, two of them conjunctions. Local memory does not imply .mem; reuse rather than device class decides .cache.
• 6.4 — price every capability, keep capability and enablement distinct, and require each protocol to have an owner, a justification, and a measurement that could falsify it.

The thread through all four is that a protocol name is a commitment, not a feature. Each one binds state on both ends, a share of a shared link, a slice of an exponential verification space, and software that must exist. That is why the discipline runs from workload to selection and not the other way, and why the most valuable artefact any of these chapters produces is a counter that could prove the architect wrong.

Module 7 takes the first of the three apart in full: CXL.io, the mandatory one, and the only protocol every CXL device must implement.