CXL 2.0 New Capabilities

20.1 built the switch, 20.2 built the pool, and 20.3 enumerated the adapter delta that makes a device eligible for either.

This chapter is the rest of the generation — the capabilities that are not switching and not pooling, that nobody demos, and that decide whether a CXL 2.0 deployment survives contact with a running data centre. Every one of them is something a CXL 1.1 device never had to do, because a 1.1 device was present at boot, owned by one host, and gone only when the machine was powered off.

1. The Engineering Problem — Runtime Is The New Requirement

Six things separate a 2.0 capability set that works from one that demos.

Hot-add is a sequence with preconditions, not a timer. Five phases, each depending on everything the phases before it established, and a sequence that advances on a counter brings capacity online that nothing can reach. Section 5.

A device that leaves without warning is a different event from one that was taken away. Only one of the two can drain, and the other must abort what is outstanding or the requests simply never complete. Section 6.

Hot-add needs a hole in the address map, and the hole costs address space whether anything arrives in it or not. A platform that did not reserve cannot grow, however much capacity is physically present. Section 7.

Persistence has a deadline measured in microseconds. Everything acknowledged and not yet committed has to reach durable media inside the hold-up window, and a flush that reports success on starting is a flush that loses data. Section 8.

Telemetry the host does not act on is not a control loop. A 2.0 device reports how loaded it is; a host that reads that and does nothing has built a dashboard, not a mechanism. Section 9.

And single level is a ceiling, not a starting point. One switch's port count is the whole fabric, and that number is where CXL 2.0 stops. Section 14.

This chapter against 20.3, stated precisely. That one owns the adapter delta — what a 1.1 device must become. This one owns what the resulting device must then do at runtime, and section 15 shows a device with flawless hot-plug and five other capabilities missing.

2. The One-Sentence Model

The CXL 2.0 capability set is delivered when hot-add is sequenced on preconditions, surprise removal aborts what cannot complete, the address map reserves room for growth, the persistent flush fits its hold-up window, QoS telemetry is acted on rather than merely read, and the register interface is standard enough for one driver — and every defect below is a device that hot-plugs perfectly and is missing one of the other five.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real implementation spans firmware, a fabric manager, a host driver, a power-loss energy budget and a device controller, and none of that is reproduced. What is reproduced is the decision or the arithmetic each of them has to get right.

Each model is built twice — a correct build and a broken build selected by a parameter. The broken builds here share a shape worth naming up front: they are all the version that works in a demo. A hot-add that advances on a timer works when nothing is wrong. A removal treated as managed works when nothing is in flight. A flush that reports success on starting works when the window is generous. Every one of them fails first in production.

Figure 1 — Three of the six capabilities are things that happen to a host that is already running, which is precisely why none of them existed in CXL 1.1. The dashed path is section 15's weak definition: capacity can be added at runtime, therefore the generation is delivered.

5. RTL 1 — Hot-Add Is A Sequence Of Preconditions

// RTL 1 - the managed hot-add sequence. Capacity arriving under a running host
// passes through phases, and each one has a precondition the next depends on.
module hot_add_sequence #(parameter int SKIP_PHASES = 0) (
  input  logic clk, rst_n,
  input  logic       step,
  input  logic [2:0] phase,          // 0 idle 1 present 2 decoded 3 announced 4 online
  input  logic       link_up, space_reserved, decoder_set, host_acked,
  output logic       precond_met, may_advance, online,
  output logic [7:0] n_steps, n_skipped,
  output logic       out_of_order_err
);
  // Each phase depends on everything the phases before it established.
  assign precond_met =
      ((phase == 3'd0) && 1'b1)
   || ((phase == 3'd1) && link_up)
   || ((phase == 3'd2) && link_up && space_reserved)
   || ((phase == 3'd3) && link_up && space_reserved && decoder_set)
   || ((phase == 3'd4) && link_up && space_reserved && decoder_set && host_acked);
  // The skipping build advances on the phase counter alone, which is how a
  // sequence written as a timer rather than a handshake behaves.
  assign may_advance = (SKIP_PHASES != 0) ? 1'b1 : precond_met;
  assign online = (phase == 3'd4) && may_advance;
  // Advancing past a phase whose precondition does not hold.
  assign out_of_order_err = step && may_advance && !precond_met;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_steps <= 8'd0; n_skipped <= 8'd0;
    end else if (step) begin
      n_steps <= n_steps + 8'd1;
      if (may_advance && !precond_met) n_skipped <= n_skipped + 8'd1;
    end
  end
endmodule

Six steps through the sequence.

Four out-of-order advances against none.

The preconditions are cumulative, not per-phase. Going online requires the link and the reservation and the decoder and the acknowledgement — not merely the acknowledgement. That is what makes it a sequence rather than four independent checks, and it is why the fourth row fails at phase 1 rather than only at phase 4: a link that is not up invalidates everything downstream of it.

The last row is the failure with the worst symptom. The link is up, space is reserved, the decoder is programmed, and the host has not said it is ready. The skipping build brings the capacity online anyway, and the host discovers memory in its address map that its own allocator does not know exists. Nothing is wrong with the device. The failure is entirely in a sequence that treated an acknowledgement as advisory.

Why the broken build is not a strawman. A hot-add sequence written as a timed state machine — advance every N microseconds — is simpler, has no deadlock, and works perfectly on a bench where every step completes in nanoseconds. It fails when one step is slow, which in a rack is every step.

The failure mode is a stall, and a stall has no error code. Every other defect in this chapter produces something countable: a lost request, an unflushed byte, a flit in the clear. A sequence that stops at phase 3 produces nothing at all — the device is present, the link is up, and the capacity never appears. That asymmetry is why section 21 asks for the phase counter specifically, and why section 22's investigation begins by reading it rather than by looking for errors.

6. RTL 2 — Surprise Removal Is Not Managed Removal

// RTL 2 - surprise removal. A device that leaves without warning is a different
// event from one the fabric manager took away, and only one of them can drain.
module surprise_removal #(parameter int TREAT_ALL_MANAGED = 0) (
  input  logic clk, rst_n,
  input  logic       removal,
  input  logic       announced,
  input  logic [7:0] inflight,
  output logic       is_surprise, can_drain, needs_abort,
  output logic [7:0] lost_requests, n_removals, n_aborted,
  output logic       silent_loss_err
);
  assign is_surprise = !announced;
  // Only an announced removal can drain: a device that is gone cannot complete
  // anything. A surprise removal must abort what is outstanding.
  assign can_drain   = announced;
  assign needs_abort = (TREAT_ALL_MANAGED != 0) ? 1'b0 : (is_surprise && (inflight != 8'd0));
  assign lost_requests = (is_surprise && !needs_abort) ? inflight : 8'd0;
  // Requests that will never complete and that nothing aborted.
  assign silent_loss_err = removal && is_surprise && (inflight != 8'd0) && !needs_abort;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_removals <= 8'd0; n_aborted <= 8'd0;
    end else if (removal) begin
      n_removals <= n_removals + 8'd1;
      if (needs_abort) n_aborted <= n_aborted + 8'd1;
    end
  end
endmodule

Five removals.

Two aborts against none, and two silent losses.

20.1 §11 asked the switch to wait for a drain. This chapter asks what happens when waiting is not an option. A managed removal is exactly that drain: the fabric manager announced it, the device is still there, and outstanding requests complete normally. A surprise removal has no such option — the device is gone, and every outstanding request is now a completion that will never arrive.

Aborting is worse than completing and far better than nothing. An aborted request returns an error the host can handle: the load faults, the driver logs it, the thread sees an exception. The all-managed build returns nothing at all, and the host waits for a completion from a device that has been physically removed — surfacing minutes later as a timeout at whatever layer has the longest patience.

One outstanding request is as much a loss as six. The abort condition is an inequality against zero, not a threshold, for the same reason 20.1's quiesce condition was: a single hung thread is a hung thread.

Why the broken build is not a strawman. Treating every removal as managed is what a device built for CXL 1.1 does, because on a 1.1 attachment there was no other kind — a device left when the machine powered down, and there was no host still running to leave requests with. The handling is correct for the topology it was written for and silently wrong for the one it now lives in, which is the shape of nearly every defect in 20.3 as well.

The counter that matters is the ratio. Section 21 asks for removals split announced against surprise, and for the abort count. Either alone is uninformative: a fleet with no surprise removals needs no aborts, and a fleet with many surprise removals and zero aborts is this model's broken build in production.

7. RTL 3 — The Address Map Has To Reserve Room

// RTL 3 - reserved address space. Hot-add needs a hole in the map, and the hole
// costs address space whether anything ever arrives in it or not.
module map_reservation #(parameter int RESERVE_NOTHING = 0) (
  input  logic clk, rst_n,
  input  logic        plan,
  input  logic [15:0] present_gb, max_gb, map_gb,
  output logic [15:0] reserved_gb, map_used_gb, reserve_pct,
  output logic        map_fits, add_possible,
  output logic [7:0]  n_plans, n_unaddable,
  output logic        no_room_err
);
  logic [31:0] r_q;
  // The reservation is the difference between what is present now and the most
  // the platform will ever admit.
  assign reserved_gb = (RESERVE_NOTHING != 0) ? 16'd0
                     : ((max_gb > present_gb) ? (max_gb - present_gb) : 16'd0);
  assign map_used_gb = present_gb + reserved_gb;
  assign map_fits    = (map_used_gb <= map_gb);
  assign r_q = (map_gb == 16'd0) ? 32'd0
                                 : (({16'd0, reserved_gb} * 32'd100) / {16'd0, map_gb});
  assign reserve_pct = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  // Capacity may be added only if the map already holds space for it.
  assign add_possible = map_fits && (reserved_gb != 16'd0);
  // A platform that admits more capacity than it reserved room for.
  assign no_room_err = plan && (max_gb > present_gb) && !add_possible;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_unaddable <= 8'd0;
    end else if (plan) begin
      n_plans <= n_plans + 8'd1;
      if (!add_possible) n_unaddable <= n_unaddable + 8'd1;
    end
  end
endmodule

Five plans. 256 GB present, a 1024 GB platform maximum.

Three plans could not add. The no-reserve build could not add on any of them.

Reserved address space is paid for up front and may never be used. 37% of the map held for capacity that does not exist and may never arrive — and that is the correct configuration. A platform that reserves nothing has a smaller map, a simpler firmware, and no ability to grow, which is exactly what a 1.1 platform was.

Row two is the failure that ships. The platform declares a 1024 GB maximum and gives itself a 512 GB map. Every number in the datasheet is consistent; the capacity simply cannot be addressed when it arrives. The map size and the declared maximum are set by different teams, and nothing checks that they agree.

Row three is not an error and the model says so: a platform already at its maximum reserves nothing and can add nothing, which is a full platform rather than a broken one. Row four is the mis-set limit — more present than the declared maximum — where the reservation floors at zero rather than wrapping.

8. RTL 4 — Persistence Has A Deadline

// RTL 4 - global persistent flush. On a power event, everything the host has
// written and the device has not committed must reach persistence in the window
// the energy budget allows.
module persistent_flush #(parameter int IGNORE_WINDOW = 0) (
  input  logic clk, rst_n,
  input  logic        power_event,
  input  logic [15:0] dirty_kb, flush_kbpm, window_us,
  output logic [15:0] flush_us, unflushed_kb,
  output logic        completes, safe,
  output logic [7:0]  n_events, n_lost,
  output logic        data_loss_err
);
  logic [15:0] capacity_kb;
  // How much the device can push to persistence inside the hold-up window.
  assign flush_us = (flush_kbpm == 16'd0) ? 16'hFFFF : (dirty_kb / flush_kbpm);
  assign capacity_kb = flush_kbpm * window_us;
  assign completes = (dirty_kb <= capacity_kb);
  assign unflushed_kb = completes ? 16'd0 : (dirty_kb - capacity_kb);
  // The ignoring build reports success on any flush it started.
  assign safe = (IGNORE_WINDOW != 0) ? 1'b1 : completes;
  // Data acknowledged to the host that does not reach persistence.
  assign data_loss_err = power_event && safe && !completes;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_events <= 8'd0; n_lost <= 8'd0;
    end else if (power_event) begin
      n_events <= n_events + 8'd1;
      if (!completes) n_lost <= n_lost + 8'd1;
    end
  end
endmodule

Five power events. A 50 microsecond hold-up window.

Two incomplete flushes, and the ignoring build called both safe.

This is the only model in the batch where the failure is silent, permanent and undetectable after the fact. 300 KB of data the host was told had been written does not exist when the machine comes back. There is no counter that fires, no error completion, no log entry — the device powered down mid-flush and the bytes were never anywhere durable.

The lever is the flush rate, and it is an energy-budget question. Doubling it from 10 to 20 KB/µs turns row two from a loss into a success without changing the window or the dirty data. That rate is bounded by how much charge the hold-up capacitors carry and how fast the media accepts writes — it is a board-design number, decided before any of this software exists.

Row three is the exact boundary: 500 KB at 10 KB/µs is exactly 50 µs and exactly completes. Row five is the device with no flush engine at all, which reports an unbounded time rather than dividing by zero and calling it instantaneous — a distinction that matters because a zero divided into anything is a zero, and a flush time of zero reads as a flush that finished.

Exactly completing is not a margin. Row three fits the window with nothing to spare, which means the next power event on a workload one kilobyte dirtier loses data. Section 21 asks for the remaining window at flush completion for that reason: an outcome counter says a flush succeeded, and only a margin counter says whether it will succeed again.

9. Waveform — A Flush Against Its Window

The flushing row going low at cycle 6 is not the flush finishing. It is the hold-up energy running out. Everything after that point is a machine that has already lost power, and the 300 KB in lost_kb was acknowledged to an application that will come back believing it was written.

10. RTL 5 — Telemetry Nobody Acts On

// RTL 5 - QoS telemetry. A 2.0 device reports how loaded it is, and the value is
// only useful if the host acts on it before the device is saturated.
module qos_telemetry #(parameter int REPORT_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        sample,
  input  logic [7:0]  load_pct, throttle_threshold,
  input  logic [15:0] host_rate, floor_rate,
  output logic        device_asks, host_throttles,
  output logic [15:0] applied_rate,
  output logic [7:0]  n_samples, n_ignored,
  output logic        saturation_err
);
  // The device raises the request; the host decides. A host that reads the
  // telemetry and never acts on it is the report-only build.
  assign device_asks    = (load_pct >= throttle_threshold);
  assign host_throttles = (REPORT_ONLY != 0) ? 1'b0 : device_asks;
  // Throttling halves the rate but never below the contracted floor.
  assign applied_rate = host_throttles
                        ? (((host_rate >> 1) < floor_rate) ? floor_rate : (host_rate >> 1))
                        : host_rate;
  // A device asking for relief and receiving none.
  assign saturation_err = sample && device_asks && !host_throttles;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_samples <= 8'd0; n_ignored <= 8'd0;
    end else if (sample) begin
      n_samples <= n_samples + 8'd1;
      if (device_asks && !host_throttles) n_ignored <= n_ignored + 8'd1;
    end
  end
endmodule

Four samples. Throttle at 80% load, a 400 Gbps host rate, a 100 Gbps contractual floor.

Three ignored requests against none.

The floor is the part that makes throttling usable. Halving 120 Gbps would give 60, below a contracted 100 Gbps minimum, so the throttle stops at 100. Without that clamp a device under sustained pressure would be throttled toward zero, and the mechanism that exists to protect the device would breach the guarantee the host was sold.

Reading telemetry is not a control loop. The report-only build samples correctly, records correctly, and would populate a perfectly accurate dashboard showing a device at 90% load. It has no actuator. This is the same shape as 19.4 §8's attribution problem — a number that goes up with nothing attached to it — and it is the most common way a well-instrumented system fails to use its own instrumentation.

11. RTL 6 — The Device Describes Itself

// RTL 6 - the device describes itself. A 2.0 device reports its own latency and
// bandwidth characteristics, and a host that ignores them places blind.
module self_description #(parameter int ASSUME_UNIFORM = 0) (
  input  logic clk, rst_n,
  input  logic        place,
  input  logic [15:0] reported_ns, reported_gbps, assumed_ns,
  input  logic [15:0] tier_limit_ns,
  output logic [15:0] used_ns, error_ns,
  output logic        fits_tier, described,
  output logic [7:0]  n_placements, n_misplaced,
  output logic        blind_placement_err
);
  assign described = (reported_ns != 16'd0);
  // The assuming host uses one number for every device in the fleet.
  assign used_ns = (ASSUME_UNIFORM != 0) ? assumed_ns
                 : (described ? reported_ns : assumed_ns);
  assign error_ns = (used_ns > reported_ns) ? (used_ns - reported_ns)
                                            : (reported_ns - used_ns);
  assign fits_tier = (used_ns <= tier_limit_ns);
  // A device placed in a tier its own reported latency does not meet. No separate
  // "it described itself" term is needed: a reported latency above a non-zero
  // tier limit is already a non-zero reported latency.
  assign blind_placement_err = place && fits_tier && (reported_ns > tier_limit_ns);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_placements <= 8'd0; n_misplaced <= 8'd0;
    end else if (place) begin
      n_placements <= n_placements + 8'd1;
      if (described && (reported_ns > tier_limit_ns)) n_misplaced <= n_misplaced + 8'd1;
    end
  end
endmodule

Five placements into a 300 ns tier, against a host assumption of 200 ns.

Two devices too slow for the tier, and the assuming host placed both.

Self-description is the capability that makes tiering possible at all, and it is the one most easily ignored, because a host that assumes a single latency for every device works perfectly on a fleet where every device is the same part. Row three is why the fall-back matters: a device that reports nothing leaves the host with only the assumption, and the correct build cannot be blamed for a placement it had no data for.

Rows four and five are the tier boundary. Exactly the limit fits; one nanosecond past does not. On a real tiering decision that boundary is where a workload's p99 either meets its target or does not, and it is decided by a number the device volunteers and the host may not read.

12. RTL 7 — Protection Before Traffic

// RTL 7 - integrity has to be established before traffic, not alongside it.
module ide_establishment #(parameter int TRAFFIC_EARLY = 0) (
  input  logic clk, rst_n,
  input  logic       attempt,
  input  logic       link_trained, keys_exchanged, ide_active,
  output logic       may_send, protected_now,
  output logic [7:0] n_attempts, n_unprotected,
  output logic       clear_traffic_err
);
  // Protection exists only once the link is trained, keys are exchanged and the
  // stream is switched on. All three, in that order.
  assign protected_now = link_trained && keys_exchanged && ide_active;
  assign may_send = (TRAFFIC_EARLY != 0) ? link_trained : protected_now;
  // Traffic sent on a link that is up and not yet protected.
  assign clear_traffic_err = attempt && may_send && !protected_now;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_attempts <= 8'd0; n_unprotected <= 8'd0;
    end else if (attempt) begin
      n_attempts <= n_attempts + 8'd1;
      if (may_send && !protected_now) n_unprotected <= n_unprotected + 8'd1;
    end
  end
endmodule

Two flits in the clear against none.

Three conditions, and the middle two fail independently. Keys exchanged with the stream not switched on is a real state — the exchange completes and the transition to protected operation is a separate step — and a device that treats the exchange as the finish line sends its first flits unprotected. 19.2 protects each flit once the channel exists. This is about the window before it does.

Row four is the case both builds get right, and for the same reason: an untrained link carries nothing at all, so there is no traffic to be unprotected. The early-traffic build's failure needs a link that is up, which is precisely the state it was written to exploit.

Figure 3 — The right-hand terminal is the one section 22 turns on. A stall is not an error: no code is returned, no counter fires, and the only evidence is a phase number that stopped advancing. Every gate here is a place a hot-add can end without anybody being told.

13. RTL 8 — One Driver, Or A Vendor Path

// RTL 8 - a standard register interface. One driver for every conforming device
// is the point; a vendor-specific escape hatch is how that is lost.
module register_conformance #(parameter int VENDOR_ESCAPE = 0) (
  input  logic clk, rst_n,
  input  logic       probe,
  input  logic [3:0] standard_mask, implemented_mask,
  input  logic [1:0] register_id,
  output logic [3:0] missing_mask,
  output logic       conformant, generic_driver_works,
  output logic [7:0] n_probes, n_vendor_paths,
  output logic       driver_break_err
);
  logic [3:0] rbit;
  logic is_standard, is_implemented;
  assign rbit = 4'b0001 << register_id;
  assign is_standard    = |(standard_mask & rbit);
  assign is_implemented = |(implemented_mask & rbit);
  // Every standard register must be implemented. Missing ones are what force a
  // vendor path into what should be one driver.
  assign missing_mask = standard_mask & ~implemented_mask;
  assign conformant   = (missing_mask == 4'd0);
  assign generic_driver_works = (VENDOR_ESCAPE != 0) ? 1'b1 : conformant;
  // A generic driver believed to work against a device missing a standard register.
  assign driver_break_err = probe && generic_driver_works && !conformant
                            && is_standard && !is_implemented;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_probes <= 8'd0; n_vendor_paths <= 8'd0;
    end else if (probe) begin
      n_probes <= n_probes + 8'd1;
      if (!conformant) n_vendor_paths <= n_vendor_paths + 8'd1;
    end
  end
endmodule

Four probes against four standard registers.

Two non-conforming devices, one driver break.

Row three is the timing of the failure, not its absence. The device is missing register 2 and the probe reads register 0, which exists. The vendor-escape build's claim is already false; nothing breaks yet. The break happens on the first probe of the missing register, which in a driver's initialisation path is a specific line, discovered by a specific customer, months after the device shipped.

Row four is the safe asymmetry, and it mirrors 20.3 §13's capability honesty: implementing more than the standard requires is wasteful and correct. Missing what the standard requires while claiming conformance is the failure, and the two errors have very different costs.

14. RTL 9 — What One Switch Level Reaches

// RTL 9 - what one switch level reaches. Single-level means the whole fabric is
// one switch's port count, and that is the rack-scale ceiling 2.0 stops at.
module single_level_reach #(parameter int ASSUME_CASCADE = 0) (
  input  logic clk, rst_n,
  input  logic       plan,
  input  logic [7:0] ports, hosts_wanted, devices_wanted,
  output logic [7:0] devices_reachable, ports_used, shortfall,
  output logic       plan_fits,
  output logic [7:0] n_plans, n_short,
  output logic       overreach_err
);
  logic [15:0] want_q;
  // Every host and every device consumes a port on the one switch.
  assign want_q = {8'd0, hosts_wanted} + {8'd0, devices_wanted};
  assign ports_used = (want_q > 16'd255) ? 8'hFF : want_q[7:0];
  assign devices_reachable = (hosts_wanted >= ports) ? 8'd0 : (ports - hosts_wanted);
  assign shortfall = (devices_wanted > devices_reachable)
                     ? (devices_wanted - devices_reachable) : 8'd0;
  assign plan_fits = (ASSUME_CASCADE != 0) ? 1'b1 : (want_q <= {8'd0, ports});
  // A plan accepted that one switch level cannot reach.
  assign overreach_err = plan && plan_fits && (want_q > {8'd0, ports});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_short <= 8'd0;
    end else if (plan) begin
      n_plans <= n_plans + 8'd1;
      if (shortfall != 8'd0) n_short <= n_short + 8'd1;
    end
  end
endmodule

Five plans against a 16-port switch.

Three plans with a shortfall; the cascading plan accepted all three.

Hosts and devices compete for the same ports, which is the part that surprises people. Every host attached consumes a port that a device could have used, so a sixteen-port switch supporting sixteen hosts reaches zero devices — a topology that is legal, buildable, and useless.

This is the ceiling of the entire 2.0 generation. 20.1 §14 said a plan needing more ports than one switch has cannot be answered by a second switch, because that is a two-hop topology CXL 2.0 cannot route. This model is that constraint expressed as a fleet-planning number: hosts plus devices, against one switch's port count, and no more. Module 21 is what lifts it.

Figure 4 — Hosts and devices draw from one pool. The dashed path at the right is the answer CXL 2.0 cannot give: a second switch is a two-hop topology, and a 2.0 address carries no path identifier to route it with. This is the ceiling Module 21 lifts.

15. RTL 10 — The Capability Set Assembled

// RTL 10 - the 2.0 capability set assembled. Everything the generation added
// beyond switching and pooling, and what each one is worth without the others.
module capability_model #(parameter int HOTPLUG_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       hotplug_sequenced,  // phases with preconditions, not a timer
  input  logic       removal_handled,    // surprise removal aborts what is outstanding
  input  logic       map_reserved,       // address space exists for what may arrive
  input  logic       flush_budgeted,     // the hold-up window covers the dirty data
  input  logic       qos_acted_on,       // the host throttles when the device asks
  input  logic       registers_standard, // one driver, no vendor escape
  output logic       complete,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_complete,
  output logic       false_complete_err
);
  assign fail_mask[0] = ~hotplug_sequenced;
  assign fail_mask[1] = ~removal_handled;
  assign fail_mask[2] = ~map_reserved;
  assign fail_mask[3] = ~flush_budgeted;
  assign fail_mask[4] = ~qos_acted_on;
  assign fail_mask[5] = ~registers_standard;
  // The hot-plug-only build checks that capacity can be added at runtime and
  // calls the 2.0 capability set delivered, which is what a feature demo shows.
  assign complete = (HOTPLUG_ONLY != 0) ? hotplug_sequenced : (fail_mask == 6'd0);
  assign false_complete_err = evaluate && complete && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_complete <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (complete) n_complete <= n_complete + 8'd1;
    end
  end
endmodule

Six configurations.

One complete of six, and four false claims.

The hot-plug-only definition is what a feature demo measures, and it is right about exactly one of the six. Rows four and five are the ones worth arguing over: a device with perfect hot-plug, correct removal, a reserved map and a budgeted flush, whose only defect is telemetry nobody acts on — or a register the generic driver cannot find. Neither shows up in a demo. Both show up in a fleet.

16. Quantitative Reasoning

Hot-add. Six steps, four out-of-order advances by the skipping build — including bringing capacity online with the host's acknowledgement never received.

Surprise removal. Six outstanding requests on an unannounced removal: the correct build aborts all six, the all-managed build loses all six silently. One outstanding request costs the same treatment as six.

Address map. 256 GB present against a 1024 GB maximum reserves 768 GB — 37% of a 2048 GB map held for capacity that does not exist. A platform declaring 1024 GB with a 512 GB map cannot reach its own maximum.

Persistent flush. 800 KB dirty at 10 KB/µs is 80 µs against a 50 µs window: 300 KB never reaches persistence, acknowledged to an application that will come back believing it was written. Doubling the flush rate to 20 KB/µs removes the loss entirely.

QoS. A device at 90% load asking for relief: the correct host halves 400 Gbps to 200; the report-only host leaves it at 400 on a saturated device. At 120 Gbps the throttle stops at the 100 Gbps floor, not 60.

Self-description. A device reporting 450 ns placed in a 300 ns tier. The assuming host uses 200 ns for every device — a 250 ns error — and places it anyway. Two of five placements were blind.

IDE. Four send attempts, two in the clear by the early-traffic build, both on a link that was trained and not yet protected.

Registers. One missing standard register of four. The vendor-escape build claims the generic driver works, and the break lands on the first probe of the missing register — not on the first probe.

Single-level reach. A 16-port switch with 4 hosts reaches 12 devices. With 16 hosts it reaches zero. Four hosts and fourteen devices is 18 ports needed, 2 short, and no second switch can answer it.

The assembled model. Six capabilities, six configurations, one complete. The hot-plug-only definition reported five.

17. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Hot-add. Each precondition is failed individually, and the cumulative nature is asserted by failing an early one and checking a later phase.

chk(aPm == 1'b0, "announcing needs the decoder set");
chk(aMa == 1'b0, "so the correct sequence stops");
chk(kMa == 1'b1, "the skipping build advances anyway");
chk(kOn == 1'b1, "the skipping build brings it online regardless");

Surprise removal. The announced-with-traffic case is asserted as a drain rather than a loss, which is what separates the two removal kinds.

chk(sCd == 1'b1, "an announced removal drains six requests");
chk(sNa == 1'b0, "so no abort is needed");
chk(sLr == 8'd0, "and none are lost");

Address map. The exact-fit boundary is driven, and both non-error cases are asserted as not errors.

chk(rMu == 16'd1024, "for a map that carries exactly 1024");
chk(rMf == 1'b1,     "which exactly fits a 1024 GB map");
chk(rNr == 1'b0,     "which is not an error, it is full");

Flush. The exact window boundary and the no-engine case are both asserted.

chk(fFu == 16'd50, "500 KB takes exactly 50 us");
chk(fCo == 1'b1,   "which exactly fits the window");
chk(fFu == 16'hFFFF, "a device that cannot flush reports an unbounded time");

QoS. The threshold is asserted at exactly its value, and the floor clamp is asserted as an exact rate.

chk(qDa == 1'b1, "exactly the threshold is a request");
chk(qAr == 16'd100, "but stops at the 100 Gbps floor rather than 60");

Self-description. The undescribed device is asserted to fall back and to produce no blind-placement error.

chk(dUn == 16'd200, "so even the correct host assumes");
chk(dBe == 1'b0,    "and cannot be blamed for a blind placement");

IDE. Each of the three conditions is failed alone.

Registers. The non-conforming device probed at an implemented register is asserted to break nothing yet, which is what dates the failure to the first probe of the missing one.

chk(vGw == 1'b1, "and the vendor build still claims the driver works");
chk(vDb == 1'b0, "but this probe hits a register that exists, so nothing breaks yet");

Reach. Hosts consuming device ports is asserted at exactly the port count and one past it.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 235 checks across two testbenches, 121 on the front five models and 114 on the back five, all passing on the unmutated sources.

18. Mutation Testing

Fifty mutations were injected one at a time.

50 injected, 50 killed, after two survivors were diagnosed.

Survivor 1 — a boundary never driven. The address-map fit comparison survived becoming exclusive because no plan had map_used_gb exactly equal to map_gb. Four plans, none of them at the boundary the guard exists for. One more stimulus — 256 GB present, a 1024 GB maximum, a 1024 GB map — kills it. Batch 019's carry-forward said to drive every threshold at exactly its value; this is the one that got past it.

Survivor 2 — a provably equivalent term. The blind-placement check carried described alongside reported_ns > tier_limit_ns. The second implies the first: a reported latency above a non-zero tier limit is necessarily non-zero. The term was documentation written as logic. It was deleted with a comment explaining why, and the mutation replaced with one that is not equivalent — dropping the tier comparison entirely, which the first correctly-placed device kills at once.

What did not happen is worth recording. Batch 019's two hardest classes — a mask bit never set, and a broken build too weak to exercise the correct one — did not recur, because both were checked for during authoring rather than discovered by the harness. Section 17's last assertion line exists for exactly that reason: each of the six mask bits is driven false alone, deliberately, before any mutation ran.

19. Verification Strategy

What a testbench for a real 2.0 device must cover.

Every precondition of a sequence, failed alone. Hot-add has four, and failing an early one must be visible at a late phase — a link that is not up invalidates the online phase, not merely the present phase.

Both kinds of every event that looks like one event. A managed removal and a surprise removal share a signal and need opposite handling. So do a full platform and an unaddressable one, and a device that reports nothing and one that reports badly.

Every threshold at exactly its value. The map fit, the flush window, the QoS threshold, the tier limit, the port count. Section 18's first survivor was the one of these that was missed.

The cases that are correct and look like failures. A platform at its maximum that cannot add capacity. A device reporting nothing, placed on an assumption. An untrained link carrying no unprotected traffic. A non-conforming device probed at a register that exists. Each trips a naive checker.

Each mask bit driven false alone. Six capabilities, six single-bit configurations plus the all-clear. Counting configurations is not covering bits, and this batch drove all six deliberately.

What a real device needs that these models do not have. Concurrency — a hot-add and a hot-remove in flight against the same port. Ordering — a power event during a hot-add sequence, where the capacity is neither fully absent nor fully present. Recovery — what a fabric manager must reassert after a link retrain mid-sequence. Every one of those is where runtime capability actually breaks.

20. Synthesis and Implementation Reality

The hot-add sequence is a handshake across three components. The device signals presence, the fabric manager programs the decoder, the host acknowledges. Each is a separate transaction on a separate path with its own latency, and the interlocks live in the fast path permanently for the sake of an event that happens rarely.

The flush engine is an energy budget before it is a design. Section 8's flush rate is bounded by hold-up capacitance and by how fast the media accepts writes, both fixed on the board. The only software lever is how much data is allowed to be dirty at once, which means a write-back policy chosen against a capacitor value.

QoS throttling needs an actuator on the host side. The device's telemetry is a register; acting on it means a rate limiter in the host's request path, which is real logic on the critical path of every request whether or not anything is throttled. That cost is why the report-only build exists.

Reserved address space is not free at any level. Page tables, IOMMU mappings and firmware memory maps all scale with the reserved region, not with what is populated. 37% of a map held for absent capacity is 37% of several structures that must be built and walked.

Register conformance is a verification problem, not a design one. Implementing the standard registers is straightforward; proving that every one behaves as the standard says — under reset, under error, at every access width — is a test suite somebody has to write and run, and the vendor escape hatch is what appears when that suite is behind schedule.

21. Silicon Observability

The flush pair is the one that has to be designed in. A device that reports flush starts and not flush completions cannot distinguish section 8's success from its failure, and the failure is invisible in every other way — the machine simply comes back with less data than it acknowledged. The margin counter matters as much as the outcome, because a flush that completed with two microseconds to spare is a flush that will fail the next time the workload is slightly dirtier.

22. Debug Lab

Symptom. A pooled CXL device is hot-added to a running host. The fabric manager reports success. The host's memory total does not change. No errors anywhere — the link is up, the device enumerates, the decoder reads back correctly, and every counter is zero.

Step 1 — did the sequence complete? Read the hot-add phase counter. It reads phase 3, announced — not phase 4. The sequence stalled one step from the end, and stalling is not an error, so nothing fired.

Step 2 — what does phase 4 need? The link (up), the reservation, the decoder (set), and the host's acknowledgement. Three of four are visible and correct. The acknowledgement is the one with no counter.

Step 3 — why did the host not acknowledge? Read the host's address map against the device's requested window. The platform declares a 1024 GB maximum; the firmware built a 512 GB map. The new capacity has nowhere to live, so the host's hot-plug handler declined it — correctly, and quietly.

Step 4 — confirm the arithmetic. 256 GB present, a 1024 GB maximum, so 768 GB should be reserved and the map should carry 1024. It carries 512. Section 7's row two, in a rack.

The finding. Nothing failed. The device is fine, the fabric manager is fine, the sequence is fine, and the host is correctly refusing capacity it cannot address. The defect was committed at platform bring-up, when a firmware map size and a declared platform maximum were chosen by different teams and nothing compared them.

Why it looked like nothing. Every layer reported success at its own level, because every layer did succeed at its own level. The only observable is a phase counter that stopped advancing, and a stall has no error code.

The fix. Rebuild the firmware map to cover the declared maximum. Then add the check that should have existed: map size against declared maximum, asserted at boot, which is one comparison and would have failed loudly at bring-up instead of silently at the first hot-add.

What made this hard. The symptom — memory not appearing — points at the device or the fabric manager. The cause was in the host's own firmware, established months earlier, and expressed only as the absence of an acknowledgement that nothing counts.

23. Design Review

1. Is hot-add sequenced on preconditions or on a timer? If nobody can name the precondition for each phase, it is a timer. Section 5.

2. What is the phase counter, and is a stalled sequence observable? Section 22 turns on this question.

3. Does a surprise removal abort what is outstanding? And is the abort count non-zero when surprises are non-zero? Section 6.

4. Does the firmware address map cover the declared platform maximum? One comparison, and nothing checks it. Section 7.

5. What is the hold-up window, and how much data may be dirty inside it? The second half is the only software lever. Section 8.

6. Does the device report flush completions, or only flush starts? Section 21, and section 8 is what "only starts" costs.

7. Is there an actuator on the QoS telemetry? Reading it is not a control loop. Section 10.

8. Does the host read the device's self-reported latency, or assume? Section 11, and the answer decides whether tiering means anything.

9. Is protected traffic gated on the stream being active, or only on the keys being exchanged? Two different conditions. Section 12.

10. How many hosts and devices does the topology need, against one switch's port count? Hosts consume device ports. Section 14, and if the answer exceeds the ceiling, the answer is Module 21.

24. How This Appears In Real Engineering

A platform bring-up team owns section 7 and usually does not know it. The firmware memory map is sized against what the board populates; the declared platform maximum is a marketing and validation number. Section 22 is the two disagreeing, and the check that reconciles them costs one line and is almost never present.

A device team implementing global persistent flush discovers the requirement is not "flush on power loss" but "flush this much in this long", and both numbers come from the board. The design conversation that follows is about write-back policy — how much data the device is allowed to hold dirty — which nobody expects to be a power-integrity decision.

A fleet operations team finds section 10 the hard way. Devices report saturation, dashboards show it accurately, and latency degrades anyway, because the telemetry was wired to a display and not to a rate limiter. The gap between "we can see it" and "we respond to it" is a piece of host-side logic somebody has to fund.

A driver team owns section 13 and is the first to feel a vendor escape hatch. One driver for every conforming device is the entire value of a standard register interface, and each non-conforming device converts that into a device-specific code path that will be maintained for the life of the product.

25. Common Misconceptions

"Hot-plug means the device can be added at runtime." It means five phases with cumulative preconditions. A sequence that advances on a timer brings capacity online that nothing can reach. Section 5.

"A removal is a removal." Announced and unannounced need opposite handling: one drains, the other must abort. Section 6.

"An orphaned request will time out and be fine." It times out at whatever layer has the longest patience, minutes later, with no attribution. An aborted request faults immediately and is handled. Section 6.

"Reserving address space is free." It costs page tables, IOMMU mappings and firmware map entries, for capacity that may never arrive. Section 7 holds 37% of the map.

"The flush will finish." Only if the dirty data fits the hold-up window. 800 KB at 10 KB/µs in a 50 µs window loses 300 KB. Section 8.

"We have QoS telemetry." Reading a load percentage is not throttling. Section 10.

"All the devices are about the same speed." A 450 ns device placed in a 300 ns tier on a 200 ns assumption. Section 11.

"The link is up, so the traffic is protected." Trained, keyed and active are three conditions. Section 12.

"It advertises a standard register interface." Implementing three of four standard registers still forces a vendor path, and the break lands on the first probe of the missing one. Section 13.

"We can add another switch for more ports." Not in CXL 2.0 — that is a two-hop topology it cannot route. Section 14.

26. Interview Reasoning

Q. What has to be true before hot-added capacity comes online?

The link up, address space reserved, the decoder programmed, and the host acknowledging — cumulatively, not individually. The follow-up worth reaching: what happens if the sequence advances without the acknowledgement. The answer is memory appearing in a host's address map that its allocator does not know about, which is a corruption with no error attached.

Q. A device is pulled from a running system with requests outstanding. What must happen?

Those requests must be aborted, because a device that is gone cannot complete them. The distinction to draw is against a managed removal, which can drain — same signal, opposite handling. The sharper point: an aborted request faults immediately and is handled; an orphaned one times out minutes later with no attribution, which is strictly worse.

Q. Why does a platform reserve address space it is not using?

Because hot-added capacity needs somewhere to live, and the map is built at boot. The follow-up: what it costs — page tables, IOMMU mappings and firmware entries scaled to the reservation rather than to what is populated. And the failure: a map smaller than the declared maximum means capacity that physically arrives and cannot be addressed.

Q. What decides whether a persistent flush succeeds?

Dirty bytes against flush rate times hold-up window. All three are board numbers except the first, which is a write-back policy — so the only software lever is how much data may be dirty at once. The follow-up worth being ready for: the failure is silent and permanent, so the device must report flush completions, not flush starts.

Q. A device reports 90% load and latency is degrading. What is missing?

An actuator. Telemetry read and not acted on is a dashboard. The follow-up is what throttling must respect — a contractual floor, or the mechanism protecting the device breaches the guarantee sold to the host.

Q. A 16-port CXL 2.0 switch, and you need 16 hosts to reach some memory. How many devices can you attach?

None. Hosts and devices compete for the same ports, and single-level means one switch is the whole fabric. The follow-up that separates candidates: the answer is not a second switch, because that is a two-hop topology CXL 2.0 cannot route — it is CXL 3.0.

27. Exercises

1. Extend RTL 1 with a per-phase timeout and show that a stalled sequence must be distinguishable from a slow one — which is section 22's missing observable.

2. Add a power event mid-hot-add to RTL 1 and RTL 4 together, and decide what state the capacity is in when the machine returns.

3. Give RTL 2 a partial-drain path: a removal announced with a deadline, draining what it can and aborting the rest. Assert the split exactly.

4. Add the boot-time check from section 22 to RTL 3 — map size against declared maximum — and show it fails loudly on the configuration that failed silently.

5. Make RTL 4's dirty-byte count a function of a write-back policy parameter, and find the policy that just fits a 50 µs window at 10 KB/µs.

6. Extend RTL 5 to a closed loop: throttle, observe the load falling, and release. Show the oscillation a threshold with no hysteresis produces.

7. Give RTL 6 a bandwidth dimension alongside latency and show that a device can fit a latency tier and fail a bandwidth one.

8. Model the window in RTL 7 as a cycle count and quantify how many flits an early-traffic build sends before protection becomes active.

9. Extend RTL 8 so each standard register is probed in turn and report the first probe at which a generic driver breaks — the number section 13 argues is the real cost.

10. Add a seventh capability to RTL 10. If it is implied by one of the six, say which; if not, give the device it catches that the current mask calls complete.

28. Summary

Switching and pooling are what CXL 2.0 is sold on. This is what it has to do afterwards, every day, on a machine that is already running.

Hot-add is five phases with cumulative preconditions. A sequence advancing on a timer went out of order four times in six steps, including bringing capacity online that the host never acknowledged.

Surprise removal is not managed removal. Six outstanding requests, aborted by the correct build and lost silently by the other — and an orphaned request is worse than an errored one, because the host has nothing to handle.

The address map must reserve room. 256 GB present against a 1024 GB maximum holds 768 GB — 37% of the map — for capacity that does not exist, and a platform declaring more than its map covers cannot reach its own maximum.

Persistence has a deadline. 800 KB dirty at 10 KB/µs in a 50 µs window loses 300 KB, acknowledged to an application that will come back believing it was written — the only failure in this batch that is silent, permanent and invisible afterwards.

Telemetry without an actuator is a dashboard. A device at 90% load asking for relief and receiving none, three samples of four, while the report-only host recorded every one of them accurately.

A device that describes itself is only useful to a host that reads it. A 450 ns device placed in a 300 ns tier on a 250 ns error, twice in five placements.

Protection needs three conditions, and keys exchanged with the stream not yet active is a real state that sent two flits of four in the clear.

One missing standard register forces a vendor path, and the driver breaks on the first probe of that register — not on the first probe, which is why the failure arrives months late and looks like a driver bug.

Single level is the ceiling. A 16-port switch with 16 hosts reaches zero devices, and no second switch answers it.

Hot-plug is one capability of six. The definition a feature demo measures called five of six configurations complete when one was.

Module 21 — CXL 3.0 begins with the constraint every chapter of Module 20 ran into: 21.1 — CXL 3.0 Fabric Enhancements is what happens when the single switch level is lifted.