CXL Switch Architecture

15.1 built the manager. 15.2 built the shape.

Both treated the switch as a box with ports. This chapter opens the box: what a port lets in, where a flit spends its time inside, and what one failing port is allowed to take with it.

1. The Engineering Problem — Three Protocols On One Link, And They Must Not Meet

A CXL switch is not a packet switch with a CXL label on it. Six things make it specific.

One link carries three protocols. CXL.io, CXL.cache and CXL.mem arrive interleaved on the same wire. Inside the switch they must become three independent paths, because a stall on one of them is not a reason to stall the others. Section 6 builds both arrangements.

The port is the only place a malformed flit can be stopped. Everything admitted is trusted by every stage after it — which is precisely why the check belongs at the boundary and nowhere else. Section 5 builds it.

One queue is a head-of-line blocking machine. A flit that cannot move stops every flit behind it, including the ones whose destination is free. Inside a switch those are usually a different protocol entirely. Section 8 measures the cost.

A switch is a pipeline, and the useful number is not the total. It is which stage the time was spent in, because that is the only form of the number anyone can act on. Section 9 attributes it.

One port will fail. Whether that is a port outage or a switch outage is decided entirely by whether the failing port can hold resources the others need. Section 10 builds the isolation.

And the switch is itself a managed element. It is the far end of 15.1's out-of-band path, and its management interface must reach it when the data path cannot — especially then. Section 12.

This chapter against 16.2 and 16.3, stated precisely. This one owns the structure: ports, protocol separation, the pipeline, isolation. 16.2 owns how a destination is chosen. 16.3 owns credits, buffering and arbitration fairness. If a section here could be moved into either without loss, it is in the wrong chapter.

2. The One-Sentence Model

A switch admits at the boundary, separates the three protocols immediately, moves each through a pipeline of stages that can each backpressure, and lets no port's failure become another port's failure — and every defect below is one of admit, separate, backpressure or isolate missing.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Four ports, three protocols, a four-stage pipeline, an eight-deep buffer. A real switch has far more of all four, and the pipeline is deeper.

What is faithful: the admission check at the boundary, the per-protocol demultiplex, head-of-line blocking, per-stage time attribution, stage-by-stage backpressure, port isolation, the ordering-domain rule, cut-through versus store-and-forward, and occupancy as the predictive statistic.

What is not: the stage count, every width, and every duration.

Every model is parameterised so the correct behaviour and a specific plausible failure are the same source under a different parameter, instantiated together, driven by one stimulus stream.

5. RTL 1 — What A Port Lets In

module port_rx #(parameter int TRUST_INPUT = 0) (
  input  logic clk, rst_n,
  input  logic       flit_valid,
  input  logic [1:0] proto,          // 0 io, 1 cache, 2 mem, 3 reserved
  input  logic [3:0] length,         // in units; 0 is not a flit
  input  logic       crc_ok,
  output logic       admit,
  output logic       bad_proto_err, bad_length_err, crc_err,
  output logic       admitted_bad_err,
  output logic [7:0] n_seen, n_admitted, n_dropped, n_bad_admitted
);
  logic proto_ok, length_ok, well_formed;
  assign proto_ok    = (proto != 2'd3);
  assign length_ok   = (length != 4'd0);
  assign well_formed = proto_ok && length_ok && crc_ok;
  // TRUST_INPUT admits whatever arrives, which pushes every one of these
  // checks onto stages that have no way to make them.
  assign admit = flit_valid && (well_formed || (TRUST_INPUT != 0));
  assign bad_proto_err  = flit_valid && !proto_ok;
  assign bad_length_err = flit_valid && !length_ok;
  assign crc_err        = flit_valid && !crc_ok;
  // A flit inside the switch that should never have been let in.
  assign admitted_bad_err = admit && !well_formed;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_seen <= 8'd0; n_admitted <= 8'd0;
      n_dropped <= 8'd0; n_bad_admitted <= 8'd0;
    end else if (flit_valid) begin
      n_seen <= n_seen + 8'd1;
      if (admit) n_admitted <= n_admitted + 8'd1;
      else       n_dropped  <= n_dropped + 8'd1;
      if (admitted_bad_err) n_bad_admitted <= n_bad_admitted + 8'd1;
    end
  end
endmodule

Five flits are offered — well-formed, reserved protocol, zero length, CRC failure, and a one-unit flit at the field's boundary:

  ingress : 2 of 5 admitted, 3 dropped (proto=1 len=1 crc=1) | trusting build admitted 3 bad

Three refusals for three different reasons, each named separately and each checked against an independent three-rule oracle:

Every error is gated on flit_valid. With nothing offered, no field can be in error however bad the fields look — and the testbench drives exactly that, with all three fields invalid and flit_valid low.

The boundary is driven in both directions: a one-unit flit is a flit, and a zero-unit one is not. != 0, not > 1.

TRUST_INPUT admits all five and puts three flits inside the switch that no later stage can route, validate, or account for. admitted_bad_err is the monitor that names it, and section 15 makes it the assembled switch's invariant.

6. RTL 2 — Three Protocols, Three Paths

module proto_demux #(parameter int SHARED_PATH = 0) (
  input  logic clk, rst_n,
  input  logic       flit_in,
  input  logic [1:0] proto,
  input  logic       io_ready, cache_ready, mem_ready,
  output logic       io_go, cache_go, mem_go,
  output logic       accepted,
  output logic       misrouted_err,
  output logic [7:0] n_io, n_cache, n_mem, n_stalled
);
  logic [2:0] sel;
  logic       target_ready, all_ready;
  assign sel = flit_in ? (3'd1 << proto[1:0]) : 3'd0;
  assign target_ready = (proto == 2'd0) ? io_ready
                      : (proto == 2'd1) ? cache_ready
                      :                   mem_ready;
  assign all_ready = io_ready && cache_ready && mem_ready;
  // SHARED_PATH gates every protocol on every channel being ready, which is
  // one queue wearing three names.
  assign accepted = flit_in && ((SHARED_PATH != 0) ? all_ready : target_ready);
  // Exactly one channel takes a flit, and it is the one the flit named.
  assign misrouted_err = accepted &&
    (({2'd0, io_go} + {2'd0, cache_go} + {2'd0, mem_go}) != 3'd1);
  assign io_go    = accepted && sel[0];
  assign cache_go = accepted && sel[1];
  assign mem_go   = accepted && sel[2];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_io <= 8'd0; n_cache <= 8'd0; n_mem <= 8'd0; n_stalled <= 8'd0;
    end else begin
      if (io_go)    n_io    <= n_io + 8'd1;
      if (cache_go) n_cache <= n_cache + 8'd1;
      if (mem_go)   n_mem   <= n_mem + 8'd1;
      if (flit_in && !accepted) n_stalled <= n_stalled + 8'd1;
    end
  end
endmodule

  demux   : io=1 cache=3 mem=0 stalled=1 | shared-path build stalled=4

One stall against four, on the same stimulus. The memory channel fills; the correct switch keeps moving io and cache flits through it, and the shared-path build stops everything — counting three stalls on a channel those flits never use.

Two properties needed explicit stimulus, and each was a mutation:

• Each channel's readiness gates only its own protocol. The testbench drives io_ready low and offers a cache flit, and asserts it goes. A demux that reads the wrong channel's ready signal works perfectly until two channels disagree.
• A flit with no channel is a misroute, not an acceptance. Protocol 3 has no sel bit, so no channel takes it. misrouted_err fires. This is the second line of defence behind section 5, and the only reason it is reachable at all is that the demux can be driven directly.

The != 3'd1 in the misroute test is exact on purpose: exactly one channel must take the flit. Not "at least one", which admits a flit going two places, and not "at most one", which admits a flit going nowhere.

7. Waveform — Eight Cycles Of A Full Memory Channel

Transcribed from the printed trace. Both builds see one stimulus stream.

Read accept against sh_acc. They agree everywhere except cycles 3, 4 and 5 — the three cycles in which one channel being full is the only thing that has changed.

8. RTL 3 — One Queue, And What It Costs The Flit Behind

module hol_block #(parameter int SINGLE_QUEUE = 0) (
  input  logic clk, rst_n,
  input  logic       push,
  input  logic [1:0] head_proto, next_proto,
  input  logic       head_blocked, next_dest_ready,
  output logic       head_go, next_go,
  output logic       hol_err,
  output logic [7:0] n_moved, n_held, max_hol_run
);
  logic [7:0] run_q;
  assign head_go = push && !head_blocked;
  // With separate queues per protocol, a blocked head only blocks its own
  // queue. SINGLE_QUEUE makes every flit wait behind the head.
  assign next_go = push && next_dest_ready
                   && ((SINGLE_QUEUE == 0) || !head_blocked);
  // A flit whose destination is free, held anyway.
  assign hol_err = push && next_dest_ready && !next_go;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_moved <= 8'd0; n_held <= 8'd0; run_q <= 8'd0; max_hol_run <= 8'd0;
    end else begin
      // Both may move in the same cycle, so the increment is computed once.
      n_moved <= n_moved + {7'd0, head_go} + {7'd0, next_go};
      if (hol_err) begin
        n_held <= n_held + 8'd1;
        run_q <= run_q + 8'd1;
        if (run_q + 8'd1 > max_hol_run) max_hol_run <= run_q + 8'd1;
      end else run_q <= 8'd0;
    end
  end
endmodule

  hol     : moved=12 held=0 | single-queue build moved=8 held=3 worst run=3

Twelve flits moved against eight, on the identical stream.

hol_err has three terms and the middle one is what makes it a defect rather than an observation:

• push — a queue with nothing in it holds nothing.
• next_dest_ready — this is the whole point. A flit whose own destination is full is not being head-of-line blocked; it is waiting for its own destination. The testbench drives that case explicitly and asserts hol_err is low in both builds.
• !next_go — it was held.

Without the middle term the monitor fires on every congested destination in the switch and is disabled within a week.

The worst run is latched, and a shorter later hold does not raise it — driven and asserted, because a "worst" that a quiet interval can rewrite is the last interval, not the worst.

9. RTL 4 — Where A Flit Actually Spends Its Time

module switch_pipe #(parameter int NO_BACKPRESSURE = 0) (
  input  logic clk, rst_n,
  input  logic       flit_in,
  input  logic       route_ready, arb_ready, egress_ready,
  output logic [2:0] stage,
  output logic       flit_out, overrun_err,
  output logic [7:0] t_decode, t_route, t_arb, t_egress, n_out, max_total
);
  logic [2:0] st_q;
  logic [7:0] tot_q;
  logic       may_advance;
  localparam logic [2:0] IDLE=3'd0, DEC=3'd1, RTE=3'd2, ARB=3'd3, EGR=3'd4;
  assign flit_out = (st_q == EGR) && egress_ready;
  // NO_BACKPRESSURE advances regardless of whether the next stage can take it.
  assign may_advance = (NO_BACKPRESSURE != 0)
                     || ((st_q == DEC) ? route_ready
                       : (st_q == RTE) ? arb_ready
                       : (st_q == ARB) ? egress_ready
                       :                 1'b1);
  // Advanced into a stage that was not ready to receive.
  assign overrun_err = (st_q != IDLE) && may_advance &&
    (((st_q == DEC) && !route_ready)
  || ((st_q == RTE) && !arb_ready)
  || ((st_q == ARB) && !egress_ready));
  ...
      if (st_q != IDLE) begin
        tot_q <= tot_q + 8'd1;
        // Time is attributed to the stage that held it, not to the total.
        case (st_q)
          DEC: t_decode <= t_decode + 8'd1;
          RTE: t_route  <= t_route + 8'd1;
          ARB: t_arb    <= t_arb + 8'd1;
          EGR: t_egress <= t_egress + 8'd1;
          default: ;
        endcase
      end
      case (st_q)
        IDLE: if (flit_in) begin st_q <= DEC; tot_q <= 8'd0; end
        DEC:  if (may_advance) st_q <= RTE;
        RTE:  if (may_advance) st_q <= ARB;
        ARB:  if (may_advance) st_q <= EGR;
        EGR:  if (egress_ready) begin
                st_q <= IDLE;
                n_out <= n_out + 8'd1;
                if (tot_q + 8'd1 > max_total) max_total <= tot_q + 8'd1;
              end
        default: st_q <= IDLE;
      endcase
    end
  end
endmodule

  pipeline: decode=3 route=4 arb=2 egress=1 total=10 | no-backpressure build overran=1

Ten cycles through a four-stage pipeline, and seven of them were spent waiting for the two stages in the middle. That is the number that is actionable; "ten cycles" is not.

The testbench asserts max_total equals the sum of the four stage counters exactly. A per-stage breakdown that does not add up to the total is a breakdown of something else.

NO_BACKPRESSURE is sampled in the cycle it decides to advance, not after it has. A stage that has already moved on shows nothing wrong — the error exists only at the instant of the decision, and the first version of this testbench looked two cycles too late and reported the mutation as killed when it was not.

10. RTL 5 — One Port Fails

module port_isolation #(parameter int SHARED_FATE = 0) (
  input  logic clk, rst_n,
  input  logic [3:0] port_up,          // which of four ports are healthy
  input  logic [3:0] port_req,
  output logic [3:0] port_grant,
  output logic       switch_up,
  output logic       collateral_err,
  output logic [7:0] n_served, n_lost, down_cycles, max_down
);
  logic any_down;
  logic [7:0] dn_q;
  assign any_down = (port_up != 4'b1111);
  assign any_down = (port_up != 4'b1111);
  // A healthy port is served whatever any other port is doing. SHARED_FATE
  // makes every port depend on every port.
  assign switch_up  = (SHARED_FATE != 0) ? !any_down : 1'b1;
  assign port_grant = port_req & port_up & {4{switch_up}};
  // A port that is up, asking, and not served.
  assign collateral_err = |(port_req & port_up & ~port_grant);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      dn_q <= 8'd0; n_served <= 8'd0; n_lost <= 8'd0; max_down <= 8'd0;
    end else begin
      // Four ports asking is four grants, not one.
      n_served <= n_served + {4'd0, port_grant[0]} + {4'd0, port_grant[1]}
                           + {4'd0, port_grant[2]} + {4'd0, port_grant[3]};
      if (collateral_err) n_lost <= n_lost + 8'd1;
      if (any_down) begin
        dn_q <= dn_q + 8'd1;
        if (dn_q + 8'd1 > max_down) max_down <= dn_q + 8'd1;
      end else dn_q <= 8'd0;
    end
  end
endmodule

  isolation: served=17 lost=0 down latched=3 | shared-fate build served=16 lost=3

One port of four fails. The isolated switch keeps serving the other three; the shared-fate build serves nobody, and collateral_err names the three healthy ports it stopped.

collateral_err is gated on port_up. A failed port asking and not being served is not collateral damage — it is the failure. The monitor counts only ports that were healthy, asking, and refused anyway, and the testbench drives a failed port that is still requesting to prove the gate is doing work.

n_served sums all four grants in one expression. Four healthy ports asking is four grants in one cycle, not one, and the testbench asserts the delta exactly — a counter that increments once per cycle regardless of how many ports were served reports a busy switch as an idle one.

11. RTL 6 — Forwarding Early, And What It Forwards

module cut_through #(parameter int CUT_THROUGH = 0) (
  input  logic clk, rst_n,
  input  logic       start_rx, end_rx, crc_ok,
  input  logic [7:0] store_cycles,
  output logic       forward, late_error,
  output logic [7:0] latency, n_forwarded, n_bad_forwarded, n_dropped_clean,
  output logic [7:0] mean_latency, total_latency
);
  // Cut-through forwards on the first cycle; store-and-forward waits for the
  // whole flit, which is also when the CRC is known.
  assign forward = (CUT_THROUGH != 0) ? start_rx : (end_rx && crc_ok);
  assign latency = (CUT_THROUGH != 0) ? 8'd1 : store_cycles;
  // A flit already forwarded and then found to be corrupt.
  assign late_error = (CUT_THROUGH != 0) && end_rx && !crc_ok;
  assign mean_latency = (n_forwarded == 8'd0) ? 8'd0
                      : (total_latency / n_forwarded);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_forwarded <= 8'd0; n_bad_forwarded <= 8'd0;
      n_dropped_clean <= 8'd0; total_latency <= 8'd0;
    end else begin
      if (forward) begin
        n_forwarded   <= n_forwarded + 8'd1;
        total_latency <= total_latency + latency;
      end
      if (late_error) n_bad_forwarded <= n_bad_forwarded + 8'd1;
      // Store-and-forward drops a corrupt flit without ever sending it.
      if (end_rx && !crc_ok && (CUT_THROUGH == 0))
        n_dropped_clean <= n_dropped_clean + 8'd1;
    end
  end
endmodule

  forward : store mean=6 dropped clean=1 | cut-through mean=1 forwarded corrupt=1

Six cycles against one — and one corrupt flit dropped before it was ever sent, against one already in the fabric.

The trade is not "which is faster". It is what the fabric does with an error it has already forwarded. Store-and-forward knows the CRC before it commits, because the flit is whole; cut-through commits at the first cycle, when the CRC does not exist yet. The corrupt flit is now somewhere else's problem, and the receiving port has to detect it for itself.

late_error exists to make that a counted event rather than a downstream mystery. It is not an error in the cut-through design — it is the design working as specified, and the count is what tells an operator how often the specification's cost was paid.

12. RTL 7 — Reaching The Switch To Configure It

module switch_mgmt #(parameter int MGMT_IN_BAND = 0) (
  input  logic clk, rst_n,
  input  logic       mgmt_req, data_congested,
  input  logic       cfg_apply, quiesced,
  input  logic [3:0] in_flight,
  output logic       mgmt_reachable, cfg_ok,
  output logic       unreachable_err, cfg_live_err,
  output logic [7:0] n_mgmt, n_blocked, n_cfg, max_block
);
  logic [7:0] blk_q;
  // Out of band means the management path does not care what the data path is
  // doing. MGMT_IN_BAND is the switch whose management shares that fate.
  assign mgmt_reachable = (MGMT_IN_BAND != 0) ? !data_congested : 1'b1;
  // A configuration change is applied only to a quiesced switch.
  assign cfg_ok = cfg_apply && mgmt_reachable && quiesced;
  assign unreachable_err = mgmt_req && !mgmt_reachable;
  assign cfg_live_err = cfg_ok && (in_flight != 4'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      blk_q <= 8'd0; n_mgmt <= 8'd0; n_blocked <= 8'd0;
      n_cfg <= 8'd0; max_block <= 8'd0;
    end else begin
      if (mgmt_req && mgmt_reachable) n_mgmt <= n_mgmt + 8'd1;
      if (cfg_ok) n_cfg <= n_cfg + 8'd1;
      if (unreachable_err) begin
        n_blocked <= n_blocked + 8'd1;
        blk_q <= blk_q + 8'd1;
        if (blk_q + 8'd1 > max_block) max_block <= blk_q + 8'd1;
      end else blk_q <= 8'd0;
    end
  end
endmodule

  mgmt    : reachable=1 cfg applied=3 live-change=1 | in-band build blocked=5 for 3 cycles

This is 15.1 section 8 seen from the switch's end, and it produces the same argument from the other side. The in-band build is unreachable for exactly as long as it is congested — which is exactly when somebody needs to reach it.

cfg_ok requires both reachability and quiescence, and each half is driven alone: a change to an unreachable switch is refused, and so is a change to one that has not been quiesced.

unreachable_err is gated on a request. A switch nobody is trying to configure is not blocked, however unreachable it is, and the testbench asserts that with the congestion still asserted and mgmt_req low.

13. RTL 8 — What Must Stay In Order

// STALL_A is the build in which a is held by something else while b is let
// through -- the only way a same-domain pair can actually be reordered.
module order_domain #(parameter int ORDER_ALL = 0, parameter int ORDER_NONE = 0,
                      parameter int STALL_A = 0) (
  input  logic clk, rst_n,
  input  logic       a_req, b_req,
  input  logic [1:0] a_proto, b_proto,
  input  logic       a_first,          // a was received before b
  output logic       a_go, b_go,
  output logic       reorder_err,
  output logic [7:0] n_parallel, n_serialised, n_reordered
);
  assign same_domain = (a_proto == b_proto);
  assign must_order = a_req && b_req && same_domain;
  assign a_go = a_req && (STALL_A == 0);
  assign b_go = b_req &&
    (((ORDER_NONE != 0) || (STALL_A != 0)) ? 1'b1
   : (ORDER_ALL  != 0) ? !a_req
   :                     !(must_order && a_first));
  assign reorder_err = must_order && a_first && b_go && !a_go;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_parallel <= 8'd0; n_serialised <= 8'd0; n_reordered <= 8'd0;
    end else if (a_req && b_req) begin
      if (a_go && b_go) n_parallel <= n_parallel + 8'd1;
      else              n_serialised <= n_serialised + 8'd1;
      if (must_order && a_first && b_go) n_reordered <= n_reordered + 8'd1;
    end
  end
endmodule

  order   : parallel=2 serialised=3 | order-all parallel=0 order-none parallel=5

Ordering is per domain, not per switch. Two flits of different protocols share no domain and move together; two of the same protocol are serialised in arrival order.

The two wrong answers bracket the right one, and the transcript prices both. ORDER_ALL never moved a pair together — correct and slow. ORDER_NONE always did — fast and wrong.

14. RTL 9 — What The Switch Has To Count

module switch_stats (
  input  logic clk, rst_n,
  input  logic        flit_in, flit_out, dropped, cycle_ev,
  input  logic [3:0]  occupancy, capacity,
  output logic [15:0] n_in, n_out, n_drop, n_cycles, occ_sum,
  output logic [7:0]  peak_occ, mean_occ_pct, drop_pct, util_pct
);
  logic [31:0] w_occ, w_drop, w_util;
  assign w_occ  = {16'd0, occ_sum} * 32'd100;
  // Mean occupancy against capacity, which is what a buffer was sized for.
  assign mean_occ_pct = (n_cycles == 16'd0 || capacity == 4'd0) ? 8'd0
                      : (w_occ / ({16'd0, n_cycles} * {28'd0, capacity}));
  assign drop_pct = (n_in == 16'd0) ? 8'd0 : (w_drop / {16'd0, n_in});
  assign util_pct = (n_cycles == 16'd0) ? 8'd0 : (w_util / {16'd0, n_cycles});
  assign w_drop = {16'd0, n_drop} * 32'd100;
  assign w_util = {16'd0, n_out}  * 32'd100;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_in <= 16'd0; n_out <= 16'd0; n_drop <= 16'd0;
      n_cycles <= 16'd0; occ_sum <= 16'd0; peak_occ <= 8'd0;
    end else begin
      if (flit_in)  n_in   <= n_in + 16'd1;
      if (flit_out) n_out  <= n_out + 16'd1;
      if (dropped)  n_drop <= n_drop + 16'd1;
      if (cycle_ev) begin
        n_cycles <= n_cycles + 16'd1;
        occ_sum  <= occ_sum + {12'd0, occupancy};
        // The peak is what a buffer has to survive; the mean is what it
        // usually holds, and they are rarely close.
        if ({4'd0, occupancy} > peak_occ) peak_occ <= {4'd0, occupancy};
      end
    end
  end
endmodule

  stats   : peak occ=8 of 8, mean=22%, drops=2 (9%), util=68%

The buffer hit its full capacity and its mean occupancy was 22 percent. That gap is the entire reason occupancy is the statistic worth exposing: a switch at 68 percent utilisation with a 22 percent mean occupancy looks comfortable, and it filled its buffer completely during the run.

Throughput is a lagging indicator here. It looks healthy right up to the moment the buffer is full, and then it looks healthy and the switch is dropping.

Three denominators, and they are three different numbers on purpose:

The testbench drives a run where arrivals, departures and cycles are all different, and asserts each ratio to an exact value. Three of the mutations here were wrong denominators that agreed with the right answer as long as two of those numbers happened to be equal.

15. RTL 10 — The Switch Assembled

module switch_top #(parameter int NO_ADMIT_CHECK = 0) (
  input  logic clk, rst_n,
  input  logic       arrive, well_formed,
  input  logic       demuxed, routed, arbitrated, departed,
  output logic [2:0] stage,
  output logic       in_switch, leaving,      // `inside` is a reserved word
  output logic       unadmitted_err,
  output logic [7:0] n_arrived, n_admitted, n_departed,
                     resident_cycles, max_resident
);
  localparam logic [2:0] OUT=3'd0, ADM=3'd1, DMX=3'd2, RTE=3'd3,
                         ARB=3'd4, EGR=3'd5;
  logic [2:0] st_q;
  logic [7:0] res_q;
  logic       bad_q;
  assign in_switch      = (st_q != OUT);
  assign leaving        = (st_q == EGR) && departed;
  // A flit inside the switch that the port never validated.
  assign unadmitted_err = in_switch && bad_q;
  ...
        OUT: if (arrive) begin
               n_arrived <= n_arrived + 8'd1;
               // The admission check is the only place this can be decided.
               if (well_formed || (NO_ADMIT_CHECK != 0)) begin
                 st_q <= ADM; res_q <= 8'd0;
                 bad_q <= !well_formed;
                 n_admitted <= n_admitted + 8'd1;
               end
             end
        ADM: st_q <= DMX;
        DMX: if (demuxed)    st_q <= RTE;
        RTE: if (routed)     st_q <= ARB;
        ARB: if (arbitrated) st_q <= EGR;
        EGR: if (departed) begin
               st_q <= OUT; bad_q <= 1'b0;   // the flag leaves with the flit
               n_departed <= n_departed + 8'd1;
             end
        default: st_q <= OUT;
      endcase

  assembled: 2 admitted of 3 arrived, residency=11 | no-check build admitted 3, unvalidated inside=1

Arrivals are counted whether or not the flit is admitted. That distinction is what makes the drop rate in section 14 meaningful, and it is one of the mutations — a switch that only counts what it admitted reports a perfect record forever.

unadmitted_err is the invariant: nothing inside the switch was unvalidated. It cannot fire in the correct build, which is what makes it an invariant, and NO_ADMIT_CHECK is the build that trips it.

The bad flag is carried with the flit and cleared when it leaves, and the testbench pushes the malformed flit all the way through the faulty build and then sends a clean one behind it, asserting the second is not reported as unvalidated. That is 15.3's run it twice lesson applied here: a flag that is set correctly and never cleared works perfectly for exactly one flit.

16. Quantitative Reasoning

Three worth a sentence.

1 stall against 4. One channel filled. The correct switch lost throughput on that channel only; the shared-path build lost it everywhere, and three of its four stalls were flits that had nothing to do with the congestion.

Peak 8 of 8 with a mean of 22%. Same run, same buffer. Only one of those two numbers would have told anyone the switch was in trouble.

7 of 10 cycles in two stages. A four-stage pipeline where two stages account for seventy percent of the residency. The total is not actionable; the breakdown is.

17. Assertions

Every property is an immediate check written as cond !== 1'b1, sampled after a settle.

18. Mutation Testing

112 mutations, one at a time, each required to make the baseline print RESULT: FAIL.

112 of 112 were killed.

The first run killed 82 and left 30 survivors — the largest first-run survivor count in this series, and the classification says why:

Two findings worth carrying forward.

Five survivors were assertions written as bounds. chk(sMeanOcc < 25) passes when the value is 0, and three wrong denominators in section 14 produce 0 or a coincidentally-similar number. Replacing every ratio check with an exact value killed all five at once. Assert the value, not the range — a bound is satisfied by the failure as often as by the success.

reorder_err was unreachable in all three builds. b_go && !a_go cannot hold when a_go is a_req and must_order requires it. A fourth build was added in which a is stalled by something outside the model, which is the realistic way a same-domain pair gets reordered — and it is now the only build in which the monitor can fire.

A representative sample:

19. Verification Strategy

Parameterised twin builds — and one triplet. TRUST_INPUT, SHARED_PATH, SINGLE_QUEUE, NO_BACKPRESSURE, SHARED_FATE, CUT_THROUGH, MGMT_IN_BAND, ORDER_ALL, ORDER_NONE, STALL_A, NO_ADMIT_CHECK. The ordering model needs four builds because two of them bracket the correct answer and a third is the only one in which the monitor can fire.

Exact values, never bounds. Every ratio in section 14 is asserted to a number. Five mutations survived a testbench full of < and > checks that the broken values also satisfied.

Every gating term falsified alone. flit_valid low with three bad fields. next_dest_ready low so head-of-line blocking is not confused with a full destination. port_up low on a port that is still asking. mgmt_req low while unreachable. Each of those is a term that reads as obviously necessary and had never been driven false.

Sample at the instant of the decision. overrun_err exists only in the cycle a stage decides to advance. A sample two cycles later shows a healthy pipeline and reports the mutation as killed.

Run it twice. A second flit through the pipeline, a second flit through the faulty assembled switch, a second shorter hold, a second shorter outage. Four separate defects live only in the second pass.

Independent oracles. Admission (three integer rules), collateral damage (a per-port loop), and ordering (a domain comparison) are each checked against a function that shares no structure with the design.

Delta discipline. Every combinational sample follows a settle.

20. Synthesis and Implementation Reality

Unlike the last two chapters, most of this is hardware.

The port receiver is real and it is on the critical path. Protocol decode, length check and CRC all have to complete before a flit is admitted, and the CRC is the expensive one. That is the actual argument behind section 11: cut-through exists because the CRC result arrives at the end of the flit and the forwarding decision wants to be made at the beginning.

The demultiplex is cheap and the three queues are not. Splitting one path into three is a decoder. Giving each of them independent buffering is three buffers, and section 8's twelve-against-eight is what that buys. This is the single largest area decision in the chapter and it is made once, at the top of the design.

The pipeline's backpressure is the part that is easy to get subtly wrong. Each stage's may_advance is one AND gate, and NO_BACKPRESSURE is what a design looks like when one of them is tied high during bring-up and never restored.

The per-stage counters are diagnostic and belong behind a configuration bit. They are the most useful thing in section 9 and they are pure observability — clock-gated or compiled out in production, with widths sized for a long run rather than a testbench.

The divisions are not synthesisable as written. Every percentage in section 14 uses /. In silicon the switch exposes occ_sum, n_cycles, n_in, n_out and n_drop as raw registers and firmware computes the ratios. What matters in hardware is that the raw counters exist and are the right ones — the mutations in section 14 were all about which denominator, and that decision is made when the register map is written.

Port isolation is a floorplan decision as much as a logic one. Section 10's switch_up is one AND gate, but the shared-fate failure it models is usually not a logic bug — it is a shared clock domain, a shared reset, or a shared credit pool. Whether ports are genuinely independent is decided long before this expression is written.

21. Silicon Observability

Three to alarm on.

admitted_bad_err non-zero at all means a flit the port could not validate is inside the switch. Every stage downstream is now operating on something it has no way to check, and the failure will surface somewhere that has no connection to the port that let it in.

peak_occ at capacity while the mean is low is the early warning in this chapter. The switch looks comfortable on every averaged metric and is filling its buffer completely during bursts. It is the state immediately before dropping.

collateral_err non-zero means ports are not independent. One is already down; the alarm is that the others are being affected, which is a different and much worse fault.

22. Debug Lab

22.1 One protocol is slow and the switch is not busy

Symptom. CXL.mem latency is poor. CXL.io and CXL.cache are fine. Switch utilisation is moderate.

The reading. n_stalled per channel, and whether the stalls are on the slow protocol's channel or elsewhere.

The diagnosis. If the stalls are on the memory channel, the memory channel is the bottleneck and the other two are correctly unaffected — which is the switch working. If the stalls are spread across all three, the paths are not actually independent, and section 6's shared-path build is what you have.

The tell that separates them. Per-channel stall counts. A single aggregate stall counter cannot distinguish "one channel is busy" from "one channel being busy stops everything", and those need completely different work.

22.2 Latency is high and every stage looks fine

Symptom. End-to-end latency through the switch is well above budget. No stage reports an error.

The reading. The four per-stage counters, not the total.

Four different investigations, and the total cannot distinguish them. This is the entire reason section 9 attributes time per stage rather than reporting a number.

22.3 A flit was delivered corrupt and the port says the CRC was fine

Symptom. A receiving port reports a corrupt flit. The transmitting port's CRC counters are clean.

The reading. n_bad_forwarded on every switch in the path.

The diagnosis. Cut-through. A switch forwarded the flit before its CRC was known, then discovered it was corrupt — and by then it was somewhere else. The switch that forwarded it is not reporting a link error, because the link was fine; it is reporting that it committed early and lost.

Why the transmitting port looks clean. It is. The corruption happened at a link between two switches, and the switch that received it had already passed it on. Only n_bad_forwarded connects the two ends.

22.4 The switch stopped and one port had failed

Symptom. A single port failed. Every port stopped.

The reading. collateral_err and n_lost on the healthy ports.

The diagnosis. Shared fate. The healthy ports were up, asking, and not served, and collateral_err names exactly that condition.

Where to look, and where not to. Not at the failed port — it failed, and that is a separate investigation. The question is what the failed port was holding that the others needed: a shared credit pool, a shared buffer, a common clock or reset domain, or an arbiter that waits for a port that will never answer. collateral_err says the coupling exists; finding it is a floorplan question more often than a logic one.

23. Design Review

1. Where is a malformed flit stopped, and is that the only place it can be? Everything after the port trusts what the port admitted.

2. Are the three protocols independent paths inside the switch, or one path with three labels? Drive one channel to full and see what the other two do.

3. Can a flit whose destination is free be held by one whose destination is not? That is head-of-line blocking, and it needs its own counter or it is invisible.

4. Does every pipeline stage backpressure, and has each one been tested with the next stage busy? One tied-high ready signal produces a switch that works until it is loaded.

5. Is residency attributed per stage? A total is not actionable. Four numbers that sum to the total are.

6. What does one port failing take with it? Not "is it isolated" — what specifically is shared: credits, buffers, clock, reset, arbitration.

7. Cut-through or store-and-forward, and is the choice counted? If cut-through, n_bad_forwarded is the number that says what it has cost.

8. Can the switch be configured while its data path is congested? That is the moment it needs configuring.

9. What are the ordering domains, and can a pair inside one be reordered by a stall the ordering logic does not see? Section 13's fourth build exists because of that case.

10. Which denominator does each exposed ratio use? Drops per arrival, utilisation per cycle, occupancy per cycle-times-capacity. Get one wrong and the metric silently reports something else.

24. How This Appears In Real Engineering

Protocol independence is an area decision made at the top of the design. Three buffers cost three times one, and the pressure to share is constant. The failure it produces — one protocol's congestion stalling the others — appears only under load and is diagnosed as a general performance problem.

The per-stage counters are the first thing cut and the first thing wanted. They are pure observability, so they lose the area argument; and when a latency problem arrives, the total residency is the only number available and it says nothing.

Backpressure gets tied off during bring-up. A stage that is not yet implemented gets a ready signal tied high, and the tie survives into a build where the stage exists. overrun_err is what catches it, and it only fires in the cycle the decision is made.

Port isolation is claimed and rarely tested. Testing it means deliberately failing a port under load and watching the others, which is a test nobody schedules until after the first incident.

Cut-through is chosen for the latency number and its cost is discovered later, at a receiving port, on a switch two hops away, with no counter connecting the two.

The peak-versus-mean gap catches teams out repeatedly. Every dashboard shows means. The buffer that filled completely for two cycles out of twenty-two does not appear on any of them, and it is the one that dropped.

25. Common Misconceptions

"A CXL switch is a PCIe switch with more protocols." The three protocols must be independent inside the switch, which is a structural requirement PCIe does not have. Section 6's shared-path build is a perfectly good PCIe switch and a broken CXL one.

"The port checks CRC, so the switch is safe." Only if the port's check gates admission. TRUST_INPUT checks the CRC, reports it, and admits the flit anyway — every counter is correct and three malformed flits are inside.

"One queue is simpler and the ordering is stronger." It is also a head-of-line blocking machine. Twelve flits against eight, on the same stream, with the difference entirely in flits whose destinations were free.

"The switch adds ten cycles." It added ten cycles on that flit, seven of them waiting for two specific stages. The number that matters is which stages, because that is what can be changed.

"Redundant ports mean an isolated switch." Only if nothing is shared. The shared-fate build has four independent ports and one shared decision, and it loses all four.

"Cut-through is strictly faster." It is six times faster here and it forwarded a corrupt flit into the fabric. The trade is real; what is not acceptable is taking it without counting what it costs.

"The switch is only 68 percent utilised, so it has headroom." Its buffer hit full capacity during the same run. Utilisation is an average over time and the buffer fills during bursts, which is when flits are dropped.

"Ordering is a switch property." It is a per-domain property. A switch that orders everything is correct and never moves a pair together; one that orders nothing is fast and delivers same-domain flits out of order.

26. Interview Reasoning

Q1. What makes a CXL switch different from a generic packet switch? Three protocols share one link and must become independent paths inside. A stall on CXL.mem is not a reason to stall CXL.io, and a switch that shares one path between them is correct for PCIe and broken for CXL.

Q2. Where must a malformed flit be stopped, and why there? At the port, because everything after it trusts what it admitted. No later stage has the framing, the length or the CRC available to make the decision, so a check anywhere else is a check that cannot be made.

Q3. Why name the three refusal reasons separately? A reserved protocol means the demux has nowhere to send it. A zero length means the framing is wrong. A CRC failure means the link is wrong and the flit may have been fine. Three causes, three responses, and merging them into "dropped" loses all of it.

Q4. Why must every ingress error be gated on the flit being valid? Otherwise the error signals report on whatever the bus happens to be holding between flits, and the counters fill with events that never occurred.

Q5. One protocol's channel is full. What should the other two do? Keep moving. That is the whole reason the paths are separate, and section 6 measures it: one stall against four, on the identical stream.

Q6. What is head-of-line blocking, in a switch specifically? A flit that cannot move holding one whose destination is free. Inside a switch the flit behind it is usually a different protocol, so the head-of-line victim is often a transaction that shares nothing at all with the blockage.

Q7. How do you distinguish head-of-line blocking from ordinary congestion? The held flit's own destination must be free. Without that term the monitor fires on every congested destination and it is disabled within a week.

Q8. Why is per-stage residency more useful than total residency? Because the total is not actionable. Seven of ten cycles in decode and route points at two specific stages; ten cycles points nowhere.

Q9. What must the per-stage counters satisfy? They must sum to the total. A breakdown that does not add up is a breakdown of something else, and the testbench asserts the equality exactly.

Q10. What is an overrun, and when can you observe it? A stage advancing into one that was not ready. Only in the cycle the decision is made — a sample afterwards shows a pipeline that has already moved on, and looks healthy.

Q11. A port fails. What decides whether that is a port outage or a switch outage? Whether the failing port holds anything the others need. Credits, buffers, a clock domain, a reset, or an arbiter that waits for it.

Q12. Why must the collateral-damage monitor exclude the failed port? A failed port not being served is the failure, not collateral damage. The monitor counts ports that were healthy, asking, and refused anyway.

Q13. Four healthy ports ask in one cycle. What must the served counter do? Increment by four. A counter that increments once per cycle regardless reports a fully-loaded switch as a quarter-loaded one.

Q14. Cut-through or store-and-forward? Depends on whether the fabric can afford to carry an error it has already forwarded. Cut-through is six times faster here and put a corrupt flit into the fabric; store-and-forward dropped it before it was ever sent.

Q15. Why does cut-through forward corrupt flits at all? Because the CRC is at the end of the flit and the forwarding decision is at the beginning. It is not a bug in the implementation — it is the definition of the technique.

Q16. A receiving port reports corruption and the transmitting port's CRC counters are clean. Explain. A cut-through switch in the path forwarded before it knew, then discovered the flit was corrupt. n_bad_forwarded on that switch is the only counter that connects the two ends.

Q17. Why must a switch's management path be out of band? Because the moment it is needed is the moment the data path is congested. An in-band management path is unreachable exactly when it matters, and section 12 measures the interval.

Q18. Why gate the unreachable monitor on a management request? An unreachable switch nobody is trying to configure is not blocked. Firing on the condition alone alarms continuously through every congestion event.

Q19. What are the two conditions for applying a configuration change to a switch? It must be reachable and it must be quiesced. Each half is driven alone, because a mutation dropping either leaves the other passing.

Q20. What is an ordering domain? The set of flits that must be delivered in the order they arrived. Here it is per protocol: two CXL.cache flits are ordered relative to each other and neither is ordered relative to a CXL.mem flit.

Q21. What do the two wrong ordering answers look like? Order everything: correct, and it never moves a pair together. Order nothing: fast, and it delivers same-domain flits out of order. The transcript prices both.

Q22. reorder_err could not fire in any of the three builds. What did you do? Added a fourth in which a is stalled by something outside the model while b is let through. That is the realistic way a same-domain pair gets reordered — by a resource the ordering logic does not know about.

Q23. Which is the more useful statistic, throughput or buffer occupancy? Occupancy, because it is predictive. Throughput looks healthy right up to the moment the buffer is full, and then it looks healthy while the switch drops.

Q24. Peak occupancy 8 of 8, mean 22 percent. What does that mean? The buffer filled completely during a burst and was mostly empty otherwise. Only one of those two numbers would have told anyone the switch was in trouble, and it is not the one on the dashboard.

Q25. Three ratios, three denominators. Which and why? Drops per arrival, because drops per departure flatters a switch that drops everything. Utilisation per observed cycle, because departures per arrival is delivery, not utilisation. Occupancy per cycle times capacity, because occupancy per cycle is a count.

Q26. Why did five mutations survive a testbench that checked every ratio? Because they were checked as bounds. chk(x < 25) is satisfied by 0, which is exactly what a missing scale factor produces. Assert the value, not the range.

Q27. Should arrivals be counted when the flit is not admitted? Yes. Otherwise the drop rate has no denominator, and a switch that refuses everything reports a perfect record.

Q28. What is the assembled switch's invariant? Nothing inside it was unvalidated. It cannot fire in a correct build, which is what makes it an invariant, and testing it needs the build without the admission check.

Q29. A flag is set correctly on admission and never cleared on departure. When does that fail? On the second flit. It works perfectly for exactly one, which is why the testbench pushes a malformed flit all the way through the faulty build and then sends a clean one behind it.

Q30. If you could expose one counter from a CXL switch, which? peak_occ alongside the mean. Every other counter here reports something that has already gone wrong; that pair is the one that says a switch is about to.

27. Exercises

1. Add a fourth protocol encoding to port_rx and proto_demux that is valid at the port and has no channel. Decide where it should be caught, and justify the answer from who can make the check.

2. Give hol_block a third queue entry and show that the worst run grows with queue depth. Derive the relationship and say what it implies for buffer sizing.

3. Add a fifth stage to switch_pipe between route and arbitrate. Show that the per-stage attribution still sums to the total, and find the mutation that would break only the sum.

4. Make port_isolation share a credit pool between ports rather than a switch_up signal. Show that collateral_err still fires, and explain why the fault is now much harder to locate.

5. Give cut_through a mode that forwards early but withholds the last unit until the CRC is known. Measure the latency against both existing modes and say what it costs in buffering.

6. Extend order_domain to a domain per protocol and direction. Show which of the existing assertions still hold and which need a second index.

7. Add a histogram of occupancy to switch_stats rather than a peak and a mean. Identify the two mutations in section 18 that a histogram would have made unnecessary.

8. Take the 112-mutation suite and change every exact ratio assertion back to a bound. Confirm the five bound-satisfied mutations return, and find every other check in Module 16 written the same way.

28. Summary

A switch admits at the boundary, separates the three protocols immediately, moves each through stages that can each backpressure, and lets no port's failure become another's.

• Admit: 2 of 5 flits, three refused for three separately named reasons — and the trusting build put three flits inside that no later stage could route, validate or account for.
• Separate: one channel filled. One stall against four, and three of the shared build's stalls were flits that never touch the congested channel.
• Backpressure: 7 of 10 residency cycles in two stages. The total is not actionable; the breakdown is, and the build with one ready signal tied high advances into a stage that cannot take it.
• Isolate: one port of four down. 17 flits served against 16, and three healthy ports stopped by one sick one.

And the number to carry: peak occupancy 8 of 8 with a mean of 22 percent, on a switch reporting 68 percent utilisation. Every averaged metric said comfortable. The buffer filled completely.

112 mutations, 112 killed. Thirty survived the first run — the most in this series — and five of them lived inside assertions written as bounds that the broken values also satisfied. chk(x < 25) is satisfied by 0, and 0 is exactly what a missing scale factor produces. Assert the value, not the range.

16.2 takes the structure as given and asks how the switch decides where a flit goes.