CXL 2.0 Switching

Every chapter of Module 19 assumed something that did not exist in CXL 1.1: a component that lets more than one host reach one device. 19.3 isolated tenants sharing a pool. 19.4 placed hundreds of them across a fleet. Neither is possible on a protocol where a device attaches to exactly one host by a point-to-point link.

CXL 2.0 added the switch. This chapter is what that component is and what it costs.

1. The Engineering Problem — A Switch Is Not Free

Six things separate a working CXL 2.0 fabric from a diagram with a box in the middle.

A switch is a decoder with ports. Routing is address comparison, and an address that matches no port must be dropped rather than sent somewhere plausible. Section 5.

One level, and exactly one. CXL 2.0 permits a single switch between a host and a device — not a fabric, not a cascade. A topology needing two switches is a topology 2.0 cannot express, and it is very easy to draw one by accident. Section 6.

The switch is crossed twice. A request goes out through it and a response comes back through it, and a latency model that counts one crossing reports half the penalty. Section 7.

The upstream port is shared. Several devices hang off one switch and reach the host through a single link, so their aggregate bandwidth is not what arrives. Section 8.

A port belongs to one host at a time, and the mechanism that enforces that is the same fabric manager Module 19 kept invoking. Section 9.

And every error that crosses the switch has to arrive with its origin intact, or the host learns that something downstream failed and nothing about which device. Section 13.

This chapter against 19.4, stated precisely. That one owns policy across a fleet of devices. This one owns the component that makes many-to-one attachment physically possible — and every property here is a way the switch itself can be the defect.

2. The One-Sentence Model

A CXL 2.0 switch is usable when it routes only what matches, sits alone in the path, fits its round-trip latency into the budget, has an upstream link sized for its downstream demand, binds each port to one host, and reports every error with the port it came from — and every defect below is a switch that routes correctly and fails one of the other five.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real switch is a decoder array, a crossbar, per-port buffering, a flow-control block, an error-aggregation path and a fabric-manager interface, and none of that is reproduced. What is reproduced is the arithmetic each of those has to get right.

Each model is built twice — a correct build and a broken build selected by a parameter — and every broken build is a real design, usually one that was correct for a point-to-point link and was carried forward unexamined.

Figure 1 — Every request the host issues crosses the decoder, and the only difference between a correct switch and a misrouting one is what happens to the requests that match nothing. The dashed path is the shortcut a switch built for a single downstream device never had to think about.

5. RTL 1 — Routing Is Address Comparison

// RTL 1 - a switch is a decoder plus a port. Routing decides which port a
// request leaves by, and an address that matches no range must not leave at all.
module switch_routing #(parameter int DEFAULT_PORT = 0) (
  input  logic clk, rst_n,
  input  logic        req,
  input  logic [15:0] addr,
  input  logic [15:0] p0_base, p0_size, p1_base, p1_size,
  output logic        hit0, hit1, any_hit, both_hit,
  output logic [1:0]  out_port,
  output logic [7:0]  n_req, n_dropped,
  output logic        misroute_err, range_conflict_err
);
  assign hit0 = (addr >= p0_base) && (addr < (p0_base + p0_size));
  assign hit1 = (addr >= p1_base) && (addr < (p1_base + p1_size));
  assign any_hit  = hit0 || hit1;
  assign both_hit = hit0 && hit1;
  // Port 2 is the drop. The default-port build sends unmatched requests to port
  // 0, which is how a switch built for one downstream device behaves.
  assign out_port = hit0 ? 2'd0
                  : hit1 ? 2'd1
                  : ((DEFAULT_PORT != 0) ? 2'd0 : 2'd2);
  // A request leaving by a port whose range did not match it.
  assign misroute_err = req && !any_hit && (out_port != 2'd2);
  // Two ports claiming the same address. The priority encoder resolves it, and
  // resolving it silently hides a decoder the fabric manager programmed wrong.
  assign range_conflict_err = req && both_hit;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_req <= 8'd0; n_dropped <= 8'd0;
    end else if (req) begin
      n_req <= n_req + 8'd1;
      if (out_port == 2'd2) n_dropped <= n_dropped + 8'd1;
    end
  end
endmodule

Seven requests. Port 0 owns [0,100), port 1 owns [100,200).

Two drops against none, and two misroutes.

Rows four and five are the port boundary. Address 99 is the last address port 0 owns and address 100 is the first that port 1 owns, and a comparison written inclusive on one end sends every request at the boundary to the wrong device — which, for a device holding a page-aligned region, is the first address of every page.

Row seven is the conflict, and it is the most interesting row in the table. Both port ranges claim address 150, the priority encoder resolves it toward port 0, and the request is served by a device that does have that address configured. Nothing looks wrong. The range_conflict_err output exists because a switch that silently resolves an overlap is a switch hiding a decoder the fabric manager programmed incorrectly — and section 18 shows the priority order was untestable until this case was driven.

Why the broken build is not a strawman. A default port is correct when there is one downstream device: everything goes there, and an address check would only be a way to fail. CXL 1.1 attachments work exactly like that. The behaviour survives into a switch with three downstream ports, where it turns every unmapped address into a request delivered to whichever device happens to be first.

6. RTL 2 — Single Level Means Exactly One

// RTL 2 - single level. CXL 2.0 permits exactly one switch between a host and a
// device, and a topology that needs two is a topology 2.0 cannot express.
module topology_depth #(parameter int ALLOW_CASCADE = 0) (
  input  logic clk, rst_n,
  input  logic       check,
  input  logic [3:0] switch_hops,
  output logic       single_level, legal_2p0,
  output logic [3:0] max_hops,
  output logic [7:0] n_checks, n_illegal,
  output logic       illegal_topology_err
);
  assign max_hops     = 4'd1;
  assign single_level = (switch_hops <= max_hops);
  // The cascading build accepts any depth, which is a CXL 3.0 fabric described
  // in 2.0 terms.
  assign legal_2p0 = (ALLOW_CASCADE != 0) ? 1'b1 : single_level;
  // A topology accepted that a 2.0 fabric cannot route.
  assign illegal_topology_err = check && legal_2p0 && !single_level;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_checks <= 8'd0; n_illegal <= 8'd0;
    end else if (check) begin
      n_checks <= n_checks + 8'd1;
      if (!single_level) n_illegal <= n_illegal + 8'd1;
    end
  end
endmodule

Two illegal topologies accepted.

Zero hops is legal, which matters more than it sounds. A CXL 2.0 fabric is not required to contain a switch; direct attachment remains valid, and a check written as "exactly one" rather than "at most one" rejects every 1.1-style attachment in the fleet.

The constraint is architectural, not a limit somebody picked. A single switch level means the routing decision happens once, which is why CXL 2.0 routing is pure address decoding with no need for a path identifier. Cascading requires knowing not just which port but which switch, and that is port-based routing — a CXL 3.0 mechanism. The two-hop topology is not slow. It does not route.

The consequence worth carrying forward is that every fan-out number in this chapter is a hard ceiling rather than a design point. On a fabric that cascades, running out of ports on one switch is an inconvenience answered by another switch. Here it is answered by a different protocol generation, and that changes who has to be in the room when the topology is sized — a platform architect rather than a purchasing decision.

It also changes the failure mode of getting it wrong. A cascaded topology drawn on a whiteboard and handed to a fabric manager does not produce a slow fabric that somebody optimises later. It produces requests with no route, which surface at bring-up as a device that never enumerates, and the diagnosis is a topology diagram rather than anything measurable in the switch.

7. RTL 3 — The Switch Is Crossed Twice

// RTL 3 - the switch costs latency on every request, in both directions.
module switch_latency #(parameter int IGNORE_RETURN = 0) (
  input  logic clk, rst_n,
  input  logic        sample,
  input  logic [15:0] direct_ns, switch_ns,
  output logic [15:0] total_ns, added_ns, penalty_pct,
  output logic        within_budget,
  input  logic [15:0] budget_ns,
  output logic [7:0]  n_samples, n_over,
  output logic        budget_err
);
  logic [31:0] p_q;
  // A request crosses the switch going out and the response crosses it coming
  // back. Counting one direction halves the number.
  assign added_ns = (IGNORE_RETURN != 0) ? switch_ns : (switch_ns + switch_ns);
  assign total_ns = direct_ns + added_ns;
  assign p_q = (direct_ns == 16'd0) ? 32'd0
                                    : (({16'd0, added_ns} * 32'd100) / {16'd0, direct_ns});
  assign penalty_pct  = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign within_budget = (total_ns <= budget_ns);
  assign budget_err = sample && !within_budget;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_samples <= 8'd0; n_over <= 8'd0;
    end else if (sample) begin
      n_samples <= n_samples + 8'd1;
      if (!within_budget) n_over <= n_over + 8'd1;
    end
  end
endmodule

A 200 ns direct access, a 320 ns budget.

One budget miss against none.

Row two is the entire argument. An 80 ns switch on a 200 ns access is a 360 ns access, and a model counting one crossing reports 280 — inside a 320 ns budget that the real path misses by 40 ns. The error is not small and it is not conservative; it is exactly a factor of two on the term that was added.

Row three is the budget boundary: exactly 320 ns meets a 320 ns budget. Row four is the sanity check that a free switch is a direct attach, which is what makes the model's zero case meaningful rather than a special case.

Why the broken build is not a strawman. Latency budgets are frequently written from a link datasheet, and a datasheet quotes a per-crossing number. Doubling it is a modelling decision somebody has to remember to make, and on a point-to-point link — where there is no intermediate hop — there was never anything to double.

8. RTL 4 — The Upstream Port Is Shared

// RTL 4 - the upstream port is a bottleneck. Several downstream devices share
// one link to the host, and their aggregate demand is not what gets through.
module upstream_bottleneck #(parameter int SUM_DOWNSTREAM = 0) (
  input  logic clk, rst_n,
  input  logic        cycle_en,
  input  logic [15:0] d0_gbps, d1_gbps, d2_gbps, up_gbps,
  output logic [15:0] demand_gbps, delivered_gbps, shortfall_gbps,
  output logic        oversubscribed_link,
  output logic [7:0]  n_cycles, n_short,
  output logic        overclaim_err
);
  assign demand_gbps = d0_gbps + d1_gbps + d2_gbps;
  // The summing build reports what the downstream devices can do. The link
  // reports what actually leaves.
  assign delivered_gbps = (SUM_DOWNSTREAM != 0) ? demand_gbps
                        : ((demand_gbps > up_gbps) ? up_gbps : demand_gbps);
  assign shortfall_gbps = (demand_gbps > up_gbps) ? (demand_gbps - up_gbps) : 16'd0;
  assign oversubscribed_link = (demand_gbps > up_gbps);
  // Reporting more bandwidth than the upstream link can carry.
  assign overclaim_err = cycle_en && (delivered_gbps > up_gbps);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_cycles <= 8'd0; n_short <= 8'd0;
    end else if (cycle_en) begin
      n_cycles <= n_cycles + 8'd1;
      if (oversubscribed_link) n_short <= n_short + 8'd1;
    end
  end
endmodule

Three devices behind one 400 Gbps upstream link.

One overclaim.

The second row is the specification a capacity model quietly makes. Three 200 Gbps devices behind a 400 Gbps upstream port is a system that can deliver 400 Gbps, and a planning spreadsheet that adds the device datasheets says 600. The devices are not lying — each one really can do 200. The link is where the arithmetic stops being additive, and the switch is exactly the component that introduces it.

The third row is the boundary: exactly 400 Gbps of demand is not oversubscription, and the summing model is correct there — which is precisely why the error survives review. It is right whenever nothing is contended.

9. RTL 5 — A Port Belongs To One Host

// RTL 5 - binding. A downstream port belongs to exactly one host at a time, and
// the fabric manager is what decides.
module port_binding #(parameter int NO_UNBIND_CHECK = 0) (
  input  logic clk, rst_n,
  input  logic       bind_req,
  input  logic       port_free,
  input  logic [3:0] current_host, new_host,
  output logic       same_host, may_bind, rebind_needed,
  output logic [7:0] n_binds, n_refused,
  output logic       double_bind_err
);
  assign same_host     = (current_host == new_host);
  assign rebind_needed = !port_free && !same_host;
  // A free port may be bound. A bound port must be released first, unless the
  // request is from the host that already holds it.
  assign may_bind = (NO_UNBIND_CHECK != 0) ? 1'b1 : (port_free || same_host);
  // A port bound to a second host while the first still holds it.
  assign double_bind_err = bind_req && may_bind && !port_free && !same_host;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_binds <= 8'd0; n_refused <= 8'd0;
    end else if (bind_req) begin
      n_binds <= n_binds + 8'd1;
      if (!may_bind) n_refused <= n_refused + 8'd1;
    end
  end
endmodule

One double binding.

Two independent routes to a legal bind. A free port may go to anyone; a bound port may only be re-requested by the host that already holds it, which is the idempotent-retry case that every fabric manager produces after a timeout. A check written as port_free alone refuses those retries and turns a recoverable timeout into a failed rebind.

This is the same invariant 19.4 section 12 enforced at the capacity level, expressed at the port. The two must agree, because a port bound to one host while its capacity is bound to another is a state neither model can describe.

10. Waveform — A Request Crossing The Switch

The final cycle is the one that costs debugging time. The request is routed, it is served, and the data is correct — the device on port 0 really does have address 150 mapped. The conflict signal is the only evidence that the fabric manager programmed two decoders to overlap, and a switch without that output produces a fabric that works until the day port 1's device is the one holding the current copy.

11. RTL 6 — Removing A Port That Is Still Busy

// RTL 6 - a port that goes away. Removing a device while requests are in flight
// leaves those requests with nowhere to land.
module port_removal #(parameter int NO_QUIESCE = 0) (
  input  logic clk, rst_n,
  input  logic       remove_req,
  input  logic [7:0] inflight,
  output logic       quiesced, may_remove,
  output logic [7:0] orphaned, n_removals, n_orphaning,
  output logic       orphan_err
);
  assign quiesced = (inflight == 8'd0);
  // A correct removal waits for the port to drain. The abrupt build removes it
  // whenever asked.
  assign may_remove = (NO_QUIESCE != 0) ? 1'b1 : quiesced;
  assign orphaned = may_remove ? inflight : 8'd0;
  // Requests left with no device to complete against.
  assign orphan_err = remove_req && may_remove && !quiesced;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_removals <= 8'd0; n_orphaning <= 8'd0;
    end else if (remove_req) begin
      n_removals <= n_removals + 8'd1;
      if (may_remove && !quiesced) n_orphaning <= n_orphaning + 8'd1;
    end
  end
endmodule

Two orphaning events against none.

One in flight is as blocking as seven. The quiesce condition is an equality against zero, not a threshold, because a single outstanding read whose response never arrives is a host thread that never returns. The count matters for how long the drain takes and not at all for whether it is required.

An orphaned request is worse than a failed one. A request that receives an error completion is a request the host can handle: the load faults, the driver logs it, the thread sees an exception. A request whose device disappears receives nothing, and the host waits for a completion that no longer has anywhere to come from — which surfaces as a timeout at whatever layer has the longest patience.

12. RTL 7 — The Switch Holds Flits

// RTL 7 - the switch holds flits. Store-and-forward needs buffering, and a
// buffer that cannot be refused fills up.
module switch_buffering #(parameter int NO_BACKPRESSURE = 0) (
  input  logic clk, rst_n,
  input  logic        push, pop,
  input  logic [7:0]  depth,
  output logic [7:0]  occupancy, headroom,
  output logic        full, accept,
  output logic [7:0]  n_pushes, n_dropped,
  output logic        overflow_err
);
  logic [7:0] occ_q;
  assign occupancy = occ_q;
  assign full     = (occ_q >= depth);
  // No floor is needed: occupancy is incremented only while it is below depth,
  // so it can never exceed depth and the subtraction cannot underflow.
  assign headroom = depth - occ_q;
  // A switch with backpressure refuses a push it cannot hold. Without it the
  // push is accepted and the flit is lost.
  assign accept = (NO_BACKPRESSURE != 0) ? push : (push && !full);
  // A flit accepted into a buffer with no room for it.
  assign overflow_err = accept && full;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      occ_q <= 8'd0; n_pushes <= 8'd0; n_dropped <= 8'd0;
    end else begin
      if (push) begin
        n_pushes <= n_pushes + 8'd1;
        if (!accept) n_dropped <= n_dropped + 8'd1;
      end
      if (accept && !pop && (occ_q < depth)) occ_q <= occ_q + 8'd1;
      else if (pop && !accept && (occ_q != 8'd0)) occ_q <= occ_q - 8'd1;
    end
  end
endmodule

A four-deep buffer.

Two refusals against none, and two overflows.

The overflow is silent. The build without backpressure accepts the push, reports it accepted, and the occupancy does not change — because there was nowhere to put it. Nothing errors. The flit is simply gone, and the failure surfaces later as a completion that never arrives, which is section 11's symptom reached by a different route.

The last row is the one a FIFO gets wrong most often: a simultaneous push and pop holds the level rather than doing either twice. The mutation that removes the !accept guard from the decrement passes every test that never drives both at once, which is the standard FIFO bug and the standard FIFO test gap.

Where the buffer comes from is worth being explicit about. A switch is not a wire; it terminates a link on one side and originates one on the other, and the two do not run in lockstep. A downstream device that stalls does not stall the host — it stalls the switch, which absorbs flits until it cannot, and what happens at that point is the only thing separating a congested fabric from a corrupted one.

Backpressure is what converts a capacity problem into a latency problem. With it, a full buffer refuses the push, the upstream side retries, and the cost of congestion is time. Without it, the same congestion is paid in flits, and the host discovers it as a completion that never arrives — the same symptom as section 11's orphan, reached from the opposite direction. Both are silent, and both are indistinguishable at the host from a device that has simply stopped responding.

A buffer sized against average demand is a buffer sized against the wrong number. Section 21's high-water mark exists because occupancy is bursty by construction: several downstream ports can present flits for the same upstream link in the same cycle, and the buffer absorbs exactly that burst. A gauge polled once a second on a switch running at gigahertz rates samples one cycle in a billion.

Figure 3 — Each device really can do 200 Gbps and the datasheets are all correct. The switch is where the arithmetic stops being additive, and the dashed path is every capacity model that adds device numbers without looking at the link they share.

13. RTL 8 — An Error Has To Keep Its Origin

// RTL 8 - an error crossing a switch. Which port it is reported on decides
// whether anybody can find the device that produced it.
module error_routing #(parameter int REPORT_AT_SWITCH = 0) (
  input  logic clk, rst_n,
  input  logic       err_in,
  input  logic [1:0] source_port,     // 0,1,2 downstream; 3 the upstream port
  input  logic [1:0] reported_port,
  output logic [1:0] effective_report,
  output logic       correctly_sourced, actionable,
  output logic [7:0] n_errors, n_actionable,
  output logic       lost_source_err
);
  // The switch-reporting build attributes every downstream error to its own
  // upstream port, which is where the host sees the error arrive.
  assign effective_report = (REPORT_AT_SWITCH != 0) ? 2'd3 : reported_port;
  assign correctly_sourced = (effective_report == source_port);
  // An error correctly attributed to the switch's own upstream port is sourced
  // correctly and names no device, so there is nobody downstream to act on.
  assign actionable = correctly_sourced && (source_port != 2'd3);
  // An error whose reported origin is not where it came from.
  assign lost_source_err = err_in && !correctly_sourced;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_errors <= 8'd0; n_actionable <= 8'd0;
    end else if (err_in) begin
      n_errors <= n_errors + 8'd1;
      if (actionable) n_actionable <= n_actionable + 8'd1;
    end
  end
endmodule

Four errors.

Two actionable against zero.

Row three is the distinction that makes the model honest. An error genuinely originating in the switch's own upstream port and reported there is correctly sourced — and there is still no device to act on, because the switch is the device. actionable and correctly_sourced are different questions, and a model that conflates them either loses real upstream errors or invents downstream ones.

Row four is the correct build getting it wrong, which is deliberate. Attribution is not a property the parameter grants; it is data the switch has to carry, and a switch that carries it incorrectly loses the source just as completely as one that does not carry it at all.

14. RTL 9 — Fan-Out Is The Topology Ceiling

// RTL 9 - fan-out. One switch has a fixed port count, and single-level means
// that count is the ceiling on the topology.
module switch_fanout #(parameter int COUNT_CASCADE = 0) (
  input  logic clk, rst_n,
  input  logic       plan,
  input  logic [7:0] ports_total, hosts_wanted, devices_wanted,
  output logic [7:0] ports_needed, ports_spare,
  output logic       fits_one_switch,
  output logic [7:0] n_plans, n_rejected,
  output logic       overcommit_err
);
  logic [15:0] need_q;
  // Every host and every device consumes one port. A cascade would need a port
  // for the link between switches too, which 2.0 does not permit.
  assign need_q = {8'd0, hosts_wanted} + {8'd0, devices_wanted};
  assign ports_needed = (need_q > 16'd255) ? 8'hFF : need_q[7:0];
  assign ports_spare  = (ports_needed >= ports_total) ? 8'd0 : (ports_total - ports_needed);
  // The cascading build says any plan fits, because it assumes a second switch.
  assign fits_one_switch = (COUNT_CASCADE != 0) ? 1'b1
                                                : (need_q <= {8'd0, ports_total});
  // A plan accepted that one switch cannot carry.
  assign overcommit_err = plan && fits_one_switch && (need_q > {8'd0, ports_total});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_rejected <= 8'd0;
    end else if (plan) begin
      n_plans <= n_plans + 8'd1;
      if (!fits_one_switch) n_rejected <= n_rejected + 8'd1;
    end
  end
endmodule

A sixteen-port switch.

One overcommitment.

This is where the single-level constraint becomes a number. A fleet designer who wants sixteen devices reachable by four hosts needs twenty ports and cannot have them on one switch — and cannot add a second switch either, because that is section 6. The plan does not need a bigger switch; it needs CXL 3.0.

The fourth row is the useful sanity case: a plan with no hosts still consumes device ports, which makes the port count a property of the topology rather than of the workload.

Figure 4 — The right-hand branches are the two that matter. Dropping is visible and counted; resolving a conflict is silent unless the switch is built to say so, which is why section 5 carries a conflict output at all.

15. RTL 10 — The Switch Assembled

// RTL 10 - the switch assembled. Every property that must hold before a switch
// can sit between a host and its memory.
module switch_model #(parameter int ROUTING_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       routes_correctly,   // no request leaves by an unmatched port
  input  logic       single_level,       // at most one switch in the path
  input  logic       latency_budgeted,   // both crossings counted and affordable
  input  logic       upstream_sized,     // the shared link carries the demand
  input  logic       binding_exclusive,  // one host per downstream port
  input  logic       errors_sourced,     // an error names the port it came from
  output logic       usable,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_usable,
  output logic       false_usable_err
);
  assign fail_mask[0] = ~routes_correctly;
  assign fail_mask[1] = ~single_level;
  assign fail_mask[2] = ~latency_budgeted;
  assign fail_mask[3] = ~upstream_sized;
  assign fail_mask[4] = ~binding_exclusive;
  assign fail_mask[5] = ~errors_sourced;
  // The routing-only build checks that requests reach the right device and calls
  // the switch working, which is the definition a bring-up test uses.
  assign usable = (ROUTING_ONLY != 0) ? routes_correctly : (fail_mask == 6'd0);
  assign false_usable_err = evaluate && usable && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_usable <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (usable) n_usable <= n_usable + 8'd1;
    end
  end
endmodule

One usable against four, and three false claims.

The routing-only definition is what a bring-up test measures. Requests reach the right device and the data comes back correct — which is exactly what a switch bring-up is trying to establish, and exactly the state row four describes: routing perfect, binding not exclusive, two hosts on one port and a corruption nobody will reproduce.

16. Quantitative Reasoning

Routing. Seven requests, two matching nothing. The default-port build delivered both to port 0 — a 100% misroute rate on unmapped addresses, which on a fabric mid-reconfiguration is a large fraction of traffic.

Topology. Two of four topologies exceeded the single level, and the cascading check accepted both. CXL 2.0's ceiling is one hop, and zero hops is legal, so the range of legal depths is exactly two values.

Latency. A 200 ns access through an 80 ns switch is 360 ns — an 80% penalty. The one-direction model reports 280 ns and 40%, meeting a 320 ns budget the real path misses by 40 ns. The factor is exactly two on the added term.

Upstream. Three 200 Gbps devices behind a 400 Gbps link: 600 Gbps of demand, 400 delivered, 200 short. The summing model claims 600 — a 50% overclaim against a link that physically cannot carry it.

Binding. Four requests, one double binding. The unchecked build refused nothing, including the one request that would have put two hosts on one port.

Removal. Seven requests in flight, removed abruptly: seven orphans in a single event, each one a completion that will never arrive.

Buffering. A four-deep buffer, two pushes while full: two flits accepted and lost with no error, and an occupancy that never moved.

Error sourcing. Four errors, two actionable under correct reporting and zero under switch reporting — a fabric where every downstream failure looks like the same upstream event.

Fan-out. Sixteen ports. Four hosts and thirteen devices is seventeen ports needed, one over, and the answer is not a bigger switch.

The assembled model. Six properties, five configurations, one usable. The routing-only definition reported four.

Two of those rows compound in the same direction. A latency model that halves the switch term and a bandwidth model that sums downstream datasheets are both optimistic, and a system sized with both is a system whose real access latency is 29% higher and whose real bandwidth is 33% lower than the plan. Neither error is visible in a component test — the switch really does cost 80 ns per crossing and each device really does do 200 Gbps — and both are certain to be discovered under load.

17. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Routing. Both port boundaries are asserted, and the drop port is asserted as a value rather than as an absence.

chk(rH0 == 1'b1, "address 99 is the last of port 0");
chk(rH0 == 1'b0, "address 100 is not port 0");
chk(rH1 == 1'b1, "it is the first of port 1");
chk(rOp == 2'd2, "so the correct switch drops it");

The range conflict asserts the resolution and the flag, because the resolution alone is indistinguishable from correct behaviour.

chk(rBh == 1'b1, "which is a range conflict");
chk(rOp == 2'd0, "resolved toward port 0");
chk(rCe == 1'b1, "and reported rather than hidden");

Topology. Zero hops and one hop are both asserted legal, which is what stops a check written as "exactly one".

Latency. The doubled crossing is asserted as an exact value, and the one-direction model's number is asserted as exactly half of the added term.

chk(lAn == 16'd160, "160 ns added by an 80 ns switch");
chk(hTn == 16'd280, "the one-direction model reports 280");
chk(hWb == 1'b1,    "and says the budget is met");
chk(lBe == 1'b1,    "while the correct model does");

Upstream. Exactly-at-capacity is asserted as not oversubscribed, and the summing model is asserted correct there.

Binding. Both routes to a legal bind are asserted separately.

Removal. One in flight is asserted as blocking as seven.

Buffering. The simultaneous push and pop is asserted to hold the level.

chk(gOc == 8'd3, "a simultaneous push and pop holds the level");

Error sourcing. The genuinely-upstream error is asserted correctly sourced and not actionable, which separates the two questions.

chk(eCs == 1'b1, "an upstream error reported upstream is correctly sourced");
chk(eAc == 1'b0, "but names no device, so there is nobody to act on");

Fan-out. Exactly the port count is asserted to fit, and one over is asserted rejected.

The assembled model. Every fail mask is asserted as an exact six-bit value.

Totals: 219 checks across two testbenches, 115 on the front five models and 104 on the back five, all passing on the unmutated sources.

18. Mutation Testing

Forty-four mutations were injected one at a time.

44 injected, 44 killed, after two survivors were diagnosed. Both diagnoses changed the design rather than the test.

Survivor 1 — an unobservable priority, which turned out to be a missing output. Reversing the priority between hit0 and hit1 survived every case, because the two port ranges were disjoint in every stimulus and the tie-break was therefore never taken. The first instinct is to call the mutation equivalent under a disjointness invariant. But the invariant is not enforced anywhere — a fabric manager can program two decoders to overlap, and the switch as written resolved it silently. The resolution was to add both_hit and range_conflict_err, drive an overlapping configuration, and assert both the resolution and the flag. A survivor that looks equivalent under an invariant is a survivor asking who checks the invariant.

Survivor 2 — an unreachable guard. The floor on headroom never fired, because occupancy is incremented only while it is strictly below depth and can therefore never exceed it. The guard was protecting against an underflow the state machine cannot produce. It was deleted with a comment saying why, and the mutation replaced with one that is not equivalent — reporting occupancy in place of headroom, which the empty-buffer case kills at once.

19. Verification Strategy

What a testbench for a real switch must cover, beyond what these models reach.

Both boundaries of every port range, from both sides. The last address of one port and the first address of the next are adjacent values that must route to different devices, and a decoder with one inclusive comparison sends the boundary of every region to the wrong place.

Overlapping port ranges, deliberately. Section 18's first survivor exists because nothing in the testbench had ever programmed two decoders to claim one address. A misconfiguration the design silently resolves is a misconfiguration the test cannot see.

Simultaneous push and pop on every buffer. The classic FIFO bug is a decrement that fires while an increment also does, and it is invisible to any test that drives one at a time.

The cases that are correct and look like failures. An error correctly attributed to the switch's own upstream port, which names no device. A summing bandwidth model that is right whenever nothing is contended. A free port bound by the host that previously held it. Each will trip a naive checker.

Zero and exactly-at-capacity everywhere. Zero switch hops. Zero switch latency. Exactly the upstream link rate. Exactly the port count. Exactly the buffer depth. Each is legal, each is a boundary, and each is where a comparison written with the wrong operator changes the answer.

What a real switch needs that these models do not have. Concurrency — several ports decoding in the same cycle against a shared crossbar. Ordering — a decoder reprogrammed while requests using the old mapping are in flight. Partial failure — a downstream link that trains down rather than failing outright. Every one of those is where a switch actually breaks.

20. Synthesis and Implementation Reality

The decoder array is the timing path. Sixteen ports means sixteen range comparisons in parallel against a 64-bit address, each needing an adder for the range end, all feeding a priority encoder. This is why real switches register the range ends rather than recomputing them, and why decoder counts per port are a spec-sheet number rather than a configuration.

The crossbar dominates area. An N-port switch is an N-by-N crossbar of flit-width datapaths plus per-port buffering, and both scale quadratically in the wiring. A sixteen-port switch is not four times a four-port switch.

Buffering is a latency choice, not just an area one. Store-and-forward means the whole flit lands before it leaves, which adds the flit's serialisation time to every crossing. Cut-through starts forwarding on the header and cannot be undone if the tail turns out to be bad. Section 7's 50 to 80 ns is mostly this decision, not the decode.

The upstream port is one link and cannot be widened after tapeout. Section 8's bottleneck is not a configuration; it is the width and rate of a physical port, chosen when the switch was designed and fixed for its lifetime.

Single level is why 2.0 routing is cheap. With one switch in the path, the destination is a pure function of the address, and no request carries a path identifier. Multi-level routing needs one, and adding it is not a switch change — it is a protocol generation.

Error aggregation is a separate path with its own fan-in. Every downstream port can raise an error in the same cycle, and the block that collects them has to preserve which port each came from while presenting a single upstream event. That is a small piece of logic with an outsized consequence: it is where section 13's attribution is either kept or thrown away, and the cheapest implementation — a wired-OR of every port's error signal — throws it away by construction.

The fabric-manager interface is not on the data path and constrains it anyway. Rebinding a port means quiescing it, changing decoder state and resuming, and every one of those steps has to be safe against traffic that is already in flight. The interface is slow and infrequent; the interlocks it needs are in the fast path, permanently.

21. Silicon Observability

The high-water mark is the counter that gets left out and pays for itself. An occupancy gauge read by polling shows whatever the buffer held at the moment of the read; a high-water mark shows what it held at the worst moment since the last clear. The failure in section 12 is a transient by definition, and a polled gauge will essentially never catch it.

22. Debug Lab

Symptom. A host attached through a switch reports intermittent data corruption on one memory region. Reads return correct data most of the time and stale data occasionally. There are no errors anywhere: no CRC, no retries, no uncorrectable errors, no dropped-request counts. The switch reports a zero error count on every port.

Step 1 — is it the device? Run the same region through a direct attachment, bypassing the switch. Clean, over hours. The device is not the problem, which points at the path.

Step 2 — is it dropped requests? The no-decoder-match counter is zero. Every request the host issued was routed somewhere. That is a finding rather than a clearance, because "routed somewhere" and "routed to the right place" are different statements and only one of them has a counter.

Step 3 — is it the buffering? The flits-dropped counter is zero and the buffer high-water mark is 6 of 32. Nothing is being lost for lack of room.

Step 4 — read the decoder configuration. Port 0 is programmed for [0, 8 GB) and port 1 for [6 GB, 14 GB). The two overlap by 2 GB, and the corrupted region sits inside the overlap.

The finding. Both decoders claim the address. The priority encoder resolves every request toward port 0, so every read is served by the device on port 0 — correctly, from that device's point of view. The stale data is whatever port 0's device holds, and it is stale because the writes the host believes it made to that region also went to port 0, while some other host bound to port 1 is writing the same addresses on the other device. The region has two backing stores and one of them is invisible.

Why nothing alarmed. Every request was decoded. Every request was routed. Every request was served, by a device that had that address mapped. Section 5's range_conflict_err is the only signal that would have named this, and a switch without it has no way to express "I routed this correctly and the configuration is wrong."

The fix. Correct the decoder programming — the fabric manager computed overlapping ranges. Then add the conflict detection, because the next overlapping configuration will be just as silent.

What made this hard. Every counter that exists read zero, correctly. The failure was a configuration error that the switch's own priority logic converted into consistent, plausible behaviour.

23. Design Review

1. What happens to an address that matches no decoder? If the answer is "goes to port 0", section 5.

2. Can two decoders claim one address, and does anything say so? Section 5, section 18 and section 22 are all this question.

3. Is the latency budget written per crossing or per round trip? Section 7, and the answer is wrong by exactly a factor of two often enough to always ask.

4. What is the upstream link rate against the sum of downstream device rates? Section 8. A ratio above one is fine and unmeasured is not.

5. How many switch hops does the deepest path have? More than one is not slow, it is unroutable. Section 6.

6. Can a downstream port be bound while already bound? And is a re-request by the current holder allowed? Two questions. Section 9.

7. Does a removal wait for the port to drain? Section 11, and the follow-up is what the timeout is when it does not drain.

8. Is there backpressure on every buffer, and is a dropped flit counted? Section 12's failure is silent without the second half.

9. Does an error forwarded upstream carry its source port? Section 13. If not, every downstream failure is the same event.

10. What is the port count against the topology you plan to reach? And if the plan needs more, the answer is a protocol generation, not a bigger part. Section 14.

24. How This Appears In Real Engineering

A platform team evaluating CXL 2.0 memory expansion does the latency arithmetic of section 7 first, because it decides whether the technology is usable for the workload at all. Getting the round trip wrong by a factor of two on the switch term is the difference between a 60% and a 30% latency penalty, and those lead to opposite decisions.

A fabric-manager team owns the decoder programming and is the source of the overlap in section 22. The relevant discipline is that the fabric manager computes ranges and the switch validates them, for the same reason 19.3 argued the device should validate what the host sends: the software changes more often than the hardware.

A switch vendor's bring-up measures exactly what section 15's routing-only model measures — requests reach the right device and the data comes back. That is the right first test and a bad final one, and every property in this chapter is something a bring-up passes with flying colours.

An SRE team owning a CXL-attached fleet discovers that the switch is the component with the least observability and the most influence. A device reports its own errors; a host reports its own latency; the switch sits between them reporting whatever its vendor chose to expose. Section 21 is a shopping list, and the counters worth arguing for before purchase are the ones that cannot be inferred from either end — dropped requests, decoder conflicts, and buffer high-water marks.

A verification team writing a switch testbench finds that the interesting cases are all configuration rather than traffic. Overlapping decoders, a reprogram mid-flight, a removal with outstanding requests, a port bound twice — none of those is a traffic pattern, and a testbench built around traffic generation will not produce any of them without being told to.

A capacity planning team runs into section 8 the first time a rack is built from a spreadsheet. Three devices at 200 Gbps behind a 400 Gbps upstream port is a system somebody sized at 600, and the discovery usually happens under load, in production, on a benchmark that was supposed to be a formality.

25. Common Misconceptions

"A switch just forwards; it does not change anything." It adds two crossings of latency, imposes a shared upstream link, holds flits in buffers that can overflow, and decides whether an error keeps its origin. Sections 7, 8, 12 and 13.

"An unmapped address is harmless." It is delivered to port 0 by a switch built for one downstream device. Section 5.

"Overlapping decoder ranges would cause an error." They cause correct-looking routing and a region with two backing stores. Section 22.

"CXL 2.0 supports switching, so it supports fabrics." One level, which is two legal depths: zero hops and one. Section 6.

"Add another switch if you need more ports." That is a two-hop topology, which 2.0 cannot route. It needs 3.0. Section 14.

"The switch latency is on the datasheet." Per crossing, and there are two. Section 7.

"Three 200 Gbps devices give 600 Gbps." Behind a 400 Gbps upstream port they give 400. Section 8.

"A dropped flit would show up as an error." Not without backpressure and a counter. Section 12.

"Removing a device is a management operation." It is a management operation with a drain, and skipping the drain orphans every request in flight. Section 11.

"If routing works, the switch works." One property of six, and the routing-only definition called four of five configurations usable. Section 15.

26. Interview Reasoning

Q. What does adding a CXL switch cost, in latency terms?

Two crossings, not one — the request goes out through it and the response comes back through it. The number worth stating is the ratio: on a 200 ns access, a 50 ns switch is a 50% penalty and an 80 ns switch is 80%. The follow-up that separates candidates is whether they catch the doubling, because a per-crossing datasheet number invites exactly that error.

Q. Three devices behind one switch, each capable of 200 Gbps. What does the host see?

Whatever the upstream link carries. If it is 400 Gbps, the answer is 400 and not 600, and the shortfall is 200. The general point is that the switch is where bandwidth stops being additive — every model that sums downstream device capability is correct only while the link is uncontended.

Q. A request arrives at a switch with an address no decoder matches. What should happen?

It should be dropped and counted, and the host should see a completion with an error rather than nothing at all. The wrong answer is a default port, which is how a switch built for one downstream device behaves and which turns unmapped addresses into requests delivered to whichever device is first.

Q. Two decoders are programmed to claim one address. What does the switch do?

Whatever its priority encoder does — which will be consistent, plausible, and wrong. That is the trap: the requests route, the data comes back, and the region has two backing stores. The right answer includes a conflict output, because the failure is otherwise unobservable.

Q. Why can CXL 2.0 not cascade switches?

Because with one switch level the destination is a pure function of the address, and routing needs no path identifier. Two levels means the address is no longer sufficient, which is port-based routing and a CXL 3.0 mechanism. The follow-up worth being ready for: this is why a topology needing twenty ports on a sixteen-port switch is not solved by a second switch.

Q. A device is being hot-removed with requests outstanding. What must the switch do?

Wait for the port to drain. One outstanding request is as blocking as a hundred, because an orphaned request receives no completion at all — which is worse than an error completion, since the host has nothing to handle and simply waits.

27. Exercises

1. Extend RTL 1 to eight ports with a proper priority encoder, and add an output naming which two ports conflict rather than only that a conflict exists.

2. Add a decoder-reprogram path to RTL 1 and show that a request decoded under the old mapping and served under the new one is a corruption no counter in section 21 detects.

3. Extend RTL 3 to model store-and-forward against cut-through: add flit serialisation time to the store-and-forward path and find the flit size at which the two crossings dominate.

4. Give RTL 4 per-port arbitration so the 200 Gbps shortfall is distributed rather than taken from whichever device is last. Assert each port's share exactly.

5. Combine RTL 5 with 19.4's rebind sequence and show that port binding and capacity binding must be ordered — and which order is safe.

6. Extend RTL 6 to report the drain time as a function of in-flight count and completion rate, and find the in-flight count at which the drain exceeds a hot-plug timeout.

7. Add a second buffer to RTL 7 and model head-of-line blocking: a flit destined for a full port blocking a flit destined for an empty one. Quantify the throughput loss.

8. Extend RTL 8 so an error carries both the source port and the switch that forwarded it, and show what that buys in a hypothetical two-level fabric.

9. Modify RTL 9 to model a switch whose ports can be configured as upstream or downstream, and find the host-to-device ratio that maximises usable devices on sixteen ports.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the configuration it catches that the current mask calls usable.

28. Summary

CXL 2.0's switch is what made every chapter of Module 19 possible, and it is a component with six ways to be the defect.

Routing is address comparison, and an unmapped address must be dropped. The default-port build delivered both unmapped requests to port 0 — a behaviour that is correct on a device with one downstream port and catastrophic on a switch.

Two decoders claiming one address routes correctly and is wrong. The priority encoder resolves it, the data comes back, and the region has two backing stores. Only an explicit conflict output names it — and section 18 shows the priority order was untestable until that case existed.

Single level means two legal depths, zero hops and one. A two-hop topology is not slow; it does not route, because 2.0 addresses carry no path.

The switch is crossed twice. An 80 ns switch on a 200 ns access is 360 ns and an 80% penalty; the one-direction model reports 280 and meets a budget the real path misses.

The upstream port is where bandwidth stops being additive. Three 200 Gbps devices behind a 400 Gbps link deliver 400, not 600 — a 50% overclaim by any model that sums datasheets.

A port belongs to one host, and a re-request by the current holder is legal — which is the idempotent retry every fabric manager produces after a timeout.

A removal must wait for the drain. Seven in flight, removed abruptly, is seven completions that never arrive — worse than seven errors, because the host has nothing to handle.

A buffer without backpressure loses flits silently. Two pushes while full: two flits accepted, two flits gone, occupancy unmoved, no error.

An error must keep its source port. Four errors, two actionable correctly sourced and zero under switch reporting — and an upstream error is correctly sourced and still not actionable, which are two different questions.

Fan-out is the topology ceiling. Seventeen ports needed on a sixteen-port switch is not answered by a bigger switch or a second one. It is answered by CXL 3.0.

Routing working is not the switch working. One configuration of five was usable; the routing-only definition — the one a bring-up measures — reported four.

20.2 — CXL 2.0 Memory Pooling takes this switch and builds the feature it exists for: capacity that moves between hosts without moving a cable.