CXL 3.0 Scalability

21.1 built a fabric that reaches 240 endpoints. 21.2 used it to remove the host from a transfer. Both assumed every link was a cable inside one rack, and at 240 endpoints that assumption is fine.

This chapter is what happens when it stops being fine. Past the rack, a link is a different medium with a different latency and a different failure rate, the fabric manager becomes a component with a capacity of its own, and the design point stops being the fabric as drawn and becomes the fabric as it actually is — which is always partly broken.

1. The Engineering Problem — Scale Changes The Assumptions, Not The Numbers

A rack-crossing hop is not a rack-internal hop. 400 ns against 80, on a fabric where the topology diagram draws both as one line. Section 5.

The failure domain grows with the fabric unless something partitions it. One fabric-wide event on an unpartitioned fabric is an event for everything on it. Section 6.

The fabric manager has a capacity nobody puts on a diagram. Every endpoint costs it operations, and a fabric large enough to need a manager is large enough to saturate one. Section 7.

What a fabric carries across its own middle is not the sum of its edge links. Summing every link counts bandwidth that never crosses the bisection, and an all-to-all workload lives entirely in the middle. Section 8.

At scale the fabric is always partly broken. Some fraction of links is down at any moment, and the design point is the degraded fabric rather than the drawn one. Section 11.

And distance becomes a placement problem. Not every endpoint is equally far, so a placer that ignores distance pays the far latency on most accesses. Section 13.

This chapter against 21.1, stated precisely. That one owns what a second switch level requires. This one owns what changes when the fabric outgrows the rack it was drawn in — and section 15 shows a fabric that reaches every endpoint it claims and does not scale.

2. The One-Sentence Model

A fabric scales when its endpoints are genuinely reachable, rack-crossing hops cost what they cost, one failure is not the whole fabric, the control plane services the endpoint count, the bisection carries the demand, and the design point is the degraded fabric rather than the drawn one — and every defect below is a fabric that reaches further and fails one of the other five.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real large fabric is a physical plant, a cabling topology, a control plane, a telemetry pipeline and a capacity planner, and none of that is reproduced. What is reproduced is the arithmetic each of them has to get right.

Each model is built twice — a correct build and a broken build selected by a parameter. The broken builds here are all the model that was right in a rack. A uniform link latency, an unpartitioned failure domain, a control plane too fast to notice, an edge-bandwidth figure, a fully healthy fabric. Every one of them is a good approximation at 240 endpoints and a bad one at 2,400.

Figure 1 — The topology diagram draws two lines that look identical and cost five times differently. The dashed path is section 5: a latency model built inside one rack, applied to a fabric that left it.

5. RTL 1 — A Rack-Crossing Hop Is A Different Hop

// RTL 1 - the rack boundary. Inside a rack a link is a cable; past it a link is
// a different medium with a different latency and a different failure rate.
module rack_boundary #(parameter int UNIFORM_LINK = 0) (
  input  logic clk, rst_n,
  input  logic        hop,
  input  logic        crosses_rack,
  input  logic [15:0] intra_ns, inter_ns, budget_ns,
  output logic [15:0] hop_ns,
  output logic        within_budget, boundary_crossed,
  output logic [7:0]  n_hops, n_crossings,
  output logic        undercount_err
);
  assign boundary_crossed = crosses_rack;
  // A uniform model uses the intra-rack figure everywhere, which is the number
  // measured on the bench where every link was a cable.
  assign hop_ns = (UNIFORM_LINK != 0) ? intra_ns
                : (crosses_rack ? inter_ns : intra_ns);
  assign within_budget = (hop_ns <= budget_ns);
  // A rack-crossing hop costed as if it stayed inside the rack.
  assign undercount_err = hop && crosses_rack && (hop_ns == intra_ns)
                          && (inter_ns != intra_ns);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_hops <= 8'd0; n_crossings <= 8'd0;
    end else if (hop) begin
      n_hops <= n_hops + 8'd1;
      if (crosses_rack) n_crossings <= n_crossings + 8'd1;
    end
  end
endmodule

Four hops. 80 ns inside a rack, 400 ns across, a 200 ns budget.

Three crossings, and the uniform model undercounted the two whose media differed.

Five times, not five percent. A cable inside a rack and a link between racks are different physical media, and the difference is not a margin an engineering budget absorbs — it is the difference between meeting a 200 ns budget and missing it by a factor of two.

Row three is why undercount_err is gated on the media differing. A fabric built entirely from one medium — an optical plant where every link is the same regardless of distance — has no boundary penalty, and the uniform model is exactly right there. The error is not "crossing a rack"; it is "costing a crossing as if it were not one".

Why the broken build is not a strawman. The uniform figure is the measured one. It comes from a bench, a rack, a prototype — every environment where the fabric fits in one enclosure — and it is correct in all of them. The model does not become wrong; the fabric moves out from under it.

That is the shape of every broken build in this chapter, and it is worth stating once rather than five times. None of them is a coding error or a bad assumption at the time it was made. A uniform link latency, an unpartitioned failure domain, an unlimited control plane, an edge-bandwidth figure, a fully healthy fabric — each was an accurate description of a fabric that fit in one rack. Scale does not falsify them by revealing a mistake; it falsifies them by changing the system they described.

6. RTL 2 — The Failure Domain Grows With The Fabric

// RTL 2 - the failure domain grows with the fabric. One fabric-wide event is an
// event for everything on the fabric.
module failure_scope #(parameter int FABRIC_WIDE = 0) (
  input  logic clk, rst_n,
  input  logic       fail,
  input  logic [7:0] endpoints_total, endpoints_per_domain,
  input  logic [7:0] domains,
  output logic [7:0] lost, survivors, loss_pct,
  output logic       contained,
  output logic [7:0] n_failures, n_uncontained,
  output logic       fleet_event_err
);
  logic [15:0] l_q;
  // A partitioned fabric loses one domain; an unpartitioned one loses all of it.
  assign lost = (FABRIC_WIDE != 0) ? endpoints_total : endpoints_per_domain;
  assign survivors = (lost >= endpoints_total) ? 8'd0 : (endpoints_total - lost);
  assign l_q = (endpoints_total == 8'd0) ? 16'd0
             : (({8'd0, lost} * 16'd100) / {8'd0, endpoints_total});
  assign loss_pct = (l_q > 16'd255) ? 8'hFF : l_q[7:0];
  assign contained = (domains > 8'd1) && (lost < endpoints_total);
  // A single failure costing more than half the fabric.
  assign fleet_event_err = fail && (loss_pct > 8'd50);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_failures <= 8'd0; n_uncontained <= 8'd0;
    end else if (fail) begin
      n_failures <= n_failures + 8'd1;
      if (!contained) n_uncontained <= n_uncontained + 8'd1;
    end
  end
endmodule

Six failures on a 64-endpoint fabric.

Four uncontained failures in the correct build, and two fleet events against five.

Containment needs more than one domain, whatever the loss was. Row five is the distinction: a single-domain fabric that loses a quarter of itself has not contained anything — it got lucky. The next failure may take all of it, because nothing structural is stopping it. contained requires domains > 1 for exactly that reason, and section 18 records that the term was untestable until a partial failure in a single domain was driven.

Row two is the boundary that decides a fleet-sizing conversation. Exactly half is not a majority, and two domains of 32 is the smallest partition that survives a failure in the sense that matters — half the fabric keeps running.

Row six is the accounting inconsistency: a domain listed with more endpoints than the fabric contains, which drift between an inventory and a topology produces. Survivors floor at zero rather than wrapping to 240.

7. RTL 3 — The Fabric Manager Has A Capacity

// RTL 3 - the fabric manager is one component managing everything, and its own
// capacity is a scaling limit nobody puts on a topology diagram.
module manager_capacity #(parameter int MANAGER_IS_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        plan,
  input  logic [15:0] endpoints, ops_per_endpoint, mgr_ops_per_sec,
  input  logic [15:0] window_sec,
  output logic [15:0] ops_needed, ops_available, shortfall,
  output logic        keeps_up,
  output logic [7:0]  n_plans, n_saturated,
  output logic        overrun_err
);
  logic [31:0] need_q;
  assign need_q = {16'd0, endpoints} * {16'd0, ops_per_endpoint};
  assign ops_needed = (need_q > 32'd65535) ? 16'hFFFF : need_q[15:0];
  assign ops_available = mgr_ops_per_sec * window_sec;
  assign shortfall = (ops_needed > ops_available) ? (ops_needed - ops_available) : 16'd0;
  // The free-manager model assumes the control plane scales with the fabric.
  assign keeps_up = (MANAGER_IS_FREE != 0) ? 1'b1 : (ops_needed <= ops_available);
  // A fabric planned around a manager that cannot service it.
  assign overrun_err = plan && keeps_up && (ops_needed > ops_available);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_saturated <= 8'd0;
    end else if (plan) begin
      n_plans <= n_plans + 8'd1;
      if (ops_needed > ops_available) n_saturated <= n_saturated + 8'd1;
    end
  end
endmodule

Four plans. Four operations per endpoint, 100 operations per second, a 10 second window.

One saturated plan, and the free-manager model overran once.

The manager is the component the topology diagram never shows. Every endpoint costs it configuration operations — decoders to program, bindings to record, telemetry to poll — and those costs are linear in the endpoint count while the manager is a fixed thing. A fabric can be perfectly routable and still be unmanageable, and the failure surfaces as a bring-up that never completes rather than as anything on a link.

Row four is the lever that is usually available and rarely acceptable. Doubling the window from 10 to 20 seconds services the same fabric — but the window is the bring-up time, the reconfiguration time, the recovery time. Making the manager keep up by giving it longer is making the fabric slower to change, and section 12 is what that costs when something has to change urgently.

8. RTL 4 — Bisection, Not Edge Bandwidth

// RTL 4 - bisection. What a fabric can carry across its own middle is what
// decides whether an all-to-all workload runs on it.
module bisection_bandwidth #(parameter int COUNT_EDGE_LINKS = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [15:0] edge_gbps, bisect_links, per_link_gbps,
  input  logic [15:0] demand_gbps,
  output logic [15:0] bisect_gbps, reported_gbps, deficit_gbps,
  output logic        meets_demand,
  output logic [7:0]  n_assess, n_short,
  output logic        overclaim_err
);
  logic [31:0] b_q;
  assign b_q = {16'd0, bisect_links} * {16'd0, per_link_gbps};
  assign bisect_gbps = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  // Summing every edge link counts bandwidth that never crosses the middle.
  assign reported_gbps = (COUNT_EDGE_LINKS != 0) ? edge_gbps : bisect_gbps;
  assign deficit_gbps = (demand_gbps > bisect_gbps) ? (demand_gbps - bisect_gbps) : 16'd0;
  assign meets_demand = (reported_gbps >= demand_gbps);
  // A fabric said to meet a demand its middle cannot carry.
  assign overclaim_err = assess && meets_demand && (demand_gbps > bisect_gbps);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assess <= 8'd0; n_short <= 8'd0;
    end else if (assess) begin
      n_assess <= n_assess + 8'd1;
      if (demand_gbps > bisect_gbps) n_short <= n_short + 8'd1;
    end
  end
endmodule

Four assessments. Eight bisection links of 400 Gbps against 6400 Gbps of edge bandwidth.

One demand exceeded the middle, and the edge-summing model overclaimed it.

6400 against 3200 is a factor of two, and it is the wrong factor of two to be casual about. Edge bandwidth is what every link on the fabric can carry at once; bisection is what can cross from one half to the other. For a workload where every endpoint talks to every other endpoint, all the traffic is bisection traffic, and the edge figure describes a pattern that workload never generates.

Row four is the fix and it is expensive. Doubling the bisection links from eight to sixteen doubles the middle — and those are links that carry no endpoint, connect no device, and exist purely to make the halves reach each other. Bisection bandwidth is bought with hardware that does nothing except be in the middle, which is why it is the first thing cut and the first thing missed.

9. RTL 5 — Anything That Reaches Every Endpoint Scales With Them

// RTL 5 - a fabric-wide operation costs the fabric. Anything that must reach
// every endpoint scales with the endpoint count.
module fabric_wide_op #(parameter int ASSUME_CONSTANT = 0) (
  input  logic clk, rst_n,
  input  logic        issue,
  input  logic [15:0] endpoints, per_endpoint_ns, deadline_ns,
  input  logic        parallel_fanout,
  output logic [15:0] serial_ns, actual_ns,
  output logic        meets_deadline,
  output logic [7:0]  n_ops, n_late,
  output logic        deadline_miss_err
);
  logic [31:0] s_q;
  assign s_q = {16'd0, endpoints} * {16'd0, per_endpoint_ns};
  assign serial_ns = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  // A fanned-out operation costs one endpoint time; a serial one costs the sum.
  assign actual_ns = (ASSUME_CONSTANT != 0) ? per_endpoint_ns
                   : (parallel_fanout ? per_endpoint_ns : serial_ns);
  assign meets_deadline = (actual_ns <= deadline_ns);
  assign deadline_miss_err = issue && !meets_deadline;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_ops <= 8'd0; n_late <= 8'd0;
    end else if (issue) begin
      n_ops <= n_ops + 8'd1;
      if (!meets_deadline) n_late <= n_late + 8'd1;
    end
  end
endmodule

One missed deadline; the constant model saw none.

Fan-out is the whole difference and it is not free. A parallel operation reaches every endpoint at once, which means the issuing point drives 200 requests simultaneously — buffering, credits and a tree of switches all sized for the burst. A serial operation needs none of that and costs 200 times as long.

Row four is the sanity case that makes the model honest: at one endpoint the two costs are identical, so a design validated on a small fabric cannot tell the two implementations apart. The distinction only becomes visible at exactly the scale where getting it wrong is expensive.

10. Waveform — A Fabric-Wide Operation Against Its Deadline

The late row rising at cycle 5 while ser_reach is at 150 is the shape of the failure. The operation is not stuck, not failing and not erroring — it is working correctly and slowly, and 50 endpoints have not been reached when whatever depended on the deadline gives up.

11. RTL 6 — The Fabric Is Always Partly Broken

// RTL 6 - a fabric that is always partly broken. At scale some fraction of
// links is down at any moment, and the design point is the degraded fabric.
module steady_state_degradation #(parameter int ASSUME_ALL_UP = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [15:0] links_total, mtbf_hours, repair_hours,
  input  logic [15:0] links_needed,
  output logic [15:0] links_down, links_up, true_down, true_up,
  output logic        capacity_met,
  output logic [7:0]  n_assess, n_degraded,
  output logic        optimism_err
);
  logic [31:0] d_q;
  // At steady state the expected down count is total times repair over MTBF.
  assign d_q = (mtbf_hours == 16'd0) ? 32'd0
             : (({16'd0, links_total} * {16'd0, repair_hours}) / {16'd0, mtbf_hours});
  // What is really down, and what the model chooses to report.
  assign true_down = (d_q > 32'd65535) ? 16'hFFFF : d_q[15:0];
  assign true_up   = (true_down >= links_total) ? 16'd0 : (links_total - true_down);
  assign links_down = (ASSUME_ALL_UP != 0) ? 16'd0 : true_down;
  assign links_up = (links_down >= links_total) ? 16'd0 : (links_total - links_down);
  assign capacity_met = (links_up >= links_needed);
  // A fabric planned against a link count that is never all present.
  assign optimism_err = assess && capacity_met && (links_needed > true_up);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assess <= 8'd0; n_degraded <= 8'd0;
    end else if (assess) begin
      n_assess <= n_assess + 8'd1;
      if (d_q != 32'd0) n_degraded <= n_degraded + 8'd1;
    end
  end
endmodule

Six assessments. 1000 links, a 10,000 hour MTBF, a 50 hour repair.

Five assessments found links down, and the all-up model was optimistic twice.

Five links of a thousand, always. Not five links that failed once — five at any instant, forever, as the steady state of a plant with a finite MTBF and a non-zero repair time. A fabric sized for 1000 links has 995, and a capacity plan written against 1000 is wrong by a margin it will discover under load.

Row four is the lever and it is operational, not architectural. Quadrupling the repair time from 50 to 200 hours quadruples the steady-state down count. The fabric's usable capacity is a function of how fast a technician gets to it, which is a staffing decision that nobody presents as a bandwidth decision.

Row six is the pathological configuration: a repair time longer than the MTBF, which describes a fabric failing faster than it is being fixed. The estimate exceeds the link count, and the up-count floors at zero rather than wrapping — which section 18 records as a guard the mutation harness had to ask for.

12. RTL 7 — Reconfiguration Scales With What Is Being Reconfigured

// RTL 7 - reconfiguration takes time proportional to what is being reconfigured,
// and a fabric large enough to need it is large enough for that to matter.
module reconfig_time #(parameter int IGNORE_SCALE = 0) (
  input  logic clk, rst_n,
  input  logic        reconfig,
  input  logic [15:0] switches, per_switch_ms, quiesce_ms, sla_ms,
  output logic [15:0] program_ms, total_ms,
  output logic        within_sla,
  output logic [7:0]  n_reconfigs, n_over,
  output logic        sla_miss_err
);
  logic [31:0] p_q;
  assign p_q = {16'd0, switches} * {16'd0, per_switch_ms};
  // The scale-free model costs a reconfiguration as one switch.
  assign program_ms = (IGNORE_SCALE != 0) ? per_switch_ms
                    : ((p_q > 32'd65535) ? 16'hFFFF : p_q[15:0]);
  assign total_ms = quiesce_ms + program_ms;
  assign within_sla = (total_ms <= sla_ms);
  assign sla_miss_err = reconfig && !within_sla;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reconfigs <= 8'd0; n_over <= 8'd0;
    end else if (reconfig) begin
      n_reconfigs <= n_reconfigs + 8'd1;
      if (!within_sla) n_over <= n_over + 8'd1;
    end
  end
endmodule

One SLA miss; the scale-free model saw none.

This is section 7's window from the other side. The fabric manager's throughput determines how long a reconfiguration takes, and a reconfiguration is exactly when the fabric is least able to wait — a failed switch being routed around, a tenant being migrated, a topology being repaired.

Doubling the switch count doubles the reconfiguration time, which means a fabric that grew to fix a capacity problem has made its own recovery slower. Section 15's assembled model treats manager capacity and reach as separate properties for that reason: growing the first without the second produces a fabric that is large and cannot be changed.

Figure 3 — The dashed edge is bandwidth that exists and never crosses the middle. For a workload where every endpoint talks to every other, the only number that matters is the accent block, and it is half what the edge sum reports.

13. RTL 8 — Distance Becomes A Placement Problem

// RTL 8 - locality. At scale not every endpoint is equally far, and a placement
// that ignores distance pays the far latency on most accesses.
module placement_locality #(parameter int IGNORE_DISTANCE = 0) (
  input  logic clk, rst_n,
  input  logic        place,
  input  logic [7:0]  near_share_pct,
  input  logic [15:0] near_ns, far_ns, budget_ns,
  output logic [15:0] blended_ns,
  output logic        meets_budget, is_local,
  output logic [7:0]  n_placements, n_over,
  output logic        budget_err
);
  logic [31:0] b_q;
  logic [7:0] share;
  // A distance-blind placer lands anywhere, which on a large fabric is far.
  assign share = (IGNORE_DISTANCE != 0) ? 8'd0 : near_share_pct;
  assign is_local = (share >= 8'd50);
  assign b_q = (({16'd0, near_ns} * {24'd0, share})
              + ({16'd0, far_ns} * (32'd100 - {24'd0, share}))) / 32'd100;
  assign blended_ns = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  assign meets_budget = (blended_ns <= budget_ns);
  assign budget_err = place && !meets_budget;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_placements <= 8'd0; n_over <= 8'd0;
    end else if (place) begin
      n_placements <= n_placements + 8'd1;
      if (!meets_budget) n_over <= n_over + 8'd1;
    end
  end
endmodule

Four placements. 200 ns near, 600 ns far, a 400 ns budget.

One placement missed the budget; the distance-blind placer missed all four.

Half near is exactly the budget, which makes the locality target a hard number rather than a preference: a placer that achieves 50% locality is exactly at the line, and 40% is over it. On a fabric where far is three times near, the locality share is the single input that decides whether a workload meets its latency target.

The distance-blind placer is not random, it is pessimal. On a large fabric the overwhelming majority of endpoints are far — that is what "large" means — so a placer choosing uniformly lands far essentially always. Ignoring distance is not neutral; it is choosing the worst case, which is why the blind placer misses the budget on every placement including the one where 100% locality was available.

14. RTL 9 — Scale Has A Power Bill

// RTL 9 - what a fabric costs to run. Links, switches and power all scale, and
// the endpoints per unit of cost is what decides whether scale is affordable.
module scale_economics #(parameter int COUNT_ENDPOINTS_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [15:0] endpoints, switches, link_watts, switch_watts,
  input  logic [15:0] power_budget_w,
  output logic [15:0] total_watts, endpoints_per_kw,
  output logic        within_power, affordable,
  output logic [7:0]  n_assess, n_over,
  output logic        budget_err
);
  logic [31:0] w_q, e_q;
  assign w_q = ({16'd0, switches} * {16'd0, switch_watts})
             + ({16'd0, endpoints} * {16'd0, link_watts});
  assign total_watts = (w_q > 32'd65535) ? 16'hFFFF : w_q[15:0];
  assign e_q = (total_watts == 16'd0) ? 32'd0
             : (({16'd0, endpoints} * 32'd1000) / {16'd0, total_watts});
  assign endpoints_per_kw = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  assign within_power = (total_watts <= power_budget_w);
  // Counting endpoints alone makes any fabric affordable.
  assign affordable = (COUNT_ENDPOINTS_ONLY != 0) ? (endpoints != 16'd0)
                                                  : ((endpoints != 16'd0) && within_power);
  assign budget_err = assess && affordable && !within_power;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assess <= 8'd0; n_over <= 8'd0;
    end else if (assess) begin
      n_assess <= n_assess + 8'd1;
      if (!within_power) n_over <= n_over + 8'd1;
    end
  end
endmodule

Four assessments. 150 W per switch, 8 W per link, a 5000 W budget.

One assessment exceeded the power budget.

Endpoints per kilowatt barely moves and the total more than doubles. 53 against 55 per kilowatt — the efficiency is essentially unchanged by scaling, which is exactly why an efficiency figure is the wrong number to plan with. The absolute is what a rack's power feed cares about, and 10,800 W is not a 5000 W budget however good the ratio is.

Row four is the clamp both models agree on: a fabric with no endpoints draws power and delivers nothing, and is not affordable at any budget.

Figure 4 — The right-hand terminal is labelled deliberately. Every one of these failures is a plan that is wrong rather than a fabric that is broken — the hardware reaches what it claims in all six configurations, and five of them still will not carry the workload.

15. RTL 10 — Scale Assembled

// RTL 10 - scale assembled. Everything that must hold before a fabric that works
// in a rack works across a room.
module scale_model #(parameter int ENDPOINT_COUNT_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       reach_achieved,     // the endpoints are genuinely reachable
  input  logic       distance_modelled,  // rack-crossing hops cost what they cost
  input  logic       blast_partitioned,  // one failure is not the whole fabric
  input  logic       manager_keeps_up,   // the control plane services the fabric
  input  logic       bisection_sized,    // the middle carries the demand
  input  logic       degradation_planned,// the design point is the degraded fabric
  output logic       scales,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_scales,
  output logic       false_scale_err
);
  assign fail_mask[0] = ~reach_achieved;
  assign fail_mask[1] = ~distance_modelled;
  assign fail_mask[2] = ~blast_partitioned;
  assign fail_mask[3] = ~manager_keeps_up;
  assign fail_mask[4] = ~bisection_sized;
  assign fail_mask[5] = ~degradation_planned;
  // The endpoint-count build counts what the fabric reaches and calls it scale,
  // which is what a capacity slide shows.
  assign scales = (ENDPOINT_COUNT_ONLY != 0) ? reach_achieved : (fail_mask == 6'd0);
  assign false_scale_err = evaluate && scales && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_scales <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (scales) n_scales <= n_scales + 8'd1;
    end
  end
endmodule

One scaling fabric of six, and four false claims.

The endpoint-count definition is what a capacity slide shows, and it is right about exactly one of the six. Rows four and five are the ones that get built: a fabric reaching every endpoint it claims, with correct distance modelling, a partitioned failure domain and a manager that keeps up — whose middle cannot carry an all-to-all workload, or whose capacity plan assumed every link was up.

16. Quantitative Reasoning

The rack boundary. 80 ns inside, 400 ns across — five times, on a diagram that draws both as one line. The uniform model reported 80 for both and met a budget the real hop missed by a factor of two.

Failure domains. 64 endpoints in four domains loses 16, 25%; unpartitioned it loses all 64. Two domains of 32 is exactly half — survivable. A single domain that loses a quarter has not contained anything, it got lucky.

The manager. 400 endpoints at four operations each is 1600 operations against 1000 available — a 600 shortfall. Doubling the window services it, and doubles every reconfiguration.

Bisection. 6400 Gbps of edge bandwidth, 3200 across the middle. A 4000 Gbps all-to-all demand is met by the edge figure and short by 800 in reality.

Fabric-wide operations. 200 endpoints at 50 ns: 50 ns fanned out, 10,000 ns serially — a 200x difference invisible at one endpoint.

Degradation. 1000 links at a 10,000 hour MTBF and a 50 hour repair: five down, always. Quadruple the repair time and it is twenty. The fabric's capacity is a function of technician response time.

Reconfiguration. 32 switches at 20 ms is 640 ms; 64 switches is 1280 against a 1000 ms SLA. The scale-free model reports 70 ms at every size.

Locality. 200 ns near, 600 ns far: 50% locality is exactly a 400 ns budget, 40% misses it. A distance-blind placer blends to 600 and misses every time.

Power. 240 endpoints and 17 switches is 4470 W; 600 and 40 is 10,800 W against a 5000 W budget — while endpoints per kilowatt moved only from 53 to 55.

The assembled model. Six properties, six configurations, one scales. The endpoint-count definition reported five.

17. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Rack boundary. The equal-media case is asserted as not an undercount, which separates crossing a boundary from mispricing one.

chk(rHn == 16'd80, "with equal media a crossing costs the same");
chk(rUe == 1'b0,   "so nothing is undercounted");

Failure scope. The single-domain partial failure is asserted uncontained, and the accounting drift is asserted to floor.

chk(fCo == 1'b0,  "which is not contained, because there is only one domain");
chk(fSu == 8'd0,  "so survivors floor at zero rather than wrapping");

Manager. The exact capacity boundary is driven.

Bisection. Exactly the bisection is asserted to meet the demand, and the edge model's claim is asserted separately from the middle's capacity.

Fabric-wide operations. The single-endpoint case is asserted identical across all three models.

chk(oSn == 16'd50, "one endpoint costs one endpoint time");
chk(cAn == 16'd50, "and every model agrees");

Degradation. The pathological repair time is asserted to floor rather than wrap.

chk(gTd == 16'd2000, "the estimate exceeds the link count");
chk(gTu == 16'd0,    "so the up-count floors at zero rather than wrapping");

Reconfiguration. The exact SLA boundary is constructed — 47 switches at 20 ms with a 60 ms quiesce is exactly 1000 ms.

Locality. The exactly-half case is asserted to be both local and exactly at budget.

Power. The exact budget is constructed — 16 switches and 325 endpoints is exactly 5000 W.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 223 checks across two testbenches, 110 on the front five models and 113 on the back five, all passing on the unmutated sources.

18. Mutation Testing

Forty-seven mutations were injected one at a time.

47 injected, 47 killed, after three survivors were diagnosed. All three shared a shape worth naming.

Every survivor needed an input the fabric should never produce. The survivor floor required a domain listed with more endpoints than the fabric contains. The up-count floor required a repair time longer than the mean time between failures. Neither is a state a healthy system reaches — and both are states a real one reports, because inventory drifts from topology and a fabric can genuinely fail faster than it is repaired.

The third was different and more interesting. Removing domains > 1 from the containment test survived because every single-domain case in the testbench also lost the entire fabric, so lost < endpoints_total was false for an unrelated reason. A single domain suffering a partial failure separates them — and the case is not exotic at all: a partial failure in an unpartitioned fabric is the most likely thing that will ever happen to one. The stimulus was missing because the obvious single-domain test is the total one.

The generalisation this batch adds to the survivor taxonomy: when two terms of a conjunction are false together in every driven case, neither is tested, and the case that separates them is usually the less dramatic one — a partial failure rather than a total one, a drifted count rather than an absurd one.

19. Verification Strategy

What a testbench for a large fabric must cover.

Inputs the fabric should never produce. All three of section 18's survivors needed one. An inventory that disagrees with a topology, a repair queue longer than the failure rate — these are not hypothetical inputs, they are what a real telemetry pipeline delivers, and every unfloored subtraction in the model is waiting for them.

Conjunctions where both terms fail together in the obvious case. Containment is "more than one domain and not everything lost", and the obvious single-domain test fails both. Find the case that separates them, and expect it to be less dramatic than the one you have.

Every threshold at exactly its value, constructed where necessary. Three of this chapter's boundaries had to be solved for: 47 switches for exactly a 1000 ms SLA, 16 switches and 325 endpoints for exactly 5000 W, 250 endpoints for exactly the manager's capacity.

The cases that are correct and look like failures. A fabric of one medium where a crossing costs nothing extra. An empty fabric that loses nothing. Exactly half a fabric lost, which is not a majority. A single endpoint where serial and parallel agree.

Each mask bit driven false alone. Six properties, six single-bit configurations plus the all-clear.

What a real fabric needs that these models do not have. Correlation — links that fail together because they share a chassis or a power feed, which makes the independent-failure model in section 11 optimistic. Repair queueing — a repair time that grows when many links are down at once. Partial reconfiguration — reprogramming a subset of switches, which is what section 12's model would have to become to be usable.

20. Synthesis and Implementation Reality

The inter-rack link is a different component, not a longer cable. Optics, retimers, a connector standard and a failure mode that copper does not have. Section 5's 400 ns is mostly serialisation and retiming, and it is not reducible by shortening the cable because it is not distance-dominated.

Failure-domain partitioning costs bisection bandwidth. Section 6's four domains are four groups that must not depend on each other — which means links between domains cannot be counted on for containment, and section 8's middle gets thinner as partitioning gets stronger. The two properties trade directly, and a fabric optimised for either alone gets the other wrong.

The fabric manager is usually one process on one machine. Section 7's capacity is a real throughput figure with a real bottleneck — typically the per-endpoint round trip rather than any computation — and scaling it means sharding the fabric across managers, which introduces exactly the consistency problem 20.2 §14 described between one manager and one device.

Bisection links carry no endpoints. They are switch-to-switch links whose entire purpose is to let the halves reach each other, and every one is a port not spent on a device. Section 8's fix — doubling the middle — is bought by halving what the same switches could otherwise attach.

Steady-state degradation is a staffing model in a bandwidth budget. Section 11's repair time is how long a technician takes to reach a rack, which makes the fabric's usable capacity a function of an operations roster. That is an uncomfortable dependency to write into a capacity plan and an accurate one.

21. Silicon Observability

"Links down, sampled continuously" is the one that gets built wrong. A counter of link-down events is not the same measurement as the fraction down at any instant, and section 11's model needs the second. A fabric with five links always down and a fabric with fifty brief outages a day can report identical event counts, and only one of them has lost capacity.

22. Debug Lab

Symptom. A CXL fabric is expanded from one rack to three. Endpoint count triples, as planned. A distributed training workload that ran at 90% of its target on the single-rack fabric now runs at 40%. No errors, no link failures beyond baseline, and per-hop latency measurements match the model.

Step 1 — is it the rack boundary? Read hops by class and latency per class. Inter-rack hops cost 400 ns and intra-rack 80, exactly as modelled. The mix is 30% inter-rack, which the model also predicted. Section 5 is accounted for.

Step 2 — is it the manager? Fabric-manager operations per second sit at 40% of ceiling. Reconfigurations complete in 700 ms against a 1000 ms SLA. Sections 7 and 12 are fine.

Step 3 — is the fabric degraded? Links down: 14 of 2,800, against a modelled steady state of 14. Section 11 is behaving exactly as designed.

Step 4 — then what is different? Read bisection utilisation. 97%. Read edge utilisation: 31%. The fabric has plenty of aggregate bandwidth and its middle is saturated.

Step 5 — why did the single-rack fabric not show this? On one rack there was no meaningful bisection — every endpoint was one hop from every other through one switch, so all-to-all traffic never crossed a constrained middle. Tripling the fabric created a bisection that did not previously exist, and the workload is all-to-all by nature.

The finding. Not a regression. The single-rack fabric had no bisection constraint to violate, and the expansion introduced one that the capacity plan — written from edge bandwidth — did not model. Section 8's argument, discovered by a training run.

The fix, in order. Add bisection links, which costs ports that would otherwise attach devices — the trade in section 20. Then, for placement: co-locate each training job's ranks within a rack where possible, which is section 13's locality share applied to a workload rather than to a device. Then fix the capacity model, because the next expansion will make the same prediction.

What made this hard. Every measurement matched its model. The hops cost what they should, the manager was fine, the degradation was as designed — and the one constrained resource was the one the single-rack fabric had never had, so no baseline existed to compare against.

23. Design Review

1. What fraction of hops cross a rack, and what does a crossing cost? Two numbers, and a latency model needs both. Section 5.

2. How many failure domains, and does containment survive a partial failure? More than one domain is the requirement; a lucky partial failure is not containment. Section 6.

3. What is the fabric manager's operations-per-second ceiling, against the endpoint count? Section 7, and nobody has this number until it is asked for.

4. What is the bisection bandwidth, separately from the edge bandwidth? Section 8, and section 22 is what happens when only the second is planned.

5. Is every fabric-wide operation fanned out? A serial one costs the endpoint count. Section 9.

6. What is the steady-state link-down count, and is it sampled or event-counted? Section 11 needs the fraction, not the events. Section 21.

7. How long is a full reconfiguration at the current switch count, and at twice it? Section 12, and the second number is the one that matters.

8. What locality share does the placer achieve, and what does the budget need? Section 13, where half is exactly the line.

9. What is the fabric's absolute power draw, not its endpoints per kilowatt? The ratio barely moves; the total more than doubles. Section 14.

10. Which of the six properties does the team believe "it scales" means? Section 15 exists because the answer is the endpoint count.

24. How This Appears In Real Engineering

A capacity planning team expanding a fabric does section 8 too late, as section 22 shows. The discipline worth adopting is to model bisection before the expansion, because a single-rack fabric has no bisection constraint and therefore provides no baseline that would reveal one.

A platform architect owns the trade in section 20 between failure-domain partitioning and bisection bandwidth, and it is genuinely a trade — stronger containment means fewer links that can be relied on to cross the middle. There is no configuration that maximises both, and a review that treats them as independent will produce a fabric that is bad at each.

An operations team discovers section 11 as a capacity problem rather than a reliability one. Five links always down is not an outage anybody pages for; it is a permanent 0.5% reduction in a fabric that was sized at 100%. The repair-time distribution is the lever, and it is owned by whoever staffs the data centre floor.

A workload team running all-to-all traffic meets section 13 and section 22 together. Locality is the input that decides the latency, and on a multi-rack fabric it is achieved by placing a job's ranks in one rack — which is a scheduler property, not a fabric one, and lives in a completely different team's system.

25. Common Misconceptions

"A link is a link." 80 ns inside a rack, 400 ns across. Five times, on a diagram that draws them identically. Section 5.

"The fabric is partitioned because it has four domains." Containment needs more than one domain and a failure that does not take everything. A single domain losing a quarter got lucky. Section 6.

"The fabric manager is not on the data path, so it does not scale." It has an operations-per-second ceiling and every endpoint costs it operations. Section 7.

"We have 6400 Gbps of bandwidth." Across the middle, 3200. An all-to-all workload lives entirely in the middle. Section 8.

"It is a fabric-wide operation, so it is fast." Only if it fans out. Serially it costs the endpoint count — 200x. Section 9.

"The fabric has 1000 links." It has 995, always, and 980 if repairs are slow. Section 11.

"Reconfiguration takes about a second." At 32 switches. At 64 it takes 1330 ms and misses the SLA. Section 12.

"The placer will find something." On a large fabric almost everything is far, so a distance-blind placer chooses the worst case essentially always. Section 13.

"Endpoints per kilowatt barely changed, so power is fine." The ratio went 53 to 55 and the total went 4470 W to 10,800. Section 14.

"It reaches 2400 endpoints, so it scales." One property of six. Section 15.

26. Interview Reasoning

Q. Your fabric grows from one rack to three. What changes?

The link medium for a third of the hops — 400 ns instead of 80 — and a bisection constraint that did not previously exist. The second is the one that surprises people: on one rack, every endpoint is one hop from every other, so there is no constrained middle to saturate. Tripling the fabric creates the constraint rather than tightening it.

Q. What is bisection bandwidth and why does it differ from what a datasheet totals?

It is what can cross from one half of the fabric to the other, and a datasheet sums every link — including the ones that never cross the middle. For an all-to-all workload every byte is bisection traffic, so the edge figure describes a pattern the workload never generates. The follow-up worth reaching: bisection links carry no endpoints, so buying more of them costs device ports.

Q. How many links does a 1000-link fabric have up?

995, at a 10,000 hour MTBF with a 50 hour repair — and that is a steady state, not an incident. The interesting follow-up is which lever moves it: not the MTBF, which is a component property, but the repair time, which is a staffing decision. Quadruple it and twenty links are down.

Q. A fabric has four failure domains. Is a failure contained?

Only if it also does not take everything. And the sharper case: a single-domain fabric that loses a quarter of itself has not contained anything — nothing structural stopped the loss, and the next one may take all of it. Containment is a property of the partitioning, not of the outcome.

Q. Why is the fabric manager a scaling limit?

Because every endpoint costs it configuration operations and it is a fixed-throughput component. A fabric can be perfectly routable and unmanageable, and the failure looks like a bring-up that never finishes. The follow-up: the obvious fix — a longer window — makes every reconfiguration slower, which is exactly what a fabric needs to be fast at when something breaks.

Q. Your placer ignores distance on a large fabric. How bad is that?

Pessimal, not neutral. On a large fabric most endpoints are far by definition, so choosing uniformly means choosing far essentially always — 600 ns against 200, missing a 400 ns budget on every placement. Locality is the single input that decides whether the workload meets its target, and half is exactly the line.

27. Exercises

1. Extend RTL 1 to three hop classes — intra-rack, intra-row and inter-row — and find the mix at which a 200 ns budget becomes unachievable.

2. Add correlated failures to RTL 6: links sharing a chassis that fail together. Show that the independent model understates the down count.

3. Make RTL 6's repair time a function of the number of links currently down, and find the failure rate at which the queue diverges.

4. Combine RTL 2 and RTL 4 and quantify the trade in section 20: bisection bandwidth as a function of failure-domain count.

5. Shard RTL 3 across two managers and show that the consistency problem of 20.2 §14 reappears between them.

6. Give RTL 7 a partial-reconfiguration path that reprograms only the switches on a changed route, and find the fraction of switches a typical repair touches.

7. Extend RTL 8 to a distance distribution rather than a near/far split, and derive the locality share needed for a given budget.

8. Model section 22 end to end: a single-rack fabric with no bisection constraint, expanded to three racks, and show the constraint appearing where no baseline existed.

9. Add a cooling term to RTL 9 and show that a power budget met at the fabric level can be exceeded per rack.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the fabric it catches that the current mask calls scaling.

28. Summary

21.1 and 21.2 both assumed one rack. Everything here is what that assumption was holding up.

A rack-crossing hop costs five times an internal one — 400 ns against 80 — on a topology diagram that draws both as one line, and the uniform model met a budget the real hop missed by a factor of two.

Containment needs more than one domain, whatever the loss was. A single-domain fabric that lost a quarter of itself got lucky, and section 18 records that the term was untestable until a partial failure was driven — because every obvious single-domain test loses everything.

The fabric manager has a ceiling nobody draws. 400 endpoints need 1600 operations against 1000 available, and the obvious fix — a longer window — makes every reconfiguration slower.

Bisection is not edge bandwidth. 6400 Gbps summed, 3200 across the middle, and an all-to-all workload lives entirely in the middle — which is section 22's training run, discovered after an expansion that created the constraint rather than tightening it.

A fabric-wide operation costs the endpoint count unless it fans out. 50 ns against 10,000, a 200x difference that is invisible at one endpoint.

The fabric is always partly broken. Five links of a thousand, always — twenty if repairs take four times as long, which makes usable capacity a function of an operations roster.

Reconfiguration doubles when the fabric doubles. 640 ms at 32 switches, 1280 at 64, against a 1000 ms SLA — so a fabric that grew to fix capacity made its own recovery slower.

A distance-blind placer is pessimal, not neutral. On a large fabric almost everything is far, so it blends to 600 ns and misses a 400 ns budget every time — while 50% locality is exactly the line.

Efficiency hides the absolute. Endpoints per kilowatt moved 53 to 55 while the total went 4470 W to 10,800 against a 5000 W budget.

Every mutation survivor needed an input the fabric should never produce — a domain larger than the fabric, a repair slower than the failures — and both are what a real telemetry pipeline delivers.

Reaching the endpoints is one property of six. The definition a capacity slide shows called five of six fabrics scaling when one did.

21.4 — The CXL 3.x Future Vision closes the module by asking what all of this is for: composable fabrics where memory, accelerators and hosts are assembled per workload rather than per machine.