Fabric-Architecture Question

27.6 chose a part. This chapter connects a room full of them — design a CXL fabric — and it is the question where a whiteboard rewards the wrong instinct, because a fabric drawn as boxes and lines looks finished long before it is.

The switches are connected. That is true, it is necessary, and it is what most answers stop at — a topology in which everything reaches everything, with no port counted, no hop charged, no cut measured and nobody named who will configure it.

1. The Engineering Problem — A Picture Is Not A Fabric

A fabric is ports. Thirty-two endpoints against two sixteen-port switches is not thirty-two ports, it is thirty — the switches spend two joining each other — and a design that does not know this is two ports short before anything else is decided. Section 5.

Every hop costs latency. Two hops of a hundred nanoseconds on a two-hundred-and-fifty-nanosecond base is four hundred and fifty against a three-hundred budget — a hundred and fifty percent — on a technology whose entire argument is load-to-use latency. Section 6.

The aggregate is not the bisection. Eight hundred gigabits per second wanting to cross a cut of two links carries two hundred and fifty-six and starves five hundred and forty-four, on a fabric whose total bandwidth is two thousand and forty-eight. Section 7.

Somebody has to attach the devices. Sixty-four bindings against a manager that can do forty and an operations team that can do ten leaves fourteen owned by nobody — and a fabric with unowned work is a fabric that does not come up. Section 9.

A switch failure is a memory failure. Eight hosts of thirty-two behind one switch is a quarter of the deployment and five hundred and twelve gigabytes, and a second path drawn on the diagram does not make a second copy of the memory. Section 10.

This chapter against 27.5, stated precisely. That one owns what a pool of memory means and what it costs — assignment, reassignment timing, oversubscription, isolation. This one owns the thing in the middle: how many switches, how deep, how wide, who runs it and what it costs when one of them fails. 26.5 owns what goes wrong with a fabric once it is built; this chapter is the design review that happens before it is.

2. The One-Sentence Model

A fabric design is sound when a topology has actually been named, when its ports have been counted including the ones the switches spend on each other, when the latency of the deepest path has been charged against a budget, when the bandwidth across the fabric's narrowest cut has been provisioned for the traffic that must cross it, when somebody owns the attach operations, and when the blast radius of a single switch has been bounded — and "the switches are connected" is one of those six.

3. What This Chapter Owns

Some vocabulary first, because a fabric question is usually asked in words that carry more than one meaning.

A switch is a device with ports that routes CXL traffic between them. It is not a PCIe switch with a different label: it carries CXL.io, CXL.mem and — in a fabric that supports them — CXL.cache flows, it participates in the address decoding that routes a host's request to the right device, and it is configured by a fabric manager rather than by the host's enumeration alone.

A port is a link. One physical connection at some width and speed, terminating at a host, a device, or another switch. This is the unit the whole of section 5 is about, and the reason the arithmetic surprises people is that the third case — a port spent on another switch — carries no endpoint and appears on no bill of materials.

A hop is a switch in the path. A host reaching a device through one switch takes one hop; through two switches, two. Each is a store-and-forward element with its own latency, and section 6 is about what that does to a load-to-use number.

The bisection is the narrowest cut through the fabric. Draw a line dividing the fabric in half such that the traffic crossing it is maximal and the bandwidth crossing it is minimal, and the bandwidth of the links the line crosses is the bisection. It is the number that decides whether traffic can actually get from one side to the other, and section 7 is about why the fabric's total bandwidth is not it.

A fabric manager is the entity that binds devices to hosts. It is software, it runs somewhere, it talks to the switches over a management interface, and it is the piece a whiteboard answer forgets. Section 9.

And a failure domain is everything that goes away together. For a fabric, the natural unit is a switch: everything behind it is unreachable when it fails. Section 10.

4. Teaching-Model Boundary

Every model in this chapter is a teaching model, not a fabric planner. It computes the one relationship the section is about and nothing else. There is no switch, no routing table and no topology generator anywhere in this file.

Each model is built twice from one source. A parameter selects between the measured build, which counts what the fabric actually costs, and the drawn build, which reports what the picture shows. Every section's headline number is the gap between them.

5. RTL 1 — A Fabric Is Ports Before It Is Anything Else

Start with the arithmetic, because it is the constraint that binds first and the one a drawing hides most completely.

Every endpoint consumes a switch port. Every host link, every device link. That much is obvious and everybody counts it. What a drawing hides is the third kind of port: the one a switch spends connecting to another switch, which carries no endpoint, appears in no inventory, and is consumed at both ends of every link between switches.

A chain of N switches therefore spends 2(N−1) ports on being a fabric at all, before a single host or device is attached. That is the difference between "we need thirty-two ports, so two sixteen-port switches" and a design that is two ports short.

// RTL 1 - a fabric is ports. The first constraint on a topology is not
// bandwidth or latency, it is whether the switches have enough ports for
// everything that has to plug into them, and "add a switch" adds ports at
// the cost of consuming some of them for the link between switches.
module port_arithmetic #(parameter int A_SWITCH_ADDS_PORTS = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] hosts, devices, ports_per_switch, switch_count,
  output logic [15:0] ports_used, ports_avail, ports_short, fabric_pct,
  output logic        fits,
  output logic [7:0]  n_evals, n_short,
  output logic        capacity_err
);
  logic [31:0] raw_avail, links_between, f_q, net_avail, sel_avail;
  logic [15:0] true_avail, true_short;
  logic        truly_short;
  // Every host link and every device link consumes one switch port.
  assign ports_used = hosts + devices;
  // Switches in a fabric are connected to each other, and each connection
  // consumes a port at both ends. A chain of N switches spends 2*(N-1).
  assign links_between = (switch_count > 16'd1)
                       ? (32'd2 * ({16'd0, switch_count} - 32'd1)) : 32'd0;
  assign raw_avail = ({16'd0, ports_per_switch} * {16'd0, switch_count});
  assign net_avail = (raw_avail > links_between) ? (raw_avail - links_between) : 32'd0;
  assign sel_avail = (A_SWITCH_ADDS_PORTS != 0) ? raw_avail : net_avail;
  assign ports_avail = (sel_avail > 32'd999) ? 16'd999 : sel_avail[15:0];
  // The ports a fabric really has are the ones left after the switches are
  // connected to each other; the reported figure is whichever view is built.
  assign true_avail = (net_avail > 32'd999) ? 16'd999 : net_avail[15:0];
  assign true_short = (ports_used > true_avail) ? (ports_used - true_avail) : 16'd0;
  assign ports_short = (ports_used > ports_avail) ? (ports_used - ports_avail) : 16'd0;
  assign f_q = (ports_avail == 16'd0) ? 32'd0
             : (({16'd0, ports_used} * 32'd100) / {16'd0, ports_avail});
  assign fabric_pct = (f_q > 32'd100) ? 16'd100 : f_q[15:0];
  assign fits = (ports_short == 16'd0) && (ports_used != 16'd0);
  assign truly_short = (true_short != 16'd0) && (switch_count != 16'd0);
  assign capacity_err = evaluate && truly_short && fits;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_short <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_short) n_short <= n_short + 8'd1;
    end
  end
endmodule

Thirty-two endpoints against two sixteen-port switches is thirty usable ports and a shortage of two — and the fabric is reported a hundred percent full, which it is, because it is over.

Figure 1 — two ports, and why they are invisible. The upper path multiplies the switch count by the ports per switch, which is the calculation anybody does in their head and is right for exactly one topology: a single switch. The lower path subtracts what the fabric costs itself. The gap is two ports on a small fabric and grows with every switch added, which is why the error scales in the direction of the designs that can least afford it.

The sixth case makes the point about scale concretely. Four switches in a chain spend six ports, not two, and the drawn view reports the fabric as seventy-eight percent used while the counted view says eighty-six. Neither number is alarming; the difference is a switch's worth of ports on a fabric of four, and a design sized from the first will be short on the day it is populated.

The second case is the one where the two views cannot disagree and is worth stating so the rule is not over-applied. A single switch spends nothing on being a fabric — there is nothing to connect to — and every calculation anybody does in their head is correct. This is why the error survives every prototype: the first fabric anybody builds has one switch in it.

The degenerate cases bound the model at both ends. A fabric with switches and nothing plugged into it is not a fabric that fits, because there is nothing for it to fit; the model declines to call it sound and declines to call it short. And endpoints with no switches at all report a shortage of the full thirty-two in both views, but the model does not call it a fault: a design with no switches is an absent fabric, not an under-provisioned one, and reporting those as the same thing loses the distinction that decides what to do next.

The clamp is defensive rather than descriptive and says so. Nine hundred endpoints against a saturated port count is not a shortage — the model bounds the figure it publishes and declines to invent a fault out of its own ceiling.

There is a topology question hiding behind the arithmetic that is worth making explicit, because the model computes a chain and a chain is not the only shape. A chain of N switches spends 2(N−1) ports; a fully-connected mesh of N switches spends N(N−1) — one port at each end of every pair — which at four switches is twelve ports rather than six and at eight is fifty-six rather than fourteen. A mesh buys hop depth: every switch is one hop from every other, which is section 6's currency. The two shapes trade ports against hops, and the reason section 13's model uses the chain's arithmetic is that it is the shape most designs reach for first and the one whose cost is least visible. A design that has chosen a mesh has chosen to spend ports to buy latency, deliberately, which is an answer rather than an oversight.

6. RTL 2 — Every Hop Costs Latency

The second constraint, and the one that decides whether the fabric should exist.

CXL's argument is load-to-use latency. A memory access that goes through the fabric is worth making only because it is faster than the alternative, and every switch in the path is a store-and-forward element between the host and the data. A hop is not free, it is not cheap, and two of them can be the difference between memory and a slow device.

The number that matters is the deepest path, not the average one. A fabric where most accesses take one hop and some take three is a fabric whose tail latency is three hops, and the tail is what a memory-bound workload feels.

// RTL 2 - every hop costs latency. A fabric is drawn as a picture in which
// switches are free, and each one is a store-and-forward element in the
// middle of a load-to-use path that the whole point of CXL was to keep
// short.
module hop_latency #(parameter int HOPS_ARE_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] hop_count, hop_ns, base_ns, budget_ns,
  output logic [15:0] total_ns, over_ns, budget_pct, hops_charged,
  output logic        within_budget,
  output logic [7:0]  n_evals, n_over,
  output logic        latency_err
);
  logic [31:0] added, raw_total, b_q;
  logic [15:0] true_total, true_over;
  logic        truly_over;
  // A fabric with more switches than the design admits to is still charged
  // for them; the clamp is on what one path can plausibly traverse.
  assign hops_charged = (hop_count > 16'd16) ? 16'd16 : hop_count;
  assign added = {16'd0, hops_charged} * {16'd0, hop_ns};
  assign raw_total = {16'd0, base_ns} + added;
  assign true_total = (raw_total > 32'd9999) ? 16'd9999 : raw_total[15:0];
  assign total_ns = (HOPS_ARE_FREE != 0) ? base_ns : true_total;
  assign true_over = (true_total > budget_ns) ? (true_total - budget_ns) : 16'd0;
  assign over_ns = (HOPS_ARE_FREE != 0) ? 16'd0 : true_over;
  assign b_q = (budget_ns == 16'd0) ? 32'd0
             : (({16'd0, total_ns} * 32'd100) / {16'd0, budget_ns});
  assign budget_pct = (b_q > 32'd999) ? 16'd999 : b_q[15:0];
  assign within_budget = (over_ns == 16'd0) && (total_ns != 16'd0);
  assign truly_over = (true_over != 16'd0) && (budget_ns != 16'd0);
  assign latency_err = evaluate && truly_over && within_budget;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_over <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_over) n_over <= n_over + 8'd1;
    end
  end
endmodule

Two hops at a hundred nanoseconds each on a base of two hundred and fifty is four hundred and fifty nanoseconds against a three-hundred-nanosecond budget — a hundred and fifty percent of it — and the hops-are-free view reports eighty-three percent and a fabric inside its budget.

It is worth being concrete about where a hop's latency comes from, because "a switch adds latency" is a sentence that can be repeated without content. A switch receives a flit, checks it, decides where it goes, and sends it — and at minimum that is the serialisation of the flit at the port's rate plus the switch's internal pipeline plus the arbitration delay if the output port is busy. The first two are fixed and are what a datasheet quotes. The third is load-dependent and is the one that makes a tail, which is why a hop's cost under contention is not the hop's cost on an empty fabric, and why section 8's blocking is a latency problem as much as a bandwidth one.

The sixth case is worth sitting with because it is the only configuration in which the drawn view is right. Hops that genuinely cost nothing are free, and the model agrees: three hops at zero nanoseconds each add nothing. That is not a real switch, but it is the assumption a drawing makes, and naming the assumption is what makes it arguable.

The fifth case is the comparison the whole section exists to enable. A direct attach takes no hops and pays the base latency alone, which is the number a fabric design has to beat. If the fabric's deepest path is not meaningfully better than the alternative the workload has — a larger local memory, a different device, more DIMM slots — then the fabric is carrying its cost for nothing, and section 12 is the place that arithmetic lands.

The degenerate case is the one that looks like a pass and is not. A path with no budget against it is unbudgeted, not within budget, and the model reports the full four hundred and fifty as over a budget of nothing while declining to call it a fault. A design that never wrote down a latency target cannot be said to have met it, and treating a missing budget as a satisfied one is how a fabric gets built without anybody deciding what it was for.

Where the budget itself comes from is worth a paragraph, because a number nobody can derive is a number that gets chosen to fit. The budget is set by what the fabric is competing against. If the alternative is local DRAM at eighty nanoseconds, a fabric path at four hundred and fifty is five and a half times slower and the workload has to be one that tolerates that — which many memory-bound workloads genuinely do, since the alternative they actually face is swapping to storage. If the alternative is a direct-attached CXL device at two hundred and fifty, the fabric is buying flexibility for an extra two hundred nanoseconds per access, and section 12 is where that trade gets priced. The budget is therefore not a property of CXL, it is a property of what the workload would otherwise do, and a design review that cannot say what the alternative is has no way to argue about the number.

7. RTL 3 — The Aggregate Is Not The Bisection

The third constraint, and the one most likely to be answered with the wrong number by somebody who is otherwise doing the arithmetic properly.

A fabric's total bandwidth is the sum of its links. It is a large number, it is easy to compute, and it is almost never the constraint. The constraint is the narrowest cut: draw a line through the fabric separating the traffic's sources from its destinations, and the bandwidth of the links that line crosses is what the traffic actually has.

This matters more for a memory fabric than for a general-purpose one because the traffic pattern is not uniform. Hosts on one side, pooled memory on the other, is exactly the shape that makes the bisection the binding constraint — every access crosses it.

// RTL 3 - the aggregate is not the bisection. A fabric can have enough total
// bandwidth for every link in it and still not be able to carry the traffic
// that has to cross from one half of it to the other, which is the number a
// topology drawing hides.
module bisect_provision #(parameter int AGGREGATE_IS_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] offered_gbps, links_across, gbps_per_link, aggregate_gbps,
  output logic [15:0] bisect_gbps, carried_gbps, starved_gbps, carried_pct,
  output logic        provisioned,
  output logic [7:0]  n_evals, n_starved,
  output logic        bisect_err
);
  logic [31:0] raw_bisect, c_q;
  logic [15:0] capacity, true_carried, true_starved;
  logic        truly_starved;
  assign raw_bisect = {16'd0, links_across} * {16'd0, gbps_per_link};
  assign bisect_gbps = (raw_bisect > 32'd9999) ? 16'd9999 : raw_bisect[15:0];
  // The measured view charges traffic against the cut it must cross. The
  // aggregate view charges it against the fabric's total, which is always
  // larger and is never the constraint that bites.
  assign capacity = (AGGREGATE_IS_ENOUGH != 0) ? aggregate_gbps : bisect_gbps;
  assign true_carried = (offered_gbps > capacity) ? capacity : offered_gbps;
  assign carried_gbps = true_carried;
  assign true_starved = (offered_gbps > capacity) ? (offered_gbps - capacity) : 16'd0;
  assign starved_gbps = true_starved;
  assign c_q = (offered_gbps == 16'd0) ? 32'd100
             : (({16'd0, true_carried} * 32'd100) / {16'd0, offered_gbps});
  // No clamp here: true_carried is a minimum against offered_gbps, so the
  // quotient cannot exceed a hundred and a ceiling would be unreachable code.
  assign carried_pct = c_q[15:0];
  assign provisioned = (true_starved == 16'd0) && (offered_gbps != 16'd0);
  assign truly_starved = (offered_gbps > bisect_gbps) && (links_across != 16'd0);
  assign bisect_err = evaluate && truly_starved && provisioned;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_starved <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_starved) n_starved <= n_starved + 8'd1;
    end
  end
endmodule

Eight hundred gigabits per second wanting to cross a cut of two links at a hundred and twenty-eight each carries two hundred and fifty-six and starves five hundred and forty-four — thirty-two percent of the traffic — on a fabric whose aggregate is two thousand and forty-eight and which the aggregate view reports as fully provisioned.

Figure 2 — the same fabric, two numbers, one of them irrelevant. Nothing on the upper path is a mistake of arithmetic: the fabric really does have two thousand and forty-eight gigabits per second of link bandwidth, and that number will appear in the design document. It is the wrong number because the traffic cannot use it — the links that carry it are on the wrong side of the cut, serving hosts to their own local switch, and no amount of capacity there helps a request that has to reach the other half.

The sixth case is instructive in the other direction and worth including so the technique does not become a reflex. A cut wider than the fabric's aggregate — eight links across, on a fabric whose total is smaller — means the bisection is no longer the constraint and the aggregate view is now the pessimistic one. It reports a shortfall the real fabric does not have. The rule is not "use the bisection", it is "use the binding constraint", and knowing which one binds requires computing both.

The fifth case is the one that should be alarming and is quiet instead. A fabric with no links across the cut at all carries nothing and starves the whole eight hundred, and the model declines to call it an under-provisioned cut — it is two fabrics, which is a different design and a different conversation. The aggregate view reports the traffic as fully carried, on a topology where the two halves cannot reach each other at all.

The boundary is driven deliberately. Traffic exactly equal to the bisection is provisioned, because a cut sized precisely to its load has been sized correctly, and a model that failed it would be punishing a design that did the arithmetic and planned to the edge of it.

8. RTL 4 — A Switch Has An Inside

The fourth constraint, and the one that a port count cannot see.

A switch is drawn as a box with ports. Whether every port can talk to every other port at full rate simultaneously is a property of what is inside the box, and a switch whose internal bandwidth is less than the sum of its ports is oversubscribed before the fabric around it is.

This is a real and common design point rather than a defect. A switch built for a workload where not every port is busy at once is cheaper, smaller and cooler, and it is the right part for that workload. It is the wrong part for a memory fabric where every host is reading pooled memory continuously, and the specification that distinguishes them is not the port count.

// RTL 4 - a switch has an inside. Ports are what a topology drawing shows;
// whether every port can talk to every other port at full rate at the same
// time is a property of the switch's internal bandwidth, and a switch that
// cannot is oversubscribed before the fabric around it is.
module switch_blocking #(parameter int A_SWITCH_IS_NONBLOCKING = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] radix, port_gbps, internal_gbps, active_ports,
  output logic [15:0] demanded_gbps, blocked_gbps, internal_pct, ports_charged,
  output logic        nonblocking,
  output logic [7:0]  n_evals, n_blocked,
  output logic        blocking_err
);
  logic [31:0] raw_demand, i_q;
  logic [15:0] true_demand, true_blocked;
  logic        truly_blocked;
  // Only ports carrying traffic contribute to the demand on the crossbar,
  // and a design cannot activate more ports than the switch has.
  assign ports_charged = (active_ports > radix) ? radix : active_ports;
  assign raw_demand = {16'd0, ports_charged} * {16'd0, port_gbps};
  assign true_demand = (raw_demand > 32'd9999) ? 16'd9999 : raw_demand[15:0];
  assign demanded_gbps = true_demand;
  assign true_blocked = (true_demand > internal_gbps)
                      ? (true_demand - internal_gbps) : 16'd0;
  assign blocked_gbps = (A_SWITCH_IS_NONBLOCKING != 0) ? 16'd0 : true_blocked;
  assign i_q = (internal_gbps == 16'd0) ? 32'd999
             : (({16'd0, true_demand} * 32'd100) / {16'd0, internal_gbps});
  assign internal_pct = (i_q > 32'd999) ? 16'd999 : i_q[15:0];
  assign nonblocking = (blocked_gbps == 16'd0) && (true_demand != 16'd0);
  // ports_charged is a minimum against radix, so a switch with no ports has
  // no demand and nothing blocked; a separate radix guard would be dead.
  assign truly_blocked = (true_blocked != 16'd0);
  assign blocking_err = evaluate && truly_blocked && nonblocking;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_blocked <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_blocked) n_blocked <= n_blocked + 8'd1;
    end
  end
endmodule

Eight ports at a hundred and twenty-eight gigabits per second each into a crossbar of five hundred and twelve is a thousand and twenty-four demanded and five hundred and twelve blocked — twice the crossbar's bandwidth, and the non-blocking view blocks none of it.

The fifth case is the one worth checking on a datasheet. A switch with ports and no stated internal bandwidth has every bit of its demand blocked as far as this model is concerned, and the non-blocking view calls it a working switch. That is not a hypothetical: a specification that lists port count and port rate and says nothing about the switching capacity has not told you whether it is non-blocking, and the absence is easy to read as a yes.

The second case is the boundary and the model puts it on the generous side. A crossbar sized exactly to the sum of its ports blocks nothing, which is the definition of non-blocking, and a model that failed at the equality would be calling every correctly-specified switch oversubscribed.

The sixth and seventh cases bound the model. A switch with no ports cannot block and a switch whose ports carry nothing is idle rather than adequate — the model declines to draw a conclusion from either, because an idle switch under no load is not evidence that it would keep up under load, and a design review that accepts "it was fine in the lab" is accepting exactly that.

What blocking costs is worth connecting back to section 6, because it is easy to file this as a throughput problem. A blocked flit waits, and waiting is latency — so an oversubscribed switch does not simply cap the bandwidth, it lengthens the tail of every access that happens to contend. On a memory fabric that is the number the workload feels, and it is why 26.6's queueing behaviour and this section's arithmetic are the same phenomenon measured two ways.

9. RTL 5 — Somebody Has To Attach The Devices

The fifth constraint, and the one that is missing from more whiteboard answers than any other.

A CXL fabric does not configure itself. A device does not decide which host it belongs to; a host does not discover pooled memory and claim it. Every binding is an operation issued by a fabric manager over a management interface to the switches, and the fabric manager is software that runs somewhere, is owned by somebody, has an availability requirement, and is the single thing that can make the fabric unusable while every link stays up.

A design that says "and the fabric manager handles it" has named the component, which is further than most answers get. A design that says how many operations it has to perform, how fast it can perform them and what happens when it cannot has done the work.

// RTL 5 - somebody has to attach the devices. A CXL fabric does not
// configure itself: every host-to-device binding is an operation issued by a
// fabric manager, and a design with no manager has moved that work into an
// operations team without saying so.
module fabric_manager #(parameter int IT_CONFIGURES_ITSELF = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] attach_ops, fm_capacity_ops, manual_rate_ops, fm_present,
  output logic [15:0] owned_ops, unowned_ops, manual_ops, owned_pct,
  output logic        managed,
  output logic [7:0]  n_evals, n_unowned,
  output logic        manager_err
);
  logic [15:0] fm_ops, true_unowned, true_manual;
  logic [31:0] o_q;
  logic        truly_unowned;
  // A manager that is not in the design owns nothing, whatever capacity it
  // would have had.
  assign fm_ops = (fm_present != 16'd0) ? fm_capacity_ops : 16'd0;
  assign owned_ops = (attach_ops > fm_ops) ? fm_ops : attach_ops;
  assign true_manual = (attach_ops > fm_ops) ? (attach_ops - fm_ops) : 16'd0;
  assign manual_ops = (true_manual > manual_rate_ops) ? manual_rate_ops : true_manual;
  assign true_unowned = (true_manual > manual_ops) ? (true_manual - manual_ops) : 16'd0;
  assign unowned_ops = (IT_CONFIGURES_ITSELF != 0) ? 16'd0 : true_unowned;
  assign o_q = (attach_ops == 16'd0) ? 32'd100
             : (({16'd0, owned_ops} * 32'd100) / {16'd0, attach_ops});
  // No clamp here: owned_ops is a minimum against attach_ops, so the quotient
  // cannot exceed a hundred and a ceiling would be unreachable code.
  assign owned_pct = o_q[15:0];
  assign managed = (unowned_ops == 16'd0) && (attach_ops != 16'd0);
  assign truly_unowned = (true_unowned != 16'd0) && (attach_ops != 16'd0);
  assign manager_err = evaluate && truly_unowned && managed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_unowned <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_unowned) n_unowned <= n_unowned + 8'd1;
    end
  end
endmodule

Sixty-four attach operations against a manager that can perform forty and an operations process that can absorb ten leaves fourteen owned by nobody — sixty-two percent of the work with an owner — and the self-configuring view reports a managed fabric.

The third case is the one that actually happens and it is not carelessness. A manager specified and not deployed — in the design document, in the diagram, and not yet in the rack — owns nothing, and the model reports fifty-four operations with no owner while the self-configuring view reports a fabric that runs itself. Fabric managers are frequently the last piece to arrive because they are software and the schedule pressure is on the hardware, and a bring-up that assumed one is a bring-up done by hand.

The seventh case is the one that makes the model's definition of "managed" precise, and it is deliberately generous. A fabric where every operation is performed by hand is managed, because somebody owns every operation. It is slow, it does not scale, and it is a legitimate answer for a fabric of four devices that changes twice a year. The model's complaint is never about automation, it is about work with no owner — and stating it that way is what stops the section from becoming an argument that everything must be automated.

The degenerate case is the empty fabric. A manager with nothing to attach is fully owned by vacuity and is not a managed fabric, it is an unpopulated one; the model reports the hundred percent and declines to call it sound.

There is a second-order point the model does not compute and a design review should ask. The fabric manager is a single point of failure for reconfiguration, though not for traffic. A fabric whose manager is down keeps carrying every binding it already has — the switches do not forget — but it cannot attach a device, detach one, or recover a failed binding. That distinction decides how available the manager has to be: a fabric that is bound once at deployment can tolerate a manager that is down for a day; a fabric that reassigns capacity hourly, which is 27.5's entire subject, cannot.

10. RTL 6 — A Switch Failure Is A Memory Failure

The sixth constraint, and the one where a correct instinct produces the wrong design.

The instinct is redundancy: draw a second path, and the failure of one switch stops mattering. That is true for reachability and false for memory. A host with two paths to a switch survives the loss of one path. A host whose memory is behind a switch does not survive the loss of that switch, because the memory is physically behind it and the second path leads somewhere else.

Path redundancy is not capacity redundancy, and a fabric diagram with two lines drawn between every pair of boxes says nothing about which one you have.

// RTL 6 - a switch failure is a memory failure. Path redundancy keeps a host
// reachable; it does not keep the capacity behind a failed switch reachable,
// because that capacity is physically behind it. A fabric drawn with two
// paths is not a fabric with two copies of the memory.
module failure_domain #(parameter int TWO_PATHS_IS_REDUNDANT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] hosts_behind, total_hosts, gb_behind, paths,
  output logic [15:0] hosts_lost, gb_lost, blast_pct, hosts_charged,
  output logic        bounded,
  output logic [7:0]  n_evals, n_exposed,
  output logic        domain_err
);
  logic [31:0] b_q;
  logic [15:0] true_hosts, true_gb;
  logic        truly_exposed;
  // A domain cannot contain more hosts than the deployment has.
  assign hosts_charged = (hosts_behind > total_hosts) ? total_hosts : hosts_behind;
  assign true_hosts = hosts_charged;
  // Capacity behind a failed switch is gone whatever the path count says.
  assign true_gb = gb_behind;
  assign hosts_lost = (TWO_PATHS_IS_REDUNDANT != 0 && paths > 16'd1)
                    ? 16'd0 : true_hosts;
  assign gb_lost = (TWO_PATHS_IS_REDUNDANT != 0 && paths > 16'd1)
                 ? 16'd0 : true_gb;
  assign b_q = (total_hosts == 16'd0) ? 32'd0
             : (({16'd0, hosts_lost} * 32'd100) / {16'd0, total_hosts});
  // No clamp here: hosts_lost is at most hosts_charged, which is itself a
  // minimum against total_hosts, so the quotient cannot exceed a hundred.
  assign blast_pct = b_q[15:0];
  assign bounded = (gb_lost == 16'd0) && (total_hosts != 16'd0);
  assign truly_exposed = (true_gb != 16'd0) && (true_hosts != 16'd0);
  assign domain_err = evaluate && truly_exposed && bounded;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_exposed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_exposed) n_exposed <= n_exposed + 8'd1;
    end
  end
endmodule

Eight hosts of thirty-two behind one switch, with five hundred and twelve gigabytes attached to it, is a quarter of the deployment and all of that capacity — and the two-paths view reports no loss at all.

The second case is the honest one and it is why the section is about a claim rather than about a topology. With one path drawn, both views agree: eight hosts lose their memory and nobody pretends otherwise. The error is not in building a fabric with failure domains — every fabric has them — it is in drawing a second line and believing it changed the answer.

The fourth case separates the two things the word "redundancy" is doing. A switch with hosts behind it and no memory behind it takes eight hosts off the fabric and loses no capacity at all, and the model calls the memory blast radius bounded while still reporting that a quarter of the deployment went dark. Those are genuinely different failures with different responses — one is a connectivity outage the hosts can survive by rebooting into a smaller configuration, the other is data or capacity that is simply gone — and a design that reports them as one number cannot tell which it has.

The last case is the one worth putting on the whiteboard next to any single-switch fabric. The whole deployment behind one switch is a hundred percent blast radius, the two-paths view reports it as bounded, and this is the default topology anybody draws first because it is the simplest one that works.

What actually bounds a blast radius is worth naming, since the model only measures it. Spread the capacity, so that a host's memory comes from more than one switch and the loss of one costs a fraction rather than all of it. Or accept the domain and size it, deciding deliberately how many hosts may fail together and building the fabric so that no switch carries more than that. Both are design decisions with costs — the first needs more switch ports and more bisection, the second needs more switches — and neither is a line on a diagram.

11. RTL 7 — Decoders Are Finite

The seventh constraint, and the one that turns a clean capacity argument into a routing problem.

A host does not reach fabric-attached memory by magic. It reaches it through HDM decoders — a small, fixed number of address-range descriptors that say which physical address range goes out to which device through which path. The number is a property of the host silicon, it is single-digit to low-double-digit, and it does not grow because the pool did.

This is the constraint that limits how finely capacity can be carved. A host with ten decoders holding a terabyte cannot describe that terabyte in thirty-two pieces; it can describe it in ten, and the grain is what the capacity divided by the decoders gives.

// RTL 7 - decoders are finite. A host routes to fabric-attached memory
// through a small fixed number of HDM decoders, and that number - not the
// capacity - is what limits how finely a pool can be carved and how many
// separate ranges a host can hold at once.
module decoder_budget #(parameter int CARVE_AS_FINE_AS_YOU_LIKE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] decoders_avail, ranges_wanted, capacity_gb, grain_wanted_gb,
  output logic [15:0] ranges_held, ranges_short, granularity_gb, held_pct,
  output logic        carveable,
  output logic [7:0]  n_evals, n_short,
  output logic        decoder_err
);
  logic [31:0] g_q, h_q;
  logic [15:0] true_short, true_gran;
  logic        truly_short;
  assign ranges_held = (ranges_wanted > decoders_avail) ? decoders_avail : ranges_wanted;
  assign true_short = (ranges_wanted > decoders_avail)
                    ? (ranges_wanted - decoders_avail) : 16'd0;
  assign ranges_short = (CARVE_AS_FINE_AS_YOU_LIKE != 0) ? 16'd0 : true_short;
  // The finest grain a host can actually address is its capacity divided by
  // the decoders it has to describe that capacity with.
  assign g_q = (decoders_avail == 16'd0) ? 32'd9999
             : ({16'd0, capacity_gb} / {16'd0, decoders_avail});
  assign true_gran = (g_q > 32'd9999) ? 16'd9999 : g_q[15:0];
  // The carve-as-fine-as-you-like view reports whatever grain was asked for.
  assign granularity_gb = (CARVE_AS_FINE_AS_YOU_LIKE != 0) ? grain_wanted_gb : true_gran;
  assign h_q = (ranges_wanted == 16'd0) ? 32'd100
             : (({16'd0, ranges_held} * 32'd100) / {16'd0, ranges_wanted});
  // No clamp here: ranges_held is a minimum against ranges_wanted, so the
  // quotient cannot exceed a hundred and a ceiling would be unreachable code.
  assign held_pct = h_q[15:0];
  assign carveable = (ranges_short == 16'd0) && (granularity_gb <= grain_wanted_gb)
                   && (ranges_wanted != 16'd0);
  assign truly_short = ((true_short != 16'd0) || (true_gran > grain_wanted_gb))
                     && (ranges_wanted != 16'd0);
  assign decoder_err = evaluate && truly_short && carveable;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_short <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_short) n_short <= n_short + 8'd1;
    end
  end
endmodule

Ten decoders against thirty-two ranges on a terabyte is twenty-two ranges that cannot be described and a grain of a hundred and two gigabytes against a request for sixteen — thirty-one percent of what was asked for — and the carve-freely view reports whatever grain was asked for and a carve that fits.

The seventh case is the one that makes the section worth a model rather than a sentence, and it is the failure that is hardest to see coming. Enough decoders and a grain too coarse — four ranges wanted, four decoders available, and a carve that still fails — is a design where every count is satisfied and the capacity still cannot be described the way the workload needs it. The decoder budget has two dimensions and satisfying the obvious one does not satisfy the other.

The third case is the degenerate one and it is a real configuration. A host with no decoders available for fabric memory — because they are all consumed by something else — cannot route to the pool at all, and the carve-freely view reports a carve on a host that cannot reach a single byte of it.

The fourth case bounds the model. A host asked for no ranges is not short of decoders, and the model reports full coverage of an empty request while declining to call the result a carve.

This constraint interacts with 27.5 in a way worth stating explicitly, because the two chapters together produce a number neither does alone. Pooling's value comes from assigning capacity in pieces small enough to match demand, and the decoder budget puts a floor under how small a piece can be. A pool that must be handed out in hundred-gigabyte grains cannot serve a request for sixteen without wasting eighty-six, which is 27.5 section 11's rounding loss arriving through the host's address decoder rather than through the fabric manager's policy. The pooling design and the decoder budget have to be solved together or the granularity argument is made twice and honoured once.

12. RTL 8 — The Switch Is Not The Cost

The eighth constraint, and the one that decides whether the fabric should have been built.

A fabric's cost is written down as switch silicon because that is the part with a part number. It is the smallest term in the sum. The reach a fabric needs is longer than a direct attach, which means retimers; the connections are cables rather than traces, which at these rates are neither cheap nor trivial; and every one of those parts consumes power and rack space and has a failure rate.

A saving computed against the switch alone is computed against perhaps half of what the fabric costs.

// RTL 8 - the switch is not the cost. A fabric is switch silicon plus the
// retimers the longer reach needs plus the cabling plus the power for all of
// it, and a saving computed against the switch alone is computed against the
// smallest term.
module fabric_cost #(parameter int THE_SWITCH_IS_THE_COST = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] switch_cost, retimer_cost, cable_cost, saving,
  output logic [15:0] fabric_cost_out, net_gain, shortfall, return_pct,
  output logic        worth_it,
  output logic [7:0]  n_evals, n_negative,
  output logic        cost_err
);
  logic [31:0] raw_cost, r_q;
  logic [15:0] true_cost, true_short;
  logic        truly_negative;
  assign raw_cost = {16'd0, switch_cost} + {16'd0, retimer_cost} + {16'd0, cable_cost};
  assign true_cost = (raw_cost > 32'd9999) ? 16'd9999 : raw_cost[15:0];
  assign fabric_cost_out = (THE_SWITCH_IS_THE_COST != 0) ? switch_cost : true_cost;
  assign net_gain = (saving > fabric_cost_out) ? (saving - fabric_cost_out) : 16'd0;
  assign true_short = (true_cost > saving) ? (true_cost - saving) : 16'd0;
  assign shortfall = (THE_SWITCH_IS_THE_COST != 0) ? 16'd0 : true_short;
  assign r_q = (fabric_cost_out == 16'd0) ? 32'd999
             : (({16'd0, saving} * 32'd100) / {16'd0, fabric_cost_out});
  assign return_pct = (r_q > 32'd999) ? 16'd999 : r_q[15:0];
  assign worth_it = (shortfall == 16'd0) && (saving != 16'd0);
  assign truly_negative = (true_short != 16'd0) && (saving != 16'd0);
  assign cost_err = evaluate && truly_negative && worth_it;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_negative <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_negative) n_negative <= n_negative + 8'd1;
    end
  end
endmodule

A hundred of switch, sixty of retimer and forty of cable against a saving of a hundred and fifty is two hundred of fabric and fifty short of paying for itself — seventy-five percent returned — and the switch-is-the-cost view books fifty of gain and a hundred and fifty percent.

The last case is the honest limit and it is why the model has a switch-only build rather than simply being right. A fabric that really is only switch silicon — short reach, no retimers, direct attach cabling already in the budget — makes the two views identical, and the switch-is-the-cost view is correct. That configuration exists, inside a single chassis, and a design there should not be charged for a cost it does not have.

The cost that is missing from even the measured build is worth naming so the model is not read as complete. Power is the term that scales worst. A switch, its retimers and the optics or cables on every port draw power continuously whether the fabric is carrying traffic or not, and in a rack whose power budget is the binding constraint — which is most of them — that power displaces compute. A fabric that pays for itself in capacity and costs a server's worth of power has not obviously paid for itself, and the arithmetic that settles it is the deployment's, not this model's.

The sixth case is the boundary and the model puts it on the permissive side. A saving exactly equal to the fabric's cost counts as worth it — it breaks even, which is a decision rather than a failure, and one worth making deliberately given that a fabric also buys flexibility the arithmetic does not price.

The fourth case is the shape of most early proposals. A fabric built with no saving claimed for it is unjustified rather than uneconomic, and the model declines to alarm: a design that never wrote down what the fabric was worth has not made a bad trade, it has not made a trade. Asking for the number is the whole of the review.

What a fabric buys that the model does not price is worth saying, because otherwise this section reads as an argument against fabrics. A fabric buys the ability to change the assignment without touching the hardware — which is 27.5's entire value proposition, and which is worth a great deal in a deployment whose demand moves and nothing at all in one whose demand does not. The arithmetic here prices the fabric; what it is worth is a property of the workload, and a design that has both numbers has a decision rather than an argument.

13. RTL 9 — The Topology That Works At Four Hosts

The ninth constraint, and the one that makes a correct small design a wrong large one.

A topology is drawn at the scale somebody has in mind and deployed at the scale the programme reaches. The port arithmetic of section 5 does not stay linear as that happens: every switch added spends ports on the switches it connects to, so the usable ports per switch fall, so more switches are needed, so more ports are spent — and at the same time the path between the two furthest endpoints gets longer, which is section 6's cost arriving through a door nobody opened.

// RTL 9 - the topology that works at four hosts. A fabric drawn for a rack
// is sized by port arithmetic that does not stay linear: every switch added
// spends ports on the switches it connects to, and the path between the two
// furthest endpoints gets longer as the count grows.
module scale_limit #(parameter int IT_SCALES_LINEARLY = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] endpoints, ports_per_switch, hop_ns, budget_ns,
  output logic [15:0] switches_needed, hops_at_scale, path_ns, over_ns,
  output logic        scales,
  output logic [7:0]  n_evals, n_over,
  output logic        scale_err
);
  logic [31:0] usable, sw_lin, sw_real, sel_sw, raw_true_path, raw_path;
  logic [15:0] true_hops, true_path, true_over, lin_over;
  logic        truly_over;
  // Each switch in a multi-switch fabric spends two ports on the fabric.
  assign usable = (ports_per_switch > 16'd2)
                ? ({16'd0, ports_per_switch} - 32'd2) : 32'd1;
  assign sw_lin = (ports_per_switch == 16'd0) ? 32'd1
                : (({16'd0, endpoints} + {16'd0, ports_per_switch} - 32'd1)
                   / {16'd0, ports_per_switch});
  assign sw_real = ({16'd0, endpoints} + usable - 32'd1) / usable;
  assign sel_sw = (IT_SCALES_LINEARLY != 0) ? sw_lin : sw_real;
  assign switches_needed = (sel_sw > 32'd999) ? 16'd999 : sel_sw[15:0];
  // One switch is one hop; anything more puts a switch-to-switch hop in the
  // path as well, and a fabric of more than three needs a level above them.
  assign true_hops = (sw_real <= 32'd1) ? 16'd1
                   : ((sw_real <= 32'd3) ? 16'd2 : 16'd3);
  assign hops_at_scale = (IT_SCALES_LINEARLY != 0) ? 16'd1 : true_hops;
  assign raw_true_path = {16'd0, true_hops} * {16'd0, hop_ns};
  assign true_path = (raw_true_path > 32'd9999) ? 16'd9999 : raw_true_path[15:0];
  assign raw_path = {16'd0, hops_at_scale} * {16'd0, hop_ns};
  assign path_ns = (raw_path > 32'd9999) ? 16'd9999 : raw_path[15:0];
  assign true_over = (true_path > budget_ns) ? (true_path - budget_ns) : 16'd0;
  assign lin_over  = (path_ns > budget_ns) ? (path_ns - budget_ns) : 16'd0;
  assign over_ns   = (IT_SCALES_LINEARLY != 0) ? lin_over : true_over;
  assign scales = (over_ns == 16'd0) && (endpoints != 16'd0);
  assign truly_over = (true_over != 16'd0) && (endpoints != 16'd0);
  assign scale_err = evaluate && truly_over && scales;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_over <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_over) n_over <= n_over + 8'd1;
    end
  end
endmodule

A hundred and twenty-eight endpoints on sixteen-port switches needs ten switches and three hops — three hundred nanoseconds against a two-hundred-and-fifty-nanosecond budget — where the linear view reports eight switches and one hop and a fabric that scales.

The third case is worth reading because it is the one that gets deployed. Forty endpoints, three switches, two hops, two hundred nanoseconds — comfortably inside the budget, and the linear view still reports one hop. The design is fine, the discrepancy is invisible, and the same topology at three times the size is over budget. The error does not announce itself at the scale where it is cheap to fix.

The eighth case is the degenerate one and it is more instructive than it looks. A switch with exactly two ports has one usable port, because the fabric consumes the other, and eight endpoints therefore need eight switches. The linear view divides by two and reports four. A two-port switch is not a product anybody sells, but it is the limit that shows what the arithmetic is doing, and the same effect at sixteen ports is the difference between eight switches and ten.

The fifth case is the one that produces a real answer from a nonsense input. Switches with no ports means one switch per endpoint, three hops and an overrun — which is the right conclusion from a zero-port switch — while the linear view divides by nothing and reports a single-switch fabric.

The degenerate case at the other end is bounded deliberately. A fabric sized for no endpoints does not scale, it is absent, and the model declines to report a latency failure for a path nobody takes.

14. RTL 10 — A CXL Fabric Design Assembled

Nine sections of constraints. This one puts them in one place and makes the confident answer visible as what it is: one bit of six.

// RTL 10 - a CXL fabric design assembled. Nine sections of inputs, one
// summary. "The switches are connected" is bit 0: true, necessary, and one
// sixth of what makes a fabric design sound.
module fabric_signoff #(parameter int CONNECTED_IS_DESIGNED = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic        topology_named, ports_accounted, hops_budgeted,
  input  logic        bisect_provisioned, manager_owned, blast_bounded,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_evals, n_sound, n_claimed,
  output logic        signoff_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~topology_named;
  assign fail_mask[1] = ~ports_accounted;
  assign fail_mask[2] = ~hops_budgeted;
  assign fail_mask[3] = ~bisect_provisioned;
  assign fail_mask[4] = ~manager_owned;
  assign fail_mask[5] = ~blast_bounded;
  assign conditions_met = {15'd0, topology_named} + {15'd0, ports_accounted}
                        + {15'd0, hops_budgeted} + {15'd0, bisect_provisioned}
                        + {15'd0, manager_owned} + {15'd0, blast_bounded};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp here: conditions_met is a sum of six one-bit values, so the
  // quotient cannot exceed a hundred and a ceiling would be unreachable.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  // The connected view reads bit 0 and stops.
  assign claimed = (CONNECTED_IS_DESIGNED != 0) ? topology_named : truly_sound;
  assign sound = claimed;
  assign signoff_err = evaluate && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When a topology has been named and any one of the other five fails, the assembled model reports that the design is not sound and the connected view reports a fabric.

Across the eight evaluations, the assembled model calls one design sound and the connected view calls six of them a fabric.

The bit order is by how early the condition binds. Bit 0 is the answer's existence. Bit 1 is the arithmetic that decides whether the topology is buildable at all. Bits 2 and 3 are the two performance constraints, latency before bandwidth because latency is the reason the technology exists. Bit 4 is the operational condition, which can be satisfied after the hardware is chosen. Bit 5 is last because a blast radius can only be bounded once there is a topology to bound it in.

"The switches are connected" is bit 0, and it is true. That is what makes it the right weak definition for this chapter: it is not a wrong statement, it is a necessary one offered where a design was wanted. A topology in which everything reaches everything is a real achievement on a whiteboard and it is the first thing to establish — and it says nothing about whether the ports exist, whether the latency is survivable, whether the traffic fits across the cut, whether anybody will configure it, or what happens when one box fails.

The five other bits fail independently, which is the property that makes the mask a summary rather than a score. A fabric can be short of ports while its latency is fine. It can be inside its latency budget and unable to carry the traffic across the cut, because those are different links. It can be perfectly provisioned and have nobody to configure it. And it can satisfy every one of those and put the whole deployment behind one switch. None of the five implies any of the others, and a design review that checks one and infers the rest is checking one.

Figure 4 — the mask ordered by how early each condition binds, which is also the order in which a design review can act on them. The first two decide whether the topology is buildable and are settled before any part is chosen. The middle two are performance constraints that a different topology can fix. The last two are operational and structural, and they are the ones a programme can still address after the switches are on order — which is precisely why they are the ones that reach deployment unaddressed.

15. Quantitative Reasoning

Two ports short on a fabric of thirty-two, because two sixteen-port switches spend two joining each other — and six ports on a fabric of four switches.

Four hundred and fifty nanoseconds against a three-hundred budget — a hundred and fifty percent, from two hops the drawing did not charge for.

Five hundred and forty-four gigabits per second starved across a cut of two links, on a fabric whose aggregate bandwidth is two thousand and forty-eight and which reports itself provisioned.

Five hundred and twelve gigabits per second blocked inside the switch — a thousand and twenty-four of port demand into a crossbar of five hundred and twelve, which is twice what it can carry.

Fourteen attach operations owned by nobody, of sixty-four, with a manager doing forty and an operations process absorbing ten. Sixty-two percent of the work has an owner.

A quarter of the deployment and five hundred and twelve gigabytes behind one switch, reported as no loss at all by a design with a second path drawn on it.

Twenty-two ranges that cannot be described and a hundred-and-two-gigabyte grain against a request for sixteen, from ten decoders holding a terabyte.

Two hundred of fabric against a hundred and fifty of saving — fifty short, seventy-five percent returned, where the switch alone returns a hundred and fifty percent.

Ten switches and three hops where the linear view sees eight and one, on a hundred and twenty-eight endpoints, and fifty nanoseconds over the budget as a result.

One design of eight sound; the connected view counts six. The assembled model's summary, and the chapter's.

16. Assertions

The testbenches carry 639 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported one, and it was the connected view's soundness percentage — parameter-independent, with its partner asserted, and now asserted in both directions anyway.

Both builds are asserted on every degenerate case. A fabric with no endpoints, a fabric with no switches, a path with no budget, a cut with no links across it, a switch with no ports, a switch with no crossbar behind it, a manager with nothing to attach, a switch with no memory behind it, a host with no decoders, a fabric with no saving claimed for it, a topology sized for nothing.

Every clamp that an input can reach is driven past its limit exactly once. More endpoints than the port count will hold, more hops than a path can traverse, a total latency past its ceiling, a bisection past its ceiling, a port demand past its ceiling, more hosts in a domain than the deployment has, more ranges than any host could describe, a grain past its ceiling, a fabric cost past its ceiling, a switch count past its ceiling, and a hop cost that saturates the path.

The qualification in that sentence is doing real work and section 17 explains why. Five clamps in this chapter's first drafts could never fire, because the quantity feeding them was already bounded by a minimum taken against the same denominator. They have been removed rather than tested, because a ceiling that cannot be reached is unreachable code and asserting against it proves nothing.

Every error output is checked in both directions in every case. Section 5's second and third cases, section 6's fifth and sixth, section 7's sixth and seventh, section 8's second, section 9's seventh and section 12's last exist to assert the quiet half — a single switch, an empty fabric, a direct attach, hops that really are free, a cut wider than the aggregate, traffic exactly equal to the bisection, a crossbar sized to its ports, a fabric run entirely by hand, and a fabric that really is only switch silicon. Each is a case where the drawn view is right, and a model that alarmed on them would be unusable.

17. Mutation Testing

137 mutations, 137 killed. Sixty-five against the first testbench, seventy-two against the second. The first run killed one hundred and twenty-seven and left nine, and the nine are the chapter's real result.

Five of the nine survivors were the same defect, and it is one no previous chapter in this module has named. A percentage clamped at a hundred is unreachable code when the quantity above the line is already a minimum taken against the quantity below it — carried against offered, owned against attached, held against wanted, lost against the deployment, met against six. The quotient cannot exceed a hundred, so the ceiling never fires, and a mutation of it is an equivalent mutant rather than a survivor.

That made a script, domcheck.py, which reads the models and reports the class without running anything — the second check in three batches that is answerable from the source alone. Run across the eight chapters already signed off in Modules 25 to 27 it finds nineteen more, in agreed_pct, achieved_pct, burn_pct, stall_pct, reach_pct and a dozen others. None of those chapters ever injected a mutation on one, so no published kill count is wrong; what is wrong is the sentence in each of their section 16 that says every clamp is driven past its limit, since some of those clamps cannot be. That sentence has been corrected in all eight.

The remaining four survivors were three different faults and each is worth its line. Two were stimulus gaps — a guard on a switch with exactly two ports, which no test exercised, and a path-latency ceiling that no hop cost in the stimulus could reach. Both were closed by adding a case rather than by weakening the mutation, and the two-port switch turned out to be the clearest illustration in section 13 of what the port arithmetic does. One was structural: the true overrun in the scale model was computed, used only to set a boolean, and never observed as a value, so reversing its subtraction changed nothing that any assertion could see. The model was restructured so the reported overrun is a view of the true one, which is the house pattern everywhere else in the chapter and should have been the pattern here.

The lesson is that a clamp is not automatically worth testing. Four batches of discipline have said drive every clamp past its limit, and the instruction quietly assumed every clamp could be reached. Five in one chapter could not, and the check that finds them costs no simulation at all.

18. Verification Strategy

Count the ports including the ones the switches spend on each other. Section 5. Two on a chain of two, six on a chain of four, and the error grows with the fabric.

Charge the deepest path, not the average one. Section 6. The tail is what a memory-bound workload feels.

Compute the bisection and the aggregate, and use the one that binds. Section 7. Neither is automatically the answer.

Ask for the switch's internal bandwidth, not its port count. Section 8. A specification that lists ports and rates and stops has not told you.

Name who issues the attach operations and how many there are. Section 9. A manager in the diagram and not in the rack owns nothing.

Separate reachability redundancy from capacity redundancy. Section 10. A second path does not make a second copy of the memory.

Divide the capacity by the decoders before promising a grain. Section 11.

Add the retimers and the cables to the switch. Section 12. The switch is the smallest term.

Run the port arithmetic forward to the scale the programme will actually reach. Section 13. The topology that works at four hosts is the question.

19. Synthesis and Implementation Reality

A CXL switch is a large, hot, expensive part. The port count, the rates and the internal switching capacity all cost silicon, and a non-blocking switch at high radix is among the more demanding digital designs in a rack.

Retimers are not optional at the reach a fabric needs. The signalling rates that make CXL worth having do not survive a cable and two connectors without retiming, and every retimer is a part, a power budget and a failure mode.

The fabric manager talks to the switches over a management interface, out of band from the CXL links themselves. That is what lets it survive a data-path problem and what makes it a separate availability question from the fabric's.

Hop latency is a published number and it is not small. A switch's cut-through latency is the figure to ask for; the store-and-forward path and the arbitration delay under load are what a datasheet is least likely to quantify, and section 8 is why the second matters.

A fabric of one switch is not a fabric, in the sense that every constraint in this chapter except sections 8, 9 and 10 is trivially satisfied by it — and the three that are not are the three that catch people.

20. Silicon Observability

Free, and on paper. Port counts, port rates, and the switch's stated switching capacity. Sections 5 and 8's inputs, when the datasheet gives the third.

Cheap. The topology's hop depth, from the diagram. Section 6's hop count is a property of the drawing, and only the per-hop latency needs a number from anywhere else.

Cheap. The number of HDM decoders a host has. Section 11, from the host's specification.

Moderate. The traffic that must cross the cut, which needs a traffic matrix rather than a total. Section 7's binding number is the one a capacity plan is least likely to contain.

Moderate. The switch's behaviour under simultaneous load on every port, which is section 8's real question and which a datasheet's headline figure does not answer.

Expensive, and usually skipped. The number of attach operations a deployment will actually issue and how fast. Section 9's input is an operational figure that nobody owns until the fabric is running.

Unobtainable before the fact. What scale the programme will reach. Section 13 is a bet, and the model's contribution is to make the bet explicit rather than to settle it.

21. Debug Lab

A fabric is built and the memory it serves is slower than the projection.

Step 1 — count the hops on the slow path. Section 6. If the path is three hops and the projection assumed one, the arithmetic is finished and the answer is topological.

Step 2 — check whether the traffic crosses a cut. Section 7. Hosts on one side and memory on the other is the shape, and the bisection is the number.

Step 3 — ask what the switch's internal bandwidth is and how many ports are busy. Section 8. A switch at two hundred percent of its crossbar is a latency problem as well as a bandwidth one.

Step 4 — check the decoder assignment. Section 11. A host routing through fewer ranges than it needs may be reaching some capacity the long way, or not at all.

Step 5 — check whether the bindings are the ones intended. Section 9. A fabric manager that failed part way through leaves a fabric that works and is not the fabric that was designed.

Step 6 — only then look at the devices. 26.6 owns what happens inside them, and reaching for it before the fabric has been eliminated is a long way round.

Steps 1 to 4 are reads from a diagram and two datasheets, which makes this a cheap investigation whose result is usually a design conclusion rather than a fix.

22. Design Review

How many ports does this topology need, including the ones spent connecting the switches?

How many hops is the deepest path, and what is the latency budget it has to meet?

What is the bisection, what has to cross it, and which of the bisection and the aggregate binds?

What is each switch's internal bandwidth, and how many of its ports are busy at once?

Who issues the attach operations, how many are there, and what happens when that component is unavailable?

If any one switch fails, how many hosts lose their memory and how much capacity goes with them?

How many HDM decoders does each host have, and what grain does that put a floor under?

What does the fabric cost with the retimers and cables included, and what is it worth?

What scale is this topology drawn for, and what does the port arithmetic say at three times that?

23. How This Appears In Real Engineering

The design is drawn early, it is correct, and it is drawn at the wrong scale.

What usually happened is section 5 and section 13 together. The topology was sized for the pilot — a few hosts, one switch, everything one hop away — and every number in the review was comfortable. The programme grew, switches were added, and the port arithmetic that was exactly right at one switch was wrong at four in the direction that costs ports. The fabric was re-planned twice, each time from the same linear calculation, and each time it came out slightly short.

The second shape is section 7, and it survives a careful process. The capacity plan totalled the fabric's bandwidth, compared it to the offered load, and found headroom. The traffic matrix was never written down, so nobody computed what had to cross the middle, and the fabric ran at a third of its planned throughput with two thousand gigabits per second of link bandwidth sitting on the wrong side of the cut.

The third is section 9 and it is an integration story rather than a design one. The fabric manager was in the architecture from the start and arrived last, because it was software and the hardware had the schedule pressure. Bring-up was done by hand, which worked, and the hand-built bindings became the deployment's real configuration — undocumented, unreproducible, and owned by whoever happened to do it.

The fourth is section 10 and it is the quietest. The diagram had two lines between every pair of boxes, redundancy was ticked in the review, and the first switch failure took a quarter of the deployment's memory with it. Nothing about the second path was wrong; it was answering a different question than the one the review thought it had asked.

The pattern is that a fabric is the part of a design most easily drawn and least easily counted, and every failure above is a number that a picture does not contain.

24. Common Misconceptions

"The switches are connected." True, necessary, and one condition of six. Section 14.

"Two sixteen-port switches give thirty-two ports." Thirty. Section 5.

"A hop is a nanosecond or two." It is tens, and the tail under load is worse. Section 6.

"The fabric has plenty of bandwidth." Across the cut that matters? Section 7.

"The switch has eight ports at a hundred and twenty-eight." And what inside it? Section 8.

"The fabric manager handles that." Which one, running where, doing how many operations? Section 9.

"There are two paths, so it is redundant." For reachability. Not for the memory behind the switch. Section 10.

"We can carve the pool however we like." Into as many pieces as the host has decoders. Section 11.

"The fabric costs one switch." Plus retimers, plus cables, plus the power for both. Section 12.

"The topology scales." Run the port arithmetic forward and check. Section 13.

25. Interview Reasoning

"Design a CXL fabric for thirty-two endpoints." Start with ports, not with a picture. Thirty-two endpoints need thirty-two endpoint ports plus the ports the switches spend on each other, so two sixteen-port switches are two short and the answer is three switches or a larger radix. Then charge the deepest path against a latency budget, compute what has to cross the bisection, name who does the attach operations, and say how many hosts lose their memory when one switch fails.

"Why not just add a switch?" Because a switch adds ports and consumes them. Each inter-switch link costs a port at both ends, so the usable ports per switch fall as the fabric grows, and the path between the furthest endpoints gets longer at the same time.

"How much bandwidth does this fabric have?" Two answers: the aggregate, which is the sum of the links and is almost never the constraint, and the bisection, which is what crosses the narrowest cut and usually is. Which one binds depends on the traffic matrix, so I would ask for that before quoting either.

"Is this switch non-blocking?" That is a question about its internal switching capacity, not its port count. Ports times port rate against the crossbar: if the crossbar is smaller, it is oversubscribed, and under simultaneous load that shows up as latency before it shows up as throughput.

"You have two paths to every switch. Is the fabric redundant?" For reachability, yes. For the memory behind a switch, no — that capacity is physically behind it and the second path leads elsewhere. If capacity redundancy is wanted, the capacity has to be spread across switches, which costs ports and bisection.

"What breaks first as this fabric grows?" Ports, then hops. The port arithmetic goes non-linear as soon as there is a second switch, and hop depth steps up at the point a level has to be added above the switches — which is also where the latency budget usually goes.

"Who configures it?" A fabric manager, out of band, issuing a binding per host-device pair. It is not a single point of failure for traffic already bound, but it is for any reconfiguration — which decides how available it needs to be, and that depends on how often the fabric reassigns.

26. Exercises

1. Sixty endpoints on twenty-four-port switches. How many switches in a chain, how many ports are spent on the fabric, and how many are left? Redo it for a topology where every switch connects to every other.

2. A base latency of 180 ns, hops costing 90 ns, and a budget of 400 ns. How many hops fit? What does the budget have to be for three?

3. A fabric of sixteen links at 256 Gb/s. Four cross the middle. Compute the aggregate and the bisection. At what offered load across the cut does the bisection stop being the constraint?

4. A twelve-port switch at 128 Gb/s per port with an internal bandwidth of 1,024 Gb/s. How many ports can be busy at once before it blocks? What internal bandwidth would make it non-blocking?

5. A deployment with 128 attach operations, a manager that does 60 and an operations process that absorbs 20. How many are unowned? What manager capacity makes the answer zero, and what else could?

6. Forty-eight hosts, six switches, capacity spread evenly. Compute the blast radius of one switch failure in hosts and in capacity. Redo it for a topology where all the capacity is behind one switch.

7. A host with 8 decoders and 2 TB of fabric memory. What grain can it describe? What does it cost to serve a request for 32 GB, and how many such requests can the host hold at once?

8. Extend the assembled model with a seventh bit for a condition this chapter does not cover — power, cabling, or thermal. Justify its position using the rule that the ordering is by how early the condition binds.

9. Take the topology from exercise 1 and run it forward to 240 endpoints. How many switches, how many hops, and at what point does the hop depth step?

27. Summary

A fabric is ports. Thirty-two endpoints on two sixteen-port switches is thirty usable ports and a shortage of two.

Every hop costs latency, and the deepest path is the one that matters — four hundred and fifty nanoseconds against a three-hundred budget, from two hops nobody charged for.

The aggregate is not the bisection. Five hundred and forty-four gigabits per second starved on a fabric with two thousand of link bandwidth.

A switch has an inside, and a port count says nothing about it — a thousand and twenty-four demanded into a crossbar of five hundred and twelve.

Somebody has to attach the devices. Fourteen operations of sixty-four owned by nobody is a fabric that does not come up.

A switch failure is a memory failure, and a second line on the diagram does not make a second copy of the memory.

Decoders are finite, and they put a floor under how finely a pool can be carved — a hundred and two gigabytes against a request for sixteen.

The switch is the smallest term in the fabric's cost, and a saving computed against it is computed against half the bill.

The topology that works at four hosts is the question, because the port arithmetic goes non-linear and the hops step up.

Six bits, and "the switches are connected" is one of them. One design of eight is sound; the connected view counts six.