CXL 3.0 Fabric Enhancements

Every chapter of Module 20 ran into the same wall. 20.1 §6 established that CXL 2.0 permits exactly one switch between a host and a device. 20.1 §14 turned that into a port count. 20.4 §14 turned it into a fleet-planning number: hosts plus devices, against one switch's ports, and no more.

CXL 3.0 lifts it. This chapter is what that costs.

The constraint was never arbitrary. With one switch in the path, the destination is a pure function of the address, and a request needs to carry nothing but where it is going in memory. Two levels breaks that — the same address can be reached through different switches, so the address alone no longer names a port. Everything below follows from that one fact.

1. The Engineering Problem — Depth Is Not Free

A request has to name its destination. Beyond one hop the address is ambiguous, so routing needs a port identifier the request carries. Section 5.

Each level multiplies reach and is crossed twice. A two-level tree of sixteen-port switches reaches 240 endpoints instead of 16, and costs 200 ns instead of 100. Section 6.

A fabric has more than one path between two points, which is the point — and an ordered stream sprayed across two of them arrives out of order. Section 7.

Doubling the signalling rate does not double the bandwidth. At 64 GT/s the correction and the retries are both larger, and the delivered rate is what is left. Section 8.

A fabric with a cycle in its dependency graph will stop, and virtual channels are what break the cycle rather than a scheduling policy. Section 9.

And every endpoint needs a name that is unique across the fabric, not merely across the switch it happens to hang from. Section 11.

This chapter against 20.1, stated precisely. That one owns the single-level switch and why it is enough for a rack. This one owns what changes when there are two of them in a path — and section 15 shows a topology that reaches further and is not a fabric.

2. The One-Sentence Model

A multi-level CXL fabric works when reach genuinely extends past one switch, requests carry the port that names their destination, an ordered stream takes one path, the PHY budget accounts for correction and retry, virtual channels break every dependency cycle, and every endpoint has one fabric-wide name — and every defect below is a topology that reaches further and fails one of the other five.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one property. A real fabric is a routing table, a crossbar per switch, a virtual-channel allocator, a PHY with a correction engine and a fabric manager that programs all of it, and none of that is reproduced. What is reproduced is the decision or the arithmetic each of them has to get right.

Each model is built twice — a correct build and a broken build selected by a parameter. The broken builds in this chapter share a shape: they are all CXL 2.0 assumptions carried into a fabric. One switch level needs no port identifier, has one path, has no dependency cycle, and can name endpoints per switch. Every one of those was correct and stops being correct at the second level.

Figure 1 — The reach is the point and the port identifier is the price. Every request in a multi-level fabric carries something a CXL 2.0 request did not need, because the address that named a destination through one switch names only a memory location through two.

5. RTL 1 — A Request Has To Name Where It Is Going

// RTL 1 - multi-level routing. With more than one switch in the path an address
// no longer names a destination, so a request has to carry where it is going.
module port_based_routing #(parameter int ADDRESS_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       req,
  input  logic [3:0] dest_pid, local_pid,
  input  logic [1:0] hops,
  input  logic       pid_present,
  output logic       routable, is_local, needs_pid,
  output logic [7:0] n_reqs, n_unroutable,
  output logic       lost_route_err
);
  assign is_local  = (dest_pid == local_pid);
  // One hop can be decoded from the address alone. Beyond that the request has
  // to name its destination, because the address is ambiguous across switches.
  assign needs_pid = (hops > 2'd1);
  assign routable  = (ADDRESS_ONLY != 0) ? (hops <= 2'd1)
                                         : (!needs_pid || pid_present);
  // A request accepted into a multi-level fabric with nothing naming its port.
  assign lost_route_err = req && routable && needs_pid && !pid_present;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reqs <= 8'd0; n_unroutable <= 8'd0;
    end else if (req) begin
      n_reqs <= n_reqs + 8'd1;
      if (!routable) n_unroutable <= n_unroutable + 8'd1;
    end
  end
endmodule

Six requests.

One unroutable request in the correct build, three in the address-only build.

Zero and one hop need no identifier, and that is the compatibility story. A CXL 3.0 fabric carries CXL 2.0 traffic patterns unchanged at one level, which is why the third and fourth rows matter as much as the second — the identifier is required only where the address stops being sufficient.

The address-only build is not broken, it is CXL 2.0. It refuses two hops rather than mis-routing them, which is exactly right for what it is: a switch that cannot express a two-level destination and declines to guess. Its failure is a capability gap, not a correctness bug — and that distinction is why lost_route_err reads zero for both builds. Neither accepts a request it cannot route; one simply routes fewer of them.

6. RTL 2 — Reach Multiplies, Latency Adds

// RTL 2 - what multi-level buys. Each level multiplies reach and adds a crossing
// to every request, in both directions.
module fabric_reach #(parameter int IGNORE_DEPTH_COST = 0) (
  input  logic clk, rst_n,
  input  logic        plan,
  input  logic [7:0]  ports, levels,
  input  logic [15:0] base_ns, per_cross_ns, budget_ns,
  output logic [15:0] endpoints, added_ns, total_ns,
  output logic        within_budget,
  output logic [7:0]  n_plans, n_over,
  output logic        budget_err
);
  logic [31:0] e_q;
  // A two-level tree of P-port switches reaches P times (P-1) endpoints; the
  // model uses the same shape for one level, where it degenerates to P.
  assign e_q = (levels == 8'd0) ? 32'd0
             : ((levels == 8'd1) ? {24'd0, ports}
                                 : ({24'd0, ports} * ({24'd0, ports} - 32'd1)));
  assign endpoints = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  // Every level is crossed going out and coming back.
  assign added_ns = (IGNORE_DEPTH_COST != 0) ? per_cross_ns
                                             : ({8'd0, levels} * per_cross_ns * 16'd2);
  assign total_ns = base_ns + added_ns;
  assign within_budget = (total_ns <= budget_ns);
  assign budget_err = plan && !within_budget;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_over <= 8'd0;
    end else if (plan) begin
      n_plans <= n_plans + 8'd1;
      if (!within_budget) n_over <= n_over + 8'd1;
    end
  end
endmodule

Five plans. Sixteen-port switches, 50 ns per crossing, a 200 ns base and a 450 ns budget.

One budget miss; the one-way model saw none.

Fifteen times the reach for twice the latency is the trade in one line. Sixteen endpoints at one level; 240 at two — and the 240 is P × (P−1), not P², because each second-level switch spends one of its ports on the uplink. That uplink port is the tax of depth, and it is why a fabric's usable width is always slightly less than its nominal one.

Row four is the one that decides deployments. Three levels is 500 ns against a 450 ns budget, and the one-way model reports 250 — wrong by exactly a factor of two on the added term, at every depth. 20.1 §7 made this error at one level and it cost 50 ns. At three levels the same error costs 150.

Why the broken build is not a strawman. Counting one crossing is what a per-hop latency figure invites, and at one level the mistake is small enough to be absorbed by margin. Depth is what makes it fatal, because the error scales with the thing being added.

7. RTL 3 — More Than One Path Is The Point And The Problem

// RTL 3 - a fabric has more than one path between two points, and using them
// needs the ordering rules that a single path made unnecessary.
module multipath_order #(parameter int SPRAY_FREELY = 0) (
  input  logic clk, rst_n,
  input  logic       send,
  input  logic [1:0] path_a, path_b,
  input  logic       same_stream, ordered_required,
  output logic       same_path, order_kept, may_spray,
  output logic [7:0] n_sends, n_reordered,
  output logic       reorder_err
);
  assign same_path = (path_a == path_b);
  // Two requests of one ordered stream must take one path, or nothing keeps
  // them in order. Independent streams may take any path.
  assign order_kept = !ordered_required || !same_stream || same_path;
  assign may_spray  = (SPRAY_FREELY != 0) ? 1'b1 : order_kept;
  // An ordered pair sent down two paths.
  assign reorder_err = send && may_spray && !order_kept;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_sends <= 8'd0; n_reordered <= 8'd0;
    end else if (send) begin
      n_sends <= n_sends + 8'd1;
      if (may_spray && !order_kept) n_reordered <= n_reordered + 8'd1;
    end
  end
endmodule

One reordering.

Three conditions, and two of them are exemptions. Rows three and four are the reason multipath is worth having at all: most traffic is either independent or unordered, and holding all of it to one path would waste every path but one. The constraint applies to exactly the traffic that needs it, which is what makes a fabric faster than a single link rather than merely wider.

Why the broken build is not a strawman. Spraying freely is the highest-throughput policy and the obvious one when the fabric is built — every path carries load, nothing is idle. It is correct for a fabric carrying no ordered streams, and there is no such fabric.

The failure is silent and looks like something else. A reordered pair does not error; it delivers, in the wrong sequence, and the corruption surfaces in an application as a value that was overwritten and came back. Section 22 spends two steps clearing exactly this hypothesis, because stale-looking data is what both a reordering and a coherence defect produce — and only one of them leaves a counter.

8. RTL 4 — 64 GT/s Is Not Twice 32

// RTL 4 - the 64 GT/s PHY. Doubling the signalling rate does not double the
// delivered bandwidth, because the error rate and the correction both change.
module phy_rate #(parameter int IGNORE_FEC = 0) (
  input  logic clk, rst_n,
  input  logic        sample,
  input  logic [15:0] raw_gbps, fec_pct, retry_pct,
  output logic [15:0] after_fec_gbps, effective_gbps, loss_pct,
  output logic        meets_target,
  input  logic [15:0] target_gbps,
  output logic [7:0]  n_samples, n_missed,
  output logic        overclaim_err
);
  logic [31:0] f_q, r_q, l_q;
  // Forward error correction is bytes on the wire; retries are time on the wire.
  assign f_q = ({16'd0, raw_gbps} * (32'd100 - {16'd0, fec_pct})) / 32'd100;
  assign after_fec_gbps = (IGNORE_FEC != 0) ? raw_gbps
                        : ((f_q > 32'd65535) ? 16'hFFFF : f_q[15:0]);
  assign r_q = ({16'd0, after_fec_gbps} * (32'd100 - {16'd0, retry_pct})) / 32'd100;
  assign effective_gbps = (r_q > 32'd65535) ? 16'hFFFF : r_q[15:0];
  assign l_q = (raw_gbps == 16'd0) ? 32'd0
             : ((({16'd0, raw_gbps} - {16'd0, effective_gbps}) * 32'd100) / {16'd0, raw_gbps});
  assign loss_pct = (l_q > 32'd65535) ? 16'hFFFF : l_q[15:0];
  assign meets_target = (effective_gbps >= target_gbps);
  assign overclaim_err = sample && !meets_target;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_samples <= 8'd0; n_missed <= 8'd0;
    end else if (sample) begin
      n_samples <= n_samples + 8'd1;
      if (!meets_target) n_missed <= n_missed + 8'd1;
    end
  end
endmodule

Five samples against an 800 Gbps raw rate and a 650 Gbps target.

Two missed targets; the FEC-free model saw one.

The two costs compose and are different in kind. Forward error correction is bytes on the wire — it is there whether or not anything goes wrong. Retries are time on the wire — they happen only when something does. Applying the retry rate to the raw figure instead of the post-FEC figure overstates the result, which is one of the mutations section 18 kills.

Row two is the argument for reading a PHY spec carefully. A doubled signalling rate arrives with a larger correction overhead, because the error rate at the higher rate is what forced the stronger correction in the first place. 28% total loss on a nominal 800 Gbps link is 576 delivered, and a bandwidth plan built on the nominal figure is out by nearly a third.

9. RTL 5 — A Cycle Will Stop The Fabric

// RTL 5 - deadlock. A fabric with cycles in its dependency graph will stop, and
// virtual channels are what break the cycle.
module fabric_deadlock #(parameter int ONE_CHANNEL = 0) (
  input  logic clk, rst_n,
  input  logic       check,
  input  logic [3:0] channels_needed, channels_available,
  input  logic       cycle_present,
  output logic       enough_channels, cycle_broken, safe,
  output logic [7:0] n_checks, n_deadlockable,
  output logic       deadlock_err
);
  assign enough_channels = (channels_available >= channels_needed);
  // A dependency cycle is broken by giving each stage of it its own channel.
  assign cycle_broken = !cycle_present || enough_channels;
  // The single-channel build assumes ordering alone prevents deadlock and never
  // counts channels against the topology, so it calls every fabric safe.
  assign safe = (ONE_CHANNEL != 0) ? 1'b1 : cycle_broken;
  // A fabric declared safe with a cycle nothing breaks.
  assign deadlock_err = check && safe && cycle_present && !enough_channels;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_checks <= 8'd0; n_deadlockable <= 8'd0;
    end else if (check) begin
      n_checks <= n_checks + 8'd1;
      if (cycle_present && !enough_channels) n_deadlockable <= n_deadlockable + 8'd1;
    end
  end
endmodule

Five topology checks. A cycle needing two channels.

One deadlockable topology, and the single-channel build called it safe.

A dependency cycle is a property of the topology, not of the traffic. Row four is the case that makes that precise: one channel, a demand for two, and no cycle — so nothing can deadlock and the channel count is irrelevant. Row three is the identical channel situation with a cycle, and it will stop.

Why the broken build is not a strawman. A single-channel fabric is what one switch level is: with no cycles possible in a star, ordering alone is sufficient and virtual channels are silicon spent on nothing. The assumption survives into a topology where cycles exist, and the fabric that results does not fail gradually — it stops.

10. Waveform — An Ordered Stream Meeting Two Paths

The reorder count freezing at cycle 5 is the model's exemption, not its limit. Stream B is unordered, so spraying it across both paths reorders nothing — there was no order to break. A fabric that could not tell the two streams apart would have to hold B to one path as well, and would give up half its bandwidth to protect an ordering nobody asked for.

11. RTL 6 — One Name Per Endpoint, Fabric-Wide

// RTL 6 - a fabric-wide identifier space. Every endpoint needs a name that is
// unique across the fabric, not merely across one switch.
module fabric_addressing #(parameter int SWITCH_LOCAL = 0) (
  input  logic clk, rst_n,
  input  logic       assign_req,
  input  logic [7:0] id_width_bits, endpoints_total,
  input  logic [3:0] switch_a, switch_b,
  input  logic [7:0] local_id_a, local_id_b,
  output logic       ids_collide, space_sufficient, unique_ok,
  output logic [15:0] id_space,
  output logic [7:0] n_assigns, n_collisions,
  output logic       collision_err
);
  logic [31:0] sp_q;
  // A switch-local identifier repeats on every switch, so two endpoints on
  // different switches can carry the same name.
  assign ids_collide = (SWITCH_LOCAL != 0) ? (local_id_a == local_id_b)
                     : ((local_id_a == local_id_b) && (switch_a == switch_b));
  assign sp_q = (id_width_bits >= 8'd16) ? 32'd65535 : (32'd1 << id_width_bits);
  assign id_space = sp_q[15:0];
  assign space_sufficient = ({8'd0, endpoints_total} <= id_space);
  assign unique_ok = !ids_collide && space_sufficient;
  // Two distinct endpoints answering to one name.
  assign collision_err = assign_req && ids_collide;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assigns <= 8'd0; n_collisions <= 8'd0;
    end else if (assign_req) begin
      n_assigns <= n_assigns + 8'd1;
      if (ids_collide) n_collisions <= n_collisions + 8'd1;
    end
  end
endmodule

Six assignments.

One collision fabric-wide; two under switch-local naming.

The first row is the whole model. Identifier 5 on switch 0 and identifier 5 on switch 1 are two different endpoints in a fabric that qualifies names by switch, and one endpoint in a fabric that does not. CXL 2.0 could name per switch because there was only one, and the assumption travels quietly into a two-level topology where it is a collision.

Rows four through six are the other half — an identifier space large enough for the fabric. Six bits holds 64 endpoints and not 200, which is the arithmetic a fabric-wide name has to satisfy before uniqueness even becomes the question.

Uniqueness and sufficiency are separate failures with the same symptom. A collision means two endpoints answer to one name; an insufficient space means some endpoint cannot be named at all. Both produce requests arriving at the wrong place, and they call for opposite responses — renumbering in the first case, a wider field in the second, which is an architectural change rather than a configuration one.

The identifier width is the field section 20 charges every request for. Making it generous costs bits on every packet in the fabric forever; making it tight costs a respin the first time a deployment outgrows it. It is the clearest example in this chapter of a decision that cannot be revisited, and it is made before anyone knows how large the fabrics will be.

12. RTL 7 — Telling A Device Its Copy Is Stale

// RTL 7 - back-invalidate. A device that caches host memory must be told when
// the host changes it, which a 2.0 device had no channel to hear.
module back_invalidate #(parameter int NO_BI_CHANNEL = 0) (
  input  logic clk, rst_n,
  input  logic       host_write,
  input  logic       device_cached, bi_sent, bi_acked,
  output logic       stale_possible, invalidated, coherent,
  output logic [7:0] n_writes, n_stale,
  output logic       stale_read_err
);
  // A cached line the host has written is stale until the device is told and
  // acknowledges.
  assign stale_possible = device_cached;
  assign invalidated = (NO_BI_CHANNEL != 0) ? 1'b0 : (bi_sent && bi_acked);
  assign coherent = !stale_possible || invalidated;
  // A host write over a line a device still holds and was never told about.
  assign stale_read_err = host_write && stale_possible && !invalidated;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_writes <= 8'd0; n_stale <= 8'd0;
    end else if (host_write) begin
      n_writes <= n_writes + 8'd1;
      if (stale_possible && !invalidated) n_stale <= n_stale + 8'd1;
    end
  end
endmodule

Five host writes.

Two stale lines in the correct build, three in the build with no channel.

Sent and acknowledged are two conditions. An invalidate in flight has not landed, and a device that acts on the send rather than the acknowledgement declares coherence a round trip early. That window is short and it is exactly where a race lives.

Row four is what keeps coherent honest. An uncached line with no invalidate sent is coherent, because there is nothing anywhere to be stale. A model written as coherent = invalidated would call it incoherent — and section 18 records that this case was missing from the testbench until a mutation asked for it.

This is the capability CXL 2.0 did not have. A 2.0 device could cache host memory and had no channel by which the host could reach back and say the line had changed. The fabric did not create the problem; it made the solution routable, because a back-invalidate has to find a device that may now be two switch levels away.

Figure 3 — The left branch at the top is CXL 2.0, unchanged: at one hop the address is still sufficient and no identifier is consulted. Everything below it is new, and the right-hand terminal is what a 2.0 switch would have had no way to express.

13. RTL 8 — A Region Several Hosts Hold

// RTL 8 - shared memory. CXL 3.0 lets several hosts hold one region coherently,
// which needs a directory that knows who holds what.
module shared_region #(parameter int NO_DIRECTORY = 0) (
  input  logic clk, rst_n,
  input  logic       write,
  input  logic [3:0] sharer_mask, writer_bit,
  output logic [3:0] to_invalidate,
  output logic [7:0] n_sharers, n_writes, n_missed,
  output logic       missed_sharer_err
);
  logic [3:0] others;
  // Everyone holding the line except the writer must be invalidated.
  assign others = sharer_mask & ~writer_bit;
  assign to_invalidate = (NO_DIRECTORY != 0) ? 4'd0 : others;
  assign n_sharers = {3'd0, sharer_mask[0]} + {3'd0, sharer_mask[1]}
                   + {3'd0, sharer_mask[2]} + {3'd0, sharer_mask[3]};
  // A sharer holding a line that has been written and was not invalidated. No
  // separate "somebody shares it" term is needed: to_invalidate can only differ
  // from others when others is non-empty.
  assign missed_sharer_err = write && (to_invalidate != others);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_writes <= 8'd0; n_missed <= 8'd0;
    end else if (write) begin
      n_writes <= n_writes + 8'd1;
      if (to_invalidate != others) n_missed <= n_missed + 8'd1;
    end
  end
endmodule

Four writes.

Two missed writes.

The writer is excluded and the exclusion is the interesting bit. A host writing a line it holds does not invalidate its own copy — it updates it. sharer_mask & ~writer_bit is one operation and getting it wrong in either direction is a full failure: invalidate everyone including the writer, and the writer loses the line it just wrote; invalidate only the writer, and every other holder keeps a stale copy.

Row three is the case people forget. The writer is not among the sharers — it is writing a line it does not currently hold — and both holders must still be invalidated. A directory that reasons "everyone except me" from the sharer list rather than from the writer identity gets this wrong, because the writer is not in the list to subtract.

14. RTL 9 — Depth Has To Be Worth It

// RTL 9 - the cost of the fabric itself. Every level adds silicon, latency and
// a management surface, and the reach has to be worth all three.
module fabric_cost #(parameter int COUNT_REACH_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        assess,
  input  logic [15:0] endpoints, latency_ns, latency_budget_ns,
  input  logic [7:0]  levels,
  output logic [15:0] reach_per_level, cost_index,
  output logic        worth_it, latency_ok,
  output logic [7:0]  n_assess, n_rejected,
  output logic        false_value_err
);
  assign reach_per_level = (levels == 8'd0) ? 16'd0 : (endpoints / {8'd0, levels});
  // The cost index is levels times latency: both grow with depth.
  assign cost_index = {8'd0, levels} * latency_ns;
  assign latency_ok = (latency_ns <= latency_budget_ns);
  // Reach alone justifies any depth. Reach against latency does not.
  assign worth_it = (COUNT_REACH_ONLY != 0) ? (endpoints != 16'd0)
                                            : ((endpoints != 16'd0) && latency_ok);
  // A fabric called worthwhile whose latency does not meet the budget.
  assign false_value_err = assess && worth_it && !latency_ok;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_assess <= 8'd0; n_rejected <= 8'd0;
    end else if (assess) begin
      n_assess <= n_assess + 8'd1;
      if (!worth_it) n_rejected <= n_rejected + 8'd1;
    end
  end
endmodule

Five assessments against a 500 ns latency budget.

Two refusals against one, and one false valuation.

Row two is the pitch every fabric vendor makes and the number every platform team has to check. Fifteen times the endpoints of row one, and a latency that misses the budget — so the reach is real, the fabric works, and the workload it was bought for will not run on it. Reach is not a benefit until it is reach within a latency the application can use.

Row four is the sanity clamp both builds agree on: a fabric reaching nothing is worth nothing however fast it is. The endpoints != 0 term is what stops the model reporting excellent value for an empty topology.

Figure 4 — The cycle at the left is a property of the topology and exists in both cases. What differs is whether each stage has a buffer the other cannot block. The lower-right terminal is the one section 21 argues about: a fabric that has stopped emits nothing, so the warning has to arrive before it.

15. RTL 10 — The Fabric Assembled

// RTL 10 - the 3.0 fabric assembled. Everything that must hold before more than
// one switch level is a fabric rather than a diagram.
module fabric_model #(parameter int REACH_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       reach_extended,     // more than one switch level is routable
  input  logic       routing_carried,    // requests name their destination port
  input  logic       order_preserved,    // one ordered stream takes one path
  input  logic       phy_budgeted,       // FEC and retry are in the bandwidth model
  input  logic       deadlock_free,      // channels break every dependency cycle
  input  logic       ids_unique,         // one name per endpoint, fabric-wide
  output logic       is_fabric,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_fabric,
  output logic       false_fabric_err
);
  assign fail_mask[0] = ~reach_extended;
  assign fail_mask[1] = ~routing_carried;
  assign fail_mask[2] = ~order_preserved;
  assign fail_mask[3] = ~phy_budgeted;
  assign fail_mask[4] = ~deadlock_free;
  assign fail_mask[5] = ~ids_unique;
  // The reach-only build counts endpoints and calls the result a fabric, which
  // is what a topology diagram shows.
  assign is_fabric = (REACH_ONLY != 0) ? reach_extended : (fail_mask == 6'd0);
  assign false_fabric_err = evaluate && is_fabric && (fail_mask != 6'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_fabric <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (is_fabric) n_fabric <= n_fabric + 8'd1;
    end
  end
endmodule

One fabric of six, and four false claims.

The reach-only definition is what a topology diagram shows, and it is right about exactly one of the six. Rows four and five are the ones that hurt: a topology that reaches 240 endpoints, routes correctly, preserves ordering and budgets its PHY — and either stops under load because a cycle has no channel to break it, or delivers requests to the wrong endpoint because two of them answer to one name.

16. Quantitative Reasoning

Routing. Six requests. The address-only build could route three of six; the correct build routed five and refused the one with no identifier. Zero and one hop need no identifier at all, which is the compatibility case.

Reach. Sixteen-port switches: 16 endpoints at one level, 240 at two — P × (P−1), because each second-level switch spends a port on its uplink. Latency 100 ns at one level, 200 at two, 300 at three, and the one-way model reports 50 at every depth — wrong by a factor of two on the added term, and the term grows with depth.

Ordering. Five requests of one ordered stream sprayed across two paths arrive 1, 3, 2, 5, 4 — every pair transposed. Two of four cases are exemptions, which is why multipath is worth having.

PHY. 800 Gbps raw at 20% FEC and 10% retry delivers 576 Gbps — a 28% total loss. The FEC-free model reports 720 and meets a target the real link misses.

Deadlock. A cycle needing two channels with one available: the correct build calls it unsafe, the single-channel build calls it safe. A deadlocked fabric does not degrade — it stops.

Addressing. Identifier 5 on two switches is two endpoints fabric-wide and one endpoint switch-locally. Six bits holds 64 endpoints, not 200.

Back-invalidate. Five writes, two stale lines in the correct build and three with no channel — and an invalidate sent and not acknowledged has not landed.

Shared regions. Four sharers, one writer: three to invalidate. The directory-less build named nobody, twice.

Cost. 3600 endpoints across three levels at 600 ns against a 500 ns budget: fifteen times the reach of a two-level fabric, and refused, because reach outside a usable latency is not reach.

The assembled model. Six properties, six configurations, one fabric. The reach-only definition reported five.

17. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Routing. The hop boundary is driven at zero, one and two, and the identifier is asserted unnecessary below the boundary.

chk(pNp == 1'b0, "one hop needs no port identifier");
chk(pNp == 1'b1, "two hops needs a port identifier");
chk(aRo == 1'b0, "the address-only build cannot route two hops at all");

Reach. The P × (P−1) shape and the doubled crossing are asserted as exact values at each depth.

chk(fEp == 16'd240, "two levels reach 240 endpoints");
chk(fAn == 16'd200, "crossing two levels twice costs 200 ns");
chk(gTn == 16'd250, "the one-way model still reports 250");

Ordering. Both exemptions are asserted as correct behaviour, which is what stops a checker holding all traffic to one path.

chk(mOk == 1'b1, "but independent streams have no order between them");
chk(mOk == 1'b1, "a stream requiring no order cannot be reordered");

PHY. The two costs are asserted separately at each stage, and the exact-target boundary is driven.

chk(hAf == 16'd720, "10 percent FEC leaves 720 Gbps");
chk(hEg == 16'd684, "5 percent retry leaves 684");
chk(hEg == 16'd650, "650 Gbps effective");
chk(hMt == 1'b1,    "which exactly meets a 650 Gbps target");

Deadlock. The exact channel boundary is driven, and the no-cycle case is asserted safe with insufficient channels.

chk(dEc == 1'b1, "exactly two channels is enough for two");
chk(dCb == 1'b1, "but with no cycle there is nothing to break");

Addressing. The exact identifier-space boundary is driven, and the two-switch case is asserted as two names.

chk(gIs == 16'd64, "six bits is a 64-endpoint space");
chk(gSs == 1'b1,   "which exactly holds 64 endpoints");
chk(gIc == 1'b0,   "and identifier 5 on two switches is two names, fabric-wide");

Back-invalidate. The uncached-and-unsent case is asserted coherent, which is the case a naive model gets wrong.

chk(bIv == 1'b0, "and no invalidate was sent");
chk(bCo == 1'b1, "which is still coherent, because there is nothing to make stale");

Shared regions. The writer-not-among-the-sharers case is asserted to invalidate both holders.

Cost. The exact latency budget is driven, and an empty fabric is asserted worthless in both builds.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 243 checks across two testbenches, 123 on the front five models and 120 on the back five, all passing on the unmutated sources.

18. Mutation Testing

Forty-seven mutations were injected one at a time.

47 injected, 47 killed, after four survivors and one design defect were diagnosed.

A broken build that could not fail — found before the harness ran. The first version of RTL 5 gave the single-channel build one channel and let it compute safe honestly, which meant it correctly reported unsafe and could therefore never produce the deadlock error it exists to model. That is batch 019's hardest class, caught at baseline rather than by a survivor: the testbench simply failed, because the assertion expected an error the design could not generate. The fix was the same in kind — the broken build now assumes ordering alone prevents deadlock and declares every fabric safe, which is both the realistic mistake and the one that exercises the check.

Survivors 1 and 2 — thresholds never driven at their value. The PHY target comparison and the identifier-space comparison both survived becoming exclusive, because no sample landed exactly on the target and no fabric had exactly as many endpoints as its space holds. Both were one stimulus each.

The second of those had a trap. The obvious exact case — an eight-bit space holding 256 endpoints — does not work, because endpoints_total is eight bits wide and 256 wraps to zero. The stimulus was driving space_sufficient with a zero endpoint count and passing for the wrong reason. A boundary case has to be reachable in the port widths the model actually has, and six bits against 64 endpoints is.

Survivor 3 — a case the stimulus never combined. Coherence reduced to invalidated survived because every uncached case in the testbench also had an invalidate sent and acknowledged. An uncached line with no invalidate sent distinguishes them, and it is the state a device spends most of its life in.

Survivor 4 — a provably equivalent guard. The missed-sharer check carried others != 0 alongside to_invalidate != others. In both builds to_invalidate can only differ from others when others is non-empty, so the first term is implied. Deleted with a comment, and the mutation replaced with one that drops the comparison instead of the guard.

19. Verification Strategy

What a testbench for a real fabric must cover.

Every hop depth, including zero and one. A fabric that requires a port identifier at one hop has broken compatibility with every CXL 2.0 traffic pattern, and that failure is only visible if the shallow cases are driven.

Both exemptions of every constraint. Multipath ordering has two — independent streams and unordered streams — and a testbench that drives only the constrained case will accept a fabric that holds all traffic to one path and wastes itself.

Every threshold at exactly its value, in the width the model has. Section 18's second survivor is the sharper version of that rule: an eight-bit port cannot be driven to 256, and a boundary case that silently wraps tests the wrong thing while appearing to pass.

Each half of every compound condition, and each exemption alone. Coherence is "not cacheable or invalidated", and the first half needs a case where the second is also false.

The cases that are correct and look like failures. An address-only switch refusing two hops. An uncached line that was never invalidated. A writer not invalidating its own copy. A fabric with insufficient channels and no cycle. Each trips a naive checker.

What a real fabric needs that these models do not have. Concurrency — several requests allocating channels at once. Adaptivity — a path chosen dynamically, where the ordering constraint has to be enforced by the allocator rather than checked afterwards. Reconfiguration — a fabric manager changing the topology while traffic is in flight, where the port identifiers a request already carries may name a port that has moved.

20. Synthesis and Implementation Reality

The port identifier is bits on every request, forever. A field wide enough to name every endpoint in the fabric rides in every packet header whether the topology is one level or three. That is the permanent tax of routability, paid by single-level traffic that does not need it, and it is why the identifier width is an architectural decision rather than an implementation one.

Virtual channels are buffers, not policy. Breaking a dependency cycle means each stage of the cycle has its own buffer that cannot be blocked by the others — so channels_needed in section 9 is a buffer count per port, multiplied by port count, multiplied by flit width. A fabric that needs four channels needs four times the buffering of one that needs one, and that is the real cost behind row three of section 9's table.

Multi-level routing tables are state that must be kept consistent. Every switch needs a mapping from port identifier to output port, and a fabric manager has to program all of them coherently. A request in flight during a reconfiguration carries an identifier resolved against the old table and arrives at a switch holding the new one.

64 GT/s changes the analogue budget before it changes the protocol. The correction overhead in section 8 is the consequence of an error rate that the channel imposes, and the choice is between a stronger code (more bytes, always) and more retries (more time, sometimes). Neither is free and the balance is a silicon decision.

Back-invalidate needs a reverse path with its own flow control. A device holding host memory has to be reachable by an invalidate that cannot be blocked behind the traffic it is invalidating — which is another dependency cycle, and another reason section 9's channel count is not one.

21. Silicon Observability

The blocked-on-channel counter is the one that has to be designed in. A deadlocked fabric produces no traffic and therefore no telemetry — every other counter simply stops. A rising blocked count is the only signal that arrives before the stop, and a fabric that reports occupancy without reporting blocking gives an operator a number that looks healthy right up to the moment nothing moves.

22. Debug Lab

Symptom. A two-level CXL 3.0 fabric is brought up. Every endpoint enumerates. Traffic flows. A database workload reports intermittent data corruption on one shared region — reads occasionally return values that were overwritten seconds earlier. Every link is clean, no CRC errors, no retries above baseline, and the FEC counters are unremarkable.

Step 1 — is it the PHY? Read effective against raw rate per link. Every link is within a point of its expected 14% loss. Section 8 is not the chapter.

Step 2 — is it ordering? Read the ordered-stream path-change counter. Zero. Every ordered stream took one path for its lifetime, so section 7 is not it either — and that is a real clearance, because a corruption that looks like stale data is exactly what reordering produces.

Step 3 — is it coherence? Read back-invalidates sent, acknowledged and timed out. Sent and acknowledged match. Timed out is zero. Every invalidate the fabric sent landed.

Step 4 — how many were sent? Compare sharers invalidated per write against the directory's sharer count. The directory reports three sharers on the corrupted region; invalidates sent per write reads two. One holder is never being told.

Step 5 — which one? Read the sharer list. Hosts 1, 2 and 3 hold the region; host 0 is writing. The invalidate mask being sent is derived from the sharer list minus the lowest set bit rather than minus the writer's bit — and host 0 is not in the list at all, so the subtraction removes host 1 instead. Section 13's row three, in silicon.

The finding. A directory that computes "everyone except me" from the sharer list rather than from the writer identity. It is correct whenever the writer is among the sharers, which is most of the time — a host usually writes a line it already holds. It fails on exactly the case where the writer does not currently hold the line, which is the first write after an eviction.

Why it looked like a PHY or ordering problem. The symptom is stale data, which is what a reordered stream or a corrupted flit produces. The cause is a coherence directory computing the wrong mask, which produces no error anywhere — every invalidate it sends is delivered and acknowledged correctly. It sends the wrong number of them.

The fix. Derive the exclusion from the writer identity, not from the list. Then add the counter comparison from step 4 as a permanent check: sharers invalidated per write against directory sharer count, which would have caught this at bring-up rather than in a database.

23. Design Review

1. What does a request carry that names its destination, and how wide is it? Section 5, and section 20 for what it costs on single-level traffic.

2. Is the latency budget written per level or per crossing? Twice per level. Section 6, and the error grows with depth.

3. How many endpoints does the topology actually reach? P × (P−1) at two levels, not P² — each second-level switch spends a port upward. Section 6.

4. Which traffic is ordered, and what holds it to one path? And what happens to the rest — because holding everything to one path wastes the fabric. Section 7.

5. What is the effective rate after FEC and retry? Not the signalling rate. Section 8, and 28% is a realistic loss at the higher rate.

6. How many virtual channels does the topology need, and how many does the switch have? The first number comes from the dependency graph, not from a preference. Section 9.

7. Is there a blocked-on-channel counter? A deadlocked fabric produces no telemetry at all. Section 21.

8. Are endpoint identifiers unique fabric-wide or per switch? Section 11, and the per-switch assumption arrives from CXL 2.0 intact.

9. Does the coherence directory derive its invalidate mask from the writer identity or from the sharer list? Section 22 is the second answer in production.

10. Which of the six properties does the team believe "a fabric" means? Section 15 exists because the answer is usually the first one.

24. How This Appears In Real Engineering

A platform architect evaluating a multi-level fabric does section 6 and section 14 together, because the two numbers are the decision: reach against latency. Fifteen times the endpoints for twice the latency is a good trade for a capacity-bound workload and a bad one for a latency-bound workload, and the same fabric is the right and wrong answer depending on which is being run.

A switch team implementing port-based routing discovers that the identifier field is the easy part and the routing-table consistency is the hard one. Every switch has to agree, a fabric manager has to program them all, and a request in flight during a change carries an identifier resolved against a table that no longer exists.

A verification team finds section 9 the most dangerous model in the chapter, because deadlock is the only failure here that produces no observable. Every other defect leaves a counter, a stale value or a missed target. A deadlocked fabric leaves silence, which is why section 21 argues so hard for the blocked-on-channel counter.

A coherence team owns sections 12 and 13 and will meet section 22. The directory logic is small, the failure mode is intermittent, and the case that breaks it — a writer that does not currently hold the line — is common in production and rare in a directed test.

25. Common Misconceptions

"CXL 3.0 just allows more switches." It requires requests to carry a port identifier, because the address stops naming a destination past one hop. Section 5.

"Two levels of sixteen-port switches reach 256 endpoints." 240 — each second-level switch spends a port on its uplink. Section 6.

"The switch latency is per hop." Per crossing, and every level is crossed twice. Section 6, and at three levels the one-way error is 150 ns.

"A fabric with many paths is faster." Only for traffic that may use them. An ordered stream must take one path. Section 7.

"64 GT/s doubles the bandwidth." 800 Gbps raw at 20% FEC and 10% retry delivers 576. Section 8.

"Deadlock is a scheduling problem." It is a topology problem, solved with buffers. Virtual channels, not policy. Section 9.

"A deadlock will show up in the telemetry." It stops the traffic that produces telemetry. Only a blocked-on-channel counter warns first. Section 21.

"Endpoint identifiers are unique." Per switch, if the naming came from CXL 2.0 — where there was only one switch. Section 11.

"An invalidate sent is a line invalidated." Not until it is acknowledged. Section 12.

"The writer is always one of the sharers." Not on the first write after an eviction, which is exactly where section 22's directory bug lives.

26. Interview Reasoning

Q. Why can CXL 2.0 not cascade switches, and what does 3.0 add?

Because with one switch the destination is a pure function of the address, and with two the same address is reachable through different switches — so the address no longer names a port. CXL 3.0 adds a port identifier the request carries. The follow-up worth reaching: one hop and zero hops still need no identifier, which is what keeps 2.0 traffic patterns working.

Q. Two levels of sixteen-port switches. How many endpoints, and at what latency?

240, not 256 — each second-level switch spends one port on its uplink. Latency is two crossings per level and two levels, so four crossings: 200 ns at 50 ns each. The error to catch is counting one crossing per level, which understates by exactly a factor of two and gets worse with depth.

Q. Your fabric has four paths between two points. Can you use all of them?

For independent or unordered traffic, yes — that is the point. For a single ordered stream, no: it must take one path, or nothing keeps it in order. The interesting follow-up is what fraction of traffic that constrains, because a fabric that holds everything to one path has bought width and thrown it away.

Q. What causes deadlock in a fabric, and what fixes it?

A cycle in the dependency graph — requests waiting on buffers held by requests waiting on them. The fix is virtual channels: each stage of the cycle gets a buffer that the others cannot block. The sharp follow-up: how would you know it happened? A deadlocked fabric produces no telemetry, so the only warning is a rising blocked-on-channel count before everything stops.

Q. A host writes a line that three other hosts hold. What must happen?

All three are invalidated; the writer is not. The follow-up that finds real bugs: what if the writer does not hold the line? Then it is not in the sharer list to subtract, and a directory computing "the list minus one of its own bits" invalidates the wrong host. That is the first write after an eviction, and it is common.

Q. Does 64 GT/s double the bandwidth over 32?

No. The higher rate needs stronger forward error correction — bytes on the wire, always — and produces more retries — time on the wire, sometimes. 800 Gbps raw at 20% and 10% delivers 576. The two costs compose and they are different in kind, which is why applying the retry rate to the raw figure is wrong.

27. Exercises

1. Extend RTL 1 to three levels and show that a port identifier resolved against a stale routing table names a port that has moved — section 20's reconfiguration hazard.

2. Generalise RTL 2 to L levels and find the depth at which added latency exceeds the base access, for a 50 ns crossing and a 200 ns base.

3. Give RTL 3 an adaptive allocator that picks a path per request, and show that the ordering constraint must be enforced at allocation rather than checked afterwards.

4. Split RTL 4's retry term into a per-flit error probability and derive the retry percentage from it, then find the error rate at which retries cost more than a stronger code.

5. Extend RTL 5 to derive channels_needed from an explicit dependency graph rather than taking it as an input, and show which topologies need more than two.

6. Add a reconfiguration path to RTL 6 in which endpoints are renumbered, and show what happens to a request carrying an old identifier.

7. Give RTL 7 a timeout on the acknowledgement and decide what a device should do when a back-invalidate is never acknowledged.

8. Fix section 22's bug in RTL 8 the wrong way — subtract the lowest set bit — and confirm it passes every case where the writer holds the line.

9. Combine RTL 2 and RTL 9 into one model and plot reach against latency for one, two and three levels, marking the depth at which a given budget is exceeded.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the topology it catches that the current mask calls a fabric.

28. Summary

Module 20 ran into one wall repeatedly. This is what it costs to remove it.

Past one hop the address stops naming a destination, so every request carries a port identifier — a field on every packet forever, paid by single-level traffic that does not need it.

Reach multiplies and latency adds. Sixteen-port switches reach 16 endpoints at one level and 240 at two — P × (P−1), because each second-level switch spends a port upward — for 200 ns instead of 100. A one-way latency model is wrong by a factor of two on the added term, and the term grows with depth.

More than one path is the point and the problem. Five requests of an ordered stream sprayed across two paths arrive 1, 3, 2, 5, 4. Two of four cases are exemptions, and a fabric that cannot tell them apart gives up half its bandwidth protecting an order nobody asked for.

64 GT/s is not twice 32. 800 Gbps raw at 20% FEC and 10% retry delivers 576 — a 28% loss — and the two costs compose differently: correction is bytes always, retries are time sometimes.

A dependency cycle stops the fabric, and virtual channels are buffers rather than policy. A deadlocked fabric produces no telemetry at all, which is why the only useful warning is a rising blocked-on-channel count.

Endpoint names must be unique fabric-wide. Identifier 5 on two switches is two endpoints, or one — and the per-switch assumption travels intact from a generation that had only one switch.

An invalidate sent is not an invalidate landed, and an uncached line that was never invalidated is coherent, which is the case a model written as coherent = invalidated gets wrong.

The writer is excluded from the invalidate, by identity and not by position. Four sharers, three invalidated — and a directory that subtracts a bit from the list instead of the writer's bit fails on the first write after an eviction.

Reach is not value until it is reach inside a usable latency. 3600 endpoints at 600 ns against a 500 ns budget is a working fabric the workload cannot run on.

And a broken build that reports the failure honestly can never exercise the check. RTL 5's first version declared itself unsafe and could not produce the deadlock error it existed to model — batch 019's hardest class, caught this time at baseline instead of by a survivor.

Reaching further is one property of six. The definition a topology diagram shows called five of six configurations a fabric when one was.

21.2 — CXL 3.0 Peer-to-Peer takes the fabric this chapter built and removes the host from the middle of a transfer.