The CXL Fabric

Chapters 3.1 and 3.2 both assumed one host, one link, one device. Every design decision in them was made under that assumption, and several of them stop being sufficient the moment it is removed.

This chapter removes it.

1. The Engineering Problem

A host and a device connected directly need no routing: there is exactly one place a request can go. They need no fairness: there is one requester. They need no isolation: there is one host, and everything the device has belongs to it.

Put a switch in the middle and all three become required — and not as refinements. Each is a new class of state that must be configured, and configuration is the part of a system that is wrong most often. A device that is perfect in a point-to-point bench can be unreachable, starved, or improperly shared the first time it sits behind a fabric, with nothing in the device at fault.

2. The One-Sentence Model

A fabric is what turns "the device" into "which device" — and the cost of that question is three kinds of state the point-to-point case never needed: routing state that says where an address lives, arbitration state that decides whose turn it is, and binding state that decides who is allowed — each of which is configured by software and can therefore be wrong while every individual link is healthy.

3. What This Chapter Owns

Switching mechanics at spec depth belong to Modules 20 and 21. This chapter is the architecture: what a fabric must contain, and what goes wrong when it does not.

4. The Mental Model — a Fabric Is Three Tables

The picture to carry is not a box with ports. It is three lookups on the path:

   Host A ──┐                                    ┌── Device 0
            ├──►  ┌───────────────────────┐  ────┤
   Host B ──┘     │ 1. where does this    │      └── Device 1
                  │    address live?      │
                  │ 2. whose turn is it?  │      ┌── Memory
                  │ 3. is this host       │  ────┘
                  │    allowed?           │
                  └───────────────────────┘

Each question has an answer that software wrote. A fabric is mostly configuration, and its characteristic failures are configuration failures with hardware symptoms.

5. Ports, Hierarchies and the Vocabulary

Some terms are worth fixing before the architecture, because they are used loosely elsewhere.

Upstream and downstream are relative to the host. A switch's upstream port faces a host root port; its downstream ports face devices. The CXL Consortium's CXL 2.0 material states the platform requirement plainly — the upstream port of a CXL 2.0 switch connects to a CPU, and downstream ports face CXL or PCIe devices.

Virtual hierarchy. A switch is shared physical hardware presenting a private-looking tree to each host. The Consortium's CXL 2.0 material names these structures: a Virtual CXL Switch (VCS), virtual PCI-to-PCI bridges (vPPB) as the per-host view, and PPBs on the physical side. The reason the abstraction exists is that host software was written for a tree it owns, and pooling means it no longer owns one.

SLD and MLD. A Single Logical Device belongs to one hierarchy. A Multiple Logical Device can be divided: the Consortium describes MLD support in CXL 2.0 as allowing "a single device to be pooled across 16 Virtual Hierarchies." That is the hardware basis of memory pooling.

Fabric Manager. The agent that discovers the topology, configures it, and — in the Consortium's words — "binds a Logical Device to VCS vPPB". CXL 2.0 defines a switch API for multi-host or memory-pooled switch configuration and management. Section 12 discusses what it is and what this chapter deliberately does not claim about it.

6. Not All Three Protocols Route the Same Way

This is the fact most often missed, and it is specific enough to be worth quoting.

The CXL Consortium's CXL 2.0 material describes switch behaviour as: "CXL.IO same as PCIe decode"; the switch "does Address Look-Up and forwards request/response between Downstream Port(s) and RP for CXL.IO and CXL.Mem"; and — the striking one — "CXL.Cache no fanout. Software must enable Single Device with CXL.Cache in Virtual Hierarchy." The same material states that CXL.cache "is direct routed between CPU and device with a single caching device within a hierarchy." Responses, it notes, "always routed to vPPB."

Why .cache is the constrained one is worth reasoning about rather than memorising. .mem requests are memory accesses whose target is determined by an address, so fanning them across devices is an address-decode problem — the Consortium notes CXL 2.0 defines a memory decode mechanism supporting 2-way, 4-way and 8-way interleave. .cache traffic is a coherence conversation, and coherence requires a single agreed authority and a bounded set of participants. Fanning coherence out across a switch multiplies exactly the thing Chapter 3.1 showed to be expensive: the set of agents that must be snooped before a writer can be granted.

Version qualification matters here. Support for multiple Type 1 / Type 2 devices per root port is listed by the Consortium as a later-revision capability, not a CXL 2.0 one, and multi-level switching likewise arrives after 2.0. A statement like "a CXL switch fans out coherent traffic" is therefore not a fact about CXL — it is a fact about a revision, and the revision has to be named.

7. What Each Revision Added

Version-qualified from CXL Consortium material, and deliberately coarse — the detail belongs to Modules 20 and 21.

CXL 3.1's fabric work is worth one more sentence because it changes the shape of what is possible. The Consortium describes Port Based Routing (PBR) in contrast to Hierarchy Based Routing (HBR): PBR supports "fabric topologies other than tree topologies that HBR switches offer", using "address-based, non-prescriptive routing for large memory fabrics", and supporting "tree, mesh, ring, star, butterfly, and multi-dimensional topologies". It also describes fabric-attached memory with a Global Integrated Memory concept, direct peer-to-peer .mem through PBR switches, and an FM API for PBR switches.

The architectural reading: HBR is a tree because PCIe is a tree, and a tree is what inherited enumeration gives you for free (Chapter 2.4). Escaping the tree is what a memory fabric at rack scale requires, and it is a genuine departure rather than an extension — which is why it needed its own routing model.

8. Quantitative Reasoning — Oversubscription

The fabric's defining arithmetic, and it is unforgiving.

N ingress ports each offering bandwidth B into one egress of capacity E:

offered   =  N × B
drained   =  E
if N × B > E, the difference must go somewhere

There are exactly three destinations for that difference, and a fabric picks one: queue it, backpressure the sources, or drop it. A memory fabric cannot drop — a lost memory request is a hang, not a retransmission — so real designs queue and then backpressure, which is what Section 11 measures.

Queueing does not remove oversubscription, it delays it. A queue of depth D absorbs a burst of D and no more; sustained oversubscription fills any finite queue in D / (N·B − E) cycles and then the backpressure arrives anyway. The measured run took a 4-into-1 ratio and filled a 6-deep queue, spending 13 of 24 cycles refusing pushes.

What the queue actually buys is tolerance of burstiness, not of rate. That distinction is the one to carry: sizing a queue against average load is how a fabric ends up permanently full.

By Little's Law the occupancy tracks arrival rate times residence time, so a fabric that is congested shows it as depth, and the depth is what turns into latency for everything behind it — including traffic destined for an entirely uncongested port.

9. RTL 1 — The Route Table

Purpose

Answer "which downstream port owns this address?", and detect a table that cannot answer it correctly.

// A small programmable routing table: which downstream port owns this address?
//
// Routing state is CONFIGURATION, and configuration is the part of a fabric
// that software gets wrong. This table therefore checks the table itself, not
// just its output.
//
// GENERIC teaching model. This is NOT a CXL decoder or any specification-
// defined address-routing mechanism.
module route_table #(
  parameter int unsigned NENT  = 4,
  parameter int unsigned NPORT = 4
) (
  input  logic        clk,
  input  logic        rst_n,
  // configuration port
  input  logic        cfg_we,
  input  logic [1:0]  cfg_idx,
  input  logic        cfg_valid,
  input  logic [31:0] cfg_base,
  input  logic [31:0] cfg_limit,
  input  logic [1:0]  cfg_port,
  input  logic [3:0]  cfg_perm,      // one bit per host: may this host use it?
  // lookup port
  input  logic        lk_valid,
  input  logic [31:0] lk_addr,
  input  logic [1:0]  lk_host,
  output logic        hit,
  output logic [1:0]  hit_port,
  output logic        unmapped,
  output logic        denied,        // matched, but this host is not permitted
  output logic        overlap_err,   // two entries claim the same address
  output logic        cfg_bad_err    // an entry with limit below base
);
  logic [NENT-1:0] match;
 
  always_comb begin
    for (i = 0; i < NENT; i = i + 1)
      match[i] = v_q[i] && (lk_addr >= base_q[i]) && (lk_addr <= lim_q[i]);
  end
 
  // Priority resolves the OUTPUT even when the table overlaps -- which is
  // exactly why the overlap must be detected on `match`, not on `hit_port`.
  // ... nmatch and first_idx computed from `match` ...
 
  assign hit      = lk_valid && first_found && perm_q[first_idx][lk_host];
  assign denied   = lk_valid && first_found && !perm_q[first_idx][lk_host];
  assign unmapped = lk_valid && !first_found;
  assign hit_port = port_q[first_idx];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      // Deny and unmap by default. An unconfigured fabric routes nothing.
    end else begin
      if (cfg_we) begin
        // ... write the entry ...
        // An inverted range matches nothing and looks like a missing device.
        if (cfg_valid && (cfg_limit < cfg_base)) cfg_bad_err <= 1'b1;
      end
      // Detect the TABLE being wrong, not the lookup output.
      if (lk_valid && (nmatch > 4'd1)) overlap_err <= 1'b1;
    end
  end
endmodule

Everything is denied and unmapped out of reset. An unconfigured fabric that routes nothing is safe; one that defaults to routing somewhere is a system where a partially applied configuration delivers requests to a resource nobody assigned.

Two distinct configuration errors get distinct flags, and this is the fabric-specific lesson. overlap_err says two entries claim an address; cfg_bad_err says an entry can never match. Both are software mistakes. Neither is visible in the lookup output.

Simulation evidence — a clean table

=== EXP1: a clean route table ===
  own range               host 0 -> hit=1 port=0 unmapped=0 denied=0
  another host's range    host 0 -> hit=0 port=1 unmapped=0 denied=1
  shared range            host 3 -> hit=1 port=2 unmapped=0 denied=0
  nothing configured      host 0 -> hit=0 port=0 unmapped=1 denied=0
  overlap_err=0 cfg_bad_err=0

Row two is the interesting one: the address matched an entry and the host was not permitted, so denied asserts rather than unmapped. Those are different diagnoses — "there is no such memory" versus "that memory is not yours" — and a fabric that reports one for both makes an isolation problem look like a mapping problem.

Simulation evidence — a reprogramming that leaves an overlap

=== EXP2: reprogramming leaves an overlap ===
  inside the overlap      host 1 -> hit=1 port=1 unmapped=0 denied=0
  overlap_err=1  <-- the lookup looked perfectly normal

The lookup is indistinguishable from a correct one. Hit asserted, a port was selected, no error appeared on any output a requester can see. Priority picked the first matching entry, which is a defensible answer and possibly the wrong port entirely.

This is Chapter 3.1's tautological-assertion finding at fabric scale, with an important difference: the host's address map is programmed once at boot, and a fabric's routing table is reprogrammed while the system runs — every time a resource is reassigned. The window in which an overlap can appear is not a boot-time window; it is every reconfiguration, forever.

Simulation evidence — an inverted range

=== EXP3: an inverted range ===
  inside the intended range  host 0 -> hit=0 port=0 unmapped=1 denied=0
  cfg_bad_err=1  <-- the device looks absent

An entry with limit < base matches nothing, so every access to the intended range reports unmapped. The symptom is a missing device, and the investigation starts at the device, the link, and the enumeration — three places where nothing is wrong. One comparator at configuration time redirects it to the actual cause.

10. RTL 2 — Arbitration

Purpose

Several upstream ports want the same downstream port. Someone must not go first.

// Several upstream ports converge on one downstream port.
//
// MODE 0 -- fixed priority by port index. The default thing to write.
// MODE 1 -- round robin: the winner moves to the back.
//
// GENERIC teaching model; this is NOT a CXL switch arbitration mechanism.
module port_arbiter #(
  parameter int unsigned NREQ = 4
) (
  input  logic            clk,
  input  logic            rst_n,
  input  logic            mode_rr,
  input  logic [NREQ-1:0] req,
  input  logic            dst_ready,     // downstream can accept this cycle
  output logic [NREQ-1:0] gnt,
  output logic [7:0]      wait_q  [NREQ-1:0],
  output logic [7:0]      max_wait_q,
  output logic            multi_grant_err
);
  logic [1:0] ptr_q;
 
  always_comb begin
    gnt = '0; found = 1'b0;
    if (dst_ready) begin
      if (!mode_rr) begin
        for (i = 0; i < NREQ; i = i + 1)
          if (!found && req[i]) begin gnt[i] = 1'b1; found = 1'b1; end
      end else begin
        for (j = 0; j < NREQ; j = j + 1) begin
          k = (ptr_q + j) % NREQ;            // start from the rotating pointer
          if (!found && req[k]) begin gnt[k] = 1'b1; found = 1'b1; end
        end
      end
    end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      if ($countones(gnt) > 1) multi_grant_err <= 1'b1;
      for (i = 0; i < NREQ; i = i + 1) begin
        if (gnt[i])                            wait_q[i] <= 8'd0;
        else if (req[i] && wait_q[i] != 8'hFF) wait_q[i] <= wait_q[i] + 8'd1;
        // The maximum wait any port ever experienced -- the number that matters.
        if (req[i] && !gnt[i] && (wait_q[i] + 8'd1 > max_wait_q))
          max_wait_q <= wait_q[i] + 8'd1;
      end
      // The winner goes to the back. This one line is the whole difference.
      for (i = 0; i < NREQ; i = i + 1)
        if (gnt[i]) ptr_q <= (i[1:0] + 2'd1);
    end
  end
endmodule

dst_ready gates the grant, which is the coupling that makes a fabric a fabric. Downstream congestion does not merely slow one path — it stops the arbiter from granting at all, so a stalled device on one port removes service from every port contending for it. Section 14's waveform shows exactly this.

Timing. Grant is combinational from req and dst_ready, and it feeds the egress queue's push path. In a real switch this is a critical path and it is why arbitration is usually pipelined; the flat version is kept here because pipelining changes the timing and not the argument.

Simulation evidence

Both modes on identical stimulus — four ports asking every cycle for 200 cycles:

=== EXP4: four upstream ports, one downstream, 200 cycles ===
  fixed priority : p0=200 p1=0 p2=0 p3=0   max wait=200
  round robin    : p0=50  p1=50 p2=50 p3=50 max wait=3

Three ports received nothing at all. Not less, not later — nothing, for the entire run, with max_wait saturating at 200 cycles. Round robin gave each port exactly a quarter and bounded the worst wait at 3 cycles, which is NREQ − 1 and is the design's actual service guarantee.

The general result is the same one Chapter 2.4 measured on protocol classes, and its recurrence is the point: a priority chain is the default thing an engineer writes, and it has no fairness property at all. At fabric scale the consequence is worse, because the starved party is not a traffic class inside your chip — it is another customer's host.

11. RTL 3 — The Egress Queue and Backpressure

Purpose

Absorb burstiness, and tell the sources when you cannot.

// The output queue on a downstream port, and the backpressure it produces.
//
// This is where oversubscription becomes visible: when several ingress ports
// offer more than one egress can drain, the queue fills and `ready` drops,
// which is the fabric telling its sources to slow down.
//
// GENERIC teaching model.
module egress_queue #(
  parameter int unsigned DEPTH = 6
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        push,
  input  logic [7:0]  push_id,
  input  logic        pop_ready,      // the downstream device can take one
  output logic        ready,          // this port can accept a push
  output logic        pop_valid,
  output logic [7:0]  pop_id,
  output logic [7:0]  level_q,
  output logic [7:0]  max_level_q,
  output logic [15:0] bp_cycles_q,    // cycles spent refusing pushes
  output logic        overflow_err
);
  assign ready     = (level_q < DEPTH[7:0]);
  assign pop_valid = (level_q != 0) && pop_ready;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      if (!ready) bp_cycles_q <= bp_cycles_q + 16'd1;
      // ... push / pop / simultaneous, tracking max_level_q ...
      if (level_q > DEPTH[7:0]) overflow_err <= 1'b1;
    end
  end
endmodule

ready must be honoured by the pusher, not merely reported. A queue whose ready is advisory is a queue that overflows, and in a memory fabric an overflow is a lost request — which presents as a host-side timeout on a transaction that no longer exists anywhere in the system. That is why overflow_err exists as an assertion target rather than as a recoverable condition: reaching it means a contract was already broken upstream.

max_level_q is the sizing number and bp_cycles_q is the pain number. The first says how deep the queue ever got; the second says how long it spent refusing. A queue that never approaches its depth is oversized; one that pins at depth is undersized or the port is genuinely oversubscribed, and only the ratio of offered to drained traffic distinguishes those.

Simulation evidence

A 4-into-1 oversubscription: sources push every cycle, the downstream device accepts one in four.

=== EXP5: oversubscription -- 4 ingress offering, 1 egress draining ===
  after 24 cycles: level=6 max_level=6 backpressure_cycles=13 ready=0
  -> the queue is the fabric telling its sources to slow down
  after draining : level=0 ready=1

The queue reached its depth and stayed there, refusing pushes on 13 of 24 cycles. That is Section 8's arithmetic in measured form: a 6-deep queue against a sustained 4:1 ratio absorbs the first burst and then contributes nothing further — it has converted a rate problem into a rate problem plus 6 entries of latency.

Note what the queue did not do. It never dropped, and overflow_err never fired, because the pusher honoured ready. The system degraded to the drain rate, which is the correct behaviour and is also why the sources have to be built to be told no.

12. RTL 4 — Isolation, and the Fabric Manager

Purpose

A pool that everyone can reach is not a pool.

Pooling means resources are assigned, and assignment is meaningless without the ability to refuse. The CXL Consortium describes the Fabric Manager as the agent that binds logical devices to a host's view — "FM Binds a Logical Device to VCS vPPB" — with an API defined for multi-host and memory-pooled switch configuration.

// Host x resource authorisation.
//
// A composable fabric assigns resources to hosts, which means it must also be
// able to REFUSE. Without this, "pooling" means every host can reach every
// resource, which is not a pool -- it is a shared bus with extra latency.
//
// GENERIC teaching model. NOT a CXL access-control mechanism; no CXL
// permission encoding, binding structure or Fabric Manager command is modelled.
module access_check #(
  parameter int unsigned NHOST   = 4,
  parameter int unsigned NRES    = 4,
  parameter bit          ENFORCE = 1'b1     // 0 = check computed but ignored
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       bind_we,
  input  logic [1:0] bind_host,
  input  logic [1:0] bind_res,
  input  logic       bind_allow,
  input  logic       req_valid,
  input  logic [1:0] req_host,
  input  logic [1:0] req_res,
  output logic       permit,
  output logic       refuse,
  output logic       cross_host_leak_err,
  output logic [15:0] n_refused_q
);
  assign authorised = allow_q[req_host][req_res];
 
  // ENFORCE=0 is the bug shape: the permission is computed and then not used.
  assign permit = req_valid && (ENFORCE ? authorised : 1'b1);
  assign refuse = req_valid && !permit;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (i = 0; i < NHOST; i = i + 1) allow_q[i] <= '0;   // deny by default
      cross_host_leak_err <= 1'b0; n_refused_q <= '0;
    end else begin
      if (bind_we) allow_q[bind_host][bind_res] <= bind_allow;
      if (refuse)  n_refused_q <= n_refused_q + 16'd1;
      // Something was permitted that the binding table does not authorise.
      if (permit && !authorised) cross_host_leak_err <= 1'b1;
    end
  end
endmodule

cross_host_leak_err compares the decision against the table, which is the only way to catch a check that is computed and then ignored. Asserting that permit follows from permit would pass on the broken design; the property has to name the authorisation source independently.

Simulation evidence

Two instances, identical bindings, identical requests:

=== EXP6: host/resource isolation ===
  host 0 -> its own resource 0 : enforcing permit=1 | ignored permit=1
  host 0 -> host 1's resource 1: enforcing permit=0 | ignored permit=1  <-- leak
  host 3 -> unbound resource 2 : enforcing permit=0 | ignored permit=1
  refusals: enforcing=2  ignored=0

Row one is identical between the two designs, which is the structural problem: every legitimate access behaves the same, so the non-enforcing fabric passes any test written from the perspective of a correctly-configured host.

Rows two and three are the two distinct leaks. One host reaches another host's assigned memory; a third host reaches memory nobody assigned. In a multi-tenant deployment those are not performance bugs.

13. RTL 5 — Fabric Counters

Purpose

The fabric is between everything, so it is where congestion is visible and nowhere else is.

// Per-port fabric instrumentation. A fabric without these is a black box in
// exactly the place congestion happens.
// GENERIC teaching model.
module fabric_counters #(
  parameter int unsigned NPORT = 4
) (
  input  logic             clk,
  input  logic             rst_n,
  input  logic [NPORT-1:0] req_pulse,
  input  logic [NPORT-1:0] gnt_pulse,
  input  logic             egress_full,
  input  logic             unmapped_pulse,
  input  logic             denied_pulse,
  output logic [15:0]      n_req_q  [NPORT-1:0],
  output logic [15:0]      n_gnt_q  [NPORT-1:0],
  output logic [15:0]      full_cycles_q,
  output logic [15:0]      max_full_run_q,
  output logic [15:0]      n_unmapped_q,
  output logic [15:0]      n_denied_q
);
  // ... per-port request and grant counts; full-cycle total AND maximum run ...
endmodule

Requests and grants are counted separately per port on purpose. Their ratio is the service rate that port actually received, and a port whose request count is high and grant count is low is being starved — which is the fixed-priority signature and is not visible from a total.

n_unmapped_q and n_denied_q are separated for the reason Section 9 gave: one means the map is wrong and the other means the binding is. Same symptom to a requester, different team.

Simulation evidence

=== counters ===
  egress full cycles=14  max full run=3  unmapped=2  denied=1

The maximum full run is the number that matters for latency. A port full for 14 cycles in short bursts of 3 is a healthy fabric under load; the same 14 cycles as a single run is a stall that every requester behind it experienced in full.

14. Waveform — Two Hosts, One Downstream Port, and a Stall

Three things are worth reading off this trace.

The hosts did not feel the stall for four cycles. dst_ready fell at cycle 4 and the grant continued through cycle 8, because the queue had room. That delay is the queue's entire function, and it is also why a host-side latency spike is later than the event that caused it — which sends the investigation to the wrong cycle.

When the queue filled, both hosts stopped, including the one whose traffic was not responsible for filling it. Congestion at a fabric port is shared by everyone contending for that port, which is the fabric's version of Chapter 3.2's head-of-line result and the reason isolation of bandwidth is a separate problem from isolation of access.

Round robin held throughout. The alternation continues across the stall and resumes after it, so no host was disadvantaged by the congestion — the fairness property and the flow-control mechanism are independent, and each has to be built.

15. Assertions

Icarus does not execute concurrent SVA, so these were not run; the table gives the procedural check.

// F1 — TABLE property: no two entries claim the same address. Asserting the
// lookup output is one-hot passes on an overlapping table, because priority
// already guarantees it.
a_table_disjoint: assert property (@(posedge clk) disable iff (!rst_n)
  lk_valid |-> $onehot0(match));
 
// F2 — every lookup resolves to exactly one of hit / denied / unmapped.
a_lookup_total: assert property (@(posedge clk) disable iff (!rst_n)
  lk_valid |-> $onehot({hit, denied, unmapped}));
 
// F3 — a configured entry can actually match something.
a_range_sane: assert property (@(posedge clk) disable iff (!rst_n)
  (cfg_we && cfg_valid) |-> (cfg_limit >= cfg_base));
 
// F4 — one grant per cycle, and never without downstream capacity.
a_grant_legal: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0(gnt) && (|gnt |-> dst_ready));
 
// F5 — LIVENESS: no upstream port waits forever. This is what fixed priority
// violates, and no safety property can express it.
a_no_starvation: assert property (@(posedge clk) disable iff (!rst_n)
  req[p] |-> ##[1:BOUND] gnt[p]);
 
// F6 — the egress queue is bounded and never accepts when full.
a_queue_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (level_q <= DEPTH) && (!ready |-> !(push && ready)));
 
// F7 — permission always follows from the binding table, not from itself.
a_permit_authorised: assert property (@(posedge clk) disable iff (!rst_n)
  permit |-> allow_q[req_host][req_res]);
 
// F8 — nothing is permitted before it is bound.
a_deny_by_default: assert property (@(posedge clk)
  $fell(rst_n) |=> !permit until_with bind_we);

F1 and F7 share a shape worth naming. Both check a decision against its source rather than against itself. Asserting $onehot0({hit_port…}) or permit |-> permit would pass on both broken designs, because in each case the output was made well-formed by logic that runs after the mistake.

F5 is liveness, and it is the only property that separates fixed priority from round robin. Every safety property in this table passes on a fabric that gave three of four ports nothing for 200 cycles.

16. Debug Lab

// The lookup output is one-hot, so the table must be consistent.
assert property (lk_valid |-> $onehot0({to_p0, to_p1, to_p2, to_p3}));

  inside the overlap    host 1 -> hit=1 port=1 unmapped=0 denied=0
  overlap_err=1  <-- the lookup looked perfectly normal

a_table_disjoint: assert property (lk_valid |-> $onehot0(match));
// and in RTL, so the running system can report it:
if (lk_valid && (nmatch > 1)) overlap_err <= 1'b1;

// Serve the lowest-numbered requesting port.
for (i = 0; i < NREQ; i++)
  if (!found && req[i]) begin gnt[i] = 1'b1; found = 1'b1; end

  fixed priority : p0=200 p1=0 p2=0 p3=0   max wait=200
  round robin    : p0=50  p1=50 p2=50 p3=50 max wait=3

k = (ptr_q + j) % NREQ;                     // start from the pointer
if (!found && req[k]) begin gnt[k] = 1'b1; found = 1'b1; end
...
for (i = 0; i < NREQ; i++) if (gnt[i]) ptr_q <= i + 1;

assign authorised = allow_q[req_host][req_res];   // computed
assign permit     = req_valid;                    // ... and not used

  host 0 -> its own resource 0 : enforcing permit=1 | ignored permit=1
  host 0 -> host 1's resource 1: enforcing permit=0 | ignored permit=1  <-- leak
  host 3 -> unbound resource 2 : enforcing permit=0 | ignored permit=1

assign permit = req_valid && authorised;
// the property must name the table independently:
a_permit_authorised: assert property (permit |-> allow_q[req_host][req_res]);
if (permit && !authorised) cross_host_leak_err <= 1'b1;

// Average offered load is below drain rate, so a small queue is plenty.
egress_queue #(.DEPTH(6)) u_eq (...);

  after 24 cycles: level=6 max_level=6 backpressure_cycles=13 ready=0

// If max_level_q pins at DEPTH and bp_cycles_q grows without bound, the answer
// is not a deeper queue -- it is more egress capacity or fewer sources.

// Program the entry from the descriptor software supplied.
base_q[idx] <= cfg_base;
lim_q[idx]  <= cfg_limit;      // no check that limit >= base

  inside the intended range  host 0 -> hit=0 port=0 unmapped=1 denied=0
  cfg_bad_err=1  <-- the device looks absent

if (cfg_valid && (cfg_limit < cfg_base)) cfg_bad_err <= 1'b1;

17. Design Review — Reading a Fabric Implementation

On routing. Is any property checking the raw match vector rather than the priority-resolved output? Is the whole table revalidated on every write, or only the entry being written? Are unmapped and denied distinguishable to a requester? Is a malformed range rejected at configuration time? What does the table route out of reset?

On arbitration. Is there a fairness mechanism at all, and is the liveness property instantiated for every port or only port 0? What is the measured worst-case wait, and is it a number the design can state? Does dst_ready gate the grant, and is that coupling understood as "one stalled device removes service from every contender"?

On queueing. Is ready binding or advisory? Is the queue sized for burst or for rate? Are max_level and backpressure_cycles both instrumented, and are they read together?

On isolation. Is the authorisation in the decision path or merely computed? Does the property name the binding table independently? Is the default deny? Are negative tests generated systematically?

On observability. Are requests and grants counted per port so the service ratio is visible? Are maxima recorded, not just totals?

And the structural question this chapter adds. Which of this fabric's behaviour is configuration rather than design? Everything in that set can be wrong while every link is healthy and every device is correct, and it is where the fabric's characteristic failures live.

18. How This Appears in Real Engineering

CXL / system architect

The fabric decisions are topology-shaped: how many hosts share a port, what oversubscription ratio is acceptable, and which revision's capabilities the design depends on. That last one is a commitment, not a preference — multi-level switching and multiple coherent devices per root port are later-revision capabilities, so a design that assumes them has chosen a floor.

RTL engineer

Five blocks and five disciplines. Check the routing table, not the lookup. Give arbitration a rotating pointer and a bounded wait. Make ready binding. Put the authorisation in the decision path. Count requests and grants separately per port.

Verification engineer

The stimulus that finds these defects is specific: an address inside a reprogrammed overlap, saturating traffic on every port simultaneously, sustained oversubscription rather than bursts, and every host against every resource it was not bound to. Three of the five measured defects are invisible to any test written from the perspective of a correctly-configured, uncontended system.

Performance engineer

Read n_req and n_gnt per port as a ratio — that is the service each port actually received, and starvation is invisible in a total. Then read max_level and bp_cycles together: burst absorption and genuine oversubscription look the same in an average and different in those two numbers.

Firmware and fabric management

Every table this chapter models is software-written, so most of its failure modes are yours. Two habits follow: revalidate the entire routing table on every write, because overlap is a property of pairs; and treat unmapped and denied as distinct diagnostics to surface, because they route the investigation to different teams.

System integrator

The fabric is where multi-tenant guarantees live or fail. Access isolation and bandwidth isolation are separate problems with separate mechanisms — the measured trace shows both hosts stalled by one congested port even with fairness working perfectly, which is a bandwidth-isolation gap that no permission table addresses.

19. Common Misconceptions

20. Interview Reasoning

21. Exercises

1. Compute the fill time. A port has four sources each offering 1 request per cycle into an egress that drains one every three cycles. How many cycles until a 16-deep queue is full? Now derive the general expression and explain why the answer does not depend on the burst pattern once the ratio is sustained.

2. Break F1 differently. The measured overlap came from adding an entry. Construct an overlap produced by widening an existing entry, and confirm that a per-entry configuration check misses it while the raw-match property still fires.

3. Bound the wait. Round robin gave a worst case of NREQ − 1. What is the worst case if dst_ready is low half the time in an adversarial pattern? Is the liveness property still satisfiable, and what assumption does it need?

4. Design the negative test. For 4 hosts and 4 resources with 4 bindings, enumerate the negative isolation tests. How does the count grow, and at what fabric size does exhaustive stop being tractable? What would you do then?

5. Separate the two isolations. The measured trace shows access isolation working and bandwidth isolation absent. Sketch what a bandwidth reservation would have to add to the arbiter, and state what it costs the tenant who was previously getting the spare capacity.

22. Summary

A fabric turns "the device" into "which device", and the cost of that question is three kinds of software-written state: routing, arbitration and binding. Each fails while every link is healthy.

Routing. A reprogrammed table with an overlap produced a lookup indistinguishable from a correct one — hit asserted, port selected, no requester-visible error — because priority makes the output well-formed regardless. Only a property on the raw match vector fires. And an inverted range made a present, healthy device report as absent.

Arbitration. Fixed priority gave one port all 200 grants and three ports zero, with every safety property passing. Round robin gave each a quarter with a bounded worst wait of 3 cycles. The liveness property is the only thing that separates them, and instantiating it on port 0 alone makes it a tautology.

Queueing. Under a sustained 4:1 ratio a 6-deep queue filled and spent 13 of 24 cycles refusing pushes. A queue absorbs burstiness, not rate — and the measured waveform shows the second consequence: the hosts did not feel the downstream stall for four cycles, so the host-side symptom is dated later than its cause.

Isolation. A permission computed and not enforced behaved identically for every legitimate access and leaked in both directions — one host into another's bound resource, and any host into an unbound one.

Two protocol-specific facts are worth carrying. The three CXL protocols do not route alike: Consortium material describes .io decoded as PCIe, .mem with fan-out and interleave, and .cache direct routed with a single caching device per hierarchy in CXL 2.0. And fabric capability is revision-specific — single-level switching at 2.0, multi-level and multiple coherent devices per root port at 3.0, non-tree topologies via Port Based Routing at 3.1 — so any claim about what a CXL fabric can do has to name a revision.

Finally, the property shape this chapter shares with Chapter 3.1: check a decision against its source, not against itself. The one-hot lookup and the self-referential permission both pass on broken designs for the same reason — the output was made well-formed by logic that runs after the mistake.

23. What Comes Next

Three chapters have now described where things are: the host, the device, and the fabric between them. None has described what they say to each other when a line of memory has to stay coherent across all three.

Chapter 3.4 is that conversation. It starts from one cache line and one question — who has the authoritative value, and who may change it — and builds the communication model from there, including the asymmetry that Consortium material describes when it says the host processor orchestrates the coherency management.

For adjacent material: The CXL Host has the home agent this chapter's .cache routing constraint protects, The CXL Device has the endpoint behind each downstream port, and Evolution of CXL has the revision chronology this chapter version-qualifies against. The path is on the CXL tutorials index.