Fabric Problems

26.4 was about an access that completed and went to the wrong place. This chapter is about an access that never goes anywhere at all — and the difference matters because the fabric will draw you a picture of a perfectly healthy path while it happens.

The topology dump shows five switches, every link up, every route programmed. Sixty-four gigabytes a second of offered traffic is moving at zero — and every tool in the rack says the destination is reachable.

1. The Engineering Problem — Reachable Is Not Usable

A path that exists is not a path that carries traffic. One hop with no credit to give leaves the route intact, the link trained, the destination pingable and the whole offered load stalled. Section 5.

Credit is granted per hop, so an end-to-end path runs at its slowest switch. One switch in the middle granting a quarter of the credit turns a sixty-four gigabyte link into sixteen, and the datasheet still says sixty-four. Section 6.

Two switches, two routing tables, one destination. Four entries out of sixty-four programmed differently at the far end is forty packets in a thousand arriving somewhere nobody chose — and reading the near table shows nothing at all. Section 7.

A cycle does not drop packets. It keeps them. Twenty packets caught in a three-switch loop with a sixteen-hop limit burn three hundred and twenty hops of somebody else's bandwidth before they retire. Section 8.

The flows that complain are not the flows that are broken. Sixteen flows on two virtual channels with one congested destination puts seven uninvolved flows behind it, and those seven file the ticket. Section 9.

This chapter against 26.2, stated precisely. That one owns a link that drops and comes back. This one owns a link that never drops and never delivers — which is why every model here is about what a path does rather than whether it is there, and why section 14's weak definition is a topology dump.

2. The One-Sentence Model

A destination is reachable in the sense that matters when a route to it is programmed, every hop on that route has credit to give, both ends compute the same next hop, the path does not revisit a switch, no unrelated flow is parked in front of it, and returns match consumption over time — and "there is a path" is one of those six.

3. What This Chapter Owns

The boundary is sharper than it looks. A fabric problem and a performance problem produce the same first complaint — this is slower than it should be — and they are separated by a single question: is the shortfall proportional, or is it total? A path running at a quarter of its rate because one switch grants a quarter of the credit is section 6, here. A path running at ninety percent of its rate because a queue is slightly too shallow is 26.6. The instrument that tells them apart is per-hop credit, and the reason engineers reach for it late is that the topology dump looks so convincing.

The fabric is the only layer in CXL where the failure and the symptom are reliably in different places. A discovery failure is at the device that did not enumerate. A coherency bug is at the line that went stale. A fabric problem is at a switch nobody is looking at, reported by a host nobody suspects, on behalf of a flow that has nothing to do with the fault. Section 9 is entirely about that displacement, and section 13 is about the cheapest way to undo it.

4. Teaching-Model Boundary

Every model in this chapter is a teaching model, not a switch. It computes the one relationship the section is about and nothing else. There is no packet, no flit, no arbiter, no virtual-channel allocator and no crossbar anywhere in this file.

Each model is built twice from one source. A parameter selects between the measured build, which computes what is actually true of the fabric, and the reported build, which computes what a particular instrument would say. The two are instantiated side by side against identical stimulus, and every section's headline number is the gap between them.

The error output in each model is deliberately shaped: it separates what is true of the fabric from what the build reports about it, and fires only when the second contradicts the first. That shape is why a mutation to either half is caught rather than absorbed.

5. RTL 1 — A Path That Exists Is Not A Path That Carries Traffic

Start with the case that wastes the most engineering time in a CXL fabric, because every instrument agrees and every instrument is answering a different question than the one you asked.

A topology dump answers is there a route from here to there. It walks the fabric, finds each switch, reads each routing table and draws a line. That line is real. The route is programmed, the links are trained, the ports are up. Ask the fabric manager and it will tell you the destination is reachable, and it will be telling the truth.

Traffic does not move along routes. Traffic moves along credit. A CXL switch forwards a flit only when the next hop has advertised room to take it, and a hop that has advertised nothing takes nothing — not slowly, not eventually, not at a reduced rate. The route stays perfectly intact while zero bytes cross it.

This model computes both facts and keeps them apart. Reachability is a property of hops; usable bandwidth is a property of the scarcest credit on the path. The measured build requires both. The reported build — the one named for the topology dump — requires only the first.

// RTL 1 - reachability against usable bandwidth. A path that exists is not a
// path that carries traffic: a hop with no credit is perfectly reachable and
// completely useless, and only one of those two facts shows up in a topology
// dump.
module fabric_reach #(parameter int THERE_IS_A_PATH = 0) (
  input  logic clk, rst_n,
  input  logic        probe,
  input  logic [15:0] hops, hops_reachable, min_credit, offered_bw,
  output logic [15:0] live_hops, usable_bw, stalled_bw, usable_pct,
  output logic        path_usable,
  output logic [7:0]  n_probes, n_unusable,
  output logic        reach_only_err
);
  logic [31:0] u_q;
  logic        path_live, truly_unusable;
  // A reachability report cannot name more live hops than the path has.
  assign live_hops  = (hops_reachable > hops) ? hops : hops_reachable;
  assign path_live  = (hops != 16'd0) && (live_hops == hops);
  // Credit, not reachability, decides whether anything moves.
  assign usable_bw  = (path_live && (min_credit != 16'd0)) ? offered_bw : 16'd0;
  assign stalled_bw = offered_bw - usable_bw;
  assign u_q = (offered_bw == 16'd0) ? 32'd0
             : (({16'd0, usable_bw} * 32'd100) / {16'd0, offered_bw});
  assign usable_pct = (u_q > 32'd100) ? 16'd100 : u_q[15:0];
  // What is true, kept separate from what the build reports.
  assign truly_unusable = path_live && (min_credit == 16'd0);
  assign path_usable = (THERE_IS_A_PATH != 0)
                       ? path_live
                       : (path_live && (min_credit != 16'd0));
  // A path every tool calls up, carrying nothing.
  assign reach_only_err = probe && truly_unusable && path_usable;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_probes <= 8'd0; n_unusable <= 8'd0;
    end else if (probe) begin
      n_probes <= n_probes + 8'd1;
      if (usable_bw == 16'd0) n_unusable <= n_unusable + 8'd1;
    end
  end
endmodule

The two builds are instantiated against the same four-hop path. Give every hop credit and they agree: sixty-four gigabytes a second offered, sixty-four moving, a hundred percent of the load delivered, both builds calling the path usable. Take the credit away from one hop and nothing about the route changes — four hops, all reachable, the same trained links — but the whole sixty-four gigabytes stalls, and the topology view still calls the path up.

Figure 1 — the same four-hop path measured two ways. The topology read on the upper path is correct about everything it looks at: four hops, all reachable, the route programmed end to end. The credit read on the lower path finds one hop advertising nothing, and the delivered rate is zero. Neither measurement contradicts the other; they answer different questions, and only one of them is the question the traffic cares about.

The fourth stimulus case is the one worth sitting with. Two of five hops reachable is a genuinely broken path, and both builds get it right — the route is not there, so neither model claims it is usable. That case is easy, it is caught by everything, and it is therefore not the case that reaches a debug session. The case that reaches a debug session is the one where the route is perfect.

The model also refuses to be fooled by its own inputs. A reachability probe that reports nine live hops on a three-hop path is clamped to three, because a count of live hops on a path cannot exceed the path. That clamp is not decoration: an over-counting probe is a common artefact of a fabric manager that walks a topology while it is changing, and a model that accepts nine would compute a path that is not live and report a fault that is not there.

Two degenerate inputs are driven deliberately. A fabric with no hops at all is not usable, and neither build claims otherwise — an absent path is a different fault, owned by 26.1. A live path with credit and no traffic offered to it is usable and is moving nothing, and the model must not confuse an idle path with a stalled one. Both cases are where a careless model computes a percentage of zero and reports a hundred.

6. RTL 2 — The Narrowest Hop Sets The Rate

Section 5 was the total case: one hop at zero credit, nothing moves. The proportional case is more common and much harder to see, because the fabric does deliver, just not what the datasheet says.

Credit in CXL is per hop and per virtual channel. A flit crossing three switches is admitted three times, and each admission is governed by the credit that hop has advertised. The end-to-end rate is the minimum of those three, not the rate of the link underneath them. A switch configured with a small shared buffer, or one whose credit returns are slow enough to keep its advertised pool shallow, sets the rate for every path that crosses it.

The measured build takes the minimum across the hops and clamps it to the link rate, because no hop delivers more than the wire it sits on. The reported build quotes the line rate, which is what a datasheet does, what a link-status register does, and what nearly every first-pass bandwidth estimate does.

// RTL 2 - the narrowest hop sets the rate. Credit is granted per hop, so an
// end-to-end path runs at its slowest switch, not at its link rate.
module hop_credit_floor #(parameter int LINK_RATE_IS_PATH_RATE = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] line_rate, hop_rate_a, hop_rate_b, hop_rate_c,
  output logic [15:0] narrow_hop, fabric_rate, rate_gap, achieved_pct,
  output logic        rate_honest,
  output logic [7:0]  n_measures, n_overstated,
  output logic        credit_floor_err
);
  logic [15:0] min_ab, min_abc;
  logic [31:0] a_q;
  logic        truly_overstated;
  assign min_ab  = (hop_rate_a < hop_rate_b) ? hop_rate_a : hop_rate_b;
  assign min_abc = (min_ab < hop_rate_c)     ? min_ab     : hop_rate_c;
  // No hop can deliver more than the link underneath it.
  assign narrow_hop = (min_abc > line_rate) ? line_rate : min_abc;
  assign fabric_rate = (LINK_RATE_IS_PATH_RATE != 0) ? line_rate : narrow_hop;
  assign rate_gap = line_rate - narrow_hop;
  assign a_q = (line_rate == 16'd0) ? 32'd0
             : (({16'd0, narrow_hop} * 32'd100) / {16'd0, line_rate});
  assign achieved_pct = (a_q > 32'd100) ? 16'd100 : a_q[15:0];
  assign rate_honest = (fabric_rate <= narrow_hop);
  assign truly_overstated = (narrow_hop < line_rate);
  // The quoted number is the line rate and the fabric cannot deliver it.
  assign credit_floor_err = measure && truly_overstated
                            && (fabric_rate == line_rate);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_overstated <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (!rate_honest) n_overstated <= n_overstated + 8'd1;
    end
  end
endmodule

With every hop at line rate the two builds agree and the gap is zero. Put one switch in the middle at sixteen and the measured build quotes sixteen — twenty-five percent of line rate, a forty-eight gigabyte gap — while the link-rate view keeps quoting sixty-four on a fabric that can carry a quarter of it.

The clamp in this model runs the other way from section 5's and is worth reading carefully. A switch can be configured with more credit than the link beneath it can drain — that is a legal, common and entirely harmless configuration — and a model that reported a hundred gigabytes a second on a thirty-two gigabyte link would be manufacturing bandwidth out of a register value. The stimulus drives exactly that case: three hops at a hundred, eighty and sixty-four on a thirty-two gigabyte link, and the answer is thirty-two.

The last case is the one that connects this section back to the previous one. A hop granting no credit at all is not a separate failure mode; it is this one at its limit. The narrowest hop is zero, the path rate is zero, the gap is the entire link rate, and the link-rate view is still quoting sixty-four. Section 5 and section 6 are one phenomenon measured with different resolution, which is why an engineer who has internalised section 6 rarely spends a day on section 5.

The error output fires on a specific contradiction: the fabric is rate-limited by a hop, and the number being quoted is the line rate anyway. It does not fire because the quoted number is wrong in some general sense — it fires because the report is the link rate on a path the link rate cannot describe.

7. RTL 3 — Two Tables, One Destination

A route is not an object. It is an agreement between switches, held in two places, and nothing in the fabric enforces that the two copies say the same thing.

Reading a routing table tells you where that switch will send a packet for a given destination. It does not tell you where the packet arrives, because the next switch has its own table and its own opinion. When a fabric manager programs both and one write fails, or when a switch reloads a stale configuration after a reset, the two tables disagree for the entries the failure touched — and the fabric keeps forwarding, confidently, to the wrong place.

This model compares entries between the two ends and converts the disagreement into traffic, because a disagreement on an entry nothing uses costs nothing and a disagreement on a hot entry costs everything.

// RTL 3 - two switches, two routing tables, one destination. Reading the table
// at one end tells you where that switch sends a packet, not where the packet
// arrives.
module route_table_skew #(parameter int ONE_TABLE_IS_THE_FABRIC = 0) (
  input  logic clk, rst_n,
  input  logic        compare_it,
  input  logic [15:0] entries, entries_matching, packets, traffic_per_entry,
  output logic [15:0] agreeing, disagreeing, misroute_pkts, misroute_pct,
  output logic        routes_agree,
  output logic [7:0]  n_compares, n_skewed,
  output logic        table_skew_err
);
  logic [31:0] m_q, p_q;
  logic        truly_skewed;
  // A match count cannot exceed the number of entries compared.
  assign agreeing    = (entries_matching > entries) ? entries : entries_matching;
  assign disagreeing = entries - agreeing;
  assign m_q = {16'd0, disagreeing} * {16'd0, traffic_per_entry};
  assign misroute_pkts = (m_q > {16'd0, packets}) ? packets : m_q[15:0];
  assign p_q = (packets == 16'd0) ? 32'd0
             : (({16'd0, misroute_pkts} * 32'd100) / {16'd0, packets});
  assign misroute_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign truly_skewed = (entries != 16'd0) && (disagreeing != 16'd0);
  // Reading only the near switch's table cannot show a disagreement.
  assign routes_agree = (ONE_TABLE_IS_THE_FABRIC != 0)
                        ? 1'b1 : (disagreeing == 16'd0);
  assign table_skew_err = compare_it && truly_skewed && routes_agree;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_compares <= 8'd0; n_skewed <= 8'd0;
    end else if (compare_it) begin
      n_compares <= n_compares + 8'd1;
      if (truly_skewed) n_skewed <= n_skewed + 8'd1;
    end
  end
endmodule

Sixty-four entries with four of them programmed differently at the far switch is a four-entry disagreement and, at ten packets per entry against a thousand packets of traffic, forty packets landing somewhere nobody chose — four percent. The measured build reports the disagreement. The near-table view reports that the routes agree, because from where it is standing they do.

The saturation case matters more than it looks. Two tables that share nothing, on light traffic, produce a disagreement on every entry — but the misroute count is clamped to the packet count, because no packet can be misrouted twice. A model that multiplied sixty-four entries by ten packets and reported six hundred and forty misroutes out of a hundred packets would be reporting a number that cannot happen, and an engineer who saw it would stop trusting the model rather than the fabric.

The degenerate case is the one that separates this fault from its neighbour. A routing table with no entries in it is not a skewed table. It is an unprogrammed fabric, which is a different failure with a different owner and a different fix, and the model is explicit about it: the truth guard requires entries to exist before a disagreement between them can mean anything. Without that guard, every freshly reset switch in the rack would report a routing skew.

8. RTL 4 — A Cycle Does Not Drop Packets, It Keeps Them

The intuition that makes topology cycles hard is that a loop sounds like a place where packets go to die. They do not die. They circulate, and while they circulate they are indistinguishable from legitimate traffic at every hop they pass.

A CXL fabric with a cycle in it — usually from a redundant link added for availability and not excluded from the routing computation — forwards a packet from switch to switch until a hop limit retires it. Each of those hops consumes real credit and real bandwidth on a real link that other flows are trying to use. The cost of a loop is not the packets in it; it is the capacity those packets burn on their way to being discarded.

The measured build computes that burn. The tree view — the build named for a tool that assumes a fabric is acyclic, which most topology visualisers do — has no way to represent a cycle and therefore reports none.

// RTL 4 - a cycle in the topology. Packets on a looped path do not vanish and
// do not arrive: they circulate, burning the bandwidth of every hop they pass
// until a hop limit retires them.
module topology_cycle #(parameter int THE_FABRIC_IS_A_TREE = 0) (
  input  logic clk, rst_n,
  input  logic        inspect,
  input  logic [15:0] hop_limit, loop_len, pkts_entering, link_capacity,
  output logic [15:0] circulating, wasted_hops, capacity_burned, burn_pct,
  output logic        cycle_free,
  output logic [7:0]  n_inspects, n_cyclic,
  output logic        cycle_err
);
  logic [31:0] w_q, b_q;
  logic        truly_cyclic;
  assign truly_cyclic = (loop_len != 16'd0) && (pkts_entering != 16'd0);
  assign circulating  = truly_cyclic ? pkts_entering : 16'd0;
  // Each trapped packet burns one hop per step until the limit retires it.
  assign w_q = {16'd0, circulating} * {16'd0, hop_limit};
  assign wasted_hops = (w_q > 32'hFFFF) ? 16'hFFFF : w_q[15:0];
  assign capacity_burned = (wasted_hops > link_capacity)
                           ? link_capacity : wasted_hops;
  assign b_q = (link_capacity == 16'd0) ? 32'd0
             : (({16'd0, capacity_burned} * 32'd100) / {16'd0, link_capacity});
  assign burn_pct = (b_q > 32'd100) ? 16'd100 : b_q[15:0];
  // A tool that assumes a tree has no way to represent a cycle.
  assign cycle_free = (THE_FABRIC_IS_A_TREE != 0) ? 1'b1 : (loop_len == 16'd0);
  assign cycle_err = inspect && truly_cyclic && cycle_free;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_inspects <= 8'd0; n_cyclic <= 8'd0;
    end else if (inspect) begin
      n_inspects <= n_inspects + 8'd1;
      if (truly_cyclic) n_cyclic <= n_cyclic + 8'd1;
    end
  end
endmodule

Twenty packets in a three-switch loop with a sixteen-hop limit burn three hundred and twenty hops of capacity: thirty-two percent of a thousand-unit link, spent on twenty packets that will never arrive. The tree view reports an acyclic fabric on the same twenty packets.

Two clamps guard the arithmetic. A loop burning more than the link can carry is capped at the link capacity, because capacity that does not exist cannot be consumed. A hop limit and a packet count large enough to overflow the counter saturate rather than wrap — a saturated waste count is still a counted loop, and a wrapped one would read as a healthy fabric, which is the single worst thing a diagnostic model can do.

The last stimulus case is the reason cycles ship. A loop in the topology that no traffic ever enters costs nothing today. The measured build reports the cycle anyway, because it is a structural fact; the burn is zero, the capacity consumed is zero, and neither build raises an error. That is exactly the state a fabric is in when the redundant link is added and the traffic pattern that will eventually route into it has not been deployed yet. The cycle is present, harmless, and invisible to the one instrument that would have caught it.

9. RTL 5 — The Victims Are The Flows That Never Touched The Hot Port

This is the section that explains why fabric tickets are filed by the wrong people.

Virtual channels exist so that traffic to a congested destination does not stop traffic to an uncongested one. There are never enough of them. When more flows share a channel than the fabric has channels, a flow blocked at the head of that channel's queue holds the channel, and every other flow on it stops — regardless of where those flows were going, regardless of whether their own destinations have credit to spare, regardless of whether they have ever addressed the congested port in their lives.

The engineer who notices is not the owner of the congested destination. That owner sees their own traffic slow down, which is expected and unremarkable. The engineer who notices is the owner of one of the other flows, whose perfectly healthy path just stopped for reasons entirely outside their component.

// RTL 5 - head-of-line blocking. One congested destination on a shared virtual
// channel stops flows that have nothing to do with it, and those victims
// complain first.
module hol_coupling #(parameter int ONLY_THE_HOT_DEST_SUFFERS = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] flows, vcs, pkts_per_flow, hot_dest_flows,
  output logic [15:0] flows_per_vc, victim_flows, hol_pkts, victim_pct,
  output logic        hol_free,
  output logic [7:0]  n_evals, n_coupled,
  output logic        hol_err
);
  logic [31:0] h_q, v_q;
  logic [15:0] sharers, true_victims;
  logic        truly_coupled;
  // With no channels declared the fabric is one channel, not zero.
  assign flows_per_vc = (vcs == 16'd0) ? flows : (flows / vcs);
  // Everything else parked behind the congested flow on the same channel.
  assign sharers = (flows_per_vc == 16'd0) ? 16'd0 : (flows_per_vc - 16'd1);
  assign true_victims = (hot_dest_flows == 16'd0) ? 16'd0 : sharers;
  assign victim_flows = (ONLY_THE_HOT_DEST_SUFFERS != 0) ? 16'd0 : true_victims;
  assign h_q = {16'd0, victim_flows} * {16'd0, pkts_per_flow};
  assign hol_pkts = (h_q > 32'hFFFF) ? 16'hFFFF : h_q[15:0];
  assign v_q = (flows == 16'd0) ? 32'd0
             : (({16'd0, true_victims} * 32'd100) / {16'd0, flows});
  assign victim_pct = (v_q > 32'd100) ? 16'd100 : v_q[15:0];
  assign truly_coupled = (true_victims != 16'd0);
  assign hol_free = (victim_flows == 16'd0);
  assign hol_err = evaluate && truly_coupled && hol_free;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_coupled <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_coupled) n_coupled <= n_coupled + 8'd1;
    end
  end
endmodule

Sixteen flows on two virtual channels is eight flows per channel. One congested destination on that channel parks seven uninvolved flows behind it — forty-three percent of every flow in the fabric — and blocks seven hundred packets that had nothing to do with it. The hot-destination view counts no victims at all, because from its point of view exactly one flow is in trouble.

Figure 2 — the displacement that makes fabric tickets land on the wrong team. The seven victims are not on the congested path, have not addressed the congested destination, and have clean hops end to end. They are also the ones who notice, because the owner of the hot destination expects to be slow and they do not.

A fabric that declares no virtual channels at all is one channel, not zero, and the model says so explicitly. This is not pedantry: a fabric manager that reports zero channels because the field is unimplemented would, in a model that divided by it, either trap or report infinite isolation. Eight flows with no channel separation are eight flows sharing one channel, and seven of them are victims.

The counter-case is equally important. Sharing a channel is not blocking on its own. Eight flows per channel with no congested destination among them produces zero victims, zero blocked packets and a clean fabric in both builds. Head-of-line blocking requires a head of line that is actually blocked, and a model that reported coupling from channel-sharing alone would raise an alarm on every correctly configured fabric in existence.

10. RTL 6 — A Credit Leak Is A Timer, Not A Failure

Every failure so far is present the moment you look. This one is not. It is a fabric that works, passes, ships, runs in production for two weeks and then stops, with no event anywhere near the stop.

Credit in CXL is consumed when a flit is sent and returned when the receiver frees the buffer. If the return path loses one credit in a thousand — a counter that increments on the wrong condition, a return that is dropped during a rare corner, a buffer freed on a path that does not signal it — nothing fails. The pool is large; the loss is small; the fabric runs. It runs until the pool is empty, and then it stops forever, and the stop has no cause near it in time.

The measured build converts the imbalance into the quantity that matters: how long the fabric survives. The symmetric build — the one that assumes returns match consumption because they are designed to — measures no leak and therefore no end.

// RTL 6 - a credit leak. Returns that are one short of consumption do not fail
// a test; they set a timer, and the fabric stops days later with no event
// anywhere near the stop.
module credit_leak #(parameter int RETURNS_ARE_SYMMETRIC = 0) (
  input  logic clk, rst_n,
  input  logic        run_it,
  input  logic [15:0] credits_total, credits_out_k, credits_back_k, hours_tested,
  output logic [15:0] leak_per_k, dead_after_k, hours_to_dead, margin_pct,
  output logic        credits_balance,
  output logic [7:0]  n_runs, n_leaking,
  output logic        leak_err
);
  logic [15:0] true_leak;
  logic [31:0] m_q;
  logic        truly_leaking;
  // A return count above consumption is a measurement artefact, not a gain.
  assign true_leak = (credits_back_k >= credits_out_k)
                     ? 16'd0 : (credits_out_k - credits_back_k);
  assign leak_per_k = (RETURNS_ARE_SYMMETRIC != 0) ? 16'd0 : true_leak;
  // Thousands of transactions the fabric survives before the last credit goes.
  assign dead_after_k = (leak_per_k == 16'd0)
                        ? 16'hFFFF : (credits_total / leak_per_k);
  assign hours_to_dead = (dead_after_k == 16'hFFFF) ? 16'hFFFF
                       : ((dead_after_k > 16'd1000) ? 16'd1000 : dead_after_k);
  assign m_q = (hours_tested == 16'd0) ? 32'd100
             : (({16'd0, hours_to_dead} * 32'd100) / {16'd0, hours_tested});
  assign margin_pct = (m_q > 32'd100) ? 16'd100 : m_q[15:0];
  assign truly_leaking = (true_leak != 16'd0);
  assign credits_balance = (leak_per_k == 16'd0);
  assign leak_err = run_it && truly_leaking && credits_balance;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_runs <= 8'd0; n_leaking <= 8'd0;
    end else if (run_it) begin
      n_runs <= n_runs + 8'd1;
      if (truly_leaking) n_leaking <= n_leaking + 8'd1;
    end
  end
endmodule

Four credits lost per thousand transactions against a sixty-four credit pool is sixteen thousand transactions of life. Expressed against a seventy-two-hour soak, that is sixteen hours: twenty-two percent of the test window, which means the fabric dies during the test and the symmetric view reports full margin.

The second leaking case is the one that ships. A leak of one credit per thousand against a twenty-thousand credit pool is twenty million transactions of life, which is past the end of any test anybody will run — the model clamps the reported hours at a thousand and quotes full margin, and still reports the leak, because a leak that passes every test is still a leak and the pool is not always twenty thousand. That distinction is the entire value of measuring the imbalance rather than the outcome.

Two input artefacts are handled explicitly. More returns than consumption is not a gain; it is a counter read at two different instants, and a model that subtracted in the other direction would compute an enormous negative leak and report a fabric that gains credit forever. And returns arriving for credits nothing consumed is a counter bug in the instrument, not a leak in the fabric — the model reports nothing, because it is not the failure this section owns.

11. RTL 7 — A Dependency Cycle Is Not Congestion

A transaction that does not complete within its timeout produces the same event regardless of why. The event says this took too long. It does not say whether waiting longer would have helped, and that is the only question that matters.

Congestion is a queue that drains. Given time, the far buffer frees, the credit returns, the transaction completes. Every instinct an engineer has — increase the timeout, add retries, widen the buffer — is correct for congestion and actively harmful for its neighbour.

A buffer-dependency cycle is two switches each holding the resource the other is waiting for. It does not drain. It cannot drain. The timeout fires, the retry re-enters the same cycle, and the entire retry budget is spent on a transaction that was never going to complete. The event log is identical to congestion's.

// RTL 7 - cyclic buffer dependency. Two switches each holding the resource the
// other is waiting for is not congestion: no amount of time clears it, and the
// timeout that fires looks exactly like a slow fabric.
module buffer_dependency #(parameter int A_TIMEOUT_IS_CONGESTION = 0) (
  input  logic clk, rst_n,
  input  logic        analyse,
  input  logic [15:0] timeout_cyc, stall_cyc, a_needs_b, b_needs_a,
  output logic [15:0] held_cyc, recovery_cyc, retries_spent, stall_pct,
  output logic        deadlock_free,
  output logic [7:0]  n_analyses, n_deadlocked,
  output logic        deadlock_err
);
  logic [31:0] s_q;
  logic        dep_cycle, truly_stuck;
  assign dep_cycle = (a_needs_b != 16'd0) && (b_needs_a != 16'd0);
  assign truly_stuck = dep_cycle && (stall_cyc >= timeout_cyc);
  // A stall cannot be measured past the timeout that ends the transaction.
  assign held_cyc = (stall_cyc > timeout_cyc) ? timeout_cyc : stall_cyc;
  // Congestion drains; a dependency cycle does not, so retrying never ends.
  assign recovery_cyc = dep_cycle ? 16'hFFFF : held_cyc;
  assign retries_spent = dep_cycle ? timeout_cyc : 16'd0;
  assign s_q = (timeout_cyc == 16'd0) ? 32'd0
             : (({16'd0, held_cyc} * 32'd100) / {16'd0, timeout_cyc});
  assign stall_pct = (s_q > 32'd100) ? 16'd100 : s_q[15:0];
  // Calling it congestion means believing it will clear on its own.
  assign deadlock_free = (A_TIMEOUT_IS_CONGESTION != 0) ? 1'b1 : ~dep_cycle;
  assign deadlock_err = analyse && truly_stuck && deadlock_free;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_analyses <= 8'd0; n_deadlocked <= 8'd0;
    end else if (analyse) begin
      n_analyses <= n_analyses + 8'd1;
      if (truly_stuck) n_deadlocked <= n_deadlocked + 8'd1;
    end
  end
endmodule

The model separates them on the structural fact rather than the symptom. A one-way wait — switch A needing something from switch B, with B needing nothing back — is congestion: it clears when the far buffer drains, and both builds agree on it. Make the need mutual and the recovery time goes from a bounded number to never, while the congestion view keeps reporting a slow fabric.

The fourth case is the one that makes this measurable in a lab. A dependency cycle that has not yet reached its timeout is already a dependency cycle. The measured build reports it at two hundred cycles into a thousand-cycle timeout — the structure is there, the recovery is already impossible — while nothing has failed yet and no event has been logged. An engineer watching the dependency relation sees it eight hundred cycles before an engineer watching timeouts.

There is a second-order cost here that the retry column in the table understates. A retry budget exists to absorb transient failures, and it is sized on the assumption that a retried transaction has a meaningfully better chance of completing than the original did. A dependency cycle violates that assumption completely: every retry re-enters the same cycle and consumes the same buffers, so the budget is spent at full rate on a transaction whose probability of completion is exactly zero. A deeper retry budget makes a deadlocked fabric fail later and no less completely, and it does so while occupying resources that the rest of the fabric needs. The instinctive response to a timeout is therefore not merely ineffective here — it actively extends the blast radius from one stalled transaction to the flows competing for the same buffers.

The last case is the degenerate one, and it is not a curiosity. A fabric with no timeout configured and a dependency cycle in it never produces an event at all. The transaction does not fail; it simply never returns, and a congestion view waits for it forever. This is the state a bring-up fabric is often in — timeouts disabled to avoid noise during initial testing — which is precisely when the dependency cycle is most likely to be present.

12. RTL 8 — A Port That Trained Is Not A Port That Belongs To You

In a multi-host fabric, a port has two independent properties: whether the link works, and which host owns it. Link training establishes the first and says nothing whatever about the second.

A port assigned to the wrong host in a fabric-manager configuration is a fully functional port. It trains at full width, at full speed, with no errors, and it serves memory to a machine that should not be able to see it. Every link-level instrument reports a healthy port, because at the link level it is one.

The measured build audits assignment and ownership. The training view — the one that treats a trained port as an accounted-for port — reports on the wire.

// RTL 8 - port ownership in a multi-host fabric. A port that trained is a port
// that trained; which host owns it is a separate fact, and a port owned by the
// wrong host works perfectly and serves the wrong machine.
module port_ownership #(parameter int TRAINED_MEANS_OWNED = 0) (
  input  logic clk, rst_n,
  input  logic        audit,
  input  logic [15:0] ports, ports_trained, ports_assigned, owner_mismatch,
  output logic [15:0] trained_ports, stray_ports, wrong_host_ports, clean_pct,
  output logic        ownership_sound,
  output logic [7:0]  n_audits, n_wrong,
  output logic        binding_err
);
  logic [15:0] assigned_ports, clean_ports;
  logic [31:0] c_q;
  logic        truly_wrong;
  // Neither count can exceed the number of ports the fabric has.
  assign trained_ports  = (ports_trained  > ports) ? ports : ports_trained;
  assign assigned_ports = (ports_assigned > ports) ? ports : ports_assigned;
  assign stray_ports = trained_ports - ((assigned_ports > trained_ports)
                                        ? trained_ports : assigned_ports);
  assign wrong_host_ports = (owner_mismatch > assigned_ports)
                            ? assigned_ports : owner_mismatch;
  assign clean_ports = trained_ports - stray_ports
                       - ((wrong_host_ports > (trained_ports - stray_ports))
                          ? (trained_ports - stray_ports) : wrong_host_ports);
  assign c_q = (ports == 16'd0) ? 32'd100
             : (({16'd0, clean_ports} * 32'd100) / {16'd0, ports});
  assign clean_pct = (c_q > 32'd100) ? 16'd100 : c_q[15:0];
  assign truly_wrong = (wrong_host_ports != 16'd0) || (stray_ports != 16'd0);
  // Link training says the wire works. It says nothing about the owner.
  assign ownership_sound = (TRAINED_MEANS_OWNED != 0)
                           ? (trained_ports == ports) : ~truly_wrong;
  assign binding_err = audit && truly_wrong && ownership_sound;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_audits <= 8'd0; n_wrong <= 8'd0;
    end else if (audit) begin
      n_audits <= n_audits + 8'd1;
      if (truly_wrong) n_wrong <= n_wrong + 8'd1;
    end
  end
endmodule

Sixteen ports, all trained, three of them serving the wrong machine: the ownership audit reports eighty-one percent clean and calls the fabric unsound, while the training view signs off sixteen healthy links. Four ports trained and assigned to nobody is the same shape of problem from the other direction — a stray port is a security problem, not a link problem, and training is the wrong instrument for it.

The last stimulus case inverts the usual relationship and is worth reading twice. Ports that exist and have not trained are reported sound by the ownership model — an untrained port is not a misowned one — and unsound by the training view. The two models disagree, and here the ownership model is the permissive one. That is not an inconsistency: each model answers its own question correctly, and the point of running both is that neither question is the whole one. Section 14 is what it looks like to stop choosing between them.

The clamps in this model are the strictest in the chapter, because ownership data comes from a fabric manager's inventory and inventories drift. A mismatch report naming forty misowned ports on a fabric with ten assigned ports is clamped to ten, because at most ten can be misowned. An inventory claiming twelve trained and twenty assigned ports on an eight-port fabric is clamped to eight of each. A model that accepted those numbers would report negative clean ports and a percentage above a hundred, and would be discarded — correctly — the first time an engineer read its output.

13. RTL 9 — Which Hop Fails Is The Cheapest Signal In The Fabric

Every previous section has been about what is wrong. This one is about how much it costs to find out, because in a fabric that cost is dominated by a single decision made before any debugging starts.

A five-hop path with a fault somewhere on it has two plans available. Read the per-hop counters, find the hop where the discrepancy appears, and confirm it — cost proportional to the hops the evidence names. Or replace hardware from the endpoints inward until the symptom changes — cost proportional to the hops the path has, at hardware-swap prices rather than counter-read prices.

The measured build uses the evidence. The endpoint build is what a team does when it does not know the counters are there, or does not trust them, or is under enough pressure to start swapping.

// RTL 9 - which hop fails is the cheapest signal in the fabric. Per-hop
// counters turn a five-switch path into one switch; without them the only
// remaining move is to replace hardware one box at a time.
module hop_localization #(parameter int REPLACE_THE_ENDPOINTS = 0) (
  input  logic clk, rst_n,
  input  logic        plan_it,
  input  logic [15:0] hops, evidence_hops, probe_cost, swap_cost,
  output logic [15:0] hop_span, isolate_cost, blind_cost, saving_pct,
  output logic        localized,
  output logic [7:0]  n_plans, n_blind,
  output logic        hop_blind_err
);
  logic [15:0] narrowed, true_span;
  logic [31:0] i_q, b_q, s_q;
  logic        truly_localizable;
  // Evidence cannot narrow a path to fewer hops than it names, or to more
  // hops than the path has.
  assign narrowed  = (evidence_hops == 16'd0) ? hops : evidence_hops;
  assign true_span = (narrowed > hops) ? hops : narrowed;
  assign hop_span  = (REPLACE_THE_ENDPOINTS != 0) ? hops : true_span;
  assign i_q = {16'd0, hop_span} * {16'd0, probe_cost};
  assign isolate_cost = (i_q > 32'hFFFF) ? 16'hFFFF : i_q[15:0];
  assign b_q = {16'd0, hops} * {16'd0, swap_cost};
  assign blind_cost = (b_q > 32'hFFFF) ? 16'hFFFF : b_q[15:0];
  assign s_q = (blind_cost == 16'd0) ? 32'd0
             : ((blind_cost > isolate_cost)
                ? ((({16'd0, (blind_cost - isolate_cost)}) * 32'd100)
                   / {16'd0, blind_cost}) : 32'd0);
  assign saving_pct = (s_q > 32'd100) ? 16'd100 : s_q[15:0];
  assign localized = (hop_span == 16'd1);
  assign truly_localizable = (true_span == 16'd1) && (hops != 16'd0);
  // The evidence names one hop and the plan still touches all of them.
  assign hop_blind_err = plan_it && truly_localizable && (hop_span > 16'd1);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_blind <= 8'd0;
    end else if (plan_it) begin
      n_plans <= n_plans + 8'd1;
      if (!localized) n_blind <= n_blind + 8'd1;
    end
  end
endmodule

Per-hop counters naming one switch on a five-hop path cost two hours to confirm, against two hundred to swap every box on the path: ninety-nine percent of the cost avoided. The endpoint plan touches all five at five times the probing cost and localizes nothing.

The counter-cases keep the model honest. No per-hop counters means the whole path, and both plans become the same plan — the measured build claims no localization it cannot justify. Evidence naming more hops than the path has is no evidence, clamped to the path and localizing nothing. And a single-hop path is the case where the blind plan is accidentally right: there is nothing to localize to, both plans touch one hop, and neither model reports a fault. An engineer who has only ever debugged single-hop paths has never needed this section, which is why the first multi-switch fabric is where careers acquire this lesson.

It is worth being honest about why teams choose the expensive plan, because it is rarely ignorance. The endpoint swap has properties the evidence-driven plan does not: it requires no cooperation from a team that owns the middle switches, it produces visible activity for a manager watching an outage, and it cannot be blocked by a counter that turns out to be unimplemented or wrong. The blind plan is chosen because it is unblockable, not because it is cheap. That is a solvable problem, and the solution is upstream — counters that are trusted, documented and readable by the team that will be debugging at three in the morning. A counter nobody trusts has the same cost as a counter that does not exist.

The last case is the one to argue in a design review. Evidence that costs nothing to read — counters already implemented, already exposed, already collected — makes the saving a hundred percent of the swap cost. The counters are cheap at design time and unobtainable at debug time, and the gap between those two prices is the entire argument for per-hop observability. Section 20 returns to this.

14. RTL 10 — A Fabric Diagnosis Assembled

Nine models, nine independent claims. This one puts them in one place and makes the weak claim visible as what it is: one bit of six.

"There is a path" is the answer a topology dump gives, and it is not wrong — it is incomplete in a specific, enumerable way. The assembled model takes the six conditions the previous nine sections established, sets a bit for each one that fails, and contrasts a build that requires all six against a build that requires only the first.

// RTL 10 - a fabric diagnosis assembled. Everything that must hold before a
// destination is reachable in the sense that matters, with "there is a path"
// as one of the six rather than the whole claim.
module fabric_signoff #(parameter int THERE_IS_A_PATH = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       path_exists,      // a route to the destination is programmed
  input  logic       hop_credit,       // every hop on it has credit to give
  input  logic       routes_agree,     // both ends compute the same next hop
  input  logic       no_cycle,         // the path does not revisit a switch
  input  logic       no_hol,           // no unrelated flow is parked behind it
  input  logic       credits_balance,  // returns match consumption over time
  output logic       fabric_sound,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_sound,
  output logic       false_reach_err
);
  assign fail_mask[0] = ~path_exists;
  assign fail_mask[1] = ~hop_credit;
  assign fail_mask[2] = ~routes_agree;
  assign fail_mask[3] = ~no_cycle;
  assign fail_mask[4] = ~no_hol;
  assign fail_mask[5] = ~credits_balance;
  // The there-is-a-path build is what a topology dump reports.
  assign fabric_sound = (THERE_IS_A_PATH != 0) ? path_exists : (fail_mask == 6'd0);
  assign false_reach_err = evaluate && fabric_sound && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_sound <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (fabric_sound) n_sound <= n_sound + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time, and the pattern is the chapter in miniature. When a path exists and any one of the other five fails, the assembled model reports the failure and the topology view reports a reachable destination. Only when the path itself is missing do the two agree — which is why the obvious case is never the one that ships.

Across the eight evaluations the stimulus drives, the assembled model calls one fabric sound and the topology view calls six of them reachable. The five it gets wrong are the five single-bit failures with the path bit set, and each one is a real fabric that a real dump has drawn as healthy.

The ordering of the bits is not arbitrary. Bit 0 is checkable in seconds from a management interface. Bit 5 requires a soak. The cost of confirming a bit rises monotonically with its index, which makes the mask a triage order as well as a result: check them in bit order and stop at the first one that is set. Section 21 runs exactly that order against a fabric.

Figure 4 — the mask read as a triage order. Each decision is one bit of section 14, and the bits are ordered by what they cost to confirm: the credit read at the top is minutes, the soak at the bottom is hours. Taking them out of order is the mistake section 13 prices — every step skipped leaves the next one searching the whole path instead of one hop.

15. Quantitative Reasoning

Numbers from the models, stated so they can be argued with rather than admired.

Zero credit at one hop stalls a hundred percent of the load. Not a proportional share, not the traffic addressed to that hop — all of it, on a path that four independent instruments call up. This is the only failure in the chapter with no partial mode: every other section degrades, this one either works or does not.

One switch at a quarter credit delivers twenty-five percent of line rate. The gap between the quoted sixty-four and the delivered sixteen is forty-eight gigabytes a second, and it is stable, reproducible and entirely invisible to a link-status register. An engineer benchmarking that fabric will report a device problem.

Four entries out of sixty-four is four percent of the traffic. The ratio holds because the model converts entries to packets: a six-percent table disagreement is a four-percent misroute rate at ten packets per entry, and would be a forty-percent misroute rate if those four entries carried a hundred packets each. Which entries disagree matters more than how many.

Twenty packets in a loop burn thirty-two percent of a link. Twenty packets is nothing. Sixteen hops each is three hundred and twenty units of somebody else's capacity, which is why the observable symptom of a topology cycle is a bandwidth shortfall on flows that never enter the loop.

Seven of sixteen flows are victims of one congested destination. Forty-three percent of the fabric's flows stop because of a destination none of them addressed. The per-destination view reports one flow affected — a factor of seven between the instrument and the fault.

Four credits per thousand is sixteen hours. The leak rate is 0.4 percent, which reads as noise in any measurement. Against a sixty-four credit pool it is sixteen thousand transactions, and against a seventy-two-hour soak it is twenty-two percent of the window. A rate that looks like noise becomes a deadline when you divide the pool by it.

A dependency cycle is visible eight hundred cycles before the timeout. Two hundred cycles into a thousand-cycle timeout, the structure is already unrecoverable and nothing has been logged. Every cycle after that is spent waiting for information the fabric already has.

Three of sixteen misowned ports is eighty-one percent clean and a hundred percent trained. The two numbers are measured on the same ports at the same instant, and only one of them is about who can read the memory.

One hop of evidence against five hops of hardware is ninety-nine percent. Two hours against two hundred. When the counters are already implemented, it is a hundred percent — the entire swap cost, avoided by reading a register.

One of eight fabrics sound, six of eight reachable. The assembled model's summary number, and the chapter's.

16. Assertions

The testbenches carry 520 checks across ten models, and their structure is deliberate.

Every output of every model is asserted as a value, in both builds. Not merely checked for being non-X — compared against a number computed by hand before the model was run. An output listing step runs before the mutation campaign and reports any output net connected to a model instance that never appears in a check; this chapter's first run reported twenty-three, of which two were real and both were the reported build's own headline figure: the leak model's "no end, full margin" and the localization model's saving percentage. An output that only the broken build computes differently is exactly the output a mutation can corrupt for free.

Both builds are asserted on every degenerate case. An empty fabric, a table with no entries, a path with no hops, a fabric with no traffic, a timeout of zero, a fabric with no ports. Nine of batch 025's sixteen mutation survivors were an input pinned at zero that realistic stimulus never generated, so the stimulus here drives the top input of every model to zero deliberately rather than incidentally.

Every clamp is driven past its limit exactly once. A reachability probe naming nine hops on a three-hop path, hop rates above the link rate, a match count above the entry count, a misroute count above the packet count, a loop burning more than the link capacity, a hop-limit product that overflows sixteen bits, a mismatch report naming more ports than were assigned, evidence naming more hops than the path has. A clamp that is never exercised is a clamp whose inversion survives.

Every error output is checked in both directions in every case. The measured build must never fire; the reported build must fire on exactly the cases the section is about and stay quiet elsewhere. Half the checks in the file are the second half of that pair.

The counters are asserted at the end of each model's block. Six probes and four of them moving nothing; five measurements with two overstated; seven audits with three wrong. A counter assertion catches a mutation that corrupts a decision without changing any single output enough to fail a value check.

17. Mutation Testing

110 mutations, 110 killed. Fifty-three against the first testbench, fifty-seven against the second, both sets clean on the first run.

The mutations fall into families rather than being scattered:

Three mutations were designed and then discarded as equivalent, which is a result rather than an omission. Tightening the hop-rate clamp from strictly-greater to greater-or-equal changes nothing, because at the equality both branches compute the same value. The same is true of the misroute clamp, whose equality case no stimulus produces, and of the held-cycle clamp, where the stall equals the timeout and both branches yield that number. A replacement mutation needs its own reachability argument, and three of them could not produce one.

The families that matter most are the last two. A counter mutation changes no published value in any single cycle — it corrupts a running decision, and only an end-of-block count assertion catches it. Two of them in this chapter — the cycle counter and the coupling counter — could not be killed by inversion on the first stimulus, because six evaluations split three-and-three and the inverted counter produced the identical total. That is an equivalent mutant created by the stimulus rather than by the code, and the fix is a seventh case that unbalances the split, not a weaker mutation. 26.7 section 17 has the general form of it. A signal substitution — the localization model judging its plan by the evidence rather than by the plan, the leak model judging the truth by the report — is the mutation that most closely resembles a real bug, because it is what a careless refactor produces: two signals with similar names and different meanings, one substituted for the other.

18. Verification Strategy

A verification plan for fabric problems inherits a structural difficulty from the subject: the fault and the symptom are in different places, so a testbench organised by component will not find any of these.

Stimulate at the path level, check at the hop level. The interesting quantity in every section is a per-hop value that a path-level measurement averages away. A test that offers traffic end to end and measures throughput will observe section 6's twenty-five percent and have no way to attribute it. The same test with per-hop credit counters attributes it in one read.

Every fabric test needs a second flow that should not be affected. Section 9 is undetectable otherwise. A test with one flow through a congested destination measures congestion correctly and reports a healthy fabric for the seven flows it did not model. The victim flow is the instrument.

Soak for the leak, and measure the imbalance rather than the outcome. A credit leak at four per thousand takes sixteen hours to manifest against a sixty-four credit pool and never manifests at all against a large one. Waiting for the fabric to stop is the wrong test; counting credits out against credits back for a minute is the right one, and it scales to leaks that would take a month.

Build the topology check into the configuration path, not the test suite. A cycle added by a redundant link is present the moment the link is added and harmless until traffic routes into it. A test run at bring-up will pass; the failure arrives with a traffic-pattern change months later. The check belongs where the topology changes.

Separate the congestion tests from the dependency tests explicitly. They produce identical events and demand opposite responses. A test plan that treats "transaction timed out" as one bucket has already lost the distinction section 11 exists to make.

Audit ownership on every fabric-manager write. Section 12's failure is a configuration error that no traffic test can detect, because the traffic works. The only instrument is a comparison between the intended assignment and the actual one.

19. Synthesis and Implementation Reality

The models are teaching models, but the observability they assume has a real cost, and the cost is where these arguments are won and lost.

Per-hop credit counters are small and are the highest-value observability in a switch. A counter per virtual channel per port, readable without disturbing traffic, resolves sections 5, 6 and 10 directly and section 13 by construction. They are among the first things cut when an area budget tightens, on the grounds that they are debug-only — which is true, and is the argument for them, not against.

Credit return paths need their own counters, not a shared one. Section 10's leak is the difference between two counts. A design that increments one counter for consumption and infers returns from buffer occupancy cannot measure the imbalance at all, because the inference assumes the property being tested.

A routing-table comparison needs both tables readable from one place. Section 7's fault is invisible to a switch-local check by definition. This is a fabric-manager requirement rather than a switch requirement, and it is routinely discovered late because the switch team and the manager team each assume the other owns it.

Hop limits are cheap and must be finite. Section 8's loop is bounded only by the hop limit, and a fabric with no limit turns a redundant link into permanent capacity loss. The limit costs a few bits per flit and a comparator.

Timeouts must be configurable and non-zero in production. Section 11's last case — a dependency cycle on a fabric with timeouts disabled — produces no event at all. Disabled timeouts are common during bring-up and must not survive to a shipping configuration.

Ownership must be checkable independently of link state. A port's owner should be readable from a register that link training does not touch, so that section 12's audit is a read rather than an inference.

Virtual-channel count is a fabric-wide decision with a per-flow consequence. Section 9's victim count is flows-per-channel minus one, which makes the channel count the single parameter that determines how far one congested destination propagates. Doubling the channels halves the victims; the area cost is the buffering, and the argument against it is always made in terms of the congested flow rather than the seven that are not. Deciding this on the basis of the destination's own latency is the standard way the number comes out too low.

20. Silicon Observability

What can be read from a real fabric, ordered by what it costs to get.

Free, already there. Link state and width per port; route tables per switch; hop counts. These answer bit 0 of section 14's mask and nothing else. Every fabric has them, which is exactly why every fabric problem starts with them and why so many stop there.

Cheap, if the counters exist. Credit advertised and credit consumed per hop per virtual channel. This is the single highest-value read in the chapter: it answers bits 1 and 5, and it converts section 13's five-hop search into a one-hop confirmation. If it exists, read it first; if it does not, this is the design review to have.

Moderate. Per-port packet and byte counters with destination attribution, which give section 7's misroute rate and section 8's burn as a discrepancy between what a switch forwarded and what the next one received. The attribution is what makes them useful; a bare packet count is nearly worthless here.

Expensive. A protocol analyser on a link, which gives ground truth on one hop and nothing about the other four. In a fabric problem, an analyser is a confirmation instrument, not a search instrument — and the cost of treating it as a search instrument is section 13's two hundred hours. 26.7 is about using it well.

Unobtainable. Simultaneous visibility into every hop's credit state at one instant. Nothing provides this, which is why the credit imbalance — a difference between two counts over time — is the practical substitute for the credit state.

21. Debug Lab

A fabric where a host reports that a device five hops away delivers a quarter of its expected bandwidth. Nothing is logged. Every link is up.

Step 1 — check the path exists. A topology read. The route is programmed, all five hops are reachable, every link trained at full width. This takes two minutes and is bit 0 of the mask. It is also the point at which most sessions declare the fabric healthy and go looking at the device.

Step 2 — check every hop has credit. Read advertised credit per hop. Four hops advertise a full pool and one advertises a quarter. This is section 6, the answer is bit 1, and the elapsed time is under ten minutes. Stop here if the number matches the shortfall — a quarter of the credit on one hop and a quarter of the expected bandwidth is not a coincidence.

Step 3 — if the shortfall is total rather than proportional, check for zero credit. Section 5. A hop at zero credit stalls everything, and the distinction from step 2 is whether the delivered rate is a fraction or is nothing.

Step 4 — compare routing tables at both ends of each hop. Section 7. This is the first step that needs data from two places at once, and it is where a fabric manager that cannot read both tables costs an afternoon. A misroute rate that matches the disagreement rate confirms it.

Step 5 — look for a cycle. Section 8. Walk the routes for a repeated switch. A cycle explains a bandwidth shortfall on flows that do not traverse the loop, which is the signature to look for: the complaining flow is not the flow in the loop.

Step 6 — check whether the complaining flow shares a channel with a congested one. Section 9. If the complaining flow's own path is clean at every hop, the fault is not on its path. This is the step that ends sessions where every hop checks out.

Step 7 — if the fabric worked and then stopped, count credits out against credits back. Section 10. A leak explains a failure with no event near it, and the imbalance is measurable in a minute even when the time-to-death is weeks.

Step 8 — if a transaction is stalled rather than slow, check whether the need is mutual. Section 11. A one-way wait clears; a mutual one does not, and increasing the timeout makes it worse.

The order is the mask's bit order, and the reason to follow it is section 13: each step narrows the hops the next step has to examine, and a step taken out of order examines all of them.

22. Design Review

Questions worth asking before a fabric exists, each of which is cheap now and impossible later.

Can I read credit per hop, per virtual channel, without perturbing traffic? If not, sections 5, 6, 10 and 13 all become hardware swaps. This is the single question with the largest cost asymmetry in the chapter.

Are credits out and credits back counted separately? An inferred return count cannot measure section 10's imbalance.

Can the fabric manager read both ends of a route? If routing tables are only readable switch-locally, section 7 is undetectable by construction.

Is the hop limit finite and configurable? An infinite one turns a redundant link into permanent capacity loss.

How many virtual channels, and how are flows mapped to them? Section 9's victim count is flows-per-channel minus one. The mapping policy is a design decision that determines how far a single congested destination propagates.

Is port ownership readable independently of link state? If ownership can only be inferred from a link being up, section 12 is a security exposure with no instrument.

Are timeouts non-zero in the shipping configuration? A fabric with timeouts disabled turns section 11's deadlock into a transaction that never returns and never reports.

What happens to the routing computation when a redundant link is added? If the answer is "it is recomputed manually", section 8 is scheduled rather than prevented.

23. How This Appears In Real Engineering

The ticket says the device is slow. It has been assigned to the device team, because the device is the thing the complaining host was talking to, and the device team has spent three days proving their device is fine — which it is.

The fabric is five switches. The device team has no access to four of them. The fabric team has not been told there is a problem, because from their side nothing is wrong: every link is up, every route is programmed, the topology dump is clean and no alarm has fired. The two teams are looking at the same fabric and neither of them can see the other's evidence.

What eventually resolves it is almost always the same move: someone reads per-hop credit. It takes minutes, it names one switch, and the three days that preceded it were spent on a device that was working correctly the whole time.

The organisational pattern under this is worth naming. The complaining party, the owning party and the faulty component are three different things in a fabric, and every process that routes a ticket by who complained will route it wrongly. Section 9 is the extreme case — the complaining flow has no relationship at all to the fault — but every section in this chapter has some version of it.

The second recurring shape is the failure with no event. Section 10's leak and section 8's dormant cycle both ship in a state where nothing is observably wrong, and both surface weeks later when something unrelated changes: a traffic pattern, a workload, a firmware update that shifts a routing decision. The change gets blamed, because it is the only thing that moved. It is almost never the cause; it is the thing that finally routed traffic into a fault that was already there.

The third shape is organisational rather than technical, and it determines how long the first two take to resolve. The instrument that resolves a fabric problem is usually owned by neither the team that reports it nor the team it is assigned to. Per-hop credit lives with the fabric team; the complaint arrives from a host team; the ticket is assigned to a device team. Nothing in that chain is malicious or incompetent, and the delay is not caused by any individual step — it is caused by the fact that the evidence and the urgency are held by different people. Teams that resolve these quickly have almost always done one specific thing in advance: made per-hop credit readable by whoever is holding the pager, rather than by whoever owns the switch.

24. Common Misconceptions

"The topology dump says it's reachable, so the path is fine." The dump answers whether a route is programmed. Five other things must hold, and four of them are invisible to it. This is the chapter.

"The link is 64 gigabytes a second, so the path is 64 gigabytes a second." The path is the minimum of the per-hop credit, clamped to the link. Section 6.

"I read the routing table and it's correct." You read one end of it. Section 7.

"A loop would drop packets." A loop keeps packets and burns capacity until a hop limit retires them. The flows that suffer are the ones sharing links with the loop, not the ones in it. Section 8.

"My flow is slow, so something on my path is broken." If your flow shares a virtual channel with a congested one, nothing on your path is broken. Section 9.

"It ran for two weeks, so it's stable." A credit leak is a timer. Two weeks of running is a measurement of the pool size divided by the leak rate, not of correctness. Section 10.

"It timed out, so I'll increase the timeout." Correct for congestion, actively harmful for a dependency cycle, and the event does not distinguish them. Section 11.

"The port trained, so it's configured." Training establishes the link. Ownership is a separate fact that no link-level instrument reads. Section 12.

"I'll swap the endpoints first, they're easiest to reach." Easiest to reach is not cheapest to test. Section 13 puts the ratio at ninety-nine percent.

25. Interview Reasoning

"A host reports a CXL device delivering a quarter of its expected bandwidth across a five-switch fabric. Every link is up. Where do you start?" Per-hop credit, before anything else. A proportional shortfall with a matching per-hop credit ratio is section 6 and takes ten minutes. Starting at the device costs days and finds nothing. The reasoning to show is that the shortfall being proportional rather than total is itself evidence — it points at credit rather than at a dead hop.

"A fabric runs correctly for two weeks and then stops. No event is logged near the stop. What is your first hypothesis?" A credit leak, and the test is to count credits out against credits back rather than to wait for it to happen again. The insight being tested is that a failure with no proximate cause is usually an accumulation, and that accumulations are measured by their rate, not their outcome.

"Two flows share a fabric. One is congested by design. The other, on a completely different path, slows down. Explain." Head-of-line blocking on a shared virtual channel. The follow-up is how to confirm it: check whether the complaining flow's own path is clean at every hop, and if it is, the fault is not on its path. The point is that a clean path is evidence for this diagnosis, not against it.

"How do you tell a fabric deadlock from fabric congestion, given that both produce a timeout?" Congestion drains and deadlock does not, so the structural question is whether the need is mutual. Increasing the timeout is correct for one and wastes the retry budget on the other. A strong answer notes that the dependency relation is observable before the timeout fires.

"A redundant link was added to a fabric six months ago for availability. Should you worry?" Only if the routing computation excludes it from forwarding paths. A cycle is harmless until traffic routes into it, which is what makes it survive bring-up and surface after a workload change. The reasoning is about when the failure becomes observable, not whether it exists.

26. Exercises

1. A four-hop path advertises 64, 64, 8 and 64 gigabytes a second of credit on a 64-gigabyte link. What rate should the datasheet quote, and what will a link-status register report? Compute the gap.

2. A fabric has 32 routing entries; 28 match between the two switches. Traffic is 50 packets per entry against 2,000 packets total. Compute the misroute count and percentage. Now move all the traffic onto the four disagreeing entries and recompute.

3. A credit pool of 512 leaks 2 credits per thousand transactions. How many transactions until the fabric stops? If the soak test runs 100 hours at 10 transactions per hour per thousand, does the test catch it?

4. Twenty-four flows share four virtual channels. One destination is congested. How many flows are victims? How many would be victims with eight channels? With twenty-four?

5. A six-hop path has a fault. Per-hop counters cost 1 hour each to read; swapping a switch costs 40 hours. Compute the cost of both plans, and the saving. Now assume the counters name two hops rather than one.

6. Take the head-of-line model and add a second congested destination on a different channel. What changes in the victim count, and what does the per-destination view report now?

7. A fabric has a three-switch cycle and a hop limit of 32. Eight packets enter the loop per second against a 2,000-unit link. What fraction of the link is consumed? At what hop limit does the loop consume the entire link?

8. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify where it goes in the bit order, using the rule that cost-to-confirm rises with index.

27. Summary

A route is not a throughput. A topology dump answers one of six questions, and the other five are the ones that produce debug sessions.

Credit is per hop, and the narrowest hop sets the rate. A quarter of the credit on one switch is a quarter of the bandwidth on every path that crosses it, and the link-status register will not mention it.

A routing table is an agreement held in two places. Reading one end tells you what that switch will do, not where the packet arrives.

A cycle keeps packets and burns capacity. The flows that suffer are not the flows in the loop, and the loop is harmless until the day traffic routes into it.

Head-of-line blocking displaces the symptom completely. Seven of sixteen flows stop because of a destination none of them addressed, and every one of those seven has a perfectly clean path.

A credit leak is a timer. Four credits per thousand against a sixty-four credit pool is sixteen hours, and a model that assumes returns match consumption reports full margin right up to the stop.

A dependency cycle is not congestion, produces the same event, and demands the opposite response.

A port that trained is a port that trained. Ownership is a separate fact, and no link-level instrument reads it.

Which hop fails is the cheapest signal in the fabric — ninety-nine percent cheaper than the alternative, and free if the counters were designed in.

Six bits, and "there is a path" is one of them. One fabric of eight is sound; a topology dump calls six of them reachable.

26.6 takes the proportional case further: a fabric where every one of these six bits is clear and the throughput is still wrong.