Multi-Endpoint Systems — Many Devices, One Fabric

Chapter 21.3 built the tree. This chapter turns everything on at once.

A real system does not have one active device. It has an NVMe drive streaming, a NIC receiving, a GPU pulling training data and a management controller polling — all behind the same switches, all issuing requests, all expecting Completions back.

And a question appears here that could not be asked before. Two of those endpoints, under different switches, issue a Memory Read in the same microsecond, and both use Tag 5. Both are entirely correct. So what keeps their Completions apart?

1. Sources and Scope

2. What Actually Changes

Nothing in the routing rules. A switch with eight busy endpoints applies exactly the rules of Chapter 21.1 to each packet, one at a time.

What changes is that state is now shared. Queues, credits, arbitration, error registers and the uplink are all touched by traffic from multiple independent sources, and any structure that cannot say whose traffic it is holding becomes a defect.

Three questions organize the rest of the chapter.

Whose Completion is this? (§3, §4) — identity in the data path.

Whose bandwidth is this? (§6, §7) — identity in accounting and arbitration.

Whose fault was this? (§8, §10) — identity in error handling.

3. A Tag Is Not Globally Unique

4. Completions Find Their Way Home

The return path uses ID routing, and nothing about it is per-endpoint special-cased.

At each hop upward from the Completer, the switch compares the Completion's Requester ID bus number against its downstream ports' Secondary/Subordinate ranges (Chapter 21.1 §5). Inside a range, the Completion goes down that port. Outside every range, it goes upstream — Chapter 21.3 §3's default, unchanged.

Which means Completion routing scales for free. The switch does not track outstanding requests, does not remember who asked, and holds no per-transaction state. The Completion carries its own return address, so the fabric is stateless with respect to it.

This is the fabric-scale instance of a law this curriculum has established repeatedly: an asynchronous response must carry its own identity. Chapter 20.3 §3 measured it inside one device; here the same property is what lets an arbitrary number of endpoints share an arbitrary tree with no coordination.

Two consequences worth stating.

A split read is still one transaction. Multiple Completions returning for one request (Chapter 20.3 §5) all carry the same (Requester ID, Tag), and the requester reassembles them. Retiring the Tag on the first Completion is a bug — §10's tracker retires only when the byte count is satisfied.

And Chapter 21.3 §5's invariant is load-bearing here. If a bridge's Secondary/Subordinate range does not contain the requesting endpoint's bus, the Completion routes upstream and never returns — the requester eventually reports a Completion Timeout, and the reported symptom is a device that hangs on reads while its own configuration is perfectly valid.

5. Many Endpoints, Many Interrupts

Interrupts multiply the same identity problem into a different subsystem.

MSI and MSI-X are Memory Writes (Chapter 19.2, 19.3), so they route by address like any other posted write — the fabric does not treat them specially and needs no interrupt-aware logic.

Which means the identity is in the message, not the path. Two endpoints raising interrupts simultaneously are distinguished by the address and data each was programmed with, not by which port they arrived on (Chapter 19.4).

Two multi-endpoint effects.

Interrupt storms are a contention source. Many endpoints interrupting at high rate produce many posted writes competing for the same uplink — the same arbitration as data traffic (§7), which is one reason coalescing matters at scale (Chapter 19.5).

And an ordering subtlety survives at fabric scale. An endpoint that writes data by DMA and then raises an MSI relies on the interrupt not passing the data. This chapter adds no new rule — Chapter 19.2 §5 owns it — but note that the guarantee is about ordering within a path, and a second endpoint's traffic on the same uplink neither strengthens nor weakens it.

6. The Traffic Matrix

With N endpoints there is no single "traffic pattern" — there is a matrix of who talks to whom. The sourced device documents Host-Centric and Peer-to-Peer traffic as distinct cases (§1), and they load the fabric very differently.

Read the second row carefully, because it is the useful one. Two endpoints under the same switch exchanging data need not consume the uplink at all — the switch can route between its own downstream ports (Chapter 21.1 §4: an address inside another downstream port's window goes there directly, not upstream).

Which makes placement a design decision, not an accident. Two devices that talk to each other heavily behave differently under the same switch than under different switches — same rules, same devices, different uplink load.

Two cautions. Peer-to-peer support is not universal — it depends on the components and on configuration, and this chapter makes no claim that any particular system permits it. And Module 22 owns the throughput analysis of these patterns; §6 is topology, not performance.

7. Whose Bandwidth Is This?

8. Failure Domains

9. The System View

Three things to read out of the figure.

The queues are per port and the uplink is not. Everything left of the arbiter is contained; everything right of it is shared — and §8's 92% versus 0% is exactly the difference between those two regions.

The completion demux is keyed on the pair. Endpoint A and Endpoint B both have Tag 5 outstanding in the figure deliberately (§3), and the demux returns each Completion to its own requester using (Requester ID, Tag). Keyed on Tag alone, that return arrow points at the wrong endpoint 76.3% of the time (§11).

And the two observers hang off the shared point, because that is where per-endpoint questions become unanswerable without deliberate instrumentation (§7, §10).

10. RTL — Identity, Accounting, Capture and a Cross-Port Monitor

// SYNTHESIZABLE. Multi-endpoint identity types (§3).
package multi_ep_pkg;
 
  parameter int N_PORT = 4;
  parameter int PORT_W = (N_PORT <= 1) ? 1 : $clog2(N_PORT);
  parameter int TAG_W  = 8;
  parameter int RID_W  = 16;                 // Bus/Device/Function
  parameter int LEN_W  = 12;
 
  // =================================================================
  // THE IDENTITY OF AN OUTSTANDING TRANSACTION. Not the Tag alone.
  // Two endpoints may both hold Tag 5 legally and simultaneously.
  // Section 11 Model 1: Tag-only matching misattributed 76.3%.
  // =================================================================
  typedef struct packed {
    logic [RID_W-1:0] requester_id;
    logic [TAG_W-1:0] tag;
  } txn_id_t;
 
  function automatic bit txn_match(input txn_id_t a, input txn_id_t b);
    return (a.requester_id == b.requester_id) && (a.tag == b.tag);
  endfunction
 
  typedef struct packed {
    logic             valid;
    txn_id_t          id;
    logic [LEN_W-1:0] bytes_expected;
    logic [LEN_W-1:0] bytes_returned;
  } outstanding_t;
 
endpackage

import multi_ep_pkg::*;
 
// SYNTHESIZABLE. Completion demultiplex, keyed on the PAIR (§3, §4).
// A split read returns several Completions for one transaction, so the
// entry retires on BYTE COUNT SATISFIED, never on first Completion
// (Chapter 20.3 §5).
module completion_demux #(parameter int DEPTH = 16) (
  input  logic clk,
  input  logic rst_n,
 
  input  logic             issue_valid,
  input  txn_id_t          issue_id,
  input  logic [LEN_W-1:0] issue_bytes,
 
  input  logic             cpl_valid,
  input  txn_id_t          cpl_id,
  input  logic [LEN_W-1:0] cpl_bytes,
 
  output logic             cpl_hit,
  output logic             cpl_unexpected,     // no matching outstanding entry
  output logic             cpl_ambiguous,      // duplicate identity -- a bug
  output logic             txn_complete,
  output txn_id_t          complete_id
);
  outstanding_t tbl [DEPTH];
  logic [DEPTH-1:0] mvec;
 
  // Match on the WHOLE identity. Comparing tag alone here is mutation 1.
  always_comb
    for (int i = 0; i < DEPTH; i++)
      mvec[i] = tbl[i].valid && txn_match(tbl[i].id, cpl_id);
 
  assign cpl_hit        = cpl_valid &&  $onehot(mvec);
  assign cpl_unexpected = cpl_valid && (mvec == '0);
  assign cpl_ambiguous  = cpl_valid && !$onehot0(mvec);   // never resolve by priority
 
  int hit_idx;
  always_comb begin
    hit_idx = 0;
    for (int i = 0; i < DEPTH; i++) if (mvec[i]) hit_idx = i;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < DEPTH; i++) tbl[i] <= '0;
      txn_complete <= 1'b0; complete_id <= '0;
    end else begin
      txn_complete <= 1'b0;
      if (issue_valid)
        for (int i = 0; i < DEPTH; i++)
          if (!tbl[i].valid) begin
            tbl[i] <= '{valid:1'b1, id:issue_id, bytes_expected:issue_bytes, bytes_returned:'0};
            break;
          end
 
      if (cpl_hit) begin
        automatic logic [LEN_W-1:0] tot = tbl[hit_idx].bytes_returned + cpl_bytes;
        tbl[hit_idx].bytes_returned <= tot;
        if (tot >= tbl[hit_idx].bytes_expected) begin   // RETIRE ONLY WHEN SATISFIED
          tbl[hit_idx].valid <= 1'b0;
          txn_complete <= 1'b1;
          complete_id  <= tbl[hit_idx].id;
        end
      end
    end
  end
endmodule

import multi_ep_pkg::*;
 
// SYNTHESIZABLE. Per-port accounting (§7). Counts GRANTS and BYTES
// separately, because equal grants can mean a 32:1 byte ratio
// (§11 Model 3). An aggregate total detects 0% of compensating
// cross-port errors (§11 Model 2), which is why nothing here is pooled.
module per_port_accounting #(parameter int N = N_PORT) (
  input  logic clk,
  input  logic rst_n,
  input  logic [N-1:0]      grant,
  input  logic [LEN_W-1:0]  grant_bytes [N],
  input  logic              clear,
 
  output logic [31:0] grants [N],
  output logic [47:0] bytes  [N],
  output logic [47:0] bytes_total
);
  logic [31:0] g_q [N];
  logic [47:0] b_q [N];
 
  always_comb begin
    bytes_total = '0;
    for (int i = 0; i < N; i++) begin
      grants[i] = g_q[i];
      bytes[i]  = b_q[i];
      bytes_total += b_q[i];      // derived FROM per-port state, never instead of it
    end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear)
      for (int i = 0; i < N; i++) begin g_q[i] <= '0; b_q[i] <= '0; end
    else
      for (int i = 0; i < N; i++)
        if (grant[i]) begin
          if (!(&g_q[i])) g_q[i] <= g_q[i] + 32'd1;         // saturate, never wrap
          b_q[i] <= b_q[i] + 48'(grant_bytes[i]);
        end
  end
endmodule

import multi_ep_pkg::*;
 
// SYNTHESIZABLE. First-failure capture (§8, §14).
// Errors cascade across ports: one stalled endpoint produces timeouts in
// several others. LAST-WRITE-WINS reports a NON-root-cause error 96.8% of
// the time (§11 Model 5); latch-once-and-lock reports the first 100%.
module first_failure_capture #(parameter int N = N_PORT) (
  input  logic clk,
  input  logic rst_n,
  input  logic [N-1:0] err_pulse,
  input  logic [3:0]   err_code [N],
  input  logic         clear,
 
  output logic             captured,
  output logic [PORT_W-1:0] first_port,
  output logic [3:0]       first_code,
  output logic [31:0]      subsequent_count      // count later ones, do NOT overwrite
);
  logic             cap_q;
  logic [PORT_W-1:0] port_q;
  logic [3:0]       code_q;
  logic [31:0]      later_q;
 
  assign captured = cap_q; assign first_port = port_q;
  assign first_code = code_q; assign subsequent_count = later_q;
 
  int idx; logic any;
  always_comb begin
    idx = 0; any = 1'b0;
    for (int i = N-1; i >= 0; i--) if (err_pulse[i]) begin idx = i; any = 1'b1; end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin cap_q<=1'b0; port_q<='0; code_q<='0; later_q<='0; end
    else if (any) begin
      if (!cap_q) begin                      // LATCH ONCE
        cap_q <= 1'b1; port_q <= PORT_W'(idx); code_q <= err_code[idx];
      end else if (!(&later_q)) later_q <= later_q + 32'd1;   // then only count
    end
  end
endmodule

import multi_ep_pkg::*;
 
// VERIFICATION INSTRUMENTATION -- not a data-path block.
// Cross-port invariant monitor: checks properties no single port can see.
module cross_port_monitor #(parameter int N = N_PORT) (
  input  logic clk,
  input  logic rst_n,
  input  logic [N-1:0] port_owns_uplink,
  input  logic [31:0]  outstanding [N],
  input  logic [31:0]  issued [N],
  input  logic [31:0]  returned [N],
 
  output logic err_multi_owner,
  output logic err_port_overreturn,
  output logic err_negative_outstanding
);
  always_comb begin
    // Exactly one port may own the shared uplink (Chapter 21.2 §11).
    err_multi_owner = !$onehot0(port_owns_uplink);
 
    // PER-PORT, deliberately. A summed check passes while two ports are
    // wrong in opposite directions -- §11 Model 2 measured 0% detection.
    err_port_overreturn      = 1'b0;
    err_negative_outstanding = 1'b0;
    for (int i = 0; i < N; i++) begin
      if (returned[i] > issued[i])                    err_port_overreturn = 1'b1;
      if (outstanding[i] != (issued[i] - returned[i])) err_negative_outstanding = 1'b1;
    end
  end
endmodule

Classification: the first four are synthesizable; cross_port_monitor is verification instrumentation.

The single most important line is txn_match. Everything else in the chapter is a consequence of getting that comparison right.

Failure — six. Matching Completions on Tag alone (76.3%, §11). Retiring on the first Completion of a split read. Resolving a duplicate identity by priority instead of reporting it. Pooling accounting instead of keeping it per port (0% detection, §11). Overwriting error status with the newest error (96.8% wrong, §11). And counting grants as a proxy for bandwidth (32:1, §7).

11. Measured Behaviour

12. Assertions

// ==================================================================
// IDENTITY -- the core of the chapter (§3, §4).
// ==================================================================
 
// P1: a matched Completion matched on the WHOLE identity, never the Tag.
property p_match_is_pairwise;
  @(posedge clk) disable iff (!rst_n)
    cpl_hit |-> txn_match(tbl[hit_idx].id, cpl_id);
endproperty
 
// P2: same Tag, different Requester ID => NOT a match. This is Model 1's
// 76.3% expressed as a property.
property p_same_tag_different_rid_no_match;
  @(posedge clk) disable iff (!rst_n)
    (cpl_valid && tbl[0].valid
      && (tbl[0].id.tag == cpl_id.tag)
      && (tbl[0].id.requester_id != cpl_id.requester_id)) |-> !mvec[0];
endproperty
 
// P3: exactly one outcome per Completion. No silent third case.
property p_cpl_outcome_total;
  @(posedge clk) disable iff (!rst_n)
    cpl_valid |-> $onehot({cpl_hit, cpl_unexpected, cpl_ambiguous});
endproperty
 
// P4: a duplicate outstanding identity is REPORTED, not priority-resolved.
property p_duplicate_identity_flagged;
  @(posedge clk) disable iff (!rst_n)
    (cpl_valid && !$onehot0(mvec)) |-> cpl_ambiguous;
endproperty
 
// P5: two outstanding entries never hold the same identity.
property p_identities_unique;
  @(posedge clk) disable iff (!rst_n)
    (tbl[0].valid && tbl[1].valid) |-> !txn_match(tbl[0].id, tbl[1].id);
endproperty
 
// P6: an unexpected Completion is surfaced, never silently absorbed.
property p_unexpected_reported;
  @(posedge clk) disable iff (!rst_n)
    (cpl_valid && (mvec == '0)) |-> cpl_unexpected;
endproperty
 
// ==================================================================
// SPLIT COMPLETIONS -- one transaction, several responses (20.3 §5).
// ==================================================================
 
// P7: retire only when the expected byte count is satisfied.
property p_retire_on_bytes_satisfied;
  @(posedge clk) disable iff (!rst_n)
    txn_complete |-> $past(tbl[hit_idx].bytes_returned + cpl_bytes
                            >= tbl[hit_idx].bytes_expected);
endproperty
 
// P8: a partial Completion does NOT free the entry.
property p_partial_keeps_entry;
  @(posedge clk) disable iff (!rst_n)
    (cpl_hit && ((tbl[hit_idx].bytes_returned + cpl_bytes)
                  < tbl[hit_idx].bytes_expected)) |=> tbl[$past(hit_idx)].valid;
endproperty
 
// P9: returned bytes never exceed expected -- an over-return is an error,
// not an arithmetic accident.
property p_no_overreturn;
  @(posedge clk) disable iff (!rst_n)
    tbl[0].valid |-> (tbl[0].bytes_returned <= tbl[0].bytes_expected);
endproperty
 
// P10: the completing transaction is identified by its pair, not its slot.
property p_complete_id_is_pair;
  @(posedge clk) disable iff (!rst_n)
    txn_complete |-> txn_match(complete_id, $past(cpl_id));
endproperty
 
// ==================================================================
// PER-PORT ACCOUNTING (§7). Model 2: an aggregate detects 0%.
// ==================================================================
 
// P11: a port's counters advance only when THAT port is granted.
property p_accounting_is_per_port;
  @(posedge clk) disable iff (!rst_n)
    (!grant[0]) |=> ($stable(grants[0]) && $stable(bytes[0]));
endproperty
 
// P12: the total is DERIVED from per-port state, never maintained instead.
property p_total_is_derived;
  @(posedge clk) disable iff (!rst_n)
    (bytes_total == bytes[0] + bytes[1] + bytes[2] + bytes[3]);
endproperty
 
// P13: counters saturate rather than wrap. A wrapped counter reads as
// starvation and sends debugging in the wrong direction.
property p_counters_saturate;
  @(posedge clk) disable iff (!rst_n)
    (grants[0] == 32'hFFFF_FFFF) |=> (grants[0] == 32'hFFFF_FFFF);
endproperty
 
// P14: bytes are counted separately from grants -- equal grants can be a
// 32:1 byte ratio (Model 3).
property p_bytes_tracked_independently;
  @(posedge clk) disable iff (!rst_n)
    (grant[0] && (grant_bytes[0] != '0)) |=> (bytes[0] != $past(bytes[0]));
endproperty
 
// ==================================================================
// FIRST-FAILURE CAPTURE (§8). Model 5: last-write-wins is wrong 96.8%.
// ==================================================================
 
// P15: once captured, the record is immutable until explicitly cleared.
property p_capture_is_sticky;
  @(posedge clk) disable iff (!rst_n)
    (captured && !clear) |=> (captured && $stable(first_port) && $stable(first_code));
endproperty
 
// P16: later errors increment a counter and never overwrite the record.
property p_later_errors_counted_not_stored;
  @(posedge clk) disable iff (!rst_n)
    (captured && |err_pulse && !clear) |=> $stable(first_port);
endproperty
 
// P17: the first error is captured on the cycle it occurs.
property p_first_error_captured;
  @(posedge clk) disable iff (!rst_n)
    (!captured && |err_pulse) |=> captured;
endproperty
 
// P18: capture cannot claim a port that did not signal an error.
property p_capture_names_real_port;
  @(posedge clk) disable iff (!rst_n)
    ($rose(captured)) |-> $past(err_pulse[first_port]);
endproperty
 
// ==================================================================
// CROSS-PORT INVARIANTS (§10) -- properties no single port can see.
// ==================================================================
 
// P19: exactly one port owns the shared uplink (21.2 §11's contract).
property p_single_uplink_owner;
  @(posedge clk) disable iff (!rst_n)
    $onehot0(port_owns_uplink);
endproperty
 
// P20: outstanding is per port and consistent per port. A summed version
// of this property passes while two ports are wrong oppositely (Model 2).
property p_outstanding_consistent_per_port;
  @(posedge clk) disable iff (!rst_n)
    (outstanding[0] == issued[0] - returned[0]);
endproperty
 
// P21: no port returns more than it was issued.
property p_no_port_overreturn;
  @(posedge clk) disable iff (!rst_n)
    (returned[0] <= issued[0]);
endproperty
 
// P22: one port's fault does not stop an unrelated port's progress
// (§8, Model 4: 92% versus 0%).
property p_fault_containment;
  @(posedge clk) disable iff (!rst_n)
    (port_stalled[0] && request[1] && uplink_operational && fc_grant)
      |-> s_eventually (port_owns_uplink[1]);
endproperty

Twenty-two properties. P1–P6 are the chapter — every other group is a consequence of identity applied to a different structure. P7–P10 exist because a read may answer in pieces, P11–P14 because a total cannot be interrogated, P15–P18 because cascades end with symptoms, and P19–P22 because some invariants are only visible above the port level.

13. Verification — Mutations

Two counterexamples worth stating explicitly.

Mutation 11 is the one that ships. Indexing the outstanding table by Tag is the natural data structure — it is O(1), it is what a single-device driver does, and it is correct for exactly as long as there is one Requester. It does not fail loudly when a second endpoint appears; it fails by returning the wrong entry, which is why P5 asserts uniqueness of the whole identity rather than of the Tag.

Mutation 28 is a design-review failure, not an assertion failure. An implementation that flags cross-endpoint Tag collisions is internally consistent and will pass its own properties. It is wrong because it encodes a false belief about PCIe (§3) — and no assertion can catch a specification error, only a review can.

14. Debugging a Multi-Endpoint System

Symptom — a device works alone and corrupts data when a second device is active. This is §3 until proven otherwise. Check what the Completion tracker matches on. If the key is the Tag rather than (Requester ID, Tag), the failure is by construction and it scales with the number of active endpoints. The tell is that severity tracks concurrency, not workload size.

Symptom — several ports report errors at once. Read the first-failure capture, not the current status. A cascade's newest error is a consequence — last-write-wins reports a non-root-cause 96.8% of the time (§11 Model 5). If the design only has a last-write-wins register, the timestamped log is more trustworthy than the register.

Symptom — several devices fail and none of them is broken. Look for a shared structure (§8). The device to investigate is the one that is not reporting errors — a stalled endpoint produces timeouts in its neighbours and complains about nothing itself. Model 4 measured 92% collateral with a shared queue.

Symptom — grant counters look perfectly fair, one endpoint is starved. Grants are not bytes (§7). Compare byte counters per port; a 32:1 byte ratio at equal grant counts is entirely possible with mixed payload sizes.

Symptom — credit or outstanding-count accounting drifts, and the total looks right. Check per port (§11 Model 2). A correct total is compatible with two ports being wrong in opposite directions, and the aggregate detected 0% of that case. If per-port bounds also pass, the remaining errors require per-transaction identity to find — which is the same tool as §3.

Symptom — one endpoint hangs on reads; its configuration is valid. Suspect the return path (§4). Verify that every bridge between the Completer and this requester has a Secondary/Subordinate range containing the requester's bus (Chapter 21.3 §5). A Completion that cannot route home produces a Completion Timeout at a device that did nothing wrong.

Symptom — throughput collapses when a specific pair of devices is active together. Consider placement (§6). Peer traffic between devices under the same switch need not cross the uplink; the same pair under different switches does. Module 22 owns the analysis; the topology question is the one to ask first.

A general rule: at fabric scale, ask whose before asking what. Every symptom in this list is produced by a structure that could not name the owner of the state it held.

15. Misconceptions

"Tag 5 identifies one transaction in the system." No. A Tag is unique per Requester (§3). Two endpoints holding Tag 5 simultaneously is legal and ordinary.

"Then Tags must be coordinated between devices." No — they never are, and nothing would be gained. The Requester ID already separates them.

"A switch tracks outstanding requests so it can return Completions." No. The switch holds no per-transaction state; the Completion carries its own return address and is routed by ID like anything else (§4).

"A Tag collision between two endpoints is an error the fabric should report." No — reporting it would reject correct traffic (mutation 28).

"One Completion means the read is done." No — a read may answer in several Completions (Chapter 20.3 §5), and retiring early releases a Tag while data is still in flight (§10, P7).

"Adding endpoints adds proportional throughput." No — Chapter 21.3 §7. Beyond the uplink, adding endpoints divides bandwidth rather than adding it.

"All traffic goes through the host." No. Peer-to-peer between two ports of one switch need not touch the uplink at all (§6) — though support is not universal.

"Equal grant counts mean equal bandwidth." No — 32:1 is achievable at equal grants (§7, §11 Model 3).

"If the aggregate counter balances, accounting is correct." No. 0% of compensating cross-port errors were detected by the total (§11 Model 2).

"Per-port counters solve accounting completely." Also no — they caught 4.3% of that injected class. The rest need per-transaction identity, the same answer as §3.

"The latest error tells you what went wrong." No — it tells you what went wrong last. It is a non-root-cause 96.8% of the time (§11 Model 5).

"One endpoint's failure is that endpoint's problem." Only if nothing is shared. With a shared queue, one stall blocked unrelated ports in 92% of runs (§8).

"Per-port queues are an optimization." They are a containment mechanism (§8) — the difference between 92% and 0% collateral.

"Interrupts need special fabric handling at scale." No — MSI/MSI-X are Memory Writes and route by address (§5). What scales is the contention they add, not the mechanism.

"A device reporting no errors during a fabric-wide failure is healthy." Frequently the opposite — it is the prime suspect (§14).

16. Understanding Check

Q1. Endpoint A (Bus 4) and Endpoint B (Bus 9) each issue a read with Tag 5. Is either wrong, and what distinguishes their Completions? Neither is wrong. Tags are per-Requester (§3). Each Completion carries the Requester ID of the device that asked, so the identities are 4:0.0/Tag 5 and 9:0.0/Tag 5 — distinct pairs. A tracker keyed on Tag alone misattributes 76.3% of the time (§11 Model 1), and misattribution means data delivered to the wrong device, not a dropped packet.

Q2. How much per-transaction state does a switch hold to route Completions back? None. The Completion carries its Requester ID, and each switch compares it against Secondary/Subordinate ranges (§4). That statelessness is why Completion routing costs nothing as endpoint count grows.

Q3. Eight ports, one shared egress queue, one endpoint stops accepting. What is reported, and by whom? Unrelated ports are blocked in 92% of runs (§11 Model 4), so several healthy devices report failures while the stalled one reports nothing. Per-port queues reduce collateral to 0%. The debugging rule: investigate the device that is not complaining (§14).

Q4. The aggregate credit total balances exactly. Can accounting still be wrong? Yes, and undetectably so at that level. A compensating cross-port error leaves the total correct; the aggregate detected 0% (§11 Model 2). Per-port bounds catch 4.3%, and the rest require per-transaction identity — the same fix as Q1, in a different subsystem.

Q5. Port A and Port B receive identical grant counts. Can B be receiving 32× A's bandwidth? Yes — 128 B versus 4096 B payloads (§7, §11 Model 3). Grants count packets; bandwidth is bytes, which is why §10 maintains both counters and P14 asserts they move independently.

Q6. Six ports latch errors within a few microseconds. Which one do you read, and why? The first-failure capture — the earliest error, latched once and locked (§10). A cascade ends with consequences, so the newest error is a non-root-cause 96.8% of the time (§11 Model 5). P15 and P16 exist to guarantee the record survives everything that follows it.

17. What's Next

Module 21 is complete. 21.1 made the routing decision, 21.2 moved a packet across a switch, 21.3 composed switches into a tree, and this chapter turned every endpoint on at once.

One idea runs through all four: a structure that cannot name the owner of the state it holds will eventually attribute it to the wrong owner. It appeared as ambiguous routing ranges, as packet-locked arbitration, as range containment, and here as (Requester ID, Tag) — four subsystems, one law.

Module 22 owns everything this module deliberately refused to quantify. This chapter published contention structure and no performance numbers: throughput analysis, latency decomposition, credit bottlenecks, payload-size effects, link efficiency and benchmark interpretation are all its scope. §7's fairness ratios and Chapter 21.3 §7's oversubscription arithmetic are topology consequences — they say who competes, not how fast anything runs.

Which is the right handoff: you now know where contention forms and whose traffic is whose. Module 22 measures what it costs.