Data Center Systems — The Endpoint Is Not the System

The five chapters before this one each traced one endpoint. Every one of them ended the same way: the fabric was innocent, the link was clean, and the fault was legible only from inside the device. This chapter is what happens when five such devices share a fabric — and the fault stops being inside any of them.

1. Sources, Scope, and What This Chapter Refuses to Do

2. The Model

At data-center scale, PCIe correctness remains transaction-local while performance, isolation and failure behaviour become topology-wide.

Four readings, and the fourth is why this chapter closes the module.

Correctness really does stay local, and that is not a consolation. Each TLP is well-formed or it is not; each Completion matches its request or it does not; each credit is returned or it is not (16.5). None of those properties get harder with more endpoints — which is precisely why every diagnostic that checks them will pass while the system fails.

Performance stops being a property of a device. 29.2 derived a device's achievable bandwidth from four factors it owned. At system scale a fifth factor appears that the device cannot see: what everyone else is doing.

Isolation and failure scope are new problems entirely. A single endpoint has no isolation question. Six endpoints sharing a switch subtree have one whether or not anybody designed an answer (§13–§14).

And the composition is not additive. Five correct devices on a shared fabric are not a correct system, in the same way five correct sentences are not an argument. Module 29's five chapters each found a device-internal blind spot; this one is about the blind spot between devices.

3. A Conceptual Topology

The one structural fact to carry forward. Each switch's upstream link is a single serving resource for everything beneath it. Nothing about that is exotic, and it is the origin of every problem in this chapter.

4. The Traffic Matrix

A topology drawing shows what is connected. A traffic matrix shows what competes — and they are different pictures.

Four readings.

The load-bearing observation: F1 and F3 share no endpoint whatsoever. An accelerator and a storage drive have no relationship — different vendors, different drivers, different software stacks, no transaction ever passes between them. They contend anyway, because both are served by A-upstream, and neither one's designers had any reason to think about the other.

Which means the competition graph is not the connectivity graph. Connectivity says "F1 and F3 both reach host memory." The competition graph says "F1 and F3 are rivals." Nothing in either endpoint's documentation, verification plan or benchmark reveals the second fact — it is a property of where they were plugged in.

The "latency-sensitive?" column is the one that gets lost. F1 at 8 GB/s and F3 at 1.2 GB/s look like a big flow and a small one. They are actually a flow that does not care and a flow that does — and the small one is the one with a deadline (29.1 §5: an NVMe command's PCIe share is small but its completion path is on the critical path).

And burstiness is a third axis that is not size. F5 offers 6 GB/s on average and, per 29.4 §9, spends most of its time near idle and the rest at its peak. Averaged into a capacity plan it is 6 GB/s; experienced by a shared link it is a series of collisions.

5. Oversubscription — Arithmetic, Not an Accusation

Peak sum ÷ shared service:

24 ÷ 16 = 1.5 : 1 oversubscription

And with F4's opposite-direction traffic and a fourth endpoint the ratio would rise. The number is not the point. Its status is.

Three readings.

Oversubscription is a design decision, and the failure is leaving it implicit. Provisioning every downstream port's peak on the upstream path is usually wasteful — most workloads genuinely do not activate everything at once. The engineering act is stating which workload you are betting on, so that a later change of workload is recognisable as invalidating the bet rather than as "the fabric got slower."

The claim that must be defensible is a concurrency claim, not a bandwidth claim. "Can the target workload activate all three simultaneously?" is answerable from the software architecture — a training job that overlaps copy with compute (26.6) answers yes; a batch pipeline that stages one phase at a time answers no. These are different systems with the same hardware.

And the ratio understates the risk when the flows are bursty. With F5's character, the peak sum is reached far more often than the average utilisation suggests — which is 29.4 §4's lesson arriving at the fabric level: averages describe the wrong quantity when the failure is a tail event.

6. RTL — Per-Flow Service Accounting

The whole chapter turns on one instrumentation decision: aggregate or per-class.

// ILLUSTRATIVE. Per-flow service accounting at a shared serving point. The
// design rule is that EVERY counter is indexed by flow class, because the
// failure this chapter is about is invisible in any total (§7).
localparam int NCLASS = 4;                 // e.g. bulk, stream, storage, control
localparam int CW     = 48;
 
logic [CW-1:0] offered_bytes_q  [NCLASS];  // presented to the serving point
logic [CW-1:0] accepted_bytes_q [NCLASS];  // actually taken (valid && ready)
logic [CW-1:0] useful_bytes_q   [NCLASS];  // delivered, retries excluded (29.2)
logic [CW-1:0] stall_cyc_q      [NCLASS];  // wanted service, did not get it
logic [15:0]   occupancy_q      [NCLASS];  // shared-resource entries held
logic [15:0]   occ_high_water_q [NCLASS];  // peak, not a sample (29.4 §7)
logic [CW-1:0] grants_q         [NCLASS];
 
// The metric that no total can express: the longest interval a class waited
// while OTHER classes were being served. This is the starvation measurement.
logic [31:0] starve_cyc_q       [NCLASS];
logic [31:0] starve_max_q       [NCLASS];
 
// Measurement epoch. A window whose configuration changed midway is not a
// measurement, so the snapshot is stamped and software compares stamps.
logic [7:0]  meas_epoch_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n || ctr_clear) begin
    for (int c = 0; c < NCLASS; c++) begin
      offered_bytes_q[c] <= '0; accepted_bytes_q[c] <= '0;
      useful_bytes_q[c]  <= '0; stall_cyc_q[c]     <= '0;
      occupancy_q[c]     <= '0; occ_high_water_q[c] <= '0;
      grants_q[c]        <= '0; starve_cyc_q[c]    <= '0;
      starve_max_q[c]    <= '0;
    end
    meas_epoch_q <= meas_epoch_q + 8'd1;   // a clear starts a NEW window
  end else begin
    for (int c = 0; c < NCLASS; c++) begin
      if (offer_fire[c])  offered_bytes_q[c]  <= offered_bytes_q[c]  + CW'(offer_bytes[c]);
      // Accepted, never offered: valid && ready, the same discipline 30.2 makes
      // a checklist item.
      if (accept_fire[c]) accepted_bytes_q[c] <= accepted_bytes_q[c] + CW'(accept_bytes[c]);
      if (deliver_fire[c] && !deliver_is_retry[c])
        useful_bytes_q[c] <= useful_bytes_q[c] + CW'(deliver_bytes[c]);
 
      // "Wanted and did not get" — the only evidence that separates a starved
      // class from an idle one. Without the want-gate this counter is noise.
      if (want_service[c] && !grant_fire[c]) stall_cyc_q[c] <= stall_cyc_q[c] + CW'(1);
 
      // Starvation interval: reset on service, high-water otherwise.
      if (grant_fire[c]) begin
        grants_q[c]      <= grants_q[c] + CW'(1);
        starve_cyc_q[c]  <= '0;
      end else if (want_service[c]) begin
        starve_cyc_q[c]  <= starve_cyc_q[c] + 32'd1;
        if (starve_cyc_q[c] > starve_max_q[c]) starve_max_q[c] <= starve_cyc_q[c];
      end
 
      // One signed next-state expression, so an insert and a retire in the same
      // cycle net correctly instead of losing one (30.2's two-NBA item).
      occupancy_q[c] <= occupancy_q[c] + 16'(insert_fire[c]) - 16'(retire_fire[c]);
      if (occupancy_q[c] > occ_high_water_q[c]) occ_high_water_q[c] <= occupancy_q[c];
    end
  end
end

Architecture. Seven per-class accumulators, a per-class peak, and a per-class starvation high-water mark. It exists because the system-level failure mode is a distribution across classes, and a distribution cannot be recovered from its sum.

State. Byte, cycle and occupancy accumulators, all per class. starve_max_q is the one metric with no aggregate equivalent at all — there is no total you can compute that tells you the longest any class waited.

Event. accept_fire is valid && ready, never valid alone; useful_bytes excludes retries (29.2 §7); stall_cyc is gated on want_service, which is what separates starved from idle.

Contract. The class index must be assigned once, at the point a request enters, and never recomputed downstream. If two stages classify independently they will disagree, and useful_bytes will not sum to the global total — which §13's first assertion checks precisely because this is the realistic error.

Failure. The likely mistakes are all classification, not arithmetic: an unclassified flow silently landing in class 0 inflates one class and hides another; want_service tied high makes idle classes look starved and destroys the metric's meaning.

DV/debug. starve_max_q versus a design-defined bound is the one-read starvation test. grants_q ratios against offered ratios show the service split. And occ_high_water_q per class shows which class is holding the shared resource, which is the head-of-line question (§9).

7. Wrong RTL — the Aggregate That Hides a Starved Class

// WRONG. ILLUSTRATIVE. The counter a shared serving point actually tends to
// get: one number, because one number is what a capacity dashboard wants.
logic [CW-1:0] total_bytes_q;
logic [CW-1:0] elapsed_cyc_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    total_bytes_q <= '0; elapsed_cyc_q <= '0;
  end else begin
    // BUG 1: no class index. Every flow's bytes land in one accumulator, so the
    //        SPLIT between flows — the only thing that matters here — is
    //        arithmetically unrecoverable afterwards.
    if (deliver_fire) total_bytes_q <= total_bytes_q + CW'(deliver_bytes);
    elapsed_cyc_q <= elapsed_cyc_q + CW'(1);
  end
end
 
// BUG 2: reported as utilisation against capacity. This is a work-conservation
//        metric. It is HIGH precisely when the serving point is busiest, which
//        is exactly when a class is most likely to be starved.
assign utilisation_pct = (total_bytes_q * 100) / (elapsed_cyc_q * BYTES_PER_CYC);
 
// BUG 3: no starvation state anywhere. There is no register whose value would
//        differ between "all classes served proportionally" and "one class
//        served, one class receiving nothing".

Architecture. One accumulator, one derived percentage.

State. total_bytes_q. The missing state is the class index — one field, and its absence makes the failure mathematically invisible rather than merely hard to see.

Event. Every delivery, unclassified.

Contract. Whoever reads utilisation_pct believes it describes whether the fabric is healthy. It describes whether the fabric is busy, and those diverge exactly under contention.

Failure — the timeline. F1 and F2 (bulk, 8 GB/s each) run alongside F3 (NVMe, 1.2 GB/s, latency-sensitive) on A-upstream's 16 GB/s.

First divergence: 20–30 s — F3's share began falling while the aggregate rose. Every subsequent interval made the dashboard look better and the system worse.

Root cause. A work-conserving metric measured at a contended resource is maximised by the outcome this chapter is about. Give one class everything and the total is at its highest. The aggregate is not merely blind to starvation; it is positively correlated with it.

BUG 3 is why nobody can argue back. There is no register anywhere in this design whose value distinguishes proportional service from total starvation. The claim "storage is being starved" has no supporting evidence available, so it loses to the dashboard.

DV/debug. The signature is rising utilisation with rising tail latency on one class, and the discriminator is §6's starve_max_q plus per-class grants_q. One number per class ends the argument in a minute.

8. Fairness Is Not Throughput

Three readings.

A system can be 100 % utilised and architecturally unacceptable, and §7's final row is exactly that state. Utilisation is a cost metric — am I wasting the resource — and it was never a correctness metric. Using it as one is the category error at the heart of this chapter.

Which means the quantities to measure are the four in row 3. Per-class accepted and delivered (they differ, and the difference is where retries and drops live), maximum starvation interval, and latency tail — never the mean (29.4 §4).

And this chapter deliberately does not tell you what arbitration policy PCIe provides. §1 explains why: no specification was inspected. What a platform architect can always do is state the required per-class floor as a system requirement, then measure whether it is met — which is policy the architect owns, and it is what 30.1 will demand be written down.

9. Head-of-Line — Holding Versus Using

The distinction that makes shared-resource contention worse than plain bandwidth sharing.

A large transfer does not merely use bandwidth; while it is in progress it holds an entry in a shared structure. A small, urgent request behind it waits for the holding to end, not for its own tiny service time to elapse.

Three readings.

This is why F3's 1.2 GB/s is not "small enough to fit." Its bytes are trivial against the link; its wait is set by what is in front of it, and a bulk flow's occupancy is long-lived by construction.

The architectural risk statement is generic and does not require knowing any switch's internals — which §1 forbids inventing. Wherever two classes share a queue, the class that occupies it longest determines the other's latency. That sentence is design-independent, and it is enough to drive the requirement.

And §6 measures it directly with a counter already present. occ_high_water_q[bulk] near the structure's capacity, with starve_cyc_q[storage] non-zero at the same time, is the head-of-line signature — one class holding, another wanting.

10. A Progress Floor — System Policy, Not a PCIe Requirement

The requirement to write down: a progress-sensitive class receives at least a stated share of service within a stated interval, whatever the bulk class offers.

And the design must be work-conserving anyway — a reservation that idles the resource when the protected class has nothing to send is a permanent tax paid for an occasional benefit. Borrowing is what makes a floor affordable.

11. RTL — A Two-Class Arbiter With a Floor

// ILLUSTRATIVE. Two classes, a guaranteed floor for the progress class, and
// full borrowing so nothing is wasted when the floor is unused. The whole
// design is three counters and one comparison.
localparam int WINDOW_CYC = 1024;   // the interval the floor is defined over
localparam int FLOOR_GRANT = 64;    // minimum grants the progress class gets
 
logic [15:0] win_cyc_q;             // position within the current window
logic [15:0] prog_grant_q;          // progress-class grants this window
logic [15:0] prog_deficit_q;        // floor grants still owed this window
 
// Accepted events only — valid && ready. A grant asserted into a stalled
// consumer is not service, and counting it as service is how a floor gets
// reported as met while the class receives nothing (§7's discipline).
wire bulk_fire = bulk_valid && bulk_ready && grant_bulk;
wire prog_fire = prog_valid && prog_ready && grant_prog;
 
// Cycles remaining in the window, and grants still owed. If the progress class
// cannot still make its floor by coasting, it MUST be served now. This is the
// entire policy in one expression.
wire [15:0] cyc_left   = 16'(WINDOW_CYC) - win_cyc_q;
wire        must_serve = prog_valid && (prog_deficit_q >= cyc_left);
 
// Priority: urgency first, then bulk, then progress opportunistically. The
// third term is the borrowing that keeps this work-conserving in reverse —
// the progress class takes spare capacity when bulk is idle.
assign grant_prog = must_serve || (prog_valid && !bulk_valid);
assign grant_bulk = !grant_prog && bulk_valid;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    win_cyc_q <= '0; prog_grant_q <= '0;
    prog_deficit_q <= 16'(FLOOR_GRANT);
  end else if (win_cyc_q == 16'(WINDOW_CYC) - 16'd1) begin
    // Window boundary: restart. The deficit is RESET, not carried, so a window
    // in which the progress class was idle does not create a debt that starves
    // bulk in the next one.
    win_cyc_q      <= '0;
    prog_grant_q   <= '0;
    prog_deficit_q <= 16'(FLOOR_GRANT);
  end else begin
    win_cyc_q    <= win_cyc_q + 16'd1;
    if (prog_fire) begin
      prog_grant_q   <= prog_grant_q + 16'd1;
      // Saturating decrement — the deficit must never underflow to a huge
      // value, which would make must_serve permanently true and invert the
      // starvation (30.2's unsigned-subtraction item).
      prog_deficit_q <= (prog_deficit_q == '0) ? '0 : prog_deficit_q - 16'd1;
    end
  end
end

Architecture. A windowed deficit rather than a fixed reservation. The floor is expressed as "grants still owed", and urgency is "owed ≥ time remaining" — so the progress class is served late by default and early only when it must be, which maximises the bulk class's uninterrupted run.

State. win_cyc_q, prog_grant_q, prog_deficit_q. prog_deficit_q is the load-bearing register: it converts a rate guarantee into a same-cycle comparison, with no division and no rate estimation.

Event. Both _fire signals are valid && ready. prog_deficit_q decrements on service delivered, never on grant asserted — and the distinction is the whole difference between a floor that holds and one that is merely reported.

Contract. WINDOW_CYC and FLOOR_GRANT must come from the progress class's actual deadline (29.1 §5 is where an NVMe latency budget is derived), not from a round number. A floor chosen without a deadline behind it cannot be argued for or against in review — which is exactly what 30.1 §46 asks architecture to supply.

Failure. Three realistic errors: the deficit carried across windows, which converts a quiet period into a debt that later starves bulk; prog_deficit_q decremented on grant_prog rather than prog_fire, which reports a met floor while the class is stalled; and unsigned underflow, guarded above, which would pin must_serve high forever.

DV/debug. prog_grant_q at each window boundary versus FLOOR_GRANT is the direct check, and §13's a_floor_met_each_window is it as an assertion. starve_max_q from §6 bounds the worst case across windows, which is the number a system requirement should actually state.

12. Wrong RTL — Strict Priority

// WRONG. ILLUSTRATIVE. Strict priority. It is the natural first implementation,
// it is correct in every functional test, and it starves.
assign grant_bulk = bulk_valid;                 // BUG: unconditional precedence
assign grant_prog = !bulk_valid && prog_valid;  // served only in the gaps

Architecture. One line of precedence and no state at all. The absence of state is the tell — a fairness or progress property is a statement about time, and a design with no time in it cannot express one.

Failure — the cycle timeline. Bulk holds valid continuously from cycle 3 (F1 and F2 together always have something to send). The progress class asserts at cycle 5.

First divergence: cycle 5 — the progress class asserted and was not served, and nothing in the design will ever change that while bulk stays valid.

Root cause. Strict priority is a statement about instants; starvation is a statement about intervals. The RTL is correct at every single cycle and wrong over every window. This is why functional tests pass: a test that checks "the right requester won" confirms the bug.

And the system-level symptom is §7's table, arrived at from the other end: total throughput is maximal, because bulk never yields. The two wrong designs — the aggregate counter and the strict-priority arbiter — reinforce each other, one producing the starvation and the other certifying it as excellent.

DV/debug. starve_max_q[progress] grows without bound while utilisation_pct sits at 100 %. A test must run long enough to see an interval, which is why §17's coverage requires sustained concurrent load rather than transaction-level checks.

13. Assertions — Labelled as Architecture-Level

// MANDATORY. English: every accepted transfer is counted in exactly one class.
// Catches the realistic instrumentation failure — two stages classifying
// independently — which silently corrupts every per-class conclusion in §6.
a_classified_exactly_once: assert property (
  @(posedge clk) disable iff (!rst_n || ctr_clear)
    accept_any |-> ($onehot(accept_fire))
);
 
// MANDATORY. English: per-class useful bytes sum to the global useful total.
// An arithmetic identity over the instrument, not the design — the same shape
// as 29.1's stage-sum and 29.2's payload closure.
a_class_bytes_sum_to_total: assert property (
  @(posedge clk) disable iff (!rst_n || ctr_clear)
    (useful_bytes_q[0] + useful_bytes_q[1]
   + useful_bytes_q[2] + useful_bytes_q[3]) == global_useful_bytes_q
);
 
// MANDATORY. English: a disabled or unowned endpoint never receives a grant.
// This is the isolation property — the one that matters for fault containment
// (§14) rather than performance.
a_disabled_never_granted: assert property (
  @(posedge clk) disable iff (!rst_n)
    grant_fire_ep |-> ep_enabled[grant_ep_id]
);
 
// MANDATORY. English: the progress class meets its floor in every completed
// window in which it had work. The "had work" antecedent is essential — an
// idle class cannot be starved, and asserting otherwise produces false fails
// that get the assertion deleted.
a_floor_met_each_window: assert property (
  @(posedge clk) disable iff (!rst_n)
    (win_cyc_q == 16'(WINDOW_CYC) - 16'd1) && prog_had_work
      |-> (prog_grant_q >= 16'(FLOOR_GRANT))
);
 
// MANDATORY. English: the measurement epoch does not change inside a window.
// A window whose configuration changed midway is not a measurement, and a
// performance conclusion drawn across an epoch boundary is not evidence.
a_epoch_stable_in_window: assert property (
  @(posedge clk) disable iff (!rst_n)
    (win_cyc_q != '0) |-> $stable(meas_epoch_q)
);
 
// MANDATORY. English: a reset confined to one fault domain never retires
// transactions belonging to another. This is §14's blast-radius property, and
// it is the single most valuable assertion in this chapter.
a_reset_scope_contained: assert property (
  @(posedge clk) disable iff (!rst_n)
    domain_reset_fire |->
      ##1 !$fell(txn_live[other_domain_txn])
);

Four readings.

The second is the identity that makes every other per-class number trustworthy. If the classes do not sum to the total, some flow is uncounted or double-counted, and §7's argument cannot be made from the data. It fails loudly at the instrument rather than quietly at the conclusion.

The fourth's prog_had_work antecedent is not a detail. Without it, every window in which the progress class was idle fails the assertion. The predictable outcome is that somebody disables it, which is worse than never having written it — the same reasoning that put stat_clear in every disable term in 29.4 §8.

The fifth protects measurements rather than function, and it belongs here because §5's oversubscription claim and §7's timeline are both statements about windows. A number gathered across a configuration change is not evidence, and 30.1 will make that a review question.

And the sixth is the one to carry into Module 30. It is a containment property, not a performance one, and §14 is why. It is also the hardest to write, because it requires the design to know what a fault domain is — which is an architecture question that 30.1 §42 puts on the checklist precisely because designs reach RTL without an answer.

14. Fault Scope Versus Ownership

Correctness is transaction-local (§2). Failure is not.

The question a platform must be able to answer before it recovers anything:

Wrong system behaviour — recovery scope wider than ownership. One accelerator beneath switch A hangs. Software resets the switch subtree.

Three readings.

The blast radius was decided long before the incident — at the moment the topology put an accelerator and a storage device under one switch, and nobody wrote down what a recovery of one may do to the other. This is an architecture decision made by omission, which is 30.1's central concern.

Step 6 is the lasting damage. The evidence now shows storage failing at the same moment an accelerator recovered, and the causal arrow points backwards from what the data suggests. Teams spend weeks on the storage stack.

And the corrective is a sentence, not a mechanism: recovery scope must match ownership. If the two cannot be separated, that is a topology finding, and the honest outcomes are to change the topology, to accept the coupling explicitly, or to make the recovery narrower. All three are fine; leaving it undecided is not.

15. Observability at System Scale

25.9 established that an analyser sees the wire. At system scale the wire is a smaller fraction of the truth than ever.

Three readings.

The last row is the trap, and it is Module 29's recurring one. The single question an analyser answers definitively is the single question whose answer is "everything is fine." Five chapters in a row ended with the fabric exonerated, and at system scale it is exonerated across every link at once.

Correlation requires a shared time base and a shared identity, and neither is free. Counters from three sources with no common timestamp cannot establish an ordering, so "the NIC burst preceded the storage timeout" becomes unprovable — which is precisely the claim an investigation needs.

And the practical minimum is smaller than it sounds. Per-class delivered bytes and starvation maximum at each shared point, plus first-fault capture per endpoint, plus one timestamp source. That is a handful of registers and it converts most of this chapter's failures into a single read.

16. Flagship Failure — Every Component Passes Alone

The report: "GPU benchmark: passes. NVMe benchmark: passes. NIC benchmark: passes. Run all three: storage tail latency goes up 40×, and there are no PCIe errors anywhere."

Three readings.

Both wrong diagnoses were reasonable and both were encouraged by the available evidence. The drive looks slow; the link looks perfect. Every instrument present pointed away from the answer, and the one that would have pointed at it did not exist.

The reframe is the transferable skill. "Which flows share a serving resource?" is a question about the topology that takes minutes to answer and is not answerable from any endpoint's documentation. It is the system architect's contribution, and nobody else is positioned to make it.

And the fix menu has three entries with different costs, which is what makes this an architecture conversation rather than a bug fix. A floor costs arbitration complexity; reducing oversubscription costs links; moving the drive costs a topology change. Choosing among them requires §5's workload claim to have been written down — and if it never was, there is no basis for choosing.

17. Verification at System Scale

Four readings.

"All classes simultaneously active" is the coverage point that does not exist in most plans, because each endpoint's team wrote their own. Nobody owns the cross product, and the cross product is where this chapter's failures live.

Aggregate throughput must not be the pass criterion, or the suite certifies §12's starving arbiter as optimal. This is the direct verification consequence of §8: if utilisation is the metric, the broken design scores best.

The third negative case is the one teams skip because it requires a testbench that models two independent domains with concurrent live work. It is also the only way to test the property that prevents §14's incident.

And a duration requirement in a verification plan is unusual enough to justify. A window-based floor (§11) cannot be evaluated in fewer than several windows; a 100-transaction directed test proves nothing about it, however carefully written.

18. Misconceptions

"If every endpoint passes, the system passes." §16: all three passed alone, and the combined system failed 40× on its most latency-sensitive path.

"High utilisation means the fabric is healthy." §7, §8: a work-conserving metric is maximised by starving one class. 100 % utilisation was the worst row in the table.

"Flows that never talk to each other don't interact." §4: F1 and F3 share no endpoint, no driver and no transaction — and they are rivals, because they share an upstream link.

"Oversubscription is a design bug." §5: it is normal and usually correct. Leaving the workload concurrency assumption unstated is the bug.

"The small flow will fit in the gaps." §9: it waits for the big flow to stop holding the shared resource, not for its own service time. Size and latency are different axes.

"An analyser will find it." §15: every TLP is legal. The analyser's confident answer is the wrong question's.

"Resetting the subtree is a safe recovery." §14: it terminated an innocent drive's outstanding commands and made the post-mortem point at the wrong device.

"Strict priority is fine because the important class is on top." §12: bulk was on top, and the design has no state, so it cannot express a property about intervals at all.

"Average per-class rates prove fairness." §8, and 29.4 §4: the quantity that matters is the maximum starvation interval, which no average contains.

19. Understanding Check

Q1. Three endpoints pass their benchmarks individually. Run together, storage tail latency rises 40× with zero PCIe errors. Where do you look?

At the competition graph, not the connectivity graph (§4, §16). Ask which flows share a serving resource: in §3's topology F1, F2 (bulk accelerators) and F3 (NVMe) all traverse A-upstream, so they are rivals despite sharing no endpoint, no driver and no transaction. That relationship appears in no endpoint's documentation — it is a property of where things were plugged in, and only a system architect is positioned to state it.

Then read per-class counters, never the aggregate. §7's timeline shows the failure signature exactly: bulk delivered 15.9 GB/s, storage 0.1 GB/s, and utilisation_pct at 100 % — the dashboard's best-looking number coinciding with the system's worst state. The first divergence is the interval where storage's share began falling while the total rose. Root cause is strict-priority service (§12) at a point oversubscribed 1.5 : 1 (§5) with no progress floor (§10). Both wrong first diagnoses — a throttling drive, a bad link — were actively supported by the available evidence, which is why the reframe matters more than the measurement.

Q2. Why is an aggregate utilisation counter not merely insufficient here, but actively misleading?

Because it is a work-conservation metric, and work conservation is maximised by the failure (§7, §8). Give one class everything and the total is at its highest — so utilisation is positively correlated with starvation, not merely blind to it. §7's table shows the aggregate rising through 98.75 % to 100 % as storage's share collapsed to 0.1 GB/s and then to timeouts.

And the structural problem is that the split is arithmetically unrecoverable. One accumulator with no class index cannot be decomposed afterwards by any analysis — the information was never captured (§7 BUG 1). Worse, §7's design contains no register anywhere whose value differs between proportional service and total starvation (BUG 3), so the claim "storage is being starved" has no supporting evidence and loses to the dashboard. The fix is per-class delivered, grants, and starve_max — and starve_max has no aggregate equivalent at all: there is no total you can compute that tells you the longest any class waited.

Q3. Why does a strict-priority arbiter pass every functional test?

Because it is correct at every instant and wrong over every interval (§12). At each individual cycle the higher-priority requester wins, which is exactly what the design intends and exactly what a transaction-level test checks — so a test that verifies "the right requester was granted" confirms the bug. The tell is that the design has no state: a progress or fairness property is a statement about time, and RTL containing no time cannot express one.

§12's timeline makes it concrete: bulk asserts at cycle 3 and holds; the progress class asserts at cycle 5 and is never served through cycle 1023 — 0 grants out of a 64-grant floor. Catching it requires stimulus with concurrent sustained multi-class load running long enough to contain many windows (§17), plus a checker on per-class delivered and starve_max rather than aggregate throughput. If the pass criterion is utilisation, the suite scores the broken arbiter as optimal — which is why §17 makes the negative case "replace the arbiter with strict priority and confirm the suite fails."

Q4. One accelerator hangs and the platform resets the switch subtree. What went wrong, and when?

The blast radius exceeded the ownership boundary, and the decision was made at topology time (§14). NVMe (A) had outstanding commands, they were terminated, and the filesystem above surfaced I/O errors — all to a device that had done nothing wrong. The lasting damage is step 6: the post-mortem sees storage errors coincident with an accelerator recovery and the causal arrow points backwards, so teams investigate the storage stack for weeks.

The failure was an architecture decision made by omission. Nobody wrote down what a recovery of one endpoint may do to its siblings, and the coupling was created the moment an accelerator and a drive were placed under one switch. The corrective is a sentence — recovery scope must match ownership — and three acceptable outcomes: change the topology, narrow the recovery, or accept the coupling explicitly. Leaving it undecided is the only unacceptable one. The property is testable: a_reset_scope_contained (§13), driven by the negative case in §17 that resets one domain with another domain's work live.

20. Module 29 Complete — The Endpoint Is Not the System

The through-line. Every one of the first five chapters found a failure that was legible from inside the device and invisible from outside it, and each was fixed by a few registers costing no meaningful area. This chapter's failures are legible from inside the system and invisible from inside any device — and the instrument is again a handful of counters, now per class instead of per stage.

Which produces the module's conclusion. The endpoint is not the system. Correctness composes; performance, isolation and failure scope do not. A design review that verifies every endpoint and never asks which flows share a serving resource, or what a recovery of one does to its siblings, has verified everything except the system.

And that is the question Module 30 exists to ask. 30.1 opens it with the gate that comes before all the others: is the architecture explicit enough that independent teams could implement, verify, integrate and debug it without tribal knowledge? Every failure in Module 29 traces back to something nobody wrote down.