Throughput Analysis

Chapter 15.4 produced a number: the package's sustainable throughput, computed from capacities and demand. The silicon will deliver less.

This chapter is about accounting for the difference — every unit of it.

1. The One-Sentence Model

Theoretical throughput is a ceiling. Achieved throughput is that ceiling minus time and capacity lost to identifiable causes. Every missing cycle must be attributable to exactly one cause, and the residual must go to zero.

The last clause is the chapter's discipline and the one most performance work skips. A 5% unexplained residual is not "close enough" — it is a 5% hole in your understanding, and holes in understanding are where the next surprise lives.

2. What This Chapter Owns

This is not 15.1's equations again. 15.1 built the model of useful bandwidth. This chapter is about measurement and attribution: classifying every observation cycle, preventing the double-counting that makes stall fractions sum to 180%, running the sweeps that identify which resource is binding, and closing the residual.

Specifically new here: cycle-state classification as a one-hot partition; one primary stall cause and the causal priority order behind it; the cycle and byte accounting invariants; atomic snapshots for a whole counter bank rather than one counter; measurement windows, warmup and drain; seven sweeps — payload size, outstanding depth, credit depth, replay window, queue depth, link rate, lane width — each of which identifies a different binding resource; six behavioural signatures; bottleneck migration as a methodology rather than an observation; A/B discipline; four worked cases; and the residual-zero principle that makes the whole thing a proof rather than a guess.

3. Sourcing

4. Start From the Ceiling

Before measuring anything, compute what the measurement should be compared against — otherwise "62 GB/s" is a number without a verdict.

STEP 1 — raw, from verified inputs (15.1 Section 12)
  raw_bw = transfer_rate × active_data_lanes × bits_per_transfer
 
STEP 2 — after framing (15.1 Section 19)
  theoretical_useful_bw = raw_bw × framing_efficiency
 
STEP 3 — after the best achievable packing
  theoretical_payload_bw = theoretical_useful_bw × max_payload_occupancy

Worked, with verified inputs and one illustrative packing figure:

  transfer rate      = 32 GT/s per lane            [VERIFIED]
  active data lanes  = 16                          [VERIFIED cluster size]
  bits per transfer  = 1                           [VERIFIED for UCIe-S @ 32 GT/s]
  flit               = 68 B, 4 B overhead          [VERIFIED]
  max occupancy      = 1.00                        [ILLUSTRATIVE: perfect packing]
 
  raw               = 32e9 × 16 × 1 = 512 Gb/s = 64.0 GB/s per direction
  framing           = 64/68 = 0.9412
  theoretical useful = 64.0 × 0.9412 = 60.24 GB/s
  theoretical payload = 60.24 × 1.00 = 60.24 GB/s

Three properties of the ceiling.

It is a ceiling, and nothing measured may exceed it. A measurement above it is always an instrumentation bug — a unit error, a boundary mismatch, or 15.1 §27's valid-without-ready counter. Never a discovery (§35).

It already excludes framing, so the gap this chapter explains is not framing. Framing is structural and known; the interesting loss is time and replay, which is what §5 decomposes.

And max_payload_occupancy must be honest. Setting it to 1.00 assumes perfectly packed containers, which most workloads do not achieve. If the workload's object sizes cannot fill a container, that is a real and permanent loss and belongs in the ceiling rather than in the residual — otherwise §33's residual can never close.

5. The Decomposition

achieved_payload_bw =
    theoretical_payload_bw
  × useful_cycle_fraction        // Section 6: cycles that transferred NEW payload
  × non_replay_fraction          // Section 22
  × actual_payload_occupancy     // Section 24, relative to max
 
Units: bytes/s × (dimensionless)³ = bytes/s              ✓

Every factor is at most 1, so the product is at most the ceiling — which makes the arithmetic self-checking.

The relationship to the earlier chapters, stated once so the vocabulary stays fixed — these three words are used consistently across Module 15 and are not interchangeable:

These are not interchangeable and this chapter does not switch between them casually. A statement about raw bandwidth says nothing about achieved throughput, and the whole discipline below exists to connect them.

6. Cycle-State Classification

The signature technique of the chapter. Every observation cycle is assigned to exactly one state.

Two structural properties to read off it. Every tap reports into one bank, which is what makes the one-hot classification of §9 possible — six independent condition detectors feeding six independent counters is §11's bug, and a single bank with a priority classifier is the fix. And the snapshot sits between the bank and the model, because every figure in this chapter is a ratio across many counters, so the model must read a coherent set (§15) or §12's and §13's invariants fail on the snapshot even though they held continuously in hardware.

Three properties of the taxonomy.

ST_SRC_EMPTY is deliberately not a loss. A link with nothing offered is idle because the workload had nothing to send. Counting it as a stall makes a lightly loaded link look broken and hides real stalls inside a large idle count — the point 13.5 §34 and 15.1 §22 both make, and it is why §7's fractions are reported against both totals.

ST_USEFUL and ST_REPLAY are both transfers. The link is busy in both. That distinction is why "the link is 100% utilised" and "the link delivers 100% of its useful capacity" are different statements (§30).

And the taxonomy must be complete. Any cycle that fits none of the ten is unattributed, and §12's invariant makes that visible rather than silently absorbed. An unattributed cycle is a hole in the classifier, not a mystery about the design.

7. Two Denominators, Both Reported

useful_cycle_fraction        = useful_cycles / observation_cycles
useful_fraction_when_offered = useful_cycles / (observation_cycles − src_empty_cycles)

Report both, always. They answer different questions:

Worked, illustrative:

observation = 1,000,000 cycles
src_empty   =   400,000
useful      =   480,000
 
against observation: 480,000 / 1,000,000        = 0.480   → 48%
against offered:     480,000 /   600,000        = 0.800   → 80%

48% reads as a serious problem; 80% reads as a well-behaved link that is under-fed. Both are true, and reporting only the first sends the investigation to the interconnect when the answer is the producer (§19).

8. The Counter Bank

// ILLUSTRATIVE performance counter bank. No UCIe counter definition is
// claimed (Section 3). One counter per cycle state, plus byte counters.
//
// 64-bit because 15.1 Section 31's arithmetic shows 32 bits wraps in tens of
// milliseconds at these rates.
typedef struct packed {
  logic [63:0] cycles;                 // observation window
  logic [63:0] useful_cycles;
  logic [63:0] replay_cycles;
  logic [63:0] src_empty_cycles;
  logic [63:0] no_credit_cycles;
  logic [63:0] replay_full_cycles;
  logic [63:0] queue_stall_cycles;
  logic [63:0] arb_stall_cycles;
  logic [63:0] target_stall_cycles;
  logic [63:0] link_down_cycles;
  logic [63:0] recovery_cycles;
  logic [63:0] unattributed_cycles;    // MUST stay at zero (Section 12)
  // bytes
  logic [63:0] transmitted_bytes;
  logic [63:0] unique_payload_bytes;
  logic [63:0] replay_bytes;
  logic [63:0] framing_bytes;
} throughput_counters_t;
 
throughput_counters_t perf_q;

Architecture. One counter per cycle state and one per byte category, so every term of §5's decomposition has a counter and every counter maps to a term. That correspondence is the design: a counter that maps to no term cannot be interpreted, and a term with no counter cannot close a residual.

State. All observation-window lifetime, cleared together. Clearing any subset breaks every ratio, because the ratios assume a common window — and §14's snapshot exists so software never has to read them at different instants.

Cycle behaviour. Exactly one cycle counter increments per cycle (§9). Byte counters increment only on transfers, qualified by valid && ready (15.1 §27).

Contract. Two invariants (§12, §13) must hold at every window boundary. They are the difference between numbers and measurements.

Failure. §11's independent counters. Or omitting unattributed_cycles, which hides a classifier hole rather than exposing it — the counter that should always read zero is the one that proves the other eleven are complete.

DV. Assert both invariants every window. Then drive a workload that produces each state in isolation and confirm only that counter moves.

9. The Cycle Classifier

// ILLUSTRATIVE one-hot cycle classifier. The PRIORITY ORDER is a design
// decision representing causal order (Section 10), not severity.
typedef enum logic [3:0] {
  ST_USEFUL      = 4'd0,
  ST_REPLAY      = 4'd1,
  ST_SRC_EMPTY   = 4'd2,
  ST_LINK_DOWN   = 4'd3,
  ST_RECOVERY    = 4'd4,
  ST_NO_CREDIT   = 4'd5,
  ST_REPLAY_FULL = 4'd6,
  ST_QUEUE       = 4'd7,
  ST_ARB         = 4'd8,
  ST_TARGET      = 4'd9,
  ST_UNATTRIB    = 4'd10
} stall_reason_t;
 
stall_reason_t reason_d;
 
always_comb begin
  unique case (1'b1)
    // --- transfers first: the link is doing work
    beat_transferred && !beat_is_replay : reason_d = ST_USEFUL;
    beat_transferred &&  beat_is_replay : reason_d = ST_REPLAY;
    // --- not the link's loss
    !work_offered                       : reason_d = ST_SRC_EMPTY;
    // --- the link is unusable at all
    !link_operational                   : reason_d = ST_LINK_DOWN;
    recovery_active                     : reason_d = ST_RECOVERY;
    // --- resources, in causal order (Section 10)
    !credit_available                   : reason_d = ST_NO_CREDIT;
    !replay_space                       : reason_d = ST_REPLAY_FULL;
    !queue_space                        : reason_d = ST_QUEUE;
    !arb_grant                          : reason_d = ST_ARB;
    !target_ready                       : reason_d = ST_TARGET;
    // --- the classifier is incomplete if this is ever reached
    default                             : reason_d = ST_UNATTRIB;
  endcase
end
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n || perf_clear) begin
    perf_q <= '0;
  end else begin
    perf_q.cycles <= perf_q.cycles + 64'd1;
    unique case (reason_d)
      ST_USEFUL      : perf_q.useful_cycles       <= perf_q.useful_cycles       + 64'd1;
      ST_REPLAY      : perf_q.replay_cycles       <= perf_q.replay_cycles       + 64'd1;
      ST_SRC_EMPTY   : perf_q.src_empty_cycles    <= perf_q.src_empty_cycles    + 64'd1;
      ST_LINK_DOWN   : perf_q.link_down_cycles    <= perf_q.link_down_cycles    + 64'd1;
      ST_RECOVERY    : perf_q.recovery_cycles     <= perf_q.recovery_cycles     + 64'd1;
      ST_NO_CREDIT   : perf_q.no_credit_cycles    <= perf_q.no_credit_cycles    + 64'd1;
      ST_REPLAY_FULL : perf_q.replay_full_cycles  <= perf_q.replay_full_cycles  + 64'd1;
      ST_QUEUE       : perf_q.queue_stall_cycles  <= perf_q.queue_stall_cycles  + 64'd1;
      ST_ARB         : perf_q.arb_stall_cycles    <= perf_q.arb_stall_cycles    + 64'd1;
      ST_TARGET      : perf_q.target_stall_cycles <= perf_q.target_stall_cycles + 64'd1;
      default        : perf_q.unattributed_cycles <= perf_q.unattributed_cycles + 64'd1;
    endcase
 
    if (beat_transferred) begin
      perf_q.transmitted_bytes <= perf_q.transmitted_bytes + 64'(beat_bytes);
      perf_q.framing_bytes     <= perf_q.framing_bytes     + 64'(beat_framing_bytes);
      if (beat_is_replay) perf_q.replay_bytes        <= perf_q.replay_bytes        + 64'(beat_payload_bytes);
      else                perf_q.unique_payload_bytes<= perf_q.unique_payload_bytes+ 64'(beat_payload_bytes);
    end
  end
end

Architecture. A single combinational classifier producing one enum, and a single unique case incrementing exactly one counter. The one-hot structure is what makes §12's invariant hold by construction rather than by discipline.

State. The counter bank of §8, observation-window lifetime.

Cycle behaviour. Exactly one cycle counter increments per clock. perf_q.cycles increments unconditionally, which is what makes the invariant checkable — the total is measured, not derived by summing.

Contract. The classifier's priority order is the analysis. Two designs with the same counters and different orders report different bottlenecks for identical behaviour, so the order must be documented alongside the numbers (§10).

Failure. §11. Also: putting !work_offered after the resource checks, which would classify an idle cycle as a credit stall whenever credits happen to be zero — making a quiet link look credit-limited, which is a real and confusing misdiagnosis.

DV. Assert one-hot (§14). Construct a stimulus for each state in isolation. Then construct one where several conditions are true simultaneously and confirm the documented priority wins.

10. Causal Priority, Not Severity

The order in §9 encodes a claim: when several conditions are true, the earliest in the causal chain owns the cycle.

Two properties.

Position 3 is the most important. If nothing was offered, no downstream condition explains anything. Classifying an unoffered cycle by a coincidentally-zero credit counter is how a producer-limited system gets diagnosed as credit-limited — and then credits get added, and nothing changes.

And position 9 is deliberately late. A cycle that lost arbitration and had no credit is credit-limited: winning the grant would not have helped. Reporting it as arbitration-limited sends the investigation to a scheduler that is working correctly.

A different architecture may justify a different order — and that is fine, provided the order is stated. What is not fine is an undocumented order, because then the same counters mean different things to different readers.

11. Wrong Classifier — Independent Counters

// WRONG — each condition increments its own counter independently.
always_ff @(posedge clk) begin
  if (!credit_available) credit_stall_q <= credit_stall_q + 1;
  if (!queue_space)      queue_stall_q  <= queue_stall_q  + 1;
  if (!target_ready)     target_stall_q <= target_stall_q + 1;
  if (!arb_grant)        arb_stall_q    <= arb_stall_q    + 1;
end

One physical cycle increments up to four counters.

ILLUSTRATIVE measurement over 1,000,000 cycles, congested:
 
  credit_stall = 500,000   → 50%
  queue_stall  = 450,000   → 45%
  target_stall = 400,000   → 40%
  arb_stall    = 450,000   → 45%
  ─────────────────────────────
  "total stall" = 180%

Four properties of this bug.

The fractions sum above 100%, which is at least visible — and §35 lists it as a signature. The dangerous case is mild congestion, where the overlap is small, the fractions sum to 103%, and nobody notices the 3% is impossible.

Every fraction is individually wrong and the ranking may be wrong too. Here credit at 50% looks dominant, but under this scheme a condition that is frequently true ranks high even when it is never the binding one. A resource that is unavailable whenever anything else is unavailable will always look like the bottleneck — which is exactly the pathology causal priority prevents.

It cannot close a residual. §33's method requires the losses to sum to the gap. With overlapping counters the sum exceeds the gap and the residual is negative, which is not a number anyone can act on.

And it survives review because each line is individually correct. if (!credit_available) credit_stall++ is a true statement about a cycle. The defect is the absence of exclusivity, which is a property of the four lines together and of none of them alone.

12. The Cycle Accounting Invariant

observation_cycles
  = useful_cycles + replay_cycles + src_empty_cycles
  + no_credit_cycles + replay_full_cycles + queue_stall_cycles
  + arb_stall_cycles + target_stall_cycles
  + link_down_cycles + recovery_cycles
  + unattributed_cycles

This is the chapter's primary invariant, and it must hold exactly at every window boundary.

What a violation means, by direction:

All three are instrumentation bugs, not performance findings. And the ordering of the investigation matters: check this invariant before believing any fraction. A regression that reports stall fractions without checking it is reporting numbers of unknown validity.

With §9's structure the invariant holds by construction, which is the argument for the one-hot classifier over independent counters. Asserting it anyway is a guard against a future refactor that "simplifies" the unique case into separate if statements.

13. The Byte Accounting Invariant

transmitted_bytes = unique_payload_bytes + replay_bytes + framing_bytes

At the transport boundary, and the boundary must be stated (15.1 §17).

Worked, illustrative:

over one window:
  unique_payload_bytes = 60,000,000
  replay_bytes         =  1,200,000
  framing_bytes        =  3,825,000
  ─────────────────────────────────
  sum                  = 65,025,000
  transmitted_bytes    = 65,025,000        ✓ invariant holds
 
derived figures:
  non_replay_fraction = 60,000,000 / (60,000,000 + 1,200,000) = 0.9804
  framing_fraction    =  3,825,000 / 65,025,000               = 0.0588   ✓ = 4/68

The framing fraction reproducing 4/68 is a useful cross-check. It confirms the framing counter and the flit format agree — and if it does not reproduce, either the counter is wrong or the format assumption is (§35).

And note what the invariant does not include. Bytes of padding inside a partially filled container are neither payload nor framing under this definition. If a design pads, padding needs its own category or the invariant will not close — which is §24's occupancy question arriving in the byte accounting.

14. SVA — One Class Per Cycle, and the Invariants

// MANDATORY. The property that makes Section 11 structurally impossible.
property p_exactly_one_class;
  @(posedge clk) disable iff (!rst_n)
    $onehot(class_onehot);            // one bit per state, exactly one set
endproperty
a_exactly_one_class: assert property (p_exactly_one_class);
 
// The cycle invariant, at every window boundary (Section 12).
property p_cycle_accounting;
  @(posedge clk) disable iff (!rst_n)
    window_end |-> (perf_q.cycles == sum_of_all_cycle_counters());
endproperty
a_cycle_accounting: assert property (p_cycle_accounting);
 
// The byte invariant (Section 13).
property p_byte_accounting;
  @(posedge clk) disable iff (!rst_n)
    window_end |-> (perf_q.transmitted_bytes ==
                    perf_q.unique_payload_bytes + perf_q.replay_bytes
                  + perf_q.framing_bytes);
endproperty
a_byte_accounting: assert property (p_byte_accounting);
 
// Nothing may be unattributed. This is the completeness claim.
property p_no_unattributed;
  @(posedge clk) disable iff (!rst_n)
    (perf_q.unattributed_cycles == '0);
endproperty
a_no_unattributed: assert property (p_no_unattributed);
 
// Counters are monotonic between clears — catches a wrap or a stray write.
property p_counters_monotonic;
  @(posedge clk) disable iff (!rst_n || perf_clear)
    (perf_q.useful_cycles >= $past(perf_q.useful_cycles))
 && (perf_q.transmitted_bytes >= $past(perf_q.transmitted_bytes));
endproperty
a_counters_monotonic: assert property (p_counters_monotonic);
 
// The physical ceiling (15.1 Section 28) — a measurement above it is a bug.
property p_never_exceeds_ceiling;
  @(posedge clk) disable iff (!rst_n)
    window_end |-> (perf_q.transmitted_bytes <=
                    perf_q.cycles * BYTES_PER_CYCLE_MAX);
endproperty
a_never_exceeds_ceiling: assert property (p_never_exceeds_ceiling);

Architecture. Six properties in three groups: exclusivity and completeness (1, 4), invariants (2, 3), and arithmetic sanity (5, 6).

p_no_unattributed is stronger than it looks. It asserts the classifier's taxonomy is complete for every reachable state of the design — a claim that is far more valuable than any individual stall fraction, because it is what licenses the residual analysis of §33. A design that cannot assert this cannot close a residual.

And p_exactly_one_class is preferable to checking the invariant alone. The invariant is a window-boundary claim; one-hot is a per-cycle claim. One-hot fails on the cycle the bug occurs, with a one-cycle counterexample, rather than a million cycles later with an arithmetic discrepancy.

DV. All six should be permanently enabled — they are cheap and instrumentation bugs are silent. They need congestion in the stimulus: with ready tied high, most states are unreachable and the exclusivity property is nearly vacuous.

15. Atomic Snapshot of the Whole Bank

15.1 §30 introduced snapshots for a counter set. Here the requirement is stricter, because every figure in this chapter is a ratio across many counters.

// ILLUSTRATIVE. The ENTIRE bank is captured on one edge. Software reads only
// the shadow.
throughput_counters_t snap_q;
logic                 snap_valid_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n)               snap_valid_q <= 1'b0;
  else if (snapshot_req) begin
    snap_q       <= perf_q;              // ALL fields, ONE edge
    snap_valid_q <= 1'b1;
  end
end

Architecture. A whole-struct copy in one assignment. snap_q <= perf_q is the mechanism — a single non-blocking assignment of the entire structure cannot tear.

Why the stricter requirement. §33's residual analysis divides sixteen counters against each other. If any two came from different instants, the invariants of §12 and §13 will not hold on the snapshot even though they held continuously in hardware — and the residual will be non-zero for a reason that has nothing to do with the design.

Contract. Software reads only snap_q. Mixing a live counter with a snapshot value reintroduces tearing in its most confusing form, because the invariants would then fail intermittently and load-dependently.

Failure. Capturing fields in a loop or across several cycles. Or snapshotting the cycle counters and letting software read the byte counters live — which produces a transmitted_bytes that exceeds cycles × max and looks exactly like §35's impossible-measurement signature.

DV. Snapshot under rapidly varying load and assert §12's and §13's invariants on the snapshot. A torn capture fails them — a neat reuse of properties written for a different purpose.

16. Measurement Window

Too short and burst noise dominates; too long and phase changes average away.

There is no universal number, and the criterion is stationarity: the window should be long enough to contain many instances of the workload's repeating unit, and short enough that the workload's character does not change within it.

And a series of windows is worth more than one long window for the same total cycles. Ten windows of 100,000 cycles show whether the workload was stable; one window of 1,000,000 cannot distinguish a steady 60% from alternating 90% and 30% phases. 15.2 §29's argument about averages applies to throughput too.

17. Warmup and Drain

A pipeline must fill before it can run at rate, and including the fill in the measurement understates steady-state throughput.

ILLUSTRATIVE, a 5-stage pipeline (15.1's structure, 13.5 Section 8's arithmetic):
 
  fill cost = stages − 1 = 4 cycles of no output
 
  over a 20-cycle burst:   16 outputs / 20 cycles = 0.80   → 80%
  over a 100-cycle burst:  96 / 100               = 0.96   → 96%
  over a 10,000-cycle run: 9,996 / 10,000         = 0.9996 → 99.96%

So the same design measures 80% or 99.96% depending only on the burst length. Neither number is wrong; they answer different questions.

A benchmark must therefore define three regions:

And "reach steady credit state" is the subtle one. 13.1 §4 established that credit return has a round-trip latency. A measurement starting before the credit loop is in steady state measures the loop filling, not the link running — which is a warmup requirement that has nothing to do with the pipeline depth.

18. Wrong Benchmark — Measuring the Startup

THE BENCHMARK
  transfer 16 objects, measure wall time, divide
  → 5-stage pipeline: 16 objects in 20 cycles = 80% of theoretical
  → CONCLUSION: "the design achieves only 80% of link throughput"
 
THE REALITY
  the same design over a 10,000-object stream: 99.96%
  the benchmark measured PIPELINE FILL, which is a LATENCY property

Three properties of this failure.

Both measurements are correct and only one answers the question that was asked. If the question is "what throughput does a sustained stream achieve?", the answer is 99.96%. If it is "how long does a 16-object transfer take?", the answer is a latency and should be reported in nanoseconds (15.2), not as a percentage of bandwidth.

The conclusion drives the wrong work. A team told the design achieves 80% will look for a 20% loss that does not exist. They will find something — there is always something — and fix a non-problem.

And it interacts with §17's credit warmup. A short burst may not even fill the credit loop, so the measurement includes the loop filling as well as the pipeline. The shorter the benchmark, the larger the fraction of it that is startup, and at 16 objects almost all of it is.

A throughput benchmark must be long enough that the fill cost is negligible, or it is a latency benchmark wearing the wrong units.

19. Six Behavioural Signatures

With §9's classifier, the dominant state names the investigation.

Two observations about using the table.

The first row is the most common and the most misdirected. A dominant SRC_EMPTY means the interconnect is fine and the workload or its concurrency is the limit. Reporting it as a UCIe performance problem wastes the interconnect team's time — and §7's two denominators are what make it obvious.

And the last row is a category error worth naming. A dominant RECOVERY fraction is a throughput symptom whose root cause is reliability. The fix is in Module 14's domain, and optimising the datapath will not touch it — which is 15.4 §24's replay-storm scenario expressed as a cycle fraction.

20. Sweep Methodology

Sweeping one parameter at a time is the most reliable way to identify a binding resource, because it produces a causal answer rather than a correlational one.

The two "flat" answers in the last two rows are the most valuable results in the table, because they are negative results that eliminate the interconnect. A design where doubling the width changes nothing is definitively not link-limited, and that conclusion is worth more than any positive measurement.

21. Payload-Size Sweep

ILLUSTRATIVE. Ceiling 60.24 GB/s payload (Section 4). Objects placed one per
64 B payload area in a verified 68 B flit.
 
  object size   occupancy   framing×occ   achieved     limited by
  ──────────────────────────────────────────────────────────────
      8 B        0.125        0.1176       7.1 GB/s    occupancy
     16 B        0.250        0.2353      14.2 GB/s    occupancy
     32 B        0.500        0.4706      28.4 GB/s    occupancy
     64 B        1.000        0.9412      56.5 GB/s    approaching ceiling
    128 B*       1.000        0.9412      56.5 GB/s    ceiling
 
ARITHMETIC CHECK for 16 B:
  achieved = raw × framing × occupancy = 64.0 × 0.9412 × 0.250 = 15.06 GB/s
  measured 14.2 → ratio 14.2/15.06 = 0.943, so ~5.7% is lost to other causes  ✓
 
* 128 B spans two flits; occupancy stays 1.0 and per-object framing is unchanged.

Two readings.

Small objects are occupancy-limited and the loss is structural, not a stall. An 8-byte object in a 64-byte payload area wastes 87.5% of the container. No amount of queue depth, credit or arbitration tuning changes that — the fix is packing multiple objects per container (13.5 §18) or a smaller transport unit.

And the sweep separates structural loss from stalls, which is its purpose. At 64 B the achieved figure is 56.5 of a 56.5 ceiling — so all remaining loss at small sizes is occupancy, and the residual 5.7% at 16 B is the stall component. One sweep, two components separated.

22. Outstanding-Depth Sweep

ILLUSTRATIVE. Response latency 40 cycles; the link can accept 1 object/cycle.
Bound from 13.5 Section 13: throughput ≤ depth / latency.
 
  depth   bound (obj/cy)   measured   plateau?
  ─────────────────────────────────────────────
      1        0.025         0.025      no
      4        0.100         0.098      no
     16        0.400         0.390      no
     32        0.800         0.780      no
     40        1.000         0.920      approaching
     64        1.000         0.930      YES — plateau
    128        1.000         0.930      YES
 
CONCLUSION
  throughput rises approximately linearly to depth ≈ 40, then plateaus.
  → latency hiding WAS the limit up to depth 40
  → required concurrency ≈ 40 = rate × latency        (13.5 Section 14)
  → beyond 40, something else binds at 0.93 — investigate that 7%

Three readings.

The plateau point estimates the required concurrency directly, and it matches rate × latency — which is the check that the measurement and the model agree.

The plateau level is the next finding. 0.93 rather than 1.00 means a second resource binds once concurrency is sufficient. That 7% is what §33's residual analysis must explain, and it is exactly the situation where an undisciplined investigation stops at "we got to 93%, good enough".

And a flat curve from depth 1 would have been equally informative — it would mean the design was never latency-limited, and outstanding depth is not the parameter to tune.

23. Credit-Depth Sweep

ILLUSTRATIVE. Credit round trip 24 cycles, launch rate 1 object/cycle.
Bound from 13.1 Section 4: bubbles unless credits ≥ rate × RTT.
 
  credits   bound      measured   NO_CREDIT cycles
  ────────────────────────────────────────────────
       4    0.167       0.165          83%
       8    0.333       0.330          67%
      16    0.667       0.655          34%
      24    1.000       0.910           4%
      32    1.000       0.930           0%
      48    1.000       0.930           0%
 
CONCLUSION
  rises to credits ≈ 24 = rate × RTT, then plateaus at 0.93.
  the NO_CREDIT fraction falling to 0% CONFIRMS credits stopped binding.

The NO_CREDIT column is the point of the sweep, and it is what §20's table means by a causal answer.

Why the counter alone is insufficient. A design could show a small NO_CREDIT fraction and still be credit-limited in effect — if credits are marginal, the stall is brief but frequent and the throughput impact exceeds the cycle fraction. The sweep proves causality: adding credits raised throughput, therefore credits were binding. The counter alone shows correlation.

And note that both this sweep and §22's plateau at 0.93. That is consistent — the same second resource binds in both cases, which is a cross-check that the two sweeps are measuring the same system. If they plateaued at different levels, one of the sweeps changed something it should not have (§28).

24. Replay-Window and Queue-Depth Sweeps

Replay window, illustrative, confirmation latency 30 cycles:

  entries   measured   REPLAY_FULL cycles
  ───────────────────────────────────────
        8     0.265         73%
       16     0.530         47%
       30     0.900          6%
       48     0.930          0%
       64     0.930          0%
 
CONCLUSION: binding up to ≈ 30 = rate × confirmation latency (14.3 Section 26),
then plateau at the same 0.93.

Queue depth, illustrative:

  depth   measured   QUEUE cycles   verdict
  ─────────────────────────────────────────
      2     0.870        12%        binding
      4     0.920         3%        nearly resolved
      8     0.930         0%        resolved
     16     0.930         0%        NO IMPROVEMENT
     64     0.930         0%        NO IMPROVEMENT — and latency is now worse

The queue sweep's flat tail is the most practically useful result in this chapter.

Beyond depth 8 the queue is not binding, and additional depth buys zero throughput while adding residence time. This is the measurement that ends the "just make the FIFO bigger" argument — and it is 13.4 §28's bufferbloat demonstrated rather than asserted.

And all three sweeps plateau at 0.93, which localises the remaining bottleneck to something none of them touched — a strong hint before §33's residual analysis even begins.

25. Link Rate and Lane Width Sweeps

The two sweeps that can eliminate the interconnect as a suspect.

ILLUSTRATIVE, with verified rates (Section 3):
 
  RATE SWEEP (x16 throughout)
    rate      raw       measured payload   scaled with rate?
    ────────────────────────────────────────────────────────
    32 GT/s   64 GB/s      56.0 GB/s         —
    48 GT/s   96 GB/s      56.2 GB/s         NO
    64 GT/s  128 GB/s      56.1 GB/s         NO
 
  WIDTH SWEEP (32 GT/s throughout, verified {1,2,4} clusters)
    clusters  raw       measured payload   scaled with width?
    ─────────────────────────────────────────────────────────
    1         64 GB/s      56.0 GB/s         —
    2        128 GB/s      56.3 GB/s         NO
    4        256 GB/s      56.1 GB/s         NO
 
CONCLUSION
  quadrupling raw capacity changed throughput by 0.2%.
  → the link is DEFINITIVELY NOT the bottleneck
  → the TARGET_STALL fraction should confirm it (Section 31)

Three properties of this experiment.

It is a negative result and it is decisive. Most performance evidence is correlational; this is causal and eliminative. Quadrupling a resource and observing no change proves that resource was not binding, with no model required.

It is cheap in simulation and often cheap in silicon, because both rate and width are usually configurable. It should be the first experiment run, before any counter analysis, because it partitions the problem space in one step.

And the width sweep has a verified constraint. The cluster count is {1, 2, 4} (§3), so the achievable steps are 64, 128 and 256 GB/s per link and nothing between. A sweep asking for 1.5× is asking for a configuration the material does not describe.

26. A/B Discipline

Change one thing, hold everything else fixed — otherwise attribution is invalid.

And repeatability differs by environment:

Keep the statistics light. For simulation, exact match is the standard and any deviation is a defect to investigate. For silicon, the spread across runs is itself a measurement — a wide spread means the workload or the environment is not stationary, which invalidates §16's window choice before any conclusion is drawn.

27. Bottleneck Migration as a Method

A successful optimisation changes what is limiting you. If throughput improved and the dominant stall state is unchanged, either the fix was not the mechanism you thought, or you have two instances of the same bottleneck.

Worked, illustrative, following the sweeps:

  step                          throughput   dominant state
  ─────────────────────────────────────────────────────────
  baseline, depth 8             0.265        REPLAY_FULL
  replay window → 48            0.780        NO_CREDIT       ← migrated
  credits → 32                  0.910        TARGET          ← migrated
  (widen target service)        0.955        ARB             ← migrated
  (fix arbitration policy)      0.978        SRC_EMPTY       ← migrated
  (raise producer concurrency)  0.993        USEFUL dominant

Three readings.

Each step's dominant state predicts the next step, which turns performance work from trial-and-error into a sequence. That is the practical payoff of the classifier.

The final state is SRC_EMPTY, which is the correct place to stop. Once the producer is the limit, the interconnect is done — and further interconnect work has no effect (§19's first row).

And a step that improves throughput without changing the dominant state is a warning. It means either the improvement came from somewhere unexpected, or the same resource binds twice — for instance credits on two different links, where fixing one leaves the other dominant. §28 is the failure of not checking.

28. Wrong Optimisation — Tuning a Non-Binding Resource

THE ACTION
  double the transmit FIFO depth from 8 to 16
 
THE RESULT
  throughput: 0.930 → 0.930        (no change)
  QUEUE cycles: 0% → 0%            (it was already 0%)
 
THE CONCLUSION DRAWN
  "the architecture is broken — more buffering doesn't even help"
 
WHAT WAS ACTUALLY TRUE
  the queue stopped binding at depth 8 (Section 24)
  TARGET_STALL was 34% and had been the binding resource all along

Four properties of this failure.

The counters already contained the answer. QUEUE was at 0% and TARGET at 34% before the change. The action was chosen without reading the classifier, which is the specific process failure.

Doubling the FIFO made latency worse for no throughput gain — 13.4 §28's bufferbloat, and now the design carries the cost permanently.

The conclusion drawn is actively harmful. "More buffering doesn't help so the architecture is broken" is a reasonable-sounding inference from a real observation, and it is wrong. It will be repeated in a review and may drive a redesign of something that works.

And the correct next action was available for free. TARGET at 34% names the remote service rate, and §25's width sweep would have confirmed it in one run. The whole misstep is avoided by reading the classifier before acting.

29. Worked Case 1 — 62% Achieved

CEILING (Section 4, verified inputs)
  theoretical payload = 60.24 GB/s
  MEASURED            = 37.35 GB/s        → 62.0% of ceiling
  GAP                 = 22.89 GB/s        → 38.0%
 
CYCLE CLASSIFICATION over the window (Section 9, mutually exclusive)
  USEFUL        62%
  NO_CREDIT     18%
  TARGET        10%
  REPLAY         5%
  SRC_EMPTY      5%
  ──────────────────
  TOTAL        100%      ✓ Section 12's invariant holds
 
ATTRIBUTION OF THE 38% GAP
  NO_CREDIT   18%  × 60.24 = 10.84 GB/s   ← the largest single loss
  TARGET      10%  × 60.24 =  6.02 GB/s
  REPLAY       5%  × 60.24 =  3.01 GB/s   (link busy, no new payload)
  SRC_EMPTY    5%  × 60.24 =  3.01 GB/s   (NOT the link's loss)
  ──────────────────────────────────────
  explained          38%  = 22.89 GB/s
 
  RESIDUAL = 22.89 − 22.89 = 0.00 GB/s    ✓ FULLY EXPLAINED
 
TWO DENOMINATORS (Section 7)
  against observation: 62 / 100 = 0.620
  against offered:     62 /  95 = 0.653   → the link converted 65% of its
                                            opportunities

Four readings.

The residual is zero, which is what makes this an analysis rather than an observation. Every one of the 38 missing percentage points has a named cause and a byte figure.

NO_CREDIT at 18% is the first thing to fix, and §23's sweep would confirm causality before any change is made.

SRC_EMPTY at 5% is not the interconnect's loss and should not be counted against it. Reporting "62% efficiency" without that caveat overstates the interconnect's shortfall — 65% against offered is the fairer figure, and both should be reported.

And 5% replay is far above the specified BER. 15.1 §23's calculation puts the expected replay fraction at around 10⁻⁹ at the verified BER. 5% means the channel is orders of magnitude outside specification, so that loss is a reliability finding rather than a performance one (14.5).

30. Worked Case 2 — 95% Link Utilisation, 70% Useful

OBSERVED
  transfer cycles / observation = 0.95      → the link is busy 95% of the time
  achieved payload              = 42.2 GB/s of a 60.24 ceiling → 70.0%
 
THE APPARENT CONTRADICTION
  how is a link that is 95% busy delivering only 70% of its useful capacity?
 
CYCLE CLASSIFICATION
  USEFUL        88%
  REPLAY         7%        ← busy, but no new payload
  SRC_EMPTY      3%
  ARB            2%
  ──────────────────
  transfers = 88 + 7 = 95%       ✓ matches the utilisation figure
 
BYTE ACCOUNTING (Section 13)
  unique_payload  = 42.2 GB/s
  replay_payload  =  3.4 GB/s
  framing         =  2.9 GB/s
  ─────────────────────────
  transmitted     = 48.5 GB/s
 
  non_replay_fraction = 42.2 / (42.2 + 3.4) = 0.9254
  occupancy (relative to max) = 0.82         [ILLUSTRATIVE]
 
RECONCILIATION against Section 5's model
  achieved = 60.24 × useful_cycle_fraction × non_replay × occupancy
           = 60.24 × 0.92 × 0.9254 × 0.82
           = 42.1 GB/s      vs measured 42.2      → residual 0.24%   ✓

Three readings.

"The link is busy" and "the link is productive" are different claims, and this case is the proof. 95% busy, 70% useful — the 25-point gap is replay, framing and partial containers.

The reconciliation closes to 0.24%, which is within measurement noise for illustrative figures. That is what a closed residual looks like and it validates both the counters and the model simultaneously.

And the actionable finding is occupancy at 0.82, not the replay at 7%. Occupancy costs 60.24 × 0.92 × 0.9254 × 0.18 = 9.2 GB/s; replay costs 60.24 × 0.92 × 0.0746 = 4.1 GB/s. Occupancy is the larger term, and it is a packing decision rather than a channel problem.

31. Worked Case 3 — x8 to x16 With No Gain

THE EXPERIMENT (Section 25's width sweep)
  1 cluster  (x16 lanes, 64 GB/s raw):   measured 44.0 GB/s payload
  2 clusters (128 GB/s raw):             measured 44.2 GB/s payload
  4 clusters (256 GB/s raw):             measured 44.1 GB/s payload
 
  quadrupling raw capacity → +0.2%
 
CYCLE CLASSIFICATION at 4 clusters
  USEFUL        73%
  TARGET        24%        ← dominant
  SRC_EMPTY      2%
  ARB            1%
  ──────────────────
  TOTAL        100%      ✓
 
CONCLUSION
  the link is not the bottleneck at ANY width.
  TARGET at 24% names the remote service rate.
 
CROSS-CHECK against the target's own capacity
  if remote service capacity = 45 GB/s payload      [ILLUSTRATIVE]
  then achieved 44.1 of 45 = 0.98 → the TARGET is at 98% of ITS capacity
  → consistent: the target is saturated, the link is idle 27% of the time

Three readings.

The negative result is decisive and cost three runs. No model, no counter analysis needed to reach the primary conclusion — quadrupling a resource with no effect proves it was not binding.

The classifier then names the actual bottleneck, and the cross-check against the target's own capacity confirms it independently. Two independent lines of evidence agreeing is what makes a diagnosis trustworthy.

And this is 15.4 §17's compute bottleneck seen from the measurement side. The correct action is to widen the target or reduce demand — and the correct action is definitively not to touch the interconnect, which is what a link-centric investigation would have done.

32. Worked Case 4 — Throughput Falls Under Error Injection

SWEEP: injected error rate, everything else held fixed (Section 26)
 
  error rate     achieved    USEFUL  REPLAY  REPLAY_FULL  SRC_EMPTY  RECOVERY
  ──────────────────────────────────────────────────────────────────────────
  0 (clean)      56.0 GB/s     93%      0%        0%          7%        0%
  1e-9           55.8 GB/s     93%      0%        0%          7%        0%
  1e-6           54.1 GB/s     90%      3%        0%          7%        0%
  1e-4           41.2 GB/s     68%     14%        9%          6%        3%
  1e-3           22.6 GB/s     38%     21%       28%          4%        9%
 
THE CAUSAL CHAIN, read across the table
  1. error rate rises           → REPLAY cycles rise (retransmission)
  2. replay entries retire slower → REPLAY_FULL rises (14.3 Section 26)
  3. admission stalls            → SRC_EMPTY FALLS (work backs up upstream)
  4. retry exhaustion            → RECOVERY rises (14.3 Section 37)
  5. useful throughput collapses  → 56.0 → 22.6 GB/s
 
INVARIANT CHECK at 1e-3
  38 + 21 + 28 + 4 + 9 = 100%     ✓ Section 12 holds throughout

Four readings.

SRC_EMPTY falls as things get worse, which is counter-intuitive and diagnostic. At a clean rate the link is idle 7% of the time waiting for work; under stress the work backs up and there is always something to send. A falling idle fraction alongside falling throughput is a signature of downstream saturation.

RECOVERY appearing at all is the category boundary. At 1e-4 and above, 3–9% of cycles are recovery — which means this stopped being a performance problem and became a reliability one (14.5 §4's event-fault-incident distinction).

The chain crosses four modules — detection, retry, congestion and recovery — and every step is a correct local response. Nothing in the chain is a bug, which is 13.4 §26's amplification demonstrated with numbers.

And the invariant holds at every row, which is what licenses reading the table as a causal chain rather than as four unrelated measurements.

33. Residual Analysis — Drive It to Zero

The chapter's signature principle.

residual = theoretical − measured − Σ(explained losses)

And the standard is zero, not "small".

Worked, illustrative, showing a non-zero residual and its resolution:

  theoretical      = 100.0 units
  measured         =  70.0 units
  gap              =  30.0 units
 
  explained:
    NO_CREDIT      =  12.0
    TARGET         =   6.0
    REPLAY         =   2.0
  ─────────────────────────
  explained total  =  20.0
  RESIDUAL         =  10.0 units       ← 10% with NO NAME
 
INVESTIGATION
  1. check Section 12's cycle invariant       → holds; no unattributed cycles
  2. check Section 13's byte invariant        → holds
  3. check the ceiling's occupancy assumption → max_payload_occupancy was set
                                                to 1.00, but the workload's
                                                object size can only fill 0.90
  4. corrected ceiling = 100.0 × 0.90 = 90.0
     corrected gap     =  90.0 − 70.0 = 20.0
     RESIDUAL          =  20.0 − 20.0 = 0.0   ✓ CLOSED

Three properties of the discipline.

A non-zero residual is a finding, not a tolerance. In the worked case it pointed at an optimistic ceiling rather than at a missing stall — which is a modelling error and exactly the kind of thing that would otherwise be discovered by an architect months later trying to reconcile two documents.

"Close enough" is where the next surprise lives. A 10% unexplained loss means there is a mechanism nobody has named. It may be benign, and it may be the thing that gets worse under a workload nobody has run yet — and there is no way to know which while it is unexplained.

And closing it validates three things at once: the counters (they sum correctly), the classifier (nothing is unattributed), and the model (the factors reproduce the measurement). A closed residual is the strongest single statement a performance analysis can make.

34. The Throughput Scoreboard

// Verification-only reference model. Not synthesisable.
class throughput_scoreboard;
 
  // Independently observed, from the monitor rather than the DUT counters.
  longint obs_cycles, obs_useful, obs_replay, obs_src_empty;
  longint obs_stall[NUM_REASONS];
  longint obs_transmitted, obs_unique, obs_replay_bytes, obs_framing;
  real    theoretical_payload_bw;
 
  // ---- Check 1: every DUT counter matches the independent observation.
  function void check_counters(throughput_counters_t dut);
    if (dut.cycles        != obs_cycles)     $error("cycles: %0d vs %0d", dut.cycles, obs_cycles);
    if (dut.useful_cycles != obs_useful)     $error("useful: %0d vs %0d", dut.useful_cycles, obs_useful);
    if (dut.replay_cycles != obs_replay)     $error("replay: %0d vs %0d", dut.replay_cycles, obs_replay);
    if (dut.unique_payload_bytes != obs_unique)
      $error("unique payload: %0d vs %0d — replay counted as unique?",
             dut.unique_payload_bytes, obs_unique);
  endfunction
 
  // ---- Check 2: the cycle invariant on the SNAPSHOT (Sections 12, 15).
  function void check_cycle_invariant(throughput_counters_t s);
    longint sum = s.useful_cycles + s.replay_cycles + s.src_empty_cycles
                + s.no_credit_cycles + s.replay_full_cycles + s.queue_stall_cycles
                + s.arb_stall_cycles + s.target_stall_cycles
                + s.link_down_cycles + s.recovery_cycles + s.unattributed_cycles;
    if (sum != s.cycles)
      $error("CYCLE INVARIANT: sum %0d != cycles %0d (%0d unaccounted)",
             sum, s.cycles, s.cycles - sum);
    if (s.unattributed_cycles != 0)
      $error("CLASSIFIER INCOMPLETE: %0d unattributed cycles", s.unattributed_cycles);
  endfunction
 
  // ---- Check 3: the byte invariant (Section 13).
  function void check_byte_invariant(throughput_counters_t s);
    if (s.transmitted_bytes != s.unique_payload_bytes + s.replay_bytes + s.framing_bytes)
      $error("BYTE INVARIANT broken: %0d != %0d + %0d + %0d",
             s.transmitted_bytes, s.unique_payload_bytes, s.replay_bytes, s.framing_bytes);
  endfunction
 
  // ---- Check 4: the ceiling was never exceeded.
  function void check_ceiling(real measured_bw);
    if (measured_bw > theoretical_payload_bw * 1.001)
      $error("IMPOSSIBLE: measured %0.2f > theoretical %0.2f — units, boundary or counter bug",
             measured_bw, theoretical_payload_bw);
  endfunction
 
  // ---- Check 5: THE RESIDUAL. Section 33's principle, enforced.
  function void check_residual(real measured_bw, throughput_counters_t s);
    real gap = theoretical_payload_bw - measured_bw;
    real explained = 0.0;
    explained += theoretical_payload_bw * real'(s.no_credit_cycles)    / real'(s.cycles);
    explained += theoretical_payload_bw * real'(s.replay_full_cycles)  / real'(s.cycles);
    explained += theoretical_payload_bw * real'(s.queue_stall_cycles)  / real'(s.cycles);
    explained += theoretical_payload_bw * real'(s.arb_stall_cycles)    / real'(s.cycles);
    explained += theoretical_payload_bw * real'(s.target_stall_cycles) / real'(s.cycles);
    explained += theoretical_payload_bw * real'(s.recovery_cycles)     / real'(s.cycles);
    explained += theoretical_payload_bw * real'(s.link_down_cycles)    / real'(s.cycles);
    explained += theoretical_payload_bw * real'(s.replay_cycles)       / real'(s.cycles);
    explained += theoretical_payload_bw * real'(s.src_empty_cycles)    / real'(s.cycles);
 
    if ($abs(gap - explained) > RESIDUAL_LIMIT)
      $error("UNEXPLAINED RESIDUAL %0.3f: gap %0.3f, explained %0.3f — a loss has no name",
             gap - explained, gap, explained);
  endfunction
 
endclass

Architecture. Five checks in three categories: counter correctness (1), accounting integrity (2, 3, 4), and the residual (5).

Check 1 is why the scoreboard exists. The counters are the basis of every conclusion, so an independently derived observation is the only thing that establishes they can be trusted. Its most valuable line is the unique_payload comparison, whose error message names the likely cause — replay counted as unique, which makes the replay tax invisible.

Check 2 asserts the classifier is complete, not merely consistent. A non-zero unattributed_cycles is a hole in the taxonomy, and it must fail the run rather than be reported as a curiosity.

Check 5 is the chapter's whole point, expressed as a check that can fail a regression. RESIDUAL_LIMIT should be tight — a fraction of a percent, not five percent — because §33 established that an unexplained residual is a gap in understanding rather than a tolerance.

35. Coverage

covergroup cg_throughput_analysis @(posedge clk);
  option.per_instance = 1;
 
  // --- Every cycle state must be reached, and USEFUL must dominate sometimes.
  cp_state : coverpoint reason_d {
    bins useful      = {ST_USEFUL};
    bins replay      = {ST_REPLAY};
    bins src_empty   = {ST_SRC_EMPTY};
    bins no_credit   = {ST_NO_CREDIT};
    bins replay_full = {ST_REPLAY_FULL};
    bins queue       = {ST_QUEUE};
    bins arb         = {ST_ARB};
    bins target      = {ST_TARGET};
    bins link_down   = {ST_LINK_DOWN};
    bins recovery    = {ST_RECOVERY};
    illegal_bins unattributed = {ST_UNATTRIB};       // must NEVER occur
  }
 
  // --- Which state DOMINATED the window (Section 19's signatures).
  cp_dominant : coverpoint dominant_state_this_window;
 
  // --- Achieved fraction of ceiling, including the full-rate case.
  cp_achieved : coverpoint achieved_bucket {
    bins poor    = {[0:4]};        // < 50%
    bins mid     = {[5:7]};        // 50-80%
    bins good    = {[8:8]};        // 80-90%
    bins near_max= {9};            // > 90% — proves the ceiling is approachable
  }
 
  // --- Sweep coverage (Section 20). Each parameter must be swept.
  cp_swept_param : coverpoint swept_parameter {
    bins payload_size = {SW_PAYLOAD};
    bins outstanding  = {SW_OUTSTANDING};
    bins credits      = {SW_CREDITS};
    bins replay_win   = {SW_REPLAY};
    bins queue_depth  = {SW_QUEUE};
    bins rate         = {SW_RATE};
    bins width        = {SW_WIDTH};
  }
  cp_plateau_seen : coverpoint sweep_reached_plateau;
 
  // --- Measurement methodology (Sections 16, 17).
  cp_window_len : coverpoint window_length_bucket {
    bins short = {0}; bins medium = {1}; bins long = {2};
  }
  cp_warmup_excluded : coverpoint warmup_was_excluded;
 
  // --- Error injection (Section 32).
  cp_error_rate : coverpoint injected_error_bucket {
    bins clean = {0}; bins spec = {1}; bins elevated = {2}; bins storm = {3};
  }
 
  // --- Snapshot and residual.
  cp_snapshot_under_load : coverpoint snapshot_taken_while_counting;
  cp_residual_closed     : coverpoint residual_within_limit;
 
  // --- Multiple flows, for fairness (15.4 Section 30).
  cp_flows_active : coverpoint num_active_flows { bins one={1}; bins few={[2:3]}; bins many={[4:$]}; }
 
  // --- Crosses.
  x_dominant_achieved : cross cp_dominant, cp_achieved;
  x_error_dominant    : cross cp_error_rate, cp_dominant;      // Section 32
  x_sweep_plateau     : cross cp_swept_param, cp_plateau_seen; // Section 20
  x_window_achieved   : cross cp_window_len, cp_achieved;      // Sections 16, 18
endcovergroup

One illegal_bins and six must-be-non-zero bins.

illegal_bins unattributed. The classifier reaching ST_UNATTRIB is not a coverage hole — it is a defect, and declaring it illegal makes the tool report it as such.

cp_achieved.near_max. Without it the ceiling has never been approached, so §14's ceiling property has never been near its bound and the theoretical figure is unvalidated against reality.

cp_dominant — every state as dominant at least once. Each of §19's six signatures must be produced, or that diagnostic path is untested.

x_sweep_plateau — every parameter swept and a plateau observed. A sweep that never plateaus has not found the parameter's limit, and §20's inference requires the plateau.

cp_warmup_excluded. Proves §17's methodology was applied; without it §18's benchmark error is reachable.

cp_residual_closed. The direct measurement of §33's principle — and it should be at 100%, not merely non-zero.

And x_error_dominant with storm and RECOVERY. §32's category boundary, where a throughput problem becomes a reliability one.

36. Debug Taxonomy

37. Debug Checklist

1. What is the theoretical ceiling, and is its occupancy assumption honest? (§4, §33).
2. Are the units and the measurement boundary consistent? (15.1 §17).
3. Was warmup excluded and drain excluded? (§17, §18).
4. How long was the window, and was the workload stationary within it? (§16).
5. What is achieved throughput, as a fraction of the ceiling?
6. Does the cycle invariant hold? If not, stop — no fraction is trustworthy (§12).
7. Is unattributed_cycles zero? If not, the classifier is incomplete (§14).
8. Does the byte invariant hold, and does the framing fraction match the format? (§13).
9. What fraction of cycles is USEFUL, against both denominators? (§7).
10. What is the dominant non-useful state? That names the investigation (§19).
11. Is the snapshot atomic across the whole bank? (§15).
12. What happens when the payload size is swept? Occupancy against stalls (§21).
13. What happens when outstanding depth, credits, replay window and queue depth are swept? Each identifies a different resource (§22–§24).
14. What happens when rate and width are swept? A flat result eliminates the link (§25).
15. Was only one thing changed between the two measurements? (§26).
16. After the fix, did the dominant state change? If not, the fix was not the mechanism you thought (§27).
17. Does the residual close to zero? If not, a loss has no name (§33).
18. What is the next binding resource? The dominant state after the fix predicts it (§27).

38. Common Misconceptions

"Theoretical throughput should equal measured throughput." It is a ceiling. The gap is real, and the discipline is to attribute every unit of it — not to expect the gap to be zero (§1, §33).

"A busy link means good throughput." 95% busy and 70% useful is a real and common combination — the difference is replay, framing and partial containers. unique_payload / transmitted is the ratio that separates them (§30).

"The biggest FIFO occupancy is the bottleneck." A full queue accumulates behind the bottleneck; the bottleneck's own input queue is typically empty (15.4 §28).

"Increasing queue depth always helps." It helps until the queue stops binding, and the sweep shows exactly where — after which additional depth buys zero throughput and adds latency (§24, §28).

"If x16 is no faster than x8, the link is broken." It means the link is not the bottleneck, which is a decisive negative result and the most useful kind of measurement in this chapter (§25, §31).

"Stall counters may overlap freely." Overlapping counters make fractions sum above 100%, misrank the causes, and make a residual impossible to close. One cycle, one cause, by causal priority (§11, §12).

"Replay only reduces bandwidth by the replayed bytes." It also holds replay entries longer, which stalls admission, which backs work upstream — and at elevated error rates it pulls in recovery cycles too, so throughput falls far more than the replayed byte count (§32).

"Short benchmarks measure steady-state throughput." A 16-object transfer through a 5-stage pipeline measures 80% of theoretical while the same design sustains 99.96%. That benchmark measured pipeline fill, which is a latency property (§17, §18).

"One aggregate counter is enough." One counter gives a number with no verdict and no attribution. Ten mutually exclusive cycle classes plus four byte categories give a decomposition that closes (§6, §8).

"Performance debugging is trial-and-error tuning." It is measurement, attribution and single-variable sweeps. The classifier's dominant state names the next action, and bottleneck migration predicts what will bind after the fix (§19, §27).

"A 5% unexplained residual is acceptable." It is a 5% hole in your understanding of the system. It may be benign; it may be the mechanism that dominates under a workload nobody has run yet, and there is no way to tell while it is unnamed (§33).

"After one optimisation, the original bottleneck remains the bottleneck." A successful optimisation changes what limits you. If throughput improved and the dominant state did not change, either the fix was not the mechanism you assumed, or the same resource binds in two places (§27, §28).

39. Understanding Check

40. Summary — and Module 15 Closed

Theoretical throughput is a ceiling; achieved throughput is that ceiling minus losses that must each have a name. The residual is the measure of how well you understand the system, and its target is zero.

Classify every cycle into exactly one state. Ten states, one-hot, by causal priority rather than severity — SRC_EMPTY third because nothing else is meaningful when nothing was offered, and ARB late because a cycle that also lacked credit was credit-limited. Independent counters make fractions sum to 180%, misrank the causes, and make the residual impossible to close.

Two invariants make the numbers measurements rather than numbers: cycles summing to the observation count, and transmitted bytes summing to payload plus replay plus framing. Check them before believing any fraction, and keep unattributed_cycles at zero — that counter is what proves the other eleven are complete.

Report both denominators. 48% of capacity and 80% of opportunities are the same link, and only the second says anything about the interconnect.

Measure steady state. A 16-object transfer through a five-stage pipeline reports 80% while the same design sustains 99.96% — that benchmark measured pipeline fill, which is a latency property in the wrong units. Define warmup, measurement and drain, and remember the credit loop has its own warmup.

Sweep one parameter at a time, because sweeps give causation where counters give correlation. Payload size separates structural loss from stalls; outstanding, credit and replay sweeps should plateau at rate × latency and cross-check the model; the queue sweep's flat tail ends the bigger-FIFO argument; and rate and width can eliminate the interconnect entirely — a flat result there is the most decisive measurement in the chapter.

The dominant state names the investigation, and after a fix it should change. If it does not, the fix was not the mechanism you assumed or the same resource binds twice. And a fix applied to a resource whose stall fraction was already zero adds latency and nothing else.

Close the residual. A non-zero residual is a finding, not a tolerance — and in the worked case it pointed at an optimistic ceiling rather than a missing stall. Closing it validates the counters, the classifier and the model at once, which is the strongest statement a performance analysis can make.

Module 15 — UCIe Performance, complete

And one vocabulary held across all five: raw is physical capacity, useful is payload after structural overhead, achieved is what the workload receives. No single number represents performance — bandwidth, throughput, latency, utilisation, fairness, scaling efficiency, tail latency and headroom are eight different questions, and observability is what lets an engineer prove which one is binding.

Module 15 quantified the interconnect. Module 16 turns to what the interconnect is increasingly asked to carry: memory and coherence that span dies as if they were one.

• 16.1 — Shared Memory Across Chiplets — what chiplet-spanning shared memory looks like.

Browse the full path on the UCIe tutorials index.