PCIe · Module 22
Throughput — What You Actually Got, and Where the Rest Went
Capacity is arithmetic. Achieved throughput is a measurement, and a measurement is only trustworthy if it names where it counted and what every non-transferring cycle was doing instead.
Chapter 6.7 already answers how much a Link can carry: signalling rate, encoding factor, width, and the exact arithmetic from GT/s to GB/s.
This chapter answers a different question. You ran the DMA. You got a number. It is lower than the capacity, and someone is going to ask why.
"PCIe is slow" is not an answer. Neither is "the link is Gen4 x8, so it should be 15.75 GB/s" — that is a statement about the Link, and the Link is rarely the thing that was limiting.
1. Sources, Scope, and What This Chapter Refuses to Re-derive
2. Capacity Is Computed; Throughput Is Measured
Capacity comes from the specification. Rate × encoding ÷ 8 × width, per direction, exactly as Chapter 6.7 §2 derives it. It is a property of the negotiated Link and nothing else.
Achieved throughput comes from a stopwatch and a counter. It is a property of the system under that workload during that window — and it changes when the workload changes, without the Link changing at all.
§11 Model 1 put the two side by side, using capacity values inherited from Chapter 6.7 and applying utilisation factors — the products are script-computed:
| Link | Capacity (one direction) | Utilisation | Achieved | Unexplained |
|---|---|---|---|---|
| Gen3 x4 | 3.938462 GB/s | 100% | 3.938462 GB/s | 0 |
| Gen3 x4 | 3.938462 GB/s | 75% | 2.953846 GB/s | 0.984615 GB/s |
| Gen4 x8 | 15.753846 GB/s | 75% | 11.815385 GB/s | 3.938462 GB/s |
| Gen4 x8 | 15.753846 GB/s | 50% | 7.876923 GB/s | 7.876923 GB/s |
| Gen5 x16 | 63.015385 GB/s | 50% | 31.507692 GB/s | 31.507692 GB/s |
The last column is the entire chapter. On Gen5 x16 at 50% utilisation, 31.5 GB/s went somewhere, and the arithmetic that produced the capacity figure cannot tell you where. §6 can.
Note what the utilisation column is not. It is not a fudge factor and not a vendor derate — it is the fraction of measurement cycles that actually moved payload, and it is directly measurable in RTL (§8).
3. The Measurement Point
4. Units, Basis, and Dimensional Analysis
Chapter 6.7 §2 and §5 own this — the recap is short because the trap is real.
GT/s is not Gb/s is not GB/s. Transfers per second become bits per second only after the encoding factor, and bytes only after ÷8. DERIVED:
[GT/s] × [bits/transfer, after encoding] = [Gb/s]
[Gb/s] ÷ 8 [bits/byte] = [GB/s]
[GB/s per lane] × [lanes] = [GB/s per direction]Carry the units through the measurement side too, because that is where they are usually dropped:
[bytes accepted] [bytes] [bytes]
──────────────────── = ───────── × ──────────────── = [bytes/second]
[cycles] × [s/cycle] [cycle] [second]Which makes the RTL decision in §8 obvious. The hardware counts bytes and cycles — two dimensionless integers — and the division happens in software where the clock period is known. A rate computed in the datapath bakes in a clock assumption that a later frequency change silently invalidates.
And one basis rule inherited from 6.7 §4, restated because it is the most common reporting error: PCIe is full duplex, so TX and RX capacity do not sum into a one-way figure. A x8 Link does not carry a one-directional stream at twice its one-direction capacity. Report one-way throughput per direction, and report simultaneous bidirectional traffic separately and say so.
5. The Ceiling Stack
Any of these can be the binding constraint, and only one of them is "the Link".
| Layer | What sets the ceiling | Who owns it |
|---|---|---|
| Physical | negotiated generation and negotiated width | 17.3, 6.7 |
| Encoding | the generation's line code or FLIT structure | 5.3, 5.6 |
| Packetization | header/payload ratio, MPS and MRRS | 22.4, 22.5 |
| Flow control | credits available per class | 16.1, 22.3 |
| Outstanding work | Tags and request depth versus round-trip | 20.5 §5 |
| Source / sink | DMA engine, host memory, application drain rate | 20.6 |
| Scheduler / fabric | arbitration and contention across ports | 21.3 §7, 21.4 §7 |
Read the first row carefully, because it produces the most embarrassing bug report in this chapter. Capability x16 is not negotiated x16 (Chapter 17.3). A Link that trained to x4 because of a marginal lane will deliver a quarter of the expected capacity while every device reports itself as x16-capable. Read the negotiated width before computing anything.
And note the row this chapter does not need to re-teach. Chapter 20.5 §5 already established that occupying a pipe requires enough independent work in flight to cover the round trip. 22.1 adds only this: when that condition fails, the cycles it costs land in a specific, countable category — no_work — and §6 is how you prove it rather than assert it.
6. Every Cycle Has Exactly One Reason
7. Counting on valid Is Not Counting Bytes
8. The Waveform
Same link rate, half the throughput — and the reasons are countable
10 cyclesThree things to read out of the figure.
The link rate is constant and the throughput is 50%. Five of ten cycles transferred. No capacity calculation can produce the number 50% from this Link's parameters — it is a property of the other five cycles.
Those five cycles have three different causes, and each indicts a different subsystem: cycles 2–3 the receiver's credit availability (22.3), cycles 4–5 the data source, cycle 6 downstream back-pressure.
And cycle 6 is the one that breaks naive counters. tx_valid is high and no bytes moved. A counter keyed on valid records a transfer here (§7); the transfer row shows the truth.
9. RTL — Measuring, and Not Lying
// SYNTHESIZABLE. Measurement types and a reusable saturating counter.
// Diagnostic counters SATURATE rather than wrap: a wrapped counter turns a
// saturated link into an apparently idle one (§11 Model 5: 255 -> 232).
package perf_pkg;
parameter int CNT_W = 32;
parameter int BYTES_W = 12; // bytes accepted per beat
parameter int WINDOW_W = 24;
typedef enum logic [2:0] {
CAT_TRANSFER = 3'd0,
CAT_STALLED = 3'd1,
CAT_NO_CREDIT = 3'd2,
CAT_NO_WORK = 3'd3,
CAT_LINK_DOWN = 3'd4
} cycle_cat_e;
function automatic logic [CNT_W-1:0] sat_add(input logic [CNT_W-1:0] v,
input logic [CNT_W-1:0] inc);
logic [CNT_W:0] wide;
wide = {1'b0, v} + {1'b0, inc};
return wide[CNT_W] ? {CNT_W{1'b1}} : wide[CNT_W-1:0];
endfunction
function automatic bit is_saturated(input logic [CNT_W-1:0] v);
return (v == {CNT_W{1'b1}});
endfunction
endpackageimport perf_pkg::*;
// SYNTHESIZABLE. Payload byte counter at the DECLARED measurement point (§3).
// Increments ONLY on the handshake -- counting on valid alone inflated the
// total by 17.7% in §11 Model 3.
module payload_byte_counter (
input logic clk,
input logic rst_n,
input logic measuring,
input logic tx_valid,
input logic tx_ready,
input logic [BYTES_W-1:0] tx_byte_count, // VALID bytes this beat, not bus width
output logic [CNT_W-1:0] payload_bytes,
output logic saturated,
output logic zero_byte_seen // sticky: a beat claiming 0 bytes
);
logic [CNT_W-1:0] acc_q;
logic zb_q;
logic xfer;
assign xfer = tx_valid && tx_ready; // THE transfer, §7
assign payload_bytes = acc_q;
assign saturated = is_saturated(acc_q);
assign zero_byte_seen = zb_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin acc_q <= '0; zb_q <= 1'b0; end
else if (measuring && xfer) begin
acc_q <= sat_add(acc_q, CNT_W'(tx_byte_count));
if (tx_byte_count == '0) zb_q <= 1'b1; // report, do not silently count
end
end
endmoduleimport perf_pkg::*;
// SYNTHESIZABLE. THE CORE BLOCK (§6). Classifies every measured cycle into
// exactly one category by declared priority, so the categories SUM to the
// window. §11 Model 2: residual 0 over 600,000 cycles, versus a 6.2%
// overcount for independent per-cause counters.
module cycle_attribution (
input logic clk,
input logic rst_n,
input logic measuring,
input logic link_operational,
input logic work_pending, // something is queued to send
input logic credit_ok, // credits suffice for the pending packet
input logic tx_valid,
input logic tx_ready,
output logic [CNT_W-1:0] c_transfer,
output logic [CNT_W-1:0] c_stalled,
output logic [CNT_W-1:0] c_no_credit,
output logic [CNT_W-1:0] c_no_work,
output logic [CNT_W-1:0] c_link_down,
output logic [CNT_W-1:0] c_cycles, // the DENOMINATOR
output cycle_cat_e cat_this_cycle,
output logic any_saturated
);
logic [CNT_W-1:0] t_q, s_q, nc_q, nw_q, ld_q, cy_q;
// ================================================================
// ONE cycle, ONE category. The priority order encodes what you would
// fix first: a down Link moots everything; nothing pending indicts the
// source, not PCIe; only then is credit the story (§6).
// ================================================================
always_comb begin
if (!link_operational) cat_this_cycle = CAT_LINK_DOWN;
else if (!work_pending) cat_this_cycle = CAT_NO_WORK;
else if (!credit_ok) cat_this_cycle = CAT_NO_CREDIT;
else if (tx_valid && tx_ready) cat_this_cycle = CAT_TRANSFER;
else cat_this_cycle = CAT_STALLED;
end
assign c_transfer = t_q; assign c_stalled = s_q;
assign c_no_credit = nc_q; assign c_no_work = nw_q;
assign c_link_down = ld_q; assign c_cycles = cy_q;
assign any_saturated = is_saturated(t_q) || is_saturated(s_q)
|| is_saturated(nc_q) || is_saturated(nw_q)
|| is_saturated(ld_q) || is_saturated(cy_q);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
t_q<='0; s_q<='0; nc_q<='0; nw_q<='0; ld_q<='0; cy_q<='0;
end else if (measuring) begin
cy_q <= sat_add(cy_q, CNT_W'(1));
unique case (cat_this_cycle)
CAT_TRANSFER: t_q <= sat_add(t_q, CNT_W'(1));
CAT_STALLED: s_q <= sat_add(s_q, CNT_W'(1));
CAT_NO_CREDIT: nc_q <= sat_add(nc_q, CNT_W'(1));
CAT_NO_WORK: nw_q <= sat_add(nw_q, CNT_W'(1));
CAT_LINK_DOWN: ld_q <= sat_add(ld_q, CNT_W'(1));
endcase
end
end
endmoduleimport perf_pkg::*;
// SYNTHESIZABLE. Sliding-window rate measurement WITHOUT a divider (§4).
// Publishes bytes-per-window; software divides by WINDOW_CYCLES and the
// clock period. Verified at WINDOW_CYCLES = 1 (§11 Model 4, §12).
module throughput_window #(parameter int WINDOW_CYCLES = 1024) (
input logic clk,
input logic rst_n,
input logic measuring,
input logic tx_valid,
input logic tx_ready,
input logic [BYTES_W-1:0] tx_byte_count,
output logic [CNT_W-1:0] window_bytes, // live accumulator
output logic [CNT_W-1:0] last_window_bytes, // published snapshot
output logic snap_valid,
output logic [WINDOW_W-1:0] window_cycle
);
// Guarded so WINDOW_CYCLES = 1 is legal: the boundary is then EVERY cycle.
localparam int unsigned WC = (WINDOW_CYCLES < 1) ? 1 : WINDOW_CYCLES;
logic [CNT_W-1:0] acc_q, snap_q;
logic [WINDOW_W-1:0] cyc_q;
logic snap_q_v;
logic at_boundary, xfer;
assign xfer = tx_valid && tx_ready;
assign at_boundary = measuring && (cyc_q == WINDOW_W'(WC - 1));
assign window_bytes = acc_q;
assign last_window_bytes = snap_q;
assign snap_valid = snap_q_v;
assign window_cycle = cyc_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
acc_q <= '0; snap_q <= '0; cyc_q <= '0; snap_q_v <= 1'b0;
end else begin
snap_q_v <= 1'b0;
if (measuring) begin
// SAME-CYCLE RULE (§10): a transfer ON the boundary cycle belongs to
// the window that is closing, not to the next one.
automatic logic [CNT_W-1:0] nxt = xfer ? sat_add(acc_q, CNT_W'(tx_byte_count))
: acc_q;
if (at_boundary) begin
snap_q <= nxt; // publish INCLUDING this cycle
snap_q_v <= 1'b1;
acc_q <= '0;
cyc_q <= '0;
end else begin
acc_q <= nxt;
cyc_q <= cyc_q + WINDOW_W'(1);
end
end
end
end
endmoduleimport perf_pkg::*;
// VERIFICATION-ONLY. Completeness monitor: the categories must sum to the
// window (§6). This is the property that makes the taxonomy a measurement
// rather than a set of opinions.
module attribution_completeness_monitor (
input logic [CNT_W-1:0] c_transfer, c_stalled, c_no_credit,
c_no_work, c_link_down, c_cycles,
input logic any_saturated,
output logic err_residual
);
logic [CNT_W+2:0] sum;
always_comb begin
sum = c_transfer + c_stalled + c_no_credit + c_no_work + c_link_down;
// Only meaningful while nothing has saturated -- saturation is lossy
// BY DESIGN, and the monitor must not report it as a taxonomy error.
err_residual = !any_saturated && (sum != {3'b0, c_cycles});
end
endmoduleClassification: four synthesizable, one verification-only.
Failure — five. Counting on valid (17.7%, §11). Independent instead of exclusive categories (6.2% overcount). Wrapping diagnostics (a saturated link reads as idle). A boundary-cycle transfer credited to the next window (§10). And a rate divider in the datapath that bakes in a clock period.
10. Same-Cycle Audit and Assertions
// ==================================================================
// BYTE ACCOUNTING (§7) -- the transfer, not the offer.
// ==================================================================
// P1: payload bytes advance ONLY on a handshake. §11 Model 3: counting on
// valid alone inflated the total by 17.7%.
property p_bytes_on_transfer_only;
@(posedge clk) disable iff (!rst_n)
(payload_bytes != $past(payload_bytes)) |->
$past(measuring && tx_valid && tx_ready);
endproperty
// P2: nothing is counted while measurement is off.
property p_no_count_when_idle;
@(posedge clk) disable iff (!rst_n)
(!measuring) |=> $stable(payload_bytes);
endproperty
// P3: the increment equals the ACCEPTED byte count, not the bus width.
property p_increment_exact;
@(posedge clk) disable iff (!rst_n)
(measuring && tx_valid && tx_ready && !saturated) |=>
(payload_bytes == $past(payload_bytes) + CNT_W'($past(tx_byte_count)));
endproperty
// P4: a zero-byte beat is REPORTED, never silently counted as progress.
property p_zero_byte_flagged;
@(posedge clk) disable iff (!rst_n)
(measuring && tx_valid && tx_ready && (tx_byte_count == '0)) |=> zero_byte_seen;
endproperty
// ==================================================================
// CYCLE ATTRIBUTION (§6) -- the completeness contract.
// ==================================================================
// P5: exactly one category per measured cycle. The enum makes this
// structural; the property makes it checkable.
property p_one_category_per_cycle;
@(posedge clk) disable iff (!rst_n)
measuring |-> (cat_this_cycle inside {CAT_TRANSFER, CAT_STALLED,
CAT_NO_CREDIT, CAT_NO_WORK, CAT_LINK_DOWN});
endproperty
// P6: exactly one counter advances per measured cycle.
property p_one_counter_advances;
@(posedge clk) disable iff (!rst_n)
(measuring && !any_saturated) |=>
($countones({c_transfer != $past(c_transfer),
c_stalled != $past(c_stalled),
c_no_credit != $past(c_no_credit),
c_no_work != $past(c_no_work),
c_link_down != $past(c_link_down)}) == 1);
endproperty
// P7: THE COMPLETENESS PROPERTY. The parts sum to the whole -- §11 Model 2
// measured residual 0 over 600,000 cycles.
property p_categories_sum_to_window;
@(posedge clk) disable iff (!rst_n)
(!any_saturated) |->
((c_transfer + c_stalled + c_no_credit + c_no_work + c_link_down)
== c_cycles);
endproperty
// P8: the priority order is respected -- a down Link is never attributed
// to credit, however credit-starved it also is.
property p_link_down_wins;
@(posedge clk) disable iff (!rst_n)
(measuring && !link_operational) |-> (cat_this_cycle == CAT_LINK_DOWN);
endproperty
// P9: no-work outranks no-credit. Nothing pending is a SOURCE problem and
// must not be reported as a PCIe credit problem (§13).
property p_no_work_outranks_credit;
@(posedge clk) disable iff (!rst_n)
(measuring && link_operational && !work_pending) |->
(cat_this_cycle == CAT_NO_WORK);
endproperty
// P10: a stall requires an unaccepted offer. valid && !ready, nothing else.
property p_stall_requires_offer;
@(posedge clk) disable iff (!rst_n)
(cat_this_cycle == CAT_STALLED) |-> (tx_valid && !tx_ready);
endproperty
// P11: a transfer category implies the handshake.
property p_transfer_implies_handshake;
@(posedge clk) disable iff (!rst_n)
(cat_this_cycle == CAT_TRANSFER) |-> (tx_valid && tx_ready);
endproperty
// ==================================================================
// SATURATION (§9) -- diagnostics must degrade honestly.
// ==================================================================
// P12: counters SATURATE, never wrap. §11 Model 5: a wrapping 8-bit
// counter read 232 after 1000 increments -- an idle-looking saturated link.
property p_never_wraps;
@(posedge clk) disable iff (!rst_n)
(c_cycles == {CNT_W{1'b1}}) |=> (c_cycles == {CNT_W{1'b1}});
endproperty
// P13: monotonic while measuring -- a diagnostic counter never decreases.
property p_counters_monotonic;
@(posedge clk) disable iff (!rst_n)
(c_transfer >= $past(c_transfer)) && (c_cycles >= $past(c_cycles));
endproperty
// P14: saturation is sticky and visible, so software can discard the window.
property p_saturation_sticky;
@(posedge clk) disable iff (!rst_n)
any_saturated |=> any_saturated;
endproperty
// ==================================================================
// WINDOW (§9) -- boundary behaviour, incl. WINDOW_CYCLES = 1.
// ==================================================================
// P15: the snapshot is published exactly once per window.
property p_snapshot_once_per_window;
@(posedge clk) disable iff (!rst_n)
snap_valid |=> !snap_valid until_with (window_cycle == WINDOW_W'(0));
endproperty
// P16: the accumulator restarts at zero after a boundary.
property p_accumulator_restarts;
@(posedge clk) disable iff (!rst_n)
snap_valid |-> (window_cycle == WINDOW_W'(0));
endproperty
// P17: the boundary-cycle transfer is INCLUDED in the closing window (§10).
property p_boundary_transfer_included;
@(posedge clk) disable iff (!rst_n)
(snap_valid && $past(tx_valid && tx_ready && measuring)) |->
(last_window_bytes >= CNT_W'($past(tx_byte_count)));
endproperty
// P18: the window counter never exceeds its modulus -- safe at WC = 1.
property p_window_counter_bounded;
@(posedge clk) disable iff (!rst_n)
(window_cycle <= WINDOW_W'(WC - 1));
endproperty
// ==================================================================
// NON-INTERFERENCE -- the measurement must not become the experiment.
// ==================================================================
// P19: no diagnostic output drives the transmit interface. Formally: tx
// behaviour is independent of every counter value.
property p_monitor_does_not_drive_tx;
@(posedge clk) disable iff (!rst_n)
$stable({tx_valid, tx_ready, tx_byte_count}) or
!$stable({c_transfer, c_cycles, payload_bytes});
endproperty
// P20: measured payload can never exceed the interface's own capacity --
// bytes/cycle is bounded by the maximum beat size.
property p_payload_bounded_by_capacity;
@(posedge clk) disable iff (!rst_n)
(!any_saturated) |-> (payload_bytes <= c_cycles * CNT_W'(2**BYTES_W - 1));
endpropertyTwenty properties. P5–P11 are the chapter — they are the only place the exclusive taxonomy becomes checkable rather than merely intended. P19 is the one people forget: a monitor that back-pressures the interface it measures reports a system that does not exist.
11. Measured Behaviour
12. Verification — DV and Mutations
DV, against an independent byte-and-cycle model (not the DUT's own functions): continuous traffic · alternating ready · no work · no credit · link unavailable · several block causes simultaneously · WINDOW_CYCLES = 1 · a transfer exactly on the final window cycle · counter saturation · reset mid-window · maximum byte count per beat · a zero-byte beat.
| # | Mutation | Symptom | Caught by |
|---|---|---|---|
| 1 | Count bytes on tx_valid alone | +17.7% inflation, worst under back-pressure (§11) | P1 |
| 2 | Count the full bus width instead of tx_byte_count | every partial beat over-counted | P3 |
| 3 | Count while measuring is low | window boundaries meaningless | P2 |
| 4 | Silently count a zero-byte beat as progress | phantom transfers, no evidence | P4 |
| 5 | Independent per-cause counters instead of exclusive | 6.2% overcount; causes cannot be ranked (§11) | P6, P7 |
| 6 | Drop the completeness check | a taxonomy nobody can falsify | P7 |
| 7 | Rank credit above no-work | idle source reported as a PCIe credit problem | P9 |
| 8 | Rank credit above link-down | a dead Link reported as credit starvation | P8 |
| 9 | Count a stall without an offer pending | idle cycles reported as back-pressure | P10 |
| 10 | Classify valid && !ready as a transfer | cycle 6 of Figure 1 counted as useful | P11 |
| 11 | Let diagnostic counters wrap | saturated link reads as idle (232 vs 255) | P12, P14 |
| 12 | Allow a counter to be cleared mid-window by a read | non-monotonic; windows silently short | P13 |
| 13 | Credit the boundary-cycle transfer to the next window | off-by-one; every window misreports its edge | P17 |
| 14 | Publish the snapshot twice at a boundary | software double-counts a window | P15 |
| 15 | Fail to reset the accumulator after a snapshot | windows accumulate; rate grows without bound | P16 |
| 16 | Compute bytes/cycles in the datapath | bakes in a clock period; wrong after any change | design review |
| 17 | Let the monitor back-pressure the TX interface | measurement changes the system it measures | P19 |
| 18 | WINDOW_CYCLES = 0 unguarded | zero-modulus counter; no boundary ever | P18 |
| 19 | Report throughput without a measurement point | two correct numbers look like a contradiction (§3) | design review |
| 20 | Count replayed wire bytes as application payload | reports throughput the application never sees | design review |
| 21 | Sum TX and RX into a one-way figure | doubles the reported result (§4) | design review |
| 22 | Use capability width instead of negotiated width | capacity overstated by the training shortfall (§5) | design review |
Two counterexamples worth stating explicitly.
Mutation 1 is the one that ships, and it ships because it passes. On an unstalled interface valid and valid && ready are identical, so every clean bench test agrees. The divergence appears only under back-pressure — that is, only in the loaded system someone is trying to diagnose, and it overstates throughput precisely there. §11 Model 3's 17.7% is the gap between a design that looks healthy and one that is.
Mutation 5 is subtler and worse. Independent counters are individually correct — each really does count cycles where its condition held. They are wrong only in combination, and the failure is that they sum past the window, so the shares do not form a distribution and no cause can be ranked against another. P7 is a one-line property that makes the whole taxonomy falsifiable, and it is the only assertion in this chapter that cannot be expressed by looking at one counter.
13. Debugging
Scenario — "Gen4 x8 should give ~15.75 GB/s and we measure 7."
Do not start at the PHY. In order: negotiated generation and width (capability is not negotiation, §5) · the measurement point on both sides (§3) · then the attribution counters (§6). §11 Model 2's shape is the common answer — the largest category is usually no_work, and that is a DMA-engine or descriptor-supply problem with no PCIe content at all.
Scenario — the analyzer looks busy, the application is slow. The Link is carrying transport, not payload. Either small TLPs are spending the Link on headers (22.4 and 22.5 own the exact accounting) or packets are being replayed — Chapter 14.4's replay counters distinguish them immediately, and replayed bytes are not application bytes (§3).
Scenario — the DMA engine reports 14 GB/s, the application reports 11. Probably neither is wrong (§3). Ask each side where it counts. The 3 GB/s is then a locatable quantity — host-side service, driver batching, or a buffer drain rate — rather than a contradiction between two instruments.
Scenario — throughput oscillates: bursts, then flat spots.
Window counters are the right tool (§9): capture per-window attribution, then look at what dominates the flat windows. no_work means the source starves periodically; no_credit means the receiver's buffering is the limit and Chapter 22.3 is the next chapter to open; stalled means downstream.
Scenario — throughput reported as excellent, users disagree.
Suspect the counters before the users. Check for mutation 1 (counting on valid), mutation 11 (a wrapped counter), and mutation 21 (TX+RX summed). All three inflate, and all three survive casual review.
Scenario — the numbers changed after a clock-frequency change and nothing else did. Look for a rate computed in hardware (mutation 16). Bytes and cycles are frequency-independent; a GB/s value computed in the datapath is not.
14. Misconceptions
"GT/s and GB/s are the same number in different units." No — transfers become bits only after the encoding factor and bytes only after ÷8 (6.7 §2, §4 here).
"The device is x16-capable, so the Link is x16." No. Negotiated width is what carries traffic (17.3), and a Link that trained down reports full capability all the way through.
"Link rate is throughput." No — link rate is a ceiling, and §11 Model 2 measured a run sitting at 49.39% of it with the single largest cause being no work.
"Full duplex means one transfer can use TX+RX bandwidth." No. A one-directional stream uses one direction (6.7 §4).
"A higher generation gives proportionally higher application throughput." Only if the binding constraint was the Link. If it was the source, credits or outstanding work (§5), doubling the Link changes nothing measurable.
"A big, fast Link means credits are never the limit." No — credits are about receiver buffering and its return loop, not link speed (22.3).
"Replayed bytes count." They cross the wire and deliver payload once (§3).
"tx_valid means bytes moved." No — cycle 6 of Figure 1 is exactly this, and it is a 17.7% error at scale (§7).
"A throughput monitor may count offered traffic; it is only a monitor." Then it is a monitor of offers, not of throughput.
"All PCIe generations share one efficiency constant." No — and applying × 128/130 to Gen6 is the specific error Chapter 6.7 §2 calls out, because Gen6 is 1b/1b at the line with its overhead relocated into the FLIT.
"50% utilisation is a finding." It is a number. The finding is the attribution (§6).
15. Understanding Check
Q1. A Gen4 x8 Link is negotiated correctly, and the TX interface transfers a 256-byte payload on only half of its cycles. Why is computing throughput from Gen4 x8 alone guaranteed to overestimate? Because that arithmetic produces capacity, which assumes every cycle transfers. Capacity is a property of the Link; achieved throughput is a property of the workload on it. The other half of the cycles are not in the formula — they are in the attribution counters, and until you read them you cannot say whether the loss was the source, credits, back-pressure, or the Link.
Q2. Your monitor counts on tx_valid. On a clean bench it matches the reference exactly. Why is it still broken, and when does it bite?
Because valid and valid && ready coincide only when nothing back-pressures. The bug appears exactly under load and inflates by 17.7% in §11 Model 3 — so it reports health precisely when the system is unhealthy.
Q3. Two per-cause counters each show 30% blocked and the window is 100%. What is structurally wrong? The categories are not exclusive — they are counting the same cycles twice, so the shares are not a distribution and cannot be ranked. §11 measured a 6.2% overcount from exactly this. P7 is the property that catches it, and it cannot be written by inspecting any single counter.
Q4. Your 32-bit cycle counter reads a smaller value after a longer, busier run. What happened, and what should the design have done? It wrapped. §11 Model 5's 8-bit illustration read 232 after saturating at 255 — a saturated link that looks idle. Diagnostics should saturate and report saturation stickily (P12, P14) so software discards the window instead of believing it.
Q5. A DMA engine reports 14 GB/s and the application reports 11 GB/s on the same run. Who is wrong? Probably nobody (§3). They measured at different points. The right response is to name both points and treat the 3 GB/s as a located quantity — driver batching, host-side service, or buffer drain — not to argue about which instrument is broken.
Q6. Why does this chapter's RTL expose bytes and cycles instead of a GB/s value?
Because the division needs a clock period, and a rate computed in the datapath embeds a frequency assumption that a later change silently invalidates (§4, mutation 16). Bytes and cycles are dimensionless integers, and they stay correct when the clock does not.
16. What's Next
This chapter answered how much, and — more usefully — where the rest went.
Chapter 22.2 asks a different question about the same system: not how much moved per second, but how long one unit of work took, and where that time went. The two are not reciprocals — §11 Model 2's run has high no_work, which is a throughput symptom with no latency component at all.
Chapter 22.3 then takes the no_credit category apart. This chapter counted those cycles; it did not say which of the six credit classes was empty, whether a packet of that class was even pending, or what ceiling the credit-return loop imposes. That is the deepest chapter of the three.
And three chapters after that stay deliberately untouched here. 22.4 owns MPS/MRRS and packetization; 22.5 owns the exact header, framing and encoding overhead accounting that §5 named only as a term; 22.6 owns reading a real benchmark without lying to yourself — for which §3's measurement point is the prerequisite this chapter supplies.