GPU Interface — Accounting for the Whole Shortfall

29.1 accounted for one command's microseconds, and the stages added up. This chapter accounts for a transfer's bytes, and they do not add — they multiply, which changes both the arithmetic and the instrument you need.

1. Sources, Scope, and What This Chapter Refuses to Do

2. Additive Budgets and Multiplicative Ones

The difference matters because it decides what instrument you need.

Two readings.

A single overall efficiency number is the natural thing to measure and it is nearly useless. useful ÷ ceiling = 0.61 tells you 39 % was lost and gives no basis for choosing what to fix. The stages are not separable from the product, so you have to measure each factor where it occurs.

And this is why "our link runs at 61 % efficiency" is a statement that ends investigations rather than starting them. §5 decomposes the same 0.61 into four factors, and three of the four have completely different fixes.

3. The Data Path

Four factors, and they belong to four different owners.

And the ownership column is the reason to decompose. A 39 % shortfall attributable to headers is a payload-size conversation; the same 39 % attributable to the window is an RTL conversation. One number cannot tell you which meeting to hold.

4. Derive the Ceiling First

Step 1 — the post-encoding ceiling. 2.0 GB/s × 16 lanes = 32.0 GB/s in one direction.

That is the number a measurement should be compared against — not a raw signalling figure, because factor 1 has already been applied. Comparing a measured useful rate against a raw rate double-counts the encoding loss, which is the most common way an efficiency figure is made to look worse than it is.

5. The Cascade

Illustrative parameters: payload 256 bytes per TLP; header and framing overhead 24 bytes per TLP; retransmission rate 3 %; device outstanding window 64 tags, round-trip 1.0 µs.

Factor 2 — header overhead.

256 / (256 + 24) = 0.914

Factor 3 — retry. Wire capacity spent carrying retransmitted payload is unavailable for new payload:

1 / (1 + 0.03) = 0.971

Factor 4 — the outstanding window. The device can sustain at most window × payload ÷ round_trip:

64 × 256 B ÷ 1.0 µs = 16.4 GB/s

As a fraction of what the previous factors would have allowed — 32.0 × 0.914 × 0.971 = 28.4 GB/s — that is:

16.4 / 28.4 = 0.577

The product.

Four readings, and the third is the one that changes behaviour.

The overall figure is 51 %, and it is dominated by one factor. Factors 2 and 3 together cost 11 %; factor 4 alone costs 42 %. A report that says "we achieve 51 % of link" invites payload-size and signal-integrity work — and both would be nearly worthless here.

Factor 4 is the device's, and its fix is arithmetic (26.2 §5). To reach 28.4 GB/s the device needs 28.4e9 × 1.0e-6 ÷ 256 ≈ 111 tags, not 64. That is an RTL parameter with a derivation, and it is the single highest-value change available.

The factors interact, which is why order-of-magnitude intuition fails. Doubling the payload to 512 bytes improves factor 2 to 0.955 and factor 4 simultaneously — the same 64 tags now carry twice the bytes, so the window supports 64 × 512 ÷ 1.0 µs = 32.8 GB/s, above the 30.5 GB/s the earlier factors permit. Factor 4 stops binding entirely, and overall efficiency jumps to about 0.928. One parameter, two factors.

And factor 3 is the one to leave alone. 3 % retransmission costs 2.9 % of throughput. It is a signal-integrity signal worth watching and almost never the largest term — but §7 is what happens when it is mismeasured.

6. RTL — Four Ratios, Measured Where They Occur

Because the factors multiply, each must be measured at its own boundary. A single ratio at the end cannot be decomposed.

// ILLUSTRATIVE. Four independent measurements, each taken where its factor
// occurs. The design point is that NO counter here divides anything — the
// hardware collects numerators and denominators, and the ratios are formed
// by software, which keeps the hardware cheap and the arithmetic auditable.
localparam int CW = 48;
 
// Factor 2 — header overhead. Both quantities at the TLP boundary.
logic [CW-1:0] tlp_payload_bytes_q;   // payload bytes in transmitted TLPs
logic [CW-1:0] tlp_total_bytes_q;     // payload + header + framing
 
// Factor 3 — retry. Retransmitted payload is counted SEPARATELY, never folded
// into the useful total (§7's defect).
logic [CW-1:0] retry_payload_bytes_q;
 
// Factor 4 — the window. Cycles in which the device WANTED to issue and could
// not because no tag was free. This is the only direct evidence for factor 4.
logic [CW-1:0] no_tag_stall_cyc_q;
logic [CW-1:0] issue_active_cyc_q;
logic [7:0]    tags_in_use_q, tags_high_water_q;
 
// The delivered total, counted at RETIREMENT of unique payload — the one number
// a throughput claim may be made from.
logic [CW-1:0] useful_bytes_q;
 
function automatic logic [CW-1:0] sat(logic [CW-1:0] a, logic [CW-1:0] b);
  sat = ((a + b) < a) ? {CW{1'b1}} : (a + b);      // saturate, never wrap
endfunction
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n || ctr_clear) begin
    tlp_payload_bytes_q <= '0; tlp_total_bytes_q   <= '0;
    retry_payload_bytes_q <= '0; no_tag_stall_cyc_q <= '0;
    issue_active_cyc_q  <= '0; useful_bytes_q      <= '0;
    tags_in_use_q <= '0; tags_high_water_q <= '0;
  end else begin
    // Factor 2 numerator and denominator, at the transmit boundary.
    if (tlp_fire) begin
      tlp_total_bytes_q <= sat(tlp_total_bytes_q, CW'(tlp_bytes + HDR_BYTES));
      if (!tlp_is_retry)
        tlp_payload_bytes_q <= sat(tlp_payload_bytes_q, CW'(tlp_bytes));
      else
        retry_payload_bytes_q <= sat(retry_payload_bytes_q, CW'(tlp_bytes));
    end
 
    // Factor 4 evidence. Stall counted only when the engine had work AND no
    // tag — distinguishing "window-limited" from "nothing to do".
    if (issue_want && !tag_free) no_tag_stall_cyc_q <= sat(no_tag_stall_cyc_q, 1);
    if (issue_want)              issue_active_cyc_q <= sat(issue_active_cyc_q, 1);
 
    // One signed next-state expression, so an allocation and a retirement in
    // the same cycle net correctly instead of losing one.
    tags_in_use_q <= tags_in_use_q + 8'(tag_alloc_fire) - 8'(tag_retire_fire);
    if (tags_in_use_q > tags_high_water_q) tags_high_water_q <= tags_in_use_q;
 
    // Useful bytes at RETIREMENT of unique payload (26.2 §8's byte accounting).
    if (retire_fire) useful_bytes_q <= sat(useful_bytes_q, CW'(retire_bytes));
  end
end

Architecture. Seven counters and one high-water mark, arranged so that each factor has its own numerator and denominator at its own boundary. Nothing is divided in hardware.

State. Byte and cycle accumulators, all saturating. tags_high_water_q is the direct evidence for factor 4 — if it never reaches the tag count, the window is not the constraint and factor 4 is not binding.

Event. Factor 2 at tlp_fire; factor 3 separated by tlp_is_retry; factor 4's stall gated on issue_want && !tag_free; useful bytes at retirement, not at transmission.

Contract. retire_fire must be the retirement of unique payload (26.2 §8) — a replayed delivery must not increment it. If retirement is wired to a transport event, §7's defect reappears one layer down.

Failure. The realistic errors are all about which event feeds which counter. issue_active_cyc_q counting all cycles rather than wanting-to-issue cycles makes factor 4's ratio meaningless — an idle device would read as window-limited.

DV/debug. tags_high_water_q versus the tag count is the one-read test for factor 4: at the limit, the window binds; well below it, look elsewhere. And no_tag_stall_cyc_q ÷ issue_active_cyc_q quantifies how much, which is the number §5's factor 4 predicts.

7. Wrong RTL — One Ratio, and Retry Counted as Useful

// WRONG. ILLUSTRATIVE. A single throughput counter at the transmit boundary.
// This is the cheapest possible instrument and it makes two errors that push
// the reported number in OPPOSITE directions — so they can cancel and hide
// each other.
logic [CW-1:0] wire_bytes_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) wire_bytes_q <= '0;
  else if (tlp_fire)
    // BUG 1: header and framing bytes are included, so "throughput" is inflated
    //        relative to useful payload.
    // BUG 2: retransmitted TLPs are counted, so the number RISES as the link
    //        degrades — the metric improves as the system gets worse.
    wire_bytes_q <= wire_bytes_q + CW'(tlp_bytes + HDR_BYTES);
end
// BUG 3: reported as one ratio against the ceiling, which gives a total loss
//        with no attribution — §2's argument.
assign reported_efficiency_x1000 = (wire_bytes_q * 1000) / expected_bytes;

Architecture. One accumulator and one ratio.

State. wire_bytes_q. The missing state is every denominator and every per-cause numerator — which is why no decomposition is possible from this design at all.

Event. Every transmitted TLP, including retransmissions, including headers.

Contract. Whoever reads reported_efficiency believes it is useful payload delivered as a fraction of the achievable rate. It is wire occupancy as a fraction of an expectation.

Failure — the timeline, with §5's parameters. A transfer where the retransmission rate rises mid-run from 0 % to 12 %.

First divergence: interval 20–30 ms — the useful rate fell and the reported figure did not move. Because retried bytes occupy the wire, wire occupancy stays constant while useful throughput drops, and the metric is blind to exactly the degradation it should detect.

Root cause. The instrument measures the wire, and the wire is busy whether or not the bytes are new. BUG 2 is the dangerous half: a retransmission-driven collapse is invisible, and in a worse case where retry rises further the reported number would increase.

And BUG 1 makes the baseline wrong in the other direction, which is why the two are worth separating. Header inclusion inflates the figure by about 9 % at 256-byte payloads. A team that "corrects for headers" without separating retry then has a number that is accurate at 0 % retry and progressively wrong as the link degrades.

DV/debug. The signature is an efficiency figure that does not move while users report degradation, and the discriminator is one comparison: useful_bytes at retirement versus wire_bytes at transmit. A widening gap between them is the retry rate, and §6 measures it directly.

8. Corrected — and the Assertions That Keep It Honest

// MANDATORY. English: transmitted payload bytes equal unique retired payload
// plus retried payload. This is the identity that makes the retry factor
// trustworthy — if it does not close, retry is being folded into useful
// somewhere and §5's factor 3 is unmeasurable.
a_payload_accounting_closes: assert property (
  @(posedge clk) disable iff (!rst_n)
    tlp_payload_bytes_q == (useful_bytes_q + retry_payload_bytes_q)
);
 
// MANDATORY. English: useful bytes only increase on retirement of unique
// payload — never on a transmit event. Catches §7 BUG 2 being reintroduced by
// someone wiring the "throughput" counter to the convenient signal.
a_useful_only_on_retire: assert property (
  @(posedge clk) disable iff (!rst_n)
    ($changed(useful_bytes_q)) |-> $past(retire_fire)
);
 
// MANDATORY. English: tags in use never exceed the tag count. Catches the
// allocation gate being bypassed, which would make the high-water mark — the
// evidence for factor 4 — meaningless.
a_tags_bounded: assert property (
  @(posedge clk) disable iff (!rst_n)
    tags_in_use_q <= 8'(N_TAG)
);

Reading the three.

The first is the load-bearing one and it is an identity rather than a temporal property. If transmitted payload does not equal retired plus retried, the two are not being separated — and factor 3 cannot be computed. It is the bandwidth analogue of 29.1 §9's sum identity, applied to a different quantity.

The second uses $changed with $past(retire_fire) because the wrong event is a transmit. It fires the first time someone connects the useful counter to tlp_fire, which is precisely §7 BUG 2.

And the third protects factor 4's evidence. tags_high_water_q is the one-read test for whether the window binds; if allocation can exceed the tag count, the high-water mark is not a measurement of anything.

9. Measured Behaviour — the Payload Sweep

Three readings.

Factor 4 binds for every payload up to 256 bytes and then stops binding entirely. That discontinuity is the whole engineering content: below the crossover the device's tag count is the answer to every performance question; above it, the tag count is irrelevant. A benchmark run at one payload size cannot see the crossover.

And the crossover is computable rather than empirical. Factor 4 stops binding when N_TAG × payload ÷ round_trip ≥ ceiling × F2 × F3. Solving for payload at 64 tags gives ≈ 445 bytes — so the transition between the two regimes sits between the 256 B and 512 B rows, exactly where the table shows it.

The 64-byte row is worth sitting with: 12.8 % overall. Headers cost 27 %, and the window costs another 82 % of what remains. Reporting this as "PCIe is inefficient for small transfers" is true and misattributed — most of the loss is the device's tag count, not the protocol's headers.

10. Executable Counterexamples

Case 3 is the one that changes decisions. Both instruments report a poor number; only §6 says which of the four owners should be in the room.

11. Verification

Two readings.

The two negative cases are the verification argument. One proves the instrument responds to link degradation; the other proves the factors are independent — that changing the window moves factor 4 and leaves factors 2 and 3 alone. An instrument whose factors move together is not decomposing anything.

And the coverage requirement is unusually specific. Bins must span the crossover (§9), because a suite that only exercises 512 B payloads never sees factor 4 bind, and a suite that only exercises 64 B never sees it stop.

12. Debugging — Which Owner Should Be in the Room

Three readings.

Row 1 and row 2 look identical in a single efficiency number and have completely different owners. That is the practical reason this chapter exists: the decomposition routes the problem to the right team, and a one-number report routes it to whoever is loudest.

Row 4 is the case the cascade makes visible by elimination. If every factor is close to one and throughput is still low, nothing on this path is the constraint — and that is a real and common finding that a single ratio can never produce.

And the minimum discriminating instrument is tags_high_water_q alone. One register, read once: at the limit, it is the window; well below it, it is not. Everything else in §6 quantifies; that one register decides.

13. Misconceptions

"Efficiency is one number." §2, §5: it is a product of four factors with four different owners, and the product cannot be decomposed after the fact.

"We're at 51 % of link, so headers and signal integrity need work." §5: factor 4 alone costs 42 % in that example. Headers and retry together cost 11 %.

"Throughput counters are simple." §7: counting wire bytes includes headers and retransmissions, so the figure does not fall when the link degrades — and in a worse case it rises.

"A wider link fixes a small-transfer shortfall." §9: at 64 B the window supports 4.1 GB/s regardless of width, because width does not shorten the round trip (6.8).

"Bigger payloads help because of header overhead." §5: they help twice — factor 2 and factor 4 — and the factor 4 effect is the larger one below the crossover.

"Measure at one payload size and scale." §9: the binding factor changes across the sweep. A benchmark at 512 B and one at 64 B are measuring different bottlenecks in the same hardware (22.6).

"Compare measured useful throughput against the raw signalling rate." §4: that double-counts encoding. Compare against the post-encoding ceiling.

14. Understanding Check

Q1. A x16 transfer measures 16.4 GB/s against a 32.0 GB/s ceiling. Where did the other half go?

Decompose it — the losses multiply (§5). With 256 B payloads, 24 B overhead, 3 % retry and 64 tags at a 1.0 µs round trip: factor 2 (header) = 0.914, factor 3 (retry) = 0.971, factor 4 (window) = 0.577. Product 0.512. Factor 4 alone costs 42 % and the other two together cost 11 % — so a report of "51 % of link" invites exactly the wrong work. The fix is arithmetic: to reach the 28.4 GB/s the earlier factors permit, the device needs 28.4e9 × 1.0e-6 ÷ 256 ≈ 111 tags, not 64. And the one-register test is tags_high_water — pinned at the tag count means the window binds.

Q2. Why can't a single useful-over-ceiling ratio tell you what to fix?

Because a product does not decompose (§2). 0.512 is consistent with many different factor combinations, and the four factors have four different owners (§3): encoding is fixed, headers belong to payload size, retry belongs to signal integrity, and the window belongs to the device's RTL. Rows 1 and 2 of §12 produce identical single-number reports and require different teams. So each factor must be measured at its own boundary, with its own numerator and denominator (§6) — and the hardware should collect those and divide nothing, keeping the arithmetic auditable in software.

Q3. A throughput counter reports a steady 56 % while users report degradation. What is wrong?

It is counting wire bytes, including retransmissions (§7 BUG 2). Retried TLPs occupy the wire, so wire occupancy stays flat while useful throughput falls — §7's timeline shows the actual rate dropping 16.4 → 13.1 GB/s as retry rises to 12 % with the reported figure unchanged at 56 %. In a worse case the metric would rise as the link degraded. A second, opposite error compounds it: including header bytes inflates the baseline by about 9 % at 256 B payloads, so a team that corrects for headers alone gets a number that is right at 0 % retry and progressively wrong afterwards. The discriminator is one comparison — useful_bytes at retirement versus tlp_payload_bytes at transmit — and the widening gap between them is the retry rate. The assertion that prevents the regression is a_payload_accounting_closes.

Q4. Design the coverage that would have caught this class of error.

Bins spanning the crossover, plus two independence tests (§9, §11). Factor 4 stops binding when N_TAG × payload ÷ round_trip ≥ ceiling × F2 × F3, which at 64 tags is ≈ 445 bytes — so payload bins must sit on both sides of that, because a suite at 512 B never sees the window bind and a suite at 64 B never sees it stop. Then two negative tests. Inject 12 % retransmission and assert the reported useful rate falls — that proves the instrument responds to degradation. And halve the tag count and assert factor 4 moves while factors 2 and 3 do not — that proves the factors are actually independent, which is the property the whole decomposition rests on. Add tags_high_water at the limit as a coverage point, because it is the evidence factor 4's attribution depends on.

15. What Comes Next

This chapter accounted for a transfer that the host initiated and the device pulled. The next one removes the transfer entirely.

29.3 traces an FPGA card streaming — a continuous flow with no per-job descriptor, where the interesting state is not a transfer's accounting but the back-pressure path from the accelerator into the DMA engine, and where the failure is not a shortfall but a stall that propagates the wrong way.