Skip to content

PCIe · Module 22

Link Efficiency — Naming the Denominator Before Quoting the Number

Payload over what? The same workload is 94% or 80% efficient depending only on which transport the denominator includes — and replay, control traffic and idle time each belong to a different question.

"PCIe is about 95% efficient" is not a fact. It is a fragment of one.

Chapter 6.7 §6 named protocol overhead as its layer 6 and did not derive it. Chapter 12.5 §10 explained at length why it published no efficiency percentage. Chapter 22.4 §3 counted the packets and said nothing about what one costs.

All three deferred to this chapter, and this chapter's first job is to explain why they were right to be careful.

1. Sources, Scope, and a Deliberate Refusal

2. Payload Over What?

The question that makes the chapter tractable is not "how efficient is PCIe" but "efficient with respect to which resource".

Four denominators, four legitimate questions:

DenominatorThe question it answersWho asks it
payload + per-packet fixed costis my packet size sensible?the DMA architect
+ replayed packet bytesis my link clean?the SI / validation engineer
+ control trafficwhat is my protocol spending?the controller architect
+ all occupied link timewhat fraction of the link did my application get?the system owner

They are nested, so each is smaller than the last, and the difference between two adjacent rows is itself the useful number — it isolates one cost.

§13 Model 8 makes this concrete on one workload and finds 94.12% / 90.50% / 89.55% / 80.00%. The 3.62-point step from the first to the second is the replay cost. The 0.95-point step to the third is the control cost. The 9.55-point step to the fourth is everything else, which is mostly idle — and idle is 22.1's subject, not this chapter's (§8).

So the rule this chapter enforces everywhere: state the numerator, state the denominator, state the workload, state the direction. A number missing any of those cannot be compared with another number, which is Chapter 22.6's entire subject.

3. The Efficiency Stack

Each layer wraps the one above it, and each adds something that is not application payload.

LayerWhat it addsExpressible identically in every generation?
application payloadyes
transaction layerheader, and optionally a digestyes, with the header form named
data link layersequence and integrity information, DLLPsyes
physical layerframing, encoding, ordered setsno — see §9
occupied link timeidle, retraining, everything elseyes, but it is utilisation (§8)

The fourth row is why this chapter cannot publish one universal formula. Gen1–Gen5 apply a line code that expands every byte; Gen6 does not, and relocates the overhead into the FLIT (6.7 §2). A single "efficiency" number that silently spans that boundary is comparing two different quantities (§9).

4. Header Amortization

5. Four Denominators, One Workload

§13 Model 8's workload, stated completely so the numbers are reproducible: 1,000 accepted packets of 256-byte payload, per-packet fixed cost F = 16 B, 40 packets replayed, 3,000 bytes of control traffic, and 320,000 byte-times of occupied link.

Unique application payload delivered: 256,000 bytes. That number is the numerator in all four rows and it never changes.

ModelDenominator includesEfficiency
Apayload + per-packet fixed cost94.12%
BA + bytes spent retransmitting replayed packets90.50%
CB + control traffic89.55%
Dtotal occupied link byte-times80.00%

The differences are the diagnosis:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
A − B = 3.62 points   ← the replay tax
B − C = 0.95 points   ← the control-traffic tax
C − D = 9.55 points   ← idle and everything else (22.1's question)

Which is why §11's RTL keeps four independent counters rather than one efficiency register. A single ratio computed in hardware forecloses every one of these questions, and the four counters cost less than the divider would.

6. Replay Adds Transport and No Payload

7. Control Traffic Is Real and Workload-Dependent

Flow Control updates and acknowledgements travel as DLLPs (16.6 §1) and occupy link opportunities that carry no application payload.

But their share is not a constant, and any chapter quoting a fixed DLLP overhead percentage has invented it. §13 Model 10 shows why — the same absolute control traffic against different packet sizes:

Payload per packetTLP bytesControl bytesControl share
64 B80,0003,0003.61%
256 B272,0003,0001.09%
1024 B1,040,0003,0000.29%
4096 B4,112,0003,0000.07%

A 50× spread from packet size alone, with the control traffic held fixed. And in reality it is not fixed — update frequency depends on the receiver's policy and its drain rate (16.6 §8).

So the only defensible way to state control overhead is to measure it, which is what §11's control_byte_counter exists for. measured_control_bytes / total_transport_bytes is a number about your system; anything else is a number about someone's memory.

8. Idle Is Not Overhead

9. Gen6 Needs a Different Basis, Not a Different Constant

10. The Waveform

Same numerator, larger denominator — replay and control at work

10 cycles
Ten cycles at a transmit interface. In cycles 0 to 3 four TLPs transfer and each delivers new unique payload. A snapshot is taken at cycle 4. In cycles 5 to 8 a TLP delivers new payload at cycle 5, a replay transmits at cycle 6 delivering no new payload, control traffic occupies cycle 7, and a further TLP delivers new payload at cycle 8. A second snapshot is taken at cycle 9.window A — every byte was new payloadwindow A — every byte wasnew payloadreplay: transport up, payload flatreplay: transport up,payload flatcontrol traffic occupies the linkcontrol traffic occupiesthe linkclktlp_xferunique_payreplay_xferctrl_xfertransportsnapshott0t1t2t3t4t5t6t7t8t9
Figure 1 — two measurement windows carrying similar unique payload. In the first window every transmission delivers new application bytes. In the second, one transmission is a replay and another is control traffic, so transport accumulates while unique payload does not. The numerator is nearly unchanged between the windows and the denominator is not, which is what efficiency loss looks like at an interface.

Three things to read out of the figure.

transport is asserted in cycles 6 and 7 and unique_pay is not. That divergence is the efficiency loss, and it is visible only because the two are counted separately (§11).

Cycle 6 is the replay. A counter that increments unique_pay here reports the second window as being as efficient as the first — blind to exactly what it was installed to see (§6).

And the two snapshots must not be mixed. Cycle 4 closes window A and cycle 9 closes window B; a ratio built from window A's numerator and window B's denominator is meaningless, which is why §11 publishes all counters from one atomic snapshot and P13 asserts it.

11. RTL — Four Counters That Must Not Become One

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SYNTHESIZABLE. Efficiency measurement types.
// Four independent accumulators. The generation basis is a compile-time
// selection because Gen6 is NOT the Gen3-5 model with a different constant
// (§9) -- an unsupported combination must fail to elaborate, not silently
// compute the wrong thing.
package eff_pkg;
 
  parameter int CNT_W  = 48;
  parameter int BYTES_W = 13;
 
  typedef enum logic [1:0] {
    BASIS_LINECODE = 2'd0,   // Gen1-Gen5: a line code expands every byte
    BASIS_FLIT     = 2'd1,   // Gen6: no line code; overhead inside the FLIT
    BASIS_NONE     = 2'd2    // byte-domain only; no physical-layer claim
  } basis_e;
 
  function automatic logic [CNT_W-1:0] sat_add(input logic [CNT_W-1:0] v,
                                               input logic [CNT_W-1:0] inc);
    logic [CNT_W:0] wide;
    wide = {1'b0, v} + {1'b0, inc};
    return wide[CNT_W] ? {CNT_W{1'b1}} : wide[CNT_W-1:0];
  endfunction
 
  function automatic bit is_sat(input logic [CNT_W-1:0] v);
    return (v == {CNT_W{1'b1}});
  endfunction
 
endpackage
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import eff_pkg::*;
 
// SYNTHESIZABLE. UNIQUE application payload -- the NUMERATOR (§6).
// Increments on FIRST successful delivery only. A replay of the same
// packet must not increment it: §13 Model 9 shows a counter that does is
// blind to the exact condition it exists to detect.
module unique_payload_counter (
  input  logic clk,
  input  logic rst_n,
  input  logic               measuring,
  input  logic               tlp_accepted,     // handshake at the measurement point
  input  logic               is_replay,        // this transmission is a retransmit
  input  logic [BYTES_W-1:0] payload_bytes,
 
  output logic [CNT_W-1:0]   unique_payload,
  output logic               saturated,
  output logic               replay_seen       // sticky: qualifies the whole window
);
  logic [CNT_W-1:0] p_q;
  logic             rs_q;
 
  assign unique_payload = p_q;
  assign saturated      = is_sat(p_q);
  assign replay_seen    = rs_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin p_q <= '0; rs_q <= 1'b0; end
    else if (measuring && tlp_accepted) begin
      if (is_replay) rs_q <= 1'b1;                       // qualify, do not count
      else p_q <= sat_add(p_q, CNT_W'(payload_bytes));   // FIRST delivery only
    end
  end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import eff_pkg::*;
 
// SYNTHESIZABLE. Transport bytes -- the DENOMINATOR's core term (§5).
// Counts EVERY transmission, including replays. The measurement point is
// the local transmit interface, which sits ABOVE the physical layer: this
// is represented transport, NOT a wire capture (§1, mutation 16).
module transport_byte_counter (
  input  logic clk,
  input  logic rst_n,
  input  logic               measuring,
  input  logic               tx_valid,
  input  logic               tx_ready,
  input  logic [BYTES_W-1:0] tx_bytes,          // header + payload for this packet
 
  output logic [CNT_W-1:0]   transport_bytes,
  output logic               saturated
);
  logic [CNT_W-1:0] t_q;
  assign transport_bytes = t_q;
  assign saturated       = is_sat(t_q);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) t_q <= '0;
    // The HANDSHAKE, never valid alone -- Chapter 22.1 §7's rule, unchanged.
    else if (measuring && tx_valid && tx_ready) t_q <= sat_add(t_q, CNT_W'(tx_bytes));
  end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import eff_pkg::*;
 
// SYNTHESIZABLE. Replay bytes, counted SEPARATELY so the replay tax is
// isolable (§5: A - B is exactly this quantity). Diagnostic only -- the
// functional replay controller is Chapter 14.4's and is untouched here.
module replay_byte_counter (
  input  logic clk,
  input  logic rst_n,
  input  logic               measuring,
  input  logic               tx_valid,
  input  logic               tx_ready,
  input  logic               is_replay,
  input  logic [BYTES_W-1:0] tx_bytes,
 
  output logic [CNT_W-1:0]   replay_bytes,
  output logic [31:0]        replay_packets,
  output logic               saturated
);
  logic [CNT_W-1:0] r_q; logic [31:0] n_q;
  assign replay_bytes   = r_q;
  assign replay_packets = n_q;
  assign saturated      = is_sat(r_q);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin r_q <= '0; n_q <= '0; end
    else if (measuring && tx_valid && tx_ready && is_replay) begin
      r_q <= sat_add(r_q, CNT_W'(tx_bytes));
      if (!(&n_q)) n_q <= n_q + 32'd1;
    end
  end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import eff_pkg::*;
 
// SYNTHESIZABLE, CONDITIONAL ON INTERFACE VISIBILITY (§1, §7).
// Only instantiate where the controller exposes categorized control
// transmissions. Where it does not, the honest design has NO counter --
// claiming to measure invisible traffic is mutation 16.
module control_byte_counter #(parameter bit HAS_CONTROL_VISIBILITY = 1'b1) (
  input  logic clk,
  input  logic rst_n,
  input  logic               measuring,
  input  logic               ctrl_valid,
  input  logic               ctrl_ready,
  input  logic [BYTES_W-1:0] ctrl_bytes,
 
  output logic [CNT_W-1:0]   control_bytes,
  output logic               control_measured   // 0 => the field is UNKNOWN
);
  logic [CNT_W-1:0] c_q;
  assign control_bytes   = HAS_CONTROL_VISIBILITY ? c_q : '0;
  assign control_measured = HAS_CONTROL_VISIBILITY;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) c_q <= '0;
    else if (HAS_CONTROL_VISIBILITY && measuring && ctrl_valid && ctrl_ready)
      c_q <= sat_add(c_q, CNT_W'(ctrl_bytes));
  end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import eff_pkg::*;
 
// SYNTHESIZABLE. Atomic snapshot of the whole window (§5, §10).
// ALL counters are published from ONE cycle. A ratio assembled from two
// different windows is meaningless, and it is the easiest bug to commit
// when counters are read one register at a time.
module efficiency_snapshot #(parameter basis_e BASIS = BASIS_NONE) (
  input  logic clk,
  input  logic rst_n,
  input  logic window_close,
 
  input  logic [CNT_W-1:0] unique_payload,
  input  logic [CNT_W-1:0] transport_bytes,
  input  logic [CNT_W-1:0] replay_bytes,
  input  logic [CNT_W-1:0] control_bytes,
  input  logic [CNT_W-1:0] active_cycles,      // 22.1's denominator, kept apart (§8)
  input  logic             control_measured,
  input  logic             replay_seen,
  input  logic             any_saturated,
 
  output logic             snap_valid,
  output logic [CNT_W-1:0] snap_unique,
  output logic [CNT_W-1:0] snap_transport,
  output logic [CNT_W-1:0] snap_replay,
  output logic [CNT_W-1:0] snap_control,
  output logic [CNT_W-1:0] snap_active,
  output logic             snap_control_valid,
  output logic             snap_replay_seen,
  output logic             snap_saturated,
  output basis_e           snap_basis
);
  assign snap_basis = BASIS;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      snap_valid<=1'b0; snap_unique<='0; snap_transport<='0; snap_replay<='0;
      snap_control<='0; snap_active<='0; snap_control_valid<=1'b0;
      snap_replay_seen<=1'b0; snap_saturated<=1'b0;
    end else begin
      snap_valid <= window_close;
      if (window_close) begin
        // ONE cycle, ALL fields. This is the whole point of the block.
        snap_unique        <= unique_payload;
        snap_transport     <= transport_bytes;
        snap_replay        <= replay_bytes;
        snap_control       <= control_bytes;
        snap_active        <= active_cycles;
        snap_control_valid <= control_measured;
        snap_replay_seen   <= replay_seen;
        snap_saturated     <= any_saturated;
      end
    end
  end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import eff_pkg::*;
 
// VERIFICATION-ONLY. Sanity monitor for the ratios software will form.
// Payload can never exceed the transport that carried it, and replay
// bytes are a SUBSET of transport -- not an addition to it.
module efficiency_sanity_monitor (
  input logic [CNT_W-1:0] snap_unique,
  input logic [CNT_W-1:0] snap_transport,
  input logic [CNT_W-1:0] snap_replay,
  input logic             snap_saturated,
  output logic            err_payload_exceeds_transport,
  output logic            err_replay_exceeds_transport
);
  always_comb begin
    err_payload_exceeds_transport =
      !snap_saturated && (snap_unique > snap_transport);
    err_replay_exceeds_transport  =
      !snap_saturated && (snap_replay > snap_transport);
  end
endmodule

Classification: five synthesizable (one gated on declared interface visibility), one verification-only.

Failure — seven. Counting replayed payload as new (blind to replay). Counting transport on valid (inflated denominator under stall). One efficiency register instead of four counters (every question in §2 foreclosed). Mixing two windows in one ratio. Claiming to measure control traffic an interface does not expose. Wrapping counters. And applying the Gen1–Gen5 basis to Gen6 — which the basis_e parameter exists to make an elaboration-time decision rather than a silent one.

12. Same-Cycle Audit and Assertions

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ==================================================================
// NUMERATOR (§6) -- unique payload, once.
// ==================================================================
 
// P1: unique payload advances only on an accepted, NON-replay TLP.
property p_unique_on_first_delivery;
  @(posedge clk) disable iff (!rst_n)
    (unique_payload != $past(unique_payload)) |->
      $past(measuring && tlp_accepted && !is_replay);
endproperty
 
// P2: a replay NEVER advances unique payload. §13 Model 9: a counter that
// breaks this reports 94.12% at every replay rate.
property p_replay_adds_no_payload;
  @(posedge clk) disable iff (!rst_n)
    (measuring && tlp_accepted && is_replay) |=> $stable(unique_payload);
endproperty
 
// P3: the increment is exactly the payload accepted.
property p_unique_increment_exact;
  @(posedge clk) disable iff (!rst_n)
    (measuring && tlp_accepted && !is_replay && !saturated) |=>
      (unique_payload == $past(unique_payload) + CNT_W'($past(payload_bytes)));
endproperty
 
// P4: a replay in the window is REPORTED stickily -- the window is
// qualified even if the replay bytes are small.
property p_replay_seen_sticky;
  @(posedge clk) disable iff (!rst_n)
    replay_seen |=> replay_seen;
endproperty
 
// ==================================================================
// DENOMINATOR (§5) -- transport, on the handshake.
// ==================================================================
 
// P5: transport advances only on an accepted transmission.
property p_transport_on_handshake;
  @(posedge clk) disable iff (!rst_n)
    (transport_bytes != $past(transport_bytes)) |->
      $past(measuring && tx_valid && tx_ready);
endproperty
 
// P6: an offered but unaccepted packet adds nothing (Chapter 22.1 §7).
property p_no_transport_on_stall;
  @(posedge clk) disable iff (!rst_n)
    (tx_valid && !tx_ready) |=> $stable(transport_bytes);
endproperty
 
// P7: transport counts replays TOO -- that asymmetry against P2 is the
// entire mechanism by which replay shows up as efficiency loss.
property p_transport_includes_replay;
  @(posedge clk) disable iff (!rst_n)
    (measuring && tx_valid && tx_ready && is_replay && !saturated) |=>
      (transport_bytes != $past(transport_bytes));
endproperty
 
// P8: replay bytes are a SUBSET of transport, never an addition.
property p_replay_subset_of_transport;
  @(posedge clk) disable iff (!rst_n)
    (!saturated) |-> (replay_bytes <= transport_bytes);
endproperty
 
// P9: control bytes accumulate independently of TLP transport.
property p_control_independent;
  @(posedge clk) disable iff (!rst_n)
    (ctrl_valid && ctrl_ready && measuring && control_measured) |=>
      (control_bytes != $past(control_bytes));
endproperty
 
// ==================================================================
// COUNTER HYGIENE.
// ==================================================================
 
// P10: counters saturate, never wrap.
property p_no_wrap;
  @(posedge clk) disable iff (!rst_n)
    (transport_bytes == {CNT_W{1'b1}}) |=> (transport_bytes == {CNT_W{1'b1}});
endproperty
 
// P11: saturation invalidates the window explicitly, so software discards
// it rather than computing a ratio from a truncated denominator.
property p_saturation_flagged;
  @(posedge clk) disable iff (!rst_n)
    (snap_valid && $past(any_saturated)) |-> snap_saturated;
endproperty
 
// P12: payload can never exceed the transport that carried it. This is
// the cheapest possible check that the two counters share a window.
property p_payload_le_transport;
  @(posedge clk) disable iff (!rst_n)
    (snap_valid && !snap_saturated) |-> (snap_unique <= snap_transport);
endproperty
 
// ==================================================================
// SNAPSHOT ATOMICITY (§5) -- one window, one set of numbers.
// ==================================================================
 
// P13: every published field comes from the SAME cycle.
property p_snapshot_atomic;
  @(posedge clk) disable iff (!rst_n)
    snap_valid |-> ((snap_unique    == $past(unique_payload))
                 && (snap_transport == $past(transport_bytes))
                 && (snap_replay    == $past(replay_bytes))
                 && (snap_active    == $past(active_cycles)));
endproperty
 
// P14: exactly one snapshot per window close -- no double publication.
property p_one_snapshot_per_close;
  @(posedge clk) disable iff (!rst_n)
    snap_valid |-> $past(window_close);
endproperty
 
// P15: a boundary-cycle transfer is inside the CLOSING window (§12).
property p_boundary_transfer_in_closing_window;
  @(posedge clk) disable iff (!rst_n)
    (window_close && tx_valid && tx_ready) |=>
      (snap_transport >= CNT_W'($past(tx_bytes)));
endproperty
 
// P16: the unmeasured control field is marked UNKNOWN, not zero-as-fact.
// Reporting an unobservable quantity as 0 is mutation 16.
property p_control_validity_declared;
  @(posedge clk) disable iff (!rst_n)
    (snap_valid && !snap_control_valid) |-> (snap_control == '0);
endproperty
 
// P17: reset publishes nothing and clears every accumulator.
property p_reset_clears_window;
  @(posedge clk)
    (!rst_n) |=> (!snap_valid && (unique_payload == '0) && (transport_bytes == '0));
endproperty
 
// P18: a window with no traffic yields a zero denominator -- defined, and
// the consumer's problem, not a fabricated 100%.
property p_zero_traffic_defined;
  @(posedge clk) disable iff (!rst_n)
    (snap_valid && (snap_transport == '0)) |-> (snap_unique == '0);
endproperty
 
// P19: the generation basis travels WITH the snapshot, so a consumer
// cannot apply the Gen1-Gen5 model to a Gen6 window (§9).
property p_basis_accompanies_snapshot;
  @(posedge clk) disable iff (!rst_n)
    snap_valid |-> (snap_basis inside {BASIS_LINECODE, BASIS_FLIT, BASIS_NONE});
endproperty
 
// P20: the monitor never drives the transmit interface.
property p_monitor_does_not_drive;
  @(posedge clk) disable iff (!rst_n)
    $stable({tx_valid, tx_ready}) or !$stable({transport_bytes, unique_payload});
endproperty

Twenty properties. P1, P2 and P7 together are the chapter — the deliberate asymmetry between a numerator that ignores replay and a denominator that does not. P13–P16 exist because a ratio is only as trustworthy as the atomicity of the two numbers in it, and P16 is the one that keeps an honest design from reporting a quantity it cannot see.

13. Measured Behaviour

14. Verification — DV and Mutations

DV, against an independent byte-accounting model — never the DUT's own counters: a clean window with no replay · a window that is entirely replay · a single packet · a zero-payload packet · maximum payload · control-only traffic · a transfer exactly on the window boundary · two consecutive window closes · counter saturation · reset mid-window · HAS_CONTROL_VISIBILITY = 0 · a window with no traffic at all · payload deliberately forced above transport (must be rejected).

#MutationSymptomCaught by
1Count replayed payload as new unique payloadreports 94.12% at every replay rate; blind to replay (§13)P2
2Omit replay bytes from transportthe replay tax vanishes from the denominatorP7, P8
3Count transport on tx_valid alonedenominator inflated under stall; efficiency understatedP5, P6
4Count unique payload twice for one deliveryefficiency above 100%P3, P12
5Publish one efficiency register instead of four countersevery question in §2 forecloseddesign review
6Assemble a ratio from two different windowsnumerator and denominator describe different runsP13
7Publish two snapshots per window closesoftware double-counts a windowP14
8Credit the boundary-cycle transfer to the next windowevery window misreports its edgeP15
9Report unmeasured control bytes as 0an unknown presented as a measurement (§7)P16
10Claim wire-level measurement from an interface above the PHYa denominator that omits physical-layer transport (§1)design review
11Let counters wrapa busy window reports a small denominatorP10, P11
12Ignore saturation when publishinga truncated denominator inflates efficiencyP11
13Allow payload > transport silentlyan impossible ratio published as factP12
14Change the denominator basis between rows of a tablethe trend is an artifact of the basis (§5)design review
15Use one header size for every TLP3 DW and 4 DW forms differ (11.3 §2)design review
16Conflate the link-layer CRC with an end-to-end digestdifferent layers, different presence conditions (§1)design review
17Quote a fixed DLLP overhead percentagemeasured spread was 3.61% to 0.07% by packet size (§13)design review
18Apply × 128/130 to Gen6"wrong and confidently precise" (6.7 §2)P19
19Compare a Gen6 post-FLIT figure with a Gen5 post-encoding figuredifferent bases; not a valid ratio (§9)P19
20Sum TX and RX into a one-direction efficiencydoubles the numerator against a one-way denominatordesign review
21Report idle cycles as packet-format overheadsends the fix to packet sizing on a starved system (§8)design review
22Equate 100% link utilisation with 100% payload efficiency§13 Model 11 row 3: 100% busy at 80.00% efficientdesign review
23Divide when the window had no trafficdivision by zero, or a fabricated 100%P18
24Type the efficiency tables by handthe basis errors above are invisible without a scriptdesign review

Two counterexamples worth stating explicitly.

Mutation 1 is self-concealing, which makes it the worst kind. A unique_payload counter that increments on every accepted TLP — replay or not — inflates the numerator by exactly the same bytes it inflates the denominator with. The reported efficiency is therefore identical at 0% and 25% replay: 94.12% in both cases (§13 Model 9 computed the true values as 94.12% and 75.29%). The instrument was installed to detect link quality problems and it is precisely blind to them. P2 is the one-line property that separates the two counters' behaviour on a replay.

Mutation 6 needs no bad code at all. Software reads unique_payload, then reads transport_bytes a few microseconds later, after a window boundary has passed. Both registers are correct; the ratio is not, because it mixes one window's numerator with another's denominator. The defect is the absence of an atomic snapshot, and it appears in real systems as an efficiency figure that occasionally exceeds 100% for no reproducible reason. P13 asserts atomicity and P12 catches the symptom.

15. Debugging

Symptom — the link looks busy and the application receives little. Separate the two questions first (§8). Read active_cycles alongside the byte counters: high activity with a low payload-to-transport ratio is an efficiency problem (small packets, or replay); high ratio with low activity is a utilisation problem and belongs to 22.1 §6. They have no fix in common.

Symptom — efficiency is far below what the packet size predicts. Read replay_bytes and replay_packets (§11). Compare §4's curve for your payload size against §13 Model 9's replay column: if the shortfall matches the replay rate, this is a link-quality problem and belongs to 14.4, not to packet sizing.

Symptom — a tiny-TLP workload saturates the link and delivers almost nothing. Expected (§4). At F = 16, a 4-byte payload is 20% efficient; four fifths of the link carried no application data. The fix is batching in the producer, and 22.4 §3 shows what the packet count becomes.

Symptom — hardware counters and a benchmark disagree about efficiency. Almost always a denominator mismatch (§5). Ask which of Model A/B/C/D each side computed, and whether both used the same direction. A 14-point disagreement is exactly the A-to-D spread, so a discrepancy of that size is a strong hint that nobody is wrong and the bases differ — the reconciliation procedure is 22.6 §3's.

Symptom — a link analyzer reports more payload than the DMA engine did. The analyzer is counting replayed bytes as payload (§6) or measuring at a different point (22.1 §3). Reconcile by subtracting replay_bytes from the analyzer's total and checking whether the residual matches; if it does, the instruments agree and only their definitions differed.

Symptom — a Gen6 device looks less efficient than a Gen5 one. Check the bases before believing it (§9). A Gen6 post-FLIT figure has already paid overhead that a Gen5 post-encoding figure has not, so the comparison is invalid as posed (6.7 §2). Recompute both in the byte domain — payload over transport, which is generation-independent — and compare those.

Symptom — the efficiency figure occasionally exceeds 100%. Not a rounding artifact. Either the numerator counted a replay (mutation 1 combined with 2), or the ratio mixed two windows (mutation 6). P12 exists precisely so this is caught in simulation rather than in a report.

16. Misconceptions

"PCIe is about 95% efficient." With respect to which denominator? The same workload measured 94.12% and 80.00% in §13 (§2).

"Efficiency is a property of the generation." It is a property of the workload, the packet size, the link quality and the chosen denominator — the generation sets the capacity (6.7).

"Larger packets are always dramatically better." 64 B → 256 B buys 14 points; 1 KiB → 4 KiB buys 1.15 (§4). Most of the gain arrives early.

"Replayed bytes are throughput." They are transport that delivers no new payload (§6).

"A replay-heavy run just measures a bit lower." At 25% replay, efficiency fell 18.82 points while delivering identical data (§13).

"DLLP overhead is about N%." Measured spread was 3.61% to 0.07% by packet size alone (§7).

"Idle time is protocol overhead." It is utilisation loss, a different question with different fixes (§8).

"100% link utilisation means 100% efficiency." §13 Model 11 row 3: 100% busy at 80.00% efficient (§8).

"One header size is close enough." Header forms are 3 DW or 4 DW (11.3 §2), and the digest is conditional (§1).

"LCRC and ECRC are the same protection." Different layers, different scope, different presence conditions (§1, mutation 16).

"Gen6 just uses a different efficiency constant." Gen6 is 1b/1b at the line with overhead relocated into the FLIT — a different basis, not a different constant (6.7 §2, §9).

"Full duplex doubles efficiency." Efficiency is per direction, like capacity (6.7 §4).

"Hardware should just report an efficiency percentage." That forecloses every question in §2 and hides which cost dominated (§11).

17. Understanding Check

Q1. Two engineers measure the same run and report 94% and 80%. Who is wrong? Probably neither (§5). 94.12% is payload over payload-plus-per-packet-cost; 80.00% is payload over total occupied link time. The 14.12-point difference is itself the finding — it decomposes into the replay tax (3.62), the control tax (0.95) and idle (9.55). The correct response is to name both denominators, not to pick a winner.

Q2. Your efficiency counter reads 94.12% whether the replay rate is 0% or 25%. What is broken? The numerator counts replayed payload as new (mutation 1). It inflates numerator and denominator identically, so the ratio is invariant to the very condition it should expose. True values are 94.12% and 75.29% (§13 Model 9). P2 is the property, and the fix is that unique_payload advances only on a first delivery.

Q3. Raising the payload from 1 KiB to 4 KiB — how much efficiency do you gain, and is it the best available lever? About 1.15 points at F = 16 (98.46% → 99.61%, §4). Almost certainly not the best lever: the same table shows 64 B → 256 B buying 14 points, and §13 Model 9 shows a 5% replay rate costing 4.48. If replay is non-zero, link quality is worth more than the last doubling of MPS — and 22.4 §3 notes the granularity costs of the larger value.

Q4. Why does §11 keep four counters instead of computing one ratio in hardware? Because the differences between the nested denominators are the diagnosis (§5): A−B is the replay tax, B−C the control tax, C−D idle. One ratio destroys all three and cannot be recovered afterwards. The four counters are also cheaper than the divider they replace.

Q5. A window reports zero transport bytes. What should software display? Not 100%, and not 0%. The denominator is zero, so the ratio is undefined — P18 guarantees the numerator is zero too, so the honest display is "no traffic" (§12). A design that fabricates 100% here reports its best-ever efficiency during a completely idle window.

Q6. Why does this chapter refuse to publish a cross-generation efficiency table? Because Gen6 has no line encoding and relocates its overhead into the FLIT (6.7 §2), so a Gen6 "after overhead" figure and a Gen5 "after encoding" figure are not on the same basis (§9). A table spanning both would change denominator mid-row — mutation 14, which is the exact sin §2 exists to prevent. The generation-independent comparison is bytes of payload over bytes of transport, and §4 publishes that.

18. What's Next

Three chapters have now measured the same system three ways. 22.1 counted the cycles, 22.4 counted the packets, and this chapter priced them. Each insisted on the same discipline: name what you divided by.

Chapter 22.6 closes Module 22 by applying that discipline to other people's numbers. Everything here is a prerequisite for it — a benchmark that does not state its denominator, its direction, its transfer size and its error state is not a weaker measurement than yours; it is an uninterpretable one.

And a specific handoff. §5's four models are exactly the ambiguity that makes two honest benchmark results differ by 14 points with neither being wrong. 22.6 turns that observation into a procedure.