Performance Implications — Why a Fast Link Can Still Be Slow

Module 12 has been about correctness. Every chapter asked what must happen, and answered it with ownership boundaries and invariants.

A design can satisfy every one of those invariants and still deliver a fraction of the bandwidth the link is capable of. That is not a contradiction, and it is not a rare pathology — it is the ordinary outcome of building a correct read engine without thinking about how much work it keeps in flight.

Why can a PCIe link with enormous raw bandwidth still deliver poor application throughput, and how do latency, payload size, outstanding depth, posted/non-posted semantics, and buffering interact?

1. Latency and Throughput Are Different Quantities

They are measured in different units, they are limited by different things, and improving one can worsen the other.

All four combinations exist and three of them are common.

And a fifth case that is not in the table because it is not about the design at all: a high-bandwidth link with poor transaction throughput. The link is idle most of the time. Nothing is broken. The design simply never has enough outstanding work to fill it.

2. What Is Verified, What Is Derived

3. Writes: Bounded by the Forward Path

A posted write needs no round trip (Chapter 12.2), so nothing about its performance depends on how long a response takes. That removes the round-trip term entirely — and leaves five others.

4. Reads: the Round Trip Is the Whole Story

A Memory Read is non-posted. The requested data does not exist at the Requester until a Completion returns, and that takes a round trip through the fabric and the target.

Request launched
   → fabric transit
   → target access
   → Completion generation and queueing        (Chapter 12.3 section 3)
   → fabric transit
   → correlation and progress accounting
   → data available

Call the whole of that RTT. With exactly one read outstanding, the Requester does this, then does it again, then does it again. Between the Request leaving and its Completion arriving, it issues nothing.

read throughput ≈ bytes returned per read ÷ RTT

This is derived arithmetic, not a PCIe formula, and it is the single most useful relation in the chapter.

5. Little's Law, Correctly Labelled

6. Worked Example — One Outstanding Read

Stated assumptions, and they are doing a lot of work.

throughput = 256 bytes / 1 µs
           = 256 bytes / 0.000001 s
           = 256,000,000 B/s
           = 256 MB/s          (decimal MB = 10^6 bytes)

On a link capable of several GB/s, that is a small fraction of capacity — and nothing is broken. Every packet is well-formed, every Completion correlates, the design passes every property in Module 12.

7. Worked Example — Eight Outstanding Reads

Same assumptions, one change: eight reads outstanding.

bytes in flight = 8 × 256 B = 2048 B
 
throughput = 2048 bytes / 1 µs
           = 2,048,000,000 B/s
           = 2.048 GB/s        (decimal GB = 10^9 bytes)

Eight times the concurrency, eight times the throughput — up to a point.

And the same table at 2 µs RTT — doubling the latency halves every row:

Read the two tables together and the design rule falls out: to hold bandwidth constant when latency doubles, you must double the work in flight.

Turning it around

The useful direction in practice is solving for what you need:

required bytes in flight = target bandwidth × RTT

The last row is the one that changes designs. Sustaining 16 GB/s of reads across a 2 µs path needs on the order of 125 concurrent 256-byte reads — which is a statement about the outstanding-transaction table, the Tag space, and the return buffer, all at once. A design with 16 contexts cannot get there, and no amount of link upgrade will help it.

8. Outstanding Count and Outstanding Bytes Are Different

A table with N entries bounds the number of reads. It does not bound the amount of data in flight, and the two dimensions cost different resources.

Two configurations with the same throughput and completely different resource profiles:

Same idealised bandwidth. Four times the tracking state in A; four times the burst size in B.

Which is better depends on what is scarce. If contexts are cheap and buffering is not, prefer A. If the local consumer handles large contiguous blocks well and correlation state is expensive, prefer B. What is never right is reasoning about only one of the two numbers — a design review that discusses "how many outstanding reads" without asking "of what size" has specified half the requirement.

And note the interaction with Chapter 12.3: larger reads are answered by more Completions each, so B's eight reads may produce as many Completion packets as A's thirty-two. The count of packets on the return path is a third dimension, and it is bounded by MPS rather than by either column above.

9. MPS and MRRS — Different Controls, Different Effects

The distinction is Chapter 11.4 §6's and Chapter 12.1 §8's. Here is what each one does to performance.

10. Packet Overhead, Carefully

Useful payload shares the link with headers and with lower-layer framing and protocol overhead. So for bulk transfer, larger payloads generally improve payload efficiency — the per-packet cost amortises over more bytes.

11. Resource Ceilings

Three buffers, three different ways to become the constraint.

The outstanding-transaction table

CTXS entries means at most CTXS unresolved reads (Chapter 12.1 §7). When it is full, loc_ready drops and no new read launches — even if the link is completely idle.

This is the ceiling §7's table quantifies. A design targeting 8 GB/s across a 2 µs path with 256 B reads needs on the order of 63 contexts; with 16, it is capped near 2 GB/s regardless of anything else.

The return buffer

Completion data arrives at whatever rate the return path delivers it. The local client consumes at its own rate, and the buffer absorbs the difference (Chapter 12.1 §12).

When it is full, backpressure eventually propagates, and effective throughput falls to the consumer's rate. Note carefully what this does not mean: the local application's ready is not asserted across PCIe. The remote transmitter never sees it. The pressure travels through local buffering and then through protocol-level mechanisms whose details belong to the flow-control chapters.

The posted write buffer

queue growth ≈ accepted rate − drained rate

A shallow queue is fine while those rates match and fails the moment they do not. Bursty producers are exactly the case where they do not match, which is why §14's third scenario is burst-specific: average throughput looks correct and the peaks are clipped.

12. Head-of-Line Blocking

A shared queue lets one stalled item block unrelated work behind it.

If reads to a slow target and reads to a fast target share one request queue, a burst to the slow one occupies the head and everything behind it waits — including work that could have completed immediately.

Mitigations exist and none of them is mandated by PCIe:

This is implementation architecture, not protocol behaviour. PCIe defines what packets mean; queue organisation is a design decision, and it is one of the larger performance levers available. A design that measures poor throughput with an under-full outstanding table and an idle link should look here before looking at the link.

13. RTL — Outstanding Occupancy and High-Water Mark

// SYNTHESIZABLE. Live occupancy telemetry for an outstanding-transaction
// table, with a high-water mark and a table-full cycle counter.
// This is IMPLEMENTATION TELEMETRY — PCIe defines no such mechanism.
// The counter widths and clear semantics are ILLUSTRATIVE.
module outstanding_occupancy #(
  parameter int CAPACITY = 32,        // table entries
  parameter int CYC_W    = 32         // full-cycle counter width
) (
  input  logic clk,
  input  logic rst_n,
 
  // One allocate and one free event per cycle, at most.
  input  logic allocate,
  input  logic free,
 
  // Synchronous clear for the sticky telemetry. Does NOT clear occupancy,
  // which is live state and is not the software's to reset.
  input  logic clear_stats,
 
  // OCC_W holds CAPACITY itself, not CAPACITY-1: a full table must be
  // representable. This is the width bug that makes a full table read as
  // empty.
  output logic [$clog2(CAPACITY+1)-1:0] occupancy,
  output logic [$clog2(CAPACITY+1)-1:0] high_water,
  output logic [CYC_W-1:0]              full_cycles,
  output logic                          occ_error
);
 
  localparam int OCC_W = $clog2(CAPACITY + 1);
 
  generate
    if (CAPACITY < 1) $error("CAPACITY must be at least 1");
  endgenerate
 
  logic [OCC_W-1:0] occ_q, hw_q;
  logic [CYC_W-1:0] full_q;
  logic             err_q;
 
  wire at_capacity = (occ_q == OCC_W'(CAPACITY));
 
  // Guard both directions. A live gauge must never wrap: an underflowed
  // occupancy reads as an enormous number and an overflowed one reads as a
  // small one, and both look like plausible telemetry.
  wire underflow = free     && !allocate && (occ_q == '0);
  wire overflow  = allocate && !free     && at_capacity;
 
  assign occupancy   = occ_q;
  assign high_water  = hw_q;
  assign full_cycles = full_q;
  assign occ_error   = err_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      occ_q <= '0; hw_q <= '0; full_q <= '0; err_q <= 1'b0;
    end else begin
      // Four explicit arms. Simultaneous allocate and free leaves occupancy
      // UNCHANGED — written as its own case rather than falling out of
      // (occ + allocate - free), so the same-cycle event is visible in the
      // source and cannot transit an illegal intermediate value.
      unique case ({allocate, free})
        2'b10:   occ_q <= overflow  ? occ_q : (occ_q + OCC_W'(1));
        2'b01:   occ_q <= underflow ? occ_q : (occ_q - OCC_W'(1));
        default: occ_q <= occ_q;          // 2'b11 and 2'b00
      endcase
 
      if (underflow || overflow) err_q <= 1'b1;
 
      // High-water tracks the LIVE gauge, so it is meaningful. Deriving a
      // peak from cumulative totals would be meaningless — that is the
      // mistake Chapter 10.5 corrects, and it is the same mistake here.
      if (clear_stats)
        hw_q <= occ_q;                    // restart from the present, not 0
      else if (occ_q > hw_q)
        hw_q <= occ_q;
 
      // Cycles spent unable to accept new work. THE most actionable single
      // number in this module (section 16).
      if (clear_stats)                    full_q <= '0;
      else if (at_capacity && (&full_q) == 1'b0) full_q <= full_q + CYC_W'(1);
    end
  end
 
endmodule

Classification: synthesizable.

Architecture. One live gauge, one peak, one saturating cycle counter. The gauge is maintained directly from allocate/free events — never derived by subtracting cumulative totals, which is the mistake Chapter 10.5 corrects and which becomes meaningless the moment either total saturates or wraps.

State. occ_q (live), hw_q (peak), full_q (saturating), a sticky error flag.

Cycle behaviour. {allocate, free} = 2'b11 leaves occupancy unchanged; 2'b10 increments unless at capacity; 2'b01 decrements unless empty. clear_stats restarts the peak from the current occupancy, not from zero — a peak reset to zero would under-report until the next rise.

Contract. occupancy is a live gauge; full_cycles is cumulative and saturating. Software must not subtract cumulative counters to infer live state.

Failure — four. Sizing the counter $clog2(CAPACITY) instead of $clog2(CAPACITY+1) cannot represent a full table, so full reads as empty. occ + allocate - free transits an intermediate value and can underflow on a simultaneous event at zero. Clearing the high-water mark to zero under-reports until occupancy next rises. And letting full_q wrap turns a long full episode into a small number — which is why it saturates, at the documented cost that the total is no longer exact.

Deliberately simplified: one allocate and one free per cycle; no per-class breakdown; no time-windowed rates.

DV. §17's P1–P6.

14. RTL — Throughput Counters

// SYNTHESIZABLE. Cumulative event and byte counters for read traffic.
// IMPLEMENTATION TELEMETRY. Rates are derived in software or the testbench
// from these integers and a known clock period — there is deliberately no
// division here (section 15).
module read_throughput_counters #(
  parameter int CNT_W = 40,           // wide enough for long runs
  parameter int LEN_W = 11
) (
  input  logic clk,
  input  logic rst_n,
  input  logic clear_stats,
 
  input  logic               req_launched,
  input  logic [LEN_W-1:0]   req_dw,          // DW requested by this read
  input  logic               chunk_credited,
  input  logic [LEN_W-1:0]   chunk_dw,        // DW returned in this chunk
  input  logic               read_resolved,
 
  output logic [CNT_W-1:0]   reads_launched,
  output logic [CNT_W-1:0]   reads_resolved,
  output logic [CNT_W-1:0]   dw_requested,
  output logic [CNT_W-1:0]   dw_returned,
  output logic [CNT_W-1:0]   elapsed_cycles,
  // Any counter reached its maximum. Once set, the totals are no longer
  // exact and MUST NOT be used for arithmetic that assumes they are.
  output logic               counters_saturated
);
 
  logic [CNT_W-1:0] rl_q, rr_q, dq_q, dr_q, cy_q;
  logic             sat_q;
 
  // Saturating helper. Chosen over wrapping so that a long run reports an
  // implausibly large number rather than a plausibly small one — a wrapped
  // counter looks like correct telemetry and is the harder failure.
  function automatic logic [CNT_W-1:0] add_sat
      (input logic [CNT_W-1:0] a, input logic [CNT_W-1:0] b);
    logic [CNT_W:0] w;
    w = {1'b0, a} + {1'b0, b};
    add_sat = w[CNT_W] ? {CNT_W{1'b1}} : w[CNT_W-1:0];
  endfunction
 
  function automatic logic is_max (input logic [CNT_W-1:0] a);
    is_max = &a;
  endfunction
 
  assign reads_launched     = rl_q;
  assign reads_resolved     = rr_q;
  assign dw_requested       = dq_q;
  assign dw_returned        = dr_q;
  assign elapsed_cycles     = cy_q;
  assign counters_saturated = sat_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      rl_q <= '0; rr_q <= '0; dq_q <= '0; dr_q <= '0; cy_q <= '0;
      sat_q <= 1'b0;
    end else if (clear_stats) begin
      rl_q <= '0; rr_q <= '0; dq_q <= '0; dr_q <= '0; cy_q <= '0;
      sat_q <= 1'b0;
    end else begin
      cy_q <= add_sat(cy_q, CNT_W'(1));
 
      if (req_launched) begin
        rl_q <= add_sat(rl_q, CNT_W'(1));
        dq_q <= add_sat(dq_q, CNT_W'(req_dw));
      end
      // Counted on the CREDITED event, not on arrival: a chunk that was
      // rejected as unknown or overrun returned no useful bytes, and
      // counting it would overstate delivered work.
      if (chunk_credited)
        dr_q <= add_sat(dr_q, CNT_W'(chunk_dw));
 
      if (read_resolved)
        rr_q <= add_sat(rr_q, CNT_W'(1));
 
      if (is_max(cy_q) || is_max(rl_q) || is_max(rr_q)
                       || is_max(dq_q) || is_max(dr_q))
        sat_q <= 1'b1;
    end
  end
 
endmodule

Classification: synthesizable.

Architecture. Five saturating cumulative counters plus a saturation flag. No division and no rate computation — §15 explains why that belongs in software.

Contract. All five are cumulative totals, not gauges. counters_saturated means the totals are no longer exact, and any consumer must check it before doing arithmetic with them.

Failure — three. Counting returned bytes on chunk arrival rather than on chunk_credited counts rejected chunks as delivered work, overstating throughput exactly when something is going wrong. Wrapping instead of saturating turns a long run into a plausible small number. And subtracting reads_launched − reads_resolved to get live occupancy is the Chapter 10.5 mistake: once either counter saturates the difference is meaningless, and §13's gauge exists so nobody has to.

Deliberately simplified: reads only; no per-context breakdown; no windowed rates.

15. Verification-Only — Latency Instrumentation and the Rate Calculator

// VERIFICATION-ONLY. Per-read latency sampling and rate derivation.
// NOT synthesizable as written, and NOT a PCIe mechanism. A production
// design that wants this implements a small timestamp table; the point here
// is the measurement discipline, not the implementation.
module read_latency_probe #(
  parameter int CTXS  = 32,
  parameter int BINS  = 8,
  parameter int TIME_W = 32
);
 
  int unsigned launch_time [CTXS];
  bit          active      [CTXS];
  int unsigned bin         [BINS];      // histogram of cycles outstanding
  int unsigned max_latency;
  int unsigned samples;
  longint unsigned latency_sum;
 
  int unsigned now;
 
  task automatic tick; now++; endtask
 
  task automatic on_launch (input int ctx);
    if ((ctx < 0) || (ctx >= CTXS)) begin
      $error("launch with out-of-range ctx %0d", ctx);   // Chapter 12.3 s8a
      return;
    end
    if (active[ctx]) $error("launch on already-active ctx %0d", ctx);
    launch_time[ctx] = now;
    active[ctx]      = 1;
  endtask
 
  task automatic on_resolve (input int ctx);
    int unsigned lat;
    int unsigned b;
    if ((ctx < 0) || (ctx >= CTXS)) begin
      $error("resolve with out-of-range ctx %0d", ctx);
      return;
    end
    if (!active[ctx]) begin
      $error("resolve on inactive ctx %0d", ctx);
      return;
    end
    lat = now - launch_time[ctx];
    active[ctx] = 0;
 
    // Log2 binning: the interesting structure in a latency distribution is
    // usually its tail, and linear bins bury the tail in the last bucket.
    b = 0;
    while ((b < BINS-1) && (lat >= (1 << (b+1)))) b++;
    bin[b]++;
 
    if (lat > max_latency) max_latency = lat;
    latency_sum += lat;
    samples++;
  endtask
 
  function automatic int unsigned mean_latency;
    mean_latency = (samples == 0) ? 0 : int'(latency_sum / samples);
  endfunction
 
endmodule

// COMPILE-TIME / VERIFICATION-ONLY. Rate derivation with EXPLICIT units.
// Kept out of the synthesizable counters deliberately: division belongs
// where the clock period is known and where a wrong unit is a compile-time
// argument rather than a silently wrong register.
package perf_calc_pkg;
 
  // All inputs integer; all units named in the argument list.
  // bytes, cycles, and picoseconds-per-cycle in; bytes-per-second out.
  function automatic longint unsigned bytes_per_second
      (input longint unsigned bytes,
       input longint unsigned cycles,
       input longint unsigned ps_per_cycle);
    longint unsigned elapsed_ps;
    if ((cycles == 0) || (ps_per_cycle == 0)) return 0;
    elapsed_ps = cycles * ps_per_cycle;
    // bytes / (elapsed_ps * 1e-12 s) = bytes * 1e12 / elapsed_ps
    return (bytes * 1_000_000_000_000) / elapsed_ps;
  endfunction
 
  // The design-time question of section 7, in one function.
  // target_MBps is DECIMAL megabytes per second (10^6 B/s).
  // rtt_ns in nanoseconds. Result in bytes.
  function automatic longint unsigned required_bytes_in_flight
      (input longint unsigned target_MBps,
       input longint unsigned rtt_ns);
    // (target_MBps * 1e6 B/s) * (rtt_ns * 1e-9 s) = target_MBps * rtt_ns / 1000
    return (target_MBps * rtt_ns) / 1000;
  endfunction
 
endpackage

Classification: verification-only (probe) and compile-time (package).

Why the division is here and not in §14. A synthesizable divider for a rate nobody reads every cycle is wasted area, and a rate computed in hardware bakes in a clock period the RTL does not actually know. Keeping the counters as integers and deriving the rate where the period is known is both cheaper and harder to get wrong.

Why the units are in the argument names. bytes_per_second(bytes, cycles, ps_per_cycle) cannot be called with a nanosecond period without someone noticing. A function taking three bare integers can, and a factor-of-1000 error in a bandwidth report is indistinguishable from a real result.

Sanity check against §7: required_bytes_in_flight(8000, 2000) = 8000 × 2000 / 1000 = 16,000 bytes — matching the 8 GB/s at 2 µs row exactly.

Why log2 histogram bins. A latency distribution's useful information is in its tail. Linear bins put every outlier in the last bucket and report a shape you already assumed; log2 bins show whether the tail is one slow target, an occasional retry, or a queue that occasionally saturates.

16. Assertions

// SVA over outstanding_occupancy and read_throughput_counters. These are
// TELEMETRY CORRECTNESS properties — they prove the instrumentation reports
// the truth. They do NOT assert that any performance target is met, which
// is a workload question and not a design contract.
 
// GAUGE — P1: occupancy never exceeds capacity and never underflows.
property p_occupancy_bounded;
  @(posedge clk) disable iff (!rst_n)
  (occupancy <= OCC_W'(CAPACITY));
endproperty
a_occ_bounded : assert property (p_occupancy_bounded);
 
property p_no_underflow;
  @(posedge clk) disable iff (!rst_n)
  (free && !allocate && (occupancy == '0)) |=> (occupancy == '0) && occ_error;
endproperty
a_no_underflow : assert property (p_no_underflow);
 
// GAUGE — P2: SIMULTANEOUS ALLOCATE AND FREE PRESERVES OCCUPANCY. The
// same-cycle case, stated as its own property because it is the one an
// (occ + allocate - free) implementation gets right by accident and an
// if/else-if implementation gets wrong on purpose.
property p_simul_preserves;
  @(posedge clk) disable iff (!rst_n)
  (allocate && free) |=> (occupancy == $past(occupancy));
endproperty
a_simul : assert property (p_simul_preserves);
 
// GAUGE — P3: occupancy changes by at most one, and only on an event.
property p_occupancy_step;
  @(posedge clk) disable iff (!rst_n)
  !$stable(occupancy)
    |-> ($past(allocate ^ free)
      && (occupancy == $past(occupancy) + OCC_W'(1)
       || occupancy == $past(occupancy) - OCC_W'(1)));
endproperty
a_occ_step : assert property (p_occupancy_step);
 
// WATERMARK — P4: the high-water mark is always at least the live gauge.
property p_hw_ge_occupancy;
  @(posedge clk) disable iff (!rst_n)
  (high_water >= occupancy);
endproperty
a_hw_ge : assert property (p_hw_ge_occupancy);
 
// WATERMARK — P5: the high-water mark never decreases except on an explicit
// clear. A peak that can fall is not a peak.
property p_hw_monotonic;
  @(posedge clk) disable iff (!rst_n)
  (high_water < $past(high_water)) |-> $past(clear_stats);
endproperty
a_hw_mono : assert property (p_hw_monotonic);
 
// FULL COUNTER — P6: full_cycles increments only when the table is
// genuinely at capacity. A counter that counted near-full episodes would
// systematically overstate the constraint.
property p_full_only_when_full;
  @(posedge clk) disable iff (!rst_n)
  (full_cycles > $past(full_cycles))
    |-> ($past(occupancy) == OCC_W'(CAPACITY));
endproperty
a_full_honest : assert property (p_full_only_when_full);
 
// COUNTERS — P7: cumulative counters never decrease except on clear.
property p_counters_monotonic;
  @(posedge clk) disable iff (!rst_n)
  ((dw_returned < $past(dw_returned)) || (reads_launched < $past(reads_launched)))
    |-> $past(clear_stats);
endproperty
a_cnt_mono : assert property (p_counters_monotonic);
 
// COUNTERS — P8: returned bytes are counted once per CREDITED chunk, never
// per arriving chunk. A rejected chunk delivered nothing.
property p_returned_counts_credited_only;
  @(posedge clk) disable iff (!rst_n)
  (dw_returned != $past(dw_returned))
    |-> ($past(chunk_credited)
      && (dw_returned == $past(dw_returned) + CNT_W'($past(chunk_dw))));
endproperty
a_credited_only : assert property (p_returned_counts_credited_only);
 
// COUNTERS — P9: resolved never exceeds launched. Catches a resolve event
// fabricated from something other than a real retirement.
property p_resolved_le_launched;
  @(posedge clk) disable iff (!rst_n)
  !counters_saturated |-> (reads_resolved <= reads_launched);
endproperty
a_resolved_le : assert property (p_resolved_le_launched);
 
// COUNTERS — P10: returned DW never exceeds requested DW while unsaturated.
property p_returned_le_requested;
  @(posedge clk) disable iff (!rst_n)
  !counters_saturated |-> (dw_returned <= dw_requested);
endproperty
a_returned_le : assert property (p_returned_le_requested);
 
// COUNTERS — P11: saturation is sticky and honest. Once any counter has
// pinned, the flag stays set — a flag that could clear itself would let a
// consumer trust inexact totals.
property p_saturation_sticky;
  @(posedge clk) disable iff (!rst_n)
  (counters_saturated && !clear_stats) |=> counters_saturated;
endproperty
a_sat_sticky : assert property (p_saturation_sticky);
 
// SAFETY — P12: telemetry is never unknown.
property p_telemetry_never_unknown;
  @(posedge clk) disable iff (!rst_n)
  !$isunknown({occupancy, high_water, full_cycles, dw_returned});
endproperty
a_no_x : assert property (p_telemetry_never_unknown);

Every property here proves the instrumentation tells the truth. None asserts a performance outcome, because performance is a property of a workload on a system, not a contract a module can satisfy. An assertion that "throughput exceeds X" would fail on a legitimately slow stimulus and pass on a fast one that is wrong.

P2 is the same-cycle property and it is the one worth writing out. An occ + allocate - free implementation satisfies it incidentally; an if (allocate) ... else if (free) ... implementation drops the free and slowly leaks occupancy upward until the table reads full forever. P2 fires on the first simultaneous event.

P9 and P10 are guarded on !counters_saturated deliberately. Once a counter pins, the inequality is no longer meaningful — and asserting it anyway would produce a failure that says nothing about the design. Guarding the property is the honest form, and it is why the saturation flag is an output rather than an internal detail.

P8 is the property that keeps the throughput number honest. Counting arriving chunks rather than credited ones inflates delivered bytes precisely when chunks are being rejected — so the telemetry looks best exactly when the design is failing.

17. Verification

Monitors observe: allocate and free events; launch, credit and resolve events with their DW; and every telemetry output.

The scoreboard maintains an independent event log and recomputes every counter from it. It must not read occ_q, hw_q or any DUT counter as its expected value — a mirrored scoreboard cannot detect a counting bug, which is the only kind of bug this module can have.

Occupancy and watermark

• One outstanding operation — allocate, free, verify the gauge goes 0 → 1 → 0.
• Fill to CAPACITY. Verify the gauge reaches capacity and does not exceed it (P1).
• Allocate at capacity. Verify no overflow and occ_error (P1).
• Free at zero. Verify no underflow and occ_error.
• Simultaneous allocate and free at every occupancy from 0 to CAPACITY (P2). Required at zero and at capacity specifically, where the naive implementations diverge.
• A long run of random allocate/free compared against an independent model every cycle.
• High-water: rise to a peak, drain, verify the peak holds (P5); then clear_stats and verify it restarts from the current occupancy, not from zero.
• CAPACITY = 1. Verify the gauge width and the full detection.

Full-cycle counter

• Drive the table full and hold for a known number of cycles. Verify full_cycles matches exactly (P6).
• Hover at CAPACITY − 1. Verify full_cycles does not increment — the test that catches a near-full counter.
• Alternate full and not-full. Verify the total is the sum of the full episodes.
• Saturate full_cycles. Verify it pins rather than wraps.

Throughput counters

• Launch and resolve a known number of reads with known DW. Verify all five counters against the independent log.
• A rejected chunk — unknown context or overrun. Verify dw_returned does not move (P8).
• A read resolved by several chunks. Verify returned DW accumulates once per credited chunk.
• Drive a counter to saturation. Verify it pins, counters_saturated sets and stays set (P11), and that P9/P10 stop being checked rather than failing.
• clear_stats mid-run. Verify all counters and the saturation flag reset and the gauge does not.

Scenario-level

• One outstanding read, long latency. Verify occupancy is 1 almost always and full_cycles is zero — the §18 starvation signature.
• Many outstanding reads, short latency. Verify high occupancy and non-zero full_cycles.
• Return burst — several chunks in quick succession. Verify byte counting under burst.
• Write burst against a shallow queue. Verify the write-side occupancy telemetry shows the fill.
• Local consumer stalled. Verify the outstanding table does not fill as a result (Chapter 12.1 §12) — a telemetry test of an architectural property.
• Reset mid-run with operations outstanding. Verify every counter and the gauge clear.

Coverage should include: occupancy at 0, 1, CAPACITY−1 and CAPACITY; simultaneous allocate/free at each of those; the high-water clear path; the saturation path on each counter; credited and rejected chunks; and CAPACITY = 1.

18. Performance Debugging

Four signatures, and the telemetry in §13–§14 distinguishes them in one reading.

Link idle, latency high, outstanding table nearly empty

Starvation. The Requester is not exposing enough parallelism.

The table is not full, so nothing is blocking new reads — the design simply is not issuing them. Look at the request generator: is it waiting for each read to complete before starting the next? Is a local dependency serialising work that has no real dependency? Is the client interface only capable of one operation at a time?

Compute §7's requirement and compare. If the target bandwidth needs 63 concurrent reads and the peak occupancy is 2, the answer is arithmetic rather than investigation.

Outstanding table always full, link still underutilised

The table is not the constraint — something behind it is. Reads are being issued as fast as contexts allow and they are not coming back.

Candidates, in order. Completion return bottleneck (Chapter 12.3 §3 — is the Completer's generation queue full?). Target latency higher than assumed. Congestion or a suboptimal path. Or reads that are not resolving at all — check whether reads_resolved is tracking reads_launched; a growing gap means reads are hanging, not merely slow.

The latency histogram (§15) separates the last two. A uniformly high distribution is a slow target; a bimodal one with a long tail is congestion or an occasional stall.

Posted write throughput collapses only during bursts

Forward queue depth versus drain rate (§11).

Average throughput is fine because the average accepted rate matches the average drain rate. Bursts exceed the queue's ability to absorb the difference, the producer stalls, and the peak is clipped.

The observation: correlate the write-queue occupancy trace against the producer's stall cycles. If the queue reaches full exactly when the producer stalls, the depth is the answer — and the required depth is roughly the burst size minus what drains during the burst.

Larger payloads raise bandwidth but latency spikes

This is a real trade, not a bug. Larger units of work amortise overhead and improve bulk throughput; they also increase the granularity of everything downstream. A large transfer occupies shared resources for longer, so an unrelated small operation arriving behind it waits longer — §12's head-of-line effect, arriving through configuration rather than through queue structure.

There is no universally correct answer. A bulk-transfer path wants large; a latency-sensitive control path wants small; a path carrying both wants separate queues rather than a compromise size.

19. Common Misconceptions

• "Link generation and lane count give you application bandwidth." They give a raw ceiling. What you get depends on latency, concurrency, payload size, buffering and the target (§1).
• "More outstanding requests always improves performance." Only until something else binds. Past the knee, deeper concurrency lengthens queues and raises latency without raising throughput (§7).
• "One outstanding read can saturate a fast link." With one read in flight, throughput is bytes-per-read ÷ RTT and is independent of link rate (§6).
• "Larger MPS always improves performance." It improves payload efficiency and coarsens granularity. It is also bounded by what every Function on the path supports (§9).
• "Larger MRRS always improves performance." It reduces request overhead and lengthens per-context occupancy. With a small context table it can reduce effective concurrency (§9).
• "Posted transactions have zero latency." They have no round trip the Requester waits on. The write still takes time to reach and update the target (§3).
• "Completion latency only matters to software." It sets the concurrency a hardware read engine must sustain to reach any given bandwidth (§5).
• "Queue depth affects correctness but not performance." It is often the binding constraint, and it is the one §18's third signature identifies.
• "Bandwidth, throughput and latency are the same thing." Bandwidth is a capacity; throughput is achieved work per time; latency is per-operation duration (§1).
• "High link utilisation proves the Endpoint is efficient." A link can be busy carrying headers for tiny payloads, or carrying retried work. Utilisation is not useful-work rate.
• "Little's Law is a PCIe rule." It is a general systems relation. PCIe defines packets and transactions, not queueing behaviour (§5).
• "More buffering always improves latency." More buffering absorbs bursts and improves throughput stability. It generally increases latency, because operations wait in it.

20. Understanding Check

21. Module 12 Complete

Five chapters have taken memory traffic from a single packet to a system.

The learner now has both lifetimes: the correctness lifetime of a transaction, from acceptance to retirement, and its performance lifetime, from issue to the concurrency required to keep a link busy.

Module 13 asks the question Module 12 kept deferring. Chapter 13.1 — Completion Types takes the Completion packet itself: why more than one form exists, and how the original Request determines whether the answer carries data or only status. Chapter 13.2 then decodes the outcome — successful, unsupported, retry, aborted — and 13.3 and 13.4 own split Completions and the ordering rules.

The idea to carry forward: bandwidth is what the link can carry; throughput is what you manage to give it — and for reads, that is set by latency and concurrency, not by the link at all.