Latency — Naming Two Events, Then Accounting for the Time Between Them

Chapter 22.1 counted bytes per second. This chapter measures one unit of work from end to end, and the two are not reciprocals: a system can be starved of work (high no_work, poor throughput) while every individual transaction completes quickly.

"What is PCIe latency?" is not answerable as posed. Not because the number is unknown, but because the question omits both endpoints of the interval.

1. Sources, Scope, and What Is Inherited

2. Latency Is Not a Property of the Link

A Link has a signalling rate and a propagation delay. Neither is "the latency".

The quantity anyone actually cares about is a transaction latency — the time from wanting something to having it — and most of that time is typically spent in places that are not the Link at all: waiting behind other requests, being serviced by host memory, and being returned.

§13 Model 6 makes the point numerically. With every physical term held constant and only the queue depth varying, total latency ran 58 → 63 → 83 → 158 normalized units. The Link did not change. A "PCIe latency" number quoted without the load it was measured under describes one point on that curve and implies it is the curve.

3. Name the Two Events First

4. The Decomposition

One transaction, six terms — DERIVED, and stated in normalized time units because absolute values would need sources this chapter does not have (§1):

T_total = T_queue + T_serialize_req + T_transport
        + T_service + T_serialize_cpl + T_return

Dimensional check (the discipline Chapter 22.1 §4 established): every term is [seconds], and serialization is [bits] ÷ [bits/second]. A term expressed in cycles must be multiplied by the clock period before it can be added to one expressed in seconds — mixing the two is the most common arithmetic error in latency budgets.

Note which terms are not PCIe. T_service is the host memory system, and it frequently dominates a small read. PCIe latency and DRAM latency are different quantities that happen to appear in the same sum (§16), and a chapter that blurs them will send optimization effort to the wrong subsystem.

5. Writes Do Not Have a Completion to Wait For

6. First Byte Is Not Last Byte

A read may be answered in several Completions (Chapter 20.3 §5). So a single read has two meaningful end events, and they are far apart.

§13 Model 10 measured the gap across 60,000 reads with 1–8 fragments: mean first-byte 9.00, mean final 40.61 — a ratio of 4.51×. And treating the first fragment as completion understated the latency in 52,574 of 60,000 reads (87.6%).

Two consequences.

For measurement, the RTL must capture both and free the context only at the final beat — freeing on the first Completion is mutation 13, and it also releases a Tag while data is still arriving, which Chapter 21.4 §10 showed is a correctness bug and not merely a measurement one.

For architecture, a pipeline that can start on the first fragment sees T_first; one that needs the whole buffer sees T_final. Quoting one to a consumer of the other overstates or understates by the fragment count, which the model puts at 4.5× on average.

7. Queueing Is Latency Too

The term engineers most often exclude is the one that usually dominates.

T_queue is the time a request waits before it is even offered to the Link — behind other requests, behind arbitration, behind a class that currently lacks credits (22.3). It is real time, experienced by the requester, and it belongs in the total.

§13 Model 6, one physical path, four loads — ILLUSTRATIVE normalized units:

At heavy load nearly two-thirds of the latency is waiting, and every physical term is unchanged. A latency figure measured at idle predicts nothing about this system under load — which is why §15's first scenario is "bad only under load".

And this is where latency and throughput meet. Chapter 12.5 §5 owns Little's Law; the consequence here is that pushing utilisation up pushes queueing latency up, so the two goals in this module are in tension rather than aligned (§16).

8. Depth Costs, Both Ways

Each switch hop adds forwarding and queueing (Chapter 21.3 §6), and a read pays it twice — outbound request and returning Completion.

§13 Model 7 — per-hop cost of 3 normalized units each direction, everything else held constant:

Linear in depth, doubled by the round trip — and this chapter publishes no absolute per-hop figure (§1), because "a PCIe switch adds N ns" is exactly the kind of universal claim that requires a source and rarely has one.

9. The Path, End to End

Three things to read out of the figure.

t0 to t1 is entirely queueing — no PCIe mechanism is involved, and §7 measured it dominating at load.

t2 to t3 is host memory, not PCIe. It is frequently the largest single term for a small read, and optimizing the Link does not touch it.

And t4 and t5 are different events. A pipeline that can start at t4 has a latency of t4 − t0; one that needs the whole buffer has t6 − t0. §6 measured the mean ratio at 4.51×.

10. The Waveform

Three things to read out of the figure.

Cycles 1 to 3 are queueing, and they are inside the latency whether or not the instrument counts them (§7). A design that timestamps at cycle 3 reports a smaller, still-correct, differently-defined number (§3).

Cycles 4 to 6 show nothing on either interface. That is the host service term — invisible at the endpoint's pins, and often the largest part of the total.

And cycles 7 and 8 are both completion beats. Stopping the clock at cycle 7 measures T_first; stopping at cycle 8 measures T_final (§6).

11. RTL — Measuring Per Transaction

// SYNTHESIZABLE. Latency measurement types.
// The transaction identity is the PAIR -- Chapter 21.4 §3. A latency
// context keyed on Tag alone cross-attributes between Requesters.
package lat_pkg;
 
  parameter int TS_W    = 32;                 // free-running cycle counter
  parameter int TAGS    = 16;
  parameter int TAG_W   = (TAGS <= 1) ? 1 : $clog2(TAGS);
  parameter int RID_W   = 16;
  parameter int LEN_W   = 12;
  parameter int BINS    = 4;
  parameter int BIN_W   = (BINS <= 1) ? 1 : $clog2(BINS);
 
  typedef struct packed {
    logic [RID_W-1:0] requester_id;
    logic [TAG_W-1:0] tag;
  } txn_id_t;
 
  typedef struct packed {
    logic             valid;
    txn_id_t          id;
    logic [TS_W-1:0]  start_ts;
    logic [TS_W-1:0]  first_ts;
    logic             first_seen;
    logic [LEN_W-1:0] bytes_expected;
    logic [LEN_W-1:0] bytes_returned;
  } lat_ctx_t;
 
  // Unsigned subtraction is correct across ONE wrap of a free-running
  // counter, provided the measured interval is shorter than the counter's
  // modulus. That assumption is documented, and P17 bounds it.
  function automatic logic [TS_W-1:0] ts_delta(input logic [TS_W-1:0] now,
                                               input logic [TS_W-1:0] then_);
    return now - then_;
  endfunction
 
endpackage

import lat_pkg::*;
 
// SYNTHESIZABLE. Free-running cycle counter -- the ONLY time source.
// It is allowed to wrap; ts_delta() is correct across one wrap (§11).
module cycle_time_source (
  input  logic clk,
  input  logic rst_n,
  output logic [TS_W-1:0] cycle_time
);
  logic [TS_W-1:0] t_q;
  assign cycle_time = t_q;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) t_q <= '0; else t_q <= t_q + TS_W'(1);
endmodule

import lat_pkg::*;
 
// SYNTHESIZABLE. THE FLAGSHIP BLOCK. Per-transaction latency contexts.
// ONE GLOBAL start register is wrong for 80.0% of completions once
// requests overlap (§13 Model 8) -- so there is one context per identity.
module latency_tracker (
  input  logic clk,
  input  logic rst_n,
  input  logic [TS_W-1:0]  cycle_time,
 
  // start: the request is ACCEPTED at the TX handshake (§3)
  input  logic             rq_valid,
  input  logic             rq_ready,
  input  txn_id_t          rq_id,
  input  logic [LEN_W-1:0] rq_bytes,
 
  // completion beats, also on the handshake (§12 P2)
  input  logic             cpl_valid,
  input  logic             cpl_ready,
  input  txn_id_t          cpl_id,
  input  logic [LEN_W-1:0] cpl_bytes,
 
  output logic             sample_first_valid,
  output logic [TS_W-1:0]  sample_first_latency,
  output logic             sample_final_valid,
  output logic [TS_W-1:0]  sample_final_latency,
  output txn_id_t          sample_id,
  output logic             err_unknown_tag,      // sticky
  output logic             err_ctx_overwrite     // sticky
);
  lat_ctx_t ctx [TAGS];
  logic [TAGS-1:0] mvec;
  logic ek_q, eo_q;
  int   hit;
 
  assign err_unknown_tag  = ek_q;
  assign err_ctx_overwrite = eo_q;
 
  // Match on the WHOLE identity (Chapter 21.4 §3).
  always_comb begin
    for (int i = 0; i < TAGS; i++)
      mvec[i] = ctx[i].valid
             && (ctx[i].id.requester_id == cpl_id.requester_id)
             && (ctx[i].id.tag          == cpl_id.tag);
    hit = 0;
    for (int i = 0; i < TAGS; i++) if (mvec[i]) hit = i;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < TAGS; i++) ctx[i] <= '0;
      sample_first_valid <= 1'b0; sample_final_valid <= 1'b0;
      sample_first_latency <= '0; sample_final_latency <= '0;
      sample_id <= '0; ek_q <= 1'b0; eo_q <= 1'b0;
    end else begin
      sample_first_valid <= 1'b0;
      sample_final_valid <= 1'b0;
 
      // ---- START: only on an ACCEPTED request -----------------------
      if (rq_valid && rq_ready) begin
        if (ctx[rq_id.tag].valid) eo_q <= 1'b1;      // live context: REPORT
        else ctx[rq_id.tag] <= '{valid:1'b1, id:rq_id, start_ts:cycle_time,
                                 first_ts:'0, first_seen:1'b0,
                                 bytes_expected:rq_bytes, bytes_returned:'0};
      end
 
      // ---- COMPLETION: only on an ACCEPTED beat (§13 Model 9) --------
      if (cpl_valid && cpl_ready) begin
        if (mvec == '0) ek_q <= 1'b1;                // unknown identity: REPORT
        else begin
          automatic logic [LEN_W-1:0] tot = ctx[hit].bytes_returned + cpl_bytes;
          ctx[hit].bytes_returned <= tot;
 
          if (!ctx[hit].first_seen) begin            // FIRST -- captured once
            ctx[hit].first_seen   <= 1'b1;
            ctx[hit].first_ts     <= cycle_time;
            sample_first_valid    <= 1'b1;
            sample_first_latency  <= ts_delta(cycle_time, ctx[hit].start_ts);
            sample_id             <= ctx[hit].id;
          end
 
          // FINAL -- only when the byte count is satisfied (§6). Freeing
          // on the first beat is mutation 13, and it also releases a Tag
          // while data is still arriving (Chapter 21.4 §10).
          if (tot >= ctx[hit].bytes_expected) begin
            ctx[hit].valid       <= 1'b0;
            sample_final_valid   <= 1'b1;
            sample_final_latency <= ts_delta(cycle_time, ctx[hit].start_ts);
            sample_id            <= ctx[hit].id;
          end
        end
      end
    end
  end
endmodule

import lat_pkg::*;
 
// SYNTHESIZABLE. Small latency histogram (§11). Bins are half-open and
// contiguous, so classification is exactly one-hot -- §13 Model 11
// checked 200,000 samples with 0 non-one-hot classifications.
module latency_histogram #(
  parameter int unsigned B1 = 16,
  parameter int unsigned B2 = 64,
  parameter int unsigned B3 = 256
) (
  input  logic clk,
  input  logic rst_n,
  input  logic            sample_valid,
  input  logic [TS_W-1:0] sample_latency,
  input  logic            clear,
 
  output logic [31:0]     bin [BINS],
  output logic [BIN_W-1:0] bin_sel,
  output logic            saturated
);
  logic [31:0] b_q [BINS];
  logic sat_q;
 
  // Half-open [lo, hi): contiguous and non-overlapping BY CONSTRUCTION.
  always_comb begin
    if      (sample_latency <  TS_W'(B1)) bin_sel = BIN_W'(0);
    else if (sample_latency <  TS_W'(B2)) bin_sel = BIN_W'(1);
    else if (sample_latency <  TS_W'(B3)) bin_sel = BIN_W'(2);
    else                                  bin_sel = BIN_W'(3);
  end
 
  always_comb begin
    for (int i = 0; i < BINS; i++) bin[i] = b_q[i];
    saturated = sat_q;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      for (int i = 0; i < BINS; i++) b_q[i] <= '0;
      sat_q <= 1'b0;
    end else if (sample_valid) begin
      if (&b_q[bin_sel]) sat_q <= 1'b1;              // saturate and REPORT
      else b_q[bin_sel] <= b_q[bin_sel] + 32'd1;
    end
  end
endmodule

import lat_pkg::*;
 
// SYNTHESIZABLE. Worst-observed latency, with the identity that caused it.
// Practical silicon debug: the outlier's Tag is usually the whole clue.
module max_latency_tracker (
  input  logic clk,
  input  logic rst_n,
  input  logic            sample_valid,
  input  logic [TS_W-1:0] sample_latency,
  input  txn_id_t         sample_id,
  input  logic            clear,
 
  output logic [TS_W-1:0] max_latency,
  output txn_id_t         max_id,
  output logic [31:0]     sample_count
);
  logic [TS_W-1:0] max_q; txn_id_t id_q; logic [31:0] n_q;
  assign max_latency = max_q; assign max_id = id_q; assign sample_count = n_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin max_q <= '0; id_q <= '0; n_q <= '0; end
    else if (sample_valid) begin
      if (!(&n_q)) n_q <= n_q + 32'd1;
      // MONOTONIC between clears. A smaller sample must NEVER lower it --
      // that is mutation 12, and it erases the outlier you were hunting.
      if (sample_latency > max_q) begin
        max_q <= sample_latency;
        id_q  <= sample_id;
      end
    end
  end
endmodule

Classification: all five synthesizable.

And one block deliberately absent: a percentile calculator. Percentiles need either the whole sample set or a sorted structure, and neither belongs in a datapath. The hardware records a histogram and a maximum; software computes median, p95 and p99 from them. Chapter 22.6 owns how to interpret those numbers — building the arithmetic in RTL would be expensive, inflexible and, for percentile estimates from coarse bins, less accurate than doing it in software.

Failure — six. One global start timestamp (80.0%, §13). Sampling on cpl_valid (3.01×). Freeing the context on the first beat (§6). Keying contexts on Tag alone. A max tracker that decays. And overlapping histogram bins, which makes the distribution unnormalizable.

12. Same-Cycle Audit and Assertions

// ==================================================================
// START -- the interval opens once, at a defined event (§3).
// ==================================================================
 
// P1: a context is created only on an ACCEPTED request.
property p_start_on_accepted_request;
  @(posedge clk) disable iff (!rst_n)
    ($rose(ctx[0].valid)) |-> $past(rq_valid && rq_ready);
endproperty
 
// P2: never on the offer alone -- the rule of Chapter 22.1 §7.
property p_no_start_on_valid_alone;
  @(posedge clk) disable iff (!rst_n)
    (rq_valid && !rq_ready) |=> $stable(ctx[0].valid);
endproperty
 
// P3: the start timestamp is STABLE while the transaction is outstanding.
// This is what one global register cannot provide (§13 Model 8).
property p_start_ts_stable;
  @(posedge clk) disable iff (!rst_n)
    (ctx[0].valid && !$rose(ctx[0].valid)) |-> $stable(ctx[0].start_ts);
endproperty
 
// P4: a live context is never silently overwritten by a reused Tag.
property p_no_overwrite_live_ctx;
  @(posedge clk) disable iff (!rst_n)
    (rq_valid && rq_ready && ctx[rq_id.tag].valid) |=> err_ctx_overwrite;
endproperty
 
// ==================================================================
// COMPLETION -- identity, and the handshake (§11).
// ==================================================================
 
// P5: a beat with no matching context produces NO sample and is reported.
property p_unknown_tag_no_sample;
  @(posedge clk) disable iff (!rst_n)
    (cpl_valid && cpl_ready && (mvec == '0))
      |=> (!sample_first_valid && !sample_final_valid && err_unknown_tag);
endproperty
 
// P6: a sample never uses another transaction's start timestamp.
property p_no_cross_tag_contamination;
  @(posedge clk) disable iff (!rst_n)
    sample_final_valid |-> ((sample_id.requester_id == $past(cpl_id.requester_id))
                         && (sample_id.tag          == $past(cpl_id.tag)));
endproperty
 
// P7: the FIRST sample is emitted exactly once per transaction.
property p_first_sample_once;
  @(posedge clk) disable iff (!rst_n)
    (sample_first_valid && $past(ctx[hit].first_seen)) |-> 1'b0;
endproperty
 
// P8: first latency never exceeds final latency. They are the same
// interval measured to two different end events (§6).
property p_first_le_final;
  @(posedge clk) disable iff (!rst_n)
    sample_final_valid |-> (sample_first_latency <= sample_final_latency);
endproperty
 
// P9: a sample is emitted only on an ACCEPTED beat. §13 Model 9:
// sampling on cpl_valid recorded 3.01x as many samples as completions.
property p_sample_on_handshake_only;
  @(posedge clk) disable iff (!rst_n)
    (sample_first_valid || sample_final_valid) |-> $past(cpl_valid && cpl_ready);
endproperty
 
// P10: a stalled Completion beat produces no repeated samples.
property p_stall_no_duplicate_samples;
  @(posedge clk) disable iff (!rst_n)
    (cpl_valid && !cpl_ready) |=> (!sample_first_valid && !sample_final_valid);
endproperty
 
// ==================================================================
// LIFETIME -- the context is freed by BYTES, not by arrival (§6).
// ==================================================================
 
// P11: the context is freed only when the expected byte count is met.
property p_free_on_bytes_satisfied;
  @(posedge clk) disable iff (!rst_n)
    ($fell(ctx[0].valid) && !$past(!rst_n))
      |-> $past(ctx[0].bytes_returned + cpl_bytes >= ctx[0].bytes_expected);
endproperty
 
// P12: a first fragment of an incomplete read never emits a FINAL sample.
property p_partial_no_final_sample;
  @(posedge clk) disable iff (!rst_n)
    (cpl_valid && cpl_ready && (mvec != '0)
      && ((ctx[hit].bytes_returned + cpl_bytes) < ctx[hit].bytes_expected))
      |=> !sample_final_valid;
endproperty
 
// P13: outstanding contexts equal valid timestamp entries -- the tracker's
// population is not an independent number.
property p_population_consistent;
  @(posedge clk) disable iff (!rst_n)
    ($countones({ctx[0].valid, ctx[1].valid}) <= 2);
endproperty
 
// ==================================================================
// AGGREGATION -- histogram and maximum.
// ==================================================================
 
// P14: exactly one bin is selected per sample; bins are half-open and
// contiguous (§13 Model 11: 0 non-one-hot over 200,000 samples).
property p_one_bin_per_sample;
  @(posedge clk) disable iff (!rst_n)
    sample_valid |-> (bin_sel < BIN_W'(BINS));
endproperty
 
// P15: exactly one bin counter advances per sample.
property p_one_bin_advances;
  @(posedge clk) disable iff (!rst_n)
    (sample_valid && !saturated) |=>
      ($countones({bin[0] != $past(bin[0]), bin[1] != $past(bin[1]),
                   bin[2] != $past(bin[2]), bin[3] != $past(bin[3])}) == 1);
endproperty
 
// P16: the maximum is MONOTONIC between clears -- a smaller sample never
// lowers it (mutation 12 erases the outlier being hunted).
property p_max_monotonic;
  @(posedge clk) disable iff (!rst_n)
    (!clear) |=> (max_latency >= $past(max_latency));
endproperty
 
// P17: the sample count advances exactly once per completed transaction.
property p_sample_count_once;
  @(posedge clk) disable iff (!rst_n)
    (sample_valid && !(&sample_count)) |=> (sample_count == $past(sample_count) + 32'd1);
endproperty
 
// P18: measured intervals stay within the counter's modulus -- the
// documented assumption behind unsigned ts_delta (§11).
property p_interval_within_modulus;
  @(posedge clk) disable iff (!rst_n)
    sample_final_valid |-> (sample_final_latency < {1'b0, {(TS_W-1){1'b1}}});
endproperty
 
// ==================================================================
// NON-INTERFERENCE.
// ==================================================================
 
// P19: reset clears every context and emits no samples.
property p_reset_clears_contexts;
  @(posedge clk)
    (!rst_n) |=> (!ctx[0].valid && !sample_first_valid && !sample_final_valid);
endproperty
 
// P20: the monitor never drives the functional interfaces.
property p_monitor_does_not_drive;
  @(posedge clk) disable iff (!rst_n)
    $stable({rq_ready, cpl_ready}) or !$stable({max_latency, sample_count});
endproperty

Twenty properties. P1–P6 are the chapter — identity and event definition. P7–P12 exist because a read answers in pieces, and P16 exists because the most valuable sample is the worst one, which a decaying tracker destroys.

13. Measured Behaviour

14. Verification — DV and Mutations

DV, against an independent dictionary keyed by (Requester ID, Tag) — never the DUT's own lookup: one request in isolation · queue delay before acceptance · a single-beat read where first and final coincide · a split read · multiple concurrent Tags · completions returning in reverse order · an unknown Tag · a stalled Completion beat · a bounded timestamp-wrap case · the exact histogram bin boundaries · a max update followed by smaller samples · reset with transactions outstanding.

Two counterexamples worth stating explicitly.

Mutation 1 is the classic, and it deserves its trace. Request A starts at t=10 and B at t=14, into a single global_start_ts. B's write clobbers A's value. A's Completion arrives at t=50: the true latency is 40, and the design computes 50 − 14 = 36. Nothing errors, the number is plausible, and it is wrong for every request except the most recent one. §13 measured that as 80% of all samples — and the design is perfectly correct with one outstanding request, which is exactly how it gets written and how it passes review.

Mutation 4 is worse because it is also a correctness bug. Declaring a read complete at the first fragment understates latency by 4.51× on average (§13), and it retires the Tag while further Completions are still arriving — so the next user of that Tag can be credited with another transaction's data (Chapter 21.4 §10). A measurement bug and a data-corruption bug from one line.

15. Debugging

Scenario — latency is fine on the bench and terrible in production. Almost always queueing (§7). §13 Model 6 moved the queue share from 0% to 63.3% with no physical change. Compare the decomposition at both loads rather than the totals; if only T_queue grew, the fix is arbitration, admission or offered load — not the Link.

Scenario — first data arrives promptly, the descriptor completes late. Fragmentation (§6). Read T_first and T_final separately; a large ratio means many Completions, which points at the Completer's splitting behaviour and at total payload — Chapter 22.4 owns the payload-size analysis.

Scenario — one Tag reports an absurd latency and the others are normal. Suspect the instrument before the fabric. This is the signature of context overwrite (mutation 9) or cross-Tag contamination (mutation 5). Check err_ctx_overwrite and err_unknown_tag first — both are sticky in §11 precisely so this question is answerable after the fact.

Scenario — latency grew after adding a switch layer. Decompose per hop (§8) and remember the round trip pays it twice. If the growth exceeds the hop terms, the extra is queueing at the new level, not the hop itself (21.3 §6).

Scenario — write latency measures as near-zero. It probably is — and it is measuring TX acceptance, not host visibility (§5). Do not report it as though a Posted write had confirmed anything about host memory.

Scenario — the mean looks fine and users complain. Read the histogram and the maximum (§11), not the mean. A distribution with a long tail has an unremarkable mean, and the tail is what users experience. The max tracker's captured Tag is usually the fastest route to the cause, which is why P16 forbids it from decaying.

Scenario — two teams report different latencies for the same operation. Ask which event pair each measured (§3) before touching anything. Sixteen defensible pairs exist, and most such disputes dissolve at this question.

16. Misconceptions

"PCIe latency is one number." It is an interval between two events you must name (§3), and it moves with load (§7).

"A newer generation halves latency." Higher rate shortens serialization, which is one term (§4). Queueing, host service and return are untouched; §13's host term alone is 40 of 58 units at idle.

"Posted writes have a Completion latency." They have no Completion (§5). Any such number measures something else.

"Every switch adds the same latency." This chapter publishes no universal per-hop figure and neither should anyone else without a source (§8).

"Host memory latency is PCIe latency." Different subsystems in the same sum (§4).

"The first Completion means the read is done." It means the first fragment arrived — 4.51× short of final on average (§13, §6).

"One timestamp register is fine, requests are issued in order." They complete out of order, and one register is wrong for 80.0% of completions (§13 Model 8).

"Measuring in software is the same as measuring on the wire." They are different event pairs with driver and OS scheduling in between (§3).

"Lower latency always means higher throughput." No — they are different quantities (12.5 §1), and pushing utilisation up raises queueing latency (§7).

"Queueing is overhead, not real latency." It is time the requester waits, and at heavy load it was 63.3% of the total (§13).

"A p99 belongs in hardware." Hardware records the histogram; software computes percentiles (§11).

17. Understanding Check

Q1. Two Memory Reads use different Tags. B starts after A but completes first. Why is one global timestamp register unusable even though the requests were issued in order? Because the register holds only the most recent start. B's start overwrote A's, so A's completion is measured against B's start and reports a latency shorter than the truth. Issue order is irrelevant — overlap is what breaks it, and §13 Model 8 measured 80.0% of samples wrong. The fix is one context per (Requester ID, Tag).

Q2. Your latency monitor samples on cpl_valid. What does the histogram look like, and why is the error not random? It records 3.01× as many samples (§13 Model 9), one per stall cycle. The bias is systematic: heavily stalled completions contribute the most duplicates, so the distribution is weighted toward exactly the transactions that were most back-pressured — the mean and every percentile shift together.

Q3. A read returns in six Completions. Give two correct latencies for it and say who each is for. T_first — start to the first accepted beat — is correct for a streaming consumer that can begin work immediately. T_final is correct for anything needing the whole buffer. §13 measured a mean ratio of 4.51×; quoting one to a consumer of the other is off by the fragment structure.

Q4. An endpoint reports "write latency: 3 cycles." What did it actually measure? The time for its own TX interface to accept the Posted write (§5). It is not a statement about the Link, the Root Complex or host memory — a Posted write returns no Completion, so nothing in the transaction reports host visibility.

Q5. Latency doubled after moving a device one switch level deeper. Is the hop the explanation? Only partly. §13 Model 7 puts one extra level at about 1.10× with pure hop terms. A doubling means the extra time is mostly queueing at the new level, plus the round trip paying the hop twice (§8) — so the investigation is contention on the shared segment, not the switch's forwarding delay.

Q6. Why is percentile computation deliberately absent from §11's RTL? Because percentiles need the sample set or a sorted structure, both expensive in a datapath, and coarse-bin estimates computed in hardware are less accurate than software working from the same bins (§11). Hardware supplies the histogram and the maximum; software owns the statistics, and Chapter 22.6 owns interpreting them.

18. What's Next

Two of the three questions in this batch are now answered. 22.1 measured how much, this chapter measured how long, and both required the same discipline: name the measurement point, then account for everything you did not measure.

Chapter 22.3 takes apart the category both chapters kept deferring. 22.1 §6 counted no_credit cycles and §7 above attributed queueing to "a class that currently lacks credits" — neither said which of the six credit classes was empty, whether a packet of that class was even pending, or what ceiling the credit-return loop imposes on throughput. That chapter is the deepest of the three.

And three chapters remain untouched by this one. 22.4 owns MPS/MRRS and how packetization changes the serialization terms §4 named. 22.5 owns exact overhead accounting. 22.6 owns benchmark interpretation — for which §3's event-pair discipline is the prerequisite, because most benchmark disagreements are definitional rather than empirical.