PCIe · Module 25
Completion Timeouts — Which Timer Fired, and Was It the Right One?
A timeout report names the timer that fired first, not the thing that broke. Three timers never conflated, the late-Completion misdelivery measured at 1.9 MB into wrong buffers, and the diagnostic that cannot see its own failure.
Chapter 25.6 built an engine that originates requests and never asked what happens when one is never answered. A non-posted request that never completes is its own failure class, and it is the first fault in Module 25 whose recovery is more dangerous than the fault.
1. Sources, Scope, and the Boundary With Modules 10, 13 and 23.5
2. A Timeout Is a Decision, Not an Observation
The phrase "the request timed out" describes something the requester did, not something that happened to it. Nothing arrived; a counter reached a threshold; the requester chose to stop waiting. The distinction sounds pedantic and it organises the entire chapter.
Because it was a decision, it has all the properties of a decision. It can be made too early. It can be made by the wrong agent. It can be made correctly and then followed by an unsafe cleanup. And crucially: the absence it responds to has several possible causes, and the timeout itself cannot distinguish them.
the requester observes: nothing came back
the possible causes: no Completion was ever sent
a Completion was sent and lost on the link
a Completion arrived with an identity that did not match
Completions arrived but the byte count was never satisfied
a Completion arrived and the local retirement path dropped itFive causes, one observation. The timer fires identically for all five, which is why §3's classification comes before any instrument, and why a design that records only "request N timed out" has thrown away every distinguishing fact at the moment it was available.
Two consequences shape the rest of the chapter.
The instrument must record what the timer could not. Bytes received so far, Completions matched so far, whether any Completion arrived at all, whether one arrived and was rejected. §10's outstanding-request table exists to hold exactly those fields, and §15's cases are almost entirely reads of them.
And the cleanup is a separate design problem from the detection. Reclaiming a Tag is the correct thing to do — a requester that never reclaims starves itself, and §13 measured that at 366 requests issued against a healthy 31,195. But reclamation creates the possibility of a late Completion matching a new request, and §7 is entirely about that.
3. Five Causes, One Observation
Class D is the one that changes a debugging session's direction, and §13 measured it as the majority of timeouts: 1,218 of 3,089 timed-out requests, 39.4%, had received part of their data.
A partial timeout means the path worked and then stopped. The completer decoded the address, generated Completions, and routed at least one back successfully — so every hypothesis about routing, address decode, or a dead completer is already eliminated. That is a fundamentally different investigation from a request that received nothing, and the only thing distinguishing them is a byte counter the outstanding table has to have been keeping.
Class E is the one people do not believe. The Completion arrived, matched, and the requester still timed out — because the retirement path had backpressure, or a result FIFO was full, or the matched Completion was written into a buffer whose owner never took it. 23.5 owns that path; what matters here is that a timeout does not prove anything about the link, and §15 case 8 is how to rule it in or out in one read.
And classes A and B cannot be separated at the requester at all. Both look identical: nothing arrived. Separating them requires evidence from the other end — a completer-side counter, or a link-side trace (25.9). A chapter that claimed otherwise would be inventing a capability, so this one states the limit plainly and §15 case 2 treats it as the point where local instrumentation is exhausted.
4. Three Timers, Never Conflated
At least three independent timers can decide that a request has taken too long, and they belong to three different agents with three different scopes.
1 the hardware Completion Timeout
owner: the requester's transaction layer
scope: one outstanding non-posted request
effect: abandons the request and reclaims its Tag
2 the local engine watchdog
owner: the DMA engine or the function's control logic
scope: one job, which may span many requests
effect: aborts the job; may reset engine state
3 the driver/software timeout
owner: host software
scope: one operation, which may span many jobs
effect: error recovery, device reset, re-enumerationTheir ordering determines what gets reported, and §14 measured exactly that. One fault population, three orderings, nothing else changed:
| ordering | completed normally | reported cause |
|---|---|---|
| CTO 200 < watchdog 300 < driver 500 (correct) | 10,016 | hardware Completion Timeout — 9,984 |
| driver 200 < watchdog 300 < CTO 500 (inverted) | 10,016 | driver timeout — 9,984 |
| all three at 300 (ambiguous) | 13,239 | decided by sweep order |
The correct ordering is innermost-first, and the reason is mechanical rather than aesthetic.
The hardware timeout must fire first because it is the only one that can safely reclaim the Tag. The Tag is the requester's resource; the requester's transaction layer is the only agent that knows whether a Completion is still legally possible. When a software timeout fires first, the request is still outstanding in hardware — software begins recovery, possibly resetting the device, while a Completion for a live Tag may still arrive. The recovery races the link.
And the equal-timer row is the one to avoid designing. When two timers expire at the same time, which one reports depends on the order the sweep happens to visit them. The reported cause becomes a property of the implementation's loop order rather than of the fault — which means the statistic is not merely imprecise, it is arbitrary, and no amount of collecting more samples improves it.
A practical rule follows, and it is a DEBUG HEURISTIC: before believing any timeout statistic from a system, establish the three timers' relative values. If they are not strictly ordered innermost-to-outermost, the attribution in every report you have is unreliable, and §15 case 1 is that check.
5. Partial Completion, and the Byte Count That Is Never Satisfied
A read is not answered by "a Completion". It is answered by enough Completions to satisfy its byte count (13.3 §2).
That distinction creates class D, and it creates a specific instrument requirement: the outstanding-request table must track bytes, not Completions.
A design that retires a request on "a Completion arrived with my Tag" is wrong in two directions. It retires early on the first of several split Completions, freeing the Tag while more are still in flight; and it cannot detect the case where the last Completion of a series never arrives, because it already retired.
§13's model split every read at RCB boundaries and tracked bytes. The result:
| value | |
|---|---|
| requests that timed out | 3,089 |
| of which had received part of their data | 1,218 (39.4%) |
Nearly two in five timeouts were partial. Every one of them is a request where routing, decode and Completion generation all demonstrably worked.
The Byte Count field is what makes this checkable rather than inferable. 13.3 §2 establishes that it carries the count remaining, including the current Completion — so a requester can verify each arriving Completion against its own running total rather than merely accumulating. A mismatch between the two is detectable at the Completion, several timer-periods before the timeout, and P14 asserts it.
One thing this chapter does not do is publish a rule for how many Completions a read "should" take. That depends on RCB, on the completer's policy, and on the request's alignment — all owned by 13.3. A count-based expectation is exactly the kind of invented constant §1 forbids. The byte count is the contract; the Completion count is not.
6. The Timer Itself
The timer is the easiest part of this chapter to get wrong in a way that measures nothing, and 25.4 §7 already established the shape of the error. Three requirements, stated as design obligations:
It must be per-request, not global. A single timer covering all outstanding requests times out the youngest along with the oldest. Each entry in the outstanding table carries its own age.
It must compare on the next value, not the current one. age == LIMIT fails whenever the counter can step by more than one — and an age counter that advances on a prescaler tick, or on a variable number of cycles, does exactly that. 25.4 §15 measured == LIMIT missing in 20.0% of trials at a 1–3 step and never firing at all at steps of three or more. The correct form is age_next >= LIMIT, and P9 states it.
And its threshold must be a configured value, read from the design. This chapter publishes no timeout value because it cannot source one (13.2 and the requester's own documentation own that). Every numeric limit in §10 is a model parameter marked as such, and a debugging session that assumes a value rather than reading it has invented its own evidence — which is 25.4 mutation 34 in a new context.
One additional requirement is specific to this chapter. The age must be sampled on a clock that keeps running when the link does not. A timer derived from a link-active gate stops counting exactly when the fault it exists to detect occurs, and P11 is the property that catches it.
7. Reclamation, and the Late Completion
This is the chapter's core result and the most dangerous mechanism in Module 25 so far.
When a timeout fires, the requester reclaims the Tag — correctly, because a requester that does not reclaim runs out. §13 measured the alternative: with reclamation removed, the requester issued 366 requests against a healthy 31,195, with zero free Tags at the end. It starves itself into silence.
But reclamation creates a window. The abandoned request's Completion may still be in flight. When it arrives, its Tag has been reissued to a different request — and (Requester ID, Tag) is the entire identity of a Completion (10.2). The late Completion is indistinguishable from the correct one.
The correct configuration saw 8,371 late Completions and misdelivered none of them. Late Completions are not rare and they are not an error condition — they are the normal consequence of abandoning a request whose answer is still in flight. The design's job is to discard them, and to count them.
The epoch is what makes that possible. Each Tag carries a generation number, incremented every time the Tag is reissued. A Completion is qualified by (RID, Tag, epoch), and one that does not match the current epoch is discarded. This is 21.4 §3's law — an asynchronous response must carry its own identity — with the identity extended to include which use of that identity.
And the third row is the result worth carrying out of this chapter.
Removing the epoch bump corrupts exactly as much as removing the check — 39,574 misdeliveries, 1,907,096 identical bytes — and the epoch-based diagnostic reports zero late Completions. It cannot report otherwise: it detects lateness by comparing epochs, and the mutation makes every epoch equal. The instrument's failure and the fault are the same event.
That is a general hazard, not a curiosity about epochs. Any diagnostic derived from the mechanism it is checking has a blind spot exactly where the mechanism fails. §13's model needed a ground-truth serial — a quantity with no hardware counterpart — to see it at all, and §15 case 5 is how to construct the equivalent check in a real system.
8. The Outstanding Request Table, Drawn
The two edges out of "Late Completion" are the whole chapter. One is a counted discard; the other is silent corruption. They differ by one comparison, and §7's third row showed that the comparison can be present and still fail — if the value it compares was never advanced.
9. The Waveform
Timeout, tag reissue, then the original Completion arrives
10 cyclesFour readings.
age_expire and tag_reissue are separated by two cycles, and the gap is where the design's safety lives. Reclamation is not instantaneous and does not need to be; what matters is that the epoch is bumped at abandonment, not at reissue (P19, mutation 22).
At cycle 7 cpl_valid and epoch_match disagree. That disagreement is the entire mechanism. Remove epoch_match from the trace and cycles 7 and 9 are identical events — a Completion carrying the right Tag for a live request.
bytes_adv does not assert at cycle 7. The late Completion contributes nothing to the new request's byte count, which is what makes the discard correct rather than merely quiet.
And cpl_discard asserts. It is counted, not dropped — 25.1 §6's rule that a detected-and-discarded condition must be visible, and §15 case 4 reads that counter as its primary evidence.
10. RTL — The Timeout Instruments
Block 1 — the package: timer taxonomy, request classes, and the epoch type.
package cto_dbg_pkg;
// §4's three timers, kept as distinct types so that a report can never
// conflate them. The enum exists to make "which timer fired" a field
// rather than an inference.
typedef enum logic [1:0] {
TMR_NONE = 2'd0,
TMR_HARDWARE = 2'd1, // the requester's Completion Timeout
TMR_WATCHDOG = 2'd2, // the local engine's job watchdog
TMR_DRIVER = 2'd3 // host software
} timer_src_e;
// §3's five causes. The table records which one applied, because the
// timer itself fires identically for all five.
typedef enum logic [2:0] {
CTO_NONE = 3'd0,
CTO_NEVER_ANSWERED= 3'd1, // A — nothing arrived at all
CTO_LOST = 3'd2, // B — indistinguishable from A locally (§3)
CTO_WRONG_ID = 3'd3, // C — arrived, identity mismatched
CTO_PARTIAL = 3'd4, // D — 0 < bytes < requested
CTO_LOCAL_DROP = 3'd5 // E — matched, retirement path lost it
} cto_cause_e;
function automatic int unsigned gw(input int unsigned n);
return (n <= 1) ? 1 : $clog2(n);
endfunction
// The epoch is a generation counter per Tag. Its width bounds how many
// reissues can occur before a very late Completion could alias onto the
// same epoch again — see P21 and mutation 24.
localparam int unsigned EPOCH_W = 4;
typedef logic [EPOCH_W-1:0] epoch_t;
endpackageBlock 2 — the outstanding request table. The chapter's central structure: it holds what the timer cannot record.
module cto_outstanding_table #(
parameter int unsigned NTAG = 32,
parameter int unsigned CTO_LIMIT = 32'd2000 // MODEL PARAMETER — see §6
)(
input logic clk,
input logic rst_n,
// allocation
input logic alloc,
input logic [cto_dbg_pkg::gw(NTAG)-1:0] alloc_tag,
input logic [31:0] alloc_bytes,
input cto_dbg_pkg::epoch_t alloc_epoch,
// completion arrival, already identity-qualified upstream
input logic cpl_ok,
input logic [cto_dbg_pkg::gw(NTAG)-1:0] cpl_tag,
input logic [15:0] cpl_bytes,
// outputs
output logic timeout_fire,
output logic [cto_dbg_pkg::gw(NTAG)-1:0] timeout_tag,
output cto_dbg_pkg::cto_cause_e timeout_cause,
output logic [31:0] timeout_bytes_got,
output logic [31:0] timeout_bytes_want
);
import cto_dbg_pkg::*;
typedef struct packed {
logic live;
logic [31:0] age;
logic [31:0] want; // bytes requested
logic [31:0] got; // bytes received so far — §5's requirement
logic saw_any; // did ANY completion match this request?
epoch_t ep;
} ent_t;
ent_t tbl [NTAG];
logic [gw(NTAG)-1:0] sweep;
// Per-entry age, and the NEXT-value comparison of §6. `age == LIMIT`
// is 25.4 §7's measured failure: it never fires when the counter can
// step by more than one.
logic [31:0] age_next;
assign age_next = tbl[sweep].age + 32'd1;
always_comb begin
timeout_fire = tbl[sweep].live && (age_next >= CTO_LIMIT);
timeout_tag = sweep;
timeout_bytes_got = tbl[sweep].got;
timeout_bytes_want = tbl[sweep].want;
// §3's classification, decided from the evidence the table kept.
// A timer alone cannot produce this; only the recorded fields can.
if (!timeout_fire) timeout_cause = CTO_NONE;
else if (tbl[sweep].got == 0 && !tbl[sweep].saw_any)
timeout_cause = CTO_NEVER_ANSWERED;
else if (tbl[sweep].got == 0 && tbl[sweep].saw_any)
timeout_cause = CTO_WRONG_ID;
else if (tbl[sweep].got < tbl[sweep].want) timeout_cause = CTO_PARTIAL;
else timeout_cause = CTO_LOCAL_DROP;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < NTAG; i++) tbl[i] <= '0;
sweep <= '0;
end else begin
// Age every live entry. Note this is NOT gated on link activity —
// a timer that stops when the link stops cannot detect a dead link
// (P11, mutation 13).
for (int i = 0; i < NTAG; i++)
if (tbl[i].live && tbl[i].age != 32'hFFFF_FFFF)
tbl[i].age <= tbl[i].age + 32'd1;
if (alloc) begin
tbl[alloc_tag].live <= 1'b1;
tbl[alloc_tag].age <= '0;
tbl[alloc_tag].want <= alloc_bytes;
tbl[alloc_tag].got <= '0;
tbl[alloc_tag].saw_any <= 1'b0;
tbl[alloc_tag].ep <= alloc_epoch;
end
if (cpl_ok) begin
tbl[cpl_tag].saw_any <= 1'b1;
tbl[cpl_tag].got <= tbl[cpl_tag].got + 32'(cpl_bytes);
// Retire on BYTE COUNT, never on "a completion arrived" (§5).
if ((tbl[cpl_tag].got + 32'(cpl_bytes)) >= tbl[cpl_tag].want)
tbl[cpl_tag].live <= 1'b0;
end
if (timeout_fire) tbl[sweep].live <= 1'b0;
sweep <= (sweep == gw(NTAG)'(NTAG-1)) ? '0 : sweep + 1'b1;
end
end
endmoduleBlock 3 — the Tag allocator with epochs. §7's mechanism, with the bump placed at abandonment rather than at reissue.
module cto_tag_alloc #(
parameter int unsigned NTAG = 32
)(
input logic clk,
input logic rst_n,
input logic request,
output logic grant,
output logic [cto_dbg_pkg::gw(NTAG)-1:0] grant_tag,
output cto_dbg_pkg::epoch_t grant_epoch,
input logic retire, // normal completion
input logic [cto_dbg_pkg::gw(NTAG)-1:0] retire_tag,
input logic abandon, // timeout
input logic [cto_dbg_pkg::gw(NTAG)-1:0] abandon_tag,
output cto_dbg_pkg::epoch_t cur_epoch [NTAG],
output logic [cto_dbg_pkg::gw(NTAG+1)-1:0] free_count
);
import cto_dbg_pkg::*;
logic [NTAG-1:0] free_mask;
epoch_t ep [NTAG];
assign cur_epoch = ep;
always_comb begin
grant = request && (free_mask != '0);
grant_tag = '0;
for (int i = NTAG-1; i >= 0; i--) if (free_mask[i]) grant_tag = gw(NTAG)'(i);
grant_epoch = ep[grant_tag];
free_count = '0;
for (int i = 0; i < NTAG; i++) free_count += gw(NTAG+1)'(free_mask[i]);
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
free_mask <= '1;
for (int i = 0; i < NTAG; i++) ep[i] <= '0;
end else begin
if (grant) free_mask[grant_tag] <= 1'b0;
if (retire) begin
free_mask[retire_tag] <= 1'b1;
ep[retire_tag] <= ep[retire_tag] + 1'b1;
end
if (abandon) begin
free_mask[abandon_tag] <= 1'b1;
// The bump happens HERE, at abandonment — not at the next reissue.
// Deferring it leaves a window in which a late Completion matches
// the current epoch (mutation 22), and §13's third row measured
// what removing the bump entirely costs: 39,574 misdeliveries that
// the epoch-based diagnostic reports as ZERO late Completions.
ep[abandon_tag] <= ep[abandon_tag] + 1'b1;
end
end
end
endmoduleBlock 4 — the Completion identity qualifier. Where a late Completion is discarded, and counted.
module cto_cpl_qualify #(
parameter int unsigned NTAG = 32
)(
input logic clk,
input logic rst_n,
input logic cpl_valid,
input logic [15:0] cpl_rid,
input logic [cto_dbg_pkg::gw(NTAG)-1:0] cpl_tag,
input cto_dbg_pkg::epoch_t cpl_epoch,
input logic [15:0] my_rid,
input cto_dbg_pkg::epoch_t cur_epoch [NTAG],
input logic [NTAG-1:0] tag_live,
output logic cpl_accept,
output logic cpl_late, // stale epoch
output logic cpl_foreign, // wrong RID
output logic cpl_orphan, // tag not live
output logic [31:0] late_count,
output logic [31:0] foreign_count,
output logic [31:0] orphan_count
);
import cto_dbg_pkg::*;
// Three distinct rejection reasons, never merged. Merging them is
// mutation 27, and it destroys exactly the distinction §3 classes C and
// the late-completion path depend on.
always_comb begin
cpl_foreign = cpl_valid && (cpl_rid != my_rid);
cpl_orphan = cpl_valid && !cpl_foreign && !tag_live[cpl_tag];
cpl_late = cpl_valid && !cpl_foreign && tag_live[cpl_tag] &&
(cpl_epoch != cur_epoch[cpl_tag]);
cpl_accept = cpl_valid && !cpl_foreign && !cpl_orphan && !cpl_late;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
late_count <= '0; foreign_count <= '0; orphan_count <= '0;
end else begin
if (cpl_late && late_count != 32'hFFFF_FFFF) late_count <= late_count + 32'd1;
if (cpl_foreign && foreign_count != 32'hFFFF_FFFF) foreign_count <= foreign_count + 32'd1;
if (cpl_orphan && orphan_count != 32'hFFFF_FFFF) orphan_count <= orphan_count + 32'd1;
end
end
endmoduleBlock 5 — the byte-count checker. §5's requirement, using the Byte Count field to detect a mismatch at the Completion rather than at the timeout.
module cto_bytecount_check (
input logic clk,
input logic rst_n,
input logic cpl_accept,
input logic [11:0] cpl_byte_count, // remaining INCLUDING this Cpl (13.3 §2)
input logic [15:0] cpl_payload_bytes,
input logic [31:0] req_want,
input logic [31:0] req_got,
output logic bc_consistent,
output logic bc_violation,
output logic would_overrun,
output logic [31:0] bc_violation_count
);
logic [31:0] expect_remaining;
always_comb begin
// What the requester believes remains, before crediting this Cpl.
expect_remaining = req_want - req_got;
// The completer's own statement of what remains, from the header.
// Comparing the two catches a lost or duplicated split Completion at
// the moment it happens, several timer periods before the timeout.
bc_consistent = cpl_accept && (32'(cpl_byte_count) == expect_remaining);
bc_violation = cpl_accept && !bc_consistent;
// A Cpl carrying more payload than the request has outstanding is a
// buffer overrun, and it is the same two-sided reasoning as 25.6 §4.
would_overrun = cpl_accept && (32'(cpl_payload_bytes) > expect_remaining);
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) bc_violation_count <= '0;
else if (bc_violation && bc_violation_count != 32'hFFFF_FFFF)
bc_violation_count <= bc_violation_count + 32'd1;
end
endmoduleBlock 6 — the three-timer arbiter. §4's ordering, made structural instead of accidental.
module cto_timer_order #(
parameter int unsigned HW_LIMIT = 32'd2_000, // MODEL PARAMETERS (§6)
parameter int unsigned WDOG_LIMIT = 32'd20_000,
parameter int unsigned DRV_LIMIT = 32'd200_000
)(
input logic clk,
input logic rst_n,
input logic [31:0] req_age,
input logic [31:0] job_age,
input logic [31:0] op_age,
output cto_dbg_pkg::timer_src_e fired,
output logic ordering_ok,
output logic ordering_ambiguous
);
import cto_dbg_pkg::*;
// The ordering is checked, not assumed. §14 measured that inverting it
// changes the reported cause of an unchanged fault population from
// "hardware Completion Timeout, 9,984" to "driver timeout, 9,984".
assign ordering_ok = (HW_LIMIT < WDOG_LIMIT) && (WDOG_LIMIT < DRV_LIMIT);
assign ordering_ambiguous = (HW_LIMIT == WDOG_LIMIT) || (WDOG_LIMIT == DRV_LIMIT) ||
(HW_LIMIT == DRV_LIMIT);
// Innermost first. This priority is only MEANINGFUL when ordering_ok
// holds; when it does not, the reported cause is a property of this
// priority list rather than of the fault (§4's third row).
always_comb begin
if (req_age >= HW_LIMIT) fired = TMR_HARDWARE;
else if (job_age >= WDOG_LIMIT) fired = TMR_WATCHDOG;
else if (op_age >= DRV_LIMIT) fired = TMR_DRIVER;
else fired = TMR_NONE;
end
endmoduleBlock 7 — the first-timeout recorder. Sticky capture of everything §3 needs, at the moment it is available.
module cto_first_record #(
parameter int unsigned NTAG = 32
)(
input logic clk,
input logic rst_n,
input logic timeout_fire,
input logic [cto_dbg_pkg::gw(NTAG)-1:0] timeout_tag,
input cto_dbg_pkg::cto_cause_e timeout_cause,
input cto_dbg_pkg::timer_src_e timeout_timer,
input logic [31:0] bytes_got,
input logic [31:0] bytes_want,
input logic clear,
output logic captured,
output cto_dbg_pkg::cto_cause_e first_cause,
output cto_dbg_pkg::timer_src_e first_timer,
output logic [31:0] first_got,
output logic [31:0] first_want,
output logic [31:0] timeout_count
);
import cto_dbg_pkg::*;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n || clear) begin
captured <= 1'b0; first_cause <= CTO_NONE; first_timer <= TMR_NONE;
first_got <= '0; first_want <= '0; timeout_count <= '0;
end else if (timeout_fire) begin
// Sticky FIRST, per 25.1 §6: later timeouts are usually consequences
// of the first (a stalled requester times out everything it holds),
// and overwriting destroys the only informative sample.
if (!captured) begin
captured <= 1'b1;
first_cause <= timeout_cause;
first_timer <= timeout_timer;
first_got <= bytes_got;
first_want <= bytes_want;
end
if (timeout_count != 32'hFFFF_FFFF) timeout_count <= timeout_count + 32'd1;
end
end
endmoduleBlock 8 — the independent misdelivery oracle. §7's blind-spot result, turned into a checker. This is verification hardware, not product hardware — it uses a serial that no design mechanism consumes.
// verilog_lint: waive-start module-filename
module cto_misdeliver_oracle #(
parameter int unsigned NTAG = 32,
parameter int unsigned SER_W = 32
)(
input logic clk,
input logic rst_n,
// A ground-truth serial stamped on every request and carried, in the
// TESTBENCH only, alongside each Completion. No RTL mechanism reads it.
input logic alloc,
input logic [cto_dbg_pkg::gw(NTAG)-1:0] alloc_tag,
input logic [SER_W-1:0] alloc_serial,
input logic cpl_credited,
input logic [cto_dbg_pkg::gw(NTAG)-1:0] cpl_tag,
input logic [SER_W-1:0] cpl_serial,
input logic [15:0] cpl_bytes,
output logic misdeliver,
output logic [31:0] misdeliver_count,
output logic [63:0] misdeliver_bytes
);
logic [SER_W-1:0] owner_serial [NTAG];
// The check is independent of the epoch mechanism ENTIRELY. §13's third
// row is why: with the epoch bump removed, an epoch-based detector
// reports 0 late Completions while 39,574 payloads land in wrong
// buffers. A diagnostic derived from the mechanism it checks is blind
// exactly where that mechanism fails.
assign misdeliver = cpl_credited && (cpl_serial != owner_serial[cpl_tag]);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
misdeliver_count <= '0; misdeliver_bytes <= '0;
for (int i = 0; i < NTAG; i++) owner_serial[i] <= '0;
end else begin
if (alloc) owner_serial[alloc_tag] <= alloc_serial;
if (misdeliver) begin
misdeliver_count <= misdeliver_count + 32'd1;
misdeliver_bytes <= misdeliver_bytes + 64'(cpl_bytes);
end
end
end
endmodule
// verilog_lint: waive-stop module-filename11. Assertions
Table-integrity properties.
// P1 — a tag is live in the table iff the allocator has it outstanding.
property p1_table_alloc_agree;
@(posedge clk) disable iff (!rst_n)
tbl_live[tag] |-> !free_mask[tag];
endproperty
a_p1: assert property (p1_table_alloc_agree);
// P2 — allocation initialises the entry completely. A partially reset
// entry inherits the previous request's byte counts (mutation 4).
property p2_alloc_clears;
@(posedge clk) disable iff (!rst_n)
alloc |=> ((tbl[$past(alloc_tag)].age == 0) &&
(tbl[$past(alloc_tag)].got == 0) &&
(tbl[$past(alloc_tag)].saw_any == 1'b0));
endproperty
a_p2: assert property (p2_alloc_clears);
// P3 — an entry leaves the table exactly once per occupancy.
property p3_single_exit;
@(posedge clk) disable iff (!rst_n)
(retire && (retire_tag == abandon_tag)) |-> !abandon;
endproperty
a_p3: assert property (p3_single_exit);
// P4 — no allocation of a tag that is already live.
property p4_no_double_alloc;
@(posedge clk) disable iff (!rst_n)
alloc |-> !tbl_live[alloc_tag];
endproperty
a_p4: assert property (p4_no_double_alloc);
// P5 — the free count never exceeds NTAG.
property p5_free_bounded;
@(posedge clk) disable iff (!rst_n)
(free_count <= NTAG);
endproperty
a_p5: assert property (p5_free_bounded);Byte-accounting properties — §5.
// P6 — retirement happens on byte count, never on completion arrival.
property p6_retire_on_bytes;
@(posedge clk) disable iff (!rst_n)
retire |-> (tbl[retire_tag].got >= tbl[retire_tag].want);
endproperty
a_p6: assert property (p6_retire_on_bytes);
// P7 — a request is never retired with bytes still owed. The direct
// statement of "a split completion series is not one completion".
property p7_no_early_retire;
@(posedge clk) disable iff (!rst_n)
(cpl_accept && (tbl[cpl_tag].got + cpl_bytes < tbl[cpl_tag].want))
|=> tbl_live[$past(cpl_tag)];
endproperty
a_p7: assert property (p7_no_early_retire);
// P8 — received bytes never exceed requested bytes. 25.6 §4's two-sided
// reasoning, applied to the completion path.
property p8_no_byte_overrun;
@(posedge clk) disable iff (!rst_n)
cpl_accept |-> !would_overrun;
endproperty
a_p8: assert property (p8_no_byte_overrun);Timer properties — §6.
// P9 — the expiry comparison is on the NEXT value and is >=, not ==.
// 25.4 §15 measured `== LIMIT` missing 20.0% of the time at a 1-3 step.
property p9_next_value_compare;
@(posedge clk) disable iff (!rst_n)
(tbl_live[sweep] && ((tbl[sweep].age + 1) >= CTO_LIMIT)) |-> timeout_fire;
endproperty
a_p9: assert property (p9_next_value_compare);
// P10 — age is per request, cleared at allocation, never shared.
property p10_age_per_request;
@(posedge clk) disable iff (!rst_n)
alloc |=> (tbl[$past(alloc_tag)].age == 32'd0);
endproperty
a_p10: assert property (p10_age_per_request);
// P11 — the age advances regardless of link activity. A timer gated on
// link-active stops counting exactly when the fault occurs (mutation 13).
property p11_age_not_link_gated;
@(posedge clk) disable iff (!rst_n)
(tbl_live[tag] && !link_active && (tbl[tag].age != 32'hFFFF_FFFF))
|=> (tbl[tag].age > $past(tbl[tag].age));
endproperty
a_p11: assert property (p11_age_not_link_gated);
// P12 — no timeout fires before the limit.
property p12_no_early_timeout;
@(posedge clk) disable iff (!rst_n)
timeout_fire |-> ((tbl[timeout_tag].age + 1) >= CTO_LIMIT);
endproperty
a_p12: assert property (p12_no_early_timeout);
// P13 — the age saturates rather than wrapping. A wrapped age restarts
// the wait silently and the request never times out at all.
property p13_age_saturates;
@(posedge clk) disable iff (!rst_n)
(tbl[tag].age == 32'hFFFF_FFFF) |=> (tbl[tag].age == 32'hFFFF_FFFF);
endproperty
a_p13: assert property (p13_age_saturates);Byte Count header properties — §5.
// P14 — the completer's stated remaining count agrees with the requester's.
// This catches a lost split Completion AT the completion, not at the
// timeout several timer periods later.
property p14_bytecount_agrees;
@(posedge clk) disable iff (!rst_n)
cpl_accept |-> bc_consistent;
endproperty
a_p14: assert property (p14_bytecount_agrees);
// P15 — a byte count disagreement is reported, never silently tolerated.
property p15_bc_violation_reported;
@(posedge clk) disable iff (!rst_n)
(cpl_accept && !bc_consistent) |-> bc_violation;
endproperty
a_p15: assert property (p15_bc_violation_reported);
// P16 — the final Completion of a series states a remaining count equal to
// its own payload.
property p16_final_cpl_bytecount;
@(posedge clk) disable iff (!rst_n)
(cpl_accept && ((tbl[cpl_tag].got + cpl_bytes) == tbl[cpl_tag].want))
|-> (cpl_byte_count == cpl_payload_bytes);
endproperty
a_p16: assert property (p16_final_cpl_bytecount);Identity and epoch properties — §7, the chapter's core.
// P17 — a Completion with a foreign Requester ID is never accepted.
property p17_reject_foreign_rid;
@(posedge clk) disable iff (!rst_n)
(cpl_valid && (cpl_rid != my_rid)) |-> !cpl_accept;
endproperty
a_p17: assert property (p17_reject_foreign_rid);
// P18 — a Completion whose epoch does not match the tag's current epoch is
// never accepted. This is the CHECK; §13's second row removed it and
// measured 39,574 misdeliveries carrying 1,907,096 bytes.
property p18_reject_stale_epoch;
@(posedge clk) disable iff (!rst_n)
(cpl_valid && tag_live[cpl_tag] && (cpl_epoch != cur_epoch[cpl_tag]))
|-> !cpl_accept;
endproperty
a_p18: assert property (p18_reject_stale_epoch);
// P19 — the epoch is bumped AT abandonment. This is the BUMP; §13's third
// row removed it and measured identical corruption while the epoch-based
// diagnostic reported ZERO late Completions. P18 and P19 are a pair and
// neither alone is sufficient.
property p19_epoch_bumped_on_abandon;
@(posedge clk) disable iff (!rst_n)
abandon |=> (cur_epoch[$past(abandon_tag)] != $past(cur_epoch[$past(abandon_tag)]));
endproperty
a_p19: assert property (p19_epoch_bumped_on_abandon);
// P20 — the epoch is also bumped on normal retirement, so a very late
// Completion from a retired request cannot match either.
property p20_epoch_bumped_on_retire;
@(posedge clk) disable iff (!rst_n)
retire |=> (cur_epoch[$past(retire_tag)] != $past(cur_epoch[$past(retire_tag)]));
endproperty
a_p20: assert property (p20_epoch_bumped_on_retire);
// P21 — a tag is not reissued until its epoch has advanced. Without this
// the bump and the reissue can race (mutation 22).
property p21_no_reissue_before_bump;
@(posedge clk) disable iff (!rst_n)
(abandon && grant && (grant_tag == abandon_tag)) |-> 1'b0;
endproperty
a_p21: assert property (p21_no_reissue_before_bump);
// P22 — the INDEPENDENT oracle sees no misdelivery. Deliberately not
// derived from the epoch, because §13 showed an epoch-derived check is
// blind exactly where the epoch mechanism fails.
property p22_no_misdelivery;
@(posedge clk) disable iff (!rst_n)
!misdeliver;
endproperty
a_p22: assert property (p22_no_misdelivery);Reclamation and liveness properties.
// P23 — every timeout reclaims its tag. Omitting this starves the
// requester: §13 measured 366 requests issued against a healthy 31,195.
property p23_timeout_reclaims;
@(posedge clk) disable iff (!rst_n)
timeout_fire |=> free_mask[$past(timeout_tag)];
endproperty
a_p23: assert property (p23_timeout_reclaims);
// P24 — a live request eventually leaves the table, by retirement or
// timeout. The chapter's only liveness property.
property p24_eventual_exit;
@(posedge clk) disable iff (!rst_n)
alloc |-> s_eventually (!tbl_live[$past(alloc_tag)]);
endproperty
a_p24: assert property (p24_eventual_exit);
// P25 — free tags are eventually available again under any fault.
property p25_no_permanent_starvation;
@(posedge clk) disable iff (!rst_n)
(free_count == 0) |-> s_eventually (free_count > 0);
endproperty
a_p25: assert property (p25_no_permanent_starvation);Timer-ordering properties — §4.
// P26 — the three limits are strictly ordered innermost-to-outermost.
// §14 measured that violating this changes the REPORTED CAUSE of an
// unchanged fault population.
property p26_timer_ordering;
@(posedge clk) disable iff (!rst_n)
ordering_ok;
endproperty
a_p26: assert property (p26_timer_ordering);
// P27 — no two limits are equal. Equal limits make the reported cause a
// property of the sweep order rather than of the fault.
property p27_no_ambiguous_order;
@(posedge clk) disable iff (!rst_n)
!ordering_ambiguous;
endproperty
a_p27: assert property (p27_no_ambiguous_order);
// P28 — when the hardware timer has expired, no outer timer reports.
property p28_innermost_reports;
@(posedge clk) disable iff (!rst_n)
(req_age >= HW_LIMIT) |-> (fired == cto_dbg_pkg::TMR_HARDWARE);
endproperty
a_p28: assert property (p28_innermost_reports);Recorder-integrity properties.
// P29 — the first timeout's record is sticky.
property p29_first_sticky;
@(posedge clk) disable iff (!rst_n)
(captured && !clear) |=> (captured && $stable(first_cause) && $stable(first_timer));
endproperty
a_p29: assert property (p29_first_sticky);
// P30 — the recorded cause is consistent with the recorded byte counts.
// An instrument whose fields contradict each other is worse than none.
property p30_cause_consistent;
@(posedge clk) disable iff (!rst_n)
(captured && (first_cause == cto_dbg_pkg::CTO_PARTIAL))
|-> ((first_got > 0) && (first_got < first_want));
endproperty
a_p30: assert property (p30_cause_consistent);
// P31 — every rejection reason is counted separately (mutation 27).
property p31_reasons_distinct;
@(posedge clk) disable iff (!rst_n)
cpl_valid |-> $onehot0({cpl_late, cpl_foreign, cpl_orphan});
endproperty
a_p31: assert property (p31_reasons_distinct);Cover — the anti-vacuity set.
// P32 — a request leaves the outstanding table by exactly one route, and
// that route is recorded. A request that exits without a recorded cause
// makes §3's five-way classification unavailable for that request forever.
property p32_exit_route_recorded;
@(posedge clk) disable iff (!rst_n)
(tbl_live[tag] && !$past(tbl_live[tag])) or
($fell(tbl_live[tag]) |-> (retire || timeout_fire));
endproperty
a_p32: assert property (p32_exit_route_recorded);
// P32's covers — the rare events must occur, or P14 through P22 never evaluate.
// §13's model required an explicit "very late" traffic class to produce
// any late Completions at all; c3 is the cover that proves the testbench
// has one.
c1_partial_timeout: cover property (@(posedge clk) disable iff (!rst_n)
timeout_fire && (timeout_bytes_got > 0) &&
(timeout_bytes_got < timeout_bytes_want));
c2_zero_byte_to: cover property (@(posedge clk) disable iff (!rst_n)
timeout_fire && (timeout_bytes_got == 0));
c3_late_cpl: cover property (@(posedge clk) disable iff (!rst_n) cpl_late);
c4_foreign_rid: cover property (@(posedge clk) disable iff (!rst_n) cpl_foreign);
c5_orphan_cpl: cover property (@(posedge clk) disable iff (!rst_n) cpl_orphan);
c6_tag_exhaust: cover property (@(posedge clk) disable iff (!rst_n) free_count == 0);
c7_bc_violation: cover property (@(posedge clk) disable iff (!rst_n) bc_violation);
c8_split_series: cover property (@(posedge clk) disable iff (!rst_n)
cpl_accept ##[1:$] cpl_accept ##[1:$] retire);12. Executable Counterexamples
Six minimal designs, each violating exactly one property, each with the failing stimulus stated.
Counterexample A — retire on Completion arrival rather than byte count (violates P6, P7).
module ce_a_retire_on_arrival #(parameter int unsigned NTAG = 32)(
input logic clk, rst_n,
input logic cpl_accept, input logic [4:0] cpl_tag,
output logic [NTAG-1:0] live
);
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) live <= '0;
else if (cpl_accept) live[cpl_tag] <= 1'b0; // <-- any Cpl retires it
endmodule
// Failing stimulus: a 256-byte read split at RCB=64 into four Completions.
// Golden: live until the 4th Cpl satisfies the byte count.
// This: retires on the 1st. The tag is freed with 192 bytes still in
// flight, and the remaining three Completions become orphans.
// P6 fails (got < want at retirement). P7 fails.
// Observable consequence: the requester reports success having received a
// quarter of its data, and 3 orphan Completions arrive for a freed tag.Counterexample B — the epoch check removed (violates P18, P22).
module ce_b_no_epoch_check (
input logic cpl_valid, input logic [15:0] cpl_rid, input logic [15:0] my_rid,
input logic tag_is_live, output logic cpl_accept
);
assign cpl_accept = cpl_valid && (cpl_rid == my_rid) && tag_is_live;
endmodule // <-- no epoch compare
// Failing stimulus: request S1 on tag 7 times out; tag 7 is reissued to S2;
// S1's Completion arrives carrying epoch 0 while tag 7's epoch is now 1.
// Golden: cpl_late, discarded and counted.
// This: accepted, credited to S2.
// P18 fails. P22 fails (the independent oracle sees serial S1 credited to
// an entry owning serial S2).
// §13 measured this configuration at 39,574 misdeliveries and 1,907,096
// bytes written into wrong buffers.Counterexample C — the epoch bump removed (violates P19, P22, and defeats its own diagnostic).
module ce_c_no_epoch_bump #(parameter int unsigned NTAG = 32)(
input logic clk, rst_n, input logic abandon, input logic [4:0] abandon_tag,
output cto_dbg_pkg::epoch_t ep [NTAG]
);
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) for (int i = 0; i < NTAG; i++) ep[i] <= '0;
// the abandon path is present, and does NOT touch the epoch
endmodule
// Failing stimulus: identical to counterexample B.
// Golden: cpl_late asserts, the Completion is discarded.
// This: every epoch is permanently 0, so cpl_epoch == cur_epoch ALWAYS.
// The epoch comparison is present and passes. The Completion is
// accepted and credited to S2.
// P19 fails. P22 fails. P18 is VACUOUSLY SATISFIED — its antecedent
// (cpl_epoch != cur_epoch) is never true.
// This is the chapter's sharpest result: §13 measured 39,574 misdeliveries
// and 1,907,096 bytes — byte-identical to counterexample B — while the
// late-Completion counter read ZERO. A diagnostic derived from the
// mechanism it checks is blind exactly where that mechanism fails.Counterexample D — the link-gated age counter (violates P11).
module ce_d_link_gated_age (
input logic clk, rst_n, input logic link_active, input logic live,
output logic [31:0] age
);
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) age <= '0;
else if (live && link_active) age <= age + 32'd1; // <-- gated
endmodule
// Failing stimulus: the link drops while a request is outstanding.
// Golden: the age continues, the request times out, recovery begins.
// This: age freezes. The request NEVER times out. The requester waits
// forever for a Completion that cannot arrive, holding its tag.
// P11 fails. P24 fails (no eventual exit).
// Observable consequence: the exact fault the timer exists to detect is
// the one condition under which the timer stops running.Counterexample E — the single global timer (violates P10, P12).
module ce_e_global_timer (
input logic clk, rst_n, input logic any_live, input logic alloc,
output logic [31:0] age, output logic expire
);
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) age <= '0;
else if (alloc) age <= '0; // <-- reset by ANY allocation
else if (any_live) age <= age + 32'd1;
assign expire = (age >= 32'd2000);
endmodule
// Failing stimulus: a steady stream of short requests alongside one long
// outstanding request.
// Golden: the long request ages independently and times out.
// This: every new allocation resets the shared age, so the old request
// never expires while traffic continues. Under load it never
// times out at all; when traffic stops, everything times out
// simultaneously.
// P10 fails (age is not per-request). P12 fails.
// Observable consequence: timeouts arrive in bursts at idle, attributed to
// whatever was outstanding then — never to the request that actually hung.Counterexample F — merged rejection reasons (violates P31, and destroys §3's classification).
module ce_f_merged_reject (
input logic clk, rst_n, input logic cpl_valid,
input logic rid_bad, epoch_bad, tag_dead,
output logic [31:0] reject_count
);
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) reject_count <= '0;
else if (cpl_valid && (rid_bad || epoch_bad || tag_dead))
reject_count <= reject_count + 32'd1; // <-- one counter for three
endmodule
// Failing stimulus: a run containing foreign-RID Completions, late
// Completions, and Completions for freed tags.
// Golden: three counters — foreign, late, orphan — separately readable.
// This: one number. A reading of 8,371 is consistent with a healthy
// requester discarding late Completions (§13's correct row) AND
// with a misrouted fabric delivering foreign traffic.
// P31 fails.
// Observable consequence: §3's classes C and the late-Completion path
// become indistinguishable, and §15 case 4's primary evidence is gone.13. Measured Behaviour — Reclamation and Misdelivery
| configuration | issued | retired | timed out | partial | late seen | misdelivered | bytes wrong |
|---|---|---|---|---|---|---|---|
| epoch bumped and checked (correct) | 31,195 | 28,074 | 3,089 | 1,218 | 8,371 | 0 | 0 |
| epoch check removed | 38,208 | 35,466 | 2,710 | 1,381 | 39,574 | 39,574 | 1,907,096 |
| epoch bump removed | 38,208 | 35,466 | 2,710 | 1,381 | 0 | 39,574 | 1,907,096 |
| Tag never reclaimed | 366 | 334 | 32 | 12 | 0 | 0 | 0 |
Four readings.
Late Completions are normal. The correct configuration saw 8,371 and misdelivered none. A design that treats a late Completion as an error condition has misunderstood the mechanism; a design that treats it as impossible has a bug.
Partial timeouts are the majority case at 1,218 of 3,089 — 39.4%. §3's class D is not an edge case.
Rows two and three are byte-identical in damage and opposite in observability. Removing the check leaves the diagnostic working (39,574 late seen). Removing the bump leaves it reading zero. Same corruption, and one of the two configurations tells you nothing.
And the last row is the cost of the obvious "safe" choice. Never reclaiming a Tag eliminates misdelivery completely — and reduces throughput from 31,195 requests to 366, with zero free Tags at the end. Refusing to reclaim is not a conservative option; it is a different failure.
14. Measured Behaviour — Which Timer Reports
| ordering | outcome distribution |
|---|---|
| CTO 200 < watchdog 300 < driver 500 (correct) | completed normally 10,016 · hardware Completion Timeout 9,984 |
| driver 200 < watchdog 300 < CTO 500 (inverted) | completed normally 10,016 · driver timeout 9,984 |
| all three at 300 (ambiguous) | completed normally 13,239 · reported cause decided by sweep order |
The same 9,984 faults are attributed to a different agent purely by changing the limits. Nothing about the underlying failures changed — not their count, not their cause, not their timing.
This is why §15 case 1 is a configuration read rather than a trace analysis. A timeout statistic from a system whose timers are not strictly ordered is not a weak signal to be improved with more samples; it is an arbitrary one, and collecting more of it changes nothing.
And the ambiguous row shows the second-order effect. With all three equal, the "completed normally" count itself changed — from 10,016 to 13,239 — because the effective limit moved. A timer ordering error alters both the attribution and the rate, which makes before-and-after comparisons across a configuration change meaningless unless the ordering is verified first.
15. Verification — Mutations
Thirty-eight mutations. Every "Caught by" entry names a property from §11.
| # | Mutation | Symptom | Caught by |
|---|---|---|---|
| 1 | Table entry marked live without allocator update | tag double-issued | P1, P4 |
| 2 | Allocator frees a tag still live in the table | orphan Completions | P1 |
| 3 | Allocation does not clear got | inherits previous request's byte count | P2 |
| 4 | Allocation does not clear saw_any | class A misreported as class C (§3) | P2, P30 |
| 5 | Retire and abandon both fire for one entry | double free | P3 |
| 6 | Free count computed over a stale mask | allocator believes tags exist | P5 |
| 7 | Retire on any Completion arrival | early retire; 3 orphans per split read | P6, P7 |
| 8 | Retire on Completion count reaching a constant | wrong for every RCB and alignment (§5) | P6 |
| 9 | Byte accumulation saturates instead of wrapping into overrun detect | overrun invisible | P8 |
| 10 | age == LIMIT instead of age_next >= LIMIT | missed in 20.0% of trials (25.4 §15) | P9 |
| 11 | Single global age shared by all entries | timeouts burst at idle (counterexample E) | P10, P12 |
| 12 | Age not cleared at allocation | new request inherits an expired age | P10 |
| 13 | Age gated on link-active | the timer stops exactly when the link dies | P11 |
| 14 | Age gated on outstanding-nonzero globally | one hung request masked by traffic | P10 |
| 15 | Age wraps at full scale | the wait silently restarts; never expires | P13 |
| 16 | Timeout fires one tick early | healthy requests abandoned under load | P12 |
| 17 | Byte Count field ignored | a lost split Cpl is found at timeout, not at arrival | P14 |
| 18 | Byte Count compared against total rather than remaining | every Completion flags a violation | P14 |
| 19 | Byte Count violation counted but not reported | detected and discarded | P15 |
| 20 | Final Completion's Byte Count not checked | a truncated series looks complete | P16 |
| 21 | Requester ID not compared | another requester's Completion accepted | P17 |
| 22 | Epoch bumped at reissue rather than abandonment | a window where a late Cpl matches | P19, P21 |
| 23 | Epoch check removed | 39,574 misdelivered, 1,907,096 bytes (§13) | P18, P22 |
| 24 | Epoch bump removed | identical corruption, diagnostic reads zero (§13) | P19, P22 |
| 25 | Epoch bumped only on timeout, not on retirement | a very late Cpl from a retired request matches | P20 |
| 26 | Epoch width reduced to one bit | aliases after two reissues | P22 |
| 27 | Foreign, late and orphan merged into one counter | §3's classes become indistinguishable | P31 |
| 28 | Orphan Completions silently dropped | no evidence a freed tag received data | P31 |
| 29 | Timeout does not reclaim the tag | 366 issued vs 31,195 healthy (§13) | P23, P25 |
| 30 | Reclaim without abandoning the table entry | tag free and entry live simultaneously | P1, P23 |
| 31 | No liveness path for a wedged entry | one tag lost permanently per fault | P24 |
| 32 | Timer limits not strictly ordered | reported cause changes with ordering (§14) | P26, P28 |
| 33 | Two timer limits equal | cause decided by sweep order (§14) | P27 |
| 34 | Outer timer allowed to report while inner has expired | recovery races the link (§4) | P28 |
| 35 | First-timeout record overwritten by later timeouts | reports a consequence, not the cause | P29 |
| 36 | Recorded cause not derived from recorded bytes | instrument contradicts itself | P30 |
| 37 | Testbench generates no Completions delayed past the timeout | P18–P22 all vacuous | P32 (c3) |
| 38 | Testbench uses only single-Completion reads | class D unreachable; P7, P16 vacuous | P32 (c1, c8) |
Mutations 23 and 24 are the pair this chapter exists for. Identical damage; one leaves the diagnostic functional and the other blinds it. A mutation campaign that scores them equally has missed the more dangerous one, and only P22's independent oracle separates them.
Mutations 37 and 38 break the testbench, and their symptom is that everything passes.
16. Debugging
Case 1 — before analysing any timeout, read the three timer limits.
§14 measured that the reported cause changes with the ordering and nothing else. Establish the hardware Completion Timeout, the engine watchdog, and the driver timeout, and confirm they are strictly ordered innermost-to-outermost (P26, P27). If they are not, every timeout attribution you have is arbitrary — and if any two are equal, the reported cause is a property of the implementation's sweep order. Confidence: this is a configuration read, not an inference, and it either holds or it does not.
Case 2 — the request received nothing at all.
Class A or B, and they cannot be separated at the requester (§3). Both look identical: zero bytes, no matched Completion. This is where local instrumentation is exhausted, and it is one of the few legitimate reasons in Module 25 to reach for an analyzer (25.9) — you need evidence from the other end of the link. Before doing so, confirm saw_any is genuinely zero rather than uninitialised (mutation 4).
Case 3 — the request received part of its data.
Class D, and it is the majority case — §13 measured 1,218 of 3,089 timeouts, 39.4%. The path worked and then stopped, which eliminates address decode, routing, and a dead completer in one read. Look at the completer's ability to continue a split series under backpressure, and at 16.4 for completion-credit exhaustion. Confidence: high; the byte counter is unambiguous evidence.
Case 4 — the late-Completion counter is large.
That is not a fault. §13's correct configuration recorded 8,371 late Completions and misdelivered none — abandoning a request whose answer is in flight produces late Completions by construction. What matters is that they were discarded and counted, which the counter's existence demonstrates. Confirm the misdelivery oracle reads zero (P22), and check that the counter is not a merged one (mutation 27, counterexample F) — a single "rejects" number cannot distinguish this healthy case from a misrouted fabric.
Case 5 — the late-Completion counter reads zero and you suspect misdelivery anyway.
Suspect the epoch bump (§7, §13's third row, mutation 24). A counter that detects lateness by comparing epochs reads zero when the epochs stopped advancing — the instrument's failure and the fault are the same event. §13 measured 39,574 misdeliveries under exactly this condition with the counter at zero. The discriminating test does not use the epoch at all: stamp requests with a serial in the testbench, or in a system, verify that the epoch register for a tag actually changes across a timeout by reading it twice. Confidence: high, and this is the chapter's most important single check.
Case 6 — the requester stops issuing requests entirely.
Tag starvation. §13 measured the never-reclaim configuration issuing 366 requests against a healthy 31,195 with zero free Tags. Read the free-Tag count (P5, P25). If it is zero and static, tags are leaking — every timeout that does not reclaim loses one permanently (mutation 29). Confidence: high; the free count is direct evidence.
Case 7 — timeouts arrive in bursts when the system goes idle.
A shared global age counter (counterexample E, mutation 11). Every new allocation resets it, so under load nothing ages out; when traffic stops, everything expires together and is attributed to whatever happened to be outstanding then. The tell is the correlation with idle rather than with load — the opposite of what a real hang produces. Confidence: high, and the fix is per-entry aging (P10).
Case 8 — the timeout says the byte count was satisfied.
Class E (§3). The Completion arrived, matched, and the retirement path lost it — a full result FIFO, backpressure, or a buffer whose owner never took it. The link is not involved. 23.5 owns that path. This case is worth its own entry because it is the one where a timeout tells you nothing whatsoever about PCIe, and hours are routinely spent on traces before anyone reads the byte counter (P30).
Case 9 — a request outstanding across a link failure never times out.
The age counter is gated on link activity (counterexample D, mutation 13). The timer stops running under exactly the condition it exists to detect. Read the counter twice during the fault; if it is static, the gate is the bug. P11 is the property. Confidence: high, and this fault is invisible in any test where the link stays up.
Case 10 — the reported timeout cause contradicts the byte counts.
Do not believe the cause; check the instrument (P30, mutation 36). A record showing CTO_PARTIAL with zero bytes received, or CTO_NEVER_ANSWERED with a nonzero count, means the cause is being derived independently of the fields rather than from them. An instrument whose fields contradict each other is worse than no instrument, because it directs the investigation confidently in a wrong direction — 25.4 mutation 34's lesson in a new form.
Case 11 — everything passes and no timeouts have ever been observed in simulation.
Read the covers (P32, mutations 37 and 38). A testbench with no Completions delayed past the timeout satisfies P18 through P22 vacuously — §13's model required an explicit very-late traffic class to generate any at all. A testbench with only single-Completion reads makes class D unreachable. The absence of observed timeouts is a statement about the stimulus, not about the design.
17. Misconceptions
"A completion timeout means the completer is broken." It means the requester stopped waiting (§2). Five distinct causes produce the identical observation (§3), and one of them — class E — does not involve the link at all.
"The timeout report tells you what failed." It names the timer that fired first (§4, §14). The same fault population reported as "hardware Completion Timeout, 9,984" under one ordering and "driver timeout, 9,984" under another.
"Late Completions indicate a problem." §13's correct configuration saw 8,371 of them. They are the normal consequence of abandoning a request whose answer is in flight. The design's obligation is to discard and count them, not to prevent them.
"Our late-Completion counter reads zero, so we have none." Or the epoch stopped advancing (§7, §13's third row). That configuration misdelivered 1,907,096 bytes with the counter at zero.
"Not reclaiming the Tag is the safe choice." It reduced throughput from 31,195 requests to 366 with zero free Tags (§13). It is a different failure, not a conservative one.
"A read is answered by a Completion." It is answered by enough Completions to satisfy its byte count (13.3, §5). Retiring on arrival frees the Tag with data still in flight (counterexample A).
"Zero bytes received means nothing was sent." It means nothing was accepted. A Completion rejected for a foreign Requester ID or a stale epoch arrived and was discarded — which is class C, and it is distinguishable only if the rejection reasons are counted separately (§3, mutation 27).
"We should compare timeout rates before and after the change." Not across a timer-configuration change. §14 measured the "completed normally" count itself shifting from 10,016 to 13,239 purely from making the limits equal.
18. Understanding Check
Q1. A system reports that 90% of its failures are driver timeouts. What do you check before believing it?
The relative values of the three timers (§4, §14, §15 case 1). §14 ran one unchanged fault population through three orderings: with the hardware Completion Timeout innermost the same 9,984 faults were reported as hardware Completion Timeout; with the driver timeout innermost, as driver timeout. A timeout report names the timer that fired first, never the thing that broke. If the three limits are not strictly ordered innermost-to-outermost (P26), the attribution is arbitrary rather than merely noisy — and if any two are equal (P27), it is decided by sweep order, so collecting more samples cannot improve it.
Q2. Your late-Completion counter reads zero. Why might that be the worst possible reading?
Because the counter detects lateness by comparing epochs, and a missing epoch bump makes every epoch equal (§7, §13, mutation 24). §13 measured that configuration at 39,574 misdeliveries carrying 1,907,096 bytes — byte-identical to removing the epoch check — while the late-Completion counter read zero. The instrument's failure and the fault are the same event. Detecting it requires a check independent of the epoch entirely: a ground-truth request serial in verification (§10 Block 8, P22), or in a live system, reading a tag's epoch register across a timeout to confirm it actually changes.
Q3. A request timed out having received 192 of 256 bytes. What has that already ruled out?
Address decode, routing, and a dead completer (§3 class D, §5). The completer decoded the request, generated Completions, and at least one was routed back successfully — the path demonstrably worked and then stopped. §13 measured partial timeouts as 1,218 of 3,089, 39.4% — the majority case, not an edge case. The remaining investigation is about the completer continuing a split series under backpressure, and about completion credit (16.4). None of that is visible unless the outstanding table counts bytes rather than Completions.
Q4. Why is retiring a request when a Completion with its Tag arrives wrong in two different directions?
It retires early and it cannot detect a truncated series (§5, counterexample A, mutations 7 and 8). A read may be answered by several Completions (13.3); retiring on the first frees the Tag with data still in flight, and the remaining Completions arrive as orphans for a freed Tag. And because the request is already retired, the case where the last Completion never arrives cannot be detected at all — the requester reported success. The contract is the byte count, which is why P6 asserts on bytes and why a Completion-count expectation would be an invented constant (§1).
Q5. Your requester stops issuing after a while and the free-Tag count reads zero. What are the two candidate faults, and how do they differ?
Tags leaking on timeout, or the requester genuinely saturated. §13 measured the never-reclaim configuration issuing 366 requests against a healthy 31,195, ending with zero free Tags — every timeout that fails to reclaim loses one permanently (mutation 29, P23). The discriminator is whether the count is static or fluctuating: a saturated requester's free count oscillates as requests retire; a leaking one decays monotonically to zero and stays there. P25 is the liveness property that distinguishes them formally.
Q6. Why must the age counter not be gated on link activity?
Because a dead link is exactly the fault the timer exists to detect, and a gated timer stops counting when it occurs (§6, counterexample D, mutation 13, P11). The request never times out, the Tag is never reclaimed, and the requester waits indefinitely for a Completion that cannot arrive. The fault is invisible in any test where the link stays up, which is every functional test — so it survives to silicon and appears only in error-injection or field conditions.
Q7. Merging "foreign Requester ID", "stale epoch" and "tag not live" into one rejection counter loses what, specifically?
§3's classification, and §15 case 4's primary evidence (mutation 27, counterexample F, P31). A merged reading of 8,371 is consistent with §13's correct configuration — a healthy requester discarding late Completions — and equally consistent with a misrouted fabric delivering another requester's traffic. Those demand opposite investigations, and no amount of further analysis of a merged counter can separate them, because the distinguishing information was discarded at the moment it existed.
Q8. Your timeout instrument reports CTO_PARTIAL with zero bytes received. What is the fault?
The instrument (P30, mutation 36, §15 case 10). CTO_PARTIAL means 0 < got < want by definition, so a record showing zero bytes contradicts its own cause field — which means the cause is being derived independently of the recorded fields rather than from them. An instrument whose fields contradict each other is worse than no instrument, because it directs an investigation confidently in a wrong direction. Check the recorder before anything it reports about the design, exactly as 25.5 §15 case 7 required for the first-rejection recorder.
19. Module 25 So Far
Seven chapters, and this one adds a failure mode the previous six did not have.
| Chapter | Asks |
|---|---|
| 25.1 Debugging Overview | which layer, and what is the last provable event? |
| 25.2 Enumeration Failures | where did the configuration conversation stop? |
| 25.3 Link Training Failures | which state is it in, and which exit is missing? |
| 25.4 LTSSM Issues | which contract inside the state is wrong? |
| 25.5 BAR Problems | who disagrees about owning this address? |
| 25.6 DMA Failures | where did ownership stop moving? |
| 25.7 (this) | which timer fired, and was it the right one? |
Every chapter before this one debugged a fault. This one debugs a recovery. The timeout is the design working correctly — a bounded wait, correctly bounded — and the damage comes from what happens afterwards. Reclamation is mandatory and reclamation is what creates the misdelivery window, and §13 measured both horns: 366 requests if you never reclaim, 1,907,096 wrong bytes if you reclaim without an epoch.
The chapter's most transferable result is not about timeouts at all. §13's third row showed a diagnostic reporting zero because the mechanism it derives from had stopped working. Any instrument built on the mechanism it checks is blind precisely where that mechanism fails — and the fix is always the same shape: an oracle with no shared machinery, which in verification is a ground-truth serial and in the field is a second, independent observation.
That result reframes 25.6 §13 in hindsight. Both counterexamples there were monitors rather than engines, and both failed for a related reason: they measured a quantity derived from the thing they were watching. A one-sided byte check, an aggregate completion counter and an epoch-derived lateness counter are the same mistake three times.
25.8 inherits the hardest version of the question. A timeout at least fires. A credit deadlock produces no event at all — no error, no expiry, no counter reaching a limit. The link is simply, permanently, waiting, and every pool reads a value that a busy healthy link reads too.