PCIe · Module 24
Error Injection — One Contract at a Time, and Prove It Recovers
A useful negative test corrupts exactly one rule and leaves everything else legal. The harder half is what comes after: detection is not recovery, and a design that reports the fault but leaks a Tag has failed the test it appeared to pass.
Module 24 built observers. 24.2 bound properties, 24.3 tracked transactions, 24.4 measured reach, 24.6 composed them.
Every one of them has branches that no correct traffic can reach. The terminal-status path in the scoreboard, the credit-exhaustion cover property, the error bins — all of it is dead code until something deliberately misbehaves.
1. Sources, Scope, and What This Chapter Refuses to Fabricate
2. One Contract at a Time
The difference between a negative test and noise:
| Test | What you learn |
|---|---|
| legal packet + wrong integrity value | whether the integrity checker fires, and what recovery follows |
| random bits on the wire | that the design dislikes garbage |
The discipline, as a procedure:
1. establish a baseline legal transaction that PASSES
2. identify the single field or event that carries the contract
3. corrupt exactly that, after everything else is computed
4. observe WHICH checker fires FIRST
5. observe the required recovery or containment
6. prove no unrelated resource leaked
7. prove the NEXT clean transaction succeedsStep 3's "after everything else is computed" is not a detail. §4 shows the specific way it goes wrong: corrupt the payload before the integrity value is computed and the integrity value is computed over the corrupted payload — so it is valid, the checker correctly says nothing, and the test silently proves nothing.
And step 4's word "first" matters. With several checkers armed, the useful evidence is which one fires first — Chapter 21.4 §11 measured last-error reporting naming a non-root-cause 96.8% of the time, and an injected fault produces exactly the cascade that exploits it.
3. The Error Matrix by Layer
Faults belong to layers, and the layer determines where the test must inject and where it must watch.
| Layer | Fault | Injected where | First checker |
|---|---|---|---|
| Transaction | request no Function claims | at the requester | Unsupported Request path (21.1 §6) |
| Completion with an unknown identity | at the completer/VIP | the context matcher (23.5 §3) | |
| Completion with an error status | at the completer/VIP | the terminal path (13.2) | |
| byte count inconsistent with the request | at the completer | scoreboard coverage (24.3 §5) | |
| Data Link | bad integrity value | after computation (§4) | the integrity checker (14.4) |
| sequence discontinuity | in the transmitted sequence field | the sequence checker | |
| acknowledgement withheld | at the link partner | the replay mechanism | |
| Flow Control | issue without credit — internal | inside the DUT's scheduler | SVA, before the wire (24.2 §10) |
| incorrect credit advertisement — peer | at the VIP/partner | the DUT's credit accounting | |
| Link | expected training event withheld | at the partner/PHY model | LTSSM progress (25.3) |
The Flow Control rows are deliberately split, and the distinction is the most important one in the table.
An internal credit fault is a DUT mutation: the scheduler issues although the credit manager says zero. The correct outcome is that an assertion fires before anything reaches the wire — nothing illegal is ever transmitted, and the test proves the internal contract.
A peer credit fault is an environment behaviour: the link partner advertises credit incorrectly. That tests the DUT's own accounting, and it is only injectable if the environment can model it.
Conflating them produces a test that cannot be interpreted — you do not know whether you proved the DUT's scheduler or its accounting, and §10's mutation 16 is exactly that confusion.
4. Injection Is Not Corruption
5. Detection Is Not Recovery
6. The Waveform
Arm, inject once, detect, recover — and the clean packet that follows
10 cyclesFour things to read out of the figure.
inject_now is one cycle wide. The fault lands on exactly one packet (§2); a corruptor still asserted at cycle 3 would damage the next one too, and the recovery test would then run against a second fault.
first_err is sticky from cycle 3 onward. Later cascading errors must not overwrite it (23.6 §7) — the whole value of the record is that it names the first cause.
recovering has a definite end. Cycle 6 is where the design says it is finished; without that event a watchdog cannot distinguish "recovering" from "hung" (§7).
And cycles 7–8 are the test. A trace that stops at cycle 6 has demonstrated detection. The clean packets after it are what demonstrate recovery (§5).
7. RTL — The Injection and Recovery Harness
// VERIFICATION-ONLY. Fault taxonomy and control.
// LOCAL VERIFICATION POLICY -- PCIe defines none of these values (§1).
package fault_pkg;
parameter int TAGS = 8;
parameter int TAG_W = (TAGS <= 1) ? 1 : $clog2(TAGS);
parameter int SEQ_W = 12;
parameter int CNT_W = 32;
typedef enum logic [3:0] {
FI_NONE = 4'd0,
FI_BAD_INTEGRITY = 4'd1, // Data Link: corrupt the COMPUTED value (§4)
FI_SEQ_CORRUPT = 4'd2, // Data Link: sequence discontinuity
FI_DROP_ACK = 4'd3, // Data Link: withhold acknowledgement
FI_BAD_CPL_ID = 4'd4, // Transaction: wrong (Requester ID, Tag)
FI_CPL_ERR_STATUS = 4'd5, // Transaction: terminal status (13.2)
FI_CREDIT_OVERSPEND= 4'd6, // INTERNAL mutation -- SVA must fire (§3)
FI_DROP_TRAIN_EVENT= 4'd7 // Link: withhold a training event
} fault_kind_e;
typedef enum logic [1:0] { FM_ONESHOT, FM_REPEAT, FM_DISABLED } fault_mode_e;
typedef struct packed {
logic armed;
fault_kind_e kind;
fault_mode_e mode;
logic [CNT_W-1:0] target_event; // fire on the Nth qualifying EVENT
logic [2:0] target_port;
logic [2:0] target_function;
} fault_cfg_t;
// Which checker SHOULD fire first for each fault. The test asserts this
// mapping -- a fault that trips the wrong checker is §10's mutation 4.
function automatic string expected_checker(input fault_kind_e k);
case (k)
FI_BAD_INTEGRITY: return "dl_integrity";
FI_SEQ_CORRUPT: return "dl_sequence";
FI_DROP_ACK: return "dl_replay";
FI_BAD_CPL_ID: return "tl_unknown_identity";
FI_CPL_ERR_STATUS: return "tl_terminal_status";
FI_CREDIT_OVERSPEND: return "sva_credit_reservation";
FI_DROP_TRAIN_EVENT: return "ltssm_progress";
default: return "none";
endcase
endfunction
endpackageimport fault_pkg::*;
// VERIFICATION-ONLY. One-shot arm and trigger (§2).
// The event counter advances on the ACCEPTED event only -- counting offers
// would fire the fault on a packet that was never sent (§10, mutation 18).
// §9: one-shot fires exactly 1; a repeating variant fired 2,987.
module fault_trigger (
input logic clk,
input logic rst_n,
input fault_cfg_t cfg,
input logic event_fire, // a qualifying event TRANSFERRED
input logic fault_consumed, // the DUT finished recovering
output logic inject_now,
output fault_kind_e active_kind,
output logic pending,
output logic [CNT_W-1:0] event_count,
output logic err_fire_unarmed // sticky: fired without an arm
);
logic [CNT_W-1:0] cnt_q;
logic armed_q, pend_q, e_q;
assign event_count = cnt_q;
assign active_kind = cfg.kind;
assign pending = pend_q;
assign err_fire_unarmed = e_q;
// SAME-CYCLE RULE (§8): arm and the target event in one cycle FIRE. The
// arm is a level, and the count is compared against the next value.
logic [CNT_W-1:0] cnt_next;
assign cnt_next = event_fire ? (cnt_q + CNT_W'(1)) : cnt_q;
assign inject_now = armed_q && event_fire
&& (cnt_next == cfg.target_event)
&& (cfg.mode != FM_DISABLED);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
cnt_q <= '0; armed_q <= 1'b0; pend_q <= 1'b0; e_q <= 1'b0;
end else begin
cnt_q <= cnt_next;
// Arming is edge-insensitive: a level that is high enables firing.
if (cfg.armed && !armed_q) armed_q <= 1'b1;
if (inject_now) begin
pend_q <= 1'b1;
if (cfg.mode == FM_ONESHOT) armed_q <= 1'b0; // FIRES EXACTLY ONCE
end
if (fault_consumed) pend_q <= 1'b0;
// A fire with no arm cannot happen; prove it rather than trust it.
if (event_fire && (cnt_next == cfg.target_event) && !armed_q && cfg.armed)
e_q <= 1'b1;
end
end
endmoduleimport fault_pkg::*;
// VERIFICATION-ONLY. Data Link integrity corruptor (§4).
// It mutates the COMPUTED value, never the payload -- corrupting the
// payload first makes the computed value CORRECT for the corrupted data,
// and the checker rightly says nothing (§10, mutation 14).
module dl_integrity_injector #(parameter int W = 32) (
input logic inject_now,
input logic [W-1:0] integrity_in, // already computed over the packet
input logic [$clog2(W)-1:0] bit_sel,
output logic [W-1:0] integrity_out,
output logic corrupted
);
// Exactly one bit, exactly one packet. Everything else is untouched, so
// the only broken contract is the integrity check (P8, P9).
assign integrity_out = inject_now ? (integrity_in ^ (W'(1) << bit_sel))
: integrity_in;
assign corrupted = inject_now;
endmoduleimport fault_pkg::*;
// VERIFICATION-ONLY. Sequence-number injector.
// The WIDTH and the legal progression are the Data Link chapter's (14.4);
// this block only perturbs the transmitted value by a parameterized delta,
// so nothing here asserts a sequence-number rule.
module dl_sequence_injector (
input logic inject_now,
input logic [SEQ_W-1:0] seq_in,
input logic signed [3:0] delta, // +1 skip, -1 duplicate, etc.
output logic [SEQ_W-1:0] seq_out,
output logic corrupted
);
assign seq_out = inject_now ? (seq_in + SEQ_W'(delta)) : seq_in;
assign corrupted = inject_now;
endmoduleimport fault_pkg::*;
// VERIFICATION-ONLY. Wrong-identity Completion injector (§3).
// The packet stays perfectly legal -- correct format, correct length, a
// valid identity. Only the WRONG one. A conformance checker sees nothing;
// the project scoreboard's matcher must catch it (24.5 §6's distinction).
module tl_cpl_identity_injector (
input logic inject_now,
input logic [15:0] rid_in,
input logic [TAG_W-1:0] tag_in,
input logic [TAG_W-1:0] wrong_tag,
output logic [15:0] rid_out,
output logic [TAG_W-1:0] tag_out,
output logic corrupted
);
assign rid_out = rid_in; // still a legal Requester ID
assign tag_out = inject_now ? wrong_tag : tag_in;
assign corrupted = inject_now;
endmoduleimport fault_pkg::*;
// SYNTHESIZABLE debug hook. First-error capture, injection-aware.
// It records WHICH checker fired first AND what fault was pending, so a
// cascade cannot bury the mapping the test is asserting (§2, §5).
module injection_first_error (
input logic clk,
input logic rst_n,
input logic [7:0] checker_id, // which checker fired
input logic checker_fire,
input fault_kind_e pending_kind,
input logic pending_valid,
input logic [15:0] pkt_rid,
input logic [TAG_W-1:0] pkt_tag,
input logic [3:0] link_state,
input logic [CNT_W-1:0] cycle,
input logic clear,
output logic captured,
output logic [7:0] first_checker,
output fault_kind_e first_fault,
output logic [15:0] first_rid,
output logic [TAG_W-1:0] first_tag,
output logic [3:0] first_link_state,
output logic [CNT_W-1:0] first_cycle,
output logic [CNT_W-1:0] later_errors
);
logic cap_q; logic [7:0] chk_q; fault_kind_e fk_q;
logic [15:0] rid_q; logic [TAG_W-1:0] tag_q; logic [3:0] ls_q;
logic [CNT_W-1:0] cyc_q, later_q;
assign captured=cap_q; assign first_checker=chk_q; assign first_fault=fk_q;
assign first_rid=rid_q; assign first_tag=tag_q; assign first_link_state=ls_q;
assign first_cycle=cyc_q; assign later_errors=later_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n || clear) begin
cap_q<=1'b0; chk_q<='0; fk_q<=FI_NONE; rid_q<='0; tag_q<='0;
ls_q<='0; cyc_q<='0; later_q<='0;
end else if (checker_fire) begin
if (!cap_q) begin
// LATCH ONCE. 21.4 §11 measured last-write-wins naming a
// non-root-cause 96.8% of the time.
cap_q<=1'b1; chk_q<=checker_id;
fk_q <= pending_valid ? pending_kind : FI_NONE;
rid_q<=pkt_rid; tag_q<=pkt_tag; ls_q<=link_state; cyc_q<=cycle;
end else if (!(&later_q)) later_q <= later_q + CNT_W'(1);
end
end
endmoduleimport fault_pkg::*;
// VERIFICATION-ONLY. Recovery watchdog (§5).
// A TEST BUDGET, not a PCIe timer. It measures how long the DUT stayed in
// its declared recovery state and flags a design that detected but never
// finished. Labelled deliberately -- 24.3 §8's distinction.
module recovery_watchdog #(parameter int unsigned BUDGET = 4096) (
input logic clk,
input logic rst_n,
input logic fault_injected, // one-cycle pulse
input logic recovery_done, // the DUT's declared completion event
output logic recovery_pending,
output logic [CNT_W-1:0] recovery_cycles,
output logic watchdog_expired, // sticky
output logic [CNT_W-1:0] worst_recovery
);
logic pend_q, exp_q;
logic [CNT_W-1:0] cyc_q, worst_q;
assign recovery_pending = pend_q;
assign recovery_cycles = cyc_q;
assign watchdog_expired = exp_q;
assign worst_recovery = worst_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin pend_q<=1'b0; cyc_q<='0; exp_q<=1'b0; worst_q<='0; end
else begin
// SAME-CYCLE: injection and completion in one cycle means the design
// recovered immediately -- a legal zero-cycle recovery, not an error.
if (recovery_done && pend_q) begin
pend_q <= 1'b0;
if (cyc_q > worst_q) worst_q <= cyc_q;
end else if (fault_injected && !pend_q) begin
pend_q <= 1'b1; cyc_q <= '0;
end else if (pend_q) begin
cyc_q <= cyc_q + CNT_W'(1);
if (cyc_q >= CNT_W'(BUDGET)) exp_q <= 1'b1; // test budget elapsed
end
end
end
endmoduleimport fault_pkg::*;
// SYNTHESIZABLE debug hook. THE RECOVERY INVARIANT MONITOR (§5).
// This is the block that turns "the checker fired" into "the design
// recovered". §9: every fault kind returned to 8/8 free Tags.
module recovery_invariant_monitor (
input logic clk,
input logic rst_n,
input logic [TAGS-1:0] tag_free_map,
input logic [TAGS-1:0] tag_owned_map,
input logic [CNT_W-1:0] buffer_reserved,
input logic replay_pending,
input logic fault_injected,
input logic recovery_done,
input logic clean_txn_completed, // a LEGAL transaction after recovery
output logic err_tag_leak,
output logic err_buffer_leak,
output logic err_replay_stuck,
output logic err_no_clean_after_recovery, // sticky
output logic recovered_cleanly
);
logic [TAGS-1:0] baseline_free_q;
logic [CNT_W-1:0] baseline_buf_q;
logic seen_fault_q, seen_recovery_q, seen_clean_q, e_q;
assign err_tag_leak = seen_recovery_q && (tag_free_map != baseline_free_q);
assign err_buffer_leak = seen_recovery_q && (buffer_reserved != baseline_buf_q);
assign err_replay_stuck = seen_recovery_q && replay_pending;
assign err_no_clean_after_recovery = e_q;
assign recovered_cleanly = seen_recovery_q && seen_clean_q
&& !err_tag_leak && !err_buffer_leak && !err_replay_stuck;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
baseline_free_q <= {TAGS{1'b1}}; baseline_buf_q <= '0;
seen_fault_q<=1'b0; seen_recovery_q<=1'b0; seen_clean_q<=1'b0; e_q<=1'b0;
end else begin
// Snapshot the resting state BEFORE the fault -- the comparison the
// whole recovery test depends on.
if (!seen_fault_q && !fault_injected) begin
baseline_free_q <= tag_free_map;
baseline_buf_q <= buffer_reserved;
end
if (fault_injected) seen_fault_q <= 1'b1;
if (recovery_done) seen_recovery_q <= 1'b1;
if (clean_txn_completed) seen_clean_q <= 1'b1;
// Detection without a subsequent clean transaction is a FAILED test,
// even though every checker behaved correctly (§5).
if (seen_recovery_q && !seen_clean_q && (tag_owned_map == '0)) e_q <= 1'b1;
end
end
endmoduleClassification: six verification-only, two synthesizable debug hooks.
Failure — eight. Corrupting before the integrity value is computed (§4). A fault that outlives its packet. Counting offers rather than accepted events. A one-shot that repeats (2,987 fires, §9). First-error overwritten by the cascade. A recovery watchdog labelled as a PCIe timer. No baseline snapshot, so leaks are invisible. And no clean transaction after recovery — the test that looks passed and is not.
8. Same-Cycle Audit and Assertions
// ---- INJECTION DISCIPLINE (§2, §4) ------------------------------
// P1: no injection unless armed.
property p_no_inject_unarmed;
@(posedge clk) disable iff (!rst_n) inject_now |-> cfg.armed;
endproperty
// P2: injection requires an ACCEPTED event, never an offer.
property p_inject_on_transfer;
@(posedge clk) disable iff (!rst_n) inject_now |-> event_fire;
endproperty
// P3: the fault lands on the EXACT target event.
property p_exact_target_event;
@(posedge clk) disable iff (!rst_n)
inject_now |-> (event_count + CNT_W'(1) == cfg.target_event);
endproperty
// P4: a one-shot fault fires exactly once. §9: a repeating variant fired
// 2,987 times against the correct design's 1.
property p_oneshot_fires_once;
@(posedge clk) disable iff (!rst_n)
(inject_now && (cfg.mode == FM_ONESHOT)) |=> !inject_now;
endproperty
// P5: a second arm while a fault is pending is refused.
property p_no_rearm_while_pending;
@(posedge clk) disable iff (!rst_n)
(pending && cfg.armed) |-> !inject_now;
endproperty
// P6: firing without an arm is REPORTED, never silent.
property p_unarmed_fire_flagged;
@(posedge clk) disable iff (!rst_n)
(event_fire && (event_count + CNT_W'(1) == cfg.target_event)
&& !armed_q && cfg.armed) |=> err_fire_unarmed;
endproperty
// P7: the event counter advances only on an accepted event.
property p_event_count_on_transfer;
@(posedge clk) disable iff (!rst_n)
(event_count != $past(event_count)) |-> $past(event_fire);
endproperty
// P8: the corruptor mutates the COMPUTED value only -- the payload is
// untouched, so exactly one contract is broken (§4).
property p_payload_untouched;
@(posedge clk) disable iff (!rst_n)
inject_now |-> $stable(packet_payload);
endproperty
// P9: a non-target packet is bit-identical to the baseline.
property p_non_target_unmodified;
@(posedge clk) disable iff (!rst_n)
(!inject_now && pkt_valid) |-> (integrity_out == integrity_in);
endproperty
// P10: exactly one fault kind is active at a time.
property p_single_active_fault;
@(posedge clk) disable iff (!rst_n)
pending |-> (active_kind != FI_NONE);
endproperty
// ---- DETECTION (§2, §3) -----------------------------------------
// P11: the injected fault trips the EXPECTED checker, not another.
property p_expected_checker_fires;
@(posedge clk) disable iff (!rst_n)
(captured && (first_fault != FI_NONE))
|-> (checker_name(first_checker) == expected_checker(first_fault));
endproperty
// P12: a clean packet never trips a checker. §9: the baseline fired NOTHING.
property p_no_false_detection;
@(posedge clk) disable iff (!rst_n)
(pkt_valid && !inject_now && !pending) |-> !checker_fire;
endproperty
// P13: an injected packet is counted as ONE transaction, not two.
property p_injected_counted_once;
@(posedge clk) disable iff (!rst_n)
inject_now |=> (txn_count == $past(txn_count) + 1);
endproperty
// P14: the first error is sticky.
property p_first_error_sticky;
@(posedge clk) disable iff (!rst_n)
(captured && !clear) |=> (captured && $stable(first_checker)
&& $stable(first_fault));
endproperty
// P15: later errors are counted, never stored over the first.
property p_later_errors_counted;
@(posedge clk) disable iff (!rst_n)
(captured && checker_fire && !clear) |=> $stable(first_cycle);
endproperty
// ---- RECOVERY (§5) ----------------------------------------------
// P16: recovery is pending for a bounded interval under the TEST BUDGET.
// This bounds a verification watchdog, not a PCIe timer (§1).
property p_recovery_bounded;
@(posedge clk) disable iff (!rst_n)
recovery_pending |-> (recovery_cycles <= CNT_W'(BUDGET)) || watchdog_expired;
endproperty
// P17: every Tag returns to the pre-fault baseline. §9: 8/8 for every kind.
property p_no_tag_leak_after_recovery;
@(posedge clk) disable iff (!rst_n)
(recovery_done && (tag_owned_map == '0)) |=> !err_tag_leak;
endproperty
// P18: buffer reservations return to baseline.
property p_no_buffer_leak;
@(posedge clk) disable iff (!rst_n)
(recovery_done && (tag_owned_map == '0)) |=> !err_buffer_leak;
endproperty
// P19: replay state clears -- a replayed packet is not a new transaction.
property p_replay_state_clears;
@(posedge clk) disable iff (!rst_n)
recovery_done |=> !err_replay_stuck;
endproperty
// P20: THE PROPERTY MOST OFTEN MISSING. A clean transaction after recovery
// completes normally. Detection without this is a failed test (§5).
property p_clean_after_recovery;
@(posedge clk) disable iff (!rst_n)
recovery_done |-> s_eventually clean_txn_completed;
endproperty
// P21: the error does not repeat indefinitely from one fault.
property p_no_error_storm;
@(posedge clk) disable iff (!rst_n)
(later_errors > CNT_W'(STORM_LIMIT)) |-> error_storm_reported;
endproperty
// ---- CROSS-LAYER AND HYGIENE ------------------------------------
// P22: reset clears the arm and any pending fault.
property p_reset_clears_injection;
@(posedge clk)
(!rst_n) |=> (!inject_now && !pending && !captured);
endproperty
// P23: a wrong-identity Completion never retires an INNOCENT context
// (23.5 §3's matcher must report unknown rather than default).
property p_wrong_identity_no_innocent_retire;
@(posedge clk) disable iff (!rst_n)
(inject_now && (active_kind == FI_BAD_CPL_ID)) |=> !unrelated_ctx_retired;
endproperty
// P24: an internal credit overspend is caught BEFORE the wire (§3).
property p_credit_fault_caught_internally;
@(posedge clk) disable iff (!rst_n)
(inject_now && (active_kind == FI_CREDIT_OVERSPEND)) |-> !tx_wire_valid;
endproperty
// P25: the credit ledger never underflows even under injection.
property p_ledger_no_underflow_under_injection;
@(posedge clk) disable iff (!rst_n)
(avail_credit <= advertised_credit);
endproperty
// P26: the packet owner is held through EOP even during injection --
// corruption does not release arbitration (21.2 §11's contract).
property p_packet_lock_holds_under_injection;
@(posedge clk) disable iff (!rst_n)
(owner_valid && inject_now) |=> (owner_valid && $stable(owner));
endproperty
// P27: injection logic never drives functional traffic.
property p_injector_non_functional;
@(posedge clk) disable iff (!rst_n)
$stable({pkt_valid, pkt_ready}) or !$stable({event_count, later_errors});
endproperty
// P28: EVIDENCE -- a fault was actually injected in this run.
property c_fault_injected;
@(posedge clk) disable iff (!rst_n) inject_now;
endproperty
// P29: EVIDENCE -- a clean transaction followed a recovery.
property c_clean_after_recovery_reached;
@(posedge clk) disable iff (!rst_n) (recovery_done ##[1:$] clean_txn_completed);
endpropertyTwenty-nine properties. P17–P20 are the recovery contract, and P20 is the one this chapter argues is most often absent. P28 and P29 are the cover properties — without them an injection suite can report a clean run because it never actually injected anything (24.2 §4's vacuity measured at 60,205 versus 0).
9. Measured Behaviour
10. Verification — Mutations
| # | Mutation | Symptom | Caught by |
|---|---|---|---|
| 1 | Fault fires before the arm | corruption in the baseline phase (§9) | P1, P6 |
| 2 | One-shot fires repeatedly | 2,987 fires; recovery never tested cleanly | P4 |
| 3 | Corruption persists into the next packet | recovery tested against a second fault | P9 |
| 4 | The fault trips a different checker than expected | the mapping the test asserts is wrong | P11 |
| 5 | A clean packet trips a checker | false failures; the baseline is unusable | P12 |
| 6 | The injected packet counted as two transactions | conservation breaks in the scoreboard | P13 |
| 7 | First error overwritten by the cascade | root cause lost 96.8% of the time | P14, P15 |
| 8 | Fault state survives reset | the next test starts pre-corrupted | P22 |
| 9 | Tag leaked after a bad Completion | wedges after ~14 jobs (23.3 §14) | P17 |
| 10 | Buffer reservation leaked after an error | throughput decays run over run | P18 |
| 11 | Replay state never retires | the retry path stays armed forever | P19 |
| 12 | The recovery watchdog called a PCIe timer | an unobserved mechanism claimed (§1) | review |
| 13 | Multi-field random corruption | the failure names nothing (§2) | review |
| 14 | Payload corrupted before the integrity value is computed | the value is valid; the test silently passes (§4) | P8 |
| 15 | The "sequence" fault mutates a transaction-layer field | the test's name and its layer disagree (§3) | review |
| 16 | Internal and peer credit faults conflated | you cannot tell scheduler from accounting (§3) | P24 |
| 17 | The error path never exercised with a TX stall | stall-only ownership bugs unreached | P26 |
| 18 | Injection triggered on valid rather than the transfer | the fault lands on a packet never sent | P2, P7 |
| 19 | Target event counter off by one | the fault lands on the wrong packet | P3 |
| 20 | No clean transaction attempted after recovery | detection mistaken for recovery (§5) | P20, P29 |
| 21 | VIP reports the first error; the project checker ignores it | two observers disagree and neither is read (24.5 §10) | review |
| 22 | Expected errors suppressed too broadly for the whole run | real failures after the injected one are hidden | review |
| 23 | Reset used to clear residue before the leak check | the leak is masked by the cleanup | P17, P18 |
| 24 | The same fault injected in every sequence | one fault class tested, the rest never | review |
| 25 | An impossible behaviour presented as a legal PCIe scenario | the DUT fails a test the protocol forbids | review |
| 26 | A wrong-identity Completion retires an innocent context | a valid transaction corrupted by the test | P23 |
| 27 | A replayed packet counted as new application payload | efficiency and throughput inflate (22.5 §6) | P19 |
| 28 | The error result pulse lost under backpressure | the failure is never reported | P21 |
| 29 | The checker and the injector share a helper function | both wrong together (24.1 §5) | review |
| 30 | Credit ledger underflows under injection | the engine then believes it owns huge credit | P25 |
| 31 | The packet owner released when corruption is detected | header and payload split across owners | P26 |
| 32 | No cover property on the injection itself | a suite that never injected reports clean (§8) | P28 |
| 33 | Injection logic gates a functional signal | the measured design is not the shipping one | P27 |
Two counterexamples worth stating explicitly.
Mutation 14 is the one that makes a whole test suite worthless while every run is green. The injector corrupts a payload byte, and then the harness computes the integrity value over the packet it is about to send. The value is correct for the corrupted payload. The receiver checks it, finds it valid, and accepts the packet. No checker fires, the test reports "error injected, no failure detected" — and if the expectation is written loosely, that reads as a pass. §7's corruptor operates on the computed value for exactly this reason, and P8 asserts the payload is untouched.
Mutation 20 is the methodological one, and §9 is built to expose it. A suite that injects, observes the checker, and ends has verified detection. Chapter 23.3 §14's engine passed that test and wedged after 14 jobs because the error path leaked a Tag. The fix costs one more transaction at the end of each test — and P20 plus the baseline comparison in §7's monitor is what turns it into an assertion rather than a habit.
11. Debugging the Testbench
Symptom — the fault is injected and no checker fires.
Suspect the injection before the DUT (§4, mutation 14). Confirm the corruption reached the wire: read the injector's corrupted output and compare the transmitted value against the baseline. The most common cause is corrupting before the protected value is computed, which makes the packet internally consistent.
Symptom — the checker fires one packet later than expected.
Target-event off-by-one (mutation 19), or the counter is advancing on offers rather than transfers (mutation 18). event_count at the moment of injection is the evidence — P3 pins it.
Symptom — the expected error appears and the design wedges afterwards.
Detection without recovery (§5). Read the recovery invariant monitor: err_tag_leak, err_buffer_leak, err_replay_stuck. §9's right-hand columns are the shape of a healthy result; a wedge means one of those three is set.
Symptom — a different checker fires first than the one the test expects.
Either the fault is in the wrong layer (mutation 15) or the DUT genuinely detects it earlier. Both are findings. The first-error record names the checker and the pending fault together, so the mapping is readable directly — and if the DUT's earlier detection is legitimate, the expectation in expected_checker() is what needs updating.
Symptom — the test passes without backpressure and fails with it. The error path has never been exercised under stall (mutation 17). Every ownership defect in Modules 22–23 needed a stall to appear (24.1 §7). Add a stalling consumer to the recovery phase specifically, not just to the traffic phase.
Symptom — reset leaves a fault armed. Mutation 8. The next test begins pre-corrupted, and its baseline is not clean — which invalidates every comparison in §9. P22 asserts the clear, and the symptom is a suite where test order changes results.
Symptom — one fault produces thousands of errors.
An error storm (P21). Distinguish a genuine cascade from a repeating injection: read the one-shot's pending and event_count. §9 measured a repeating trigger firing 2,987 times — which looks exactly like a design that cannot recover.
Symptom — the project scoreboard fails while the protocol checker reports successful recovery. They are answering different questions (24.5 §6). The link recovered; the transaction did not — most often a Tag or context that the recovery path did not release. This is the wrong-identity injection's signature, and P23 is the property.
12. Misconceptions
"Error injection means flipping random bits." It means breaking exactly one contract and leaving everything else legal (§2).
"Corrupt the payload — that's the data." Then the protecting value is computed over the corruption and the packet is consistent (§4).
"If the checker fired, the test passed." Detection is half the test (§5).
"The design reported the error, so it recovered." Chapter 23.3 §14's engine reported and wedged after 14 jobs.
"A recovery timer is the Completion Timeout." It is a test budget unless it is the actual mechanism (§1, §7).
"One-shot versus repeat is a detail." 1 fire versus 2,987 (§9).
"Inject on valid — the packet is right there." The packet may never be sent (P2).
"An internal credit bug and a bad peer are the same test." One tests the scheduler, the other the accounting (§3).
"Suppress expected errors for the whole test." Then the failures after the injected one are hidden too (mutation 22).
"Clear the residue with a reset before checking." That masks the leak the test exists to find (mutation 23).
"A replayed packet is new traffic." It is the same payload again (22.5 §6).
"If the suite is green, faults were injected." Not without a cover property on the injection itself (P28).
13. Understanding Check
Q1. Your bad-integrity test injects a fault, the checker never fires, and the suite reports a pass. What happened? Almost certainly the corruption preceded the computation (§4, mutation 14). The protecting value was computed over the already-corrupted packet, making it internally consistent — so the receiver correctly accepted it. The injector must mutate the computed value, after everything else is final, and P8 asserts the payload is untouched by injection.
Q2. All five fault kinds fire their expected checkers. What have you still not tested? Recovery (§5). The middle of §9's table would look identical for a design that leaks a Tag on every errored transaction. The right-hand columns — return to 8/8 free Tags and clean transactions afterwards — are the other half, and 23.3 §14 measured what their absence costs: a wedge after 14 jobs.
Q3. Why does the trigger count events rather than cycles, and why on the transfer? Because a fault must land on a specific packet to be interpretable (§2). Counting cycles lands it wherever the pipeline happens to be; counting offers lands it on a packet that may never be sent, so the corruption is discarded and nothing is tested. P2 and P7 pin both, and §9 shows the injector waiting for a packet of the targeted class — which is why 1,965 injections produced 787 integrity errors.
Q4. An internal credit overspend and a peer advertising credit incorrectly — why must these be separate tests? Because they prove different contracts (§3). The internal mutation must be caught by an assertion before anything reaches the wire (P24) — nothing illegal is transmitted, and the test proves the scheduler honours the reservation. The peer fault tests the DUT's own credit accounting. Conflating them yields a failure you cannot attribute to either.
Q5. Your recovery watchdog expires at 4,096 cycles. Is that a Completion Timeout violation? No. It is a test budget (§7). The watchdog measures how long the DUT stayed in its declared recovery state and flags a design that never finished — a true statement about the simulation. A Completion Timeout is a device mechanism with its own configuration, which this chapter does not model and 25.7 owns diagnosing.
Q6. Which single property would you add to an error-injection suite you inherited? P20 — a clean transaction completes after recovery. It requires no protocol knowledge, it is one extra transaction per test, and it is the property that separates "the design noticed" from "the design survived." P28's cover property is the close second, because a suite that silently stopped injecting reports the same green as one that never had a bug.
14. Module 24 Complete
Seven chapters built a verification stack. 24.1 framed the three contracts, 24.2 proved the local ones, 24.3 proved end-to-end conservation, 24.4 measured reach, 24.5 divided the labour with a vendor participant, 24.6 composed them — and this chapter gave them something to catch.
Every observer in Module 24 has branches that only a fault reaches. The terminal-status path, the unknown-identity error, the credit-exhaustion cover property, the coverage error bins. They are now reachable, and §9 shows each one firing exactly when it should and never otherwise.
Module 25 changes the question entirely. Verification asks did the design meet its contract, under stimulus I chose. Debugging asks what is wrong with this system, which I did not build and cannot re-run at will — fewer observation points, no golden model, and a symptom instead of a testbench.
Chapter 25.1 establishes the method: symptom → layer → last known-good event → first missing event → distinguishing experiment. And the first-error capture in §7 is the bridge — the same block, in silicon instead of simulation, is what makes that method possible at all.