Protocol Analyzer Debug — What the Link Cannot Show You

Chapter 25.1 §3 argued the analyzer should never be the first tool. This chapter is the argument for why it is the right one once the previous eight have produced a question — and an honest account of the questions it cannot answer at all.

1. Sources, Scope, and What This Chapter Refuses to Do

2. What the Analyzer Is

A protocol analyzer is a passive recorder placed in the physical path between two link partners. It captures the symbol stream, decodes it into TLPs and DLLPs, timestamps them, and stores as much as its buffer holds.

Three properties follow from "placed between two link partners", and they determine everything:

It sees one link. A fabric with a switch has several links; an analyzer on one of them sees traffic on that one. What a switch did internally between its ingress and egress is not on any link (§8).

It sees what was transmitted, not what was intended or what was done with it. A TLP on the wire is evidence that a device transmitted it. It is not evidence that the device meant to, that the receiver acted on it correctly, or that the value it carries is the value the host expected.

And it sees nothing that never became a packet. This is the largest category and §3 enumerates it.

What it is genuinely excellent at is worth stating plainly, because the rest of this chapter is about limits and the limits are not a reason to avoid the tool:

• It is the only view of the fabric between devices. For any question of the form "did this leave A and arrive at B", nothing else answers it.
• It has an accurate, common timebase across both directions of a link, which lets request/response latencies be measured directly rather than inferred (22.2 owns the measurement definition).
• It records what a device's own instrumentation would have lost — a device that has hung, reset, or overwritten its trace buffer still left its packets on the wire.
• And it is neutral. It does not depend on the correctness of the device under suspicion, which matters when the device's own counters are the thing you doubt (25.5 §16 case 7).

3. What the Analyzer Cannot See

§13 measured the consequence rather than asserting it. Three fault classes drawn from that list — a wrong local target, a Tag allocator reusing a live Tag, and a descriptor ownership violation — produced traces byte-identical to healthy, while generating 790, 313 and 313 internal events respectively that never reached the wire.

The wrong-register case is the sharpest and deserves spelling out. The request is well-formed. The address is one the device owns. The completer returns a Successful Completion with the correct Tag, correct length, correct Byte Count. Every field a decoder can show is correct. The only thing wrong is which register inside the device supplied the data — 25.5 §4's Direction B — and that is a fact about the device's internals, not about the transaction.

Two practical rules follow, and they are the chapter's operating discipline.

Decide whether your question is a link question before capturing. "Did this request leave the endpoint?" is a link question. "Why did the endpoint return that value?" is not, and no capture will answer it. §16's cases each state which kind they are.

And when the answer is "not a link question", the response is instrumentation, not persistence. §9 through §11 build the on-chip side. The gap is closed by exporting internal state into the trace, not by a better trigger.

4. Forming a Question a Trace Can Answer

The most expensive analyzer session is the one started without a question. A capture buffer fills in a bounded time; a trace of everything, taken hopefully, usually contains the interesting moment somewhere in a region nobody can identify.

A usable question has three parts:

1  an EVENT the analyzer can recognise      (a trigger)
2  a WINDOW around it worth keeping          (pre/post capture)
3  a PREDICTION about what the trace shows    (what confirms or refutes)

The third part is the one most often skipped, and skipping it is what makes a trace unfalsifiable. Without a prediction, any trace can be read as consistent with any hypothesis — and traces are large enough that a determined reader will always find something that looks meaningful.

The eight previous chapters exist to produce part 3. That is the actual reason 25.1 §3 puts the analyzer last:

Every one of those is falsifiable and answerable from a trace. None of them could have been formed without the earlier chapter's work.

5. Triggering

A trigger is a hypothesis expressed in the analyzer's language, and the categories that exist are consistent across instruments even though the syntax is not. Read your instrument's documentation for the syntax (§1); what follows is how to choose.

Trigger on the rarest thing you can name. A trigger on "any Memory Read" fills the buffer with ordinary traffic. A trigger on a specific address, a specific Tag, or an error status keeps the interesting event and its neighbourhood.

Prefer error statuses when the fault produces one. UR and CA are rare, unambiguous, and almost always closer to the fault than anything else in the trace (13.2). §13 measured the BAR-rejection fault producing 790 URs against a healthy zero — a trigger that cannot miss.

Trigger on absence only via a sequence, never directly. No analyzer triggers on "a Completion that never came", because nothing happens. The workable form is a request followed by a long interval without a matching Completion — and this is exactly the unmatched-request analysis of §6, which is usually better done as post-processing than as a trigger.

Set the pre-trigger window generously. The cause precedes the symptom, often by a long way. A capture that begins at the error contains the consequence and not the cause — which is 25.1 §6's first-failure rule in the analyzer's terms.

And know what your trigger cannot see. A fault from §3's list produces no triggerable event at all. §13's three identical-trace faults have no trigger condition that exists — not a difficult one, none. Recognising that before a capture session saves the session.

6. Unmatched Requests — the Strongest Signal

The analyzer's single most valuable derived measurement is the set of non-posted requests with no matching Completion.

It is valuable because it is structural rather than interpretive. Every non-posted request must be answered (10.2); a request with no answer is an unambiguous fact about the fabric, computed by matching (Requester ID, Tag) pairs across the trace with no judgement involved.

§13's healthy baseline had 16 unmatched requests out of 15,620 — the ones still in flight when the capture ended. That non-zero floor is important: a capture always ends mid-conversation, and treating every unmatched request as a fault produces false findings at exactly the rate of the outstanding depth.

What it tells you is where ownership stopped moving, in the sense 25.6 §2 established — and across a fabric, which is something no single device's counters can do. Given probes on more than one link, the unmatched set localises the failure to a segment:

request seen at link 1, not at link 2   -> the switch did not forward it
request seen at both, no Cpl anywhere    -> the completer never answered
request seen, Cpl seen at link 2 only    -> the switch did not return it
request and Cpl both seen at link 1      -> the requester dropped it locally

The last row is the one people find surprising, and it is 25.7 §3's class E. The Completion arrived on the wire and the requester still timed out, because its internal retirement path lost it. The trace proves the link was fine — which is a genuinely valuable negative result, and one the analyzer is uniquely able to deliver.

What it never tells you is why. A request that was never answered has at least the five causes 25.7 §3 enumerates, and the trace distinguishes only whether a Completion appeared. Localisation is not diagnosis, and the distinction is this chapter's central discipline.

7. Reading Time in a Trace

An analyzer's timestamps are accurate and are routinely over-interpreted. Three cautions.

A timestamp records when a packet crossed the probe point, not when it was generated or acted upon. The interval between a request and its Completion on the wire includes the completer's internal latency, and it excludes the requester's own queuing on both ends. 22.2 owns the definition of which event pair constitutes "latency" for a given question, and using the wire timestamps for a different pair than intended is the most common measurement error in trace analysis.

Ordering on the wire is not ordering in the device. Two TLPs adjacent in the trace may have been generated by unrelated agents inside the transmitter, and their adjacency says nothing about causality. The trace shows one serialisation of many concurrent internal activities.

And a gap is not evidence of anything by itself. 25.8 §14 measured a healthy link entering 818 zero-credit episodes, all of which resolved. Those appear in a trace as gaps. A gap means the link had nothing to send or could not send; distinguishing those requires the flow control DLLPs (16.6), which the trace does carry — so the analysis is possible, and it requires looking at the credit returns rather than at the gap.

8. An ACK Is Not a Delivery Receipt

This is the most consequential misreading of a trace, and §14 measured it.

An ACK is a Data Link Layer acknowledgement to the immediate link neighbour (14.4, 21.2 §3). Its meaning is precise and narrow: the receiving link partner accepted this TLP at the Data Link Layer, and the transmitter may now release it from the replay buffer.

It says nothing about:

• whether the receiving device forwarded the TLP onward,
• whether any downstream device received it,
• whether the destination acted on it,
• or whether the payload was correct.

In a fabric with a switch, an ACK on the ingress link is generated by the switch, not by the endpoint the request is addressed to. The endpoint may be unreachable, unpowered, or discarding everything — and the ingress trace looks perfect throughout.

The practical consequence is a probe-placement rule, and it is the main reason multi-probe captures exist: to prove delivery across a switch you need a probe on both sides. One probe proves transmission. Two prove forwarding. No number of probes on the ingress side proves anything about the egress side, which is why §16 case 3 treats a single-probe "the packets were ACKed" observation as no evidence at all.

9. Closing the Gap — On-Chip Trace Correlation

§3's list is not a limitation to work around. It is a specification for what a device must export.

The gap is closed by making internal events visible in a form that can be correlated with the wire trace, and that requires three things from the device:

A shared timebase. An on-chip trace timestamped in a clock domain with no defined relationship to the analyzer's is not correlatable. What is needed is a periodic, recognisable event visible on both — a marker TLP, or a known-address write emitted at a known internal moment — so the two timelines can be aligned.

A correlation key on the packets themselves. The most useful single technique in this chapter: make the Tag, or a field the device controls, carry an identifier that also appears in the on-chip trace. Then a packet in the wire trace and an event in the internal trace can be matched exactly rather than by timestamp proximity.

And export of the specific state §3 lists — descriptor ownership transitions, Tag allocations with epochs, queue occupancy, decoded local target, and the wait-for graph. Each of those was built as an instrument in 25.5 through 25.8; this chapter's contribution is getting them into the same timeline as the trace.

One design decision matters more than the rest, and §11's Block 3 is built around it. The internal trace must be exportable after the device has stopped. A hung device cannot emit a marker TLP; a reset device has lost its buffer (25.8 §2). The export path must therefore be readable through configuration space or a sideband, and the buffer must live on a reset domain that recovery does not clear.

10. The Waveform

Four readings.

wire_req, wire_cpl and wire_status are identical across both transactions. Same cycles, same values, same status. This is §13's identical-trace result drawn out in time — a decoder shows two successful reads.

int_tgt_ok is the only signal that differs, and it does not exist on the Link. It is an internal comparison the device performs and exports.

int_export pulses once, at the moment of the discrepancy. Exporting continuously would flood any buffer; exporting on the discrepancy is what makes the trace finite and readable — §11's Block 4.

And corr_key spans both the wire event and the internal event. That is the mechanism of §9: a value carried in a device-controlled packet field and in the internal trace, so the two records can be matched exactly rather than by timestamp proximity.

11. RTL — The Correlation Instruments

Block 1 — the package: event taxonomy and the correlation key.

package pa_dbg_pkg;
 
  // The internal events of §3 — the ones that never become packets.
  // Each maps to an instrument built in 25.5 through 25.8.
  typedef enum logic [3:0] {
    EV_NONE         = 4'd0,
    EV_BAR_REJECT   = 4'd1,   // 25.5 — which boundary rejected
    EV_BAR_TARGET   = 4'd2,   // 25.5 — which local target was selected
    EV_DESC_OWNER   = 4'd3,   // 25.6 — a descriptor ownership transition
    EV_DESC_VIOLATE = 4'd4,   // 25.6 — an ownership law violation
    EV_TAG_ALLOC    = 4'd5,   // 25.7 — tag allocated, with epoch
    EV_TAG_ABANDON  = 4'd6,   // 25.7 — timeout reclaimed a tag
    EV_CPL_DISCARD  = 4'd7,   // 25.7 — a late completion was discarded
    EV_QUEUE_FULL   = 4'd8,   // 25.8 — internal backpressure
    EV_CLASS_STALL  = 4'd9,   // 25.8 — a traffic class stopped
    EV_WAITFOR_EDGE = 4'd10,  // 25.8 — a wait-for edge formed
    EV_MARKER       = 4'd15   // timebase alignment (§9)
  } pa_event_e;
 
  function automatic int unsigned gw(input int unsigned n);
    return (n <= 1) ? 1 : $clog2(n);
  endfunction
 
  // The correlation key travels in a device-controlled packet field AND in
  // the internal trace, so a wire record and an internal record can be
  // matched exactly rather than by timestamp proximity (§9).
  localparam int unsigned CORR_W = 12;
  typedef logic [CORR_W-1:0] corr_t;
 
endpackage

Block 2 — the correlation key generator. The single most useful technique in the chapter, and the one that costs least.

module pa_corr_key #(
  parameter int unsigned NTAG = 32
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                              alloc,
  input  logic [pa_dbg_pkg::gw(NTAG)-1:0]   alloc_tag,
  output pa_dbg_pkg::corr_t                 key_out,
  output pa_dbg_pkg::corr_t                 key_for_tag [NTAG]
);
 
  import pa_dbg_pkg::*;
 
  corr_t next_key;
 
  // A monotonically increasing key stamped on every request and recorded
  // against its tag.  The key is what the analyzer sees (carried in a
  // device-controlled field) and what the internal trace records — one
  // value, two timelines (§9).
  //
  // Note this does NOT change any protocol field's meaning: the key is
  // carried in a field the device already owns the encoding of, and it
  // must never displace information the protocol requires (P4).
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      next_key <= '0;
      for (int i = 0; i < NTAG; i++) key_for_tag[i] <= '0;
    end else if (alloc) begin
      key_for_tag[alloc_tag] <= next_key;
      next_key               <= next_key + 1'b1;
    end
  end
 
  assign key_out = next_key;
 
endmodule

Block 3 — the internal trace buffer. Survives the recovery that erases everything else.

module pa_trace_buffer #(
  parameter int unsigned DEPTH = 256,
  parameter int unsigned TS_W  = 48
)(
  input  logic clk,
  // NOTE: this reset is deliberately NOT the watchdog/recovery reset.
  // 25.8 §2 established that recovery destroys the evidence; a trace that
  // is cleared by the recovery it exists to explain is worthless.
  input  logic                     rst_n_persistent,
  input  logic                     ev_valid,
  input  pa_dbg_pkg::pa_event_e    ev_kind,
  input  pa_dbg_pkg::corr_t        ev_key,
  input  logic [31:0]              ev_data,
  input  logic [TS_W-1:0]          timestamp,
  input  logic                     freeze,        // stop recording, keep contents
  input  logic                     rd_en,
  input  logic [pa_dbg_pkg::gw(DEPTH)-1:0] rd_addr,
  output logic [TS_W+48-1:0]       rd_data,
  output logic [pa_dbg_pkg::gw(DEPTH)-1:0] wr_ptr,
  output logic                     wrapped,
  output logic [31:0]              dropped_count
);
 
  import pa_dbg_pkg::*;
 
  typedef struct packed {
    logic [TS_W-1:0] ts;
    pa_event_e       kind;
    corr_t           key;
    logic [31:0]     data;
  } rec_t;
 
  rec_t mem [DEPTH];
 
  always_ff @(posedge clk or negedge rst_n_persistent) begin
    if (!rst_n_persistent) begin
      wr_ptr <= '0; wrapped <= 1'b0; dropped_count <= '0;
    end else if (ev_valid && !freeze) begin
      mem[wr_ptr] <= '{ts: timestamp, kind: ev_kind, key: ev_key, data: ev_data};
      if (wr_ptr == gw(DEPTH)'(DEPTH-1)) begin
        wr_ptr <= '0; wrapped <= 1'b1;
      end else begin
        wr_ptr <= wr_ptr + 1'b1;
      end
    end else if (ev_valid && freeze) begin
      // Events arriving after the freeze are COUNTED, never silently lost.
      // A frozen buffer with an unknown number of missed events cannot be
      // reasoned about (P12, and 25.1 §6's rule).
      if (dropped_count != 32'hFFFF_FFFF) dropped_count <= dropped_count + 32'd1;
    end
  end
 
  assign rd_data = rd_en ? {mem[rd_addr].ts, mem[rd_addr].kind,
                            mem[rd_addr].key, mem[rd_addr].data}[TS_W+47:0] : '0;
 
endmodule

Block 4 — the discrepancy exporter. Exports on difference rather than continuously, which is what keeps the buffer finite.

module pa_discrepancy_export #(
  parameter int unsigned NTARGET = 4
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                                  access_valid,
  input  logic [pa_dbg_pkg::gw(NTARGET)-1:0]    decoded_target,
  input  logic [pa_dbg_pkg::gw(NTARGET)-1:0]    expected_target,
  input  logic                                  expected_valid,
  input  pa_dbg_pkg::corr_t                     key,
  output logic                                  ev_valid,
  output pa_dbg_pkg::pa_event_e                 ev_kind,
  output pa_dbg_pkg::corr_t                     ev_key,
  output logic [31:0]                           ev_data
);
 
  import pa_dbg_pkg::*;
 
  // §13's wrong-target fault produced a byte-identical wire trace while
  // generating 790 internal events.  Exporting EVERY decode would flood
  // the buffer; exporting only the discrepancies makes the trace finite
  // and every record in it meaningful.
  logic mismatch;
  assign mismatch = access_valid && expected_valid &&
                    (decoded_target != expected_target);
 
  always_comb begin
    ev_valid = mismatch;
    ev_kind  = EV_BAR_TARGET;
    ev_key   = key;
    ev_data  = {16'd0, {(16-gw(NTARGET)){1'b0}}, decoded_target};
  end
 
endmodule

Block 5 — the timebase marker. §9's alignment mechanism.

module pa_timebase_marker #(
  parameter int unsigned PERIOD = 32'd10_000     // MODEL PARAMETER
)(
  input  logic clk,
  input  logic rst_n,
  input  logic        link_up,
  output logic        emit_marker,       // request a marker TLP on the wire
  output logic        ev_valid,
  output pa_dbg_pkg::pa_event_e ev_kind,
  output logic [31:0] ev_data,
  output logic [31:0] marker_seq
);
 
  import pa_dbg_pkg::*;
 
  logic [31:0] cnt;
 
  // A periodic event recorded in BOTH timelines: a marker TLP on the wire
  // and a marker record internally, sharing a sequence number.  Without a
  // common event the two timelines cannot be aligned at all (§9), and
  // timestamp-proximity matching produces wrong pairings under load.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      cnt <= '0; marker_seq <= '0;
    end else if (link_up) begin
      if (cnt == PERIOD) begin
        cnt <= '0; marker_seq <= marker_seq + 32'd1;
      end else begin
        cnt <= cnt + 32'd1;
      end
    end
  end
 
  assign emit_marker = link_up && (cnt == PERIOD);
  assign ev_valid    = emit_marker;
  assign ev_kind     = EV_MARKER;
  assign ev_data     = marker_seq;
 
endmodule

Block 6 — the unmatched-request tracker. §6's analysis, computed on-chip so it is available without a capture.

module pa_unmatched_track #(
  parameter int unsigned NTAG = 32
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                              req_issued,
  input  logic [pa_dbg_pkg::gw(NTAG)-1:0]   req_tag,
  input  logic                              cpl_matched,
  input  logic [pa_dbg_pkg::gw(NTAG)-1:0]   cpl_tag,
  input  logic                              capture_end,
  output logic [pa_dbg_pkg::gw(NTAG+1)-1:0] outstanding_now,
  output logic [31:0]                       unmatched_at_end,
  output logic [31:0]                       total_issued,
  output logic [31:0]                       total_matched
);
 
  import pa_dbg_pkg::*;
 
  logic [NTAG-1:0] live;
 
  // §13's healthy baseline had 16 unmatched of 15,620 — the requests still
  // in flight when the capture ended.  That floor is NOT a fault, and a
  // tool that reports every unmatched request as one produces false
  // findings at exactly the rate of the outstanding depth (§6).
  always_comb begin
    outstanding_now = '0;
    for (int i = 0; i < NTAG; i++) outstanding_now += gw(NTAG+1)'(live[i]);
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      live <= '0; total_issued <= '0; total_matched <= '0; unmatched_at_end <= '0;
    end else begin
      if (req_issued) begin
        live[req_tag] <= 1'b1;
        total_issued  <= total_issued + 32'd1;
      end
      if (cpl_matched) begin
        live[cpl_tag] <= 1'b0;
        total_matched <= total_matched + 32'd1;
      end
      if (capture_end) unmatched_at_end <= 32'(outstanding_now);
    end
  end
 
endmodule

Block 7 — the link-state annotator. Correlating TLP activity with the LTSSM, so a trace gap can be attributed.

module pa_linkstate_annotate #(
  parameter int unsigned TS_W = 48
)(
  input  logic clk,
  input  logic rst_n,
  input  logic [4:0]        ltssm_state,      // encoding owned by 18.1 §10
  input  logic [TS_W-1:0]   timestamp,
  output logic              ev_valid,
  output pa_dbg_pkg::pa_event_e ev_kind,
  output logic [31:0]       ev_data,
  output logic [31:0]       transition_count
);
 
  import pa_dbg_pkg::*;
 
  logic [4:0] prev_state;
 
  // §7's third caution: a gap in a trace is not evidence by itself.  A gap
  // that coincides with a link-state change has a cause; one that does not
  // is a credit or an idle question.  Exporting every transition with a
  // timestamp is what lets the two be told apart.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      prev_state <= '0; transition_count <= '0;
    end else begin
      prev_state <= ltssm_state;
      if (ltssm_state != prev_state && transition_count != 32'hFFFF_FFFF)
        transition_count <= transition_count + 32'd1;
    end
  end
 
  always_comb begin
    ev_valid = (ltssm_state != prev_state);
    ev_kind  = EV_CLASS_STALL;
    ev_data  = {22'd0, prev_state, ltssm_state};
  end
 
endmodule

Block 8 — the event arbiter. Several sources, one buffer, and an explicit policy about what is lost.

// verilog_lint: waive-start module-filename
module pa_event_arbiter #(
  parameter int unsigned NSRC = 6
)(
  input  logic clk,
  input  logic rst_n,
  input  logic [NSRC-1:0]           src_valid,
  input  pa_dbg_pkg::pa_event_e     src_kind [NSRC],
  input  pa_dbg_pkg::corr_t         src_key  [NSRC],
  input  logic [31:0]               src_data [NSRC],
  output logic                      ev_valid,
  output pa_dbg_pkg::pa_event_e     ev_kind,
  output pa_dbg_pkg::corr_t         ev_key,
  output logic [31:0]               ev_data,
  output logic [31:0]               arb_dropped [NSRC]
);
 
  import pa_dbg_pkg::*;
 
  logic [gw(NSRC)-1:0] winner;
 
  // Fixed priority, lowest index first, and every LOSER is counted per
  // source.  An arbiter that silently drops events makes the trace's
  // completeness unknowable — and a trace whose gaps cannot be
  // distinguished from quiet periods cannot support any negative
  // conclusion (P13, and this is the same rule as 25.1 §6).
  always_comb begin
    ev_valid = (src_valid != '0);
    winner   = '0;
    for (int i = NSRC-1; i >= 0; i--) if (src_valid[i]) winner = gw(NSRC)'(i);
    ev_kind = src_kind[winner];
    ev_key  = src_key[winner];
    ev_data = src_data[winner];
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < NSRC; i++) arb_dropped[i] <= '0;
    end else begin
      for (int i = 0; i < NSRC; i++)
        if (src_valid[i] && (gw(NSRC)'(i) != winner) && arb_dropped[i] != 32'hFFFF_FFFF)
          arb_dropped[i] <= arb_dropped[i] + 32'd1;
    end
  end
 
endmodule
// verilog_lint: waive-stop module-filename

12. Checks and Assertions

Trace-integrity properties.

// P1 — timestamps are monotonic within one trace.  A non-monotonic trace
// cannot be ordered, and every causal reading of it is unfounded.
property p1_timestamps_monotonic;
  @(posedge clk) disable iff (!rst_n)
    ev_valid |=> (ev_valid -> (timestamp >= $past(timestamp)));
endproperty
a_p1: assert property (p1_timestamps_monotonic);
 
// P2 — the timestamp counter does not wrap within a capture, or the wrap
// is recorded.  A silent wrap reorders the trace.
property p2_no_silent_ts_wrap;
  @(posedge clk) disable iff (!rst_n)
    (timestamp == '1) |=> (ts_wrap_flag);
endproperty
a_p2: assert property (p2_no_silent_ts_wrap);
 
// P3 — every recorded event carries a correlation key or is a marker.
// A record that can be matched to nothing is unusable (§9).
property p3_every_event_keyed;
  @(posedge clk) disable iff (!rst_n)
    (ev_valid && (ev_kind != pa_dbg_pkg::EV_MARKER)) |-> (ev_key != '0);
endproperty
a_p3: assert property (p3_every_event_keyed);

Correlation-key properties — §9.

// P4 — the key never displaces protocol-required information.  This is the
// obligation that makes the technique safe: the key rides in a field the
// device owns the encoding of, and the protocol's use of that field is
// unaffected (§11 Block 2).
property p4_key_does_not_displace;
  @(posedge clk) disable iff (!rst_n)
    req_issued |-> (tlp_protocol_fields == expected_protocol_fields);
endproperty
a_p4: assert property (p4_key_does_not_displace);
 
// P5 — a key is unique among concurrently live requests.  A reused key
// makes two wire records match one internal record.
property p5_key_unique_among_live;
  @(posedge clk) disable iff (!rst_n)
    alloc |-> !key_live[next_key];
endproperty
a_p5: assert property (p5_key_unique_among_live);
 
// P6 — the key recorded against a tag matches the key transmitted.
property p6_key_matches_transmitted;
  @(posedge clk) disable iff (!rst_n)
    req_issued |-> (tlp_key == key_for_tag[req_tag]);
endproperty
a_p6: assert property (p6_key_matches_transmitted);
 
// P7 — the key advances monotonically, so ordering is recoverable even if
// timestamps are lost.
property p7_key_monotonic;
  @(posedge clk) disable iff (!rst_n)
    alloc |=> (next_key == ($past(next_key) + 1));
endproperty
a_p7: assert property (p7_key_monotonic);

Timebase-alignment properties.

// P8 — a marker appears in BOTH timelines with the same sequence number.
// Without this the two traces cannot be aligned at all (§9).
property p8_marker_in_both;
  @(posedge clk) disable iff (!rst_n)
    emit_marker |-> (ev_valid && (ev_kind == pa_dbg_pkg::EV_MARKER) &&
                     (ev_data == marker_seq));
endproperty
a_p8: assert property (p8_marker_in_both);
 
// P9 — markers are emitted only when the link can carry them.
property p9_marker_needs_link;
  @(posedge clk) disable iff (!rst_n)
    emit_marker |-> link_up;
endproperty
a_p9: assert property (p9_marker_needs_link);
 
// P10 — the marker sequence never repeats within a capture.
property p10_marker_seq_unique;
  @(posedge clk) disable iff (!rst_n)
    emit_marker |=> (marker_seq != $past(marker_seq));
endproperty
a_p10: assert property (p10_marker_seq_unique);

Buffer-integrity properties.

// P11 — the trace buffer is not cleared by the recovery reset.  25.8 §2:
// a trace erased by the recovery it exists to explain is worthless.
property p11_buffer_survives_recovery;
  @(posedge clk) disable iff (!rst_n_persistent)
    watchdog_reset |=> $stable(wr_ptr);
endproperty
a_p11: assert property (p11_buffer_survives_recovery);
 
// P12 — events arriving after a freeze are COUNTED, never silently lost.
property p12_frozen_drops_counted;
  @(posedge clk) disable iff (!rst_n_persistent)
    (ev_valid && freeze) |=> (dropped_count > $past(dropped_count));
endproperty
a_p12: assert property (p12_frozen_drops_counted);
 
// P13 — arbitration losers are counted per source.  A trace whose gaps
// cannot be distinguished from quiet periods supports no negative
// conclusion (§11 Block 8).
property p13_arb_drops_counted;
  @(posedge clk) disable iff (!rst_n)
    (src_valid[i] && (i != winner)) |=> (arb_dropped[i] > $past(arb_dropped[i]));
endproperty
a_p13: assert property (p13_arb_drops_counted);
 
// P14 — wrap is flagged.  A wrapped buffer read as a complete record
// silently omits the beginning, which is where the cause usually is.
property p14_wrap_flagged;
  @(posedge clk) disable iff (!rst_n_persistent)
    (ev_valid && !freeze && (wr_ptr == DEPTH-1)) |=> wrapped;
endproperty
a_p14: assert property (p14_wrap_flagged);
 
// P15 — the write pointer never advances while frozen.
property p15_freeze_holds;
  @(posedge clk) disable iff (!rst_n_persistent)
    freeze |=> $stable(wr_ptr);
endproperty
a_p15: assert property (p15_freeze_holds);

Discrepancy-export properties.

// P16 — an export occurs exactly when the decoded and expected values
// differ, so the buffer holds only meaningful records (§11 Block 4).
property p16_export_on_mismatch;
  @(posedge clk) disable iff (!rst_n)
    (access_valid && expected_valid && (decoded_target != expected_target))
      |-> ev_valid;
endproperty
a_p16: assert property (p16_export_on_mismatch);
 
// P17 — no export when they agree.  Exporting every decode floods the
// buffer and makes the wrap of P14 certain.
property p17_no_export_on_match;
  @(posedge clk) disable iff (!rst_n)
    (access_valid && expected_valid && (decoded_target == expected_target))
      |-> !ev_valid;
endproperty
a_p17: assert property (p17_no_export_on_match);
 
// P18 — an export carries the key of the access that produced it.
property p18_export_keyed_correctly;
  @(posedge clk) disable iff (!rst_n)
    ev_valid |-> (ev_key == key);
endproperty
a_p18: assert property (p18_export_keyed_correctly);

Unmatched-request properties — §6.

// P19 — outstanding equals issued minus matched, always.
property p19_outstanding_derived;
  @(posedge clk) disable iff (!rst_n)
    (total_issued >= total_matched);
endproperty
a_p19: assert property (p19_outstanding_derived);
 
// P20 — a completion never matches a tag that is not live.
property p20_no_match_dead_tag;
  @(posedge clk) disable iff (!rst_n)
    cpl_matched |-> live[cpl_tag];
endproperty
a_p20: assert property (p20_no_match_dead_tag);
 
// P21 — the unmatched-at-end value is captured at capture_end, once.
// §13's healthy baseline had 16 unmatched of 15,620; that floor must be
// read as in-flight work, not as a fault (§6).
property p21_unmatched_captured_once;
  @(posedge clk) disable iff (!rst_n)
    capture_end |=> $stable(unmatched_at_end);
endproperty
a_p21: assert property (p21_unmatched_captured_once);

Cover — the anti-vacuity set.

// P22 — every recorded event is readable: the write pointer and the wrap
// flag together determine which records are valid, and a reader that
// cannot bound the valid range cannot make a negative claim about the trace.
property p22_valid_range_bounded;
  @(posedge clk) disable iff (!rst_n_persistent)
    (wr_ptr < DEPTH) && (wrapped || (wr_ptr <= DEPTH));
endproperty
a_p22: assert property (p22_valid_range_bounded);
 
// P22's covers — the interesting records must actually occur.  A capture with no
// discrepancy, no wrap and no freeze exercises none of P11 through P18.
c1_discrepancy:  cover property (@(posedge clk) disable iff (!rst_n) mismatch);
c2_wrap:         cover property (@(posedge clk) disable iff (!rst_n_persistent) wrapped);
c3_freeze_drop:  cover property (@(posedge clk) disable iff (!rst_n_persistent)
                   ev_valid && freeze);
c4_arb_conflict: cover property (@(posedge clk) disable iff (!rst_n)
                   $countones(src_valid) > 1);
c5_marker:       cover property (@(posedge clk) disable iff (!rst_n) emit_marker);
c6_unmatched:    cover property (@(posedge clk) disable iff (!rst_n)
                   capture_end && (outstanding_now > 0));
c7_link_change:  cover property (@(posedge clk) disable iff (!rst_n)
                   ltssm_state != prev_state);

Completeness properties — the two §17 depends on most.

// P23 — every recorded event that is not a marker carries a key that was
// actually issued.  An internal event whose key resolves to no request
// cannot be placed on the wire timeline, and treating it as if it could
// produces wrong pairings under load — silently.
property p23_key_resolves;
  @(posedge clk) disable iff (!rst_n)
    (ev_valid && (ev_kind != pa_dbg_pkg::EV_MARKER))
      |-> (ev_key < next_key);
endproperty
a_p23: assert property (p23_key_resolves);
 
// P24 — the trace is complete for the window being analysed: no event was
// lost at a freeze or an arbiter.  This is the property that licenses a
// NEGATIVE conclusion.  Without it, an absence in the trace and a dropped
// record are indistinguishable, and §17 case 10 is unanswerable.
property p24_window_complete;
  @(posedge clk) disable iff (!rst_n)
    analysis_window |-> ((dropped_count == 32'd0) && (arb_dropped_total == 32'd0));
endproperty
a_p24: assert property (p24_window_complete);

13. Measured Behaviour — What the Wire Reveals

Healthy wire: MRd 15,620 · CplD 15,604 · UR 0 · unmatched 16 · 31,224 TLP records.

Four readings.

Three of five faults are invisible, and "invisible" is literal. Not subtle, not requiring a clever filter — byte-for-byte identical. There is no trigger condition for these faults because there is no event.

The two visible ones are visible in different ways. The BAR rejection produces 790 URs against a healthy zero — an unmissable, unambiguous signal, and the best possible case for the tool. The internal queue stall produces no error at all; it shows up only as slightly fewer requests and fewer completions, which is a statistical difference requiring a healthy baseline to notice.

That second case is worth its own note. A fault whose only signature is "somewhat less traffic" is detectable from a trace only if you have a known-good trace to compare against. §16 case 10 is about keeping one.

And the invisible three generated 790, 313 and 313 internal events. The information existed. It was never exported. That is a design decision, not a physical limit — and §9 through §11 are the argument for making the other decision.

14. Measured Behaviour — The ACK

The ingress ACK count is identical in both rows. An analyzer on the ingress link records 20,000 successful acknowledgements whether the switch forwards everything or discards nearly two thousand TLPs.

This is not a defect in the analyzer. The ACK genuinely happened, and it genuinely means what it means: the switch's Data Link Layer accepted the TLP and the transmitter may release it from its replay buffer (14.4, 21.2 §3). The error is entirely in reading it as a delivery receipt.

The practical rule is about probe placement, and it is absolute: one probe proves transmission; two prove forwarding. No amount of analysis of the ingress trace establishes anything about the egress side, because the information is not present in it.

15. Executable Counterexamples

Three minimal cases, each showing an instrument or a procedure that produces a confident wrong answer.

Counterexample A — the trace buffer on the recovery reset domain (violates P11).

// The internal trace buffer is reset by the same signal the watchdog
// asserts when it recovers a hung device.
module ce_a_buffer_on_wrong_reset #(parameter int unsigned DEPTH = 256)(
  input  logic clk,
  input  logic rst_n,          // <-- asserted by the watchdog recovery
  input  logic ev_valid,
  input  logic [47:0] ev_rec,
  output logic [pa_dbg_pkg::gw(DEPTH)-1:0] wr_ptr
);
  logic [47:0] mem [DEPTH];
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) wr_ptr <= '0;                    // <-- evidence erased here
    else if (ev_valid) begin mem[wr_ptr] <= ev_rec; wr_ptr <= wr_ptr + 1'b1; end
endmodule
 
// Failing stimulus: the device deadlocks (25.8), the watchdog fires, the
// link recovers, and an engineer reads the trace buffer.
// Golden: the buffer holds the events leading up to the hang.
// This:   wr_ptr is 0 and the buffer is empty.  The only record of the
//         fault was destroyed by the mechanism that recovered from it.
// P11 fails.
// Observable consequence: the fault is unreproducible and undiagnosable,
// and each recurrence destroys its own evidence again.  25.8 §2 named
// this for deadlocks; it applies to every fault whose recovery is a reset.

Counterexample B — the arbiter that drops silently (violates P13, and defeats every negative conclusion).

// Six event sources, one buffer port, fixed priority, losers discarded.
module ce_b_silent_arbiter #(parameter int unsigned NSRC = 6)(
  input  logic [NSRC-1:0] src_valid,
  input  logic [47:0]     src_rec [NSRC],
  output logic            ev_valid,
  output logic [47:0]     ev_rec
);
  logic [2:0] winner;
  always_comb begin
    ev_valid = (src_valid != '0);
    winner   = '0;
    for (int i = NSRC-1; i >= 0; i--) if (src_valid[i]) winner = 3'(i);
    ev_rec   = src_rec[winner];
  end                                    // <-- losers vanish, uncounted
endmodule
 
// Failing stimulus: a burst in which the descriptor-ownership source and
// the tag-allocator source are both active on the same cycles, repeatedly.
// Golden: the losers are counted per source, so the analyst knows the
//         trace is incomplete for that window and by how much.
// This:   the lower-priority source's events are absent from the trace and
//         nothing indicates they ever existed.
// P13 fails.  P24 fails (the window is not complete and does not say so).
// Observable consequence: an absence in the trace is indistinguishable
// from a quiet period, so NO negative conclusion drawn from this trace is
// sound — including the correct-looking one that the fault did not occur.

Counterexample C — the confounded observability experiment (a procedure, not a module).

// Not RTL.  This is the analysis error that was actually made while
// producing §13, recorded because it is the error the section is about.
//
// Procedure as first written:
//   for each fault F:
//       run the model with a SINGLE random stream, injecting F
//       compare the resulting TLP COUNTS against the healthy run
//       report "visible" if any count differs
//
// Failing stimulus: the wrong-local-target fault, which by construction
// leaves the Link untouched.
// Golden: identical trace; the fault is invisible.
// This:   MRd 19,752 vs 19,752, CplD 19,741, unmatched 11 vs 7 — every
//         count differed, and the fault was reported VISIBLE.
//
// Why: the fault's coin flips were drawn from the SAME random stream as
// the traffic generator, so injecting any fault shifted the entire
// traffic sequence.  The differing counts measured RNG divergence, not
// observability.  The experiment could not have reported "invisible" for
// any fault, including one that is invisible by construction.
//
// Corrected procedure:
//   give fault decisions an INDEPENDENT random stream, and compare the
//   full trace RECORD BY RECORD rather than comparing aggregate counts.
// With that correction three of five faults produce a byte-identical
// trace (§13), which is a decisive result rather than a suggestive one.
//
// The general form: an observability experiment whose control and
// treatment arms do not share a stimulus is not measuring observability.

All three share the shape this module keeps producing. Counterexample A destroys the evidence, B makes absence uninterpretable, and C measures the wrong quantity entirely. None of them is a fault in a PCIe device — each is a fault in the apparatus used to look at one, which is why §16's procedural mutations sit in the same table as the structural ones.

16. Verification — Mutations

Thirty-six mutations. Every "Caught by" entry names a property from §12. These mutate the instrumentation and the analysis procedure, because those are what this chapter builds.

Mutations 27 through 35 are procedural rather than structural, and they are in the same table deliberately. A correct instrument used with an incorrect procedure produces confident wrong answers, and this chapter's failure mode is far more often the procedure than the hardware.

Mutation 20 is the one that matters most for the chapter's argument. The discrepancy was detected — the device knew — and it was not exported. §13 measured 790 such events on a byte-identical wire trace.

17. Debugging

Case 1 — before capturing, decide whether this is a link question.

Write down the question and check it against §3's list. "Did this request leave the endpoint?" is a link question. "Why did the endpoint return that value?" is not. §13 measured three fault classes producing byte-identical traces — for those, no capture at any depth with any trigger helps. The productive response to "not a link question" is instrumentation (§9), not a longer capture. Confidence: this test costs a minute and routinely saves a day.

Case 2 — state the prediction before capturing.

A capture without a prediction is unfalsifiable (§4). Traces are large enough that a determined reader finds support for any hypothesis. The prediction should name what you expect to see and what would refute you — and the eight previous chapters exist to produce exactly that (§4's table).

Case 3 — "the packets were all ACKed, so they arrived."

They were accepted by the immediate link neighbour (§8, §14). §14 measured a switch dropping 10% internally while the ingress trace recorded 20,000 ACKs, identical to healthy. Delivery across a switch requires a probe on the egress side. One probe proves transmission; two prove forwarding. Confidence: high; this is the most common trace misreading in the chapter.

Case 4 — a request has no Completion anywhere in the trace.

The strongest thing the tool does (§6). With probes on multiple links this localises the failure to a segment: seen at link 1 and not link 2 means the switch did not forward it; seen at both with no Completion means the completer never answered. Then stop — the trace has localised, not diagnosed. 25.7 §3's five causes all look identical here, and separating them needs requester-side or completer-side state.

Case 5 — the unmatched count is small and nonzero on a healthy system.

That is the in-flight floor, not a fault (§6, mutation 26). §13's healthy baseline had 16 unmatched of 15,620 — requests outstanding when the capture ended. A tool that flags every unmatched request produces false findings at exactly the rate of the outstanding depth. Compare against the requester's outstanding capacity before concluding anything.

Case 6 — the trace shows a gap.

A gap is not evidence (§7). 25.8 §14 measured a healthy link entering 818 zero-credit episodes, every one of which appears as a gap. The discriminator is in the trace: look at the flow control DLLPs (16.6) to see whether credits were being returned, and at the LTSSM correlation (§11 Block 7) to see whether the link changed state. A gap with credits flowing and the link in L0 is an idle transmitter, not a fault.

Case 7 — everything in the trace is correct and the data is wrong.

25.5 §4's Direction B, and the analyzer cannot see it (§3, §13). Correct MRd, correct CplD, Successful Completion, correct Tag, correct Byte Count — and the wrong register supplied the data. §13 measured this fault producing a byte-identical trace alongside 790 internal events. Stop capturing. The answer is on-chip: the decoded target index, exported on discrepancy (§11 Block 4).

Case 8 — the trace shows slightly less traffic than expected and no errors.

This is the internal-queue-stall signature (§13: MRd 15,604 versus 15,620, no UR, unmatched 14 versus 16). It is detectable only against a known-good baseline — the differences are small and there is no error to trigger on. If you do not have a baseline trace, you cannot make this call, and capturing one during known-good operation is cheap insurance (mutation 35).

Case 9 — you have both a wire trace and an on-chip trace and cannot align them.

Check the markers (§9, §11 Block 5, mutations 9 to 11). Without a periodic event visible in both timelines carrying the same sequence number, alignment falls back to timestamp proximity — which produces wrong pairings under load, silently. Then check the correlation keys (P3, P6): a key recorded internally but not transmitted, or vice versa, makes every match wrong rather than merely imprecise.

Case 10 — the internal trace shows nothing in the window of interest.

Check the drop counters before concluding anything (P23, P24, mutations 14 and 17). A freeze that discarded events, or an arbiter that starved one source, produces an empty window that looks exactly like a quiet one. An absence is evidence only when the trace is known to be complete for that window — which is why §11 counts every dropped event rather than dropping it silently.

Case 11 — the device hung and its trace buffer is empty after recovery.

The buffer was on the reset domain the watchdog clears (mutation 13, P11). 25.8 §2 established this for deadlocks and it generalises: a trace erased by the recovery it exists to explain is worthless. The fix is architectural and must be made before the bug — a persistent reset domain and a freeze-on-trigger policy (§11 Block 3).

18. Misconceptions

"The analyzer sees everything on the link, so it sees everything." It sees the Link. §13 measured three of five faults producing byte-identical traces because they never became packets (§3).

"A better trigger would have caught it." Not for a fault with no event. There is no trigger condition for §13's invisible three — not a difficult one, none.

"The TLPs were ACKed, so they were delivered." An ACK is link-local (21.2 §3). §14 measured 20,000 ingress ACKs with 1,972 TLPs dropped inside the switch.

"The trace shows a gap, so something stalled." 25.8 §14 measured 818 gaps on a healthy link, all resolving. Read the flow control DLLPs, not the gap (§7).

"Unmatched requests mean lost packets." §13's healthy baseline had 16 of 15,620 — work in flight at capture end (§6). The floor equals the outstanding depth.

"These two packets are adjacent, so one caused the other." The trace is one serialisation of many concurrent internal activities (§7). Adjacency on the wire implies nothing about causality inside a device.

"The analyzer's decoder shows all the fields, so that is the protocol." It shows one vendor's reading, possibly incomplete or out of date (§1). Protocol facts come from the specification and from the chapters that cite it.

"Capture everything and look for something odd." A capture without a prediction is unfalsifiable (§4), and a buffer filled with ordinary traffic usually contains the interesting moment somewhere nobody can find.

"The internal trace is empty, so nothing happened." Or events were dropped at a freeze or an arbiter (§17 case 10). An absence is evidence only when the trace is known complete (P23, P24).

19. Understanding Check

Q1. A device returns plausible but wrong data. What will an analyzer show, and what should you do?

A byte-identical trace, and stop capturing (§3, §13, §17 case 7). §13 measured the wrong-local-target fault producing MRd 15,620, CplD 15,604, UR 0, unmatched 16 — identical to healthy in every record — while generating 790 internal events. Correct request, correct Successful Completion, correct Tag and Byte Count; the only thing wrong is which register supplied the data, and that is a fact about the device's internals. The answer is on-chip: export the decoded target on discrepancy (§11 Block 4). This is 25.5 §4's Direction B, and no trigger exists for it.

Q2. Your ingress trace shows every TLP ACKed and the endpoint never received them. Explain.

The ACK came from the switch, not from the endpoint (§8, §14). An ACK is a Data Link Layer acknowledgement to the immediate link neighbour, meaning that neighbour accepted the TLP and the transmitter may release it from its replay buffer (14.4, 21.2 §3). It is link-local and not a delivery receipt. §14 measured a switch dropping 10% internally with ingress ACKs at 20,000 in both the healthy and faulty cases and egress at 20,000 versus 18,028. One probe proves transmission; two prove forwarding.

Q3. Why does 25.1 §3 say the analyzer should not be the first tool, and what makes it the right tool later?

Because a capture without a prediction is unfalsifiable (§4). A trace is large enough that any hypothesis finds support in it, so a trigger, a window and a prediction about what would refute you are all required — and the first eight chapters of Module 25 exist to produce that third part. §4's table lists what each one yields: "does a Configuration Request appear at all", "did any Completion for this Tag appear and when", "which class stopped first". Every one is falsifiable and answerable from a trace; none could be formed without the earlier work.

Q4. What is the analyzer's strongest single measurement, and what is its limit?

Unmatched requests — non-posted requests with no matching Completion (§6). It is structural rather than interpretive: every non-posted request must be answered (10.2), and matching (Requester ID, Tag) pairs across a trace involves no judgement. With probes on more than one link it localises the failure to a fabric segment, which no single device's counters can do. Its limit is that localisation is not diagnosis — 25.7 §3's five causes for an unanswered request all look identical in the trace.

Q5. Your unmatched-request count is 16 out of 15,620. Is that a fault?

No — that is the in-flight floor (§6, mutation 26). A capture always ends mid-conversation, so requests outstanding at the end have no Completion in the trace. §13's healthy baseline was exactly this: 16 unmatched of 15,620. A tool that flags every unmatched request produces false findings at exactly the rate of the requester's outstanding depth, so the count must be compared against that capacity before any conclusion.

Q6. Two internal traces are impossible to align with a wire trace. What is missing?

A common event carrying the same identifier in both timelines (§9, §11 Block 5, mutations 9 to 11). Without a periodic marker visible on the wire and recorded internally with a shared sequence number, alignment falls back to timestamp proximity, which produces wrong pairings under load and does so silently. The stronger mechanism is a correlation key carried in a device-controlled packet field and recorded against the internal event (§9, P3, P6) — that matches records exactly rather than approximately, and it must never displace protocol-required information (P4).

Q7. Your on-chip trace is empty for the window you care about. What must you check before concluding nothing happened?

The drop counters (§17 case 10, P23, P24, mutations 14 and 17). A freeze that discarded arriving events, or an arbiter that starved one source, produces an empty window indistinguishable from a quiet one. An absence is evidence only when the trace is known to be complete for that window — which is why §11's buffer counts events dropped after a freeze and the arbiter counts every loser per source, rather than dropping either silently. This is 25.1 §6's rule applied to the debug hardware itself.

Q8. Why does this chapter cite no analyzer vendor documentation for any PCIe fact?

Because a decoder is one vendor's reading of the protocol, not the protocol (§1). It can be incomplete, out of date, or specific to a capture mode, and treating a decoded field list as a specification imports errors with total confidence — 25.4 mutation 34's failure in a new form. Vendor documentation is a legitimate source for tool behaviour, which is why this chapter describes categories of trigger and capture capability and tells you to read your instrument's own documentation for the specifics, while every protocol fact is cited to a live chapter that carries its own citation.

20. Module 25 Complete

Nine chapters. One method, applied at nine depths.

One idea produced all nine chapters, and it is 25.1 §2's: find the last event that is provably correct, and the first that is not. Every chapter is that method applied to a different chain — a configuration conversation, a state machine's exits, five ownership boundaries, four descriptor handoffs, an outstanding-request table, a wait-for graph, and now a fabric.

A second idea emerged that no single chapter set out to produce. It appeared first in 25.6 §13, where both counterexamples turned out to be monitors rather than engines. It appeared again in 25.7 §13, where an epoch-derived diagnostic read zero while 1.9 MB landed in wrong buffers. It appeared again in 25.8 §16, where a global liveness check reported health while a traffic class was permanently wedged. And this chapter is the same lesson at fabric scale: the analyzer is an instrument that cannot see the quantity the fault changes.

An instrument derived from the mechanism it checks is blind exactly where that mechanism fails.

The remedy is the same every time and it is not subtle: an independent oracle. A ground-truth serial with no hardware counterpart. A per-class counter instead of an aggregate. A two-sided conservation check instead of a one-sided one. A second probe on the other side of the switch. In every case the fix was a measurement that shares no machinery with the thing under suspicion.

And the third idea is the one this chapter ends on. Eight chapters built instruments inside devices — a first-rejection recorder, an ownership tracker, an outstanding table, a wait-for graph. This chapter's measurement is that the fabric's own view cannot see any of them, and that the gap is closed by exporting internal state into a shared timeline rather than by looking harder from outside.

That is a design obligation, not a debugging technique. §13's three invisible faults each generated hundreds of internal events; the information existed and was not exported. A device that cannot explain itself is not debuggable in a fabric — and every instrument in Module 25 is worth building before the bug arrives, because after it arrives there is nothing left to read.

Module 26 turns outward, from finding faults to the systems that carry PCIe. It opens with 26.1 — PCIe in CPUs, where the Root Complex translates between two ordering domains, and continues through GPUs, SSD controllers and network adapters — four endpoints with four different bottlenecks on one link.