Debug Checklist — What to Check First, and Why That Order

Module 25 taught how to debug PCIe. This gate is about the first ten minutes.

The individual checks are not new — they are the instruments Module 25 built. What this chapter adds is the sequence, and §11 measured that the sequence is worth as much as the checks.

1. What This Gate Owns — and What It Must Not Steal

It owns the triage order. Not the checks, not the mechanisms, not the fixes.

Module 25 owns every check on this list. If a reviewer needs one explained during triage, the triage is not the problem.

What it must not steal. It must not re-teach the LTSSM, re-derive credit accounting, or re-explain descriptor ownership. A debug checklist that contains explanations is a textbook, and nobody reads a textbook during an incident.

And one boundary with 30.5. That gate asks why is this slower than declared; this one asks why does this not work. They share almost no checks, because a performance question presumes the system functions.

2. Disposition Semantics — for Triage Rather Than Sign-Off

Triage produces a different set of verdicts from a sign-off gate.

Two readings, and the first is the discipline that makes triage converge.

A hypothesis is eliminated by a positive observation, never by a missing one. "No errors are logged" eliminates nothing — 25.5 §4 measured a fault class that returns Successful Completions with wrong data, and 26.4 §5 measured 23,495 dropped frames with every PCIe counter clean. "The link is in L0 and the width is x8" is a positive observation. "I didn't see anything wrong" is not.

And UNREACHABLE is a real verdict that must be written down. 25.9 §13 measured three of five fault classes producing byte-identical traces; if the only available instrument is a link probe, those hypotheses are genuinely unreachable and the honest record says so. The alternative — leaving them OPEN forever — is what produces investigations that never converge.

3. The Ordering Principle

Order checks by information per unit effort, not by suspicion.

rank = (hypotheses eliminated) ÷ (cost to perform)

Three properties make a check cheap and informative, and the first four items on §12's list have all three:

It reads existing state rather than creating it. A configuration-space read costs nothing to set up. A capture costs a probe, a trigger, a run and an analysis pass.

Its answer is unambiguous. The LTSSM state is a value; "the trace looked normal" is a judgement.

And its answer is positive in both directions. Reading the link state as L0 eliminates the whole training family; reading it as anything else confirms it. A check that only informs you when it fails is worth half as much.

The instinct this principle fights is reaching for the most powerful instrument first. §11 measured that instinct costing 2.8× the mean effort — and the power is illusory for most faults, because the analyzer sees the Link and 25.9 §3 enumerates what is not on it: descriptor ownership, tag-allocator state, queue occupancy, host software intent, the BAR-local target, and internal scheduler state.

4. The First Four Checks

These resolve four of seven fault classes for a cost of 1 to 5 (§11), and all four are register reads.

1 — Link state, width and speed. Is the LTSSM in L0, and at the expected width and speed? Cost 1. Eliminates the entire training family (25.3, 25.4). A link not in L0 makes every downstream question premature, and this is 25.1 §5's "prove L0 first" as a single read.

2 — Configuration space and BAR readback. Does the device enumerate, and do its BARs hold the values software programmed? Cost 1. Eliminates enumeration and gross address-ownership faults (25.2, 25.5). A correct readback rules out one of five ownership boundaries and no more — 25.5 §7 is explicit that the register is not the comparator.

3 — Bus-master enable. Is the device permitted to originate transactions? Cost 1. The single highest-yield check for "device enumerates and moves no data" (28.3 §14 case 1). It is host-controlled, frequently the answer, and it costs one read.

4 — Descriptor and queue occupancy. Does the device have work, and somewhere to put results? Cost 2. Separates "the device is broken" from "the device has nothing to do" (25.6, 28.3 §14 case 2). A device with no posted buffers cannot write host memory regardless of how healthy everything else is.

5. The Next Three

Costlier, and each is targeted at a specific class the first four cannot reach.

5 — Outstanding-tag occupancy. Cost 2. Pinned means saturated; far below the pool means starving (27.4 §14). This is the check that separates two causes producing identical throughput.

6 — Completion-timeout counters, with the byte counts. Cost 2. 25.7 §3 established that a partial timeout — some bytes arrived — is a fundamentally different investigation from a request that received nothing, and that the majority of timeouts are partial.

7 — Host-memory stall counters. Cost 3. Distinguishes a link limit from a platform limit (26.1 §16 case 5). The most expensive of the register reads and still far cheaper than a capture.

6. When the Analyzer Is Correct

It is the right instrument for exactly one class of question, and knowing that class is what keeps it in position 8 rather than position 1.

Use it when the question is: did this leave A and arrive at B?

That is a fabric question, and nothing else answers it. 25.9 §6 establishes unmatched requests — non-posted requests with no Completion — as its strongest derived measurement, because it localises where ownership stopped moving across a fabric no single device can observe.

Do not use it when the question is: why did the device return that value?

§11 measured analyzer-only resolving three of seven fault classes. The four it cannot reach are internal, and no capture depth or trigger changes that.

And two placement rules that follow from 25.9 §8:

One probe proves transmission; two prove forwarding. An ACK on the ingress link of a switch is a link-local Data Link Layer acknowledgement, not a delivery receipt — measured at 20,000 ingress ACKs while the switch discarded 1,972 TLPs internally.

And set the pre-trigger window generously. The cause precedes the symptom. A capture starting at the error contains the consequence.

7. The Waveform — What "Working" Looks Like

Three readings.

The first three signals are identical in both cases. A debugger who checks the link and the BARs, sees both healthy, and reaches for a capture has spent the capture to discover that bus_master_en is low — which check 3 answers for a cost of 1.

The failing case has no error anywhere. Nothing is malformed; no status bit sets. The device is behaving correctly for a device that has not been permitted to originate.

And the ordering in the trace is the ordering of the checklist. Each signal depends on the one before it, which is why checking them out of order wastes the earlier answers.

8. Classify Before You Investigate

Five classes, and naming the class is most of the work (25.9 §12).

The discriminating experiment for the first two is whether the transactions on the wire are legal. If they are legal and the data is wrong, the analyzer is done helping and the investigation moves inside the device.

And the discriminating experiment for the last two is substitution: a platform fault moves with the platform, a software fault moves with the driver, and an integration fault appears only in the combination.

9. RTL — The Triage Register

Block 1 — the triage summary register. The single most useful piece of debug hardware a PCIe endpoint can carry.

// One read answers checks 1-5. The value is intended for a human under time
// pressure, so every field is a POSITIVE fact rather than an error flag —
// §2's rule that hypotheses are eliminated by positive observation.
module triage_register #(
  parameter int unsigned NTAG = 64,
  parameter int unsigned NQ   = 8
)(
  input  logic clk,
  input  logic rst_n,
  input  logic [4:0]                    ltssm_state,      // encoding owned by 18.1 §10
  input  logic                          link_up,
  input  logic [5:0]                    neg_width,        // NEGOTIATED, not capability
  input  logic [3:0]                    neg_speed,
  input  logic                          cfg_done,
  input  logic                          bus_master_en,
  input  logic [$clog2(NTAG+1)-1:0]     tag_occupancy,
  input  logic [$clog2(NQ+1)-1:0]       queues_with_work,
  input  logic [$clog2(NQ+1)-1:0]       queues_with_buffers,
  output logic [31:0]                   triage_lo,
  output logic [31:0]                   triage_hi
);
 
  // Packed so a single 64-bit read (or two 32-bit reads) answers checks 1-5.
  // The negotiated width is reported, never the capability: 26.2 §4 and
  // 27.4 §20 both record engineers concluding a link is healthy from a
  // capability register that describes what the port COULD do.
  always_comb begin
    triage_lo = {  8'd0,
                   1'b0,
                   bus_master_en,          // check 3
                   cfg_done,               // check 2
                   link_up,                // check 1
                   neg_speed,              // check 1
                   neg_width,              // check 1
                   3'd0,
                   ltssm_state };          // check 1
    triage_hi = { 16'd0,
                  {(8-$clog2(NQ+1)){1'b0}}, queues_with_buffers,   // check 4
                  {(8-$clog2(NTAG+1)){1'b0}}, tag_occupancy };     // check 5
  end
 
endmodule

Block 2 — the first-failure latch. 25.1 §6's rule: the first failure, not the most recent.

// The most recent failure is usually a CONSEQUENCE of the first, and a
// register that overwrites destroys the only informative sample. This block
// captures once and holds, on a reset domain the recovery path does not
// clear — 25.8 §2 measured a watchdog reset erasing the evidence it existed
// to explain.
module first_failure_latch #(
  parameter int unsigned NTAG = 64
)(
  input  logic clk,
  input  logic rst_n_persistent,      // NOT the watchdog/recovery reset
  input  logic [4:0]  ev_link_left_l0,
  input  logic        ev_cpl_timeout,
  input  logic        ev_ur_received,
  input  logic        ev_tag_exhausted,
  input  logic        ev_desc_starved,
  input  logic [4:0]  ltssm_now,
  input  logic [$clog2(NTAG+1)-1:0] occ_now,
  input  logic [31:0] cycle_now,
  input  logic        clear,
  output logic        captured,
  output logic [2:0]  first_cause,
  output logic [4:0]  cap_ltssm,
  output logic [$clog2(NTAG+1)-1:0] cap_occ,
  output logic [31:0] cap_cycle,
  output logic [31:0] event_count
);
 
  logic [2:0] this_cause;
 
  // Priority is the ORDER OF DEPENDENCE, not severity: a link that left L0
  // explains every downstream symptom, so it must win. Ranking by severity
  // would record a completion timeout that the link drop caused.
  always_comb begin
    if      (|ev_link_left_l0) this_cause = 3'd1;
    else if (ev_desc_starved)  this_cause = 3'd2;
    else if (ev_tag_exhausted) this_cause = 3'd3;
    else if (ev_ur_received)   this_cause = 3'd4;
    else if (ev_cpl_timeout)   this_cause = 3'd5;
    else                       this_cause = 3'd0;
  end
 
  always_ff @(posedge clk or negedge rst_n_persistent) begin
    if (!rst_n_persistent || clear) begin
      captured <= 1'b0; first_cause <= 3'd0;
      cap_ltssm <= '0; cap_occ <= '0; cap_cycle <= '0; event_count <= '0;
    end else if (this_cause != 3'd0) begin
      if (!captured) begin
        captured    <= 1'b1;
        first_cause <= this_cause;
        cap_ltssm   <= ltssm_now;
        cap_occ     <= occ_now;
        cap_cycle   <= cycle_now;
      end
      // Saturating: a wedged system that wraps its event counter reports a
      // small number, which reads as "this is rare" — the opposite of true.
      if (event_count != 32'hFFFF_FFFF) event_count <= event_count + 32'd1;
    end
  end
 
endmodule

10. Same-Cycle Audit

11. Measured — Check Order

Per-fault detail for the checklist order:

Three readings.

The first four checks resolve four of seven faults for a cost of 1 to 5. That is the entire justification for the ordering, and it is the reason §4 exists as its own section.

Analyzer-only leaves four unresolved at maximum cost. It is not that the analyzer is slow — it is that four of these faults are internal and it cannot see them at any price (25.9 §3).

And analyzer-first is the worst of the four orders on the mean, at 2.8× cheapest-first, because the capture is paid before every register read that would have answered the question — and then the register reads are still needed for the four internal classes.

12. The Debug Checklist — 22 Questions

Before touching anything

1. What is the symptom, stated as an observation rather than a diagnosis? ("no data moves", not "DMA is broken")
2. What was the last known good state, and what changed since? (25.1 §5)
3. What is the expected next event that is not happening? (25.1 §2)
4. Which of §8's five classes does the symptom suggest, and what would falsify that?

Checks 1–4 — cost 1 to 5, resolve four of seven classes

5. Is the LTSSM in L0? (25.3, 25.4)
6. Is the negotiated width and speed as expected — not the capability? (26.2 §4)
7. Does configuration space read back, and do the BARs hold what software programmed? (25.2, 25.5)
8. Is bus-master enable set? (28.3 §14 case 1)
9. Does the device have work — descriptors posted, queues non-empty? (25.6)
10. Does it have somewhere to put results — free buffers? (28.3 §14 case 2)

Checks 5–7 — targeted

11. Is outstanding-tag occupancy pinned or starving? (27.4 §14)
12. Are there completion timeouts, and are they partial or zero-byte? (25.7 §3)
13. If credit stalls are nonzero, is occupancy simultaneously below the pool? (27.4 §15)
14. Are host-memory stalls dominating? (26.1 §16 case 5)

Instrument integrity — before believing any of the above

15. Has the first-failure record been read, and does it name a cause or a consequence? (§10 audit A)
16. Is the event counter saturated rather than small? (§9 Block 2)
17. Was the debug state captured on a reset domain the recovery does not clear? (25.8 §2)
18. Has any instrument been validated against a known fault? (25.5 §15 case 7)

Only now, the analyzer

19. Is the question a fabric question — did this leave A and arrive at B? (§6)
20. If the question is internal, has it been recorded as UNREACHABLE by this instrument? (§2)
21. Are there probes on both sides of any switch in the path? (25.9 §8)
22. Is the pre-trigger window long enough to contain the cause rather than the consequence? (25.9 §5)

13. Misconceptions

"Start with the analyzer — it sees everything." Why it sounds plausible: it is the most capable instrument in the room, and it does see every packet. What really happens: it sees the Link. §11 measured analyzer-only resolving 3 of 7 classes and analyzer-first costing 2.8× the mean of cheapest-first. What it causes: a capture session that ends by discovering a bus-master enable bit is clear — an answer that cost 1 unit and was paid for with 12.

"No errors are logged, so the hardware is fine." Why it sounds plausible: error logging exists precisely to report faults. What really happens: 25.5 §4 measured a fault class returning Successful Completions with wrong data, and 26.4 §5 measured 23,495 dropped frames with every PCIe counter clean. What it causes: a hypothesis eliminated by an absence, which §2 forbids — and the real fault stays OPEN while the investigation moves on.

"The link is at the right width and speed, so PCIe is working." Why it sounds plausible: the physical negotiation is the hardest part to get right, and it succeeded. What really happens: it eliminates the training family and nothing else. Bus-mastering, descriptors, tags, translation and the host path are all downstream and all untested by it. What it causes: the investigation skipping checks 3 and 4 — which §11 measured as two of the four cheapest resolutions.

"Read the most recent error register." Why it sounds plausible: it is the freshest information available. What really happens: the most recent failure is usually a consequence of the first (25.1 §6). A link that dropped explains every timeout that followed. What it causes: an investigation that begins at 25.7 for a fault that belongs to 25.3 — §10 audit A is the hardware that prevents it.

"The system recovered, so we lost nothing." Why it sounds plausible: recovery is the desired outcome and the system is working again. What really happens: the recovery reset erases the evidence unless the debug state lives on a domain it does not clear (25.8 §2). What it causes: a fault that recurs indefinitely, destroying its own evidence each time.

"Zero credits means we found it." Why it sounds plausible: zero permission means nothing can be sent, which is what a stuck link looks like. What really happens: 25.8 §14 measured a healthy link entering 818 zero-credit episodes and resolving every one. What it causes: triage stopping at a reading that a working system produces routinely, and the wait-for-cycle test — the only structural one — never being run.

14. Understanding Check

Q1. A device enumerates, its BARs read back correctly, and it never writes host memory. You have a protocol analyzer set up. What should you do first and why?

Read the bus-master enable bit — check 3, cost 1 (§4, §11). The capture cannot answer this cheaply and may not answer it at all: a device that never originates produces no traffic to capture, so the trace is empty and an empty trace is consistent with several causes. §11's per-fault table shows this fault confirmed at cost 3 by the checklist order and at cost 12 by an analyzer-first order. The general principle is §3's: order by information per unit effort, and a register read that eliminates a whole class costs less than any capture.

Q2. Your triage record says "completion timeout." Why might that be the wrong place to start, and what would you check?

Because a timeout is frequently a consequence rather than a cause (§10 audit A, 25.1 §6). A link that left L0 produces completion timeouts for every outstanding request, and a record ranked by severity would name the timeout. Check the captured LTSSM state in the same record: if the link left L0, the investigation belongs to 25.3, not 25.7. This is why §9 Block 2 orders its priority list by dependence and documents that choice in the code — the ordering is a causal claim.

Q3. A colleague eliminates "DMA ownership bug" because no errors are logged. Evaluate.

Not eliminated (§2). A hypothesis is eliminated by a positive observation, and "no errors" is an absence. The specific counter-evidence is measured: 25.6 §12 measured descriptor-ownership faults producing wrong data with no error, and 25.9 §13 measured three of five fault classes producing byte-identical traces. The positive observation that would eliminate it is a conservation check — bytes described versus bytes moved, and status signalled exactly once per job — which is what 25.6 §4's four laws provide.

Q4. §11 shows analyzer-only resolving 3 of 7 fault classes. Why is that a property of the instrument rather than of the model?

Because the split follows a measured result rather than an assumption (25.9 §13). That chapter injected five faults into a model with separated traffic and fault RNGs and compared full traces: three produced byte-identical output while generating 790, 313 and 313 internal events that never reached the wire. The classes it cannot see — descriptor ownership, tag-allocator state, queue occupancy, the BAR-local target — are the same classes §11 marks internal. No capture depth or trigger changes this, because the information never becomes a packet.

Q5. Why does the first-failure latch take a different reset from the rest of the device?

Because the usual response to a hung link is a reset, and that reset would erase the record (25.8 §2, §10 audit C). A debug register cleared by the recovery it exists to explain provides nothing: the fault recurs, recovers, and destroys its own evidence each time. The record must survive the recovery to be read afterwards, which is checklist item 17 — and it is an architectural decision that has to be made before the bug appears, not after.

15. What Comes Next

Two gates have now closed: a number, and a triage order.

The last one closes a person. 30.7 is the revision gate — not a summary of the curriculum, but the short list of things a PCIe engineer must be able to derive under pressure rather than recall. The distinction matters, because 27.4 §21 established the senior standard: an engineer who knows the shape of an answer without the number is more useful than one who knows the number and not the shape.