Credit Deadlocks — Zero Credits Is Not Evidence

Chapter 25.7 debugged a fault that at least fires. This one produces no event at all: no error, no status bit, no counter reaching a limit. The link is simply, permanently, waiting — and every pool reads a value that a busy healthy link reads too.

1. Sources, Scope, and the Boundary With Module 16 and 22.3

2. Why This Fault Has No Event

Every other fault in Module 25 announces itself. A link that will not train has a state to read (25.3). A BAR mismatch produces a UR or a wrong value (25.5). A DMA stall freezes a pointer (25.6). A request that is never answered expires a timer (25.7).

A credit deadlock produces none of those. Nothing is in error. Every agent is doing exactly what it is designed to do — waiting for a resource it correctly requires, using logic that is correct in isolation. The link is not broken; it is blocked, and the difference is not observable from any single agent's state.

That has three consequences, and they shape every instrument in the chapter.

First, there is no natural trigger. 25.9 will show a trace that simply stops, with the last packets before the stop looking entirely ordinary. There is no error packet to trigger on because no error occurred.

Second, the local view is always innocent. Ask any agent what it is doing and it answers "waiting for credit", which is a legal and common thing to be doing. §14's healthy link was doing it 818 times. The fault is not in any agent — it is in the relation between them, which is why §4's instrument is a graph rather than a register.

Third, timeouts do not help, and can hurt. A watchdog can detect that nothing has moved, but it cannot say why, and its recovery — a reset — destroys the evidence. A deadlock is one of the few faults where the recovery mechanism is the main obstacle to diagnosis, and §16 case 11 is about capturing state before it fires.

3. Zero Credits Proves Nothing

The healthy link entered a zero-credit state 818 times and left it 818 times. The longest such episode lasted 20 steps. A register dump taken during any of them shows a pool at zero and an agent waiting — which is precisely what a deadlocked link shows.

Read the DEP-1 column too, because it is the more instructive one. That configuration had 10,754 zero-credit episodes — thirteen times the healthy rate — and 10,753 of them resolved. It is a link under genuine pressure, visibly degraded in throughput from 51,376 to 36,219 sends, and it is not deadlocked. A diagnosis based on "how often do we hit zero" would rank it as the sickest configuration in the table. It is the healthiest of the two faulted ones.

Three things follow.

Pool value is not evidence. Neither is the frequency of hitting zero, nor the fraction of time spent at zero. All three are properties of load, and 22.3 owns their interpretation as a performance signal.

Duration is weak evidence, and only in one direction. The healthy longest episode was 20 steps; the deadlocked one was 79,465 and still counting when the run ended. A very long episode is suggestive. But the threshold separating them is a property of the traffic, not of the design — change the load and the healthy maximum moves. §10's detector uses duration only as a gate on the real test, never as the test.

And the real test is structural. A deadlock is a cycle. §4 builds the graph.

4. The Wait-For Graph

The instrument is a directed graph with one edge rule:

node    an agent that can be blocked  (a port, a virtual channel, a queue)
edge    A -> B  exists when A is blocked waiting for a resource
                that B currently holds and will not release
                until A releases something first
 
deadlock  <=>  a cycle exists  AND  every node on it is blocked

Both halves matter. A cycle in a graph of agents that are merely slow is not a deadlock — B will release eventually, A will proceed. The edge must mean "blocked", not "busy", and the difference is the single most common error in building this instrument (mutation 18).

The graph is small. Real deadlocks in PCIe systems involve two or three agents, because the dependencies that close a cycle come from ordering obligations and buffer-sharing decisions, and those exist between a handful of adjacent blocks. A cycle-detection algorithm over four nodes is a few gates, which is why §10 Block 6 implements one in hardware rather than leaving it to software.

And a cycle is stable in a way a stall is not. That is what makes it a usable test: the graph can be sampled repeatedly, and a genuine deadlock produces the same cycle every time, indefinitely. §10's detector requires the cycle to persist across a configurable window for exactly this reason — a transient cycle formed by two agents that happen to be blocked in the same cycle is a coincidence, and §14's healthy run produced 818 opportunities for that coincidence.

5. Starvation, Congestion, and Deadlock

Three conditions produce the same register values and need three different responses. Getting the classification wrong wastes the most time of any error in this chapter.

Starvation is a rate problem. The credits come back; they come back too slowly, or too few were advertised for the round trip. Every remedy is quantitative — a deeper pool, a shorter return latency, a different traffic mix — and 22.3 owns all of them.

Deadlock is a structural problem, and no quantitative remedy touches it. This is worth being emphatic about because the instinct is always to increase the pool depth:

A deeper pool does not remove a deadlock. It delays it.

§14's cycle closed at step 1,540 with pools 8 deep. Doubling them moves the closure later and changes nothing about whether it closes — the cycle exists in the dependency structure, and the pool depth only determines how much traffic can be absorbed before the structure matters. A "fix" that makes the problem appear less often in testing and still appear in the field is worse than no fix, because it converts a reproducible bug into an intermittent one.

And starvation has a diagnostic use that deadlock does not. A link that starves is telling you something true about its sizing. §14's DEP-1-alone configuration starved heavily — 10,754 episodes against a healthy 818 — and that number is a genuine measurement of a genuine cost. The mistake is reading it as a deadlock precursor. It is not; DEP-1 alone never deadlocked in 80,000 steps.

6. Two Safe Dependencies, One Cycle

This is the chapter's central result and the reason it is an Expert chapter.

DEP-2 alone is indistinguishable from healthy — 51,376 sends, 818 episodes, longest 20. It costs nothing measurable. A reviewer looking for a performance regression finds none.

DEP-1 alone costs throughput and never deadlocks. 36,219 sends against 51,376, and 10,754 starvation episodes of which 10,753 resolved. It is visibly a design with a cost, and it is safe.

Together: 258 sends, 118 retirements, and no progress for the final 79,462 steps. The detected cycle is A -> B -> A.

Three lessons follow, and the third is the one that generalises.

A deadlock is not a defect in a module. It is a property of a composition, and it can be assembled from parts that are each correct, each reviewed, and each shipped independently. No module-level review can find it, because at module level there is nothing to find.

The two dependencies were probably introduced by different people at different times. DEP-2 is an implementation choice inside B; DEP-1 is a protocol choice about how A signals completion. Neither author could see the other's decision, and neither decision is wrong.

And this is why the wait-for graph must be built from the design's dependencies, not from its symptoms. You cannot find this fault by looking harder at either agent. You find it by writing down what each agent must acquire before it can release, and checking the result for cycles — which is a design-time analysis that §10's hardware detector then confirms at run time.

7. Availability Is Not Permission

Chapter 23.6 §4 established the law: a resource that will be needed later must be reserved before the action that commits to needing it. This chapter is where the law's absence is most expensive, and also — measurably — where its absence alone is not sufficient to deadlock.

The rule for non-posted requests. A read commits the completer to returning Completions. Those Completions need completion-buffer space at the requester. Issuing a read because NPH credit is available, without reserving the completion space the reply will need, is "availability is not permission" exactly.

§14 measured it alone, and the result is not what the law's importance suggests:

Statistically identical, and no deadlock. The reason is structural: in that topology the requester's completion buffer always drains, so unreserved reads produce backpressure — a bounded queue that grows and shrinks — rather than a cycle.

Missing reservation becomes fatal only when combined with a dependency that blocks the drain, and §15 measures that composition. This is the same shape as §6 and it is worth stating as a general principle for this chapter:

A missing reservation is not a deadlock. It is a precondition for one — it removes the bound that would otherwise keep a queue finite, so that any dependency blocking the drain becomes permanent instead of temporary.

That reframes how to prioritise the finding. A code review that finds an unreserved read should not report "deadlock risk" with a measurement that does not support it. It should report "an unbounded queue whose drain path must now be proven never to block" — a claim that is precise, checkable, and true.

8. Credit Accounting That Can Be Trusted

Before any deadlock analysis, the accounting itself must be correct, because a credit counter that is wrong produces every symptom in this chapter without any structural fault at all.

Three requirements, all from Module 16 by citation.

Consumed and returned are separate counters. 16.5 and 16.6 own the mechanism; available credit is a derived quantity. A design that maintains a single up/down counter cannot distinguish "no credits were returned" from "too many were consumed", and both are common bugs.

Data credits are computed from the payload, not from the packet. The unit is 4 DW = 16 bytes and a payload consumes Roundup(Length / FC unit size) credits — 16.1 §1. A design that charges one data credit per packet under-charges every payload above 16 bytes, and the receiver's buffer overflows long before the sender believes it should.

Header and data credits are independent and both must be checked. A transmitter with data credit and no header credit cannot send. Checking only one is mutation 5, and its symptom is a receiver overrun rather than a deadlock — a different and more damaging fault.

And one arithmetic hazard deserves naming, because it converts a stall into corruption. Credit counters are unsigned. A decrement below zero wraps to a very large value, and the transmitter concludes it has abundant credit and floods a receiver with no space. The check must be available >= needed before the subtraction, never a subtraction followed by a sign test — there is no sign. P9 and P10 assert both halves, and this is the single highest-severity mutation in §17.

9. The Waveform

Five readings, and the first two are the section's whole argument.

Cycles 1–2 and cycles 6–7 are identical in every credit signal. cred_pd_b is zero in both; a_blocked is high in both. A register dump cannot tell them apart — this is §3's 818 episodes drawn out in time.

edge_b_to_a is what differs, and it is a derived signal that does not exist unless §10 builds it. In the first episode B is not blocked, so no edge points back and there is no cycle. In the second, both edges exist.

cycle_det asserts at cycle 8, two cycles after the cycle forms. The delay is the persistence window (§4): a cycle that exists for one cycle is a coincidence, and §14's healthy run offered 818 chances for one.

cred_pd_b never returns in the second episode, and it never will — B cannot send the Completion that would let A retire, and A cannot retire the Completion that would free B. Nothing in the trace after cycle 6 changes, ever.

And there is no error signal anywhere in the figure. That is not an omission (§2).

10. RTL — The Deadlock Instruments

Block 1 — the package: pools, classes, node identifiers.

package cd_dbg_pkg;
 
  // The six pools of 16.1.  Kept as an enum so that every counter, every
  // assertion and every report names a pool rather than an index, and a
  // header/data confusion becomes a type error rather than a silent bug.
  typedef enum logic [2:0] {
    FC_PH = 3'd0, FC_PD = 3'd1,
    FC_NPH= 3'd2, FC_NPD= 3'd3,
    FC_CPLH=3'd4, FC_CPLD=3'd5
  } fc_pool_e;
 
  // The three traffic classes.  §15's result is that liveness must be
  // tracked per class, because PCIe ordering is per class.
  typedef enum logic [1:0] {
    TC_POSTED = 2'd0, TC_NONPOSTED = 2'd1, TC_COMPLETION = 2'd2
  } tclass_e;
 
  function automatic int unsigned gw(input int unsigned n);
    return (n <= 1) ? 1 : $clog2(n);
  endfunction
 
  // The data credit unit is 4 DW = 16 bytes, and a payload consumes
  // Roundup(Length / unit) credits — 16.1 §1, sourced there to
  // Microchip AN2948 DS00002948B §3.3.7.  Nothing here derives a new
  // constant; this function is the cited arithmetic, once, in one place.
  localparam int unsigned FC_DATA_UNIT_BYTES = 16;
  function automatic int unsigned data_credits(input int unsigned len_bytes);
    return (len_bytes + FC_DATA_UNIT_BYTES - 1) / FC_DATA_UNIT_BYTES;
  endfunction
 
endpackage

Block 2 — credit accounting with separate consumed and returned counters. §8's first requirement, and the module every other instrument reads.

module cd_credit_account #(
  parameter int unsigned CNT_W = 16
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                 advertised_valid,
  input  logic [CNT_W-1:0]     advertised,      // from FC init (15.2)
  input  logic                 consume,
  input  logic [CNT_W-1:0]     consume_amount,
  input  logic                 ret,             // UpdateFC received (16.6)
  input  logic [CNT_W-1:0]     ret_amount,
  output logic [CNT_W-1:0]     limit,
  output logic [CNT_W-1:0]     consumed,
  output logic [CNT_W-1:0]     available,
  output logic                 underflow_attempt
);
 
  // Consumed and returned are SEPARATE.  A single up/down counter cannot
  // distinguish "nothing was returned" from "too much was consumed", and
  // §16 case 3 needs exactly that distinction.
  logic [CNT_W-1:0] returned_total;
 
  // Unsigned arithmetic: a decrement below zero WRAPS to a huge value and
  // the transmitter concludes it has abundant credit (§8).  The guard is
  // the comparison BEFORE the subtraction — there is no sign bit to test.
  always_comb begin
    available         = limit - consumed;
    underflow_attempt = consume && (consume_amount > available);
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      limit <= '0; consumed <= '0; returned_total <= '0;
    end else begin
      if (advertised_valid) limit <= advertised;
      if (ret) begin
        returned_total <= returned_total + ret_amount;
        limit          <= limit + ret_amount;
      end
      // The subtraction happens ONLY when the guard passes.
      if (consume && !underflow_attempt) consumed <= consumed + consume_amount;
    end
  end
 
endmodule

Block 3 — the blocked-agent detector. The edge rule of §4 depends entirely on this being "blocked", not "busy".

module cd_blocked_detect #(
  parameter int unsigned HOLD = 8          // MODEL PARAMETER
)(
  input  logic clk,
  input  logic rst_n,
  input  logic want_to_send,               // this agent has work queued
  input  logic can_send,                   // it has the credit to send it
  input  logic sent,                       // it actually sent something
  output logic blocked,
  output logic [15:0] blocked_for
);
 
  // "Blocked" means: has work, cannot proceed, and has not proceeded for
  // HOLD consecutive cycles.  Using want_to_send && !can_send alone is
  // mutation 18 — it is true for one cycle constantly on a busy link, and
  // it turns §14's 818 healthy episodes into 818 false cycles.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      blocked_for <= '0; blocked <= 1'b0;
    end else if (sent || !want_to_send) begin
      blocked_for <= '0; blocked <= 1'b0;
    end else if (!can_send) begin
      if (blocked_for != 16'hFFFF) blocked_for <= blocked_for + 16'd1;
      blocked <= (blocked_for >= HOLD);
    end
  end
 
endmodule

Block 4 — the wait-for edge builder. One module per agent; its output is one row of the adjacency matrix.

module cd_waitfor_edges #(
  parameter int unsigned NNODE = 4
)(
  input  logic                    blocked,
  // For each other node: does it currently hold the resource this agent
  // is waiting for?  This is a DESIGN-SPECIFIC mapping and must be
  // written from the dependency analysis of §6, not inferred at run time.
  input  logic [NNODE-1:0]        holder_mask,
  output logic [NNODE-1:0]        edges_out
);
 
  // An edge exists only while this agent is BLOCKED.  A busy agent that
  // is merely waiting its turn creates no edge, which is the distinction
  // §4 insists on and §5 measures the cost of getting wrong.
  assign edges_out = blocked ? holder_mask : '0;
 
endmodule

Block 5 — the credit reservation tracker. §7's law, instrumented so the precondition is visible even when it has not yet caused a deadlock.

module cd_reserve_track #(
  parameter int unsigned CNT_W = 16
)(
  input  logic clk,
  input  logic rst_n,
  input  logic             issue_np,          // a non-posted request is issued
  input  logic [CNT_W-1:0] cpl_credits_needed,
  input  logic             cpl_returned,
  input  logic [CNT_W-1:0] cpl_credits_freed,
  input  logic [CNT_W-1:0] cpl_pool_limit,
  output logic [CNT_W-1:0] reserved,
  output logic             may_issue,
  output logic             unreserved_issue,  // §7's precondition, counted
  output logic [31:0]      unreserved_count
);
 
  // Reserve BEFORE issue: the completion space a reply will need is
  // committed at the moment the request is committed (23.6 §4).
  always_comb begin
    may_issue        = (reserved + cpl_credits_needed) <= cpl_pool_limit;
    unreserved_issue = issue_np && !may_issue;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      reserved <= '0; unreserved_count <= '0;
    end else begin
      if (issue_np && may_issue) reserved <= reserved + cpl_credits_needed;
      if (cpl_returned)          reserved <= reserved - cpl_credits_freed;
      // Counted even when it does not deadlock.  §14 measured that an
      // unreserved issue alone produced NO deadlock — it is a precondition
      // (§7), and reporting it as a deadlock would be unsupported.
      if (unreserved_issue && unreserved_count != 32'hFFFF_FFFF)
        unreserved_count <= unreserved_count + 32'd1;
    end
  end
 
endmodule

Block 6 — the cycle detector. Iterative reachability over a small adjacency matrix; a cycle exists iff any node reaches itself.

module cd_cycle_detect #(
  parameter int unsigned NNODE   = 4,
  parameter int unsigned PERSIST = 64        // MODEL PARAMETER (§4)
)(
  input  logic clk,
  input  logic rst_n,
  input  logic [NNODE-1:0] adj [NNODE],      // adj[i][j] = i waits for j
  input  logic [NNODE-1:0] node_blocked,
  output logic             cycle_now,
  output logic             cycle_stable,     // persisted PERSIST cycles
  output logic [NNODE-1:0] cycle_nodes,
  output logic [15:0]      cycle_age
);
 
  // Transitive closure by repeated squaring.  NNODE is small (§4), so the
  // full closure is a handful of gates and needs no sequencing.
  logic [NNODE-1:0] reach [NNODE];
 
  always_comb begin
    for (int i = 0; i < NNODE; i++) reach[i] = adj[i];
    for (int k = 0; k < NNODE; k++)
      for (int i = 0; i < NNODE; i++)
        for (int j = 0; j < NNODE; j++)
          if (reach[i][k] && reach[k][j]) reach[i][j] = 1'b1;
 
    cycle_nodes = '0;
    for (int i = 0; i < NNODE; i++)
      // A node on a cycle reaches itself, AND must itself be blocked —
      // the second half of §4's rule.  Omitting it is mutation 19.
      if (reach[i][i] && node_blocked[i]) cycle_nodes[i] = 1'b1;
 
    cycle_now = (cycle_nodes != '0);
  end
 
  // Persistence.  §14's healthy run produced 818 zero-credit episodes,
  // each an opportunity for a transient cycle; requiring the SAME cycle to
  // hold for PERSIST cycles is what makes the test usable.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      cycle_age <= '0; cycle_stable <= 1'b0;
    end else if (!cycle_now) begin
      cycle_age <= '0; cycle_stable <= 1'b0;
    end else begin
      if (cycle_age != 16'hFFFF) cycle_age <= cycle_age + 16'd1;
      cycle_stable <= (cycle_age >= PERSIST);
    end
  end
 
endmodule

Block 7 — per-class progress monitors. §15's result: a global liveness check misses a per-class deadlock.

module cd_class_progress #(
  parameter int unsigned IDLE_LIMIT = 32'd1000    // DEBUG HEURISTIC
)(
  input  logic clk,
  input  logic rst_n,
  input  logic p_progress,      // any Posted forward progress
  input  logic np_progress,
  input  logic cpl_progress,
  output logic [31:0] p_idle,
  output logic [31:0] np_idle,
  output logic [31:0] cpl_idle,
  output logic        any_class_starved,
  output cd_dbg_pkg::tclass_e starved_class,
  output logic        global_progress
);
 
  import cd_dbg_pkg::*;
 
  // The global signal is computed too, precisely so that §15's comparison
  // can be made: it is TRUE while Posted traffic flows, even when the
  // Completion class has been wedged for tens of thousands of cycles.
  assign global_progress = p_progress || np_progress || cpl_progress;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      p_idle <= '0; np_idle <= '0; cpl_idle <= '0;
    end else begin
      p_idle   <= p_progress   ? 32'd0 : (p_idle   + 32'd1);
      np_idle  <= np_progress  ? 32'd0 : (np_idle  + 32'd1);
      cpl_idle <= cpl_progress ? 32'd0 : (cpl_idle + 32'd1);
    end
  end
 
  always_comb begin
    any_class_starved = (p_idle   >= IDLE_LIMIT) ||
                        (np_idle  >= IDLE_LIMIT) ||
                        (cpl_idle >= IDLE_LIMIT);
    // Report the MOST idle class, so the diagnosis names one thing.
    if (cpl_idle >= p_idle && cpl_idle >= np_idle)      starved_class = TC_COMPLETION;
    else if (np_idle >= p_idle)                          starved_class = TC_NONPOSTED;
    else                                                 starved_class = TC_POSTED;
  end
 
endmodule

Block 8 — the episode classifier. §5's three-way distinction, computed rather than guessed.

module cd_episode_classify #(
  parameter int unsigned LONG_EPISODE = 32'd5000   // DEBUG HEURISTIC
)(
  input  logic clk,
  input  logic rst_n,
  input  logic any_pool_zero,
  input  logic cycle_stable,
  input  logic clear,
  output logic [31:0] episodes,
  output logic [31:0] episodes_resolved,
  output logic [31:0] longest_episode,
  output logic [31:0] current_episode,
  output logic        verdict_deadlock,
  output logic        verdict_starvation
);
 
  logic in_episode;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      episodes <= '0; episodes_resolved <= '0; longest_episode <= '0;
      current_episode <= '0; in_episode <= 1'b0;
    end else if (any_pool_zero) begin
      if (!in_episode) begin
        in_episode <= 1'b1; current_episode <= 32'd1; episodes <= episodes + 32'd1;
      end else begin
        current_episode <= current_episode + 32'd1;
        if (current_episode > longest_episode) longest_episode <= current_episode;
      end
    end else begin
      // An episode that ENDS is evidence of health, and counting the
      // resolutions is what makes §14's "818 of 818" statement possible.
      if (in_episode) episodes_resolved <= episodes_resolved + 32'd1;
      in_episode <= 1'b0; current_episode <= '0;
    end
  end
 
  // The verdict is STRUCTURAL.  Duration is only a gate on reporting, never
  // the test — §3 measured that a healthy link's episodes reach 20 steps
  // and a starving one's reach 76, with no fixed threshold separating
  // either from a deadlock.
  assign verdict_deadlock   = cycle_stable;
  assign verdict_starvation = in_episode && !cycle_stable &&
                              (current_episode >= LONG_EPISODE);
 
endmodule

Block 9 — the freeze-frame capture. §2's third consequence: the recovery destroys the evidence, so the evidence must be latched first.

module cd_freeze_frame #(
  parameter int unsigned NNODE = 4,
  parameter int unsigned NPOOL = 6,
  parameter int unsigned CNT_W = 16
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                    trigger,          // cycle_stable
  input  logic [NNODE-1:0]        adj      [NNODE],
  input  logic [NNODE-1:0]        blocked,
  input  logic [CNT_W-1:0]        avail    [NPOOL],
  input  logic [31:0]             p_idle, np_idle, cpl_idle,
  input  logic                    clear,
  output logic                    frozen,
  output logic [NNODE-1:0]        f_adj    [NNODE],
  output logic [NNODE-1:0]        f_blocked,
  output logic [CNT_W-1:0]        f_avail  [NPOOL],
  output logic [31:0]             f_p_idle, f_np_idle, f_cpl_idle
);
 
  // Captured ONCE, at the trigger.  A watchdog reset is the usual response
  // to a hung link and it erases every value here; reading these registers
  // after recovery is the only way to see the state that caused it (§16
  // case 11).  They must therefore survive the reset that follows —
  // in a real design, on a reset domain that the watchdog does not clear.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      frozen <= 1'b0;
    end else if (trigger && !frozen) begin
      frozen     <= 1'b1;
      f_adj      <= adj;
      f_blocked  <= blocked;
      f_avail    <= avail;
      f_p_idle   <= p_idle;
      f_np_idle  <= np_idle;
      f_cpl_idle <= cpl_idle;
    end
  end
 
endmodule

Block 10 — the aggregate verdict. One structure a debugger reads, with the classification already made.

// verilog_lint: waive-start module-filename
module cd_verdict #(
  parameter int unsigned NNODE = 4
)(
  input  logic                    clk,
  input  logic                    rst_n,
  input  logic                    cycle_stable,
  input  logic [NNODE-1:0]        cycle_nodes,
  input  logic                    any_class_starved,
  input  cd_dbg_pkg::tclass_e     starved_class,
  input  logic                    verdict_starvation,
  input  logic [31:0]             unreserved_count,
  input  logic                    underflow_seen,
  output logic [2:0]              verdict,
  output logic [NNODE-1:0]        verdict_nodes
);
 
  import cd_dbg_pkg::*;
 
  // 0 healthy · 1 credit underflow (accounting) · 2 starvation ·
  // 3 per-class stall · 4 DEADLOCK
  // Underflow ranks ABOVE deadlock because a wrapped counter produces
  // every deadlock symptom without any structural fault (§8), and
  // diagnosing a cycle on top of broken accounting wastes the session.
  always_comb begin
    verdict_nodes = cycle_nodes;
    if      (underflow_seen)     verdict = 3'd1;
    else if (cycle_stable)       verdict = 3'd4;
    else if (any_class_starved)  verdict = 3'd3;
    else if (verdict_starvation) verdict = 3'd2;
    else                         verdict = 3'd0;
  end
 
endmodule
// verilog_lint: waive-stop module-filename

11. Assertions

Credit accounting properties — §8.

// P1 — available is exactly limit minus consumed, always.
property p1_available_derived;
  @(posedge clk) disable iff (!rst_n)
    (available == (limit - consumed));
endproperty
a_p1: assert property (p1_available_derived);
 
// P2 — consumed never exceeds limit.  The invariant a wrapped unsigned
// counter violates catastrophically (§8).
property p2_consumed_bounded;
  @(posedge clk) disable iff (!rst_n)
    (consumed <= limit);
endproperty
a_p2: assert property (p2_consumed_bounded);
 
// P3 — nothing is sent without sufficient credit.
property p3_no_send_without_credit;
  @(posedge clk) disable iff (!rst_n)
    consume |-> (consume_amount <= available);
endproperty
a_p3: assert property (p3_no_send_without_credit);
 
// P4 — an underflow ATTEMPT is reported, never silently absorbed.
property p4_underflow_reported;
  @(posedge clk) disable iff (!rst_n)
    (consume && (consume_amount > available)) |-> underflow_attempt;
endproperty
a_p4: assert property (p4_underflow_reported);
 
// P5 — the guard prevents the subtraction.  P3 states the obligation;
// this states that the implementation actually enforces it.
property p5_guard_blocks_subtract;
  @(posedge clk) disable iff (!rst_n)
    underflow_attempt |=> $stable(consumed);
endproperty
a_p5: assert property (p5_guard_blocks_subtract);
 
// P6 — consumed and returned are tracked separately, so a return is
// always visible as an increase in limit rather than a decrease elsewhere.
property p6_return_raises_limit;
  @(posedge clk) disable iff (!rst_n)
    (ret && (ret_amount > 0)) |=> (limit > $past(limit));
endproperty
a_p6: assert property (p6_return_raises_limit);
 
// P7 — data credits are charged from the payload length using the cited
// 16-byte unit (16.1 §1), not one credit per packet.
property p7_data_credit_arithmetic;
  @(posedge clk) disable iff (!rst_n)
    (consume && (pool == cd_dbg_pkg::FC_PD))
      |-> (consume_amount == cd_dbg_pkg::data_credits(payload_bytes));
endproperty
a_p7: assert property (p7_data_credit_arithmetic);
 
// P8 — header and data credit are both required before a send.
property p8_header_and_data;
  @(posedge clk) disable iff (!rst_n)
    send_posted |-> ((avail_ph >= 1) && (avail_pd >= data_credits(payload_bytes)));
endproperty
a_p8: assert property (p8_header_and_data);
 
// P9 — the comparison precedes the subtraction (no sign test on unsigned).
property p9_compare_before_subtract;
  @(posedge clk) disable iff (!rst_n)
    consume |-> !underflow_attempt;
endproperty
a_p9: assert property (p9_compare_before_subtract);
 
// P10 — available never wraps upward across a consume.
property p10_no_upward_wrap;
  @(posedge clk) disable iff (!rst_n)
    (consume && !underflow_attempt) |=> (available <= $past(available));
endproperty
a_p10: assert property (p10_no_upward_wrap);

Pool independence properties — §1's six pools.

// P11 — spending in one pool never changes another.  The independence of
// the six pools is what makes per-class analysis meaningful (§15).
property p11_pool_independence;
  @(posedge clk) disable iff (!rst_n)
    (consume && (pool == cd_dbg_pkg::FC_PD)) |=> $stable(avail_cpld);
endproperty
a_p11: assert property (p11_pool_independence);
 
// P12 — a Completion never consumes Posted credit.
property p12_cpl_uses_cpl_pool;
  @(posedge clk) disable iff (!rst_n)
    send_completion |-> (pool inside {cd_dbg_pkg::FC_CPLH, cd_dbg_pkg::FC_CPLD});
endproperty
a_p12: assert property (p12_cpl_uses_cpl_pool);
 
// P13 — a Non-Posted request never consumes Completion credit at issue.
property p13_np_uses_np_pool;
  @(posedge clk) disable iff (!rst_n)
    send_nonposted |-> (pool inside {cd_dbg_pkg::FC_NPH, cd_dbg_pkg::FC_NPD});
endproperty
a_p13: assert property (p13_np_uses_np_pool);

Blocked-detection properties — §4's edge rule.

// P14 — blocked means work queued AND unable to proceed, for HOLD cycles.
// Using instantaneous (want && !can) is mutation 18: it is true for one
// cycle constantly, turning §14's 818 healthy episodes into false cycles.
property p14_blocked_requires_hold;
  @(posedge clk) disable iff (!rst_n)
    blocked |-> (blocked_for >= HOLD);
endproperty
a_p14: assert property (p14_blocked_requires_hold);
 
// P15 — sending clears blocked immediately.
property p15_send_clears_blocked;
  @(posedge clk) disable iff (!rst_n)
    sent |=> !blocked;
endproperty
a_p15: assert property (p15_send_clears_blocked);
 
// P16 — an idle agent with no work is never blocked.
property p16_idle_not_blocked;
  @(posedge clk) disable iff (!rst_n)
    !want_to_send |-> !blocked;
endproperty
a_p16: assert property (p16_idle_not_blocked);
 
// P17 — an edge exists only while the source is blocked.
property p17_edge_needs_blocked;
  @(posedge clk) disable iff (!rst_n)
    (edges_out != '0) |-> blocked;
endproperty
a_p17: assert property (p17_edge_needs_blocked);
 
// P18 — no self-edge.  An agent waiting on itself is a modelling error,
// not a deadlock, and it produces a spurious trivial cycle.
property p18_no_self_edge;
  @(posedge clk) disable iff (!rst_n)
    !adj[node_id][node_id];
endproperty
a_p18: assert property (p18_no_self_edge);

Cycle-detection properties — §4.

// P19 — a node reported on a cycle is itself blocked.  The second half of
// §4's rule; omitting it reports cycles among busy agents (mutation 19).
property p19_cycle_nodes_blocked;
  @(posedge clk) disable iff (!rst_n)
    (cycle_nodes[n]) |-> node_blocked[n];
endproperty
a_p19: assert property (p19_cycle_nodes_blocked);
 
// P20 — cycle_now implies at least two distinct nodes (given P18).
property p20_cycle_needs_two;
  @(posedge clk) disable iff (!rst_n)
    cycle_now |-> ($countones(cycle_nodes) >= 2);
endproperty
a_p20: assert property (p20_cycle_needs_two);
 
// P21 — the stable verdict requires persistence.  §14's healthy run gave
// 818 opportunities for a one-cycle coincidence.
property p21_stable_needs_persist;
  @(posedge clk) disable iff (!rst_n)
    cycle_stable |-> (cycle_age >= PERSIST);
endproperty
a_p21: assert property (p21_stable_needs_persist);
 
// P22 — the cycle age resets the moment the cycle breaks.
property p22_age_resets;
  @(posedge clk) disable iff (!rst_n)
    !cycle_now |=> (cycle_age == '0);
endproperty
a_p22: assert property (p22_age_resets);
 
// P23 — a stable cycle never spontaneously clears.  Once a genuine
// deadlock is detected it does not resolve; a detector that flickers is
// detecting something else.
property p23_stable_is_stable;
  @(posedge clk) disable iff (!rst_n)
    cycle_stable |=> (cycle_stable || clear || !rst_n);
endproperty
a_p23: assert property (p23_stable_is_stable);
 
// P24 — the reachability closure is complete: if i reaches k and k reaches
// j, then i reaches j.  A truncated closure misses longer cycles.
property p24_closure_transitive;
  @(posedge clk) disable iff (!rst_n)
    (reach[i][k] && reach[k][j]) |-> reach[i][j];
endproperty
a_p24: assert property (p24_closure_transitive);

Reservation properties — §7.

// P25 — a non-posted request is issued only with completion space reserved.
property p25_reserve_before_issue;
  @(posedge clk) disable iff (!rst_n)
    (issue_np && issued) |-> may_issue;
endproperty
a_p25: assert property (p25_reserve_before_issue);
 
// P26 — reserved never exceeds the pool limit.
property p26_reserved_bounded;
  @(posedge clk) disable iff (!rst_n)
    (reserved <= cpl_pool_limit);
endproperty
a_p26: assert property (p26_reserved_bounded);
 
// P27 — a returned completion releases exactly what it reserved.
property p27_reserve_released;
  @(posedge clk) disable iff (!rst_n)
    cpl_returned |=> (reserved == ($past(reserved) - $past(cpl_credits_freed)));
endproperty
a_p27: assert property (p27_reserve_released);
 
// P28 — an unreserved issue is COUNTED even though §14 measured that it
// alone does not deadlock.  Reporting it as a deadlock would be
// unsupported; not reporting it at all loses §7's precondition.
property p28_unreserved_counted;
  @(posedge clk) disable iff (!rst_n)
    unreserved_issue |=> (unreserved_count > $past(unreserved_count));
endproperty
a_p28: assert property (p28_unreserved_counted);

Per-class liveness properties — §15.

// P29 — each class makes progress.  Stated PER CLASS because §15 measured
// a Completion-class deadlock hiding behind flowing Posted traffic.
property p29_completion_progress;
  @(posedge clk) disable iff (!rst_n)
    (cpl_work_pending) |-> s_eventually cpl_progress;
endproperty
a_p29: assert property (p29_completion_progress);
 
property p30_posted_progress;
  @(posedge clk) disable iff (!rst_n)
    (p_work_pending) |-> s_eventually p_progress;
endproperty
a_p30: assert property (p30_posted_progress);
 
property p31_nonposted_progress;
  @(posedge clk) disable iff (!rst_n)
    (np_work_pending) |-> s_eventually np_progress;
endproperty
a_p31: assert property (p31_nonposted_progress);
 
// P32 — the global progress signal is NOT used as the liveness criterion.
// Stated as an implication that a starved class is reported even while
// global progress is true — §15's measured blind spot.
property p32_class_starvation_visible;
  @(posedge clk) disable iff (!rst_n)
    (cpl_idle >= IDLE_LIMIT) |-> any_class_starved;
endproperty
a_p32: assert property (p32_class_starvation_visible);

Instrument-integrity properties.

// P33 — a resolved episode is counted as resolved.  §14's "818 of 818"
// statement is only possible if resolutions are counted.
property p33_resolution_counted;
  @(posedge clk) disable iff (!rst_n)
    (in_episode && !any_pool_zero) |=> (episodes_resolved > $past(episodes_resolved));
endproperty
a_p33: assert property (p33_resolution_counted);
 
// P34 — the verdict is structural, never derived from duration alone.
property p34_verdict_structural;
  @(posedge clk) disable iff (!rst_n)
    verdict_deadlock |-> cycle_stable;
endproperty
a_p34: assert property (p34_verdict_structural);
 
// P35 — the freeze frame captures once and holds.  The watchdog reset that
// follows a hang erases everything else (§2).
property p35_freeze_once;
  @(posedge clk) disable iff (!rst_n)
    frozen |=> (frozen && $stable(f_adj) && $stable(f_avail));
endproperty
a_p35: assert property (p35_freeze_once);

Cover — the anti-vacuity set.

// P36 — the verdict is total: every sampled state maps to exactly one of
// the five verdict codes.  A state that maps to none leaves a hung link
// with no classification, which is the one outcome §5 exists to prevent.
property p36_verdict_total;
  @(posedge clk) disable iff (!rst_n)
    (verdict inside {3'd0, 3'd1, 3'd2, 3'd3, 3'd4});
endproperty
a_p36: assert property (p36_verdict_total);
 
// P36's covers — the interesting states must actually occur.  A testbench that
// never composes two blocking dependencies never forms a cycle, and
// P19-P24 are all vacuously satisfied.  §14 needed BOTH DEP-1 and DEP-2
// installed to produce a single one.
c1_zero_credit:    cover property (@(posedge clk) disable iff (!rst_n) any_pool_zero);
c2_episode_resolve:cover property (@(posedge clk) disable iff (!rst_n)
                     any_pool_zero ##[1:$] !any_pool_zero);
c3_cycle_transient:cover property (@(posedge clk) disable iff (!rst_n)
                     cycle_now && !cycle_stable);
c4_cycle_stable:   cover property (@(posedge clk) disable iff (!rst_n) cycle_stable);
c5_underflow_try:  cover property (@(posedge clk) disable iff (!rst_n) underflow_attempt);
c6_unreserved:     cover property (@(posedge clk) disable iff (!rst_n) unreserved_issue);
c7_class_starve:   cover property (@(posedge clk) disable iff (!rst_n)
                     any_class_starved && global_progress);
c8_all_pools_zero: cover property (@(posedge clk) disable iff (!rst_n)
                     (avail_ph == 0) && (avail_pd == 0) && (avail_cpld == 0));

12. Executable Counterexamples

Seven minimal designs, each violating one property, each with the failing stimulus stated.

Counterexample A — the unsigned credit decrement (violates P2, P9, P10).

module ce_a_credit_underflow #(parameter int unsigned W = 8)(
  input  logic clk, rst_n, input logic consume, input logic [W-1:0] amount,
  output logic [W-1:0] available
);
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) available <= 8'd16;
    else if (consume) available <= available - amount;   // <-- no guard
endmodule
 
// Failing stimulus: available = 8'd4, a packet needing 8 data credits.
// Golden: the send is blocked; available stays 4.
// This:   4 - 8 = 8'hFC = 252.  The transmitter now believes it has 252
//         credits and floods a receiver with space for none.
// P2 fails, P9 fails, P10 fails (available INCREASED across a consume).
// Observable consequence: receiver buffer overrun — a more damaging fault
// than the deadlock this chapter is about, produced by the same counter.

Counterexample B — the instantaneous blocked signal (violates P14, and defeats the detector).

module ce_b_instant_blocked (
  input logic want_to_send, can_send, output logic blocked
);
  assign blocked = want_to_send && !can_send;      // <-- no persistence
endmodule
 
// Failing stimulus: §14's healthy run — 818 zero-credit episodes, each a
// few cycles long, each resolving.
// Golden: blocked never asserts (episodes are shorter than HOLD).
// This:   blocked asserts in 818 separate episodes, and whenever two
//         agents happen to be momentarily blocked at once, a cycle is
//         reported on a link that is working perfectly.
// P14 fails.
// Observable consequence: 818 false deadlock reports on a healthy link.
// After the first few, the detector is disabled — and the real deadlock,
// when it comes, is not reported at all.

Counterexample C — the cycle detector without the blocked check (violates P19).

module ce_c_cycle_ignores_blocked #(parameter int unsigned N = 4)(
  input  logic [N-1:0] reach [N], output logic [N-1:0] cycle_nodes
);
  always_comb begin
    cycle_nodes = '0;
    for (int i = 0; i < N; i++) if (reach[i][i]) cycle_nodes[i] = 1'b1;
  end                                           // <-- node_blocked ignored
endmodule
 
// Failing stimulus: two agents each waiting their turn in an arbiter, both
// making progress, with edges present because each holds a resource the
// other wants.
// Golden: no cycle — neither node is blocked.
// This:   a cycle is reported on a link that is transferring at full rate.
// P19 fails.
// Observable consequence: the detector reports deadlock during normal
// arbitration, which is §5's starvation-versus-deadlock confusion built
// into hardware.

Counterexample D — the duration-threshold verdict (violates P34).

module ce_d_duration_verdict (
  input logic [31:0] current_episode, output logic verdict_deadlock
);
  assign verdict_deadlock = (current_episode >= 32'd100);   // <-- duration only
endmodule
 
// Failing stimulus A: §14's DEP-1-alone configuration, whose episodes
//   reach 76 steps and always resolve.
// Failing stimulus B: a heavier traffic mix on a HEALTHY link, whose
//   episodes exceed 100 steps and always resolve.
// Golden: starvation in both; no deadlock.
// This:   stimulus B is reported as a deadlock.
// P34 fails.
// Observable consequence: the threshold is a property of the TRAFFIC, not
// of the design (§3).  Any value chosen produces false positives on a
// heavier load and false negatives on a lighter one.

Counterexample E — the global liveness watchdog (violates P29, P32).

module ce_e_global_watchdog (
  input  logic clk, rst_n, input logic any_tlp_sent,
  output logic hung
);
  logic [31:0] idle;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) idle <= '0;
    else idle <= any_tlp_sent ? 32'd0 : (idle + 32'd1);
  assign hung = (idle >= 32'd1000);
endmodule
 
// Failing stimulus: §15's DEP-1 + DEP-2 + no-reservation configuration.
// Golden: the Completion class is deadlocked; report it.
// This:   Posted traffic continues at 20,915 sends, so any_tlp_sent keeps
//         asserting and idle never reaches the limit.  hung stays low for
//         the entire run while the Completion class is wedged for 79,788
//         steps.
// P29 fails.  P32 fails.
// Observable consequence: §15 measured the global detector reporting
// NOTHING while a per-class detector fired at step 1,210.

Counterexample F — deeper pools as the fix (violates P34's spirit; measured, not asserted).

// Not a module — a configuration change, presented as a counterexample
// because it is the most common proposed "fix" in this chapter.
//   DEP-1 + DEP-2, pools 8 deep  -> cycle closes at step 1,540
//   DEP-1 + DEP-2, pools N deep  -> cycle closes later; it still closes
//
// The cycle exists in the DEPENDENCY STRUCTURE (§6).  Pool depth
// determines only how much traffic is absorbed before the structure
// matters.  Doubling the pools halves the reproduction rate in testing
// and changes nothing in the field.
// Observable consequence: a reproducible bug becomes an intermittent one,
// which is strictly worse — the same fault, now much harder to catch.

Counterexample G — reservation released on issue rather than on return (violates P27).

module ce_g_release_on_issue #(parameter int unsigned W = 16)(
  input  logic clk, rst_n, input logic issue_np, input logic [W-1:0] need,
  output logic [W-1:0] reserved
);
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) reserved <= '0;
    else if (issue_np) begin
      reserved <= reserved + need;
      reserved <= reserved;          // released immediately on the next issue
    end
endmodule
 
// Failing stimulus: a burst of non-posted requests issued back to back
// before any Completion returns.
// Golden: reserved accumulates; may_issue goes false; issuing throttles.
// This:   reserved never accumulates, so may_issue is always true and the
//         burst issues without bound.
// P27 fails.  P26 fails once the burst exceeds the pool.
// Observable consequence: §7's precondition — an unbounded queue whose
// drain path must now be proven never to block.  Alone it does not
// deadlock (§14 measured 51,372 sends, no deadlock).  Composed with a
// blocking drain it is permanent (§15).

13. What the Model Is, and Its Two Corrections

14. Measured Behaviour — Zero Credits, and the Cycle

Two agents, pools 8 deep, 80,000 steps:

Detected cycle: A -> B -> A, with no progress for the final 79,462 steps.

Five readings.

The healthy link resolved 818 of 818 zero-credit episodes. Zero credits is the normal state of a busy link and carries no diagnostic information (§3).

DEP-1 alone is the sickest-looking configuration and is not deadlocked. 10,754 episodes — thirteen times the healthy rate — and 10,753 resolved. A frequency-based diagnosis ranks it worst; it is safe.

DEP-2 alone is indistinguishable from healthy on every measure. It costs nothing detectable and is half of a permanent failure.

The composition collapses throughput by 199×, from 51,376 sends to 258, and never recovers.

And the longest-episode column is the honest one. 20, 20, 76, 20, then 79,465. The gap is large — but the healthy maximum is a property of the load, and §12's counterexample D shows why no fixed threshold on that column is a usable test.

15. Measured Behaviour — Global Versus Per-Class Detection

The last row is the important one and it took a correction to see. The global detector was written first, and it reported no deadlock on a configuration that had retired 43 Completions in 80,000 steps. The initial reading was that the configuration simply was not deadlocked. It was — in the Completion class, permanently, from step 1,210 — while Posted traffic kept flowing at 20,915 sends and kept the global "has anything moved recently" check satisfied throughout.

Three consequences.

Liveness must be tracked per traffic class, because PCIe's ordering and flow control are per class (16.1, 13.4). A link where one class is permanently blocked and another flows freely is deadlocked, and it looks busy from every aggregate measure.

The Non-Posted idle count of 2 is the clue that makes it diagnosable. Two of three classes are wedged for ~79,790 steps and one is fully active. That asymmetry names the class, and naming the class is most of the diagnosis — it points directly at which pool and which dependency to examine.

And this is the more common shape in real systems than the total-freeze case. A completely stopped link gets noticed in seconds. A link that is 90% functional with one traffic class wedged runs for a long time being described as "slow" or "intermittent" — which is 25.6 §3's observation about corrupting-versus-stalling faults, appearing again in a different layer.

16. Verification — Mutations

Forty-two mutations. Every "Caught by" entry names a property from §11.

Mutations 16, 19 and 26 are the three that break the detector rather than the design, and all three produce false positives on a healthy link. That failure mode is worse than a missed detection, because a detector that cries wolf on §14's 818 healthy episodes gets disabled — and then the real deadlock is not reported at all.

Mutation 39 is in the table because it is the most commonly proposed fix, and §12's counterexample F is the measurement that it is not one.

Mutations 40 to 42 break the testbench. Their symptom is that everything passes.

17. Debugging

Case 1 — a pool reads zero and the link looks stuck.

Do not conclude anything. §14 measured a healthy link at 818 zero-credit episodes, all resolved, longest 20 steps — a register dump inside any of them is indistinguishable from a deadlock. Sample the pool again after a known interval. If it changes at all, it is starvation or congestion and 22.3 owns it. If it never changes, that is still not proof — go to case 4. Confidence: this case exists to prevent a conclusion, not to reach one.

Case 2 — before anything structural, verify the credit accounting.

A wrapped unsigned counter produces every symptom in this chapter with no structural fault at all (§8, counterexample A). Check that consumed <= limit (P2), that data credits are charged as Roundup(bytes / 16) rather than one per packet (P7, mutation 4), and that both header and data are checked before a send (P8). This is why §10's verdict ranks underflow above deadlock — diagnosing a cycle on top of broken accounting wastes the entire session. Confidence: high; these are three register reads.

Case 3 — verify the detector before believing it, in either direction.

If it reports a deadlock, check that blocked has a persistence window (P14, mutation 16) and that cycle nodes are required to be blocked (P19, mutation 19). §14's healthy run offers 818 chances for a false cycle, and a detector without both guards will take them. If it reports nothing, check the covers (P36) — mutation 40's symptom is silence. A detector never validated against a known composition is not evidence (25.5 §15 case 7's rule).

Case 4 — the link has not moved and no error is set anywhere.

That is the expected signature (§2): nothing is in error, every agent is legally waiting. Build the wait-for graph (§4). If a cycle is present and stable, it is a deadlock and the cycle names the agents. If no cycle is present, it is starvation with a very slow return path, and the question becomes which pool and why — 16.6 and 22.3. Confidence: high; this is the chapter's primary test.

Case 5 — throughput is 90% normal and one operation type never completes.

The per-class case, and it is the most common real shape (§15). §15 measured Posted flowing at 20,915 sends while the Completion class was wedged from step 1,210, with idle counts of Posted 79,790, Non-Posted 2, Completion 79,788. Read the per-class idle counters; the asymmetry names the class, and the class names the pool. A global "has anything moved" check reports healthy throughout — do not use one. Confidence: high, and this case is why P29–P32 are per class.

Case 6 — the problem appears only under sustained load and never in short tests.

Expected for a deadlock, and it is a consequence of pool depth rather than of rarity. The cycle exists in the dependency structure from the first cycle of operation; the pools absorb traffic until they cannot. §14's cycle closed at step 1,540 with pools 8 deep. Do not conclude the fault is intermittent — it is deterministic in structure and load-dependent in timing. Confidence: high when the reproduction time scales with pool depth, which is itself a diagnostic worth measuring.

Case 7 — someone proposes deeper pools as the fix.

Deeper pools delay a deadlock; they never remove it (§5, counterexample F). The cycle is structural. The change converts a reproducible bug into an intermittent one, which is strictly worse — the same fault, now much harder to catch, and now likely to be found by a customer rather than by a test. The only fix is breaking the cycle, and §6 shows the break can be made at either dependency.

Case 8 — a code review found a non-posted request issued without reserving completion space.

Report it accurately. §14 measured that configuration at 51,372 sends with no deadlock — statistically identical to healthy. It is a precondition, not a deadlock (§7): it removes the bound that keeps a queue finite. The correct finding is "an unbounded queue whose drain path must now be proven never to block" — precise, checkable, and true. Reporting it as a deadlock is an unsupported claim (mutation 33), and one that will be dismissed the first time someone runs the configuration and sees it work.

Case 9 — two design decisions each passed review and the composition hangs.

That is the normal way a deadlock arrives (§6). §14 measured DEP-1 alone at 36,219 sends with no deadlock, DEP-2 alone indistinguishable from healthy, and the pair at 258 sends with a permanent cycle at step 1,540. No module-level review can find this, because at module level there is nothing to find. The analysis has to be done on the composition: write down, for each agent, what it must acquire before it can release, and check the result for cycles.

Case 10 — the zero-credit rate is thirteen times higher than baseline.

That is a starvation measurement and it may be entirely benign. §14's DEP-1-alone configuration had 10,754 episodes against a healthy 818 — and 10,753 resolved. It was the highest-frequency configuration in the table and it never deadlocked. The frequency of hitting zero is a load and sizing signal owned by 22.3; it is not a deadlock precursor. Confidence: high — check the resolution count, which is the number that matters.

Case 11 — the watchdog reset recovers the link and you have no evidence.

Capture before the recovery, not after (§2, §10 Block 9, mutations 37 and 38). The freeze frame must latch the adjacency matrix, the blocked vector, all six pool values and the per-class idle counts at the trigger, and it must live on a reset domain the watchdog does not clear. Reading those registers after recovery is the only way to see the state that caused it — and this is the one fault in Module 25 where the recovery mechanism is the main obstacle to diagnosis.

Case 12 — the design has run for years and hung once, in the field.

Do not assume a marginal-timing or transient cause. A structural cycle that requires a specific composition of queue occupancies can go years without closing and is fully deterministic when it does. The productive move is the design-time analysis of case 9 — enumerate the acquire-before-release dependencies and look for cycles — rather than an attempt to reproduce. §6's cycle needed both DEP-1 and DEP-2 and a particular occupancy to close; the first two are readable from the design without reproducing anything.

18. Misconceptions

"Zero credits means the link is stuck." §14's healthy link hit zero 818 times and resolved every one (§3). The pool value carries no diagnostic information.

"A high zero-credit rate is a deadlock precursor." §14's DEP-1-alone configuration had 10,754 episodes to a healthy 818, and 10,753 resolved. It never deadlocked. That number is a sizing signal, owned by 22.3.

"Deeper pools fix it." They delay it (§5, counterexample F). The cycle is structural; pool depth determines only how much traffic is absorbed first. The change makes a reproducible bug intermittent.

"Each block was reviewed, so the system is safe." §14 measured two individually safe dependencies composing into a permanent cycle (§6). A deadlock is a property of a composition and is invisible at module level.

"A missing reservation is a deadlock bug." §14 measured it at 51,372 sends with no deadlock (§7). It is a precondition — it removes a bound. Reporting more than that is unsupported.

"Our watchdog would catch it." Not if it is global. §15 measured Posted traffic flowing at 20,915 sends while the Completion class was wedged from step 1,210, with the global detector never firing (counterexample E).

"If it were deadlocked, something would report an error." Nothing is in error (§2). Every agent is legally waiting for a resource it correctly requires.

"A long zero-credit episode proves deadlock." The threshold is a property of the traffic (counterexample D). §14's healthy maximum was 20 steps and its starving one 76; a heavier load moves both.

"The analyzer will show it." It shows a trace that stops, with ordinary packets before the stop and no error to trigger on (25.9). §2 explains why there is nothing to see.

19. Understanding Check

Q1. A pool reads zero and an agent is waiting. What have you established?

Nothing (§3). §14 measured a healthy link at 818 zero-credit episodes, all 818 resolved, longest 20 steps — a register dump inside any of them is identical to a deadlocked one. Zero credits is the normal state of a busy link. The next step is to sample again after a known interval: if the value changes, it is starvation or congestion (22.3); if it never changes, that is still not proof, and the actual test is the wait-for graph (§4).

Q2. What distinguishes starvation from deadlock, and why is duration not the test?

A stable cycle in the wait-for graph (§4, §5). Starvation resolves; deadlock never does. Duration fails as a test because the threshold is a property of the traffic, not of the design (counterexample D): §14's healthy maximum episode was 20 steps and its most-starved configuration reached 76, and a heavier load moves both upward. Any fixed threshold produces false positives on heavier traffic and false negatives on lighter. §10's detector uses duration only as a gate on the structural test, never as the test (P34).

Q3. DEP-1 alone: 36,219 sends, no deadlock. DEP-2 alone: indistinguishable from healthy. Together: 258 sends, permanent. What does this tell you about how deadlocks are found?

That they cannot be found at module level (§6). Neither dependency is a defect — a completion retire that emits a status write, and a single in-order egress queue, are both ordinary and both defensible. They were probably introduced by different people at different times, and neither author could see the other's decision. A deadlock is a property of the composition, so the analysis must be done on the composition: for each agent, write down what it must acquire before it can release, and check the resulting graph for cycles (§17 case 9).

Q4. Your team proposes doubling the credit pools to fix a hang. What do you say?

That it delays the deadlock and does not remove it (§5, counterexample F, §17 case 7). The cycle exists in the dependency structure; pool depth determines only how much traffic is absorbed before the structure matters. §14's cycle closed at step 1,540 with pools 8 deep, and a deeper pool moves that number without changing whether it closes. The change converts a reproducible bug into an intermittent one — strictly worse, because the same fault is now much harder to catch and likely to be found in the field rather than in test.

Q5. A code review finds a read issued without reserving completion space. How should the finding be worded?

As an unbounded queue whose drain path must be proven never to block (§7, §17 case 8). §14 measured that configuration at 51,372 sends with no deadlock — statistically identical to healthy — because the completion buffer always drained, so unreserved reads produced bounded backpressure rather than a cycle. Calling it a deadlock is an unsupported claim (mutation 33) that will be dismissed the first time someone runs it successfully. It is a precondition: it removes the bound that would otherwise keep the queue finite, so any dependency blocking the drain becomes permanent instead of temporary.

Q6. Throughput is 90% of normal and one operation type never completes. Why will a watchdog miss this?

Because a global liveness check is satisfied by the classes that are still moving (§15, counterexample E). §15 measured Posted traffic at 20,915 sends while the Completion class was wedged from step 1,210 — idle counts Posted 79,790, Non-Posted 2, Completion 79,788 — and the global detector never fired, while a per-class detector fired at 1,210. Liveness must be tracked per traffic class, because PCIe's flow control and ordering are per class (16.1, 13.4). The asymmetry between the class idle counts is what names the class, and naming the class is most of the diagnosis.

Q7. Your deadlock detector fires regularly on a link that is working. What are the two likely causes?

An instantaneous blocked signal, or a cycle test that does not require the nodes to be blocked (mutations 16 and 19, counterexamples B and C). Without a persistence window, want && !can is momentarily true constantly, and §14's 818 healthy episodes give 818 opportunities for two agents to be momentarily blocked at once (P14). Without the blocked check, agents merely taking turns in an arbiter form a cycle while transferring at full rate (P19). Both failures are worse than a missed detection, because a detector that cries wolf gets disabled — and then the real deadlock is never reported.

Q8. Why does §10's verdict rank a credit underflow above a deadlock?

Because a wrapped unsigned counter produces every deadlock symptom with no structural fault at all (§8, counterexample A). Credit counters are unsigned; a decrement below zero wraps to a very large value, the transmitter concludes it has abundant credit, and it floods a receiver with no space — which is both a worse fault and a completely different one. Diagnosing a cycle on top of broken accounting wastes the whole session, so the accounting must be verified first (§17 case 2, P2, P9, P10) and the verdict must say so.

Q9. A link hung once, in the field, after years of service. Why is "marginal timing" the wrong first hypothesis?

Because a structural cycle is fully deterministic and can still go years without closing (§17 case 12, §6). The cycle needs both dependencies and a particular composition of queue occupancies; the occupancies are load-dependent and the dependencies are always present. The productive move is design-time analysis rather than reproduction — enumerate each agent's acquire-before-release obligations and look for cycles, which is readable from the design without reproducing anything. And if a freeze frame existed (§10 Block 9), the adjacency matrix captured at the trigger answers the question outright, which is the argument for building one before it is needed.

20. Module 25 So Far

Eight chapters, and this is the one where the fault has no event.

Every previous chapter had something to read. A state, a residency counter, a boundary code, a byte deficit, a timer. This chapter's fault leaves no register in an abnormal state — every value it produces is a value a healthy link produces too, and §14's 818 resolved episodes are the measurement of exactly that.

So the instrument had to change shape. 25.5 recorded which boundary rejected. 25.6 counted conservation. 25.7 recorded which timer and what the timer could not. This chapter builds a graph — because the fault is not in any agent's state but in the relation between agents, and no per-agent instrument can represent a relation.

Two results generalise past credits.

A property of a composition cannot be found by reviewing the parts. §14 measured two individually safe, individually reviewed dependencies producing a permanent failure together. That is not specific to flow control — it is the general reason integration failures survive module-level verification, and it is the strongest argument in the module for system-level assertions over unit-level ones.

And an aggregate liveness check is not liveness. §15 measured a global detector reporting health while a traffic class was permanently wedged. Any "is it still working?" signal computed as an OR over independent channels has this blind spot — the same shape as 25.7's epoch-derived diagnostic and 25.6's aggregate completion counter. Three chapters, three instruments, one mistake: measuring a quantity that the fault can leave unchanged.

25.9 closes the module, and it inherits this chapter's hardest limitation directly. A deadlock produces a trace that simply stops, with ordinary packets before the stop and no error to trigger on. The analyzer sees the Link — and every instrument this chapter needed was inside a device.