AMBA AXI · Module 17

Timeout Debugging

Use timeouts systematically to localize where an AXI transaction stalled — per-channel and per-transaction watchdogs that turn a silent hang into a labeled error naming the channel, transaction, and wait condition, how to set thresholds, and how the first timer to fire points nearest the root of any stall, hang, or deadlock.

The hang chapters (17.1, 17.7) all relied on one tool: timeouts. A stalled or deadlocked AXI transaction produces a silent failure — nothing happens, no error fires, the simulation just stops progressing — which is the least debuggable symptom possible. A timeout converts that silence into information: a watchdog that fires when a channel or transaction has been waiting too long, emitting an error that names what stalled, where, and what it was waiting on. Done systematically — per channel, per transaction, per resource — timeouts don't just detect a hang, they localize it: the first timer to fire points nearest the root. This chapter is the instrumentation behind all hang debugging: how timeouts work, where to place them, how to set thresholds, and how to read which-fired-first to find the root of any stall, hang, or deadlock.

1. Why Timeouts: Turning Silence into Information

A liveness failure (something that should happen never does) is invisible by nature — there's no event to catch, the system just freezes. Without instrumentation, you discover it only because the test never finishes, and you're left manually staring at a frozen waveform inferring every party's wait condition. A timeout makes the failure announce itself: a counter increments while a transaction/channel is blocked and, past a threshold, fires an error with context. This is the difference between "the sim hung somewhere" and "the AW channel of master 2 has held VALID without READY for 5000 cycles" — the latter is a localized, actionable finding.

Figure 1 — a timeout turns a silent liveness failure into a localized, actionable error. Without a timeout, a stalled transaction is invisible: the simulation simply stops progressing with no event, and debugging means manually inferring wait conditions from a frozen waveform. A watchdog counter increments while blocked and, past a threshold, fires an error naming the channel/transaction and what it was waiting on — converting 'the sim hung' into 'this specific thing stalled waiting on that'.

2. Where to Place Timeouts: Per Channel, Per Transaction, Per Resource

Granularity determines how much a timeout localizes. Per-channel watchdogs (each VALID-without-READY, or each request awaiting acceptance) catch stuck handshakes at the channel level. Per-transaction watchdogs (each outstanding transaction from issue to completion) catch a transaction that never completes even if its channels individually progressed — and name the transaction (ID, address). Per-resource watchdogs (each buffer slot, outstanding slot, arbiter grant) catch a resource held too long — supplying the wait-for edges (17.7) needed to reconstruct a deadlock. The finer the instrumentation, the more precisely the firing localizes the problem.

Figure 2 — timeout placement granularity. Per-channel watchdogs catch a stuck handshake (VALID without READY too long) and name the channel. Per-transaction watchdogs catch a transaction that never completes (issue to response) and name the transaction (ID/address). Per-resource watchdogs catch a held buffer slot / outstanding slot / grant and supply the wait-for edges for deadlock analysis. Finer granularity localizes more precisely; a complete instrumentation has all three.

The per-transaction watchdog is the workhorse — it tracks each outstanding transaction and fires if completion takes too long:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

// Per-transaction watchdog: start a timer when a transaction is issued,
// clear it on completion, fire if it exceeds the threshold.
foreach (outstanding[i]) begin
  if (outstanding[i].active) begin
    outstanding[i].age <= outstanding[i].age + 1;
    if (outstanding[i].age > TXN_TIMEOUT)
      $error("TIMEOUT: txn id=%0d addr=%0h issued @%0t never completed, "
             "stalled in phase %s (waiting on %s)",
             outstanding[i].id, outstanding[i].addr, outstanding[i].t_issue,
             outstanding[i].phase, outstanding[i].waiting_on);
  end
end
// The 'phase' and 'waiting_on' fields are what localize the stall.

3. Setting the Threshold

A timeout threshold must be long enough to not fire on legitimate backpressure/latency (a false positive that masks real behavior) but short enough to catch a hang promptly. The right value is derived from the worst-case legitimate latency: maximum outstanding depth × per-transaction latency, plus margin — anything beyond the longest a transaction could legitimately take is a hang. Too short → false timeouts on slow-but-correct traffic (and you start ignoring timeouts, the worst outcome); too long → hangs take forever to surface. Set it from the system's real latency bounds, not a guess.

Figure 3 — setting the timeout threshold. The threshold must sit above the worst-case legitimate latency (max outstanding × per-transaction latency + margin) so it never fires on correct-but-slow traffic, and below 'forever' so hangs surface promptly. Too short causes false timeouts (which lead to ignoring timeouts — the worst outcome); too long delays detection. Derive it from the system's real latency bounds, not a guess. The window between worst-case-legitimate and a true hang is where the threshold belongs.

4. Which Fired First: Localizing the Root

The diagnostic power of systematic timeouts is temporal ordering: in any hang, many timers eventually fire (every blocked party times out), but the first to fire is nearest the root, and the later ones are downstream victims waiting on it. So you don't just collect that timeouts fired — you record when each fired and read the earliest. For a single stuck point, the first timeout is the originally-blocked channel; for a deadlock, the set of timers that fire together (with no single first, all interdependent) reveals the cycle. The waveform shows a transaction's age counter climbing past the threshold and the timeout firing at the exact stall point.

Per-transaction timeout firing at the stall point

10 cycles

Figure 4 — a per-transaction timeout firing, and which-fired-first localization. A transaction's age counter increments each cycle it remains incomplete; when it crosses the threshold, the timeout fires, marking the exact point and naming the stalled transaction and its wait condition. In a multi-party hang, the FIRST timer to fire is nearest the root (downstream victims fire later, waiting on it); a set of timers firing together with no clear first indicates a deadlock cycle. Record when each fires and read the earliest.

5. Common Misconceptions

Myths to unlearn

"A hang doesn't need instrumentation — you'll see the sim stop." You'll see it stop, but with no information about what or where. A timeout is what converts "the sim hung somewhere" into "this transaction/channel stalled waiting on that" — the difference between hours of waveform staring and an immediate localization.
"One global timeout is enough." A single "the test took too long" timeout tells you nothing about where. Per-channel/per-transaction/per-resource timeouts localize the stall and supply the wait-for edges; granularity is the whole point.
"Set the timeout short to catch hangs fast." Too short fires on legitimate backpressure/latency (false positives), and the real danger is that people start ignoring timeouts — so a real hang's timeout gets dismissed as noise. Set it above the worst-case legitimate latency.
"If timeouts fire, the timeout value is wrong." Maybe — but first check whether it's a real hang. A fired timeout is a finding to investigate, not automatically a threshold problem. Only after confirming the traffic was legitimately within latency bounds should you raise the threshold.
"The channel whose timeout fired is the bug." Not necessarily — in a propagating hang, downstream victims also time out. The first timer to fire is nearest the root; later ones are waiting on it. Read the timing of the timeouts, not just which ones fired.
"Timeouts are only for simulation." Production systems also need timeouts — a real AXI master/interconnect that can hang forever on a missing response is fragile; hardware watchdogs/timeouts that abort or recover are part of robust design, not just a debug aid.

6. Debugging Insight

🔍 Debugging insight — read the first timeout, then its named wait condition

Use systematic timeouts to localize any hang:

Confirm it's a hang, then read the timeouts. If the sim stops progressing and timeouts fired, you have a hang — go straight to the timeout logs rather than the raw waveform. The logs name the stalled parties and their wait conditions.
Find the first timer to fire. Sort the timeouts by fire time. The earliest is nearest the root; the later ones are downstream victims waiting on it. This single step often points directly at the bug.
Read the named wait condition. A good per-transaction/per-channel timeout reports phase and waiting-on (e.g. "txn id=3 stalled in write-data phase waiting on WREADY"). That tells you exactly where to look — the channel and the condition.
Distinguish stall/hang/deadlock by the timeout pattern. One timeout (others clear) → a transient stall that did exceed threshold (maybe a too-short threshold, or a slow-but-correct path) — verify against latency bounds. One timeout with a clear root → single stuck point (17.1). A set firing together with no clear first, mutually waiting → deadlock cycle (17.7).
If many false timeouts, re-derive the threshold. Confirm the traffic was within worst-case legitimate latency; if so, raise the threshold (max outstanding × per-txn latency + margin). But never just disable timeouts — that re-blinds you.
Use per-resource timeouts to build the deadlock graph. When the pattern indicates a deadlock, the per-resource timeout logs ("resource X waiting on Y") are the wait-for edges — feed them into the cycle analysis (17.7).

The method: timeouts make the hang speak; the first to fire and its named wait condition localize the root. Systematic, granular timeouts turn the hardest class of bug (silent liveness failures) into the most localizable.

7. Verification Insight

✅ Verification insight — instrument comprehensively, label richly, threshold from latency bounds, and read which-fired-first

Timeouts are the detection and localization backbone for every liveness failure in AXI verification — stalls, hangs, and deadlocks all surface through them — so the discipline is comprehensive, well-labeled instrumentation. (1) Instrument at every granularity — per-channel (stuck handshake), per-transaction (never-completes), and per-resource (held buffer/outstanding/grant); the per-transaction watchdog is the workhorse, the per-resource ones supply deadlock wait-for edges (17.7), and together they localize any hang to a named party. (2) Label richly — a timeout that just says "timed out" is barely better than the silent hang; it must report the channel/transaction (ID, address), the phase it stalled in, and what it was waiting on, because that context is the localization. (3) Threshold from worst-case legitimate latency — set it above max-outstanding × per-transaction-latency + margin so it never fires on correct-but-slow traffic (false positives are corrosive — they train people to ignore timeouts, defeating the purpose), and below "forever" so hangs surface; derive it from the system's real bounds, not a guess. (4) Read which-fired-first — the temporal ordering of timeouts is the localization signal: the earliest is nearest the root, later ones are downstream victims, and a set firing together with mutual waits indicates a deadlock cycle; capture fire times, not just the fact of firing. (5) Make timeouts findings, not noise — a fired timeout is something to investigate (real hang vs. threshold-too-low), never to silence by disabling, since disabling re-blinds the environment to the worst bug class. This chapter formalizes the instrumentation the whole hang-debugging family depends on: 17.1 (stuck handshake) and 17.7 (deadlock) both assumed watchdogs — here we make them systematic. The unifying principle across Module 17's hang chapters is that liveness failures are silent, and the engineering response is to make the system announce, in localized and labeled form, what stalled and what it waited on — which converts the hardest debugging problem (a frozen black box) into a tractable one (read the first labeled timeout). Robust designs carry this into silicon too: hardware timeouts that detect and recover from a hung transaction are part of production robustness, not merely a simulation aid. With timeouts as the detection backbone and the specific-bug methods of 17.1–17.7, the final chapter assembles everything into one repeatable waveform-debugging methodology.

8. Interview Questions

9. Summary

Timeouts are the detection and localization backbone for every liveness failure in AXI — stalls, hangs, and deadlocks are all silent (no event fires; the system just freezes), and a timeout converts that silence into a localized, labeled error: a watchdog increments while a transaction/channel/resource is blocked and, past a threshold, fires naming what stalled, where, and what it waited on. Place them at every granularity — per-channel (stuck handshake), per-transaction (never-completes, the workhorse), and per-resource (held slot/grant, supplying the deadlock wait-for edges) — because granularity is what localizes. Set the threshold from worst-case legitimate latency (max outstanding × per-transaction latency + margin): too short causes false positives that train people to ignore timeouts (the worst outcome), too long delays detection.

The diagnostic power is which-fired-first: in a propagating hang the earliest timer is nearest the root and later ones are downstream victims, while a set firing together with mutual waits and no clear first signals a deadlock cycle — so record fire times, not just the fact of firing, and read the earliest plus its named wait condition. The timeout pattern even classifies the hang (clears → stall; single root → stuck point; mutually-waiting set → deadlock) before you open the waveform. This formalizes the instrumentation that 17.1 and 17.7 assumed. The unifying principle is observability for liveness: safety failures self-announce (catch the event), but liveness failures are silent and must be made observable by bounding time and watching the bound — timeouts are that instrument, which is why they underlie the whole hang-debugging family and belong in silicon too. Next, the final chapter assembles every method in Module 17 into one repeatable waveform-debugging methodology.

10. What Comes Next

You now have the detection backbone; the final chapter assembles the whole method:

17.9 — Waveform-Based Debug Methodology (coming next) — a single repeatable method for debugging any AXI waveform, integrating the handshake, burst, response, ID, deadlock, and timeout techniques of Module 17.

Previous: 17.7 — Debugging Deadlock. Related: 17.1 — Stuck VALID / Stuck READY and 17.7 — Debugging Deadlock for the hangs timeouts detect, and 13.2 — Latency Analysis for the latency bounds that set the threshold.