Timeouts, Hung Jobs and Retry Policy

Part II · Module 26 · Chapter 26.5 · Timeouts, Hung Jobs and Retry Policy

1. Every Job Bounded

JOBS = [("axi_smoke", 240), ("axi_reset", 300), ("axi_hang", None), ("axi_soak", 21000)]
DEFAULT_LIMIT = 3600
PER_TEST = {"axi_smoke": 600, "axi_soak": 28800}

print(f"{'test':<11} {'typical':>8} {'limit':>7} {'headroom':>9}")
for test, typical in JOBS:
    limit = PER_TEST.get(test, DEFAULT_LIMIT)
    headroom = f"{limit / typical:.1f}x" if typical else "unknown"
    print(f"{test:<11} {str(typical):>8} {limit:>7} {headroom:>9}")

print()
print("every job has a limit:", True)
print("a job with no typical runtime:", "axi_hang — new, or never finished")

test         typical   limit  headroom
axi_smoke        240     600      2.5x
axi_reset        300    3600     12.0x
axi_hang        None    3600   unknown
axi_soak       21000   28800      1.4x

every job has a limit: True
a job with no typical runtime: axi_hang — new, or never finished

Chapter 23.8 Section 6 built this table for one tool and the argument is unchanged: look at the headroom, not the limit. What a regression adds is that the limits now interact with each other — four hundred jobs at 3600 seconds is a bound on the whole night, and Chapter 26.9's scheduler will have its own opinion about that.

The numbers above are configured limits, which are inputs. Nothing in this chapter publishes a measured duration.

2. The Condition That Never Fires

if res.status == "TIMEOUT" and attempt < retries:
    requeue(job)

STATUSES = ("PASS", "FAIL", "COMPILE_ERROR", "NO_FINISH", "ERROR")

timed_out = {"status": "NO_FINISH", "reason": "TIMEOUT"}

print("Chapter 25.6's statuses :", STATUSES)
print("'TIMEOUT' is one of them:", "TIMEOUT" in STATUSES)
print()
print("a timed-out result is   :", timed_out)
print("the annotation's test   :", timed_out["status"] == "TIMEOUT")
print("the correct test        :",
      timed_out["status"] == "NO_FINISH" and timed_out["reason"] == "TIMEOUT")
print()
print("so with retries=2, the number of retries performed:", 0)
print("and the number of errors raised                   :", 0)

Chapter 25.6's statuses : ('PASS', 'FAIL', 'COMPILE_ERROR', 'NO_FINISH', 'ERROR')
'TIMEOUT' is one of them: False

a timed-out result is   : {'status': 'NO_FINISH', 'reason': 'TIMEOUT'}
the annotation's test   : False
the correct test        : True

so with retries=2, the number of retries performed: 0
and the number of errors raised                   : 0

The condition is never true, so nothing is ever retried, and nothing says so.

Chapter 25.6 Section 6 deliberately kept the status vocabulary at five and used reason to distinguish causes within one. A timeout is status="NO_FINISH", reason="TIMEOUT" — so comparing status against "TIMEOUT" compares a value against a string from a different vocabulary, which Python is perfectly happy to do.

The symptom is the worst available one: a regression configured with retries=2 performs zero retries, reports infrastructure failures as real results, and gives no indication anything is disabled. Somebody notices months later when they count.

This is Chapter 26.1 Section 5's rule arriving with teeth. A summary's columns and a result's statuses are different vocabularies on purpose, and the place they meet is a mapping that must be written down and tested — not a string comparison typed from memory.

def is_retryable(status, reason):
    return status == "NO_FINISH" and reason == "TIMEOUT"


CASES = [("NO_FINISH", "TIMEOUT"), ("NO_FINISH", "NO_END_MARKER"),
         ("TIMEOUT", None), ("FAIL", "UVM_ERROR"), ("ERROR", "no licence")]

for status, reason in CASES:
    print(f"status={status:<14} reason={str(reason):<15} retryable={is_retryable(status, reason)}")

status=NO_FINISH      reason=TIMEOUT         retryable=True
status=NO_FINISH      reason=NO_END_MARKER   retryable=False
status=TIMEOUT        reason=None            retryable=False
status=FAIL           reason=UVM_ERROR       retryable=False
status=ERROR          reason=no licence      retryable=False

Note the third row: a status of "TIMEOUT" — the thing the annotation expected — is correctly not retryable, because no such status exists and a function that silently accepted it would be papering over the original confusion.

3. What May Be Retried

POLICY = [
    ("ERROR", "any", "yes", "the infrastructure failed; the design was never exercised"),
    ("NO_FINISH", "TIMEOUT", "maybe", "could be machine load, could be a real hang"),
    ("NO_FINISH", "NO_END_MARKER", "no", "the testbench has an end-condition bug"),
    ("COMPILE_ERROR", "any", "no", "deterministic — it will fail again identically"),
    ("FAIL", "any", "never", "retrying hides the thing the regression exists to find"),
    ("PASS", "any", "n/a", ""),
]

print(f"{'status':<15} {'reason':<16} {'retry':<7} why")
for status, reason, retry, why in POLICY:
    print(f"{status:<15} {reason:<16} {retry:<7} {why}")

print()
print("retryable at all:", [s for s, _, r, _ in POLICY if r in ("yes", "maybe")])
print("same status, two policies:", "NO_FINISH")

status          reason           retry   why
ERROR           any              yes     the infrastructure failed; the design was never exercised
NO_FINISH       TIMEOUT          maybe   could be machine load, could be a real hang
NO_FINISH       NO_END_MARKER    no      the testbench has an end-condition bug
COMPILE_ERROR   any              no      deterministic — it will fail again identically
FAIL            any              never   retrying hides the thing the regression exists to find
PASS            any              n/a     

retryable at all: ['ERROR', 'NO_FINISH']
same status, two policies: NO_FINISH

Three rows deserve their reasoning spelled out.

ERROR is the clear yes. Chapter 23.9 Section 7 established that those runs tested nothing — a licence that was unavailable for ninety seconds says nothing about the design, and re-running is simply finishing the work.

COMPILE_ERROR is a clear no, for an unusual reason: it is deterministic. The same sources and the same simulator produce the same compile error, so a retry costs a compile and produces an identical result. Chapter 25.3 Section 7's cache would not even rebuild.

NO_FINISH splits on its reason, and that is only possible because Chapter 25.6 kept the reason. A TIMEOUT on a loaded night may genuinely be load; a missing end marker means the testbench stopped without finishing, which will happen again. Same status, opposite policies — the strongest argument for the reason field anywhere in this curriculum.

4. Why FAIL Is Never Retried

P_FAIL = 1 / 3

print("a genuine bug that fires on one run in three")
print(f"{'retries':>8} {'P(reported FAIL)':>18} {'P(reported PASS)':>18}")
for retries in (0, 1, 2, 3):
    p_all_fail = P_FAIL ** (1 + retries)
    print(f"{retries:>8} {p_all_fail:>18.4f} {1 - p_all_fail:>18.4f}")

print()
p2 = P_FAIL ** 3
print("with 2 retries, a real 1-in-3 bug is reported green:", f"{100 * (1 - p2):.2f}% of the time")
print("the bug did not change:", True)
print("the regression's ability to see it:", "almost gone")

a genuine bug that fires on one run in three
 retries   P(reported FAIL)   P(reported PASS)
       0             0.3333             0.6667
       1             0.1111             0.8889
       2             0.0370             0.9630
       3             0.0123             0.9877

with 2 retries, a real 1-in-3 bug is reported green: 96.30% of the time
the bug did not change: True
the regression's ability to see it: almost gone

96.30%. That is the whole argument, and it is arithmetic rather than opinion — a pass-if-any-attempt-passes policy over three attempts reports a genuine one-in-three bug as green in thirty-one cases out of thirty-two.

Note what the table does not say. It is not about flaky infrastructure; P_FAIL here is a real design bug that happens to be seed- or timing-sensitive, which is the most common shape of the bugs a constrained-random regression exists to find. Retrying is precisely the operation that removes them from the report.

Chapter 25.7 Section 6 put this as a warning — a flow that fails two runs in a hundred gets a retry wrapper, and the retry wrapper hides the next real bug. Here it is the same claim with a probability attached, and the number is worse than people expect because the suppression compounds.

5. Two Budgets

JOBS, PER_JOB_RETRIES = 400, 2

worst_unbounded = JOBS * (1 + PER_JOB_RETRIES)
GLOBAL_CAP = 20
worst_capped = JOBS + GLOBAL_CAP

print("jobs                      :", JOBS)
print("retries allowed per job   :", PER_JOB_RETRIES)
print()
print("worst case, per-job only  :", worst_unbounded, "runs")
print("worst case, global cap 20 :", worst_capped, "runs")
print("ratio                     :", f"{worst_unbounded / worst_capped:.1f}x")
print()
print("what the per-job budget protects against:", "one job retrying forever")
print("what the global cap protects against   :", "a bad night tripling the regression")

jobs                      : 400
retries allowed per job   : 2

worst case, per-job only  : 1200 runs
worst case, global cap 20 : 420 runs
ratio                     : 2.9x

what the per-job budget protects against: one job retrying forever
what the global cap protects against   : a bad night tripling the regression

Two budgets, two different protections, and most flows have only the first.

A per-job limit of 2 is perfectly reasonable per job and says nothing at all about the total: if the licence server is down, every job is an ERROR, every one is retried twice, and a four-hundred-job nightly runs twelve hundred times before reporting that the licence server is down.

A global cap turns that into a different and much better outcome: twenty retries are spent, the cap is reached, and the remaining failures are reported as they are. Hitting the global cap is itself the diagnosis — a regression that exhausts its retry budget is telling you the problem is systemic rather than incidental, which is information a per-job budget cannot produce.

6. A Retry Needs Its Own Directory

from pathlib import Path


def job_dir(root, test, seed, attempt):
    name = f"{test}_{seed}" + (f"_a{attempt}" if attempt > 1 else "")
    return Path(root) / "runs" / name


for attempt in (1, 2, 3):
    print(f"attempt {attempt}: {job_dir('root', 'axi_hang', 1839274, attempt)}")

dirs = {job_dir("root", "axi_hang", 1839274, a) for a in (1, 2, 3)}
print()
print("distinct directories:", len(dirs))
print("attempt 1's evidence survives attempt 2:", True)
print("a reviewer can compare the two logs    :", True)

attempt 1: root/runs/axi_hang_1839274
attempt 2: root/runs/axi_hang_1839274_a2
attempt 3: root/runs/axi_hang_1839274_a3

distinct directories: 3
attempt 1's evidence survives attempt 2: True
a reviewer can compare the two logs    : True

Chapter 26.4 Section 3 showed what the alternative costs: a retry into the same directory overwrites the log that explains why the retry happened, leaving a green result and no evidence.

The comparison in the last line is the underrated benefit. Two logs from the same job at the same seed, one failing and one passing, is the single most informative artefact a flaky failure can produce — it isolates what differed when nothing was supposed to. Discarding it to save a directory is a poor trade.

7. Green on the Second Attempt Is Not Green

RESULTS = [("axi_smoke", 1, "PASS"), ("axi_reset", 2, "PASS"),
           ("axi_burst", 1, "FAIL"), ("axi_lic", 3, "PASS"), ("axi_hang", 3, "NO_FINISH")]


def reported(status, attempts):
    if status == "PASS" and attempts > 1:
        return "PASS_ON_RETRY"
    return status


print(f"{'test':<11} {'attempts':>9} {'final':<11} reported")
for test, attempts, status in RESULTS:
    print(f"{test:<11} {attempts:>9} {status:<11} {reported(status, attempts)}")

retried_green = [t for t, a, s in RESULTS if reported(s, a) == "PASS_ON_RETRY"]
print()
print("clean passes       :", sum(1 for t, a, s in RESULTS if reported(s, a) == "PASS"))
print("passed only on retry:", retried_green)
print("hidden if reported as PASS:", len(retried_green), "of", len(RESULTS))

test         attempts final       reported
axi_smoke           1 PASS        PASS
axi_reset           2 PASS        PASS_ON_RETRY
axi_burst           1 FAIL        FAIL
axi_lic             3 PASS        PASS_ON_RETRY
axi_hang            3 NO_FINISH   NO_FINISH

clean passes       : 1
passed only on retry: ['axi_reset', 'axi_lic']
hidden if reported as PASS: 2 of 5

Two of five jobs passed only because they were run again, and a summary reporting plain PASS for them says the night was cleaner than it was.

PASS_ON_RETRY is not a sixth status for Chapter 25.1's result type — the result of attempt 3 genuinely is PASS. It is a summary-level distinction, which is exactly the separation Chapter 26.1 Section 5 drew: the report may describe a run in terms the result type does not carry, as long as the derivation is explicit.

What it buys is a trend. One PASS_ON_RETRY a week is noise; forty a night is Chapter 25.7's non-determinism rate, visible in the one report everybody already reads. Folded into PASS, that number does not exist anywhere.

8. The Policy, Assembled

from dataclasses import dataclass, replace
from typing import Optional


@dataclass(frozen=True)
class Job:
    test: str
    seed: int
    attempt: int = 1


@dataclass(frozen=True)
class Result:
    job: Job
    status: str
    reason: Optional[str] = None


RETRYABLE = {("ERROR", None), ("NO_FINISH", "TIMEOUT")}


def retryable(result):
    if result.status == "ERROR":
        return True
    return (result.status, result.reason) in RETRYABLE


def run_with_retries(jobs, outcomes, *, per_job=2, global_cap=20):
    """outcomes maps (test, seed, attempt) -> (status, reason). Pure, for the example."""
    final, spent, log = {}, 0, []
    queue = list(jobs)
    while queue:
        job = queue.pop(0)
        status, reason = outcomes[(job.test, job.seed, job.attempt)]
        res = Result(job, status, reason)
        if retryable(res) and job.attempt <= per_job and spent < global_cap:
            spent += 1
            log.append(f"retry {job.test}/{job.seed} after {status}/{reason}")
            queue.append(replace(job, attempt=job.attempt + 1))
            continue
        final[(job.test, job.seed)] = res
    return final, spent, log


JOBS = [Job("axi_smoke", 1), Job("axi_lic", 2), Job("axi_burst", 3), Job("axi_hang", 4)]
OUTCOMES = {
    ("axi_smoke", 1, 1): ("PASS", None),
    ("axi_lic", 2, 1): ("ERROR", "no licence"),
    ("axi_lic", 2, 2): ("PASS", None),
    ("axi_burst", 3, 1): ("FAIL", "UVM_ERROR"),
    ("axi_hang", 4, 1): ("NO_FINISH", "TIMEOUT"),
    ("axi_hang", 4, 2): ("NO_FINISH", "TIMEOUT"),
    ("axi_hang", 4, 3): ("NO_FINISH", "TIMEOUT"),
}

final, spent, log = run_with_retries(JOBS, OUTCOMES)

for key in sorted(final):
    res = final[key]
    label = "PASS_ON_RETRY" if res.status == "PASS" and res.job.attempt > 1 else res.status
    print(f"{key[0]:<11} attempt {res.job.attempt}  {label:<14} reason={res.reason}")

print()
print("retries spent :", spent)
print("retry log     :")
for line in log:
    print("   " + line)
print()
print("every job has exactly one final result:", len(final) == len(JOBS))

axi_burst   attempt 1  FAIL           reason=UVM_ERROR
axi_hang    attempt 3  NO_FINISH      reason=TIMEOUT
axi_lic     attempt 2  PASS_ON_RETRY  reason=None
axi_smoke   attempt 1  PASS           reason=None

retries spent : 3
retry log     :
   retry axi_lic/2 after ERROR/no licence
   retry axi_hang/4 after NO_FINISH/TIMEOUT
   retry axi_hang/4 after NO_FINISH/TIMEOUT

every job has exactly one final result: True

Five decisions, and the last is what keeps Chapter 26.1's invariant intact.

FAIL is never requeued, so axi_burst is reported after one attempt. Section 4's arithmetic, implemented.

axi_hang exhausted its per-job budget — three attempts, two retries — and is reported as NO_FINISH, not as a mysterious absence. A job that runs out of retries still produces a result.

The retry log names what it retried and why. A regression that silently performed three extra runs has spent compute nobody can account for; spent and log together make that auditable, which is Chapter 24.4 Section 6's report-what-you-did rule.

replace(job, attempt=...) makes a new frozen Job rather than mutating one. The original is still in the list, the new one has a different identity, and Chapter 26.4's directory naming therefore gives it its own directory for free.

Exactly one final result per (test, seed), which is the property Chapter 26.6 reconciles against the job list. A retry must not add a row to the summary — three attempts at axi_hang are one job, and counting them as three would break the invariant in the direction nobody checks.

9. Predict, Run, Observe

from dataclasses import dataclass, replace

STATUSES = ("PASS", "FAIL", "COMPILE_ERROR", "NO_FINISH", "ERROR")


@dataclass(frozen=True)
class J:
    test: str
    seed: int
    attempt: int = 1


def retryable(status, reason):
    return status == "ERROR" or (status == "NO_FINISH" and reason == "TIMEOUT")


# A
print("A:", "TIMEOUT" in STATUSES)

# B
print("B:", {"status": "NO_FINISH"}["status"] == "TIMEOUT")

# C
print("C:", retryable("NO_FINISH", "TIMEOUT"), retryable("NO_FINISH", "NO_END_MARKER"))

# D
print("D:", retryable("FAIL", "UVM_ERROR"), retryable("ERROR", "no licence"))

# E
print("E:", f"{(1/3)**3:.4f}", f"{1 - (1/3)**3:.4f}")

# F
print("F:", 400 * 3, 400 + 20)

# G
j = J("a", 1)
k = replace(j, attempt=2)
print("G:", j.attempt, k.attempt, j == k)

# H
print("H:", len({j, k}))

The verified output:

A: False
B: False
C: True False
D: False True
E: 0.0370 0.9630
F: 1200 420
G: 1 2 False
H: 2

A — "TIMEOUT" is not a status, which is why the annotation's test cannot succeed.

B — and comparing the status field against it is simply False.

C — the same status, two policies, separated by the reason.

D — a FAIL is never retryable and an ERROR always is.

E — two retries report a one-in-three bug green 96.30% of the time.

F — the per-job worst case against the globally capped one.

G — replace produces a new job with a different attempt, and it is not equal to the original.

H — so the two are distinct in a set, which is what gives each its own directory.

10. Exercises

Exercise 1 — headroom. Build the limits table for your own suite and identify the tightest.

Exercise 2 — the night. Compute the worst-case wall time of your regression from its limits alone.

Exercise 3 — the dead condition. Reproduce res.status == "TIMEOUT" being false and say how long it would take to notice.

Exercise 4 — the two vocabularies. Explain this as an instance of Chapter 26.1 Section 5.

Exercise 5 — write the predicate. Implement retryable(status, reason) and test all five statuses.

Exercise 6 — reject the ghost. Say why a status of "TIMEOUT" should be non-retryable rather than quietly accepted.

Exercise 7 — three justifications. Defend retrying ERROR, not retrying COMPILE_ERROR, and splitting NO_FINISH.

Exercise 8 — determinism. Say why a deterministic failure is the clearest case against retrying.

Exercise 9 — the reason field earns its keep. Give the strongest example in this curriculum of why reason exists.

Exercise 10 — the arithmetic. Compute the reported-green probability for your own suspected flaky rate at one, two and three retries.

Exercise 11 — resampling. Restate a FAIL retry as a sampling operation and say why nobody would agree to it stated that way.

Exercise 12 — two budgets. Compute your worst case with a per-job budget only, then with a global cap.

Exercise 13 — the cap as a diagnosis. Say what hitting the global cap tells you that a per-job budget cannot.

Exercise 14 — a directory per attempt. Implement the naming and confirm three attempts give three directories.

Exercise 15 — two logs. Say why a failing and a passing log from the same job and seed is so informative.

Exercise 16 — PASS_ON_RETRY. Add it as a reported category and count how many of your last regression's passes it would catch.

Exercise 17 — not a sixth status. Explain why it belongs in the summary rather than in the result type.

Exercise 18 — the trend. Say what forty PASS_ON_RETRY rows in one night means, citing Chapter 25.7.

Exercise 19 — audit the spend. Make your retry loop report what it retried and why.

Exercise 20 — one result per job. Show that three attempts produce one row, and say what counting three would do to Chapter 26.1's invariant.

11. Summary

• Every job gets a limit, and the number to look at is the headroom — Chapter 23.8 Section 6, now interacting across four hundred jobs.
• res.status == "TIMEOUT" is never true. Chapter 25.6 made TIMEOUT a reason under NO_FINISH, so the usual retry condition silently performs zero retries.
• Nothing raises when it fails, which is why a flow can be configured with retries=2 and do none for months.
• Write the retryable predicate as a tested function over (status, reason) rather than a remembered string comparison.
• ERROR is always retryable — the design was never exercised.
• COMPILE_ERROR is not, because it is deterministic; the retry produces an identical result.
• NO_FINISH splits on its reason: a timeout may be load, a missing end marker is a testbench bug. The strongest justification for the reason field anywhere in this curriculum.
• FAIL is never retried. Two retries report a genuine one-in-three bug as green 96.30% of the time.
• A FAIL retry is a resampling that reports the best draw, which nobody would accept stated plainly.
• A per-job budget and a global cap protect against different things; without the cap, a licence outage turns 400 jobs into 1200 runs.
• Hitting the global cap is itself the diagnosis — the problem is systemic rather than incidental.
• Give every attempt its own directory, or the retry destroys the evidence that justified it.
• A failing and a passing log from one job and seed is the most informative artefact a flaky failure produces.
• Report PASS_ON_RETRY separately — it is a summary distinction, not a sixth status, and folded into PASS the non-determinism rate exists nowhere.
• Audit the retry spend, naming what was retried and why.
• Three attempts are one job and one row, or Chapter 26.1's invariant breaks in the direction nobody checks.

The next chapter collects all of those final results — and the jobs that never produced one.