Linux & Bash · Part II · Module 34 · Defensive Automation Around EDA Tools

Timeouts and Hung Jobs

A tool that neither succeeds nor fails consumes a whole night and leaves nothing to report. This chapter bounds every invocation on a machine where the obvious tool for the job is not installed, and measures why the bound must record its own verdict: the simulator exits zero when it is politely killed, and ignores the signal most scripts send first.

Advanced25 min readLinuxPart IIEDATimeoutsSignals

Part II · Module 34 · Chapter 34.4 · Timeouts and Hung Jobs

Chapter 34.3 reacted to a failure. A hang produces none: the tool is running, the log is growing or not, and nothing will ever decide. An unbounded invocation is a flow with no upper bound on its own runtime, and the cost is not the hung job — it is the two hundred queued behind it.

1. The Obvious Tool Is Not Installed

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ command -v timeout  || echo "NOT INSTALLED"
NOT INSTALLED
$ command -v gtimeout || echo "NOT INSTALLED"
NOT INSTALLED
$ ls /opt/homebrew/opt/coreutils/libexec/gnubin/timeout 2>/dev/null || echo no
no

timeout(1) is GNU coreutils and this is BSD userland. The registry's own command hint for this chapter is

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

timeout 2h ./simv +UVM_TESTNAME="$t"

which is the right command on a Linux compute farm and does not run here. That is worth stating rather than quietly substituting, because the same gap appears on any machine where a flow is developed on a Mac and deployed on Linux, and the failure mode is timeout: command not found at two in the morning.

So the rest of this chapter builds the bound by hand — and the hand-rolled version turns out to need things timeout would not have given you anyway.

2. What the Tool Does When You Ask It to Stop

A bound has to end the process, and which signal you send determines what you learn. vvp as the direct child of the shell, with a sleep through the identical harness as a control:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  kill -TERM  ended, wait rc=0    log 105 bytes
  kill -INT   IGNORED (still running after 1.5s)
  kill -HUP   ended, wait rc=0    log 105 bytes
  kill -KILL  ended, wait rc=137  log 0 bytes
  control: sleep + TERM -> 143

Four findings, and the control row is what makes them trustworthy — sleep gives the textbook 128 + 15 = 143, so the harness is not what distorts the simulator's numbers.

TERM yields exit status 0. Not 143. The tool installs a handler, shuts down and exits successfully. A timeout wrapper that kills a hung run and then reads the exit status concludes that the run passed.

INT is ignored entirely. Ctrl-C does nothing to it, so a script escalating INT → TERM → KILL wastes its first step, and an engineer interrupting a regression at the terminal leaves simulators running. Chapter 30.4 measured the same tool declining INT, and Chapter 30.1 found it resetting SIGHUP's ignore request so nohup did not protect it.

HUP works and also gives 0, which is the same trap as TERM.

And KILL gives 137 with a zero-byte log. The status is finally informative and the evidence is gone — the buffered output never reached the file. Chapter 30.4 measured that cost at exactly 88 log lines.

The bound must record its own verdict, because the status cannot carry it

Put those two rows together:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  kill -TERM  wait rc=0    (log present)
  kill -KILL  wait rc=137  (log empty)

The polite kill loses the status and keeps the evidence; the hard kill keeps the status and loses the evidence. There is no signal that gives you both.

So the timeout wrapper cannot infer anything useful from what it gets back. It has to write down what it did:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf '%s\n' "$timedout" > "$d/timedout"

One file, one bit, written by the only process that knows. Chapter 32.3's verdict function checks it first, before reading the log at all, and the reason is precisely this measurement: a TERMed run exits 0 and its log is whatever happened to be flushed — which on a multi-phase testbench can include a TEST PASSED from an earlier phase. A timeout checked after the log can be overruled by the log.

This is the generalisation worth carrying: when an action's outcome cannot be read back from the system, the actor records it. The same argument produced Chapter 33.1's build manifest and Chapter 33.8's machine snapshot — three places where the only reliable source is the process that was there.

As a tool, because every tool needs asking

Those four rows are a property of one tool, and the next tool in the flow will answer differently. So the probe is worth keeping:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# signalprobe COMMAND... -- how does this tool respond to each stop signal?
# Run once per tool. The answers decide how a timeout must be written, and
# they are not guessable: this simulator exits 0 on TERM and ignores INT.
set -u
[ $# -gt 0 ] || { echo "signalprobe: give me a command that runs for a while" >&2; exit 2; }
w=$(mktemp -d) || exit 2
trap 'rm -rf "$w"' EXIT INT TERM
printf '  %-8s %-22s %-10s %s\n' SIGNAL BEHAVIOUR 'wait rc' 'log bytes'
one() {
  local sig=$1
  shift                      # BEFORE launching: "$@" must not include the signal
  ( exec "$@" ) > "$w/$sig.log" 2>&1 & local p=$!
  sleep 0.8
  kill -"$sig" "$p" 2>/dev/null
  local n=0
  while kill -0 "$p" 2>/dev/null && [ "$n" -lt 15 ]; do n=$((n+1)); sleep 0.1; done
  if kill -0 "$p" 2>/dev/null; then
    kill -9 "$p" 2>/dev/null; wait "$p" 2>/dev/null
    printf '  %-8s %-22s %-10s %s\n' "$sig" "IGNORED" "-" "$(wc -c < "$w/$sig.log" | tr -d ' ')"
  else
    wait "$p" 2>/dev/null; local rc=$?
    printf '  %-8s %-22s %-10s %s\n' "$sig" "ended" "$rc" "$(wc -c < "$w/$sig.log" | tr -d ' ')"
  fi
}
for s in TERM INT HUP QUIT KILL; do one "$s" "$@"; done
# the control: a tool known to die properly, through the identical harness
one TERM sleep 30

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ signalprobe vvp "$PWD/logs/sim.vvp" +TESTNAME=hang +SEED=1
  SIGNAL   BEHAVIOUR              wait rc    log bytes
  TERM     ended                  0          105
  INT      IGNORED                -          151
  HUP      ended                  0          105
  QUIT     IGNORED                -          0
  KILL     ended                  137        0
  TERM     ended                  143        0

The last row is the control — sleep 30 through the identical harness, giving the textbook 143 — and it is part of the tool rather than a separate step, because a probe whose harness is broken produces a confident table of wrong numbers.

Which it did, the first time:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  SIGNAL   BEHAVIOUR              wait rc    log bytes
  TERM     ended                  127        48
  INT      ended                  127        47
  HUP      ended                  127        48

Every row 127 — command not found. The function took the signal name as $1 and shifted after launching the subshell, so "$@" still contained it and the shell tried to execute TERM vvp …. One line in the wrong order:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

one() {
  local sig=$1
  shift                      # BEFORE launching: "$@" must not include the signal
  ( exec "$@" ) > "$w/$sig.log" 2>&1 & local p=$!
  sleep 0.8
  kill -"$sig" "$p" 2>/dev/null
  # … then poll, report, and fall back to KILL if it was ignored
}

And the tell was the control row, which should have read 143 and read 127 like everything else. A control that agrees with the thing it is controlling for is a control that has failed — the one check that would have caught this at a glance, which is the argument for building it into the probe rather than remembering to run it.

3. The Bound

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# bounded LIMIT_S OUTDIR -- run a tool under a time bound, and record whether
# the bound fired. The status cannot carry that: TERM makes this tool exit 0.
set -u
limit=${1:?need a limit in seconds}; d=${2:?need an output directory}; shift 2
root=$(cd "$(dirname "$0")/.." && pwd)
mkdir -p "$d" || exit 2

( cd "$d" && exec "$@" ) > "$d/run.log" 2>&1 & p=$!
timedout=0
n=0; lim=$(( limit * 10 ))
while kill -0 "$p" 2>/dev/null && [ "$n" -lt "$lim" ]; do n=$((n+1)); sleep 0.1; done

if kill -0 "$p" 2>/dev/null; then
  timedout=1
  # TERM first: it is the one that lets the tool flush its log.
  kill -TERM "$p" 2>/dev/null
  m=0; while kill -0 "$p" 2>/dev/null && [ "$m" -lt 20 ]; do m=$((m+1)); sleep 0.1; done
  # and KILL only if it refused, which costs the buffered output.
  if kill -0 "$p" 2>/dev/null; then
    printf 'bounded: TERM ignored, escalating to KILL (the log will be lost)\n' >&2
    kill -9 "$p" 2>/dev/null
  fi
fi
wait "$p" 2>/dev/null
rc=$?
printf '%s\n' "$rc"       > "$d/tool.rc"
printf '%s\n' "$timedout" > "$d/timedout"
printf '%s\n' "$limit"    > "$d/limit_s"
[ "$timedout" -eq 1 ] && printf 'bounded: the %ds limit fired\n' "$limit" >&2
exit 0

Five decisions.

exec in the subshell, so $! is the tool and not a shell. Chapter 31.5 measured what happens otherwise — an RSS sampler that read a 1 MB wrapper instead of a 9 MB simulator — and here it matters because kill has to reach the tool.

A polling loop rather than a blocking wait, because the shell must stay alive to enforce the bound. And sleep 0.1 rather than sleep 1, so a one-second limit is meaningful.

TERM before KILL, with a grace period. The measurement above is the justification: TERM keeps the log and KILL empties it, so the hard kill is a last resort and the wrapper says so when it reaches for it.

timedout recorded, and limit_s with it. A run that hit a four-second bound and a run that hit a two-hour bound are different events, and the limit is part of the record for the same reason Chapter 32.2 records the slot count.

And it exits 0 regardless. The wrapper's job is to bound and record, not to judge — Chapter 32.2's separation of dispatch from interpretation. The verdict comes from Chapter 32.3 reading timedout and the log together.

4. The Bound Is Approximate, and by How Much

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  LIMIT_S    actual     overshoot
  1s         1s         0s
  2s         2s         0s
  4s         5s         1s

A four-second limit produced a five-second run. Two reasons, both unavoidable in a shell.

sleep 0.1 is at least 0.1 seconds, and the loop spawns a process per iteration, so forty iterations take more than four seconds.

And the grace period adds up to two more. TERM, then up to 2 s of polling, then KILL — paid only when the bound fires, which is exactly the case being measured.

So the useful statement is: a shell time bound is accurate to within a few seconds and a small percentage, and both matter at different scales.

limit	what the imprecision costs
1–5 s	the grace period dominates; the overshoot can be 25%
minutes	negligible
hours	negligible in relative terms; the grace period is still ~2 s

Which gives the sizing rule: set the limit an order of magnitude above the expected runtime, never just above it. A limit chosen to be 10% above a two-hour run will fire on the first busy night and report a timeout for a run that was merely slow — and Chapter 33.8 measured why that is expensive, because a TIMEOUT that disappears on the next run is recorded as a fix.

What a timeout is and is not for

It is for bounding the damage of a hang. A hung job holds a slot, a licence and a node until somebody notices, and "somebody notices" on a nightly regression means eight hours. A bound converts that into one limit's worth of waste.

It is not a performance check. A test that is getting slower will eventually trip the bound, and the report will say TIMEOUT rather than "this test is 40% slower than last week". The runtime belongs in the results file as a number — Chapter 32.4's seconds column — where a trend can see it. Chapter 32.8's trend is where that question is answered.

It is not a substitute for a watchdog inside the testbench. A simulation that stops making progress is better caught by the testbench itself, which knows what progress looks like:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

// the testbench's own bound, in simulation time rather than wall time
initial begin
  #TIMEOUT_CYCLES;
  $display("TB  ERROR no progress for %0d cycles -- giving up", TIMEOUT_CYCLES);
  $display("TB  TEST FAILED  test=%0s checks=%0d errors=%0d", testname, checks, errors+1);
  $finish;
end

That version produces a log with a verdict in it, which every tool downstream can read, instead of a killed process and a timedout flag. The shell's bound is the backstop for the case where the testbench's own watchdog is also stuck — and both are worth having, because the simulation-time bound cannot catch a tool spinning outside simulation time.

And it is not free of false positives. Chapter 31.5 measured eight parallel heavy runs taking 9.8 s against 43.8 s for one at a time — a 4.5× spread from contention alone. A bound set from a single uncontended measurement will fire under load, which is an argument for measuring the limit at the parallelism the regression actually uses.

5. Composing It With the Rest of the Module

A bound on its own answers "did it finish". The other chapters answer "did it work", and the order they compose in is fixed:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# run_one -- one simulation, with every guard in this module around it.
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
t=${1:?need a test}; s=${2:-1}; d=${3:?need an output directory}
mkdir -p "$d" || exit 2

# 34.5 -- bound what one run can write, so it cannot take the others with it
( ulimit -f "${MAX_RUN_BLOCKS:-5242880}"
  # 34.4 -- bound how long it may take, and record that the bound fired
  "$root/bin/bounded" "${LIMIT_S:-7200}" "$d" \
      vvp "$root/logs/sim.vvp" "+TESTNAME=$t" "+SEED=$s" "+CYCLES=${CYCLES:-200}" )

# 34.2 -- the log must exist, end properly, and state exactly one verdict
"$root/bin/logcheck" "$d/run.log" >/dev/null 2>&1
printf '%s\n' "$?" > "$d/logcheck.rc"

# 34.5 -- was it a space failure rather than a design one?
"$root/bin/spaceclass" "$d" > "$d/space.txt" 2>&1 || true

# 33.8 -- the machine facts that are gone in seconds, recorded non-fatally
"$root/bin/snapshot" "$d/machine" 2>/dev/null || echo "snapshot failed" >&2
exit 0

Four properties of that composition.

The caps are outermost and the checks are innermost. ulimit must be set before the process starts and cannot be applied afterwards; logcheck can only run once there is a log. That ordering is forced, not chosen.

Every guard writes a file and none of them decides. tool.rc, timedout, limit_s, logcheck.rc, space.txt, machine/ — six artifacts, each from the component that knows, and Chapter 32.4's collector turns them into columns. This is Chapter 32.2's separation of dispatch from interpretation, and it is why the wrapper ends in exit 0.

The || true appears exactly twice, on spaceclass and snapshot — both notes beside the result rather than inputs to it, which is Chapter 34.7's test. Everywhere else a failure must reach the caller.

And each limit comes from the environment with a default. ${LIMIT_S:-7200}, ${MAX_RUN_BLOCKS:-5242880} — so one long test can be given a longer bound without editing the wrapper, which is the difference between a wrapper that is reused and one that is copied.

6. Where the Bound Goes

On every tool invocation, not on the regression. A bound around the whole regression stops it mid-flight and loses every run still in progress — which Chapter 31.7 measured as NOSTATUS, four directories with zero-byte logs after the launcher was killed. A bound per invocation loses one run.

Including the compile. A compile can hang on a network filesystem or a licence wait, and an unbounded compile holds up everything behind it. Chapter 34.1's wrapper should be called through the bound, not instead of it.

And the limit comes from configuration, not from the script:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

LIMIT_S=${LIMIT_S:-7200} bounded "$LIMIT_S" "$d" vvp "$image" "+TESTNAME=$t"

A limit hardcoded in a wrapper is a limit nobody can raise for one long test, which is how the wrapper gets copied and modified instead of reused — the failure mode Module 35 is about.

Practice

Check whether timeout is installed on the machine your flow runs on, and on the machine you develop on.
Run the registry's own hint and see what happens here.
Send each of TERM, INT, HUP and KILL to a hung tool and record the status and the log size.
Run the same harness against sleep as a control.
Find which signal keeps the log and which keeps the status.
Explain why no signal gives both.
Record a timedout flag and say which process is the only one that can.
Check it before the log in a verdict function, and construct the case that justifies the order.
Write the bound with exec, a polling loop, and a TERM-then-KILL escalation.
Explain why exec matters for the kill.
Explain why a blocking wait cannot be used.
Measure requested against actual for three limits.
Account for the overshoot in two parts.
Decide a sizing rule and say what a 10%-above limit does on a busy night.
Record limit_s alongside the flag and justify it.
Make the wrapper exit 0 regardless and say who judges.
Add a watchdog inside the testbench and compare what it produces with what a kill produces.
Say what the simulation-time bound cannot catch.
Measure one heavy run alone and eight in parallel, then set a bound that survives both.
Put the bound on an invocation rather than on the regression, and say what the other way loses.
Bound the compile too.
Take the limit from configuration and say what hardcoding it causes.

Check Your Understanding

Why is a hang worse than a failure?
Is timeout(1) available here? What about gtimeout?
Why state that rather than silently substituting?
Give the four signals and what each produced.
What does the sleep control row establish?
Why is TERM yielding 0 dangerous for a timeout wrapper?
What does INT do, and what does that mean for an escalation?
What does KILL cost?
Why does no signal give you both the status and the evidence?
What does the wrapper do instead, and who is the only one who can?
Why must the flag be checked before the log?
State the generalisation about recording outcomes.
Why exec in the subshell?
Why a polling loop rather than wait?
Why sleep 0.1?
Why TERM before KILL, and why announce the escalation?
What were the three measured limits and overshoots?
Account for the overshoot.
At which scales does the imprecision matter?
State the sizing rule and what violating it causes.
What is a timeout for?
Why is it not a performance check, and where does that question belong?
What does a testbench watchdog produce that a kill does not?
What can a simulation-time bound not catch?
Why can a bound measured on an idle machine fire under load?
Why bound each invocation rather than the regression?
Why bound the compile?
Why take the limit from configuration?

Answers

Because it produces no decision at all, holds a slot, a licence and a node, and costs the queue behind it rather than just itself.
No, and no — and there is no Homebrew coreutils either. This is BSD userland.
Because the same gap appears whenever a flow is developed on a Mac and deployed on Linux, and the failure is timeout: command not found at two in the morning.
TERM ended it with rc 0 and a 105-byte log; INT was ignored; HUP ended it with rc 0 and 105 bytes; KILL gave rc 137 and a zero-byte log.
That the harness is sound — sleep gives the textbook 143, so the simulator's zero is the simulator's behaviour.
Because the wrapper kills a hung run and then reads 0, concluding the run passed.
Nothing at all, so an INT → TERM → KILL escalation wastes its first step and Ctrl-C leaves simulators running.
The buffered log — zero bytes, measured elsewhere at 88 lines.
TERM loses the status and keeps the evidence; KILL keeps the status and loses the evidence.
Records a timedout flag. The only process that knows is the one that did the killing.
Because a TERMed run exits 0 and its log is whatever was flushed, which can include a TEST PASSED from an earlier phase — so a later check can be overruled by the log.
When an action's outcome cannot be read back from the system, the actor records it.
So $! is the tool rather than a shell, which is what kill has to reach.
Because the shell must stay alive to enforce the bound.
So a one-second limit is meaningful.
Because TERM lets the tool flush its log and KILL empties it, so the hard kill is a last resort — and announcing it tells the reader why the log is empty.
1s → 1s, 2s → 2s, 4s → 5s.
sleep 0.1 is at least 0.1 s and the loop spawns a process per iteration; and the grace period adds up to two more seconds when the bound fires.
At seconds, where the grace period can be 25% of the limit. At minutes and hours it is negligible in relative terms.
Set the limit an order of magnitude above the expected runtime. A limit 10% above will fire on the first busy night and report a timeout for a run that was merely slow.
Bounding the damage of a hang — converting eight hours of a held slot into one limit's worth.
Because it reports TIMEOUT rather than "40% slower than last week". The runtime belongs in the results file's seconds column where a trend can see it.
A log with a verdict in it, which everything downstream can read — instead of a killed process and a flag.
A tool spinning outside simulation time.
Because contention alone gave a 4.5× spread between one heavy run and eight in parallel.
Because bounding the regression loses every run still in progress — four directories of zero-byte logs — while bounding an invocation loses one run.
Because a compile can hang on a network filesystem or a licence wait and holds up everything behind it.
Because a hardcoded limit is one nobody can raise for a single long test, so the wrapper gets copied and modified instead of reused.

7. Remember

timeout(1) is not installed here, and nor is gtimeout. The registry's own hint does not run on this machine — say so rather than substituting quietly.
TERM makes the simulator exit 0 and keeps its log; KILL gives 137 and empties it; INT is ignored. No signal gives you both the status and the evidence.
So the wrapper records timedout itself, and the verdict function checks it first — a TERMed log can contain a stale TEST PASSED.
When an outcome cannot be read back, the actor records it. Same rule as the build manifest and the machine snapshot.
exec so $! is the tool; poll rather than wait; sleep 0.1 so a short limit means something.
TERM, a grace period, then KILL — and announce the escalation, because it explains the empty log.
A 4-second limit took 5 seconds. The overshoot is the loop plus up to 2 s of grace: 25% at seconds, negligible at hours.
Size the limit an order of magnitude above the expected runtime, and measure it at the parallelism the regression actually uses — contention alone gave a 4.5× spread.
A timeout is not a performance check. Put the runtime in the results file and let a trend read it.
Prefer a watchdog inside the testbench, which produces a log with a verdict; keep the shell bound as the backstop.
Bound every invocation, including the compile, never the regression — and take the limit from configuration.

8. What Is Next

A bound stops a run that will not finish. Chapter 34.5 handles the run that cannot finish — one that runs out of space halfway — and measures what lands in the log when that happens, which turns out to be a message from the wrong program entirely.

Where this fits

Part of the Linux & Bash curriculum.