Stopping Work Properly

Part II · Module 30 · Chapter 30.4 · Stopping Work Properly

Chapter 30.3 established that the job is stuck rather than slow. Now it has to be stopped, and the choice of signal decides how much of the evidence survives — which matters, because the reason you are stopping it is usually that you want to know why.

1. Four Signals, One Simulation

A long-running simulation with waveform dumping on, allowed to run for two seconds and then sent exactly one signal:

Three things in that table need separate explanation, and the second one is the chapter.

vvp survives SIGINT. That is what Ctrl-C sends. Three of them do not help:

$ ( cd sig2/INT2 && exec vvp ../../logs/dumphang.vvp > run.log 2>&1 ) & p=$!
$ sleep 2
$ for i in 1 2 3; do kill -INT $p; sleep 0.5; done
$ kill -0 $p && echo "STILL ALIVE" || echo died
STILL ALIVE
$ kill -TERM $p; sleep 1; kill -0 $p && echo alive || echo died
died

So the thing an engineer reaches for first — Ctrl-C — does nothing to this tool, and the thing that works is kill with no options, which sends TERM. Chapter 30.1 measured the same tool declining SIGHUP's ignore request; this is the same program making its own decisions about signals, in the other direction.

No signal produced a verdict line. Every row is 0. A signalled run is always Chapter 29.5's NO-VERDICT state — which means you cannot tell from the log whether you stopped it or it died, and the only record that it was deliberate is the one you write yourself.

And KILL left zero log lines while leaving a 45 MB waveform. That is §2.

2. What a Hard Kill Costs, to the Byte

The table above has a confound: each run was stopped after two seconds but the signals arrived at slightly different points, so the file sizes are not comparable. Repeating it with exactly three seconds for both:

$ for s in TERM KILL; do
>   d="sig2/$s"; rm -rf "$d"; mkdir -p "$d"
>   ( cd "$d" && exec vvp ../../logs/dumphang.vvp > run.log 2>&1 ) & p=$!
>   sleep 3                      # identical for both
>   kill -$s $p 2>/dev/null; sleep 1.5
>   kill -9 $p 2>/dev/null; wait $p 2>/dev/null
>   printf '  %-7s %-12s %-12s %s\n' "$s" "$(grep -c . $d/run.log)" \
>          "$(wc -c < $d/run.log | tr -d ' ')" "$(wc -c < $d/dump.vcd | tr -d ' ')"
> done
  SIGNAL  log lines    log bytes    vcd bytes
  TERM    292          5949         69634736
  KILL    204          4096         50679808

3. So the Order Is TERM, Wait, Then KILL

#!/bin/bash
# stopjob PID [GRACE] — stop a run and keep as much evidence as possible.
set -u
pid=${1:?usage: stopjob PID [GRACE]}
grace=${2:-10}

kill -0 "$pid" 2>/dev/null || { echo "stopjob: no such process: $pid" >&2; exit 2; }

# 1. ask. INT is not used: the simulator ignores it (measured in 30.4 §1).
printf 'stopjob: sending TERM to %s, allowing %ss\n' "$pid" "$grace"
kill -TERM "$pid" 2>/dev/null

# 2. wait, so the process can flush and close what it is holding
i=0
while [ "$i" -lt "$grace" ]; do
  kill -0 "$pid" 2>/dev/null || { printf 'stopjob: stopped cleanly after %ss\n' "$i"; exit 0; }
  sleep 1
  i=$((i + 1))
done

# 3. only now, and say what it will cost
printf 'stopjob: still alive after %ss -- sending KILL\n' "$grace" >&2
printf 'stopjob: the unflushed tail of its log will be lost\n' >&2
kill -KILL "$pid" 2>/dev/null
sleep 1
kill -0 "$pid" 2>/dev/null && { echo "stopjob: $pid survived KILL -- it is stuck in the kernel" >&2; exit 1; }
printf 'stopjob: killed\n'
exit 0

$ ( cd t1 && exec vvp ../logs/dumphang.vvp > run.log 2>&1 ) & stopjob $! 10
stopjob: sending TERM to 2354, allowing 10s
stopjob: stopped cleanly after 1s
$ echo $?
0

The escalation path needs a process that genuinely ignores TERM, which is three lines of C:

#include <signal.h>
#include <unistd.h>
int main(void) { signal(SIGTERM, SIG_IGN); for (;;) sleep(1); return 0; }

$ /tmp/stubborn & stopjob $! 3
stopjob: sending TERM to 2406, allowing 3s
stopjob: still alive after 3s -- sending KILL
stopjob: the unflushed tail of its log will be lost
stopjob: killed
$ echo $?
0
$ kill -0 2406 2>/dev/null && echo "STILL ALIVE" || echo gone
gone

And a pid that is not there:

$ stopjob 999999; echo "exit=$?"
stopjob: no such process: 999999
exit=2

A note on how that was tested, because the obvious shell version does not work as a substitute. trap "" TERM in a bash script does protect the shell:

$ printf '#!/bin/bash\ntrap "" TERM\nsleep 60\n' > /tmp/ig.sh; chmod +x /tmp/ig.sh
$ /tmp/ig.sh & sleep 1; kill -TERM $!; sleep 1
$ kill -0 $! 2>/dev/null && echo alive || echo died
alive

— but a shell script's lifetime is driven by the command it is waiting on, which makes it a muddier subject than a single process with an explicit SIG_IGN. When you need to prove an escalation path fires, use the unambiguous subject.

Four decisions.

INT is not in the sequence. §1 measured the tool ignoring it, and a signal that does nothing is worse than no signal at all: it makes the stop appear to have been attempted.

The grace period is a loop with a one-second poll, not a sleep "$grace". A job that stops in half a second should not hold the script for ten.

The cost of the KILL is printed before it is sent. The person reading the output is about to lose log lines and should know it from the tool rather than from this chapter.

A process surviving KILL is reported, not retried. SIGKILL cannot be caught, so a process still present afterwards is blocked in an uninterruptible kernel call — an unresponsive filesystem being the usual cause on a shared machine — and no amount of signalling will help. That exits 1 so the caller knows the job is still there.

4. Stopping the Right Process

Two ways to send a signal to something other than what you intended, both measured in earlier chapters and both worth restating here because this is the chapter where you actually send one.

$! is the subshell, not the tool, unless you exec:

# $! is the subshell; vvp is orphaned and keeps writing
( cd "$d" && vvp sim.vvp > run.log 2>&1 ) &

# $! is vvp
( cd "$d" && exec vvp sim.vvp > run.log 2>&1 ) &

Chapter 27.7 measured the consequence: wait reported 137 as though the job had been killed while the simulator carried on from 27.8 MB to 43.6 MB of output and then wrote a verdict.

A pid can be reused. By the time a cleanup script reads a pid file and signals it, that pid may belong to something else entirely. The guard is to check identity before signalling:

# is this still the process we think it is?
want=$(cat "$dir/pid")
if ps -o command= -p "$want" 2>/dev/null | grep -q 'vvp'; then
  kill -TERM "$want"
else
  echo "pid $want is not our simulation any more -- not signalling" >&2
fi

Measured in both directions on a real run:

$ ( cd t2 && exec vvp ../logs/dumphang.vvp > run.log 2>&1 ) & echo $! > t2/pid
$ want=$(cat t2/pid)
$ ps -o command= -p "$want" | grep -q vvp && echo "pid $want IS our simulation"
pid 2367 IS our simulation
$ kill -9 "$want"; sleep 1
$ ps -o command= -p "$want" 2>/dev/null | grep -q vvp \
>   && echo "still matches" || echo "no longer matches -- would NOT signal"
no longer matches -- would NOT signal

That check is cheap and the failure it prevents is not: a cleanup job that signals a recycled pid on a shared machine kills somebody else's work, and nothing in your logs will show that you did it.

5. Recording That the Stop Was Deliberate

§1's measurement — no signal produces a verdict — has a direct consequence: a stopped run is indistinguishable in its log from a crashed one. The fix is not technical, it is a note:

# in stopjob, before signalling
{ printf 'stopped   %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
  printf 'by        %s@%s\n' "$(id -un)" "$(hostname -s)"
  printf 'signal    TERM then KILL after %ss\n' "$grace"
  printf 'reason    %s\n' "${REASON:-not given}"
} > "$dir/stopped.txt"

Four lines, and they convert a NO-VERDICT run from a mystery into a record. Three weeks later the question "did this fail or did somebody stop it" has an answer, and on a shared machine the question "who stopped my job" has one too.

REASON being optional with a default of not given is deliberate: a required field would be skipped by making the script non-interactive, and an optional one that records its own absence is honest. Chapter 28.4's cleanup made the opposite choice — a marker file that cannot be satisfied by accident — because there the cost of being wrong was deleted data rather than a missing note.

And the complementary check, for whoever finds the run later:

# why does this run have no verdict?
if [ -f "$dir/stopped.txt" ]; then
  echo "stopped deliberately:"; cat "$dir/stopped.txt"
elif grep -q 'progress' "$dir/run.log" 2>/dev/null; then
  echo "no verdict, but it was making progress -- killed, evicted, or timed out"
  tail -1 "$dir/run.log"
else
  echo "no verdict and no progress -- see Chapter 29.5's three shapes"
fi

6. What Survives, Summarised

Putting §1 and §2 together, the evidence after each way of stopping:

Two readings of that table matter.

Rows three and four are identical in their artifacts. A hard kill and a power loss leave the same evidence, which is why stopped.txt is the only thing that distinguishes them — and why a run with no verdict and no note should be assumed to have died rather than been stopped.

The waveform is complete to the moment in every row. Which means Chapter 27.6's dumping decision is what determines whether a stopped run is debuggable at all: a run with no dump leaves a truncated log and nothing else, and a windowed dump leaves nothing if the window had not opened yet. On a long job that you might have to stop, whole-run dumping at a reduced depth is the arrangement that survives being interrupted — the only case in this track where the expensive option is the right default.

Practice

1. Send SIGINT to your simulator and check whether it stops.
2. Send three, then one TERM, and record what each did.
3. Build the four-signal table for your own tool.
4. Check whether any signal produced a verdict line.
5. Stop the same run with TERM and with KILL after identical elapsed times and compare log sizes.
6. Confirm the KILL log size is a whole multiple of the buffer block, and work out how many lines that cost.
7. Compare the waveform sizes in the same pair and explain the asymmetry.
8. Write stopjob with the poll loop, and confirm it exits 0 when the job stops in under a second.
9. Make a job ignore TERM and confirm the script escalates and says what it will cost.
10. Launch without exec, signal $!, and check whether the tool is still running.
11. Add the pid-identity check and confirm it refuses to signal a recycled pid.
12. Check whether setsid exists on your machine, and decide what you would do about orphans.
13. Write stopped.txt and then use the complementary check to classify a stopped run.
14. Decide what your dumping policy should be for a job you might have to interrupt.

Check Your Understanding

1. Which signal does the simulator ignore, and what sends it?
2. What happened after three of them, and what worked instead?
3. How many of the four signals produced a verdict line?
4. What size was the log after KILL, and why that number exactly?
5. How many log lines did the hard kill cost, and which ones were they?
6. Why did the waveform survive while the log did not?
7. Why is that the opposite of the intuition?
8. Why is INT left out of the stop sequence?
9. Why a poll loop instead of sleep "$grace"?
10. What does a process surviving SIGKILL mean?
11. What does exec fix about signalling?
12. What does the pid-identity check prevent on a shared machine?
13. Why does this chapter not recommend killing the process group?
14. Why is REASON optional when the cleanup chapter's marker file is not?
15. Which two rows of the final table are indistinguishable, and what follows?
16. Which dumping policy survives an interruption, and why is that unusual advice?

7. Remember

• The simulator survives SIGINT — three of them did nothing. Plain kill, which sends TERM, works.
• No signal produces a verdict line. A signalled run is always a NO-VERDICT run.
• KILL left the log at exactly 4,096 bytes against TERM's 5,949 at identical elapsed time: 88 lines lost, and the most recent ones.
• The waveform survived both. The streamed artifact is resilient and the buffered one is fragile — the opposite of the intuition.
• TERM, poll, then KILL, with the cost printed before the escalation.
• Leave INT out. A signal the tool ignores makes the stop look attempted.
• A process surviving KILL is stuck in the kernel. Report it; do not retry.
• exec so $! is the tool, and check the pid's identity before signalling it on a shared machine.
• Do not kill the process group — your child shares your shell's group and setsid is absent here.
• Write stopped.txt. A hard kill and a power loss leave identical artifacts, and the note is the only difference.
• For a job you might interrupt, dump the whole run at reduced depth — a window that never opened leaves nothing.

8. What Is Next

Stopping and starting work by hand is bounded by how much you can keep track of. Chapter 30.5 hands the job to a scheduler instead — and measures the three things submission changes about your command, each of which breaks a script that works perfectly at your own prompt.