Starting Work That Outlives Your Session

Part II · Module 30 · Chapter 30.1 · Starting Work That Outlives Your Session

Module 29 made the artifacts of a finished run trustworthy. This module is about the run while it is happening, on a machine you share, and it starts with the most-repeated piece of advice in the field.

1. The Advice, and the Measurement

nohup ./run_regression.sh > nightly.log 2>&1 &

The claim is that nohup makes the job survive your session ending. Tested against four programs, each started in the background and then sent SIGHUP directly:

nohup protected sleep and a shell script, and did not protect the simulator.

That is not a measurement error, and it is not a quirk of how the test was staged. The next two sections establish both halves: that nohup really does what it claims, and that the simulator undoes it.

2. nohup Works. Here Is the Proof.

nohup sets SIGHUP to ignore and then executes your command. An ignored disposition survives exec — that is specified behaviour — so the question is whether it actually arrives. A short C program can read its own disposition at startup and say:

#include <stdio.h>
#include <signal.h>
int main(void) {
    struct sigaction sa;
    sigaction(SIGHUP, NULL, &sa);
    printf("SIGHUP disposition at startup: %s\n",
           sa.sa_handler == SIG_IGN ? "SIG_IGN (inherited from nohup)" :
           sa.sa_handler == SIG_DFL ? "SIG_DFL (default)" : "a handler");
    return 0;
}

$ cc -o /tmp/sigq /tmp/sigq.c
$ /tmp/sigq
SIGHUP disposition at startup: SIG_DFL (default)
$ nohup /tmp/sigq
SIGHUP disposition at startup: SIG_IGN (inherited from nohup)

So the mechanism is sound: nohup sets SIG_IGN, and a freshly execed program sees it.

3. Why the Simulator Dies Anyway

Because a disposition inherited at startup can be changed at startup. A program that calls signal(SIGHUP, …) or sigaction() for itself overrides whatever it inherited, and vvp does.

The cleanest confirmation is to stack the inheritance two deep — nohup a shell, which execs the simulator, so the SIG_IGN passes through twice:

$ cat /tmp/wrap.sh
#!/bin/bash
exec vvp /…/m30/logs/long.vvp
$ nohup /tmp/wrap.sh > /tmp/w.log 2>&1 &
$ sleep 1; kill -HUP $!
$ sleep 1; kill -0 $! 2>/dev/null && echo SURVIVED || echo died
died

4. What Actually Protects a Long Run

The useful question is not how to make a process ignore SIGHUP, but why the signal is sent at all — and the answer makes most of the problem disappear.

SIGHUP is delivered by the kernel to a session's foreground process group when its controlling terminal goes away. Two things follow.

The child shares your process group, which is why it is in the blast radius:

$ ( vvp logs/long.vvp > logs/pg.log 2>&1 ) & ch=$!
$ printf 'this shell:  pid=%-7s pgid=%s\n' "$$" "$(ps -o pgid= -p $$ | tr -d ' ')"
this shell:  pid=97058   pgid=97054
$ printf 'the child:   pid=%-7s pgid=%s\n' "$ch" "$(ps -o pgid= -p $ch | tr -d ' ')"
the child:   pid=97061   pgid=97054

Same group. (Demonstrating the kill is awkward for exactly this reason: kill -HUP -97054 takes down the shell running the demonstration along with its subject. That is not a limitation of the measurement — it is the measurement.)

And with no controlling terminal, nothing sends it. A background child of a non-interactive script already survives its parent, with or without nohup:

$ cat /tmp/parent_plain.sh
#!/bin/bash
cd "$1"
vvp logs/long.vvp > logs/plain.log 2>&1 &
echo $! > /tmp/plain.pid
exit 0
$ /tmp/parent_plain.sh "$PWD"
$ sleep 2; p=$(cat /tmp/plain.pid)
$ kill -0 $p && echo "child alive, log $(wc -c < logs/plain.log) bytes"
child alive, log    28672 bytes

The parent exited and the child kept working. Repeating it with the parent killed by HUP rather than exiting cleanly gives the same answer — the child survives, because the HUP went to the parent and nothing propagated it.

So the real rule is about where the job is started from, not about nohup:

The last two rows are the ones a real flow uses, and the fourth is why a nightly regression launched from cron has never needed nohup.

5. Launching It So Somebody Else Can Find It

A nine-hour job that nobody can identify is nearly as bad as one that died. Four things belong in the launch, and all of them are cheap:

#!/bin/bash
# launch-long — start a long run detached, identified, and findable.
set -u
name=${1:?usage: launch-long NAME -- CMD...}; shift
[ "${1:-}" = "--" ] && shift
[ $# -gt 0 ] || { echo "launch-long: no command" >&2; exit 2; }

runs=${RUNS:?launch-long: set RUNS to the runs area}
[ -d "$runs" ] || { echo "launch-long: no such directory: $runs" >&2; exit 2; }
dir="$runs/${name}_$(date -u '+%Y%m%dT%H%M%SZ')"
mkdir -p "$dir" || exit 2

# 1. who, what, where and when -- before anything starts
{ printf 'name     %s\n' "$name"
  printf 'user     %s\n' "$(id -un)"
  printf 'host     %s\n' "$(hostname -s)"
  printf 'started  %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
  printf 'cwd      %s\n' "$PWD"
  printf 'cmd     '; printf ' %q' "$@"; printf '\n'
} > "$dir/launch.txt"

# 2. The job runs IN the run directory, so any relative path in its argv would
#    resolve there instead of here. Make existing paths absolute first.
args=()
for a in "$@"; do
  if [ -e "$a" ]; then args+=("$(cd "$(dirname "$a")" && pwd)/$(basename "$a")")
  else args+=("$a"); fi
done

# 3. record the RESOLVED argv, one argument per line, so Chapter 27.9's replay
#    works from inside the run directory. launch.txt keeps what was typed.
: > "$dir/cmd.args"
for a in "${args[@]}"; do printf '%s\n' "$a" >> "$dir/cmd.args"; done

# 4. detach, with output to a file rather than to a terminal
( cd "$dir" && exec nohup "${args[@]}" ) > "$dir/run.log" 2>&1 &
pid=$!

# 5. the pid, so it can be found and signalled without guessing
printf '%s\n' "$pid" > "$dir/pid"
disown "$pid" 2>/dev/null || true

# 6. one line back, because the caller needs the directory more than the pid
printf '%s  pid=%s\n' "$dir" "$pid"

$ RUNS=$PWD/runs launch-long nightly -- vvp logs/long.vvp
/…/m30/runs/nightly_20261009T065510Z  pid=1463
$ d=$(ls -d runs/nightly_* | head -1)
$ ls $d
cmd.args  launch.txt  pid  run.log
$ cat $d/launch.txt
name     nightly
user     azharuddinsaifi
host     MacBook-Air
started  2026-10-09T06:54:47Z
cwd      /…/m30
cmd      vvp logs/long.vvp
$ cat -n $d/cmd.args
     1	vvp
     2	/…/m30/logs/long.vvp
$ kill -0 $(cat $d/pid) && echo alive
alive

Six details are deliberate.

Five details are deliberate.

The run directory is created before the job starts, so launch.txt exists even if the command fails immediately — Chapter 27.3's arrangement. Verified:

$ RUNS=$PWD/runs launch-long bogus -- /no/such/tool
$ b=$(ls -d runs/bogus_* | head -1)
$ [ -f "$b/launch.txt" ] && echo "launch.txt present"; head -1 "$b/run.log"
launch.txt present
nohup: /no/such/tool: No such file or directory

The argv is made absolute before the cd — see the callout above.

exec nohup "$@" inside the subshell, so $! is the tool and not an intervening shell. Chapter 27.7 measured what happens without exec: $! is the subshell, a kill lands there, and the simulator is orphaned and keeps writing. The nohup is kept despite §3 — it costs nothing and it helps for the tools that honour it.

disown removes the job from the shell's table so an interactive shell will not warn on exit and will not HUP it. It is a bash builtin and a no-op in a non-interactive shell, hence the || true.

The pid is written to a file. $! is available for one line in one shell; the file is available to whoever is on call.

The printed line is the directory, not the pid. The next thing anybody does is read the log, and the directory is what they need.

And the three refusals, each at the point of use:

$ ( unset RUNS; launch-long x -- true )
launch-long: line 8: RUNS: launch-long: set RUNS to the runs area
$ RUNS=/no/such launch-long x -- true
launch-long: no such directory: /no/such
$ RUNS=$PWD/runs launch-long x --
launch-long: no command

6. Checking On It Later

Three questions, three commands — and the third is the one people get wrong:

# is it still running?
pid=$(cat "$dir/pid"); kill -0 "$pid" 2>/dev/null && echo running || echo gone

# what has it said?
tail -5 "$dir/run.log"

# how long has it been going, and how hard is it working?
ps -o pid,etime,rss,pcpu -p "$pid"

The trap is in the second one. A log with nothing new in it does not mean the job has stopped making progress — Chapter 30.3 measures a simulation that worked for seven seconds with its log at exactly 0 bytes, because output to a file is block-buffered. ps is the reliable signal and the log is not.

And when the job is gone, the question becomes whether it finished:

# gone is not the same as finished -- Chapter 29.5's three states
if grep -qE 'TEST (PASSED|FAILED)|Report Summary' "$dir/run.log"; then
  echo "finished: $(grep -oE 'TEST (PASSED|FAILED)' "$dir/run.log" | tail -1)"
else
  echo "gone with no verdict -- killed, evicted, or starved of events"
fi

That is Chapter 29.5's NO-VERDICT state arriving in its most common real form: you come back in the morning and the process is not there.

Practice

1. Test nohup on sleep and confirm it survives a direct SIGHUP.
2. Test it on your own simulator with the two-command check from §3's callout.
3. Write the C probe and confirm SIG_IGN is inherited with nohup and not without.
4. Stack the inheritance — nohup a script that execs the tool — and see whether that changes the answer.
5. Print your shell's pgid and a background child's and confirm they match.
6. Try kill -HUP -<pgid> from a shell you do not mind losing, and explain what happened to the demonstration.
7. Launch a background job from a non-interactive script that then exits, and confirm the child survives without nohup.
8. Do the same with the parent killed by HUP rather than exiting.
9. Fill in the six-row survival table for your own environment.
10. Write launch-long and confirm launch.txt exists even when the command is bogus.
11. Remove the exec from the subshell, kill $!, and check whether the tool is still running.
12. Check on a running job three ways, then let its log go quiet and decide which signal you trust.
13. Come back to a finished job and distinguish "finished" from "gone".

Check Your Understanding

1. Which of the four tested programs did nohup protect, and which did it not?
2. How was it established that nohup genuinely sets SIG_IGN?
3. Why does the simulator die anyway?
4. What does stacking nohup over an execing wrapper demonstrate?
5. Why might a simulator legitimately refuse to ignore SIGHUP?
6. Why is "the wrapper survives and the tool does not" the worst outcome?
7. What sends SIGHUP, and to whom?
8. Why is kill -HUP -<pgid> awkward to demonstrate?
9. Why does a background child of a non-interactive script survive without nohup?
10. Which two rows of the survival table does a real flow rely on?
11. Why is the run directory created before the job starts?
12. What does exec inside the subshell fix?
13. Why print the directory rather than the pid?
14. Why is a quiet log not evidence that a job has stalled?

7. Remember

• nohup protected sleep and a shell script and did not protect the simulator. Measured on both.
• nohup does work: a C probe showed SIG_DFL plainly and SIG_IGN under nohup, inherited across exec.
• The tool overrides it. A program may set its own SIGHUP disposition, and simulators commonly do — often for good reasons.
• So nohup is advisory. Test it on your own tool; it is two commands.
• The wrapper surviving while the tool dies is the worst outcome, because nothing looks wrong.
• SIGHUP goes to the foreground process group of a session losing its terminal, and your child shares your group.
• With no controlling terminal nothing sends it — a background child of a script survives its parent inherently.
• Real flows rely on a scheduler or a screen/tmux session, not on nohup.
• Create the run directory and write launch.txt first, and exec inside the subshell so $! is the tool.
• Write the pid to a file, print the directory, and disown.
• A quiet log is not a stalled job. Output to a file is buffered; use ps.

8. What Is Next

One long job is launched and findable. Chapter 30.2 runs several at once from one shell — where the obvious way to collect their results returns success even when two of them failed.