Linux & Bash · Part II · Module 30 · Running Work on Shared Engineering Machines

Several Jobs at Once, Without Losing Track

Four simulations from one shell is two lines of bash, and collecting their results correctly is not. This chapter measures the obvious way — which reports success for a batch in which two jobs failed — finds that the modern idiom for fixing it does not exist on this bash, and builds the pattern that works anyway.

Intermediate21 min readLinuxPart IIJobsExit StatusParallelism

Part II · Module 30 · Chapter 30.2 · Several Jobs at Once, Without Losing Track

Chapter 30.1 launched one long job and made it findable. Four at once is almost the same two lines, and the difference is entirely in how you collect the results.

1. The Obvious Way Reports Success

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ( exit 0 ) & ( exit 3 ) & ( exit 7 ) &
$ wait
$ echo $?
0

Three jobs exiting 0, 3 and 7, and the shell reports 0.

wait with no arguments waits for every child and returns 0 — not the worst status, not an aggregate, just zero. It is documented behaviour and it is the single most common way a parallel regression reports a clean night that was not.

The shape to recognise in a real script:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# this reports success whatever happens
for t in smoke fairness random burst; do
  ./run_one.sh "$t" > "logs/$t.log" 2>&1 &
done
wait
echo "all tests passed"      # printed unconditionally

Nothing about that is obviously wrong, which is why it survives review.

2. The Modern Fix Does Not Exist Here

Newer bash offers wait -n, which returns when the next job finishes and gives you its status — the building block for bounded parallelism. On this machine:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'sleep 0.1 & wait -n'
bash: line 0: wait: -n: invalid option
wait: usage: wait [n]
$ echo "$BASH_VERSION"
3.2.57(1)-release

Write for the bash that is on the server

wait -n arrived in bash 4.3. macOS ships 3.2.57 for licensing reasons, and a great many engineering servers are not far ahead — so a regression driver written against wait -n fails on the machine it was meant for, with an error about an invalid option rather than anything about jobs.

This is the same class of problem as Chapter 28.3's du --apparent-size and Chapter 22.1's GNU-versus-BSD spellings: the feature exists, the documentation describes it, and the machine in front of you does not have it.

Check once, at the top of anything that depends on a version:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# fail at the start with a clear message, not in the middle with a usage error
case "$BASH_VERSION" in
  (1.*|2.*|3.*) echo "this driver needs bash 4.3+ for 'wait -n'; found $BASH_VERSION" >&2; exit 2 ;;
esac

Or — better, and what the rest of this chapter does — do not depend on it. Everything needed is in bash 3.2.

3. The Pattern That Collects Every Status

wait with a pid argument returns that job's status. So keep the pids:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

pids=""
for code in 0 3 7 0; do
  ( exit $code ) & pids="$pids $!"
done

i=0; fails=0
set -- $pids
for p in "$@"; do
  wait "$p"; rc=$?
  i=$((i + 1))
  printf '  job %d pid=%-7s rc=%d\n' "$i" "$p" "$rc"
  [ "$rc" -ne 0 ] && fails=$((fails + 1))
done
printf '  collected %d statuses, %d non-zero\n' "$i" "$fails"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  job 1 pid=98387   rc=0
  job 2 pid=98388   rc=3
  job 3 pid=98389   rc=7
  job 4 pid=98390   rc=0
  collected 4 statuses, 2 non-zero

Every status, correctly attributed. Three things make it work.

$! is captured immediately. It holds the pid of the most recent background command and is overwritten by the next one, so it must be read on the line after the & — never later.

The pids are accumulated as a string and re-split with set --. Bash 3.2 has arrays and they would be cleaner (pids+=("$!")), and the string form is shown here because it is what survives being pasted into a sh-only context. Either works; what does not work is relying on $! to still be meaningful after the loop.

Each wait is separate. Waiting on a pid blocks until that job finishes, so the loop finishes in the order you wrote rather than the order they completed. For collecting statuses that is fine. For bounded parallelism — starting a new job as soon as any slot frees — it is not, and without wait -n that needs a different structure, which is Chapter 31.5's subject.

Waiting twice on the same pid is safe here

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ( exit 5 ) & p=$!
$ wait "$p"; echo "first  rc=$?"
first  rc=5
$ wait "$p" 2>/dev/null; echo "second rc=$?"
second rc=5

The second wait returned 5, not an error and not 127. Bash remembers the status of a reaped child, so a double-wait is idempotent rather than a bug — which matters because a loop that waits on a pid it may already have waited on does not need a guard.

Do not read too much into it. The memory is per-shell and finite, and a pid that was never your child gives a different answer entirely:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ out=$(wait 1 2>&1); echo "rc=$?  msg=$out"
rc=127  msg=bash: wait: pid 1 is not a child of this shell
$ out=$(wait 999999 2>&1); echo "rc=$?  msg=$out"
rc=127  msg=bash: wait: pid 999999 is not a child of this shell

127 for "not my child" — and the same 127 and the same wording for a pid that never existed at all, so the message does not distinguish "somebody else's process" from "no such process". Against a real status for "already reaped", that gives three cases with two outcomes:

the pid	`wait` returns
your child, still running	its status when it finishes
your child, already reaped	its status again
not your child, or nonexistent	127, with an identical message for both

4. Knowing Which Job Is Which

jobs -l lists them with pids, and it works in a script even though job control is off there:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ set -o | awk '/monitor/{print $2}'
off
$ sleep 5 & sleep 5 &
$ jobs -l
[3]  98356 Exit 7                  ( exit 7 )
[4]- 98361 Running                 sleep 5 &
[5]+ 98362 Running                 sleep 5 &

Three things in that output are worth knowing.

Job control is off and jobs still works. What monitor mode adds is process groups per job, fg/bg, and job-control signals — not the table. So a script can enumerate its own jobs.

A finished job is still listed, with its status: [3] Exit 7. Bash retains a terminated job until it has been reported once, which is why a jobs call in a loop can show a job that is already gone.

- and + mark the previous and current jobs. Those are what a bare fg or %+ refers to, and they shift as jobs finish — so a script should never use them. Use the pid.

For a batch you want the mapping written down rather than inferred:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# runbatch TEST... — run each test in its own directory, collect every status.
set -u
[ $# -gt 0 ] || { echo "usage: runbatch TEST..." >&2; exit 2; }
runs=${RUNS:?runbatch: set RUNS}
[ -d "$runs" ] || { echo "runbatch: no such directory: $runs" >&2; exit 2; }

pids=""; names=""
for t in "$@"; do
  d="$runs/$t"
  mkdir -p "$d" || exit 2
  ( cd "$d" && exec vvp "$SIM" "+TESTNAME=$t" +STRICT ) > "$d/run.log" 2>&1 &
  pids="$pids $!"
  names="$names $t"
  printf 'started %-12s pid=%-7s %s\n' "$t" "$!" "$d/run.log"
done

# the mapping, on disk, because $! and the jobs table are both transient
paste <(printf '%s\n' $pids) <(printf '%s\n' $names) > "$runs/batch.map"

bad=0
set -- $pids
i=0
for p in "$@"; do
  i=$((i + 1))
  t=$(printf '%s\n' $names | sed -n "${i}p")
  wait "$p"; rc=$?
  log="$runs/$t/run.log"
  if   grep -q 'TEST PASSED' "$log" 2>/dev/null; then v=PASS
  elif grep -q 'TEST FAILED' "$log" 2>/dev/null; then v=FAIL
  else v=NO-VERDICT; fi
  printf '%-12s pid=%-7s rc=%-4s %s\n' "$t" "$p" "$rc" "$v"
  [ "$v" = PASS ] || bad=$((bad + 1))
done
printf '%d of %d not passing\n' "$bad" "$#"
exit $(( bad > 0 ))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ RUNS=$PWD/batch SIM=$PWD/logs/plusarg.vvp runbatch smoke fairness random bogus
started smoke        pid=1959    /…/batch/smoke/run.log
started fairness     pid=1961    /…/batch/fairness/run.log
started random       pid=1963    /…/batch/random/run.log
started bogus        pid=1965    /…/batch/bogus/run.log
smoke        pid=1959    rc=0    PASS
fairness     pid=1961    rc=0    PASS
random       pid=1963    rc=0    PASS
bogus        pid=1965    rc=0    FAIL
1 of 4 not passing
$ echo $?
1
$ cat batch/batch.map
1959	smoke
1961	fairness
1963	random
1965	bogus

Look at the rc column: all four exited 0, including the one that failed. The PASS/FAIL column came from the log, and the two columns beside each other are the whole of Module 29's argument in one screen.

And the refusals:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ runbatch
usage: runbatch TEST...
$ ( unset RUNS; runbatch smoke )
runbatch: line 5: RUNS: runbatch: set RUNS
$ RUNS=/no/such runbatch smoke
runbatch: no such directory: /no/such

Four decisions.

The mapping is written to batch.map. Both $! and the jobs table are transient — the first is overwritten immediately and the second is cleared as jobs are reported — so the only durable record is a file.

The verdict comes from the log, and rc is printed beside it. Module 29 in one line: a failing simulation exits 0, so the status is evidence and not the answer.

NO-VERDICT is a third state, for the job that was killed or starved — Chapter 29.5.

It exits non-zero if anything is not a pass, so the batch is usable in a condition.

5. Where the Output Goes

The reason each job gets its own directory is Chapter 27.3's measurement — four concurrent runs sharing one directory produced a single waveform that matched none of them. Running several at once makes two further redirection mistakes easy, and both are silent.

Appending instead of truncating. >> on a shared log interleaves four jobs' output at line granularity if they are line-buffered, and at block granularity if they are not:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# wrong: four jobs, one log, interleaved at whatever granularity the buffers use
./run_one.sh a >> batch.log 2>&1 &
./run_one.sh b >> batch.log 2>&1 &

Chapter 30.3 measures the buffering directly; the consequence here is that the interleaving is not even predictable.

Forgetting 2>&1 on one of them. Three jobs' stderr goes to a file and the fourth's goes to your terminal, so one job's errors are the only ones you see and the other three look clean:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# the fourth job's stderr is missing from its log
./run_one.sh d > logs/d.log &

The defence is to not write the redirection four times:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

run() {                      # one place where redirection is decided
  local t=$1 d="$RUNS/$1"
  mkdir -p "$d" || return 2
  ( cd "$d" && exec vvp "$SIM" "+TESTNAME=$t" +STRICT ) > "$d/run.log" 2>&1 &
}
for t in "$@"; do run "$t"; pids="$pids $!"; done

$! after a function call that backgrounds something still works, and it is more robust than it looks:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ run()  { ( sleep 2 ) & }
$ run2() { ( sleep 2 ) & echo started >/dev/null; }
$ run;  p1=$!; kill -0 "$p1" && echo "last command was the &: \$!=$p1 valid"
last command was the &: $!=1923 valid
$ run2; p2=$!; kill -0 "$p2" && echo "a command AFTER the &:  \$!=$p2 valid"
a command AFTER the &:  $!=1927 valid

Both work, because $! is only updated by the background operator — an ordinary command afterwards does not disturb it, and neither does returning from the function. What does overwrite it is the next &, which is why §3 captures it immediately inside the loop.

Practice

Run three subshells exiting 0, 3 and 7, then bare wait, and record $?.
Write the four-line loop that looks correct and reports success unconditionally.
Try wait -n and record the exact error and your bash version.
Add the version guard and confirm it fires with a clear message.
Capture $! one line later than the & and see which pid you get.
Collect four statuses with the pid loop and confirm two are non-zero.
Wait twice on the same pid and record both statuses.
Wait on pid 1 and on a pid that cannot exist, and compare the two messages.
Check whether monitor is on in a script, then call jobs -l anyway.
Find a completed job in jobs -l output and explain why it is still listed.
Write runbatch and confirm it exits non-zero when one test fails.
Delete batch.map and try to work out which pid was which from the jobs table.
Send four jobs' output to one appended log and look at how the lines interleave.
Omit 2>&1 from one job and find where its errors went.

Check Your Understanding

What does bare wait return when three jobs exit 0, 3 and 7?
Why does the four-line loop survive code review?
What is wait -n for, and what happens here?
Which bash version introduced it, and which is on this machine?
What updates $!, and what does not?
Why does the collection loop finish in the order written rather than the order completed?
What is that unsuitable for, and which module addresses it?
What does a second wait on the same pid return?
What does wait 1 return, and how does it differ from wait 999999?
Is job control on in a script, and does jobs work?
Why is a finished job still in the jobs listing?
Why should a script never use %+ or %-?
Why is the pid-to-name mapping written to a file?
Give the two silent redirection mistakes that appear only with several jobs.

Answers

0. It waits for every child and returns zero — not the worst status and not an aggregate.
Because nothing about it looks wrong: the loop is correct, the redirections are correct, and only the result collection is missing.
It returns when the next job finishes and gives its status, which is the building block for bounded parallelism. Here it is an invalid option error.
Bash 4.3; this machine has 3.2.57.
Only the background operator & updates it. An ordinary command afterwards does not, and neither does returning from a function — but the next & does, which is why it is captured immediately inside a loop.
Because wait <pid> blocks until that job finishes, regardless of whether others completed first.
Bounded parallelism — starting a new job the moment any slot frees. Chapter 31.5 covers it.
The same status again — 5 — because bash remembers a reaped child's status, so a double-wait is idempotent.
127, with pid 1 is not a child of this shell — and wait 999999 gives the identical status and wording, so the message does not distinguish somebody else's process from no process at all. Against a real status for an already-reaped child, that is three cases with two outcomes.
monitor is off, and jobs works anyway. What job control adds is per-job process groups, fg/bg and job-control signals — not the table.
Because bash retains a terminated job until it has been reported once, so a jobs call can show a job that has already gone.
Because they refer to the current and previous jobs, which shift as jobs finish. Use the pid.
Because $! is overwritten immediately and the jobs table is cleared as jobs are reported — a file is the only durable record.
>> to a shared log, which interleaves at whatever granularity the buffers use; and omitting 2>&1 on one job, so three jobs' errors are captured and the fourth's go to your terminal.

6. Remember

Bare wait returns 0 even when jobs exited 3 and 7. It is the commonest way a parallel regression reports a clean night that was not.
wait -n needs bash 4.3 and this is 3.2.57. Guard the version or do not depend on it.
Only & updates $!. A command after it, or returning from a function, leaves it intact; the next & overwrites it.
wait <pid> per job collects every status, correctly attributed.
Waiting twice on a pid returns the same status; a non-child or a nonexistent pid returns 127 with the same message for both.
jobs -l works with job control off, lists finished jobs until reported, and marks +/- positions that shift — so scripts use pids.
Write the pid-to-name mapping to a file. Both $! and the jobs table are transient.
Take the verdict from the log and print rc beside it. A failing simulation exits 0.
One directory per job, and decide redirection in exactly one place so no job loses its stderr.

7. What Is Next

Four jobs are running and their results will be collected correctly. Chapter 30.3 watches them while they run — and finds that a log with nothing new in it is no evidence at all that a job has stopped working.

Where this fits

Part of the Linux & Bash curriculum.