Bounded Parallelism: wait, xargs -P and Job Slots

Part II · Module 31 · Chapter 31.5 · Bounded Parallelism

Chapter 31.4 produced a list of 240 runs and priced it at 76 seconds sequential. This box has 8 cores. The obvious move is to run several at once, and the two questions that matter are how many and how do you know.

1. Unbounded Is Not an Option

The shortest way to run everything at once:

# DO NOT do this with a real matrix
while read -r spec; do
  run_one.sh "$spec" &
done < specs.txt
wait

With 240 specs that forks 240 simulators. Each one of this fixture's heavy runs holds about 9 MB resident and wants a core; 240 of them want 2 GB and 240 cores, and the machine spends its time context-switching and paging rather than simulating. On a real block — where one simulation is hundreds of megabytes — the same loop is a machine that stops responding.

So the pattern is slots: a fixed number of runs in flight, a new one starting as one finishes.

2. Three Ways to Bound It

xargs -P

$ xargs -P 8 -I{} run_one.sh {} < specs.txt

Eight at a time, a ninth starting the moment one exits. It is one line, it is in POSIX-adjacent daily use, and it is what the rest of this chapter measures.

A hand-rolled slot counter

Useful when each job needs more setup than a single command:

slots=8
running=0
while IFS= read -r spec; do
  run_one.sh "$spec" &
  running=$((running + 1))
  if [ "$running" -ge "$slots" ]; then
    wait                      # <-- waits for ALL of them, not one
    running=0
  fi
done < specs.txt
wait

This works and it is worse than it looks. Plain wait blocks until every background job finishes, so the loop runs in batches of eight and the whole batch waits for its slowest member. With one 30-second run among seven 1-second runs, seven slots sit idle for 29 seconds.

Why not wait -n

The fix in a modern bash is wait -n, which returns when any one job finishes, so a new job can start immediately:

$ /bin/bash -c 'sleep 0.1 & wait -n; echo "rc=$?"'
/bin/bash: line 0: wait: -n: invalid option
wait: usage: wait [n]
rc=2

3. Measuring the Curve — the Light Jobs

A warm single run of this fixture, three times:

$ bash bin/probe_speed.sh
=== a warm single run, three times ===
  run 1: 405 ms
  run 2: 370 ms
  run 3: 402 ms

So 16 of them is about 6.2 seconds sequential. The sweep, three repetitions at each slot count:

=== the -P sweep, three repetitions each (16 jobs, 8 cores) ===
  -P    rep1      rep2      rep3      median    vs -P 8
  1     5915ms    10588ms   6205ms    6205ms    -
  2     3246ms    3389ms    4753ms    3389ms    -
  4     2175ms    2216ms    3264ms    2216ms    -
  8     8527ms    5636ms    3195ms    5636ms    1.00x
  16    2259ms    2360ms    1650ms    2259ms    .40x
  32    1780ms    1367ms    1228ms    1367ms    .24x

Read the -P 8 row before reading anything else. Its median is 5636 ms, which is slower than -P 4's 2216 ms, on a machine with 8 cores, for work that is not memory-bound. That cannot be true. Its three repetitions were 8527, 5636 and 3195 — a spread of 2.7× within one configuration.

The honest conclusion is not "-P 8 is slow". It is the run-to-run noise on this measurement is larger than the effect being measured, so the table cannot support a claim about the shape of the curve.

Which is worth stating plainly, because the table is otherwise very persuasive. It has repetitions, it has a median, it has a normalised column, and the normalised column is nonsense.

What the table can support

Two things survive the noise, because they are much larger than it.

-P 1 at 6205 ms and -P 32 at 1367 ms differ by 4.5×, far beyond the spread. Parallelism helps a great deal.

And one clean repetition is monotone. Reading rep3 alone — a single sweep, in order, with nothing else contending:

That is a sensible curve, and it says something surprising: it is still improving at 32 slots on 8 cores. Which brings us to the measurement that matters.

4. The Heavy Jobs Say the Opposite

The light jobs above are about 370 ms each, and most of that is process startup, dynamic linking and reading the compiled image — not simulation. A job that is 80% waiting is a job where oversubscription fills the gaps, so more slots than cores keeps helping.

Real regression jobs are not like that. This fixture also has a deliberately heavy simulation, around 5.5 seconds of actual compute each:

=== 8 memory-heavy runs, one rep per -P ===
  -P    elapsed    per-job avg  vs -P 8
  1     43847ms    5480ms       -
  4     12620ms    1577ms       -
  8     9796ms     1224ms       1.00x
  16    10561ms    1320ms       1.07x

Here the curve does what the textbook says: it falls to a minimum at the core count and then turns back up. -P 16 is 7% slower than -P 8, on 8 cores, from nothing but oversubscription.

Three numbers to take from this table.

The speedup at the core count is 4.48×, not 8×. 43847 / 9796. Eight slots on eight cores bought four and a half times, because the cores are shared with the operating system, the slots contend for memory bandwidth, and the eight jobs do not finish together.

Beyond the core count it gets worse, not flat. The penalty here is small — 7% — and it grows with the per-job memory footprint, because that is where swapping starts.

And the two tables disagree. The light jobs improve past 8 slots; the heavy jobs degrade past 8 slots. Both measurements are correct. They are measuring different things: one measures process startup, the other measures simulation.

5. The First Measurement I Published Was Backwards

The first version of the sweep in §3 was run once, cold, on a freshly created fixture. It produced this:

  -P 8     1839ms
  -P 32    3052ms       1.66x

— which is the clean, expected, textbook result: oversubscription hurts. It agreed with everything I expected to write, and I nearly wrote it.

Repeating the same sweep on a warm page cache reversed the ordering: -P 32 came in at 1413 ms, 0.82× of -P 8.

The mechanism is Chapter 31.1's cold-start cost. The first sweep ran -P 1 first, which paid the cost of faulting in the simulator binary, the compiled image and every shared library. By the time the sweep reached -P 32, the cache was warm — except that the fixture had been rebuilt between configurations, so the high slot counts were paying it again. The number being compared was not parallelism, it was which configuration happened to run first on a cold filesystem.

6. Exit Status From Many Children

A parallel launcher has to report whether the regression passed, and xargs has one answer for a wide range of outcomes. Every case, measured:

=== xargs exit status, every combination ===
  CASE                                           rc
  1 child, exit 0                                0
  1 child, exit 7 (no -P)                        1
  3 children, all exit 0                         0
  3 children, all exit 7 (no -P)                 1
  3 children exit 7, -P 1                        1
  3 children exit 7, -P 2                        1
  3 children exit 7, -P 4                        1
  mixed 0,7,0 with -P 2                          1
  mixed 0,0,7 with -P 2                          1
  1 child exit 255, -P 2                         1
  missing command, -P 2                          127

And the manual says why:

EXIT STATUS
     The xargs utility exits with a value of 0 if no error occurs.  If utility
     cannot be found, xargs exits with a value of 127, otherwise if utility
     cannot be executed, xargs exits with a value of 126.  If any other error
     occurs, xargs exits with a value of 1.

So the whole vocabulary is three values:

One failing run and two hundred failing runs are both 1. So is a run that exited 3 because its log had no verdict, which Chapter 31.1 deliberately made a distinct status. All of that distinction is discarded at the xargs boundary.

This is not a flaw to work around in xargs; it is a consequence of there being one exit status and many children. The answer is that the per-run status has to be recorded by the run, on disk, where it survives the collapse — which is Chapter 31.7.

What xargs's own status is still good for is the two values that are not about the children at all:

xargs -P 8 -I{} "$BIN/run_one.sh" {} < specs.txt
rc=$?
case $rc in
  126|127) echo "launcher problem: run_one.sh not found or not executable" >&2; exit 2;;
esac
# rc 0 or 1 tells you almost nothing; read the status files instead

127 and 126 mean the regression never ran, which is a completely different report from "some tests failed" — and the one that a script treating any non-zero as "failures" will mislabel. Chapter 30.6 made the same distinction for a submitted job.

7. Sizing the Slot Count

Cores are the first bound and rarely the binding one. The others:

$ sysctl -n hw.ncpu
8

Memory. Slots times peak per-run RSS has to fit, and the per-run figure needs measuring, not guessing:

  the wrapper pid=30293 (/bin/bash)
  its children: 30297
  the vvp pid=30297
  peak RSS of the WRAPPER: 1056 KB (1 MB)
  peak RSS of VVP itself:  8896 KB (8 MB)

Those two lines are a measurement error of 8.4×, and it is the one I made. The sampler watched $! — which is the wrapper script, not the simulator. A wrapper is a few hundred kilobytes of bash forever; the process that actually holds the design is its child. Sizing 8 slots from the wrapper's 1 MB gives a memory budget of 8 MB for something that needs 72.

Two caveats on even the corrected number. It is sampled at 0.1 s intervals, so it is a lower bound on the true peak — a short spike between samples is invisible. And it is this fixture; a real design is three orders of magnitude larger, which is the whole reason the slot count usually comes from memory rather than cores.

Licences. Chapter 25.4 covered diagnosing a licence failure; the scheduling consequence is that if the site has 4 simulator licences, 8 slots means 4 runs and 4 failures, and the failures look like tool errors rather than a scheduling mistake.

And disk. Chapter 28.2 measured what one run leaves behind; eight at once produce it eight times as fast, and a waveform-enabled regression can fill a filesystem in the middle of the night.

So:

# slots: the smallest of the four bounds, overridable
cores=$(sysctl -n hw.ncpu 2>/dev/null || echo 4)
slots=${SLOTS:-$cores}
[ "$slots" -lt 1 ] && slots=1
echo "regression: $slots slots on $cores cores" >&2
xargs -P "$slots" -I{} "$BIN/run_one.sh" {} < specs.txt

sysctl -n hw.ncpu is the macOS spelling. nproc is not installed here, and getconf _NPROCESSORS_ONLN is the more portable fallback — hence the || echo 4, which keeps the script running on a machine where neither answers.

8. The Whole Launcher

#!/bin/bash
# regress SPECS -- run a matrix with a bounded number of slots.
set -u
specs=${1:-}
[ -r "$specs" ] || { echo "regress: cannot read '$specs'" >&2; exit 2; }
[ -n "${ROOT:-}" ] || { echo "regress: set ROOT" >&2; exit 2; }
[ -n "${RUNS:-}" ] || { echo "regress: set RUNS" >&2; exit 2; }

cores=$(sysctl -n hw.ncpu 2>/dev/null || getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)
slots=${SLOTS:-$cores}
n=$(grep -cv '^[[:space:]]*\(#.*\)\{0,1\}$' "$specs")
[ "$n" -gt 0 ] || { echo "regress: '$specs' has no runs" >&2; exit 2; }

echo "regress: $n runs, $slots slots on $cores cores" >&2
"$ROOT/bin/run_one.sh" smoke:0 >/dev/null 2>&1    # warm-up, status ignored
s=$(date +%s)
xargs -P "$slots" -I{} "$ROOT/bin/run_one.sh" {} < "$specs"
rc=$?
e=$(date +%s)
echo "regress: finished in $((e-s))s, xargs rc=$rc" >&2
case $rc in
  126|127) echo "regress: run_one.sh not found or not executable -- nothing ran" >&2; exit 2;;
esac
exit 0      # the verdict comes from the status files, not from here

The last line is deliberate and it is the hinge into the next chapter. This launcher does not report whether the regression passed, because xargs cannot tell it — and a launcher that exits 1 for "something failed" invites a caller to treat 1 as the whole answer.

Practice

1. Fork 240 background jobs on a toy command and watch the load average.
2. Write the batching wait loop and time it with one deliberately slow job among fast ones.
3. Run wait -n on this machine and read the exit status.
4. Measure a warm single run three times and compute the sequential cost of 16.
5. Sweep -P with three repetitions and print the raw values next to the median.
6. Find a row whose spread exceeds the difference between rows and say what the table can still support.
7. Sweep a heavy job and find where the curve turns up.
8. Compute the speedup at the core count and explain why it is not the core count.
9. Run time on one job and compute the compute fraction.
10. Sweep once cold and once warm and see whether your conclusion changes.
11. Tabulate xargs's exit status for all-pass, one-fail, all-fail, missing and non-executable.
12. Handle 126 and 127 specifically and say what report they deserve.
13. Sample the RSS of $! and then of its child, and compare.
14. Compute your memory bound from the corrected figure.
15. Write the launcher, with a warm-up, and have it exit 0 whatever the children did.
16. Explain to someone else why that exit 0 is right.

Check Your Understanding

1. What does an unbounded fork of 240 runs cost?
2. Why is the batching wait loop worse than it looks?
3. What does wait -n do on this machine, and what is the danger?
4. In §3's table, which row is impossible and how do you know?
5. What is the honest conclusion from that table?
6. What two claims does it still support?
7. Why do the light jobs keep improving past 8 slots?
8. What do the heavy jobs do past 8 slots, and by how much here?
9. Why is the speedup at the core count 4.48× and not 8×?
10. How do you tell which table describes your regression?
11. Why is the cost of too many slots worse than too few?
12. What made the first, cold sweep wrong?
13. Give the three rules for a timing claim.
14. Why is a confirming measurement suspicious?
15. What are the only four values xargs can exit with, and what does 1 cover?
16. Which two of them are worth branching on, and why?
17. What was the 8.4× error in the RSS measurement?
18. Name the four bounds on the slot count.
19. Why does the launcher exit 0 even when runs failed?

9. Remember

• Bound the parallelism. An unbounded fork of a real matrix is a machine that stops responding.
• Plain wait batches, so the loop waits for each batch's slowest member. wait -n does not exist in bash 3.2 — it exits 2 and the loop runs unbounded.
• xargs -P maintains the slot count itself, in one process, with no version dependency.
• Look at the spread before the median. A 2.7× spread inside one configuration makes the whole normalised column meaningless, repetitions and all.
• Light jobs and heavy jobs give opposite answers. Startup-dominated work keeps improving past the core count; real simulation turns up at it.
• Measure the compute fraction with time to know which regime you are in. Default to bounding at the core count: the cost of too many slots is asymmetric.
• 8 slots on 8 cores bought 4.48×, not 8×.
• Warm up before the first measured configuration, or the table measures the page cache.
• Distrust a measurement that confirms what you were about to write. The cold sweep gave the textbook answer and was wrong.
• xargs exits 0, 1, 126 or 127. 1 means "at least one child failed" and nothing more; branch on 126 and 127, which mean nothing ran.
• Size slots from the simulator's RSS, not the wrapper's — that mistake was 8.4× here — and from licences and disk as well as cores.
• A parallel launcher should not report the verdict. It cannot know it.

10. What Is Next

Eight runs are now in flight at once, writing into whatever directories and files they were told to. Chapter 31.6 measures what happens when two of them are told the same name — including four runs that produced one log, and two passing tests recorded as failures.