Job Schedulers: the Queue Model

Part II · Module 30 · Chapter 30.5 · Job Schedulers: the Queue Model

Everything so far in this module runs the job yourself, on the machine in front of you. A scheduler does it for you, somewhere else, and the universal first experience is that the command which worked at your prompt does not work when submitted.

1. The Queue Model

Submission replaces running a command with describing a job:

# FORMS, from the documented interfaces -- not run here
bsub -q normal -o run.log -n 4 -R "rusage[mem=8000]" ./run_sim.sh
qsub -q normal -o run.log -pe smp 4 -l h_vmem=8G ./run_sim.sh
sbatch --partition=normal --output=run.log --cpus-per-task=4 --mem=8G ./run_sim.sh

Three vendors, one shape: a queue, somewhere for the output, a resource request, and the command. And four things change the moment you use any of them.

Somebody else decides when. Your job enters a queue and runs when the scheduler chooses. Everything about elapsed time from Chapter 30.3 now includes an unknown waiting period.

Somebody else decides where. A machine you did not pick, which may not have your tools, your filesystems, or your architecture.

The environment is not yours. §2.

The working directory may not be yours. §3.

The first two are a scheduler's job and you adapt to them. The last two are the ones that break scripts, and they are measurable here.

2. The Environment Is Not Yours

A representative flow script — it needs a project variable and it needs the tool on PATH:

#!/bin/bash
# a typical flow script: depends on PROJ, on PATH, and on the cwd
set -u
: "${PROJ:?run_sim.sh: PROJ is not set}"
echo "PROJ=$PROJ"
echo "cwd=$PWD"
echo "which iverilog: $(command -v iverilog || echo NOT-FOUND)"
iverilog -o sim.vvp "$PROJ/src_dump.sv" 2>&1 | head -2
[ -s sim.vvp ] && echo "built sim.vvp" || echo "no artifact"

At your own prompt it works:

$ PROJ="$PWD" bin/run_sim.sh
PROJ=/…/m30
cwd=/…/m30
which iverilog: /opt/homebrew/bin/iverilog
built sim.vvp

Now the environment a scheduler hands a job — env -i keeps nothing, and a minimal PATH is supplied the way a batch system does:

$ env -i HOME="$HOME" PATH=/usr/bin:/bin bash -c "cd $PWD && bin/run_sim.sh"
bin/run_sim.sh: line 4: PROJ: run_sim.sh: PROJ is not set

PROJ was set by your shell profile, which never ran. And with PROJ supplied so the script gets further, the second cause appears:

$ env -i HOME="$HOME" PATH=/usr/bin:/bin PROJ="$PWD" bash -c "cd $PWD && bin/run_sim.sh"
PROJ=/…/m30
cwd=/…/m30
which iverilog: NOT-FOUND
bin/run_sim.sh: line 8: iverilog: command not found
built sim.vvp

Read the last two lines together.

3. The Working Directory Is Not Yours

With the environment intact and only the directory changed:

$ ( cd / && PROJ="$SC/m30" "$SC/m30/bin/run_sim.sh" )
PROJ=/…/m30
cwd=/
which iverilog: /opt/homebrew/bin/iverilog
sim.vvp: Read-only file system
error: Code generator failure: -1
no artifact

PROJ is fine and the tool is found. The script failed because -o sim.vvp is a relative path, and relative to / it is not writable.

This is the fifth appearance of the same mechanism: Chapter 26.6 for filelist sources, Chapter 27.8 for a compile in the wrong directory, Chapter 28.5 for recorded argv, Chapter 30.1 for a launcher's cd, and now a scheduler's choice of directory. The mechanism never changes: a relative path means something different depending on where the process is standing, and nothing warns you when it moves.

A scheduler's default is vendor-specific — some start in your submission directory, some in your home, some in a scratch area they created — and the only portable assumption is that you do not know. So a submitted script states its own directory:

# the first two lines of anything that will be submitted
cd "${PROJ:?PROJ is not set}" || exit 2

Or, better, uses no relative paths at all:

out="${RUNDIR:?RUNDIR is not set}/sim.vvp"
iverilog -o "$out" "$PROJ/src_dump.sv"

4. Why a Login Shell Is Not the Answer

The obvious fix for §2 is to make the job run a login shell so your profile executes. It helps and it is not equivalent:

$ printf 'interactive PATH entries: %s\n' "$(printf '%s' "$PATH" | tr ':' '\n' | grep -c .)"
interactive PATH entries: 15
$ env -i HOME="$HOME" bash -lc 'printf "login PATH has %s entries\n" "$(printf %s "$PATH" | tr : "\n" | grep -c .)"'
login PATH has 11 entries

Fifteen against eleven. The four missing entries were added by something that runs for an interactive shell and not for a login shell — a tool-setup script sourced from an interactive rc file, a directory manager, a per-project activation.

Which is Chapter 25.6's point arriving from the other side: an environment that was assembled by a chain of sourced files is not reproducible by running one of them. The reliable approach is the one that chapter arrived at — have the job set up its own environment explicitly, from a file that is version-controlled and that a human can read:

# the first thing a submitted script does
set -u
: "${PROJ:?PROJ is not set}"
. "$PROJ/setup/tools.sh"          # explicit, committed, idempotent
command -v iverilog >/dev/null || { echo "iverilog not on PATH after setup" >&2; exit 2; }

The last line matters more than it looks. Chapter 25.1 measured a tool check that reported success with a library missing; this is the cheap version of the same idea — assert the environment is right before doing any work, because a failure here is one line and a failure forty minutes in is a wasted slot.

5. Testing Submission Without Submitting

All of §2, §3 and §4 collapse into one reusable command:

#!/bin/bash
# as-submitted -- CMD...   Run CMD the way a scheduler would: clean environment,
# a directory it chose, output to a file. No scheduler required.
set -u
[ "${1:-}" = "--" ] && shift
[ $# -gt 0 ] || { echo "usage: as-submitted -- CMD..." >&2; exit 2; }

sandbox=$(mktemp -d -t submitted.XXXXXX) || exit 2
log="$sandbox/job.out"

# the minimal environment a batch system typically provides, and nothing else
env -i \
  HOME="$HOME" \
  USER="$(id -un)" \
  PATH=/usr/bin:/bin \
  SHELL=/bin/sh \
  TMPDIR=/tmp \
  bash -c "cd '$sandbox' && \"\$@\"" bash "$@" > "$log" 2>&1
rc=$?

printf 'as-submitted: exit %d\n' "$rc"
printf 'as-submitted: ran in %s\n' "$sandbox"
# printf takes options, so a format beginning with '-' needs '%s' or '--'
printf '%s\n' '--- output ---'
cat "$log"
printf '%s\n' '--------------'
[ "$rc" -eq 0 ] || printf 'as-submitted: this is what submission will do\n' >&2
exit $rc

$ as-submitted -- /…/m30/bin/run_sim.sh
as-submitted: exit 1
as-submitted: ran in /var/folders/…/submitted.XXXXXX.1BJ8oy28MZ
--- output ---
/…/m30/bin/run_sim.sh: line 4: PROJ: run_sim.sh: PROJ is not set
--------------
as-submitted: this is what submission will do
$ echo $?
1

And the argv survives intact, including an argument with a space, while the working directory is the sandbox:

$ as-submitted -- /tmp/showargv.sh one "two three" four
as-submitted: exit 0
as-submitted: ran in /var/folders/…/submitted.XXXXXX.YHaV4qbnXR
--- output ---
argc=3
  [1] one
  [2] two three
  [3] four
cwd=/var/folders/…/submitted.XXXXXX.YHaV4qbnXR
--------------

Four decisions.

The sandbox is a fresh temporary directoryFour decisions.

The sandbox is a fresh temporary directory, so a relative output path fails the way it will on a compute node rather than quietly landing in your project.

The environment list is explicit and short. HOME, USER, PATH, SHELL, TMPDIR — what a batch system typically sets. Adding more makes the test weaker; the point is for it to fail.

bash -c '…' bash "$@" passes the argv through properly. The "$@" inside the quoted script refers to the positional parameters bash is given after the script, and the extra bash is the $0 placeholder. Getting this wrong is how a test harness silently runs a different command than the one you handed it.

It exits with the command's status and says what the failure means, so it can be a gate in a pre-submission check.

6. What Is Measured Here and What Is Not

Three claims in this chapter are measured, and the rest of a scheduler's behaviour is not reproducible without one. Being explicit about which is which:

Measured. A clean environment breaks a script that depends on a profile-set variable or on a PATH entry; a different working directory breaks relative paths; a login shell does not reproduce an interactive environment; a stale artifact makes a failed build report success; and test -nt is unreliable for freshness.

Documented, not measured. Queue priority and fair-share ordering. Preemption and eviction of a running job. Resource enforcement — a job exceeding its memory request being killed by the scheduler rather than by the kernel. Per-queue limits on wall-clock time. Array jobs and dependencies. Everything about when and whether your job runs.

The practical consequence of that split is a division of labour worth stating: the failures you can reproduce locally are the ones that are your script's fault, and the ones you cannot are the ones to ask the cluster team about. A job that fails identically under as-submitted is a script defect and fixing it needs no cluster access. A job that runs correctly under as-submitted and still fails when submitted is about resources, scheduling or the node — and Chapter 30.6 is about reading what the scheduler says in that case.

Practice

1. Check whether any scheduler command exists on your machine and on your compute server.
2. Write a flow script that depends on a variable and on a PATH entry.
3. Run it interactively and confirm it works.
4. Run it under env -i with a minimal PATH and record the first failure.
5. Supply the variable and record the second failure.
6. Leave a stale artifact in place and watch the script report success with the compiler missing.
7. Add rm -f before the build and confirm the report changes.
8. Try the -nt freshness check on a build that completes in under a second.
9. Compare two files created in the same second with -nt in both directions.
10. Find the sub-second mtime with stat and explain why -nt does not use it.
11. Run the script from a different directory with the environment intact, and read the failure.
12. Count your interactive PATH entries against bash -lc's.
13. Write as-submitted and use it on a script you believe is ready.
14. Add more variables to its environment list and work out why that makes the test worse.
15. Classify three of your own submission failures as reproducible or not.

Check Your Understanding

1. Which scheduler commands exist on this machine?
2. What are the four things submission changes, and which two break scripts?
3. Which two local tools reproduce those two?
4. What was the first failure under a clean environment, and why?
5. With the variable supplied, what was the second failure?
6. Why did the script print built sim.vvp with the compiler missing?
7. How does that sharpen Chapter 29.2's rule?
8. Why is a stale artifact the classic "worked once, failed the second time" cause?
9. Why did the -nt freshness check report a freshly built file as stale?
10. What does stat -f '%Fm' show, and why does it not help?
11. What is the more reliable way to know something is new?
12. What failed when only the working directory changed, and why?
13. How many previous chapters has that same mechanism appeared in?
14. How many PATH entries did an interactive shell have against a login shell?
15. Why does as-submitted use a fresh temporary directory?
16. Why does adding variables to its environment list weaken the test?
17. What is the division of labour between failures you can and cannot reproduce locally?

7. Remember

• No scheduler exists here, so every scheduler command is a form — and the three failures that matter are reproducible anyway.
• Submission changes four things; the environment and the working directory are the two that break scripts, and env -i and cd reproduce both.
• A clean environment loses your profile's variables and PATH entries.
• A stale artifact made a failed build report success. Requiring the artifact is only sound for this run's artifact — rm -f first.
• test -nt is false for equal mtimes, so a freshness check against a timestamp fails on a fast build. Sub-second mtimes exist and test does not use them.
• Deleting first beats timing when a check needs to know something is new.
• A relative output path broke on a read-only directory — the fifth appearance of the same mechanism in this track.
• A login shell is not an interactive shell: 11 PATH entries against 15. Have the job set up its environment explicitly and then assert it.
• Test with as-submitted: a fresh temporary directory, a short explicit environment, and the command's status passed through.
• Adding variables to that environment weakens the test, because the point is for it to fail.
• A failure you can reproduce locally is yours to fix; one you cannot is a question for the cluster team.

8. What Is Next

The script is ready to be submitted and tested as if it had been. Chapter 30.6 submits it — the resource request, following it to completion, and reading the three ways a scheduler can refuse you: rejected, queued forever, or evicted.