Linux & Bash · Part II · Module 30 · Running Work on Shared Engineering Machines

Submitting and Tracking Scheduled Jobs

Submission is one line and everything interesting happens afterwards: the job is refused, or it waits forever, or it starts and is thrown off a machine for using more than it asked for. This chapter produces all four outcomes and reads each one — including a resource cap that turned out to be twice as generous as it was declared.

Advanced22 min readLinuxPart IISchedulersResourcesMonitoring

Part II · Module 30 · Chapter 30.6 · Submitting and Tracking Scheduled Jobs

Chapter 30.5 established what submission changes about your script and how to test it without a scheduler. This chapter is about the submission itself, and about the four things that can happen next.

1. The Submission Line

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# FORMS, from each vendor's documented interface -- see 30.5's note
bsub   -q normal -o run.log -n 4 -R "rusage[mem=8000]" -W 240 ./run_sim.sh
qsub   -q normal -o run.log -pe smp 4 -l h_vmem=8G -l h_rt=4:00:00 ./run_sim.sh
sbatch --partition=normal --output=run.log --cpus-per-task=4 --mem=8G --time=4:00:00 ./run_sim.sh

Four requests in every one of them, and each has a cost if you get it wrong:

request	too small	too large
queue	rejected, or the wrong limits	a long job in a short queue is killed
cores	the tool runs single-threaded	the job waits for cores it will not use
memory	evicted mid-run	the job waits for memory it will not use
wall-clock	killed at the limit, mid-run	lower priority, longer wait

The asymmetry is the point. Under-requesting kills a job that was working; over-requesting makes it wait. Which is why the useful habit is to measure what a job actually uses — Chapter 30.3's rss maximum — and request a margin above it, rather than guessing in either direction.

2. A Queue You Can Observe

There is no scheduler here, so the four outcomes are produced by a stand-in that reproduces the parts affecting your script: the queue wait, the clean environment, a working directory of its own choosing, output to a file, a resource cap, and a job record.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# qsub-stub — a queue stand-in. There is no scheduler here, so this reproduces
# the parts that affect your script: a clean environment, a directory of the
# queue's choosing, output to a file, a resource cap, and a job record.
# Usage: qsub-stub -q QUEUE [-o OUT] [-f MAXKB] -- CMD...
set -u
queue=normal out="" maxkb=0
while [ $# -gt 0 ]; do
  case "$1" in
    (-q) queue=${2:?-q needs a queue}; shift 2 ;;
    (-o) out=${2:?-o needs a path};    shift 2 ;;
    (-f) maxkb=${2:?-f needs a size};  shift 2 ;;
    (--) shift; break ;;
    (*)  echo "qsub-stub: unknown option '$1'" >&2; exit 2 ;;
  esac
done
[ $# -gt 0 ] || { echo "qsub-stub: no command" >&2; exit 2; }

# a queue the site does not have is a REJECTION, before anything is spent
case "$queue" in
  (normal|long|debug) : ;;
  (*) echo "qsub-stub: job rejected: no such queue '$queue'" >&2; exit 3 ;;
esac

q=${QDIR:?qsub-stub: set QDIR to the queue area}
[ -d "$q" ] || { echo "qsub-stub: no such queue area: $q" >&2; exit 2; }
id=$(( $(cat "$q/next" 2>/dev/null || echo 1000) + 1 ))
printf '%s\n' "$id" > "$q/next"
jd="$q/$id"; mkdir -p "$jd" || exit 2
: "${out:=$jd/job.out}"

{ printf 'id       %s\n' "$id"
  printf 'queue    %s\n' "$queue"
  printf 'user     %s\n' "$(id -un)"
  printf 'host     %s\n' "$(hostname -s)"
  printf 'submitted %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
  printf 'subdir   %s\n' "$PWD"
  printf 'maxkb    %s\n' "$maxkb"
  printf 'cmd     '; printf ' %q' "$@"; printf '\n'
} > "$jd/job.info"
: > "$jd/cmd.args"; for a in "$@"; do printf '%s\n' "$a" >> "$jd/cmd.args"; done
printf 'PEND\n' > "$jd/state"

printf 'Job <%s> is submitted to queue <%s>.\n' "$id" "$queue"

(
  sleep 1                                     # the queue wait
  printf 'RUN\n' > "$jd/state"
  printf 'started  %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" >> "$jd/job.info"
  # the execution environment: clean, in a directory of the queue's choosing
  wd="$jd/work"; mkdir -p "$wd"
  args=(); while IFS= read -r a; do args+=("$a"); done < "$jd/cmd.args"
  env -i HOME="$HOME" USER="$(id -un)" PATH=/usr/bin:/bin SHELL=/bin/sh TMPDIR=/tmp \
      bash -c "cd '$wd' && { [ '$maxkb' -gt 0 ] && ulimit -f $maxkb; } ; exec \"\$@\"" \
      bash "${args[@]}" > "$out" 2>&1
  rc=$?
  printf 'exit %d\n' "$rc" > "$jd/status"
  printf 'finished %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" >> "$jd/job.info"
  case "$rc" in
    (0)   printf 'DONE\n' > "$jd/state" ;;
    (153) printf 'EVICTED\n' > "$jd/state" ;;   # SIGXFSZ: exceeded the file cap
    (*)   printf 'EXIT\n' > "$jd/state" ;;
  esac
) > /dev/null 2>&1 &
disown 2>/dev/null || true

and something to read it back with:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# qstat-stub [ID] — what the queue knows about your jobs.
set -u
q=${QDIR:?qstat-stub: set QDIR}
[ -d "$q" ] || { echo "qstat-stub: no such queue area: $q" >&2; exit 2; }
printf '%-7s %-9s %-9s %-7s %s\n' JOBID STATE QUEUE EXIT COMMAND
n=0
for jd in "$q"/*/; do
  [ -d "$jd" ] || continue
  id=$(basename "$jd")
  [ $# -gt 0 ] && [ "$id" != "$1" ] && continue
  n=$((n+1))
  printf '%-7s %-9s %-9s %-7s %s\n' "$id" \
    "$(cat "$jd/state" 2>/dev/null || echo '?')" \
    "$(sed -n 's/^queue *//p' "$jd/job.info" 2>/dev/null)" \
    "$(sed -n 's/^exit //p' "$jd/status" 2>/dev/null || echo '-')" \
    "$(sed -n 's/^cmd  *//p' "$jd/job.info" 2>/dev/null | cut -c1-40)"
done
[ "$n" = 0 ] && { printf 'no jobs found\n'; exit 1; }
exit 0

3. The Four Outcomes

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ QDIR=$PWD/queue qsub-stub -q normal -- "$PWD/bin/run_sim.sh"
Job <1001> is submitted to queue <normal>.
$ QDIR=$PWD/queue qsub-stub -q normal -- /bin/echo hello
Job <1002> is submitted to queue <normal>.
$ QDIR=$PWD/queue qsub-stub -q bogus -- /bin/true
qsub-stub: job rejected: no such queue 'bogus'
$ echo $?
3

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ QDIR=$PWD/queue qstat-stub
JOBID   STATE     QUEUE     EXIT    COMMAND
1001    EXIT      normal    1       /…/m30/bin/run_sim.sh
1002    DONE      normal    0       /bin/echo hello
1003    EXIT      normal    127     vvp /…/m30/logs/dumphang.vvp
1004    EVICTED   normal    153     /tmp/writer.sh

Five states, each with a different meaning and a different response.

REJECTED, before the job exists. A queue that is not configured, a resource request above the queue's maximum, a missing permission. The important property is that nothing has been spent — no wait, no slot, no partial output. A rejection is the cheapest failure available and the only one that is pure good news, because you find out in a second rather than in four hours.

PEND — queued. Observed in the stand-in at t=1s before it moved to RUN. On a real cluster this is where a job can sit indefinitely, and §5 is about telling "waiting its turn" from "will never run".

EXIT with 1 — job 1001. That is Chapter 30.5 §2's measurement arriving through the queue: the clean environment had no PROJ, so the script refused to start. Nothing about the scheduler is wrong; the script was not ready to be submitted.

EXIT with 127 — job 1003. The minimal PATH did not contain the simulator. Same cause, different symptom, and 127 is specifically "command not found" — which is worth recognising on sight because it means the environment, never the design.

EVICTED with 153 — job 1004. The job exceeded a resource it had declared, and §4 is about that number.

4. The Resource Cap, and the Unit That Was Not What It Said

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ QDIR=$PWD/queue qsub-stub -q normal -f 50 -- /tmp/writer.sh
Job <1005> is submitted to queue <normal>.
$ sleep 3; printf 'state=%s status=%s output=%s bytes (cap 50 KB)\n' \
>   "$(cat queue/1005/state)" "$(cat queue/1005/status)" "$(wc -c < queue/1005/job.out | tr -d ' ')"
state=EVICTED status=exit 153 output=51200 bytes (cap 50 KB)

51,200 bytes for a 50 KB cap — exact. And the status decodes:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ echo $((153 - 128))
25
$ kill -l 25
XFSZ

153 is 128 + 25, the shell's convention for "terminated by signal 25", and signal 25 here is SIGXFSZ — file size limit exceeded. A job killed by its own declared limit reports it as a signal, not as an error from the tool, which is why the tool's log usually ends mid-sentence with nothing explaining why.

The cap was twice what it was declared, because of a unit

The first version of the stand-in asked for ulimit -f $(( maxkb * 2 )), on the standard understanding that ulimit -f counts 512-byte blocks. ulimit -a encourages it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ulimit -a | grep 'file size'
file size               (blocks, -f) unlimited

"blocks" with no size given. Measured:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ for n in 1 10 100; do
>   ( ulimit -f $n; /bin/dd if=/dev/zero bs=1024 count=10000 2>/dev/null > /tmp/cap.out )
>   printf '  ulimit -f %-5s -> wrote %-9s bytes = %s x %s\n' "$n" "$(wc -c < /tmp/cap.out | tr -d ' ')" "$n" "$(( $(wc -c < /tmp/cap.out) / n ))"
>   rm -f /tmp/cap.out
> done
  ulimit -f 1     -> wrote 1024      bytes = 1 x 1024
  ulimit -f 10    -> wrote 10240     bytes = 10 x 1024
  ulimit -f 100   -> wrote 102400    bytes = 100 x 1024

1024 bytes per block, not 512. So the doubling made a 50 KB request into a 100 KB cap — a job allowed twice what it had asked for, which on a shared filesystem is exactly the error that fills a disk nobody budgeted for.

Two general points, and the second is the one worth keeping.

A unit labelled "blocks" is not a unit. ulimit, du, df and stat all use the word for different sizes on the same machine — Chapter 28.3 measured du reporting 512-byte blocks by default while stat -f '%b' counts 512s and the filesystem allocates 4096s.

And the direction of the error is the dangerous one. A cap that is too tight evicts a job and you find out immediately. A cap that is too loose silently permits what you were trying to prevent, and the only evidence is a full filesystem weeks later. Verify a limit by exceeding it on purpose, which is three lines and the only way to know which unit you got.

And one limit this platform does not have at all:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ( ulimit -v 65536 )
bash: ulimit: virtual memory: cannot modify limit: Invalid argument

So an address-space cap — the thing a scheduler's mem request usually becomes — cannot be demonstrated or enforced here. A memory limit on this machine is advisory; on a real cluster it is not, and a job exceeding it is killed in a way that looks exactly like job 1004.

5. Queued Forever, or Waiting Its Turn

The state that needs most interpretation is PEND, because it has two completely different causes and they look identical.

Waiting its turn is normal. Other people's jobs are ahead of you, the queue is busy, your priority is what it is.

Never going to run is a request that cannot be satisfied: more cores than any node has, more memory than the queue permits, a combination no machine matches. A scheduler will usually say so if asked, and the asking is the skill:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# FORMS -- the "why is this pending" question, per vendor
bjobs -l 12345          # LSF: the PENDING REASONS block
qstat -j 12345          # SGE/UGE: the "scheduling info" lines
squeue -j 12345 -o '%r' # Slurm: the Reason field
scontrol show job 12345 # Slurm: the full record

What they report falls into three groups, and the group tells you what to do:

the reason says	what it means	what to do
resources busy, hosts occupied, priority	waiting its turn	wait, or request less
no host matches, resource not available	unsatisfiable	fix the request
licence not available, dependency not met	blocked on something external	fix the dependency

The second row is the one to recognise quickly, because waiting on it is free of any possibility of success. And the practical move for all three is the same one that applies without any scheduler: ask for what you measured. A job requesting 64 GB because somebody once saw a big number pends behind every job that asked for 8.

Which of this chapter's claims are measured

Measured here, in the stand-in and on this machine: the five states and their transitions; a rejection costing nothing; exit 1 and exit 127 from a clean environment and a minimal PATH; eviction at exactly the declared cap, reported as 128 + SIGXFSZ; ulimit -f counting 1024-byte blocks; and ulimit -v being unavailable.

Documented, not measured: real pending reasons and their wording; priority and fair-share ordering; preemption of a running job; per-queue wall-clock enforcement; array jobs and dependencies; and anything about how long a real queue takes.

The split matters because it tells you who to ask. Chapter 30.5 §6 made the same division for script failures; this is its operational half: a job that fails the same way under the stand-in is your script's problem, and a job that only misbehaves on the real cluster is a question for the people who run it — asked with the job id, the submission line and the pending reason, which is a conversation that takes one message instead of five.

6. Tracking a Job to Completion

The loop that follows a job is short, and the care is all in the terminal conditions:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# follow JOBID — wait for a job to reach a terminal state, then report.
set -u
id=${1:?usage: follow JOBID}
q=${QDIR:?follow: set QDIR}
jd="$q/$id"
[ -d "$jd" ] || { echo "follow: no such job: $id" >&2; exit 2; }

poll=${2:-2}
last=""
while :; do
  st=$(cat "$jd/state" 2>/dev/null || echo UNKNOWN)
  [ "$st" != "$last" ] && { printf '%s  %s\n' "$(date -u '+%H:%M:%SZ')" "$st"; last=$st; }
  case "$st" in
    (DONE|EXIT|EVICTED) break ;;
    (UNKNOWN) echo "follow: job $id has no state -- the queue lost it" >&2; exit 1 ;;
  esac
  sleep "$poll"
done

rc=$(sed -n 's/^exit //p' "$jd/status" 2>/dev/null)
printf '\n-- job %s finished in state %s, exit %s --\n' "$id" "$st" "${rc:-?}"

# a terminal state is not a verdict: Module 29's rule, one more time
out=$(sed -n 's/^cmd  *//p' "$jd/job.info" | head -1)
printf 'command: %s\n' "$out"
if [ ! -s "$jd/job.out" ]; then
  printf 'no output at all -- see Chapter 29.5\n'
elif grep -qI . "$jd/job.out" 2>/dev/null; then
  printf 'last 5 lines of output:\n'; tail -5 "$jd/job.out" | sed 's/^/  /'
else
  # a job's output is not necessarily text, and BSD sed ABORTS on NUL bytes
  printf 'output is %s bytes of binary data -- not shown\n' "$(wc -c < "$jd/job.out" | tr -d ' ')"
fi

case "$st" in
  (DONE)    exit 0 ;;
  (EVICTED) printf 'evicted: it exceeded a resource it declared. Raise the request or reduce the use.\n' >&2; exit 1 ;;
  (*)       exit 1 ;;
esac

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ QDIR=$PWD/queue follow 1005
07:33:45Z  EVICTED

-- job 1005 finished in state EVICTED, exit 153 --
command: /tmp/writer.sh
output is 51200 bytes of binary data -- not shown
evicted: it exceeded a resource it declared. Raise the request or reduce the use.
$ echo $?
1

Following a job from submission shows the transitions:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ QDIR=$PWD/queue qsub-stub -q normal -- /bin/echo hello
Job <1006> is submitted to queue <normal>.
$ QDIR=$PWD/queue follow 1006 1
07:33:03Z  PEND
07:33:04Z  RUN
07:33:05Z  DONE

-- job 1006 finished in state DONE, exit 0 --
command: /bin/echo hello
last 5 lines of output:
  hello
$ echo $?
0

And the two failure modes of the tracker itself:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ QDIR=$PWD/queue follow 9999        # a job directory with no state file
07:33:05Z  UNKNOWN
follow: job 9999 has no state -- the queue lost it
$ echo $?
1
$ QDIR=$PWD/queue follow 7777        # no such job
follow: no such job: 7777
$ echo $?
2

A job's output is not necessarily text, and BSD sed does not error on that — it aborts

The binary-output guard in that script is there because the first version did not have one:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

last 5 lines of output:
Assertion failed: (advance > 0), function substitute, file process.c, line 462.
bin/follow: line 31:  3168 Done   tail -5 "$jd/job.out"
      3169 Abort trap: 6          | sed 's/^/  /'

Job 1005's output is 51,200 bytes of NUL from dd if=/dev/zero, and piping that through sed produced an assertion failure and SIGABRT — not a diagnostic, a crash.

Each tool handles it differently, which is worth knowing before a log-reading script meets a core dump, a waveform written to stdout, or a tool that emits a progress bar with control characters:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ tail -5 job.out | sed 's/^/x/'
Assertion failed: (advance > 0), function substitute, file process.c, line 462.
$ tail -5 job.out | cat | head -c 16 | od -An -c
          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
$ tail -5 job.out | tr -d '\0' | sed 's/^/x/'
                                     (empty, and no crash)
$ tail -5 job.out | cat -v | head -c 30
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

And the clean way to decide in advance is grep's own text test:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ grep -qI . queue/1005/job.out && echo text || echo BINARY
BINARY
$ grep -qI . queue/1006/job.out && echo text || echo BINARY
text

-I means "treat a binary file as non-matching", so grep -qI . succeeds only for a file grep considers text. That is one call, it is portable, and it turns a crash into a line of output — which matters most in exactly the place this happened: an unattended tracker reading the output of a job that failed in a way nobody anticipated.

Four decisions.

**It prints a line only when the state changes.**Four decisions.

It prints a line only when the state changes. A poll loop that prints every tick produces a log nobody reads; one that prints transitions produces a timeline.

UNKNOWN is a failure, not a state to wait in. A job directory with no state file means the queue lost track of it, and waiting forever for a state that will never be written is the commonest defect in a tracking script.

The exit status and the output are both reported. Module 29 in one line: a terminal state is not a verdict, and DONE with exit 0 still needs the log read to know whether the simulation passed.

EVICTED gets its own message and its own advice, because it is the one outcome where the fix is a number in the submission line rather than anything in the design.

7. What Module 30 Leaves You

The module began with one job you started yourself and ends with a queue you do not control, and the same three facts held at every step.

The exit status is not the verdict. A failing simulation exits 0 (Module 27), a 256-error compile exits 0 (Chapter 29.2), a run that exhausts its events exits 0 (Chapter 29.5), and a job that reaches DONE has still only told you it finished.

Relative paths move when the process does. Five chapters measured it, the last being a scheduler's choice of directory.

And the tools do not do what the folklore says. nohup did not protect the simulator, wait returned 0 for failed jobs, log growth was not a progress signal, kill -9 cost 88 lines of evidence, a stale artifact reported success, and ulimit -f counted 1024-byte blocks. Every one of those was found by running it rather than by reading about it, which is the only method this module recommends.

What remains is the part no amount of shell can supply: a regression that submits two thousand jobs, decides each verdict, collects the results, and returns a status something else can gate on. Module 32 is that, and Module 31 is the batch mechanics it needs first — test lists as data, seeds, a run matrix, and the bounded parallelism that Chapter 30.2 could not express without wait -n.

Practice

Write out the four resource requests for your own flow, with the cost of getting each one too small and too large.
Measure a job's actual peak memory with Chapter 30.3's watcher and compare it with what you have been requesting.
Submit to a queue that does not exist and confirm the rejection costs nothing.
Observe a job in PEND before it runs.
Submit a script that depends on a profile variable and read the exit status.
Submit one whose tool is not on the minimal PATH and recognise 127 on sight.
Cap a job's output below what it writes and record the state and status.
Decode the status into 128 + signal and name the signal.
Measure what unit ulimit -f counts with three different values.
Work out what a 512-byte assumption would have done to a declared cap.
Try ulimit -v on your machine and on your compute server.
Ask a real scheduler why a job is pending and classify the reason into one of the three groups.
Write follow and confirm it reports transitions rather than every poll.
Delete a job's state file while follow is watching, and confirm it fails rather than waiting.
Follow a job to DONE with exit 0 and then read its log to find out whether the simulation actually passed.

Check Your Understanding

What are the four resource requests, and why is the error asymmetric?
What is the right way to choose a memory request?
Which outcome is the only pure good news, and why?
What are the two causes of PEND, and why do they look the same?
What did exit 1 mean for job 1001, and exit 127 for job 1003?
What does 153 decode to, and which signal is it?
Why does a job killed by its own limit usually have a log that ends mid-sentence?
How many bytes does ulimit -f 100 allow here, and what does that make a block?
What did the 512-byte assumption do to a 50 KB request?
Which direction of a wrong cap is dangerous, and why?
What is the only way to know which unit a limit uses?
What happens with ulimit -v here, and what does that mean for memory limits?
Which of this chapter's claims are measured and which are documentation?
Why does follow treat UNKNOWN as a failure?
Why is DONE with exit 0 still not a verdict?

Answers

Queue, cores, memory and wall-clock. Under-requesting kills a job that was working; over-requesting only makes it wait — so the two errors cost completely different amounts.
Measure the actual peak rss and request a margin above it, rather than guessing in either direction.
A rejection. Nothing has been spent — no wait, no slot, no partial output — and you find out in a second instead of four hours.
Waiting its turn, and a request that can never be satisfied. Both show the same state, so the pending reason has to be asked for.
The clean environment had no PROJ, so the script refused to start; and the minimal PATH did not contain the simulator, which is specifically what 127 means.
128 + 25, and kill -l 25 is XFSZ — file size limit exceeded.
Because the kill arrives as a signal rather than as an error the tool can report, so the tool never gets to explain itself.
102,400 bytes, which makes a block 1024 bytes — not the 512 that ulimit -a's "(blocks)" label implies.
It doubled it: a 50 KB request became a 100 KB cap, letting a job use twice what it asked for.
Too loose. A tight cap evicts the job and you find out immediately; a loose one silently permits what you were preventing, and the evidence is a full filesystem weeks later.
Exceed it on purpose. Three lines, and the only way to be sure.
cannot modify limit: Invalid argument — it is unavailable, so a memory limit here is advisory. On a real cluster it is enforced and looks like job 1004.
Measured: the five states, a costless rejection, exit 1 and 127 from the environment, eviction at the exact cap as 128 + SIGXFSZ, ulimit -f counting 1024s, and ulimit -v being absent. Documented: real pending reasons, priority and fair-share, preemption, wall-clock enforcement, array jobs and dependencies.
Because a job directory with no state file means the queue lost it, and waiting for a state that will never be written is the commonest defect in a tracking script.
Because a terminal state only says the job finished. A failing simulation exits 0, so the log still has to be read.

8. Remember

Four requests — queue, cores, memory, wall-clock — and the error is asymmetric. Too small kills a working job; too large only makes it wait.
Request what you measured, from a peak rss, with a margin.
A rejection costs nothing and is the only outcome that is pure good news.
PEND has two causes that look identical: waiting its turn, and unsatisfiable. Ask for the reason.
Exit 1 and exit 127 from a submitted job are environment failures, not design failures — 127 specifically means the tool was not on PATH.
Eviction reports as 128 + SIGXFSZ = 153, and the tool's log ends mid-sentence because a signal gives it nothing to say.
ulimit -f counts 1024-byte blocks here, not 512. The 512 assumption made a 50 KB request into a 100 KB cap.
A too-loose cap is the dangerous direction, and exceeding a limit on purpose is the only way to learn its unit.
ulimit -v does not work here, so memory limits are advisory on this machine and enforced on a cluster.
Report transitions, not polls; treat a missing state as a failure; and read the log even on DONE.
A job that fails under the stand-in is yours to fix. One that only fails on the cluster is a question for the cluster team — with the job id, the submission line and the pending reason.

9. What Is Next

Module 30 leaves one job submitted, tracked and understood. Module 31 turns one invocation into many — test lists as data, explicit seeds, a run matrix you can see before launching it, and the bounded parallelism this module could not express — and Module 32 assembles the whole regression on top of it.

Where this fits

Part of the Linux & Bash curriculum.