Python Part II · Module 26 · Regression Automation

Handing Jobs to a Scheduler

Four hundred simulations do not belong on one workstation, and the moment they move to a compute farm the whole shape of this module inverts. Python stops being the thing that runs the work and becomes the thing that describes it, hands it over, and finds out later what happened — and every habit built up over the last four modules about return codes and process status now applies to the wrong process. The command that submits a job succeeds when the job is queued, which is a long way from succeeding.

Advanced19 min readPythonPart IIAutomationRegressionVerification Flow

Part II · Module 26 · Chapter 26.9 · Handing Jobs to a Scheduler

1. Python Stops Owning the CPU

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

BEFORE = [
    ("starts the simulator", "subprocess.run blocks until it exits"),
    ("knows the return code", "directly, from the process it started"),
    ("enforces the timeout", "timeout= kills its own child"),
    ("knows when it finished", "when run() returned"),
]
AFTER = [
    ("starts a submission", "bsub returns in milliseconds"),
    ("knows the submission's code", "which says the job was queued"),
    ("asks for a limit", "the scheduler enforces it, not Python"),
    ("finds out later", "by polling, or by looking for a record"),
]

print("owning the CPU (Modules 23–25):")
for what, how in BEFORE:
    print(f"   {what:<28} {how}")
print()
print("handing it over (this chapter):")
for what, how in AFTER:
    print(f"   {what:<28} {how}")
print()
print("what Python still owns:", "the job list, the identities, the reconciliation")
print("what it no longer owns :", "the process, the clock, the machine")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

owning the CPU (Modules 23–25):
   starts the simulator         subprocess.run blocks until it exits
   knows the return code        directly, from the process it started
   enforces the timeout         timeout= kills its own child
   knows when it finished       when run() returned

handing it over (this chapter):
   starts a submission          bsub returns in milliseconds
   knows the submission's code  which says the job was queued
   asks for a limit             the scheduler enforces it, not Python
   finds out later              by polling, or by looking for a record

what Python still owns: the job list, the identities, the reconciliation
what it no longer owns : the process, the clock, the machine

Everything in the bottom half of that table is a habit that has to be unlearned, and the first row is the one that causes real damage: bsub returns in milliseconds and the simulation has not started.

Chapter 23.8's timeout= now bounds the submission, which takes no time at all. Chapter 23.4's return code now describes the submission, which almost always succeeds. The outer bound Chapter 23.8 Section 8 pointed at — the scheduler's own limit — is now the only one that matters.

No scheduler is installed on this machine, per Chapter 23.1 Section 1, so what follows constructs the shapes these tools produce and verifies the handling. Nothing here claims a particular scheduler's exact wording.

2. The Annotation Throws Away the Job Id

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

subprocess.run(["bsub", "-J", job_name, "-o", str(log), *cmd])

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
import sys

SUBMIT = "print('Job <884213> is submitted to queue <normal>.')"

discarded = subprocess.run([sys.executable, "-c", SUBMIT])
print("return code      :", discarded.returncode)
print("captured stdout  :", discarded.stdout)
print("the job id       :", "gone")
print()
captured = subprocess.run([sys.executable, "-c", SUBMIT], capture_output=True, text=True)
print("with capture     :", repr(captured.stdout.strip()))
print("the job id is in it:", "884213" in captured.stdout)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

Job <884213> is submitted to queue <normal>.
return code      : 0
captured stdout  : None
the job id       : gone

with capture     : 'Job <884213> is submitted to queue <normal>.'
the job id is in it: True

The banner appears in the transcript above because the uncaptured child inherited the terminal — Chapter 23.1 Section 3 — which is exactly the problem. It went to a human and not to the program.

subprocess.run(...) without capture_output leaves .stdout as None, so the annotation discards the single most valuable thing the submission produced: the identifier by which every later question about this job has to be asked.

Chapter 23.5 argued that capturing is usually the wrong choice for a tool, because a simulator's output is enormous and belongs in a log file. A submission is the opposite case: one line, and that line is the handle. Capture it.

3. Zero Means Queued

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
import sys

ACCEPTED = "print('Job <884213> is submitted to queue <normal>.')"
REJECTED = ("import sys; sys.stderr.write('Queue <nosuch> does not exist.\\n'); "
            "raise SystemExit(255)")

for label, code in (("accepted", ACCEPTED), ("rejected", REJECTED)):
    cp = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True)
    print(f"{label:<9} rc={cp.returncode:<4} stdout={cp.stdout.strip()!r}")
    print(f"{'':<9} stderr={cp.stderr.strip()!r}")

print()
print("rc=0 means  :", "the scheduler accepted the work")
print("rc=0 does not mean:", "the job ran, finished, or passed")
print("the test's verdict is:", "somewhere else entirely, later")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

accepted  rc=0    stdout='Job <884213> is submitted to queue <normal>.'
          stderr=''
rejected  rc=255  stdout=''
          stderr='Queue <nosuch> does not exist.'

rc=0 means  : the scheduler accepted the work
rc=0 does not mean: the job ran, finished, or passed
the test's verdict is: somewhere else entirely, later

This is where four modules of training become a liability. Chapter 23.4 established that the return code is the tool's verdict and must be checked; every flow in Modules 23 to 25 treats rc == 0 as the good case. Here it is still worth checking and it means something far weaker: the request was well formed and the queue exists.

The failures it does catch are real and worth catching early — a queue name typo, a resource request the farm cannot satisfy, an expired allocation. Chapter 23.9's classification applies exactly: a rejected submission is ERROR, infrastructure, and the design was never exercised.

What it cannot catch is anything about the simulation, because the simulation has not begun.

4. Parsing the Handle

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re

BANNERS = {
    "lsf": "Job <884213> is submitted to queue <normal>.\n",
    "sge": 'Your job 55120 ("axi_smoke_1839274") has been submitted\n',
    "slurm": "Submitted batch job 7781234\n",
}
PATTERNS = {
    "lsf": re.compile(r"Job <(\d+)>"),
    "sge": re.compile(r"Your job (\d+)"),
    "slurm": re.compile(r"Submitted batch job (\d+)"),
}


def job_id(flavour, banner):
    m = PATTERNS[flavour].search(banner)
    if m is None:
        raise ValueError(f"no job id in {flavour} submission output: {banner.strip()!r}")
    return m.group(1)


for flavour, banner in BANNERS.items():
    print(f"{flavour:<6} -> {job_id(flavour, banner)}")

print()
print("a bare r'(\\d+)' on the same banners:",
      [re.search(r"(\d+)", b).group(1) for b in BANNERS.values()])
print()
try:
    job_id("lsf", "Job not submitted.\n")
except ValueError as exc:
    print("no id found:", exc)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

lsf    -> 884213
sge    -> 55120
slurm  -> 7781234

a bare r'(\d+)' on the same banners: ['884213', '55120', '7781234']

no id found: no job id in lsf submission output: 'Job not submitted.'

These are constructed specimens in the documented shape of each tool; the parser's behaviour over them is what is being demonstrated.

The bare (\d+) works on all three, which is precisely why it is the wrong choice. It is a pattern that happens to be right on today's banners and will silently take the wrong number the first time one of them carries another — a numeric queue name, a cluster identifier, a version string. Chapter 19.12's argument about anchoring to the stable part of a format applies, and the per-flavour pattern is three lines.

A missing id raises. The alternative is a None job id that propagates into the tracking table and makes one job permanently unqueryable — which, under Chapter 26.1's invariant, surfaces much later as an unexplained not_run.

5. A Scheduler State Is Not a Verdict

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

STATES = [
    ("DONE", "the job ran and its command exited 0"),
    ("EXIT", "the job ran and its command exited non-zero"),
    ("PEND", "still queued when we stopped waiting"),
    ("UNKWN", "the execution host stopped responding"),
    (None, "no accounting entry at all"),
]

MEANS = {"DONE": "a record may be on disk — classify it",
         "EXIT": "a record may be on disk — classify it",
         "PEND": "no record: not_run",
         "UNKWN": "no record: not_run",
         None: "no record: not_run"}

print(f"{'state':<7} {'what it says':<44} {'what we do'}")
for state, what in STATES:
    print(f"{str(state):<7} {what:<44} {MEANS[state]}")

print()
print("states that imply a verdict:", 0)
print("states that imply a record may exist:", sum(1 for s, _ in STATES if MEANS[s].startswith("a record")))
print()
print("mapping EXIT to FAIL would mean:", "the scheduler classified your design")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

state   what it says                                 what we do
DONE    the job ran and its command exited 0         a record may be on disk — classify it
EXIT    the job ran and its command exited non-zero  a record may be on disk — classify it
PEND    still queued when we stopped waiting         no record: not_run
UNKWN   the execution host stopped responding        no record: not_run
None    no accounting entry at all                   no record: not_run

states that imply a verdict: 0
states that imply a record may exist: 2

mapping EXIT to FAIL would mean: the scheduler classified your design

No scheduler state is a test result. DONE and EXIT both mean the job ran, so a record may be on disk and Chapter 25.6's classifier decides what it says; everything else means no record can exist, which is Chapter 26.1's not_run.

The tempting shortcut is DONE → PASS and EXIT → FAIL, and it is wrong in both directions. A UVM testbench that reports UVM_ERROR without calling $fatal exits zero — Chapter 25.6 Section 3's first disagreement — so DONE would report a failing test as a pass. And a job killed by the scheduler for exceeding its limit exits non-zero, which is Chapter 26.1's timeout, not a design failure.

The state's only job is to tell the aggregator whether to expect a record, which is useful and much narrower than it looks.

6. The Job Name Is the Join Key

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re

MAX_NAME = 60
ALLOWED = re.compile(r"[^A-Za-z0-9_.-]")


def job_name(test, seed, attempt=1):
    base = f"{test}_{seed}" + (f"_a{attempt}" if attempt > 1 else "")
    safe = ALLOWED.sub("_", base)
    return safe[:MAX_NAME]


def parse_name(name):
    body, _, tail = name.rpartition("_")
    if tail.startswith("a") and tail[1:].isdigit():
        test, _, seed = body.rpartition("_")
        return test, int(seed), int(tail[1:])
    return body, int(tail), 1


CASES = [("axi_smoke", 1839274, 1), ("axi_smoke", 1839274, 2),
         ("grp/axi burst", 77, 1)]

for test, seed, attempt in CASES:
    name = job_name(test, seed, attempt)
    print(f"{str((test, seed, attempt)):<34} -> {name}")

print()
print("round-trips (clean name):", parse_name(job_name("axi_smoke", 1839274, 2)))
print("does NOT round-trip     :", parse_name(job_name("grp/axi burst", 77)))
print("because the name was sanitised:", job_name("grp/axi burst", 77))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

('axi_smoke', 1839274, 1)          -> axi_smoke_1839274
('axi_smoke', 1839274, 2)          -> axi_smoke_1839274_a2
('grp/axi burst', 77, 1)           -> grp_axi_burst_77

round-trips (clean name): ('axi_smoke', 1839274, 2)
does NOT round-trip     : ('grp_axi_burst', 77, 1)
because the name was sanitised: grp_axi_burst_77

The sanitised name does not decode back to the job it names. grp/axi burst became grp_axi_burst, and parsing it back gives a test name that does not exist in the job list.

This is Chapter 26.8 Section 7's round-trip hazard, and it is sharper here because schedulers impose real charset and length limits that a filesystem does not. A name truncated at sixty characters or with its separators rewritten cannot be parsed back reliably at all.

The fix is to stop trying:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

from dataclasses import dataclass


@dataclass(frozen=True)
class Job:
    test: str
    seed: int
    attempt: int = 1


SUBMITTED = {}


def submit(job, job_id, name):
    """Keep the mapping. Never recover a job by parsing its name."""
    SUBMITTED[job_id] = {"job": job, "name": name}


submit(Job("grp/axi burst", 77), "884213", "grp_axi_burst_77")
submit(Job("axi_smoke", 1839274, 2), "884214", "axi_smoke_1839274_a2")

for job_id, entry in SUBMITTED.items():
    j = entry["job"]
    print(f"{job_id}  name={entry['name']:<24} job={(j.test, j.seed, j.attempt)}")

print()
print("recovering the job needs no parsing:", SUBMITTED["884213"]["job"].test)
print("the name is for humans reading bjobs:", True)
print("the id is the key:", list(SUBMITTED))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

884213  name=grp_axi_burst_77         job=('grp/axi burst', 77, 1)
884214  name=axi_smoke_1839274_a2     job=('axi_smoke', 1839274, 2)

recovering the job needs no parsing: grp/axi burst
the name is for humans reading bjobs: True
the id is the key: ['884213', '884214']

Keep the mapping; do not parse the name back. The scheduler's id is the key, the Job is stored next to it, and the name exists so that somebody running bjobs sees something recognisable. A lossy encoding is fine as long as nothing depends on inverting it — Chapter 26.8 chose the opposite trade for a state file because there the encoding is the key.

That mapping must be persisted, for the same reason Chapter 26.8 persists state: if the submitting process dies, the ids are the only way to find the jobs again, and they exist nowhere else.

7. The Shape of a Submitted Regression

Figure 1 — six stages, and the join between them is the scheduler's job id. Python builds and submits, records the identity against the id, waits without owning anything, then collects records from disk and reconciles against the same job list Chapter 26.1 fixed before any of it started.

Steps five and six are Chapter 26.6 unchanged, which is the point of having built them against a job list rather than against a process. The runner did not need to know the jobs went to a farm, and neither does the aggregator.

8. Waiting Without Owning

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

STATES_OVER_TIME = [
    {"884213": "PEND", "884214": "PEND", "884215": "PEND"},
    {"884213": "RUN", "884214": "PEND", "884215": "RUN"},
    {"884213": "DONE", "884214": "RUN", "884215": "RUN"},
    {"884213": "DONE", "884214": "DONE", "884215": "UNKWN"},
]
TERMINAL = {"DONE", "EXIT", "UNKWN"}

for poll, states in enumerate(STATES_OVER_TIME, start=1):
    pending = [j for j, s in states.items() if s not in TERMINAL]
    counts = {}
    for s in states.values():
        counts[s] = counts.get(s, 0) + 1
    print(f"poll {poll}: {dict(sorted(counts.items()))}  outstanding={len(pending)}")

final = STATES_OVER_TIME[-1]
print()
print("all terminal:", all(s in TERMINAL for s in final.values()))
print("ran to completion:", [j for j, s in final.items() if s == "DONE"])
print("never produced a record:", [j for j, s in final.items() if s == "UNKWN"])

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

poll 1: {'PEND': 3}  outstanding=3
poll 2: {'PEND': 1, 'RUN': 2}  outstanding=3
poll 3: {'DONE': 1, 'RUN': 2}  outstanding=2
poll 4: {'DONE': 2, 'UNKWN': 1}  outstanding=0

all terminal: True
ran to completion: ['884213', '884214']
never produced a record: ['884215']

Polling until nothing is outstanding, with UNKWN counted as terminal — a job whose host vanished will never progress, and waiting for it is waiting forever.

Three properties a real poll loop needs, and none of them is the polling itself.

A deadline. The outer bound Chapter 23.8 Section 8 described is the scheduler's, and the waiter needs one too: a farm outage leaves jobs pending indefinitely, and a regression that waits forever has the same symptom as the hung tool Chapter 23.8 opened with.

A poll interval that does not hammer the scheduler. Querying four hundred job states every second is a denial of service against shared infrastructure, and most schedulers will rate-limit or refuse. Chapter 24.2's version-cache reasoning applies: ask rarely, and in bulk.

Terminal states treated as terminal. UNKWN is the one people miss, because it reads like a transient. It is not — it means the scheduler has lost track, and the correct response is to stop waiting and let Chapter 26.6 record not_run.

9. Submission, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import json
import re
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path


class SubmitError(Exception):
    """The scheduler refused the work, or said something unparseable."""


@dataclass(frozen=True)
class Job:
    test: str
    seed: int
    attempt: int = 1


JOB_ID = re.compile(r"Job <(\d+)>")
UNSAFE = re.compile(r"[^A-Za-z0-9_.-]")
SUBMITTER = "print('Job <88%04d> is submitted to queue <normal>.')"


def job_name(job):
    base = f"{job.test}_{job.seed}" + (f"_a{job.attempt}" if job.attempt > 1 else "")
    return UNSAFE.sub("_", base)[:60]


def submit(job, cmd, root, *, n):
    """Submit one job. Returns its scheduler id; never guesses one."""
    run_dir = Path(root) / "runs" / job_name(job)
    run_dir.mkdir(parents=True, exist_ok=True)
    sched_log = run_dir / "scheduler.out"

    submit_cmd = [sys.executable, "-c", SUBMITTER % n]  # stands in for bsub
    cp = subprocess.run(submit_cmd, capture_output=True, text=True)

    if cp.returncode != 0:
        raise SubmitError(f"{job_name(job)}: submission refused "
                          f"(rc={cp.returncode}) {cp.stderr.strip()!r}")
    m = JOB_ID.search(cp.stdout)
    if m is None:
        raise SubmitError(f"{job_name(job)}: no job id in {cp.stdout.strip()!r}")
    return m.group(1), {"job": job, "name": job_name(job),
                        "run_dir": str(run_dir), "sched_log": str(sched_log)}


JOBS = [Job("axi_smoke", 1839274), Job("axi_burst", 1839275), Job("axi_burst", 1839275, 2)]

table = {}
for n, job in enumerate(JOBS, start=1):
    job_id, entry = submit(job, ["simv"], "root", n=n)
    table[job_id] = entry

Path("root").mkdir(exist_ok=True)
Path("root/submitted.json").write_text(
    json.dumps({k: {"test": v["job"].test, "seed": v["job"].seed,
                    "attempt": v["job"].attempt, "name": v["name"],
                    "run_dir": v["run_dir"]} for k, v in table.items()},
               indent=2, sort_keys=True) + "\n", encoding="utf-8")

for job_id, entry in sorted(table.items()):
    print(f"{job_id}  {entry['name']:<24} {entry['run_dir']}")

print()
print("submitted           :", len(table))
print("jobs in the list    :", len(JOBS))
print("all accounted for   :", len(table) == len(JOBS))
print("mapping persisted   :", Path("root/submitted.json").exists())
print("each has its own dir:", len({e['run_dir'] for e in table.values()}) == len(JOBS))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

880001  axi_smoke_1839274        root/runs/axi_smoke_1839274
880002  axi_burst_1839275        root/runs/axi_burst_1839275
880003  axi_burst_1839275_a2     root/runs/axi_burst_1839275_a2

submitted           : 3
jobs in the list    : 3
all accounted for   : True
mapping persisted   : True
each has its own dir: True

Six decisions, and the last two are what make this module's earlier chapters still apply.

The submission output is captured and the id is parsed from it, rather than being discarded as the annotation does.

A non-zero submission raises SubmitError. Chapter 23.9's shape: a refusal is infrastructure, the design was never exercised, and the message names the job and quotes the scheduler.

An unparseable banner raises too, rather than producing a None id. Chapter 26.1 Section 2's missing job, prevented at the point it would be created.

The mapping is written to disk immediately. If the submitting process dies, four hundred jobs are running on the farm and submitted.json is the only record that they belong to this regression.

The scheduler's own log goes in the run directory alongside the tool's. They are different files answering different questions — the scheduler's says why a job was killed, the tool's says what the design did — and Chapter 26.4's isolation covers both.

Each job still has its own directory, by Chapter 26.4's naming, including the retry. Nothing about moving to a farm changed that, which is why Chapter 26.6's aggregation works unmodified.

10. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re
import subprocess
import sys

PAT = re.compile(r"Job <(\d+)>")
TERMINAL = {"DONE", "EXIT", "UNKWN"}

# A
cp = subprocess.run([sys.executable, "-c", "print('Job <7> is submitted')"])
print("A:", cp.returncode, cp.stdout)

# B
cp = subprocess.run([sys.executable, "-c", "print('Job <7> is submitted')"],
                    capture_output=True, text=True)
print("B:", repr(cp.stdout.strip()))

# C
print("C:", PAT.search(cp.stdout).group(1))

# D
print("D:", PAT.search("Submitted batch job 7781234"))

# E
cp = subprocess.run([sys.executable, "-c", "raise SystemExit(255)"], capture_output=True)
print("E:", cp.returncode, "submission refused, design untested")

# F
states = {"a": "DONE", "b": "PEND", "c": "UNKWN"}
print("F:", [j for j, s in states.items() if s not in TERMINAL])

# G
print("G:", re.sub(r"[^A-Za-z0-9_.-]", "_", "grp/axi burst_77"))

# H
print("H:", "axi_smoke_1839274_a2".rpartition("_")[2])

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

Job <7> is submitted
A: 0 None
B: 'Job <7> is submitted'
C: 7
D: None
E: 255 submission refused, design untested
F: ['b']
G: grp_axi_burst_77
H: a2

A — the uncaptured banner reaches the terminal and .stdout is None. The annotation's loss.

B — captured, it reaches the program.

C — and the id parses out of it.

D — the LSF pattern correctly declines a Slurm banner rather than guessing.

E — a refused submission is ERROR, not a test result.

F — only PEND is outstanding; UNKWN is terminal.

G — sanitising rewrites both the slash and the space, which is why the name is not invertible.

H — the attempt suffix is recoverable from a clean name, and only from a clean one.

11. Exercises

Exercise 1 — the inversion. For your own flow, list what Python owns locally and what it would still own on a farm.

Exercise 2 — milliseconds. Say what subprocess.run with a timeout= bounds once the command is bsub.

Exercise 3 — the discarded id. Run a submission without capturing and show .stdout is None.

Exercise 4 — the one case for capture. Contrast this with Chapter 23.5's advice and say why it reverses.

Exercise 5 — zero means queued. Write down, in one sentence each, what rc=0 and rc≠0 from a submit command mean.

Exercise 6 — which failures it catches. List three submission failures worth catching, and classify them.

Exercise 7 — three banners. Write a per-flavour pattern for every scheduler your site uses.

Exercise 8 — the bare pattern. Say why (\d+) working on all three today is an argument against it.

Exercise 9 — no id. Make an unparseable banner raise, and say what a None id would eventually look like.

Exercise 10 — five states. Map your scheduler's states onto "a record may exist" and "no record".

Exercise 11 — the tempting shortcut. Explain why DONE → PASS and EXIT → FAIL are both wrong, with an example each.

Exercise 12 — the join key. Show a job name being sanitised so it no longer decodes to its job.

Exercise 13 — stop inverting. Keep an id-to-job mapping instead, and say what the name is now for.

Exercise 14 — persist it. Say what is lost if the submitting process dies and the mapping was only in memory.

Exercise 15 — a deadline. Add one to a poll loop and say which earlier chapter's failure it prevents.

Exercise 16 — the interval. Choose a poll interval for four hundred jobs and justify it.

Exercise 17 — UNKWN is terminal. Say why treating it as transient makes a regression hang.

Exercise 18 — give up and cancel. Describe what an abandoned pending job does to a later regression's aggregator.

Exercise 19 — two logs. Say what the scheduler's log answers that the tool's log cannot.

Exercise 20 — unchanged. Confirm that your Chapter 26.6 aggregation needs no modification to work with submitted jobs, and say why.

12. Summary

Handing jobs to a scheduler inverts the module: Python keeps the job list, the identities and the reconciliation, and gives up the process, the clock and the machine.
bsub returns in milliseconds, so Chapter 23.8's timeout= now bounds the submission and the scheduler's own limit is the real one.
The annotation discards the job id. Without capture_output, .stdout is None and the only handle on the work is gone.
This is the one place Chapter 23.5's advice reverses — the interesting output is one line, and that line is the handle.
A submit command's zero means queued, not passed, and four modules of return-code training point the wrong way here.
A refused submission is ERROR — infrastructure, design never exercised.
Parse the id with a per-scheduler pattern; a bare (\d+) works on today's banners and will take the wrong number eventually.
Raise when no id is found, or a None id becomes an unexplained not_run much later.
No scheduler state is a verdict. DONE and EXIT mean a record may exist and Chapter 25.6 classifies it; everything else means not_run.
DONE → PASS is wrong because a UVM bench can report errors and exit zero; EXIT → FAIL is wrong because a scheduler kill is a timeout.
A sanitised job name does not decode back to its job, and scheduler charset and length limits make that worse than a filesystem's.
Keep an id-to-job mapping and never parse the name back; the name is for whoever reads the queue.
Persist that mapping immediately, or a dead submitter leaves four hundred unidentifiable jobs on the farm.
A poll loop needs a deadline, a sane interval, and UNKWN treated as terminal.
Abandoning a pending job is not stopping it — cancel, or it writes a record a later regression will find.
Keep the scheduler's log beside the tool's; they answer different questions.
Chapter 26.6's aggregation works unmodified, because it was written against the job list rather than against a process.

That completes Module 26. A regression now expands a suite into a job list that fixes its own total, gives every job an isolated directory, bounds and selectively retries it, collects a result for every job or records that none arrived, groups the failures into a handful of causes, resumes without redoing finished work, and does all of it whether the jobs run here or on a farm. Module 27 runs them at the same time.

Where this fits

Part of the Python curriculum.