Linux & Bash · Part II · Module 32 · Regression Automation End to End

Nightly Runs, Trends and Archiving

A regression that runs unattended every night raises two problems the interactive one never had: the environment it inherits is not yours, and ninety nights of output will not fit. This chapter measures a nightly run that lost its tools and its image digest while every stage reported success, builds a trend that refuses to compare incomparable nights, and quantifies what to throw away.

Advanced26 min readLinuxPart IIRegressionSchedulingArchiving

Part II · Module 32 · Chapter 32.8 · Nightly Runs, Trends and Archiving

Chapter 32.7 finished the regression as a thing you invoke. Running it unattended, every night, for a year, adds exactly two new problems — the environment is not the one you tested in, and the output does not fit — and one new capability, which is that a year of results is a measurement you cannot otherwise make.

1. The Night It Ran With No Tools

A scheduled job does not inherit your shell. Whatever runs it — cron, a CI agent, a batch system — hands the job a minimal environment, and the cheapest way to find out what that does to a regression is to reproduce it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ echo $PATH | tr ':' '\n' | grep -c .
15
$ env -i /bin/bash -lc 'echo $PATH | tr ":" "\n" | grep -c .'
11
$ env -i /bin/bash -c 'echo "$PATH"'
/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.

Fifteen PATH entries interactively, eleven in a login shell, and five with no environment at all. Chapter 30.5 measured the same gap for a submitted job and made the same point: env -i is a free, local, exact reproduction of the thing that breaks scheduled runs.

Running the driver that way:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ env -i HOME="$HOME" /bin/bash -c "cd $PWD && SLOTS=2 LIMIT_S=3 bin/regress"
bin/regress: line 22: md5sum: command not found
regress: /…/regr/20261009T100443Z
regress: 24 runs, 2 slots, 3s limit each
regress: dispatch finished in 2s (xargs rc=0)
regress: 24 result rows

Two seconds for twenty-four simulations, which is six times faster than it has any right to be. The report says why:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

 24 runs: 0 PASS  0 FAIL  0 TIMEOUT  24 CRASH
          0 NO_VERDICT  0 INCONSISTENT  0 VACUOUS
          0 MISSING  0 NOSTATUS

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat regr/20261009T100443Z/runs/b2b_s1/run.log
/…/bin/runsim: line 12: exec: vvp: not found

The simulator is not on a minimal PATH. Twenty-four crashes, and the system handled it well: Chapter 32.3's classifier called every one of them CRASH rather than FAIL, Chapter 32.5's report put all twenty-four in the environment column, and the exit status was 1. Nothing lied.

Except for one line at the very top, which went to standard error, and which cron would have mailed to an address nobody reads.

2. The Line That Did Lie

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

bin/regress: line 22: md5sum: command not found

Line 22 was the image digest — Chapter 32.2's env.txt entry and the thing Chapter 32.6 called "the single highest-value thing a regression records about itself":

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

echo "image_md5=$(md5 -q "$root/logs/sim.vvp" 2>/dev/null || md5sum "$root/logs/sim.vvp" | cut -d' ' -f1)"

That line is a fallback chain and it has no floor. Here is why both halves failed:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ command -v md5
/sbin/md5
$ env -i /bin/bash -c 'command -v md5 || echo "md5 NOT on this PATH"'
md5 NOT on this PATH
$ env -i /bin/bash -c 'command -v md5sum || echo "md5sum NOT on this PATH"'
md5sum NOT on this PATH

Both of them live in /sbin, and /sbin is not on that PATH. The fallback chain reads as "try the BSD name, then the GNU name", which would be sound if the two names were in different places — and they are not:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ls -li /sbin/md5 /sbin/md5sum
1152921500312571498 -rwxr-xr-x  6 root  wheel  136576 /sbin/md5
1152921500312571498 -rwxr-xr-x  6 root  wheel  136576 /sbin/md5sum
$ env -i /bin/bash -c 'echo "PATH=[$PATH]"'
PATH=[/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.]

The same inode, under two names, six hardlinks to one multi-call binary — so md5sum does exist here (md5sum (Darwin) 1.0, and it even prints GNU-style output with the trailing -). The fallback was never a second platform; it was the same file in the same unreachable directory. So the first command fails silently (its stderr was redirected), the second fails loudly, and the substitution yields the empty string:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ grep '^image_md5=' regr/20261009T100443Z/env.txt
image_md5=
$ grep '^image_md5=' regr/20261009T100443Z/env.txt | sed 's/^image_md5=//' | wc -c
       0

An empty digest. Not absent — present and empty, which is worse, because the field is there and looks filled in.

Two empty digests compare equal, so the guard verified nothing

Chapter 32.6's guard is a string comparison:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if [ "$was" != "$now" ]; then … refuse … fi

Re-running that night's failures from the same cron-like environment, so that now is also empty:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ env -i HOME="$HOME" /bin/bash -c "cd $PWD && bin/rerun regr/20261009T100443Z"
bin/rerun: line 18: md5sum: command not found
rerun: 24 spec(s) from 20261009T100443Z

  SPEC               was            now
  b2b:1              CRASH          CRASH          reproduced
  b2b:3              CRASH          CRASH          reproduced
  crash:1            CRASH          CRASH          reproduced
  fairness:1         CRASH          CRASH          reproduced
  flaky:1            CRASH          CRASH          reproduced
  …

It did not refuse. "" != "" is false, so the guard passed, and the rerun went on to report a perfect reproduction — of a regression in which nothing ran, against an environment in which nothing could run. Every stage was telling the truth about its own job and the composition was nonsense.

b2b:1 and fairness:1 are tests that pass. They are listed as CRASH → CRASH, reproduced.

The fix is not a better comparison; it is refusing to produce an empty digest in the first place:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# A digest, or a refusal. md5 lives in /sbin on macOS, which is NOT on a cron
# PATH, and md5sum does not exist here at all -- so an unguarded fallback chain
# silently yields an EMPTY digest, and two empty digests compare equal.
digest() {
  local f=$1 d=""
  for t in /sbin/md5 /usr/bin/md5 /usr/bin/md5sum /sbin/md5sum; do
    [ -x "$t" ] || continue
    case $t in
      *md5sum) d=$("$t" "$f" | cut -d' ' -f1);;
      *)       d=$("$t" -q "$f");;
    esac
    [ -n "$d" ] && break
  done
  [ -n "$d" ] || { echo "digest: no working md5 tool found for '$f'" >&2; return 1; }
  printf '%s\n' "$d"
}

Absolute candidate paths, because PATH is the thing that is wrong. A check that the result is non-empty, because a tool can exist and fail. And an explicit refusal, because there is no safe default for "I cannot tell you what ran".

Measured three ways — the working case, the refusal, and the environment that broke it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== normal ===
  bfa18ecfa441cb8a19f0cf9e4213215c
    rc=0
=== with every candidate path broken (the refusal) ===
  digest: no working md5 tool found for 'logs/sim.vvp'
    rc=1
=== and from a cron-like environment ===
  bfa18ecfa441cb8a19f0cf9e4213215c
    rc=0
$ env -i HOME="$HOME" /bin/bash -c "cd $PWD && SLOTS=2 LIMIT_S=2 bin/regress" >/dev/null 2>&1
$ grep '^image_md5=' regr/…/env.txt
image_md5=bfa18ecfa441cb8a19f0cf9e4213215c

The general shape is one this track has met in several forms, and it is worth naming: a fallback chain with no floor turns a missing tool into a missing value, and a missing value into a silent wrong answer. a || b where both can fail needs a third branch, and that branch is almost always a refusal.

What a nightly job has to do about its environment

So the driver cannot assume anything it has not asserted:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# set up the environment explicitly, then CHECK it -- do not hope
. /etc/profile.d/eda-tools.sh 2>/dev/null || true
for t in iverilog vvp; do
  command -v "$t" >/dev/null || { echo "regress: $t not on PATH -- refusing" >&2; exit 3; }
done
digest() { … }                                  # absolute paths, with a refusal

Three obligations, each matching something measured above. Source the setup rather than inheriting it — Chapter 25.6's subject. Assert the tools are reachable, and exit 3 (setup) rather than producing twenty-four crashes. And never depend on a tool's location being on PATH when you can name it.

Exiting 3 is the right value here precisely because of Chapter 32.7's division by who acts: a missing simulator is not a design question, and twenty-four CRASH rows invite somebody to go looking at twenty-four testbenches.

2b. Scheduling It

What actually exists on this machine:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  crontab      /usr/bin/crontab
  cron         /usr/sbin/cron
  at           /usr/bin/at
  launchctl    /bin/launchctl
  systemctl    ABSENT
  anacron      ABSENT

cron is present and has no entries here (crontab: no crontab for …), and the shape of an entry is worth getting right once:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# m  h  dom mon dow   command
  30 22  *   *   1-5   /proj/chip_a/bin/nightly >> /proj/chip_a/logs/nightly.out 2>&1

Four things about that line, all of which follow from Sections 1 and 2 rather than from cron itself.

It calls a wrapper, not the driver. nightly is where the environment setup, the tool assertions and the retention policy live. Putting them in crontab makes them invisible and untestable.

It redirects both streams to a file. cron mails whatever a job writes, which means the md5sum: command not found line above goes to a mailbox rather than a log — and on a machine with no mail configured, nowhere at all. Append to a file somebody can read.

It appends rather than truncates, so last night's output is still there when tonight's job fails in the first second.

And >> file 2>&1, in that order. The reverse — 2>&1 >> file — points stderr at the terminal's stdout before stdout is redirected, so the diagnostics do not reach the file. Chapter 12.3 measured the ordering; this is the one line where getting it backwards loses exactly the message that explains the failure.

The one cron-specific thing worth knowing is that % is special in a crontab command and needs escaping as \%, which matters the moment a date format enters the line — and it is a reason to keep the command a bare wrapper invocation.

3. The Trend, and When Not to Draw One

Ninety nights of results.tsv is a dataset. Three of them, on this fixture:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  REGRESSION              runs  PASS  FAIL TIMEO CRASH NO_VE INCON VACUO MISSI NOSTA
  20261009T093929Z          24     9     3     2     2     2     4     2     0     0
  20261009T094027Z          24    13     1     2     2     2     2     2     0     0
  20261009T094111Z          24    10     2     2     2     2     4     2     0     0

Nine passes, then thirteen, then ten. Read as a trend: progress, then a regression on the third night. And the per-spec comparison between consecutive nights says otherwise:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  verdict changes between consecutive regressions:
    20261009T093929Z -> 20261009T094027Z
      inconsistent:1     INCONSISTENT -> PASS
      inconsistent:2     INCONSISTENT -> PASS
      prio0:1            FAIL -> PASS
      prio0:2            FAIL -> PASS
    20261009T094027Z -> 20261009T094111Z
      inconsistent:1     PASS -> INCONSISTENT
      prio0:1            PASS -> FAIL
      MATRIX CHANGED: 12 spec(s) gone, 12 new -- counts above are not comparable

The second transition shows two changed specs, not four, and then the line that matters: twelve specs left the matrix and twelve arrived, because the seed list changed. Chapter 32.5 measured that in detail; here it appears as an annotation on a trend table, which is the only place it can do any good.

A trend table without that annotation is actively misleading, because a chart is more persuasive than a report and nobody checks a chart's denominators. So the rule is: print the matrix change next to the counts, and say in words that the counts are not comparable. A trend over a changing matrix is not a trend; it is two different measurements plotted on one axis.

A trend has to read archived nights, which is what decides the archive's shape

Ninety nights means eighty-nine of them are archived. If reading a trend requires unpacking archives, nobody draws one.

Which is why Chapter 32.4's results file is kept outside the archive, and why the trend resolves either form transparently:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

resolve() {
  case $1 in
    *.results.tsv) printf '%s\n' "$1";;
    *) if   [ -r "$1/results.tsv" ];   then printf '%s\n' "$1/results.tsv"
       elif [ -r "$1.results.tsv" ];   then printf '%s\n' "$1.results.tsv"
       else return 1; fi;;
  esac
}

The first regression in the table above is an archive — a 14 KB .tar.gz plus a 1 KB .results.tsv beside it — and its row is byte-identical to the row it produced when it was a live directory. That is the whole design goal: a year of trend data is ninety small files and no unpacking.

Note also that resolve returns failure rather than guessing, and the caller prints (no results) for that night rather than omitting the row. A trend with a silently missing night is a trend with a gap nobody sees — the same reason Chapter 32.4 emits a row for a run that never happened.

4. What a Regression Costs to Keep

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ find regr/20261009T093929Z -type f | wc -l
     151
$ find regr/20261009T093929Z -type f -print0 | xargs -0 wc -c | tail -1
    8585 total
$ du -sk regr/20261009T093929Z
604     regr/20261009T093929Z

8,585 bytes of content in 618,496 bytes of disk — a factor of 72. One hundred and fifty-one files, most of them a handful of bytes, each occupying a whole filesystem block. That is what Chapter 28.3 measured in general; here it is the dominant cost of keeping history.

And what the obvious remedies are worth:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ tar -cf /tmp/r.tar  -C regr 20261009T093929Z
$ tar -czf /tmp/r.tgz -C regr 20261009T093929Z
  tar     : 534016 bytes
  tar.gz  : 14469 bytes  (2.3% of the on-disk size)

tar alone barely helps — it pads every file to a 512-byte record, so 151 tiny files are still 151 records plus headers. tar.gz is 2.3% of the directory's footprint, a factor of 42, because the content is highly repetitive text.

Now the instinct everybody has, which is to throw away the passing runs' logs:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ # remove run.log from the 9 PASS runs
$ awk -F'\t' '$4=="PASS" {print $2"_s"$3}' results.tsv | while read -r d; do rm -f "runs/$d/run.log"; done
  logs removed  : 9
  content now   : 7343 bytes (was 8585)
  tar.gz now    : 13563 bytes (was 14469)

A 6% saving, for the permanent loss of the evidence that nine runs passed.

Compress; do not prune

Those two numbers decide the policy, and they are the opposite way round from intuition:

action	saving	cost
delete the passing runs' logs	6%	the evidence that those runs passed
`tar.gz` the whole regression	97.7%	one `tar -xzf` before investigating

Pruning feels like the aggressive option and is nearly free of benefit. The reason is arithmetic: the passing runs are the small logs — a pass is 133 to 143 bytes here, while a failing run's log is 170 to 207 — and compression finds most of its win in exactly the repetition that many similar small files provide.

So the retention policy writes itself:

Compress every finished regression, immediately. There is no window in which the uncompressed directory is more useful, because tar -tzf lists it and tar -xzf restores it in under a second at this size.

Keep results.tsv and env.txt unpacked beside the archive, which is what makes Section 3's trend possible.

And delete whole regressions on age, not files within them. A regression is the unit that makes sense — it is internally consistent, and half a regression answers nothing. Chapter 28.4 is the mechanics of doing that safely; the policy decision is simply that the thing you delete is a night, not a file.

The one case for pruning is the artifact this fixture does not have: a waveform. Chapter 27.6 measured dumps at megabytes per run, and a waveform does not compress well because it is already dense. If a regression dumps waves, the passing runs' dumps are the right thing to delete and the only thing worth deleting — and the general rule is to prune by size measured, not by category assumed.

Figure 1 — what a finished regression costs, measured. One hundred and fifty-one mostly tiny files hold 8.5 KB of content and occupy 604 KB of disk blocks. Deleting the passing runs' logs — the instinctive saving — recovers 6 percent, because those are the smallest logs. Compressing the whole directory recovers 97.7 percent.

5. Archiving That Can Be Trusted

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# archive REGRDIR -- turn a finished regression into one compressed file, and
# keep the summary readable without unpacking it.
# Measured on this fixture: the directory costs 72x its own content in disk
# blocks, and gzip gets the whole thing to 2.3% of that. Deleting the passing
# runs' logs first saves 6% -- so compress, do not prune.
set -u
d=${1:-}
[ -n "$d" ] || { echo "archive: need a regression directory" >&2; exit 2; }
[ -d "$d" ] || { echo "archive: '$d' is not a directory" >&2; exit 2; }
[ -r "$d/results.tsv" ] || { echo "archive: '$d' has no results.tsv -- refusing to archive an unfinished run" >&2; exit 2; }
base=$(cd "$(dirname "$d")" && pwd); name=$(basename "$d")
tgz=$base/$name.tar.gz
[ -e "$tgz" ] && { echo "archive: '$tgz' already exists" >&2; exit 2; }

# Keep the two small files OUTSIDE the archive, so a trend or a search never
# has to unpack anything.
cp "$d/results.tsv" "$base/$name.results.tsv"
cp "$d/env.txt"     "$base/$name.env.txt" 2>/dev/null

before=$(( $(du -sk "$d" | cut -f1) * 1024 ))
tar -czf "$tgz" -C "$base" "$name" || { echo "archive: tar failed" >&2; rm -f "$tgz"; exit 2; }

# Verify the archive before removing anything. An archive nobody opened is a
# claim, not a backup.
n_in=$(find "$d" -type f | grep -c .)
n_ar=$(tar -tzf "$tgz" | grep -vc '/$')
if [ "$n_in" -ne "$n_ar" ]; then
  echo "archive: $n_in files in, $n_ar in the archive -- NOT deleting the directory" >&2
  exit 1
fi
after=$(wc -c < "$tgz" | tr -d ' ')
printf 'archive: %s -> %s\n' "$name" "$(basename "$tgz")"
printf '         %s files, %s bytes on disk -> %s bytes (%s%%)\n' \
  "$n_in" "$before" "$after" "$(echo "scale=1; 100*$after/$before" | bc)"
printf '         summary kept unpacked: %s.results.tsv\n' "$name"
rm -rf "$d"

One run of it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ archive regr/20261009T093929Z
archive: 20261009T093929Z -> 20261009T093929Z.tar.gz
         151 files, 618496 bytes on disk -> 14469 bytes (2.3%)
         summary kept unpacked: 20261009T093929Z.results.tsv

The part that matters is the four lines before the rm -rf:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

n_in=$(find "$d" -type f | grep -c .)
n_ar=$(tar -tzf "$tgz" | grep -vc '/$')
if [ "$n_in" -ne "$n_ar" ]; then
  echo "archive: $n_in files in, $n_ar in the archive -- NOT deleting the directory" >&2
  exit 1
fi

The archive is read back and counted before anything is deleted. An archive nobody has opened is a claim, not a backup — and this is the one script in the module whose bug is unrecoverable, so it gets the only verification in the module that runs every time rather than only in a test.

Proven able to refuse, by making tar skip the logs:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ sed 's|tar -czf "$tgz" -C "$base" "$name"|tar -czf "$tgz" --exclude "*.log" -C "$base" "$name"|' archive > /tmp/amut.sh
$ bash /tmp/amut.sh /tmp/at
archive: 151 files in, 127 in the archive -- NOT deleting the directory
  mutant exit status: 1
  directory still present: YES - refused to delete

One hundred and fifty-one in, one hundred and twenty-seven archived, and the directory survived. And the other refusal:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ archive /tmp/at2          # a regression with no results.tsv
archive: '/tmp/at2' has no results.tsv -- refusing to archive an unfinished run
  rc=2

An unfinished regression is not archived at all, because archiving is how a regression stops being re-runnable, and a run that never produced results is exactly the one somebody will want to look at.

Three smaller decisions in the same script:

It refuses to overwrite. [ -e "$tgz" ] && exit 2. An archive silently replacing another night's archive is the only way this script could destroy data twice.

It copies the summary out before archiving, not after, so a failure part-way through leaves the summary present rather than an archive with no index.

And tar -tzf | grep -vc '/$' counts entries that are not directories, because find -type f does not count directories either. Comparing a file count against an entry count would fail every time, and a verification that always fails gets switched off — which is the same end state as no verification, reached by a more annoying route.

6. The Module, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# nightly -- what cron actually calls.
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
cd "$root" || exit 3

# 1. the environment, asserted rather than inherited   (32.8 §1)
. /etc/profile.d/eda-tools.sh 2>/dev/null || true
for t in iverilog vvp; do
  command -v "$t" >/dev/null || { echo "nightly: $t not on PATH" >&2; exit 3; }
done

# 2. build once, and let the digest identify it        (32.2, 32.6)
iverilog -o logs/sim.vvp -f cfg/files.f || exit 3

# 3. the regression                                     (32.2 - 32.5)
set -o pipefail
bin/regress | tee -a logs/nightly.out
rc=$?

# 4. a stable path for whatever reads the result        (32.7)
ln -sfn "$(ls -dt regr/2026* | head -1)" regr/latest

# 5. keep the history affordable                        (32.8 §4, §5)
for d in $(ls -dt regr/2026*/ 2>/dev/null | tail -n +8); do
  bin/archive "${d%/}" || echo "nightly: archive failed for $d" >&2
done
find regr -name '*.tar.gz' -mtime +90 -print -delete

exit "$rc"

Five stages again, at a different scale, and every line traceable to something measured:

line	why	measured in
assert the tools	24 `CRASH` rows from a missing `PATH`	§1
the digest helper	an empty digest "reproduced" 24 crashes	§2
`set -o pipefail`	`\| tee` returns `tee`'s status	32.7
`tee -a`	tonight must not erase last night	§2b
`ln -sfn`	without `-n` the link lands inside the target	32.7
archive the 8th-newest onward	72× disk overhead, 2.3% compressed	§4
delete whole archives on age	half a regression answers nothing	§4

One thing to be explicit about: -mtime +90 skips the file that is exactly 90 days old, because find's -mtime +n means strictly greater than n days. Chapter 28.4 measured that off-by-one. It does not matter for a 90-day policy; it matters a great deal for a 7-day one, and it is the kind of thing that is only ever discovered when the file you wanted is gone.

Practice

Run your driver under env -i and read the first line of output.
Count your PATH entries interactively, in a login shell, and under env -i.
Find a tool your flow needs that is not on a minimal PATH.
Find a fallback chain in your scripts where both branches can fail, and work out what the result is.
Produce an empty digest and then compare two of them.
Write a digest function with absolute candidate paths and an explicit refusal.
Prove it refuses by breaking every candidate path.
Prove it works under env -i.
Assert your tools exist before running anything, and return the setup status.
Write the crontab line calling a wrapper, with >> file 2>&1 in that order.
Reverse the redirection order and find out where the diagnostics went.
Truncate instead of appending and then fail in the first second.
Build a trend over three or more nights and read the counts.
Change your seed list between two nights and confirm the trend says the counts are not comparable.
Remove that annotation and show the trend to somebody; see what they conclude.
Archive a night, then draw the trend across it without unpacking anything.
Make the resolver return failure for a night with no results, and print the row anyway.
Measure your own files, content and disk usage, and compute the ratio.
Compare tar against tar.gz and explain the difference.
Delete the passing runs' logs and measure what you saved.
Decide your retention policy from those two numbers, and write it down.
Verify an archive by counting entries before deleting the directory.
Break the archiving so it misses files and confirm the directory survives.
Try to archive an unfinished regression and confirm it refuses.
Archive over an existing archive and confirm it refuses.
Check whether your retention find includes the boundary day.

Check Your Understanding

What two new problems does running nightly introduce?
How many PATH entries did env -i give, against interactive?
What happened to the 24 runs, and how did the system handle it?
Which single line of that output was misleading, and where did it go?
Why did both halves of the digest fallback fail?
Why is an empty digest worse than an absent one?
What did rerun do when both digests were empty?
Why is b2b:1 CRASH → CRASH reproduced the most alarming line in that output?
State the general rule about fallback chains.
What three things must a nightly driver do about its environment?
Why exit 3 for a missing simulator rather than letting the runs crash?
Why does the crontab line call a wrapper?
Why >> file 2>&1 and not 2>&1 >> file?
Why append rather than truncate?
What is special about % in a crontab command?
The pass counts went 9, 13, 10. What does the per-spec comparison add?
Why is a trend table more dangerous than a report?
What makes a trend across archived nights possible?
Why does the resolver fail rather than guess, and what does the caller print?
151 files, 8,585 bytes of content — how much disk, and why?
Why does tar alone barely help?
What does tar.gz achieve, and why?
What does deleting the passing runs' logs save, and why so little?
State the retention policy in three rules.
What is the one artifact worth pruning, and what is the general rule?
Why is the archive read back before the directory is deleted?
How was that verification proven able to refuse?
Why refuse to archive an unfinished regression?
Why compare entry counts rather than file counts, and what happens if you get it wrong?
What does -mtime +90 skip, and when does that matter?

Answers

The environment it inherits is not the one you tested in, and ninety nights of output will not fit.
Five against fifteen — and eleven in a login shell.
All 24 crashed because vvp was not on the PATH. The classifier called them CRASH rather than FAIL, the report put them in the environment column, and the status was 1 — nothing lied.
md5sum: command not found, on standard error, which cron would mail to an unread address.
md5 is in /sbin, which is not on a cron PATH, and md5sum is the GNU spelling and does not exist on this BSD userland at all.
Because the field is present and looks filled in, so nothing downstream treats it as missing.
Nothing — "" != "" is false, so the guard passed and it reported a perfect reproduction of a regression in which nothing ran.
Because b2b is a test that passes. It is being reported as a reproduced crash.
A fallback chain with no floor turns a missing tool into a missing value and a missing value into a silent wrong answer. a || b where both can fail needs a third branch, and that branch is a refusal.
Source the setup rather than inheriting it; assert the tools are reachable; and never depend on a tool's location being on PATH when you can name it.
Because a missing simulator is a setup question, not a design one, and 24 CRASH rows invite somebody to inspect 24 testbenches.
So the environment setup, tool assertions and retention policy live somewhere testable and visible, rather than inside a crontab entry.
Because 2>&1 first points stderr at the current stdout — the terminal — before stdout is redirected, so diagnostics never reach the file.
So last night's output survives tonight's job failing in its first second.
It is special to cron and needs escaping as \%, which bites as soon as a date format enters the command.
That only two specs changed between nights two and three, and that twelve specs left the matrix and twelve arrived — so the counts are not comparable.
Because a chart is more persuasive than a report and nobody checks a chart's denominators.
Keeping results.tsv outside the archive, and a resolver that reads either a live directory or an archived summary transparently.
So a night with no results is not silently skipped; the caller prints (no results) so the gap is visible.
618,496 bytes — 72× — because 151 mostly-tiny files each occupy a whole filesystem block.
Because it pads every file to a 512-byte record, so 151 tiny files are still 151 records plus headers.
2.3% of the on-disk size, a factor of 42, because the content is highly repetitive text.
About 6%, because the passing runs have the smallest logs and compression finds its win in exactly the repetition that many similar files provide.
Compress every finished regression immediately; keep results.tsv and env.txt unpacked beside it; and delete whole regressions on age rather than files within them.
Waveforms — they are megabytes per run and do not compress well. The general rule is to prune by size measured, not by category assumed.
Because an archive nobody has opened is a claim rather than a backup, and this is the one script whose bug is unrecoverable.
By making tar exclude the logs: it reported 151 files in, 127 in the archive, exited 1, and left the directory in place.
Because archiving is how a regression stops being re-runnable, and a run that produced no results is exactly the one somebody will want.
Because find -type f excludes directories and tar -tzf lists them, so comparing the two raw counts fails every time — and a verification that always fails gets switched off, which is no verification by a more annoying route.
The file that is exactly 90 days old, since -mtime +n is strictly greater than n. Negligible at 90 days, significant at 7.

7. Remember

A scheduled job gets five PATH entries where you had fifteen. env -i reproduces that for free, locally, exactly.
The simulator was not on a cron PATH — 24 runs, 24 CRASH, and the classifier, report and status all handled it correctly.
md5 is in /sbin and md5sum does not exist here, so an unguarded fallback produced an empty digest, reported only on stderr.
Two empty digests compare equal, so the reproduction guard passed and declared 24 crashes "reproduced" — including tests that pass.
A fallback chain with no floor needs a third branch, and it is a refusal.
Assert the tools before running anything and exit with the setup status, not 24 crashes.
cron calls a wrapper, redirects with >> file 2>&1 in that order, appends rather than truncates, and escapes %.
Print the matrix change beside a trend's counts. A trend over a changing matrix is two measurements on one axis, and a chart is more persuasive than a report.
Keep results.tsv outside the archive, so a year of trend data is ninety small files and no unpacking.
151 files, 8.5 KB of content, 604 KB on disk — 72×. tar.gz is 2.3%; deleting the passing logs is 6%. Compress, do not prune.
Delete whole regressions on age, because half a regression answers nothing. Prune waveforms only, and by measured size.
Read the archive back and count before deleting, refuse to overwrite, refuse to archive an unfinished run — and compare entry counts, not file counts.
-mtime +90 skips day 90.
A nightly flow decays without being edited. Chapter 35.8 measures the decay that prints the same line and the same status while two of five blocks stop being tested.

8. What Is Next

Module 32 has turned Module 31's mechanics into a system that runs unattended, decides each verdict from evidence rather than from one number, records enough to act on, returns a status a machine can gate on, and keeps a year of history in a few megabytes.

What it cannot do is tell you why a run failed. Every class this module reports — FAIL, TIMEOUT, CRASH, NO_VERDICT, INCONSISTENT, VACUOUS — names a situation and then hands you a log.

Module 33, Debugging Failed Jobs and Broken Flows, is about the case where the flow itself is what is broken: triaging a failure to a layer before investigating it, reproducing one failing run by hand exactly, finding why a command works at your prompt and fails inside the script, and using tracing to recover what a long script actually did rather than what it was written to do.

Where this fits

Part of the Linux & Bash curriculum.