Linux & Bash · Part II · Module 28 · Run Directories, Generated Files and Workspace Hygiene

Archiving a Failure Before It Is Cleaned

A retention policy will delete a run somebody still needs, so something has to run first and keep enough. This chapter establishes what enough means by testing it the only way that counts — archiving, deleting the original, and reproducing the failure from what was kept — and finds a tar option that silently keeps fifteen thousand times too much.

Intermediate21 min readLinuxPart IIArchivingtarReproducibility

Part II · Module 28 · Chapter 28.5 · Archiving a Failure Before It Is Cleaned

Chapter 28.4's cleanup is correct, guarded and scheduled, and in three weeks it will delete the run directory containing the failure somebody is still arguing about. Something has to run before it. The interesting question is not how to make an archive — it is how to know the archive is sufficient, and that has an answer you can test.

1. What "Enough" Means, Operationally

Chapter 28.2 classified one run's artifacts and found the sizes and the values ordered oppositely. The archive is where that classification pays:

artifact	keep?	why
`cmd.args`, `tool.txt`, `env.txt`, `status.txt`	yes	nothing can regenerate them
`run.log`, `compile.log`	yes	the verdict; a rerun is a different run
the sources the run actually read	yes	the tree will move on
`sim.vvp` and other compiled artifacts	no	rebuildable in under a second
`dump.vcd` / `dump.fst`	no	reproducible, and the largest thing by far

The last two rows are where the discipline is. Keeping the dump is the instinct — it is the thing you would open in a viewer — and it is the wrong call, because it is both the biggest artifact and a reproducible one. Chapter 27.6 measured the reproduction: a windowed re-dump at the failure time costs 6% of the runtime and 23 KB.

But the table is a claim, and the claim needs testing. §3 tests it.

2. The `tar` Option That Does Nothing

First, a defect that invalidates the archive before any of this matters. Two commands differing only in word order:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ tar czf ar/before.tgz --exclude='*.vcd' --exclude='*.vvp' -C runs fail_s1
$ tar czf ar/after.tgz  -C runs fail_s1 --exclude='*.vcd' --exclude='*.vvp'
$ for f in before after; do
>   printf '  --exclude %-7s %7s bytes %2s entries   sim.vvp? %s  dump.vcd? %s\n' "$f" \
>     "$(stat -f '%z' ar/$f.tgz)" "$(tar tzf ar/$f.tgz | wc -l | tr -d ' ')" \
>     "$(tar tzf ar/$f.tgz | grep -c 'sim.vvp')" "$(tar tzf ar/$f.tgz | grep -c 'dump.vcd')"
> done
  --exclude before      1326 bytes  7 entries   sim.vvp? 0  dump.vcd? 0
  --exclude after       4614 bytes  9 entries   sim.vvp? 1  dump.vcd? 1

--exclude placed after the path operand is silently ignored. No warning, no error, exit status 0. The archive is simply larger and contains the two things you asked to leave out.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ tar --version
bsdtar 3.5.3 - libarchive 3.7.4 zlib/1.2.12 liblzma/5.4.3 bz2lib/1.0.8

Scale it to a realistic dump — 60 MB of incompressible data, because zeros would flatter the comparison:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ dd if=/dev/urandom of=runs/fail_s1/huge.vcd bs=1m count=60
$ du -sh runs/fail_s1
 60M	runs/fail_s1
$ tar czf ar/kept.tgz --exclude='*.vcd' --exclude='*.vvp' -C runs fail_s1
$ tar czf ar/oops.tgz -C runs fail_s1 --exclude='*.vcd' --exclude='*.vvp'
$ tar czf ar/all.tgz  -C runs fail_s1
$ for f in kept oops all; do
>   printf '  %-6s %9s  %2s entries\n' "$f" "$(du -h ar/$f.tgz | awk '{print $1}')" \
>          "$(tar tzf ar/$f.tgz | wc -l | tr -d ' ')"
> done
  kept        4.0K   7 entries
  oops         60M  10 entries
  all          61M  10 entries

4.0 KB intended, 60 MB archived — and oops is indistinguishable from all, which is the real cost: a retention policy that thinks it keeps 4 KB per failure and actually keeps 60 MB will fill the archive area and then somebody will turn archiving off.

Why this one is worth a guard rather than a habit

"Put the options first" is correct and insufficient, because the failure is silent and the command looks right. Nobody re-reads a working tar line.

The check is one line, and it belongs in the archive script rather than in your memory:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# assert the archive does NOT contain what we excluded
if tar tzf "$out" | grep -qE '\.(vcd|fst|fsdb|vvp)$'; then
  echo "archive-failure: $out contains excluded artifacts -- check option order" >&2
  exit 1
fi

That is the same principle as Chapter 27.8's cross-check: the command reported success, so the only way to know what it did is to inspect the artifact it produced. An option that is silently ignored can only be caught by looking at the result.

3. The Test That Establishes Sufficiency

Now the claim from §1. Build an archive of the record plus the sources it names, and nothing else:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ tar cf ar/failure.tar --exclude='*.vcd' --exclude='*.vvp' -C runs fail_s1
$ tar rf ar/failure.tar faults/arb_bad.v tb/tb_arb.v filelists/sim.f
$ gzip -f ar/failure.tar
$ printf '  archive: %s, %s entries\n' "$(du -h ar/failure.tar.gz | awk '{print $1}')" \
>        "$(tar tzf ar/failure.tar.gz | wc -l | tr -d ' ')"
  archive: 4.0K, 10 entries
$ tar tzf ar/failure.tar.gz
fail_s1/
fail_s1/env.txt
fail_s1/run.log
fail_s1/tool.txt
fail_s1/compile.log
fail_s1/run.args
fail_s1/compile.args
fail_s1/status.txt
faults/arb_bad.v
tb/tb_arb.v

Then destroy the original, which is the step that makes this a test rather than a demonstration:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ rm -rf runs/fail_s1
$ [ -d runs/fail_s1 ] && echo yes || echo no
no

Restore somewhere unrelated and reproduce:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ mkdir -p /tmp/restore && tar xzf ar/failure.tar.gz -C /tmp/restore
$ ( cd /tmp/restore && iverilog -o sim.vvp faults/arb_bad.v tb/tb_arb.v > c.log 2>&1 \
>     && vvp sim.vvp +TESTNAME=smoke +SEED=1 +STRICT ) | grep -E 'ERROR|TEST'
TB  ERROR t=45 gnt=10 expected 01
TB  TEST FAILED  test=smoke checks=3 errors=1

And compare against what the original run recorded, read straight out of the archive without unpacking it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ tar xzf ar/failure.tar.gz -O fail_s1/run.log | grep -E 'ERROR|TEST'
TB  ERROR t=45 gnt=10 expected 01
TB  TEST FAILED  test=smoke checks=3 errors=1

Identical. Same error, same time, same check and error counts. 4 KB preserved what a 60 MB archive would have, and the 60 MB artifact was the one thing not needed, because it was the one thing reproducible.

What the test does and does not establish

It establishes that this failure is reproducible from this archive in this environment, today. It does not establish that it will be in a year, and the reasons are the ones Chapter 28.1 set out: reproducibility is a property of the environment.

So the archive should carry the evidence needed to detect that, which is exactly what tool.txt and env.txt are for. A year from now the replay may produce a different answer, and the useful outcome is not "it still works" — it is knowing which of the two runs to believe, which needs the recorded tool path, version and environment from Chapter 27.9.

And one thing no archive can carry: if the failure depends on a machine, a licence server, or a filesystem that no longer exists, the archive preserves the question and not the answer. Which is a reason to write down the conclusion of a debug somewhere durable, not only its inputs.

4. Keeping the Sources: the Record Is Per Stage, Not Per Run

The archive above includes three source files and I chose them by hand, which does not scale. The run recorded its own command — so derive the list from that:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat runs/fail_s1/cmd.args
vvp
sim.vvp
+TESTNAME=smoke
+SEED=1
+STRICT
+DUMP
+DUMP_FILE=dump.vcd

There are no source files in it. Not one. And that is not a gap in the record — it is correct: this is the run command, and the run does not read source. The sources were arguments to the compile, which Chapter 27.1 established as a separate stage with its own argument list. A record of one command cannot describe a flow of three.

So the first version of the archive script, which read cmd.args and added every path it found, silently produced an archive with no sources at all — 7 entries where 10 were intended, exit status 0, no warning.

There is a second, independent reason it found nothing. sim.vvp is a real path in that list, and it did not resolve:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ while IFS= read -r a; do
>   case "$a" in (-*|+*) continue ;; esac
>   printf '  %-24s cwd:%-4s rundir:%-4s\n' "$a" \
>     "$( [ -f "$a" ] && echo yes || echo no )" \
>     "$( [ -f "runs/fail_s1/$a" ] && echo yes || echo no )"
> done < runs/fail_s1/cmd.args
  vvp                      cwd:no   rundir:no
  sim.vvp                  cwd:no   rundir:yes

The recorded paths are relative to the directory the stage ran in, which for the run stage is the run directory (Chapter 27.3). Resolved from the project root they do not exist. This is the cwd-relative rule from Chapter 26.6 arriving for the third time in two modules, and it will keep arriving: a relative path in a record is meaningless without the directory it was recorded against.

Both fixes are structural. Record one argv file per stage:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# in the wrapper, per stage, not once per run
: > "$rundir/compile.args"
for a in iverilog -o "$rundir/sim.vvp" faults/arb_bad.v tb/tb_arb.v; do
  printf '%s\n' "$a" >> "$rundir/compile.args"
done

and resolve against both bases, over every stage file, excluding the run's own artifacts:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

sources() {
  local rundir=$1 f a
  for f in "$rundir"/*.args; do
    [ -r "$f" ] || continue
    while IFS= read -r a; do
      case "$a" in (-*|+*|'') continue ;; esac
      if   [ -f "$a" ];         then printf '%s\n' "$a"
      elif [ -f "$rundir/$a" ]; then printf '%s\n' "$rundir/$a"
      fi
    done < "$f"
  done | sort -u
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ sources runs/fail_s1
faults/arb_bad.v
runs/fail_s1/sim.vvp
tb/tb_arb.v

$ sources runs/fail_s1 | grep -v "^runs/fail_s1/"
faults/arb_bad.v
tb/tb_arb.v

Exactly the two sources this failure needs, derived rather than chosen.

Why the filelist is not the authority either

The obvious alternative is to read the filelist, since that is what names the design. Chapter 23.3's flattener does it in a dozen lines:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ flatten filelists/sim.f
rtl/arb.v
tb/tb_arb.v

Correct, and wrong for this run. The failing run compiled faults/arb_bad.v, a mutant that appears in no filelist. The filelist describes what the project builds; the record describes what this run built, and a debug needs the second one.

The rule that generalises: derive the archive's contents from what the run recorded, never from what the project currently contains. By the time anybody opens the archive those two have diverged — which is the whole reason the archive exists.

5. The Archive Script

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# archive-failure RUNDIR [ARCHIVE_DIR] — keep enough of a failed run, verified.
set -u
rundir=${1:?usage: archive-failure RUNDIR [ARCHIVE_DIR]}
ardir=${2:-${FAILURE_ARCHIVE:-./archive}}

[ -d "$rundir" ] || { echo "archive-failure: no such run: $rundir" >&2; exit 2; }
[ -r "$rundir/run.log" ] || { echo "archive-failure: $rundir has no run.log" >&2; exit 2; }

# only archive things that actually failed -- an archive of passes is noise
if grep -q 'TEST PASSED' "$rundir/run.log"; then
  echo "archive-failure: $rundir passed; nothing to archive"; exit 0
fi

mkdir -p "$ardir" || exit 2
name=$(basename "$rundir")
out="$ardir/${name}.tar.gz"
[ -e "$out" ] && { echo "archive-failure: $out exists; refusing to overwrite" >&2; exit 2; }

tmp="$ardir/.${name}.$$.tar"
# EVERY --exclude goes BEFORE the path operand. Placed after, it is ignored.
tar cf "$tmp" \
    --exclude='*.vcd' --exclude='*.fst' --exclude='*.fsdb' --exclude='*.shm' \
    --exclude='*.vvp' --exclude='*.daidir' --exclude='work' \
    -C "$(dirname "$rundir")" "$name" || { rm -f "$tmp"; exit 1; }

# Add the sources the run named -- across EVERY stage, not just the run.
# The run command names no sources at all: those were arguments to the
# compile, which is a separate stage with its own argv (Chapter 27.1).
# And a recorded path may be relative to the run directory (Chapter 27.3),
# so try both bases.
srcs=$(for f in "$rundir"/*.args; do
         [ -r "$f" ] || continue
         while IFS= read -r a; do
           case "$a" in (-*|+*|'') continue ;; esac
           if   [ -f "$a" ];         then printf '%s\n' "$a"
           elif [ -f "$rundir/$a" ]; then printf '%s\n' "$rundir/$a"
           fi
         done < "$f"
       done | sort -u | grep -v "^$rundir/")
if [ -n "$srcs" ]; then
  printf '%s\n' "$srcs" | while IFS= read -r a; do tar rf "$tmp" "$a"; done
else
  echo "archive-failure: warning -- no source files identified from $rundir/*.args" >&2
fi

gzip -f "$tmp" && mv "$tmp.gz" "$out"

# VERIFY: the exclusions must have taken effect, and the log must be present
if tar tzf "$out" | grep -qE '\.(vcd|fst|fsdb|vvp)$'; then
  echo "archive-failure: $out contains excluded artifacts -- check option order" >&2
  exit 1
fi
tar tzf "$out" | grep -q "$name/run.log" || {
  echo "archive-failure: $out has no run.log -- refusing to call it an archive" >&2
  exit 1; }

printf '%s  %s  %s entries\n' "$out" "$(du -h "$out" | awk '{print $1}')" \
       "$(tar tzf "$out" | wc -l | tr -d ' ')"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ archive-failure runs/fail_s1 archive
archive/fail_s1.tar.gz  4.0K  10 entries
$ tar tzf archive/fail_s1.tar.gz
fail_s1/
fail_s1/env.txt
fail_s1/run.log
fail_s1/tool.txt
fail_s1/compile.log
fail_s1/run.args
fail_s1/compile.args
fail_s1/status.txt
faults/arb_bad.v
tb/tb_arb.v

Every refusal, in order:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ archive-failure runs/smoke_s1 archive
archive-failure: runs/smoke_s1 passed; nothing to archive      # exit 0
$ archive-failure runs/fail_s1 archive
archive-failure: archive/fail_s1.tar.gz exists; refusing to overwrite   # exit 2
$ archive-failure runs/nope archive
archive-failure: no such run: runs/nope                        # exit 2
$ archive-failure runs/nolog archive
archive-failure: runs/nolog has no run.log                     # exit 2
$ archive-failure runs/bare archive
archive-failure: warning -- no source files identified from runs/bare/*.args
archive/bare.tar.gz  4.0K  2 entries                           # exit 0, with a warning

Five decisions worth stating.

It refuses to archive a passing run, and exits 0 when it does. An archive of everything is an archive of nothing: the point is that the retained set is small enough to keep indefinitely.

It refuses to overwrite. An archive is append-only by nature; silently replacing one with a rerun's output destroys the thing it exists to protect.

It verifies the exclusions by listing the archive. §2 is the reason. The tar command reported success either way.

It verifies the log is present. This is the check that would catch the whole script having been pointed at an empty directory — an archive with no log is a file, not evidence, and calling it an archive is worse than failing.

It warns when it identified no sources, and still succeeds. The last transcript line is that case: a run with no .args files produces a 2-entry archive of the log alone. That is genuinely better than nothing — the verdict is preserved — so it is a warning rather than a refusal, and the warning is what tells you the wrapper is not recording its stages.

6. Where the Archive Goes, and When

Three operational points, all of which are ordering rather than code.

It must run before the cleanup. Chapter 28.4 said the same thing from the other side: the ordering lives in whatever schedules the two, and nothing in either script can check it. The cheapest insurance is to make the cleanup refuse to delete a failed run that has no archive:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# in the cleanup, before deleting
if grep -q 'TEST FAILED' "$d/run.log" 2>/dev/null \
   && [ ! -e "$FAILURE_ARCHIVE/$(basename "$d").tar.gz" ]; then
  printf 'SKIP   %s  (failed and not archived)\n' "$d"; continue
fi

It must go on a different filesystem from the runs area, or the archive is deleted by whatever fills the disk. This is the one requirement that cannot be satisfied by a script and has to be satisfied by whoever set up the project areas — Chapter 22.4 is where those areas were introduced.

Archiving converts many small files into one. Chapter 28.3 measured the per-file allocation floor at 4096 bytes, so a run of eight small files occupies 44 KB on disk and compresses into a 4 KB archive — a factor of eleven that has nothing to do with compression ratios and everything to do with block overhead. For thousands of retained failures that is the dominant effect.

Practice

Build two archives differing only in where --exclude appears and compare the entry counts.
Confirm the misplaced version exits 0 and prints no warning.
Add a 60 MB incompressible file and compare the two archive sizes.
Check whether the misplaced version differs from one with no excludes at all.
Write the verification line that lists the archive and fails if an excluded extension is present.
Archive a failing run's record plus its sources, then delete the run directory entirely.
Restore elsewhere and reproduce the failure. Compare against the archived log.
Read a file out of the archive without unpacking it, using tar -O.
Derive the source list from cmd.args instead of naming files by hand, and count how many sources it finds. Explain the number.
Record one argv file per stage and re-derive the list. Then check which of the recorded paths resolve from the project root and which only from the run directory.
Explain why the filelist is not the authority for what a run compiled.
Write archive-failure and confirm it refuses a passing run, refuses to overwrite, refuses an archive with no log, and warns when it finds no sources.
Make the cleanup skip a failed run with no archive, and test it by deleting the archive.
Compare the run's size on disk against its archive's, and work out how much of the difference is block overhead rather than compression.

Check Your Understanding

Which two artifacts should not be archived, and why is that counter-intuitive?
What happens to --exclude placed after the path operand, and what is the exit status?
How much was archived instead of 4.0 KB when the option was misplaced?
Why is the misplaced version worse than simply being large?
What is the only way to detect a silently ignored option?
What made §3 a test rather than a demonstration?
What did the restored archive reproduce, and what was it compared against?
What does the sufficiency test establish, and what does it not?
Why are tool.txt and env.txt worth archiving even though they are tiny?
How many source files did cmd.args yield, and why?
Why is the filelist not the authority on what a run compiled?
Why should the archive script refuse a passing run?
Why must it refuse to overwrite?
Why does archiving save more space than the compression ratio suggests?

Answers

The compiled artifact and the waveform dump. Counter-intuitive because the dump is the thing you would open in a viewer — and it is both the largest artifact and a reproducible one.
It is silently ignored: no warning, no error, exit status 0. The archive simply contains what you asked to leave out.
60 MB instead of 4.0 KB, and the result was indistinguishable from an archive with no exclusions at all.
Because a policy that believes it keeps 4 KB per failure and actually keeps 60 MB will fill the archive area, and the response to that is usually to switch archiving off.
Inspect the artifact. The command reported success either way, so only listing the archive reveals what it did.
Deleting the original. Without that step the reproduction could have been reading something still present.
TB ERROR t=45 gnt=10 expected 01 and TEST FAILED checks=3 errors=1, compared against the same two lines read out of the archived run.log with tar -O.
That this failure reproduces from this archive in this environment today. Not that it will in a year — reproducibility is a property of the environment.
Because when a future replay disagrees with the archived log, they are what tells you which of the two runs to believe.
None. cmd.args records the run command, and the run reads no source — the sources were arguments to the compile, a separate stage with its own argv. The first version of the script therefore built a 7-entry archive with no sources, exit status 0, no warning.
Because the run may have compiled something not in it — the failing run used faults/arb_bad.v, absent from sim.f. The recorded argv is the authority.
Because an archive of everything is an archive of nothing; the retained set has to be small enough to keep indefinitely.
Because an archive is append-only by nature, and silently replacing one with a rerun's output destroys the evidence it exists to protect.
Because of per-file block allocation: eight small files occupy 44 KB on disk and 4 KB in an archive, which is block overhead rather than compression.

7. Remember

Do not archive the dump or the compiled artifact. They are the largest things and the reproducible ones.
--exclude after the path operand is silently ignored in bsdtar 3.5.3 — exit status 0, no warning, and the result is identical to having no exclusions.
That cost 60 MB instead of 4.0 KB on a realistic dump.
Verify by listing the archive. A silently ignored option can only be caught by inspecting the artifact.
Prove sufficiency by deleting the original, restoring elsewhere, and reproducing. 4 KB reproduced the failure byte for byte.
Read from an archive without unpacking it with tar -O.
Record one argv file per stage. cmd.args for the run names no sources at all — those belonged to the compile — so a single-command record builds an archive with nothing in it.
Resolve recorded paths against both the cwd and the run directory. A relative path in a record is meaningless without the directory it was recorded against.
Derive the source list from what the run recorded, not from the current tree — and the recorded argv outranks the filelist.
Refuse a passing run, refuse to overwrite, and refuse an archive with no log.
Archive before cleaning, and make the cleanup skip a failed run that has no archive.
Put the archive on a different filesystem, or it dies with whatever filled the disk.
Archiving beats its compression ratio because it collapses per-file block overhead: 44 KB on disk became 4 KB.

8. What Is Next

Archiving one failure by hand scales into a retention policy in Chapter 32.8, which measures the surprising part: deleting the passing runs' logs saves 6%, and compressing the whole regression saves 97.7%.

The runs are inventoried, measured, cleaned and archived. Chapter 28.6 closes the module with the last place derived files cause trouble — a repository where they are committed, a review full of tool noise, and the .gitignore rule that does nothing because the file is already tracked.

Where this fits

Part of the Linux & Bash curriculum.

1. What "Enough" Means, Operationally

2. The tar Option That Does Nothing

3. The Test That Establishes Sufficiency

4. Keeping the Sources: the Record Is Per Stage, Not Per Run

5. The Archive Script

6. Where the Archive Goes, and When

Practice

Check Your Understanding

7. Remember

8. What Is Next

Where this fits

2. The `tar` Option That Does Nothing