Isolation: One Directory and One Log per Run

Part II · Module 31 · Chapter 31.6 · Isolation: One Directory and One Log per Run

Chapter 31.5 put eight runs in flight. Every one of them writes a log, a results line, a waveform and a status, and the whole of this chapter follows from one observation: a name that two runs share is a name through which they can destroy each other.

None of the failures below produce an error. All three were measured on this machine.

1. The Results File Everybody Writes

The natural way to accumulate results is one line per run, appended to a shared file:

printf '%s\t%s\t%s\n' "$spec" "$verdict" "checks=$n" >> results.tsv

Two hundred runs at sixteen slots, each appending one such line:

$ seq 1 200 | xargs -P 16 -I{} bash -c \
    'printf "%s\t%s\t%s\t%s\n" "job{}" "PASS" "checks=3" "seed={}" >> results.tsv'
  200 appends with -P 16 -> 200 lines, 200 well-formed (4 fields)

Two hundred for two hundred. The append is a single write with the file opened in append mode, and the kernel serialises those against the end of the file, so the lines interleave in an arbitrary order but none of them is cut in half.

It is very tempting to stop here and conclude that appending is safe.

2. The Same Thing, Built in Pieces

Real reporters do not emit the line in one printf. They emit a field, then look something up, then emit another field — because the line is assembled as the information arrives:

printf '%s\t' "$spec"    >> results.tsv
printf '%s\t' "$verdict" >> results.tsv
printf '%s\n' "seed=$s"  >> results.tsv

The same two hundred jobs at the same sixteen slots:

=== B. the same, but each line built with SEVERAL appends (the real shape) ===
  200 three-part appends -> 200 lines, 192 with exactly 3 fields
  malformed lines: 8

Eight corrupt rows out of two hundred. Here they are, with tabs shown as ^I:

job18^Ijob19^IPASS^IPASS^Iseed=18
seed=19
job97^Ijob98^IPASS^IPASS^Iseed=97
seed=98
job112^IPASS^Ijob113^Iseed=112
PASS^Iseed=113
job193^IPASS^Ijob194^Iseed=193
PASS^Iseed=194

Four collisions, each producing two broken lines. Read the first pair: job 18 wrote its name, job 19 wrote its name before job 18 got to its verdict, then both verdicts landed, then job 18's newline ended the line — so one row claims to be job18 job19 PASS PASS seed=18 and the next row is the orphaned remainder seed=19.

Nothing failed. Every printf succeeded. Every job exited 0.

3. One Append Is Not Safe Either — It Depends on the Length

§1's conclusion was that a single printf is atomic. That holds only while the line is short. The same 200 jobs, one printf each, at four line lengths:

=== one printf per job, at four line lengths, -P 16, 200 jobs ===
  line bytes  lines   well-formed malformed  note
  60          200     200         0          intact
  1024        200     190         10         TORN
  8192        200     170         30         TORN
  65536       200     140         60         TORN

A 60-byte line survived 200 of 200. A 1 KB line tore ten times. A 64 KB line tore sixty times — nearly a third of the file.

And the line count is 200 in every row of that table, including the one where 60 of the lines are wrong.

The practical rule is not "short appends are safe", because "short" is a property of the buffer, not of your results format, and a results line grows: somebody adds a field, somebody puts the failing assertion's message in it, and the line that was 60 bytes is 2000. Treat a shared append file as unreliable regardless of line length — 200 of 200 at 60 bytes is evidence of nothing stronger than "I did not observe tearing at this length on this run".

4. The Shape That Cannot Race

One file per run, concatenated afterwards:

$ seq 1 200 | xargs -P 16 -I{} bash -c \
    'printf "%s\t%s\t%s\n" "job{}" "PASS" "seed={}" > parts/{}.tsv'
$ cat parts/*.tsv > results3.tsv
  200 separate files -> 200 lines, 200 with exactly 3 fields, 0 malformed

Zero malformed, and it is not luck: no two jobs name the same file, so there is nothing to serialise. The ordering is lost — parts/10.tsv sorts before parts/2.tsv — which is a sorting problem, and sorting problems are fixable after the fact in a way that corrupted rows are not.

That is the general shape and it is worth stating as a rule. Parallel jobs should write to private names and be merged by a single process afterwards. Everything else in this chapter is that rule applied to a different artifact.

# write
printf '%s\t%s\t%s\n' "$spec" "$verdict" "checks=$n" > "$RUNS/$name/result.tsv"

# merge, once, after the regression -- sorted by the matrix order, not by filename
while IFS= read -r spec; do
  t=${spec%%:*}; s=${spec#*:}
  cat "$RUNS/${t}_s${s}/result.tsv" 2>/dev/null
done < specs.txt > results.tsv

Note that the merge walks the matrix, not the directory. A directory scan cannot know which runs are missing, which is Chapter 31.7's subject.

5. The Shared Run Directory

A results line is the small case. The large one is the run directory, because that is where the log, the waveform and the artifacts live.

A wrapper that names the directory after the test — forgetting that Chapter 31.3 made the seed part of the run's identity:

d="$RUNS/$test"            # the defect: the seed is not in the name

Four seeds of one test, at four slots:

=== 4 seeds into ONE shared directory, each recording its own verdict ===
  run directories: 1 (expected 4)
  log files:       1 (expected 4)
  status files:    4
  the four recorded verdicts:
    random-1       PASS
    random-2       PASS
    random-3       PASS
    random-4       PASS
  the one surviving log, in full:
    TB  test=random seed=3 cycles=120000 verbosity=LOW
    TB  TEST PASSED  test=random checks=29746 errors=0
    tb/tb_arb.v:83: $finish called at 1497485 (1s)

Four runs, one log. The log is seed 3's — seed=3 is in its first line — and it is the only evidence that exists. Seeds 1, 2 and 4 ran, consumed four seconds of machine, and left nothing behind.

The four status files are the dangerous part. Each one is a real file written by a real run, and each says PASS. But all four wrappers computed their verdict by grepping $d/run.log, which is one file, so random-1 PASS is a statement about seed 1 derived from seed 3's output. The four verdicts agree here only because all four seeds happen to pass.

6. The Same Collision, With a Failing Run in the Matrix

The directory does not have to be keyed badly to collide. The commonest real version is a wrapper that writes every run into one fixed place:

d="$RUNS/latest"           # the defect: one directory for every run

Four specs, two of which fail. First the ground truth, one at a time, each in its own directory:

=== ground truth, run one at a time in its own directory ===
  fairness:1       PASS
  error_inject:1   FAIL
  fairness:2       PASS
  error_inject:2   FAIL

error_inject fails for a real and ordinary reason — this testbench does not implement a test by that name, so it reports ERROR unknown test 'error_inject' and fails, which is the behaviour Chapter 31.2 argued a missing test should have.

Now the same four at four slots, sharing RUNS/latest:

=== the same four, -P 4, all sharing RUNS/latest ===
  recorded verdicts:
    error_inject-1   FAIL
    error_inject-2   FAIL
    fairness-1       FAIL
    fairness-2       FAIL
  log files: 1
  the one surviving log:
    TB  test=error_inject seed=1 cycles=120000 verbosity=LOW
    TB  ERROR unknown test 'error_inject'
    TB  TEST FAILED  test=error_inject checks=0 errors=1
    tb/tb_arb.v:83: $finish called at 25 (1s)

Both passing tests were recorded as failures. fairness:1 and fairness:2 pass when run properly; here they are FAIL, because they read their verdict out of error_inject's log.

That is the whole hazard in one table. Not lost evidence — wrong answers, in a results file that looks completely normal, on a regression where every job exited and the launcher reported in the usual way.

7. The Fix, and What It Has to Include

d="$RUNS/${test}_s${seed}"

The same four seeds from §5, with the seed in the name:

=== the same 4 seeds, one directory each (the fix) ===
  run directories: 4
  log files:       4
  the four verdicts, each from its own log:
    random_s1        PASS 0 0
    random_s2        PASS 0 0
    random_s3        PASS 0 0
    random_s4        PASS 0 0

Four directories, four logs, four verdicts each derived from its own run's output.

The name must contain every axis the matrix varies. That is the operational statement of Chapter 31.3's identity rule: if the matrix crosses tests, seeds and configurations, then all three are in the directory name, because two runs that differ in an axis the name omits will share a directory.

# every axis in the name, in a fixed order
d="$RUNS/${test}_s${seed}_${cfg}"

Four more things have to be inside that directory, not beside it.

The log, which Chapter 29.1 showed is the only record of what a run did.

The waveform. A dump file is the easiest collision to miss, because its name is often fixed inside the testbench rather than passed in — $dumpfile("dump.vcd") with no path writes to the current directory, which is why Chapter 27.6 runs each simulation from inside its own run directory. Eight parallel runs with one dump.vcd and no cd produce one corrupt dump and seven lost ones.

Any scratch or temporary file the tool creates, for the same reason.

And the status, which the next chapter builds.

8. What Is Legitimately Shared

Not everything needs a copy per run, and making one is its own mistake — eight copies of a compiled image is eight times the disk and eight times the build.

The distinction is read-only versus written. Anything a run only reads can be shared by all of them; anything a run writes needs a name no other run will use. A compiled image is the clearest case of the first, and Chapter 27.1's separation of compile from run is what makes it possible.

The one thing to check about the shared read-only artifacts is that nothing rebuilds them while the regression is running. A launcher that compiles and then immediately runs 240 simulations against the image is fine; a launcher that recompiles per run into the same output path is the §6 collision again, with the compile as the victim.

Practice

1. Append one line per job from 200 parallel jobs and count the well-formed rows.
2. Split the line into three appends and count again.
3. Print the field-count histogram and confirm the line total is unchanged.
4. Write the count-based check and watch it pass on the corrupt file.
5. Write the shape check and watch it fail.
6. Repeat the single-append test at 1 KB, 8 KB and 64 KB and find where it tears.
7. Explain why "short appends are safe" is not a rule you can rely on.
8. Make your harness fail when it produces no output, and say why that is not optional.
9. Switch to one file per job and concatenate; confirm zero malformed rows.
10. Merge by walking the matrix rather than the directory, and say what that buys.
11. Key a run directory on the test only and run four seeds in parallel.
12. Count the logs, then read the one that survived and find whose it is.
13. Check the recorded verdicts and explain where each one came from.
14. Repeat with a failing test in the matrix and find the false failures.
15. Argue the other direction — the same defect reporting a failure as a pass.
16. Add every axis to the directory name and re-run.
17. Find a fixed dump file name in a testbench and predict what eight parallel runs do to it.
18. Break a relative argv with a cd, then fix it by resolving the root first.
19. Classify your flow's artifacts into shared-read-only and per-run.

Check Your Understanding

1. Why did 200 single-printf appends of short lines all survive?
2. What changed in §2 to produce eight corrupt rows?
3. Trace what happened to jobs 18 and 19.
4. Why does the line count come out right anyway?
5. What check does see the corruption?
6. At what line length did the single append start tearing, and how badly at 64 KB?
7. Why is "keep the line short" not a usable rule?
8. Why is 200-of-200 at 60 bytes weak evidence?
9. How did my own probe report "intact" on an empty file?
10. What were the two root causes of that false green?
11. Why can one file per job not race?
12. What does one-file-per-job cost, and why is that cost acceptable?
13. Why should the merge walk the matrix rather than the directory?
14. In §5, how many logs did four runs produce, and whose was it?
15. Where did random-1 PASS come from, and why is it not trustworthy?
16. In §6, what were the four true verdicts and the four recorded ones?
17. Why is the direction of that error not something you can rely on?
18. What single observable in §6's output indicated that anything was wrong?
19. What must a run directory's name contain?
20. Why is a waveform the easiest collision to miss?
21. Why must $root be absolute before the cd?
22. What is the test for whether an artifact can be shared?
23. What must not happen to a shared read-only artifact during a regression?

9. Remember

• A name two runs share is a name through which they destroy each other. Every failure in this chapter is that sentence.
• One short append survived 200 of 200; three appends per line produced 8 corrupt rows; and a single 64 KB append tore 60 times.
• The line count was correct in every case. A count cannot see corruption inside a line — check the field count per row.
• Treat a shared append file as unreliable at any length. "Short" is a property of the buffer, not of your format.
• One file per run, merged by one process afterwards, cannot race. You lose ordering, which is recoverable; you do not lose rows.
• Merge by walking the matrix, because a directory scan cannot know what is missing.
• Four runs sharing a directory produced one log — and four status files, each derived from somebody else's output.
• With a failing test in the matrix, two passing tests were recorded as FAIL. The direction is a coin flip; the other side of it is a failure reported as a pass.
• Put every axis in the run directory's name, and put the log, the waveform, the scratch files and the status inside it.
• Run from inside the directory to catch the names you do not control — and resolve the root to an absolute path first, or the cd breaks the argv.
• Share what runs only read, isolate what runs write, and make sure nothing rebuilds a shared artifact mid-regression.

10. What Is Next

Every run now has its own directory and writes its own verdict there. Chapter 31.7 turns 240 of those into one answer — and finds a reporter that said 7 runs found, 7 PASS, exit 0 about a regression of eight, and another that reported success for a regression in which nothing ran at all.