Creating Run Directories Idempotently

Part II · Module 17 · Chapter 17.5 · Creating Run Directories Idempotently

1. mkdir Is Not Enough

use strict;
use warnings;
use File::Path qw(make_path);
use File::Temp qw(tempdir);
use File::Spec;

my $base = File::Spec->catdir(tempdir(CLEANUP => 1), "runs");
my $show = sub { (my $p = shift) =~ s{^.*/runs/?}{}; return $p eq "" ? "runs/" : $p };

# ---- 1. first creation: three levels at once
my @made = make_path(File::Spec->catdir($base, "alu_1", "logs"), { error => \my $e1 });
printf "1. first make_path   -> created %d dir(s), %d error(s)\n",
    scalar @made, scalar @{ $e1 || [] };

# ---- 2. again, on a path that now exists
my @again = make_path(File::Spec->catdir($base, "alu_1", "logs"), { error => \my $e2 });
printf "2. second make_path  -> created %d dir(s), %d error(s)   <- idempotent\n",
    scalar @again, scalar @{ $e2 || [] };

# ---- 3. a FILE sitting where a directory should be
open my $w, ">", File::Spec->catfile($base, "blocker") or die "cannot write: $!\n";
close $w;
my @blocked = make_path(File::Spec->catdir($base, "blocker", "logs"), { error => \my $e3 });
printf "3. a file in the way -> created %d, %d error(s):\n", scalar @blocked, scalar @{ $e3 || [] };
for my $item (@{ $e3 || [] }) {
    my ($file, $message) = %$item;
    printf "      %-16s %s\n", $show->($file), $message;
}

# ---- 4. the return value names what was actually created
my @deep = make_path(File::Spec->catdir($base, "x", "y", "z"));
printf "4. three new levels  -> created %d: %s\n", scalar @deep,
    join(" ", map { $show->($_) } @deep);

# ---- 5. plain mkdir, for contrast
my $ok = mkdir File::Spec->catdir($base, "alu_1");
printf "5. plain mkdir on an existing dir -> %s%s\n",
    ($ok ? "ok" : "FAILED"), ($ok ? "" : " ($!)");

1. first make_path   -> created 3 dir(s), 0 error(s)
2. second make_path  -> created 0 dir(s), 0 error(s)   <- idempotent
3. a file in the way -> created 0, 2 error(s):
      blocker          File exists
      blocker/logs     Not a directory
4. three new levels  -> created 3: x x/y x/y/z
5. plain mkdir on an existing dir -> FAILED (File exists)

Read row 5 first, because it is the reason this chapter exists. mkdir on a directory that already exists fails, with File exists in $!. So this, which is what people write:

mkdir $rundir or die "cannot create $rundir: $!\n";

works perfectly the first time and kills the job on every re-run. And mkdir creates one level, so runs/alu_1/logs needs two calls in the right order.

make_path fixes both. Row 1 created three directories in one call; row 2 on the same path created zero and reported no error at all. That is what idempotent means here: calling it twice is not a failure.

2. The Return Value Is the Useful Part

make_path returns the list of directories it actually created, which answers a question a script needs to ask.

• Row 1 returned 3 — three levels did not exist.
• Row 2 returned 0 — everything was already there.
• Row 4 returned x, x/y, x/y/z — the three it made, in creation order.

So you can tell a fresh run from a re-run without a separate -d test:

my @created = make_path($rundir, File::Spec->catdir($rundir, "logs"));
if (@created) {
    print STDERR "fresh run directory: $rundir\n";
}
else {
    print STDERR "re-using existing run directory: $rundir\n";
    # ... and now DECIDE, rather than carrying on by accident
}

That else branch is the whole decision, and section 3 is about it.

3. Not Overwriting Last Night's Evidence

A run directory that already exists means one of three things, and they need different responses.

A re-run after a crash. Normal, and the right response is usually to clear the directory and start clean — because a half-written log mixed with a new one is worse than either. Chapter 17.6 is how to clear it safely.

A second run of the same test at the same seed. The output would be identical, so reusing the directory loses nothing — unless the first run is the one somebody is looking at right now.

A different run that collided in naming. This is the bug, and it is a naming bug rather than a directory bug.

The fix for the third case is in the name. runs/<test> collides the moment you run two seeds. The convention from Chapter 17.2 exists for this:

my $rundir = File::Spec->catdir($RUNS, "${test}_${seed}");

And if runs can overlap in time — a nightly regression and somebody's interactive debug of the same test and seed — add something that distinguishes the invocation:

# only when runs can genuinely overlap: a job id beats a timestamp
my $tag    = $ENV{LSB_JOBID} // $ENV{SLURM_JOB_ID} // $$;
my $rundir = File::Spec->catdir($RUNS, "${test}_${seed}_$tag");

Note what that uses and what it avoids. A scheduler job id is stable, unique and meaningful — you can find the job later. $$, the process id, is the fallback. A timestamp is the tempting choice and the wrong one, because it makes the directory name different on every run, which means two runs of the same test can never be compared by path and Chapter 17.8's reproducibility argument is lost.

4. When Creation Legitimately Fails

Row 3 is the case worth understanding, because the error message is a pair rather than one line:

runs/blocker       File exists
runs/blocker/logs  Not a directory

A file named runs/blocker was in the way. make_path reported two failures: it could not create blocker because something is already there, and it could not create blocker/logs because the thing in the way is not a directory. Both are true and the second explains the first.

This happens in practice when a tool writes a file where a later step expects a directory — commonly a log written as runs/alu_1 instead of runs/alu_1/sim.log.

The other failures to expect, and what each means:

Permission denied. The run area is not writable by this user. Check it once at startup with -w rather than discovering it on directory ninety — Chapter 17.1's validation argument, and Chapter 13.6's file tests.

No space left on device. Creation succeeds and the first write fails, so a -w test does not catch it. There is no clean way to pre-empt this; the answer is to check the write, per Chapter 13.5.

The parent does not exist and cannot be created. make_path creates intermediate directories, so this only happens when a parent is a dangling symlink or an unmounted filesystem — and the message names which component failed, which is why the per-file error list is worth printing.

A startup check that prevents most of it:

die "run area does not exist: $RUNS\n"   unless -d $RUNS;
die "run area is not writable: $RUNS\n"  unless -w $RUNS;

5. What To Remember

• Measured: plain mkdir on an existing directory FAILS with File exists, so mkdir ... or die kills every re-run.
• mkdir makes one level; runs/alu_1/logs needs two calls in order.
• Measured: make_path created three levels in one call, and on the second call created 0 with no error — idempotent.
• The return value is the list of directories actually created. Zero means it was already there.
• Use that instead of a separate -d test to tell a fresh run from a re-run.
• Pass { error => \my @err }, or make_path croaks and you cannot say which directory failed.
• Each error item is a one-key hash — unpack with my ($file, $msg) = %$item;.
• An existing run directory means one of three things: a crash re-run, a duplicate run, or a naming collision.
• Name directories <test>_<seed>, or two seeds collide.
• Add a scheduler job id only if runs can overlap — and not a timestamp, which makes every run's path different and destroys comparability.
• Put the re-run policy behind --clean / --force and refuse by default with exit 2.
• Measured: a file in the way gives two errors — File exists and Not a directory, the second explaining the first.
• Check -d and -w on the run area at startup, not on directory ninety.
• -w cannot catch a full filesystem — check the write itself.

Chapter 17.6 is the other half, and the one rated Critical. Clearing a run directory means deleting files, and a deletion that is slightly too broad is the one mistake in this module that cannot be undone.