Timestamps, File Ages and Stale Artifacts

Part II · Module 17 · Chapter 17.9 · Timestamps, File Ages and Stale Artifacts

1. The Check, and Why It Matters

The rule is one sentence: an output must be newer than every input it was produced from. If it is not, it is stale and nothing downstream should trust it.

my $out_mtime = (stat $report)[9];
my $in_mtime  = (stat $netlist)[9];
if ($out_mtime < $in_mtime) {
    print STDERR "stale: $report is older than $netlist\n";
    exit 3;                                 # 15.4: could not reach a verdict
}

stat returns a thirteen-element list and element 9 is the modification time, in seconds since the epoch. Element 8 is the access time and element 10 is the inode-change time; 9 is the one you want, and getting the index wrong gives a plausible number that means something else.

This is the same logic make uses, and the reason to write it yourself is that a flow usually has dependencies make does not know about: a report generated by a script that read three files, a netlist checked against a constraint file nobody listed, a coverage database merged from directories that come and go.

2. Resolution Decides Whether the Check Works

use strict;
use warnings;
use Time::HiRes qw(sleep);
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
my $path = sub { File::Spec->catfile($dir, shift) };
my $write = sub {
    my ($name) = @_;
    open my $w, ">", $path->($name) or die "cannot write: $!\n";
    print $w "content\n";
    close $w;
};

# ---- 1. what resolution does stat give?
$write->("a.txt");
my $m = (stat $path->("a.txt"))[9];
printf "1. stat element 9 is an integer?        %s\n", ($m == int($m) ? "yes" : "no");
my $hm = (Time::HiRes::stat($path->("a.txt")))[9];
printf "   Time::HiRes::stat has a fraction?    %s\n", ($hm != int($hm) ? "yes" : "no");

# ---- 2. two files written back to back, in the same second
$write->("b.txt");
my ($sa, $sb) = ((stat $path->("a.txt"))[9], (stat $path->("b.txt"))[9]);
my ($ha, $hb) = ((Time::HiRes::stat($path->("a.txt")))[9],
                 (Time::HiRes::stat($path->("b.txt")))[9]);
printf "\n2. a.txt then b.txt, written immediately after each other:\n";
printf "   stat        : difference %d s        -> is b newer? %s\n",
    $sb - $sa, ($sb > $sa ? "yes" : "NO - they compare EQUAL");
# the exact fraction varies per run, so report the VERDICT not the value
printf "   Time::HiRes : difference non-zero? %s -> is b newer? %s\n",
    ($hb != $ha ? "yes" : "no"), ($hb > $ha ? "yes" : "no");
printf "   -M on the two: %s\n",
    ((-M $path->("a.txt")) == (-M $path->("b.txt"))
        ? "equal -- indistinguishable" : "different");

# ---- 3. once a full second has passed, stat is enough
sleep 1.1;
$write->("netlist.v");
my ($rep, $net) = ((stat $path->("a.txt"))[9], (stat $path->("netlist.v"))[9]);
printf "\n3. netlist.v written a second later:\n";
printf "   netlist is newer than the report by %d s -> report is %s\n",
    $net - $rep, ($rep < $net ? "STALE" : "current");

# ---- 4. what -M actually measures. Both files above were created AFTER the
#      script started, so to show both signs we need a genuinely old file.
$write->("old.txt");
utime time() - 10 * 24 * 3600, time() - 10 * 24 * 3600, $path->("old.txt")
    or die "cannot set mtime: $!\n";
printf "\n4. -M is measured from \$^T, the script's START, not from now:\n";
my $old = $path->("old.txt");
printf "   old.txt     backdated 10 days: -M is %s, about %d day(s)\n",
    ((-M $old) > 0 ? "POSITIVE" : "negative"), int((-M $old) + 0.5);
# -M is a NAMED UNARY operator, so its precedence is LOWER than +. Without
# the parentheses this is -M ($old + 0.5) -- a stat of the file named "0.5".
printf "   PRECEDENCE: int(-M \$f + 0.5) = %d, int((-M \$f) + 0.5) = %d\n",
    int(-M $old + 0.5), int((-M $old) + 0.5);
printf "   netlist.v   written during the run: -M is %s\n",
    ((-M $path->("netlist.v")) < 0 ? "NEGATIVE" : "positive");
printf "   so a file newer than \$^T has a negative age: %s\n",
    ((-M $path->("netlist.v")) < 0 ? "yes" : "no");

1. stat element 9 is an integer?        yes
   Time::HiRes::stat has a fraction?    yes

2. a.txt then b.txt, written immediately after each other:
   stat        : difference 0 s        -> is b newer? NO - they compare EQUAL
   Time::HiRes : difference non-zero? yes -> is b newer? yes
   -M on the two: equal -- indistinguishable

3. netlist.v written a second later:
   netlist is newer than the report by 1 s -> report is STALE

4. -M is measured from $^T, the script's START, not from now:
   old.txt     backdated 10 days: -M is POSITIVE, about 10 day(s)
   PRECEDENCE: int(-M $f + 0.5) = 0, int((-M $f) + 0.5) = 10
   netlist.v   written during the run: -M is NEGATIVE
   so a file newer than $^T has a negative age: yes

stat's modification time is a whole number of seconds. And two files written immediately after one another come back with the same timestamp — so $b > $a is false, and a check asking "is the output newer than the input" answers no for an output that was written second.

That is the failure mode to understand. A flow step that regenerates the netlist and the report within the same second produces a report whose timestamp equals the netlist's. A check written as < correctly says "not older". A check written as > — "the report must be strictly newer" — reports a stale artifact that is perfectly fine, every time the flow is fast.

Time::HiRes::stat resolves it, distinguishing the two files by a fraction of a millisecond. It is a core module, so it costs nothing to use.

3. -M Is Not "Age Now"

-M gives a file's age in days, and it is convenient for "is this older than a week". It also has a property that surprises people, measured in section 4 of the output.

-M is measured from $^T, the time the script started — not from the current time. So:

• a file written before the script started has a positive -M — the measurement backdates one by ten days to show it
• a file written during the run has a negative -M, which is why a.txt and b.txt above both came back negative: the script created them itself

That is correct and documented behaviour, and it makes -M stable across a long script: every -M in one run is relative to one fixed instant, so two comparisons taken minutes apart are consistent. It also means this is wrong:

unlink $_ for grep { -M $_ > 7 } @files;        # "older than a week"

In a script that has been running for two hours, -M still measures from the start, so the boundary is "older than a week as of two hours ago". Usually harmless. But in a long-lived process it drifts, and the fix is to set $^T yourself or to compute from time():

my $cutoff = time() - 7 * 24 * 3600;
unlink $_ for grep { (stat $_)[9] < $cutoff } @files;

That is also clearer about what it means, which matters for a line that deletes things — and Chapter 17.6's point applies: age is not ownership. Check containment first, then age.

Two more -M details worth knowing:

-M returns days as a fraction, so an hour is about 0.0417. Comparing against an integer silently means "whole days".

And -M is a named unary operator, whose precedence is lower than +. Measured in section 4 of the output above, on a file backdated ten days:

int(-M $f + 0.5)     ->   0      # parses as -M ($f + 0.5)
int((-M $f) + 0.5)   ->  10      # what was meant

Without the parentheses Perl numifies the filename, gets 0.5, and stats a file called 0.5. It does not exist, so the test returns undef and the arithmetic yields 0 — an age check written that way reports every file as brand new, with no warning at all. Parenthesise any file test that appears inside a larger expression.

-M, -A and -C use the stat cache. After a plain stat $file, you can write -M _ with an underscore to reuse that result rather than calling stat again — which matters in a loop over thousands of files, and is a real idiom you will meet in old scripts.

4. A Dependency Check That Is Honest

Putting it together, the useful shape is a function that takes one output and its inputs and reports what it found.

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;
use Time::HiRes qw(sleep);

my $dir = tempdir(CLEANUP => 1);
my $p = sub { File::Spec->catfile($dir, shift) };
my $mk = sub {
    open my $w, ">", $p->($_[0]) or die "cannot write: $!\n";
    print $w "x\n";
    close $w;
};

# returns (verdict, explanation). Never exits, so it can be tested -- the
# 15.3 argument for check functions that return data.
sub staleness {
    my ($out, @inputs) = @_;
    return ("MISSING", "the output does not exist") unless -e $out;
    my $om = (stat $out)[9];
    my @newer;
    for my $in (@inputs) {
        return ("UNKNOWN", "input missing: " . (File::Spec->splitpath($in))[2])
            unless -e $in;
        push @newer, (File::Spec->splitpath($in))[2] if $om < (stat $in)[9];
    }
    return ("STALE", "newer than the output: " . join(", ", sort @newer)) if @newer;
    return ("CURRENT", "the output is at least as new as every input");
}

# build inputs, then an output, then touch one input again
$mk->($_) for qw(netlist.v constraints.sdc);
sleep 1.1;
$mk->("timing.rpt");
my @cases = (
  [ "fresh report",        "timing.rpt",  [qw(netlist.v constraints.sdc)] ],
  [ "output missing",      "missing.rpt", [qw(netlist.v)]                 ],
  [ "an input missing",     "timing.rpt",  [qw(netlist.v nosuch.sdc)]      ],
);
printf "%-18s %-9s %s\n", "CASE", "VERDICT", "EXPLANATION";
for my $c (@cases) {
    my ($label, $out, $in) = @$c;
    my ($v, $why) = staleness($p->($out), map { $p->($_) } @$in);
    printf "%-18s %-9s %s\n", $label, $v, $why;
}

# now re-touch an input so the report becomes stale
sleep 1.1;
$mk->("netlist.v");
my ($v, $why) = staleness($p->("timing.rpt"), $p->("netlist.v"), $p->("constraints.sdc"));
printf "%-18s %-9s %s\n", "input re-touched", $v, $why;

CASE               VERDICT   EXPLANATION
fresh report       CURRENT   the output is at least as new as every input
output missing     MISSING   the output does not exist
an input missing   UNKNOWN   input missing: nosuch.sdc
input re-touched   STALE     newer than the output: netlist.v

Four cases, four distinct verdicts, and the distinctions are the point.

CURRENT — the output is at least as new as every input. Note at least as new, per section 2's <.

MISSING — the output does not exist. Different from stale: nothing ran, rather than something ran too early.

UNKNOWN — an input does not exist, so the question cannot be answered. This is Chapter 15.4's "could not reach a verdict", and collapsing it into STALE would tell a flow that the report needs regenerating when actually the flow is misconfigured.

STALE — and crucially, the explanation names which input is newer. "The report is stale" sends somebody looking; "newer than the output: netlist.v" tells them which step to re-run.

5. What To Remember

• An output must be newer than every input it was produced from; otherwise it is stale and nothing should trust it.
• (stat $f)[9] is the modification time. Element 8 is access, 10 is inode change — a wrong index gives a plausible wrong number.
• Write your own check for dependencies make does not know about.
• Measured: stat's mtime is a whole number of seconds.
• Measured: two files written back to back compare EQUAL.
• So use < for "stale", not > for "must be newer" — they differ exactly in the fast, healthy case.
• Time::HiRes::stat gives sub-second resolution and is core, if you genuinely need it.
• Measured: -M is relative to $^T, the script's start — a file written during the run has a negative age.
• That makes -M consistent within one run, and wrong as "age now" in a long-lived process.
• For a real cutoff, compute from time() — clearer and drift-free.
• Age is not ownership — containment first, then age (17.6).
• -M is days as a fraction; comparing to an integer means whole days.
• Measured: -M is a named unary operator with precedence below +. int(-M $f + 0.5) gave 0 for a file ten days old, because it parses as -M ($f + 0.5) and stats a file named 0.5. Parenthesise any file test used in an expression.
• -M _ with an underscore reuses the last stat — a real idiom in old scripts, and it matters in a big loop.
• Report four verdicts, not two: CURRENT, STALE, MISSING, UNKNOWN.
• UNKNOWN must not collapse into STALE — a missing input is a misconfigured flow, not a stale output.
• Name which input is newer, so the reader knows which step to re-run.
• A timestamp check is advisory: a touch without a change, a restored backup, or clock skew across hosts all defeat it.

That closes Module 17. A script can now anchor itself to the project root regardless of who called it, recognise the four roles a directory can have, collect sources without being fooled by a wildcard that does not recurse, skip the trees that would build the wrong design, create run directories that survive a re-run, delete only what it owns, edit a tool setup file without destroying the notes in it, produce output that two runs agree on, and notice when a flow step quietly did not happen.