Temporary Files and Atomic Writes

Part I · Module 13 · Chapter 13.13 · Temporary Files and Atomic Writes

1. A Name Nothing Can Collide With

use strict;
use warnings;
use File::Temp qw(tempfile tempdir);

my $dir = tempdir(CLEANUP => 1);

# tempfile gives a unique name AND an open handle, with no race.
my ($fh, $path) = tempfile(DIR => $dir, SUFFIX => ".tmp");
print $fh "work in progress\n";
close $fh;

printf "created a temp file : %s\n", (-e $path ? "yes" : "no");
printf "name is unique      : %s\n", ($path =~ /\.tmp$/ ? "ends in .tmp" : "?");
printf "it is inside our dir: %s\n", (index($path, $dir) == 0 ? "yes" : "no");

created a temp file : yes
name is unique      : ends in .tmp
it is inside our dir: yes

tempfile returns two things: an already-open filehandle and the path it chose.

Returning both is the whole point, and it is worth understanding why. The obvious way to make a temporary file is to invent a name, check it is free, and open it. That is Chapter 13.6's race with a worse outcome — two jobs can both find the name free and both decide to use it, and one of them loses its data. tempfile asks the operating system to create and open the file in a single step, so there is no gap for a second job to slip into.

The options you will use:

• DIR => $dir puts the file where you want it. The default is the system temporary directory, which is fine for scratch work and wrong for an atomic write — section two explains why.
• SUFFIX => ".tmp" gives the name a recognisable ending, which helps a human looking at a directory mid-run.
• UNLINK => 1 deletes the file when your program exits. Right for genuine scratch work, wrong when you intend to rename it into place.

tempdir(CLEANUP => 1) is the directory equivalent, and it is what every script on these pages has used to keep its examples self-contained.

2. The Atomic Write

Now the problem the chapter exists for.

use strict;
use warnings;
use File::Temp qw(tempdir tempfile);
use File::Spec;

my $dir   = tempdir(CLEANUP => 1);
my $final = File::Spec->catfile($dir, "report.txt");

# A reader always sees either the OLD file or the COMPLETE new one.
open my $seed, ">", $final or die $!; print $seed "OLD COMPLETE\n"; close $seed;

# Write to a temp file in the SAME directory, then rename over the target.
my ($fh, $tmp) = tempfile(DIR => $dir, SUFFIX => ".part");
print $fh "NEW COMPLETE\n";
close $fh or die "close failed: $!";

printf "before rename, target holds : %s", do { open my $r, "<", $final or die $!; scalar <$r> };
rename $tmp, $final or die "rename failed: $!";
printf "after  rename, target holds : %s", do { open my $r, "<", $final or die $!; scalar <$r> };
printf "temp file gone              : %s\n", (-e $tmp ? "no" : "yes");
print  "\nrename within one filesystem is atomic: no reader sees a half-written file\n";

before rename, target holds : OLD COMPLETE
after  rename, target holds : NEW COMPLETE
temp file gone              : yes

rename within one filesystem is atomic: no reader sees a half-written file

The target held OLD COMPLETE, and then it held NEW COMPLETE. It never held anything else.

Compare that with the straightforward version from Chapter 13.4:

open my $fh, ">", $final or die $!;    # the file is empty from this instant
print $fh $line for @many_lines;       # partial for however long this takes
close $fh;

Between the open and the close, the file on disk is incomplete. A reader during that window gets a truncated report. And if the script dies in the middle — a full disk, a killed job, a bug in line 400 — the truncated version is what remains, with no sign that it is not the whole thing.

The atomic version has four parts, and each one is load-bearing:

1. Write to a temporary file. Nothing is reading it, so a partial state is harmless.

2. Put the temporary file in the same directory as the target. This is the part people get wrong. rename is only atomic within one filesystem; across filesystems it becomes a copy and a delete, which has exactly the partial-file window you were trying to avoid. The system temporary directory is usually a different filesystem from your scratch area, so DIR => $dir is not a tidiness preference — it is what makes the next step work.

3. close and check it. Chapter 13.4 measured that close is when the data actually reaches the disk. Renaming a file whose close failed puts a truncated file into place with full confidence, which is worse than not writing at all.

4. rename over the target. One operation. A reader either opens the old file or the new one.

One more detail worth knowing: a renamed temporary file keeps the permissions tempfile gave it, which are deliberately restrictive — readable only by you. If the report needs to be readable by colleagues or by another job, set the permissions before the rename with chmod 0644, $tmp;. A report nobody else can read is a confusing way to discover this.

3. Putting It Together

The complete pattern, as you would actually write it:

use File::Temp qw(tempfile);
use File::Spec;
use File::Basename qw(dirname);

my $final = "/scratch/runs/run2/report.txt";
my ($fh, $tmp) = tempfile(DIR => dirname($final), SUFFIX => ".part");

print $fh "...the whole report...\n";

close $fh            or die "cannot close $tmp: $!\n";
chmod 0644, $tmp     or warn "cannot chmod $tmp: $!\n";
rename $tmp, $final  or die "cannot rename $tmp to $final: $!\n";

Seven lines, every one of which is a chapter from this module: dirname from 13.11, the or die habit from 13.5, the close check from 13.4, and the same-filesystem rule above.

Make this your default for any file another person or job will read. It costs three extra lines over the naive version and removes an entire class of problem — the one where somebody shows you a report that is obviously wrong, and the real answer is that they read it while it was being written.

4. What To Remember

• tempfile returns an open handle and the path, created in one step, so there is no window for another job to take the name.
• DIR, SUFFIX and UNLINK are the options you will use. tempdir(CLEANUP => 1) is the directory version.
• Use it whenever you hand a path to another program, because two parallel runs of your script is normal.
• A plain > leaves the target incomplete from the open until the close, and leaves it truncated for good if the script dies.
• The fix is write-then-rename. Measured: the target held OLD COMPLETE, then NEW COMPLETE, and nothing in between.
• Put the temporary file in the same directory as the target. rename is atomic only within one filesystem; across filesystems it is a copy and a delete, which reopens the window.
• Check close before you rename, or you may put a truncated file into place confidently.
• rename replaces silently — it stops readers seeing a partial file, it does not stop two jobs overwriting each other. For that, use a unique name per job.
• tempfile's permissions are restrictive. chmod before the rename if others must read it.

Chapter 13.14 closes the module with the format you will read and write most often after plain lines: delimited text, the split option that stops you losing empty columns, and an honest account of where a simple split stops working.