Perl Part I · Module 13 · File Handling

Recursive Traversal With File::Find

The previous chapter listed one directory. Real work usually needs a whole tree: every timing report under a results directory, every log in every run. You could write that recursion yourself with opendir and a stack, and the reason not to is that the core module already handles the cases you would get wrong. File::Find takes a starting directory and a piece of your code, and runs your code once for every file and directory beneath it. The interface is unusual enough to be worth a chapter of its own, because it hands you information in variables rather than arguments, and because it quietly changes your working directory before each call — which makes one kind of code work and another kind fail. Both are measured. The chapter ends with the feature that turns a slow scan into a fast one: telling it not to descend into a directory at all.

Intermediate10 min readPerlPart IFile Handling

Part I · Module 13 · Chapter 13.10 · Recursive Traversal With File::Find

1. Walking a Tree

How File::Find calls your code

You hand find two things and it drives the rest. The step worth noticing is the third: your callback runs with the working directory already changed to the entry's own directory, which is why a bare file test works inside it and why a relative path you built earlier does not.

Azvya Education Pvt. Ltd.VLSI Mentor

find-basics.pl

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Path qw(make_path);
use File::Find qw(find);
use File::Spec;

my $root = tempdir(CLEANUP => 1);
make_path(File::Spec->catdir($root, "run1", "timing"));
make_path(File::Spec->catdir($root, "run2", "timing"));
make_path(File::Spec->catdir($root, "run2", "power"));

for my $p (
    ["run1","timing","setup.rpt"], ["run1","timing","hold.rpt"],
    ["run2","timing","setup.rpt"], ["run2","power","ir.rpt"],
    ["run2","notes.txt"],
) {
    my $f = File::Spec->catfile($root, @$p);
    open my $w, ">", $f or die $!; print $w "x\n"; close $w;
}

my @rpt;
find(sub { push @rpt, $File::Find::name if -f $_ && /\.rpt$/ }, $root);

printf "found %d .rpt file(s)\n", scalar @rpt;
for my $f (sort @rpt) {
    my $rel = File::Spec->abs2rel($f, $root);
    printf "  %s\n", $rel;
}

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

found 4 .rpt file(s)
  run1/timing/hold.rpt
  run1/timing/setup.rpt
  run2/power/ir.rpt
  run2/timing/setup.rpt

Four reports found across three nested directories, and notes.txt correctly left out. The whole traversal is one line:

find(sub { ... }, $root);

Two arguments: a callback — a subroutine reference, which is Chapter 7.11's code reference — and one or more starting directories. find walks everything beneath them and calls your callback once per entry, files and directories alike.

The callback in that script does three things, and the order matters: -f $_ to skip directories, /\.rpt$/ to pick the reports, and then push @rpt, $File::Find::name to keep the full path. The next section is why those are the right variables.

2. The Three Variables, and the Changed Directory

Azvya Education Pvt. Ltd.VLSI Mentor

find-chdir.pl

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Path qw(make_path);
use File::Find qw(find);
use File::Spec;
use Cwd qw(getcwd);

my $root = tempdir(CLEANUP => 1);
make_path(File::Spec->catdir($root, "deep", "deeper"));
my $f = File::Spec->catfile($root, "deep", "deeper", "a.rpt");
open my $w, ">", $f or die $!; print $w "x\n"; close $w;

my $outside = getcwd();
find(sub {
    return unless /\.rpt$/;
    printf "\$_                  : %s\n", $_;
    printf "\$File::Find::name   : %s\n", File::Spec->abs2rel($File::Find::name, $root);
    printf "\$File::Find::dir    : %s\n", File::Spec->abs2rel($File::Find::dir, $root);
    printf "cwd inside callback : %s\n",
        (getcwd() eq $outside ? "unchanged" : "CHANGED to the file's directory");
    printf "-s \$_ works         : %s\n", (-s $_ ? "yes" : "no");
}, $root);

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

$_                  : a.rpt
$File::Find::name   : deep/deeper/a.rpt
$File::Find::dir    : deep/deeper
cwd inside callback : CHANGED to the file's directory
-s $_ works         : yes

Three variables, each answering a different question:

$_ — the bare name, a.rpt. This is what your pattern matches against, which is why /\.rpt$/ with no variable reads so naturally.
$File::Find::name — the full path. This is what you keep.
$File::Find::dir — the directory being scanned.

And then the behaviour that explains all of it: the working directory was changed. Before calling your callback, find moves into the entry's own directory.

That is why -s $_ works on the last line. A bare name would normally be the wrong path — exactly the Chapter 13.9 trap — but here it is correct, because the process is standing in that directory.

The same change breaks the opposite kind of code. A relative path you computed before calling find will not resolve inside the callback, and neither will a relative output filename. Two habits avoid it entirely:

Collect, then work. Have the callback do nothing but push the full paths into an array, and process the array after find returns. That is what section one does, and it keeps all your real logic outside the strange environment.

Make paths absolute before you start. If the callback must write somewhere, compute that path with File::Spec->rel2abs before the traversal.

If you would rather not have the chdir at all, find({ no_chdir => 1, wanted => sub { ... } }, $root) turns it off — and then $_ becomes the full path rather than the bare name, so your patterns need basename or an anchored match.

3. Skipping a Subtree

Azvya Education Pvt. Ltd.VLSI Mentor

find-prune.pl

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Path qw(make_path);
use File::Find qw(find);
use File::Spec;

my $root = tempdir(CLEANUP => 1);
make_path(File::Spec->catdir($root, "keep"));
make_path(File::Spec->catdir($root, "scratch", "huge"));
for my $p (["keep","a.rpt"], ["scratch","b.rpt"], ["scratch","huge","c.rpt"]) {
    my $f = File::Spec->catfile($root, @$p);
    open my $w, ">", $f or die $!; print $w "x\n"; close $w;
}

# Without pruning: everything.
my @all;
find(sub { push @all, $_ if /\.rpt$/ }, $root);
printf "no prune : %d file(s)  %s\n", scalar @all, join(",", sort @all);

# With pruning: skip the whole scratch subtree.
my @kept;
find(sub {
    if (-d $_ && $_ eq "scratch") { $File::Find::prune = 1; return }
    push @kept, $_ if /\.rpt$/;
}, $root);
printf "pruned   : %d file(s)  %s\n", scalar @kept, join(",", sort @kept);
print  "\n\$File::Find::prune = 1 skips the whole subtree, not just that entry\n";

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

no prune : 3 file(s)  a.rpt,b.rpt,c.rpt
pruned   : 1 file(s)  a.rpt

$File::Find::prune = 1 skips the whole subtree, not just that entry

Three reports without pruning, one with it.

Setting $File::Find::prune = 1 while looking at a directory tells find not to descend into it. Note what was skipped: b.rpt sitting directly inside scratch, and c.rpt two levels down inside scratch/huge. Pruning removes the entire subtree, not just the one entry.

This matters for speed more than for correctness. On real infrastructure the directories you want are mixed in with ones you emphatically do not want to walk — a scratch area with a million intermediate files, a .git directory, a mounted filesystem that is slow to stat. Without pruning, find visits every one of them and your scan takes minutes instead of seconds.

The shape to remember is the early return:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if (-d $_ && $_ eq "scratch") { $File::Find::prune = 1; return }

Set the flag and return, so the rest of the callback does not also try to treat the directory as a file.

A practical note on the alternative. File::Find is core and always present, which is its main advantage. If you are starting fresh and may install modules, Path::Tiny offers a nicer interface for the same work — but Chapter 12.10's constraint applies, and on a shared farm "already installed" usually wins.

4. A Worked Case: Collect, Then Work

Putting the three pieces together on a tree shaped like a real results directory.

Azvya Education Pvt. Ltd.VLSI Mentor

find-worked.pl

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Path qw(make_path);
use File::Find qw(find);
use File::Spec;

# A results tree as a flow would leave it.
my $root = tempdir(CLEANUP => 1);
for my $run (qw(run1 run2 run3)) {
    make_path(File::Spec->catdir($root, $run, "timing"));
    make_path(File::Spec->catdir($root, $run, "scratch"));
    for my $f (qw(setup.rpt hold.rpt)) {
        my $p = File::Spec->catfile($root, $run, "timing", $f);
        open my $w, ">", $p or die $!;
        print $w "u_alu/A " . ($run eq "run2" ? "-0.95" : "-0.12") . "\n";
        close $w;
    }
    # junk we must not walk
    for my $j (1 .. 5) {
        my $p = File::Spec->catfile($root, $run, "scratch", "tmp$j.rpt");
        open my $w, ">", $p or die $!; print $w "junk\n"; close $w;
    }
}

# Collect only the real reports, skipping every scratch directory.
my @reports;
find(sub {
    if (-d $_ && $_ eq "scratch") { $File::Find::prune = 1; return }
    push @reports, $File::Find::name if -f $_ && /\.rpt$/;
}, $root);

printf "reports collected : %d\n", scalar @reports;
printf "scratch files seen: %d\n", scalar grep { m{/scratch/} } @reports;

# Now do the work OUTSIDE the callback, where cwd is normal.
my $worst = { slack => 0, pin => "(none)", file => "" };
for my $f (sort @reports) {
    open my $fh, "<", $f or die "cannot read $f: $!";
    while (my $line = <$fh>) {
        chomp $line;
        my ($pin, $slack) = split ' ', $line;
        next unless defined $slack;
        if ($slack < $worst->{slack}) {
            $worst = { slack => $slack, pin => $pin,
                       file => File::Spec->abs2rel($f, $root) };
        }
    }
    close $fh;
}
printf "worst slack       : %s at %s\n", $worst->{slack}, $worst->{pin};
printf "found in          : %s\n", $worst->{file};

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

reports collected : 6
scratch files seen: 0
worst slack       : -0.95 at u_alu/A
found in          : run2/timing/hold.rpt

Three runs, each with two real reports and five pieces of junk — twenty-one .rpt files on disk. The scan collected 6 and saw 0 scratch files. The fifteen junk files were never even visited, because prune stopped the traversal at each scratch directory.

The structure is the one to copy, and it has two halves on purpose:

The callback does nothing but decide and collect. One prune test, one match, one push. No file is opened inside it, nothing is parsed, nothing is printed.

All the real work happens after find returns. The loop that opens each report and finds the worst slack runs in a normal environment: the working directory is where you started, the paths in @reports are absolute, and or die messages name a path you can actually use.

That separation is what makes the code debuggable. Everything inside a File::Find callback happens somewhere surprising, with $_ meaning something unusual, which is a poor place to put logic you may need to inspect. Everything after it is ordinary Perl.

Note the last line of output as well: abs2rel turned an absolute temporary path into run2/timing/hold.rpt, which is what you want in a report a person reads. Chapter 13.11 covers it.

5. What To Remember

find(sub { ... }, $root) walks a whole tree and calls your code once per entry, files and directories alike.
Three variables: $_ the bare name, $File::Find::name the full path — keep this one — and $File::Find::dir the directory.
find changes the working directory before each call. Measured. That is why a bare -s $_ works inside the callback.
So a relative path from outside does not resolve inside. Collect full paths in the callback and do the work after find returns.
no_chdir => 1 turns the chdir off, and then $_ is the full path instead of the bare name.
$File::Find::prune = 1 on a directory skips the whole subtree. Measured: 3 files became 1, including one two levels down.
Set prune and return, so the callback does not also process the directory.
Prune the expensive directories — scratch areas, version-control directories, slow mounts — or a scan takes minutes.
File::Find is core, which is usually the deciding argument over a nicer third-party module.

Chapter 13.11 fills in the piece every example so far has quietly relied on: taking paths apart and putting them together correctly, with File::Basename and File::Spec.

Where this fits

Part of the Perl curriculum.