Discovering Sources by Pattern

Part II · Module 17 · Chapter 17.3 · Discovering Sources by Pattern

1. Why a Hand-Kept List Goes Stale

A checked-in files.f has one advantage — it is explicit — and one fatal problem: it has to be edited by whoever adds a file, and sometimes they do not.

What happens then is not a build failure. The new module is in the tree, absent from the list, and the design still elaborates because nothing in the tested configuration instantiates it. The error arrives weeks later as an unresolved reference in a configuration somebody finally exercised, and by then the change that caused it is buried.

Generating the list makes that class of error impossible. The tree is the source of truth, which it already was in practice.

2. What glob Actually Does

glob is the obvious first tool. It is worth measuring before trusting.

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;
use File::Path qw(make_path);

# a tree with a .v file at FOUR different depths
use Cwd qw(getcwd);
my $START = getcwd();          # so CLEANUP can remove the tree at exit
END { chdir $START if defined $START }

my $base = tempdir(CLEANUP => 1);
my $a = File::Spec->catdir($base, "a");
make_path(File::Spec->catdir($a, "b", "c", "d"));
my %at = ( "top.v" => [], "top.sv" => [], "one.v" => ["b"],
           "two.v" => ["b","c"], "three.v" => ["b","c","d"] );
for my $name (sort keys %at) {
    my $p = File::Spec->catfile($a, @{ $at{$name} }, $name);
    open my $w, ">", $p or die "cannot write: $!\n";
    print $w "// $name\n";
    close $w;
}

chdir $base or die "cannot chdir: $!\n";
print "depths present: a/top.v=1  a/b/one.v=2  a/b/c/two.v=3  a/b/c/d/three.v=4\n\n";
for my $pat ("a/*.v", "a/*/*.v", "a/**/*.v", "a/*/*/*.v", "a/**",
             "a/{*.v,*.sv}", "a/*.{v,sv}") {
    my @g = glob($pat);
    printf "  %-15s -> %d: %s\n", "'$pat'", scalar @g, join(" ", @g);
}

print "\nif ** recursed, 'a/**/*.v' would find 3 files (depths 2, 3, 4)\n";

# ---- and the ordering question, which matters for Chapter 17.8
print "\nordering:\n";
my $o = File::Spec->catdir($base, "ord");
make_path($o);
for my $n (qw(b.v a.sv c.v d.sv)) {
    open my $w, ">", File::Spec->catfile($o, $n) or die "cannot write: $!\n";
    close $w;
}
chdir $o or die "cannot chdir: $!\n";
for my $pat ("*.v", "{*.v,*.sv}", "{*.sv,*.v}") {
    my @g = glob($pat);
    printf "  glob(%-13s) -> %-24s sorted? %s\n", "'$pat'", join(" ", @g),
        (join(",", @g) eq join(",", sort @g) ? "yes" : "NO");
}

depths present: a/top.v=1  a/b/one.v=2  a/b/c/two.v=3  a/b/c/d/three.v=4

  'a/*.v'         -> 1: a/top.v
  'a/*/*.v'       -> 1: a/b/one.v
  'a/**/*.v'      -> 1: a/b/one.v
  'a/*/*/*.v'     -> 1: a/b/c/two.v
  'a/**'          -> 3: a/b a/top.sv a/top.v
  'a/{*.v,*.sv}'  -> 2: a/top.v a/top.sv
  'a/*.{v,sv}'    -> 2: a/top.v a/top.sv

if ** recursed, 'a/**/*.v' would find 3 files (depths 2, 3, 4)

ordering:
  glob('*.v'        ) -> b.v c.v                  sorted? yes
  glob('{*.v,*.sv}' ) -> b.v c.v a.sv d.sv        sorted? NO
  glob('{*.sv,*.v}' ) -> a.sv d.sv b.v c.v        sorted? NO

a/**/*.v returned exactly what a/*/*.v returned: one file.

That is the headline. Perl's glob has no recursive wildcard. ** is parsed as *, matching one level. The tree has .v files at four depths and the pattern found the one at depth two.

This catches people arriving from elsewhere. bash with shopt -s globstar makes ** recursive. Python's glob.glob(..., recursive=True) makes ** recursive. Perl's does not, and it does not warn — it returns a short list, and a filelist generator built on it collects part of the design. Chapter 14.1's third kind of failure, from one wildcard.

Three more measured facts about glob:

Brace expansion works. a/{*.v,*.sv} and a/*.{v,sv} both return two files, so braces are a real feature.

A braced glob is not sorted. Each alternative is sorted internally and the alternatives are concatenated in the order you wrote them — so {*.v,*.sv} gives the .v files then the .sv files, and swapping the braces changes the output order. A plain glob("*.v") is sorted. Chapter 17.8 explains why this matters enough to sort explicitly anyway.

glob skips dotfiles. A file named .hidden.v is not returned, which is usually what you want and is worth knowing when a file appears to be missing.

3. File::Find Walks the Whole Tree

use strict;
use warnings;
use File::Find;
use File::Temp qw(tempdir);
use File::Spec;
use File::Path qw(make_path);

# a tree shaped like a real project, including the parts a walk must skip
use Cwd qw(getcwd);
my $START = getcwd();          # so CLEANUP can remove the tree at exit
END { chdir $START if defined $START }

my $root = File::Spec->catdir(tempdir(CLEANUP => 1), "t");
my @rel = qw(
  rtl/alu.v rtl/core/cpu.sv rtl/core/pkg.svh rtl/core/.hidden.v
  rtl/sub/fifo.v rtl/generated/gen_regs.sv
  tb/tb_top.sv third_party/uvm/uvm_pkg.sv
  rtl/.svn/entries backup/alu.v.bak
);
for my $rel (@rel) {
    my @part = split m{/}, $rel;
    my $name = pop @part;
    make_path(File::Spec->catdir($root, @part));
    open my $w, ">", File::Spec->catfile($root, @part, $name) or die "cannot write: $!\n";
    print $w "// $rel\n";
    close $w;
}

# ---- glob, one level at a time
chdir $root or die "cannot chdir: $!\n";
my @by_glob = glob("rtl/*.{v,sv,svh}");
printf "glob 'rtl/*.{v,sv,svh}'  -> %d file(s)\n", scalar @by_glob;
print  "   $_\n" for @by_glob;

# ---- File::Find, the whole tree
my @by_find;
find(sub {
    return unless -f $_;
    return unless /\.(?:v|sv|svh)\z/;
    (my $rel = $File::Find::name) =~ s{^\./}{};
    push @by_find, $rel;
}, ".");

printf "\nFile::Find, whole tree   -> %d file(s)\n", scalar @by_find;
print  "   $_\n" for sort @by_find;

printf "\n   glob saw %d, the walk saw %d\n", scalar @by_glob, scalar @by_find;
my @unwanted = grep { m{(?:/|\A)(?:generated|third_party)/} || m{/\.} } sort @by_find;
printf "   of those %d, %d should NOT be in a filelist: %s\n",
    scalar @by_find, scalar @unwanted, join(", ", @unwanted);

glob 'rtl/*.{v,sv,svh}'  -> 1 file(s)
   rtl/alu.v

File::Find, whole tree   -> 8 file(s)
   rtl/alu.v
   rtl/core/.hidden.v
   rtl/core/cpu.sv
   rtl/core/pkg.svh
   rtl/generated/gen_regs.sv
   rtl/sub/fifo.v
   tb/tb_top.sv
   third_party/uvm/uvm_pkg.sv

   glob saw 1, the walk saw 8
   of those 8, 3 should NOT be in a filelist: rtl/core/.hidden.v, rtl/generated/gen_regs.sv, third_party/uvm/uvm_pkg.sv

The walk found eight files where the single-level glob found one. Three things to take from that.

File::Find recurses by default, and the callback decides what to keep. $File::Find::name is the path relative to where you started; $_ is the bare filename and the current directory is the file's directory, which is why -f $_ works.

It returns dotfiles. rtl/core/.hidden.v is in the list where glob omitted it. Usually unwanted — and the .svn/entries row shows why a walk needs rules.

Three of the eight should not be in a filelist: the generated source, the vendor code, and the version-control metadata. Finding the files is the easy half; Chapter 17.4 is the other half, and it is the one that decides whether you built the right design.

4. What the Walk Must Still Handle

Four things a production source walk needs beyond recursion.

Deterministic order. File::Find visits in readdir order, which is not alphabetical and differs between filesystems. Every example above sorts before printing. A filelist whose order changes between runs produces compile logs that cannot be diffed, and in some tools a different order is a different elaboration. This is Chapter 17.8 and it is rated Critical for exactly this reason.

Extension rules in one place. /\.(?:v|sv|svh)\z/ belongs in a variable, not repeated in three callbacks. Note \z rather than $ — Chapter 11.8's anchors — so a file called alu.v.bak does not match.

Exclusions. The generated, third_party and .svn rows above. Chapter 17.4.

A count, reported. A walk that returns zero files should say so loudly rather than writing an empty filelist, which is Chapter 15.4's "could not reach a verdict" case:

die "no source files found under @roots\n" unless @files;
printf STDERR "collected %d source file(s)\n", scalar @files;

The message goes to standard error, per Chapter 15.12, so the filelist itself stays clean on standard output.

5. What To Remember

• A hand-kept filelist is wrong from the first merge that adds a file, and the failure is quiet rather than a build error.
• Measured: a/**/*.v returns exactly what a/*/*.v returns. Perl's glob has no recursive wildcard; ** means *.
• bash globstar and Python's recursive=True do recurse — Perl's does not, and does not warn.
• A filelist generator built on ** collects a fraction of the design and the design still compiles.
• Brace expansion works: a/{*.v,*.sv} and a/*.{v,sv} both find two files.
• A braced glob is NOT sorted — alternatives concatenate in written order, and swapping the braces changes the order. Plain glob("*.v") is sorted.
• glob skips dotfiles.
• Use glob only when you know the depth, and bsd_glob with GLOB_NOMAGIC for paths that come from data.
• Measured: File::Find found 8 files where the one-level glob found 1.
• In the callback, $_ is the bare filename and the cwd is the file's directory, so -f $_ works; $File::Find::name is the relative path.
• File::Find returns dotfiles, including .svn/entries.
• Three of the eight found should not be in a filelist — generated, vendor, and version-control metadata.
• File::Find does not follow symlinks by default, and that is correct — following produces duplicate module definitions.
• Sort before output — readdir order is neither alphabetical nor stable (17.8).
• Anchor the extension pattern with \z, or alu.v.bak matches.
• Report the count to stderr and die on zero, rather than writing an empty filelist.

Chapter 17.4 is the half that decides correctness. Three of the eight files found above would build the wrong design, and skipping them turns out to need more than a pattern on the filename.