Filtering Out Generated and Third-Party Trees

Part II · Module 17 · Chapter 17.4 · Filtering Out Generated and Third-Party Trees

1. What a Filename Filter Misses

use strict;
use warnings;
use File::Find;
use File::Temp qw(tempdir);
use File::Spec;
use File::Path qw(make_path);

# ---- a SAMPLE tree with every kind of thing a walk must skip
my $root = File::Spec->catdir(tempdir(CLEANUP => 1), "t");
my @rel = qw(
  rtl/alu.v
  rtl/core/cpu.sv
  rtl/core/cpu.v.orig
  rtl/generated/regs.sv
  rtl/.svn/entries
  rtl/core/.git/HEAD
  third_party/uvm/uvm.sv
  backup/old.v
);
for my $rel (@rel) {
    my @part = split m{/}, $rel;
    my $name = pop @part;
    make_path(File::Spec->catdir($root, @part));
    open my $w, ">", File::Spec->catfile($root, @part, $name) or die "cannot write: $!\n";
    print $w "// $rel\n";
    close $w;
}

my %SKIP_DIR  = map { $_ => 1 } qw(generated third_party backup .svn .git);
my $SKIP_FILE = qr/\.(?:orig|bak|rej)\z/;
my $SRC       = qr/\.(?:v|sv|svh)\z/;

# ---- A: filter on the filename only
my @naive;
find(sub { push @naive, $File::Find::name if -f $_ && /$SRC/ }, $root);

# ---- B: prune the directory, so the walk never descends into it.
#      In the callback, $_ is the BASENAME and the cwd is the containing
#      directory -- so `-d $_` is the test that works. Using
#      `-d $File::Find::name` prunes NOTHING and silently returns A's answer.
my (@good, @pruned, $visited);
find(sub {
    $visited++;
    if (-d $_ && $SKIP_DIR{$_}) {
        $File::Find::prune = 1;
        push @pruned, $File::Find::name;
        return;
    }
    return unless -f $_;
    return if /$SKIP_FILE/;
    push @good, $File::Find::name if /$SRC/;
}, $root);

my $all = 0;
find(sub { $all++ }, $root);

my $strip = sub { (my $p = shift) =~ s{^.*/t/}{}; return $p };
printf "A. filename filter only -> %d file(s)\n", scalar @naive;
printf "     %s\n", $strip->($_) for sort @naive;
printf "\nB. with directory prune -> %d file(s)\n", scalar @good;
printf "     %s\n", $strip->($_) for sort @good;
printf "\n   pruned %d director(ies): %s\n", scalar @pruned,
    join(" ", map { $strip->($_) } sort @pruned);
printf "   entries visited: %d with pruning, %d without\n", $visited, $all;

my %keep = map { $_ => 1 } @good;
my @wrong = grep { !$keep{$_} } @naive;
printf "\n   %d of A's %d would build the WRONG design:\n", scalar @wrong, scalar @naive;
printf "     %s\n", $strip->($_) for sort @wrong;

A. filename filter only -> 5 file(s)
     backup/old.v
     rtl/alu.v
     rtl/core/cpu.sv
     rtl/generated/regs.sv
     third_party/uvm/uvm.sv

B. with directory prune -> 2 file(s)
     rtl/alu.v
     rtl/core/cpu.sv

   pruned 5 director(ies): backup rtl/.svn rtl/core/.git rtl/generated third_party
   entries visited: 11 with pruning, 17 without

   3 of A's 5 would build the WRONG design:
     backup/old.v
     rtl/generated/regs.sv
     third_party/uvm/uvm.sv

Five files from the filename filter, two from the pruning walk. The three extras are the whole point:

rtl/generated/regs.sv — a source another step produces. Collecting it means compiling whatever was generated last time, which may predate the specification change you are testing. Chapter 17.9 is about noticing that.

third_party/uvm/uvm.sv — a vendor library. It is compiled, but separately and with different options, and including it in the design filelist usually produces a duplicate-definition error or, worse, silently shadows the version the tool would have used.

backup/old.v — somebody's copy. This is the dangerous one, because it is a valid, slightly older version of a real module. Compile both and the tool picks one, and which one is not something you control.

Notice also that rtl/core/cpu.v.orig was excluded by both, because $SKIP_FILE catches it. That is worth noting as the easy case: a filename rule handles filename problems. It cannot handle a directory problem.

2. Prune the Directory

The mechanism is one assignment:

if (-d $_ && $SKIP_DIR{$_}) {
    $File::Find::prune = 1;
    return;
}

Setting $File::Find::prune tells File::Find not to descend into the directory it is currently looking at. Three details make it work.

-d $_, not -d $File::Find::name. Inside the callback, File::Find has already changed directory to the one containing the entry, and $_ is the bare name. A relative $File::Find::name like t/rtl/generated does not resolve from there.

I got this wrong first time and it is worth repeating why it matters: a broken prune is silent. My version with -d $File::Find::name pruned zero directories and returned exactly the naive answer — five files, no error, no warning. The only way I noticed was printing the count of pruned directories, which is why the script above prints it.

Prune on the directory, before the file test. The return after setting prune is important: without it, execution falls through to the file checks, which is harmless here but confusing to read.

It also makes the walk cheaper. Measured: 11 entries visited with pruning against 17 without. On a real tree with a vendor library of several thousand files, that is the difference between a walk you notice and one you do not.

3. The Five Things to Skip

In rough order of how much damage they do:

1. Backup and editor files. *.orig, *.rej from a merge, *.bak, *~, .#file. A filename rule handles these. They are first because they are valid older versions of real modules.

2. Generated trees. generated/, gen/, auto/, build/. Real inputs, produced by another step, and a source walk runs at the wrong time to collect them. Let the step that produces them declare them.

3. Third-party and vendor code. third_party/, vendor/, ip/, a UVM installation. Compiled separately, usually precompiled into a library.

4. Version-control metadata. .git/, .svn/, CVS/. Harmless to compile — there is nothing in them with a source extension — but walking a large .git directory is slow, and .svn in older checkouts contained copies of the sources with their original extensions, which is a genuine duplicate-definition source.

5. Run and scratch areas. runs/, work/, scratch/. A simulator writes intermediate .sv files into its work area on some flows. Chapter 17.2's role table already marks these.

And one category that is not a skip rule at all: testbench sources in a design filelist. tb/ has the right extensions and is a legitimate source tree — for the testbench. Whether it belongs in a given filelist depends on which filelist, which is a question about roles, not about filtering. Chapter 17.2's %TREE table answers it by having sources be a list you choose from.

4. Getting the Rules Right

The script above uses a deny-list: walk everything, skip named directories. There is an alternative, and the choice matters.

A deny-list is what you want when the tree is mostly sources and the exceptions are few and named. It is also the one that fails open: a directory nobody added to the list gets collected. A new generated_v2/ appears in the tree and silently joins your filelist.

An allow-list inverts that: start only from the directories declared as source roots, and nothing outside them is reachable.

my @roots = map { File::Spec->catdir($ROOT, $_) } @{ $TREE{sources} };
find(\&collect, @roots);        # only rtl/ and tb/ are ever visited

That fails closed: a new source directory nobody declared is missing from the filelist, which produces an unresolved reference — a loud failure rather than a quiet wrong one. Chapter 14.1's third kind of failure is what a deny-list risks and an allow-list avoids.

Use both. Start from declared roots, and still prune inside them, because rtl/generated/ lives under a source root:

find(\&collect, @roots);        # allow-list at the top
# ... and inside the callback, the prune rules from section 2

5. What To Remember

• Finding the files is the easy half. Filtering decides whether you built the right design.
• Measured: a filename filter kept 5 files; a pruning walk kept 2.
• Three of the five would build the wrong design — a generated source, a vendor library, and somebody's backup.
• The backup copy is the most dangerous — a valid, slightly older version of a real module.
• A filename rule handles filename problems only (*.orig was caught by both).
• $File::Find::prune = 1 stops the descent.
• Test with -d $_, not -d $File::Find::name — the cwd is the containing directory and $_ is the basename.
• A broken prune is silent. Mine pruned zero and returned the naive answer with no error. Print the count of pruned directories.
• Pruning is also cheaper — 11 entries visited against 17.
• prune does nothing under follow => 1, because the contents are already collected.
• Five things to skip: backup/editor files, generated trees, vendor code, version-control metadata, run areas.
• Old .svn directories contained copies of sources with real extensions — a genuine duplicate-definition source.
• tb/ is not a skip rule — whether it belongs is a question about which filelist, answered by 17.2's role table.
• A deny-list fails open; a new generated_v2/ joins the filelist silently.
• An allow-list fails closed, producing an unresolved reference instead — a loud failure.
• Use both: start from declared roots and prune inside them.
• Neither list covers a symlink out of a source root — so do not follow symlinks, and if you resolve one, check it lands under the project root.

Chapter 17.5 moves from reading the tree to writing into it. A run directory has to be created before a tool can write into it, and creating one twice must neither fail nor destroy the evidence from last night.