Perl Part II · Module 19 · RTL Filelist Generation

Validating a Filelist Before the Tool Sees It

Every chapter in this module has run into the same wall: the compiler's error message does not name the cause. A wrong order and a missing file look identical, and three quite different duplicate bugs produce one error string between them. The fix is not better debugging — it is a handful of checks that run on the generated list before the tool is invoked, each of which can say exactly what is wrong in your own terms. They take a page of Perl, they are measurably able to fail, and they stay quiet on a list that is right.

Intermediate15 min readPerlPart IIFilelists

Part II · Module 19 · Chapter 19.7 · Validating a Filelist Before the Tool Sees It

1. The Validator, And Eight Cases

One good filelist and seven broken ones, each broken in exactly one way.

Azvya Education Pvt. Ltd.VLSI Mentor

validate.pl

use strict;
use warnings;
use Cwd qw(abs_path);

# ---- the validator: five checks, run on a filelist before the tool sees it
sub validate {
    my ($path) = @_;
    my @err;

    # 1. the filelist itself must exist and be non-empty (18.9's three states)
    return ("NO_FILELIST: $path does not exist") unless -e $path;
    return ("EMPTY_FILELIST: $path is zero bytes") if -z $path;

    open my $fh, "<", $path or return ("UNREADABLE: $path: $!");
    my (@files, @opts);
    while (my $l = <$fh>) {
        chomp $l;
        next if $l =~ /^\s*(?:\/\/|#)/ || $l !~ /\S/;   # comments and blanks
        if ($l =~ /^\s*[-+]/) { push @opts, $l; next }  # +define+, +incdir+, -I
        push @files, $l;
    }
    close $fh;

    # 2. non-empty source list
    push @err, "NO_SOURCES: the list carries options but no source files"
        unless @files;

    # 3. every path exists and is readable
    for my $f (@files) {
        push @err, "MISSING: $f"      unless -e $f;
        push @err, "UNREADABLE: $f"   if -e $f && !-r $f;
    }

    # 4. no file listed twice -- by path string AND by resolved path (19.3)
    my (%by_str, %by_real);
    for my $f (@files) {
        push @{ $by_str{$f} }, 1;
        my $r = abs_path($f);
        push @{ $by_real{$r} }, $f if defined $r;
    }
    for my $f (sort keys %by_str) {
        push @err, "DUPLICATE PATH: $f listed " . scalar(@{ $by_str{$f} }) . " times"
            if @{ $by_str{$f} } > 1;
    }
    for my $r (sort keys %by_real) {
        my %spell = map { $_ => 1 } @{ $by_real{$r} };
        push @err, "SAME FILE, TWO SPELLINGS: " . join(" and ", sort keys %spell)
            if keys %spell > 1;
    }

    # 5. no module name declared twice (19.1's correction, 19.3 section 4)
    my %by_module;
    for my $f (@files) {
        open my $s, "<", $f or next;
        while (my $l = <$s>) { push @{ $by_module{$1} }, $f if $l =~ /^\s*module\s+(\w+)/ }
        close $s;
    }
    for my $m (sort keys %by_module) {
        my @in = @{ $by_module{$m} };
        push @err, "DUPLICATE MODULE '$m' declared in: " . join(", ", @in) if @in > 1;
    }
    return @err;
}

# ---- build one good filelist and five broken ones, and check each
my @GOOD = qw(rtl/core/chip_pkg.sv rtl/core/types_pkg.sv
              rtl/core/counter.sv rtl/core/alu.sv tb/tb_counter.sv);
sub write_f { my ($n, @l) = @_; open my $o, ">", $n or die $!; print $o "$_\n" for @l; close $o }

write_f("v_good.f",      "// GENERATED", "+incdir+rtl/include", @GOOD);
write_f("v_empty.f");
write_f("v_optsonly.f",  "// GENERATED", "+incdir+rtl/include");
write_f("v_missing.f",   @GOOD, "rtl/core/fifo.sv");
write_f("v_duppath.f",   @GOOD, "rtl/core/counter.sv");
symlink "counter.sv", "rtl/core/counter_alias.sv" unless -e "rtl/core/counter_alias.sv";
write_f("v_symlink.f",   @GOOD, "rtl/core/counter_alias.sv");
write_f("v_dupmod.f",    @GOOD, "gate/counter_gate.sv");

my @CASES = (
  ["a correct filelist",            "v_good.f"],
  ["the filelist is empty",         "v_empty.f"],
  ["options but no sources",        "v_optsonly.f"],
  ["a path that does not exist",    "v_missing.f"],
  ["the same path listed twice",    "v_duppath.f"],
  ["one file, two spellings",       "v_symlink.f"],
  ["two files, one module name",    "v_dupmod.f"],
  ["the filelist is not there",     "v_absent.f"],
);

printf "%-30s %-7s %s\n", "CASE", "ERRORS", "first message";
my $fired = 0;
for my $c (@CASES) {
    my ($label, $file) = @$c;
    my @e = validate($file);
    $fired++ if @e;
    printf "%-30s %-7d %s\n", $label, scalar @e, (@e ? $e[0] : "-- clean --");
}
printf "\n%d of %d cases produced an error; the correct one produced none.\n",
    $fired, scalar @CASES;

# ---- and what the TOOL says about the same broken lists, for comparison
print "\nwhat the compiler says about the same lists:\n";
for my $c (@CASES) {
    my ($label, $file) = @$c;
    next if $file eq "v_absent.f" || $file eq "v_empty.f";
    my $log = `iverilog -g2012 -I rtl/include -o /dev/null -c $file 2>&1`;
    my $rc  = $? >> 8;
    my ($first) = grep { /\S/ } split /\n/, ($log // "");
    printf "  %-30s rc=%-3s %s\n", $label, $rc,
        ($rc == 0 ? "compiles" : ($first // "failed"));
}
unlink "rtl/core/counter_alias.sv";

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

CASE                           ERRORS  first message
a correct filelist             0       -- clean --
the filelist is empty          1       EMPTY_FILELIST: v_empty.f is zero bytes
options but no sources         1       NO_SOURCES: the list carries options but no source files
a path that does not exist     1       MISSING: rtl/core/fifo.sv
the same path listed twice     2       DUPLICATE PATH: rtl/core/counter.sv listed 2 times
one file, two spellings        2       SAME FILE, TWO SPELLINGS: rtl/core/counter.sv and rtl/core/counter_alias.sv
two files, one module name     1       DUPLICATE MODULE 'counter' declared in: rtl/core/counter.sv, gate/counter_gate.sv
the filelist is not there      1       NO_FILELIST: v_absent.f does not exist

7 of 8 cases produced an error; the correct one produced none.

what the compiler says about the same lists:
  a correct filelist             rc=0   compiles
  options but no sources         rc=1   iverilog: no source files.
  a path that does not exist     rc=1   rtl/core/fifo.sv: No such file or directory
  the same path listed twice     rc=2   rtl/core/counter.sv:2: error: 'counter' has already been declared in this scope.
  one file, two spellings        rc=2   rtl/core/counter_alias.sv:2: error: 'counter' has already been declared in this scope.
  two files, one module name     rc=2   gate/counter_gate.sv:3: error: 'counter' has already been declared in this scope.

Seven of eight cases produced an error, and the correct one produced none. Both halves matter. A check that cannot fire is decoration — Chapter 18.10's rule that a measurement agreeing with your expectation is not evidence until you know it could have disagreed. And a check that fires on a correct list is worse than no check, because people learn to ignore it.

2. The Five Checks

One: the filelist itself. Three states, not two — Chapter 18.9's argument applied to the generator's output instead of a tool's log:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

return ("NO_FILELIST: $path does not exist") unless -e $path;
return ("EMPTY_FILELIST: $path is zero bytes") if -z $path;

-e before -z, because Chapter 18.9 measured that -z is false for a file that does not exist. A zero-byte filelist is what a generator that died halfway produces, and it is the single most likely artefact of a broken generator.

Two: options but no sources. The parse separates lines starting with + or - from paths, so a list holding +incdir+ and nothing else is caught by name rather than by the tool's no source files:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

push @err, "NO_SOURCES: the list carries options but no source files" unless @files;

This is the case a configuration bug produces — a root that matched nothing, an exclusion that excluded everything. The defines are all present, which is why it looks plausible.

Three: every path exists and is readable. -e and -r (Chapter 13.6). Readability is a separate test from existence for the reason Chapter 18.9 gave: a file can exist, be non-empty, and still be unreadable after a job ran as another user.

Four: no file twice — by path and by resolved path. Both of Chapter 19.3's tests, because a repeated path and a symlink are different bugs with different fixes:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

DUPLICATE PATH: rtl/core/counter.sv listed 2 times
SAME FILE, TWO SPELLINGS: rtl/core/counter.sv and rtl/core/counter_alias.sv

Five: no module name twice. The scan from Chapter 19.3's section 4, which catches what no path-based test can — two genuinely different files declaring one module:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

DUPLICATE MODULE 'counter' declared in: rtl/core/counter.sv, gate/counter_gate.sv

The comment and blank-line handling in the parse is not boilerplate:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

next if $l =~ /^\s*(?:\/\/|#)/ || $l !~ /\S/;
if ($l =~ /^\s*[-+]/) { push @opts, $l; next }

Without the first line, the generated-file stamp from Chapter 19.1 — // GENERATED for config 'rtl' -- do not edit — is treated as a path, and the validator reports MISSING: // GENERATED ... on a perfectly good filelist. That is the false-positive case that teaches people to ignore the tool, and it is produced by the module's own recommended practice.

Without the second, +incdir+rtl/include is a missing path too. Both lines exist because the validator has to accept the format the generator actually emits.

3. Three Causes, One Compiler Message

Now the comparison that justifies the whole chapter. The same broken lists, handed to the compiler:

cause	compiler says
the same path listed twice	`'counter' has already been declared`
one file, two spellings (symlink)	`'counter' has already been declared`
two files, one module name	`'counter' has already been declared`

Counted: one distinct message across the three. Three different bugs in the generator, three different fixes — remove a duplicated root, resolve symlinks before deduplicating, decide which of two files is the real one — and the compiler says the same sentence for all of them.

The line number and filename differ slightly, which is worse than no difference at all, because it suggests the location is a clue when it is only wherever the second declaration happened to land.

And the validator, on the same three:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

DUPLICATE PATH: rtl/core/counter.sv listed 2 times
SAME FILE, TWO SPELLINGS: rtl/core/counter.sv and rtl/core/counter_alias.sv
DUPLICATE MODULE 'counter' declared in: rtl/core/counter.sv, gate/counter_gate.sv

Each names its own cause, in the project's own terms, pointing at the generator rather than at a source file nobody edited. That difference is the entire return on a page of Perl, and it is available only before the tool runs, because afterwards the information has been collapsed into one string.

Two limits worth stating plainly, both in the spirit of Chapter 18.11.

The module scan is a pattern, not a parser. ^\s*module\s+(\w+) misses a module keyword that is not first on its line and counts one inside a block comment. For this job the trade is right — a false clash is a loud message somebody dismisses in a minute, a missed clash is the compile error you were trying to avoid — but it is not a guarantee.

Validation cannot check currency. Chapter 19.6 measured the stale list that compiled: every path existed, nothing was duplicated, every check above passed, and the list was missing a file that had been in the tree for weeks. No check on a filelist can know what the design should contain. That is why the list has to be generated every run rather than validated into correctness — validation catches a broken generator, not a stale one.

4. Where To Run It, And What To Exit With

Run it in the generator, on its own output, before writing the file. Not as a separate script somebody remembers to call. The generator already has the list in memory, the checks are a few milliseconds, and a generator that can produce an invalid filelist will eventually produce one at an inconvenient moment.

Then exit with a status that distinguishes the cases — Chapter 15.4's three values, the same distinction Chapter 18.8 drew for an unrecognised field:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if (@err) {
    print STDERR "filelist validation failed for '$path':\n";
    print STDERR "  $_\n" for @err;
    exit 3;                 # the filelist is not usable -- not a design failure
}

Exit 3, not 1. A broken filelist is not "the design failed" and not "the tests failed" — it is "this run could not happen". A flow that maps it onto 1 reports a design failure for a generator bug, and somebody spends an afternoon in the wrong place.

And report every error, not the first. The measurement shows two of the cases produce two errors each; a validator that dies on the first one turns a single fix-up into three rounds of run-and-fix. Collect into an array, print the array, exit once.

One more check worth adding once a project has a few configurations, which the measurement here does not cover: compare the generated list against the previous run's.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

previous: 142 files   now: 141 files   (-1: rtl/core/fifo.sv)

Nothing is wrong with a filelist shrinking — somebody deleted a module. But a filelist that shrinks by one file nobody mentioned is worth a line of output, and a filelist that shrinks by forty means a root stopped matching. This is Chapter 18.8's range check in a different setting: a number you cannot sanity-check is a number you should not act on, and the cheapest sanity check available is the same number from yesterday.

5. What To Remember

The compiler's message does not name the cause — that is this module's recurring result, and validation is the answer to it.
Measured, 8 cases: 7 produced an error and the correct filelist produced none. Both halves are the point.
A check that cannot fire is decoration; a check that fires on correct input is worse than none.
Check the filelist itself first: -e then -z, because -z is false for a file that does not exist (18.9).
A zero-byte filelist is the signature of a generator that died halfway.
Check for options-but-no-sources — a root that matched nothing looks plausible because the defines are all there.
Check existence and readability separately (13.6) — a file can exist and still be unreadable.
Check duplicates by path AND by resolved path (19.3) — different bugs, different fixes.
Check duplicate module names, which no path test can see.
Skip comments and option lines when parsing, or the generated-file stamp from 19.1 is reported as a missing path — a false positive produced by the module's own advice.
Measured: one distinct compiler message across three different duplicate causes.
The differing line numbers are worse than no difference, because they imply the location is a clue.
Run the validator inside the generator, on its own output, before the file is written.
Exit 3, not 1 (15.4) — a broken filelist is "this run could not happen", not a design failure.
Report every error, not the first — two of the cases produced two each.
Validation catches a broken generator, not a stale one. No check on a filelist knows what the design should contain (19.6).
Compare the count against the previous run — the cheapest sanity check is yesterday's number (18.8).

That closes Module 19. You can take a source tree and produce a filelist a tool will accept: find the sources and reject what merely looks like one, collapse duplicates by file identity and by module name, derive the include path from what the sources actually include, order the list from the dependencies the files declare about each other, generate every configuration from one description, and check the result before the tool ever sees it — with every step measured against a compiler rather than argued from habit.

Where this fits

Part of the Perl curriculum.