Perl Part II · Module 19 · RTL Filelist Generation

Per-Configuration Filelists

No real project builds one thing. There is an RTL simulation and a gate-level one, a run with the assertion library and a run without, a lint configuration that wants the sources but not the testbench. The usual outcome is four filelists in a directory, each a copy of another made at some point in the past, drifting apart. This chapter generates all of them from one description and then measures the failure that makes hand-maintenance indefensible: a filelist that has gone stale still compiles.

Advanced14 min readPerlPart IIFilelists

Part II · Module 19 · Chapter 19.6 · Per-Configuration Filelists

1. One Description, Three Filelists

The description is a hash of hashes (Chapter 7.8), one entry per configuration, each naming its search roots, its exclusions and its defines. Everything else — the walk, the filter, the deduplication, the dependency sort — is the machinery from Chapters 19.2 to 19.5, called once per configuration.

Azvya Education Pvt. Ltd.VLSI Mentor

configs.pl

use strict;
use warnings;
use File::Find;

# ---- ONE description. Every filelist in the project comes from this.
my %CONFIG = (
  rtl => {
    roots   => ["rtl/core", "tb"],
    exclude => [],
    defines => { SIM => 1 },
  },
  gate => {
    roots   => ["rtl/core", "gate", "tb"],
    exclude => ["rtl/core/counter.sv"],      # replaced by the gate netlist
    defines => { SIM => 1, GATE_LEVEL => 1 },
  },
  rtl_assert => {
    roots   => ["rtl/core", "verif", "tb"],
    exclude => [],
    defines => { SIM => 1, ASSERT_ON => 1 },
  },
);

sub sources_for {
    my ($name) = @_;
    my $cfg = $CONFIG{$name} or die "no config '$name'\n";
    my %skip = map { $_ => 1 } @{ $cfg->{exclude} };
    my @hits;
    find({ no_chdir => 1, wanted => sub {
        my $p = $File::Find::name;
        return unless -f $p && $p =~ /\.sv$/;
        return if $p =~ m{(^|/)generated/};
        return if (split m{/}, $p)[-1] =~ /^\./;
        return if $skip{$p};
        push @hits, $p;
    }}, @{ $cfg->{roots} });
    # dependency order (19.5), reduced to what this config needs
    my (%def, %need);
    for my $f (@hits) {
        open my $fh, "<", $f or die "$f: $!";
        while (my $l = <$fh>) {
            $def{$f}{$1}  = 1 if $l =~ /^\s*package\s+(\w+)\s*;/;
            $need{$f}{$1} = 1 if $l =~ /^\s*import\s+(\w+)\s*::/;
        }
        close $fh;
    }
    my %prov;
    for my $f (@hits) { $prov{$_} = $f for keys %{ $def{$f} // {} } }
    my (%st, @ord);
    my $visit; $visit = sub {
        my ($f) = @_;
        return if ($st{$f} // "") eq "done";
        return if ($st{$f} // "") eq "open";
        $st{$f} = "open";
        for my $n (sort keys %{ $need{$f} // {} }) {
            my $d = $prov{$n};
            $visit->($d) if defined $d && $d ne $f;
        }
        $st{$f} = "done";
        push @ord, $f;
    };
    $visit->($_) for sort @hits;
    return @ord;
}

sub emit {
    my ($name, @files) = @_;
    my $cfg = $CONFIG{$name};
    open my $o, ">", "$name.f" or die $!;
    print $o "// GENERATED for config '$name' -- do not edit\n";
    print $o "+define+$_=$cfg->{defines}{$_}\n" for sort keys %{ $cfg->{defines} };
    print $o "+incdir+rtl/include\n";
    print $o "$_\n" for @files;
    close $o;
    return scalar @files;
}

sub compile {
    my ($name, @files) = @_;
    my $cfg = $CONFIG{$name};
    open my $o, ">", "c_$name.f" or die $!;
    print $o "$_\n" for @files;
    close $o;
    my $defs = join " ", map { "-D $_=$cfg->{defines}{$_}" } sort keys %{ $cfg->{defines} };
    my $log = `iverilog -g2012 -I rtl/include $defs -o c_$name.vvp -c c_$name.f 2>&1`;
    my $rc = $? >> 8;
    return ($rc, $rc == 0 ? "compiles" : ((grep { /\S/ } split /\n/, ($log // ""))[0] // "failed"));
}

printf "%-12s %-7s %-34s %s\n", "CONFIG", "FILES", "DEFINES", "COMPILER";
my %generated;
for my $name (sort keys %CONFIG) {
    my @files = sources_for($name);
    $generated{$name} = \@files;
    emit($name, @files);
    my ($rc, $msg) = compile($name, @files);
    printf "%-12s %-7d %-34s rc=%s %s\n", $name, scalar @files,
        join(",", map { "$_=$CONFIG{$name}{defines}{$_}" } sort keys %{ $CONFIG{$name}{defines} }),
        $rc, $msg;
}

print "\nhow much do the configs share?\n";
my %count;
for my $name (sort keys %generated) { $count{$_}++ for @{ $generated{$name} } }
my $all = grep { $count{$_} == scalar keys %generated } keys %count;
printf "  files in every config: %d of %d distinct files\n", $all, scalar keys %count;
printf "  files unique to one config: %s\n",
    join(", ", map { (split m{/}, $_)[-1] } sort grep { $count{$_} == 1 } keys %count);

print "\nwhat if somebody merges rtl and gate by hand?\n";
my @merged = do { my %s; grep { !$s{$_}++ } (@{ $generated{rtl} }, @{ $generated{gate} }) };
my ($mrc, $mmsg) = compile("rtl", @merged);
printf "  merged list: %d files, rc=%s  %s\n", scalar @merged, $mrc, $mmsg;

print "\nnow a new source file appears (rtl/core/decoder.sv):\n";
open my $nf, ">", "rtl/core/decoder.sv" or die $!;
print $nf "module decoder (input logic [1:0] sel, output logic [3:0] oh);\n",
          "  always_comb begin oh = 4'b0000; oh[sel] = 1'b1; end\n",
          "endmodule\n";
close $nf;
for my $name (sort keys %CONFIG) {
    my @files = sources_for($name);
    printf "  %-12s regenerated: %d files, picked up decoder.sv? %s\n", $name, scalar @files,
        ((grep { m{/decoder\.sv$} } @files) ? "YES" : "no");
}
# a hand-maintained copy of the rtl list, written before decoder.sv existed
my ($hrc, $hmsg) = compile("rtl", @{ $generated{rtl} });
printf "  hand-kept list: %d files, still compiles? %s\n",
    scalar @{ $generated{rtl} }, ($hrc == 0 ? "yes -- and it is now INCOMPLETE" : "no: $hmsg");
printf "  does it contain decoder.sv? %s\n",
    ((grep { m{/decoder\.sv$} } @{ $generated{rtl} }) ? "yes" : "NO");
unlink "rtl/core/decoder.sv";

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

CONFIG       FILES   DEFINES                            COMPILER
gate         5       GATE_LEVEL=1,SIM=1                 rc=0 compiles
rtl          5       SIM=1                              rc=0 compiles
rtl_assert   6       ASSERT_ON=1,SIM=1                  rc=0 compiles

how much do the configs share?
  files in every config: 4 of 7 distinct files
  files unique to one config: counter_gate.sv, count_checker.sv

what if somebody merges rtl and gate by hand?
  merged list: 6 files, rc=2  gate/counter_gate.sv:3: error: 'counter' has already been declared in this scope.

now a new source file appears (rtl/core/decoder.sv):
  gate         regenerated: 6 files, picked up decoder.sv? YES
  rtl          regenerated: 6 files, picked up decoder.sv? YES
  rtl_assert   regenerated: 7 files, picked up decoder.sv? YES
  hand-kept list: 5 files, still compiles? yes -- and it is now INCOMPLETE
  does it contain decoder.sv? NO

Three configurations, three filelists, all three compile. They differ in file count (5, 5 and 6) and in defines, and the differences come from three lines of description rather than three maintained files.

The generated output is a real filelist, in the commercial spelling from Chapter 19.1:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

// GENERATED for config 'rtl' -- do not edit
+define+SIM=1
+incdir+rtl/include
rtl/core/chip_pkg.sv
rtl/core/types_pkg.sv
rtl/core/alu.sv
rtl/core/counter.sv
tb/tb_counter.sv

2. What The Description Holds

Three keys carry everything, and the third is the one people leave out.

Roots. Where to look. gate adds gate/, rtl_assert adds verif/. Adding a configuration is adding a root, not copying a list.

Exclusions. What to drop even though the walk found it. The gate configuration excludes rtl/core/counter.sv because gate/counter_gate.sv replaces it. This is the key that cannot be derived — the tree cannot tell you that two files are alternatives.

Defines. The macros the tool is given. SIM everywhere, plus GATE_LEVEL or ASSERT_ON. These belong in the description because they are part of what makes a configuration that configuration; a filelist with the right files and the wrong defines is a different build.

The sharing measurement is there to make one point about scale: 4 of the 7 distinct files appear in every configuration. Only two files are unique to one. That is the normal ratio, and it is exactly why copy-paste filelists are so tempting and so bad — they are 80% identical, so the diff between two of them is small enough to look harmless and large enough to hide a real difference.

3. The Merge Mistake

What happens if somebody decides two configurations are similar enough to combine?

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

merged list: 6 files, rc=2
gate/counter_gate.sv:3: error: 'counter' has already been declared in this scope.

The gate netlist and the RTL module both declare module counter — which is the whole point of a gate-level netlist, that it is the same module built differently. Merging the lists gives Chapter 19.3's duplicate-module error, naming both files.

This is the friendly case. The two configurations are mutually exclusive by construction, so the mistake is impossible to miss.

The dangerous version is two configurations whose overlap declares nothing in common — a lint configuration merged with a simulation one, say, where the result compiles and runs and is simply not either configuration. Nothing collides, so nothing complains, and the run reports results for a build that corresponds to no configuration anybody defined.

That is the argument for making configurations named objects rather than ad-hoc command lines: a named configuration can be wrong, but it cannot be accidental.

4. Drift: The Stale List Compiles

Now the measurement that settles it. A new source file appears — somebody adds rtl/core/decoder.sv.

All three generated configurations pick it up, with no edit to anything: 6, 6 and 7 files, each reporting picked up decoder.sv? YES.

The hand-kept list — the same rtl list, written before decoder.sv existed:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

hand-kept list: 5 files, still compiles? yes -- and it is now INCOMPLETE
does it contain decoder.sv? NO

It still compiles. rc=0. Nothing instantiates decoder yet, so the design has no unresolved reference, and the compiler has no reason to object.

So the stale filelist is not detected now. It is detected later, by whoever first instantiates decoder in a module and gets an error about an unresolved module — in a file they just wrote, from a filelist they did not know existed, about a file that has been in the tree for three weeks.

This is the same shape as Chapter 19.2's result that two of three wrong lists compiled, and it generalises to the rule this module keeps arriving at:

The compiler validates sufficiency, never currency. It can tell you the list you gave it is enough to build what you asked for. It cannot tell you the list is the list you should have given it. Nothing in the toolchain can, because the toolchain has no opinion about what your design should contain — so the only defence is that the list is computed from the tree every time and no human is in a position to let it go stale.

Two practical habits that follow from the measurement.

Regenerate, never edit. The generated-file stamp from Chapter 19.1 is the mechanism: a header comment naming the generator and the command to re-run it. It does not stop somebody editing the file, but it means the next person knows the edit is about to be lost.

Never commit a generated filelist. If it is in version control, somebody will edit it, and a reviewer cannot tell an edit from a regeneration. Generate into the run directory — Chapter 17.5's run-directory creation is where this belongs — and let the only copy be the one this run produced.

The measurement for the second habit is the same decoder.sv result: a committed filelist and a stale filelist are indistinguishable to the tool, and an engineer comparing the committed list against the tree will not find the difference either, because git status has nothing to say about a file that nobody edited.

5. What To Remember

No real project builds one thing — RTL, gate-level, assertions on or off, lint.
One description, many filelists. A hash of hashes (7.8), one entry per configuration.
Measured: three configurations, 5, 5 and 6 files, all three compile, with different defines.
Three keys carry it: roots, exclusions, defines.
Exclusions cannot be derived — only the description can say that two files are alternatives.
Defines belong in the description: right files plus wrong defines is a different build.
Do not put a file list in the description. A files => [...] key turns a derived configuration back into a hand-maintained one that still looks generated.
Measured: 4 of 7 distinct files are in every configuration — configurations are mostly identical, which is why copy-paste is tempting and why the real difference hides.
A topological order is a PARTIAL order. An unconstrained file can be emitted anywhere, so two configurations may order shared files differently and both be right.
Measured: a hand merge of rtl and gate gives 'counter' has already been declared — the gate netlist redeclares the module deliberately.
That is the friendly merge. The dangerous one is two configurations with no colliding declaration: it compiles, runs, and is neither configuration.
A named configuration can be wrong but cannot be accidental.
Measured: a new source file appeared; all three generated lists picked it up.
Measured: the stale hand-kept list still compiled — decoder is never instantiated, so nothing objects.
The compiler validates sufficiency, never currency. It cannot know what your design should contain.
Regenerate, never edit — and keep the generated-file stamp (19.1).
Do not commit a generated filelist; generate into the run directory (17.5), so a stale copy cannot exist.

Chapter 19.7 closes the module with the checks that run between the generator and the tool. Three of them are cheap, and each turns one of this module's confusing compile errors into a sentence that names the problem.

Where this fits

Part of the Perl curriculum.