Generating +incdir+ Include Lists

Part II · Module 19 · Chapter 19.4 · Generating +incdir+ Include Lists

1. Derive It In Three Steps

The derivation is mechanical. Ask the sources what they include, find those headers in the tree, and collect the directories.

use strict;
use warnings;
use File::Find;

my @SRC = qw(rtl/core/chip_pkg.sv rtl/core/types_pkg.sv
             rtl/core/counter.sv rtl/core/alu.sv tb/tb_counter.sv);

# ---- 1. which headers do the sources actually ask for?
my %wanted;                       # header name -> which sources include it
for my $f (@SRC) {
    open my $fh, "<", $f or die "$f: $!";
    while (my $l = <$fh>) {
        push @{ $wanted{$1} }, $f if $l =~ /^\s*`include\s+"([^"]+)"/;
    }
    close $fh;
}
print "headers the sources include:\n";
for my $h (sort keys %wanted) {
    printf "  %-14s asked for by %d file(s)\n", $h, scalar @{ $wanted{$h} };
}

# ---- 2. where do those headers live?
my %found;                        # header name -> list of directories holding it
find({ no_chdir => 1, wanted => sub {
    my $p = $File::Find::name;
    return unless -f $p;
    my $base = (split m{/}, $p)[-1];
    return unless $wanted{$base};
    my $dir = $p; $dir =~ s{/[^/]+$}{};
    push @{ $found{$base} }, $dir;
}}, "rtl", "tb");

print "\nwhere each one lives:\n";
for my $h (sort keys %wanted) {
    printf "  %-14s %s\n", $h,
        ($found{$h} ? join(", ", @{ $found{$h} }) : "NOT FOUND IN THE TREE");
}

# ---- 3. the derived include path: the set of those directories
my %dirset;
$dirset{$_} = 1 for map { @{ $found{$_} // [] } } keys %wanted;
my @derived = sort keys %dirset;
printf "\nderived include path (%d dir(s)): %s\n", scalar @derived, join(" ", @derived);

# ---- 4. a header that is included but absent is a FATAL gap, found here
my @missing = grep { !$found{$_} } sort keys %wanted;
printf "headers included but not found in the tree: %s\n",
    (@missing ? join(", ", @missing) : "none");

# ---- 5. compare three include paths against the compiler
sub try_incdirs {
    my ($label, @dirs) = @_;
    open my $o, ">", "i.f" or die $!;
    print $o "$_\n" for @SRC;
    close $o;
    my $flags = join " ", map { "-I $_" } @dirs;
    my $log = `iverilog -g2012 $flags -o /dev/null -c i.f 2>&1`;
    my $rc  = $? >> 8;
    my ($first) = grep { /\S/ } split /\n/, ($log // "");
    printf "  %-34s rc=%-3s %s\n", $label, $rc,
        ($rc == 0 ? "compiles" : ($first // "failed"));
}
print "\nhand each include path to the compiler:\n";
try_incdirs("derived from the sources", @derived);
try_incdirs("none at all");
try_incdirs("a stale hardcoded guess", "rtl/includes");   # note the 's'

# ---- 6. somebody moves a header. The derived path follows; a hardcoded one does not.
print "\n--- move rtl/include/opcodes.svh to rtl/common/ ---\n";
mkdir "rtl/common" unless -d "rtl/common";
rename "rtl/include/opcodes.svh", "rtl/common/opcodes.svh" or die "rename: $!";

my %found2;
find({ no_chdir => 1, wanted => sub {
    my $p = $File::Find::name;
    return unless -f $p;
    my $base = (split m{/}, $p)[-1];
    return unless $wanted{$base};
    my $dir = $p; $dir =~ s{/[^/]+$}{};
    push @{ $found2{$base} }, $dir;
}}, "rtl", "tb");
my %ds2; $ds2{$_} = 1 for map { @{ $found2{$_} // [] } } keys %wanted;
my @derived2 = sort keys %ds2;
printf "re-derived include path (%d dir(s)): %s\n", scalar @derived2, join(" ", @derived2);
try_incdirs("re-derived after the move", @derived2);
try_incdirs("the OLD hardcoded path", "rtl/include");

# ---- 7. two directories, one header name: which wins?
print "\n--- put a SECOND widths.svh in rtl/common (DATA_W 16, not 8) ---\n";
open my $w, ">", "rtl/common/widths.svh" or die $!;
print $w "`ifndef WIDTHS_SVH\n`define WIDTHS_SVH\n`define DATA_W 16\n`endif\n";
close $w;

sub width_seen {
    my (@dirs) = @_;
    open my $o, ">", "w.f" or die $!;
    print $o "$_\n" for @SRC;
    close $o;
    my $flags = join " ", map { "-I $_" } @dirs;
    unlink "w.vvp";
    `iverilog -g2012 $flags -o w.vvp -c w.f 2>&1`;
    return "compile failed" if ($? >> 8) != 0;
    # A 16-bit counter still reaches 4, so the testbench cannot show which
    # header won. Ask the preprocessor directly with a one-line probe.
    open my $t, ">", "probe.sv" or die $!;
    print $t "`include \"widths.svh\"\nmodule probe; initial \$display(\"DATA_W=%0d\", `DATA_W); endmodule\n";
    close $t;
    unlink "p.vvp";
    `iverilog -g2012 $flags -o p.vvp probe.sv 2>&1`;
    return "probe failed" if ($? >> 8) != 0;
    my $out = `vvp p.vvp 2>&1`;
    my ($v) = $out =~ /DATA_W=(\d+)/;
    return defined $v ? $v : "?";
}
printf "  -I rtl/include -I rtl/common  ->  DATA_W = %s\n", width_seen("rtl/include", "rtl/common");
printf "  -I rtl/common -I rtl/include  ->  DATA_W = %s\n", width_seen("rtl/common", "rtl/include");
print "  so the FIRST directory on the command line wins, and a derived list\n";
print "  that is merely sorted alphabetically decides this by accident.\n";

rename "rtl/common/opcodes.svh", "rtl/include/opcodes.svh";
unlink "rtl/common/widths.svh";
rmdir "rtl/common";

headers the sources include:
  opcodes.svh    asked for by 1 file(s)
  widths.svh     asked for by 3 file(s)

where each one lives:
  opcodes.svh    rtl/include
  widths.svh     rtl/include

derived include path (1 dir(s)): rtl/include
headers included but not found in the tree: none

hand each include path to the compiler:
  derived from the sources           rc=0   compiles
  none at all                        rc=1   rtl/core/counter.sv:2: Include file widths.svh not found
  a stale hardcoded guess            rc=1   rtl/core/counter.sv:2: Include file widths.svh not found

--- move rtl/include/opcodes.svh to rtl/common/ ---
re-derived include path (2 dir(s)): rtl/common rtl/include
  re-derived after the move          rc=0   compiles
  the OLD hardcoded path             rc=1   rtl/core/alu.sv:2: Include file opcodes.svh not found

--- put a SECOND widths.svh in rtl/common (DATA_W 16, not 8) ---
  -I rtl/include -I rtl/common  ->  DATA_W = 8
  -I rtl/common -I rtl/include  ->  DATA_W = 16
  so the FIRST directory on the command line wins, and a derived list
  that is merely sorted alphabetically decides this by accident.

Three steps, and each one is a few lines:

Step one: what do the sources ask for? One scan, one anchored pattern:

push @{ $wanted{$1} }, $f if $l =~ /^\s*`include\s+"([^"]+)"/;

That is Chapter 18.2's line scan with Chapter 11.8's anchor and Chapter 11.10's capture. The ^\s* matters: an `include mentioned in a comment mid-line should not create an include directory, and anchoring is the cheapest defence. Chapter 18.11's caveat applies — this is a pattern over source, so a commented-out `include at the start of a line still counts. For this job that is the safe direction to be wrong in: you get a directory you did not strictly need, not a missing one.

Step two: where does each header live? A File::Find walk (Chapter 13.10) that keeps only files whose basename is one the sources asked for. Note the shape — the walk is driven by the wanted set, not by an extension. A header is defined by being included, not by being named .svh.

Step three: collect the directories as a set. Chapter 6.9's hash-as-set again, because several headers in one directory must produce one entry:

my %dirset;
$dirset{$_} = 1 for map { @{ $found{$_} // [] } } keys %wanted;
my @derived = sort keys %dirset;

The measurement shows the result: widths.svh is asked for by three files, opcodes.svh by one, both live in rtl/include, and the derived path is that single directory. Handed to the compiler it builds; with no include path at all, or with a plausible-looking wrong guess, it does not.

2. A Header That Is Included But Absent

Step four of the script is three lines and it is the reason to derive rather than hardcode:

my @missing = grep { !$found{$_} } sort keys %wanted;

The sources asked for a header; the walk did not find it anywhere in the tree. The derivation knows this, and a hardcoded list cannot.

Measured here it reports none, which is the result you want on a healthy tree — and the check is worth keeping precisely because it is usually silent. When it does fire, it means one of three things, and all three are worth a message:

The header is outside the search roots — in a vendor tree, a generated directory, or a sibling project. The fix is a root, not an include directory.

The header does not exist at all — a typo in the `include, or a file deleted in a branch merge. The compiler would eventually say Include file ... not found; saying it from the generator names the source file that wants it, which the compiler's message does too but after a longer walk.

The header is created during the build — a generated configuration header. Then it legitimately does not exist yet, and the generator needs to be told about that directory explicitly.

Reporting it as a warning rather than a fatal error is the right default, because the third case is real. What matters is that it is said.

3. The Move: Derived Follows, Hardcoded Breaks

The second half of the measurement is the case that justifies the whole chapter.

Somebody moves opcodes.svh from rtl/include to rtl/common. Nothing else changes — no source edit, no filelist edit.

The re-derived path picks up two directories (rtl/common and rtl/include) and compiles.

The old hardcoded path fails, and the error is alu.sv:2: Include file opcodes.svh not found.

Read that error as the engineer who gets it would. It names a source file they did not touch and a header they did not touch, and says nothing about a directory having changed. The person who moved the header is not the person who gets the error, and the error does not mention the move. The gap between cause and message is exactly what Chapter 19.1's anchor finding was about, and a derived include path closes it by never having a stale list to begin with.

4. Two Directories, One Header Name

Now the sharp one. Put a second widths.svh in rtl/common, defining DATA_W as 16 instead of 8, and both directories are legitimately on the include path.

-I rtl/include -I rtl/common  ->  DATA_W = 8
-I rtl/common -I rtl/include  ->  DATA_W = 16

The first directory on the command line wins. The same sources, the same filelist, the same tree — and the design is built eight bits wide or sixteen depending on the order of the include path. No warning, no diagnostic; the preprocessor takes the first match and stops looking.

This is why a derived include path needs one more step than "collect and sort". A sorted list is deterministic, which Chapter 17.8 requires, but determinism is not correctness here: rtl/common sorts before rtl/include, so alphabetical order silently chooses the 16-bit definition, and it chose it for no reason anybody decided.

The right response is not to order around it. It is to notice:

for my $h (sort keys %found) {
    next unless @{ $found{$h} } > 1;
    warn "header '$h' exists in " . scalar(@{ $found{$h} }) . " directories: "
       . join(", ", @{ $found{$h} }) . "\n";
    warn "  the include path order decides which one the design uses\n";
}

A header name present in two directories is a question about the design, like Chapter 19.3's module-name clash: either one of them is stale and should be deleted, or they belong to two configurations and should be selected deliberately (Chapter 19.6's job), or the two are genuinely different headers that need different names. A generator cannot decide which. It can refuse to decide silently.

5. What To Remember

• The include path is derivable, not a thing to maintain by hand.
• Three steps: ask the sources what they include, find those headers, collect the directories as a set.
• Anchor the scan with /^\s*`include\s+"([^"]+)"/ so a mid-line mention does not add a directory (11.8, 18.2).
• It is a pattern over source, so a commented-out include at line start still counts (18.11). That errs toward an extra directory, which is the safe direction.
• Drive the header walk from the wanted set, not from an extension — a header is defined by being included.
• Collect with a hash-as-set (6.9) so several headers in one directory give one entry.
• // [] on the lookup is load-bearing — one missing header would otherwise kill the generator with a reference error instead of reporting a header (3.8).
• Report headers that are included but not found. Measured none here; when it fires it means a missing root, a typo, or a build-time generated header — so warn, do not die.
• Measured: derived compiles; no include path fails; a plausible wrong guess fails identically.
• Measured: after moving a header, the re-derived path (2 dirs) compiles and the old hardcoded path fails with Include file opcodes.svh not found.
• That error names a source nobody touched and never mentions the move — the cause-to-message gap a derived list removes.
• Measured: the FIRST -I directory wins — DATA_W = 8 or 16 from identical sources.
• Determinism is not correctness: sorting makes the order stable, and alphabetical order then picks a definition nobody chose.
• Detect a header name in two directories and say so — it is a design question (like 19.3's module clash), not a generator decision.
• A measurement needs an instrument that can distinguish the cases — a 16-bit counter still reaches 4, so the testbench could not answer; a one-line preprocessor probe could.

Chapter 19.5 takes the last and hardest of the four things a filelist carries. Order cannot be derived from filenames or directories at all — it has to come from what the files say about each other.