Nested Filelists and Reuse Across Flows

Part II · Module 26 · Chapter 26.6 · Nested Filelists and Reuse Across Flows

Lint wants the design. Simulation wants the design plus a testbench. Synthesis wants the design minus anything non-synthesisable. Three lists, one design — and the only arrangement that does not drift is one where the design is written down once.

1. One Tool, Two Base Directories

Before composing anything, a fact that decides how every nested path must be written. Chapter 26.1 §5 measured that source paths are relative to the current directory. Nested filelist paths are not, and the same tool does both.

A filelist in filelists/ whose nested path is written relative to the repository root:

$ cat filelists/n1.f
-f filelists/blocks/axi.f
$ iverilog -g2012 -o /tmp/b1.vvp -f filelists/n1.f
Error: unable to read nested command file (filelists/filelists/blocks/axi.f) at filelists/n1.f:1.
$ echo $?
1

filelists/filelists/blocks/axi.f — the path was appended to the containing list's own directory. Written relative to that directory instead:

$ cat filelists/n2.f
-f blocks/axi.f
$ iverilog -g2012 -o /tmp/b2.vvp -f filelists/n2.f
rtl/axi/axi_slave.sv:2: Include file chip_defs.svh not found

The nested list was found — the error has moved on to a different subject entirely, which is the include path.

And source paths, re-confirmed in the same session:

$ iverilog -g2012 -o /tmp/a.vvp -f filelists/srconly.f
$ echo $?
0
$ cd filelists && iverilog -g2012 -o /tmp/a2.vvp -f srconly.f
rtl/core/chip_top.sv: No such file or directory
$ echo $?
1

Plus +incdir+, which follows the source-path rule:

$ cat filelists/bad_inc.f
+incdir+../rtl/include
-f blocks/axi.f
$ iverilog -g2012 -o /tmp/d.vvp -f filelists/bad_inc.f
rtl/axi/axi_slave.sv:2: Include file chip_defs.svh not found
$ echo $?
1

Two base directories, in one tool, with nothing in the file to say which applies. That is the single most surprising fact in this module, and it is why composition feels fragile until you know it.

2. The Hierarchy

Given §1, the layout writes itself: nested paths relative to the filelist, source paths relative to the repository root.

filelists/
├── blocks/
│   ├── core.f      rtl/core/chip_top.sv, rtl/core/clk_div.sv
│   ├── axi.f       rtl/axi/axi_master.sv, rtl/axi/axi_slave.sv
│   └── uart.f      rtl/uart/uart_tx.sv
├── design.f        +incdir+rtl/include, -f blocks/core.f, -f blocks/axi.f, -f blocks/uart.f
├── sim.f           -f design.f, tb/tests/tb_chip.sv
└── lint.f          -f design.f

$ cat filelists/design.f
+incdir+rtl/include
-f blocks/core.f
-f blocks/axi.f
-f blocks/uart.f
$ cat filelists/sim.f
-f design.f
tb/tests/tb_chip.sv
$ cat filelists/lint.f
-f design.f

And it compiles, from the repository root:

$ iverilog -g2012 -o /tmp/c.vvp -f filelists/sim.f
$ echo $?
0
$ vvp /tmp/c.vvp < /dev/null | tail -2
TEST PASSED
tb/tests/tb_chip.sv:12: $finish called at 35 (1s)

Three levels — block, design, flow — and the design is written down once.

3. The Divergence This Prevents, Measured

The argument for sharing is usually made by assertion. Here it is measured both ways.

With a shared block list. Add a file to blocks/axi.f and ask what each flow sees:

$ printf 'rtl/axi/axi_master.sv\nrtl/axi/axi_slave.sv\nrtl/axi/axi_fifo.sv\n' > filelists/blocks/axi.f
$ ./flat filelists/sim.f  | wc -l
       7
$ ./flat filelists/lint.f | wc -l
       6

Six and seven — the difference is the testbench, as it should be. Both flows picked up the new file, because both read the same blocks/axi.f. Nobody had to remember anything.

With an independent copy. Give lint its own axi_lint_copy.f that was correct yesterday:

$ ./flat filelists/sim.f      | wc -l
       7
$ ./flat filelists/lint_bad.f | wc -l
       5
$ diff <(./flat filelists/lint_bad.f | sort) <(./flat filelists/sim.f | sort)
0a1
> rtl/axi/axi_fifo.sv
5a7
> tb/tests/tb_chip.sv

Two differences where there should be one. axi_fifo.sv is in the design and lint does not see it, so the new block is never linted — and nothing fails, which is the point. The flows agree about everything except the one file that matters, and the discrepancy surfaces whenever somebody eventually looks.

That is the whole case for composition: a shared list cannot drift, and two copies always do — not because anyone is careless, but because the two copies are edited by different people for different reasons on different days.

4. Flattening, and Why You Need It

A composed hierarchy is unreadable as a hierarchy. Two questions need a flat answer: "what does this flow compile?" and "how do two flows differ?"

#!/usr/bin/env bash
# flat — print every source a filelist pulls in, following nested lists
# usage: flat [-b cwd|filelist] <filelist>
# exit:  0 ok · 2 unusable
set -o nounset
export LC_ALL=C

base=filelist                            # the measured default for iverilog
while getopts ':b:' o; do
  case $o in
    (b) case $OPTARG in (cwd|filelist) base=$OPTARG ;;
          (*) printf 'flat: -b must be cwd or filelist\n' >&2; exit 2 ;; esac ;;
    (\?) printf 'flat: unknown option -%s\n' "$OPTARG" >&2; exit 2 ;;
  esac
done
shift $(( OPTIND - 1 ))
[ $# -eq 1 ] || { printf 'usage: flat [-b cwd|filelist] <filelist>\n' >&2; exit 2; }
[ -r "$1" ] || { printf 'flat: cannot read %s\n' "$1" >&2; exit 2; }

root=$(pwd -P)
seen=" "

walk() {
  local fl=$1 line path here anchor
  here=$(cd -- "$(dirname -- "$fl")" && pwd -P) || return 2
  case $base in (cwd) anchor=$root ;; (filelist) anchor=$here ;; esac
  case $seen in (*" $fl "*) printf '# cycle: %s already included\n' "$fl" >&2; return 0 ;; esac
  seen="$seen$fl "

  while IFS= read -r line || [ -n "$line" ]; do
    line=${line%%#*}; line=${line%$'\r'}
    [ -n "${line//[[:space:]]/}" ] || continue
    case $line in
      (-[fF]*)
        path=${line#-[fF]}; path=${path# }
        case $path in (/*) : ;; (*) path=$anchor/$path ;; esac
        if [ -r "$path" ]; then walk "$path"
        else printf '# unreadable nested list: %s\n' "$path" >&2; fi ;;
      (-*|+*) continue ;;
      (*) printf '%s\n' "$line" ;;
    esac
  done < "$fl"
}

walk "$1"

$ flat filelists/sim.f
rtl/core/chip_top.sv
rtl/core/clk_div.sv
rtl/axi/axi_master.sv
rtl/axi/axi_slave.sv
rtl/uart/uart_tx.sv
tb/tests/tb_chip.sv
$ diff <(flat filelists/lint.f | sort) <(flat filelists/sim.f | sort)
5a6
> tb/tests/tb_chip.sv

One line of difference between lint and simulation, and it is the testbench. That diff is the artifact worth putting in a review, and it is the thing nobody can produce from a hierarchy by reading it.

Six decisions:

base defaults to filelist, which is §1's measured behaviour for this tool — and the option exists because other tools differ and a filelist does not say.

The cycle guard writes to stderr and returns 0, so a cyclic include produces a complete list plus a diagnostic rather than a hang or a truncated list. Chapter 25.3 §4's visited set, with the spaces around $fl for the substring reason that chapter gives.

Options are skipped, not printed. This tool answers "which sources", and Chapter 26.2 §6's flopts answers the other half. Two tools, two questions.

Order is preserved and nothing is sorted, because the output is the compile order — which Chapter 26.1 §3 established is load-bearing. Sorting is the caller's choice, which is why the diff above pipes through sort explicitly.

An unreadable nested list is a stderr note and the walk continues, so one broken branch does not hide the rest of the list. That is the opposite of Chapter 23.3 §4's resolver, which returns 2 — and the difference is deliberate: that one produces a list to compile from, where a partial list is dangerous, and this one produces a report, where partial information beats none.

And export LC_ALL=C, so that a caller's sort on the output is stable — Chapter 26.4 §2.

5. What Each Flow Actually Needs

Composition is only useful if the per-flow lists differ in the right way:

The first two compose cleanly. Synthesis is the one that does not, and it is worth being honest about why: a design frequently contains code that simulates and does not synthesise — a $display, an initial block, a behavioural model of a memory — and the usual mechanisms are a `ifdef SYNTHESIS inside the file or a separate block list naming a synthesisable variant.

blocks/
├── axi.f           the simulation view
└── axi_syn.f       the synthesis view: axi_master_syn.sv instead of the model

That is duplication, and it is the kind §3 warns about. The honest positions are two:

Prefer `ifdef and one file list, so there is nothing to diverge; the cost is `ifdefs in the RTL, which reviewers dislike.

Or accept two block lists and check they agree on everything except the deliberate difference:

diff <(flat filelists/blocks/axi.f | sort) <(flat filelists/blocks/axi_syn.f | sort)

A diff that is expected to be non-empty still needs checking, because what you want to know is whether it is the same non-empty diff as last week. That is a golden-file check on a diff — Chapter 26.3 §6's pattern, applied to the divergence rather than to the list.

6. Module 26 in One Page

• 26.1 — the single statement of what is compiled. It exists because a command line cannot hold 40,000 paths (measured: status 126), and it is a sequence: swapping two lines turned a working compile into exit 2.
• 26.2 — +incdir+ and +define+ inside the list take effect, measured; moving a define there makes it visible, not authoritative, because an unguarded header `define still wins.
• 26.3 — find -printf works at your prompt and fails in your script, because the prompt's find is bfs. Pin the tools; six things any generation rule gets wrong.
• 26.4 — find's order is stable and name-hashed, not random; sort -u f > f leaves zero bytes; the locale reorders a mixed-case list completely.
• 26.5 — the obvious validator is wrong in three ways, an unexpanded variable is not a missing file, and a passing check does not mean a correct list.
• 26.6 — one tool, two base directories; and a shared block list cannot drift while two copies always do.

Three themes ran through all six, and they are the module's content.

A filelist is an artifact with semantics, not a list of strings. Order, option scope, and two different base directories are all invisible in the file and all change what gets compiled.

Every tool that reads one must handle four kinds of line, and every shortcut that treats them all as paths produced a confident wrong answer — in the registry's own suggested one-liner, in my first flatten-filelist, and in flopts.

And the authority is the tool, not the artifact. Chapter 23.3 §3 established that in the context of finding a file; this module needed it twice more, and the second time it corrected a note already written in Chapter 26.2.

Module 27 takes the list and runs the tool: compile and elaborate as separate steps, where the output goes, and the arguments that differ between vendors.

Practice

1. Write a filelist in a subdirectory with a nested -f path relative to the repository root. What error do you get?
2. Rewrite that path relative to the filelist and confirm the nested list is found.
3. Confirm source paths are current-directory-relative by compiling the same list from two directories.
4. Make a +incdir+ path filelist-relative and confirm the include is not found.
5. Build the three-level hierarchy from §2 and compile it.
6. Write flat and print what each flow sees. Is the diff exactly what you expect?
7. Add a file to a shared block list and confirm both flows pick it up without any other edit.
8. Give one flow a private copy of a block list, add a file to the other, and diff. How many differences?
9. Check your own project: does any flow list name design sources directly?
10. Check whether any two block lists overlap, with flat … | sort | uniq -d.
11. Make two lists include each other and confirm flat reports a cycle rather than hanging.
12. If your project has a synthesis view, diff it against the simulation view and decide whether the difference is the one you expect.
13. Save that diff as a golden file and make the flow check it.

Check Your Understanding

1. What three kinds of filelist entry are there, and which base directory does each use?
2. What error names the doubled path, and what does it tell you?
3. Why was the note in Chapter 26.2 wrong, and what rule would have caught it?
4. Why does the -b option still exist if this tool's behaviour is known?
5. How should the three levels of the hierarchy be arranged?
6. What did adding a file to a shared block list demonstrate?
7. What did the independent copy demonstrate, and why is it dangerous rather than merely untidy?
8. Why does flat preserve order rather than sorting?
9. Why does flat continue past an unreadable nested list when the Chapter 23.3 resolver exits 2?
10. Which flow does not compose cleanly, and what are the two honest responses?
11. What are the three composition mistakes, and what one rule avoids all of them?

7. Remember

• One tool, two base directories: source and +incdir+ paths are current-directory-relative; nested -f paths are relative to the containing filelist.
• Nothing in a filelist says which convention it was written for. Keep the -b option.
• The nested-list error names the resolved path and the line, so a doubled component is visible — read it.
• Three levels: blocks name files, the design list includes blocks, flow lists include the design list.
• Nothing points sideways or upward.
• A shared block list cannot drift. Measured: adding a file gave lint 6 and simulation 7, the difference being the testbench.
• Two copies always drift, and the failure is silent — the new block is simply never linted.
• Flatten to compare flows. A one-line diff between lint and simulation is a reviewable artifact.
• Preserve order when flattening; let the caller sort.
• A report may continue past a broken branch; a list to compile from may not.
• Synthesis does not compose cleanly. Either `ifdef or golden-file the diff between two views.
• Check that flow lists name no sources and that block lists do not overlap — flat … | sort | uniq -d.
• Guard against cycles, which arise from the same good instinct that makes composition work.
• The authority is the tool, not the artifact. This module needed that rule twice, and the second time it corrected a chapter already written.

8. What Is Next

The list is correct, ordered, validated and composed. Module 27 hands it to the tool: compile and elaborate as separate steps, where the output goes, incremental builds, and the arguments that differ between vendors.