Generating a Filelist from the Tree

Part II · Module 26 · Chapter 26.3 · Generating a Filelist from the Tree

A hand-maintained filelist is wrong the first time somebody adds a file and does not update it. A generated one is never incomplete and is frequently wrong in a different way. Both are defensible; what is not defensible is not knowing which you have.

1. The Obvious Command Fails in a Script

This chapter's own curriculum entry suggests:

find rtl -name '*.sv' -printf '%p\n' | sort > rtl.f

At an interactive prompt on this machine it works. In a script it does not, and the reason is the same one Chapter 23.5 §3 measured for grep:

$ type -a find
find is a shell function from /Users/engineer/.../snapshot-zsh-….sh
find is /usr/bin/find
$ find --version | head -1
bfs 4.1.1

The find at the prompt is a shell function dispatching to bfs, a modern reimplementation that supports GNU's -printf. The real find does not:

$ /usr/bin/find rtl -name '*.sv' -printf '%p\n' > /tmp/o.txt 2> /tmp/e.txt
$ echo $?
1
$ cat /tmp/e.txt
find: -printf: unknown primary or operator
$ wc -c < /tmp/o.txt
0

Status 1, zero bytes of output. And a script gets that one:

$ bash -c 'command -v find'
/usr/bin/find

So the suggested command produces a filelist interactively and an empty filelist in a flow — which then fails at compile time with no source files, a hundred lines away from the cause.

-printf is a GNU extension. The portable spelling is -print, which is also the default:

$ /usr/bin/find rtl -name '*.sv' -print | sort
rtl/axi/axi_master.sv
rtl/axi/axi_slave.sv
rtl/core/chip_top.sv
rtl/core/clk_div.sv
rtl/uart/uart_tx.sv

Five files, and %p was only ever "the path", which -print already gives.

2. The Six Things Generation Gets Wrong

A rule like "every .sv under rtl/" is never exactly the design. Six specific gaps, in the order they cause trouble:

1. Order. find | sort is alphabetical, and Chapter 26.1 §3 measured that a package after its user does not compile. Alphabetical order is a deterministic order, not a correct one — pkg.sv sorts after axi_master.sv.

2. Files that are not part of the design. A *.sv under rtl/ may be an old copy, a work-in-progress, a testbench someone left there, or — as Chapter 23.1 measured — a stale duplicate in a generated directory the rule did not exclude.

3. Files that are part of the design and not under rtl/. Generated RTL, vendor IP, a package in common/.

4. Headers. A .svh that is `included must not be in the list — compiling it standalone is usually an error — while one that is a compilation unit must be. The rule cannot tell them apart from the extension.

5. Conditional membership. A design with two configurations compiles different files. One generated list cannot express that; the generator needs a parameter.

6. And the options. Chapter 26.2's +incdir+ and +define+ lines are not files and cannot be generated from the tree at all.

So a generated list is a starting point that needs an ordering rule, an exclusion rule, an inclusion rule and a header rule — at which point it is no longer "just find", and the honest generators in real projects are forty to a hundred lines.

3. Exclusions, and the Two Good Ways

The exclusion problem is Chapter 23.5's, and the same two answers apply.

A positive list of roots, which cannot go stale:

"$FIND" rtl tb -name '*.sv' -print | sort

Or let git decide, which is better when the repository's .gitignore is maintained:

git ls-files -z 'rtl/**/*.sv' 'tb/**/*.sv' | tr '\0' '\n' | sort

git ls-files excludes exactly the untracked and ignored files, so a stale copy in a run directory cannot appear — and Chapter 23.5 §2 measured that it finds the real sources and nothing else. Its limitation is the one that matters for generated RTL: a file that is generated and not committed is invisible to it, so a flow that generates RTL must add those paths separately.

And the deny-list form, which is what most projects actually have:

"$FIND" . -name '*.sv' \
  \( -path './work/*' -o -path './csrc/*' -o -path './run_*' -o -path './.git/*' \) -prune \
  -o -print | sort

Chapter 14.6's prune idiom, with its two details: -prune before the -o, and the final -print because any action disables the implicit one. It is the weakest of the three, for Chapter 23.5 §2's measured reason — it excludes what you remembered.

4. Ordering: What To Do About §2's First Gap

Four approaches, in increasing order of effort and correctness.

Alphabetical, and make it work. Name packages so they sort first — 00_pkg/, or a pkgs/ directory listed before rtl/. Crude, effective, and visible:

{ "$FIND" rtl/pkgs -name '*.sv' -print | sort
  "$FIND" rtl/core rtl/axi rtl/uart -name '*.sv' -print | sort
} > rtl.f

A hand-written prologue plus a generated body, which is the commonest real arrangement:

cat filelists/prologue.f > rtl.f          # options, packages, interfaces — by hand
"$FIND" rtl -name '*.sv' ! -path '*/pkgs/*' -print | sort >> rtl.f

Topological order from the source, which means parsing — import p::* implies a dependency on the file defining package p. That is a real tool, not a shell script, and several exist.

Or let the tool do it. Some simulators accept files in any order and resolve packages themselves, or offer a two-pass mode. Where that is true, ordering is a non-problem — and finding out whether it is true for your tool is worth ten minutes, because it makes everything above unnecessary.

5. The Generator

#!/usr/bin/env bash
# genlist — generate a filelist from the tree, with a declared rule
#
# usage: genlist [-o out] [-p prologue] <root>...
#   -o  write here (default: stdout)
#   -p  prepend this file verbatim (options, packages, interfaces)
# exit: 0 wrote a non-empty list · 1 no sources found · 2 bad usage
set -o nounset

FIND=${FIND:-/usr/bin/find}
out=; prologue=

while getopts ':o:p:' opt; do
  case $opt in
    (o) out=$OPTARG ;;
    (p) prologue=$OPTARG ;;
    (:) printf 'genlist: -%s needs a value\n' "$OPTARG" >&2; exit 2 ;;
    (\?) printf 'genlist: unknown option -%s\n' "$OPTARG" >&2; exit 2 ;;
  esac
done
shift $(( OPTIND - 1 ))

[ $# -ge 1 ] || { printf 'usage: genlist [-o out] [-p prologue] <root>...\n' >&2; exit 2; }
for r in "$@"; do
  [ -d "$r" ] || { printf 'genlist: not a directory: %s\n' "$r" >&2; exit 2; }
done
[ -z "$prologue" ] || [ -r "$prologue" ] || {
  printf 'genlist: cannot read prologue: %s\n' "$prologue" >&2; exit 2; }

emit() {
  if [ -n "$prologue" ]; then
    printf '# ---- from %s ----\n' "$prologue"
    cat -- "$prologue"
    printf '\n'
  fi
  printf '# ---- generated by genlist from: %s ----\n' "$*"
  printf '# rule: *.sv and *.v, excluding */pkgs/* and anything untracked\n'
  LC_ALL=C "$FIND" "$@" \( -name '*.sv' -o -name '*.v' \) ! -path '*/pkgs/*' -print |
    LC_ALL=C sort
}

n=$(emit "$@" | grep -cvE '^[[:space:]]*(#|$)')
[ "$n" -gt 0 ] || { printf 'genlist: no sources under %s\n' "$*" >&2; exit 1; }

if [ -n "$out" ]; then
  tmp=$(mktemp "${out%/*}/.genlist.XXXXXX") || exit 2
  trap 'rm -f -- "$tmp"' EXIT
  emit "$@" > "$tmp" || exit 2
  mv -- "$tmp" "$out" || exit 2
  trap - EXIT
  printf 'genlist: %s (%s source lines)\n' "$out" "$n" >&2
else
  emit "$@"
fi

$ genlist -o /tmp/rtl.f rtl
genlist: /tmp/rtl.f (5 source lines)
$ cat /tmp/rtl.f
# ---- generated by genlist from: rtl ----
# rule: *.sv and *.v, excluding */pkgs/* and anything untracked
rtl/axi/axi_master.sv
rtl/axi/axi_slave.sv
rtl/core/chip_top.sv
rtl/core/clk_div.sv
rtl/uart/uart_tx.sv
$ genlist -o /tmp/none.f docs; echo "rc=$?"
genlist: no sources under docs
rc=1

Eight decisions, each with a reason:

FIND is pinned and overridable, for §1's measured reason.

LC_ALL=C on both find and sort. Chapter 26.4 measures why: without it the order depends on the user's locale, so two engineers generate two different lists from one tree.

The rule is written into the file as a comment. A generated list that does not say how it was generated is a list nobody dares regenerate. This is the single highest-value line in the script.

A prologue is prepended verbatim, which is §4's second approach and keeps the hand-ordered part reviewable and separate.

An empty result is exit 1, not an empty file. A generator that writes a zero-line list on a typo'd root produces Chapter 26.1's no source files a hundred lines later. Failing here names the cause.

The count excludes comments, so "5 source lines" means five sources — a count that included the header would be a small lie that compounds.

Write to a temporary and mv, so an interrupted run cannot leave a half-written filelist that compiles to a half-written design. Chapter 20.6 is mktemp; the mv is atomic within a filesystem, which is why the temporary is created next to the output rather than in /tmp.

And trap - EXIT after the successful mv — which I wrote believing it was load-bearing, and it is not. Removing it and running the generator leaves the output intact:

$ sed '/trap - EXIT/d' genlist > genlist-notrap && chmod +x genlist-notrap
$ ./genlist-notrap -o /tmp/nt.f rtl
genlist: /tmp/nt.f (5 source lines)
$ ls /tmp/nt.f
/tmp/nt.f

The file is there. mv -- "$tmp" "$out" renames the temporary, so by the time the trap fires $tmp no longer exists and rm -f on it is a no-op — -f suppresses the error, so nothing is even printed.

The line is therefore tidiness, not a fix: it stops the trap from running a pointless command, and it would matter if the temporary were ever copied rather than moved. It is kept because an explicit disarm documents that the temporary's life has ended, and because a future edit from mv to cp would turn the harmless no-op into a real deletion.

The general point is the one this module keeps arriving at: a defensive line whose failure mode you have not reproduced may be defending against nothing. Checking cost one sed and one run.

6. Generated or Committed?

The decision this chapter exists to inform, and it is not "generated is better".

The question that decides it: when a file appears in the tree that should not be compiled, do you want the build to include it or to ignore it? A generated list includes it; a committed list ignores it. Neither is wrong, and a project should know which it chose.

The common and good middle position: generate into the run directory, commit a checksum. The list is always complete, and a change in it is visible:

genlist -o "$WORK_DIR/rtl.f" rtl tb
if ! cmp -s "$WORK_DIR/rtl.f" filelists/rtl.f.expected; then
  printf 'the generated filelist differs from the committed expectation:\n' >&2
  diff filelists/rtl.f.expected "$WORK_DIR/rtl.f" >&2
  printf 'if this is intended, copy the new list over the expectation and commit it\n' >&2
  exit 1
fi

That is Chapter 21.8's golden-file pattern applied to a filelist: the build fails when the set of compiled files changes, and the fix is a reviewed commit. It catches a new file that nobody meant to add and a deleted file nobody noticed — which neither pure approach does.

Practice

1. Run type -a find and find --version. Which find is at your prompt?
2. Run bash -c 'command -v find'. Is it the same one?
3. Try /usr/bin/find . -name '*.sv' -printf '%p\n' and record the status and the byte count of its output.
4. Run bash --noprofile --norc -c 'command -v find grep sed awk sort'. Any surprises?
5. Generate a filelist for a tree you have. How many files, and how many should not be there?
6. For each of §2's six gaps, say whether it applies to your project.
7. Find a .svh in your tree and determine whether it is `included or compiled. How did you tell?
8. Compare find-based generation with git ls-files. Which files differ, and why?
9. Does your project generate any RTL? Would git ls-files miss it?
10. Write the prune-based deny-list and find a generated directory it misses.
11. Write genlist and run it. Confirm the rule comment is in the output.
12. Run it on a directory with no sources and confirm exit 1 with no file written.
13. Remove the trap - EXIT line and check whether the output file survives. Explain the result in terms of what mv did to the temporary.
14. Set up the golden-file check from §6, add a new .sv, and confirm the build fails with a readable diff.

Check Your Understanding

1. Why does find -printf work at the prompt and fail in a script here?
2. What is the portable spelling, and what did %p add?
3. What is the general pattern behind both this and the grep/ugrep divergence?
4. What two lines fix it, and why does it matter more in a generator than elsewhere?
5. What are the six things a generation rule gets wrong?
6. Why can the rule not tell an includable .svh from a compilable one?
7. What are the three exclusion approaches, and what does git ls-files miss?
8. Why LC_ALL=C on both find and sort?
9. Why is an empty generated list an error rather than an empty file?
10. Why write to a temporary and mv, and why create the temporary next to the output?
11. Is trap - EXIT after the mv load-bearing? Test it.
12. What question decides between a committed and a generated list?

7. Remember

• find -printf is a GNU extension. Measured: /usr/bin/find rejects it with status 1 and zero output, while the prompt's find is a function dispatching to bfs and accepts it.
• A script gets /usr/bin/find, so the suggested command yields an empty filelist in a flow.
• Use -print, which is also the default.
• Pin the tools: FIND=${FIND:-/usr/bin/find}. The same trap exists for grep, sed, cat, ls.
• bash --noprofile --norc -c 'command -v …' shows what the flow will use.
• Six gaps in any generation rule: order, unwanted files, missed files, headers, conditional membership, options.
• Alphabetical is deterministic, not correct. A package sorts after its user.
• Three exclusion approaches: positive roots, git ls-files, a prune deny-list — and git ls-files misses generated RTL.
• LC_ALL=C on find and sort, or the list depends on the user's locale.
• Write the generation rule into the file as a comment.
• An empty result is exit 1, never an empty file.
• Generate to a temporary beside the output and mv — atomic within a filesystem, so an interrupt cannot leave a half-written list.
• trap - EXIT after the mv is tidiness, not a fix. Measured: mv already removed the temporary, so the trap's rm -f is a no-op. A defensive line whose failure you have not reproduced may be defending against nothing.
• Committed versus generated is a real choice: does an unexpected file get compiled or ignored?
• Generate into the run directory and commit a checksum — a golden-file check catches both additions and deletions.

8. What Is Next

Chapter 26.4 takes determinism seriously: what find's unsorted order actually is on this filesystem, why sort -u file > file destroys the file, and the locale measurement that makes two engineers' generated lists differ.