Linux & Bash · Part II · Module 26 · Compile and Simulation File Lists

Validating a Filelist Before the Tool Sees It

A filelist with one bad path produces a compile error that names the file and then a second, unrelated-looking failure — and in a four-hundred-line log the first is easy to miss. Checking the list first costs a second and turns that into one line. The obvious way to do the check is wrong in three ways.

Intermediate20 min readLinuxPart IIFilelistsValidationScripting

Part II · Module 26 · Chapter 26.5 · Validating a Filelist Before the Tool Sees It

A missing file in a filelist is the cheapest possible failure to diagnose and one of the most commonly mis-diagnosed, because of what the tool does with it.

1. What the Tool Tells You

From Chapter 26.1 §5, a filelist whose paths do not resolve:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cd filelists && iverilog -g2012 -o /tmp/s.vvp -f chip.f
rtl/core/chip_top.sv: No such file or directory
No top level modules, and no -s option.
$ echo $?
1

Two messages. The first names the problem and the second does not, and in a real compile log the first is line 12 of four hundred while the second is the last line — which is the one people read.

Worse, with several missing files a tool may report only the first, or report none and fail later at elaboration with a message about a module that does not exist. The compile error is a downstream symptom, and the filelist is where the fact lives.

2. The Obvious Check, and Its Three Defects

This chapter's own curriculum entry suggests:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

while read -r f; do [ -f "$f" ] || echo "missing: $f"; done < rtl.f

Right idea, and wrong in three ways that each produce a false report.

It treats options as paths. +incdir+rtl/include is not a file, so the check reports it missing — and a validator that cries wolf on every well-formed filelist is one nobody runs.

It loses the last line when there is no trailing newline. Chapter 17.5 §4 measured that read returns false on the final partial line, so the loop never sees it. A hand-edited filelist is exactly the file that ends without a newline, and the entry it silently skips is the one most recently added — the one most likely to be wrong.

And it cannot see an unexpanded variable. $PROJ/rtl/axi/axi_master.sv does not exist as a path, so it is reported as missing. The file is there; the variable was not set. Those are different problems with different fixes, and conflating them sends someone looking for a file that is where it should be.

3. The Four-Case Reader

Each line is one of four things, and each gets its own verdict:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat v.f
# a comment
rtl/core/chip_top.sv
$PROJ/rtl/axi/axi_master.sv
rtl/nope.sv
+incdir+rtl/include
$ while IFS= read -r f || [ -n "$f" ]; do
>   case $f in (''|'#'*|'+'*|'-'*) continue ;; esac
>   case $f in (*'$'*) printf 'UNEXPANDED: %s\n' "$f"; continue ;; esac
>   [ -f "$f" ] || printf 'MISSING:    %s\n' "$f"
> done < v.f
UNEXPANDED: $PROJ/rtl/axi/axi_master.sv
MISSING:    rtl/nope.sv

Two lines of output for two problems, and the comment, the option and the good path produce nothing. That is the whole chapter: a validator's value is in what it does not say.

Four details in those four lines:

|| [ -n "$f" ] so the final newline-less line is processed.

IFS= read -r so a path with leading whitespace or a backslash survives — Chapter 17.5 §2.

The option test comes before the existence test, so +incdir+ never reaches [ -f ].

And UNEXPANDED is checked before MISSING, because a path containing $ cannot meaningfully be tested for existence. Order matters in a validator exactly as it does in a filelist.

Five more things worth checking, and one that is not

A path that exists and is not a regular file. A directory in a filelist is a real mistake — usually a glob that matched a directory — and [ -f ] already catches it, which is why -f rather than -e is right.

A path that exists and is not readable. On a shared filesystem this is common enough to deserve its own message, because the fix is chmod rather than finding the file:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ -r "$f" ] || printf 'UNREADABLE: %s\n' "$f"

An empty file, which compiles silently and contributes nothing. Often a truncated copy or a touch that was meant to be a cp:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ -s "$f" ] || printf 'EMPTY:      %s\n' "$f"

A duplicate, which Chapter 26.4 §4 covers and which belongs in the same pass.

And a path outside the project, which is worth flagging rather than failing: a filelist entry under /home/someone/ compiles today and breaks when they leave.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

case $f in (/*) [ "${f#$PROJECT_ROOT/}" != "$f" ] || printf 'OUTSIDE:    %s\n' "$f" ;; esac

What is not worth checking: whether the file is valid Verilog. That is the compiler's job, it needs a compiler to do, and a validator that attempts it becomes a bad compiler. The division of labour is the point — this check exists to make the compiler's errors meaningful, not to replace them.

4. Following Nested Lists

A validator that does not follow -f checks a fraction of the real list. Chapter 23.3 §4's resolver already has the mechanism, including the base-directory decision that Chapter 26.2 §6 measured going wrong:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

check() {
  local fl=$1 depth=$2 line path anchor
  case $base in (cwd) anchor=$root ;; (filelist) anchor=$(cd -- "$(dirname -- "$fl")" && pwd -P) ;; esac
  local n=0
  while IFS= read -r line || [ -n "$line" ]; do
    n=$(( n + 1 ))
    line=${line%%#*}; line=${line%$'\r'}
    [ -n "${line//[[:space:]]/}" ] || continue
    case $line in
      (-[fF]*)
        path=${line#-[fF]}; path=${path# }
        case $path in (/*) : ;; (*) path=$anchor/$path ;; esac
        if [ -r "$path" ]; then check "$path" $(( depth + 1 ))
        else say FAIL "$fl:$n" "nested list not readable: $path"; fi ;;
      (-*|+*) continue ;;
      (*'$'*) say WARN "$fl:$n" "unexpanded variable: $line" ;;
      (*)
        case $line in (/*) path=$line ;; (*) path=$anchor/$line ;; esac
        [ -e "$path" ] || { say FAIL "$fl:$n" "missing: $line"; continue; }
        [ -f "$path" ] || { say FAIL "$fl:$n" "not a regular file: $line"; continue; }
        [ -r "$path" ] || { say FAIL "$fl:$n" "not readable: $line"; continue; }
        [ -s "$path" ] ||   say WARN "$fl:$n" "empty: $line" ;;
    esac
  done < "$fl"
}

Every message carries file:line, which is the single most useful thing a filelist validator can do: filelists/axi.f:17: missing: rtl/axi/axi_fifo.sv is a complete bug report, and "missing: rtl/axi/axi_fifo.sv" is half of one. Chapter 24.4 §3 made the same point about log lines carrying both a line number and a time.

5. The Validator

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/usr/bin/env bash
# flcheck — validate a filelist before the tool sees it
#
# usage: flcheck [-b cwd|filelist] [-r root] [-w] <filelist>
#   -b  base for relative entries (default: cwd — what iverilog/vcs do)
#   -r  treat this as the project root, and flag entries outside it
#   -w  treat warnings as failures
# exit: 0 clean · 1 problems found · 2 unusable
set -o nounset
export LC_ALL=C

base=cwd; root=; wfatal=0
while getopts ':b:r:w' o; do
  case $o in
    (b) case $OPTARG in (cwd|filelist) base=$OPTARG ;;
          (*) printf 'flcheck: -b must be cwd or filelist\n' >&2; exit 2 ;; esac ;;
    (r) root=$OPTARG ;;
    (w) wfatal=1 ;;
    (:) printf 'flcheck: -%s needs a value\n' "$OPTARG" >&2; exit 2 ;;
    (\?) printf 'flcheck: unknown option -%s\n' "$OPTARG" >&2; exit 2 ;;
  esac
done
shift $(( OPTIND - 1 ))
[ $# -eq 1 ] || { printf 'usage: flcheck [-b cwd|filelist] [-r root] [-w] <filelist>\n' >&2; exit 2; }
[ -r "$1" ] || { printf 'flcheck: cannot read %s\n' "$1" >&2; exit 2; }

cwdroot=$(pwd -P)
[ -z "$root" ] || root=$(cd -- "$root" 2>/dev/null && pwd -P) || {
  printf 'flcheck: cannot resolve root: %s\n' "$root" >&2; exit 2; }

fails=0; warns=0; files=0; seen=" "
say() { printf '%-5s %-28s %s\n' "$1" "$2" "$3"; }

check() {
  local fl=$1 line path anchor n=0
  case $base in
    (cwd)      anchor=$cwdroot ;;
    (filelist) anchor=$(cd -- "$(dirname -- "$fl")" && pwd -P) || return 2 ;;
  esac
  case $seen in (*" $fl "*) say WARN "$fl" "already included: skipping"; warns=$(( warns + 1 )); return 0 ;; esac
  seen="$seen$fl "

  while IFS= read -r line || [ -n "$line" ]; do
    n=$(( n + 1 ))
    line=${line%%#*}; line=${line%$'\r'}
    [ -n "${line//[[:space:]]/}" ] || continue
    case $line in
      (-[fF]*)
        path=${line#-[fF]}; path=${path# }
        case $path in (/*) : ;; (*) path=$anchor/$path ;; esac
        if [ -r "$path" ]; then check "$path"
        else say FAIL "${fl##*/}:$n" "nested list not readable: $path"; fails=$(( fails + 1 )); fi ;;
      (-*|+*) continue ;;
      (*'$'*)
        say WARN "${fl##*/}:$n" "unexpanded variable: $line"; warns=$(( warns + 1 )) ;;
      (*)
        case $line in (/*) path=$line ;; (*) path=$anchor/$line ;; esac
        files=$(( files + 1 ))
        if   [ ! -e "$path" ]; then say FAIL "${fl##*/}:$n" "missing: $line";          fails=$(( fails + 1 ))
        elif [ ! -f "$path" ]; then say FAIL "${fl##*/}:$n" "not a regular file: $line"; fails=$(( fails + 1 ))
        elif [ ! -r "$path" ]; then say FAIL "${fl##*/}:$n" "not readable: $line";     fails=$(( fails + 1 ))
        elif [ ! -s "$path" ]; then say WARN "${fl##*/}:$n" "empty: $line";            warns=$(( warns + 1 ))
        fi
        if [ -n "$root" ]; then
          case $path in ("$root"/*) : ;; (*) say WARN "${fl##*/}:$n" "outside the project root: $line"; warns=$(( warns + 1 )) ;; esac
        fi ;;
    esac
  done < "$fl"
}

check "$1" || exit 2
printf -- '-- %s source entries, %s failure(s), %s warning(s)\n' "$files" "$fails" "$warns"

[ "$fails" -eq 0 ] || exit 1
[ "$wfatal" -eq 0 ] || [ "$warns" -eq 0 ] || exit 1
exit 0

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ flcheck v.f
WARN  v.f:3                        unexpanded variable: $PROJ/rtl/axi/axi_master.sv
FAIL  v.f:4                        missing: rtl/nope.sv
-- 2 source entries, 1 failure(s), 1 warning(s)
$ echo $?
1
$ flcheck filelists/chip.f
-- 4 source entries, 0 failure(s), 0 warning(s)
$ echo $?
0

Eight decisions:

-b is an option with cwd as the default, because Chapter 26.2 §6 measured a tool guessing this wrong and producing filelists/filelists/sub.f. The default matches what iverilog and the commercial tools actually do.

FAIL and WARN are separated, and -w promotes warnings. An unexpanded variable may be perfectly fine — the tool may expand it — so it cannot be a hard failure by default. In CI, -w makes it one.

Every message is file:line, with the filelist's basename to keep the column narrow.

A re-included list is a warning, not a silent skip. The seen set prevents infinite recursion on a cyclic include, and saying so tells you the composition has a loop — which is Chapter 26.6's subject.

The four file tests are an if/elif chain, not four independent tests, so a missing file produces one message rather than four. A validator that reports "missing", "not a regular file" and "not readable" for one absent path is unreadable.

Empty is a warning, because an empty source file is legal and occasionally deliberate.

The count of source entries is printed even when clean, so a run that validated nothing is visible — a 0 source entries, 0 failures line is the signature of a filelist whose paths were all options, or of a -b mismatch that made every line a nested list.

And exit 0 is explicit at the end rather than falling off the end with whatever status the last test left. Chapter 15.6 measured that a script running off the end returns its last command's status, which here would be the [ "$warns" -eq 0 ] test.

One parenthesis too many: (( is arithmetic, not a case pattern

The first version of that -r line was written as:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

case $path in (("$root"/*)) : ;; ((*)) say WARN … ;; esac

and the script would not parse at all:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ./flcheck /tmp/v.f
./flcheck: line 53: syntax error near unexpected token `('
rc=2

Every invocation returned 2 — including the clean list and the usage errors, which made it look as though the argument handling was broken rather than the body.

The cause is over-applying a good rule. Chapter 25.1's callout established that a case pattern inside $( ) needs a leading parenthesis under Bash 3.2, so this script uses (cwd), (-[fF]*), (*'$'*) throughout. Writing a second one produces ((, which the shell reads as the start of an arithmetic expression, and the diagnostic points at the parenthesis rather than at the doubling.

Two things worth taking from it:

bash -n finds this in a second. A syntax check costs nothing and catches the entire class:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -n flcheck && echo "syntax: OK"
syntax: OK

Chapter 20.8 introduced bash -n; this is the case where it would have saved the whole diagnosis, and it belongs in the quality gate from Chapter 21.7 §7 for exactly this reason.

And a uniform status is a signal, not noise. When every path — success, failure and usage error — returns the same code, the fault is almost always before any of them: a parse error, a missing interpreter, a set -e firing in the preamble. Noticing that "even the clean case returns 2" is faster than reading the body.

6. Where to Run It

Three places, and all three are worth it:

In the flow, before the compile. One second, and it converts a late confusing error into an early clear one:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

flcheck -r "$PROJECT_ROOT" "$FILELIST" || { printf 'flow: filelist is not valid\n' >&2; exit 2; }
iverilog -g2012 -o "$WORK_DIR/sim.vvp" -f "$FILELIST"

Note the exit 2: a bad filelist is a setup problem, not a test failure — Chapter 21.4's distinction, and it keeps a regression's failure count honest.

In a pre-commit hook or CI, with -w, so an unexpanded variable or an outside-the-root path is caught before it is committed.

And after generating a list, as the second half of Chapter 26.3's generator — because a generated list can contain a path that existed when find ran and has since been removed, which is a real race in a tree that is being rebased under you.

A validator that passes is not a filelist that works

Three things flcheck cannot tell you, and it is worth being explicit because a green check invites trust:

It cannot check order. Every path can exist and the compile still fail, because a package follows its user — Chapter 26.1 §3's measured failure. Checking that would need a parser.

It cannot see what -y pulls in. A library search path compiles files the list never names, so a validated four-line filelist can compile four hundred files, none of them checked. Chapter 23.3 §4 and Chapter 26.2 §6 both flag -y for exactly this reason.

And it cannot know whether the list is complete. Every named file exists; whether a file that should be named is missing from the list is a different question, and the only mechanical answer is the golden-file check in Chapter 26.3 §6.

So the honest claim for this tool is narrow and worth making precisely: every path the list names resolves to a readable, non-empty regular file, and no variable is unexpanded. That is a real guarantee and it is not "the filelist is correct" — and a tool that claimed the latter would be the false green Chapter 25.1's callout was about.

Practice

Point a compile at a filelist with one bad path. Count the lines of output and find the one that names the file.
Run the registry's one-liner on a real filelist. How many false reports from option lines?
Strip the final newline from a filelist and confirm the one-liner silently skips the last entry.
Put $PROJ/x.sv in a list with PROJ unset and see what the one-liner says about it.
Write the four-case reader and confirm it reports exactly the two real problems.
Swap the UNEXPANDED and MISSING checks and explain what changes.
Add the -r, -s and readability checks. Which fires on your real lists?
Make a filelist entry a directory and confirm -f catches it where -e would not.
Write flcheck and run it on a clean list and a broken one. Confirm 0 and 1.
Run it with the wrong -b and read the output. How many false failures?
Make two filelists include each other and confirm the cycle warning rather than a hang.
Add flcheck to a flow before the compile, with exit 2 on failure. Why 2 and not 1?
Build a filelist whose every line is an option and confirm the 0 source entries line makes that visible.

Check Your Understanding

What two messages does a tool give for a bad filelist path, and which one do people read?
What are the three defects in the obvious one-liner?
Why is an unexpanded variable a different problem from a missing file?
Why must the option test come before the existence test?
Why -f rather than -e?
Why is an unreadable file worth its own message?
What should a validator deliberately not check?
Why does every message carry file:line?
Why are the four file tests an if/elif chain?
Why is a re-included list a warning rather than a silent skip?
Why does the flow exit 2 rather than 1 on a bad filelist?
Name the three things a passing validator does not guarantee.

Answers

1. rtl/core/chip_top.sv: No such file or directory and No top level modules, and no -s option. The second is the last line of the log, so it is the one people read — and it does not name the cause. 2. It treats option lines as paths and reports them missing; it loses the final line when there is no trailing newline; and it reports an unexpanded variable as a missing file. 3. Because the file is there and the variable was not set. The fixes are completely different, and conflating them sends someone looking for a file that is where it should be. 4. So that +incdir+rtl/include never reaches [ -f ] and is not reported as missing. 5. Because -f requires a regular file, so a directory in a filelist — usually a glob that matched one — is caught; -e would accept it. 6. Because the fix is chmod or a group membership, not finding the file, and on a shared filesystem it is common. 7. Whether the file is valid Verilog. That needs a compiler, and a validator that attempts it becomes a bad compiler. The check exists to make the compiler's errors meaningful, not to replace them. 8. Because filelists/axi.f:17: missing: rtl/axi/axi_fifo.sv is a complete bug report and the message without the location is half of one. 9. So one absent path produces one message instead of three — "missing", "not a regular file" and "not readable" are all true of a file that is not there. 10. Because the seen set is needed to prevent infinite recursion on a cyclic include, and saying so reports that the composition has a loop. 11. Because a bad filelist is a setup problem rather than a test failure, and conflating them corrupts a regression's failure count. 12. It cannot check order (a package after its user still fails); it cannot see what -y pulls in; and it cannot know whether the list is complete.

7. Remember

A bad filelist path gives two messages, and the useful one is not the last line of the log.
The obvious one-liner has three defects: options reported as missing, the final newline-less line skipped, and an unexpanded variable called missing.
An unexpanded $VAR is not a missing file. Check for $ before checking existence.
Check options before paths, or every well-formed list reports failures.
-f, not -e — a directory in a filelist is a real mistake.
Give unreadable and empty their own messages: different fixes.
Do not check whether the file is valid HDL. That is the compiler's job.
Every message needs file:line.
Use if/elif for the file tests, so one absent path gives one message.
Follow nested -f lists, with a cycle guard that warns rather than skipping silently.
Separate FAIL from WARN and let CI promote warnings with a flag.
Print the entry count even when clean, so a run that validated nothing is visible.
A bad filelist is exit 2, not 1 — setup, not a test failure.
A passing validator does not mean a correct filelist: not order, not -y, not completeness.
(( is arithmetic, not a doubled case pattern. One leading parenthesis, never two — and bash -n catches it instantly.
If every path returns the same status, suspect the preamble, not the body.

8. What Is Next

Chapter 26.6 closes the module with composition: per-block lists assembled into per-flow ones, so lint, simulation and synthesis read one hierarchy instead of three copies that quietly diverge.

Where this fits

Part of the Linux & Bash curriculum.