Building Filelists a Tool Will Accept

Part II · Module 18 · Chapter 18.7 · Building Filelists a Tool Will Accept

1. What a Filelist Is

Every simulator accepts a file naming the sources to compile, passed with -f. The format is simple enough to describe in four lines:

# a comment
/proj/chipA/rtl/alu.sv
/proj/chipA/rtl/core/fifo.sv
+incdir+/proj/chipA/rtl/include
-f /proj/chipA/tb/tb.f

One path per line. +incdir+ adds a directory to the include search path. -f nests another filelist. Lines beginning with # or // are comments in most tools.

Beyond that the switches are vendor-specific, and learning them is not durable knowledge. What is durable is the three properties that make the file usable, which are the subject of the rest of the chapter.

2. Writing One

From a discovered and filtered set (Chapters 18.5 and 18.6) to the artefact:

module alu; endmodule

module fifo; endmodule

module decoder; endmodule

from pathlib import Path

ROOT = Path(".").resolve()
RTL = ROOT / "rtl"
SKIP = {"generated", "third_party"}

srcs = sorted(p for p in RTL.rglob("*.sv") if not (SKIP & set(p.parts)))
incdirs = sorted({p.parent for p in srcs})

out = ROOT / "files.f"
with open(out, "w") as f:
    print("# generated -- do not edit", file=f)
    for d in incdirs:
        print(f"+incdir+{d}", file=f)
    for s in srcs:
        print(s, file=f)

text = out.read_text()
print("lines written:", len(text.splitlines()))
print("all absolute :", all(line.startswith("/") for line in text.splitlines()
                            if not line.startswith(("#", "+"))))
print()
for line in text.splitlines():
    if line.startswith(("#", "+")):
        print(" ", line.split("+incdir+")[0] or "+incdir+<abs>")
    else:
        print(" ", Path(line).relative_to(ROOT))

lines written: 6
all absolute : True

  # generated -- do not edit
  +incdir+<abs>
  +incdir+<abs>
  rtl/alu.sv
  rtl/core/fifo.sv
  rtl/decoder.sv

The file itself contains absolute paths; what is printed here is each one made relative to the project root, because the absolute form differs on every machine. The all absolute check confirms what was actually written.

Note incdirs is built as a set comprehension over the sources' parents, then sorted — one include directory per directory containing a source, deduplicated automatically.

3. The Ordering Is the Point

The canonical annotation sorts both loops, and Chapter 18.5 Section 4 gave the reason. Here is the consequence at the artefact:

SRCS_A = ["/p/rtl/alu.sv", "/p/rtl/core/fifo.sv", "/p/rtl/decoder.sv"]
SRCS_B = ["/p/rtl/core/fifo.sv", "/p/rtl/decoder.sv", "/p/rtl/alu.sv"]


def filelist(srcs):
    return "\n".join(["# generated"] + srcs) + "\n"


a, b = filelist(SRCS_A), filelist(SRCS_B)

print("same set of files:", set(SRCS_A) == set(SRCS_B))
print("same filelist    :", a == b)
print("same when sorted :", filelist(sorted(SRCS_A)) == filelist(sorted(SRCS_B)))
print()
print("the diff a reviewer sees between two nightly runs:")
for la, lb in zip(a.splitlines(), b.splitlines()):
    mark = "  " if la == lb else "~ "
    print(f"{mark}{la:<26} | {lb}")

same set of files: True
same filelist    : False
same when sorted : True

the diff a reviewer sees between two nightly runs:
  # generated                | # generated
~ /p/rtl/alu.sv              | /p/rtl/core/fifo.sv
~ /p/rtl/core/fifo.sv        | /p/rtl/decoder.sv
~ /p/rtl/decoder.sv          | /p/rtl/alu.sv

Nothing changed and every line differs. Two discoveries of identical content produced two filelists, and a reviewer comparing last night's against tonight's sees three modified lines rather than the nothing that actually happened.

Three concrete costs, beyond the unreadable diff:

• Compile order can change behaviour — a macro defined in two places, a package compiled after a file that imports it.
• A build cache keyed on the filelist misses every time, because the key changed when the design did not.
• Reproducing a failure becomes unreliable: "the same commit" no longer means "the same compile".

The fix is sorted() and it costs nothing. The habit is to apply it to anything that becomes a file.

4. Absolute Paths, Because the Tool Has Its Own Directory

The most common filelist bug, and it follows directly from Chapter 18.1.

A filelist containing rtl/alu.sv is a relative path. When the simulator reads it, that path is resolved against the simulator's working directory — which is wherever the tool was launched, typically the run directory, not the directory the filelist lives in.

from pathlib import Path

ROOT = Path(".").resolve()
(ROOT / "rtl").mkdir()
(ROOT / "rtl" / "alu.sv").write_text("")
(ROOT / "runs" / "smoke_1234").mkdir(parents=True)

relative = "rtl/alu.sv"
absolute = str(ROOT / "rtl" / "alu.sv")

print("from the project root:")
print("  relative resolves to an existing file:", Path(relative).is_file())
print("  absolute resolves to an existing file:", Path(absolute).is_file())

import os
os.chdir("runs/smoke_1234")
print("from the run directory, where the tool actually runs:")
print("  relative resolves to an existing file:", Path(relative).is_file())
print("  absolute resolves to an existing file:", Path(absolute).is_file())
os.chdir(ROOT)

from the project root:
  relative resolves to an existing file: True
  absolute resolves to an existing file: True
from the run directory, where the tool actually runs:
  relative resolves to an existing file: False
  absolute resolves to an existing file: True

The relative path stopped working the moment the reader moved, which is precisely what happens between the script that writes the filelist and the tool that reads it.

So: write absolute paths into a filelist. They are longer and less pretty and they are correct from any launching directory, which is the only property that matters for an artefact another program consumes.

The same applies to +incdir+ entries, and more sharply — an include directory that does not resolve produces a missing-header error naming a file, several steps from the filelist that was actually wrong.

5. The Filelist Is an Interface

Chapter 15.1 made the case that a file format is an interface between two programs that never meet. A filelist is the same fact with the roles reversed: you are the producer, and the consumer is a tool whose parser you cannot change and whose error messages are rarely about the filelist.

That has practical consequences for what you write into it:

Be conservative about syntax. Comment markers, line continuations and switch spellings vary between VCS, Xcelium and Questa. One path per line and +incdir+ are near-universal; anything else deserves checking against the tool you actually run.

Emit a provenance comment. A generated file that does not say it is generated gets hand-edited, and the edit is lost on the next run:

from pathlib import Path

out = Path("files.f")
with open(out, "w") as f:
    print("# generated by bin/build_filelist.py -- do not edit", file=f)
    print("# regenerate with: make filelist", file=f)
    print("/p/rtl/alu.sv", file=f)

print(out.read_text(), end="")

# generated by bin/build_filelist.py -- do not edit
# regenerate with: make filelist
/p/rtl/alu.sv

Two lines that tell a reader what to do instead of editing. The second one matters more than the first — "do not edit" without "here is how to change it" is an instruction people ignore.

Fail loudly on an empty discovery. Chapter 18.5 Section 7 noted that a pattern matching nothing returns an empty list rather than an error. A filelist with zero sources is syntactically valid and compiles nothing:

from pathlib import Path

srcs = []

if not srcs:
    print("refusing to write a filelist with no sources")
    print("exit status would be:", 2)
else:
    print("writing", len(srcs), "sources")

refusing to write a filelist with no sources
exit status would be: 2

That is Chapter 1.8's EMPTY_LOG reasoning applied to an output: an empty artefact is a distinct failure state and deserves a distinct, non-zero verdict rather than a successful run producing a useless file.

6. Write It Atomically

Chapter 15.6 Section 7 established the pattern and a filelist is exactly the artefact it was meant for:

import os
from pathlib import Path

ROOT = Path(".").resolve()
SOURCES = ["/p/rtl/alu.sv", "/p/rtl/core/fifo.sv", None, "/p/rtl/decoder.sv"]


def write_filelist(path, srcs):
    tmp = path.with_suffix(".f.tmp")
    with open(tmp, "w") as f:
        print("# generated -- do not edit", file=f)
        for s in srcs:
            print(s.strip(), file=f)
    os.replace(tmp, path)


out = ROOT / "files.f"
out.write_text("# last night's good filelist\n")

try:
    write_filelist(out, SOURCES)
except AttributeError as e:
    print("failed partway through:", type(e).__name__)

print()
print("files.f still contains:", out.read_text().strip())
print("a .tmp was left behind:", out.with_suffix(".f.tmp").exists())

failed partway through: AttributeError

files.f still contains: # last night's good filelist
a .tmp was left behind: True

The write died on the third source — a None in the list where a path was expected. Last night's filelist survived, because the real path was never opened for writing; the partial content went to files.f.tmp, and the os.replace that would have published it never ran.

Without the temp-and-rename, files.f would now contain a comment and two sources: a valid filelist naming half the design. The compile would fail on an unresolved module, and the error would be about the RTL rather than about the script that truncated its own output.

The leftover .tmp is deliberate — it is visible evidence that a run failed, where a silently truncated files.f is not.

7. Predict, Run, Observe

from pathlib import Path

ROOT = Path(".").resolve()
for d in ["rtl/core", "rtl/include"]:
    (ROOT / d).mkdir(parents=True)
for f in ["rtl/alu.sv", "rtl/core/fifo.sv"]:
    (ROOT / f).write_text("")
(ROOT / "rtl" / "include" / "defs.svh").write_text("")

srcs = sorted((ROOT / "rtl").rglob("*.sv"))
incdirs = sorted({p.parent for p in srcs})
headers = sorted((ROOT / "rtl").rglob("*.svh"))

print("A:", len(srcs))
print("B:", [str(p.relative_to(ROOT)) for p in incdirs])
print("C:", len(headers))
print("D:", [str(p.relative_to(ROOT)) for p in srcs])
print("E:", (ROOT / "rtl" / "include") in incdirs)

A: 2
B: ['rtl', 'rtl/core']
C: 1
D: ['rtl/alu.sv', 'rtl/core/fifo.sv']
E: False

A and C — headers are not sources. *.sv does not match .svh, so the header is correctly absent from the compile list.

B and E together are the trap. incdirs was derived from the sources' parents, so it contains rtl and rtl/core — and not rtl/include, which is where the header actually lives. A file doing `include "defs.svh" would fail to compile.

Deriving include directories from source locations is a reasonable default and is not sufficient: the directories containing headers must be added too. The fix is to glob for *.svh as well and take those parents, which is one more line and the kind of thing that is obvious only after it has cost an afternoon.

D confirms the sorted order, which is what makes this filelist comparable with tomorrow's.

8. Exercises

Exercise 1 — write one. Discover the sources in a small tree and write a filelist with a provenance comment. Confirm the line count.

Exercise 2 — two orders. Build the same filelist from two orderings and diff them. Then sort and diff again.

Exercise 3 — relative breaks. Write a filelist with relative paths, then check from a run directory whether they resolve. Fix it with absolute paths.

Exercise 4 — include directories. Reproduce Section 7's trap: a header in a directory containing no sources. Then add the line that fixes it.

Exercise 5 — empty discovery. Make the glob match nothing and confirm a filelist is still written. Add the guard and give it an exit status.

Exercise 6 — atomic write. Reproduce Section 6's failure. Confirm the old filelist survived and the .tmp remains.

Exercise 7 — half a design. Remove the temp-and-rename and run the same failing build. Describe the error a simulator would report.

Exercise 8 — provenance. Write the two comment lines for a filelist in your own flow, including the exact command to regenerate it.

9. Summary

• A filelist is one path per line, plus +incdir+ for search paths and -f for nesting; everything else is vendor-specific.
• Sort both the sources and the include directories — an unsorted artefact is a different artefact every run.
• Unsorted output costs an unreadable diff, a build cache that never hits, and "the same commit" no longer meaning the same compile.
• Write absolute paths, because the tool resolves relative ones against its working directory, not the filelist's.
• The same applies to +incdir+, where the resulting error is a missing header several steps from the cause.
• A filelist is an interface to a tool you did not write — be conservative about syntax.
• Emit a provenance comment saying it is generated and how to regenerate it.
• An empty discovery produces a valid filelist that compiles nothing — refuse it with a non-zero status.
• Write to a temporary name and rename, or a dying script leaves a valid filelist naming half the design.
• Include directories derived from source locations miss header-only directories, which is a compile failure pointing at the wrong thing.

The next chapter creates the directory a run writes into, which has its own version of the half-finished problem.