Python Part II · Module 35 · Industry VLSI Automation Projects
A Filelist and Project Checker
The filelist is the one input every tool in the project reads, and nothing checks it. A compile failure tells you a file is missing; nothing tells you a file is listed twice, or that the module somebody added last week is in the tree and not in the list — which compiles cleanly and simulates the old behaviour. This chapter builds the checker, and two of its three jobs turn out to depend on facts about the filesystem that the obvious implementation gets wrong.
Advanced25 min readPythonPart IIAutomationFilelistsProject Hygiene
Part II · Module 35 · Chapter 35.7 · A Filelist and Project Checker
1. Three Questions About One File
CHECKS = [
("does every listed file exist?", "the compiler", "immediately",
"a hard error"),
("is anything listed twice?", "nothing", "never",
"a redefinition, or silence"),
("is anything in the tree missing from the list?", "nothing", "never",
"the old behaviour, silently"),
]
print(f"{'the question':<46} {'who asks':<12} {'when':<12} symptom")
for q, who, when, sym in CHECKS:
print(f"{q:<46} {who:<12} {when:<12} {sym}")
print()
print("questions the flow already answers:", 1)
print("questions nothing answers :", 2)
print("and the worst symptom is :", "the third — it compiles")the question who asks when symptom
does every listed file exist? the compiler immediately a hard error
is anything listed twice? nothing never a redefinition, or silence
is anything in the tree missing from the list? nothing never the old behaviour, silently
questions the flow already answers: 1
questions nothing answers : 2
and the worst symptom is : the third — it compilesOnly the first question is answered by anything today, and the third has the worst symptom. A module added to the tree and not to the filelist produces a design that compiles, elaborates and simulates — using whatever was there before — so the regression passes and the new logic is simply absent.
That is the shape Chapter 23.1's chip_a fixture had: three copies of one
module in the tree and one in the filelist, with nothing reporting it.
2. The Annotation
PARTS = [
("files.f", "the filelist to check", "the subject"),
("--root rtl/", "what relative entries resolve against", "required — §4"),
("--strict", "orphans become errors, not warnings", "a policy switch"),
]
print(f"{'part':<16} {'meaning':<42} note")
for part, meaning, note in PARTS:
print(f"{part:<16} {meaning:<42} {note}")
print()
MODES = [
("no --strict", "missing and duplicate are errors; orphans warn", 1),
("--strict", "orphans are errors too", 1),
("a clean project", "nothing reported", 0),
]
print(f"{'mode':<20} {'behaviour':<48} exit")
for mode, behaviour, code in MODES:
print(f"{mode:<20} {behaviour:<48} {code}")
print()
print("why --strict is a flag and not the default:",
"an orphan is sometimes deliberate — §8")
print("why --root has no default :",
"every default would be wrong somewhere — §4")part meaning note
files.f the filelist to check the subject
--root rtl/ what relative entries resolve against required — §4
--strict orphans become errors, not warnings a policy switch
mode behaviour exit
no --strict missing and duplicate are errors; orphans warn 1
--strict orphans are errors too 1
a clean project nothing reported 0
why --strict is a flag and not the default: an orphan is sometimes deliberate — §8
why --root has no default : every default would be wrong somewhere — §4--root is required and --strict is optional, and the asymmetry is
deliberate. Without a root the tool cannot answer any question correctly;
without --strict it can still answer two of three, and the third is a policy
decision rather than a fact.
3. The First Version
from pathlib import Path
# a small project: three modules in the tree, two in the filelist
Path("proj/rtl").mkdir(parents=True, exist_ok=True)
for name in ("a.v", "b.v", "c.v"):
(Path("proj/rtl") / name).write_text(
f"module {name[0]}; endmodule\n", encoding="utf-8")
Path("proj/rtl/files.f").write_text("rtl/a.v\nrtl/b.v\n", encoding="utf-8")
# v1: read the lines and test each one
filelist = Path("proj/rtl/files.f")
entries = [l.strip() for l in filelist.read_text(encoding="utf-8").splitlines()
if l.strip() and not l.strip().startswith("//")]
print("v1 — for each line, check that the path exists")
missing = []
for entry in entries:
ok = Path(entry).exists()
print(f" {entry:<16} exists={ok}")
if not ok:
missing.append(entry)
print()
print(" entries :", len(entries))
print(" missing :", len(missing))
print()
print("every file is really there:", True)
print("v1 reports them missing :", len(missing) == len(entries))
print("so v1 is wrong about :", "all of them")v1 — for each line, check that the path exists
rtl/a.v exists=False
rtl/b.v exists=False
entries : 2
missing : 2
every file is really there: True
v1 reports them missing : True
so v1 is wrong about : all of themBoth files exist and v1 reports both missing. The tool is not subtly wrong on an edge case — it is wrong about every entry, and it is wrong for a reason Section 4 makes precise.
A v1 that happened to be run from the right directory would have reported everything fine, which is worse: it would have been adopted.
4. The Limitation: There Is No Implicit Root
from pathlib import Path
Path("proj/rtl").mkdir(parents=True, exist_ok=True)
for name in ("a.v", "b.v", "c.v"):
(Path("proj/rtl") / name).write_text("module m; endmodule\n",
encoding="utf-8")
filelist = Path("proj/rtl/files.f")
filelist.write_text("rtl/a.v\nrtl/b.v\n", encoding="utf-8")
entries = [l.strip() for l in filelist.read_text(encoding="utf-8").splitlines()
if l.strip()]
BASES = [
("the current directory", Path(".")),
("the filelist's own directory", filelist.parent),
("the filelist's parent's parent", filelist.parent.parent),
]
print(f"{'resolved against':<34} ", end="")
print(" ".join(f"{e:>10}" for e in entries))
for label, base in BASES:
found = [(base / e).exists() for e in entries]
print(f"{label:<34} " + " ".join(f"{str(f):>10}" for f in found))
print()
print("bases tried :", len(BASES))
print("bases that find both files :",
sum(1 for _, b in BASES if all((b / e).exists() for e in entries)))
print()
print("the filelist's own directory works:",
all((filelist.parent / e).exists() for e in entries))
print("which is the base people assume :", True)
print()
print("so a filelist alone is :", "not interpretable")
print("and --root is :", "part of the input, not a flag")resolved against rtl/a.v rtl/b.v
the current directory False False
the filelist's own directory False False
the filelist's parent's parent True True
bases tried : 3
bases that find both files : 1
the filelist's own directory works: False
which is the base people assume : True
so a filelist alone is : not interpretable
and --root is : part of the input, not a flagTwo of three bases find nothing, including the one everybody assumes. The
entries say rtl/a.v and the filelist itself lives in rtl/, so resolving
against the filelist's directory looks for proj/rtl/rtl/a.v.
That combination — entries relative to the project root, in a filelist stored inside a subdirectory — is the normal arrangement, and it means the two implicit bases a tool might choose are both wrong. The only correct behaviour is to take the root as an argument and refuse to guess.
5. The Architecture
Read, resolve, then three independent checks
The resolver keeps both the written spelling and the resolved path, and that
is not redundancy. Every message a user reads must quote the line as they wrote
it — rtl/../rtl/a.v, not the tidy absolute form — or they cannot find it in
the file; and every comparison must use the resolved form, because that is what
the tool will actually open.
6. Reading the Filelist
from pathlib import Path
Path("proj/rtl").mkdir(parents=True, exist_ok=True)
Path("proj/tb").mkdir(parents=True, exist_ok=True)
for p in ("rtl/a.v", "rtl/b.v", "rtl/c.v", "tb/tb_top.v"):
(Path("proj") / p).write_text("module m; endmodule\n", encoding="utf-8")
Path("proj/rtl/files.f").write_text(
"// the axi_fifo rtl filelist\n"
"+incdir+rtl/include\n"
"rtl/a.v\n"
"\n"
"rtl/b.v // the second one\n"
"-f tb/files.f\n", encoding="utf-8")
Path("proj/tb/files.f").write_text(
"// testbench sources\n"
"tb/tb_top.v\n", encoding="utf-8")
def read_filelist(path, root, _seen=None):
"""Flatten a filelist. Returns (entries, directives, includes)."""
seen = _seen if _seen is not None else set()
path = Path(path)
if path.resolve() in seen:
return [], [], [f"{path}: already included — cycle"]
seen.add(path.resolve())
entries, directives, problems = [], [], []
for raw in path.read_text(encoding="utf-8").splitlines():
line = raw.split("//", 1)[0].strip()
if not line:
continue
if line.startswith("+"):
directives.append(line)
continue
if line.startswith("-f"):
nested = line.split(None, 1)[1]
sub, subdir, subprob = read_filelist(Path(root) / nested, root,
seen)
entries.extend(sub)
directives.extend(subdir)
problems.extend(subprob)
continue
entries.append(line)
return entries, directives, problems
entries, directives, problems = read_filelist("proj/rtl/files.f", "proj")
print("entries :", entries)
print("directives:", directives)
print("problems :", problems or "none")
print()
print("comments stripped :", not any("//" in e for e in entries))
print("blank lines dropped :", "" not in entries)
print("trailing comment handled :", "rtl/b.v" in entries)
print("nested filelist expanded :", "tb/tb_top.v" in entries)
print("entry order preserved :", entries == ["rtl/a.v", "rtl/b.v",
"tb/tb_top.v"])entries : ['rtl/a.v', 'rtl/b.v', 'tb/tb_top.v']
directives: ['+incdir+rtl/include']
problems : none
comments stripped : True
blank lines dropped : True
trailing comment handled : True
nested filelist expanded : True
entry order preserved : TrueThree source entries from two files, with the directive kept separately.
+incdir+ is not a source and must not be existence-checked as one — it names
a directory, and treating it as a file is how a checker produces a false
positive on a perfectly good filelist.
The _seen set exists because a filelist can include itself, directly or
through a chain, and the natural recursive reader then does not terminate. The
cycle is reported rather than ignored, because a self-including filelist is
always a mistake.
7. Duplicates: Strings, Paths and Files
from pathlib import Path
Path("proj/rtl").mkdir(parents=True, exist_ok=True)
for name in ("a.v", "b.v"):
(Path("proj/rtl") / name).write_text("module m; endmodule\n",
encoding="utf-8")
ENTRIES = ["rtl/a.v", "./rtl/a.v", "rtl/../rtl/a.v", "rtl//a.v", "rtl/A.v",
"rtl/b.v"]
ROOT = Path("proj")
print("six entries naming two real files")
print()
print(f"{'entry':<20} {'exists':<8} {'resolve() name':<16} file")
identity, order = {}, []
for e in ENTRIES:
p = ROOT / e
st = p.stat()
ident = (st.st_dev, st.st_ino)
if ident not in identity:
identity[ident] = f"file#{len(identity) + 1}"
order.append(ident)
print(f"{e:<20} {str(p.exists()):<8} {p.resolve().name:<16} "
f"{identity[ident]}")
by_string = len(set(ENTRIES))
by_resolve = len({(ROOT / e).resolve() for e in ENTRIES})
by_identity = len(identity)
print()
print("distinct by string :", by_string)
print("distinct by resolve() :", by_resolve)
print("distinct by file identity :", by_identity)
print("real distinct files :", 2)
print()
print("resolve() collapses . .. and //:", True)
print("resolve() collapses CASE :", False)
print("this filesystem is case-insensitive:", True)six entries naming two real files
entry exists resolve() name file
rtl/a.v True a.v file#1
./rtl/a.v True a.v file#1
rtl/../rtl/a.v True a.v file#1
rtl//a.v True a.v file#1
rtl/A.v True A.v file#1
rtl/b.v True b.v file#2
distinct by string : 6
distinct by resolve() : 3
distinct by file identity : 2
real distinct files : 2
resolve() collapses . .. and //: True
resolve() collapses CASE : False
this filesystem is case-insensitive: TrueString comparison says six, resolve() says three, file identity says two.
Only the last is right, and the gap between three and two is the finding: this
filesystem is case-insensitive, so rtl/A.v is rtl/a.v — exists() is
True and the file opens — while resolve() preserves the case you asked for
and therefore reports a different path.
The identifiers printed as file#1 and file#2 are first-seen surrogates. The
real values come from st_dev and st_ino and are specific to this machine and
this filesystem, so they are not published — Chapter 33.14 Section 3's rule
applied to the checker's own output.
from pathlib import Path
Path("proj/rtl").mkdir(parents=True, exist_ok=True)
(Path("proj/rtl") / "a.v").write_text("module a; endmodule\n",
encoding="utf-8")
ENTRIES = ["rtl/a.v", "rtl/A.v", "rtl/../rtl/a.v", "rtl/missing.v"]
ROOT = Path("proj")
def find_duplicates(entries, root):
"""Group by file identity; fall back to resolve() for missing files."""
groups, order, problems = {}, [], []
for entry in entries:
p = root / entry
if not p.exists():
problems.append((entry, "missing"))
continue
st = p.stat()
key = (st.st_dev, st.st_ino)
if key not in groups:
groups[key] = []
order.append(key)
groups[key].append(entry)
dups = [groups[k] for k in order if len(groups[k]) > 1]
return dups, problems
dups, problems = find_duplicates(ENTRIES, ROOT)
print("entries given :", len(ENTRIES))
print("missing :", [e for e, _ in problems])
print("duplicate groups:", len(dups))
for group in dups:
print(" one file listed", len(group), "times as:", group)
print()
print("a missing file cannot be identity-checked:", True)
print("so it is reported as missing, not deduped:", True)
print("and the dedup never silently drops it :", True)
print()
print("the group is reported with the SPELLINGS:",
"so the user can find the lines")entries given : 4
missing : ['rtl/missing.v']
duplicate groups: 1
one file listed 3 times as: ['rtl/a.v', 'rtl/A.v', 'rtl/../rtl/a.v']
a missing file cannot be identity-checked: True
so it is reported as missing, not deduped: True
and the dedup never silently drops it : True
the group is reported with the SPELLINGS: so the user can find the linesThree spellings of one file, grouped, and reported as the user wrote them.
The ordering matters: existence is checked first, because a missing file has
no st_ino to group by, and a dedup pass that called stat() on it would raise
rather than report.
Reporting the spellings rather than the resolved path is what makes the message
actionable. rtl/../rtl/a.v is findable with a text search; the absolute
resolved form is not in the file at all.
8. Orphans Need a Declared Scope
from pathlib import Path
for d in ("proj/rtl", "proj/rtl/include", "proj/tb", "proj/rtl/.old"):
Path(d).mkdir(parents=True, exist_ok=True)
FILES = ["proj/rtl/a.v", "proj/rtl/b.v", "proj/rtl/c.v",
"proj/rtl/scratch.v", "proj/rtl/.old/a_v1.v",
"proj/rtl/include/defs.vh", "proj/tb/tb_top.v"]
for p in FILES:
Path(p).write_text("module m; endmodule\n", encoding="utf-8")
LISTED = ["rtl/a.v", "rtl/b.v", "tb/tb_top.v"]
ROOT = Path("proj")
SOURCE_SUFFIXES = {".v", ".sv"}
IGNORE = [
("rtl/.old/**", "superseded versions, kept for reference", "amit",
"2026-01-20"),
("rtl/scratch.v", "an experiment, not in any build", "priya",
"2026-02-03"),
]
def walk_sources(root):
return sorted(p.relative_to(root).as_posix()
for p in root.rglob("*")
if p.is_file() and p.suffix in SOURCE_SUFFIXES)
def ignored(rel, patterns):
from fnmatch import fnmatch
return any(fnmatch(rel, pat.replace("/**", "/*"))
or rel.startswith(pat.replace("/**", "/"))
for pat, *_ in patterns)
tree = walk_sources(ROOT)
listed_set = set(LISTED)
orphans = [r for r in tree if r not in listed_set]
unexplained = [r for r in orphans if not ignored(r, IGNORE)]
print("source files in the tree :", len(tree))
for r in tree:
mark = ("listed" if r in listed_set
else "ignored" if ignored(r, IGNORE) else "ORPHAN")
print(f" {r:<26} {mark}")
print()
print("listed :", len(listed_set))
print("orphans :", len(orphans))
print("of those, explained by the ignore list:", len(orphans) - len(unexplained))
print("unexplained orphans :", unexplained)
print()
print("the .vh include file is an orphan:", "rtl/include/defs.vh" in orphans)
print("is it a source? :", False)
print("so SOURCE_SUFFIXES excludes it :",
".vh" not in SOURCE_SUFFIXES)source files in the tree : 6
rtl/.old/a_v1.v ignored
rtl/a.v listed
rtl/b.v listed
rtl/c.v ORPHAN
rtl/scratch.v ignored
tb/tb_top.v listed
listed : 3
orphans : 3
of those, explained by the ignore list: 2
unexplained orphans : ['rtl/c.v']
the .vh include file is an orphan: False
is it a source? : False
so SOURCE_SUFFIXES excludes it : TrueThree orphans, two explained, one real. The one real orphan is rtl/c.v —
a module in the tree that no build includes — and it is the finding the whole
check exists for.
The other two are why --strict is a flag rather than the default. rtl/.old/
holds superseded versions somebody deliberately kept, and rtl/scratch.v is an
experiment; reporting them every run trains people to ignore the output, which
is the same dynamic Chapter 35.6 Section 8 described for a --check that always
fails.
The ignore list carries the same four fields as Chapter 35.5's waiver file — pattern, reason, owner, date — for the same reason:
IGNORE = [
("rtl/.old/**", "superseded versions, kept for reference", "amit",
"2026-01-20"),
("rtl/scratch.v", "an experiment, not in any build", "priya",
"2026-02-03"),
]
print(f"{'pattern':<18} {'owner':<8} {'since':<12} reason")
for pat, reason, who, when in IGNORE:
print(f"{pat:<18} {who:<8} {when:<12} {reason}")
print()
print("entries :", len(IGNORE))
print("with a reason :", sum(1 for e in IGNORE if e[1]))
print("with an owner :", sum(1 for e in IGNORE if e[2]))
print("with a date :", sum(1 for e in IGNORE if e[3]))
print()
print("an ignore entry with no reason is:", "a file nobody will ever ask about")
print("and the date lets you :", "re-review it when the block changes")
print("which is 35.5 §9's argument :", "applied to project hygiene")pattern owner since reason
rtl/.old/** amit 2026-01-20 superseded versions, kept for reference
rtl/scratch.v priya 2026-02-03 an experiment, not in any build
entries : 2
with a reason : 2
with an owner : 2
with a date : 2
an ignore entry with no reason is: a file nobody will ever ask about
and the date lets you : re-review it when the block changes
which is 35.5 §9's argument : applied to project hygiene9. The Checker, Assembled
from dataclasses import dataclass
from fnmatch import fnmatch
from pathlib import Path
EXIT_OK, EXIT_PROBLEMS, EXIT_USAGE = 0, 1, 2
SOURCE_SUFFIXES = {".v", ".sv"}
@dataclass(frozen=True)
class Problem:
kind: str
entry: str
detail: str
def read_filelist(path, root, seen=None):
seen = set() if seen is None else seen
path = Path(path)
key = path.resolve()
if key in seen:
return [], [Problem("cycle", str(path), "filelist includes itself")]
seen.add(key)
entries, problems = [], []
if not path.is_file():
return [], [Problem("missing-filelist", str(path), "not a file")]
for raw in path.read_text(encoding="utf-8").splitlines():
line = raw.split("//", 1)[0].strip()
if not line or line.startswith("+"):
continue
if line.startswith("-f"):
nested = line.split(None, 1)[1]
sub, subprob = read_filelist(Path(root) / nested, root, seen)
entries.extend(sub)
problems.extend(subprob)
continue
entries.append(line)
return entries, problems
def check(filelist, root, ignore=(), strict=False):
root = Path(root)
entries, problems = read_filelist(filelist, root)
# 1. existence
present, groups, order = [], {}, []
for entry in entries:
p = root / entry
if not p.is_file():
problems.append(Problem("missing", entry, f"no file at {p}"))
continue
present.append(entry)
st = p.stat()
k = (st.st_dev, st.st_ino)
if k not in groups:
groups[k] = []
order.append(k)
groups[k].append(entry)
# 2. duplicates, by file identity
for k in order:
if len(groups[k]) > 1:
problems.append(Problem("duplicate", groups[k][0],
"also listed as "
+ ", ".join(groups[k][1:])))
# 3. orphans, against a declared scope
listed = {entry for entry in present}
tree = sorted(p.relative_to(root).as_posix() for p in root.rglob("*")
if p.is_file() and p.suffix in SOURCE_SUFFIXES)
resolved_listed = {(root / e).resolve() for e in present}
for rel in tree:
if (root / rel).resolve() in resolved_listed:
continue
if any(fnmatch(rel, pat.replace("/**", "/*"))
or rel.startswith(pat.replace("/**", "/"))
for pat, *_ in ignore):
continue
problems.append(Problem("orphan" if strict else "orphan-warning",
rel, "in the tree, not in the filelist"))
return entries, problems
def report(entries, problems, strict):
errors = [p for p in problems if p.kind != "orphan-warning"]
warnings = [p for p in problems if p.kind == "orphan-warning"]
lines = [f"{len(entries)} entries, {len(errors)} error(s), "
f"{len(warnings)} warning(s)"]
for p in sorted(errors + warnings, key=lambda p: (p.kind, p.entry)):
lines.append(f" [{p.kind}] {p.entry}: {p.detail}")
return "\n".join(lines), (EXIT_PROBLEMS if errors else EXIT_OK)
# --- a project with one of everything
for d in ("proj/rtl", "proj/rtl/.old", "proj/tb"):
Path(d).mkdir(parents=True, exist_ok=True)
for p in ("rtl/a.v", "rtl/b.v", "rtl/c.v", "rtl/.old/a_v1.v", "tb/tb_top.v"):
(Path("proj") / p).write_text("module m; endmodule\n", encoding="utf-8")
Path("proj/rtl/files.f").write_text(
"// axi_fifo\n+incdir+rtl/include\n"
"rtl/a.v\nrtl/A.v\nrtl/b.v\nrtl/gone.v\n-f tb/files.f\n",
encoding="utf-8")
Path("proj/tb/files.f").write_text("tb/tb_top.v\n", encoding="utf-8")
IGNORE = [("rtl/.old/**", "superseded", "amit", "2026-01-20")]
for strict in (False, True):
entries, problems = check("proj/rtl/files.f", "proj", IGNORE, strict)
text, code = report(entries, problems, strict)
print(f"--- --strict {'on' if strict else 'off'}")
for line in text.splitlines():
print(" ", line)
print(" exit:", code)
print()
print("the duplicate found is a CASE variant:", True)
print("the missing entry is :", "rtl/gone.v")
print("the orphan is :", "rtl/c.v")
print("the ignored file never appears :", True)--- --strict off
5 entries, 2 error(s), 1 warning(s)
[duplicate] rtl/a.v: also listed as rtl/A.v
[missing] rtl/gone.v: no file at proj/rtl/gone.v
[orphan-warning] rtl/c.v: in the tree, not in the filelist
exit: 1
--- --strict on
5 entries, 3 error(s), 0 warning(s)
[duplicate] rtl/a.v: also listed as rtl/A.v
[missing] rtl/gone.v: no file at proj/rtl/gone.v
[orphan] rtl/c.v: in the tree, not in the filelist
exit: 1
the duplicate found is a CASE variant: True
the missing entry is : rtl/gone.v
the orphan is : rtl/c.v
the ignored file never appears : TrueAll three problems found, with --strict moving the orphan from a warning to
an error. The duplicate is the case variant from Section 7, which a
string-comparison checker would have missed entirely and a resolve()-based one
would also have missed.
Two details of report are worth copying. Problems are sorted by (kind, entry) so the output is stable run to run — Chapter 33.14 Section 6 — and the
exit code is 1 when there is any error and 0 only when there is none,
regardless of warnings, which is Chapter 33.8's contract: a warning must not
fail a build or nobody can distinguish it from an error.
10. How It Is Tested
import io
import shutil
import tempfile
import unittest
from fnmatch import fnmatch
from pathlib import Path
SOURCE_SUFFIXES = {".v", ".sv"}
def read_entries(text):
out = []
for raw in text.splitlines():
line = raw.split("//", 1)[0].strip()
if line and not line.startswith(("+", "-f")):
out.append(line)
return out
def identity_groups(entries, root):
groups, order = {}, []
for e in entries:
p = root / e
if not p.is_file():
continue
st = p.stat()
k = (st.st_dev, st.st_ino)
if k not in groups:
groups[k] = []
order.append(k)
groups[k].append(e)
return [groups[k] for k in order if len(groups[k]) > 1]
def orphans(root, listed, ignore=()):
resolved = {(root / e).resolve() for e in listed if (root / e).is_file()}
out = []
for p in sorted(root.rglob("*")):
if not p.is_file() or p.suffix not in SOURCE_SUFFIXES:
continue
rel = p.relative_to(root).as_posix()
if p.resolve() in resolved:
continue
if any(fnmatch(rel, pat.replace("/**", "/*"))
or rel.startswith(pat.replace("/**", "/")) for pat in ignore):
continue
out.append(rel)
return out
class Checker(unittest.TestCase):
def setUp(self):
self.root = Path(tempfile.mkdtemp())
self.addCleanup(shutil.rmtree, self.root, True)
(self.root / "rtl").mkdir()
for n in ("a.v", "b.v"):
(self.root / "rtl" / n).write_text("module m; endmodule\n",
encoding="utf-8")
# --- reading
def test_comments_are_stripped(self):
self.assertEqual(read_entries("rtl/a.v // first\n"), ["rtl/a.v"])
def test_a_full_line_comment_is_dropped(self):
self.assertEqual(read_entries("// nothing\nrtl/a.v\n"), ["rtl/a.v"])
def test_blank_lines_are_dropped(self):
self.assertEqual(read_entries("\n\nrtl/a.v\n\n"), ["rtl/a.v"])
def test_incdir_is_not_a_source(self):
self.assertEqual(read_entries("+incdir+rtl/inc\nrtl/a.v\n"),
["rtl/a.v"])
def test_order_is_preserved(self):
self.assertEqual(read_entries("rtl/b.v\nrtl/a.v\n"),
["rtl/b.v", "rtl/a.v"])
# --- resolution
def test_a_relative_entry_needs_the_root(self):
self.assertFalse(Path("rtl/a.v").is_absolute())
self.assertTrue((self.root / "rtl/a.v").is_file())
# --- duplicates
def test_string_duplicates_are_found(self):
self.assertEqual(identity_groups(["rtl/a.v", "rtl/a.v"], self.root),
[["rtl/a.v", "rtl/a.v"]])
def test_dot_and_dotdot_spellings_are_one_file(self):
got = identity_groups(["rtl/a.v", "./rtl/a.v", "rtl/../rtl/a.v"],
self.root)
self.assertEqual(len(got), 1)
self.assertEqual(len(got[0]), 3)
def test_distinct_files_are_not_a_duplicate(self):
self.assertEqual(identity_groups(["rtl/a.v", "rtl/b.v"], self.root),
[])
def test_a_missing_entry_is_skipped_by_the_dedup(self):
self.assertEqual(identity_groups(["rtl/gone.v"], self.root), [])
# --- orphans
def test_an_unlisted_source_is_an_orphan(self):
self.assertEqual(orphans(self.root, ["rtl/a.v"]), ["rtl/b.v"])
def test_an_ignored_orphan_is_not_reported(self):
self.assertEqual(orphans(self.root, ["rtl/a.v"], ("rtl/b.v",)), [])
def test_a_fully_listed_tree_has_no_orphans(self):
self.assertEqual(orphans(self.root, ["rtl/a.v", "rtl/b.v"]), [])
def test_a_non_source_suffix_is_not_an_orphan(self):
(self.root / "rtl" / "defs.vh").write_text("`define X 1\n",
encoding="utf-8")
self.assertEqual(orphans(self.root, ["rtl/a.v", "rtl/b.v"]), [])
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
print("checker suite:", run(Checker))
print()
print("reading tests :", 5)
print("duplicate tests :", 4)
print("orphan tests :", 4)
print()
print("what is NOT asserted here:", "the case-variant duplicate")
print("why :",
"it would fail on a case-sensitive filesystem")
print("so it belongs in :", "a platform-conditional test, or neither")checker suite: {'run': 14, 'failures': 0, 'ok': True}
reading tests : 5
duplicate tests : 4
orphan tests : 4
what is NOT asserted here: the case-variant duplicate
why : it would fail on a case-sensitive filesystem
so it belongs in : a platform-conditional test, or neitherFourteen behaviours pinned, and one deliberately absent. The case-variant
duplicate from Section 7 is the chapter's most interesting finding and it is
not in the suite, because a test asserting that rtl/A.v duplicates
rtl/a.v passes here and fails on Linux — and a suite that fails on half the
team's machines gets disabled.
That is a real decision rather than an omission, and it has a correct form: a test guarded by a runtime probe of the filesystem's behaviour, asserting the checker agrees with the filesystem rather than asserting a fixed answer. The property is platform-independent even though the answer is not, which is the same move Chapter 35.5 made in preferring a property to a value.
test_a_missing_entry_is_skipped_by_the_dedup encodes Section 7's ordering
requirement: existence first, because a missing file has no identity to group
by and stat() on it raises.
11. Predict, Run, Observe
from pathlib import Path
Path("proj/rtl").mkdir(parents=True, exist_ok=True)
(Path("proj/rtl") / "a.v").write_text("module a; endmodule\n",
encoding="utf-8")
ROOT = Path("proj")
# A
print("A:", Path("rtl/a.v").exists(), (ROOT / "rtl/a.v").exists())
# B
print("B:", (ROOT / "rtl/../rtl/a.v").resolve()
== (ROOT / "rtl/a.v").resolve())
# C
upper = ROOT / "rtl/A.v"
print("C:", upper.exists(), upper.resolve() == (ROOT / "rtl/a.v").resolve())
# D
print("D:", upper.stat().st_ino == (ROOT / "rtl/a.v").stat().st_ino)The verified output:
A: False True
B: True
C: True False
D: TrueA — the same entry, unresolvable from the current directory and resolvable
from the root. Section 4 in two booleans, and the reason --root is required.
B — resolve() collapses .., so the three punctuation variants of one
path are correctly identified.
C — the case variant exists and resolve() says it is a different
path. That pair is the Section 7 finding, and it is why resolve() alone is
not a duplicate detector here.
D — and the same two paths share an inode, so file identity gets it right. The two mechanisms disagree, and only one of them agrees with the filesystem.
12. Exercises
Exercise 1 — three questions. Check which of them anything in your flow answers today.
Exercise 2 — the silent one. Describe what happens when a new module is in the tree and not in the filelist.
Exercise 3 — required and optional. Explain why --root is required and
--strict is not.
Exercise 4 — run v1 from the wrong place. Confirm it reports every file missing.
Exercise 5 — the worse outcome. Say why a v1 that happened to work would have been adopted.
Exercise 6 — three bases. Measure which of them finds your own filelist's entries.
Exercise 7 — the assumed base. Confirm whether the filelist's own directory works for you, and explain the layout that breaks it.
Exercise 8 — the simulator's base. Find out what your simulator resolves against.
Exercise 9 — both spellings. Explain why the resolver keeps the written form as well as the resolved one.
Exercise 10 — +incdir+. Confirm your reader does not existence-check it as
a file.
Exercise 11 — a nested filelist. Add a -f include and confirm it is
flattened.
Exercise 12 — a cycle. Make a filelist include itself and check your reader terminates.
Exercise 13 — three counts. Measure distinct-by-string, by resolve() and
by identity on your own filelist.
Exercise 14 — what resolve() collapses. List what it does and does not
normalise.
Exercise 15 — your filesystem. Determine whether it is case-insensitive.
Exercise 16 — existence first. Explain why the dedup pass must run after the existence check.
Exercise 17 — report the spellings. Say why the resolved path is the wrong thing to print.
Exercise 18 — the portability hazard. Describe a filelist that is clean on a laptop and broken on the build server.
Exercise 19 — walk your tree. Count the source files and compare with your filelist.
Exercise 20 — your real orphan. Find one and decide what it is.
Exercise 21 — the ignore list's four fields. Write a real entry.
Exercise 22 — training people to ignore output. Explain what reporting deliberate orphans every run achieves.
Exercise 23 — a warning must not fail a build. Confirm your exit code ignores warnings.
Exercise 24 — stable output. Sort your problems and say which chapter requires it.
Exercise 25 — the absent test. Decide whether to write the case-variant test and justify either way.
Exercise 26 — a property, not an answer. Write the platform-independent version of that test.
13. Summary
- Only one of the three questions is answered by anything today, and the unanswered third has the worst symptom: a module in the tree and not in the filelist compiles, elaborates, and simulates the old behaviour.
- A filelist is not interpretable on its own. Two of three resolution bases found nothing, including the one everybody assumes — entries relative to the project root in a filelist stored inside a subdirectory.
--rootis part of the input, not a convenience flag, and the simulator has the same problem resolved against the current directory, which is why flows wrap every invocation in acd.- Keep both the written spelling and the resolved path: messages must quote what the user typed, comparisons must use what the tool will open.
+incdir+is not a source and existence-checking it produces a false positive on a good filelist.- A filelist can include itself, so the reader needs a seen-set and should report the cycle rather than ignore it.
- Six entries naming two files: string comparison says 6,
resolve()says 3, file identity says 2. Only the last is right. resolve()collapses.,..and//and does not collapse case, and this filesystem is case-insensitive — sortl/A.vopens the same file and resolves to a different path.- Check existence before deduplicating, because a missing file has no identity to group by.
- Inode numbers are machine-specific and are never published — a first-seen surrogate id says everything the reader needs.
- On a case-sensitive filesystem the same filelist behaves differently, which makes this a portability hazard rather than a curiosity.
- An orphan needs a declared scope. Three orphans, two explained by a reviewed ignore list carrying a pattern, reason, owner and date, and one real.
--strictis a flag because an orphan is sometimes deliberate, and reporting deliberate ones every run trains people to ignore the output.- Sort the problems and let only errors set the exit code — a warning that fails a build is indistinguishable from an error.
- One finding is deliberately not in the test suite. A case-variant assertion passes here and fails on Linux, so the correct form probes the filesystem and asserts the checker agrees with it — a property rather than an answer.
The next chapter checks the environment rather than the project: the script an engineer runs before filing a tool bug.
Where this fits
Part of the Python curriculum.