Finding RTL Sources and Logs by Pattern

Part II · Module 18 · Chapter 18.5 · Finding RTL Sources and Logs by Pattern

1. The List That Goes Stale

A filelist checked into the repository, or a Python list at the top of a script:

SOURCES = [
    "rtl/alu.sv",
    "rtl/decoder.sv",
    "rtl/core/fifo.sv",
]

print("files listed:", len(SOURCES))
print("files actually in rtl/ today: whatever the last merge left")
print("the two agree until: somebody adds one")

files listed: 3
files actually in rtl/ today: whatever the last merge left
the two agree until: somebody adds one

When a colleague adds rtl/core/arbiter.sv and forgets the filelist, the compile does not fail on a missing file — it succeeds, having compiled three modules instead of four, and fails later with an unresolved reference. The error names the module, not the list, and the connection is not obvious.

The fix is to stop maintaining a list and start describing one.

2. Discovery

sources = sorted(Path("rtl").glob("**/*.sv"))
logs = Path("runs").glob("*/sim.log")

glob takes a pattern and returns the paths that match it:

module alu; endmodule

module decoder; endmodule

module fifo; endmodule

from pathlib import Path

RTL = Path("rtl")

sources = sorted(RTL.glob("**/*.sv"))

print("found", len(sources), "sources:")
for s in sources:
    print("  ", s)

found 3 sources:
   rtl/alu.sv
   rtl/core/fifo.sv
   rtl/decoder.sv

Add a fourth file and the next run finds four, with no edit. That is the entire argument, and the rest of the chapter is about the two ways to get it wrong.

The wildcards:

3. One Level or All of Them

The distinction that decides whether a filelist is complete:

module alu; endmodule

module decoder; endmodule

module fifo; endmodule

module regs; endmodule

from pathlib import Path

RTL = Path("rtl")

top = sorted(RTL.glob("*.sv"))
deep = sorted(RTL.glob("**/*.sv"))
rdeep = sorted(RTL.rglob("*.sv"))

print("one level  (*.sv)   :", len(top), [str(p) for p in top])
print("all levels (**/*.sv):", len(deep))
for p in deep:
    print("   ", p)
print("rglob is the same   :", deep == rdeep)

one level  (*.sv)   : 2 ['rtl/alu.sv', 'rtl/decoder.sv']
all levels (**/*.sv): 4
    rtl/alu.sv
    rtl/core/fifo.sv
    rtl/decoder.sv
    rtl/generated/regs.sv
rglob is the same   : True

Two files versus four, on the same tree. *.sv stopped at the top level and silently omitted everything in subdirectories — which is exactly the stale-filelist failure arriving by a different route, and harder to spot because the code looks like discovery.

rglob(pattern) is shorthand for glob("**/" + pattern), and the last line confirms they agree. Use whichever reads better; rglob is shorter and ** is more explicit about what it does.

A pattern can also constrain the directory level deliberately, which is the point of Section 5.

4. sorted() Is Not Tidiness

The canonical annotation wraps the first call in sorted() and not the second, and the asymmetry is worth taking seriously.

glob returns results in an order chosen by the filesystem. It is not alphabetical, it is not the creation order, and it is not specified by Python — the documentation says the order is arbitrary. In practice it depends on the filesystem's directory structure, which means it can differ between a developer's machine and a compute node even for identical content.

For anything you merely count or search, the order is irrelevant. For anything that becomes an artefact, it is not:

ORDER_A = ["rtl/alu.sv", "rtl/core/fifo.sv", "rtl/decoder.sv"]
ORDER_B = ["rtl/core/fifo.sv", "rtl/alu.sv", "rtl/decoder.sv"]

print("same files       :", set(ORDER_A) == set(ORDER_B))
print("same filelist    :", "\n".join(ORDER_A) == "\n".join(ORDER_B))
print("same sorted form :", sorted(ORDER_A) == sorted(ORDER_B))

same files       : True
same filelist    : False
same sorted form : True

Two discoveries of identical content, two different filelists. Three consequences, all real:

• Compile order can matter. A macro defined in two files, an include that resolves differently, a package compiled after its user — order-dependent in ways that produce confusing failures.
• A diff between two runs is unreadable. Comparing last night's filelist with tonight's shows forty moved lines instead of the one file that was added.
• A build cache keyed on the filelist misses every time, because the key changes when nothing did.

Sort anything that becomes a file, a key or a comparison. The habit is cheap and the failure it prevents is expensive and intermittent.

This is Chapter 15.6 Section 6's rule — deterministic output is a property you have to choose — now applying to a source whose order looks stable because it usually is.

5. Patterns That Name a Level They Do Not Know

The log half of the canonical annotation does something a hard-coded path cannot:

Simulation complete

Simulation complete

Simulation complete

from pathlib import Path

RUNS = Path("runs")

logs = sorted(RUNS.glob("*/sim.log"))

print("found", len(logs), "logs")
for log in logs:
    print(f"  {log.parent.name:<14} {log}")

run_dirs = sorted(p for p in RUNS.glob("*") if p.is_dir())
print("run directories:", [p.name for p in run_dirs])

found 3 logs
  burst_99       runs/burst_99/sim.log
  cdc_7          runs/cdc_7/sim.log
  smoke_1234     runs/smoke_1234/sim.log
run directories: ['burst_99', 'cdc_7', 'smoke_1234']

The pattern */sim.log says: one directory down, whatever it is called, a file named sim.log. The script does not know and does not need to know what runs exist — which is the whole point, since the run names are generated from test names and seeds that vary nightly.

The last line discovers the run directories themselves rather than their logs, using the same pattern with an is_dir() guard — which is how a summariser reports a run that produced no log at all. A discovery of logs alone cannot see the runs that are missing one, and those are Chapter 1.8's NO_LOG cases.

p.parent.name recovers the run directory from the log path, which is how a summariser labels its rows without parsing strings — Chapter 18.3's parts family doing useful work.

Note also what */sim.log excludes: a sim.log directly in runs/, and one nested two levels down. Being specific about the level is usually what you want, because it encodes the tree's shape as a check.

6. What glob Cannot Do

The honest limit, and the reason the next chapter exists.

Look again at Section 3's result: four files, one of which is rtl/generated/regs.sv. Generated sources usually must not go into a hand-built filelist — they are produced by the flow, compiled separately, or deliberately excluded.

The instinct is to find a cleverer pattern. There is not one:

from pathlib import Path

RTL = Path("rtl")
for d in ["core", "generated"]:
    (RTL / d).mkdir(parents=True)
for f in ["alu.sv", "core/fifo.sv", "generated/regs.sv"]:
    (RTL / f).write_text("")

all_sv = sorted(RTL.rglob("*.sv"))
print("everything        :", [str(p) for p in all_sv])

wanted = sorted(p for p in all_sv if "generated" not in p.parts)
print("without generated :", [str(p) for p in wanted])

everything        : ['rtl/alu.sv', 'rtl/core/fifo.sv', 'rtl/generated/regs.sv']
without generated : ['rtl/alu.sv', 'rtl/core/fifo.sv']

The filtering happens in Python, not in the pattern. Glob patterns match names; they have no way to express "except". Nor can they express "only files modified since", "only those over a certain size", or "only those containing a module declaration" — all of which are ordinary flow requirements.

Note "generated" not in p.parts rather than "generated" not in str(p). The second would also reject a file called regenerated.sv, which is Chapter 18.3 Section 4's exact-membership point and a real bug in inherited scripts.

Chapter 18.6 is about doing this properly, including the directories you want to skip entirely rather than walk and discard.

7. Predict, Run, Observe

from pathlib import Path

for d in ["rtl/core", "rtl/generated"]:
    Path(d).mkdir(parents=True)
for f in ["rtl/alu.sv", "rtl/decoder.sv", "rtl/legacy.v",
          "rtl/core/fifo.sv", "rtl/generated/regs.sv"]:
    Path(f).write_text("")

RTL = Path("rtl")
print("A:", len(sorted(RTL.glob("*.sv"))))
print("B:", len(sorted(RTL.glob("**/*.sv"))))
print("C:", len(sorted(RTL.glob("*.*"))))
print("D:", sorted(str(p) for p in RTL.glob("al?.sv")))
print("E:", list(RTL.glob("*.vhd")))
print("F:", sorted(str(p) for p in RTL.glob("**/*") if p.is_dir()))
print("G:", sorted(str(p) for p in RTL.glob("*/*.sv")))

A: 2
B: 4
C: 3
D: ['rtl/alu.sv']
E: []
F: ['rtl/core', 'rtl/generated']
G: ['rtl/core/fifo.sv', 'rtl/generated/regs.sv']

C is 3, not 2 — *.* matches legacy.v as well, because the pattern constrains the shape of the name and not the extension. A filelist built with *.* quietly acquires Verilog files in a SystemVerilog compile.

E is an empty list, not an error. A pattern matching nothing is not a failure, which means a typo in a pattern produces an empty discovery that looks like an empty directory. A script that compiles zero files and reports success is Chapter 1.8's EMPTY_LOG in another form, and worth an explicit check.

F shows glob returns directories too, so is_file() or is_dir() is usually needed when the pattern could match either.

G matches exactly one level down — rtl/core/fifo.sv but not rtl/alu.sv — which is the complement of A rather than a subset of it.

8. Exercises

Exercise 1 — discover a tree. Build a small rtl/ with two levels and find every .sv with both **/*.sv and rglob. Confirm they agree.

Exercise 2 — one level versus all. Count the results of *.sv and **/*.sv on the same tree. Explain what a filelist built from the first would omit.

Exercise 3 — the empty discovery. Glob for an extension that does not exist. Record the result and write the check you would add.

Exercise 4 — order matters. Build a filelist from two different orderings of the same files and diff them. Then sort both and diff again.

Exercise 5 — one log per run. Create three run directories with logs and find them all with */sim.log. Label each with its run name using parent.name.

Exercise 6 — exclude by parts. Filter out a generated directory using p.parts. Then try it with str(p) and find an input where the two disagree.

Exercise 7 — directories in the results. Write a pattern that matches both files and directories, and add the guard that keeps only files.

Exercise 8 — the stale list. Take a checked-in filelist from your flow and compare it against a discovery. Report any difference.

9. Summary

• A hard-coded file list is correct on the day it is written, and its failure mode is a compile that succeeds with a file missing.
• glob describes files rather than listing them, so it cannot go stale.
• * matches within one directory level; ** matches any number — two files versus four on the same tree.
• rglob(p) is shorthand for glob("**/" + p).
• glob returns results in a filesystem-chosen order that is not alphabetical and not guaranteed between machines.
• Sort anything that becomes a file, a key or a comparison — compile order, diffs and build caches all depend on it.
• A pattern can name a level it does not know: runs/*/sim.log finds one log per run without knowing any run's name.
• parent.name recovers the run directory from a discovered log.
• A pattern that matches nothing returns an empty list, not an error, so a typo looks like an empty directory.
• Glob matches names and cannot express "except" — filtering happens in Python, and in p.parts is the exact test where in str(p) would match a substring.

The next chapter does the filtering properly, including the directories worth skipping rather than walking.