Python Part II · Module 24 · EDA Environment Engineering
Artefacts, Temporary Data and What to Keep
Chapter 23.7 decided what a single run should delete, and Chapter 24.3 moved the bulk of it somewhere it could be deleted wholesale. What neither settled is how long the survivors live. A regression that keeps everything is not making a careful choice — it is making the same choice as a regression with no policy at all, only more slowly, and the day the volume fills somebody deletes things in a hurry with no rule at all. This chapter writes the rule down, and it starts by showing that the obvious form of it throws away the waveform.
Advanced18 min readPythonPart IIAutomationEDA ToolsEnvironment
Part II · Module 24 · Chapter 24.4 · Artefacts, Temporary Data and What to Keep
1. The Obvious Policy
KEEP = {"sim.log", "compile.log", "cmd.sh", "waves.fsdb"}
# everything else in the run dir is disposableA set of names and one comment. It is readable, it is reviewable, it puts the decision in one place a team can argue about, and it is the right shape — Chapter 23.7 Section 6 argued for exactly this over a cleanup spread through a script.
It is also, applied literally, a bug.
2. What a Real Run Directory Contains
KEEP = {"sim.log", "compile.log", "cmd.sh", "waves.fsdb"}
ACTUAL = ["sim.log", "compile.log", "cmd.sh", "waves_axi_smoke_1839274.fsdb",
"axi_smoke.vcd", "record.json", "simv", "csrc", "ucli.key"]
print(f"{'file in the run directory':<32} kept?")
for name in ACTUAL:
print(f"{name:<32} {name in KEEP}")
kept = sum(1 for n in ACTUAL if n in KEEP)
print()
print(f"kept {kept} of {len(ACTUAL)}")file in the run directory kept?
sim.log True
compile.log True
cmd.sh True
waves_axi_smoke_1839274.fsdb False
axi_smoke.vcd False
record.json False
simv False
csrc False
ucli.key False
kept 3 of 9The waveform is deleted. The annotation names waves.fsdb and the file on disk is waves_axi_smoke_1839274.fsdb, because Chapter 24.3 Section 2 required the test and seed in every name that can collide — and a wave file absolutely can. The one artefact a failing run exists to produce is the one the policy throws away.
record.json is lost too, which is the file Chapter 21.1 built the whole reporting chain on.
This is worth dwelling on because of how it fails. The policy is readable and reviewable, somebody did write it down, and it looks right when you read it. It is wrong about the data, and nothing in a code review surfaces that — you would have to have a real run directory in front of you.
A keep-list is a claim about filenames. Test it against real ones.
3. Patterns, Not Names
import fnmatch
KEEP_PATTERNS = ("*.log", "cmd.sh", "record.json", "*.fsdb", "*.vcd", "env.json")
ACTUAL = ["sim.log", "compile.log", "cmd.sh", "waves_axi_smoke_1839274.fsdb",
"axi_smoke.vcd", "record.json", "simv", "csrc", "ucli.key"]
def keep_reason(name):
return next((p for p in KEEP_PATTERNS if fnmatch.fnmatch(name, p)), None)
print(f"{'file':<32} kept by")
for name in ACTUAL:
print(f"{name:<32} {keep_reason(name)}")
print()
print("kept :", sum(1 for n in ACTUAL if keep_reason(n)))
print("disposed:", [n for n in ACTUAL if not keep_reason(n)])file kept by
sim.log *.log
compile.log *.log
cmd.sh cmd.sh
waves_axi_smoke_1839274.fsdb *.fsdb
axi_smoke.vcd *.vcd
record.json record.json
simv None
csrc None
ucli.key None
kept : 6
disposed: ['simv', 'csrc', 'ucli.key']Six kept, three disposed, and the three disposed are the right three: the simulator binary, the generated C tree, and a lock file.
Two properties of this form earn their keep.
keep_reason returns which pattern matched, not just a boolean. A sweeper that logs the reason is one you can debug when it keeps something it should not, and the reason is the line of policy to change.
fnmatch.fnmatch matches a name against a shell-style pattern, which is the same syntax everyone already reads in a .gitignore or a Makefile. Chapter 19 would do this with a regular expression; a regular expression is more powerful and, for a list of file extensions a team maintains, Chapter 19.12's advice applies — the simpler tool that everyone can edit is the better one.
4. Three Questions, Not One
POLICY_QUESTIONS = [
("what", "which files are worth keeping at all", "Chapter 23.7 §6"),
("when", "does the answer depend on how the run ended", "Chapter 23.7 §6"),
("how long", "for how many days do the survivors survive", "this chapter"),
]
for name, question, where in POLICY_QUESTIONS:
print(f"{name:<9} {question:<48} {where}")
print()
print("a policy that answers only the first two:")
print(" keeps every passing run's log forever")
print(" keeps every failing run's waveform forever")
print(" fills the volume at a slower rate, and then fills it")what which files are worth keeping at all Chapter 23.7 §6
when does the answer depend on how the run ended Chapter 23.7 §6
how long for how many days do the survivors survive this chapter
a policy that answers only the first two:
keeps every passing run's log forever
keeps every failing run's waveform forever
fills the volume at a slower rate, and then fills itChapter 23.7 Section 6 answered what and when, and that is a genuine improvement over deleting by habit. It is still unbounded. A nightly producing four hundred runs keeps four hundred logs a day whether or not anybody reads them, and the difference between that and no policy is a multiplier, not a limit.
The third question is what turns it into a policy:
Keep this, in this circumstance, for this long.
And the horizon differs per artefact, which is the point of asking separately:
RETENTION_DAYS = {
"*.log": 90, # small; the evidence trail, worth months
"cmd.sh": 90, # tiny; how to reproduce anything above
"record.json": 365, # tiny; the data a trend report reads
"*.fsdb": 7, # large; debugged within a week or not at all
"*.vcd": 7,
"env.json": 365, # tiny; which tools ran
}
print(f"{'pattern':<14} {'days':>5} rationale")
for pattern, days in sorted(RETENTION_DAYS.items(), key=lambda kv: kv[1]):
size = "large" if days <= 7 else "small"
print(f"{pattern:<14} {days:>5} {size}, kept {'briefly' if days <= 7 else 'long'}")
print()
print("longest horizon :", max(RETENTION_DAYS.values()), "days")
print("shortest horizon:", min(RETENTION_DAYS.values()), "days")
print("the short ones are the large ones:",
all(d <= 7 for p, d in RETENTION_DAYS.items() if p.endswith(("fsdb", "vcd"))))pattern days rationale
*.fsdb 7 large, kept briefly
*.vcd 7 large, kept briefly
*.log 90 small, kept long
cmd.sh 90 small, kept long
record.json 365 small, kept long
env.json 365 small, kept long
longest horizon : 365 days
shortest horizon: 7 days
the short ones are the large ones: TrueThe horizon is inversely related to the size, which is what makes the policy affordable. A year of record.json files is a few megabytes and is what every trend report in Chapter 29 will read. A week of waveforms is most of the disk.
5. Age, Measured
import os
import time
from pathlib import Path
DAY = 86400
now = time.time()
base = Path("runs/retention")
base.mkdir(parents=True, exist_ok=True)
for name, age_days in (("sim.log", 1), ("waves_axi_smoke_1839274.fsdb", 3),
("waves_axi_reset_771002.fsdb", 30), ("old_sim.log", 120)):
path = base / name
path.write_text("x" * 10, encoding="utf-8")
stamp = now - age_days * DAY
os.utime(path, (stamp, stamp))
RETENTION_DAYS = {"*.log": 90, "*.fsdb": 7}
import fnmatch
def age_days(path, reference):
return (reference - path.stat().st_mtime) / DAY
def horizon_for(name):
for pattern, days in RETENTION_DAYS.items():
if fnmatch.fnmatch(name, pattern):
return days
return 0
print(f"{'file':<32} {'age':>6} {'horizon':>8} expired")
for path in sorted(base.iterdir()):
age = age_days(path, now)
horizon = horizon_for(path.name)
print(f"{path.name:<32} {age:>5.0f}d {horizon:>7}d {age > horizon}")file age horizon expired
old_sim.log 120d 90d True
sim.log 1d 90d False
waves_axi_reset_771002.fsdb 30d 7d True
waves_axi_smoke_1839274.fsdb 3d 7d FalseTwo expired, two not, and the two that expired are a four-month-old log and a one-month-old waveform.
The ages here are constructed with os.utime rather than waited for, which is the only way to demonstrate a retention policy in a reproducible example. The same technique is how you test one: a sweeper whose behaviour you can only observe by waiting ninety days is a sweeper nobody tests.
Every number printed is a difference from the script's own clock reading. No timestamp appears, which keeps the output identical on every machine and every day — Chapter 19.16's rule about publishing figures, applied to time.
6. Report Before You Remove
import fnmatch
import os
import time
from pathlib import Path
DAY = 86400
now = time.time()
RETENTION_DAYS = {"*.log": 90, "cmd.sh": 90, "record.json": 365, "*.fsdb": 7}
base = Path("runs/sweep")
base.mkdir(parents=True, exist_ok=True)
for name, age, size in (("sim.log", 1, 2_000), ("old.log", 120, 3_000),
("cmd.sh", 120, 100), ("record.json", 120, 500),
("waves_a.fsdb", 30, 1_048_576), ("simv", 1, 40_000)):
path = base / name
path.write_bytes(b"x" * size)
os.utime(path, (now - age * DAY, now - age * DAY))
def horizon_for(name):
for pattern, days in RETENTION_DAYS.items():
if fnmatch.fnmatch(name, pattern):
return days
return 0
def plan_sweep(root, reference):
"""Decide what would go. Deletes nothing."""
doomed, total = [], 0
for path in sorted(Path(root).rglob("*")):
if not path.is_file():
continue
age = (reference - path.stat().st_mtime) / DAY
horizon = horizon_for(path.name)
if age > horizon:
size = path.stat().st_size
doomed.append((path.name, round(age), horizon, size))
total += size
return doomed, total
doomed, total = plan_sweep(base, now)
print(f"{'file':<16} {'age':>5} {'keep':>5} {'bytes':>9}")
for name, age, horizon, size in doomed:
print(f"{name:<16} {age:>4}d {horizon:>4}d {size:>9,}")
print()
print(f"would remove {len(doomed)} files, {total:,} bytes ({total / 1048576:.2f} MiB)")
print("files still on disk:", len([p for p in base.rglob('*') if p.is_file()]))file age keep bytes
cmd.sh 120d 90d 100
old.log 120d 90d 3,000
simv 1d 0d 40,000
waves_a.fsdb 30d 7d 1,048,576
would remove 4 files, 1,091,676 bytes (1.04 MiB)
files still on disk: 6Nothing was deleted. The function returns a plan, and the last line proves all six files are still there.
That separation is not politeness; it is what makes the sweeper deployable. A program that walks a shared results volume deleting things is about as destructive as unattended automation gets, and three things follow from writing it this way.
It can be run in anger before anyone trusts it. Point it at the real volume, read the list, argue about the three surprising entries, change the patterns.
The byte total answers the question people actually ask. "Should we enable this?" is really "how much would it free?", and a plan that reports 1.04 MiB on a test tree reports something far more persuasive on a real one.
simv shows up with a horizon of 0d — it matches no pattern, so nothing keeps it and it goes on the first sweep regardless of age. That is the default this policy wants, and seeing it in the plan is how you confirm the default is doing what you think.
7. Keep the Newest, Not Just the Newest-Enough
import os
import time
from pathlib import Path
DAY = 86400
now = time.time()
base = Path("runs/newest")
base.mkdir(parents=True, exist_ok=True)
for i, age in enumerate((0, 1, 2, 3, 8, 20)):
path = base / f"waves_run{i}.fsdb"
path.write_bytes(b"x" * 1000)
os.utime(path, (now - age * DAY, now - age * DAY))
waves = sorted(base.glob("*.fsdb"), key=lambda p: p.stat().st_mtime, reverse=True)
KEEP_NEWEST = 3
print("newest first:", [p.name for p in waves])
print("keep :", [p.name for p in waves[:KEEP_NEWEST]])
print("delete :", [p.name for p in waves[KEEP_NEWEST:]])
print()
print("under a 7-day horizon alone, this many survive:",
sum(1 for p in waves if (now - p.stat().st_mtime) / DAY <= 7))newest first: ['waves_run0.fsdb', 'waves_run1.fsdb', 'waves_run2.fsdb', 'waves_run3.fsdb', 'waves_run4.fsdb', 'waves_run5.fsdb']
keep : ['waves_run0.fsdb', 'waves_run1.fsdb', 'waves_run2.fsdb']
delete : ['waves_run3.fsdb', 'waves_run4.fsdb', 'waves_run5.fsdb']
under a 7-day horizon alone, this many survive: 4Age and count answer different questions, and a good policy uses both.
An age horizon alone has no upper bound on volume. A bad week producing two hundred failing runs puts two hundred waveforms inside the seven-day window, all legitimately young, all retained.
A count alone has no lower bound on usefulness. Keep-the-newest-three on a quiet month discards a waveform from eight days ago that is still the only example of a rare failure.
The combination — keep the newest N, and nothing older than D regardless — bounds both the disk and the surprise. The sort is by st_mtime descending, and reverse=True on the sort rather than reversing afterwards is the readable form.
8. Never Delete What You Cannot Regenerate
ARTEFACTS = [
("sim.log", False, "the only record of what happened"),
("record.json", False, "derived from the log, but the log may be gone"),
("cmd.sh", False, "the instructions for regenerating everything else"),
("waves_*.fsdb", True, "re-run cmd.sh with waves enabled"),
("simv", True, "re-run the compile step in cmd.sh"),
("csrc/", True, "produced by the compile step"),
("obj_dir/", True, "produced by the compile step"),
]
print(f"{'artefact':<16} {'regenerable':<12} how")
for name, regenerable, how in ARTEFACTS:
print(f"{name:<16} {str(regenerable):<12} {how}")
print()
keepers = [n for n, r, _ in ARTEFACTS if not r]
print("never-delete set:", keepers)
print("and every 'how' above depends on:", "cmd.sh" in keepers)artefact regenerable how
sim.log False the only record of what happened
record.json False derived from the log, but the log may be gone
cmd.sh False the instructions for regenerating everything else
waves_*.fsdb True re-run cmd.sh with waves enabled
simv True re-run the compile step in cmd.sh
csrc/ True produced by the compile step
obj_dir/ True produced by the compile step
never-delete set: ['sim.log', 'record.json', 'cmd.sh']
and every 'how' above depends on: TrueThat is the rule that makes aggressive deletion defensible:
Delete freely what you can regenerate. Keep what you cannot, and keep the thing that regenerates the rest.
Every entry in the regenerable column says re-run cmd.sh, which means Chapter 23.10's one-line artefact is carrying the entire policy. Deleting a four-gigabyte build tree is a safe decision because a hundred-byte file can reproduce it; delete that hundred-byte file and the four gigabytes become irreplaceable retrospectively.
Chapter 24.6 closes the remaining gap, and it is worth naming here: re-run cmd.sh assumes the same tools are still installed. The command reproduces the run only to the extent the environment has not moved, which is why the environment goes in the record too.
9. The Sweeper
import fnmatch
import os
import shutil
import time
from pathlib import Path
DAY = 86400
RETENTION_DAYS = {"*.log": 90, "cmd.sh": 90, "record.json": 365, "*.fsdb": 7}
KEEP_NEWEST = {"*.fsdb": 2}
def horizon_for(name):
for pattern, days in RETENTION_DAYS.items():
if fnmatch.fnmatch(name, pattern):
return days
return 0
def sweep(root, reference, dry_run=True):
"""Return (plan, bytes). Deletes only when dry_run is False."""
root = Path(root).resolve()
protected = set()
for pattern, count in KEEP_NEWEST.items():
matches = sorted((p for p in root.rglob("*")
if p.is_file() and fnmatch.fnmatch(p.name, pattern)),
key=lambda p: p.stat().st_mtime, reverse=True)
protected.update(matches[:count])
plan, freed = [], 0
for path in sorted(root.rglob("*")):
if not path.is_file() or path in protected:
continue
age = (reference - path.stat().st_mtime) / DAY
horizon = horizon_for(path.name)
if age <= horizon:
continue
size = path.stat().st_size
plan.append((str(path.relative_to(root)), round(age), horizon, size))
freed += size
if not dry_run:
path.unlink()
return plan, freed
now = time.time()
base = Path("runs/final")
base.mkdir(parents=True, exist_ok=True)
for name, age, size in (("sim.log", 1, 100), ("old.log", 200, 100),
("cmd.sh", 200, 50), ("waves_a.fsdb", 2, 900),
("waves_b.fsdb", 9, 900), ("waves_c.fsdb", 40, 900),
("simv", 1, 5000)):
p = base / name
p.write_bytes(b"x" * size)
os.utime(p, (now - age * DAY, now - age * DAY))
plan, freed = sweep(base, now, dry_run=True)
print("DRY RUN")
for name, age, horizon, size in plan:
print(f" {name:<16} {age:>4}d (keep {horizon}d) {size:>6,} bytes")
print(f" would free {freed:,} bytes from {len(plan)} files")
print(" still on disk:", len([p for p in base.rglob('*') if p.is_file()]))
plan, freed = sweep(base, now, dry_run=False)
print()
print("FOR REAL")
print(" removed:", len(plan), "files,", f"{freed:,}", "bytes")
print(" left :", sorted(p.name for p in base.rglob('*') if p.is_file()))DRY RUN
cmd.sh 200d (keep 90d) 50 bytes
old.log 200d (keep 90d) 100 bytes
simv 1d (keep 0d) 5,000 bytes
waves_c.fsdb 40d (keep 7d) 900 bytes
would free 6,050 bytes from 4 files
still on disk: 7
FOR REAL
removed: 4 files, 6,050 bytes
left : ['sim.log', 'waves_a.fsdb', 'waves_b.fsdb']Read the two halves together: the dry run named exactly four files and 6,050 bytes, and the live run removed exactly those four and exactly that many bytes. The plan is the contract, and a sweeper whose dry run and live run can disagree is one nobody should enable.
Four details in sweep are worth copying.
cmd.sh is removed, and that is correct. It is 200 days old against a 90-day horizon, so the policy says go — and it is worth noticing because Section 8 called cmd.sh the thing that regenerates everything else. Both are true: the never-delete rule protects it from being deleted as disposable, and the retention horizon eventually expires it as old. If a 200-day-old cmd.sh should outlive a 200-day-old log, say so by giving it a longer horizon — the policy is one dict and the fix is one number.
KEEP_NEWEST protects before the age test runs. waves_a.fsdb and waves_b.fsdb are the two most recent dumps and survive; waves_c.fsdb at 40 days does not. Note that waves_b.fsdb is 9 days old against a 7-day horizon — the count protected a file the age rule would have deleted, which is Section 7's lower bound on usefulness doing its job.
simv goes on the first sweep at one day old, because it matches no pattern and gets the zero horizon. Seeing it in the dry run is how you confirm the default is what you intended.
protected is computed once, over the whole tree, before the deletion loop. Computing it inside the loop would be both slower and wrong — the set of "newest two" must be decided against the full listing, not against whatever is left at the moment each file is considered.
10. Predict, Run, Observe
import fnmatch
import os
import time
from pathlib import Path
DAY = 86400
now = time.time()
d = Path("runs/pro")
d.mkdir(parents=True, exist_ok=True)
for name, age in (("a.log", 5), ("b.fsdb", 5), ("c.fsdb", 50)):
p = d / name
p.write_bytes(b"x" * 10)
os.utime(p, (now - age * DAY, now - age * DAY))
# A
print("A:", "waves_axi_1.fsdb" in {"waves.fsdb"})
# B
print("B:", fnmatch.fnmatch("waves_axi_1.fsdb", "*.fsdb"))
# C
print("C:", fnmatch.fnmatch("cmd.sh", "*.log"))
# D
R = {"*.log": 90, "*.fsdb": 7}
print("D:", next((v for k, v in R.items() if fnmatch.fnmatch("x.fsdb", k)), 0))
# E
print("E:", next((v for k, v in R.items() if fnmatch.fnmatch("simv", k)), 0))
# F
print("F:", round((now - (d / "c.fsdb").stat().st_mtime) / DAY))
# G
ordered = sorted(d.glob("*.fsdb"), key=lambda p: p.stat().st_mtime, reverse=True)
print("G:", [p.name for p in ordered])
# H
print("H:", sum(p.stat().st_size for p in d.iterdir() if p.is_file()))The verified output:
A: False
B: True
C: False
D: 7
E: 0
F: 50
G: ['b.fsdb', 'c.fsdb']
H: 30A — an exact-name keep-list misses a real wave filename. Section 2.
B — a pattern catches it.
C — and does not over-catch: cmd.sh is not a log.
D — the first matching pattern supplies the horizon.
E — an unmatched file gets 0, which means delete on the first sweep. That default is deliberate.
F — age is a difference in days, so the number is stable on any machine.
G — newest first, by st_mtime descending.
H — sizes are exact when the content is.
11. Exercises
Exercise 1 — list a real one. Take an actual run directory and list every entry. Count them.
Exercise 2 — test the keep-list. Run an exact-name keep-list over that listing. Report what it would delete that it should not.
Exercise 3 — the wave file. Say why a real dump's name cannot be a constant, citing Chapter 24.3.
Exercise 4 — patterns. Convert your keep-list to patterns and re-run it over the same listing.
Exercise 5 — report the reason. Make your matcher return which pattern matched, and say what that buys when a sweep surprises you.
Exercise 6 — too broad. Construct a filename your patterns keep but should not. Say how you would have found it.
Exercise 7 — three questions. For your flow, answer what, when and how long — separately, in writing.
Exercise 8 — the multiplier. Explain why answering only the first two is not a policy.
Exercise 9 — inverse relation. Show that your long horizons are on your small artefacts. If not, justify it.
Exercise 10 — construct an age. Use os.utime to build a directory with four known ages, and say why waiting is not a test strategy.
Exercise 11 — relative only. Explain why the example prints differences in days rather than timestamps.
Exercise 12 — dry run. Write plan_sweep and prove with a file count that it deleted nothing.
Exercise 13 — the persuasive number. Run your plan against a real volume and report the bytes it would free.
Exercise 14 — the zero horizon. Say what a file matching no pattern should do, and why seeing it in the plan is useful.
Exercise 15 — age and count. Give a scenario where an age horizon alone fails, and one where a count alone fails.
Exercise 16 — both. Implement keep-newest-N combined with a hard age limit.
Exercise 17 — regenerable. Classify your artefacts, and confirm every "how" depends on something in your never-delete set.
Exercise 18 — the dependency. Say what deleting cmd.sh does to the regenerability of everything else.
Exercise 19 — the assumption. State what re-run cmd.sh assumes, and which chapter addresses it.
Exercise 20 — possible is not free. Give an artefact that is regenerable and still worth keeping, and price the trade.
Exercise 21 — report the reason. Add the matched pattern to sweep's plan output. Re-run Section 9's scenario and say which line the extra column explains fastest.
12. Summary
- A keep-list of exact names is a claim about filenames, and it is wrong about real ones:
waves.fsdbdoes not matchwaves_axi_smoke_1839274.fsdb. - The artefact the policy deletes is the waveform — the one thing a failing run exists to produce — and
record.jsonwith it. - Test the keep-list against a real directory listing. Nothing in a code review surfaces this.
- Use
fnmatchpatterns, which read like a.gitignoreand which a team can edit. - Return which pattern matched, not a boolean, so a surprising sweep can be debugged.
- A pattern that is too broad fails the other way, and the remedy is the same: run it over real names.
- A policy answers three questions — what, when, and how long. Chapter 23.7 answered the first two.
- Without a horizon, a policy is a slower rate of filling, not a limit.
- Long horizons belong on small artefacts — a year of records is megabytes, a week of waveforms is most of the disk.
- Construct ages with
os.utimerather than waiting; a sweeper you can only observe over ninety days is one nobody tests. - Print differences, never timestamps, so the output is the same on any machine on any day.
- A sweeper must report before it removes, and prove it removed nothing by counting the files afterwards.
- The byte total is what persuades anyone to enable it.
- A file matching no pattern gets a horizon of zero and goes on the first sweep — a deliberate default, visible in the plan.
- Combine a count with an age: an age alone has no bound on volume, a count alone no bound on usefulness.
- Delete freely what you can regenerate, keep what you cannot, and keep the thing that regenerates the rest — which is Chapter 23.10's
cmd.sh. - "Regenerable" means the instructions exist, not that re-running is free, and the horizon is where that trade gets priced.
- The dry run and the live run must agree exactly — same files, same byte total — and a sweeper should print the pattern that decided each file, not just the verdict.
KEEP_NEWESTis applied over the whole listing before any deletion, and it legitimately rescues a file the age rule would have taken.
The next chapter stops the failures this module has been describing from reaching a run at all, by checking the environment before anything launches.
Where this fits
Part of the Python curriculum.