Source vs Derived: What You Are Allowed to Delete

Part II · Module 28 · Chapter 28.1 · Source vs Derived: What You Are Allowed to Delete

Module 27 left a run directory that is self-describing and replayable. It is also the largest thing in the project, and in a month there will be two thousand of them. Before any of that can be cleaned up, one question has to be answerable about every file: if I delete this, can it come back?

1. Measuring What a Flow Creates

The first thing to establish is what is actually appearing, because nobody's mental model of this is complete. Snapshot the tree, run the flow, snapshot again:

snap() { find . -mindepth 1 \( -type f -o -type d \) | sort; }
snap > /tmp/before.txt
# ... compile, run, dump, and a three-stage nvc run ...
snap > /tmp/after.txt
comm -13 /tmp/before.txt /tmp/after.txt

On the fixture — ten entries of source, one simulation each from two simulators:

  d ./runs
  d ./runs/smoke_s1
  f ./runs/smoke_s1/analyse.log               0 bytes
  f ./runs/smoke_s1/compile.log               0 bytes
  f ./runs/smoke_s1/dump.fst                416 bytes
  f ./runs/smoke_s1/dump.vcd               1470 bytes
  f ./runs/smoke_s1/elaborate.log             0 bytes
  f ./runs/smoke_s1/nvcrun.log              298 bytes
  f ./runs/smoke_s1/run.log                 222 bytes
  f ./runs/smoke_s1/sim.vvp                17015 bytes
  d ./runs/smoke_s1/work
  f ./runs/smoke_s1/work/WORK.ARB           450 bytes
  f ./runs/smoke_s1/work/WORK.ARB-RTL      1034 bytes
  f ./runs/smoke_s1/work/WORK.TB_ARB        504 bytes
  f ./runs/smoke_s1/work/WORK.TB_ARB-SIM   3787 bytes
  f ./runs/smoke_s1/work/WORK.TB_ARB.elab  1214 bytes
  f ./runs/smoke_s1/work/_NVC_LIB            11 bytes
  f ./runs/smoke_s1/work/_index              97 bytes

  created: 18 entries, 26518 bytes
  removed: 0 entries

Two observations that shape everything after.

Seven of the eighteen entries are a library directory whose filenames nobody would have guessed. WORK.TB_ARB-SIM, _NVC_LIB, _index. This is why Chapter 27.3 argued for a directory per run rather than a naming convention: you cannot write a pattern for filenames you have never seen, and every tool invents its own.

Three of the logs are zero bytes, and all three stages succeeded. Measured per stage:

  stage            rc  log bytes
  compile          0   0
  analyse          0   0
  elaborate        0   0
  run              0   134
  analyse (broken) 1   61

A clean stage says nothing, so on this flow the relationship is inverted: a non-empty compile or elaborate log means something went wrong. Two rules fall immediately. A cleanup of "delete empty files" would remove the evidence that those stages ran at all. And a validation of "an empty log means the stage failed" would be wrong three times out of three — while "a non-empty compile log deserves a look" is a usable heuristic that cost nothing to discover.

2. Three Discriminators, Measured on One Repository

The three things people actually use to classify a file are version control, a banner in the file, and the directory it sits in. The fixture contains four files chosen to make them disagree.

$ git check-ignore -q runs/smoke_s1/sim.vvp && echo ignored
ignored
$ git ls-files --error-unmatch rtl/generated/regs_pkg.sv >/dev/null && echo tracked
tracked

Read the two middle rows. rtl/generated/regs_pkg.sv is committed to version control and stamped as generated — so git says source and the banner says derived. rtl/generated/hand_patch.sv is hand-written in a directory called generated — so the directory convention says derived and the truth is source.

And common/pinmap.svh is the worst case: tracked, no banner, in an ordinary directory. Every discriminator says source. It happens to be generated, and nothing in the repository records that.

3. The Only Test That Settles It

Derived does not mean "a tool wrote it". It means this can be reconstructed, and that is a claim you can test:

rebuildable() {                 # $1 = file, $2 = the command that should produce it
  local f=$1 cmd=$2
  cp "$f" /tmp/keep.$$ 2>/dev/null || { echo "missing"; return; }
  rm -f "$f"
  eval "$cmd" >/dev/null 2>&1
  if [ -s "$f" ]; then echo "REBUILT"
  else cp /tmp/keep.$$ "$f"; echo "NOT rebuildable (restored)"; fi
  rm -f /tmp/keep.$$
}

The cp before and the restore after are not politeness. They are what makes this safe to run on a real tree: the test is destructive by construction, so it carries its own undo. Note also [ -s "$f" ] rather than [ -e "$f" ] — a tool that creates the file and then fails leaves a zero-byte artifact, and the empty logs in §1 are proof that zero-byte files are ordinary here.

Three files, with the command each one claims to come from:

$ command -v regtool || echo ABSENT
ABSENT
$ [ -e regs.yaml ] && echo present || echo ABSENT
ABSENT
$ head -2 rtl/generated/regs_pkg.sv
// AUTOMATICALLY GENERATED by regtool 2.4 -- DO NOT EDIT
// source: regs.yaml

The file stamped DO NOT EDIT is the one file in this repository that cannot be regenerated. The generator is not installed and its input is not in the tree. By the only test that matters it is source — more precisely, it is the sole surviving copy of something, which is the most valuable category of file there is and the one a naive cleanup is most likely to delete, because every label on it says it is disposable.

4. Two More Tests That Look Convincing and Are Not

"Derived files are newer than their sources." Appealing, and circular. Here is the fixture's own mtime ordering:

$ for f in rtl/arb.v rtl/generated/regs_pkg.sv runs/smoke_s1/sim.vvp; do
>   printf '  %-34s %s\n' "$f" "$(stat -f '%Sm' -t '%Y-%m-%d %H:%M:%S' "$f")"
> done
  rtl/arb.v                          2026-10-09 03:28:33
  rtl/generated/regs_pkg.sv          2026-10-09 03:29:24
  runs/smoke_s1/sim.vvp              2026-10-09 03:29:24

regs_pkg.sv is tied for newest. Is it newer than the sources, or is it one of the sources? The test cannot say, because answering it requires the classification the test was supposed to produce. And the practical failure is worse than the logical one: a git clone or an rsync -a without -t rewrites mtimes wholesale, after which every file is the same age and the test returns nothing at all.

"A generated file's banner tells you." Trustworthy about the past and not about the present. The banner records what wrote the file once. It does not record whether that tool still exists, whether its input still exists, or whether somebody has since edited the file by hand in spite of the instruction not to. A banner is evidence; the rebuild test is a measurement.

5. A Classification You Can Act On

Four categories, each with a different cleanup rule, and the boundary between the first two is the rebuild test rather than a label:

#!/bin/bash
# classify FILE... — four categories, from evidence rather than convention.
set -u
for f in "$@"; do
  [ -e "$f" ] || { printf '%-44s MISSING\n' "$f"; continue; }

  tracked=no;  git ls-files --error-unmatch "$f" >/dev/null 2>&1 && tracked=yes
  ignored=no;  git check-ignore -q "$f" 2>/dev/null && ignored=yes
  banner=no;   head -5 "$f" 2>/dev/null | grep -qiE 'automatically generated|do not edit|generated by' && banner=yes

  if   [ "$tracked" = yes ] && [ "$banner" = no ];  then cat=SOURCE
  elif [ "$tracked" = yes ] && [ "$banner" = yes ]; then cat=COMMITTED-GENERATED
  elif [ "$ignored" = yes ];                        then cat=DERIVED-DECLARED
  else                                                   cat=UNCLASSIFIED
  fi
  printf '%-44s %-20s tracked=%-3s ignored=%-3s banner=%s\n' "$f" "$cat" "$tracked" "$ignored" "$banner"
done

$ classify rtl/arb.v rtl/generated/regs_pkg.sv common/pinmap.svh \
>          rtl/generated/hand_patch.sv runs/smoke_s1/sim.vvp
rtl/arb.v                                    SOURCE               tracked=yes ignored=no  banner=no
rtl/generated/regs_pkg.sv                    COMMITTED-GENERATED  tracked=yes ignored=no  banner=yes
common/pinmap.svh                            SOURCE               tracked=yes ignored=no  banner=no
rtl/generated/hand_patch.sv                  SOURCE               tracked=yes ignored=no  banner=no
runs/smoke_s1/sim.vvp                        DERIVED-DECLARED     tracked=no  ignored=yes banner=no

Four of those five are right, and the fourth is right for a reason worth noticing: hand_patch.sv sits in a directory called generated and the classifier calls it SOURCE, because it never consults the directory name. Dropping the weakest of the three discriminators is what made that row correct.

Now commit a build artifact by accident, the way somebody does once per project:

$ git add -f runs/smoke_s1/sim.vvp && git commit -qm "oops, committed an artifact"
$ classify runs/smoke_s1/sim.vvp
runs/smoke_s1/sim.vvp                        SOURCE               tracked=yes ignored=no  banner=no
$ git check-ignore runs/smoke_s1/sim.vvp || echo "NOT ignored"
NOT ignored

A compiled simulation executable is now classified as SOURCE. Two things combined to do that: .gitignore lists *.vvp, but being tracked outranks being ignored — a fact Chapter 28.6 measures directly — so git check-ignore returns false, and the cascade reaches SOURCE on the first branch.

The fix is to untrack it, and then the classification corrects itself:

$ git rm --cached -q runs/smoke_s1/sim.vvp && git commit -qm untrack
$ classify runs/smoke_s1/sim.vvp
runs/smoke_s1/sim.vvp                        DERIVED-DECLARED     tracked=no  ignored=yes banner=no

The lesson is about the tool rather than the file: a classifier built from weak signals produces a confident answer, and its confidence is not evidence. Note which direction this one fails in — a stray commit promotes an artifact to SOURCE, so the mistake costs disk rather than data. That is the correct direction to fail in, and it is a design choice rather than luck: the cascade tests tracked first precisely so that anything a human once committed is protected.

The rules that follow have to be conservative in the same way:

• SOURCE — never deleted by any automatic rule.
• COMMITTED-GENERATED — never deleted either, until the rebuild test passes in your environment. Then, and only then, it may be treated as derived.
• DERIVED-DECLARED — deletable, subject to the age and archive rules in Chapter 28.4 and Chapter 28.5.
• UNCLASSIFIED — never deleted, and reported. An unclassified file is a gap in the project's own policy, and the correct response is to write a rule, not to guess.

That last category is the one that makes the scheme safe. A classifier with no "I do not know" state has to put every file into one of the others, and for cleanup the cost of the two mistakes is wildly asymmetric: a derived file misfiled as source wastes disk, and a source file misfiled as derived is gone.

6. Where This Leaves the Run Directory

Apply it to Module 27's output and the answer is clean, which is the payoff for having built it deliberately:

The third and fourth rows are the interesting ones. A log is technically derived and practically irreplaceable, because re-running does not reproduce the run that failed — it produces a new run, which may pass. And the record files from Chapter 27.9 are data about an event, so no tool can regenerate them at all.

This is why Chapter 28.5 archives the small files and discards the large one: the 17 KB executable and the multi-megabyte dump are both reproducible, and the 70-byte cmd.args is not.

Practice

1. Snapshot a tree, run your flow, snapshot again, and list exactly what appeared. Count the entries you would not have predicted.
2. Find the library or intermediate directory your tool creates and look at the filenames. Try to write a glob that would have matched them.
3. Count the zero-byte files your successful flow produced, and then reconsider any rule you have that treats an empty file as a failure.
4. Build the three-discriminator table for five files in your own repository, and find one where two discriminators disagree.
5. Write the rebuildable function with its save-and-restore, and run it on a file you are confident is derived.
6. Run it on a committed generated file. Check whether the generator is installed and whether its input is in the tree.
7. Explain why the mtime test is circular, and then verify the practical failure by copying a tree with rsync -a and re-running the test.
8. Write classify and run it over a directory. Find a file it gets wrong and work out which signal misled it.
9. Remove the UNCLASSIFIED category so every file is forced into one of the other three, and decide which of your files you have just put at risk.
10. Classify every artifact in one run directory and identify the ones that are technically derived and practically irreplaceable.
11. Find the single most valuable file in your project by asking which one has no other copy and no generator.

Check Your Understanding

1. Why can you not write a filename pattern for the files a tool will create?
2. Three logs were zero bytes. What had happened, and which two rules does that break?
3. What question does git tracking actually answer?
4. Why is a generated file sometimes committed on purpose?
5. What is the only test that establishes whether a file is derived?
6. Why does that function copy the file before deleting it?
7. Why [ -s "$f" ] rather than [ -e "$f" ]?
8. The file stamped DO NOT EDIT turned out to be source. What made it so?
9. In what sense is "derived" a property of the environment rather than the file?
10. Why is the mtime test circular, and how does rsync -a break it in practice?
11. Why must a classifier have an UNCLASSIFIED state?

7. Remember

• Measure what a flow creates by snapshotting before and after. On the fixture: 18 entries, 7 of them a library with unguessable filenames.
• Zero-byte logs come from successful stages. Do not delete empty files, and do not treat an empty log as a failure.
• Git tracking, a banner, and a directory name answer three different questions, and on this repository they disagreed about half the files.
• Derived means reconstructible, and the test is to delete it and rebuild it — with a save-and-restore, because the test is destructive.
• The file stamped DO NOT EDIT was the one that could not be rebuilt: no generator installed, no input in the tree.
• Derived is a property of your environment, not of the file. "We can always regenerate it" is a claim about an installed tool, a retained input and a recorded version.
• The mtime test is circular, and a fresh clone or rsync -a destroys it outright.
• Keep an UNCLASSIFIED category and report it. The two mistakes cost wildly different amounts.
• A log is technically derived and practically irreplaceable; a record file cannot be regenerated at all.

8. What Is Next

The categories exist. Chapter 28.2 applies them to a single run in detail — every file one simulation leaves, how large each gets on a real design, and which of them a debug three weeks later actually opens.