Keeping Generated Files Out of Version Control

Part II · Module 28 · Chapter 28.6 · Keeping Generated Files Out of Version Control

Chapter 28.1 classified files; this chapter is about telling git the answer. The stakes are smaller than Chapter 28.4's — nothing is destroyed — and the failure is more insidious, because a repository that has absorbed tool output never recovers on its own. Reviews fill with diffs nobody wrote, and after a while nobody reads them.

1. The Line That Changes Nothing

Here is what everybody does, and what it accomplishes:

$ printf 'x\n' >> common/pinmap.svh          # a generated file, already committed
$ echo 'common/pinmap.svh' >> .gitignore
$ git check-ignore -v common/pinmap.svh || echo "(not ignored)"
(not ignored)
$ git status --porcelain common/pinmap.svh
 M common/pinmap.svh

The pattern has no effect. .gitignore governs whether git will start tracking a path; it says nothing about a path git already tracks. The file still shows up modified, still appears in every diff, still conflicts on merge.

The fix is to untrack it, which is a separate operation:

$ git rm --cached -q common/pinmap.svh
$ git check-ignore common/pinmap.svh >/dev/null && echo ignored
ignored
$ git status --porcelain common/pinmap.svh
D  common/pinmap.svh

Now the pattern applies. Note what the status says: D — a staged deletion. git rm --cached removes the path from the index and leaves it on disk, so the next commit records that the repository no longer tracks it. That commit is a real change with real consequences: anybody who pulls it will find the file disappears from their working tree unless they also have it locally.

2. Patterns That Miss How the Flow Actually Writes

A plausible .gitignore for a simulation project:

runs/
*.vvp
*.vcd
*.fst

Measured against what tools actually create:

Two things in that table are worth separating.

The first three are caught by runs/, not by the extension patterns. dump.vcd.gz does not match *.vcd and dump.fsdb does not match anything in the list — they are ignored purely because they are under an ignored directory. Which means the extension lines are doing less work than they appear to, and the directory line is doing all of it.

The last four are the output of tools this .gitignore was not written for. csrc/ and INCA_libs/ belong to commercial simulators; work/ is nvc's library, which Chapter 28.2 inventoried at seven files under names nobody documents. Every new tool in the flow adds a directory, and the .gitignore is always one tool behind.

The structural answer is the same one Chapter 27.3 reached for run directories: ignore by location, not by name, because you can enumerate the places your flow writes and you cannot enumerate the filenames it will invent.

# everything the flow writes goes under one of these, by project policy
/runs/
/build/
/work/
/archive/

# plus the handful of tools that insist on writing beside the sources
/csrc/
/INCA_libs/
/simv.daidir/
ucli.key
vc_hdrs.h

The extensions can stay as a backstop for the inevitable tool that writes where it was told not to, but they are the second line of defence, not the first.

3. The Anchor Is Not What You Assume

Three patterns that look like variations on one idea:

$ for pat in 'build' '/build' 'build/'; do
>   printf '%s\n' "$pat" > .gitignore
>   printf '  pattern %-9s -> build/a.out %-8s sub/build/x.o %s\n' "'$pat'" \
>     "$(git check-ignore -q build/a.out && echo ignored || echo NO)" \
>     "$(git check-ignore -q sub/build/x.o && echo ignored || echo NO)"
> done
  pattern 'build'   -> build/a.out ignored  sub/build/x.o ignored
  pattern '/build'  -> build/a.out ignored  sub/build/x.o NO
  pattern 'build/'  -> build/a.out ignored  sub/build/x.o ignored

A pattern with no slash matches at any depth — build catches sub/build/x.o. A leading slash anchors it to the directory containing the .gitignore. A trailing slash restricts the match to directories but does not anchor it.

For tool output, recursive matching is usually what you want: a per-block build/ directory should be ignored wherever it appears. For a single top-level area, the leading slash is correct and says so. The failure to avoid is writing build meaning "the build directory at the top" and then being surprised when a source directory called build elsewhere in the tree vanishes from git's view — which produces the worst possible symptom, a source file that exists on your disk and in nobody else's clone.

4. A Negation Cannot Rescue a File Inside an Excluded Directory

The natural thing to try when one file inside an ignored area matters:

$ printf 'runs/\n!runs/keep/important.log\n' > .gitignore
$ git check-ignore -q runs/keep/important.log && echo "STILL ignored"
STILL ignored
$ git check-ignore -v runs/keep/important.log
.gitignore:1:runs/	runs/keep/important.log

The negation does nothing, and check-ignore -v names the rule that won: line 1, runs/. Once a directory is excluded, git does not descend into it, so no pattern about its contents is ever consulted.

The working form re-includes the directory on the way down:

$ printf 'runs/*\n!runs/keep/\n!runs/keep/**\n' > .gitignore
$ git check-ignore -q runs/keep/important.log || echo "not ignored -- the negation works"
not ignored -- the negation works

runs/* excludes the entries in runs rather than the directory itself, which leaves git free to descend; !runs/keep/ re-includes that one entry; !runs/keep/** re-includes what is under it.

5. Fixing a Repository That Already Has Output In It

The operation is mechanical and the ordering matters, because one of the steps is a commit everybody else will feel.

#!/bin/bash
# untrack-generated — list, or remove from the index, tracked files that the
# ignore rules say should never have been committed. DRY RUN by default.
set -u
apply=0
[ "${1:-}" = "--apply" ] && apply=1

git rev-parse --git-dir >/dev/null 2>&1 || { echo "untrack-generated: not a git repository" >&2; exit 2; }

# tracked files that WOULD be ignored if they were not tracked.
# --no-index is what makes check-ignore answer for tracked paths at all.
offenders=$(git ls-files | git check-ignore --no-index --stdin 2>/dev/null || true)

if [ -z "$offenders" ]; then
  echo "untrack-generated: nothing tracked matches an ignore rule"
  exit 0
fi

n=$(printf '%s\n' "$offenders" | grep -c .)
if [ "$apply" = 1 ]; then verb='untracking'; else verb='would untrack'; fi
printf '%s %d tracked file(s) that match an ignore rule:\n' "$verb" "$n"
printf '%s\n' "$offenders" | sed 's/^/  /'

if [ "$apply" = 1 ]; then
  printf '%s\n' "$offenders" | while IFS= read -r f; do git rm --cached -q -- "$f"; done
  printf 'done -- the files remain on disk; commit to record the untracking\n'
else
  printf 'dry run -- pass --apply to untrack them\n'
fi

$ untrack-generated
would untrack 1 tracked file(s) that match an ignore rule:
  runs/smoke_s1/sim.vvp
dry run -- pass --apply to untrack them

$ untrack-generated --apply
untracking 1 tracked file(s) that match an ignore rule:
  runs/smoke_s1/sim.vvp
done -- the files remain on disk; commit to record the untracking
$ git status --porcelain | head -2
D  runs/smoke_s1/sim.vvp

Three details that are easy to get wrong.

git check-ignore --no-index. Without it, check-ignore refuses to answer for a tracked path — which is §1's behaviour, and exactly what makes the offenders invisible to the obvious script. --no-index says "ignore the index and answer from the patterns alone", which is the question being asked here.

git rm --cached, never git rm. The first removes the path from the index and leaves the file; the second deletes it. On generated files the difference is recoverable; on anything misclassified it is not, which is why this script untracks rather than deletes and leaves the commit to a human.

It is a dry run by default. The same rule as Chapter 28.4: the destructive branch is asked for by name.

And one thing the script deliberately does not do: rewrite history. Removing a large file from past commits requires rewriting every commit after it, which changes every hash and forces everyone to re-clone. That is occasionally justified — a committed multi-gigabyte dump, or a leaked credential — and it is a decision for whoever owns the repository, not a cleanup script. Untracking from here forward costs nothing and solves the review-noise problem; shrinking the repository is a different project.

6. What Belongs in Which File

Three places to put ignore rules, with a rule for choosing:

.gitignore, committed. Everything that is true for everybody: the project's output directories, the known tools' artifacts. This is the one that belongs in review, because it is a statement of project policy.

.gitignore in a subdirectory, committed. Rules local to one block, so a per-block build/ can be handled where it is understood. Git reads these on the way down, and the nearest file wins for a given path — which is also how a subdirectory can re-include something its parent excluded, with the runs/* technique from §4.

.git/info/exclude, not committed. Your own scratch files — a personal notes.md, an editor directory, a one-off experiment. Putting these in the committed .gitignore is a small act of imposition, and it is the usual reason a project's ignore file grows a section nobody can explain.

A fourth, for completeness: core.excludesFile is a global setting for things true of your machine everywhere, like editor backup files. The ordering is nearest-first, and a later ! negation in a more specific file can re-include what a broader rule excluded — subject to §4's directory restriction, which no amount of file layering escapes.

Practice

1. Add a pattern for a file git already tracks and confirm with check-ignore and status that nothing changed.
2. Untrack it with git rm --cached and confirm the pattern now applies. Note the status letter.
3. Commit an artifact deliberately and then run Chapter 28.1's classifier on it. Explain the category it lands in.
4. Test a .gitignore with runs/ *.vcd against dump.vcd.gz and work out which line caught it.
5. Find three directories your flow writes that your current ignore rules miss.
6. Compare build, /build and build/ against a nested path, and record which match.
7. Create a source directory called build deeper in the tree and watch an unanchored pattern hide it.
8. Try to re-include one file inside an ignored directory with a ! negation, and read check-ignore -v.
9. Make it work with the runs/* plus !runs/keep/ form.
10. Write the gstate helper and use it on a tracked-and-matching file.
11. Write untrack-generated and confirm it finds nothing until --no-index is added.
12. Run it with --apply and check that the files are still on disk.
13. Put a personal scratch file in .git/info/exclude and confirm it is ignored without appearing in any committed file.

Check Your Understanding

1. What does adding a pattern do for a file git already tracks?
2. What does git rm --cached do, and what status letter results?
3. Why did that interaction misfile a build artifact as SOURCE in 28.1?
4. What question does git check-ignore actually answer?
5. In the four-line .gitignore, which rule caught dump.vcd.gz?
6. Why is the extension list doing less work than it appears to?
7. Why ignore by location rather than by filename?
8. What is the difference between build, /build and build/?
9. What is the worst symptom of an unanchored pattern?
10. Why does !runs/keep/important.log fail after runs/?
11. What form works instead, and why?
12. What does --no-index change about check-ignore?
13. Why does untrack-generated not rewrite history?
14. Which ignore rules belong in .git/info/exclude rather than .gitignore?

7. Remember

• Adding a pattern does nothing for a tracked file. git rm --cached is the separate operation that makes it apply, and it stages a D.
• git check-ignore answers "would this be ignored if new", not "is this ignored". Anything built on it inherits that, which is how a .vvp classified as SOURCE.
• --no-index is what makes check-ignore answer for tracked paths, and without it the offenders are invisible to the obvious script.
• Ignore by location, not by filename. Extension lists are a backstop; the directory rules do the work, and dump.vcd.gz proved it.
• Your ignore file is always one tool behind. csrc/, work/, INCA_libs/ each arrived with a tool.
• build, /build and build/ differ: no slash is recursive, a leading slash anchors, a trailing slash means directories only.
• An unanchored pattern can hide a source directory of the same name elsewhere in the tree — present on your disk, absent from every clone.
• A negation cannot reach inside an excluded directory. Use runs/* plus !runs/keep/ so git still descends.
• check-ignore -v names the file, line and pattern that decided. Use it for every question in this chapter.
• Untrack going forward; do not rewrite history. That is a repository-owner's decision.
• Personal rules go in .git/info/exclude, not in the committed file.

8. What Is Next

Module 28 leaves a workspace whose contents are classified, measured, cleaned, archived and no longer leaking into review. Everything retained is either irreplaceable or cheap, and the logs — kept deliberately, because they carry the verdict the exit status does not — are now the largest body of evidence in the project. Module 29 reads them: a tool log as a structured document, finding the first real error among its hundreds of consequences, and the runs that print errors and still exit zero.