Linux & Bash · Part II · Module 23 · Navigating RTL and Verification Repositories

Source, Derived and Imported Trees

Chapter 23.1 read a repository's classes off its directory names, which works until a project uses names you have not seen. This chapter proves the classification instead — four tests, run against a real repository, in order of how much they settle.

Intermediate21 min readLinuxPart IIGitRepositoryGenerated Files

Part II · Module 23 · Chapter 23.2 · Source, Derived and Imported Trees

The classification matters because three different rules follow from it:

Authored — you may edit it, and your edit is the point.
Imported — you may read it; editing it in place means your fix is destroyed by the next import.
Generated — you may delete it, and it will come back. Editing it is always a mistake, and searching it is the mistake that wastes afternoons.

Guessing from names gets this right most of the time, which is exactly why it is dangerous. Here are four tests that settle it.

1. Test One: Is It Tracked?

Almost decisive on its own, and it takes one command. Generated files are not committed; source files are.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ for d in rtl tb ip work csrc run_2026_10_01 third_party; do
>   printf '%-18s %s\n' "$d" "$(git ls-files "$d" | wc -l) tracked"
> done
rtl                3 tracked
tb                 1 tracked
ip                 1 tracked
csrc               0 tracked
work               0 tracked
run_2026_10_01     0 tracked
third_party        1 tracked

Four directories contain tracked files and three contain none. That single number separates source from generated with no knowledge of the project's conventions, no reading of documentation, and no risk of being misled by a directory called build that someone actually commits.

git ls-files <dir> lists the index entries under a path — so a zero means git has never been told about anything in here. On a repository that is properly maintained, that is the definition of generated.

The two cases where test one is wrong, and both are informative

Generated files that were committed. Some teams commit a generated file deliberately — a register map produced from a spreadsheet, a package built by a script — so that users do not need the generator. These show as tracked and are not editable by hand, because the next generator run overwrites them. The signal is a header: nearly all of them begin with some form of "DO NOT EDIT — generated by". Which gives a test:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

git grep -l -iE 'do not edit|automatically generated|generated by' -- '*.sv' '*.v' '*.vhd'

Source files that are not committed yet. Your own new file is untracked and is certainly source. This is why git status matters as much as git ls-files:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git status --porcelain --untracked-files=all
$ echo $?
0

Empty. In this repository every file is either tracked or deliberately ignored, which is the healthy state. An untracked, unignored file is the one real anomaly — it is either source somebody forgot to add, or generated output somebody forgot to ignore, and it is worth resolving either way.

2. Test Two: Is It Ignored?

Test one tells you git does not track it. Test two tells you whether that was intentional, and git answers directly:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ for d in rtl work csrc run_2026_10_01 cov; do
>   git check-ignore -q "$d" && echo "$d -> IGNORED" || echo "$d -> not ignored"
> done
rtl -> not ignored
work -> IGNORED
csrc -> IGNORED
run_2026_10_01 -> IGNORED
cov -> IGNORED

git check-ignore -q exits 0 if the path is ignored and 1 if it is not — a status-only query, which makes it exactly the right shape for a shell test. Chapter 16.7 is the pattern: a command used as a condition, with its output discarded.

Ignored plus untracked is a positive statement that a directory is generated, made by whoever wrote the .gitignore. That is stronger evidence than any heuristic, because it is a human assertion recorded in the repository:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat .gitignore
work/
simv.daidir/
csrc/
cov/
run_*/

run_*/ is the line worth noticing. The author knew run directories would be created with names they could not predict, and wrote a pattern rather than a list — which is Chapter 3.5's glob being used as a policy. Compare that with the deny-list in Chapter 23.5, which has to be maintained by hand and therefore goes stale.

check-ignore is also how you find out WHY something is ignored

Without -q it prints the path; with -v it prints the rule that matched and where that rule lives:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

git check-ignore -v run_2026_10_01
# .gitignore:5:run_*/   run_2026_10_01

File, line number, pattern, path. That is the answer to "why is git ignoring my new source file", which happens the first time someone adds rtl/work_in_progress/ to a tree that ignores work*. The question is unanswerable by reading .gitignore yourself when there are several of them at different depths, and -v names the one that won.

3. Test Three: Who Has Touched It?

Tests one and two separate source from generated. They do not separate authored from imported, because imported IP is committed and not ignored — it looked like source in §1, and it is source. It is just not yours.

The history separates them:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ for d in rtl tb ip third_party constraints; do
>   printf '%-14s commits=%-3s authors: %s\n' "$d" \
>     "$(git log --oneline -- "$d" | wc -l)" \
>     "$(git log --format='%ae' -- "$d" | sort -u | tr '\n' ' ')"
> done
rtl            commits=3   authors: asha@chip ben@chip e@x
tb             commits=2   authors: asha@chip e@x
ip             commits=1   authors: e@x
third_party    commits=1   authors: e@x
constraints    commits=1   authors: e@x

ip/ and third_party/ have exactly one commit each, by one person. That is the shape of an import: a tree arrives whole, in a single commit, usually with a message like "import vendor_axi 2.4.1", and then nobody touches it until the next version arrives as another single commit.

rtl/ and tb/ have several commits by several people, which is the shape of authored code.

Two refinements that make this a real test rather than a suggestion:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# the commit that introduced a tree, with its message — usually says "import"
git log --oneline --diff-filter=A -- ip | tail -1

# how long since anyone touched it
git log -1 --format='%ar  %an  %s' -- ip

--diff-filter=A restricts to commits that added files, and tail -1 takes the oldest — so this is "how did this directory come to exist". An import's message nearly always says so, and when it does, the classification is settled.

Three more import signals, in order of reliability

A submodule or subtree. The strongest signal, because it is structural rather than inferred:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

cat .gitmodules 2>/dev/null          # submodules, with their upstream URLs
git log --grep='git-subtree-dir' --oneline | head

A path listed in .gitmodules is imported by definition, and the file names its upstream.

A vendor's own metadata. LICENSE, CHANGELOG, version.txt, a doc/ directory with a datasheet, a release note naming a version number. Authored trees inside a company rarely carry these; shipped IP almost always does.

And a different style. Tab-indented where the rest of the tree uses spaces, a different header comment, a different naming convention for signals. This is the weakest signal and the one you notice first, which is worth being aware of — it suggests imported code and does not establish it, because a company with several teams has several styles.

4. Test Four: Is It Newer Than What It Built?

The fourth test asks a different question — not "what kind of directory is this" but "is this generated directory still valid" — and it is the one that catches the failure Chapter 22.6 §6 measured.

A generated artifact is only meaningful if it is newer than every source that went into it. Edit one source after a build and that stops being true:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ls -l work/sim.vvp
-rw-r--r--  1 engineer  staff  1832 Oct  8 23:02 work/sim.vvp
$ printf '// edited after the build\n' >> rtl/axi/axi_master.sv
$ while read -r f; do
>   [ "$f" -nt work/sim.vvp ] && printf 'STALE: %s is newer than the artifact\n' "$f"
> done < filelists/chip.f
STALE: rtl/axi/axi_master.sv is newer than the artifact
STALE: rtl/uart/uart_tx.sv is newer than the artifact
STALE: tb/tests/tb_chip.sv is newer than the artifact

Three of the four, because the other two edits happened earlier in the session. The point is that the question is answerable in four lines, and that the answer is a fact rather than a feeling about whether you remembered to rebuild.

As a reusable check:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/usr/bin/env bash
# build-is-stale — exit 0 if the artifact is up to date, 1 if not, 2 if unusable
# usage: build-is-stale <artifact> <filelist>
set -o nounset

art=${1:?usage: build-is-stale <artifact> <filelist>}
fl=${2:?usage: build-is-stale <artifact> <filelist>}

[ -f "$art" ] || { printf 'build-is-stale: no artifact: %s\n' "$art" >&2; exit 2; }
[ -r "$fl" ]  || { printf 'build-is-stale: no filelist: %s\n' "$fl" >&2; exit 2; }

stale=0
while IFS= read -r f; do
  case $f in ''|'#'*|'+'*|'-'*) continue ;; esac
  [ -e "$f" ] || { printf 'build-is-stale: missing source: %s\n' "$f" >&2; exit 2; }
  if [ "$f" -nt "$art" ]; then printf 'stale: %s\n' "$f"; stale=$(( stale + 1 )); fi
done < "$fl"

[ "$stale" -eq 0 ]

Five decisions, all Part I:

${1:?usage: …} so a missing argument stops the script with a message naming the script — Chapter 20.3 and Chapter 21.1.

The case skips blanks, comments and option lines. A real filelist contains +incdir+..., -y ... and # comments alongside file paths, and treating an option as a filename produces a confident, wrong answer. Chapter 16.6's pattern list, and Module 26 is where filelist syntax is taken seriously.

A missing source is exit 2, not "stale". It is a different problem — a broken filelist — and conflating the two sends someone to rebuild when they should be fixing a path. Chapter 21.4's separation of "it failed" from "you called it wrong".

while IFS= read -r f, so a path containing a space or a backslash survives — Chapter 17.5.

And the bare [ "$stale" -eq 0 ] as the last line, so the script's status is the answer — Chapter 16.7.

5. What Each Verdict Licenses

The reason to do any of this:

Verdict	Edit	Search	Delete	Commit
Authored	Yes	Always	No	Yes
Imported	No — patch upstream, or wrap	Yes, read-only	No	Only a version bump
Generated	Never	Never	Freely	Never

Two rows need the explanation, because both are advice people ignore once and remember forever.

Why not edit imported IP in place. The fix survives until the next import, which overwrites the tree wholesale and takes your change with it — silently, because the import is one commit touching a thousand files and nobody reads that diff. The alternatives, in order of preference: get the fix upstream; keep a patch file applied by the flow; or wrap the block in a module of your own under rtl/. All three are visible; an in-place edit is not.

Why not search generated directories. Because they contain copies of source, often several generations old. Chapter 23.3 measures a repository where grep -r 'module axi_master' returns three files, one of which is a stale copy inside a run directory — and if you open that one and start reading, every conclusion you draw is about last week's design.

6. The Whole Classification, in One Loop

The four tests, applied to every top-level directory:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/usr/bin/env bash
# classify-trees — authored / imported / generated, for each top-level directory
set -o nounset

git rev-parse --is-inside-work-tree > /dev/null 2>&1 || {
  printf 'classify-trees: not a git repository\n' >&2; exit 2; }

printf '%-18s %8s %8s %8s  %s\n' DIRECTORY TRACKED IGNORED SIZE VERDICT
for d in $(git rev-parse --show-toplevel > /dev/null && \
           find . -maxdepth 1 -type d ! -name . ! -name .git | sed 's|^\./||' | sort); do
  tracked=$(git ls-files "$d" | wc -l | tr -d ' ')
  ignored=$(git check-ignore -q "$d" && echo yes || echo no)
  size=$(du -sk "$d" 2>/dev/null | awk '{print $1}')
  authors=$(git log --format='%ae' -- "$d" | sort -u | wc -l | tr -d ' ')
  commits=$(git log --oneline -- "$d" | wc -l | tr -d ' ')

  if   [ "$tracked" -eq 0 ] && [ "$ignored" = yes ]; then verdict=GENERATED
  elif [ "$tracked" -eq 0 ];                         then verdict="UNTRACKED — resolve"
  elif [ "$commits" -le 1 ];                         then verdict="IMPORTED (one commit)"
  elif [ "$authors" -le 1 ];                         then verdict="IMPORTED? (one author)"
  else                                                    verdict=AUTHORED
  fi
  printf '%-18s %8s %8s %7sK  %s\n' "$d" "$tracked" "$ignored" "$size" "$verdict"
done

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ./classify-trees
DIRECTORY           TRACKED  IGNORED     SIZE  VERDICT
constraints               1       no       4K  IMPORTED (one commit)
cov                       0      yes       0K  GENERATED
csrc                      0      yes       4K  GENERATED
docs                      1       no       4K  IMPORTED (one commit)
filelists                 1       no       4K  IMPORTED (one commit)
ip                        1       no       4K  IMPORTED (one commit)
rtl                       3       no      12K  AUTHORED
run_2026_10_01            0      yes       4K  GENERATED
run_2026_10_07            0      yes       0K  GENERATED
scripts                   1       no       4K  IMPORTED (one commit)
simv.daidir               0      yes       4K  GENERATED
tb                        1       no       4K  AUTHORED
third_party               1       no       4K  IMPORTED (one commit)
work                      0      yes       8K  GENERATED

And now read the output critically, because four rows are wrong. constraints, docs, filelists and scripts are authored, not imported — they are simply small and young, with one commit each in a toy repository. ip and third_party get the right verdict for the right reason, and tb gets the right verdict despite also being small.

That is not a bug to hide; it is the honest limit of the test. The commit-count heuristic needs history to work, and a repository five commits old does not have any. On a real tree with two years of history the separation is sharp; on a new one, test three is silent and you fall back on §3's callout — .gitmodules, a vendor LICENSE, an import commit message.

A classifier that reports a confident wrong answer is worse than one that abstains

The script above states IMPORTED (one commit) where the truth is "not enough evidence". Chapter 21.8 argued that a check you have never watched fail is not evidence; this is the companion point — a check that cannot say "I do not know" will say something false instead.

The fix is one more branch and a threshold:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  if   [ "$tracked" -eq 0 ] && [ "$ignored" = yes ]; then verdict=GENERATED
  elif [ "$tracked" -eq 0 ];                         then verdict="UNTRACKED — resolve"
  elif [ "$total_commits" -lt 20 ];                  then verdict="UNKNOWN — too little history"
  elif [ "$commits" -le 1 ] && [ "$tracked" -gt 5 ]; then verdict=IMPORTED
  elif [ "$authors" -ge 2 ];                         then verdict=AUTHORED
  else                                                    verdict="UNKNOWN — check .gitmodules"
  fi

with total_commits=$(git log --oneline | wc -l) computed once outside the loop. Two changes carry the weight: an abstention when the repository as a whole is too young to judge, and [ "$tracked" -gt 5 ] so a one-commit directory holding one file is not called an import. Run it on this repository and every row becomes UNKNOWN — too little history, which is the correct answer.

7. The Habit

Three questions, asked before editing anything in a tree you do not know:

"Is this tracked?" — git ls-files <path>. If not, do not spend an hour reading it.

"Who wrote this?" — git log -1 --format='%an %ar %s' -- <path>. One commit from two years ago by someone who left means imported; three commits last week means you are in live code and should talk to someone before changing it.

"Is this file the one being compiled?" — the filelist, which is Chapter 23.3.

None of those takes ten seconds, and each of them has saved somebody a day.

Practice

Run test one on every top-level directory of a repository you use. Does anything surprise you?
Find a tracked file that says "DO NOT EDIT" in its header. If there is none, search for "generated".
Run git status --porcelain --untracked-files=all. Resolve one entry: is it source to add, or output to ignore?
Run git check-ignore -v on a path you expected to be tracked. Which rule, in which file, matched?
Run test three on your repository. Which directories have one commit and one author?
For each of those, run git log --oneline --diff-filter=A -- <dir> | tail -1 and read the message.
Check whether your repository has a .gitmodules. If it does, compare its paths to your test-three results.
Write build-is-stale and run it against a real artifact and filelist. Then touch one source and run it again.
Give build-is-stale a filelist containing a +incdir+ line and confirm the case skips it rather than reporting a missing source.
Then delete the case line and see what it reports. Which failure would have cost you more time?
Write classify-trees and run it. For every row you believe is wrong, say which test was silent.
Add the abstention branch from §6's callout and confirm a young repository reports UNKNOWN throughout.

Check Your Understanding

What are the three classes, and what does each license you to do?
What single command separates source from generated most reliably?
What two cases make test one wrong, and how do you detect each?
What does git check-ignore -q return, and why is that shape convenient?
Why do tests one and two fail to separate authored from imported?
What is the commit-shape signature of an imported tree?
What is the strongest, non-inferred import signal?
Why must you never edit imported IP in place?
In build-is-stale, why is a missing source exit 2 rather than "stale"?
Why did classify-trees label scripts/ as imported, and what is the right fix?

Answers

1. Authored — edit it, search it, commit it. Imported — read it, do not edit in place, patch upstream or wrap it. Generated — delete it freely, never edit it, and never search it. 2. git ls-files <dir> | wc -l. Zero tracked files means git was never told about anything there, which on a maintained repository is the definition of generated. 3. Generated files that were deliberately committed (detect them by a "DO NOT EDIT / generated by" header), and new source not yet added (detect it with git status --porcelain --untracked-files=all). 4. A status only — 0 if the path is ignored, 1 if not. That makes it directly usable as a shell condition with its output discarded. 5. Because imported IP is source: it is committed and not ignored, so it looks exactly like authored code to both tests. 6. One commit, by one person, adding the whole tree at once, with a message naming a version — and then nothing until the next version arrives the same way. 7. A .gitmodules entry (or a subtree commit), because it is structural rather than inferred, and it names the upstream. 8. Because the next import overwrites the tree wholesale and silently takes your change with it — the import is one commit touching a thousand files and nobody reads that diff. 9. Because it is a different problem: a broken filelist, not an out-of-date build. Conflating them sends someone to rebuild when they should be fixing a path. 10. Because it has one commit and one author in a five-commit repository — the commit-count heuristic needs history. The fix is to abstain: report UNKNOWN — too little history when the repository as a whole has too few commits to judge.

8. Remember

Three classes, three licences: authored (edit), imported (read, patch upstream), generated (delete, never search).
Test one — git ls-files <dir> | wc -l — is nearly decisive. Zero tracked files means generated.
Two exceptions: committed generated files (look for a "DO NOT EDIT" header) and your own new source (look at git status).
An untracked, unignored file is the real anomaly — either source to add or output to ignore.
Test two — git check-ignore -q — turns "git does not track it" into "someone declared it generated". -v names the rule and its line.
run_*/ in a .gitignore is a policy, not a list, and it does not go stale the way a --exclude-dir deny-list does.
Tests one and two cannot separate authored from imported, because imported IP is source.
Test three — one commit, one author, whole tree at once — is the import signature. --diff-filter=A finds the commit that created it.
.gitmodules is the strongest signal because it is structural and names the upstream.
Test four — [ "$src" -nt "$artifact" ] — answers whether a build is still valid in four lines.
A missing source is a different exit code from a stale build.
Never edit imported IP in place. The next import takes your fix silently.
A classifier that cannot abstain will say something false. Add the "too little history" branch.

9. What Is Next

Chapter 23.3 uses all of this on the question that actually comes up: there are three files defining the module named in your log, they differ, and you need the one the compiler read.

Where this fits

Part of the Linux & Bash curriculum.