Linux & Bash · Part II · Module 23 · Navigating RTL and Verification Repositories

Reading a Repository You Did Not Write

The first hour on a new project is usually spent wandering. This chapter makes it a routine instead — eight commands in an order chosen so each answers the most, assembled from everything Module 23 established, and ending with where the documentation actually is.

Intermediate20 min readLinuxPart IIGitRepositoryOnboarding

Part II · Module 23 · Chapter 23.6 · Reading a Repository You Did Not Write

You have been given a repository and a task. The repository has four hundred directories, nine filelists, two build systems, and a README last edited two years ago.

Reading it front to back is not an option. What follows is a routine, in a deliberate order: each step is chosen because it answers more than the step after it.

1. The Order, and Why

#	Question	Command	Why here
1	What shape is it?	`find . -maxdepth 2 -type d`	cheapest, and frames everything else
2	What is source?	`git ls-files`, `git check-ignore`	decides what the next six steps look at
3	How big is the job?	line and module counts	sets your expectations honestly
4	How is it built?	tracked executables, filelists	the single most load-bearing answer
5	What is alive?	`git log` recency and churn	points you at the code that matters
6	Who do I ask?	`git log --format='%an'`	the fastest route past any blocker
7	Where are the conventions?	the newest files, not the oldest	tells you how to write code that fits
8	Where is the real documentation?	commit messages and tests	see §6

Steps 1 and 2 are Chapter 23.1 and Chapter 23.2. Steps 3 to 8 are this chapter.

2. How Big Is the Job

Ask before you start reading, because the answer changes your approach entirely — and ask it over source only, which Chapter 23.2 made possible:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git ls-files '*.sv' '*.v' | xargs wc -l | tail -1
      38 total
$ git grep -h -E '^[[:space:]]*module[[:space:]]' -- '*.sv' '*.v' | wc -l
5
$ grep -cvE '^\s*(#|$|-|\+)' filelists/chip.f
4

Thirty-eight lines, five modules, four compiled files — a toy, as intended. The numbers you actually meet are more like two hundred thousand lines and eight hundred modules, and the ratio between them is the useful part:

Lines per module tells you the style. Two hundred lines per module is a well-decomposed design; three thousand is a few enormous blocks and a different reading strategy.

Modules defined versus files compiled tells you how much is dead or alternative. If a tree defines eight hundred modules and the filelist compiles three hundred files, there is a great deal you can ignore — and Chapter 23.3's intersection tells you which.

xargs wc -l and the two traps in it

git ls-files '*.sv' | xargs wc -l | tail -1 is the idiom, and it has two failure modes worth knowing before you quote the number.

With many files, xargs runs wc more than once, and you get several total lines — so tail -1 reports the last batch, not the sum. Chapter 22.1 §9's ARG_MAX is the cause. The fix:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

git ls-files '*.sv' '*.v' | xargs wc -l | awk '$2=="total" {n+=$1} END {print n}'

And a path with a space breaks it, because xargs splits on whitespace by default. Chapter 14.7 is the NUL-safe form, and git has it built in:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

git ls-files -z '*.sv' '*.v' | xargs -0 wc -l | awk '$2=="total" {n+=$1} END {print n}'

-z from git, -0 from xargs. It is two characters and it is the difference between a number you can quote and a number that is quietly wrong.

3. How Is It Built

The most valuable answer in the routine, because it makes everything else checkable. Two independent searches, and agreement between them is the point:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git ls-files -s | awk '$1 ~ /755$/ { print $4 }'
scripts/run_sim.sh
$ git grep -ln -E 'iverilog|vcs|xrun|vsim|verilator|nvc' -- 'scripts/*' '*.sh' 'Makefile*'
scripts/run_sim.sh

The first asks "what did someone commit as executable"; the second asks "what mentions a simulator". Both name the same file, which is Chapter 23.3 §3's rule applied to a different question: two independent methods agreeing is evidence.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat scripts/run_sim.sh
#!/bin/bash
# scripts/run_sim.sh
iverilog -g2012 -o work/sim.vvp -f filelists/chip.f

Three lines, and they tell you the tool, the filelist, the output location and the language version. In a real repository the equivalent file is four hundred lines, and the four things to extract from it are the same four:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# which tool, and which version flag
git grep -n -E 'iverilog|vcs|xrun|vsim|verilator|nvc' -- scripts | head
# which filelist is the real one
git grep -n -E '\-f |\-F |\.f\b' -- scripts | head
# where output goes
git grep -n -E '\-o |work|results|OUTDIR|RUNDIR' -- scripts | head
# and what the entry point expects from the environment
git grep -n -E '\$\{?[A-Z_]{3,}' -- scripts | sed 's/.*\(\$[{]\?[A-Z_]*\).*/\1/' | sort -u | head

That last one is the question "what must be set up before this works", and it is the bridge to Module 25.

4. What Is Alive

A repository's git history is a map of where the work is, and three queries extract most of it.

Recency — what is being changed right now:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git log --format='%ar  %an  %s' -5
29 minutes ago  Asha  tb: add check
29 minutes ago  Ben  uart: fix stat
29 minutes ago  Asha  axi: widen status
29 minutes ago  Asha  rtl: fix axi reset
30 minutes ago  n  init

Churn — which files change most, which is a good proxy for where the difficulty is:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git log --name-only --format= | grep -v '^$' | sort | uniq -c | sort -rn | head
   2 tb/tests/tb_chip.sv
   2 rtl/uart/uart_tx.sv
   2 rtl/axi/axi_master.sv
   1 third_party/cde/notes.txt
   1 scripts/run_sim.sh
   1 rtl/core/chip_top.sv

And ownership — who to ask:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git log --format='%an <%ae>' | sort | uniq -c | sort -rn
   3 Asha <asha@chip>
   1 Ben <ben@chip>
   1 n <e@x>
$ git log --format='%an' -- rtl | sort | uniq -c | sort -rn
   1 Asha
   1 Ben
   1 n

Per-directory ownership is the one worth running, because "who owns the AXI block" is answerable and "who owns the repository" usually is not.

A measured git trap: a --format with no percent sign is a format NAME

A natural way to group each commit's files is to print a separator between commits. It does not work:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git log --name-only --format='---'
fatal: invalid --pretty format: ---
$ git log --name-only --format=':::'
fatal: invalid --pretty format: :::
$ git log --name-only --format='COMMIT'
fatal: invalid --pretty format: COMMIT

Three different separators, the same refusal. The cause is not the dashes:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git log --format='oneline' -1
8bca565d0166b560ed7a72a9024a80dbca85a2a4 tb: add check

A --format argument containing no % is interpreted as the name of a built-in format — oneline, short, medium, full — rather than as a literal string. oneline is a name it knows; --- and ::: are not, so it reports an invalid format.

The fix is to make the separator contain a %, using a hex escape for the literal characters:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git log --format='%x2D%x2D%x2D' -1
---
$ git log --format='C%x3A' -1
C:

%x2D is a hyphen and %x3A a colon. So the co-change query — which pairs of files are changed together, a strong signal for coupling in an unfamiliar design — becomes:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

git log --format='%H' | while read -r h; do
  files=$(git show --name-only --format= "$h" | grep -v '^$' | sort)
  [ "$(printf '%s\n' "$files" | grep -c .)" -ge 2 ] || continue
  printf '%s\n' "$files" |
    awk '{a[NR]=$0} END {for(i=1;i<=NR;i++) for(j=i+1;j<=NR;j++) print a[i]" + "a[j]}'
done | sort | uniq -c | sort -rn | head

That form sidesteps the separator problem entirely by taking one commit at a time, and it was checked against a %x00-separated single-pass version: both produced identical output. Two implementations agreeing is the same discipline Chapter 23.3 §3 argued for, applied to a script rather than a search.

The error message deserves one more note, because it is a good one: it says invalid --pretty format when you passed --format. --format is a synonym for --pretty=format: and the diagnostic uses the older name — which is the kind of mismatch that makes an error hard to search for. Chapter 21.1 §6's point about quoting back what the user actually typed.

5. Where the Conventions Are

Not in a style guide. Read the newest files, not the oldest — the newest code shows what the team does now, and the oldest shows what they did before they knew better.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# the most recently added source files
git log --diff-filter=A --name-only --format='%x2D%x2D%x2D %ar' -- '*.sv' | head -30

# or by modification time, for a tree without useful history
git ls-files '*.sv' | while read -r f; do
  printf '%s %s\n' "$(git log -1 --format='%at' -- "$f")" "$f"
done | sort -rn | head -5 | cut -d' ' -f2

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git ls-files '*.sv' | while read -r f; do
>   printf '%s %s\n' "$(git log -1 --format='%at' -- "$f")" "$f"
> done | sort -rn | head -3 | cut -d' ' -f2
tb/tests/tb_chip.sv
rtl/uart/uart_tx.sv
rtl/axi/axi_master.sv

%at is the author timestamp as a Unix integer, which sorts numerically — whereas %ar ("29 minutes ago") is for humans and sorts meaninglessly. Picking the right format specifier for the consumer is the whole of that line.

Four conventions to extract once you have the newest few files open, each answering a question you will otherwise get wrong on your first commit:

Naming. Is it clk/rst_n or i_clk/i_rst_n? Are instances u_foo, foo_inst or foo_i? Chapter 23.4's walk depends on the instance convention, so this is worth writing down.

File layout. One module per file, or packages grouped? Where do interfaces live?

Indentation and width. Measurable rather than guessable:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

git ls-files '*.sv' | head -20 | xargs grep -hn '^	' | wc -l     # tab-indented lines
git ls-files -z '*.sv' | xargs -0 awk '{ print length }' | sort -rn | head -1   # longest line

And the header block. Copy it. Every project has one, nobody documents it, and omitting it is the first thing a reviewer notices.

6. Where the Real Documentation Is

The honest answer, in order of how often it is right:

The commit messages. Usually the largest body of accurate prose about the design, because they were written while the change was fresh and by the person making it. In chip_a the ratio is already visible:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ wc -c docs/spec.md README.md
      17 docs/spec.md
      63 README.md
      80 total
$ git log --format='%s%n%b' | wc -c
      76

Comparable in a five-commit toy; in a two-year-old repository the commit messages outweigh docs/ by an order of magnitude, and unlike docs/ they cannot be stale, because each one describes a change that definitely happened.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

git log --grep='axi' --format='%h %ar %an%n   %s%n%b' -- rtl/axi | head -40

Searching history for a block's name is frequently faster than reading the block.

The tests. A testbench is an executable statement of what the design is supposed to do, and it is maintained because it fails when it is wrong. A specification is prose that nothing checks. When the two disagree, the test is what the design actually does — and Chapter 23.3 §6 is how to find the one that covers your block.

The filelist. Underrated: it is the authoritative list of what is in the design, in compile order, maintained because the build breaks otherwise. Reading one top to bottom is a good twenty minutes.

The build script. §3's file, which encodes every assumption about tools, environment and output layout — all of it true, because the flow runs.

And docs/ last. Not because it is worthless, but because it is the only item on this list with no mechanism forcing it to stay true. Read it for intent and architecture; verify every specific against the source.

The two questions to ask a person, after the routine and not before

The routine answers "what is here" and "how is it built". It cannot answer two things, and those are the ones worth a colleague's time:

"What is the part that surprises people?" Every design has one — a clock that is not what its name says, a block that is deliberately not reset, an interface whose handshake is inverted. It is never written down, because whoever knows it stopped finding it surprising.

"What should I not touch?" Code that is fragile, scheduled for replacement, or owned by another team. Discovering this by touching it is expensive for everyone.

Asking these after the routine is a much better use of the conversation than asking "where do I start", which the routine answers for free — and it demonstrates that you did the reading, which is how you get a useful answer to the first question.

7. The Routine as a Script

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/usr/bin/env bash
# onboard — the first hour in an unfamiliar design repository, in one pass
# usage: onboard [dir]
# exit:  0 ok · 2 not a git repository
set -o nounset

cd -- "${1:-.}" || exit 2
git rev-parse --is-inside-work-tree > /dev/null 2>&1 ||
  { printf 'onboard: not a git repository: %s\n' "$(pwd -P)" >&2; exit 2; }

printf '================ %s\n\n' "$(pwd -P)"

printf -- '-- shape\n'
find . -maxdepth 1 -type d ! -name . ! -name .git | sed 's|^\./|   |' | sort

printf -- '\n-- source size\n'
git ls-files -z '*.sv' '*.svh' '*.v' '*.vhd' 2>/dev/null |
  xargs -0 wc -l 2>/dev/null | awk '$2=="total" {n+=$1} END {printf "   %d tracked HDL lines\n", n+0}'
printf '   %s modules defined\n' \
  "$(git grep -h -E '^[[:space:]]*module[[:space:]]' -- '*.sv' '*.v' 2>/dev/null | wc -l | tr -d ' ')"

printf -- '\n-- entry points\n'
git ls-files -s | awk '$1 ~ /755$/ { print "   " $4 }'
git grep -ln -E 'iverilog|vcs|xrun|vsim|verilator|nvc|dc_shell' -- \
  'scripts/*' '*.sh' 'Makefile*' 2>/dev/null | sed 's/^/   mentions a tool: /'

printf -- '\n-- filelists\n'
git ls-files | grep -E '\.f$|filelist' | sed 's/^/   /'

printf -- '\n-- recent activity\n'
git log --format='   %ar  %<(11)%an %s' -8

printf -- '\n-- who to ask, overall\n'
git log --format='%an <%ae>' | sort | uniq -c | sort -rn | head -6 | sed 's/^/   /'

printf -- '\n-- most-changed files\n'
git log --name-only --format= | grep -v '^$' | sort | uniq -c | sort -rn | head -8 | sed 's/^/   /'

printf -- '\n-- newest source (read these for the conventions)\n'
git ls-files '*.sv' '*.v' | while IFS= read -r f; do
  printf '%s %s\n' "$(git log -1 --format='%at' -- "$f")" "$f"
done | sort -rn | head -5 | cut -d' ' -f2- | sed 's/^/   /'

printf -- '\n-- generated directories git does not know about\n'
for d in */; do
  d=${d%/}
  [ "$(git ls-files "$d" | wc -l)" -eq 0 ] && ! git check-ignore -q "$d" &&
    printf '   %s (untracked and not ignored — resolve this)\n' "$d"
done
printf '   (nothing above means .gitignore is complete)\n'

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ./onboard
================ /home/engineer/chip_a

-- shape
   constraints
   cov
   csrc
   ...

-- source size
   38 tracked HDL lines
   5 modules defined

-- entry points
   scripts/run_sim.sh
   mentions a tool: scripts/run_sim.sh

-- filelists
   filelists/chip.f

-- recent activity
   29 minutes ago  Asha       tb: add check
   29 minutes ago  Ben        uart: fix stat
   ...

Five decisions:

git ls-files -z … | xargs -0 for the line count, so a path with a space does not break it and awk sums every total rather than taking the last — §2's two traps, fixed in the one place it matters.

n+0 in the awk END block, so a repository with no HDL prints 0 rather than an empty field. An onboarding script that prints a blank where a number should be reads as a bug in the script.

Two independent entry-point searches, printed one after the other, so agreement or disagreement is visible rather than hidden behind a merge.

%<(11)%an in the log format, which pads the author name to eleven columns so the subject lines align.

This one was wrong in the first draft, and the way it was wrong is worth the space. I wrote %-10an, reasoning by analogy with printf's %-10s. Git's format language does not work that way:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git log --format='[%-10an]' -2
[10an]
[10an]

%- is a real git placeholder meaning suppress the preceding empty line, so %- was consumed and 10an printed as literal text. The author name vanished entirely and nothing errored. Git's padding is a parenthesised prefix:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ git log --format='[%<(10)%an] %s' -3
[Asha      ] tb: add check
[Ben       ] uart: fix stat
[Asha      ] axi: widen status

%<(N) pads right, %>(N) pads left, and %<(N,trunc) truncates instead of overflowing. The general lesson is the one Chapter 21.6 keeps making: a format language that looks like printf is not printf, and the failure mode here was silent output rather than an error — which is why the script's output has to be read, not assumed.

And the last section is Chapter 23.2's combined test, printed even when it finds nothing — because "nothing above means .gitignore is complete" is itself a useful result, and a silent section reads as a broken script.

Practice

Run the eight-step routine by hand on a repository you already know. Which step told you something you did not know?
Count tracked HDL lines with xargs wc -l | tail -1 and with the awk-summing form. Do they differ? Make them differ.
Add a file with a space in its name and confirm the -z/-0 form still works and the plain one does not.
Compute lines per module for your repository. What does the ratio say about the style?
Compare modules defined against files compiled. How much of the tree can you ignore?
Find your repository's entry point two ways. Do the two agree?
Extract from the entry point: tool, filelist, output location, and required environment variables.
Run git log --name-only --format='---' and read the error. Then explain why --format='oneline' works.
Make a separator work with %x2D. Then run the co-change query and name the most-coupled pair.
Run the co-change query both ways from §4 and confirm they agree.
Find the five newest source files. Open one and write down four conventions.
Compare the byte count of docs/ against the byte count of all commit messages.
git log --grep for a block's name. Was it faster than reading the block?
Write onboard and run it on three repositories. Then run it somewhere that is not a git repository and confirm exit 2.

Check Your Understanding

Why does the routine establish what is source before anything else?
What does the ratio of lines to modules tell you?
Why run two independent entry-point searches?
Why does git log --format='---' fail while --format='oneline' works?
How do you get a literal separator into a git log format?
Why read the newest files for conventions rather than the oldest?
Why are commit messages usually more accurate than docs/?
Why is a testbench better documentation than a specification?
Which two questions are worth asking a person, and why after the routine?
In onboard, why print the "generated directories git does not know about" section even when it is empty?

Answers

1. Because it decides what every later step looks at — a size count, a churn ranking or a search that includes build output is wrong in a way that looks right. 2. The decomposition style, and therefore your reading strategy: a couple of hundred lines per module is well-decomposed; a few thousand means a handful of enormous blocks. 3. Because agreement between two methods that fail for different reasons is evidence, while one method is a guess. Here: "committed as executable" and "mentions a simulator". 4. Because a --format argument containing no % is read as the name of a built-in format. oneline is a name git knows; --- is not. 5. Include a % — use hex escapes such as %x2D for a hyphen or %x3A for a colon. 6. Because the newest code shows what the team does now and the oldest shows what they did before they knew better. Your commit will be reviewed against the current convention. 7. Because they were written by the person making the change while it was fresh, and they describe something that definitely happened — nothing forces docs/ to stay true. 8. Because it is executable and fails when it is wrong, so it is maintained; a specification is prose nothing checks. When they disagree, the test describes the design. 9. "What surprises people?" and "What should I not touch?" — neither is written down anywhere, and asking them after the routine shows you did the reading, which is how you get a real answer. 10. Because "nothing found" is a useful result, and a section that prints nothing at all reads as a broken script rather than as a clean bill of health.

Chapter 28.6 deals with the opposite problem — a repository that has absorbed tool output, and the ignore rule that does nothing because the file is already tracked.

8. Remember

Eight steps, in order of answers per command: shape, what is source, size, how it is built, what is alive, who to ask, conventions, documentation.
Establish what is source first. Every later count is wrong otherwise.
git ls-files -z | xargs -0, and sum every total with awk — tail -1 reports the last batch when ARG_MAX forces several.
Lines per module tells you the style; modules defined versus files compiled tells you how much to ignore.
Find the entry point two ways — committed executables, and files mentioning a tool — and check they agree.
A git log --format with no % is read as a format NAME. Use %x2D for a literal hyphen.
%at sorts, %ar does not. Pick the specifier for the consumer.
Read the newest files for conventions, never the oldest.
Commit messages are the most accurate documentation, because nothing makes them stale.
A testbench outranks a specification — it is executable and fails when wrong.
The filelist is an authoritative inventory in compile order, maintained because the build breaks otherwise.
Read docs/ last, for intent, and verify every specific.
Ask a person two things, afterwards: what surprises people, and what not to touch.
Print a section even when it is empty, and say what empty means.

9. Module 23 in One Page

23.1 — three classes and seven kinds of directory; tree is often absent, so find -maxdepth 2 is the map; the biggest directories are the generated ones.
23.2 — prove the classification: tracked, ignored, who has touched it, and newer-than-its-artifact. A classifier that cannot abstain will say something false.
23.3 — three copies of one module is normal; the filelist decides, the intersection finds it, and the running tool confirms it independently.
23.4 — the first element of a hierarchy path is a module and every later one is an instance; head -1 on an ambiguous module gave the stale copy, and the fix was the filelist plus a warning.
23.5 — a deny-list excludes what you remembered; name what you want; and your grep may already be honouring .gitignore without telling you.
23.6 — the routine, and the measured reminder that even git log --format has a trap in it.

Three things recurred in every chapter, and they are the module's real content. The filelist is the authority, not the filesystem. Two independent methods agreeing is evidence, one is a guess. And a tool that reports a confident wrong answer is worse than one that admits uncertainty — which showed up in 23.2's classifier, 23.4's resolver and 23.5's silent grep divergence.

Module 24 narrows from navigating the tree to searching inside it: finding a signal's declaration across hundreds of files, and the different problem of searching forty thousand lines of tool output.

Where this fits

Part of the Linux & Bash curriculum.