Linux & Bash · Part II · Module 24 · Searching Design, Verification and Log Text

Turning a Search Into an Answer

Nobody ever asked for matching lines. They asked how many, which ones, or where — and the gap between grep output and an answer is three or four Part I stages. This chapter closes the module by crossing it, with the one sorting mistake that quietly ranks every number wrong.

Intermediate21 min readLinuxPart IIPipelinesawkReporting

Part II · Module 24 · Chapter 24.6 · Turning a Search Into an Answer

A lead asks "how many modules are in the AXI block", "which tests touch the scoreboard", or "where are the errors coming from". None of those is answered by a screen of matching lines, and pasting one into a message is how you get asked again.

Three shapes answer almost every question: a count, a list, a table.

Three shapes, and the stages that produce them

Each shape is the previous one plus a stage. A count is a search and a tally; a list adds extraction so you can say which; a table adds grouping and alignment so you can say where and how many at once. Choosing the shape before writing the pipeline is what stops a search becoming a screen of matching lines nobody reads.

1. A Count

The whole answer is one number, and the pipeline is grep then wc:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /usr/bin/grep -rhE '^[[:space:]]*module[[:space:]]' --include='*.sv' rtl | wc -l
       5

Two details make this correct rather than nearly correct.

-h suppresses the file name, which matters because grep -rc gives a per-file count and summing it needs awk:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /usr/bin/grep -rhcE '^[[:space:]]*module[[:space:]]' --include='*.sv' rtl | awk '{s+=$1} END {print s}'
5

Both give 5. -c with awk is the one to use on a huge tree, because it never transfers the matching lines; -h | wc -l is clearer and fine up to millions of lines.

And wc -l right-pads on BSD, which is why the output above is 5. In a report that is ugly and in a comparison it is a bug:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

n=$(... | wc -l | tr -d ' ')          # or
n=$(... | wc -l); n=$(( n ))          # arithmetic strips it

Chapter 4.5 measured that padding. The second form is the tidier one: $(( )) parses the number and discards the whitespace.

A bare number is still not an answer. Restate the question with it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ printf 'modules in rtl/: %s\n' "$(/usr/bin/grep -rhE '^[[:space:]]*module[[:space:]]' --include='*.sv' rtl | wc -l | tr -d ' ')"
modules in rtl/: 5

Five words of context, and the recipient does not have to ask what was counted or where.

2. A List

A count says how many; a list says which. The extra stage is extraction, and sed with a capture group is the tool:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /usr/bin/grep -rhE '^[[:space:]]*module[[:space:]]+' --include='*.sv' rtl |
>   sed -E 's/^[[:space:]]*module[[:space:]]+([a-zA-Z_][a-zA-Z0-9_]*).*/\1/' | sort
axi_master
axi_slave
chip_top
clk_div
uart_tx

Five names, alphabetical, nothing else. Three decisions:

sed -E with \1, which replaces the whole line with the captured identifier. The trailing .* is essential — without it the rest of the line survives the substitution, because sed replaces only what matched. Chapter 13.6 §4 is the capture-group mechanism.

sort at the end, not sort -u — here the names are already unique, and using -u would silently hide a duplicate module name, which is a defect you want to see. When you do want deduplication, sort | uniq -c tells you how many duplicates, which sort -u throws away.

And grep before sed, not sed alone. sed could do both jobs, and grep is faster at rejecting lines — which is Chapter 13.9 §5's rule that the cheapest, most selective stage goes first.

grep -o as a shorter list, and what it costs

For a simple extraction, -o skips the sed:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

/usr/bin/grep -rhoE '^[[:space:]]*module[[:space:]]+[a-zA-Z_][a-zA-Z0-9_]*' --include='*.sv' rtl |
  awk '{print $NF}' | sort

-o prints only the matched text, so the match has to be what you want — and here it includes the keyword, so an awk '{print $NF}' is still needed to drop it. Two stages either way.

-o wins when the thing you want is a self-contained token: a message id in brackets, a hex address, a time stamp. Chapter 24.4 §4's census is exactly that case, and there -o replaces the entire extraction stage.

sed wins when you want part of a match, or want to reorder fields, because -o cannot drop or rearrange anything.

3. A Table

A table answers "where" and "how many" at once, and it is the shape most questions actually want.

Counted by directory:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /usr/bin/grep -rlE '^[[:space:]]*module[[:space:]]' --include='*.sv' rtl |
>   sed 's|/[^/]*$||' | sort | uniq -c | sort -rn
   2 rtl/core
   2 rtl/axi
   1 rtl/uart

grep -l for file names, sed 's|/[^/]*$||' to strip the last component — dirname for a stream, with no process per line — then the sort | uniq -c | sort -rn idiom from Chapter 13.4.

Or as a lookup, which is the most useful table in an unfamiliar repository:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /usr/bin/grep -rnE '^[[:space:]]*module[[:space:]]+' --include='*.sv' rtl |
>   sed -E 's|^([^:]*):([0-9]*):[[:space:]]*module[[:space:]]+([a-zA-Z_][a-zA-Z0-9_]*).*|\3\t\1:\2|' |
>   sort | awk -F'\t' 'BEGIN { printf "%-14s %s\n", "MODULE", "DEFINED AT" }
>                            { printf "%-14s %s\n", $1, $2 }'
MODULE         DEFINED AT
axi_master     rtl/axi/axi_master.sv:2
axi_slave      rtl/axi/axi_slave.sv:2
chip_top       rtl/core/chip_top.sv:2
clk_div        rtl/core/clk_div.sv:1
uart_tx        rtl/uart/uart_tx.sv:1

Five modules, where each is defined, with a header. Four things are doing work:

grep -n to keep the line number, so the table's second column is a location you can open rather than just a file.

The sed reorders three captures into two tab-separated fields. Reordering is why this is sed and not -o.

A tab as the field separator and awk -F'\t', because a path contains no tab while it may contain a space. Choosing a separator the data cannot contain is the whole trick — and it is why the log tables in §4 use |.

And awk doing the alignment, with a BEGIN block for the header. Chapter 13.7 is awk; printf with %-14s is what turns columns into a table rather than a list with spaces in it.

4. A Table From a Log

The same three shapes, applied to Chapter 24.4's log. Every warning and error, in time order, as columns:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /usr/bin/grep -E '^UVM_(ERROR|WARNING) [^ ]*\([0-9]+\) @ ' run_2026_10_09/axi_burst_test.log |
>   sed -E 's/^(UVM_[A-Z]+) ([^ ]+) @ ([0-9]+): ([^ ]+) \[([A-Z_]+)\].*/\1|\3|\4|\5|\2/' |
>   sort -t'|' -k2,2n |
>   awk -F'|' 'BEGIN { printf "%-12s %8s  %-26s %-10s %s\n", "SEVERITY","TIME","SCOPE","ID","SOURCE" }
>                    { printf "%-12s %8s  %-26s %-10s %s\n", $1,$2,$3,$4,$5 }'
SEVERITY         TIME  SCOPE                      ID         SOURCE
UVM_WARNING      2500  uvm_test_top.env.sb        SB_LATE    tb/env/axi_sb.sv(88)
UVM_WARNING      7500  uvm_test_top.env.sb        SB_LATE    tb/env/axi_sb.sv(88)
UVM_WARNING     12500  uvm_test_top.env.sb        SB_LATE    tb/env/axi_sb.sv(88)
UVM_ERROR       13370  uvm_test_top.env.sb        MISMATCH   tb/env/axi_sb.sv(201)
UVM_WARNING     17500  uvm_test_top.env.sb        SB_LATE    tb/env/axi_sb.sv(88)
UVM_WARNING     22500  uvm_test_top.env.sb        SB_LATE    tb/env/axi_sb.sv(88)
UVM_WARNING     27500  uvm_test_top.env.sb        SB_LATE    tb/env/axi_sb.sv(88)
UVM_ERROR       29000  uvm_test_top.env.sb        MISMATCH   tb/env/axi_sb.sv(201)
UVM_WARNING     32500  uvm_test_top.env.sb        SB_LATE    tb/env/axi_sb.sv(88)
UVM_WARNING     37500  uvm_test_top.env.sb        SB_LATE    tb/env/axi_sb.sv(88)

Ten rows out of 4021 lines, and the shape of the failure is now visible: the warnings are regular — every 5000 time units — and the two errors are not. A regular warning is a configuration artefact; an irregular error is a bug. You cannot see that in a list of matching lines and you can see it immediately in a time-sorted table.

The sort that ranks 100 above 20

sort -t'|' -k2,2n is doing more than it looks. Drop the n:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ printf 'a|100\na|20\na|3\n' | sort -t'|' -k2,2n
a|3
a|20
a|100
$ printf 'a|100\na|20\na|3\n' | sort -t'|' -k2,2
a|100
a|20
a|3

Without n the field is compared as text, so 100 precedes 20 because 1 precedes 2. A time-ordered table becomes a nonsense order that happens to look sorted — and with simulation times, which are routinely five to nine digits, the wrong order is not obviously wrong.

Three parts of that key specification matter:

-t'|' sets the separator. Without it sort splits on whitespace, and the |-joined line is one field.

-k2,2 means "field two only". Plain -k2 means "from field two to the end of the line", so two rows with the same time would then be ordered by their scope and id — usually harmless, occasionally not, and never what you meant to say.

And n is attached to the key, not given globally. sort -n -t'|' -k2,2 works too, but -k2,2n keeps the numeric-ness next to the field it applies to, which is what you want the moment a second key is added: -k1,1 -k2,2n sorts by severity then numerically by time.

Chapter 13.3 is sort's key syntax. The reason it earns a callout here is that this is the stage where a table stops being a list, and a silently mis-sorted table is more misleading than no table at all.

5. A Census With a Total

The last shape, and the one a lead actually reads:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /usr/bin/grep -oE '\[[A-Z_]+\]' run_2026_10_09/axi_burst_test.log |
>   sort | uniq -c | sort -rn |
>   awk 'BEGIN { printf "%8s  %s\n", "COUNT", "MESSAGE ID" }
>              { printf "%8s  %s\n", $1, $2; t += $1 }
>        END   { printf "%8s  %s\n", t, "(total)" }'
   COUNT  MESSAGE ID
    4000  [AXI_MON]
       8  [SB_LATE]
       2  [MISMATCH]
       1  [TEST_DONE]
       1  [RNTST]
       1  [BUILD]
    4013  (total)

The total row is the point. 4013 of 4021 lines are accounted for, so eight lines have no message id — the banner, the summary and the verdict. A census whose total does not reconcile with the file is a census that is quietly dropping something, and printing the total is how you find out.

That reconciliation is worth doing deliberately:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /usr/bin/grep -c '' run_2026_10_09/axi_burst_test.log
4021

4021 lines, 4013 categorised, 8 unaccounted for. Being able to say where every line went is the difference between a summary and a guess — and it is the same discipline as Chapter 24.4 §2's three agreeing counts.

6. The One-Line Row

A regression produces hundreds of logs, and each needs to become one line:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ printf '%-22s %-6s %4s %4s %4s\n' 'axi_burst_test' \
>   "$(/usr/bin/grep -qE '^TEST PASSED' "$L" && echo PASS || echo FAIL)" \
>   "$(/usr/bin/grep -cE '^UVM_ERROR [^ ]*\([0-9]+\) @ ' "$L")" \
>   "$(/usr/bin/grep -cE '^UVM_FATAL [^ ]*\([0-9]+\) @ ' "$L")" \
>   "$(/usr/bin/grep -cE '^UVM_WARNING [^ ]*\([0-9]+\) @ ' "$L")"
axi_burst_test         FAIL      2    0    8

One row per test, fixed-width, so a hundred of them is a table without any further work. Two notes:

grep -q for the verdict, which exits on the first match and prints nothing — the status is the answer. Chapter 16.7's command-as-condition.

And four separate grep passes over the same file, which is wasteful and correct at this size. At regression scale it becomes one awk pass:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '
  /^TEST PASSED/                              { v = "PASS" }
  /^TEST FAILED/                              { v = "FAIL" }
  /^UVM_ERROR [^ ]*\([0-9]+\) @ /             { e++ }
  /^UVM_FATAL [^ ]*\([0-9]+\) @ /             { f++ }
  /^UVM_WARNING [^ ]*\([0-9]+\) @ /           { w++ }
  END { printf "%-22s %-6s %4d %4d %4d\n", name, (v?v:"NO-VERDICT"), e, f, w }
' name="$(basename "${L%.log}")" "$L"

One traversal, five questions — and (v?v:"NO-VERDICT") keeps §24.4's third verdict state. Reading a multi-gigabyte log once instead of five times is the difference between a regression summary that takes a minute and one that takes five. Chapter 13.7 §6 made the general argument; this is where it pays.

7. Choosing the Shape

The question	The shape	The pipeline
"How many?"	a count, with the question restated	`grep -c` → `awk` sum, or `grep -h` → `wc -l`
"Which ones?"	a sorted list	`grep` → `sed` capture → `sort`
"Where, and how many?"	a counted table	`grep -l` → `sed` → `sort \| uniq -c \| sort -rn`
"What happened, in order?"	a column table	`grep` → `sed` to fields → `sort -k2,2n` → `awk`
"What is this made of?"	a census with a total	`grep -o` → `sort \| uniq -c` → `awk` with `END`
"Pass or fail?"	one fixed-width row	one `awk` pass

And the rule that makes all six readable: put the question in the output. A header row, a restated count, a labelled total. The recipient did not run your command and cannot reconstruct what the columns are.

Practice

Count the modules in a repository you use. Then restate the count as a sentence.
Do it both ways — -h | wc -l and -c | awk — and confirm they agree.
Strip wc -l's padding two ways and check both.
Produce the sorted module list. Use sort rather than sort -u, then find out whether any name is duplicated.
Build the module-to-file table. Open one of the locations it reports.
Change the tab separator to a space and find a path that breaks the awk.
Build a time-ordered table from a log you have. Is the irregular event visible?
Drop the n from the sort key and compare the two orders. How obvious is the wrong one?
Change -k2,2 to -k2 and construct two rows where it matters.
Build a census with a total. Does the total reconcile with grep -c ''?
Find the lines your census does not categorise. What are they?
Write the one-line row for a log, then the single-pass awk version, and confirm they agree.
Time both on the largest log you have.

Check Your Understanding

What are the three shapes that answer most questions?
Why does grep -h matter when counting?
Why is a bare number not an answer?
Why sort rather than sort -u for the module list?
Why is the trailing .* essential in the sed capture?
When does -o beat sed, and when does it not?
Why choose a tab or | as a field separator?
What does sort -t'|' -k2,2 do wrong, and why is it hard to spot?
What is the difference between -k2 and -k2,2?
What does the total row in a census tell you?
Why replace four grep passes with one awk pass?

Answers

1. A count, a list and a table. 2. Because without it grep -r prefixes each line with the file name, and grep -rc gives a per-file count that has to be summed rather than a single number. 3. Because the recipient did not run the command and cannot tell what was counted or where — restating the question costs five words. 4. Because -u would silently hide a duplicated module name, which is a defect you want to see. When you do want deduplication, sort | uniq -c keeps the count. 5. Because sed replaces only the matched part, so without .* the rest of the line survives alongside the captured name. 6. -o wins when the thing you want is a self-contained token — a bracketed id, a hex address. It loses when you want only part of a match or need to reorder fields, because it cannot drop or rearrange anything. 7. Because the data cannot contain it: a path may contain a space but not a tab, so field splitting is unambiguous. 8. It sorts the field as text, so 100 comes before 20. It is hard to spot because the output still looks ordered, and simulation times have enough digits that the wrong order is not obviously wrong. 9. -k2 means from field two to the end of the line; -k2,2 means field two only. With -k2, rows that tie on field two are ordered by everything after it. 10. Whether the categories account for every line. 4013 of 4021 here, so eight lines — the banner, summary and verdict — are uncategorised, and knowing that is the difference between a summary and a guess. 11. Because at regression scale the file is read once instead of five times, which is the difference between a summary that takes a minute and one that takes five.

Chapter 29.6 is the case where there is nothing to search for: an uninitialised value poisoned every expression it touched and the simulator issued no message at all.

8. Remember

Three shapes: a count, a list, a table. Nobody asked for matching lines.
grep -h when counting, or the file names come too; grep -rc | awk sums without transferring the lines.
BSD wc -l right-pads. Strip it with tr -d ' ' or $(( )).
Restate the question in the output. "modules in rtl/: 5", not "5".
sed -E with a capture and a trailing .* to extract; without the .* the rest of the line survives.
sort, not sort -u — a duplicate is a finding. sort | uniq -c keeps the count.
grep before sed: the cheapest, most selective stage goes first.
-o for a self-contained token, sed when you need part of a match or a reorder.
Pick a separator the data cannot contain — a tab for paths, | for log fields.
sort -t'|' -k2,2n: without n the field sorts as text and 100 precedes 20. -k2,2, not -k2.
awk with a BEGIN header and printf "%-14s" is what makes a table.
Print a total and reconcile it with grep -c ''. 4013 of 4021 means eight lines went somewhere.
One fixed-width row per test, and one awk pass rather than four greps, at regression scale.

9. Module 24 in One Page

24.1 — ten hits for one signal, one of them the declaration. Narrow in four stages; -w's word characters are letters, digits and underscore only, so it fixes substrings and nothing else.
24.2 — match the construct, not the string. #$[^)]*$ found two of three instantiations because a negated class cannot span nested parentheses.
24.3 — --include='*.{sv,v,c,py}' finds nothing and exits 1, because quoting suppresses brace expansion. Repeated options or an array are the only forms that behave the same in bash, zsh and dash.
24.4 — logs are the opposite problem. grep -c '^UVM_ERROR' reported 3 where the log said 2, and invented a UVM_FATAL that never happened, because the summary lines match the pattern.
24.5 — half of what a design does is decided in build scripts. -DADDR_W=8 was accepted, exited 0, printed nothing, and had no effect.
24.6 — turn it into a count, a list or a table, and put the question in the output.

Two themes ran through all six. Every chapter's headline failure was silent: a wrong count, a missing instance, an empty result, an invented fatal, an ignored flag. None of them errored, and each was found only by knowing what the answer should have been. And the fix was never a cleverer pattern — it was a second, independent way of getting the same number, which is the habit Chapter 23.3 §3 introduced and which this module has now used five more times.

Module 25 moves from reading the build description to running it: module systems, environment files, licence servers, and what source setup.sh actually did to your shell.

Where this fits

Part of the Linux & Bash curriculum.