find and Multi-File Processing

Part III · Module 12 · Chapter 12.3 · find and Multi-File Processing

1. Why This Matters

awk takes a list of filenames and reads them in order, maintaining FILENAME and FNR as it goes. That is exactly what a regression reduction needs, and the difficulty is entirely in getting the list there.

Three things go wrong, in increasing order of how hard they are to notice: a path with a space becomes two paths, a long list becomes several awk runs, and an empty file becomes no file at all.

2. A Log Tree With the Awkward Cases

rm -rf logs && mkdir -p logs "logs/run 2"
for t in alu_smoke alu_random fifo_basic; do
  printf 'UVM_INFO ./env/mon.sv(40) @ 10: top.mon [XACT] ok\nUVM_ERROR ./env/sb.sv(148) @ 20: top.sb [MISMATCH] bad\n' > "logs/$t.log"
done
printf 'UVM_INFO ./env/mon.sv(40) @ 10: top.mon [XACT] ok\n' > "logs/run 2/spaced.log"
printf 'UVM_ERROR ./env/sb.sv(1) @ 5: top.sb [MISMATCH] x\nUVM_ERROR ./env/sb.sv(2) @ 6: top.sb [TIMEOUT] y\n' > "logs/run 2/two errors.log"
: > logs/empty.log
find logs -name '*.log' | sort

logs/alu_random.log
logs/alu_smoke.log
logs/empty.log
logs/fifo_basic.log
logs/run 2/spaced.log
logs/run 2/two errors.log

Six logs, five with content, one empty. A directory named run 2 and a file named two errors.log — both unremarkable in a real flow, where run directories carry dates and test names carry descriptions.

Note the sort. find emits paths in directory order, which is stable while the directory is unchanged and changes the moment a file is added — an earlier version of this chapter's probe produced a different ordering after logs/empty.log was created. Any multi-file output that is going to be compared against anything needs sorting.

3. The Space Splits the Path

awk '/^UVM_ERROR/ { n++ } END { printf "  rows=%d\n", n+0 }' $(find logs -name '*.log' | sort) 2>&1 | head -4

awk: can't open file logs/run
 input record number 2, file logs/run
 source line number 1

The unquoted $(…) is split on whitespace by the shell, so logs/run 2/spaced.log arrives as two arguments — logs/run and 2/spaced.log — and neither exists.

What makes this more than an inconvenience is that it is a partial failure. awk reports the file it cannot open, carries on with the ones it can, and exits with a status that reflects the problem only because it could not open a file at all. A flow that redirects stderr to a log nobody reads gets a count computed over a subset of the logs.

4. -print0 and xargs -0

find logs -name '*.log' -print0 | sort -z | xargs -0 awk '/^UVM_ERROR/ { n++ } END { printf "  total=%d over %d file(s)\n", n+0, ARGC-1 }'

  total=5 over 6 file(s)

-print0 separates paths with a NUL byte, which cannot occur in a filename, and xargs -0 splits on it. sort -z keeps the NUL convention through the sort.

ARGC-1 is the number of filenames awk received — a useful self-check, and the basis of section 7.

The portable alternative, when -print0 is unavailable or the per-file work is substantial:

find logs -name '*.log' -print | sort | while IFS= read -r f; do printf "  %-28s %s\n" "$f" "$(awk '/^UVM_ERROR/ { n++ } END { print n+0 }' "$f")"; done

  logs/alu_random.log          1
  logs/alu_smoke.log           1
  logs/empty.log               0
  logs/fifo_basic.log          1
  logs/run 2/spaced.log        0
  logs/run 2/two errors.log    2

IFS= read -r is the whole trick: IFS= stops word splitting so the space survives, and -r stops backslash interpretation. "$f" is quoted at the point of use. This form runs one awk per file, which is slower but gives a per-file answer directly — and it is the only one of the two that reports logs/empty.log at all.

5. END Runs Once Per Invocation

This is the finding that costs people a nightly report.

find logs -name '*.log' -print0 | sort -z | xargs -0 -n 2 awk '/^UVM_ERROR/ { n++ } END { printf "[%d]", n+0 }'; echo

[2][1][2]

Three numbers. xargs -n 2 runs awk once per two files, so END executes three times and emits three partial totals. The single-invocation form in section 4 printed one total=5.

-n 2 forces it here for demonstration, and xargs does the same thing on its own once the argument list exceeds the system limit:

printf "  ARG_MAX = %s bytes\n" "$(getconf ARG_MAX)"
printf "  a list of 100000 paths averaging 60 bytes = %s bytes -> xargs WILL split it\n" "$((100000*60))"

  ARG_MAX = 1048576 bytes
  a list of 100000 paths averaging 60 bytes = 6000000 bytes -> xargs WILL split it

6. Summing the Partials

find logs -name '*.log' -print0 | sort -z | xargs -0 -n 2 awk '/^UVM_ERROR/ { n++ } END { print n+0 }' | awk '{ t += $1 } END { printf "  partial totals summed: %d\n", t }'

  partial totals summed: 5

Five, matching the single-invocation answer. The second awk is a one-line adder and it makes the pipeline correct regardless of how many times xargs chose to run the first one.

Per-file results survive a split too, because FILENAME is per-record rather than per-invocation:

find logs -name '*.log' -print0 | sort -z | xargs -0 awk '/^UVM_ERROR/ { e[FILENAME]++ } END { for (f in e) printf "  %-28s %d\n", f, e[f] }' | sort

  logs/alu_random.log          1
  logs/alu_smoke.log           1
  logs/fifo_basic.log          1
  logs/run 2/two errors.log    2

Four files, not six — only the files with a matching record appear, which is section 7.

The trailing | sort is not decoration. for (f in e) has no defined order, which Chapter 10.4 section 8 measured; sorting makes the output comparable between runs.

7. An Empty File Fires Nothing

find logs -name '*.log' -print0 | sort -z | xargs -0 awk 'FNR == 1 { n++ } END { printf "  %d of %d file(s) fired FNR==1 -- the empty one never does\n", n+0, ARGC-1 }'

  5 of 6 file(s) fired FNR==1 -- the empty one never does

FNR == 1 is the standard idiom for "the first record of each file", and it does not fire for a file with no records. So a file counter built on it reports five where six files were read.

That matters because an empty log is a meaningful event: in Chapter 9.5's terms it means the test never ran, which is a flow problem rather than a design problem. A reduction that counts files by FNR == 1 cannot see it, and a reduction that counts errors sees a clean zero.

ARGC-1 is the number awk was given, so the comparison is available inside awk:

find logs -name '*.log' -print0 | sort -z | xargs -0 awk 'FNR == 1 { seen++ } END { printf "  given %d, read %d, silent %d\n", ARGC-1, seen+0, ARGC-1-(seen+0) }'

  given 6, read 5, silent 1

One silent file, named rather than inferred:

find logs -name '*.log' -print0 | sort -z | xargs -0 awk 'FNR == 1 { r[FILENAME] = 1 } END { for (i = 1; i < ARGC; i++) if (!(ARGV[i] in r)) printf "  no records: %s\n", ARGV[i] }'

  no records: logs/empty.log

Walking ARGV in END is the technique from Chapter 9.5 section 6 — enumerate from the list you were given, not from the records you saw.

One portability note, measured rather than assumed:

awk 'BEGIN { a["x"]=1; a["y"]=2; print "  length(a) =", length(a) }'

  length(a) = 2

length() on an array works in this awk. It is not POSIX — it is a gawk extension that one-true-awk has adopted — so a script that must run on an unknown awk should keep its own counter, as the commands above do.

8. Common Mistakes

awk … $(find …). The shell splits a path on its spaces — section 3.

Treating that as a total failure. awk carries on with the files it can open.

Not sorting a multi-file listing. find emits directory order, which changes when the directory does — section 2.

Assuming xargs runs awk once. END runs per invocation and the total arrives in fragments — section 5.

Reading the last line of a split run. tail -1 gave 2 against a true 5.

Testing the reduction on a small directory. The split appears only past ARG_MAX.

Counting files with FNR == 1. An empty file never fires it — section 7.

for (f in e) without a sort. No defined order.

Relying on length(array). It works here and is not POSIX.

9. Exercises

Exercise 1 — A NUL-safe reduction

find logs -name '*.log' -print0 | sort -z | xargs -0 awk '/^UVM_ERROR/ { n++ } END { printf "%d error(s) across %d file(s)\n", n+0, ARGC-1 }'

5 error(s) across 6 file(s)

Exercise 2 — Per-file counts, including the silent ones

find logs -name '*.log' -print0 | sort -z | xargs -0 awk 'FNR == 1 { seen[FILENAME] = 1 } /^UVM_ERROR/ { e[FILENAME]++ } END { for (i = 1; i < ARGC; i++) { f = ARGV[i]; printf "%-28s %s\n", f, (f in seen ? e[f] + 0 : "NO RECORDS") } }'

logs/alu_random.log          1
logs/alu_smoke.log           1
logs/empty.log               NO RECORDS
logs/fifo_basic.log          1
logs/run 2/spaced.log        0
logs/run 2/two errors.log    2

Six rows for six files, with 0 and NO RECORDS distinguished — the first ran and found nothing, the second produced nothing at all.

Exercise 3 — Debug a path that cannot be opened

A reduction reports can't open file logs/run.

awk '/^UVM_ERROR/ { n++ } END { printf "rows=%d\n", n+0 }' $(find logs -name '*.log' | sort) 2>&1 | head -3

awk: can't open file logs/run
 input record number 2, file logs/run
 source line number 1

There is no logs/run — there is logs/run 2, and the unquoted substitution split it.

find logs -name '*.log' -print0 | sort -z | xargs -0 awk '/^UVM_ERROR/ { n++ } END { printf "rows=%d\n", n+0 }'

rows=5

Exercise 4 — Debug a total that arrives in pieces

A nightly job reports far fewer errors than the per-test logs contain.

find logs -name '*.log' -print0 | sort -z | xargs -0 -n 2 awk '/^UVM_ERROR/ { n++ } END { print n+0 }'

2
1
2

Three totals, because xargs -n 2 ran awk three times and END fired three times. A script taking the last line gets 2.

find logs -name '*.log' -print0 | sort -z | xargs -0 -n 2 awk '/^UVM_ERROR/ { n++ } END { print n+0 }' | awk '{ t += $1 } END { printf "summed: %d\n", t }'

summed: 5

Exercise 5 — Engineering application

A reduction that survives spaces, survives a split argument list, and reports the files that produced nothing.

#!/bin/sh
# reduce.sh — reduce a tree of simulation logs to one error count.
#
# Usage:  sh reduce.sh <dir>
# Exit:   0 every log was read and none holds an error
#         1 errors present
#         2 a log produced no records at all, or the file count disagrees
#
# Correct for any list length: the per-invocation partials are summed by a
# second awk, so it does not matter how many times xargs runs the first one.

dir=${1:-logs}
[ -d "$dir" ] || { echo "no such directory: $dir" >&2; exit 2; }

# the authoritative file count, from the same sorted NUL-separated list
want=$(find "$dir" -name '*.log' -print0 | sort -z | tr -dc '\0' | wc -c | tr -d ' ')

find "$dir" -name '*.log' -print0 | sort -z | xargs -0 awk '
    FNR == 1 { seen[FILENAME] = 1 }
    /^UVM_ERROR (\.|\/|@)/ { n++ }
    END {
        silent = 0
        for (i = 1; i < ARGC; i++) if (!(ARGV[i] in seen)) { silent++; printf "SILENT %s\n", ARGV[i] }
        printf "PART %d %d %d\n", n + 0, ARGC - 1, silent
    }
' > parts.txt

awk -v want="$want" '
    /^SILENT / { sub(/^SILENT /, ""); silent[++ns] = $0; next }
    /^PART /   { errs += $2; files += $3; sil += $4; nparts++ }
    END {
        printf "# %d invocation(s) of awk, %d file(s) read of %d found\n", nparts, files, want
        printf "# %d error(s)\n", errs
        for (i = 1; i <= ns; i++) printf "# no records: %s\n", silent[i]
        if (files != want) { printf "ERROR: awk saw %d file(s), find produced %d\n", files, want > "/dev/stderr"; exit 2 }
        if (sil) exit 2
        if (errs) exit 1
    }
' parts.txt

sh reduce.sh logs
echo "status: $?"

# 1 invocation(s) of awk, 6 file(s) read of 6 found
# 5 error(s)
# no records: logs/empty.log
status: 2

The invocation count is reported rather than assumed, the file count is checked against find's own, and the empty log is named. Status 2 rather than 1 because a log with no records means a test did not run — a different problem from a test that failed.

Removing the empty log leaves the ordinary failing case:

rm logs/empty.log
sh reduce.sh logs
echo "status: $?"

# 1 invocation(s) of awk, 5 file(s) read of 5 found
# 5 error(s)
status: 1

The tr -dc '\0' | wc -c idiom counts NUL bytes, which is how many paths the NUL-separated stream carried — counting lines would have been wrong on a path containing a newline, which is legal in a filename and which -print0 is there to survive.

Exercise 6 — Challenge: show that the file count is the only check that detects a split

reduce.sh sums the partials, so its total is right either way. Show that nothing about the total reveals whether awk ran once or many times, and that the file count does.

for n in 1 2 3 6; do
  printf "  -n %-2s " "$n"
  find logs -name '*.log' -print0 | sort -z | xargs -0 -n $n awk '/^UVM_ERROR (\.|\/|@)/ { c++ } END { print c+0 }' | awk -v n=$n '{ t += $1; p++ } END { printf "%d invocation(s), summed total %d\n", p, t }'
done

  -n 1  5 invocation(s), summed total 5
  -n 2  3 invocation(s), summed total 5
  -n 3  2 invocation(s), summed total 5
  -n 6  1 invocation(s), summed total 5

The total is 5 in every case. Summing the partials makes the answer independent of the batching, which is exactly what it is for — and it means the total carries no information about whether a split occurred.

That is fine when the partials are summed and fatal when they are not:

for n in 1 2 3 6; do
  printf "  -n %-2s last partial only: " "$n"
  find logs -name '*.log' -print0 | sort -z | xargs -0 -n $n awk '/^UVM_ERROR (\.|\/|@)/ { c++ } END { print c+0 }' | tail -1
done

  -n 1  last partial only: 2
  -n 2  last partial only: 2
  -n 3  last partial only: 2
  -n 6  last partial only: 5

The answer changes with the batch size and is wrong in three of the four cases — 2 where the true total is 5. The correct value appears only at -n 6, where the batch happens to hold every file, and 2 is a perfectly plausible error count for a five-log regression. Note also that these runs follow Exercise 5's rm logs/empty.log, so the directory holds five logs and the last one sorted is logs/run 2/two errors.log, which is where the 2 comes from; with the empty log still present the final batch would have ended on it and reported 0.

So the detection cannot come from the number. It comes from counting what awk was given against what find produced, which is the check reduce.sh runs and the reason it prints the invocation count. A pipeline whose answer depends on an invisible batching decision must report the batching, which is the same argument as Chapter 11.8's rule that a figure must name its aggregation.

10. Summary

• awk … $(find …) splits a path on its spaces and fails partially — section 3.
• -print0 | sort -z | xargs -0 is the NUL-safe list; IFS= read -r with "$f" is the per-file alternative.
• find emits directory order, which changes when the directory does. Sort anything you will compare.
• xargs may run awk more than once, so END runs more than once — 2 1 2 against a single total=5.
• ARG_MAX is 1,048,576 bytes here, so a list of 100,000 paths splits.
• Reading the last partial gave 0, 2, 2 or 5 depending only on the batch size — section 9.
• Sum the partials with a one-line second awk, and the total is correct for any list length.
• FILENAME survives a split because it is per-record; END-block state does not.
• An empty file never fires FNR == 1 — 5 of 6. Enumerate from ARGV to name it.
• Check ARGC-1 against the path count. The total cannot reveal a split; the file count can.
• length(array) works here and is not POSIX.

Chapter 12.4 — Parsing Tcl-Generated Reports moves the boundary inside the EDA tool: a Tcl script writes the report and awk reduces it outside, and the chapter's point is that choosing the format at the Tcl end makes the awk trivial or impossible.