Processing Multiple Input Files

Part II · Module 7 · Chapter 7.5 · Processing Multiple Input Files

1. Why This Matters

Chapter 7.4 covered reporting per file. This chapter is about the other two multi-file jobs.

One program over many files, where the files are interchangeable — every log in a regression directory, scanned for the same pattern. That is section 2 and it is nearly free.

Two files with different roles, where the first supplies keys and the second supplies data. That is the FNR == NR idiom, it is how awk does a join, and it is the subject of the rest of the chapter — because it is both indispensable and quietly fragile.

2. One Program, Many Files

Name as many files as you like; awk reads them in order as a single record stream:

UVM_INFO start
UVM_ERROR mismatch
UVM_INFO done

UVM_INFO start
UVM_INFO done

UVM_FATAL deadlock

awk '/UVM_ERROR|UVM_FATAL/ { print FILENAME ":" FNR ": " $0 }' logs/alu.log logs/uart.log logs/ddr.log

logs/alu.log:2: UVM_ERROR mismatch
logs/ddr.log:1: UVM_FATAL deadlock

Counting across the same three:

awk '/UVM_ERROR|UVM_FATAL/ { n++ } END { printf "%d error record(s) across %d log(s)\n", n + 0, ARGC - 1 }' logs/alu.log logs/uart.log logs/ddr.log

2 error record(s) across 3 log(s)

In practice the file list comes from a shell glob — logs/*.log — which expands to whatever the directory holds. That is convenient and it means the denominator depends on the directory rather than on the command, so the count is only as trustworthy as your knowledge of what is in there.

Note ARGC - 1 for the log count rather than a counter — Chapter 7.4 section 8's technique, and the only way to include logs that produced no records.

The glob is expanded by the shell, not by awk, which has two consequences worth knowing. A pattern matching nothing is passed through literally and awk reports it as a missing file. And a directory with thousands of logs can exceed the command-line length limit, which is what Chapter 12.3's find … | xargs exists for.

3. The Two-File Idiom

The job: keep only the records whose first field appears in a watchlist.

alu
uart

alu PASS
fifo FAIL
uart PASS
spi FAIL

awk 'FNR == NR { want[$1]; next } $1 in want' keep.txt all.txt

alu PASS
uart PASS

Two tests kept out of four. This is the standard awk join and it is worth reading slowly, because every part of it is doing something.

FNR == NR is true only while the first file is being read. want[$1] adds a key to a set, using Chapter 6.1 section 6's no-assignment form. next stops that record going any further. And $1 in want is a bare pattern with no action, so matching records print whole — Chapter 4.1's rule.

4. Why FNR == NR Works

The condition is not magic; it is Chapter 7.4 section 2's divergence used as a phase detector:

awk '{ printf "%-10s NR=%d FNR=%d  %s\n", FILENAME, NR, FNR, (FNR == NR ? "KEY PHASE" : "data phase") }' keep.txt all.txt

keep.txt   NR=1 FNR=1  KEY PHASE
keep.txt   NR=2 FNR=2  KEY PHASE
all.txt    NR=3 FNR=1  data phase
all.txt    NR=4 FNR=2  data phase
all.txt    NR=5 FNR=3  data phase
all.txt    NR=6 FNR=4  data phase

FNR resets with each file while NR keeps counting, so they are equal only during the first file. Once the second file starts, NR is ahead by however many records the first file had.

That phrasing contains the bug. The two are equal only during the first file provided the first file had at least one record.

5. An Empty Key File Swallows the Data File

Give the same command an empty first file:

: > emptykeep.txt
awk 'FNR == NR { want[$1]; next } $1 in want' emptykeep.txt all.txt
echo "status: $?"

status: 0

No output, exit status 0. It looks exactly like a watchlist that matched nothing.

The phase trace shows what actually happened:

awk '{ printf "%-16s NR=%d FNR=%d  %s\n", FILENAME, NR, FNR, (FNR == NR ? "KEY PHASE" : "data phase") }' emptykeep.txt all.txt

all.txt          NR=1 FNR=1  KEY PHASE
all.txt          NR=2 FNR=2  KEY PHASE
all.txt          NR=3 FNR=3  KEY PHASE
all.txt          NR=4 FNR=4  KEY PHASE

Every record of the data file was treated as a key. The empty file contributed nothing to NR, so when all.txt began, NR and FNR were both 1 — and stayed equal for the whole file.

So the program loaded the data into want[], ran next on every record, and never reached the data rule at all. Instrumenting it confirms both halves:

awk 'FNR == NR { want[$1]; keys++; next } { seen++ } END { printf "keys collected: %d\ndata records seen: %d\n", keys + 0, seen + 0 }' emptykeep.txt all.txt

keys collected: 4
data records seen: 0

Four keys from a two-line watchlist that was empty, and zero data records from a four-line data file.

6. The Obvious Guard Does Not Fire

The natural check is "did I collect any keys?" — and it passes:

awk 'FNR == NR { want[$1]; keys++; next } { seen++ } END { if (!keys) { print "ERROR: key file was empty" > "/dev/stderr"; exit 2 } printf "keys=%d seen=%d\n", keys, seen + 0 }' emptykeep.txt all.txt 2>/dev/null
echo "status: $?"

keys=4 seen=0
status: 0

Status 0, and keys=4. The guard did not fire because keys is not zero — the data file's four records were counted as keys.

The assertion fails for the same reason the program failed. Asking "did the key phase collect anything?" cannot detect a fault whose symptom is the key phase collecting too much.

7. The Guard That Does

Assert on the data phase instead:

awk 'FNR == NR { want[$1]; keys++; next } { seen++; if ($1 in want) print } END { if (!seen) { print "ERROR: data phase saw 0 records (was the key file empty?)" > "/dev/stderr"; exit 2 } printf "(matched against %d keys over %d records)\n", keys, seen }' emptykeep.txt all.txt 2>&1 >/dev/null
echo "status: $?"

ERROR: data phase saw 0 records (was the key file empty?)
status: 2

It fires, names the likely cause, and exits 2 — Chapter 5.6's "could not evaluate" status.

On the correct inputs the same command works normally:

awk 'FNR == NR { want[$1]; keys++; next } { seen++; if ($1 in want) print } END { if (!seen) { print "ERROR: data phase saw 0 records" > "/dev/stderr"; exit 2 } printf "(matched against %d keys over %d records)\n", keys, seen }' keep.txt all.txt
echo "status: $?"

alu PASS
uart PASS
(matched against 2 keys over 4 records)
status: 0

Two matches, with the key count and the record count stated — so a reader can see the filter had something to work with.

A second, independent guard is worth adding when the file count is fixed, because it catches a different mistake — a caller passing the wrong number of files:

awk 'BEGIN { if (ARGC != 3) { printf "ERROR: need exactly 2 files, got %d\n", ARGC - 1 > "/dev/stderr"; exit 2 } } FNR == NR { want[$1]; next } $1 in want' keep.txt 2>&1
echo "status: $?"

ERROR: need exactly 2 files, got 1
status: 2

Without that guard, a single-file invocation reads the whole file as keys and produces nothing — the same silent-empty failure from a different cause. The check runs in BEGIN, before any input, which is Chapter 7.4 section 8's point about ARGV being available early.

8. FILENAME == ARGV[1], and Its Own Limit

There is another way to detect the phase: compare FILENAME to the first argument rather than comparing counters.

It is immune to the empty-key-file problem:

awk 'FILENAME == ARGV[1] { want[$1]; next } { seen++; if ($1 in want) print } END { printf "(data phase saw %d records)\n", seen + 0 }' emptykeep.txt all.txt

(data phase saw 4 records)

Four records reached the data phase, correctly, because the phase is decided by which file rather than by a record count. Nothing was swallowed.

And it has a failure the counter form does not. When the same file is named twice — a legitimate thing to do, and exactly what Chapter 7.2 section 11 did to measure column widths — it matches both passes:

awk 'FILENAME == ARGV[1] { a++; next } { b++ } END { printf "phase1=%d phase2=%d\n", a + 0, b + 0 }' all.txt all.txt

phase1=8 phase2=0

All eight records went to the first phase and the second never ran. FNR == NR handles that case correctly:

awk 'FNR == NR { a++; next } { b++ } END { printf "phase1=%d phase2=%d\n", a + 0, b + 0 }' all.txt all.txt

phase1=4 phase2=4

Four and four, which is what a two-pass program needs.

9. Three or More Files

FNR == NR only distinguishes the first file from the rest. For three roles, compare FILENAME or count file boundaries:

awk 'FNR == 1 { phase++ } phase == 1 { a[$1]; next } phase == 2 { b[$1]; next } { n++; if ($1 in a) ina++; if ($1 in b) inb++ } END { printf "%d data record(s); %d in list A, %d in list B\n", n + 0, ina + 0, inb + 0 }' keep.txt keep2.txt all.txt

fifo
uart

4 data record(s); 2 in list A, 2 in list B

FNR == 1 fires once per file, so phase counts files — and this form is robust to an empty file in a way FNR == NR is not, because an empty file produces no FNR == 1 record and so does not consume a phase number. That is a real benefit and also a hazard: the phases then shift, and file three is processed as phase 2.

For anything beyond two files, naming the roles explicitly is clearer than inferring them:

awk 'FILENAME == ARGV[1] { a[$1]; next } FILENAME == ARGV[2] { b[$1]; next } { n++; if ($1 in a) ina++; if ($1 in b) inb++ } END { printf "%d data record(s); %d in A, %d in B\n", n + 0, ina + 0, inb + 0 }' keep.txt keep2.txt all.txt

4 data record(s); 2 in A, 2 in B

Same answer, and the rules say which file they mean.

10. Common Mistakes

Using FNR == NR with a key file that might be empty. The data file becomes keys and the output is silently empty — section 5.

Guarding with a key count. It does not fire, because the keys are the data — section 6.

No guard at all. Empty output from this idiom is indistinguishable from no matches.

Forgetting next in the key rule. Key records then fall through to the data rule as well.

Testing membership with want[$1] != "". It creates the key — Chapter 6.1 section 5.

Using FILENAME == ARGV[1] on a file named twice. Both passes match the first phase — section 8.

Assuming FNR == NR separates three files. It separates the first from all the rest — section 9.

Letting the shell glob pass an unmatched pattern through. awk then reports a missing file.

Counting input files with a counter instead of ARGC - 1. A counter misses empty files.

11. Exercises

Exercise 1 — Predict the phase

awk 'FNR == NR { print "key:", $0; next } { print "data:", $0 }' keep.txt all.txt

key: alu
key: uart
data: alu PASS
data: fifo FAIL
data: uart PASS
data: spi FAIL

Two keys then four data records — FNR == NR is true only while the first file is read.

Exercise 2 — Filter by a watchlist

Keep only the records whose test name is in the watchlist.

alu
uart

alu PASS
fifo FAIL
uart PASS
spi FAIL

awk 'FNR == NR { want[$1]; next } $1 in want' keep.txt all.txt

alu PASS
uart PASS

Exercise 3 — Invert the filter

Report the records not in the watchlist.

awk 'FNR == NR { want[$1]; next } !($1 in want)' keep.txt all.txt

fifo FAIL
spi FAIL

The parentheses are required — !$1 in want parses as (!$1) in want and is always false, per Chapter 6.1 section 6.

Exercise 4 — Debug a filter that returns nothing

A watchlist filter produced no output and exited 0. The watchlist file turned out to be empty.

: > emptykeep.txt
awk 'FNR == NR { want[$1]; next } $1 in want' emptykeep.txt all.txt
echo "status: $?"

status: 0

awk '{ printf "%-16s NR=%d FNR=%d  %s\n", FILENAME, NR, FNR, (FNR == NR ? "KEY PHASE" : "data phase") }' emptykeep.txt all.txt

all.txt          NR=1 FNR=1  KEY PHASE
all.txt          NR=2 FNR=2  KEY PHASE
all.txt          NR=3 FNR=3  KEY PHASE
all.txt          NR=4 FNR=4  KEY PHASE

The empty file added nothing to NR, so FNR == NR held for the whole of all.txt and every data record was loaded as a key.

Exercise 5 — Build a guard that fires

Show that a key-count guard misses this, then write one that does not.

awk 'FNR == NR { want[$1]; keys++; next } { seen++ } END { if (!keys) { print "ERROR: key file was empty" > "/dev/stderr"; exit 2 } printf "keys=%d seen=%d\n", keys, seen + 0 }' emptykeep.txt all.txt 2>/dev/null
echo "status: $?"

keys=4 seen=0
status: 0

keys is 4, so the guard is silent. The seen=0 beside it is the real signal.

awk 'FNR == NR { want[$1]; keys++; next } { seen++; if ($1 in want) print } END { if (!seen) { print "ERROR: data phase saw 0 records (was the key file empty?)" > "/dev/stderr"; exit 2 } }' emptykeep.txt all.txt 2>&1 >/dev/null
echo "status: $?"

ERROR: data phase saw 0 records (was the key file empty?)
status: 2

And it stays quiet when the input is good:

awk 'FNR == NR { want[$1]; keys++; next } { seen++; if ($1 in want) print } END { if (!seen) { print "ERROR: data phase saw 0 records" > "/dev/stderr"; exit 2 } }' keep.txt all.txt
echo "status: $?"

alu PASS
uart PASS
status: 0

A guard that has never been shown firing is a comment; this one has been shown both ways.

Exercise 6 — Engineering application

Filter a results file against a watchlist of tests, reporting how many keys were loaded, how many records were examined, how many matched, and which watched tests never appeared — with a non-zero status if the filter could not be evaluated.

awk 'FNR == NR { want[$1]; keys++; next } { seen++; if ($1 in want) { hit[$1]++; matched++; print } } END { if (!seen) { print "ERROR: no data records (empty key file?)" > "/dev/stderr"; exit 2 } miss = 0; for (k in want) if (!(k in hit)) { printf "watched but absent: %s\n", k; miss++ } printf "(%d key(s), %d record(s) examined, %d matched, %d absent)\n", keys, seen, matched + 0, miss }' keep.txt all.txt
echo "status: $?"

alu PASS
uart PASS
(2 key(s), 4 record(s) examined, 2 matched, 0 absent)
status: 0

The absent column is the one that earns its place. A watchlist entry that never appears in the results usually means the test did not run — the same class of finding as Chapter 7.4's empty log, detected here by comparing the keys loaded against the keys hit.

Adding a watched test that is not in the results shows it working:

printf 'alu\nuart\nnosuchtest\n' > keep3.txt
awk 'FNR == NR { want[$1]; keys++; next } { seen++; if ($1 in want) { hit[$1]++; print } } END { if (!seen) { print "ERROR: no data records (empty key file?)" > "/dev/stderr"; exit 2 } miss = 0; for (k in want) if (!(k in hit)) { printf "watched but absent: %s\n", k; miss++ } printf "(%d key(s), %d record(s) examined, %d absent)\n", keys, seen, miss }' keep3.txt all.txt
echo "status: $?"

alu PASS
uart PASS
watched but absent: nosuchtest
(3 key(s), 4 record(s) examined, 1 absent)
status: 0

nosuchtest is named. Note the membership test is !(k in hit) rather than hit[k] == 0, which would create a key for every absent test and make a later count of hit wrong — Chapter 6.1 section 5 again.

Exercise 7 — Challenge: a two-file program that cannot fail silently

Combine every guard in this chapter — the file count, the same-file check, and the data-phase assertion — so that no bad invocation produces a confident empty result.

A first attempt, with the guards in BEGIN:

awk 'BEGIN { if (ARGC != 3) { printf "ERROR: need 2 files, got %d\n", ARGC - 1 > "/dev/stderr"; exit 2 } } FNR == NR { want[$1]; keys++; next } { seen++ } END { printf "keys=%d seen=%d\n", keys + 0, seen + 0 }' keep.txt 2>&1
echo "status: $?"

ERROR: need 2 files, got 1
keys=0 seen=0
status: 2

The error fired and the status is right — and the summary printed anyway. exit in BEGIN skips the input but still runs END, which is Chapter 5.6 section 7's rule arriving as a reporting bug: the run emitted both a refusal and a result.

On a terminal that is merely untidy. In a pipeline it is worse, because a consumer reading stdout gets keys=0 seen=0 — a plausible-looking summary — while the refusal went to stderr where nothing may be looking.

The repair is a flag the END rule honours:

awk 'BEGIN { if (ARGC != 3) { printf "ERROR: need 2 files, got %d\n", ARGC - 1 > "/dev/stderr"; bail = 1; exit 2 } if (ARGV[1] == ARGV[2]) { print "ERROR: both files are the same" > "/dev/stderr"; bail = 1; exit 2 } } FNR == NR { want[$1]; keys++; next } { seen++; if ($1 in want) matched++ } END { if (bail) exit 2; if (!seen) { print "ERROR: data phase saw 0 records" > "/dev/stderr"; exit 2 } printf "keys=%d seen=%d matched=%d\n", keys, seen, matched + 0 }' keep.txt 2>&1
echo "status: $?"

ERROR: need 2 files, got 1
status: 2

One line of output and status 2. Each guard now refuses a different bad call. The same file twice:

awk 'BEGIN { if (ARGC != 3) { printf "ERROR: need 2 files, got %d\n", ARGC - 1 > "/dev/stderr"; bail = 1; exit 2 } if (ARGV[1] == ARGV[2]) { print "ERROR: both files are the same" > "/dev/stderr"; bail = 1; exit 2 } } FNR == NR { want[$1]; keys++; next } { seen++; if ($1 in want) matched++ } END { if (bail) exit 2; if (!seen) { print "ERROR: data phase saw 0 records" > "/dev/stderr"; exit 2 } printf "keys=%d seen=%d matched=%d\n", keys, seen, matched + 0 }' all.txt all.txt 2>&1
echo "status: $?"

ERROR: both files are the same
status: 2

An empty key file, which BEGIN cannot detect at all:

awk 'BEGIN { if (ARGC != 3) { printf "ERROR: need 2 files, got %d\n", ARGC - 1 > "/dev/stderr"; bail = 1; exit 2 } if (ARGV[1] == ARGV[2]) { print "ERROR: both files are the same" > "/dev/stderr"; bail = 1; exit 2 } } FNR == NR { want[$1]; keys++; next } { seen++; if ($1 in want) matched++ } END { if (bail) exit 2; if (!seen) { print "ERROR: data phase saw 0 records" > "/dev/stderr"; exit 2 } printf "keys=%d seen=%d matched=%d\n", keys, seen, matched + 0 }' emptykeep.txt all.txt 2>&1
echo "status: $?"

ERROR: data phase saw 0 records
status: 2

And the correct invocation:

awk 'BEGIN { if (ARGC != 3) { printf "ERROR: need 2 files, got %d\n", ARGC - 1 > "/dev/stderr"; bail = 1; exit 2 } if (ARGV[1] == ARGV[2]) { print "ERROR: both files are the same" > "/dev/stderr"; bail = 1; exit 2 } } FNR == NR { want[$1]; keys++; next } { seen++; if ($1 in want) matched++ } END { if (bail) exit 2; if (!seen) { print "ERROR: data phase saw 0 records" > "/dev/stderr"; exit 2 } printf "keys=%d seen=%d matched=%d\n", keys, seen, matched + 0 }' keep.txt all.txt
echo "status: $?"

keys=2 seen=4 matched=2
status: 0

Four invocations, four distinct outcomes, and no combination that yields a quiet empty result.

Two closing notes. The bail flag is needed because exit is not abort — it routes through END, so any END rule that prints must be told not to. And the same-file guard would be wrong in a measure-then-format program, where naming one file twice is the whole technique (Chapter 7.2 section 11): a guard encodes an assumption, so it belongs only in a program that actually makes it.

12. Summary

• awk reads many files as one record stream; a shell glob is the usual way to name them.
• ARGC - 1 counts the input files, including ones that yield no records.
• FNR == NR is true only during the first file — the two-file join idiom, with next to keep the phases apart.
• An empty first file breaks it completely: the data file is loaded as keys, nothing prints, and the status is 0.
• A key-count guard does not fire, because the fault is the key phase collecting too much.
• Assert on the data phase — if (!seen) detects it and names the likely cause.
• FILENAME == ARGV[1] survives the empty file and fails on a file named twice. Neither form is universally right.
• A guard must assert on something the failure cannot corrupt.
• For three or more roles, name the files rather than inferring phases from counters.
• Add an ARGC check when the file count is fixed; it catches a different caller error.

Two files can now be read together safely. What comes out still has to be consumable: Chapter 7.6 — Writing CSV and TSV Output covers emitting a delimited file a spreadsheet or a Python script can load, and the field containing your own delimiter — which awk will happily write and cannot reliably read back.