NR, FNR, and FILENAME

Part II · Module 7 · Chapter 7.4 · NR, FNR, and FILENAME

1. Why This Matters

Chapter 5.2 covered these three variables as a reference: NR counts every record read, FNR resets per file, FILENAME names the current one. If you want the rules, that is where they are.

This chapter is the practice, and it adds four things that reference did not:

• Labelling output so a reader knows which log a line came from.
• A readable name derived from a path like logs/alu_run.log.
• ARGC and ARGV — awk's view of its own command line, not covered in 5.2 at all.
• The repair for 5.2's hole. That chapter established that a per-file rule never runs for a file with no records. Section 8 is how you report on it anyway.

That last one is the reason this chapter is rated as it is. A regression summary that silently omits the tests which produced no output is worse than one that omits nothing, because the missing rows are the most urgent.

2. The Three Variables Across Two Files

A trace, with each line labelled by its source:

UVM_INFO start
UVM_ERROR mismatch
UVM_INFO done

UVM_INFO start
UVM_INFO done

awk '{ printf "%-14s %2d %2d  %s\n", FILENAME, FNR, NR, $0 }' logs/alu.log logs/uart.log

logs/alu.log    1  1  UVM_INFO start
logs/alu.log    2  2  UVM_ERROR mismatch
logs/alu.log    3  3  UVM_INFO done
logs/uart.log   1  4  UVM_INFO start
logs/uart.log   2  5  UVM_INFO done

FNR runs 1, 2, 3 then restarts at 1. NR runs 1 to 5 without a break. FILENAME changes with the reset.

So the division of labour is: FNR for position within a log, NR for position in the whole job, FILENAME for which log. Everything else in this chapter is built from those three.

3. Labelling Output by Source File

The simplest and most useful application. When a pattern matches across many logs, the match is useless without its origin:

awk '/UVM_ERROR|UVM_FATAL/ { print FILENAME ":" FNR ": " $0 }' logs/alu.log logs/uart.log logs/ddr.log

UVM_FATAL deadlock

logs/alu.log:2: UVM_ERROR mismatch
logs/ddr.log:1: UVM_FATAL deadlock

That is the file:line: message format every compiler and linter uses, and it is worth matching deliberately: editors and IDEs parse it, so the output becomes clickable.

Note FNR rather than NR. FNR is the line number within its file, which is what a reader needs to go and look. NR would give the cumulative position across the job, which points nowhere.

4. A Header per File

FNR == 1 is true on the first record of each file, which is the per-file header idiom:

awk 'FNR == 1 { printf "\n== %s ==\n", FILENAME } { print "  " $0 }' logs/alu.log logs/uart.log

== logs/alu.log ==
  UVM_INFO start
  UVM_ERROR mismatch
  UVM_INFO done

== logs/uart.log ==
  UVM_INFO start
  UVM_INFO done

Two rules: one fires once per file, the other on every record. Chapter 2.3's source order puts the header before the records of its own file.

The leading blank line comes from the \n at the start of the format, which separates the sections. Putting it there rather than at the end means the output does not finish with a trailing blank line.

This idiom has the hole section 7 is about. It is the standard way to structure multi-file output and it silently skips a file with no records.

5. A Readable Name from a Path

FILENAME holds whatever was on the command line, which for a glob is a full path. Trimming it to something a report can use:

awk 'FNR == 1 { n = FILENAME; sub(/.*\//, "", n); sub(/\.log$/, "", n); print n }' logs/alu.log logs/uart.log logs/ddr.log

alu
uart
ddr

Two substitutions: sub(/.*\//, "", n) removes everything up to and including the last slash — the regex is greedy, so .*\/ matches as far as the final / — and sub(/\.log$/, "", n) removes the extension.

sub() is Module 8's function. What matters here is the pattern: derive the label once per file into a variable, rather than recomputing it on every record. On a million-record log that is a million substitutions saved, and more importantly the label is then available to every later rule under a name that says what it is.

Note n is assigned in the FNR == 1 rule and read in later rules — Chapter 5.1's between-records persistence doing exactly what it is for. It survives until the next file's first record overwrites it.

6. Per-File Aggregation

Counting per file needs a way to notice the file changing. FILENAME != prev is the idiom:

awk 'FILENAME != prev { if (prev != "") printf "%-16s %d\n", prev, n; prev = FILENAME; n = 0 } { n++ } END { if (prev != "") printf "%-16s %d\n", prev, n }' logs/alu.log logs/uart.log

logs/alu.log     3
logs/uart.log    2

Three mechanisms in one line, and each is load-bearing. FILENAME != prev detects the boundary. The prev != "" guard stops a report being printed before the first file has been read. And the END rule flushes the last file, which nothing else would — the boundary rule only fires when a new file starts, and no file starts after the last one.

An array keyed on FILENAME is usually clearer, and is what Chapter 6.3 would suggest:

awk '{ n[FILENAME]++; if (/UVM_ERROR|UVM_FATAL/) bad[FILENAME]++ } END { for (f in n) printf "%-16s %d record(s), %d error(s)\n", f, n[f], (f in bad ? bad[f] : 0) | "sort" }' logs/alu.log logs/uart.log logs/ddr.log

logs/alu.log     3 record(s), 1 error(s)
logs/ddr.log     1 record(s), 1 error(s)
logs/uart.log    2 record(s), 0 error(s)

No boundary detection, no prev, no END flush — the array handles all of it, and (f in bad ? bad[f] : 0) avoids the phantom key Chapter 6.1 section 5 warned about and Chapter 7.3's final exercise demonstrated.

Both forms have the same hole.

7. The Empty File Is Invisible

Add an empty log to the list:

: > logs/spi.log
awk '{ n[FILENAME]++ } END { for (f in n) printf "%-16s %d\n", f, n[f] | "sort" }' logs/alu.log logs/spi.log logs/uart.log

logs/alu.log     3
logs/uart.log    2

logs/spi.log is absent. Not reported as zero — not reported at all.

The reason is the one Chapter 5.2 section 6 established: every rule here is driven by a record, and an empty file supplies none. awk opens it, finds nothing, and moves on. The FILENAME != prev form behaves identically, for the same reason.

8. Driving the Report from ARGV

awk exposes its own command line. ARGC is the argument count and ARGV the arguments, with ARGV[0] being the program name:

awk 'BEGIN { printf "ARGC=%d", ARGC; for (i = 0; i < ARGC; i++) printf "  ARGV[%d]=%s", i, ARGV[i]; print "" }' logs/alu.log logs/uart.log

ARGC=3  ARGV[0]=awk  ARGV[1]=logs/alu.log  ARGV[2]=logs/uart.log

So the input files are ARGV[1] through ARGV[ARGC-1]. That list exists in BEGIN, before any file is read — which is the whole point, because it does not depend on any file producing a record.

Seeding the counters from ARGV and reporting from the same list:

awk 'BEGIN { for (i = 1; i < ARGC; i++) count[ARGV[i]] = 0 } { count[FILENAME]++ } END { for (i = 1; i < ARGC; i++) printf "%-16s %d%s\n", ARGV[i], count[ARGV[i]], (count[ARGV[i]] == 0 ? "   <- EMPTY" : "") }' logs/alu.log logs/spi.log logs/uart.log

logs/alu.log     3
logs/spi.log     0   <- EMPTY
logs/uart.log    2

Three rows for three files, with the empty one named and flagged.

Two properties make this work. The BEGIN loop creates an entry for every file whether or not it ever yields a record, so the keys exist independently of the data. And the END loop walks ARGV by position rather than iterating the array — which also gives a defined output order, since Chapter 6.1 section 10 established that for (f in count) has none. The rows come out in command-line order, reproducibly, with no sort.

Turning it into a gate:

awk 'BEGIN { for (i = 1; i < ARGC; i++) count[ARGV[i]] = 0 } { count[FILENAME]++ } END { for (i = 1; i < ARGC; i++) { printf "%-16s %d\n", ARGV[i], count[ARGV[i]]; if (count[ARGV[i]] == 0) empty++ } if (empty) { printf "ERROR: %d of %d log(s) were empty\n", empty, ARGC - 1 > "/dev/stderr"; exit 2 } }' logs/alu.log logs/spi.log logs/uart.log 2>/dev/null
echo "status: $?"

logs/alu.log     3
logs/spi.log     0
logs/uart.log    2
status: 2

The diagnostic went to stderr, which this command discarded. Reading only that stream:

awk 'BEGIN { for (i = 1; i < ARGC; i++) count[ARGV[i]] = 0 } { count[FILENAME]++ } END { for (i = 1; i < ARGC; i++) { printf "%-16s %d\n", ARGV[i], count[ARGV[i]]; if (count[ARGV[i]] == 0) empty++ } if (empty) { printf "ERROR: %d of %d log(s) were empty\n", empty, ARGC - 1 > "/dev/stderr"; exit 2 } }' logs/alu.log logs/spi.log logs/uart.log 2>&1 >/dev/null

ERROR: 1 of 3 log(s) were empty

The two streams are shown separately on purpose. Combining them with 2>&1 is not reliable here — stderr is unbuffered while stdout to a pipe is not, so the diagnostic can appear before the rows it refers to, which is Chapter 6.2 section 10's ordering problem in another guise.

Status 2 for "could not evaluate", per Chapter 5.6's three-way convention — distinguishing a regression that ran and failed from one that did not run.

One limitation to know: ARGV can also contain var=value assignments, which are not files. Chapter 8.2 covers those, and a script that must handle both checks each element rather than assuming it is a path.

9. BEGIN and END

Two timing facts from Chapter 5.2, restated because every command in this chapter depends on them.

FILENAME is empty in BEGIN, since no file has been opened:

awk 'BEGIN { print "BEGIN: FILENAME=[" FILENAME "] NR=" NR " FNR=" FNR }' logs/alu.log

BEGIN: FILENAME=[] NR=0 FNR=0

Which is exactly why section 8 uses ARGV in BEGIN and not FILENAME.

In END, NR is the grand total while FNR and FILENAME hold the last file's values:

awk 'END { print "END: NR=" NR " FNR=" FNR " FILENAME=" FILENAME }' logs/alu.log logs/uart.log

END: NR=5 FNR=2 FILENAME=logs/uart.log

NR=5 is the total across both. FNR=2 and logs/uart.log are leftovers from the last file, not summaries — so END is the right place to report a job total and the wrong place to report anything per file unless you accumulated it yourself.

10. Common Mistakes

Using NR where FNR was meant in a file:line: label. NR points nowhere in any file.

Expecting FILENAME in BEGIN. It is empty — section 9.

Expecting FNR in END to mean something. It is the last file's count.

A per-file summary driven by records. It omits every empty file — section 7.

Counting the rows of such a summary as a count of logs. They differ exactly when something went badly wrong.

Recomputing a label from FILENAME on every record. Derive it once in the FNR == 1 rule.

Iterating for (f in count) for the report. No defined order; walk ARGV by position instead.

Starting an ARGV loop at 0. ARGV[0] is the program name, not a file.

Assuming every ARGV element is a filename. It can be a var=value assignment — Chapter 8.2.

11. Exercises

Exercise 1 — Predict the output

awk 'FNR == 1 { print FILENAME, "starts at NR =", NR }' logs/alu.log logs/uart.log

logs/alu.log starts at NR = 1
logs/uart.log starts at NR = 4

logs/alu.log has three records, so the second file's first record is the fourth overall.

Exercise 2 — Label matches for an editor

Report every error or fatal record in the compiler's file:line: message format.

awk '/UVM_ERROR|UVM_FATAL/ { print FILENAME ":" FNR ": " $0 }' logs/alu.log logs/uart.log logs/ddr.log

logs/alu.log:2: UVM_ERROR mismatch
logs/ddr.log:1: UVM_FATAL deadlock

FNR is the line number inside its own file, which is what makes the location usable.

Exercise 3 — Derive a test name

Turn each log's path into a bare test name.

awk 'FNR == 1 { n = FILENAME; sub(/.*\//, "", n); sub(/\.log$/, "", n); print n }' logs/alu.log logs/uart.log logs/ddr.log

alu
uart
ddr

Exercise 4 — Debug a summary with a missing row

A nightly job summarises three logs and prints two rows.

awk '{ n[FILENAME]++ } END { for (f in n) printf "%-16s %d\n", f, n[f] | "sort" }' logs/alu.log logs/spi.log logs/uart.log

logs/alu.log     3
logs/uart.log    2

logs/spi.log is empty. Every rule is record-driven, so no rule ever ran for it and no key was created.

awk 'BEGIN { for (i = 1; i < ARGC; i++) count[ARGV[i]] = 0 } { count[FILENAME]++ } END { for (i = 1; i < ARGC; i++) printf "%-16s %d%s\n", ARGV[i], count[ARGV[i]], (count[ARGV[i]] == 0 ? "   <- EMPTY" : "") }' logs/alu.log logs/spi.log logs/uart.log

logs/alu.log     3
logs/spi.log     0   <- EMPTY
logs/uart.log    2

Seeding from ARGV in BEGIN creates every key before any file is read, so an empty file still has a row to report.

Exercise 5 — Per-file counts with an array

Report records and error records per log, in sorted order.

awk '{ n[FILENAME]++; if (/UVM_ERROR|UVM_FATAL/) bad[FILENAME]++ } END { for (f in n) printf "%-16s %d record(s), %d error(s)\n", f, n[f], (f in bad ? bad[f] : 0) | "sort" }' logs/alu.log logs/uart.log logs/ddr.log

logs/alu.log     3 record(s), 1 error(s)
logs/ddr.log     1 record(s), 1 error(s)
logs/uart.log    2 record(s), 0 error(s)

(f in bad ? bad[f] : 0) rather than bad[f] + 0, so reading the error count does not create a phantom key — Chapter 6.1 section 5, and the exact bug Chapter 7.3's Exercise 6 produced.

Exercise 6 — Engineering application

Produce a regression index over a directory of logs: one row per log in command-line order, with its record count, error count and a status — and a non-zero exit if any log was empty.

awk 'BEGIN { for (i = 1; i < ARGC; i++) { n[ARGV[i]] = 0; e[ARGV[i]] = 0 } } { n[FILENAME]++; if (/UVM_ERROR|UVM_FATAL/) e[FILENAME]++ } END { for (i = 1; i < ARGC; i++) { f = ARGV[i]; t = f; sub(/.*\//, "", t); sub(/\.log$/, "", t); st = (n[f] == 0 ? "NO OUTPUT" : (e[f] > 0 ? "FAIL" : "PASS")); printf "%-8s %3d rec %3d err  %s\n", t, n[f], e[f], st; if (n[f] == 0) empty++ } printf "(%d log(s); threshold: any UVM_ERROR or UVM_FATAL is a FAIL)\n", ARGC - 1; if (empty) { printf "ERROR: %d log(s) produced no output\n", empty > "/dev/stderr"; exit 2 } }' logs/alu.log logs/spi.log logs/uart.log logs/ddr.log 2>/dev/null
echo "status: $?"

alu        3 rec   1 err  FAIL
spi        0 rec   0 err  NO OUTPUT
uart       2 rec   0 err  PASS
ddr        1 rec   1 err  FAIL
(4 log(s); threshold: any UVM_ERROR or UVM_FATAL is a FAIL)
status: 2

And the stderr stream on its own:

awk 'BEGIN { for (i = 1; i < ARGC; i++) { n[ARGV[i]] = 0; e[ARGV[i]] = 0 } } { n[FILENAME]++; if (/UVM_ERROR|UVM_FATAL/) e[FILENAME]++ } END { for (i = 1; i < ARGC; i++) { f = ARGV[i]; t = f; sub(/.*\//, "", t); sub(/\.log$/, "", t); st = (n[f] == 0 ? "NO OUTPUT" : (e[f] > 0 ? "FAIL" : "PASS")); printf "%-8s %3d rec %3d err  %s\n", t, n[f], e[f], st; if (n[f] == 0) empty++ } printf "(%d log(s); threshold: any UVM_ERROR or UVM_FATAL is a FAIL)\n", ARGC - 1; if (empty) { printf "ERROR: %d log(s) produced no output\n", empty > "/dev/stderr"; exit 2 } }' logs/alu.log logs/spi.log logs/uart.log logs/ddr.log 2>&1 >/dev/null

ERROR: 1 log(s) produced no output

Four rows for four logs, in the order given, with NO OUTPUT as a third status distinct from PASS and FAIL — which is the whole point, since spi neither passed nor failed and reporting it as either would be a lie.

The rule is printed so the FAIL labels are checkable, and the exit status is 2 rather than 1, distinguishing "the regression could not be fully evaluated" from "the regression ran and failed".

The engineering caveat: 1 err counts matching records, not defects — one root cause can emit many, which Chapter 9.3 treats properly. And this command trusts that the four logs named are the four expected; a real index is driven by the list of tests the regression was told to run, so that a log file missing altogether is caught as well as one that is empty.

Exercise 7 — Challenge: prove the ARGV form sees what the record form misses

Write a single command that reports both counts side by side, so the gap is visible rather than argued.

awk 'BEGIN { for (i = 1; i < ARGC; i++) known[ARGV[i]] = 0 } { seen[FILENAME]++ } END { nk = 0; ns = 0; for (i = 1; i < ARGC; i++) { f = ARGV[i]; nk++; if (f in seen) ns++; else printf "  not seen in any record: %s\n", f } printf "files on the command line: %d\nfiles that yielded a record: %d\ndifference: %d\n", nk, ns, nk - ns }' logs/alu.log logs/spi.log logs/uart.log

  not seen in any record: logs/spi.log
files on the command line: 3
files that yielded a record: 2
difference: 1

Two independently derived numbers — one from ARGV, one from the records — and their disagreement names the file responsible.

This is the technique Chapters 5.1, 5.3 and 5.5 all arrived at from different directions: a count derived one way, checked against a count derived another way, turns a silent omission into a stated number. Here the two sources are the command line and the data, which is as independent as it gets inside one program.

Note the membership test is f in seen, not seen[f] > 0 — the latter would create a key for every missing file and make ns wrong on a second pass through the same loop.

12. Summary

• FNR is position within a file, NR within the job, FILENAME is which file.
• Label matches as FILENAME ":" FNR — the compiler format editors can parse. NR points nowhere.
• Derive a readable name once per file in the FNR == 1 rule, not per record.
• FILENAME != prev detects a file boundary, and needs a prev != "" guard plus an END flush.
• An array keyed on FILENAME is clearer and needs neither.
• Both forms omit a file with no records, because every rule is record-driven.
• An empty log means a test did not run — the most urgent row, and the one a record-driven summary drops.
• ARGV[1] to ARGV[ARGC-1] are the input files, available in BEGIN before anything is read.
• Seed counters from ARGV and report by walking it — every file gets a row, and the order is defined without sort.
• FILENAME is empty in BEGIN; in END, FNR and FILENAME are the last file's leftovers.

You can now report on a set of logs rather than on a stream of records. Chapter 7.5 — Processing Multiple Input Files goes further: running one program over a whole regression directory, and the two-file FNR == NR idiom for reading a key file before a data file — including what that idiom does when the key file is empty, which is worse than anything in this chapter.