Built-In Variables

Part II · Module 5 · Chapter 5.2 · Built-In Variables

1. Why This Matters

Chapter 5.1 ended on a question. Variables in awk are created by assignment, have no declared type, and can be written at any time. NF is a variable. So — what happens if you assign to it?

That question is the reason this chapter exists in the form it does. Most references list the built-ins with a sentence each, which tells you what they mean and leaves three more useful things unsaid:

• Who sets it — awk, or you, or either.
• Exactly when it changes, which matters because a rule can run before or after the update.
• Whether you may assign to it, and what happens if you do.

This chapter answers all three for every built-in you will actually use, with output rather than assertion. It is the most interview-relevant chapter in the module, and the questions asked are almost always about timing: when is NR updated? what is FILENAME inside BEGIN?

2. The Map

The built-ins worth knowing, grouped by what they are for:

The split is the useful part. The first four are awk reporting on the input — awk writes them, you read them. The next six are you configuring awk — you write them, awk reads them.

That is the normal direction of traffic. Section 9 is about going against it.

There are two more, RSTART and RLENGTH, which only become meaningful after a match() call. They belong with match() in Module 8 and are not used here.

3. NR — the Record Number

NR is the count of records awk has read, and it is incremented before your rules run for that record. So inside a rule, NR is the number of the record you are looking at — the first record sees NR as 1, not 0.

alu PASS 120
fifo FAIL 95

awk '{ print NR, $1 }' alu.log

1 alu
2 fifo

Chapter 2.1 introduced NR as the record counter; what is worth adding now is that it is cumulative across files and does not reset. That is the entire difference between it and FNR, and section 5 shows the divergence.

NR == 1 is therefore the idiom for "the first record of the whole input" — which is what Chapter 5.1 used to seed a worst-case tracker.

4. NF — the Field Count

NF holds the number of fields in the current record, recomputed for every record. On ragged input it changes line by line:

DFF 120
LATCH 95 extra
MUX

awk '{ print NR, NF, "[" $0 "]" }' ragged.txt

1 2 [DFF 120]
2 3 [LATCH 95 extra]
3 1 [MUX]

Two, three, one. Chapter 3.2 used $NF to reach the last field precisely because this number is not a constant.

The engineering use is as a shape check. A report whose rows should all have five fields can be audited in one command:

awk 'NF != 2 { print "line", NR, "has", NF, "fields, expected 2" }' ragged.txt

line 2 has 3 fields, expected 2
line 3 has 1 fields, expected 2

That is a more valuable habit than it looks. A tool whose output format changed between versions will silently shift every $3 in your scripts, and an NF check is how you find out on the first run rather than the fiftieth.

5. FNR and FILENAME — Reading Several Files

With one file, NR and FNR are identical and FILENAME is redundant. With two, they diverge — and here is awk reporting on itself:

uart PASS 88
spi FAIL 77
ddr PASS 150

awk '{ print NR, FNR, FILENAME, $0 }' alu.log uart.log

1 1 alu.log alu PASS 120
2 2 alu.log fifo FAIL 95
3 1 uart.log uart PASS 88
4 2 uart.log spi FAIL 77
5 3 uart.log ddr PASS 150

Read row three. NR continues to 3 because it counts every record awk has read. FNR resets to 1 because a new file started. FILENAME changed with it.

So: NR for position in the whole job, FNR for position in this file.

FILENAME has one timing wrinkle worth knowing because it is a common interview question:

awk 'BEGIN { print "begin filename=[" FILENAME "]" } NR==1 { print "rule filename=[" FILENAME "]" }' alu.log

begin filename=[]
rule filename=[alu.log]

Empty in BEGIN. BEGIN runs before awk opens any input, so there is no current file to name. This is consistent with Chapter 2.4's phase model — BEGIN happens before reading starts — and it catches people who try to print a header naming the file.

6. FNR == 1 and the Empty File

FNR == 1 is the standard way to do something once per file. It is in every awk reference. And it has a gap that none of them mention.

Three files, the middle one empty:

: > empty.log
awk 'FNR == 1 { print "first record of", FILENAME }' alu.log empty.log uart.log

first record of alu.log
first record of uart.log

empty.log is absent. Not reported as empty — simply not mentioned.

The reason is mechanical and, once stated, obvious. FNR == 1 is a pattern, and a pattern is only ever tested against a record. An empty file contains no records, so awk opens it, finds nothing, and moves on. No rule runs for it at all, including this one.

7. FS and RS — Where to Set Them

Module 3 covered both in depth. The part that belongs in a chapter about built-ins is when the assignment takes effect.

FS must be set before the record it should apply to is split. In practice that means BEGIN, or the command line:

a:b:c

awk 'BEGIN { FS=":" } { print $2 }' colon.txt

b

awk -F: '{ print $2 }' colon.txt

b

Identical. Chapter 3.3 established that -F is the same assignment written more conveniently, and recommended it for one-liners.

The failure mode Chapter 3.3 demonstrated is setting FS in a rule, where it applies to the next record and leaves the first one split by whitespace. If you need a different separator partway through the input, that is FS's weakness and a reason to reach for the technique in Chapter 3.5 instead.

RS follows the same timing rule. Its one special value is worth repeating because it is unusually useful:

one
two

three
four

awk 'BEGIN { RS="" } { print "record", NR, "has", NF, "fields" }' para.txt

record 1 has 2 fields
record 2 has 2 fields

RS="" is paragraph mode, from Chapter 3.4 — a blank line separates records, and newlines inside a record become field separators. Two paragraphs, two records, two fields each.

8. OFS and ORS — the Output Side

OFS goes between the items of a comma-separated print:

awk 'BEGIN { OFS="|" } { print $1, $2 }' alu.log

alu|PASS
fifo|FAIL

That works immediately — the comma in the print list is what OFS fills in, so no rebuild is involved.

The case that needs care is a bare print, which emits $0 as a single string and so never consults OFS at all. Chapter 3.6 covered this: to apply a new OFS to a whole record, you have to make awk rebuild $0 by assigning to a field.

awk 'BEGIN { OFS="|" } { $1=$1; print }' alu.log

alu|PASS|120
fifo|FAIL|95

$1=$1 changes nothing about the value and everything about the representation: assigning to any field makes awk reassemble $0 from the fields, joined with the current OFS.

ORS goes after each record, and defaults to a newline. Changing it is how you put output on one line:

awk 'BEGIN { ORS=" " } { print $1 } END { printf "\n" }' alu.log uart.log | cat -e

alu fifo uart spi ddr $

The END rule supplies the final newline, because ORS has replaced it with a space. Without it the shell prompt would land on the same line as the output.

cat -e is used here to expose something the plain output hides: there is a space before that $. ORS is appended after every record including the last, so the line ends with a separator. For a space that is cosmetic. Exercise 7 shows the case where it is not.

9. Which Built-Ins You May Assign

Now the question section 1 opened with. The built-ins are ordinary variables, and awk will let you assign to any of them. The outcomes differ sharply:

The two to understand are NF and NR.

Assigning NR really does work, and really is a bad idea:

awk 'NR==1 { NR=100 } { print NR, $1 }' uart.log

100 uart
101 spi
102 ddr

awk accepted the new value and continued counting from it. There is no error and no warning. Any later rule using NR — a seeding pattern, a line-number report, a modulo sample — is now reporting a fiction. Chapter 5.1 used NR == 1 to seed a tracker; in a program that has reassigned NR, that seed may never fire.

The rule is simple: read NR, FNR, NF and FILENAME; write FS, RS, OFS and ORS. Going the other way is legal, which is precisely why it has to be a habit rather than a thing the language prevents.

10. NF = n Truncates — and Pads

Assigning to NF is the one that damages data, and it does two different things depending on the record.

awk '{ NF=2; print }' ragged.txt | cat -e

DFF 120$
LATCH 95$
MUX $

cat -e marks the end of each line with $, which is necessary here because the third line's real content is invisible otherwise.

Three distinct outcomes from one assignment:

• DFF 120 had exactly 2 fields — unchanged.
• LATCH 95 extra had 3 — truncated. extra is gone from $0.
• MUX had 1 — padded. It is now MUX with a trailing space, a record with an empty second field.

Setting NF rebuilds $0 to have exactly that many fields, inventing empty ones if the record was short. Truncation loses data; padding invents a field that was never in the input and adds trailing whitespace that a later diff will flag.

It has a legitimate use — dropping a known trailing column — but NF=2 on data you have not shape-checked is a quiet way to corrupt a report. The NF != 2 audit from section 4 is the thing to run first.

11. The Built-Ins in END

END runs after the last record, and the built-ins still hold values. Which values is worth checking rather than assuming:

awk 'END { print "NR =", NR, "FNR =", FNR, "FILENAME =", FILENAME }' alu.log uart.log

NR = 5 FNR = 3 FILENAME = uart.log

NR is the total across both files — 5 records, which is why END { print NR } is the standard record count. FNR and FILENAME hold the last file's values, not a summary of all of them; they are leftovers rather than totals.

On empty input, NR is 0:

awk 'END { print "NR =", NR }' empty.log

NR = 0

Which is the guard Chapter 5.1's final exercise used, and the only reliable way to tell "no records" from "nothing matched".

12. CONVFMT and OFMT

Two formatting built-ins that explain a surprise you will eventually hit. Their defaults:

awk 'BEGIN { print "CONVFMT=[" CONVFMT "] OFMT=[" OFMT "]" }'

CONVFMT=[%.6g] OFMT=[%.6g]

%.6g means six significant digits. OFMT controls how print renders a number; CONVFMT controls how a number becomes a string when you concatenate it.

Six digits is enough that most engineering numbers survive, and not always enough:

awk 'BEGIN { x = 1234567.89; print x; print "concatenated: " x }'

1.23457e+06
concatenated: 1.23457e+06

The value is intact internally; the display lost precision and switched to exponent form. For a cell count or a large area total that is a real misreport.

awk 'BEGIN { OFMT="%.2f"; CONVFMT="%.2f"; x = 1234567.89; print x; print "concatenated: " x }'

1234567.89
concatenated: 1234567.89

Both now readable. In practice most people reach for printf with an explicit format instead, which is Chapter 7.2's subject and is clearer because the format sits next to the value it applies to. Knowing these two variables exist is still worth it — when a number prints in scientific notation you did not ask for, this is why.

Integers are unaffected regardless, since %g renders them exactly:

awk 'BEGIN { print 1234567 }'

1234567

13. Common Mistakes

Expecting FILENAME in BEGIN. It is empty. BEGIN runs before any file is opened.

Using NR where FNR was meant across multiple files. NR does not reset. A per-file header keyed on NR == 1 fires once for the whole job.

Expecting FNR == 1 to run for an empty file. It does not — section 6. No records, no rules.

Setting OFS and printing bare. print emits $0 and ignores OFS. Assign to a field to force a rebuild.

Setting FS in a rule instead of BEGIN. It applies from the next record, so record 1 is split by the old separator.

Assigning NF. Truncates long records and pads short ones with an empty field and a trailing space.

Assigning NR. Legal, silent, and it falsifies every later use of the counter.

Relying on $0 in END. Unspecified by POSIX. Save the record yourself.

Trusting six significant digits. OFMT defaults to %.6g, so a large value prints in exponent form.

14. Exercises

Exercise 1 — Predict the output

awk 'BEGIN { print NR, NF, "[" FILENAME "]" }' alu.log

0 0 []

All three are unset in BEGIN. No record has been read, so NR and NF are 0, and no file has been opened, so FILENAME is empty — which is Chapter 5.1's unset-variable rule applying to built-ins exactly as it does to your own.

Exercise 2 — NR against FNR

Print only the first record of each file, using the two files below. Then explain why NR == 1 would be wrong.

awk 'FNR == 1 { print FILENAME ":", $0 }' alu.log uart.log

alu.log: alu PASS 120
uart.log: uart PASS 88

awk 'NR == 1 { print FILENAME ":", $0 }' alu.log uart.log

alu.log: alu PASS 120

NR is cumulative, so it equals 1 exactly once in the whole job. FNR resets per file, which is what "first record of each file" actually asks for.

Exercise 3 — Shape-check a report

Report every record in the ragged file that does not have exactly two fields, naming the line number and the count found.

awk 'NF != 2 { print "line " NR ": " NF " fields — [" $0 "]" }' ragged.txt

line 2: 3 fields — [LATCH 95 extra]
line 3: 1 fields — [MUX]

This is the audit worth running before any script that depends on a fixed column position.

Exercise 4 — Debug a per-file counter

An engineer wants a per-file record count and gets the wrong numbers. Their command and its output:

awk 'FNR == 1 { print FILENAME, n+0; n=0 } { n++ } END { print FILENAME, n+0 }' alu.log uart.log

alu.log 0
uart.log 2
uart.log 3

Two problems. Find both.

First, the per-file total is printed on the first record of the next file, so each label names the wrong file — uart.log 2 is actually alu.log's count. Second, alu.log 0 is a count printed before anything was read.

The fix is to key off FNR in END and track per-file totals as awk goes, reporting when the file changes rather than when the next one starts:

awk 'FILENAME != prev { if (prev != "") print prev, n; prev = FILENAME; n = 0 } { n++ } END { if (prev != "") print prev, n }' alu.log uart.log

alu.log 2
uart.log 3

Correct — 2 records in alu.log, 3 in uart.log. The prev != "" guard stops the first file from triggering a report before there is anything to report, and the END rule flushes the last file, which nothing else would.

Note that this still cannot report an empty file, for section 6's reason: FILENAME != prev is a pattern and no record ever arrives to test it.

Exercise 5 — NF assignment

Predict all three lines before running. Use cat -e so nothing is invisible.

awk '{ NF=2; print }' ragged.txt | cat -e

DFF 120$
LATCH 95$
MUX $

Unchanged, truncated, padded. The third is the one people miss — MUX now has an empty second field and a trailing space.

Exercise 6 — Engineering application

From the two synthetic logs, report the total runtime per file and the overall total, labelling each correctly.

awk 'FILENAME != prev { if (prev != "") print prev, t; prev = FILENAME; t = 0 } { t += $3; grand += $3 } END { if (prev != "") print prev, t; print "TOTAL", grand + 0, "over", NR, "tests" }' alu.log uart.log

alu.log 215
uart.log 315
TOTAL 530 over 5 tests

215 for alu.log (120 + 95), 315 for uart.log (88 + 77 + 150), 530 overall, across 5 records.

The engineering caveat Chapter 1.2 established applies to the total: 530 seconds is the sum of recorded runtimes, not the wall-clock time of the regression. Tests running in parallel overlap, so the sum exceeds the elapsed time — often by a large factor on a farm. And a test that crashed before writing its row contributes nothing, so this total is biased low in exactly the situation where somebody is investigating why the regression took so long.

Printing over 5 tests is what makes that readable. A bare 530 invites the reader to assume it means something it does not.

Exercise 7 — Challenge: ORS and a one-line summary

Produce a single line listing every failing test name, comma-separated, with no trailing comma — and a newline at the end.

awk '$2 == "FAIL" { printf "%s%s", sep, $1; sep = ", " } END { printf "\n" }' alu.log uart.log | cat -e

fifo, spi$

Clean — the line ends immediately after spi. The separator trick is worth keeping: sep is unset on the first match so nothing is printed before it, and is set to ", " afterwards, which puts a separator between items rather than after each. That is Chapter 5.1's unset-variable rule doing something genuinely useful.

The obvious ORS approach gets close and cannot finish the job:

awk 'BEGIN { ORS=", " } $2 == "FAIL" { print $1 } END { printf "\n" }' alu.log uart.log | cat -e

fifo, spi, $

Compare the two endings: spi$ against spi, $. The ORS version leaves a comma and a space before the newline, because ORS is appended after every record and awk has no notion of a last one. There is no value of ORS that fixes this — the trailing separator is inherent to what ORS is.

So the rule is: when output needs separators between items, use printf with a primed variable. ORS is for when every record genuinely should end the same way.

This is not a cosmetic difference once the output is consumed by something. A trailing comma makes a CSV row parse as having an extra empty field, and a comma-separated list fed to a tool as an argument usually fails outright.

15. Summary

• Four built-ins are awk reporting to you — NR, FNR, NF, FILENAME. Read them.
• Six are you configuring awk — FS, RS, OFS, ORS, CONVFMT, OFMT. Write them.
• NR is cumulative; FNR resets per file. The divergence trace in section 5 is awk's own output.
• FILENAME is empty in BEGIN, because no file has been opened yet.
• FNR == 1 does not run for an empty file. No records means no rules — so a per-file header or reset silently skips it, and a carried-over accumulator becomes a wrong number.
• OFS applies to print a, b immediately, and never to a bare print, which emits $0. Assign to a field to force a rebuild.
• NF = n truncates long records and pads short ones with an empty field and a trailing space.
• Assigning NR is legal and silent, and falsifies every later use of the counter.
• In END, NR is the grand total while FNR and FILENAME are the last file's leftovers.
• $0 in END is unspecified by POSIX. Save the record yourself if you need it.
• OFMT defaults to %.6g, so a large number can print in exponent form.

You now know what awk is telling you about the input, and what you are allowed to tell awk.

Chapter 5.1 built counters and totals with += and ++ without examining either. Chapter 5.3 — Arithmetic and String Expressions does that properly: operator precedence, the difference between i++ and ++i, what division by zero does, and the two ways a malformed measurement in a data column becomes a confidently wrong total.