Records

Part I · Module 2 · Chapter 2.1 · Records

1. Why This Matters

Module 1 was about judgement — what awk is good at, and when to reach for grep, sed or Python instead. You can make all of those decisions correctly and still be unable to explain why a command gave the answer it did.

This module is about the machinery. By the end of it you will be able to take any awk program in this curriculum and say, line by line, what awk does with it and in what order. That turns awk from a set of recipes into something you can reason about, which is the difference between copying a one-liner and debugging one.

We start with the unit awk works in. One sentence, which the chapter then earns:

awk reads its input one record at a time, and holds only the current one.

Both halves matter. "One at a time" explains the order things happen in. "Only the current one" explains why awk can read a file far larger than your machine's memory, and why anything you want to remember has to be put in a variable.

2. What a Record Is

A record is one chunk of input that awk processes as a unit. By default, a record is one line.

Here is a small file to work with:

alice 85
bob 72
carol 91

Three lines, so three records. awk processes alice 85 completely — tests every rule against it, runs whatever matches — and only then moves on to bob 72.

What marks the end of a record is the newline character at the end of each line. That is awk's default and it is held in a variable called RS, the record separator. You can change RS to make awk treat something else as the boundary, which is how a paragraph-shaped report becomes one record; that is Chapter 3.4's subject. Everything in this module uses the default.

3. One at a Time, Not All at Once

This is the part beginners most often get wrong, usually by assuming awk reads the whole file first and then starts working.

It does not. awk reads a record, runs your rules on it, discards it, and reads the next one.

You do not have to take that on trust. Two commands make it observable.

The first uses two rules at once. Do not worry about the syntax yet — Chapter 2.3 is about rules — just read the output order:

awk 'NR == 1 { print "-> acted on record 1 already" } { print "   read:", $1 }' scores.txt

-> acted on record 1 already
   read: alice
   read: bob
   read: carol

The -> acted on record 1 already line appears before read: carol. If awk had read the whole file before doing anything, that message could not have come out first. It acted on record 1 while records 2 and 3 were still unread.

The second is sharper. exit tells awk to stop immediately — Chapter 5.6 covers it properly, and all you need here is that it stops reading:

awk 'NR == 2 { print "stopping at", NR; exit }' scores.txt

stopping at 2

awk stopped at the second record. The third was never read. A tool that loaded the file up front could not do that.

4. NR: The Record Number

awk keeps a running count of the records it has read, in a variable called NR — short for Number of Records. You do not create it or update it; awk does.

awk '{ print NR, $0 }' scores.txt

1 alice 85
2 bob 72
3 carol 91

Three things to notice.

NR starts at 1, not 0. The first record is record 1. This trips up programmers used to zero-based indexing:

awk 'NR == 1 { print "first record is NR=" NR }' scores.txt

first record is NR=1

NR has no dollar sign. NR is a plain variable holding a number. $1 means "field 1". Writing $NR would mean "the field numbered by NR", which on record 2 is field 2 — almost never what you want.

NR is the current value, not the total — while records are being read. It only equals the total number of records once the input is exhausted, which is what makes the next section work.

5. Counting Records

Because NR is updated for every record, the last value it holds is the number of records in the file. To read it after the input runs out, put the print in an END block:

awk 'END { print NR }' scores.txt

3

END means "after the last record has been read". Chapter 1.1 used it informally and Chapter 2.4 covers it properly; here it is simply the only honest way to report a total.

Notice there is no ordinary rule at all in that program — no braces that run per record. awk still read all three records, still updated NR three times, and then ran the END block once. Reading the input is awk's job, not something your rules have to ask for.

6. What Counts as a Record

Three cases where "a record is a line" is true but not the whole story. All three cause real bugs.

A blank line is a record. It is an empty record, but it counts:

awk '{ print NR ": [" $0 "]" }' blanks.txt

Given this file:

alice 85

bob 72

1: [alice 85]
2: []
3: [bob 72]

Three records for two rows of data. The square brackets are there to make the empty one visible. So:

awk 'END { print NR }' blanks.txt

3

If you report that as "two people scored" you are wrong by one, and nothing warned you. Reports separated by blank lines are extremely common, which is why this matters.

A final line with no newline is still a record. Text files conventionally end with a newline, but not all do — a truncated file, or one written by a program that died, often does not:

A fenced example cannot show you a missing final newline, so build the file with printf, which writes exactly the bytes you give it and nothing more:

printf 'alice 85\nbob 72' > nonewline.txt
awk '{ print NR, $0 }' nonewline.txt

1 alice 85
2 bob 72

Both records, as you would hope. awk treats the end of input as ending the record. This is worth knowing precisely because it is the behaviour you want and you should not waste time suspecting it.

Whitespace inside a line does not split it. A record is ended by RS, which is a newline — spaces and tabs have nothing to do with it:

awk 'END { print "records:", NR }' spaces.txt

a   b     c

records: 1

One record, despite three chunks of text. Whitespace splits a record into fields, which is a completely separate mechanism and the subject of Chapter 2.2. Keeping those two apart is the single most useful distinction in this module.

7. An Engineering Example

A synthetic simulation log. Four events, each with a time, a severity and a short message:

100 INFO reset_started
150 WARN timeout_near
200 ERROR data_mismatch
250 INFO simulation_finished

Number the events and show their severity:

awk '{ print NR, $2 }' sim.log

1 INFO
2 WARN
3 ERROR
4 INFO

How many events the log contains:

awk 'END { print NR }' sim.log

4

Four records. Now the engineering caveat, because this is where a number starts being used as evidence.

NR tells you how many records awk read. On this file that happens to equal the number of logged events, because the file was built so that one event is one line. A real simulator log does not promise that: a single message can wrap across lines, a banner occupies lines that are not events at all, and blank lines separate sections.

So NR is an honest count of records and only a count of events if you have checked that one event is one record. Chapter 1.2 made the same point about error counts not being failure counts; this is the same discipline applied to the most basic number awk can give you.

8. The Empty File

Worth doing deliberately, because it is the input your script will eventually meet and the one nobody tests.

The : > empty.txt below creates a file of zero bytes, which is the whole point of the test:

: > empty.txt
awk 'END { print "records:", NR }' empty.txt

records: 0

Three things happened. awk opened the file, found no records, and still ran the END block. NR was 0 because no record was ever read. And awk exited with status 0, because nothing went wrong — reading an empty file is not an error.

9. Common Mistakes

Assuming awk reads the whole file first. Section 3. It reads one record at a time, which is why exit can stop early and why the interleaved output appears in the order it does.

Expecting NR to start at 0. It starts at 1. A loop-counter habit from other languages is the usual cause.

Writing $NR when you meant NR. NR is the record number; $NR is the field numbered by it. On record 2, $NR is $2 — which looks like it works on exactly one record and then quietly does not.

Reading NR inside a per-record rule and calling it a total. Inside the loop NR is the current record's number. Only after the input is exhausted — in END — does it hold the total.

Forgetting that a blank line counts. Section 6. This is the most common off-by-a-few error in report parsing.

Confusing record separation with field separation. A newline ends a record; whitespace splits a record into fields. Different mechanisms, different variables, different chapters.

10. Exercises

Create the files as shown and compare your output.

Exercise 1 — Predict the output

Before running it, predict what this prints, then check.

alice 85
bob 72
carol 91

awk '{ print NR }' scores.txt

1
2
3

One line per record, each carrying that record's number. If you predicted a single 3, you were thinking of END { print NR } — the difference between a rule that runs per record and one that runs once at the end is exactly what Chapter 2.4 is about.

Exercise 2 — Modify a command

Print only the second record, whatever it contains.

awk 'NR == 2 { print }' scores.txt

bob 72

NR == 2 is a condition on the record number. Note == for comparison, not =, which would assign to NR and is a mistake awk will happily let you make.

Exercise 3 — Trace the execution

For the program below, write down what awk does at each step before running it: which records it reads, which rule fires on each, and in what order the output appears.

awk 'NR == 1 { print "A" } NR == 3 { print "B" } { print "C" NR }' scores.txt

A
C1
C2
B
C3

The trace. Record 1: the first rule matches so A prints; the second does not; the third has no pattern so it always runs, printing C1. Record 2: only the third rule runs, printing C2. Record 3: the first does not match; the second matches so B prints; the third runs, printing C3.

The thing to take from this is that more than one rule can fire on the same record, and they fire in the order written. If you expected awk to stop after the first match, Chapter 2.3 is where that is corrected properly.

Exercise 4 — Debug a wrong count

An engineer counts the entries in a report with awk 'END { print NR }' blanks.txt and gets 3, but there are only two people in the file. Explain the discrepancy and write a command that counts only the non-empty records.

alice 85

bob 72

awk 'NF > 0 { n++ } END { print n + 0 }' blanks.txt

2

The blank line is a record, so NR counts it. NF is the number of fields on the current record — Chapter 2.2's subject — and a blank line has zero, so NF > 0 selects the records that actually carry data. The + 0 makes the output a 0 rather than an empty line if nothing matched, which Chapter 1.6 explained.

Exercise 5 — Engineering application

Using the synthetic log, report how many records it holds and how many of them are ERROR records, as two numbers on one line.

100 INFO reset_started
150 WARN timeout_near
200 ERROR data_mismatch
250 INFO simulation_finished

awk '$2 == "ERROR" { e++ } END { print NR, e + 0 }' sim.log

4 1

Four records, one of which has ERROR in the severity column. Both numbers come out of the same single pass, which is the shape almost every awk summary takes.

And the caveat, which is the point of putting this exercise last: 4 is a count of records, and 1 is a count of records whose second field is ERROR. Neither is a count of bugs. One root cause commonly prints many error records, and a simulator that died before starting prints none at all — Chapter 1.2 covered both.

Exercise 6 — Challenge: the empty file

Write a command that prints the number of records, and additionally prints a warning when the file turned out to be empty. Test it against both scores.txt and empty.txt.

: > empty.txt
awk 'END { if (NR == 0) print "WARNING: no records read"; print "records:", NR }' empty.txt

WARNING: no records read
records: 0

awk 'END { if (NR == 0) print "WARNING: no records read"; print "records:", NR }' scores.txt

records: 3

The same program on both files, and the difference is a message rather than a different exit status. This is the smallest useful version of section 8's point: make the zero say that it is a measured zero, because otherwise it is indistinguishable from a file nobody produced.

11. Summary

• awk reads one record at a time and holds only the current one. The file may be any size; the program's view is $0 plus whatever you have accumulated.
• A record is one line by default, because the record separator RS is a newline. Changing RS is Chapter 3.4.
• Streaming is observable, not a claim: two rules produce interleaved output, and exit at record 2 means record 3 is never read.
• Anything you need at the end must be accumulated as you go, since awk is not keeping the file for you.
• NR is the record number, starts at 1, and carries no dollar sign. Inside the loop it is the current record; only after the input is exhausted does it hold the total.
• A blank line is a record. A final line with no newline is a record. Whitespace inside a line is not a record boundary — it splits fields, which is a different mechanism.
• An empty file gives NR of 0, still runs END, and exits 0. That is indistinguishable from a clean run unless you check for it.
• NR counts records, not events. It equals the number of log events only if you have checked that one event is one record.

You now know what awk is holding at any moment and how it counts what it has seen. What you do not yet know is how awk takes that record apart.

Section 6 kept deferring to it: whitespace does not end a record, it splits one. Chapter 2.2 — Fields is about that second mechanism — how $1, $2 and $NF come to exist, what the default splitting rule really does with runs of spaces and tabs, and why NF is the most useful number on a malformed line.