Fields

Part I · Module 2 · Chapter 2.2 · Fields

1. Why This Matters

Two mechanisms are at work when awk reads a file, and confusing them is the commonest source of puzzlement in this module.

The first was Chapter 2.1's subject: the record separator decides where one record ends and the next begins. By default that is a newline.

The second is this chapter's: once awk has a record, the field separator decides how that record is cut into pieces. By default that is whitespace.

They are independent. Changing one does not change the other. A record always has exactly one record-ending boundary and may have any number of fields, including zero.

A newline ends a record. Whitespace splits a record into fields.

Everything in this chapter is a consequence of the second half.

2. What a Field Is

A field is one of the pieces a record is split into. awk numbers them from 1 and gives you each by putting $ in front of its number.

Take the first record of a familiar file:

alice 85
bob 72
carol 91

On the record alice 85, every reference awk offers maps to a value like this:

Two of those rows are the ones people get wrong, and both have their own section below. NF carries no dollar sign because it is a count, not a field. And $3 is not an error — it is the empty string, which is section 8.

3. The Default Splitting Rule

By default, awk splits a record on whitespace, and the rule has three parts that are easiest to see all at once. This fixture is built to exercise all three:

The three records differ only in their whitespace, and whitespace is invisible in a code block — so build the file with printf, where \t is a tab and every space is explicit:

printf 'DFF    120\nAND\t\t45\n   MUX   18   \n' > padded.txt
awk '{ print NF, "[" $1 "]", "[" $2 "]" }' padded.txt

2 [DFF] [120]
2 [AND] [45]
2 [MUX] [18]

Two fields on every record, cleanly extracted in all three cases. The square brackets prove there is no stray whitespace clinging to the values. So:

A run of whitespace is one separator, not several. DFF 120 has four spaces between the values and still splits into exactly two fields. A tool that split on "each space" would have produced five. This is why a report padded out so its columns line up on screen still gives you the fields you wanted.

A tab is whitespace too. Line 2 is separated by tabs rather than spaces, and splits identically. Under the default rule, spaces and tabs are interchangeable.

Leading and trailing whitespace is discarded. Line 3 begins with three spaces, and $1 is MUX rather than an empty field:

awk 'NR == 3 { print "[" $1 "]" }' padded.txt

[MUX]

If leading whitespace created a field, $1 would have been empty and every column would be off by one. It does not, which is exactly the behaviour you want from an indented report.

4. $0 Is the Original Text

A natural assumption is that $0 is awk sticking the fields back together. It is not — $0 is the record exactly as it was read.

awk 'NR == 1 { print "[" $0 "]" }' padded.txt

[DFF    120]

All four original spaces are still there, even though NF is 2 and the fields themselves came out clean. awk split the record to give you $1 and $2; it did not alter the record to do so.

This matters when you print. print with no arguments, and print $0, both emit the original text with its original spacing — which is usually what you want when passing a record through. But print $1, $2 builds a new line from the fields, joined by a single space, which is why the output of section 3's command looked tidier than the input.

5. NF: How Many Fields

NF holds the number of fields on the current record. awk sets it for every record, and it changes as the records change:

awk '{ print NR, NF }' blanks.txt

Given a file with an empty line in the middle:

a 1

b 2

1 2
2 0
3 2

Record 2 is the blank line, and it has zero fields. That is the cleanest way to recognise a blank record, and Chapter 2.1's Exercise 4 used it for exactly that: NF > 0 selects records that carry data.

NF is also the number that tells you whether a record has the shape you assumed, which is what section 10 is built on.

6. Counting From the End: $NF and $(NF-1)

Because $ applies to whatever number follows it, and NF is a number, $NF means "the field numbered NF" — the last one.

A synthetic area report with three columns: an instance, a cell category, and an area value.

u_alu comb 1820
u_regfile seq 2410
u_io pad 640

awk '{ print $NF }' area.rpt

1820
2410
640

The value column, without having to know it is column 3. And arithmetic inside the reference gives you the one before it:

awk '{ print $(NF-1) }' area.rpt

comb
seq
pad

The parentheses are required. $NF-1 would be read as ($NF) - 1 — the last field minus one, which on this file would print 1819, 2409 and 639. A plausible set of numbers, and completely wrong.

7. Computed Field References

More generally, the thing after $ can be any expression that produces a number:

awk '{ print $(1+1) }' scores.txt

85
72
91

$(1+1) is $2. You would never write it that way deliberately, but the fact that it works is what makes $(NF-1) possible, and later it is what lets a loop walk across the fields of a record — Chapter 5.5's subject.

8. Asking for a Field That Is Not There

Chapter 1.6 showed this for one case; here is the rule.

awk '{ print NF, "[" $5 "]" }' scores.txt

2 []
2 []
2 []

Every record has two fields, so $5 does not exist. awk does not complain. An out-of-range field is the empty string, and the brackets show it is genuinely empty rather than a space.

In arithmetic the empty string counts as zero, which is where this turns from a curiosity into a bug: a total over a column that does not exist comes out as 0, with no error and a successful exit status. Chapter 1.2 and Chapter 1.5 both met this; Chapters 8.3 and 8.4 defend against it properly.

9. An Engineering Example

Two questions about the synthetic area report, each one command.

The instance and its area, dropping the category column:

awk '{ print $1, $3 }' area.rpt

u_alu 1820
u_regfile 2410
u_io 640

The sum of the area column:

awk '{ total += $3 } END { print total }' area.rpt

4870

4870 is the arithmetic sum of column 3. What it is not is the design's total area, for two reasons already established and one new one. Chapter 1.2 noted that a hierarchical report commonly lists a parent and its children, so summing every row double-counts; this fixture is three flat rows by construction. And Chapter 1.6 noted that cell counts are not areas — here the column genuinely is an area, in whatever units the tool used, which is a reminder to check what a column holds before summing it.

The new reason is the subject of the next section: the sum is only right if every record really has three fields.

10. The Ragged Record, and Where $NF Betrays You

This is the most important section in the chapter.

$NF is attractive because it means "the value column" without counting. But it does not mean that. It means "the last field" — and on a record that lost a column, the last field is a different column.

alice 85
bob
carol 91

awk '{ print NR, NF, "[" $2 "]" }' ragged.txt

1 2 [85]
2 1 []
3 2 [91]

Record 2 has one field, so $2 is empty — visible, and section 8 explains it.

Now the same record through $NF:

awk 'NR == 2 { print "NF=" NF, "$NF=[" $NF "]", "$1=[" $1 "]" }' ragged.txt

NF=1 $NF=[bob] $1=[bob]

On this record $NF is bob. Not empty — a name, delivered where a number was expected.

The defence is NF. Check the record has the shape you assumed before reading from it:

awk 'NF == 2 { print $1, $2 }' ragged.txt

alice 85
carol 91

Two records out of three, and the malformed one was skipped rather than misread. That is better than silence but still not ideal, because it drops a row without telling you. The complete version counts what it skipped:

awk 'NF == 2 { print $1, $2 } NF != 2 { bad++ } END { if (bad) print "skipped", bad, "malformed record(s)" }' ragged.txt

alice 85
carol 91
skipped 1 malformed record(s)

Now nothing is silently lost. That program also happens to use two ordinary rules on the same input, which is the subject of the next chapter.

11. Common Mistakes

Confusing record separation with field separation. A newline ends a record; whitespace splits one. Section 1.

Writing $NF-1 for the second-to-last field. That is the last field minus one. You need $(NF-1). Section 6, and on the area report the wrong version produces believable numbers.

Using $NF as a synonym for a particular column. It is the last field, not column 3. Section 10.

Expecting NF to have a dollar sign. NF is a count; $NF is a value. Chapter 1.6 flagged this and it remains the most common slip.

Assuming leading whitespace creates an empty first field. Under the default rule it does not. Section 3 — and this assumption reverses the moment you set FS to a single character, which is why Chapter 3.3 exists.

Assuming $0 is the fields joined back together. It is the original text, spacing included. Section 4.

Summing a column without checking NF. A record short of a field contributes 0 to the total silently. Sections 8 and 10.

12. Exercises

Exercise 1 — Predict the output

Before running it, write down what this prints.

u_alu comb 1820
u_regfile seq 2410
u_io pad 640

awk '{ print NF }' area.rpt

3
3
3

Three fields on every record, printed once per record. If you predicted a single 3, you were thinking of a once-at-the-end rule again — the distinction Chapter 2.4 settles.

Exercise 2 — Modify a command

Print the category column — the middle one — using a reference counted from the end rather than from the start.

awk '{ print $(NF-1) }' area.rpt

comb
seq
pad

$(NF-1) with the parentheses. On this file $2 would give the same answer; the two stop agreeing the moment a record has a different number of fields, which is section 10's point.

Exercise 3 — Trace the splitting

For each record of padded.txt, write down NF, $1 and $2 before running anything. The three records are deliberately spaced differently.

printf 'DFF    120\nAND\t\t45\n   MUX   18   \n' > padded.txt
awk '{ print NR, NF, "[" $1 "]", "[" $2 "]" }' padded.txt

1 2 [DFF] [120]
2 2 [AND] [45]
3 2 [MUX] [18]

All three identical in structure, despite a four-space run, a tab, and leading plus trailing spaces. A run of whitespace is one separator, tabs are whitespace, and surrounding whitespace is discarded. If you predicted an empty $1 for record 3, that is the assumption section 3 corrects.

Exercise 4 — Debug a wrong total

An engineer sums the area column of ragged_area.rpt with awk '{ total += $3 } END { print total }' and gets a number they think is too low. Find out why, and produce a total that reports the problem instead of hiding it.

u_alu comb 1820
u_regfile seq
u_io pad 640

awk '{ total += $3 } END { print total }' ragged_area.rpt

2460

awk 'NF == 3 { total += $3 } NF != 3 { bad++ } END { print "total:", total + 0; if (bad) print "WARNING:", bad, "record(s) had no area value" }' ragged_area.rpt

total: 2460
WARNING: 1 record(s) had no area value

The total is the same number both times — 2460 — because the missing field contributed 0 either way. The difference is that the second version says a record was unusable. The arithmetic was never wrong; the reporting was. This is the same lesson Chapter 1.5 drew from comparing awk with Python, reached here from inside awk.

Exercise 5 — Engineering application

Using area.rpt, print each instance with its area, but only for instances whose area exceeds 1000, and report how many qualified.

awk '$3 > 1000 { print $1, $3; n++ } END { print "above 1000:", n + 0 }' area.rpt

u_alu 1820
u_regfile 2410
above 1000: 2

One rule doing two jobs — printing the matching records and counting them — with the count emitted once at the end. The + 0 keeps a no-match run printing 0 rather than an empty line.

Exercise 6 — Challenge: make $NF safe

A colleague's slack extractor is awk '{ print $1, $NF }' slack.rpt. It works on clean input. Explain what it does on the record that lost a column, then write a version that cannot report a name as a slack value.

path_a setup -0.12
path_b setup
path_c setup -1.20

awk '{ print $1, $NF }' slack.rpt

path_a -0.12
path_b setup
path_c -1.20

There it is: path_b setup. $NF on that record is the word setup, because that is genuinely the last field. The output looks like a table of paths and slacks, and one row holds a word where a number belongs — which in any later arithmetic counts as 0.

awk 'NF == 3 { print $1, $3 } NF != 3 { print $1, "NO_SLACK" }' slack.rpt

path_a -0.12
path_b NO_SLACK
path_c -1.20

Two changes. The field is addressed by position ($3) rather than from the end, so it cannot drift. And the malformed record is labelled rather than guessed at, so the gap is visible to whatever reads this next.

The output order is worth looking at closely, because it reveals something about how awk runs. The rows come out in record order — path_a, path_b, path_c — even though path_b was handled by the second rule and the other two by the first. awk did not run rule one over the whole file and then rule two. It took record 1, tried both rules; took record 2, tried both rules; and so on. The record loop is the outer loop and the rules are the inner one, which is exactly what Chapter 2.3 is about.

13. Summary

• A newline ends a record; whitespace splits a record into fields. Two independent mechanisms.
• Fields are numbered from 1. $0 is the whole record, $1 the first field, NF the field count, $NF the last field.
• The default rule is generous: a run of whitespace is one separator, tabs count as whitespace, and leading or trailing whitespace is discarded rather than creating empty fields.
• That generosity is specific to the default. A single-character separator such as a comma does produce empty fields from adjacent delimiters — Chapter 3.3.
• $0 is the original text, not a re-join of the fields; the original spacing survives.
• NF has no dollar sign. A blank record has NF of 0, which is how you recognise one.
• $(NF-1) needs the parentheses. $NF-1 is the last field minus one, and it produces believable wrong numbers.
• An out-of-range field is the empty string, which is 0 in arithmetic — so a total over a column that does not exist is 0 with no error.
• $NF is not a column. On a record that lost a field it returns a different column, and a name arriving where a number belonged is harder to spot than an empty value. Guard with NF, and report what you skipped.

You now know what awk holds (Chapter 2.1) and how it takes that record apart (this chapter). What you have been using without explanation is the thing that decides whether your code runs on a given record at all.

Both of the last two exercises quietly used two rules at once and relied on them firing in the order written. Chapter 2.3 — Pattern-Action Processing is about that: what a pattern is, what an action is, what happens when you supply only one of the two, and the correction that matters most — awk does not stop at the first rule that matches.