Missing and Empty Fields

Part II · Module 8 · Chapter 8.4 · Missing and Empty Fields

1. Why This Matters

Chapter 5.3 showed a malformed value corrupting a mean. This chapter is about a missing field, which does the same damage through a different door and is far more common — a tool that omits a column for one row, a log line truncated mid-write, a CSV whose last column is blank.

It is rated critical for one reason, in section 4: the obvious guard does not work. if ($2 == 0) reads as "is this field missing or zero?" and it detects only the second.

2. A Field Past NF Is Empty

Referring to a field that does not exist is not an error:

alu 95 100 ok
fifo 72
uart 88 100 ok

awk '{ printf "NR=%d NF=%d  $4=[%s]  $9=[%s]\n", NR, NF, $4, $9 }' ragged.txt

NR=1 NF=4  $4=[ok]  $9=[]
NR=2 NF=2  $4=[]  $9=[]
NR=3 NF=4  $4=[ok]  $9=[]

$9 is empty on every record and $4 is empty on the short one. No warning, no error, no change in exit status.

That is deliberate and it is what makes awk tolerant of ragged input — but tolerance and correctness are different things, and section 5 is the difference.

3. Reading a Field Does Not Create It

Unlike an array element, reading a non-existent field is harmless:

awk '{ before = NF; x = $9; printf "NF before=%d after=%d\n", before, NF }' ragged.txt

NF before=4 after=4
NF before=2 after=2
NF before=4 after=4

NF is unchanged. This is worth stating explicitly because Chapter 6.1 section 5 established the opposite for arrays — a["k"] read into existence creates the key — and it is natural to assume fields behave the same way.

They do not. Reading $9 is safe; reading a[9] is not. Section 7 covers the one field operation that does modify the record.

4. Three Kinds of Nothing

Now the distinction this chapter exists for. Three values that all look empty, and the tests that tell them apart:

a 0
b 
c

awk '{ printf "NR=%d NF=%d  $2=[%s] len=%d  ($2==\"\")=%d  ($2==0)=%d  (2<=NF)=%d\n", NR, NF, $2, length($2), ($2==""), ($2==0), (2<=NF) }' three.txt

NR=1 NF=2  $2=[0] len=1  ($2=="")=0  ($2==0)=1  (2<=NF)=1
NR=2 NF=1  $2=[] len=0  ($2=="")=1  ($2==0)=0  (2<=NF)=0
NR=3 NF=1  $2=[] len=0  ($2=="")=1  ($2==0)=0  (2<=NF)=0

Read the ($2==0) column. It is 1 for the real zero and 0 for the missing field.

And compare an uninitialised variable, which behaves differently again:

awk 'BEGIN { printf "uninitialised var: (u==0)=%d (u==\"\")=%d\n", (u == 0), (u == "") ; printf "string constant:   (\"\"==0)=%d\n", ("" == 0) }' </dev/null

uninitialised var: (u==0)=1 (u=="")=1
string constant:   (""==0)=0

An uninitialised variable equals both 0 and the empty string. A string constant "" equals the empty string and not 0. And a missing field behaves like the constant, not like the variable.

5. The Silent Zero

What the missing field costs, measured. A naive average over the third column:

awk '{ s += $3; n++ } END { printf "sum=%d over n=%d -> mean=%.2f\n", s, n, s/n }' ragged.txt

sum=200 over n=3 -> mean=66.67

Guarded on the record's shape:

awk 'NF < 3 { bad++; next } { s += $3; n++ } END { printf "sum=%d over n=%d -> mean=%.2f (%d skipped)\n", s, n, s/n, bad + 0 }' ragged.txt

sum=200 over n=2 -> mean=100.00 (1 skipped)

Same sum, different mean. 66.67 against 100.00 — a 33% error.

The sum is identical because the missing field contributed 0 and adding 0 changes nothing. The mean differs because the short record was counted in the denominator while contributing nothing to the numerator.

That is exactly Chapter 5.3 section 10's finding, arriving through a different cause. There the field was present and malformed; here it is absent. Both corrupt the denominator and leave the sum checkable, which is the worst combination — the number a reviewer would verify still agrees.

6. Blank and Absent Are Indistinguishable by Default

Look again at rows 2 and 3 of section 4's output. Row 2 is b — a name, a space, and nothing else. Row 3 is c — a name and nothing. They agree on every test, including NF.

The reason is Chapter 3.1's: the default FS splits on runs of whitespace and ignores leading and trailing whitespace entirely. So a trailing space creates no field, and b has exactly one field just as c does.

With an explicit separator they differ:

a,0
b,
c

awk -F, '{ printf "NR=%d NF=%d  $2=[%s]  ($2==\"\")=%d  (2<=NF)=%d\n", NR, NF, $2, ($2==""), (2<=NF) }' three.csv

NR=1 NF=2  $2=[0]  ($2=="")=0  (2<=NF)=1
NR=2 NF=2  $2=[]  ($2=="")=1  (2<=NF)=1
NR=3 NF=1  $2=[]  ($2=="")=1  (2<=NF)=0

Now row 2 has two fields, the second empty, and NF tells it apart from row 3.

Read the two test columns together, because they divide the work differently. $2 == "" is 1 for both the blank field and the absent one — it reports emptiness and cannot say why. 2 <= NF is 1 only for the blank one — it reports presence.

So neither test alone covers three states, and the pair does:

Under the default FS the blank row is not representable at all — a trailing space creates no field — so NF is the only test you need. With an explicit FS all three states exist and both tests are required. That is why validation logic does not transfer between an aligned EDA report and a CSV.

7. Assigning Past NF Extends the Record

Reading a non-existent field is harmless (section 3). Assigning to one is not:

awk 'NR == 2 { printf "before: NF=%d [%s]\n", NF, $0; $5 = "added"; printf "after:  NF=%d [%s]\n", NF, $0 }' ragged.txt

before: NF=2 [fifo 72]
after:  NF=5 [fifo 72   added]

NF went from 2 to 5, and $0 was rebuilt with three spaces between 72 and added — fields 3 and 4 were created empty and joined by OFS.

This is the mirror of Chapter 5.2 section 10, where assigning NF = 2 truncated a longer record. The same variable, both directions: writing a field past the end extends, and lowering NF discards.

It has a legitimate use — adding a computed column — and the thing to be aware of is the gap. Writing $5 on a two-field record does not append a third field; it appends a fifth and leaves two empty ones behind, which a downstream consumer will see as empty columns rather than a short row.

Appending safely means using NF + 1:

awk 'NR == 2 { $(NF + 1) = "added"; printf "NF=%d [%s]\n", NF, $0 }' ragged.txt

NF=3 [fifo 72 added]

Three fields, no gap. $(NF + 1) is the idiom for "one past the end, whatever the end is" — and the parentheses are required, since $NF + 1 is the last field's value plus one.

8. Writing the Guard

Collecting the chapter into a usable pattern. Three ways to handle a short record, each right in a different situation:

Skip it and count the skips — for an average, where including it corrupts the denominator:

awk 'NF < 3 { skipped++; next } { s += $3; n++ } END { printf "mean %.2f over %d row(s), %d skipped\n", s/n, n, skipped + 0 }' ragged.txt

mean 100.00 over 2 row(s), 1 skipped

Default it and say so — when a missing value genuinely means zero, which is rare and should be stated:

awk '{ v = (NF >= 3 ? $3 : 0); if (NF < 3) defaulted++; s += v; n++ } END { printf "mean %.2f over %d row(s), %d defaulted to 0\n", s/n, n, defaulted + 0 }' ragged.txt

mean 66.67 over 3 row(s), 1 defaulted to 0

Note this gives 66.67 — the naive answer from section 5 — and that is now defensible, because the output says a row was defaulted. The number is the same; the honesty is not.

Refuse the file — when a short record means the format is wrong, which Chapter 8.3 section 4 argued is the usual case for a fixed-format report:

awk 'NF < 3 { printf "ERROR: record %d has %d field(s), expected at least 3\n", NR, NF > "/dev/stderr"; bail = 2; exit 2 } { s += $3; n++ } END { if (bail) exit bail; printf "mean %.2f over %d row(s)\n", s/n, n }' ragged.txt 2>&1
echo "status: $?"

ERROR: record 2 has 2 field(s), expected at least 3
status: 2

Which to choose is an engineering decision, not a style one. The question is whether a short record is data (skip or default it) or a format change (refuse). For a nightly report whose format is fixed, it is a format change.

9. Common Mistakes

if ($2 == 0) as a missing-field test. It misses the missing field and rejects a legitimate zero — section 4.

Assuming a field behaves like an unset variable. A variable equals both 0 and ""; a field equals only "".

Assuming reading $9 creates it. It does not — but reading a[9] does, per Chapter 6.1 section 5.

Averaging over NR on ragged input. The short rows inflate the denominator — section 5.

Expecting a trailing space to create a field under the default FS. It does not — section 6.

Using the same validation for whitespace and comma-separated data. Three states are distinguishable in one and not the other.

Assigning $5 to append a field. It leaves empty fields behind; use $(NF + 1) — section 7.

Writing $NF + 1 when you meant $(NF + 1). The first is arithmetic on the last field.

Defaulting a missing value to 0 without saying so. The number is then indistinguishable from a measured zero.

10. Exercises

Exercise 1 — Predict the output

awk 'BEGIN { printf "%d %d %d\n", (u == 0), (u == ""), ("" == 0) }' </dev/null

1 1 0

An uninitialised variable equals both; the string constant "" does not equal 0.

Exercise 2 — Tell the three states apart

Report, for each record, whether field 2 is absent, blank or filled.

a,0
b,
c

awk -F, '{ printf "NR=%d %s\n", NR, (NF < 2 ? "ABSENT" : ($2 == "" ? "blank" : "filled: " $2)) }' three.csv

NR=1 filled: 0
NR=2 blank
NR=3 ABSENT

NF separates absent from present; $2 == "" separates blank from filled. Both tests are needed.

Exercise 3 — Debug a wrong average

A script reports a mean of 66.67 for a column whose two usable values are 100 and 100.

alu 95 100 ok
fifo 72
uart 88 100 ok

awk '{ s += $3; n++ } END { printf "sum=%d over n=%d -> mean=%.2f\n", s, n, s/n }' ragged.txt

sum=200 over n=3 -> mean=66.67

fifo has no $3, so it added 0 to the sum and 1 to the count.

awk 'NF < 3 { bad++; next } { s += $3; n++ } END { printf "sum=%d over n=%d -> mean=%.2f (%d skipped)\n", s, n, s/n, bad + 0 }' ragged.txt

sum=200 over n=2 -> mean=100.00 (1 skipped)

The sum was right all along — the denominator was not.

Exercise 4 — Debug a guard that does not fire

An engineer guards against a missing field with $3 == 0 and the mean is still wrong.

awk '$3 == 0 { bad++; next } { s += $3; n++ } END { printf "mean=%.2f over %d (%d skipped)\n", s/n, n, bad + 0 }' ragged.txt

mean=66.67 over 3 (0 skipped)

Zero skipped. $3 on the short record is the empty string, and "" == 0 is false — section 4.

awk 'NF < 3 { bad++; next } { s += $3; n++ } END { printf "mean=%.2f over %d (%d skipped)\n", s/n, n, bad + 0 }' ragged.txt

mean=100.00 over 2 (1 skipped)

And the $3 == 0 guard has the second fault too — it would reject a record whose slack is a measured zero:

alu 95 100
fifo 72 0
uart 88 100

awk '$3 == 0 { bad++; next } { s += $3; n++ } END { printf "mean=%.2f over %d (%d skipped)\n", s/n, n, bad + 0 }' haszero.txt

mean=100.00 over 2 (1 skipped)

A legitimate zero discarded, and the mean is now the mean of the other two. The NF guard keeps it:

awk 'NF < 3 { bad++; next } { s += $3; n++ } END { printf "mean=%.2f over %d (%d skipped)\n", s/n, n, bad + 0 }' haszero.txt

mean=66.67 over 3 (0 skipped)

Three records, nothing skipped, and the zero included — which is the correct mean of 100, 0 and 100.

Exercise 5 — Append a field safely

Add a computed column to each record without leaving gaps.

awk 'NF >= 3 { $(NF + 1) = ($3 + 0 >= 100 ? "FULL" : "PARTIAL"); print }' ragged.txt

alu 95 100 ok FULL
uart 88 100 ok FULL

$(NF + 1) appends one past the end. Using $5 instead would have worked here by coincidence — both records have four fields — and would leave a gap on any record of a different width.

Exercise 6 — Engineering application

Summarise a ragged report, classifying every record and proving the categories account for all of it.

awk '{ total++ } NF < 3 { short++; next } $3 !~ /^[0-9]+$/ { nonnum++; next } { s += $3; n++ } END { printf "usable=%d short=%d non-numeric=%d of %d; sum=%d\n", n, short + 0, nonnum + 0, total, n + short + nonnum; if (n) printf "mean %.2f over %d usable row(s)\n", s/n, n; else { print "ERROR: no usable rows" > "/dev/stderr"; exit 2 } }' ragged.txt
echo "status: $?"

usable=2 short=1 non-numeric=0 of 3; sum=3
mean 100.00 over 2 usable row(s)
status: 0

The sum=3 column is the check: three categories adding to the record count, so no record took an unaccounted path — Chapter 8.3 section 6's arithmetic, and the same device as Chapter 5.4's three-category classifier.

The mean is reported with its denominator and the two exclusion counts beside it, so a reader can see that one third of the rows were dropped. On a file where that fraction were large, the mean would be worth distrusting — and the output is what makes that judgement possible.

The engineering caveat: short=1 says a record lacked the column, which is a statement about the file, not about the design. A tool that omits a column for an unanalysed block and one that omits it because of a crash produce the same row, and distinguishing them needs the tool's own log — Chapter 7.4's point about an empty file, at the granularity of a field.

Exercise 7 — Challenge: prove both guard faults

Build one check that demonstrates $n == 0 failing in both directions on the same command.

for f in ragged.txt haszero.txt; do
  printf "%-12s " "$f"
  awk -v tag="$f" '$3 == 0 { bad++; next } { n++ } END { printf "$3==0 guard: kept %d skipped %d   ", n, bad + 0 }' "$f"
  awk 'NF < 3 { bad++; next } { n++ } END { printf "NF guard: kept %d skipped %d\n", n, bad + 0 }' "$f"
done

ragged.txt   $3==0 guard: kept 3 skipped 0   NF guard: kept 2 skipped 1
haszero.txt  $3==0 guard: kept 2 skipped 1   NF guard: kept 3 skipped 0

The two rows are the two faults, side by side:

• On ragged.txt the $3==0 guard kept 3 — it let the short record through, which is the miss.
• On haszero.txt it kept 2 — it threw away the record whose value is a measured zero, which is the false positive.

The NF guard gets both right: it skips the short record and keeps the zero.

The table is the argument. A guard that is wrong in both directions cannot be fixed by inverting it or by adding a clause — the test is asking the wrong question. NF asks about the record's shape and $n == "" asks about the field's content; $n == 0 asks about neither.

11. Summary

• A field past NF is the empty string, not an error, and reading it does not change NF.
• Reading a non-existent field is safe; reading a non-existent array element creates it (Chapter 6.1).
• An uninitialised variable equals both 0 and "". A missing field equals only "".
• So $2 == 0 does not detect a missing field, and it does reject a legitimate zero — wrong in both directions.
• Use NF < n to test the record's shape, and $n == "" to test the field's content.
• A missing field corrupts the denominator and leaves the sum intact — the same shape as Chapter 5.3's malformed value.
• Under the default FS, a trailing space creates no field, so blank and absent are indistinguishable; with an explicit FS they differ.
• Assigning past NF extends the record and leaves empty fields in the gap; $(NF + 1) appends cleanly.
• Decide whether a short record is data or a format change, then skip it, default it with a stated count, or refuse the file.
• Count every category and check the total against the record count.

Shape and presence are now handled. Chapter 8.5 — Numeric Conversion and Comparison finishes the job by settling the rule this chapter kept invoking: exactly when awk compares as numbers and when as text, why "10" < "9" can be true, and which of + 0 and "" forces the comparison you meant.