Validating Input

Part II · Module 8 · Chapter 8.3 · Validating Input

1. Why This Matters

awk's greatest convenience is that it never refuses. A missing field is empty, a non-numeric value is 0, an absent file produces no records. Chapters 5.3, 6.1 and 7.5 each found a case where that silence produced a confident wrong answer.

This chapter is the general remedy. It is rated critical because of what the scripts in this track are for: a number in a signoff report. A wrong number there is acted on, and the cost of the wrong number is unrelated to the cost of the bug that produced it.

The four checks are cheap — three or four lines in total — and the discipline is to write them before the script is trusted rather than after it has misled somebody.

2. Check One: Any Records

The cheapest check and the one most often omitted:

test,covered,total
alu,95,100
fifo,72,100

awk -F, 'END { if (!NR) { print "ERROR: file is empty" > "/dev/stderr"; exit 2 } printf "%d line(s)\n", NR }' metrics.csv

3 line(s)

: > empty.csv
awk -F, 'END { if (!NR) { print "ERROR: file is empty" > "/dev/stderr"; exit 2 } printf "%d line(s)\n", NR }' empty.csv 2>&1
echo "status: $?"

ERROR: file is empty
status: 2

Chapter 6.4 section 3 made this point about division by zero; the broader version is that an empty input is usually evidence that an earlier step failed, and reporting it as such is more useful than any number.

A header-only file is a different condition and deserves its own message:

test,covered,total

awk -F, 'NR == 1 { next } { n++ } END { if (!NR) { print "ERROR: file is empty" > "/dev/stderr"; exit 2 } if (!n) { print "ERROR: header only, no data rows" > "/dev/stderr"; exit 2 } printf "%d data row(s)\n", n }' hdr_only.csv 2>&1
echo "status: $?"

ERROR: header only, no data rows
status: 2

Same status, different diagnosis. !NR catches a file with nothing in it; !n catches one whose only line was the header. A tool that wrote its header and then died produces the second, and conflating the two costs you the information that the tool started.

3. Check Two: The Header

A header check is the cheapest protection against a format change:

awk -F, 'NR == 1 { if ($1 != "test") { printf "ERROR: unexpected header: [%s]\n", $1 > "/dev/stderr"; bail = 2; exit 2 } print "header ok"; next } END { if (bail) exit bail }' metrics.csv
echo "status: $?"

header ok
status: 0

name,covered,total
alu,95,100

awk -F, 'NR == 1 { if ($1 != "test") { printf "ERROR: unexpected header: [%s]\n", $1 > "/dev/stderr"; bail = 2; exit 2 } print "header ok"; next } END { if (bail) exit bail }' wrong_hdr.csv 2>&1
echo "status: $?"

ERROR: unexpected header: [name]
status: 2

The message quotes what it actually found, which is what makes it actionable — unexpected header alone sends the reader to look, while [name] tells them the column was renamed.

Checking the whole header is stricter and better when the format is fixed:

awk -F, 'NR == 1 { if ($0 != "test,covered,total") { printf "ERROR: header is [%s], expected [test,covered,total]\n", $0 > "/dev/stderr"; bail = 2; exit 2 } next } { n++ } END { if (bail) exit bail; printf "%d data row(s) after a verified header\n", n }' metrics.csv
echo "status: $?"

2 data row(s) after a verified header
status: 0

That catches a reordering as well as a rename, which a $1 check cannot — and reordering is the more dangerous change, because every column index in the script shifts while the header still contains all the expected words.

4. Check Three: The Field Count

A tool that gains a column breaks every positional reference. Comparing each row against the header catches it:

test,covered,total
alu,95
fifo,72,100

awk -F, 'NR == 1 { want = NF; next } NF != want { printf "record %d has %d field(s), header has %d\n", NR, NF, want; bad++ } END { printf "%d malformed of %d data row(s)\n", bad + 0, NR - 1; if (bad) exit 2 }' ragged.csv 2>&1
echo "status: $?"

record 2 has 2 field(s), header has 3
1 malformed of 2 data row(s)
status: 2

The header's own NF is the expected count, so nothing is hard-coded — the check adapts if the format legitimately changes and the header changes with it.

Note it reports both numbers: the malformed count and the total. Chapter 5.3's denominator habit, and here it distinguishes "one bad row in a thousand" from "every row is bad", which call for different responses.

This check is also how Chapter 7.6 section 7 detected an unquoted CSV. A field count that varies by row is the signature of a delimiter inside a field.

5. Check Four: Field Types

The checks so far verify the shape. This one verifies that the fields the script will do arithmetic on are usable, which Chapter 5.3 section 11 established needs a positive pattern:

test,covered,total
alu,95,100
fifo,n/a,100
uart,88,100

awk -F, 'NR == 1 { next } $2 !~ /^-?[0-9]+(\.[0-9]+)?$/ { printf "record %d: covered is not numeric: [%s]\n", NR, $2 > "/dev/stderr"; bad++; next } { hit += $2; tot += $3; n++ } END { printf "%.2f%% over %d row(s)", 100 * hit / tot, n; if (bad) printf "; %d row(s) unusable", bad; print "" }' bad_types.csv 2>&1
echo "status: $?"

record 3: covered is not numeric: [n/a]
91.50% over 2 row(s); 1 row(s) unusable
status: 0

The offending record is named on stderr, the percentage is computed from the two rows that were usable, and the count of skipped rows is printed beside it — Chapter 5.3's "count what you excluded" rule.

Note the status is 0. One unusable row in a file is a data-quality note rather than a reason to refuse the whole input, and section 10's convention reserves a non-zero status for a criterion that was not met or an input that could not be read at all. Whether that is the right call is a judgement the script should state, which is what printing 1 row(s) unusable does.

6. Report What Was Skipped

A validator that silently drops rows is only half a validator. The percentage in section 5 is correct for the data it used, and the number a reader needs alongside it is how much data that was.

The counts, printed from the accumulators directly:

awk -F, 'NR == 1 { next } $2 !~ /^-?[0-9]+(\.[0-9]+)?$/ { bad++; next } { n++ } END { printf "usable=%d unusable=%d of %d data row(s)\n", n, bad + 0, NR - 1 }' bad_types.csv

usable=2 unusable=1 of 3 data row(s)

Three data rows, two usable, one not — and the three numbers are consistent, which is the property worth checking.

They are consistent because they come from one place. A tempting alternative is to count the usable rows in the accumulating rule and derive the unusable count as NR - 1 - n, which works until a third category appears — a row skipped for a different reason, or a blank line — at which point the derived number silently absorbs it.

awk -F, 'NR == 1 { next } /^$/ { blank++; next } $2 !~ /^-?[0-9]+(\.[0-9]+)?$/ { bad++; next } { n++ } END { printf "usable=%d unusable=%d blank=%d of %d; sum=%d\n", n, bad + 0, blank + 0, NR - 1, n + bad + blank }' bad_types.csv

usable=2 unusable=1 blank=0 of 3; sum=3

The sum column is the check: every category counted explicitly, and their total compared against the denominator. If they ever disagree, a record took a path nobody accounted for — and on a real report that is how you discover a row format you did not know existed.

This is the same arithmetic check Chapter 5.4's three-category classifier used and Chapter 7.8's merge assertion used: derive the total two ways and compare. Here the two ways are the sum of the categories and NR - 1.

7. One Fault, One Message

Chapter 5.6 section 7 established that exit still runs END. That makes a multi-check validator emit several messages for one fault unless each guard records that it has already refused.

Without a flag, a bad header also trips the no-data check:

awk -F, 'NR == 1 && $1 != "test" { print "ERROR: unexpected header" > "/dev/stderr"; exit 2 } NR == 1 { next } { n++ } END { if (!n) { print "ERROR: no data records" > "/dev/stderr"; exit 2 } printf "%d row(s)\n", n }' wrong_hdr.csv 2>&1
echo "status: $?"

ERROR: unexpected header
ERROR: no data records
status: 2

Two errors for one fault. The second is a consequence of the first — exit stopped the input, so no data rows were counted — and it is misleading: a reader may conclude the file is both malformed and empty.

A bail variable fixes it:

awk -F, 'NR == 1 && $1 != "test" { print "ERROR: unexpected header" > "/dev/stderr"; bail = 2; exit 2 } NR == 1 { next } { n++ } END { if (bail) exit bail; if (!n) { print "ERROR: no data records" > "/dev/stderr"; exit 2 } printf "%d row(s)\n", n }' wrong_hdr.csv 2>&1
echo "status: $?"

ERROR: unexpected header
status: 2

One message, the right one, same status. And the good path still works:

awk -F, 'NR == 1 && $1 != "test" { print "ERROR: unexpected header" > "/dev/stderr"; bail = 2; exit 2 } NR == 1 { next } { n++ } END { if (bail) exit bail; if (!n) { print "ERROR: no data records" > "/dev/stderr"; exit 2 } printf "%d row(s)\n", n }' metrics.csv
echo "status: $?"

2 row(s)
status: 0

8. Where Each Check Belongs

The ordering in Figure 1 follows from that table rather than being a separate rule: BEGIN first because it needs no input, the header next because it is record one, the per-record checks as the data arrives, and the two emptiness checks last because they are the only ones that cannot be decided early.

9. Common Mistakes

No emptiness check. An empty input usually means an earlier step failed — section 2.

Conflating empty with header-only. Different faults, different messages.

Checking only $1 of the header. A reordering passes — section 3.

Hard-coding the expected field count. Use the header's NF — section 4.

A negative type check. It misses the value you did not think of — Chapter 5.3 section 11.

No bail flag. exit still runs END, so one fault emits several messages — section 7.

Not reporting what was skipped. A percentage without its denominator hides how much data it used — section 6.

Using status 1 for a malformed input. 1 means evaluated and failing; 2 means could not evaluate — section 10.

Validating after computing. The arithmetic has already consumed the bad field.

10. The Status Convention

Every script in Modules 5 to 8 has used the same three statuses, and this is the chapter that states it:

The distinction between 1 and 2 is the engineering point. A regression that ran and failed needs a debugging effort; a regression that did not run needs a flow fix, and the two go to different people. A script that returns 1 for both sends every malformed-input problem to whoever triages test failures, where it will be investigated as a test failure.

And returning 0 for a malformed input is the worst case, because a nightly pipeline records success and nobody looks at all — Chapter 5.6 section 9's point about printing FAIL and exiting 0.

11. Exercises

Exercise 1 — Predict the status

: > e1.csv
awk 'END { if (!NR) exit 2 }' e1.csv
echo "status: $?"

status: 2

NR is 0 on an empty file, so the guard fires.

Exercise 2 — Check a header

Refuse a file whose header is not exactly the expected line.

test,covered,total
alu,95,100
fifo,72,100

awk -F, 'NR == 1 { if ($0 != "test,covered,total") { printf "ERROR: header is [%s]\n", $0 > "/dev/stderr"; bail = 2; exit 2 } next } { n++ } END { if (bail) exit bail; printf "%d data row(s)\n", n }' metrics.csv
echo "status: $?"

2 data row(s)
status: 0

name,covered,total
alu,95,100

awk -F, 'NR == 1 { if ($0 != "test,covered,total") { printf "ERROR: header is [%s]\n", $0 > "/dev/stderr"; bail = 2; exit 2 } next } { n++ } END { if (bail) exit bail; printf "%d data row(s)\n", n }' wrong_hdr.csv 2>&1
echo "status: $?"

ERROR: header is [name,covered,total]
status: 2

Exercise 3 — Check the field count

Report rows whose field count differs from the header's.

test,covered,total
alu,95
fifo,72,100

awk -F, 'NR == 1 { want = NF; next } NF != want { printf "record %d has %d field(s), header has %d\n", NR, NF, want; bad++ } END { printf "%d malformed of %d data row(s)\n", bad + 0, NR - 1; if (bad) exit 2 }' ragged.csv 2>&1
echo "status: $?"

record 2 has 2 field(s), header has 3
1 malformed of 2 data row(s)
status: 2

Exercise 4 — Debug a doubled error message

A validator reports two faults for one bad file.

awk -F, 'NR == 1 && $1 != "test" { print "ERROR: unexpected header" > "/dev/stderr"; exit 2 } NR == 1 { next } { n++ } END { if (!n) { print "ERROR: no data records" > "/dev/stderr"; exit 2 } printf "%d row(s)\n", n }' wrong_hdr.csv 2>&1
echo "status: $?"

ERROR: unexpected header
ERROR: no data records
status: 2

exit 2 in the header rule stopped the input and went to END, where the no-data check also fired — Chapter 5.6 section 7.

awk -F, 'NR == 1 && $1 != "test" { print "ERROR: unexpected header" > "/dev/stderr"; bail = 2; exit 2 } NR == 1 { next } { n++ } END { if (bail) exit bail; if (!n) { print "ERROR: no data records" > "/dev/stderr"; exit 2 } printf "%d row(s)\n", n }' wrong_hdr.csv 2>&1
echo "status: $?"

ERROR: unexpected header
status: 2

Exercise 5 — Distinguish empty from header-only

Give the two conditions different messages.

: > empty.csv
awk -F, 'NR == 1 { next } { n++ } END { if (!NR) { print "ERROR: file is empty" > "/dev/stderr"; exit 2 } if (!n) { print "ERROR: header only, no data rows" > "/dev/stderr"; exit 2 } printf "%d data row(s)\n", n }' empty.csv 2>&1
echo "status: $?"

ERROR: file is empty
status: 2

test,covered,total

awk -F, 'NR == 1 { next } { n++ } END { if (!NR) { print "ERROR: file is empty" > "/dev/stderr"; exit 2 } if (!n) { print "ERROR: header only, no data rows" > "/dev/stderr"; exit 2 } printf "%d data row(s)\n", n }' hdr_only.csv 2>&1
echo "status: $?"

ERROR: header only, no data rows
status: 2

A tool that wrote its header and died gives the second — evidence it started.

Exercise 6 — Engineering application

Write a validating coverage summary with all four checks, the three-status convention, and one message per fault.

# validate.awk — summarise a coverage CSV, refusing input it cannot trust.
#
# Usage:  awk -v min_cov=80 -f validate.awk metrics.csv
# Exit:   0 coverage at or above min_cov
#         1 coverage below min_cov
#         2 could not evaluate

BEGIN {
    FS = ","
    HEADER = "test,covered,total"
    if (min_cov !~ /^[0-9]+(\.[0-9]+)?$/) {
        printf "ERROR: min_cov is not a number: [%s]\n", min_cov > "/dev/stderr"
        bail = 2; exit 2
    }
}

# Check 2 — the header, exactly.
NR == 1 {
    if ($0 != HEADER) {
        printf "ERROR: header is [%s], expected [%s]\n", $0, HEADER > "/dev/stderr"
        bail = 2; exit 2
    }
    want = NF
    next
}

# Check 3 — field count against the header.
NF != want {
    printf "ERROR: record %d has %d field(s), header has %d\n", NR, NF, want > "/dev/stderr"
    bail = 2; exit 2
}

# Check 4 — the fields we will do arithmetic on.
$2 !~ /^[0-9]+$/ || $3 !~ /^[0-9]+$/ {
    printf "WARNING: record %d has a non-numeric count: [%s] [%s]\n", NR, $2, $3 > "/dev/stderr"
    unusable++
    next
}

{ hit += $2; tot += $3; rows++ }

END {
    if (bail) exit bail
    if (!NR) { print "ERROR: file is empty" > "/dev/stderr"; exit 2 }
    if (!rows) { print "ERROR: no usable data rows" > "/dev/stderr"; exit 2 }
    if (!tot) { print "ERROR: total bin count is zero" > "/dev/stderr"; exit 2 }
    pc = 100 * hit / tot
    printf "coverage %.2f%% (%d of %d bins over %d row(s)", pc, hit, tot, rows
    if (unusable) printf ", %d unusable", unusable
    printf "); threshold %s%%\n", min_cov
    if (pc + 0 < min_cov + 0) exit 1
}

awk -v min_cov=80 -f validate.awk metrics.csv
echo "status: $?"

coverage 83.50% (167 of 200 bins over 2 row(s)); threshold 80%
status: 0

Above the threshold, status 0. Below it:

awk -v min_cov=90 -f validate.awk metrics.csv
echo "status: $?"

coverage 83.50% (167 of 200 bins over 2 row(s)); threshold 90%
status: 1

Status 1 — evaluated, criterion not met. And each refusal gives status 2 with one message:

awk -v min_cov=80 -f validate.awk wrong_hdr.csv 2>&1
echo "status: $?"

ERROR: header is [name,covered,total], expected [test,covered,total]
status: 2

awk -v min_cov=80 -f validate.awk ragged.csv 2>&1
echo "status: $?"

ERROR: record 2 has 2 field(s), header has 3
status: 2

awk -f validate.awk metrics.csv 2>&1
echo "status: $?"

ERROR: min_cov is not a number: []
status: 2

Four distinct refusals, one message each, all status 2 — distinguishable from the status 1 that means the design genuinely missed its target.

Note the type check is a warning that skips the row rather than a refusal, because one unusable row in a large file is a data-quality note while a bad header is a format change. That is a judgement call and the script states it: the summary prints unusable alongside the denominator so the reader can see how much was skipped — Chapter 5.3's rule.

Exercise 7 — Challenge: prove every guard can fire

A guard that has never been observed failing is a comment. Exercise the whole validator.

for f in metrics.csv wrong_hdr.csv ragged.csv hdr_only.csv empty.csv; do
  printf "%-16s -> " "$f"
  out=$(awk -v min_cov=80 -f validate.awk "$f" 2>&1); rc=$?
  printf "rc=%d  %s\n" "$rc" "$(echo "$out" | head -1)"
done

metrics.csv      -> rc=0  coverage 83.50% (167 of 200 bins over 2 row(s)); threshold 80%
wrong_hdr.csv    -> rc=2  ERROR: header is [name,covered,total], expected [test,covered,total]
ragged.csv       -> rc=2  ERROR: record 2 has 2 field(s), header has 3
hdr_only.csv     -> rc=2  ERROR: no usable data rows
empty.csv        -> rc=2  ERROR: file is empty

Five inputs, five distinct outcomes, each guard observed firing on the input it exists for — and the good file still passing.

The one guard not exercised there is the non-numeric threshold, which depends on the command line rather than the file:

awk -v min_cov=eighty -f validate.awk metrics.csv 2>&1 | head -1
awk -v min_cov=80 -f validate.awk bad_types.csv 2>&1

ERROR: min_cov is not a number: [eighty]
WARNING: record 3 has a non-numeric count: [n/a] [100]
coverage 91.50% (183 of 200 bins over 2 row(s), 1 unusable); threshold 80%

The second command shows the type check behaving as designed: it warns, skips the row, and the summary still appears with 1 unusable stated — a warning rather than a refusal, per section 5.

Six guards, six demonstrations. A validator's test matrix is one input per guard plus one that passes — and if a guard has no input that triggers it, either the input is missing or the guard is dead code, which Chapter 6.3's final exercise made the same argument for.

12. Summary

• awk never refuses, so a changed format produces a wrong number rather than an error.
• Check any records at all, and distinguish an empty file from a header-only one — different faults.
• Check the whole header, not just $1: a reordering passes a $1 check and shifts every column index.
• Check the field count against the header's NF, so nothing is hard-coded.
• Check field types with a positive pattern before the arithmetic consumes them.
• exit is not abort — set bail = N in each guard and begin END with if (bail) exit bail.
• Report each count from one place; two derivations of the same number can disagree.
• 0 means evaluated and met, 1 means evaluated and not met, 2 means could not evaluate. Conflating 1 and 2 sends format problems to whoever triages test failures.
• Validate -v parameters in BEGIN, before any input is read.
• Exercise every guard once. One input per guard plus one that passes.

Shape and type are now checked. Chapter 8.4 — Missing and Empty Fields goes a level deeper into what awk does with a field that is simply not there — which is not quite the same as an unset variable, and the difference decides whether your check for it works.