awk Part IV · Module 14 · AWK Debugging, Review, and Engineering Mastery
Debugging Incorrect or Empty Output
awk printed nothing, or printed the wrong thing, and there is an order in which to ask why — because the five common causes have very different fixes and four of them look identical from the output. This chapter establishes that order and gives each cause a test that distinguishes it, including the one a careful engineer is most likely to miss: a carriage return at the end of every line, which breaks every anchored pattern and which a bracketed dump of the raw record does not reveal.
Intermediate18 min readawkPart IVDebuggingChecklistCRLF
Part IV · Module 14 · Chapter 14.1 · Debugging Incorrect or Empty Output
1. Why the Order Matters
Empty output has no diagnostic content. A pattern that never matched, a field index past NF, a program the shell rewrote and a file that does not exist all produce the same nothing — so guessing picks one of four with equal odds and the wrong guess costs a round of edits.
The order below is not arbitrary. Each step either explains the symptom or rules out a whole class of cause, and the steps are ordered so that the cheapest and most decisive come first.
2. The Fixtures
printf 'Startpoint: u_alu/sum_reg[0]\r\n slack (VIOLATED) -0.018\r\n' > dos.rpt
printf 'name\tcells\tarea\nu_alu\t178\t1310.000\nu_ctrl\t170\t1240.000\n' > tabs.tsv
printf 'u_alu 178 1310.000\nu_ctrl 170 1240.000\n' > aligned.txt
: > empty.txt
wc -c dos.rpt tabs.tsv aligned.txt empty.txt | awk 'NF == 2 && $2 != "total" { printf " %-12s %d byte(s)\n", $2, $1 }' dos.rpt 59 byte(s)
tabs.tsv 55 byte(s)
aligned.txt 52 byte(s)
empty.txt 0 byte(s)A CRLF report, a tab-separated file, a space-aligned file, and an empty one.
3. Step One: NR, NF, and the Raw Record
for f in tabs.tsv aligned.txt dos.rpt empty.txt; do
printf " --- %s ---\n" "$f"
awk '{ printf " NR=%d NF=%d [%s]\n", NR, NF, $0 }' "$f" | head -2
done --- tabs.tsv ---
NR=1 NF=3 [name cells area]
NR=2 NF=3 [u_alu 178 1310.000]
--- aligned.txt ---
NR=1 NF=3 [u_alu 178 1310.000]
NR=2 NF=3 [u_ctrl 170 1240.000]
--- dos.rpt ---
NR=1 NF=2 [Startpoint: u_alu/sum_reg[0]
]
NR=2 NF=3 [ slack (VIOLATED) -0.018
]
--- empty.txt ---This one command answers three questions at once:
- Did any record arrive?
empty.txtproduced no output at all, which no pattern or field problem can do. - How many fields does awk see? Three on the first three files — so
$4and beyond are empty and$3is the last real one. - What is actually in the record? The brackets delimit it, so leading and trailing spaces are visible.
Run it first, every time. It costs one command and eliminates more than half the possibilities.
4. Step Two: Is the Pattern Matching?
awk '{ n = length($0); last = substr($0, n); printf " NR=%d length=%d last byte=%d%s\n", NR, n, index("\r", last) ? 13 : 0, (index("\r", last) ? " <- CARRIAGE RETURN" : "") }' dos.rpt
cat -e dos.rpt | sed 's/^/ /' NR=1 length=29 last byte=13 <- CARRIAGE RETURN
NR=2 length=28 last byte=13 <- CARRIAGE RETURN
Startpoint: u_alu/sum_reg[0]^M$
slack (VIOLATED) -0.018^M$And the consequence:
printf " /VIOLATED\$/ matches : "; awk '/VIOLATED$/ { n++ } END { print n+0 }' dos.rpt
printf " /VIOLATED/ matches : "; awk '/VIOLATED/ { n++ } END { print n+0 }' dos.rpt /VIOLATED$/ matches : 0
/VIOLATED/ matches : 1Zero against one. $ anchors to the end of the record, and the last character is the carriage return — so every anchored pattern on a CRLF file fails.
The general technique, and the reason this is step two: widen the pattern until it matches, then narrow it back one change at a time. Dropping the $ changed 0 to 1, which localises the fault to the anchor in a single step. Chapter 14.4 covers the sub(/\r$/, "") fix and the other malformed-input cases.
5. Step Three: Is FS Right?
NF from step one already answers this, which is why it comes third rather than first:
printf " default FS on tabs.tsv : NF=%s \$2=[%s]\n" "$(awk 'NR==2 { print NF }' tabs.tsv)" "$(awk 'NR==2 { print $2 }' tabs.tsv)"
printf " -F'\\t' on tabs.tsv : NF=%s \$2=[%s]\n" "$(awk -F'\t' 'NR==2 { print NF }' tabs.tsv)" "$(awk -F'\t' 'NR==2 { print $2 }' tabs.tsv)"
printf " -F'\\t' on aligned.txt : NF=%s \$2=[%s]\n" "$(awk -F'\t' 'NR==1 { print NF }' aligned.txt)" "$(awk -F'\t' 'NR==1 { print $2 }' aligned.txt)" default FS on tabs.tsv : NF=3 $2=[178]
-F' ' on tabs.tsv : NF=3 $2=[178]
-F' ' on aligned.txt : NF=1 $2=[]Three results, and the asymmetry is the useful part:
- The default
FShandles a tab-separated file, because a tab is whitespace. Both forms giveNF=3. - Forcing
-F'\t'on a space-aligned file givesNF=1and every field reference past$1is empty.
So the dangerous direction is adding -F'\t' to a file that only looked tab-separated — which Chapter 14.2 takes further, because a column that lines up on screen says nothing about what separates it.
NF=1 on a file with visible columns is the signature, and it is diagnostic on its own.
6. Step Four: Did the Shell Change the Program?
printf " single-quoted : "; awk '{ print $1 }' aligned.txt | head -1
printf " double-quoted : "; awk "{ print $1 }" aligned.txt | head -1
printf " the program the shell built: [%s]\n" "$(echo "{ print $1 }")" single-quoted : u_alu
double-quoted : u_alu 178 1310.000
the program the shell built: [{ print }]The double-quoted form prints the whole line, because the shell expanded $1 to nothing and print with no argument prints $0. Chapter 12.1 section 3 measured this; here it is in the checklist because the symptom — too much output rather than none — sends people looking at the pattern.
The test is to echo the program. echo "{ print $1 }" shows exactly what awk received, and if it differs from what you typed, nothing else about the program matters yet.
7. Step Five: Is the Action Attached to the Pattern?
printf " /u_alu/ { n++ } END { print n+0 } -> "; awk '/u_alu/ { n++ } END { print n+0 }' aligned.txt
printf " /u_alu/; { n++ } END { print n+0 } -> "; awk '/u_alu/; { n++ } END { print n+0 }' aligned.txt /u_alu/ { n++ } END { print n+0 } -> 1
/u_alu/; { n++ } END { print n+0 } -> u_alu 178 1310.000
2One against two, from a single semicolon — and the second form also printed the matching record, which is the other half of the tell.
/u_alu/; is a complete rule — a pattern with no action, which prints matching records. The { n++ } that follows is then a separate rule with no pattern, so it fires on every record, and n counts the file rather than the matches.
The symptom is a count that is too high and looks plausible, which is why this is a checklist item rather than a footnote. The test is to count what the file has: if n equals NR, the action is unconditional.
8. Step Six: Empty or Unreadable?
for f in empty.txt no-such.txt aligned.txt; do
printf " %-14s " "$f"
awk 'END { printf "NR=%d\n", NR }' "$f" 2>/dev/null || printf "awk exit %d (cannot read)\n" $?
done empty.txt NR=0
no-such.txt awk exit 2 (cannot read)
aligned.txt NR=2Three outcomes: a readable empty file gives NR=0 and exit 0; an unreadable one gives exit 2 and no NR at all. Chapter 12.6 measured why these must not be conflated — getline's return values distinguish them and a boolean test gets both wrong.
This step is last in the written order and first in the actual order, because step one already performs it: no output from the NR/NF probe means either an empty file or an unreadable one, and the exit status says which.
9. The Checklist
10. All Five Causes, Side by Side
printf " %-34s %s\n" "cause" "distinguishing test"
printf " %-34s %s\n" "----------------------------------" "-------------------"
printf " %-34s %s\n" "file empty or unreadable" "no output from the NR/NF probe; exit 2 if unreadable"
printf " %-34s %s\n" "carriage return at line end" "length() one more than visible; cat -e shows ^M"
printf " %-34s %s\n" "pattern does not match" "widening it changes the count"
printf " %-34s %s\n" "FS wrong" "NF=1 on a file with visible columns"
printf " %-34s %s\n" "shell rewrote the program" "echo the program string"
printf " %-34s %s\n" "action not attached to a pattern" "the count equals NR" cause distinguishing test
---------------------------------- -------------------
file empty or unreadable no output from the NR/NF probe; exit 2 if unreadable
carriage return at line end length() one more than visible; cat -e shows ^M
pattern does not match widening it changes the count
FS wrong NF=1 on a file with visible columns
shell rewrote the program echo the program string
action not attached to a pattern the count equals NR11. Common Mistakes
Guessing at the pattern first. Four causes look identical from the output — section 1.
Trusting a bracketed dump of $0. A carriage return leaves no visible mark.
Assuming no output means no match. An empty file and an unreadable one produce the same nothing.
Adding -F'\t' because the columns line up. On a space-aligned file that gives NF=1.
Double-quoting the program. The shell rewrites it into a different working program.
A semicolon after a pattern. The following action becomes unconditional.
Narrowing a pattern to find a bug. Widen it until it matches, then narrow one change at a time.
Reading $? after a pipe. | head replaces awk's status — Chapter 12.1 section 7.
12. Exercises
Exercise 1 — The first command
awk '{ printf "NR=%d NF=%d [%s]\n", NR, NF, $0 }' aligned.txtNR=1 NF=3 [u_alu 178 1310.000]
NR=2 NF=3 [u_ctrl 170 1240.000]Exercise 2 — Find the carriage return
awk '{ printf " NR=%d length=%d\n", NR, length($0) }' dos.rpt
cat -e dos.rpt | sed 's/^/ /' NR=1 length=29
NR=2 length=28
Startpoint: u_alu/sum_reg[0]^M$
slack (VIOLATED) -0.018^M$^M before each $. The length includes it.
Exercise 3 — Debug an anchored pattern that matches nothing
printf " /VIOLATED\$/ : %s\n" "$(awk '/VIOLATED$/ { n++ } END { print n+0 }' dos.rpt)"
printf " /VIOLATED/ : %s\n" "$(awk '/VIOLATED/ { n++ } END { print n+0 }' dos.rpt)"
printf " /sum_reg[0]\$/ without strip : %s\n" "$(awk '/sum_reg\[0\]$/ { n++ } END { print n+0 }' dos.rpt)"
printf " /sum_reg[0]\$/ after strip : %s\n" "$(awk '{ sub(/\r$/, "") } /sum_reg\[0\]$/ { n++ } END { print n+0 }' dos.rpt)" /VIOLATED$/ : 0
/VIOLATED/ : 1
/sum_reg[0]$/ without strip : 0
/sum_reg[0]$/ after strip : 1Widening the pattern changed 0 to 1, which localised the fault to the anchor. The last two lines are the same anchored pattern with and without sub(/\r$/, "") — 0 then 1 — which is the fix confirmed rather than assumed.
The anchor has to match what is really at the end of the record, which is worth checking rather than assuming:
awk '{ sub(/\r$/, ""); printf " NR=%d ends with [%s]\n", NR, substr($0, length($0)-7) }' dos.rpt NR=1 ends with [m_reg[0]]
NR=2 ends with [ -0.018]So /VIOLATED$/ could never work on the second record even with the CR gone — that record ends with the slack value.
Exercise 4 — Debug a count that equals the record count
printf " records in the file : %s\n" "$(awk 'END { print NR }' aligned.txt)"
printf " /u_alu/ { n++ } : %s\n" "$(awk '/u_alu/ { n++ } END { print n+0 }' aligned.txt)"
printf " /u_alu/; { n++ } : %s\n" "$(awk '/u_alu/; { n++ } END { print n+0 }' aligned.txt)" records in the file : 2
/u_alu/ { n++ } : 1
/u_alu/; { n++ } : u_alu 178 1310.000
2The third line shows both symptoms at once: the matching record was printed (a pattern with no action prints $0), and the count that follows is 2, equal to NR. Either one alone identifies it.
Exercise 5 — Engineering application
A reusable first-look script, so the first step is one command rather than one typed command.
#!/bin/sh
# look.sh — the first thing to run when awk prints nothing or the wrong thing.
#
# Usage: sh look.sh <file> [n]
#
# Answers, in one pass: can the file be read, how many records and fields,
# are there carriage returns, and what do the first n records actually hold.
f=$1
n=${2:-3}
[ -n "$f" ] || { echo "usage: look.sh <file> [n]" >&2; exit 2; }
if [ ! -r "$f" ]; then echo "UNREADABLE: $f" >&2; exit 2; fi
awk -v n="$n" '
{ if (length($0) > 0 && substr($0, length($0)) == "\r") cr++
nf[NF]++
if (NR <= n) printf " NR=%d NF=%d len=%d [%s]\n", NR, NF, length($0), $0 }
END {
if (NR == 0) { print " EMPTY: 0 record(s) -- the file is readable and has no content"; exit 0 }
printf " %d record(s)\n", NR
printf " field counts seen:"
for (k in nf) printf " NF=%s(%d)", k, nf[k]
print ""
if (cr) printf " %d record(s) end with a CARRIAGE RETURN -- every anchored pattern will fail\n", cr
else print " no carriage returns"
}
' "$f"sh look.sh dos.rpt NR=1 NF=2 len=29 [Startpoint: u_alu/sum_reg[0]
]
NR=2 NF=3 len=28 [ slack (VIOLATED) -0.018
]
2 record(s)
field counts seen: NF=2(1) NF=3(1)
2 record(s) end with a CARRIAGE RETURN -- every anchored pattern will failsh look.sh empty.txt
sh look.sh no-such.txt 2>&1
echo "status: $?" EMPTY: 0 record(s) -- the file is readable and has no content
UNREADABLE: no-such.txt
status: 2Three distinct answers for three distinct situations, and the CRLF warning is explicit rather than something to notice. Note where the closing ] lands in the first two rows — captured to a file rather than written to a terminal, the carriage return becomes a visible line break, which is the one context in which a bracketed dump does reveal it. The for (k in nf) listing has no defined order and is acceptable here only because it is a diagnostic read by a person — a script that consumed it would need a sort.
Exercise 6 — Challenge: show that the field-count distribution is the single most informative line
look.sh prints a distribution of NF rather than one value. Show what that catches which a single NF does not.
printf 'a b c\nd e\nf g h i\n' > ragged.txt
awk '{ printf " NR=%d NF=%d\n", NR, NF }' ragged.txt
sh look.sh ragged.txt | tail -3 NR=1 NF=3
NR=2 NF=2
NR=3 NF=4
3 record(s)
field counts seen: NF=2(1) NF=3(1) NF=4(1)
no carriage returnsThree records, three different field counts. A script that read $3 would get a value, an empty string and a value — and the empty one is the record that silently produces nothing.
Chapter 11.3 measured the same thing on a formatted report whose rows had 9, 10 or 11 fields depending on optional annotations, and Chapter 13.3 Exercise 6 found it again in a script's own output. A single NF from the first record asserts that the file is regular; the distribution proves it, and when it is not regular the distribution says exactly how many records are affected.
So the most useful single line of a first look is not NF=3 — it is NF=2(1) NF=3(1) NF=4(1), because one value is a sample and a distribution is a measurement.
13. Summary
- Empty output has no diagnostic content. Four of the six causes produce output that looks reasonable.
- Step one is
NR,NFand[$0]— it answers whether records arrived, how many fields there are, and what is in them. - A bracketed dump cannot show a carriage return. Use
length($0)orcat -e; the CR breaks every anchored pattern (/VIOLATED$/matched 0 against/VIOLATED/'s 1). - Widen a pattern until it matches, then narrow back one change at a time.
NF=1on a file with visible columns meansFSis wrong. The default handles tabs; forcing-F'\t'on space-aligned text is the dangerous direction.- Echo the program. Double quotes turned
{ print $1 }into{ print }, which prints the whole line. - A semicolon after a pattern makes the next action unconditional — the count then equals
NR. - An empty file gives
NR=0; an unreadable one gives exit 2. Never conflate them. - A distribution of
NFbeats a singleNF. One value is a sample;NF=2(1) NF=3(1) NF=4(1)is a measurement.
Chapter 14.2 — Field and Delimiter Problems takes the cause this chapter could only point at: columns that line up on screen tell you nothing about what separates them, and a report whose format changed between tool versions changes NF without changing how it looks.
Where this fits
Part of the awk curriculum.