awk Part IV · Module 14 · AWK Debugging, Review, and Engineering Mastery
Field and Delimiter Problems
Columns that line up on screen tell you nothing about what separates them, and that is the whole difficulty: a tab-separated report and a space-aligned one look identical and behave identically under awk's default splitting, right up until someone adds a delimiter to be explicit and every field past the first goes empty. This chapter gives a two-command test that classifies any file's format without guessing, and covers the three ways a field index goes wrong — including the one where setting FS to a single space does not do what the syntax says.
Intermediate17 min readawkPart IVDebuggingFSFields
Part IV · Module 14 · Chapter 14.2 · Field and Delimiter Problems
1. Why This Is Hard to See
awk's default field splitting is forgiving in a way that hides the problem. It splits on runs of blanks — spaces and tabs — and ignores leading and trailing ones. So a tab-separated file and a space-aligned file give the same NF and the same $1, $2, $3, and a script written against one works on the other.
The trouble starts when someone makes the delimiter explicit. -F'\t' on a file that was only ever space-aligned gives NF=1, and -F' ' does something different again from what the syntax suggests.
2. Four Files That Look the Same
printf 'module\tcells\tarea\nu_alu\t178\t1310.000\nu_ctrl\t170\t1240.000\n' > v1.tsv
printf 'module cells area\nu_alu 178 1310.000\nu_ctrl 170 1240.000\n' > v2.txt
printf ' u_alu 178\n u_ctrl 170\n' > lead.txt
printf 'a\tb\tc\nd\te\tf\ng h i\n' > mixed.txt
printf 'a,b,c\n1,,3\n4,5,6\n' > holes.csv
printf 'a b\n' > two.txt
cat v1.tsv | sed 's/^/ /'
cat v2.txt | sed 's/^/ /' module cells area
u_alu 178 1310.000
u_ctrl 170 1240.000
module cells area
u_alu 178 1310.000
u_ctrl 170 1240.000v1.tsv is tab-separated and v2.txt is space-aligned with the same three columns. On a terminal with eight-column tab stops they can look very nearly identical — and under awk's default splitting they are:
printf " v1.tsv default FS: NF=%s \$2=[%s]\n" "$(awk 'NR==2 { print NF }' v1.tsv)" "$(awk 'NR==2 { print $2 }' v1.tsv)"
printf " v2.txt default FS: NF=%s \$2=[%s]\n" "$(awk 'NR==2 { print NF }' v2.txt)" "$(awk 'NR==2 { print $2 }' v2.txt)" v1.tsv default FS: NF=3 $2=[178]
v2.txt default FS: NF=3 $2=[178]Indistinguishable. Same NF, same $2. Any script written against either works on both — which is why the problem surfaces only when the format is asserted rather than assumed.
3. The Two-Distribution Test
The diagnostic is to take NF under the default splitting and under the delimiter you suspect, as distributions rather than single values:
for f in v1.tsv v2.txt mixed.txt; do
d=$(awk '{ print NF }' "$f" | sort -u | tr '\n' ',' | sed 's/,$//')
t=$(awk -F'\t' '{ print NF }' "$f" | sort -u | tr '\n' ',' | sed 's/,$//')
printf " %-10s default={%s} tab={%s} -> %s\n" "$f" "$d" "$t" \
"$([ "$t" = "1" ] && echo 'no tabs: space-aligned' || { [ "$d" = "$t" ] && echo 'tab-separated' || echo 'MIXED -- some rows have tabs, some do not'; })"
done v1.tsv default={3} tab={3} -> tab-separated
v2.txt default={3} tab={1} -> no tabs: space-aligned
mixed.txt default={3} tab={1,3} -> MIXED -- some rows have tabs, some do not4. FS Set to a Single Space Means the Default
printf " default NF=%s \$2=[%s]\n" "$(awk '{ print NF }' two.txt)" "$(awk '{ print $2 }' two.txt)"
printf " -F' ' NF=%s \$2=[%s]\n" "$(awk -F' ' '{ print NF }' two.txt)" "$(awk -F' ' '{ print $2 }' two.txt)"
printf " -F'[ ]' NF=%s \$2=[%s]\n" "$(awk -F'[ ]' '{ print NF }' two.txt)" "$(awk -F'[ ]' '{ print $2 }' two.txt)" default NF=2 $2=[b]
-F' ' NF=2 $2=[b]
-F'[ ]' NF=3 $2=[]The input is a b — two spaces. And:
-F' 'gives exactly the default behaviour. POSIX specifies that anFSof a single space means "split on runs of blanks, ignoring leading and trailing ones". It does not mean "split on one space character".-F'[ ]'does mean that. A single-character regular expression bypasses the special case, so the two spaces produce an empty field between them andNFis 3.
On the leading-whitespace file the difference is larger:
printf " default NF=%s \$1=[%s]\n" "$(awk 'NR==1 { print NF }' lead.txt)" "$(awk 'NR==1 { print $1 }' lead.txt)"
printf " -F' ' NF=%s \$1=[%s]\n" "$(awk -F' ' 'NR==1 { print NF }' lead.txt)" "$(awk -F' ' 'NR==1 { print $1 }' lead.txt)"
printf " -F'[ ]' NF=%s \$1=[%s]\n" "$(awk -F'[ ]' 'NR==1 { print NF }' lead.txt)" "$(awk -F'[ ]' 'NR==1 { print $1 }' lead.txt)" default NF=2 $1=[u_alu]
-F' ' NF=2 $1=[u_alu]
-F'[ ]' NF=4 $1=[]NF=4 with $1 empty. The record is u_alu 178 — two leading spaces, which under a literal single-space split are two separators producing two empty fields before u_alu lands in $3.
So the practical rule has two halves:
- To split on whitespace, use the default and do not write
-F' 'thinking it is more explicit. It is identical and it misleads the next reader. - To split on exactly one space, write
-F'[ ]'— and then handle the empty fields that leading whitespace produces.
5. An Empty Field Means Two Different Things
printf " -F, NF=%s \$2=[%s]\n" "$(awk -F, 'NR==2 { print NF }' holes.csv)" "$(awk -F, 'NR==2 { print $2 }' holes.csv)"
printf " default FS NF=%s \$2=[%s]\n" "$(awk 'NR==2 { print NF }' holes.csv)" "$(awk 'NR==2 { print $2 }' holes.csv)" -F, NF=3 $2=[]
default FS NF=1 $2=[]Both report an empty $2 and only one of them has a second field. NF is what separates them, and the record is 1,,3:
awk 'NR==2 { printf " raw record [%s], length %d\n", $0, length($0) }' holes.csv
printf " -F, fields: "; awk -F, 'NR==2 { for (i=1;i<=NF;i++) printf "[%s]", $i; print "" }' holes.csv
printf " default FS fields: "; awk 'NR==2 { for (i=1;i<=NF;i++) printf "[%s]", $i; print "" }' holes.csv raw record [1,,3], length 4
-F, fields: [1][][3]
default FS fields: [1,,3]-F, gives three fields with the middle one genuinely empty. The default gives one field containing the whole record, because no comma is whitespace — so its $2 is empty for the entirely different reason that there is no $2.
The distinction matters because an empty $2 has two possible meanings:
- With a non-whitespace
FS— the field exists and holds nothing. On a coverage CSV that is "no score", which Chapter 13.3 kept distinct from a score of zero. - With a whitespace
FS— there is no field 2 at all, because whitespace splitting cannot produce an empty field.
NF is what separates them, and comparing NF against the expected field count is the check — which is why Chapter 13.3's reader asserted NF != want on every record.
6. Common Mistakes
Assuming aligned columns are delimited. A space-aligned file and a tab-separated one are identical under the default FS — section 2.
Adding -F'\t' to be explicit. On a space-aligned file that gives NF=1.
Taking NF from one record. The distribution is the measurement; tab={1,3} is a mixed file.
Writing -F' ' for clarity. It is exactly the default and it misleads — section 4.
Expecting -F'[ ]' to be harmless. Leading whitespace then produces empty leading fields.
Reading an empty $2 as an empty field. Under a whitespace FS it means there is no field 2 — section 5.
Not asserting NF. It is the only thing that distinguishes the two cases.
Trusting a format across tool versions. The mixed distribution is what a version change looks like.
7. Exercises
Exercise 1 — Classify a file's format
for f in v1.tsv v2.txt holes.csv; do
printf " %-10s default={%s} tab={%s} comma={%s}\n" "$f" \
"$(awk '{ print NF }' "$f" | sort -u | tr '\n' ',' | sed 's/,$//')" \
"$(awk -F'\t' '{ print NF }' "$f" | sort -u | tr '\n' ',' | sed 's/,$//')" \
"$(awk -F, '{ print NF }' "$f" | sort -u | tr '\n' ',' | sed 's/,$//')"
done v1.tsv default={3} tab={3} comma={1}
v2.txt default={3} tab={1} comma={1}
holes.csv default={1} tab={1} comma={3}Three files, three distinct fingerprints. The separator whose distribution is not {1} and is uniform is the real one.
Exercise 2 — The single-space special case
printf 'x y\n' > three.txt
for spec in "default" "-F' '" "-F'[ ]'" "-F'[ ]+'"; do
case $spec in
default) printf " %-10s NF=%s\n" "$spec" "$(awk '{ print NF }' three.txt)" ;;
"-F' '") printf " %-10s NF=%s\n" "$spec" "$(awk -F' ' '{ print NF }' three.txt)" ;;
"-F'[ ]'") printf " %-10s NF=%s\n" "$spec" "$(awk -F'[ ]' '{ print NF }' three.txt)" ;;
*) printf " %-10s NF=%s\n" "$spec" "$(awk -F'[ ]+' '{ print NF }' three.txt)" ;;
esac
done default NF=2
-F' ' NF=2
-F'[ ]' NF=4
-F'[ ]+' NF=2Three spaces between the fields. -F'[ ]' finds four fields (two empty); -F'[ ]+' collapses the run and finds two, matching the default.
Exercise 3 — Debug every field after the first going empty
printf " -F'\\t' on v2.txt : NF=%s \$2=[%s] \$3=[%s]\n" "$(awk -F'\t' 'NR==2 { print NF }' v2.txt)" "$(awk -F'\t' 'NR==2 { print $2 }' v2.txt)" "$(awk -F'\t' 'NR==2 { print $3 }' v2.txt)"
printf " tabs in the file : %s\n" "$(awk '/\t/ { n++ } END { print n+0 }' v2.txt)" -F' ' on v2.txt : NF=1 $2=[] $3=[]
tabs in the file : 0NF=1 and zero records containing a tab. The file has none, so the delimiter was asserted rather than observed.
Exercise 4 — Debug a file that changed format mid-stream
awk '{ printf " NR=%d default NF=%d\n", NR, NF }' mixed.txt
awk -F'\t' '{ printf " NR=%d tab NF=%d%s\n", NR, NF, (NF == 1 ? " <- no tab on this record" : "") }' mixed.txt NR=1 default NF=3
NR=2 default NF=3
NR=3 default NF=3
NR=1 tab NF=3
NR=2 tab NF=3
NR=3 tab NF=1 <- no tab on this recordUniform under the default splitting and non-uniform under the tab, which localises the inconsistency to record 3. A parser using -F'\t' would silently produce an empty $2 and $3 for that one record.
Exercise 5 — Engineering application
#!/bin/sh
# fsprobe.sh — identify a file's real field separator before writing a parser.
#
# Usage: sh fsprobe.sh <file>
#
# Reports the NF distribution under each candidate separator. A distribution of
# {1} means the separator is absent; a non-uniform one means the file is
# inconsistent; uniform and greater than 1 means the separator is PLAUSIBLE --
# Exercise 6 shows why that is not the same as correct.
f=$1
[ -r "$f" ] || { echo "UNREADABLE: $f" >&2; exit 2; }
printf "%-12s %-14s %s\n" "separator" "NF seen" "verdict"
for name in default tab comma semicolon pipe; do
case $name in
default) d=$(awk '{ print NF }' "$f" | sort -un | tr '\n' ',' | sed 's/,$//') ;;
tab) d=$(awk -F'\t' '{ print NF }' "$f" | sort -un | tr '\n' ',' | sed 's/,$//') ;;
comma) d=$(awk -F, '{ print NF }' "$f" | sort -un | tr '\n' ',' | sed 's/,$//') ;;
semicolon) d=$(awk -F';' '{ print NF }' "$f" | sort -un | tr '\n' ',' | sed 's/,$//') ;;
pipe) d=$(awk -F'|' '{ print NF }' "$f" | sort -un | tr '\n' ',' | sed 's/,$//') ;;
esac
case $d in
1) v="separator absent" ;;
*,*) v="INCONSISTENT -- the file is not uniform" ;;
"") v="no records" ;;
*) v="uniform at $d field(s)" ;;
esac
printf "%-12s %-14s %s\n" "$name" "{$d}" "$v"
donesh fsprobe.sh v1.tsvseparator NF seen verdict
default {3} uniform at 3 field(s)
tab {3} uniform at 3 field(s)
comma {1} separator absent
semicolon {1} separator absent
pipe {1} separator absentsh fsprobe.sh mixed.txtseparator NF seen verdict
default {3} uniform at 3 field(s)
tab {1,3} INCONSISTENT -- the file is not uniform
comma {1} separator absent
semicolon {1} separator absent
pipe {1} separator absentThe second run is the diagnosis section 3 arrived at, now in one command: uniform under the default splitting and inconsistent under the tab, which is a file where some rows are tab-separated and some are not. A parser using -F'\t' would silently produce empty fields for the rows without one.
Exercise 6 — Challenge: show that "uniform and greater than 1" is not sufficient
fsprobe.sh calls a separator plausible when its distribution is uniform and above 1. Construct a file where three candidates all pass that test.
printf 'a,b\tc,d\te,f\n1,2\t3,4\t5,6\n' > ambig.txt
cat -t ambig.txt | sed 's/^/ /'
for name in default tab comma; do
case $name in
default) d=$(awk '{ print NF }' ambig.txt | sort -un | tr '\n' ',' | sed 's/,$//') ;;
tab) d=$(awk -F'\t' '{ print NF }' ambig.txt | sort -un | tr '\n' ',' | sed 's/,$//') ;;
comma) d=$(awk -F, '{ print NF }' ambig.txt | sort -un | tr '\n' ',' | sed 's/,$//') ;;
esac
printf " %-8s {%s}\n" "$name" "$d"
done a,b^Ic,d^Ie,f
1,2^I3,4^I5,6
default {3}
tab {3}
comma {4}Three distributions, all uniform, all greater than one. Every candidate passes the test and only one is right — the file is three tab-separated columns, each of which happens to contain a comma.
The distributions cannot settle it. The field contents can:
printf " -F tab fields of row 2: "; awk -F'\t' 'NR==2 { for (i=1;i<=NF;i++) printf "[%s]", $i; print "" }' ambig.txt
printf " -F, fields of row 2: "; awk -F, 'NR==2 { for (i=1;i<=NF;i++) printf "[%s]", $i; print "" }' ambig.txt -F tab fields of row 2: [1,2][3,4][5,6]
-F, fields of row 2: [1][2 3][4 5][6]The comma reading produces fields that contain tabs — [2 3] and [4 5]. A field holding what you believe to be the record separator is the signature of a wrong separator, and it is visible only by printing the fields.
So the probe narrows the candidates and does not choose among them, which is the honest description of what a distribution can do. The rule that completes it:
for fs in tab comma; do
printf " -F %-6s fields containing the OTHER candidate: " "$fs"
case $fs in
tab) awk -F'\t' '{ for (i=1;i<=NF;i++) if ($i ~ /,/) n++ } END { print n+0 }' ambig.txt ;;
comma) awk -F, '{ for (i=1;i<=NF;i++) if ($i ~ /\t/) n++ } END { print n+0 }' ambig.txt ;;
esac
done -F tab fields containing the OTHER candidate: 6
-F comma fields containing the OTHER candidate: 4Both readings leave the other candidate inside fields, so even this does not decide it mechanically — which is the point. A separator is a fact about the format, not about the bytes, and the only thing that settles it is the producer: the tool's documentation, its -delimiter option, or a header row naming the columns. Chapter 12.4's conclusion was that a delimited format should carry its own validity conditions; this is why — a reader given only the bytes cannot always recover the format, and a header plus a declared field count removes the ambiguity entirely.
8. Summary
- A space-aligned file and a tab-separated one are identical under the default FS — same
NF, same fields. - Two NF distributions classify the format: uniform and greater than 1 is the real separator,
{1}means it is absent, and a non-uniform one means the file is inconsistent. default={3}for all three formats. Only the candidate-separator distribution separates them.FSset to a single space is special — it means the default whitespace splitting, not "one space".-F'[ ]'does mean one space, and on leading whitespace it produces empty leading fields (NF=4,$1empty).-F'[ ]+'collapses runs and matches the default.- An empty
$2means two opposite things: an empty field under a non-whitespace FS, or no second field at all under a whitespace one.NFseparates them. - Whitespace splitting cannot produce an empty field. That is what makes
NFthe discriminator. - A fixture that changes between two commands makes their outputs incomparable — regenerate or read back.
Chapter 14.3 — Regular Expression Problems takes the other half of a failed match: a missing anchor, an unescaped metacharacter, a dot that matched a literal character, and a shell that consumed a backslash before awk ever saw it.
Where this fits
Part of the awk curriculum.