Field-Aware Processing with awk

Part I · Module 13 · Chapter 13.7 · Field-Aware Processing with awk

Chapter 13.2 hit two limits and named the tool that passes them. Chapter 13.6 hit a third. This is it.

1. Three Jobs

2. Fields

$ awk -F, '{print $1, $3}' results.csv
test result
alu PASS
fifo FAIL

$1, $2, $3 are the fields; -F sets the separator. $0 is the whole line.

And immediately, the thing cut cannot do:

$ awk -F, '{print $3, $1}' results.csv
result test
PASS alu
FAIL fifo

Reordered. awk addresses fields individually, so they can be emitted in any order, repeated, or mixed with literal text.

$ awk -F, 'NR>1 {print $1 " took " $4 " cycles"}' results.csv
alu took 1200 cycles
fifo took 340 cycles
cache took 98000 cycles

Adjacent strings are concatenated — there is no operator for it, which looks odd and is correct.

Two useful built-in variables

$ awk -F, '{print NF}' results.csv | head -1
4

$ awk -F, '{print $NF}' results.csv | head -2
cycles
1200

$ awk -F, 'NR==3 {print $1}' results.csv
fifo

$NF is the one you will use. It picks the last column without knowing how many there are, which matters for log lines whose field count varies.

NR>1 is how you skip a header, and it appears in almost every real awk invocation on a CSV.

The default separator fixes chapter 13.2's problem

$ printf 'alu    7   PASS\n' | awk '{print $2}'
7

With no -F, awk splits on runs of whitespace and ignores leading and trailing space. That is the behaviour a person reading aligned columns expects, and it is why awk works on tool output where cut returns nothing.

3. Filtering

$ awk -F, '$3=="FAIL" {print $1}' results.csv
fifo
alu

A condition before the braces selects which lines the action runs on. This is awk's central idea: a program is a list of pattern { action } pairs, and each line is tested against each pattern.

Either part can be omitted:

awk -F, '$3=="FAIL"'                  # condition only — print matching lines
awk -F, '{print $1}'                  # action only — runs on every line
awk -F, '$3=="FAIL" {print $1}'       # both

With no action, the default is to print the line. So awk -F, '$3=="FAIL"' is a field-aware grep:

$ awk -F, '$3=="FAIL"' results.csv
fifo,9,FAIL,340
alu,11,FAIL,560

That matters more than it looks. grep FAIL results.csv would also match a test named fifo_FAIL_test, or a comment containing the word. Testing $3 exactly cannot.

Conditions

awk -F, '$4 > 1000'                   # numeric comparison
awk -F, '$3 == "PASS"'                # string equality
awk -F, '$1 ~ /^alu/'                 # regex match on a field
awk -F, '$1 !~ /^test/'               # regex non-match
awk -F, 'NR > 1 && $4 > 1000'         # combined
awk     '/ERROR/'                     # regex against the whole line
awk -F, 'NF != 4'                     # lines with the wrong field count

That last one is a cheap data-quality check — it finds the malformed rows in a file that is supposed to be uniform, which is often the first thing to do with data somebody sent you.

4. Accumulating

This is the job nothing else in the toolkit does at all.

$ awk -F, '$3=="FAIL" {n++} END {print n+0}' results.csv
2

END { } runs once, after every line has been read. Variables persist across lines, so n++ counts and END reports.

$ awk -F, 'NR>1 {s+=$4} END {print s}' results.csv
102400

$ awk -F, 'NR>1 {s+=$4; n++} END {printf "%.1f\n", s/n}' results.csv
20480.0

Sum and average in one pass. No temporary file, no second read, no shell loop.

There is a matching BEGIN { } that runs before the first line — used for setting variables and printing headers:

$ awk -F, 'BEGIN{OFS=" | "} NR==2 {print $1,$3}' results.csv
alu | PASS

OFS is the output field separator, used wherever you write print a,b with a comma. (The comma means "join with OFS"; a space between them means "concatenate with nothing".)

Per-key totals

$ awk -F, 'NR>1 {t[$3]+=$4} END {for (k in t) print k"="t[k]}' results.csv | sort
FAIL=900
PASS=101500

t[$3] is an associative array keyed by whatever is in field 3. This is the thing that replaces sort | uniq -c when you need to count by a field rather than a whole line (Chapter 13.4), and it needs no sorted input.

awk '{t[$1]++} END {for (k in t) print t[k], k}' access.log | sort -rn | head

Counting by the first field, ranked. One pass, one process.

Note the | sort on the output: for (k in t) iterates in an unspecified order. It is not insertion order and not sorted order. Always sort the output if the order matters.

5. Quoting

awk -F, '$3=="FAIL" {print $1}' results.csv

Single quotes around the whole program, always. The script is full of $ and " characters that belong to awk, and Chapter 9.3's "quote by owner" rule applies exactly:

$ awk -F, "{ print \$2 }" results.csv

That works and is miserable. Worse:

$ awk -F, "{ print $2 }" results.csv

The shell expands $2 — usually to nothing — and awk receives { print }, which prints whole lines. No error, wrong output.

To get a shell value into awk, use -v:

threshold=1000
awk -F, -v t="$threshold" '$4 > t {print $1}' results.csv

-v name=value sets an awk variable before the program runs. The program stays single-quoted and the value crosses cleanly, whatever it contains.

6. When to Use It, and When Not

The replacements table, since several tools overlap:

7. In Practice

# a results summary in one pass
awk -F, 'NR>1 {
  total++
  if ($3 == "PASS") pass++; else fail++
  cycles += $4
}
END {
  printf "total  : %d\n", total
  printf "passed : %d (%.1f%%)\n", pass+0, 100*pass/total
  printf "failed : %d\n", fail+0
  printf "cycles : %d (avg %.0f)\n", cycles, cycles/total
}' results.csv

That is four lines of body and five of report, and it replaces a shell loop that would need a separate grep per count and a separate pass per total.

Shorter ones that earn their place:

# the slowest ten tests
awk -F, 'NR>1 {print $4, $1}' results.csv | sort -rn | head -10

# total simulation time from a log of durations
awk '/SIMULATION TIME/ {s += $3} END {printf "%.1f hours\n", s/3600}' */sim.log

# malformed rows in a file that should be uniform
awk -F, 'NF != 4 {print FILENAME ":" NR ": " NF " fields"}' *.csv

# which user owns the most processes
ps -eo user= | awk '{t[$1]++} END {for (u in t) print t[u], u}' | sort -rn | head -5

# a filelist with a prefix added to every line
awk -v root="$PROJECT" '{print root "/" $0}' filelist.f

FILENAME in the third one is another built-in — the name of the file currently being read, which is what makes a check across many files report where the problem is.

Practice

mkdir -p /tmp/awkd && cd /tmp/awkd
cat > results.csv <<'EOF'
test,seed,result,cycles
alu,7,PASS,1200
fifo,9,FAIL,340
cache,3,PASS,98000
alu,11,FAIL,560
uart,5,PASS,2300
EOF
printf 'alu    7   PASS\nfifo   9   FAIL\n' > spaced.txt

1 — Pick and reorder.

awk -F, '{print $1, $3}' results.csv
awk -F, '{print $3, $1}' results.csv
cut -d, -f3,1 results.csv

The third does not reorder. That is the whole reason for this chapter.

2 — Use the built-ins.

awk -F, '{print NF}' results.csv | head -1
awk -F, '{print $NF}' results.csv | head -3
awk -F, 'NR==3 {print $1}' results.csv
awk -F, 'NR>1 {print $1}' results.csv

3 — Beat cut on aligned columns.

cut -d' ' -f2 spaced.txt
awk '{print $2}' spaced.txt

4 — Filter on a field.

awk -F, '$3=="FAIL"' results.csv
awk -F, '$4 > 1000 {print $1, $4}' results.csv
awk -F, '$1 ~ /^alu/ {print $0}' results.csv
awk -F, 'NF != 4' results.csv

5 — Count, and meet the empty-counter trap.

awk -F, '$3=="FAIL" {n++} END {print n+0}' results.csv
awk -F, '$3=="NOPE" {n++} END {print n}'  results.csv
awk -F, '$3=="NOPE" {n++} END {print n+0}' results.csv

The middle one prints a blank line.

6 — Sum and average.

awk -F, 'NR>1 {s+=$4} END {print s}' results.csv
awk -F, 'NR>1 {s+=$4; n++} END {printf "%.1f\n", s/n}' results.csv

7 — Count by key.

awk -F, 'NR>1 {t[$3]++} END {for (k in t) print t[k], k}' results.csv | sort -rn

8 — Get a shell value in, properly.

threshold=1000
awk -F, -v t="$threshold" 'NR>1 && $4 > t {print $1}' results.csv
awk -F, "NR>1 && \$4 > $threshold {print \$1}" results.csv

Both work; one is readable.

9 — Watch the quoting failure.

awk -F, '{print $2}' results.csv | head -2
awk -F, "{print $2}" results.csv | head -2

The second prints whole lines, because the shell ate $2.

10 — Clean up. cd /tmp && rm -rf /tmp/awkd

Check Your Understanding

1. What are the three jobs awk is unmatched at?

2. Why does awk '{print $2}' work on space-aligned output where cut -d' ' -f2 does not?

3. What does awk -F, '$3=="FAIL"' do with no action block?

4. awk '/X/ {n++} END {print n}' prints a blank line when nothing matched. Why, and what is the fix?

5. awk -F, "{print $2}" prints whole lines. What happened?

6. When should you not use awk?

8. Remember

• $1, $2, $NF are fields; $0 is the whole line; -F sets the separator.
• It can reorder fields, which is the thing cut cannot do at all.
• With no -F it splits on runs of whitespace, which is why it works on aligned output.
• NF is the field count, NR the line number, and NR>1 skips a header.
• A program is pattern { action } pairs. Omit the action to print matching lines; omit the pattern to act on all of them.
• Conditions can test a specific field — $3=="FAIL" cannot match the word elsewhere on the line, which grep can.
• NF != 4 finds malformed rows — the cheapest data-quality check there is.
• END { } runs once at the end, and variables persist across lines. That is how you sum, count and average in one pass.
• n+0 prints zero for a counter that never incremented; bare n prints an empty line.
• t[$1]++ counts by field with no sort needed, and for (k in t) iterates in unspecified order — sort the output.
• OFS sets the output separator, used where print a,b has a comma.
• Single-quote the program and pass shell values with -v. Double quotes let the shell eat your field references.
• Stop at about three lines. Beyond that, the problem wants a real script.
• A BEGIN header plus printf "%-14s" is what turns a list into a table — Chapter 24.6 builds the counts, lists and tables a question actually needs.

Two chapters remain. The next combines two sources into one table — and contains a tool that gives a wrong answer rather than an error when its input is not sorted.

• awk stops being the right tool at a measurable point. Chapter 35.4 finds it: the shell version of one report read its input thirty-one times to Python's once, and the cost is rows × groups × metrics.