awk Inside Bash Scripts

Part III · Module 12 · Chapter 12.1 · awk Inside Bash Scripts

1. Why This Matters

Everything in Modules 9 through 11 was a command. A flow needs those commands inside scripts, and that crossing introduces three problems that have nothing to do with awk's language:

• Two languages share $. The shell expands it before awk ever sees the program.
• Only text crosses the boundary. A count becomes a string, and an absent result becomes an empty string.
• Only one number comes back. The exit status is the script's whole view of whether the extraction worked.

The third is where flows break silently, and section 6 is the measurement.

2. The Log

UVM_INFO @ 0: reporter [RNTST] Running test alu_smoke_test...
UVM_INFO ./env/scoreboard.sv(142) @ 1200: uvm_test_top.env.sb [MATCH] expected=32'hA5 got=32'hA5
UVM_ERROR ./env/scoreboard.sv(148) @ 2400: uvm_test_top.env.sb [MISMATCH] expected=32'h5A got=32'hFF
UVM_INFO ./env/scoreboard.sv(142) @ 3600: uvm_test_top.env.sb [MATCH] expected=32'h11 got=32'h11
UVM_ERROR ./env/driver.sv(88) @ 4800: uvm_test_top.env.agt.drv [TIMEOUT] no response within 100 cycles
UVM_INFO @ 5000: reporter [TEST_DONE]
--- UVM Report Summary ---
UVM_INFO :    4
UVM_WARNING :    0
UVM_ERROR :    2
UVM_FATAL :    0

Two error records, and a summary line that also begins with UVM_ERROR.

3. Single-Quote the Program

awk '/^UVM_ERROR/ { n++ } END { print n+0 }' sim.log
awk "/^UVM_ERROR/ { n++ } END { print n+0 }" sim.log

3
3

Both forms agree, because that program contains no $. Add one and they diverge:

echo "  the program the shell actually hands awk:"
printf '  [%s]\n' "$(echo "{ print $1 }")"
awk "{ print $1 }" sim.log | wc -l | awk '{ printf "  it printed %d line(s) -- the whole file\n", $1 }'
awk '{ print $1 }' sim.log | wc -l | awk '{ printf "  the single-quoted form printed %d line(s) too, but of field 1 only\n", $1 }'

  the program the shell actually hands awk:
  [{ print  }]
  it printed 11 line(s) -- the whole file
  the single-quoted form printed 11 line(s) too, but of field 1 only

4. Pass Shell Values With -v

sev=UVM_ERROR
awk -v s="$sev" '$1 == s { n++ } END { printf "%s count = %d\n", s, n+0 }' sim.log

UVM_ERROR count = 3

The program stays single-quoted and the value arrives as data. Splicing it into the program text happens to give the same answer here:

sev=UVM_ERROR
awk "\$1 == \"$sev\" { n++ } END { printf \"spliced: %d\n\", n+0 }" sim.log

spliced: 3

— and it is the construct Chapter 8.4 identified as code injection: a value containing " or { becomes program syntax rather than data. -v cannot do that, which is the reason to prefer it even when the value is one you chose yourself.

-v has one limitation worth remembering: it processes escape sequences, so a value containing a backslash arrives altered. ENVIRON["name"] is the backslash-safe route, as Chapter 8.4 measured.

5. The Count Depends on the Pattern

Before wiring anything into a script, the number has to be right. Chapter 9.3's finding, now inside a capture:

errors=$(awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { print n+0 }' sim.log)
echo "  errors=[$errors]"
echo "  the naive pattern gives: [$(awk '/UVM_ERROR/ { n++ } END { print n+0 }' sim.log)]"
echo "  the log's own summary says: [$(awk '/^UVM_ERROR :/ { print $3 }' sim.log)]"

  errors=[2]
  the naive pattern gives: [3]
  the log's own summary says: [2]

Two against three, with the log's own summary agreeing on two. The naive pattern counts the summary line UVM_ERROR : 2, which both contains and begins with UVM_ERROR — so anchoring alone does not fix it and the record shape has to be matched. /^UVM_ERROR (\.|\/|@)/ requires a path, a ./ or an @ after the severity, which a summary line does not have.

That the log reports its own count is free ground truth, and a script should use it:

awk '/^UVM_ERROR (\.|\/|@)/ { n++ } /^UVM_ERROR :/ { s = $3 } END { printf "  parsed %d, log says %d -> %s\n", n+0, s+0, (n+0 == s+0 ? "AGREE" : "DISAGREE") }' sim.log

  parsed 2, log says 2 -> AGREE

A disagreement here means the pattern is wrong, which is a class of error no arithmetic check can find — Chapter 9.2 section 7's point.

6. An Empty Capture Equals Zero

This is the section that matters.

good=$(awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { print n+0 }' sim.log)
missing=$(awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { print n+0 }' no-such.log 2>/dev/null)
printf "  from a real log:    [%s]\n  from a missing log: [%s]   <- empty, and \$((missing+0)) is %d\n" "$good" "$missing" "$((missing+0))"

  from a real log:    [2]
  from a missing log: []   <- empty, and $((missing+0)) is 0

A log that does not exist captures as the empty string, and the empty string is zero in shell arithmetic. So if [ "$missing" -eq 0 ] passes, if (( missing == 0 )) passes, and a flow that treats zero errors as a pass reports a clean run for a test that never produced a log at all.

That is Chapter 9.5's finding — a record-driven table omits the test whose log is empty — arriving at the shell boundary, where it is worse, because there is no row to be missing. There is a variable holding a number.

awk's exit status does distinguish the two:

if awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { print n+0; exit (n == "") }' no-such.log >/dev/null 2>&1; then echo "  missing log: awk reported success"; else echo "  missing log: awk exit $? -- the file could not be read"; fi

  missing log: awk exit 2 -- the file could not be read

Exit 2. awk cannot open the file, says so on stderr, and returns a non-zero status — which is the only signal that crosses the boundary intact.

So the rule for any capture: check the status before using the value.

for f in sim.log no-such.log; do
  if errors=$(awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { print n+0 }' $f 2>/dev/null); then
    printf "  %-12s read, %s error(s)\n" "$f" "$errors"
  else
    printf "  %-12s COULD NOT READ IT (awk exit %s)\n" "$f" "$?"
  fi
done

  sim.log      read, 2 error(s)
  no-such.log  COULD NOT READ IT (awk exit 2)

Both branches, side by side: the if tests awk's status, so the unreadable log takes the else rather than contributing a zero.

7. A Deliberate Exit Status, and Where It Goes

awk's exit takes a value, so the extraction can carry its own verdict:

awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { exit (n > 0) }' sim.log; echo "  awk exit with errors present: $?"
awk '/NOTHING_MATCHES_THIS/ { n++ } END { exit (n > 0) }' sim.log; echo "  awk exit with none:          $?"

  awk exit with errors present: 1
  awk exit with none:          0

exit (n > 0) exits 1 when anything matched and 0 otherwise — the grep convention, which composes with if and && without the script needing to parse a number.

And then the trap:

awk 'END { exit 1 }' sim.log | cat
echo "  \$? after the pipe = $?   (cat's)"
printf '  pipestatus = [%s]   PIPESTATUS = [%s]\n' "${pipestatus[1]-unset}" "${PIPESTATUS[0]-unset}"

  $? after the pipe = 0   (cat's)
  pipestatus = [unset]   PIPESTATUS = [0]

$? is 0 and awk exited 1. After a pipeline, $? is the last command's status, and cat succeeded.

The second line is why this chapter does not recommend the usual workaround. pipestatus is a zsh array and PIPESTATUS a bash one; this block runs under /bin/sh, where neither exists — pipestatus[1] is unset and PIPESTATUS[0] reads as a bare 0 that means nothing. A script with #!/bin/sh cannot use either, and a script that does use one is pinned to the shell it was written in.

The portable remedy is to not put awk last in a pipeline when its status matters:

awk 'END { exit 1 }' sim.log > out.txt; st=$?
echo "  redirect instead of pipe: awk status = $st"

  redirect instead of pipe: awk status = 1

A redirect writes the same bytes to the same place and leaves $? as awk's. When a genuine downstream stage is needed — a sort, a tee — run awk into a file first and pipe the file, or set set -o pipefail in a shell that has it and accept the shell dependency explicitly.

One more boundary detail, verified because it bites when a program emits more than one line:

v=$(awk 'BEGIN { printf "7\n\n\n" }')
printf "  captured [%s] length %d  (three newlines emitted)\n" "$v" "${#v}"

  captured [7] length 1  (three newlines emitted)

Command substitution strips every trailing newline. Convenient for a single value, and a reason a multi-line result should go to a file or be read with a loop rather than into one variable.

8. The Shape of a Correct Wrapper

#!/bin/sh
# checklog.sh — count real UVM_ERROR records in a simulation log.
#
# Usage:  sh checklog.sh sim.log
# Exit:   0 the log was read and holds no errors
#         1 the log was read and holds errors
#         2 the log could not be read, or the parse disagrees with the log

log=$1
if [ -z "$log" ]; then echo "usage: checklog.sh <log>" >&2; exit 2; fi

# single-quoted program; the filename arrives as an argument, not spliced in.
# awk's own exit status is the verdict: 0 clean, 1 errors, 2 unreadable or
# inconsistent with the log's own summary.
awk '
    /^UVM_ERROR (\.|\/|@)/ { n++ }
    /^UVM_ERROR :/         { s = $3 }
    END {
        if (NR == 0) { print "ERROR: log is empty" > "/dev/stderr"; exit 2 }
        if (s != "" && n + 0 != s + 0) {
            printf "ERROR: parsed %d error(s) but the log summary says %d\n", n + 0, s + 0 > "/dev/stderr"
            exit 2
        }
        printf "%d error(s) in %s\n", n + 0, FILENAME
        exit (n > 0)
    }
' "$log" || status=$?

status=${status:-0}
case $status in
  0) echo "PASS: no errors" ;;
  1) echo "FAIL: errors present" ;;
  *) echo "BROKEN: could not evaluate $log" ;;
esac
exit $status

sh checklog.sh sim.log
echo "status: $?"

2 error(s) in sim.log
FAIL: errors present
status: 1

sh checklog.sh no-such.log 2>/dev/null
echo "status: $?"

BROKEN: could not evaluate no-such.log
status: 2

Three outcomes rather than two. FAIL and BROKEN are different events with different owners — one goes to whoever debugs the design, the other to whoever runs the flow. Chapter 12.6 section 7 takes this to its full form, where six states are needed rather than three and a two-state reduction gets two of five verdicts wrong.

9. Common Mistakes

Double-quoting the program. $1 becomes a shell parameter and the result is a different working program — section 3.

Splicing a shell value into the program. Code injection; use -v — section 4.

-v with a backslash in the value. It processes escapes; use ENVIRON.

Trusting a capture without checking the status. A missing log captures as empty and is zero in arithmetic — section 6.

$? after a pipeline. That is the last command's status; awk's is in pipestatus — section 7.

Reading pipestatus after another command. Copy the array immediately.

Assuming pipestatus is 0-based. It is 1-based in zsh; bash's PIPESTATUS is 0-based.

Capturing multi-line output into one variable. Trailing newlines are stripped and the rest is one blob.

Collapsing "errors" and "could not run" into one failure. They have different owners — section 8.

10. Exercises

Exercise 1 — A counted capture

errors=$(awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { print n+0 }' sim.log)
echo "errors=$errors"

errors=2

Exercise 2 — A shell value passed as data

for sev in UVM_INFO UVM_ERROR UVM_FATAL; do
  printf "%-12s " "$sev"
  awk -v s="$sev" '$1 == s && /(\.|\/|@)/ { n++ } END { print n+0 }' sim.log
done

UVM_INFO     4
UVM_ERROR    2
UVM_FATAL    0

Exercise 3 — Debug a clean report from a test that never ran

A nightly flow reports zero errors for a test whose log was never written.

missing=$(awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { print n+0 }' no-such.log 2>/dev/null)
printf "captured [%s]  -> \$((missing+0)) = %d  -> reads as a clean run\n" "$missing" "$((missing+0))"

captured []  -> $((missing+0)) = 0  -> reads as a clean run

The capture is empty because awk never ran successfully, and empty is zero. Checking the status instead:

if awk '/^UVM_ERROR/ { n++ } END { print n+0 }' no-such.log >/dev/null 2>&1; then echo "awk succeeded"; else echo "awk exit $? -- the log could not be read"; fi

awk exit 2 -- the log could not be read

Exercise 4 — Debug a lost exit status

A wrapper reports success although awk signalled errors.

awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { exit (n > 0) }' sim.log | cat
printf "with a pipe: \$? = %s\n" "$?"

with a pipe: $? = 0

The | cat was added to tee output somewhere and it replaced the status the script branches on. Without the pipe:

awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { exit (n > 0) }' sim.log
printf "no pipe: \$? = %s\n" "$?"

no pipe: $? = 1

Or with a redirect, which keeps the output and the status:

awk '/^UVM_ERROR (\.|\/|@)/ { n++ } END { exit (n > 0) }' sim.log > /dev/null
printf "redirected: \$? = %s\n" "$?"

redirected: $? = 1

Exercise 5 — Engineering application

checklog.sh from section 8, exercised on the three cases it distinguishes. Two variants of the log first — one whose summary contradicts its records, and one whose records have been stripped while the summary remains:

sed 's/^UVM_ERROR :    2/UVM_ERROR :    5/' sim.log > inconsistent.log
grep -v '^UVM_ERROR \./' sim.log > clean.log
for f in sim.log inconsistent.log clean.log; do
  printf "  %-18s " "$f"
  awk '/^UVM_ERROR (\.|\/|@)/ { n++ } /^UVM_ERROR :/ { s = $3 } END { printf "records %d, summary %s -> %s\n", n+0, (s == "" ? "none" : s), (s != "" && n+0 != s+0 ? "DISAGREE" : "agree") }' $f
done

  sim.log            records 2, summary 2 -> agree
  inconsistent.log   records 2, summary 5 -> DISAGREE
  clean.log          records 0, summary 2 -> DISAGREE

clean.log is the interesting one: grep -v '^UVM_ERROR \./' removed the two error records and left the summary claiming two errors, which is exactly what a hand-edited or partially-filtered log looks like. Running the script on all three:

for f in sim.log inconsistent.log clean.log; do
  printf "=== %s ===\n" $f
  sh checklog.sh $f 2>&1
  echo "status: $?"
done

=== sim.log ===
2 error(s) in sim.log
FAIL: errors present
status: 1
=== inconsistent.log ===
ERROR: parsed 2 error(s) but the log summary says 5
BROKEN: could not evaluate inconsistent.log
status: 2
=== clean.log ===
ERROR: parsed 0 error(s) but the log summary says 2
BROKEN: could not evaluate clean.log
status: 2

Three inputs, three statuses: 1, 2 and 2. The cross-check against the log's own summary refuses both doctored logs rather than reporting a count from them — and note that it refuses clean.log, which a count-only parser would have called a clean pass with zero errors. That is the strongest argument for the cross-check: the dangerous edit is the one that removes evidence.

Exercise 6 — Challenge: lose the status in your own test

Exercise 5 ran the script without a pipe. Add the pipe a test harness naturally adds — to indent the output, or to tee it to a file — and watch the verdict disappear.

sh checklog.sh clean.log 2>&1 | cat
echo "status after | cat: $?"

ERROR: parsed 0 error(s) but the log summary says 2
BROKEN: could not evaluate clean.log
status after | cat: 0

Status 0 on a log the script just refused. Every diagnostic line is still printed — the stderr message, the BROKEN verdict — and the one value a caller would branch on reads success.

This is section 7's trap landing on the test of the script rather than on the script, and that is what makes it worth an exercise. The wrapper is correct; the harness checking it is not, and a harness that reports status: 0 for a correctly-refused input will certify the wrapper as broken in exactly the cases where it works.

Two defences, both of which this chapter has already used:

• Do not pipe the thing whose status you are testing. Redirect to a file and read the file afterwards.
• Assert the status and the output separately. A harness that only greps the output would have passed here too, because the output is right.

The general form, stated here for the first time and returned to throughout Part IV: when a verified result comes back wrong, suspect the measurement before the thing measured. Section 6 is the instance above it — an empty capture entering arithmetic as zero — and it is worth distinguishing from Chapter 11.5 Exercise 6, which is the adjacent but different finding that two different distributions produced one identical headline. A status read through a pipe is not that command's status, and a diagnostic written to stderr is invisible without 2>&1.

11. Summary

• Single-quote every awk program. Double quotes let the shell expand $1, producing { print } — a different program that runs and prints the whole line.
• The broken form depends on the script's arguments, so the same script builds different programs.
• Pass shell values with -v, never by splicing into the program text; ENVIRON when the value may contain a backslash.
• Get the pattern right first. The naive error count is 3 against a true 2, and the log's own summary is free ground truth.
• An empty capture is zero in shell arithmetic, so a missing log reports a clean run — section 6.
• awk exits 2 on an unreadable file. Check the status before using the value.
• exit (n > 0) carries the verdict in the grep convention.
• After a pipeline $? is the last command's status. awk's is in pipestatus[1] (zsh, 1-based) or ${PIPESTATUS[0]} (bash, 0-based), and the array must be copied immediately.
• Command substitution strips every trailing newline.
• Distinguish "errors" from "could not run." Different owners, different actions.
• When a result contradicts a verified check, suspect the harness — a stderr diagnostic is invisible without 2>&1.

Chapter 12.2 — grep and awk Pipelines takes the most common two-stage shape in any flow, and the question nobody asks about it: whether the grep in front of the awk is doing anything at all.