grep and awk Pipelines

Part III · Module 12 · Chapter 12.2 · grep and awk Pipelines

1. Why This Matters

Two questions get asked about grep … | awk …, and both have answers that are usually given from memory rather than measurement.

The first is whether the grep is doing anything. awk matches patterns, so the pipeline looks like a habit from before people knew that — and the stock advice is to collapse it. Section 4 measures it instead.

The second is whether the two stages agree about what they are matching. They are different programs with different regular-expression dialects, and section 5 shows the same pattern text selecting a different line in each.

2. A Log Worth Measuring On

awk 'BEGIN { for (i = 1; i <= 200000; i++) {
  if (i % 2000 == 0) printf "UVM_ERROR ./env/sb.sv(%d) @ %d: uvm_test_top.env.sb [MISMATCH] expected=%d got=%d\n", 100+i%50, i*10, i%256, (i+7)%256
  else printf "UVM_INFO ./env/mon.sv(%d) @ %d: uvm_test_top.env.mon [XACT] addr=%x len=%d\n", 40+i%30, i*10, i%65536, i%16
} }' > big.log
wc -l < big.log | awk '{ printf "  %d lines, ", $1 }'; ls -l big.log | awk '{ printf "%.1f MB\n", $5/1048576 }'
grep -c '^UVM_ERROR' big.log | awk '{ printf "  %d error lines\n", $1 }'

  200000 lines, 15.4 MB
  100 error lines

One hundred matching records in two hundred thousand — a selectivity of 0.05%, which is realistic for error extraction from a regression log.

3. Both Forms Give the Same Answer

grep '^UVM_ERROR' big.log | awk '{ n++ } END { printf "  grep|awk: %d\n", n }'
awk '/^UVM_ERROR/ { n++ } END { printf "  awk only: %d\n", n }' big.log

  grep|awk: 100
  awk only: 100

Correctness is not the question for a pattern this simple. Cost is.

4. The Measurement

Timings are not reproducible output, so they are reported here as measurements rather than shown as command results — the runners are below, and the figures are medians of five warm runs on one machine. Run them yourself; the ratios transfer, the seconds do not.

# Each runner is its own file because a shell's built-in `time` is a keyword
# whose output escapes a subshell redirect -- use /usr/bin/time -p and read
# the `real` line.

# a_pipe.sh        -- RARE pattern, pre-filtered
grep '^UVM_ERROR' big.log | awk '{ n++ } END { print n }' > /dev/null

# b_awk.sh         -- RARE pattern, one awk pass
awk '/^UVM_ERROR/ { n++ } END { print n }' big.log > /dev/null

# c_pipe_common.sh -- COMMON pattern, pre-filtered
grep '^UVM_INFO' big.log | awk '{ n++ } END { print n }' > /dev/null

# d_awk_common.sh  -- COMMON pattern, one awk pass
awk '/^UVM_INFO/ { n++ } END { print n }' big.log > /dev/null

# e_awk_fields.sh  -- RARE pattern, and the action touches a field
awk '/^UVM_ERROR/ { s = s $NF } END { print length(s) }' big.log > /dev/null

# f_grep_c.sh      -- the grep stage alone, for attribution
grep -c '^UVM_ERROR' big.log > /dev/null

# timed with, for each runner:
#   for r in 1 2 3 4 5; do /usr/bin/time -p sh RUNNER.sh 2>&1 | awk '/^real/ { print $2 }'; done

On a 200,000-line, 15.4 MB log with 100 matching records:

On the rare pattern the pipeline is 0.18 s against 0.83 s — the grep makes it 4.6× faster. On the common pattern the two are within noise, 0.65 s against 0.70 s.

5. The Same Pattern Selects Different Lines

addr=32+4
addr=3244
err+or here
UVM_ERROR x
UVM_WARNING y

printf "  grep '2+4'      : "; grep -c '2+4' re.txt
printf "  grep -E '2+4'   : "; grep -c -E '2+4' re.txt
printf "  awk /2+4/       : "; awk '/2+4/ { n++ } END { print n+0 }' re.txt

  grep '2+4'      : 1
  grep -E '2+4'   : 1
  awk /2+4/       : 1

One line each. Which line:

printf "  grep '2+4'   matched: ["; grep '2+4' re.txt | tr -d '\n'; echo "]"
printf "  awk /2+4/    matched: ["; awk '/2+4/' re.txt | tr -d '\n'; echo "]"

  grep '2+4'   matched: [addr=32+4]
  awk /2+4/    matched: [addr=3244]

More of the same, in both directions:

printf "  grep 'err+or'   : "; grep -c 'err+or' re.txt
printf "  awk /err+or/    : "; awk '/err+or/ { n++ } END { print n+0 }' re.txt
printf "  grep 'ERROR|WARNING'    : "; grep -c 'ERROR|WARNING' re.txt
printf "  grep -E 'ERROR|WARNING' : "; grep -c -E 'ERROR|WARNING' re.txt
printf "  awk /ERROR|WARNING/     : "; awk '/ERROR|WARNING/ { n++ } END { print n+0 }' re.txt

  grep 'err+or'   : 1
  awk /err+or/    : 0
  grep 'ERROR|WARNING'    : 0
  grep -E 'ERROR|WARNING' : 2
  awk /ERROR|WARNING/     : 2

• err+or: grep finds the literal, awk finds nothing — a pattern that worked in a grep silently matches zero records when pasted into an awk program.
• ERROR|WARNING: grep without -E finds nothing because | is literal; grep -E and awk both find two — a pattern that worked in awk silently matches zero when pasted into a grep.

Both directions fail, and both fail by matching nothing rather than by erroring.

6. The Pipeline That Discards What grep Found

printf "  grep 'err+or' alone                : %s line(s)\n" "$(grep -c 'err+or' re.txt)"
printf "  grep 'err+or' | awk '/err+or/'     : %s line(s)\n" "$(grep 'err+or' re.txt | awk '/err+or/ { n++ } END { print n+0 }')"
printf "  grep 'err+or' | awk '{ n++ }'      : %s line(s)\n" "$(grep 'err+or' re.txt | awk '{ n++ } END { print n+0 }')"

  grep 'err+or' alone                : 1 line(s)
  grep 'err+or' | awk '/err+or/'     : 0 line(s)
  grep 'err+or' | awk '{ n++ }'      : 1 line(s)

Three outcomes from one intention. The middle one — re-applying the pattern in the second stage — is a common defensive habit and it is how the record grep found gets thrown away, because awk reads the same text under different rules.

The third line is the correct shape: once grep has selected the records, awk should not re-select them. { n++ } with no pattern, or an action keyed on fields rather than on the original pattern. The filter belongs in exactly one stage.

7. A Literal Metacharacter in awk

printf "  awk /err\\\\+or/ : "; awk '/err\+or/ { n++ } END { print n+0 }' re.txt
printf "  awk index()   : "; awk 'index($0, "err+or") { n++ } END { print n+0 }' re.txt

  awk /err\+or/ : 1
  awk index()   : 1

Both recover grep's answer. index($0, "…") is the better of the two when the target is a fixed string: it has no dialect, nothing to escape, and Chapter 7.8 measured the case where $0 ~ "|" is a fatal regex error while index() simply works.

8. Common Mistakes

Dropping the grep for speed. On a selective pattern it is 4.7× faster — section 4.

Keeping the grep for speed on an unselective pattern. Two processes doing one's job.

Blaming awk's field splitting. Referencing $NF costs nothing; splitting is lazy — section 4.

Reusing a pattern across the two tools. '2+4' matches a different line in each — section 5.

grep without -E for a pattern with | or +. Those are literals in BRE and it matches nothing.

Re-applying the pattern in the second stage. It discards records the first stage found — section 6.

A regex for a fixed string. Use index(); $0 ~ "|" is a fatal error.

Comparing the two forms by count. Both returned 1 on disjoint records.

9. Exercises

Exercise 1 — Both forms, one answer

grep '^UVM_ERROR' big.log | awk '{ n++ } END { printf "grep|awk %d\n", n }'
awk '/^UVM_ERROR/ { n++ } END { printf "awk only %d\n", n }' big.log

grep|awk 100
awk only 100

Exercise 2 — Extract a field after the filter

grep '^UVM_ERROR' big.log | awk '{ print $NF }' | sort | uniq -c | sort -k1,1rn -k2,2 | head -3

   7 got=119
   7 got=167
   7 got=215

The second stage has no pattern — the filter was done once, in the grep. Field 4 is the timestamp, which varies; field 2 would not, because this synthetic log derives the file and line from i % 50 and every error record is a multiple of 2000.

Exercise 3 — Debug a pipeline that finds nothing

A script greps for a signal name containing a plus and the awk stage returns zero.

printf "grep stage alone: %s\n" "$(grep -c 'err+or' re.txt)"
printf "full pipeline:    %s\n" "$(grep 'err+or' re.txt | awk '/err+or/ { n++ } END { print n+0 }')"

grep stage alone: 1
full pipeline:    0

grep finds the line and awk discards it: in awk's extended syntax err+or means er followed by one-or-more r followed by or, which the literal text err+or does not match.

printf "awk, pattern escaped:     %s\n" "$(grep 'err+or' re.txt | awk '/err\+or/ { n++ } END { print n+0 }')"
printf "awk, no re-match at all:  %s\n" "$(grep 'err+or' re.txt | awk '{ n++ } END { print n+0 }')"

awk, pattern escaped:     1
awk, no re-match at all:  1

Exercise 4 — Debug two counts that agree about nothing

A reviewer validates a rewrite by checking that the grep form and the awk form find the same number of lines.

printf "grep '2+4' count: %s    awk /2+4/ count: %s\n" "$(grep -c '2+4' re.txt)" "$(awk '/2+4/ { n++ } END { print n+0 }' re.txt)"

grep '2+4' count: 1    awk /2+4/ count: 1

The counts match. The records do not:

printf "  grep selected: ["; grep '2+4' re.txt | tr -d '\n'; echo "]"
printf "  awk selected:  ["; awk '/2+4/' re.txt | tr -d '\n'; echo "]"
printf "  the same line? %s\n" "$([ "$(grep '2+4' re.txt)" = "$(awk '/2+4/' re.txt)" ] && echo yes || echo NO)"

  grep selected: [addr=32+4]
  awk selected:  [addr=3244]
  the same line? NO

Comparing the selected text rather than the count is one [ ... = ... ] and it is the only check that would have caught this.

Exercise 5 — Engineering application

A wrapper that decides for itself whether the grep is worth keeping, by measuring the selectivity rather than guessing.

#!/bin/sh
# prefilter.sh — extract a field from matching records, pre-filtering only
#                when the pattern is selective enough to be worth it.
#
# Usage:  sh prefilter.sh <ere-pattern> <field> <file>
# Notes:  the pattern is an EXTENDED regex, passed to grep -E and to awk
#         unchanged, so both stages read it the same way (section 5).

pat=$1; fld=$2; file=$3
[ -n "$file" ] || { echo "usage: prefilter.sh <ere-pattern> <field> <file>" >&2; exit 2; }

total=$(awk 'END { print NR }' "$file") || exit 2
hits=$(grep -c -E "$pat" "$file")
[ -n "$hits" ] || hits=0

# selectivity as a percentage, computed in awk because sh has no float
pct=$(awk -v h="$hits" -v t="$total" 'BEGIN { printf "%.3f", (t ? 100 * h / t : 0) }')

printf "# %s of %s record(s) match (%s%%)\n" "$hits" "$total" "$pct"
if awk -v h="$hits" -v t="$total" 'BEGIN { exit !(t > 0 && h < t / 10) }'; then
    printf "# selective -- pre-filtering with grep -E\n"
    grep -E "$pat" "$file" | awk -v f="$fld" '{ print $f }'
else
    printf "# not selective -- one awk pass\n"
    awk -v f="$fld" -v p="$pat" '$0 ~ p { print $f }' "$file"
fi

sh prefilter.sh '^UVM_ERROR' 4 big.log > sel.txt; head -4 sel.txt

# 100 of 200000 record(s) match (0.050%)
# selective -- pre-filtering with grep -E
20000:
40000:

sh prefilter.sh '^UVM_INFO' 4 big.log > uns.txt; head -3 uns.txt

# 199900 of 200000 record(s) match (99.950%)
# not selective -- one awk pass
10:

The output is redirected to a file before head reads it rather than piped into head — a head on a live pipeline closes it early, and how much the upstream stage wrote before noticing varies between runs.

The threshold is one tenth, chosen because section 4's measurement showed the benefit present at 0.05% and absent at 99.95% — it is a tunable, not a law, and it is in one place rather than in the author's head.

Two details that matter more than the branch. The pattern is passed to both stages as the same extended regex, so section 5's dialect divergence cannot occur — grep -E "$pat" and $0 ~ p agree by construction. And the awk stage after the grep has no pattern, so section 6's discard cannot occur either.

Exercise 6 — Challenge: show that the selectivity branch cannot be decided statically

prefilter.sh measures the selectivity at run time, which costs a full grep -c pass. Show that the cheaper alternative — deciding from the pattern alone — gets it wrong.

for p in '^UVM_ERROR' '^UVM_INFO' 'MISMATCH' 'addr='; do
  h=$(grep -c -E "$p" big.log)
  printf "  %-14s %6s of 200000  (%s%%)\n" "$p" "$h" "$(awk -v h=$h 'BEGIN { printf "%.3f", 100*h/200000 }')"
done

  ^UVM_ERROR        100 of 200000  (0.050%)
  ^UVM_INFO      199900 of 200000  (99.950%)
  MISMATCH          100 of 200000  (0.050%)
  addr=          199900 of 200000  (99.950%)

Four patterns of similar shape and length, two selective and two not. Nothing about the text of MISMATCH versus addr= indicates which is rare — both are short literal strings with no metacharacters, and the answer depends entirely on the contents of this particular log.

So the branch needs a measurement, and the measurement costs a pass. Whether that pass is worth it has a measured answer too: section 4's table puts the grep -c probe at 0.17 s and the one-pass awk extraction it replaces at 0.83 s, so the probe costs a fifth of what the decision saves on a selective pattern — and would not pay on a log small enough that the whole job finishes inside the probe.

Which is the honest answer to "should I put a grep in front of my awk": it depends on the data, it is cheap to find out, and the folklore in both directions is wrong often enough to be worth not repeating.

10. Summary

• Both forms give the same answer on a simple pattern; the question is cost.
• Selectivity decides. 200 of 400,000: grep|awk 0.34s against awk-only 1.61s, a 4.7× speedup. 399,800 of 400,000: 1.29s against 1.31s, indistinguishable.
• The cost is awk's record loop, not field splitting — referencing $NF is free because splitting is lazy.
• grep -c alone costs the same as the whole pipeline, so grep is essentially all of it.
• grep is BRE, awk is ERE. '2+4' matched one line in each — addr=32+4 and addr=3244, different lines, equal counts.
• Both directions fail silently: err+or is 1 in grep and 0 in awk; ERROR|WARNING is 0 in grep and 2 in awk.
• Use grep -E so one pattern means one thing in both stages.
• Do not re-apply the pattern downstream. It discarded the record grep found.
• index($0, "...") for a fixed string — no dialect, and $0 ~ "|" is fatal.
• Compare the selected records, not the counts. One [ "$a" = "$b" ] catches what two matching counts hide.

Chapter 12.3 — find and Multi-File Processing takes the other half of the shell boundary: feeding a generated file list into awk so that a path containing a space does not silently split into two filenames that do not exist.