Pipelines with grep, find, and awk

Part III · Module 8 · Chapter 8.1 · Pipelines with grep, find, and awk

1. One Job Per Stage

A shell pipeline is a line of tools, each reading what the last one wrote. The useful discipline is to give each one a single responsibility:

You will rarely use all four. The two shapes that come up constantly are grep then sed and sed then awk, and they are sections 3 and 4.

Why bother with the division at all, when sed can select lines itself with an address and grep can be skipped? Three reasons: each stage is separately testable, each one is readable on its own, and when the answer is wrong you can run the pipeline a stage at a time to find out where. Section 5 covers the case where collapsing two stages into one is the better call.

2. The Fixtures

cat -n logs/sim.log

     1	UVM_INFO @ 120: reset released
     2	UVM_ERROR @ 980: data mismatch at addr @ 0x40
     3	UVM_INFO @ 1200: scoreboard drain
     4	UVM_ERROR @ 1510: timeout waiting for ready

Line 2 has two @ characters — a timestamp and an address. That detail is the whole of section 3.

cat -n sta/timing.rpt

     1	Startpoint: u_alu/rd_data_reg[3]
     2	Endpoint:   u_fifo/wr_ptr_reg[0]
     3	  data arrival time                        1.842
     4	  slack (MET)                              0.258
     5	Startpoint: u_fifo/rd_ptr_reg[1]
     6	  slack (VIOLATED)                        -0.110

3. Shape One — grep Selects, sed Rewrites

The job: get the timestamp of every error. grep picks the error lines; sed extracts the number.

grep UVM_ERROR logs/sim.log

UVM_ERROR @ 980: data mismatch at addr @ 0x40
UVM_ERROR @ 1510: timeout waiting for ready

Two lines, correctly selected. Now the registered extraction:

grep UVM_ERROR logs/sim.log | sed 's/^.*@ \([0-9]*\).*/\1/'

0
1510

The first answer is wrong. It should be 980.

.* is greedy, so ^.*@ ran to the last @ on the line — the one in addr @ 0x40 — and the capture group then took the digits that follow it, which are the 0 of 0x40. Chapter 4.2 measured greediness; here is what it costs in a real pipeline, on a log line that is entirely ordinary.

The fix is the negated class, which cannot cross the character it excludes:

grep UVM_ERROR logs/sim.log | sed 's/^[^@]*@ \([0-9]*\).*/\1/'

980
1510

[^@]* stops at the first @. Both timestamps correct.

4. Shape Two — sed Normalises, awk Computes

The job: pull the first and last field from each line of a padded report. The registered pipeline squeezes the column padding with sed, then lets awk pick the fields:

sed 's/[[:space:]]\{2,\}/ /g' sta/timing.rpt | awk '{ print $1, $NF }'

Startpoint: u_alu/rd_data_reg[3]
Endpoint: u_fifo/wr_ptr_reg[0]
data 1.842
slack 0.258
Startpoint: u_fifo/rd_ptr_reg[1]
slack -0.110

Correct. Now remove the sed stage entirely:

awk '{ print $1, $NF }' sta/timing.rpt

Startpoint: u_alu/rd_data_reg[3]
Endpoint: u_fifo/wr_ptr_reg[0]
data 1.842
slack 0.258
Startpoint: u_fifo/rd_ptr_reg[1]
slack -0.110

Identical. Not approximately — checked by checksum:

sed 's/[[:space:]]\{2,\}/ /g' sta/timing.rpt | awk '{ print $1, $NF }' | md5 -q

90ca4473782f0c1c181291c0e65deae7

awk '{ print $1, $NF }' sta/timing.rpt | md5 -q

90ca4473782f0c1c181291c0e65deae7

The sed stage does nothing, because awk's default field separator already splits on runs of whitespace and ignores leading and trailing space. Squeezing the padding before handing it to awk is solving a problem awk does not have.

When the sed stage is load-bearing. Two real cases, both measured.

First, when you override awk's clever default with a literal single-space separator, the padding suddenly matters:

awk -F'[ ]' '{ print NF }' sta/timing.rpt

2
4
29
34
2
28

Twenty-nine fields on a padded line — every single space is now a separator, so the run of padding creates a crowd of empty fields. Squeeze first and the counts become sane:

sed 's/[[:space:]]\{2,\}/ /g' sta/timing.rpt | awk -F'[ ]' '{ print NF }'

2
2
5
4
2
4

Second, and more usefully, when you need a separator that is not whitespace at all — because a field itself contains spaces. Here the block name and the status are single words but the padding is the only thing separating them:

cat -et sta/cols.rpt

alu.sv   1.842   MET$
fifo.sv  -0.110  VIOLATED$

sed 's/[[:space:]]\{2,\}/:/g' sta/cols.rpt | awk -F: '{ print $3, $1 }'

MET alu.sv
VIOLATED fifo.sv

sed manufactured a delimiter that did not exist in the input. That is a sed stage earning its place: it is doing a job awk cannot do for itself.

5. When to Collapse Two Stages Into One

grep X | sed 's/a/b/' can always be written as one sed, because an address does the selecting:

sed -n 's/^UVM_ERROR @ \([0-9]*\).*/\1/p' logs/sim.log

980
1510

One process instead of two, and the same answer as section 3's fixed pipeline. Which is better?

Prefer the single sed when the pattern is simple and you are writing a script that will be read often. It is fewer moving parts, and the -n … p idiom from Chapter 5.1 already means "select and transform".

Prefer the pipeline when the selection and the transformation are independently interesting — especially while developing, because you can run grep UVM_ERROR logs/sim.log on its own and confirm the selection before worrying about the extraction. That debuggability is worth a process.

Always prefer the pipeline when the selecting tool is better at selecting. grep -r, grep -c, grep -l and grep -v are all things sed either cannot do or does clumsily, and grep -F for fixed strings avoids regex escaping entirely.

The one thing not to do is use both halves badly: grep X file | sed -n '/X/p' selects twice.

6. A Pipeline's Exit Status Is Usually a Lie

The stage people forget to check. In a POSIX shell, $? after a pipeline is the status of the last command only:

grep NOSUCHPATTERN logs/sim.log | sed 's/a/b/'; echo "status=$?"

status=0

The grep found nothing and exited 1. The sed succeeded on empty input and exited 0. The pipeline reports success — and in a flow script that is a stage silently producing nothing while claiming it worked.

bash keeps every stage's status in an array:

bash -c 'grep NOSUCH logs/sim.log | sed "s/a/b/"; echo "pipestatus=${PIPESTATUS[0]} ${PIPESTATUS[1]}"'

pipestatus=1 0

There it is — stage one failed. Or make the pipeline itself fail:

bash -c 'set -o pipefail; grep NOSUCH logs/sim.log | sed "s/a/b/"; echo "status=$?"'

status=1

set -o pipefail makes the pipeline return the rightmost non-zero status. For any flow script that runs sed pipelines, this one line is the cheapest reliability improvement available.

7. Think It Through

1. grep ERROR log | sed 's/^.*: //' returns the wrong part of some lines. Why, and what is the fix?

.* is greedy, so ^.*: runs to the last colon on the line, not the first. Any line with two colons loses more than you intended — section 3 measured exactly this with @. Use s/^[^:]*: //, which cannot cross the first colon.

2. How do you tell whether a stage in your pipeline is doing anything?

Remove it and compare. Section 4 did this with a checksum, and the two outputs were byte-identical. cmd_with | md5 -q against cmd_without | md5 -q is a two-command test, and a matching pair means the stage is decoration.

3. Your extraction pipeline prints nothing and the script carries on. What did you not check?

The exit status of the stage that failed. $? after a pipeline is the last command's status, and a sed reading empty input succeeds — section 6 measured status=0 on a pipeline whose grep found nothing. Add set -o pipefail, or inspect PIPESTATUS.

4. When is grep X file | sed 's/a/b/' better than sed -n '/X/s/a/b/p' file?

When you want to check the selection independently, when grep is doing something sed cannot (-r, -l, -c, -F, -v), or when the two halves are maintained by different people. For a short script that will be read often, the single sed is fewer moving parts. Both are defensible; using both halves badly — selecting twice — is not.

5. Why does awk '{ print $1, $NF }' not need a sed stage, and when would it?

awk's default field separator splits on runs of whitespace and ignores leading and trailing space, so column padding is already handled. It needs help when you override that default with a literal separator — -F'[ ]' turned a padded line into 29 fields — or when the real separator is not whitespace and sed has to manufacture one, as the : example did.

6. Write a pipeline that reports the number of distinct error messages in a log, ignoring timestamps.

grep UVM_ERROR log | sed 's/@ [0-9]*: //' | sort | uniq -c | sort -rn. Four stages, four jobs: select, normalise, group, rank. Note that sed's job here is purely to make two lines that differ only by a timestamp compare equal — which is Chapter 6.4's normalisation idea in a one-liner.

8. Summary

• One job per stage: find chooses files, grep selects lines, sed rewrites them, awk computes across them.
• The registry's extraction example demonstrates its own bug. s/^.*@ \([0-9]*\).*/\1/ returned 0 instead of 980 on a line with two @, because greedy .* ran to the last one. ^[^@]*@ fixes it.
• A single-@ sample would hide that, which is the fixture lesson from Chapters 3.2, 3.3 and 4.2 in its most expensive form.
• The registry's normalisation example has a redundant sed stage — output byte-identical with and without it, confirmed by checksum, because awk's default FS already splits on runs of whitespace. Delete it.
• The sed stage is load-bearing when awk's default is overridden (-F'[ ]' gave 29 fields on a padded line) or when sed must manufacture a non-whitespace delimiter.
• Test whether a stage matters by removing it and comparing checksums.
• grep X | sed can usually collapse into one sed with an address. Keep the pipeline when the stages are independently testable or grep is doing something sed cannot.
• A pipeline's $? is the last stage only. A failed grep followed by a successful sed reported status=0. Use set -o pipefail, or bash's PIPESTATUS (zsh: pipestatus, 1-indexed).

Every command in this chapter was typed inside single quotes without comment. That is not an accident, and it is not always enough — the moment a pipeline needs a variable in its sed script, the shell starts rewriting your command before sed ever sees it. Chapter 8.2 — Bash Quoting and Variable Expansion is that boundary, including the measured case where a perfectly good command breaks because a path happened to contain the delimiter.