Choosing Between grep, find, sed and awk

Part I · Module 14 · Chapter 14.8 · Choosing Between grep, find, sed and awk

Chapter 13.9 was about assembling tools. This is about picking them.

1. Four Different Questions

The overlap is real — you can count lines with grep -c, awk 'END{print NR}', wc -l or sed -n '$='. That makes "which tool" feel like a matter of taste. It is not, because each one is shaped around a different question:

2. The Overlaps, Resolved

Several jobs can be done three ways. These are the ones worth settling.

The general principle: prefer the simplest tool that does the job, and prefer one tool over two.

Two tools where one would do is a wasted process, an extra thing to get wrong, and — when the pipeline has a status — Chapter 12.8's problem.

3. Combining Them Properly

The common shapes, and which direction the data flows.

Find files, then search inside them:

find . -name '*.v' -newer last_build -print0 | xargs -0 grep -ln 'clk_div'

find answers "which files", grep answers "which of those contain it". The Chapter 14.7 pair is not optional.

Search for files, then act on exactly those:

grep -rlZ --include='*.v' 'OLD_MACRO' . | xargs -0 sed -i.bak 's/OLD_MACRO/NEW_MACRO/g'

grep -l narrows to the files that need changing, so sed runs on twelve files rather than four thousand.

Search, extract, then calculate:

grep -h 'SIMULATION TIME' */sim.log | awk '{s += $3} END {printf "%.1f hours\n", s/3600}'

grep selects the lines, awk does the arithmetic. Each does what it is best at.

Search, normalise, then count:

grep -h ERROR *.log | sed 's/[0-9][0-9]*/N/g' | sort | uniq -c | sort -rn | head

The Chapter 13.4 pipeline, with the normalisation step that makes it useful.

4. What Each One Cannot Do

Knowing the limits is faster than discovering them.

find cannot look inside a file. It matches on name, type, time, size, permissions and ownership — never contents. Pair it with grep.

grep cannot do arithmetic, cannot address a field by number, and cannot match across lines. grep '$3 == "FAIL"' is not a thing; that is awk. And a pattern spanning two lines cannot match at all, because grep works a line at a time.

sed cannot address a field by number without an unreadable pattern, and cannot calculate. It also has no portable in-place edit (Chapter 13.6's -i trap).

awk cannot edit a file in place portably, and is the wrong tool once the program passes about three lines.

None of them understand nesting. JSON, XML and real CSV have records that span lines and structure that line-based tools cannot see (Chapter 13.1). jq exists for JSON precisely because this toolkit does not fit it.

5. When It Stops Being a One-Liner

Chapter 13.9 put the limit at about six stages. The other signals are about shape rather than length.

The transition is usually painless, because a script can contain the pipeline:

#!/bin/bash
set -uo pipefail

runs="${1:?usage: $0 <runs-dir>}"

# Which runs failed — a question about file contents.
failed=$(grep -rl 'TEST FAILED' "$runs"/*/sim.log | wc -l)

# Which tests failed most often — narrow, transform, summarise.
echo "=== most frequent failures ==="
grep -rl 'TEST FAILED' "$runs"/*/sim.log \
  | sed 's|.*/run_||; s|/sim.log||; s/_seed[0-9]*$//' \
  | sort | uniq -c | sort -rn | head -5

# How much time they consumed — a calculation, so awk.
echo "=== time spent on failures ==="
grep -h 'SIMULATION TIME' $(grep -rl 'TEST FAILED' "$runs"/*/sim.log) \
  | awk '{s += $3} END {printf "%.1f minutes\n", s/60}'

[ "$failed" -eq 0 ]

Each block is a pipeline with a comment saying which question it answers. Nothing was lost by making it a script — the pipelines are unchanged — and it gained names, comments, an argument and an exit status.

6. A Worked Decision

"Which modules changed since the last release still use the deprecated reset macro, and how many times?"

Four questions in one sentence, which §5 says is a script — but it is a good illustration of taking a sentence apart.

"changed since the last release" — existence and time. find:

find rtl/ -name '*.v' -newer .last_release

"still use the deprecated macro" — contents, and which files:

find rtl/ -name '*.v' -newer .last_release -print0 | xargs -0 grep -l 'OLD_RESET'

"how many times" — a count per file:

find rtl/ -name '*.v' -newer .last_release -print0 | xargs -0 grep -c 'OLD_RESET' | grep -v ':0$'

"ranked" — the sorting from Chapter 13.3:

find rtl/ -name '*.v' -newer .last_release -print0 \
  | xargs -0 grep -c 'OLD_RESET' \
  | grep -v ':0$' \
  | sort -t: -k2,2 -rn

Four stages, each one answering one clause of the original sentence, built up in the Chapter 13.9 way. It is at the limit of what should stay a one-liner, and a comment above it would be the right next step.

7. A Reference for Both Modules

Practice

mkdir -p /tmp/ch/{rtl,runs/run_alu_seed3,runs/run_fifo_seed9} && cd /tmp/ch
printf 'module alu;\n  OLD_RESET;\n  OLD_RESET;\nendmodule\n' > rtl/alu.v
printf 'module fifo;\n  NEW_RESET;\nendmodule\n' > rtl/fifo.v
printf 'TEST FAILED: timeout after 1203 cycles\nSIMULATION TIME 262.4 seconds\n' > runs/run_alu_seed3/sim.log
printf 'TEST PASSED\nSIMULATION TIME 130.0 seconds\n' > runs/run_fifo_seed9/sim.log
touch .last_release; sleep 1; touch rtl/alu.v

1 — Name the tool for each sentence before running anything:

• Which .v files are larger than 1 kilobyte?
• Which files mention OLD_RESET?
• How many seconds did all the runs take in total?
• What is the seed in each run directory name?

2 — Then check yourself.

find rtl -name '*.v' -size +1c
grep -rl OLD_RESET rtl/
grep -h 'SIMULATION TIME' runs/*/sim.log | awk '{s+=$2} END {print s}'
ls runs/ | sed 's/.*_seed//'

The third needs care — count the fields before trusting $2.

3 — Resolve an overlap three ways.

grep -c OLD_RESET rtl/alu.v
grep OLD_RESET rtl/alu.v | wc -l
awk '/OLD_RESET/ {n++} END {print n+0}' rtl/alu.v

Same answer. Say which you would commit.

4 — Count lines versus occurrences.

printf 'OLD_RESET OLD_RESET\n' > rtl/double.v
grep -c  OLD_RESET rtl/double.v
grep -o  OLD_RESET rtl/double.v | wc -l

5 — Build the worked example.

find rtl/ -name '*.v' -newer .last_release
find rtl/ -name '*.v' -newer .last_release -print0 | xargs -0 grep -l OLD_RESET
find rtl/ -name '*.v' -newer .last_release -print0 | xargs -0 grep -c OLD_RESET | grep -v ':0$'

6 — Narrow, transform, summarise.

grep -rl 'TEST FAILED' runs/*/sim.log
grep -rl 'TEST FAILED' runs/*/sim.log | sed 's|.*/run_||; s|/sim.log||; s/_seed[0-9]*$//'
grep -rl 'TEST FAILED' runs/*/sim.log | sed 's|.*/run_||; s|/sim.log||; s/_seed[0-9]*$//' | sort | uniq -c

7 — Spot the wrong tool. Each of these works and each has a better form. Write the better one:

ls rtl/ | grep '\.v$'
cat rtl/alu.v | grep OLD_RESET
grep OLD_RESET rtl/alu.v | head -1
sed -n '/OLD_RESET/p' rtl/alu.v

8 — Clean up. cd /tmp && rm -rf /tmp/ch

Check Your Understanding

1. What question is each of the four tools shaped around?

2. grep FAIL results.csv and awk -F, '$3=="FAIL"' results.csv can both find failures. When does the difference matter?

3. Why prefer grep -c to grep | wc -l?

4. What is the usual order of stages in a pipeline, and why?

5. Name three signs that a one-liner should become a script.

6. Why is none of these the right tool for JSON?

8. Remember

• Each tool answers a different question: which files exist, which contain something, which part of each line, what the numbers add up to.
• Say the question as a sentence first. The sentence names the tool; two tools means it is a script.
• Prefer the simplest tool, and one tool over two. grep -c, sort -u, grep -m1 each replace a pipeline.
• grep matches anywhere on the line; awk can test one field. That difference is the whole reason to use awk for structured data.
• cut for strictly delimited, awk for aligned or reordered.
• Never parse ls. find asks the filesystem.
• The usual order is narrow, transform, summarise — and narrowing first is also the fast way.
• find cannot see inside a file; grep cannot calculate; sed cannot address a field; awk cannot edit in place.
• None of them understand nesting. JSON and real CSV need a tool built for them.
• Stop at a conditional, a reused intermediate, a third awk, or anything you cannot comment.
• A script loses nothing — the pipelines go inside it unchanged, and gain names and an exit status.
• In a real repository, deciding where to search matters more than which tool. Chapter 23.5 measures a search whose answer changed from three files to two because of which grep was installed.

That completes Module 14, and with it the searching and text-processing half of Part I. You can locate a file by any property the filesystem records, find text anywhere in a tree and shape the answer to the question, write a pattern rather than copy one, act on what you found without breaking on a space, and choose between a dozen overlapping tools on purpose.

Module 15 stops typing commands one at a time. Everything from here — scripts, conditionals, loops, functions, arrays and defensive automation — is about putting what you now know into a file that somebody else can run.