awk vs sed

Part I · Module 1 · Chapter 1.4 · awk vs sed

1. The Question This Chapter Answers

Chapter 1.3 separated awk from grep by asking which part of a line you care about. That line will not work for sed, because sed has no notion of fields either — so repeating it would teach you nothing new.

The distinction that matters for sed is about what comes out of the command.

sed rewrites a stream and hands you back text of the same shape. awk reads a stream and hands you back values pulled out of it.

Three lines in, three lines out, some characters different — that is sed. Three lines in, one number out — that is awk. Neither is more powerful in general; they produce different kinds of thing, and knowing which kind you want is the whole decision.

2. What sed Is

sed is a stream editor. It reads input line by line, applies editing instructions, and writes the result out. The instruction you will use ninety percent of the time is substitution.

test_a PASS
test_b FAIL
test_c PASS
test_d FAIL

sed 's/FAIL/FAILED/' results.txt

test_a PASS
test_b FAILED
test_c PASS
test_d FAILED

Read s/FAIL/FAILED/ as three parts separated by slashes: the operation s for substitute, the text to find, and the text to put in its place.

Look carefully at the output. Four lines went in and four lines came out. The two PASS lines passed through untouched — sed did not select or discard anything, it edited what matched and copied the rest. The result is still a test-results file. That is the characteristic sed outcome.

By default substitution replaces only the first match on each line. Adding g for "global" replaces every match on the line:

sed 's/0/#/g' cells.txt

DFF 12#
AND 45
OR 3#
MUX 18

120 became 12# and 30 became 3# — each had one zero. Had a line contained 100, the un-suffixed form would have changed only the first zero while the g form would change both. This catches people out constantly.

The find part is a regular expression, which is why sed is powerful and why Module 4 is a whole module. The slash is only the conventional separator, and you can use another character — which matters when the text itself contains slashes, as paths do:

sed 's|u_core/|TOP/core/|' area.rpt

TOP/core/u_alu  area 1820
TOP/core/u_regfile  area 2410
u_pad/u_io  area 640

Using | as the separator avoids escaping every slash in a path. The third line is untouched because it does not contain u_core/. This is a real engineering job — rewriting a hierarchy prefix across a report or a filelist — and sed is the natural tool for it.

3. What awk Does Differently

A short reminder, since Chapters 1.1 to 1.3 have covered it: awk splits every line into fields before your instruction runs. $1 is the first field, $2 the second, $NF the last, $0 the whole line.

That gives awk two things sed does not have: it can address a column by position, and it can do arithmetic.

DFF 120
AND 45
OR 30
MUX 18

awk '{ total += $2 } END { print total }' cells.txt

213

Four lines in, one number out. There is no sed command that does this, and the reason is not that sed is badly designed — it is that adding numbers is not an editing operation. sed's whole model is text in, text out.

The shape difference is the point:

4. Extraction and Arithmetic: awk's Home Ground

Two jobs that are awk's and not sed's.

Selecting records by a field and printing part of them:

awk '$2 == "FAIL" { print $1 }' results.txt

test_b
test_d

Two lines out of four, and only the first column of each. The shape changed completely.

Filtering on a number:

awk '$2 > 100 { print $1 }' cells.txt

DFF

Only DFF has a count above 100. As Chapter 1.3 established, no text pattern expresses "greater than 100", so this is not a job sed can be talked into doing.

Here is the clearest demonstration of the boundary. sed can isolate the numbers in the area report — substitution plus -n and the p flag prints only what was rewritten:

sed -n 's/.*area //p' area.rpt

1820
2410
640

That is genuinely useful: sed extracted the values. But now add them up. There is no next step in sed, because sed cannot add. awk finishes the job in one command:

awk '{ total += $3 } END { print total }' area.rpt

4870

5. The Overlap Is Real

If you stop at section 4 you will carry a false model: sed edits, awk analyses, and never the twain. In practice the two overlap substantially, and you will meet scripts that exploit it.

sed can select lines. With -n, sed prints nothing unless told to, and p prints the current line. That makes a line selector much like grep:

sed -n '/FAIL/p' results.txt

test_b FAIL
test_d FAIL

sed can address lines by number, which neither grep nor the awk you have learned so far does as directly:

sed -n '2p' results.txt

test_b FAIL

sed can delete, which is selection inverted:

sed '/PASS/d' results.txt

test_b FAIL
test_d FAIL

That output is identical to the -n '/FAIL/p' result above, but the operations are different: one kept the matching lines, the other removed the non-matching ones. On this file they coincide. On a file with a third kind of line they would not.

And awk can transform text. Writing two values next to each other in a print joins them, which is enough to relabel output without any new syntax:

awk '$2 == "FAIL" { print $1 ": needs triage" }' results.txt

test_b: needs triage
test_d: needs triage

So awk did a text transformation, and a useful one. awk also has dedicated substitution functions — sub() and gsub() — which Chapter 4.5 covers properly; they are deliberately not used here because this chapter needs no new syntax to make its point.

6. A Semiconductor Example: One Report, Two Jobs

A synthetic hierarchical area report. Three columns: an instance path, the literal word area, and a value.

u_core/u_alu  area 1820
u_core/u_regfile  area 2410
u_pad/u_io  area 640

Two genuinely different requests arrive about this file.

Request one: the hierarchy was renamed, so update the paths. The design was re-partitioned and everything under u_core/ now lives under TOP/core/. You need the report again, with corrected paths, still readable as a report. That is a rewrite, and sed owns it:

sed 's|u_core/|TOP/core/|' area.rpt

TOP/core/u_alu  area 1820
TOP/core/u_regfile  area 2410
u_pad/u_io  area 640

Three lines in, three lines out, still an area report. Exactly what was asked for.

Request two: what is the total area, and which instances are above 1000? That is a measurement, and awk owns it:

awk '{ total += $3 } END { print total }' area.rpt

4870

awk '$3 > 1000 { print $1, $3 }' area.rpt

u_core/u_alu 1820
u_core/u_regfile 2410

Same file, same two tools available, and the right answer is different each time — decided entirely by the shape of the thing being asked for.

7. Combining Them Responsibly

The two tools pipe together, and the natural order is sed first — normalise the text — then awk — measure it:

sed 's/FAIL/FAILED/' results.txt | awk '$2 == "FAILED" { print $1 }'

test_b
test_d

That works. It is also, on this input, pointless: awk '$2 == "FAIL" { print $1 }' gives the same answer in one command. A sed stage earns its place when it genuinely makes the input easier for awk — collapsing an irregular separator, stripping a wrapper character, normalising two spellings of the same value into one — and not when it merely renames something awk could have matched directly.

The reason to care is maintenance. Every stage in a pipeline is a place where the next engineer has to work out what shape the data is in. Two stages that could be one is a cost with no benefit. Module 12 builds real multi-stage pipelines once the language is in hand.

8. In-Place Editing and Portability

You will meet sed -i in other people's scripts. It edits the file on disk instead of printing to the screen, and its syntax is not portable — which is worth seeing rather than being warned about.

On the BSD sed that ships with macOS, -i requires an argument giving the backup suffix. Writing sed -i 's/FAIL/X/' results.txt therefore does not do what it looks like: sed takes s/FAIL/X/ as the backup suffix, and then has to read your script from the next word — results.txt — which is not a sed program.

It fails, and the message is confusing because the cause is not where it points. It is also not worth memorising: testing this one command produced two different errors on the same machine depending on how the shell was invoked, one complaining about an undefined label in the filename and one about -i being used with standard input. Treat the lesson as "this command is broken on BSD sed", not as a string to recognise.

On BSD sed you must supply an explicit empty suffix:

cp results.txt scratch.txt
sed -i '' 's/FAIL/X/' scratch.txt
cat scratch.txt

test_a PASS
test_b X
test_c PASS
test_d X

That edits the file in place with no backup — which is why the example copies results.txt to a disposable name first. Run the bare in-place command on results.txt itself and the rest of this chapter's examples will quietly give different answers, which is the hazard in miniature. On GNU sed, which is what most Linux distributions and therefore most compute farms provide, -i takes no argument and sed -i '' ... is an error instead — it reads the empty string as the script. The two forms are mutually exclusive.

9. Common Mistakes

Expecting substitution to replace every match on a line. s/x/y/ changes the first occurrence per line; s/x/y/g changes all of them.

Forgetting that sed printed, and expecting the file to have changed. Without -i the original is untouched. This is a feature.

Escaping slashes in a path instead of changing the separator. s|old/|new/| is much easier to read than the same thing with every slash escaped.

Trying to get a total out of sed. Section 4. sed can isolate the numbers; it cannot add them.

Using awk where sed is clearer. If the job is "replace this text everywhere and give me the file back", sed says that in a few characters and awk takes more work to say less clearly.

Assuming sed -i works the same everywhere. Section 8.

Double quotes around either program. The same hazard Chapter 1.1 flagged for awk applies to sed. Single-quote the program so the shell leaves it alone.

10. Exercises

Exercise 1 — Recognise the output shape

Before running anything, predict how many lines each of these produces from the four-line results.txt, then check.

test_a PASS
test_b FAIL
test_c PASS
test_d FAIL

sed 's/PASS/OK/' results.txt

test_a OK
test_b FAIL
test_c OK
test_d FAIL

Four lines, because sed edits and copies rather than selecting. If you predicted two — the number of lines it changed — that is exactly the habit this chapter is correcting.

Exercise 2 — Modify a demonstrated command

Rewrite every test_ prefix to tc_, keeping the file's shape.

sed 's/test_/tc_/' results.txt

tc_a PASS
tc_b FAIL
tc_c PASS
tc_d FAIL

No g is needed because test_ appears once per line. Adding g would be harmless here and necessary if a line held two test names.

Exercise 3 — Cross the boundary

Using cells.txt, report the number of cell types listed and the total count, as two numbers.

DFF 120
AND 45
OR 30
MUX 18

awk '{ n++; total += $2 } END { print n, total }' cells.txt

4 213

Two accumulators in one pass: n counts records, total adds the second field. This is firmly awk's side of the line — sed can neither count nor add.

Exercise 4 — Debug a failing command

An engineer wants the area values alone from area.rpt and runs sed 's/.*area //' area.rpt. They get the values but also the untouched third line. Explain why, then fix it.

u_core/u_alu  area 1820
u_core/u_regfile  area 2410
u_pad/u_io  area 640

sed -n 's/.*area //p' area.rpt

1820
2410
640

Their version printed every line, because sed copies lines it did not change — and all three lines do contain area, so all three were rewritten and printed. The problem is that plain sed prints unconditionally. Adding -n silences the automatic printing and p prints only the lines that were actually substituted. The same job in awk is awk '{ print $3 }', which needs no regular expression at all, and that is a fair illustration of when each tool is the simpler one.

Exercise 5 — Challenge: pick the tool

For each request, say sed or awk and why.

1. Replace every occurrence of u_old_name with u_new_name across a filelist, keeping the filelist usable.
2. Report the worst slack in a two-column timing table.
3. Strip a trailing ns from a column of values so something else can read them as numbers.
4. Count how many instances in an area report are above 1000.
5. Change VIOLATED to FAIL in a report and then count the failures.

Answers. 1 — sed: a consistent rewrite, output is still a filelist. 2 — awk: a worst case is arithmetic. 3 — sed: a text rewrite that keeps the file's shape, and a good example of a sed stage genuinely earning its place in front of awk. 4 — awk: a numeric comparison and a count. 5 — both, in that order, and this is the one worth thinking about: the rewrite is sed's and the count is awk's, but check first whether you need the rewrite at all — awk '$2 == "VIOLATED" { n++ }' counts them without renaming anything, and section 7 is about resisting the unnecessary stage.

11. Summary

• sed rewrites a stream and gives back text of the same shape. awk reads a stream and gives back values pulled out of it. Choose on the shape of the output you want.
• s/find/replace/ changes the first match per line; add g for every match. The separator need not be a slash, which makes path rewrites readable.
• sed has no fields and no arithmetic. It can isolate numbers with a substitution, and it cannot add them. That crossing is one-way.
• The overlap is real. sed selects lines with -n and p, addresses them by number, and deletes them; awk joins strings in a print. Overlap is a reason to choose deliberately, not to stop choosing.
• One report, two jobs. Renaming a hierarchy prefix in an area report is sed's; totalling the area column and filtering above a threshold is awk's.
• A sed stage in front of awk should earn its place by making the input genuinely easier, not by renaming something awk could match directly.
• sed -i is not portable. BSD sed requires a suffix argument and GNU sed forbids one. Prefer stream output, and never edit a tool report in place — it is the evidence.

You can now place a job on the sed-or-awk axis by asking what shape the answer should be, which is a different question from Chapter 1.3's and is the reason both chapters exist.

When the job really is a transformation rather than a measurement, sed is the better tool and is taught in full in the sed track.

Both comparisons so far have been against small, single-purpose commands. The next one is against a general-purpose programming language, where the trade-off stops being about capability and becomes about how much program the job deserves. Chapter 1.5 — awk vs Python works through the same tasks in both and is honest about where a one-liner stops being the right answer.