sed vs grep, awk, and Python

Part I · Module 1 · Chapter 1.3 · sed vs grep, awk, and Python

1. Four Tools, One Log

Chapter 1.2 sorted the files a flow produces by how risky they are to edit. This chapter takes one of those files and asks a different question: once you know a text job is reasonable, which tool should do it?

The honest answer is that these four overlap heavily, and a tutorial that draws hard walls between them will mislead you the first time you read a real script. What actually separates them is the shape of the question each one answers without effort.

• grep answers which lines match.
• sed answers rewrite each line as it goes past.
• awk answers reason across the fields of each record, and remember things between records.
• Python answers this needs structure, real logic, and tests of its own.

Reach for the one whose natural question is the question you actually have. When you find yourself fighting a tool, that is usually the signal that you picked the wrong one — not that you need a cleverer expression.

Here is the log the next four sections work on. It is simplified example data, not the output of any particular simulator.

UVM_INFO @ 0: reporter [RNTST] Running test apb_burst_test
UVM_ERROR @ 4500: uvm_test_top.env.sb [MISCMP] addr=0x40 exp=0xA5 act=0x5A
UVM_INFO @ 5000: uvm_test_top [SEQ] retrying
UVM_ERROR @ 6200: uvm_test_top.env.sb [MISCMP] addr=0x40 exp=0xA5 act=0x5A
UVM_ERROR @ 7100: uvm_test_top.env.sb [MISCMP] addr=0x44 exp=0x3C act=0x00
UVM_FATAL @ 9100: uvm_test_top [TIMEOUT] no response

2. Which Lines? That Is grep

The first question anyone asks a log is did anything fail, and where. That is selection, and grep exists for it.

grep UVM_ERROR sim.log

UVM_ERROR @ 4500: uvm_test_top.env.sb [MISCMP] addr=0x40 exp=0xA5 act=0x5A
UVM_ERROR @ 6200: uvm_test_top.env.sb [MISCMP] addr=0x40 exp=0xA5 act=0x5A
UVM_ERROR @ 7100: uvm_test_top.env.sb [MISCMP] addr=0x44 exp=0x3C act=0x00

Three lines out of six, printed unchanged. grep's default job is to decide, per line, whether it matches, and to print the ones that do. No pattern of yours describes an output format, because grep is not rewriting anything.

Counting is the same question with a different answer shape:

grep -c UVM_ERROR sim.log

3

Hold on to that 3. Section 3 shows why it is a dangerous number to report.

3. Rewrite Each Line? That Is sed

Selection tells you what is there. Transformation changes it on the way past, which is the behaviour Chapter 1.1 installed. A timing team respinning a corner needs every reference to one corner name replaced across a signoff script:

set corner ss_0p72v_125c
read_sdc constraints/ss_0p72v_125c.sdc
report_timing -corner ss_0p72v_125c

sed 's/ss_0p72v_125c/ss_0p70v_125c/g' corners.tcl

set corner ss_0p70v_125c
read_sdc constraints/ss_0p70v_125c.sdc
report_timing -corner ss_0p70v_125c

Three lines in, three lines out, each one rewritten, the file on disk untouched. grep could have told you which lines mention the corner. It would not have produced the corrected script.

Now the 3 from section 2. Those three UVM_ERROR lines are not three failures. Strip the message prefix and look at what remains distinct:

grep UVM_ERROR sim.log | sed 's/.*\[MISCMP\] //' | sort -u

addr=0x40 exp=0xA5 act=0x5A
addr=0x44 exp=0x3C act=0x00

Two. The 0x40 mismatch was reported twice, once before a retry and once after. A count of matching lines is a count of messages, and one root cause commonly prints many. This is the first place in the module where the tool did exactly what it was asked and the engineering conclusion would still have been wrong — the distinction Chapter 1.2's section 7 made about exit status, now applied to a number.

Notice also that this command is a pipeline: grep narrows, sed reshapes, sort -u removes duplicates. That is how these tools are actually used.

4. Where They Genuinely Overlap

Before going further it is worth killing three claims that tutorials repeat and real scripts contradict.

sed can select lines. The -n and p combination from 1.1 does grep's job:

sed -n '/UVM_ERROR/p' sim.log | wc -l

       3

grep can extract, not just select. With -o it prints the matching part rather than the whole line:

grep -o 'addr=0x[0-9A-F]*' sim.log | sort -u

addr=0x40
addr=0x44

awk can rewrite text. Its sub function does a substitution, and print emits the modified record:

awk '{ sub(/ss_0p72v_125c/, "ss_0p70v_125c"); print }' corners.tcl

set corner ss_0p70v_125c
read_sdc constraints/ss_0p70v_125c.sdc
report_timing -corner ss_0p70v_125c

Identical output to the sed command in section 3.

So the choice is not about capability. It is about which tool states your intent most directly, which one a colleague will read correctly at review, and which one you can verify. For a straight token rewrite the sed form says substitute this for that and nothing else; the awk form says the same thing with a function call and an explicit print, which is more moving parts for no gain. Reverse the job — sum a column — and the preference reverses with it.

5. Reason Across Fields? That Is awk

Here is a synthesis area report. The third column is an area figure; the second is a cell category.

alu_core       combinational   1842.50
regfile        sequential      3120.75
fifo_ctrl      combinational    640.25
apb_bridge     sequential       915.00

The total area is a question about a column, and this is where the other tools start to strain:

awk '{ total += $3 } END { print total }' area.rpt

6518.5

Two things happened that neither grep nor sed gives you. awk split each line into fields automatically, so $3 is the area without any pattern describing the whitespace. And total persisted between lines, with END running once after the last record. That is state, and it is the real dividing line: grep and sed look at one line and forget it, while awk accumulates.

Add a condition and it becomes a grouped summary:

awk '$2 == "sequential" { n++; a += $3 } END { print n, a }' area.rpt

2 4035.75

Two sequential blocks totalling 4035.75. Writing that with sed would mean inventing arithmetic that sed does not have, and writing it with grep is not possible at all — this is genuinely awk's question. The awk track covers the language properly; the point here is only to recognise the shape of a job that belongs to it.

6. Structure and Tests? That Is Python

Some jobs are not a line question, a rewrite or a column sum. Suppose you want each distinct mismatch address and how many times it was reported — the de-duplication from section 3, but keeping the counts:

python3 report.py

0x40 reported 2 time(s)
0x44 reported 1 time(s)
distinct addresses: 2

The program behind it:

import re, collections

pat = re.compile(r"UVM_ERROR .*\[MISCMP\] addr=(\S+) exp=(\S+) act=(\S+)")
seen = collections.Counter()

for line in open("sim.log"):
    m = pat.search(line)
    if m:
        seen[m.group(1)] += 1

for addr, n in sorted(seen.items()):
    print(f"{addr} reported {n} time(s)")
print("distinct addresses:", len(seen))

An awk one-liner can also count into an array, and for this size that would be a reasonable choice. What Python adds is everything around the logic: pat is a named thing you can test in isolation, the counting is an ordinary data structure, and the whole file can have a test that feeds it three known lines and asserts the answer. The moment a transformation becomes something a team depends on — something that will be changed next quarter by someone who did not write it — those properties matter more than brevity.

The cost is proportionate. A ten-line program needs a file, a home for that file, and someone to maintain it. For a one-off rewrite of a corner name, that overhead buys nothing, and the sed command in section 3 is the better engineering answer precisely because it is disposable.

7. The Problem None of Them Solves

Everything so far has been a choice between tools. This section is about a job where the choice does not matter, because all four fail the same way.

A design review asks you to rename the signal clk_en to gate_en. Here is the file:

// drive clk_en from the gating cell
logic clk_en;
logic clk_en_q;
assign clk_en = cfg.clk_enable & ~scan_mode;
always_ff @(posedge clk) clk_en_q <= clk_en;
`define CLK_EN_POLARITY 1

The obvious command:

sed 's/clk_en/gate_en/g' snippet.sv

// drive gate_en from the gating cell
logic gate_en;
logic gate_en_q;
assign gate_en = cfg.gate_enable & ~scan_mode;
always_ff @(posedge clk) gate_en_q <= gate_en;
`define CLK_EN_POLARITY 1

Three things are now wrong. clk_en_q is a different signal and has been renamed. cfg.clk_enable is a different field of a config object and has been renamed. The comment was rewritten, which is harmless here and would not be if it named something else. The command exited zero and changed every line it touched.

The experienced reaction is to add word boundaries. On BSD sed that attempt fails in a way worth seeing:

sed 's/\bclk_en\b/gate_en/g' snippet.sv

// drive clk_en from the gating cell
logic clk_en;
logic clk_en_q;
assign clk_en = cfg.clk_enable & ~scan_mode;
always_ff @(posedge clk) clk_en_q <= clk_en;
`define CLK_EN_POLARITY 1

Nothing changed, and the exit status was 0. \b is a GNU extension; BSD sed reads it as a literal b, which matches nothing here. On GNU sed the same command would perform the rename. This is the portability trap from Chapter 1.2, and it is why the next command uses the BSD form:

sed -E 's/[[:<:]]clk_en[[:>:]]/gate_en/g' snippet.sv

// drive gate_en from the gating cell
logic gate_en;
logic clk_en_q;
assign gate_en = cfg.clk_enable & ~scan_mode;
always_ff @(posedge clk) clk_en_q <= gate_en;
`define CLK_EN_POLARITY 1

Better. clk_en_q and cfg.clk_enable are correctly left alone. The comment is still rewritten, because a word boundary knows nothing about comments.

Now the part that matters. Do the same job in Python, the most capable tool here:

python3 -c "import re; print(re.sub(r'\bclk_en\b','gate_en',open('snippet.sv').read()),end='')"

// drive gate_en from the gating cell
logic gate_en;
logic clk_en_q;
assign gate_en = cfg.clk_enable & ~scan_mode;
always_ff @(posedge clk) clk_en_q <= gate_en;
`define CLK_EN_POLARITY 1

Byte-identical to the sed output — the two were compared and their checksums match. Python's extra power bought exactly nothing, because the limitation was never the tool. It is that a regular expression matches characters, and a correct rename depends on meaning: whether this clk_en is the signal you mean, whether it is inside a comment or a string, whether a macro expands into it, whether some other scope declares a different signal with the same name.

That is refactoring, and it needs something that has parsed SystemVerilog — a linter or editor with language awareness, or an elaboration-based tool — followed by a compile and a review. Chapter 9.3 takes the boundary up in full. The lesson for tool selection is narrower and worth stating plainly: when a job needs semantics, moving up the power ladder from grep to sed to awk to Python does not help, because none of them is on a ladder that leads there.

8. Choosing, in Practice

Six jobs, and the reasoning rather than the verdict.

Find every UVM_ERROR message. grep. It is selection, the output is lines, and nothing needs rewriting.

Replace a controlled path prefix in a generated filelist. sed. One line-local substitution, no state, no arithmetic.

Sum a numeric column in a report. awk. Fields come free and the running total needs state.

Summarise records that span several lines. awk, if the grouping is simple enough to hold in variables; Python once the record structure needs a data model. sed can hold state across lines, but Module 6 shows what that costs in readability — reach for it when you are already in sed, not as a first choice.

A transformation the team will rely on and change later. Python. Not because it is more powerful, but because it can be named, imported, reviewed and tested, and a one-liner cannot.

Rename an HDL identifier correctly. None of them. Section 7 is the argument.

Two questions settle most real cases. What shape is my question — lines, a rewrite, fields, or structure? And who has to maintain this, and how will they know it still works? The first picks the tool; the second decides whether a one-liner is acceptable at all.

9. Think It Through

Three short problems. Work each one out before reading the answer.

1. A colleague reports "7 errors in the nightly run" from grep -c UVM_ERROR. What do you ask?

Whether those are seven distinct failures or seven messages. Section 3 showed three lines collapsing to two root causes; the ratio is usually worse on a real run, where one broken bus transaction can report on every beat. The useful follow-up is to de-duplicate on the part of the message that identifies the failure, and to check whether the run even reached the end — a test that died during elaboration prints no errors at all.

2. You need the average of column 3 rather than the total. Which tool, and why not sed?

awk. You already have total += $3; an average needs a count as well, which is a second piece of state, and both are naturally available at END. sed has no arithmetic and no concept of a field, so expressing this in sed would mean building both from scratch in a language that offers neither.

3. This command is meant to rename a corner across every signoff script, and it changes nothing. Why?

sed 's/\bss_0p72v_125c\b/ss_0p70v_125c/g' sta/corners.tcl

\b is a GNU extension. On BSD sed it is a literal b, so the pattern looks for bss_0p72v_125c b and finds nothing — no match, no error, exit status 0. On a GNU system the same command works, which is exactly what makes it dangerous: it will pass on one engineer's machine and silently do nothing on another's. The portable fix is to drop the boundaries, since a corner name is already distinctive, or to use the form each sed supports. Checking that the output actually changed is the habit that catches it either way.

10. Summary

• The four tools overlap heavily. sed can select, grep can extract, awk can rewrite. Capability is not what separates them.
• Pick by the shape of the question. Which lines — grep. Rewrite each line — sed. Reason across fields, with state — awk. Structure, logic and tests — Python.
• State is the real boundary between sed and awk: grep and sed consider one line and forget it; awk accumulates across records and can act at END.
• Pipelines are normal. grep | sed | sort -u is not a compromise; narrowing before rewriting is good practice.
• A count of matching lines is not a count of failures. Three error messages were two defects, and the command was correct in both readings.
• Python earns its cost through maintainability, not power — a named, importable, testable unit. For a disposable one-off rewrite that overhead buys nothing.
• Portability is part of correctness. \b is GNU-only; on BSD sed it matches nothing and exits 0.
• No text tool renames an HDL identifier safely. A word-boundary rename in Python was byte-identical to the same rename in sed, and both were wrong in the same way, because the limit is text matching rather than tool power.

You can now defend a tool choice in engineering terms rather than by habit, and recognise the one class of job where the whole comparison is beside the point.

What remains is the mechanics. You have read a good many sed commands across three chapters without yet running one end to end and checking the result. Chapter 1.4 — Running Your First sed Commands closes Module 1 by doing exactly that: file input and standard input, what success looks like, and how to tell a real substitution from a silent no-op.