Stream Editing with sed, One Substitution at a Time

Part I · Module 13 · Chapter 13.6 · Stream Editing with sed

Chapter 13.5 stopped at one character for one character. This is the tool for everything longer.

1. Substitute

$ sed 's/WARN/warning/' log.txt
warning: one
INFO: two
warning: three

s/old/new/ replaces text. It is the command you will use ninety per cent of the time, and the shape is always the same: s, a delimiter, the thing to find, the delimiter, the replacement, the delimiter.

$ echo 'a-b-c' | sed 's/-/+/'
a+b-c

$ echo 'a-b-c' | sed 's/-/+/g'
a+b+c

Without g, only the first match on each line is replaced. With g — global — every match on the line is.

That default surprises people and is worth internalising: sed works line by line, and s/// acts once per line unless told otherwise. A substitution that fixed "some of them" is almost always a missing g.

The delimiter is yours to choose

$ echo '/usr/local/bin' | sed 's|/usr|/opt|'
/opt/local/bin

Any character can be the delimiter; sed takes whatever follows the s. With paths, | or # avoids escaping every slash:

sed 's/\/usr\/local/\/opt/'      # the notorious leaning-toothpick form
sed 's|/usr/local|/opt|'         # the same thing, readable

Use a delimiter that does not appear in your text. This single habit removes most of the visual noise people associate with sed.

2. Delete and Print

$ sed '/INFO/d' log.txt
WARN: one
WARN: three

/pattern/d deletes matching lines. The pattern before the command selects which lines it applies to.

$ sed -n '/WARN/p' log.txt
WARN: one
WARN: three

p prints a line, and -n suppresses the automatic printing that sed otherwise does for every line. Without -n you would see matching lines twice.

3. Selecting Lines

$ sed -n '2p' log.txt
INFO: two

$ sed -n '1,2p' log.txt
WARN: one
INFO: two

$ sed -n '100,120p' run.log

A number selects one line; N,M selects a range. That last one is genuinely useful on a large log — far cheaper than head -120 | tail -21 and clearer about intent.

The pattern range is the one grep cannot do:

sed -n '/BEGIN SIMULATION/,/END SIMULATION/p' run.log

Extracting one section from a long log, by its markers rather than by line number. That is a common need in EDA work and there is no simpler way to express it.

Negating

$ sed -n '/WARN/!p' log.txt
INFO: two

! inverts the selection — apply the command to lines that do not match. sed '/^#/d' and sed -n '/^#/!p' both strip comments; the first is clearer.

4. Patterns

sed uses regular expressions, which get a whole module of their own (Module 14). Four pieces cover most of what you will write here:

sed 's/^ *//'              # strip leading spaces
sed 's/ *$//'              # strip trailing spaces
sed 's/.*ERROR: //'        # keep only what follows "ERROR: "
sed 's/[0-9][0-9]*/N/g'    # replace every run of digits with N
sed '/^$/d'                # delete blank lines
sed '/^#/d'                # delete comment lines

That third one is worth a second look. .*ERROR: matches as much as possible — regular expressions are greedy — so on a line with two occurrences it keeps only what follows the last. That is usually what you want for a log prefix and occasionally a surprise.

The fourth is Chapter 13.4's normalisation step, which turns hundreds of counts of one into a useful frequency table.

Capturing part of the match

$ echo 'alu_seed7' | sed 's/.*_seed\([0-9]*\)/\1/'
7

\( \) captures and \1 refers back to it. The parentheses need backslashes in basic regular expressions, which is ugly and is what -E fixes:

$ echo 'alu_seed7' | sed -E 's/.*_seed([0-9]+)/\1/'
7

-E enables extended regular expressions, where (, ), +, ? and | work without backslashes. It is supported by both GNU and BSD sed and makes almost every non-trivial pattern more readable.

sed -E 's/run_([0-9]+)_seed([0-9]+)/test \1, seed \2/'

Two captures, reordered into a sentence. Module 14 covers the pattern language properly; -E is the flag to default to.

5. Editing a File In Place

Here is the thing that costs people the most time in this chapter.

$ sed -i 's/WARN/W/' t1.txt
sed: 2: "t1.txt
": undefined label '1.txt'

That error has nothing to do with labels. On BSD sed — which is what macOS ships — -i requires an argument: the suffix for a backup file. With none given, sed read s/WARN/W/ as the suffix and t1.txt as the script, where t is the label command.

$ sed -i '' 's/WARN/W/' t2.txt       # BSD: empty suffix means no backup
$ sed -i    's/WARN/W/' t2.txt       # GNU: no argument at all

Each form fails on the other system. The GNU form on BSD gives the nonsense above; the BSD form on GNU treats '' as the script and does nothing useful.

6. Several Commands at Once

sed -e 's/WARN/warning/' -e 's/INFO/info/' log.txt
sed 's/WARN/warning/; s/INFO/info/' log.txt

-e adds a command, and a semicolon separates them in one argument. Both forms are common; the semicolon version is more compact and the -e version is easier to comment line by line:

sed \
  -e 's/\r$//' \
  -e 's/^ *//' \
  -e '/^$/d' \
  filelist.f

Strip carriage returns, strip leading spaces, drop blank lines — three transformations in one pass, each on its own line where a reader can see it.

For anything longer, put the commands in a file:

sed -f cleanup.sed filelist.f

7. Where sed Stops

sed is line-oriented and pattern-based. Two things it is poor at:

Anything field-aware. "Replace the third comma-separated value" is awkward in sed and trivial in awk:

sed 's/^\([^,]*,[^,]*,\)[^,]*/\1NEW/'      # fragile and unreadable
awk -F, 'BEGIN{OFS=","} {$3="NEW"} 1'      # obvious

Anything spanning lines. sed holds one line at a time by default. Multi-line work is possible — there is a hold space and an N command — and it is where sed genuinely becomes a language to learn. If you need it, that is the signal to use awk, Python or Perl instead.

8. In Practice

# normalise a log before counting — the step from 13.4
sed -E 's/[0-9]+/N/g; s/0x[0-9a-fA-F]+/HEX/g' run.log \
  | grep ERROR | sort | uniq -c | sort -rn | head

# extract one phase from a long log
sed -n '/BEGIN ELABORATION/,/END ELABORATION/p' compile.log

# clean a filelist that arrived from somewhere else
sed -e 's/\r$//' -e 's/^[ \t]*//' -e 's/[ \t]*$//' -e '/^$/d' -e '/^#/d' raw.f > clean.f

# retarget a path across a config, portably
sed "s|/opt/tools/2023.03|/opt/tools/2024.09|g" setup.sh > setup.new && mv setup.new setup.sh

# pull the seed out of a run directory name
echo "run_alu_seed42" | sed -E 's/.*_seed([0-9]+)/\1/'

The filelist cleanup is the one to keep. Five transformations in one pass — carriage returns, leading and trailing whitespace, blank lines, comments — and it turns a file that arrived from a colleague's Windows machine into something a tool will accept.

Practice

mkdir -p /tmp/sedd && cd /tmp/sedd
printf 'WARN: one\nINFO: two\nWARN: three\n' > log.txt
printf 'timeout after 1203 cycles\ntimeout after 1204 cycles\n' > t.log
printf '# comment\n\n  rtl/alu.sv  \nrtl/fifo.sv\n' > raw.f

1 — Substitute, with and without g.

sed 's/WARN/warning/' log.txt
echo 'a-b-c' | sed 's/-/+/'
echo 'a-b-c' | sed 's/-/+/g'

2 — Choose a better delimiter.

echo '/usr/local/bin' | sed 's/\/usr\/local/\/opt/'
echo '/usr/local/bin' | sed 's|/usr/local|/opt|'

3 — Delete and select.

sed '/INFO/d' log.txt
sed -n '/WARN/p' log.txt
sed -n '2p' log.txt
sed -n '1,2p' log.txt
sed -n '/WARN/!p' log.txt

4 — Extract a section by markers.

printf 'before\nBEGIN\nmiddle\nEND\nafter\n' > sect.txt
sed -n '/BEGIN/,/END/p' sect.txt

5 — Normalise digits before counting.

sort t.log | uniq -c
echo "--- normalised ---"
sed 's/[0-9][0-9]*/N/g' t.log | sort | uniq -c

6 — Capture, both ways.

echo 'alu_seed7' | sed 's/.*_seed\([0-9]*\)/\1/'
echo 'alu_seed7' | sed -E 's/.*_seed([0-9]+)/\1/'

7 — Meet the in-place trap.

cp log.txt a.txt; sed -i 's/WARN/W/' a.txt
cp log.txt b.txt; sed -i.bak 's/WARN/W/' b.txt && head -1 b.txt && ls b.txt.bak
cp log.txt c.txt; sed 's/WARN/W/' c.txt > c.new && mv c.new c.txt && head -1 c.txt

The first may fail. Read the error and work out what sed thought you meant.

8 — Chain several commands.

sed -e 's/\r$//' -e 's/^[ \t]*//' -e 's/[ \t]*$//' -e '/^$/d' -e '/^#/d' raw.f

9 — Watch the quoting trap.

sed 's/$HOME/X/' <<< 'my $HOME here'
sed "s/\$HOME/X/" <<< 'my $HOME here'

10 — Clean up. cd /tmp && rm -rf /tmp/sedd

Check Your Understanding

1. sed 's/-/+/' on a-b-c gives a+b-c. Why, and what fixes it?

2. Why do -n and p always appear together?

3. sed -i 's/a/b/' file gives undefined label. What happened, and what is the portable form?

4. When is sed -n '/x/,/y/p' the right tool where grep is not?

5. Why should a sed script be single-quoted?

6. sed 's/.*ERROR: //' on a line containing two ERROR: prefixes keeps only the text after the second. Why?

9. Remember

• s/old/new/ is ninety per cent of real sed use. Learn it first and well.
• Without g, only the first match on each line is replaced.
• Any character can be the delimiter. Use | or # for paths and the escaping disappears.
• /pattern/d deletes lines; -n with p prints selected ones. The two always travel together.
• Line selectors: 5, 5,10, 5,$, /start/,/end/, $, and ! to invert.
• The marker range is what grep cannot do — extracting a section by its banners.
• -E gives extended regular expressions, so (, ), + and ? work without backslashes. Default to it.
• \1 refers to a captured group, which is how you reorder or extract part of a match.
• sed -i differs between systems. BSD needs a suffix; GNU must not have one. sed -i.bak works on both.
• sed '...' f > f empties the file. Use a temporary and && mv in anything portable.
• -e or a semicolon chains commands in one pass.
• Single-quote the script, so the shell leaves it alone.
• .* is greedy and matches as much as it can.
• sed is line-oriented and pattern-based. Anything addressed by field number, or spanning lines, belongs to awk.

Which is the next chapter — and the point at which the toolkit stops selecting and starts calculating.