sed Syntax, Commands, and Options

Part I · Module 2 · Chapter 2.3 · sed Syntax, Commands, and Options

1. The Shape of an Invocation

Every sed command line has the same three parts, in this order:

sed  [options]  script  [file ...]
     └───┬───┘  └──┬─┘  └───┬────┘
   how to behave  what    where to
                 to do    read it

In the simplest form all three are visible at once:

sed 's/alu/ALU/' filelists/top.f

/proj/rtl/ALU.sv
/proj/rtl/fifo.sv
/proj/rtl/regfile.sv
/proj/rtl/mux.sv
/proj/rtl/top.sv
/proj/rtl/pkg.sv

No options, one script (s/alu/ALU/), one file. The script is quoted because it contains characters the shell would otherwise interpret — Module 8 covers quoting properly; for now, single-quote every script and you will not be bitten.

Two parts are optional and one is not. Drop the file and sed reads standard input, which is what makes it a pipeline tool. Drop the options and it uses defaults. Drop the script and nothing works, because of a rule worth stating plainly.

2. The First Operand Is the Script

sed's grammar is positional. If you have not supplied a script with -e or -f, the first thing on the command line that is not an option is the script — whatever it looks like.

This is the mistake everybody makes once:

sed filelists/top.f

sed: 1: "filelists/top.f
": invalid command code f

Exit status 1. sed took filelists/top.f as the script, tried to read it as sed commands, and reported that f — the first character of filelists — is not a command it knows. The message is confusing precisely because it answers a question you did not think you were asking.

Once you know the grammar, the message is self-explanatory: "line 1 of your script says filelists/top.f, and I do not have a command called f." If you ever see invalid command code followed by a character that happens to start a filename, this is what happened — you forgot the script, or you forgot its quotes.

3. Options Come Before the Script — and Fail Quietly If Not

Since the first non-option operand is the script, everything after the script is a file. Including things that look like options.

sed 's/alu/ALU/' -n filelists/top.f

sed: -n: No such file or directory
/proj/rtl/ALU.sv
/proj/rtl/fifo.sv
/proj/rtl/regfile.sv
/proj/rtl/mux.sv
/proj/rtl/top.sv
/proj/rtl/pkg.sv

Read that carefully, because three different things happened:

1. -n was treated as a filename. sed tried to open it, failed, and wrote sed: -n: No such file or directory to stderr.
2. It then carried on to the next operand, filelists/top.f, processed it normally, and wrote six correct-looking lines to stdout.
3. It exited with status 1.

So the substitution worked, the auto-print suppression silently did not, and the only evidence is on stderr and in the exit status.

4. -n — Suppress Auto-Print

Chapter 2.2 established what -n does: it removes the automatic print at the end of each cycle, so nothing reaches the output unless a command asks for it. The practical consequence is that -n almost always travels with a p.

sed -n '/UVM_ERROR/p' sim.log

UVM_ERROR @ 980 ns: /runs/20261012/sim.log mismatch

One matching line, printed once — a filter. Without -n, the same script prints every line and duplicates the matches:

sed '/UVM_ERROR/p' sim.log

UVM_INFO @ 120 ns: /runs/20261012/sim.log started
UVM_ERROR @ 980 ns: /runs/20261012/sim.log mismatch
UVM_ERROR @ 980 ns: /runs/20261012/sim.log mismatch

There is a second, denser idiom worth learning now, because it is everywhere in real scripts: -n combined with the p flag on a substitution, which prints only the lines that were actually changed.

sed -n 's/alu/ALU/p' filelists/top.f

/proj/rtl/ALU.sv

Six lines in, one line out — the only line the substitution touched. This is the standard way to audit an edit before running it for real: it shows you the new text of exactly the lines that would change, and stays silent about everything else. Module 7 builds its dry-run discipline on top of it.

The registered example for this chapter pairs -n with a line-range address:

sed -n -e '1,5p' filelists/top.f

/proj/rtl/alu.sv
/proj/rtl/fifo.sv
/proj/rtl/regfile.sv
/proj/rtl/mux.sv
/proj/rtl/top.sv

The first five lines of a six-line filelist. 1,5 is an address range — addresses are Chapter 4.4's subject, and they are used here only to have something concrete for -n and -e to act on.

5. -e and -f — Where the Script Comes From

-e adds a script fragment. With a single fragment it is redundant:

sed -e 's/alu/ALU/' filelists/top.f

/proj/rtl/ALU.sv
/proj/rtl/fifo.sv
/proj/rtl/regfile.sv
/proj/rtl/mux.sv
/proj/rtl/top.sv
/proj/rtl/pkg.sv

Identical to section 1's output, because one -e fragment is just the script. -e earns its place when there is more than one, and the fragments then run in the order written — which is the whole subject of Chapter 2.2:

sed -n -e 's/alu/ALU/p' -e 's/fifo/FIFO/p' filelists/top.f

/proj/rtl/ALU.sv
/proj/rtl/FIFO.sv

-e is also what you need when a fragment cannot sit next to another on one line, and it is the explicit form that makes a multi-command script readable in a shell script — one -e per line, each one reviewable on its own.

-f reads the script from a file. Once an edit has more than two or three commands, or is going to be reviewed, or is going to be reused across a regression, it belongs in a file rather than in shell quoting. Here is a two-command script that normalises a simulation log so two runs can be diffed:

cat -n scripts/normalize-log.sed

     1	s/@ [0-9]* ns/@ <time>/
     2	s|/runs/[0-9]*/|/runs/<id>/|

cat -n sim.log

     1	UVM_INFO @ 120 ns: /runs/20261012/sim.log started
     2	UVM_ERROR @ 980 ns: /runs/20261012/sim.log mismatch

sed -f scripts/normalize-log.sed sim.log

UVM_INFO @ <time>: /runs/<id>/sim.log started
UVM_ERROR @ <time>: /runs/<id>/sim.log mismatch

Timestamps and run IDs are replaced with placeholders, so the two logs differ only where the behaviour differed. Notice the second command uses | as its delimiter rather than /, because the pattern contains path separators — a technique Chapter 1.4 introduced and Module 3 formalises.

A script file needs no quoting, can carry comments, and shows up in a diff when it changes. If a sed command is going to outlive the afternoon, this is the form it should be in.

Finally, -e and -f can be combined, and the fragments are concatenated into one script in command-line order:

sed -e 's/UVM_INFO/INFO/' -f scripts/normalize-log.sed sim.log

INFO @ <time>: /runs/<id>/sim.log started
UVM_ERROR @ <time>: /runs/<id>/sim.log mismatch

One command from the command line, two from the file, all three applied to each line in order.

6. -E — Extended Regular Expressions

sed understands two regular-expression dialects. By default it uses POSIX basic regular expressions (BRE); -E switches it to extended ones (ERE).

The practical difference is which characters are operators. Under -E, (, ), |, + and ? are operators directly:

sed -n -E '/(alu|fifo)/p' filelists/top.f

/proj/rtl/alu.sv
/proj/rtl/fifo.sv

In BRE those same characters are literal text unless backslashed — and this is where a real portability trap lives. The BRE spelling with backslashed alternation is a GNU extension, not POSIX. On the BSD sed used for these transcripts:

sed -n '/\(alu\|fifo\)/p' filelists/top.f

No output. And — this is the part that matters — exit status 0. sed did not complain; it simply looked for a literal | inside a group, found none, and matched nothing. A script written on a GNU Linux machine and run on macOS will quietly return zero results and report success.

Two portable ways out, both verified:

sed -n -E '/(alu|fifo)/p' filelists/top.f

/proj/rtl/alu.sv
/proj/rtl/fifo.sv

sed -n '/alu/p;/fifo/p' filelists/top.f

/proj/rtl/alu.sv
/proj/rtl/fifo.sv

-E is specified by POSIX and present in both GNU and BSD sed, which makes it the better habit for anything with alternation or grouping. Chapter 4.1 covers the two dialects properly, including the inversion that catches people out — +incdir+ needs no escaping in BRE and does need it under -E.

7. Which sed Am I Running?

Because the dialects and some options differ, it is worth being able to answer this in one command. GNU sed implements --version and prints its version. BSD sed does not implement it at all, and the failure is itself the answer:

sed --version

sed: illegal option -- -
usage: sed script [-EHalnru] [-i extension] [file ...]
	sed [-EHalnu] [-i extension] [-e script] ... [-f script_file] ... [file ...]

Exit status 1. If you get a version string you are on GNU sed; if you get that usage block you are on the BSD lineage, which is what macOS ships.

The usage block is also a compact reference worth reading once. Two details in it:

• The option letters it accepts are listed — a quick check for whether an option exists at all on this sed.
• -i extension takes an argument here, where GNU's -i does not. That difference is the single most common portability failure in real scripts, and it is Chapter 7.1's subject. -i edits files in place; it is deliberately not demonstrated in this chapter, because nothing you run while learning to read command lines should be writing to your disk.

8. Think It Through

1. What is wrong with sed -i 's/old/new/' file.sv when run on macOS?

-i on BSD sed requires an argument — the backup extension — so it consumes s/old/new/ as that argument, which leaves file.sv as the script. Run on a disposable one-line file, that is exactly what happens: sed: 1: "file.sv": invalid command code f, exit status 1, and the file unchanged. It is section 2's error message arriving by a different route. The portable BSD form supplies an explicit empty extension, sed -i '' 's/old/new/' file.sv, which edits the file as intended. GNU sed's -i takes no argument, so the two forms are not interchangeable in either direction. Chapter 7.1 covers in-place editing properly; it is in this chapter's exercises because the reason for the failure is pure argument grammar, which is what section 2 just taught.

2. A regression script runs sed 's/+UVM_TESTNAME=smoke/+UVM_TESTNAME=stress/' run.sh -n > run.new.sh and the new file looks correct. What is the defect?

-n is after the script, so it is being treated as a filename. sed emits a No such file or directory diagnostic on stderr and exits 1, but it still processes run.sh, so run.new.sh is written and looks right. The -n had no effect — which in this particular command is harmless, since nothing needed suppressing. The real defect is a command line that returns a failing status while producing usable output; the next person to add set -e to that script will see it break for no visible reason.

3. You are handed sed -n -E -e '/^(input|output)/p' -e '/^wire/p' rtl/top.sv. Describe it part by part.

Three options: -n suppresses auto-print, -E selects extended regular expressions, and the two -e fragments make up the script. The script has two commands, each an address plus p: print lines starting with input or output, and print lines starting with wire. One file operand. Because auto-print is off, the output is exactly the lines matched by either command — and a line matching both would be printed twice, which the cycle in Chapter 2.2 explains.

4. Why would you move a working one-liner into a -f script file?

Because quoting stops being free as the script grows, because comments become possible, because it can be reviewed in a diff and committed rather than retyped from shell history, and because -f is how the same edit gets applied identically across many files and many runs. The trade is one more file to keep in sync with the task it was written for.

9. Summary

• The shape is sed [options] script [file ...], and the order is not negotiable.
• The first non-option operand is the script unless -e or -f supplied one. sed file.f therefore fails with invalid command code f.
• Anything after the script is a file, so an option written there becomes a filename: plausible stdout, a diagnostic on stderr, exit 1. Options first, and check the status.
• No file operand means standard input, which is what makes sed a pipeline stage.
• -n removes auto-print; pair it with p. sed -n 's/x/y/p' prints only the lines an edit would change — the basis of a dry run.
• -e adds a script fragment and makes multi-command scripts readable; -f reads the script from a file, which is where any edit that will be reused or reviewed belongs. The two can be combined and concatenate in order.
• -E selects extended regular expressions and is portable. The BRE \| alternation is a GNU extension that fails on BSD sed by matching nothing and returning 0.
• sed --version identifies GNU sed; on BSD it fails and prints a usage block that tells you the same thing.

One part of the shape is still unexamined: the file operands. Giving sed several files is not the same as running it several times — the files become one stream, so line numbers continue across the boundary and $ means the last line of the last file. Chapter 2.4 — Input, Output, and Multiple Files works through that, and it closes Module 2.