What Is sed and Why Does It Exist?

Part I · Module 1 · Chapter 1.1 · What Is sed and Why Does It Exist?

1. The Problem sed Was Built For

Suppose you have a file with three lines in it.

hello student
hello engineer
hello designer

You need every hello to say welcome instead. With three lines, you would open the file in an editor and fix them by hand. That is perfectly reasonable.

Now change the numbers. The file has three thousand lines. Or there are three hundred files, each needing the same change. The work has not become harder — it is the same edit, repeated — but doing it by hand has become slow, dull, and unreliable. Dull and repetitive is exactly when a person starts missing lines.

This shape of work comes up constantly:

• A library moved, and every file that points at the old location needs updating.
• A tool was upgraded, and an option it used to accept is now spelled differently.
• A log file is forty thousand lines long, and you want to read the few that matter.

In each case the change is mechanical. Something needs to be found, and something else needs to go in its place. Nothing about it requires judgement, which means it does not require a person.

sed exists for that. You describe the change once, and sed applies it to every line.

The name is short for stream editor. The editor part is familiar: it changes text. The stream part is what makes it different from the editor you already know. A normal editor opens a file, holds it on screen, and waits for you to type. sed does not open anything on screen and never waits for you. Text flows through it, and sed changes it on the way past. Three thousand lines take longer than three — there is more text to read — but the command you write is the same either way, and it fits inside automated scripts that run with nobody watching.

2. What sed Actually Does

The model is short enough to state in one line:

input text → sed command → output text

Five words name the parts of it, and they are worth learning now because every later chapter uses them.

The input is the text going in — from a file you name, or from another command. The command is what you want done; in this chapter it is always a substitution: find something, replace it. The output is the text coming out, after the change.

Inside the command, the pattern is what sed looks for on each line, and the replacement is what sed puts there instead.

Two facts about that picture matter more than anything else on this page.

First, by default, sed reads and processes input one line at a time. It takes a line, applies your command to it, sends the result out, and then reads the next one. It does not normally need the whole file in memory at once, which is what makes a 50 MB report workable: reading it still takes longer than reading three lines, but sed's memory use stays roughly the same. (Some of sed's advanced commands deliberately work across more than one line. Module 6 covers those; nothing in this chapter does.)

Second, the commands in this chapter print their results and leave the input file alone. This is the fact beginners are most often surprised by. sed reads the file and prints the changed version for you to look at. The file on disk is exactly as it was.

That is not true of sed in general — sed does have a way to write changes back, which is why Module 7 is devoted to doing it safely. But it is true of everything on this page, and while you are learning it is useful: a command that turns out to be wrong has printed something wrong, and that is all it has done.

3. Your First sed Command

Here is the smallest useful thing sed can do. Type it exactly as written.

echo "hello world" | sed 's/world/VLSI/'

hello VLSI

Short as it is, that line has four parts. Taking them in order:

echo "hello world" prints the words hello world. On its own, that is all echo does — it writes its argument out. We use it here because it is the quickest way to produce one line of text to experiment on, with no file needed.

| is called a pipe. It connects two commands: it takes whatever the command on the left printed and feeds it to the command on the right as input. So the text hello world does not appear on your screen — it goes into sed instead. A pipe is how Unix commands are joined together, and you will meet it constantly.

sed is the command itself.

's/world/VLSI/' is the instruction you are giving sed. Read it one character at a time:

• s means substitute. It is sed's find-and-replace command.
• / is a separator. It marks where each part of the instruction begins and ends. sed needs some character for this job, and / is the traditional choice.
• world is the pattern — the text to look for.
• VLSI is the replacement — the text to put there instead.
• The final / closes the instruction.

So s/world/VLSI/ reads as: substitute, find world, replace it with VLSI. sed received the line hello world, found world in it, put VLSI there, and printed the result.

4. The Same Idea on a File

Pipes are convenient for experiments. Real work usually involves a file, and the only change is that you name the file instead of piping text in.

Create a file called messages.txt with these three lines:

hello student
hello engineer
hello designer

Now run the substitution, giving sed the filename:

sed 's/hello/welcome/' messages.txt

welcome student
welcome engineer
welcome designer

All three lines changed, and you wrote the instruction once. This is section 2 happening in front of you: sed read the first line, applied s/hello/welcome/, printed the result, then did the same to the second line, then the third. Your instruction says nothing about repeating, and it does not need to. Going through the input line by line is simply what sed does.

Now the part worth pausing on. Look at the file again:

cat messages.txt

hello student
hello engineer
hello designer

The file is unchanged. Every hello is still a hello — the output you saw was sed's result, printed to your screen and then gone.

This catches almost everyone once, and while you are learning it works in your favour: you can run a command, read the output, and decide whether it did what you meant, knowing the file is intact either way. That order — run it, read it, then decide — is the foundation of the review discipline in Module 7.

5. A Practical Example: A Simulation Log

Here is the same single idea on text closer to real work. Verification and simulation tools produce log files, and a log is simply lines of text.

INFO: Simulation started
ERROR: Timeout detected
INFO: Simulation stopped

Suppose you want to see that log with the word ERROR shown as ISSUE. The command has the shape you already know:

sed 's/ERROR/ISSUE/' simulation.log

INFO: Simulation started
ISSUE: Timeout detected
INFO: Simulation stopped

Two of the three lines came through untouched. That is worth noticing: a line with no match for the pattern is not skipped or dropped — it is printed exactly as it arrived. sed passes everything through, and changes only what matches.

And as in section 4, simulation.log itself still says ERROR — you changed what was displayed, nothing more.

6. A Semiconductor Example: Updating a Filelist

Now the job sed is genuinely reached for every week.

To compile a design, a tool needs to be told which source files to read. That list usually lives in a plain text file — often with a .f extension — holding one path per line. It is called a filelist, and it is just text.

/old_lib/rtl/top.sv
/old_lib/rtl/fifo.sv
/old_lib/tb/top_tb.sv

Then the library moves. The files that used to live under /old_lib/ now live under /new_lib/. The filelist still points at the old place, so a compile using it will fail to find the sources. Every line needs the same small correction.

Nothing about this needs judgement, so it is sed's kind of problem:

sed 's|/old_lib/|/new_lib/|g' files.f

/new_lib/rtl/top.sv
/new_lib/rtl/fifo.sv
/new_lib/tb/top_tb.sv

Two things in that command are new. Both are small.

The separator is | instead of /. sed does not insist on / — the character immediately after the s becomes the separator for that command. Here the pattern and replacement are both paths, and paths are full of slashes. With / as the separator each of those slashes would have to be marked so sed did not mistake it for the end of a part, and the command would read s/\/old_lib\//\/new_lib\//. That is the same instruction, and it is much harder to read. Choosing a separator that does not appear in your text is a habit worth picking up immediately. Chapter 3.3 covers separators and escaping in full.

The g on the end stands for global. Without it, sed replaces only the first match on each line. With it, sed replaces every match on the line.

Here is where this chapter has to be careful with you. On files.f above, g changes nothing at all: each of those three lines contains /old_lib/ exactly once, so there is no second match for g to catch. The output is identical either way. It is written here because engineers write it by habit on path substitutions, and because the moment a line does contain the path twice, leaving it out gives you a half-finished line. A filelist line can name an include directory as well as a source file — +incdir+ is how a compile filelist marks a directory to search for included files — and such a line holds the path twice:

echo '+incdir+/old_lib/rtl /old_lib/rtl/top.sv' | sed 's|/old_lib/|/new_lib/|'

+incdir+/new_lib/rtl /old_lib/rtl/top.sv

The first path was fixed and the second was left behind — a line that now points at two different libraries, which is worse than not having run the command. Adding g is the difference:

echo '+incdir+/old_lib/rtl /old_lib/rtl/top.sv' | sed 's|/old_lib/|/new_lib/|g'

+incdir+/new_lib/rtl /new_lib/rtl/top.sv

Chapter 3.2 covers the occurrence flags properly. For now: g means every match on the line. Whether you want it depends on the line, not on the task — add it when a line can hold more than one match, which for path substitutions is often but not always.

And files.f itself is unchanged, so what you have is a preview of the corrected filelist.

7. What sed Is Good At

Pulling the chapter together, sed earns its place when the work is line-by-line text change on text you can describe:

• Simple replacement. One piece of text becomes another, across every line.
• Filtering lines. Printing only the lines that interest you, or dropping the ones that do not. Module 5 covers this.
• Tidying logs and reports. Removing boilerplate, collapsing spacing, or replacing run-specific details so two runs can be compared.
• Updating controlled path patterns. The filelist case from section 6, and the include paths, script options and constraint references of Module 9 onward.
• Processing generated text. Output from another program, corrected on its way past.
• Working inside shell automation. sed needs no setup, waits for nobody, and reads from a pipe, so it drops into a script that runs unattended.

The common thread is that the change is mechanical and the text can be described. When that holds, sed is often one short command. When it does not, something else is the right tool, and Chapter 1.3 draws that line between sed, grep, awk and Python.

8. What sed Cannot Safely Do

This section matters more than its length suggests.

sed is a text-processing tool. It is not a Verilog or SystemVerilog parser. It sees characters. It does not know what they mean.

Concretely, sed has no idea about:

• module hierarchy — which module instantiates which, or where a name came from;
• language semantics — what a declaration is, what an expression is, what is legal;
• scope — whether two identical names are the same thing or two unrelated things;
• UVM object relationships — what a component is, or how one connects to another;
• timing constraints as engineering models — a constraint file is a file of text to sed, not a description of clocks.

The practical consequence is this: a substitution matches wherever its pattern appears, including places you were not thinking about. The same run of characters can turn up inside a comment, inside a string, inside a longer identifier that merely starts the same way, or in a line that has nothing to do with the change you intended. sed will change all of them, report no error, and exit normally.

So when a change really is mechanical and describable — a path that moved, an option that was renamed — sed is the right tool and this curriculum will show you how to do it with the checks it deserves.

When a change depends on meaning — renaming a signal correctly everywhere it is genuinely that signal, and nowhere it is not — that is refactoring, and it needs a tool that has actually parsed the language, plus a compile and a review to confirm the result. Chapter 9.3 is devoted to where that line falls, because knowing which side of it you are on is the difference between a five-minute job and a bad afternoon.

Nothing in this curriculum will tell you to run a sweeping replacement across a design and hope.

9. Common Beginner Mistakes

Five things catch nearly everyone. None of them is a sign you are doing badly.

Expecting the file to have changed. You run a substitution, see the right output, open the file, and find it untouched. Nothing went wrong — section 4 is this, by design. The output was printed, not saved.

Leaving out the quotes. If the instruction contains a space and is not quoted, the shell splits it into pieces before sed sees it, and sed receives something incomplete. It reports an error about an unterminated command. The exact wording differs between sed versions, so do not pattern-match on the message — just put the instruction in single quotes and the problem does not arise.

Mixing up the instruction and the filename. The instruction comes first, the filename second: sed 's/a/b/' file.txt. Giving the filename where the instruction belongs makes sed try to read file.txt as a command, and it complains about an unknown command rather than about a missing file — which is confusing until you know why. Relatedly, if you give sed an instruction but no filename at all, it is not broken and has not frozen: with no file to read it waits for you to type input, because reading from the keyboard is a perfectly sensible thing for a stream editor to do. Press Ctrl-D to finish, or Ctrl-C to abandon it.

Assuming every match is replaced. By default, only the first match on each line changes. This one is easy to miss, because a file whose lines each contain a single match behaves identically with or without g — which is exactly the trap section 6 walked through. The exercise in section 10 is this mistake, so it is worth meeting it there rather than being told the answer here.

Not looking at the output. The most expensive mistake, and the easiest to avoid. A sed command that finishes without an error has only told you that your instruction was well formed — not that it matched what you intended, and not that it left everything else alone. Those are different claims, and only reading the output settles them. Every habit in Module 7 is built on this one.

10. Check Your Understanding

First, predict an output

Here is a line of text with the word hello on it twice:

hello hello

And here is a substitution that replaces hello with hi:

echo "hello hello" | sed 's/hello/hi/'

Before reading on, decide what you think this prints. Commit to an answer — there are only a few possibilities, and choosing one is what makes the next paragraph useful rather than obvious.

Most people predict hi hi. The actual output is:

hi hello

Only the first match on the line was replaced, because that is what s/// does on its own. The second hello was left exactly as it was. If you predicted hi hi, you have just met the most common sed surprise in the least expensive possible way — on one line of text, rather than on three hundred files.

Adding g gives the result you probably expected:

echo "hello hello" | sed 's/hello/hi/g'

hi hi

Now look back at section 6. The filelist there produced the same output with and without g, because each of its lines held one match — so that example could never have taught you this. One line of text just did.

Then answer these

They are about the ideas, not the syntax — none of them needs you to remember a flag.

1. What problem does sed solve?

Making the same mechanical change to text, in many places, without doing it by hand. The change has to be one you can describe — find this, put that in its place — so that no judgement is needed per line. If the work needs judgement, sed is not the tool.

2. Does a basic sed substitution change the original file?

No. It reads the input and prints the changed version. The file on disk is untouched. This is why experimenting is safe, and it is why writing results back is treated as a separate decision in Module 7.

3. What does the pipe symbol do in echo "hello world" | sed 's/world/VLSI/'?

It connects the two commands: whatever echo printed becomes sed's input. Without it, echo would print to your screen and sed would have nothing to read.

4. Why should an RTL engineer look carefully at a text replacement before trusting it?

Because sed matches characters, not meaning (section 8). A pattern can match inside a comment, inside a string, or inside a longer name that merely begins the same way, and sed changes all of them without any error. The command succeeding is not evidence that it was right.

One more, if you want to think rather than type. On the filelist in section 6, g changed nothing. Knowing what you now know from the exercise above, would you still write it there? Give a reason either way. There is a defensible answer in both directions, and noticing that there is a decision to make at all is the point.

11. Summary

• sed is a stream editor. By default it reads text one line at a time and can change each line on the way past. It opens nothing on screen and waits for nobody, which is why it works unattended and keeps its memory use modest on a large file.
• The model is input → command → output. The pattern is what sed looks for; the replacement is what it puts there instead.
• s/pattern/replacement/ is the substitution command. s means substitute, and the separator marks where the parts begin and end.
• The substitutions in this chapter print their result and leave the input file alone. sed does have a way to write changes back — Module 7 — so this is a property of these commands, not a guarantee about sed. Reading the result before going further is the habit the whole curriculum is built on.
• Lines that do not match are printed unchanged. sed passes everything through and alters only what matches.
• The separator does not have to be /. Choosing one your text does not contain — | for paths — keeps a command readable.
• g means every match on the line, not just the first. On a line with one match it makes no difference; on a line with two, leaving it out finishes half the job.
• Always put the instruction in single quotes, so the shell hands it to sed exactly as you wrote it.
• A command that runs without an error has not been proved correct. It only means the instruction was well formed. Whether it matched what you meant is a separate question, settled by reading the output.
• sed does not understand HDL. It matches characters, with no knowledge of hierarchy, scope or semantics. Mechanical, describable changes are its job; meaning-dependent refactoring is not.

You can now read a sed substitution, predict what it will print, run it on a line or a file, and say why the file did not change — which is already enough to be useful on a real filelist.

What that leaves open is where this gets used. A chip flow produces filelists, scripts, logs and reports at every stage, and they are not equally good candidates for a text tool. Chapter 1.2 — sed in Semiconductor Engineering maps the places sed actually earns its keep, from compile filelists and include paths through simulator run scripts, constraint references and log normalisation, so the rest of Part I has somewhere concrete to point.