How Stream Editing Works

Part I · Module 2 · Chapter 2.1 · How Stream Editing Works

1. Two Ways to Change a File

Open a report in a text editor and the editor reads the whole file into memory, shows you a window onto it, and waits. That model is built around a person: you need to scroll, to search backwards, to change your mind. Holding the document is what makes those things possible.

sed is built around a different assumption — that nobody is watching, the change is already decided, and the file may be larger than you want to think about. So it does not hold the document at all. It reads one line, applies your script to that line, writes the result out, and moves on with no memory of what it just did.

The name says it: a stream editor. Text flows through, and the edit happens in transit.

2. Output Before the Input Ends

The clearest proof that sed is not holding your file is that it answers before the file is finished.

seq 1 1000000 | sed 's/^/line /' | head -3

line 1
line 2
line 3

seq is generating a million lines. If sed had to read all of them before it could produce anything, this command would pause while that happened. It does not — the first three lines come straight through, and head stops the pipeline once it has what it asked for.

Push it further. This stream never ends at all:

yes "UVM_INFO ok" | sed 's/ok/seen/' | head -1

UVM_INFO seen

yes prints its argument forever. A tool that waited for end-of-input would hang here permanently. sed transformed the first line and handed it on, which is only possible because "read a line, write a line" is the whole of its loop.

Engineering implication. This is why sed composes. A command that must see all of its input before emitting any is a barrier in a pipeline — everything downstream waits. sed is not a barrier, so grep … | sed … | awk … begins producing results immediately and uses a bounded amount of memory no matter how long the run is. That matters when the input is a regression log being written while you watch it.

3. Nothing Is Remembered Between Lines

The other half of the model is what sed forgets. Each line arrives, the script runs, the result is written, and the line is gone.

printf 'a\nb\n' | sed 's/a/A/'

A
b

The second line was not compared with the first, counted, or accumulated. Lines that do not match are written out unchanged — not skipped, not dropped — which is why the output of a substitution is a whole file rather than just the parts that changed.

This is also the limit of what Module 1 taught you to do. A question like how many lines matched or what was the total of column three needs something carried from one line to the next, and Chapter 1.3 is where that pointed at awk.

4. What It Actually Costs

The folklore version of this property is "sed uses constant memory". That is close enough to be useful and wrong enough to be worth correcting, so here it is measured.

Two inputs. One is a five-line timing report; the other is 18 MB of short lines.

wc -c timing.rpt big.txt

     156 timing.rpt
18777780 big.txt

Running the same substitution over each and recording peak resident memory gives, on the machine these examples were verified on:

timing.rpt   156 bytes      peak RSS  1,359,872 bytes
big.txt      18,777,780 B   peak RSS  1,474,560 bytes

The input grew by a factor of about 120,000 and the memory grew by about 8%. That is the property worth having: sed's footprint is set by its own machinery, not by how much text you point it at. A 50 MB STA report costs essentially what a five-line one costs.

Now the correction. Make a file that is 20 MB of text in a single line and measure again:

big.txt      18,777,780 B, 2M short lines   peak RSS   1,474,560 bytes
oneline.txt  20,000,001 B, ONE long line    peak RSS  63,602,688 bytes

Forty-three times the memory, for a file of almost the same size. The accurate statement is therefore:

sed's memory tracks the longest line in the input, not the size of the input.

Because the unit it holds is one line, a file of a billion short lines is free and a file containing one enormous line is not.

5. Where "One Line at a Time" Needs Qualifying

Everything above describes sed's default behaviour, and it is the behaviour that applies to every command in Module 1 and to most real sed in a flow. It is not an absolute limit on the tool.

sed has a second buffer, the hold space, which survives from one line to the next, and commands that deliberately pull an extra line into the one being worked on. With those, sed can compare neighbouring lines, reverse a file, or match a pattern that spans a line break — all of which contradict a strict reading of "nothing is remembered".

So the correct framing is a default rather than a rule: sed processes one line at a time unless you explicitly ask it not to, and the memory property above holds exactly as long as you do not. Module 6 covers those commands, and Chapter 1.3's point stands either way — when a job genuinely needs state, reaching for awk is usually the better engineering answer than making sed remember things.

6. Think It Through

1. This command appears to hang. What is actually happening?

sed 's/foo/bar/' | head -1

Nothing is feeding it. There is no filename, and nothing is piped in, so sed is reading standard input — your keyboard. It is not stuck; it is waiting for a line, exactly as section 2's yes example was fed by a producer. Type a line and press Enter and you will see it transformed. Ctrl-D ends the input, Ctrl-C abandons it. Chapter 1.4 met the same behaviour from the other direction.

2. A colleague says sed will not work on a 40 GB waveform dump because it will not fit in memory. Are they right?

Not for the reason given. Size alone is not the problem: sed holds one line, so a 40 GB file of ordinary lines streams through in a bounded footprint. The question to ask is how long the longest line is. If the dump is one enormous record with no line breaks, section 4 says sed will try to hold all of it and your colleague's concern becomes correct — not because the file is large but because the line is.

3. Why does putting sed in the middle of a long pipeline not delay the first result?

Because it never waits for end-of-input. Each line is transformed and written as it arrives, so downstream stages begin working immediately. A stage that must see all of its input before emitting anything — a sort, for example — does delay everything behind it. Knowing which of your stages are barriers is most of what makes a long pipeline feel fast or slow.

7. Summary

• sed does not open your file the way an editor does. It reads a line, applies the script, writes the result, and forgets it.
• It can produce output before the input ends, which is why it works in a pipeline and why it survives an unbounded stream.
• Non-matching lines pass through unchanged, so output is a whole file rather than a changed fragment.
• Nothing is carried between lines by default — counting and totalling are not line-local questions, which is why Chapter 1.3 sent them to awk.
• Memory tracks the longest line, not the file. Measured here: 18 MB of short lines cost 1.47 MB; 20 MB on a single line cost 63.6 MB.
• "One line at a time" is a default, not a limit. Hold space and multiline commands exist, and Module 6 covers them.

What this chapter has not said is what happens inside one of those steps. "Apply the script to the line" is doing a lot of work: where the line is held, what order several commands run in, when the result is printed, and what happens if one command deletes the line before another sees it. Chapter 2.2 — Pattern Space and the Execution Cycle opens that step up, and it is the chapter the rest of the curriculum leans on most.