Input, Output, and Multiple Files

Part I · Module 2 · Chapter 2.4 · Input, Output, and Multiple Files

1. Three Ways In, One Way Out

By default sed writes to standard output and nowhere else. It never touches your input file unless you explicitly ask it to with -i, and that is the single most important safety property of the tool — every example in Part I has been non-destructive for that reason.

Input is where the choices are. There are three forms, and the first two are equivalent:

cat filelists/top.f | sed -n '$p'

/proj/rtl/pkg.sv

sed -n '$p' < filelists/top.f

/proj/rtl/pkg.sv

sed -n '$p' filelists/top.f

/proj/rtl/pkg.sv

Same answer three times — the last line of a six-line filelist. The difference is who opens the file: the shell in the first two cases, sed itself in the third. For one file it rarely matters. For several it matters a great deal, which is the rest of this chapter.

2. Several Files Are One Stream

Give sed more than one file operand and it does not run once per file. It reads them end to end as a single stream, exactly as if you had cat-ed them together first.

The two fixtures:

wc -l filelists/top.f filelists/tb.f

       6 filelists/top.f
       2 filelists/tb.f
       8 total

Line numbering first. Line 7 of the stream is line 1 of the second file:

sed -n '7p' filelists/top.f filelists/tb.f

/proj/tb/env.sv

env.sv is the first line of tb.f, and sed called it line 7. There is no way to ask for "line 1 of each file" with a line-number address, because as far as the cycle is concerned there is only one line 1:

sed -n '1p' filelists/top.f filelists/tb.f

/proj/rtl/alu.sv

3. What $ Means With Several Files

$ is the address that matches the last line of input. With one file that is unambiguous:

sed -n '$p' filelists/top.f

/proj/rtl/pkg.sv

sed -n '$p' filelists/tb.f

/proj/tb/test.sv

Run together, only one of those two lines is still "last":

sed -n '$p' filelists/top.f filelists/tb.f

/proj/tb/test.sv

pkg.sv was the last line of top.f, but top.f no longer has an end — it has a join. The only end-of-input in the stream is the end of tb.f.

The = command makes the arithmetic explicit. It prints the current line number rather than the line, so $= prints the number of the last line, which is the line count:

sed -n '$=' filelists/top.f

6

sed -n '$=' filelists/top.f filelists/tb.f

8

Six for the first file alone; eight for both together — 6 + 2, numbering straight through the join. Compare that with wc -l in section 2 and the model is confirmed from two directions.

Why this bites in practice. A one-liner that extracts the last line of a report is correct on one file and wrong on a glob, with no error to tell you. sed -n '$p' sta/*.rpt gives you the last line of the alphabetically final report, not the last line of each — and if you wrote it while testing against a single file, it will keep looking right until the day there are two.

4. Asking For Each File Separately

GNU sed has two features for this. Neither exists on the BSD sed that macOS ships, and both fail in a way worth seeing.

-s tells GNU sed to treat each file as a separate stream, so $ matches the last line of every file:

sed -s -n '$p' filelists/top.f filelists/tb.f

sed: illegal option -- s
usage: sed script [-EHalnru] [-i extension] [file ...]
	sed [-EHalnu] [-i extension] [-e script] ... [-f script_file] ... [file ...]

Exit status 1. The option does not exist here at all, and the usage block confirms it — -s is not among the letters listed. This failure is at least loud.

GNU's F command, which prints the name of the current input file, is equally absent:

sed -n '${F;p;}' filelists/top.f filelists/tb.f

sed: 1: "${F;p;}
": invalid command code F

Exit status 1, and the same invalid command code message Chapter 2.3 met from a different direction — sed has no F, so it is reported as an unknown command.

The portable answer is to stop asking sed to iterate and let the shell do it:

for f in filelists/top.f filelists/tb.f; do sed -n '$p' "$f"; done

/proj/rtl/pkg.sv
/proj/tb/test.sv

Both last lines, because each sed invocation gets exactly one file and therefore one end-of-input. This form works on every sed, and it has a second advantage — the loop variable is available, so you can label the output, which $p alone cannot do:

for f in filelists/top.f filelists/tb.f; do printf '%s: ' "$f"; sed -n '$p' "$f"; done

filelists/top.f: /proj/rtl/pkg.sv
filelists/tb.f: /proj/tb/test.sv

5. When One of the Operands Is Not There

Hand sed a list of files where one is missing, and it does not stop:

sed -n '$=' filelists/top.f filelists/nope.f filelists/tb.f

sed: filelists/nope.f: No such file or directory
8

Exit status 1. Look at what that output is: 8 is a perfectly true line count — of the two files sed could read. It is not the answer to the question asked, which involved three files. The diagnostic went to stderr; the wrong-but-plausible number went to stdout.

The same shape appears when you try to use - to mean standard input, which GNU sed supports and BSD sed does not:

cat filelists/top.f | sed -n '$p' -

sed: -: No such file or directory

Exit 1, and here nothing was printed. But mix - with a real file and the partial behaviour returns:

sed -n '$p' filelists/top.f - < filelists/tb.f

sed: -: No such file or directory
/proj/rtl/pkg.sv

Exit 1, one plausible line on stdout, the - operand rejected. On BSD sed, the way to feed standard input is to give no file operand at all, as section 1 showed.

6. The Redirection That Destroys the File

Everything so far has been about reading. Here is the output-side mistake, and it is the one to remember out of all of Part I.

You have a filelist to rewrite, and the obvious thing is to send sed's output back to the file:

cat list.f; wc -c list.f

/proj/old/alu.sv
/proj/old/fifo.sv
      35 list.f

Thirty-five bytes, two lines. Now the edit:

sed 's|/proj/old/|/proj/new/|' list.f > list.f

No output, no diagnostic, exit status 0. And the file:

cat list.f; wc -c list.f

       0 list.f

Zero bytes. The content is gone — not corrupted, not half-edited, gone — and sed reported success.

Why. The redirection is the shell's work, and the shell does it first. Before sed is even started, > list.f truncates list.f to zero length. sed then opens it, finds an empty file, runs its cycle zero times, writes nothing, and exits 0 having done its job correctly on the input it was given. Nothing in the chain malfunctioned. The ordering was simply not what the command looked like it meant.

7. A Pattern Worth Naming

Three commands in this module produced the same unhelpful shape — believable output on stdout, a diagnostic on stderr, and a non-zero exit status:

And one that is worse, because it has no diagnostic at all:

The practical conclusions are small and dull and worth doing anyway. Check the exit status of any sed whose result you are going to rely on — set -e in a shell script, or an explicit $? test. Do not discard stderr on a command you have not verified. And never trust that plausible-looking output means the command did what you asked, because in three of the four rows above it did not.

8. Think It Through

1. sed -n '$p' sta/*.rpt is supposed to print the slack line from each timing report. The glob matches eleven files. How many lines does it print?

One — the last line of the eleventh report, in whatever order the glob expanded. The eleven files are concatenated into one stream, so there is a single end-of-input. for f in sta/*.rpt; do sed -n '$p' "$f"; done is the portable fix, and labelling each line with $f is worth the extra printf when the answer is a list of numbers.

2. Why does sed -n '$=' a.f b.f print one number rather than two?

Because $= means "at the last line of input, print the current line number", and there is only one last line. The number is the total line count of both files, since numbering runs through the join. Two numbers requires two invocations.

3. A colleague reports that sed -i 's/old/new/' *.f worked on their Linux box and "did nothing" on their Mac. What are the two separate problems?

First, BSD -i requires an argument, so s/old/new/ was consumed as the backup extension and the first filename became the script — Chapter 2.3's exercises traced that failure. Second, even with the portable -i '' form, the two seds differ in extension handling, so a script that hard-codes one form is not portable. Chapter 7.1 covers both; the general lesson is that -i is the least portable part of sed and deserves an explicit per-platform branch rather than hope.

4. You want the first line of every .f file in a directory, each prefixed with its filename. Why can sed not do this alone on macOS?

Because the address 1 refers to the first line of the concatenated stream, not of each file, and the two GNU features that would fix it — -s for separate streams and F for the current filename — are both absent on BSD sed, as section 4 showed. The shell loop supplies both the per-file iteration and the filename. This is a good illustration of a Chapter 1.3 point: the boundary of sed is reached quickly, and the answer is usually to let another tool own the part sed is bad at rather than to find a cleverer sed.

9. Summary

• sed writes to standard output and never to the input file unless -i is used. That is why everything in Part I has been safe to run.
• Three input forms — a pipe, a shell redirect, or a file operand — are equivalent for a single file. With no operand, sed reads standard input.
• Several file operands are one stream, not several runs: the files are joined in the order written.
• Line numbers do not reset at the join. Line 1 of the second file was line 7 of the stream.
• $ is the last line of the last file. sed -n '$=' top.f tb.f printed 8, not 6 and 2.
• GNU's -s and F do not exist on BSD sed — both fail with exit 1. A shell for loop is the portable way to get per-file behaviour, and it gives you the filename as well.
• A missing operand does not stop the run. sed reports it on stderr, processes the rest, and prints a number that is true of fewer files than you asked about.
• Never redirect output onto the input. sed … list.f > list.f left the file at zero bytes with exit status 0, because the shell truncates the target before sed starts. Write elsewhere and mv, diff first, or use -i with Chapter 7.1's caveats.
• Check the exit status. Three commands in this module produced believable output while failing.

That completes Module 2. You now have the model the rest of the curriculum assumes: a stream processed one line at a time, a buffer that commands edit in order, a positional command line, and a clear picture of where the bytes come from and go. Part II opens with Module 3 — Substitution and Replacement, which takes the s command apart properly: how many occurrences it replaces and why /g exists, which delimiter to choose, what & and \1 mean on the replacement side, and which characters have to be escaped. Section 5 of Chapter 2.2 showed what happens without that knowledge; Module 3 is where it is supplied.