sed Script Files and Reusable Transformations

Part II · Module 6 · Chapter 6.4 · sed Script Files and Reusable Transformations

1. When a One-Liner Has Outgrown Itself

Chapter 2.3 introduced -f as one of the three places a script can come from. Everything since has been typed on a command line, which is right for a substitution you will run once.

Four things say it is time to move to a file:

• It has a label. Chapter 6.3 measured why: on BSD sed a label swallows the rest of its line, so :a; s/…/; ta silently does nothing and exits 0. In a file, one command per line, the problem cannot occur.
• You will run it more than once. A normalisation step used by every regression is not a thing to retype from shell history.
• Somebody has to review it. A file appears in a diff. A pasted one-liner does not.
• It needs explaining. A script file can carry comments; a command line cannot.

The last one matters more than it sounds. A t loop and a hold-space pairing are both opaque at a glance — Chapter 6.1's reverse-a-file idiom is three commands that nobody reads correctly the first time. A comment costs one line.

2. The Rules

A script file is the same commands, one per line:

   # a comment — the whole line, starting with #
   :label                 on its own line
   s/old/new/             one command per line
   /addr/{                a block can span lines
       s/a/b/
   }

Three details:

• # starts a comment and runs to the end of the line. There is one exception, which is section 4.
• One command per line — this is the rule that makes branching work.
• No shell quoting. The file is read by sed directly, so a backslash is sed's backslash and nothing else gets a say. This removes an entire class of bug that Module 8 is otherwise about.

Run it with -f:

   sed -f scripts/normalize-timing.sed sta/timing.rpt

3. A Real One: Making Two Reports Diffable

Here is a script that does the job Module 6 has been building towards. It has two stages, a comment for each, and a named label:

cat -n scripts/normalize-timing.sed

     1	# normalize-timing.sed — make two timing reports diffable
     2	# 1. squeeze column padding to a single space
     3	:squeeze
     4	s/[[:space:]][[:space:]]/ /
     5	tsqueeze
     6	# 2. mask the numeric slack value so only MET/VIOLATED matters
     7	s/\(slack ([A-Z]*)\) .*/\1 <value>/

Run against the report:

cat -n sta/timing.rpt

     1	Startpoint: u_alu/rd_data_reg[3]
     2	Endpoint:   u_fifo/wr_ptr_reg[0]
     3	  data arrival time                        1.842
     4	  data required time                       2.100
     5	  slack (MET)                              0.258
     6	Startpoint: u_fifo/rd_ptr_reg[1]
     7	Endpoint:   u_alu/op_reg[2]
     8	  data arrival time                        2.410
     9	  slack (VIOLATED)                        -0.110

sed -f scripts/normalize-timing.sed sta/timing.rpt

Startpoint: u_alu/rd_data_reg[3]
Endpoint: u_fifo/wr_ptr_reg[0]
 data arrival time 1.842
 data required time 2.100
 slack (MET) <value>
Startpoint: u_fifo/rd_ptr_reg[1]
Endpoint: u_alu/op_reg[2]
 data arrival time 2.410
 slack (VIOLATED) <value>

Both stages fired. The column padding is gone, and both slack numbers are now <value> while MET and VIOLATED are preserved.

Two things in that script are worth pointing at:

The label is named squeeze, not a. Single letters are the convention in one-liners because they are shorter. In a file there is no reason not to say what the loop does — tsqueeze reads as a sentence and survives someone adding a second loop below it.

Line 7 uses unescaped parentheses as literal text. \(slack ([A-Z]*)\) captures the string slack (MET) — the escaped pair \( \) is the capture group, and the bare ( ) are the literal brackets that appear in the report. That only works in basic mode, which Chapter 4.1 is about: under -E those bare parentheses would become a second group and the pattern would mean something else entirely.

4. #n — the Comment That Is Not a Comment

If the very first line of a script file is exactly #n, sed behaves as though you had passed -n. Auto-print is off.

cat -n scripts/extract-errors.sed

     1	#n
     2	/UVM_ERROR/p

sed -f scripts/extract-errors.sed logs/sim.log

UVM_ERROR @ 980: data mismatch
UVM_ERROR @ 1510: timeout

Two lines from a seven-line log — a filter, with no -n on the command line. That is useful: a script whose whole purpose is to extract something carries its own output mode, so a caller cannot get it wrong by forgetting an option.

It only works as the first line. Move it down one and it is an ordinary comment:

cat -n scripts/notfirst.sed

     1	/UVM_ERROR/p
     2	#n

sed -f scripts/notfirst.sed logs/sim.log | wc -l

       9

Nine lines instead of two. Auto-print stayed on, so every one of the seven lines printed, and the two matching lines printed again from p — Chapter 5.1's duplication, arriving here as a consequence of a misplaced directive.

5. Why This Is the Point of the Whole Module

The practical value of section 3's script is not the squeezing. It is that two reports become comparable.

Run the same normalisation over yesterday's report and today's, then diff the two results. Without it, a diff of raw tool output is dominated by noise — column widths shift when a value gains a digit, timestamps differ, run directories differ — and the real change is buried. With it, the diff shows the values that actually moved.

That is a flow-engineering task rather than a text-editing one, and it has the properties flow code needs: it is the same transformation every time, it is reviewable, and when someone asks "why is the slack masked?" the answer is on line 6 of the file.

The same pattern covers the other Module 6 idioms:

• a hold-space pairing (Chapter 6.1) that turns a report into one line per path;
• a sliding window (Chapter 6.2) that joins continuation lines so each message is one grepable line;
• a t loop (Chapter 6.3) that collapses arbitrary padding.

Each is three to six lines in a file, each wants a comment, and none of them is something you want to retype correctly under pressure.

6. What This Chapter Deliberately Does Not Cover

A script file is where a transformation lives. Making it safe to apply is a different subject and the whole of Part III:

• how to prove the edit is right before it writes anything;
• what -i actually does, and the GNU/BSD split that makes it the least portable part of sed;
• how to choose the file list so a sweep touches what you intended;
• when the job has outgrown sed entirely.

Nothing in this chapter writes to disk. Module 7 — Safe File Editing and Debugging picks up from here.

7. Think It Through

1. Why is a script file the only reliable way to write branching?

Because a label on BSD sed extends to the end of its line, so a label and a command cannot share one. In a file, one command per line, that is automatic. On a command line it takes one -e per command, which works but is easy to collapse back into a single quoted string — and the failure is silent, with exit status 0, as Chapter 6.3 measured.

2. What does a # do in a sed script file, and when does it not mean what you expect?

It starts a comment to end of line. The exception is #n as the entire first line, which means "behave as if -n had been passed". Anywhere else, #n is just a comment — section 4 measured nine lines instead of two.

3. In s/\(slack ([A-Z]*)\) .*/\1 <value>/, how many capture groups are there?

One. \( \) is the group; the bare ( ) are literal brackets matching the ones in slack (MET). In basic mode unescaped parentheses are ordinary characters — Chapter 4.1's table. Under -E the same text would have two groups and \1 would refer to a different thing, which is a good reason to state the dialect in a comment if a script relies on it.

4. Your script file works and a colleague says it is unreadable. What are the three cheapest improvements?

Name the labels after what they do rather than a. Add a one-line comment above each stage saying what it is for, not what it does — "make two reports diffable" rather than "substitute spaces". And put the file next to the thing it transforms, in the repository, so the reader can see what it is applied to.

5. When should a transformation stop being a sed script?

When it needs arithmetic, more than one piece of carried state, or its own tests. Chapter 1.3 drew the line at the start of this track and Module 6 has not moved it: the hold space can carry one thing, and a t loop can repeat one substitution. A script that wants to total a column, correlate two files, or make a decision based on a count is an awk or Python job. Chapter 8.4 is the chapter that makes that call explicitly.

6. A normalisation script is applied to every regression log. What should happen when someone changes it?

The same thing that happens when anyone changes flow code: a diff, a review, and a check that previously-normalised output has not shifted. That is the real argument of this chapter — a pasted one-liner has no version, no author and no review, and a -f file has all three.

8. Summary

• Move a script to a -f file when it has a label, will be run more than once, needs reviewing, or needs explaining.
• For branching it is not optional. One command per line is what stops a label swallowing the rest of the script.
• A script file is the same commands with # comments, one per line, and no shell quoting — which removes an entire class of bug.
• Name your labels. tsqueeze reads; ta does not.
• #n as the exact first line implies -n. Two lines came out of a seven-line log.
• Anywhere else, #n is just a comment — the same script with it on line 2 produced nine lines, exit 0. If it feels too subtle, pass -n explicitly and say so in a comment.
• A basic-mode pattern can mix \( \) groups with literal ( ), as the slack mask does. State the dialect in a comment when a script depends on it.
• The point is reusability and review. Normalising two tool reports is what makes a diff show changed values instead of moved columns.
• Nothing here writes to disk. Safety is Module 7.

That closes Module 6 and Part II. You now have the whole of sed as a language: the cycle, the pattern dialects, addresses, the substitution and line commands, two buffers, a multi-line pattern space, control flow, and the file to keep it all in. Everything from here is about applying it without causing damage. Part III — Safe Engineering Automation opens with Module 7 — Safe File Editing and Debugging: what -i really does to your file, the GNU and BSD forms that are not interchangeable, how to prove an edit before it lands, and how to read the failure when a pattern matched more than you meant.