Deleting Lines

Part II · Module 5 · Chapter 5.2 · Deleting Lines

1. The Command

d deletes the pattern space. With an address, it deletes the lines that address selects:

   sed '/^#/d'       delete lines starting with #
   sed '1,10d'       delete the first ten lines
   sed '$d'          delete the last line
   sed '/^#/!d'      delete lines NOT starting with #  (keep only comments)
   sed 'd'           delete every line — output nothing at all

There is no "undelete" and no output from a deleted line. Since sed writes to standard output and never to your file (Chapter 3.1), deleting is entirely safe to experiment with.

2. Deleting What You Do Not Want

A filelist with the usual clutter:

cat -n filelists/top.f

     1	+define+DEBUG
     2	  # indented comment
     3	+incdir+/proj/old/rtl
     4	
     5	./rtl/alu.sv
     6	# a commented-out file
     7	   
     8	/proj/old/tb/env.sv
     9	./rtl/top.sv

Nine lines. Two are comments — one of them indented. Two are blank — one of them empty, one containing three spaces. Both of those details matter in a moment.

Strip the comments:

sed '/^[[:space:]]*#/d' filelists/top.f

+define+DEBUG
+incdir+/proj/old/rtl

./rtl/alu.sv
   
/proj/old/tb/env.sv
./rtl/top.sv

Seven lines — both comments gone, including the indented one, because ^[[:space:]]*# allows any amount of leading whitespace before the hash.

3. The Strip That Works, and the One That Looks Like It Does

The registered example for this chapter does comments and blank lines in one pass:

sed '/^[[:space:]]*#/d; /^[[:space:]]*$/d' filelists/top.f

+define+DEBUG
+incdir+/proj/old/rtl
./rtl/alu.sv
/proj/old/tb/env.sv
./rtl/top.sv

Five lines: the five real entries. Nine in, five out, four removed.

Now the version most people write first:

sed '/^#/d; /^$/d' filelists/top.f

+define+DEBUG
  # indented comment
+incdir+/proj/old/rtl
./rtl/alu.sv
   
/proj/old/tb/env.sv
./rtl/top.sv

Seven lines. Two of the four targets survived — the indented comment on line 2, because ^# requires the hash at column one, and the whitespace-only line, because ^$ requires the line to be strictly empty.

Two spellings worth having, both portable:

   /^[[:space:]]*#/d        a comment, indented or not
   /^[[:space:]]*$/d        a blank line, empty or whitespace-only

For a Tcl or shell script, where # must be the first non-blank character to be a comment, the first is exactly right. For a filelist, where a # could in principle appear inside a path, you may want to be stricter — which is a judgement about your data, not about sed.

4. d Ends the Cycle

This is the behaviour that makes d different from p and s. When d runs, sed does not merely blank the pattern space and carry on — it stops processing that line entirely and goes to the next one. Commands written after d never run for that line.

The cleanest demonstration is two commands that contradict each other:

printf 'keep\ndrop\n' | sed -e '/drop/d' -e 's/drop/CHANGED/'

keep

The substitution never happened. drop matched the delete, the cycle ended there, and s/drop/CHANGED/ was never reached for that line.

Reverse them and both commands get a turn:

printf 'keep\ndrop\n' | sed -e 's/drop/CHANGED/' -e '/drop/d'

keep
CHANGED

Now the substitution runs first, so by the time /drop/d is tested the pattern space says CHANGED — which does not contain drop, so the delete does not fire and the line survives, renamed. Chapter 2.2 measured this from the pattern-space side; here it is the practical consequence.

Two orderings, two completely different results, no error either way. When a script mixes d with anything else, the order is part of the logic, not a formatting preference.

5. Deleting a Region

An address range deletes everything between its endpoints. The classic use is stripping a tool's banner — the block of version and copyright lines before the real output starts:

cat -n synth/dc.log

     1	Design Compiler Graphical
     2	Version R-2020.09
     3	Copyright (c) 1988-2020 Synopsys, Inc.
     4	==========================================
     5	Information: Building the design 'alu'.
     6	Warning: Design 'alu' has unconnected port 'test_en'.
     7	Information: Timing loop detected.

The banner ends at the rule of equals signs on line 4. Delete from line 1 up to and including it:

sed '1,/^=\{10,\}$/d' synth/dc.log

Information: Building the design 'alu'.
Warning: Design 'alu' has unconnected port 'test_en'.
Information: Timing loop detected.

Three lines — the actual messages. Reading the address: 1 starts at the first line, /^=\{10,\}$/ is "a line of ten or more equals signs and nothing else", and the range covers both endpoints. The \{10,\} is the basic-mode interval from Chapter 4.2, and the ^ and $ make sure a line that merely contains equals signs does not end the range early.

6. Delete or Print?

Filtering can be written either way, and they can be exactly equivalent:

sed -n '/UVM_ERROR/p' logs/sim.log

UVM_ERROR @ 980: data mismatch exp=8'hA5 act=8'h5A
UVM_ERROR @ 1510: timeout waiting for ready

sed '/UVM_ERROR/!d' logs/sim.log

UVM_ERROR @ 980: data mismatch exp=8'hA5 act=8'h5A
UVM_ERROR @ 1510: timeout waiting for ready

Identical. The first says "print what I want"; the second says "delete everything I do not want", using the negated address from Chapter 4.4.

Choose by which description is simpler to state correctly:

• Print when you can describe the thing you want. One pattern, one p, -n on the command line.
• Delete when the thing you want is "everything except a few known kinds of junk". Three deletes in a row reads better than one enormous pattern trying to describe every survivor.

There is one real difference. d genuinely removes the line from the rest of the script, so later commands do not see it; -n with p leaves every line in play and merely declines to print most of them. If your script does something else after the filter — a substitution on the survivors — deleting the noise first is both clearer and the only one of the two that actually protects the later commands.

7. Two Relatives, Named and Deferred

Two commands sit next to d and belong to Module 6:

• D deletes up to the first newline inside the pattern space. With a one-line pattern space it does the same thing as d, so there is nothing here it could usefully demonstrate. It matters once N has pulled a second line in.
• q quits sed entirely — it stops reading input, where d only stops processing one line. sed '5q' prints five lines and exits without reading the rest of the file, which on a large log is the difference between instant and not. Chapter 5.1 section 7 touched on this.

Both are named here so that neither is a surprise in somebody else's script.

8. Think It Through

1. sed '/^#/d' script.tcl leaves comments in the file. Why, and what is the fix?

The comments are indented. ^# requires the hash at column one, and a comment inside a proc body is indented. Use /^[[:space:]]*#/d, which allows any leading whitespace. The giveaway is that the output still looks like a clean script — count lines before and after rather than skimming.

2. Predict the output of printf 'a\nb\n' | sed -e '/a/d' -e 's/a/Z/' and of the same two commands reversed.

First: b. The a line is deleted and the cycle ends, so the substitution never runs on it. Second (s first): Z then b — the substitution changes the line to Z, after which /a/ no longer matches, so the delete does not fire. Both exit 0.

3. Why is 1,/^=\{10,\}$/d safe on a log containing two banner rules, while /^Design/,/^=\{10,\}$/d would not be?

Because a line-number start matches once. The second form starts on a pattern, so it reopens every time a line begins with Design — a second banner later in the log would start a second deletion. Chapter 4.4 measured the reopening; the fix is to anchor the start to a number whenever you mean "once, from the top".

4. You need to drop blank lines from a file that uses tabs for indentation. Is /^ *$/d enough?

No. ^ *$ allows only spaces, so a line containing a single tab survives — and looks identical to the ones you removed. /^[[:space:]]*$/d covers spaces, tabs and anything else the class includes. This is Chapter 4.2's argument for named classes, in the one place where getting it wrong is invisible.

5. You have a script that deletes comments, then substitutes a path. A colleague moves the substitution above the delete. Does anything change?

Functionally, usually not — the substitution would now also run on comment lines, which are then deleted anyway, so the output is the same. What changes is cost and blast radius: you are rewriting lines you are about to throw away, and if the substitution has a side effect you did not intend on a comment line, you have widened the edit for nothing. Deletes first is the better habit.

6. How do you check that a delete removed what you meant, rather than something adjacent?

Count, and look. wc -l before and after gives you the number removed; sed -n '/^[[:space:]]*#/p' on the original shows you exactly which lines the address selects, before you delete them. Chapter 4.4 makes the related point that after a delete, output line numbers no longer match input line numbers — so inspect the original, not the result.

9. Summary

• d deletes the pattern space, and with an address deletes exactly the lines that address selects.
• d also ends the cycle. Commands written after it never run for that line — -e '/drop/d' -e 's/drop/CHANGED/' printed only keep.
• Order is logic, not formatting. The same two commands reversed produced keep and CHANGED.
• /^[[:space:]]*#/d; /^[[:space:]]*$/d is the strip that works. Nine lines in, five out.
• /^#/d; /^$/d left two of four targets behind — an indented comment and a whitespace-only line — and the output still looked like a clean filelist. Check the count.
• A range deletes a region. 1,/^=\{10,\}$/d strips a tool banner, and cannot reopen because its start is a line number rather than a pattern.
• Deleting and printing are interchangeable for pure filtering — sed -n '/E/p' and sed '/E/!d' gave identical output — but only d removes the line from the rest of the script.
• D and q are relatives for Module 6: D matters only with a multi-line pattern space, and q stops reading the input entirely.

You can now keep lines and remove them. What you cannot yet do is produce a line that was never in the input — a generated-file header, an extra flag after a matched option, a whole line replaced with a different one. Chapter 5.3 — Inserting, Appending, and Changing Text adds i, a and c, and it is the chapter where GNU and BSD sed differ the most sharply of anywhere in this track.