Line Numbers, Pattern Addresses, and Ranges

Part II · Module 4 · Chapter 4.4 · Line Numbers, Pattern Addresses, and Ranges

1. Address, Then Command

Every sed command has the same shape:

   [address]  command
   └───┬───┘  └──┬──┘
   which lines   what to do

   /^#/d        on lines matching /^#/, delete
   1,20p        on lines 1 to 20, print
   $=           on the last line, print the line number
   d            on every line, delete        ← no address means ALL lines

Leave the address off and the command runs on every line. That is why sed 's/a/b/' edits the whole file — not because s is special, but because it had no address.

You have been writing addresses since Chapter 2.2 without the name. sed -n '/UVM_ERROR/p' is the address /UVM_ERROR/ plus the command p. This chapter is the address half, properly.

There are five kinds, and the fifth is where the chapter spends its time:

2. Line Numbers and $

The simplest address is a number. Here is a timing report, 29 lines long:

sed -n '1,20p' sta/timing.rpt | wc -l

      20

Twenty lines, as asked. A single number takes a single line:

sed -n '6p' sta/timing.rpt

Startpoint: u_alu/rd_data_reg[3]

And $ is the last line, whatever its number:

sed -n '$p' sta/timing.rpt

  slack (VIOLATED)                        -0.110

Asking for a line that does not exist is not an error:

sed -n '999p' sta/timing.rpt

No output, exit status 0 — the same quiet "nothing to do" that Chapter 3.2 found with an out-of-range occurrence number.

3. Pattern Addresses

A pattern between slashes selects every line it matches. The = command prints line numbers, which makes it the best tool for seeing what an address selects:

sed -n '/^Startpoint/=' sta/timing.rpt

6
26

Two lines. The report contains two timing paths. And the slack lines:

sed -n '/slack/=' sta/timing.rpt

24
29

Keep those four numbers in mind — 6 and 26 start the blocks, 24 and 29 end them. Section 5 depends on them.

4. Negation: Every Line Except

A ! after the address inverts the selection. The registered example for this chapter keeps only the comment lines, by deleting everything that is not a comment:

cat -n filelists/top.f

     1	+define+DEBUG
     2	+incdir+/proj/old/rtl
     3	
     4	./rtl/alu.sv
     5	./rtl/fifo.sv   
     6	# a commented-out file
     7	/proj/old/tb/env.sv
     8	
     9	#another comment
    10	./rtl/top.sv

sed '/^#/!d' filelists/top.f

# a commented-out file
#another comment

Read it as "on lines that do not start with #, delete". What survives is the comments. Drop the ! and you get the opposite — the far more common operation of stripping comments out:

sed '/^#/d' filelists/top.f

+define+DEBUG
+incdir+/proj/old/rtl

./rtl/alu.sv
./rtl/fifo.sv   
/proj/old/tb/env.sv

./rtl/top.sv

Both comment lines are gone — line 6 and line 9 — and the file went from ten lines to eight:

sed '/^#/d' filelists/top.f | wc -l

       8

That count is worth taking seriously as a habit. After a delete, the output no longer lines up with the input's line numbers: the output's line 7 is the input's line 8. Reading a shifted listing is how people convince themselves a delete did not happen, or deleted the wrong thing. If you need to reason about positions, get the numbers from the original first with sed -n '/^#/=', or compare counts as above.

Negation works on any address, including a number:

sed -n '1!p' filelists/blanks.f

   
fifo.sv

Everything except line 1 — which here means the empty line, the whitespace-only line, and fifo.sv.

5. The Range That Reopens

Now the main event. A two-address range addr1,addr2 starts at the first line matching addr1 and ends at the next line matching addr2.

The registered example pulls a timing path out of a report:

sed -n '/^Startpoint/,/slack/p' sta/timing.rpt

Startpoint: u_alu/rd_data_reg[3]
            (rising edge-triggered flip-flop clocked by clk)
Endpoint:   u_fifo/wr_ptr_reg[0]
            (rising edge-triggered flip-flop clocked by clk)
Path Group: clk
Path Type : max

  Point                          Incr      Path
  ---------------------------------------------
  clock clk (rise edge)          0.000     0.000
  u_alu/rd_data_reg[3]/CK        0.000     0.000
  u_alu/rd_data_reg[3]/Q         0.182     0.182
  u_fifo/wr_ptr_reg[0]/D         1.660     1.842
  data arrival time                        1.842

  data required time                       2.100
  data arrival time                       -1.842
  ---------------------------------------------
  slack (MET)                              0.258
Startpoint: u_fifo/rd_ptr_reg[1]
Endpoint:   u_alu/op_reg[2]
  data arrival time                        2.410
  slack (VIOLATED)                        -0.110

Look at the end of that output. There are two Startpoint lines in it. Counted:

sed -n '/^Startpoint/,/slack/p' sta/timing.rpt | wc -l

      23

sed -n '/^Startpoint/,/slack/p' sta/timing.rpt | grep -c '^Startpoint'

2

Twenty-three lines, two blocks. The arithmetic confirms it against the line numbers from section 3: the first range ran 6 to 24, which is 19 lines, and the second ran 26 to 29, which is 4. Nineteen plus four is 23.

A pattern range does not fire once. When it closes, sed goes back to looking for the start address, and opens again the next time it matches. So /^Startpoint/,/slack/ does not mean "the first timing path" — it means "every timing path".

6. The End Is Never Tested on the Start Line

The second surprise, and the one that explains a shape that otherwise looks broken.

When a range opens on a line, sed does not check the end address against that same line. It starts looking from the next line. So a pattern range always spans at least two lines.

A four-line fixture makes it unambiguous:

cat -n fixtures/marks.f

     1	X
     2	a
     3	X
     4	b

sed -n '/X/,/X/p' fixtures/marks.f

X
a
X

If the end address were tested on the start line, line 1 would match /X/ as both start and end and the output would be the single line X. Instead the range opened at line 1, looked from line 2 onward, and closed at line 3 — three lines.

The practical trap is a range whose start and end patterns are the same and which occurs only once. It looks like "select just that line" and it is not:

sed -n '/^Path Group/,/^Path Group/p' sta/timing.rpt | wc -l

      19

Nineteen lines — from Path Group: clk at line 10 all the way to the end of the file, because ^Path Group never matches again so the range never closes. Section 7 is that behaviour in general.

A line-number range of one line does work, because numbers are compared rather than searched:

sed -n '2,2p' fixtures/marks.f

a

And for a pattern, the one-line form is the +0 count from section 8.

7. An Unmatched End Runs to the End of the File

The third surprise. If the end address never matches, the range does not error and does not cancel — it stays open to the last line.

sed -n '/^Startpoint/,/NOSUCHTHING/p' sta/timing.rpt | wc -l

      24

Twenty-four lines: from line 6 to line 29, the whole rest of the file. Exit status 0.

This is a fail-open design, and it is worth naming because the failure is maximally unhelpful. A typo in the end pattern does not give you nothing, which you would notice; it gives you everything from the start pattern onward, which looks like a plausible extraction. Combined with section 5's reopening, a range with a mistyped end can select almost the entire file while looking like it selected a block.

The defence is to count. wc -l on the output, compared against what the block should be, catches both this and the reopening in one check.

8. Count-Based and GNU-Only Forms

A few more address forms, with their portability measured rather than assumed.

addr,+N — N lines after the start. This is usually described as a GNU extension, and the BSD sed used here implements it too:

cat -n fixtures/steps.f

     1	L1
     2	L2 START
     3	L3
     4	L4
     5	L5
     6	L6 START
     7	L7
     8	L8

sed -n '/START/,+2p' fixtures/steps.f

L2 START
L3
L4
L6 START
L7
L8

Each START plus the two lines after it — and note it reopened, exactly as section 5 described. +0 gives the start line alone, which is the one-line pattern range section 6 said was missing:

sed -n '/START/,+0p' fixtures/steps.f

L2 START
L6 START

One more property worth knowing: a range already open does not restart. With +5, the first range covers lines 2 to 7, which contains the second START at line 6 — and it is ignored:

sed -n '/START/,+5p' fixtures/steps.f

L2 START
L3
L4
L5
L6 START
L7

Six lines, one range, ending where the count said. addr,+N is not POSIX, so a script that must run on every sed should avoid it — but it works on both seds you are likely to meet, which is more than can be said for the next two.

0,/re/ — GNU only, and silent here. GNU sed allows a start address of 0 so that the end pattern can match on line 1, which is the idiomatic GNU way to say "up to the first match". On this sed:

sed -n '0,/^Endpoint/p' sta/timing.rpt

No output, exit status 0. A silent failure of the worst kind — a script relying on it returns nothing and reports success. The POSIX form starts at 1 and does work:

sed -n '1,/^Endpoint/p' sta/timing.rpt | wc -l

       8

Eight lines, up to and including the first Endpoint. The difference from GNU's 0,/re/ only shows when the pattern matches on line 1 itself.

first~step — GNU only, and loud. GNU sed's "every Nth line" form fails cleanly:

sed -n '1~5p' sta/timing.rpt

sed: 1: "1~5p
": invalid command code ~

Exit 1. So does addr,~N:

sed -n '/^Startpoint/,~5p' sta/timing.rpt

sed: 1: "/^Startpoint/,~5p
": expected context address

Exit 1. Both refuse to run, which is the better kind of portability failure — unlike 0,/re/, you find out immediately.

Scoping several commands. An address can govern a block of commands in braces — sed '/^Startpoint/,/slack/{ … }' — which is how you apply three edits to one region. That is Chapter 5.4's subject, once Module 5 has given you more commands than p and d to put in it.

9. Think It Through

1. sed -n '/^Startpoint/,/slack/p' report.rpt is supposed to extract one timing path and gives you 23 lines from a 29-line report. Why?

The range reopened. There are two paths in the file, so the range fired twice — 6 to 24 and 26 to 29. A pattern range means "every block", not "the first block". To get only the first, end the range on a line number, use ,+N, or find the boundaries with = first and then address by number.

2. Why does /X/,/X/ select three lines on a file whose lines are X a X b, rather than one?

Because the end address is not tested on the start line. The range opens at line 1 and sed begins searching for the end from line 2, finding it at line 3. A pattern range therefore spans two lines minimum. If you want exactly the matching line, use /X/,+0 or just /X/ with no range at all.

3. You mistype the end pattern of a range. What does the output look like?

Everything from the start pattern to the end of the file, with exit status 0. The range fails open. This is why a range extraction deserves a wc -l sanity check — "did I get about the number of lines a block should be" catches a typo that no error message will.

4. A colleague's script uses 0,/^read_sdc/ and returns nothing on your Mac. What is going on and what do you change it to?

0 as a start address is a GNU extension; BSD sed matches nothing and exits 0. Change it to 1,/^read_sdc/, which is POSIX. The two differ only when the end pattern can match on line 1 — GNU's 0,/re/ would stop there, while 1,/re/ has already consumed line 1 as the start and will run on to the next match.

5. After sed '/^#/d' file, why can you not trust the output's line numbers?

Because deleting lines shifts everything after them up. The output's line 9 is not the input's line 9. If you need to reason about positions, get the numbers first with sed -n '/^#/=' on the original, or compare line counts before and after — reading a shifted listing is how people convince themselves a delete did not happen.

6. Write a command that strips trailing whitespace only inside the first twenty lines of a report.

sed '1,20s/[[:space:]]*$//' report.rpt. The address 1,20 is a line-number range, so it cannot reopen — numbers are compared, not searched. That is a real advantage of number ranges when you know the geometry: they do exactly one thing exactly once.

10. Summary

• Every command takes an address. No address means every line, which is why an unaddressed s edits the whole file.
• Five kinds: a line number, $ for the last line, /pattern/, a two-address range, and ! to negate any of them.
• $ in an address is the last line; $ in a pattern is the end of a line. With several files, Chapter 2.4 measured that the address $ is the last line of the last file.
• = takes only one address — command = expects up to 1 address(es), found 2, exit 1 — so it cannot inspect a range.
• /^#/!d keeps only comments; /^#/d strips them. After a delete, output line numbers no longer match input line numbers.
• A pattern range reopens. /^Startpoint/,/slack/ gave 23 lines containing two Startpoint lines, and a substitution scoped to it changed a line in the second block too. It means "every block".
• The end address is never tested on the start line, so a pattern range spans at least two lines. /^Path Group/,/^Path Group/ ran to EOF. ,+0 is the one-line form; a number range like 2,2 works because numbers are compared.
• An unmatched end address fails open to EOF — 24 lines, exit 0. Check the line count of any range extraction.
• addr,+N works on both seds here though it is not POSIX; an open range does not restart inside itself.
• 0,/re/ is GNU-only and silent on BSD (no output, exit 0) — use 1,/re/. first~step and addr,~N are GNU-only and loud, exit 1.

That closes Module 4, and with it the matching half of Part II. You can now say which lines a command runs on and describe precisely what the text on them looks like. What you have been doing to those lines is still just two things: printing and deleting. Module 5 — Filtering and Line Operations widens that to the rest of sed's line-level commands — printing and suppressing in full, deleting, inserting and appending text that was never in the input, and then scoping several commands to one region with the braces this chapter stopped short of.