Multiline Processing

Part II · Module 6 · Chapter 6.2 · Multiline Processing

1. Why a Pattern Cannot Match Across Lines

The pattern space holds one line, without its trailing newline — Chapter 2.2 established both halves. So there is no newline in the buffer for a pattern to match, and no second line to match against.

That is why this fails to find anything useful:

   sed -n '/UVM_ERROR.*expected/p' logs/sim.log

The error text and the word expected are on different lines, and no pattern can span them while the buffer holds only one.

Three commands change that:

   N    append the NEXT input line to the pattern space, with a newline between
   P    print only the FIRST line of the pattern space (up to the first newline)
   D    delete the first line, then RESTART the cycle without reading new input

N is the one that creates the situation. P and D are what make it useful more than once.

2. N Joins Two Lines

printf 'a\nb\nc\nd\n' | sed 'N; s/\n/+/'

a+b
c+d

Two output lines from four input lines. Each cycle read one line, N pulled in the next, and the substitution replaced the newline between them with a +.

The newline really is in the buffer and really is matchable — \n in a pattern matches it, which is the whole point. Note this is different from $: $ matches the end of the pattern space, not each embedded newline, so s/$/X/ on a two-line pattern space appends one X at the very end rather than one per line.

3. The Cycle With N, P and D

4. Without the Guard, N Eats Your Last Line

Run the section 2 command on an odd number of lines:

printf 'a\nb\nc\n' | sed 'N; s/\n/+/'

a+b

One line of output. c is gone — not merged, not printed, gone. And the exit status is 0, with nothing on stderr.

What happened: on the third cycle the pattern space held c, N tried to read a fourth line, found end of input, and on this sed discarded the pattern space and stopped. The guard $!N — "append the next line unless this is the last line" — prevents it:

printf 'a\nb\nc\n' | sed '$!N; s/\n/+/'

a+b
c

Two lines, and c survived as itself. The substitution found no newline to replace on that cycle, which is fine — a substitution that matches nothing is not an error.

5. Fixed Pairs Are Not a Sliding Window

This is the distinction that decides whether a multiline script works, and it is easy to get wrong because the simple form looks general.

N alone pairs lines 1-2, then 3-4, then 5-6. It consumes both lines of each pair, so the pair boundaries are fixed by position. If what you want is "wherever a line is followed by an indented continuation, join them", fixed pairs will catch only the ones that happen to straddle an even boundary.

P and D give you the other behaviour. Here is the registered example, on a log where errors have indented detail lines:

cat -n logs/sim.log

     1	UVM_INFO @ 120: reset released
     2	UVM_ERROR @ 980: data mismatch
     3	    expected 8'hA5 actual 8'h5A
     4	UVM_INFO @ 1200: scoreboard drain
     5	UVM_ERROR @ 1510: timeout
     6	    waiting for ready on channel 3
     7	UVM_INFO @ 2000: test done

Seven lines; two of them are continuations that belong to the line above.

sed '$!N; s/\n[[:space:]]\{4,\}/ /; P; D' logs/sim.log

UVM_INFO @ 120: reset released
UVM_ERROR @ 980: data mismatch expected 8'hA5 actual 8'h5A
UVM_INFO @ 1200: scoreboard drain
UVM_ERROR @ 1510: timeout waiting for ready on channel 3
UVM_INFO @ 2000: test done

Five lines, both continuations joined onto their errors. Now each error is one grepable line — which is the actual reason to do this, since grep UVM_ERROR on the original gives you the message without its detail.

The pairs that mattered here were 2-3 and 5-6. Note that 5-6 is an odd boundary, so N alone would never have examined it as a pair.

6. How D Makes the Window Slide

D is the mechanism, and its definition has three clauses:

   D   1. delete the pattern space up to the first newline
       2. if anything is left, restart the cycle WITHOUT reading new input
       3. if nothing is left, behave like d and read the next line

Clause 2 is the one people drop, and it is the whole trick. Walk through the log:

1. Pattern space UVM_INFO @ 120…; $!N appends line 2. The substitution needs a newline followed by four or more spaces — line 2 starts with UVM_ERROR, so no match. P prints the first line. D deletes it, leaving UVM_ERROR @ 980…, and restarts without reading.
2. Pattern space is now UVM_ERROR @ 980…; $!N appends line 3, which is indented, so the substitution joins them. P prints the joined line. D deletes it; nothing is left, so sed reads line 4 normally.
3. The same alternation continues to the end.

Because D hands the second line back as the next cycle's first line, every adjacent pair is examined. That is a sliding window, and it is why this idiom appears in every multiline sed script you will see.

P matters for the same reason. A plain p would print the whole two-line buffer, so the second line would be emitted now and again after the next D — doubling. P prints only up to the first newline, which is exactly the part about to be deleted.

printf 'a\nb\n' | sed -n 'N; P'

a

printf 'a\nb\n' | sed -n 'N; p'

a
b

One line versus two, from the same pattern space. That is the whole difference between P and p, and it is why Chapter 5.1 could only name P and not demonstrate it — with a one-line pattern space the two are identical.

7. Two Buffers, Two Ways to Get More Than One Line

It is worth separating the two mechanisms this module gives you, because they are easy to conflate:

They compose — a script can use both — but they answer different questions. If the two lines you care about are next to each other, N is simpler. If they are five lines apart, as the startpoint and slack were in Chapter 6.1, the hold space is the tool.

One more command to name and not develop: lowercase n prints the pattern space (unless -n) and reads the next line into it, replacing the contents. Uppercase N appends. So n advances the window; N grows it. The registry's subject for this chapter is N, P and D, and n is mentioned so it is not a surprise elsewhere.

8. Think It Through

1. sed 'N; s/\n/ /' report.rpt joins pairs of lines and your 999-line report comes out with 499 lines. What happened to the last one?

N hit end of input on the final cycle and discarded the pattern space — section 4 measured exactly this on a three-line file. 999 lines make 499 pairs plus one orphan, and the orphan is gone. Use $!N and you get 500 lines: 499 joined pairs and the last line alone.

2. Why does the log-joining script use P and D rather than p and d?

P prints only the first line of the buffer, which is the one D is about to remove — so each line is printed exactly once. With p the whole two-line buffer would print, and the second line would print again on the next cycle. d would discard the whole buffer including the line that has not been examined yet, and would read fresh input, so you would be back to fixed pairs with half the lines missing.

3. You need to join a line to the next two lines. How?

N twice: $!N; $!N; s/\n/ /g gives a three-line window, with a guard before each. Beyond two or three this stops being readable, and a pattern that loops — :a; $!N; s/\n[[:space:]]/ /; ta using Chapter 6.3's branching — is the general form. At that point consider whether awk's record handling is a better fit, which is Chapter 8.4's question.

4. What does s/$/X/ do on a two-line pattern space?

Appends one X at the very end, not one per line. $ matches the end of the pattern space, not each embedded newline. To act on every line inside the buffer you match the newline itself — s/\n/X\n/g puts an X before each one.

5. Your joining script works on a sample and drops some continuations in production. What is the first thing to check?

Whether the continuation lines are indented the way your pattern assumes. [[:space:]]\{4,\} requires four or more whitespace characters; a tool that indents detail lines with a single tab satisfies [[:space:]] once, not four times. sed -n 'l' logs/sim.log or cat -et shows you the real whitespace. Chapter 4.2's argument for [[:space:]] over a literal space applies, and so does its warning that a fixture which cannot discriminate proves nothing.

6. Is there a line-count check that would have caught the missing-guard bug?

Yes, and it is the cheapest possible one. For a joining script, output lines should equal input lines minus the number of joins. Count the joins first — grep -c '^[[:space:]]\{4,\}' logs/sim.log gives 2 here — then confirm 7 − 2 = 5. Any other number means something was consumed that should not have been.

9. Summary

• N appends the next input line to the pattern space with a real newline between, so a pattern can match across a line break.
• \n in a pattern matches that newline; $ still means the end of the whole buffer, not each line inside it.
• Without $!N, N on the last line discarded it — a three-line file produced one output line, exit 0, no diagnostic. This is the most destructive silent failure in the track. Always write the guard, and check wc -l.
• N alone gives fixed pairs (1-2, 3-4, …), which misses any pair straddling an odd boundary.
• P and D give a sliding window. D deletes the first line and restarts without reading, so the appended line becomes the next cycle's first line and every adjacent pair is examined.
• P prints only up to the first newline — with p instead, the second line prints twice.
• The registered example joined two indented continuations, taking a seven-line log to five, so each error became one grepable line.
• H/G bring text from the hold space; N brings it from the input. Adjacent lines want N; distant ones want the hold space.
• Lowercase n replaces the pattern space; uppercase N appends to it.

You now have both ways of holding more than one line. What the scripts so far have lacked is any notion of repeating an operation until it is done, or of skipping part of a script — the three-line window in exercise 3 wanted a loop and had to be written out twice. Chapter 6.3 — Branching and Conditional Execution adds labels, b and t, and it comes with the sharpest syntax divergence in the module: the compact one-line form you will find everywhere fails on BSD sed while reporting success.