Range Patterns

Part I · Module 4 · Chapter 4.3 · Range Patterns

1. Why This Matters

Chapter 3.4 solved a version of this problem by changing what a record is: set RS to the empty string and a blank-line-separated block becomes one record.

That works when the blocks really are separated by blank lines. Often they are not — the boundaries are content. A timing report's path block begins at a line saying Startpoint and ends at a line saying slack, with no blank line anywhere. You cannot express that as a record separator.

A range pattern selects every record from the first match of one pattern to the next match of another.

It leaves records alone and selects a span of them, which is a different tool from RS rather than a better one.

2. A Range Is a Pattern With Memory

A synthetic report with two path blocks and some surrounding noise:

HEADER junk
Startpoint: u_alu/a
  data arrival 1.200
  slack -0.124
Startpoint: u_io/pad
  data arrival 0.900
  slack 0.081
FOOTER junk

The range is written as two patterns separated by a comma:

awk '/^Startpoint/,/slack/' paths.rpt

Startpoint: u_alu/a
  data arrival 1.200
  slack -0.124
Startpoint: u_io/pad
  data arrival 0.900
  slack 0.081

Six records selected, and the HEADER junk and FOOTER junk lines discarded. The ^ in /^Startpoint/ anchors the match to the start of the record so an occurrence mid-line would not trigger it — Chapter 4.4 covers anchors properly.

What makes this different from everything before it is that awk had to remember something. On the data arrival record, nothing in that record matches either pattern — it was selected because the range was already open.

3. Inclusive at Both Ends, and It Re-Triggers

Two properties the output above already demonstrated, worth stating explicitly.

Both boundary records are included. The Startpoint line and the slack line are in the output, not just the lines between them. Count them:

awk '/^Startpoint/,/slack/ { n++ } END { print n }' paths.rpt

6

Six, which is two blocks of three — start, middle, end.

The range re-triggers. After the first slack line turned it off, the second Startpoint turned it on again. One file can contain any number of ranges, and that is usually what you want from a report with repeated blocks.

4. When Both Patterns Match One Record

The end pattern is tested on the record that started the range. If it matches there, the range is one record long:

awk '/START/,/END/ { print "[" $0 "]" }' same.txt

x
START and END here
y

[START and END here]

One record. awk opened the range on that record, tested the end pattern against the same record, found it, and closed again — selecting exactly that line.

The degenerate version makes it obvious:

awk '/BOTH/,/BOTH/ { print "[" $0 "]" }' both.txt

a
BOTH
b

[BOTH]

A range whose two patterns are identical selects each matching record on its own. That is occasionally a convenient way to write "just these lines", though a plain pattern says it more clearly.

5. When the End Pattern Never Comes

Here is the first real hazard. If the end pattern never matches, the range stays open to the end of the file:

awk '/^Startpoint/,/NEVERMATCHES/' paths.rpt

Startpoint: u_alu/a
  data arrival 1.200
  slack -0.124
Startpoint: u_io/pad
  data arrival 0.900
  slack 0.081
FOOTER junk

Seven records — everything from the first Startpoint to the end of the file, including the FOOTER junk line the earlier command correctly excluded.

6. A Range Can Swallow the Next Block's Start

The second hazard, and the one most people get wrong when reasoning rather than testing.

A file with two start markers and only one end marker:

S
a
S
b
E

awk '/S/,/E/ { print "[" $0 "]" }' reopen.txt

[S]
[a]
[S]
[b]
[E]

All five records. Not two ranges — one.

The range opened at record 1 and stayed open looking for E. Record 3 is another S, but the start pattern is not tested while the range is open, so that record was simply selected as part of the ongoing range. The range finally closed at E.

7. Ranges on Line Numbers

The boundary patterns can be any expressions, not just regexes. Comparisons on NR give a positional range:

awk 'NR == 2, NR == 4 { print NR, $0 }' paths.rpt

2 Startpoint: u_alu/a
3   data arrival 1.200
4   slack -0.124

Records 2 to 4 inclusive — the first path block, selected by position. Useful for pulling a known section out of a file whose layout is fixed, and for isolating a span while debugging.

Mixed boundaries work too: NR == 2, /slack/ starts at a line number and ends at a content match.

8. The Two-Flag Alternative

Anything a range does can be written with an ordinary variable as a flag, and the longer form buys control:

awk '/^Startpoint/ { inblk = 1 } inblk { print } /slack/ { inblk = 0 }' paths.rpt

Startpoint: u_alu/a
  data arrival 1.200
  slack -0.124
Startpoint: u_io/pad
  data arrival 0.900
  slack 0.081

Identical output to section 2. Three rules, firing in source order on each record — which is exactly Chapter 2.3's model, and worth tracing: on the Startpoint record the first rule sets the flag, then the second prints; on the slack record the second prints before the third clears the flag, which is why the end record is included.

The flag form is worth the extra characters when you need something a range cannot express:

That fourth row is the important one, because it addresses section 5's hazard directly — and section 10 uses it.

9. An Engineering Example

Pull the slack line out of each path block, and verify that the file's blocks were well-formed.

First the simple version, combining a range with an ordinary pattern — the range selects the block and the inner test picks the line:

awk '/^Startpoint/,/slack/ { if (/slack/) print }' paths.rpt

  slack -0.124
  slack 0.081

The if is Chapter 5.4's subject; here it is only filtering within the selected span. (A plain awk '/slack/' would give the same answer on this file and would also match a slack line outside any block, which is precisely the kind of looseness the range removes.)

Now the version that checks its own assumptions, using the flag form so that an unterminated block is reported rather than silently swallowing the trailer:

awk '/^Startpoint/ { if (inblk) merged++; inblk = 1; blocks++ } inblk && /slack/ { print $2; inblk = 0 } END { print "blocks:", blocks+0, "merged:", merged+0, "unterminated:", (inblk ? 1 : 0) }' paths.rpt

-0.124
0.081
blocks: 2 merged: 0 unterminated: 0

Three numbers that make the extraction trustworthy: two blocks were seen, none of them started while another was still open, and the file did not end mid-block.

The same program on a file whose last block is truncated:

printf 'Startpoint: u_alu/a\n  slack -0.124\nStartpoint: u_io/pad\n  data arrival 0.900\n' > truncated.rpt
awk '/^Startpoint/ { if (inblk) merged++; inblk = 1; blocks++ } inblk && /slack/ { print $2; inblk = 0 } END { print "blocks:", blocks+0, "merged:", merged+0, "unterminated:", (inblk ? 1 : 0) }' truncated.rpt

-0.124
blocks: 2 merged: 0 unterminated: 1

Two blocks seen, one slack value extracted, and unterminated: 1 says the file ended with a block still open. A plain range would have selected the trailing records silently and reported nothing unusual.

The caveats from Chapter 1.2 stand. The slack values are whatever that column holds in this synthetic layout, a real report's block is far longer and carries a delay table, and the number of blocks is a count of checks rather than of physical paths.

10. Common Mistakes

Assuming an unterminated range selects nothing. Section 5. It selects everything to end of file.

Assuming two start markers make two ranges. Section 6. The start pattern is ignored while the range is open.

Expecting the boundaries to be excluded. A range is inclusive at both ends. Use the flag form to exclude them.

Forgetting the end pattern is tested on the start record. Section 4. If both match one record, the range is that one record.

Using a range where a plain pattern would do. /slack/,/slack/ is a long way to write /slack/.

Using a range when RS = "" is the better tool. If the blocks are blank-line separated and you want their contents as fields, Chapter 3.4 is the chapter.

Not counting what you selected. A range gives no feedback about how many spans it opened or whether the last one closed. Section 9 adds that.

Writing /a/, /b/ with a space and expecting trouble. The space is harmless; awk reads the comma. It is the comma that makes it a range, and /a/ /b/ without one is a concatenation, not a range.

11. Exercises

Exercise 1 — Predict the record count

HEADER junk
Startpoint: u_alu/a
  data arrival 1.200
  slack -0.124
Startpoint: u_io/pad
  data arrival 0.900
  slack 0.081
FOOTER junk

awk '/^Startpoint/,/slack/ { n++ } END { print n }' paths.rpt

6

Six: two blocks of three records each, boundaries included. If you said 2 you were counting blocks; if you said 8 you were counting the whole file.

Exercise 2 — Modify a command

Select from the header line to the first slack line.

awk '/HEADER/,/slack/' paths.rpt

HEADER junk
Startpoint: u_alu/a
  data arrival 1.200
  slack -0.124

Four records. The range closed at the first slack and did not re-open, because HEADER appears only once.

Exercise 3 — Trace the state

For the file below, write down for each record whether the selector is on or off before and after it, then check your prediction against the output.

S
a
S
b
E

awk '/S/,/E/ { print NR, "[" $0 "]" }' reopen.txt

1 [S]
2 [a]
3 [S]
4 [b]
5 [E]

Record 1 turns it on. Records 2, 3 and 4 are selected because it is already on — and record 3 matching the start pattern is irrelevant, since awk is only looking for E. Record 5 matches the end, is selected, and turns it off.

All five records. If you predicted two separate ranges, section 6 is the correction, and it is the most commonly mispredicted behaviour in this chapter.

Exercise 4 — Debug a range that selected too much

An engineer extracts path blocks with the command below and reports that their block count is far too high and includes summary text. Diagnose it.

Startpoint: u_alu/a
  slack -0.124
Startpoint: u_io/pad
  slack 0.081
=== SUMMARY ===
total paths 2

awk '/^Startpoint/,/Slack/ { n++ } END { print n }' paths_typo.rpt

6

Six records from a file with two two-record blocks. Their end pattern is /Slack/ with a capital S, and the report says slack — Chapter 4.1 section 8 established that matching is case-sensitive. The end pattern never matched, so the range ran to EOF and swallowed the summary.

awk '/^Startpoint/,/slack/ { n++ } END { print n }' paths_typo.rpt

4

Four records, two blocks of two. A one-character typo in the end pattern changed the answer by 50% with no error — which is section 5's hazard in its most ordinary form.

Exercise 5 — Engineering application

Print only the slack value from each path block, and report how many blocks were seen.

awk '/^Startpoint/ { blocks++ } /^Startpoint/,/slack/ { if (/slack/) print $2 } END { print "blocks:", blocks+0 }' paths.rpt

-0.124
0.081
blocks: 2

Two rules: one counts the block starts, the other selects each block and picks its slack line. The count is the cross-check — two slack values from two blocks is consistent, and a mismatch would say a block was malformed.

Exercise 6 — Challenge: exclude the boundaries

A range is inclusive. Write something that selects only the records between the markers, excluding both, and explain why a range cannot do it.

awk '/slack/ { inblk = 0 } inblk { print } /^Startpoint/ { inblk = 1 }' paths.rpt

  data arrival 1.200
  data arrival 0.900

Only the middle records. A range cannot express this because its boundary records are selected by definition — the construct has no way to say "turn on after this one".

The flag version can, and the trick is the rule order, which is Chapter 2.3's lesson doing real work. Read it in the order awk does, for each record:

• On the Startpoint record: the first rule does not match, the second sees inblk still unset so prints nothing, the third sets the flag. The boundary is excluded because the flag is set after the print rule has already run.
• On a data arrival record: only the middle rule fires, and it prints.
• On the slack record: the first rule clears the flag before the print rule runs, so it is excluded too.

Moving any of those three rules changes which boundaries are included. That is the same "source order is semantics" point Chapter 2.3 made with a counter, and it is why the flag form gives control that a range's fixed inclusivity cannot.

12. Summary

• A range selects every record from the first match of one pattern to the next match of another, written as two patterns separated by a comma.
• It is the first stateful construct in this track. A record with nothing matching can be selected because the range is already open.
• Both boundary records are included, and the range re-triggers, so one file can hold many ranges.
• The end pattern is tested on the starting record. If both match one record, the range is that single record.
• An unterminated range fails open — it selects everything to the end of the file, trailer included. A one-character typo in the end pattern is enough.
• Two start markers do not make two ranges. While the range is open, the start pattern is ignored, so a block missing its end marker merges with everything after it.
• Boundaries can be comparisons, so NR == 2, NR == 4 is a positional range.
• The two-flag form is longer and more capable: it can exclude the boundaries, count how many spans opened, and detect an unterminated one in END.
• Range or RS = ""? RS when blocks are blank-line separated and you want their contents as fields; a range when the boundaries are content.

You can now select a span of records whose edges are marked by content, and check that the spans were well-formed.

Every boundary pattern in this chapter has been plain text, with one exception that was waved through: the ^ in /^Startpoint/, which anchored the match to the start of the record. That character belongs to a language this track has been using since Chapter 1.1 and has never explained. Chapter 4.4 — Regular Expressions in awk is where the slashes stop holding plain text and start holding patterns that can describe a shape.