Regular Expressions in awk

Part I · Module 4 · Chapter 4.4 · Regular Expressions in awk

1. Why This Matters

A pattern between slashes has always been a regular expression. Used with plain text it means "contains this", which has carried every example so far.

What a regular expression can actually do is describe a shape: a number, a line starting with a particular word, one of several alternatives. That is what the durable report parsers in Part III are built from, and it is what Chapter 3.4 pointed at when it said matching the slack: label beats counting to six fields.

awk uses POSIX extended regular expressions. Most special characters work without a backslash, which is not true of default grep.

A report to work with, deliberately awkward:

slack (VIOLATED) -0.124
  slack (MET) 0.081
SLACK (VIOLATED) -1.20
slack_margin 0.5

2. Anchors: ^ and $

^ matches the start of the record and $ matches the end. Neither consumes a character — they assert a position.

awk '/^slack/ { print }' sl.rpt

slack (VIOLATED) -0.124
slack_margin 0.5

Two records start with slack. The indented one does not, because its first character is a space. Note slack_margin matched — ^slack says "starts with these five characters", not "is this word".

Chapter 4.3 used a leading-space-tolerant version, and now it is readable:

awk '/^ *slack \(/ { print }' sl.rpt

slack (VIOLATED) -0.124
  slack (MET) 0.081

Read it left to right: start of record, then zero or more spaces, then slack, a space, and a literal open parenthesis. The * is section 4 and the \( is section 5. This excluded slack_margin because that line has no ( after the word.

$ anchors the other end:

awk '/MET\) 0.081$/ { print }' sl.rpt

  slack (MET) 0.081

3. Character Classes

Square brackets match any one character from a set:

awk '/^[sS]lack/ { print $1 }' sl.rpt

slack
slack_margin

[sS] is one character, either case. This is the portable case-insensitivity workaround Chapter 4.1 promised — tedious for a long word, fine for the first letter.

A leading ^ inside the brackets negates the class:

awk '$1 ~ /^[^s]/ { print $1 }' sl.rpt

SLACK

"Field 1 starts with any character that is not a lowercase s." Only the uppercase record qualifies.

That last row is worth noting now: most special characters lose their meaning inside a class, which is often the simplest way to avoid escaping.

4. Quantifiers

A quantifier says how many times the thing before it may repeat.

a
aa
aaa

* means zero or more:

awk '/^a*$/ { print "[" $0 "]" }' q.txt

[a]
[aa]
[aaa]

+ means one or more:

awk '/^a+$/ { print }' q.txt

a
aa
aaa

Those agree here because the file has no empty record — * would also match one.

? means zero or one, which is how you make something optional:

awk '/colou?r/ { print }' c.txt

color
colour
colr

color
colour

Both spellings, and not colr. The ? applies to the u immediately before it.

Braces give an exact count or a range:

awk '/^a{2,3}$/ { print }' q.txt

aa
aaa

5. The Dot, and Why Escaping Matters

. matches any single character:

awk '/sl.ck/ { print $1 }' sl.rpt

slack
slack
slack_margin

Three records contain sl, any character, then ck.

That generosity is usually unwanted when you mean a literal dot — a decimal point, a version number, a filename extension. Here are the two versions side by side:

0.124
0x124

awk '/0\.124/ { print }' d.txt

0.124

awk '/0.124/ { print }' d.txt

0.124
0x124

The escaped version matched only the decimal. The unescaped dot matched the x as well, so a pattern meant to find one value found two.

6. Alternation and Grouping

| separates alternatives and parentheses group them:

awk '/UVM_(ERROR|FATAL)/ { print $1 }' sev.txt

UVM_ERROR x
UVM_FATAL y
UVM_INFO z

UVM_ERROR
UVM_FATAL

UVM_(ERROR|FATAL) means UVM_ followed by either word. Without the parentheses, UVM_ERROR|FATAL would mean "UVM_ERROR or FATAL" — the alternation would span everything on each side, which is rarely what you want and is the same precedence trap Chapter 4.2 covered for && and ||.

7. Putting It Together: A Decimal Number

The pieces combine into a pattern that recognises a signed decimal:

awk '/-?[0-9]+\.[0-9]+/ { print $1, "has a decimal" }' sl.rpt

slack has a decimal
slack has a decimal
SLACK has a decimal
slack_margin has a decimal

Read it piece by piece: an optional minus sign, one or more digits, a literal dot, one or more digits. Every record in the file contains such a number, so all four match.

That is a test, though. It tells you a record contains a decimal; it does not hand you the number. Pulling the matched text out needs match() and substr(), which is the whole of Chapter 4.5 — and this pattern is the one that chapter opens with.

8. awk's Extended Regex vs grep's Basic Regex

Chapter 1.3 said grep understands regular expressions, which is true and incomplete: default grep understands a different dialect. Here is the same pattern in both tools, with the same data.

awk takes + bare:

awk '/^a+$/ { print }' q.txt

a
aa
aaa

Default grep treats + as an ordinary character, so nothing matches:

grep '^a+$' q.txt

Escaped, grep's basic dialect accepts it:

grep '^a\+$' q.txt

a
aa
aaa

And grep -E switches to the extended dialect, matching awk exactly:

grep -E '^a+$' q.txt

a
aa
aaa

The same split applies to alternation:

grep 'UVM_(ERROR|FATAL)' sev.txt

grep -E 'UVM_(ERROR|FATAL)' sev.txt

UVM_ERROR x
UVM_FATAL y

The practical rule: a pattern that works in grep -E works in awk. A pattern written for default grep usually needs its backslashes removed.

9. What awk Does Not Support — Silently

Two constructs you will meet in patterns copied from elsewhere, and awk supports neither.

Backreferences. In some dialects \1 means "whatever the first group matched", so (ab)\1 finds abab:

awk '/(ab)\1/ { print "matched:", $0 }' br.txt

abab
abcd

No output. POSIX extended regex has no backreferences, so this matched nothing.

Word boundaries. GNU tools accept \< and \> to mean the edge of a word:

awk '/\<aa\>/ { print "matched:", $0 }' q.txt

Again nothing, though aa is plainly a whole word in that file.

10. Dynamic Patterns Need Doubled Backslashes

Chapter 4.1 introduced a pattern held in a variable and promised this section.

A pattern written between slashes is read by awk as a pattern. A pattern held in a string is read twice — once as a string, where backslashes are processed, and then as a pattern. So a backslash the pattern needs must survive the string pass, which means writing two:

awk -v p='[0-9]+\\.[0-9]+' '$0 ~ p { print "matched:", $0 }' d.txt

matched: 0.124

The variable holds [0-9]+\.[0-9]+ after the string pass, which is the pattern that matches a decimal and not 0x124. Written with a single backslash, the string pass would consume it and the pattern would see a bare dot — matching both records, which is section 5's bug arriving by a different route.

The practical advice from Chapter 4.1 stands: keep dynamic patterns to plain words where you can, and when you cannot, remember the rule is double every backslash.

11. An Engineering Example

A pattern that identifies a slack line robustly, rather than by counting fields — which is what Chapter 3.4 said was the durable answer.

awk '/^ *slack *\(/ { print NR, $0 }' sl.rpt

1 slack (VIOLATED) -0.124
2   slack (MET) 0.081

Start of record, optional spaces, the word, optional spaces, a literal parenthesis. It accepts both indentation styles in the file and rejects slack_margin — whose next character is _ rather than a space or ( — and rejects SLACK on case.

Accepting the uppercase variant too, if the tool is inconsistent:

awk '/^ *[sS][lL][aA][cC][kK] *\(/ { print NR, $1 }' sl.rpt

1 slack
2 slack
3 SLACK

Three records now, and the clumsiness of that pattern is the honest cost of portable case-insensitivity — which is a good reason to check whether the tool is actually inconsistent before writing it.

One detail worth noticing in that output: record 2 is indented in the file, and slack prints flush here. The pattern matched the leading spaces, but $1 is a field, and Chapter 2.2's default split discards leading whitespace — so the indentation is in $0 and never in $1.

And a verdict extracted by shape rather than position:

awk '/\((VIOLATED|MET)\)/ { print $1, ($0 ~ /VIOLATED/ ? "FAIL" : "ok") }' sl.rpt

slack FAIL
slack ok
SLACK FAIL

The caveats are the ones Chapter 1.2 established. VIOLATED is the tool's own verdict rather than an independent measurement, these are records rather than physical paths, and this synthetic layout is not any real tool's format. What the chapter adds is that a pattern describing the shape of a slack line survives a report that gains a column, where a positional reader does not.

12. Common Mistakes

Leaving a dot unescaped. Section 5. 0.124 matches 0x124 too, and the output still looks like numbers.

Expecting ^word to mean a whole word. It means "starts with these characters", so it matches word_suffix as well.

Omitting the parentheses around an alternation. Section 6. UVM_ERROR|FATAL splits at the top level.

Writing a default-grep pattern in awk. Section 8. \+ in awk is a literal plus sign; drop the backslash.

Copying a pattern with \1 or \<. Section 9. awk matches nothing and reports success.

Using a single backslash in a dynamic pattern. Section 10. Double them.

Trusting an interval brace on an unknown awk. Section 4's callout.

Assuming a cleverer pattern fixes a whole-record match. Section 6's callout. Only anchoring to a field does that.

13. Exercises

Exercise 1 — Predict the output

slack (VIOLATED) -0.124
  slack (MET) 0.081
SLACK (VIOLATED) -1.20
slack_margin 0.5

awk '/^slack/ { print $1 }' sl.rpt

slack
slack_margin

Two records start with those five characters. The indented record starts with a space and SLACK differs in case. If you expected three, the slack_margin row is the reminder that ^word is a prefix test, not a word test.

Exercise 2 — Modify a pattern

Match only the records whose first field is exactly slack, in lowercase, regardless of indentation.

awk '$1 == "slack" { print NR, $2 }' sl.rpt

1 (VIOLATED)
2 (MET)

A field comparison rather than a regex — which is Chapter 4.1's point that == is the tightest tool when you know the exact value. Indentation is irrelevant because the default field split discards it (Chapter 2.2).

Exercise 3 — Trace the escaping

Predict which records each of these two patterns matches, then check.

0.124
0x124

awk '/0\.124/ { print }' d.txt

0.124

awk '/0.124/ { print }' d.txt

0.124
0x124

The escaped dot is a literal dot and matches one record. The unescaped dot matches any character, so it also accepts the x. If you predicted both patterns matching one record each, this is the bug section 5 exists to prevent — and note that the looser pattern's output is still a plausible list of values.

Exercise 4 — Debug a pattern that matches nothing

An engineer copied a pattern from a GNU grep command and reports that it finds no errors, though the log clearly contains them. Diagnose it.

UVM_ERROR x
UVM_FATAL y
UVM_INFO z

awk '/\<UVM_ERROR\>/ { print }' sev.txt

No output, exit status 0. The pattern uses \< and \>, GNU word-boundary markers that POSIX extended regex does not define — so awk did not recognise them and the pattern matched nothing. Nothing errored, which is why this is hard to spot.

awk '$1 == "UVM_ERROR" { print }' sev.txt

UVM_ERROR x

Anchoring to the field is the portable way to say "this whole word in this place", and it is tighter than the pattern they were trying to write.

Exercise 5 — Engineering application

Write one pattern that selects records whose severity is exactly UVM_ERROR or exactly UVM_FATAL, anchored to the severity field so a message mentioning either word does not match.

100 UVM_INFO Test started
250 UVM_WARNING Retry detected
420 UVM_ERROR Data mismatch
480 UVM_INFO Checking UVM_ERROR recovery
500 UVM_FATAL Bus hung

awk '$2 ~ /^UVM_(ERROR|FATAL)$/ { print $1, $2 }' sim.log

420 UVM_ERROR
500 UVM_FATAL

Two records, and record 4 correctly excluded. The ^ and $ anchor to the start and end of the field, because that is what the pattern is applied to — so the alternation cannot match a substring of a longer value either.

Compare with the registry's whole-record form:

awk '/UVM_(ERROR|FATAL)/ { print $1, $2 }' sim.log

420 UVM_ERROR
480 UVM_INFO
500 UVM_FATAL

Three records, the middle one an informational message. Same alternation, different place to apply it — which is Chapter 4.1's lesson and this chapter's syntax working together.

Exercise 6 — Challenge: write the pattern three ways

Match a signed decimal number three ways — permissively, strictly, and with an optional unit suffix — and say what each accepts that the others do not.

0.124
-1.20
42
-0.05ns
0x124

awk '/[0-9]+\.[0-9]+/ { print "permissive:", $0 }' vals.txt

permissive: 0.124
permissive: -1.20
permissive: -0.05ns

awk '/^-?[0-9]+\.[0-9]+$/ { print "strict:", $0 }' vals.txt

strict: 0.124
strict: -1.20

awk '/^-?[0-9]+\.[0-9]+(ns|ps)?$/ { print "suffixed:", $0 }' vals.txt

suffixed: 0.124
suffixed: -1.20
suffixed: -0.05ns

The permissive form has no anchors, so it finds a decimal anywhere in the record — which is why it accepted the suffixed value, and why it would also accept a decimal buried in a sentence. It rejected 42 (no dot) and 0x124 (no dot either).

The strict form anchors both ends, so the whole record must be exactly a signed decimal. That excludes the suffixed value.

The third adds an optional group — (ns|ps)? — accepting a unit if present. This is the pattern you want for a slack column, and it pairs with Chapter 4.2's + 0 habit: recognise the suffixed value with the pattern, then force the arithmetic when you compare it.

None of the three accepted 0x124, because all three require a literal dot. Had the dot been unescaped, all three would have.

14. Summary

• awk uses POSIX extended regular expressions, so +, ?, |, () and {n,m} work without backslashes.
• ^ and $ anchor to the start and end of whatever the pattern is applied to — the record for /…/, the field for $1 ~ /…/.
• ^word is a prefix test, not a word test. It matches word_suffix.
• A character class matches one character, and [^…] negates it. Most special characters are ordinary inside a class.
• Escape every dot you mean literally. An unescaped dot matched 0x124 as well as 0.124, and the output still looked like numbers.
• Parenthesise an alternation. UVM_ERROR|FATAL splits at the top level.
• A pattern that works in grep -E works in awk. A default-grep pattern usually needs its backslashes removed.
• awk supports neither backreferences nor \< word boundaries, and both fail silently — no match, exit status 0. Test a pattern against a record you know should match.
• A dynamic pattern needs every backslash doubled, because the string is processed before the pattern is.
• A cleverer pattern does not fix a whole-record match. Only anchoring to a field does.

You can now describe a shape rather than a literal, which is what makes a report parser survive a layout change.

Everything so far has used a pattern as a test — does this record match, yes or no. Section 7 built a pattern that recognises a decimal number and then had to admit it could not hand you the number. That is the last gap in this module. Chapter 4.5 — match(), sub(), and gsub() is about using a pattern to find where something is and to rewrite it, which is how a value gets lifted out of a line that has no usable columns at all.