match(), sub(), and gsub()

Part I · Module 4 · Chapter 4.5 · match(), sub(), and gsub()

1. Why This Matters

A pattern used as a test tells you a record is interesting. It does not tell you which part of it is interesting.

That is fine when the interesting part is a field — Modules 2 and 3 were about reaching fields. It fails when the value is embedded in text with no column structure, which describes most log messages and plenty of report lines.

match() reports where a pattern matched. substr() lifts that span out. sub() and gsub() rewrite it.

A synthetic report line of the awkward kind — a value wrapped in text, not in a column:

slack (VIOLATED) -0.124 ns
  slack (MET) 0.081 ns

2. match(): Finding Where

match() takes a string and a pattern and returns the position where the pattern matched, counting from 1. Zero means no match.

awk '{ print match($0, /-?[0-9]+\.[0-9]+/) }' sl2.rpt

18
15

The decimal starts at character 18 of the first record and character 15 of the second — the difference being that the second record is indented by two spaces and has a shorter verdict.

Those numbers are not useful on their own. What makes match() work is the two variables it sets as a side effect.

3. RSTART and RLENGTH

Alongside its return value, match() sets RSTART to where the match began and RLENGTH to how long it was:

awk '{ m = match($0, /-?[0-9]+\.[0-9]+/); print "pos=" m, "RSTART=" RSTART, "RLENGTH=" RLENGTH }' sl2.rpt

pos=18 RSTART=18 RLENGTH=6
pos=15 RSTART=15 RLENGTH=5

RSTART always equals the return value. RLENGTH is the new information: the first match is 6 characters (-0.124, including the minus sign) and the second is 5 (0.081).

Those two numbers describe a span, and it is worth seeing exactly which characters they pick out of the first record — slack (VIOLATED) -0.124 ns:

The matched span begins at character 18 and runs for 6 characters, which is -0.124 including its minus sign. Those are precisely the two arguments substr() wants, which is the next section.

4. Extracting With substr()

substr(string, start, length) returns part of a string. Hand it RSTART and RLENGTH and you get the matched text:

awk 'match($0, /-?[0-9]+\.[0-9]+/) { print substr($0, RSTART, RLENGTH) }' sl2.rpt

-0.124
0.081

The numbers, extracted from lines where they sit in no particular column. This is the registry's own example for this chapter and the single most useful idiom in it.

Read the whole command as a sentence: if a signed decimal is found anywhere in this record, print the part of the record where it was found. Note that match() is doing two jobs — it is the pattern that decides whether the rule runs, and it is what sets up the extraction.

5. When Nothing Matches

The no-match case is worth knowing precisely, because one of the three values is not what you would guess:

awk 'NR == 1 { m = match($0, /ZZZ/); print "m=" m, "RSTART=" RSTART, "RLENGTH=" RLENGTH }' sl2.rpt

m=0 RSTART=0 RLENGTH=-1

match() returned 0 and RSTART is 0, both as expected. RLENGTH is -1, not 0.

That matters if you ever test RLENGTH directly: RLENGTH == 0 is not "no match", and RLENGTH > 0 is a correct test for "something matched". The reliable check is the return value or RSTART, both of which are 0 on failure and therefore false.

6. match() as a Pattern

Because a non-zero number is true in awk, match() works as a pattern directly, which is how section 4's command was written:

awk 'match($0, /VIOLATED/) { print $1 }' sl2.rpt

slack

That is the same selection a plain /VIOLATED/ would make, so there is no reason to write it this way unless you also want RSTART and RLENGTH — which is exactly the case in section 4. Use match() as a pattern when you need the position; use a plain pattern when you only need the test.

7. sub() and gsub(): Rewriting

sub(pattern, replacement) replaces the first match in the record:

awk '{ sub(/ns/, "nanosec"); print }' sl2.rpt

slack (VIOLATED) -0.124 nanosec
  slack (MET) 0.081 nanosec

gsub() — the g is for global — replaces every match:

awk '{ gsub(/[()]/, ""); print }' ar.rpt

u_core (alu) area=1820
u_pad (io) area=640

u_core alu area=1820
u_pad io area=640

Both parentheses on each record were removed, because the replacement was the empty string. This is the registry's second example here and a genuinely common cleanup — stripping punctuation a tool added for human readers so that the values underneath can be used.

With no third argument, both functions operate on $0. Both can take one: sub(/x/, "y", $3) works on field 3 only, which section 10 is about.

8. The Return Value Is a Count

This is the part most people never use, and it is the cheapest sanity check available in awk.

sub() and gsub() do not return the new string. They return how many replacements they made:

awk '{ n = sub(/ns/, "nanosec"); print n, $0 }' sl2.rpt

1 slack (VIOLATED) -0.124 nanosec
1   slack (MET) 0.081 nanosec

One replacement each, as expected from sub().

awk '{ n = gsub(/[()]/, ""); print n, $0 }' ar.rpt

2 u_core alu area=1820
2 u_pad io area=640

Two per record — one open parenthesis and one close.

And when nothing matched:

awk '{ n = gsub(/ZZZ/, "x"); print n, $0 }' ar.rpt

0 u_core (alu) area=1820
0 u_pad (io) area=640

Zero, and the record unchanged.

9. The Ampersand in a Replacement

In the replacement text, & means "whatever the pattern matched":

awk '{ gsub(/[0-9]+/, "<&>"); print }' ar.rpt

u_core (alu) area=<1820>
u_pad (io) area=<640>

Each run of digits was wrapped in angle brackets without having to know what the digits were. This is how you annotate, quote or pad a value whose content you do not control.

To mean a literal ampersand, escape it:

awk 'NR == 1 { gsub(/area/, "\\&rea"); print }' ar.rpt

u_core (alu) &rea=1820

The \\& reached awk as \&, which the replacement machinery reads as a literal &. The doubling is the same string-then-pattern processing Chapter 4.4 section 10 described for dynamic patterns — the backslash has to survive the string pass first.

10. Rewriting a Field Rebuilds the Record

sub() and gsub() accept a third argument naming what to operate on, and giving them a field connects this chapter straight to Chapter 3.6:

awk 'BEGIN { OFS = "|" } { gsub(/u_/, "", $1); print }' ar.rpt

core|(alu)|area=1820
pad|(io)|area=640

Two things happened. The u_ prefix was stripped from field 1, which is what was asked. And every separator became a pipe, which was not — because writing to a field makes awk rebuild $0 using OFS, exactly as Chapter 3.6 established. A gsub on a field is a field assignment.

Now the more surprising direction. A gsub on $0 reassigns the record, and Chapter 3.6's other rule says assigning $0 makes awk re-split it. Normally that changes nothing visible:

awk '{ before = NF; gsub(/[()]/, ""); print before, NF, $0 }' ar.rpt

3 3 u_core alu area=1820
3 3 u_pad io area=640

Three fields before, three after — the parentheses were inside fields, so removing them did not move any boundary.

But replace the separators and the record collapses:

awk '{ gsub(/ /, "_"); print NF, $0 }' ar.rpt

1 u_core_(alu)_area=1820
1 u_pad_(io)_area=640

NF is 1. Every space became an underscore, so after the re-split there are no field boundaries left and the whole record is one field.

11. An Engineering Example

A log line where the value is embedded in free text — no column holds it, so no positional read can reach it:

420 UVM_ERROR Data mismatch at addr 0x1f40 after 12.75 us
500 UVM_ERROR Parity error after 3.10 us
250 UVM_INFO Checkpoint at 99.99 us

Pull out the elapsed time — the decimal before us:

awk '$2 == "UVM_ERROR" && match($0, /[0-9]+\.[0-9]+ us/) { print $1, substr($0, RSTART, RLENGTH) }' msg.log

420 12.75 us
500 3.10 us

Two things worth noticing. The severity test is field-anchored, so the UVM_INFO checkpoint is excluded even though it contains a perfectly good decimal — that is Chapter 4.1's lesson still doing work. And including us in the pattern is what narrows the claim from section 4's callout: without it, the first record's 0x1f40 has no dot so it would be skipped anyway, but on a record carrying two decimals the bare pattern would take the wrong one.

Just the number, dropping the unit:

awk '$2 == "UVM_ERROR" && match($0, /[0-9]+\.[0-9]+(?= us)/) { print substr($0, RSTART, RLENGTH) }' msg.log

No output — and that is a deliberate demonstration rather than a mistake. (?= …) is a lookahead, which POSIX extended regex does not have, so the pattern matched nothing and awk reported success, exactly as Chapter 4.4 section 9 warned. The portable way is to match including the unit and then trim:

awk '$2 == "UVM_ERROR" && match($0, /[0-9]+\.[0-9]+ us/) { t = substr($0, RSTART, RLENGTH); sub(/ us$/, "", t); print $1, t + 0 }' msg.log

420 12.75
500 3.1

Match the wider span, extract it, then sub() the unit off the extracted copy. Note 3.10 printed as 3.1 — the + 0 made it a number, and the trailing zero is not part of a number's value. Drop the + 0 if you need the text as written.

The caveats are the ones Chapter 1.2 established. These are counts of records, UVM_ERROR is the simulator's own classification, and the synthetic message layout is not any real tool's. What this chapter adds is that a value with no column can still be extracted by shape — and that the shape is itself a claim worth stating.

12. Common Mistakes

Expecting sub() to return the new string. Section 8. It returns a count and modifies in place.

Testing RLENGTH == 0 for no match. Section 5. It is -1. Test the return value or RSTART.

Using match() where a plain pattern would do. Section 6. Only reach for it when you want the position.

Forgetting that gsub on $0 re-splits the record. Section 10. A following $2 may mean something else.

Using sub() when you meant every occurrence. sub() is the first match only; gsub() is all of them.

Writing a bare & in a replacement and meaning the character. Section 9. Escape it as \\&.

Using lookahead, lookbehind or \d. None is POSIX extended regex, and all fail silently — section 11 shows one doing exactly that.

Reaching for gensub(). It returns the modified string rather than editing in place, which is often what you want — and it is a gawk extension, not POSIX, so it will not run on the awk used here. Chapter 8.6 covers that split.

13. Exercises

Exercise 1 — Predict the output

slack (VIOLATED) -0.124 ns
  slack (MET) 0.081 ns

awk '{ print match($0, /\(/) }' sl2.rpt

7
9

The position of the open parenthesis: character 7 on the first record, 9 on the second because of its two-space indent. The parenthesis is escaped because it is a grouping character in a pattern — Chapter 4.4 section 5.

Exercise 2 — Modify an extraction

Extract the verdict — the word inside the parentheses — from each record.

awk 'match($0, /\([A-Z]+\)/) { print substr($0, RSTART + 1, RLENGTH - 2) }' sl2.rpt

VIOLATED
MET

The pattern includes the parentheses so it can find them; the arithmetic on RSTART and RLENGTH then steps inside them — start one character later, and take two fewer characters. Adjusting the span after a match is a standard move and is why match() giving you two numbers is more useful than a boolean.

Exercise 3 — Trace the no-match values

Predict all three values before running this, paying attention to RLENGTH.

awk 'NR == 1 { print match($0, /QQQ/), RSTART, RLENGTH }' sl2.rpt

0 0 -1

Zero, zero, minus one. If you predicted 0 0 0 you have company — it is the natural guess and it is the reason section 5 exists. A test of RLENGTH > 0 is correct; RLENGTH == 0 never means "no match".

Exercise 4 — Debug a cleanup that did nothing

An engineer strips parentheses from a report and says the output is unchanged. Their command is below. Find out what happened, using the return value.

u_core [alu] area=1820
u_pad [io] area=640

awk '{ n = gsub(/[()]/, ""); print n, $0 }' ar_sq.rpt

0 u_core [alu] area=1820
0 u_pad [io] area=640

The count is 0 on every record: the pattern looks for round parentheses and the file has square brackets. Without printing n, the command's output is identical to its input and nothing suggests a problem.

awk '{ n = gsub(/[][]/, ""); print n, $0 }' ar_sq.rpt

2 u_core alu area=1820
2 u_pad io area=640

Two replacements each. The class [][] is the portable way to write "a square bracket, either kind" — a closing bracket must come first inside a class, otherwise it would end the class early.

Exercise 5 — Engineering application

From the message log, print the time and the hex address of every error record that carries one.

420 UVM_ERROR Data mismatch at addr 0x1f40 after 12.75 us
500 UVM_ERROR Parity error after 3.10 us
250 UVM_INFO Checkpoint at 99.99 us

awk '$2 == "UVM_ERROR" && match($0, /0x[0-9a-f]+/) { print $1, substr($0, RSTART, RLENGTH) }' msg.log

420 0x1f40

One record. The second error has no address, so match() returned 0 and the rule did not fire — the && short-circuited, which is Chapter 4.2 section 7 being useful rather than theoretical. The UVM_INFO record was excluded by the field test before the pattern was even tried.

Exercise 6 — Challenge: extract, trim, and check

Write one command that extracts the elapsed time from every record, strips the unit, reports it as a number, and flags any record where the extraction failed.

awk 'match($0, /[0-9]+\.[0-9]+ us/) { t = substr($0, RSTART, RLENGTH); n = sub(/ us$/, "", t); if (n != 1) { print $1, "TRIM_FAILED"; next } print $1, t + 0; next } { print $1, "NO_TIME" }' msg.log

420 12.75
500 3.1
250 99.99

Three records, all extracted. Walk through what the command is asserting:

• match() both selects the record and locates the span.
• substr() copies the matched text into t, leaving $0 untouched — important, because the next step modifies t rather than the record.
• sub() removes the trailing unit and its return value is checked. If the trim ever failed, n would not be 1 and the record is reported rather than silently yielding a value with a unit stuck to it.
• The final rule, with no pattern, catches any record that match() rejected and labels it NO_TIME.

So every record produces exactly one line of output, and three different outcomes are distinguishable: a number, a failed trim, or no time found. Compare that with section 11's first version, which simply printed nothing for records it could not handle.

The UVM_INFO record is included here because this command does not filter on severity — the point was extraction coverage. Adding $2 == "UVM_ERROR" && to the first pattern would restrict it, and the catch-all rule would then label the info record NO_TIME, which would be misleading. That tension — a catch-all that reports on records you never meant to process — is worth noticing, and Chapter 8.3 is where input validation gets designed properly rather than bolted on.

14. Summary

• A pattern used as a test cannot hand you what it matched. match() can.
• match() returns the 1-based position and sets RSTART and RLENGTH.
• On no match the values are 0, 0 and -1. RLENGTH is -1, not 0, so test the return value or RSTART.
• substr($0, RSTART, RLENGTH) is the extraction idiom, and arithmetic on those two numbers lets you step inside the matched span.
• This makes no claim about layout — only about shape — which is why it survives a report that gains a column. The claim moves to "the first match on the line is the one I want".
• sub() replaces the first match, gsub() all of them, both in place, and both default to $0.
• They return a count, not a string. Checking it is free and turns a silent no-op into a reported one.
• & in a replacement is the matched text; \\& is a literal ampersand.
• A gsub on a field rebuilds the record with OFS; a gsub on $0 re-splits it — so replacing whitespace can collapse NF to 1 and break every later field reference.
• Lookahead, \d and gensub() are not POSIX awk, and the regex ones fail by matching nothing.

That completes Module 4's machinery. You can describe a shape, find where it occurs, extract it and rewrite it.

What the module has not done is put it all to work at once. Each chapter has used one tool on a small fixture chosen to isolate it. Chapter 4.6 — Conditional Filtering combines a text pattern with a numeric test into the selection rules a report script actually contains, and Chapter 4.7 — Filtering Real Logs and Reports is the module's capstone, running everything over several files in a single pass.