Pattern-Action Processing

Part I · Module 2 · Chapter 2.3 · Pattern-Action Processing

1. Why This Matters

Everything you have written so far has been a single rule, so the question of order never came up. Real awk programs have several, and the moment they do, two questions decide whether the output makes sense: which rules run, and in what order.

Most people guess wrong, and they guess wrong in the same way — by assuming awk works like a switch or an if / else if chain, picking the first match and moving on.

awk tries every rule against every record, and runs all the ones that match, in the order you wrote them.

That one sentence resolves most "why did awk print that" confusion. The rest of this chapter earns it and then shows what follows from it.

2. What a Rule Is

An awk program is a list of rules. Each rule has two parts:

pattern { action }

The pattern decides whether the action runs on the current record. The action, in braces, says what to do.

A synthetic simulation log to work with — four events, each with a time, a severity and a short message:

100 INFO reset_started
150 WARN timeout_near
200 ERROR data_mismatch
250 INFO simulation_finished

A complete rule:

awk '/ERROR/ { print }' sim.log

200 ERROR data_mismatch

/ERROR/ is the pattern — it matches a record containing that text. { print } is the action. Only record 3 matched, so the action ran once.

3. A Pattern With No Action

Leave the action off and awk supplies one for you:

awk '/ERROR/' sim.log

200 ERROR data_mismatch

Identical output. A pattern with no action means { print } — print the whole record. That default exists because filtering and printing is overwhelmingly the common case, and it is why awk '/ERROR/' file reads so much like grep ERROR file.

Chapter 1.3 drew the real distinction between those two commands; the point here is only that the short form is a shorthand, not a different mechanism.

4. An Action With No Pattern

Leave the pattern off instead, and the action runs on every record:

awk '{ print $2 }' sim.log

INFO
WARN
ERROR
INFO

Four records in, four lines out. An action with no pattern runs unconditionally. Every command in Module 1 that started with { was using this form.

So a rule can omit either half, and the omission has a sensible default each way:

5. More Than One Rule Can Fire

Here is the correction. Two rules, and a record that satisfies both:

awk '$2 == "ERROR" { print "flagged:", $3 } { print "seen:", $1 }' sim.log

seen: 100
seen: 150
flagged: data_mismatch
seen: 200
seen: 250

Look at record 3. It produced two lines of output — flagged: data_mismatch and seen: 200. The first rule matched it and ran; then the second rule, which has no pattern, also ran. awk did not stop after the first match.

And the two lines appear in the order the rules were written: the flagging rule is first in the program, so its output comes first for that record.

Three rules all matching one record behave the same way:

awk 'NR==3 { print "r1" } /ERROR/ { print "r2" } $1 == 200 { print "r3" }' sim.log

r1
r2
r3

Record 3 is the third record, contains ERROR, and has 200 in its first field. All three patterns are true, so all three actions ran, in source order.

6. The Record Loop Is the Outer Loop

Knowing that every rule is tried leaves one thing unsettled: does awk run rule 1 over the whole file, then rule 2 over the whole file? It does not, and the output order proves it.

awk 'NR == 1 { print "ruleA on", NR } NR == 3 { print "ruleB on", NR } { print "  always", NR }' sim.log

ruleA on 1
  always 1
  always 2
ruleB on 3
  always 3
  always 4

Read that carefully. ruleA on 1 is followed immediately by always 1 — the same record passing through both rules. Then always 2 alone, because neither position rule matched record 2. Then ruleB on 3 and always 3 together.

If awk had run each rule over the entire file in turn, the output would have been grouped: ruleA on 1, then ruleB on 3, then four always lines. It is interleaved instead.

The record loop is the outer loop. The rule list is the inner loop.

For each record, awk walks the whole rule list. Then it fetches the next record and walks the list again.

7. Source Order Changes the Answer

This is the section to remember. The two rules below are identical in both commands — only their order in the program differs.

alice 85
bob 72
carol 91

Increment first, then test:

awk '{ n++ } n == 2 { print "picked:", $0 }' scores.txt

picked: bob 72

Test first, then increment:

awk 'n == 2 { print "picked:", $0 } { n++ }' scores.txt

picked: carol 91

The same two rules, the same input, a different record.

The trace explains it. In the first version, n is incremented before the test, so by the time record 2 reaches the second rule n is already 2 and the record prints. In the second version the test comes first: on record 2, n is still 1 — it has only been incremented once, after record 1 — so nothing prints; n becomes 2 at the end of record 2, and the test finally succeeds on record 3.

8. next: Skipping the Rest of the Rules

There is exactly one ordinary construct that stops awk trying the remaining rules on a record: next. It means "I am finished with this record — fetch the next one".

Without it, both rules fire on the warning record:

awk '$2 == "WARN" { print "warn:", $3 } { print "other:", $3 }' sim.log

other: reset_started
warn: timeout_near
other: timeout_near
other: data_mismatch
other: simulation_finished

Record 2 produced two lines: warn: timeout_near from the first rule and other: timeout_near from the second. That is section 5's behaviour, and here it is unwanted — the record has been reported twice.

Add next to the first rule:

awk '$2 == "WARN" { print "warn:", $3; next } { print "other:", $3 }' sim.log

other: reset_started
warn: timeout_near
other: data_mismatch
other: simulation_finished

Now each record is reported exactly once. The warning record hit next, so awk skipped the second rule for that record only and went on to record 3 — where the second rule ran normally.

This is the standard way to express "handle this kind of record specially, and everything else the default way". Chapter 5.6 covers next properly alongside exit, which stops the input entirely rather than just the current record.

9. Kinds of Pattern

Three forms are enough for this module.

A text pattern between slashes matches records containing that text:

awk '/ERROR/' sim.log

200 ERROR data_mismatch

The slashes delimit a regular expression, which is a language of its own and the subject of Module 4. Used with plain text, as here, it simply means "contains this".

An expression matches when it is true:

awk '$2 >= 80 { print $1 }' scores.txt

alice
carol

This is the field-aware form Module 1 kept reaching for, and the one grep has no equivalent of.

Expressions combined with && for and, || for or, and ! for not:

awk '$2 >= 80 && $1 != "carol" { print $1 }' scores.txt

alice

Both conditions must hold, so carol is excluded despite her score.

A pattern that matches nothing is not an error:

awk '$2 == "FATAL" { print }' sim.log

No output, and awk exits 0. Chapter 1.6's warning applies in full: a successful run that matched nothing looks exactly like a successful run over data with no problems.

10. An Engineering Example

A first-pass triage of the synthetic log, built as several rules so each kind of record is handled once.

awk '$2 == "ERROR" { err++; print "ERROR at t=" $1 ": " $3; next } $2 == "WARN" { warn++; next } { info++ } END { print "errors:", err + 0, "warnings:", warn + 0, "info:", info + 0 }' sim.log

ERROR at t=200: data_mismatch
errors: 1 warnings: 1 info: 2

Four rules. The first reports and counts errors, then uses next so no later rule sees that record. The second counts warnings and does the same. The third catches everything left over. The END rule reports the totals once.

The counts add up to 4, which is NR — every record was counted exactly once, and that is the next calls doing their job. Remove them and the third rule would also count the error and warning records, giving info: 4 and a total of 6 for a four-record file. That is a realistic bug and it comes entirely from the rule-order behaviour this chapter is about.

The engineering caveats are the ones Chapter 1.2 established and they have not weakened: errors: 1 counts records whose second field is ERROR, not distinct failures — one root cause often prints many error records. And a simulator that died before starting prints none at all, so errors: 0 would not mean the run was clean. The + 0 on each counter matters for exactly that case: without it, a log with no errors prints an empty space where a 0 belongs.

11. Common Mistakes

Assuming awk stops at the first matching rule. The subject of section 5. It is the misconception that makes multi-rule output look random.

Assuming each rule runs over the whole file in turn. Section 6. The record loop is outer; output is interleaved.

Reordering two rules and expecting the same answer. Section 7, where it changes which record is selected.

Using next when you meant exit, or the reverse. next ends the record; exit ends the input.

Forgetting next when rules are meant to be exclusive. The triage in section 10 miscounts without it, and the miscount is plausible.

Writing = where you meant ==. NR = 2 assigns to NR rather than testing it, and awk will let you. The result is a program that quietly renumbers records.

Expecting a pattern with no action to do nothing. It prints $0. A stray /ERROR/ on its own line is a rule, not a comment.

12. Exercises

Exercise 1 — Predict the output

Before running it, say how many lines this prints and what they are.

100 INFO reset_started
150 WARN timeout_near
200 ERROR data_mismatch
250 INFO simulation_finished

awk '/INFO/ { print "a" } /reset/ { print "b" }' sim.log

a
b
a

Three lines. Record 1 matches both patterns, so it prints a then b — source order within the record. Record 4 matches only the first, printing a. Records 2 and 3 match neither. If you predicted two lines, you assumed one rule per record.

Exercise 2 — Modify a command

Rewrite awk '$2 == "ERROR" { print }' sim.log in the shortest form that produces the same output.

awk '$2 == "ERROR"' sim.log

200 ERROR data_mismatch

A pattern with no action prints $0, so the braces are unnecessary here. Keep them when the action is anything other than printing the whole record.

Exercise 3 — Trace the execution

Work out the output of this program by hand before running it. Write down, for each record, which rules fire and what each prints.

alice 85
bob 72
carol 91

awk '{ total += $2 } $2 > 80 { print $1, total }' scores.txt

alice 85
carol 248

The trace. Record 1: the first rule adds 85, so total is 85; the second rule matches and prints alice 85. Record 2: total becomes 157; 72 is not above 80, so nothing prints. Record 3: total becomes 248; 91 is above 80, so it prints carol 248.

The number beside carol is 248 — the running total of all three scores, not just the high ones. The first rule has no pattern, so it accumulated every record including bob. This is section 7's lesson: the value a pattern sees depends on what the rules above it already did.

Exercise 4 — Debug a double count

An engineer writes this triage and reports 4 info records in a 4-record log, which cannot be right alongside 1 error and 1 warning. Find the bug and fix it.

awk '$2 == "ERROR" { err++ } $2 == "WARN" { warn++ } { info++ } END { print err + 0, warn + 0, info + 0 }' sim.log

1 1 4

The third rule has no pattern, so it runs on every record — including the ones already counted as an error and a warning. The counts sum to 6 for a 4-record file.

awk '$2 == "ERROR" { err++; next } $2 == "WARN" { warn++; next } { info++ } END { print err + 0, warn + 0, info + 0 }' sim.log

1 1 2

Adding next to the first two rules makes them exclusive, and the counts now sum to 4. Note the bug produced a plausible set of numbers — nothing errored, and only arithmetic against NR reveals the problem.

Exercise 5 — Engineering application

Using the synthetic log, print each WARN and ERROR record prefixed with its severity, count them, and report the two counts at the end. Each record should be reported at most once.

awk '$2 == "ERROR" { print "E:", $3; e++; next } $2 == "WARN" { print "W:", $3; w++; next } END { print "errors:", e + 0, "warnings:", w + 0 }' sim.log

W: timeout_near
E: data_mismatch
errors: 1 warnings: 1

The W: line comes first because timeout_near is record 2 and data_mismatch is record 3 — record order, not rule order, exactly as section 6 established. The next on each rule is what keeps a record from being reported twice, and there is no catch-all rule, so INFO records produce nothing.

Exercise 6 — Challenge: make order matter

Write two programs that use the same two rules in opposite orders and produce different output, then explain the difference. Use scores.txt and a running maximum.

awk '{ if ($2 > max) max = $2 } { print $1, max }' scores.txt

alice 85
bob 85
carol 91

awk '{ print $1, "[" max "]" } { if ($2 > max) max = $2 }' scores.txt

alice []
bob [85]
carol [85]

In the first version the maximum is updated before it is printed, so each line shows the maximum including the current record. In the second it is printed first, so each line shows the maximum of the records before this one — which is why alice shows []: max had never been assigned when record 1 was printed. The brackets are there to make that emptiness visible, since an empty value prints as nothing at all.

That empty value is also a reminder from Chapter 1.6 — an unset variable is the empty string, and printing it gives you nothing rather than a zero. Neither version is wrong; they answer different questions, and only the order says which.

13. Summary

• An awk program is a list of rules, each a pattern and an action. The pattern decides whether the action runs on the current record.
• A pattern with no action means { print }. An action with no pattern runs on every record.
• awk does not stop at the first matching rule. Every rule is tried against every record and all matching ones run, in source order. A record matching three patterns runs three actions.
• The record loop is the outer loop and the rule list is the inner one. Output follows record order across records, and rule order within a record.
• Source order changes the answer. { n++ } n == 2 { print } selects the second record; the same two rules reversed select the third. A pattern sees whatever the rules above it have already done.
• next ends the current record and skips the remaining rules for it. exit ends the whole input. Rules meant to be exclusive need next, and without it the resulting miscount is plausible.
• Patterns can be a slash-delimited text match, an expression, or expressions combined with &&, || and !.
• A pattern matching nothing is not an error — no output, exit status 0, indistinguishable from clean data unless you check.
• BEGIN and END are not ordinary rules. They name phases rather than conditions on a record.

You can now read any multi-rule awk program and say which rules fire on which records and in what order. One gap is left, and both the triage in section 10 and three of the exercises leaned on it without explanation: the END rule that reported the totals.

END is not a condition about a record, and neither is BEGIN. They are the two phases that bracket the record loop this chapter has been describing. Chapter 2.4 — BEGIN, END, and Execution Order puts the whole model together, including what happens when there is no input at all.