for and while Loops

Part II · Module 5 · Chapter 5.5 · for and while Loops

1. Why This Matters

Chapter 3.2 taught $1, $2 and $NF. Those are enough when you know the shape of the record.

They stop being enough the moment the width is part of the data. A coverage report with one column per corner has as many columns as corners were run. A timing summary has one per clock group. A tool's output gains a column in the next release and every hardcoded $4 in your scripts quietly starts reading the wrong thing.

A loop over 1 to NF handles all of those without caring how wide the record is. That is the point of this chapter.

The other point is that loops introduce a new class of mistake. Everything in Module 4 either matched or did not. A loop can run one time too few, one time too many, or once when it should not have run at all — and in each case it produces output, which is what makes the four bugs in this chapter worth more than the syntax.

2. The for Loop

Three clauses, separated by semicolons: initialise, test, advance.

awk 'BEGIN { for (i = 1; i <= 5; i++) { printf "%s%s", sep, i; sep = " " }; print "" }' | cat -e

1 2 3 4 5$

i = 1 runs once at the start. i <= 5 is tested before each pass. i++ runs after each pass. The body is { … }.

The output uses Chapter 5.2's primed-separator idiom rather than printf "%s ", i, which would leave a trailing space before the newline — cat -e confirms the line ends right after the 5.

3. The Loop, Traced

Rather than draw a flowchart whose back-edge is hard to read, here is a loop reporting its own execution:

awk 'BEGIN { for (i = 1; i <= 3; i++) printf "test i<=3: i=%d -> true, body runs, then i becomes %d\n", i, i+1; printf "test i<=3: i=%d -> false, loop ends\n", i }'

test i<=3: i=1 -> true, body runs, then i becomes 2
test i<=3: i=2 -> true, body runs, then i becomes 3
test i<=3: i=3 -> true, body runs, then i becomes 4
test i<=3: i=4 -> false, loop ends

The last row is the one worth keeping. The condition is tested one more time than the body runs, and the loop variable survives the loop holding the first value that failed the test:

awk 'BEGIN { print "init: i = 1"; for (i = 1; i <= 3; i++) { print "  body with i =", i }; print "after: i =", i }'

init: i = 1
  body with i = 1
  body with i = 2
  body with i = 3
after: i = 4

i is 4 afterwards, not 3. Chapter 5.1 established that variables outlive the statements that create them, and a loop variable is an ordinary variable — there is no loop-local scope in awk. If a later rule uses i for something else, these two uses collide.

4. Walking Every Field

The idiom this chapter exists for:

a b c
d e

awk '{ for (i = 1; i <= NF; i++) print NR, i, $i }' fields.txt

1 1 a
1 2 b
1 3 c
2 1 d
2 2 e

Three fields from the first record, two from the second. NF is re-evaluated per record (Chapter 5.2), so the loop adapts to each width automatically.

$i is the part worth pausing on: the field reference takes an expression, not just a literal. $i means "the field whose number is the current value of i" — which is why this works at all, and is the same mechanism as Chapter 3.2's $NF.

Summing a row of a numeric table:

u_alu 120 95 88
u_fifo 75 60 45

awk '{ t = 0; for (i = 2; i <= NF; i++) t += $i; print $1, t }' wide.txt

u_alu 303
u_fifo 180

120 + 95 + 88 = 303, and 75 + 60 + 45 = 180. The loop starts at 2 to skip the name in field 1, and t = 0 resets per record — without it, t would accumulate across records, because Chapter 5.1's rule cuts both ways.

Walking backwards reverses the fields:

awk '{ for (i = NF; i >= 1; i--) { printf "%s%s", sep, $i; sep = " " }; sep = ""; print "" }' fields.txt | cat -e

c b a$
e d$

Note the sep = "" after the loop. Without it, sep would still hold " " when the next record starts, and record 2 would begin with a leading space. That is Chapter 5.1's between-records persistence showing up as a formatting bug.

5. The Off-By-One That Drops the Last Field

The first of this chapter's four bugs, and the most common. One character changes:

awk '{ for (i = 1; i < NF; i++) print NR, i, $i }' fields.txt

1 1 a
1 2 b
2 1 d

c and e are gone. i < NF stops before NF, so the last field of every record is never visited.

Compare with section 4's correct version, which produced five lines. This produces three.

A count check catches it immediately:

awk '{ for (i = 1; i <= NF; i++) n++ } END { print n, "fields visited" }' fields.txt

5 fields visited

awk '{ for (i = 1; i < NF; i++) n++ } END { print n, "fields visited" }' fields.txt

3 fields visited

Five against three, from the same five-field input. If a loop over fields reports a count you can verify independently, the off-by-one cannot hide.

6. Index 0 Is the Whole Record

The second bug, and it is specific to awk — this one has no analogue in C or Python.

awk '{ for (i = 0; i <= NF; i++) print NR, i, "[" $i "]" }' fields.txt

1 0 [a b c]
1 1 [a]
1 2 [b]
1 3 [c]
2 0 [d e]
2 1 [d]
2 2 [e]

Look at the i = 0 rows. $0 is not empty, and it is not "the field before the first". It is the entire record — which Chapter 2.1 established and which becomes a trap the moment a loop index reaches zero.

Starting a field loop at 0 therefore visits every field plus a pseudo-field containing all of them. In a sum, that double-counts:

awk '{ t = 0; for (i = 0; i <= NF; i++) t += $i; print $1, t }' wide.txt

u_alu 303
u_fifo 180

Here it happens to be harmless — $0 is u_alu 120 95 88, which has no numeric prefix, so Chapter 5.3's conversion rule makes it 0 and the sum is unaffected.

Change the data so the record starts with a number and the same loop corrupts the total:

120 95 88

awk '{ t = 0; for (i = 1; i <= NF; i++) t += $i; print "correct:", t }' numfirst.txt

correct: 303

awk '{ t = 0; for (i = 0; i <= NF; i++) t += $i; print "from zero:", t }' numfirst.txt

from zero: 423

423 instead of 303. $0 is the string 120 95 88, whose numeric prefix is 120, so the loop added an extra 120.

That is three of this curriculum's rules compounding: $0 is the whole record, a string converts to its numeric prefix without complaint (Chapter 5.3), and a loop from 0 includes it. No error, and a total that is wrong by exactly the first column.

The same reasoning is worth applying to the upper bound. i <= NF is correct and i <= NF + 1 would read $(NF+1), which is an empty field rather than an error — awk returns the empty string for any field number past NF, so an over-run is as silent as the under-run in section 5:

awk '{ print "NF is", NF, "and $(NF+1) is [" $(NF+1) "]" }' fields.txt

NF is 3 and $(NF+1) is []
NF is 2 and $(NF+1) is []

Empty, not an error. In a sum it contributes 0, so an off-by-one in the other direction is harmless to a total and fatal to a count — which is the denominator problem from Chapter 5.3 arriving through a loop bound.

7. while

while (condition) body repeats as long as the condition holds. The three for clauses, written out:

awk 'BEGIN { i = 1; while (i <= 3) { print "i =", i; i++ } }'

i = 1
i = 2
i = 3

Identical to the equivalent for. The difference is bookkeeping: for puts the initialisation, test and advance in one place where you can see all three, and while scatters them.

Use for when you are counting — which is nearly always, for field loops. Use while when the continuation condition is not a count, such as reading until something runs out. Chapter 7.4's getline loops are the natural while case.

The hazard is that while makes it easy to forget the advance, which is section 12.

8. do…while Runs at Least Once

The third bug. do { body } while (condition) tests after the body, so the body always runs at least once:

awk 'BEGIN { i = 10; do { print "ran with i =", i; i++ } while (i < 3) }'

ran with i = 10

The condition i < 3 was false from the very start — i was 10. The body ran anyway.

The while form with the same values does nothing:

awk 'BEGIN { i = 10; while (i < 3) { print "ran with i =", i; i++ }; print "while finished, i =", i }'

while finished, i = 10

No ran with line at all.

This is not a defect in do…while — it is its defining property, and occasionally what you want. It is a bug when the loop is processing data, because a do…while over an empty collection processes one item that is not there.

awk 'BEGIN { nf = 0; i = 1; do { print "processing field", i; i++ } while (i <= nf) }'

processing field 1

A record with no fields, and the loop reports processing field 1. In a script that then reads $i, it reads an empty field and treats it as data — which, by Chapter 5.3's conversion rule, becomes 0 and enters a total.

do…while is rare in awk and rarer still in field processing. If you reach for it, check what happens on empty input first.

9. break and continue

break leaves the loop immediately. continue skips to the next iteration:

awk 'BEGIN { for (i = 1; i <= 10; i++) { if (i == 3) continue; if (i == 5) break; printf "%s%s", sep, i; sep = " " }; print "" }' | cat -e

1 2 4$

Reading the output: 1 and 2 print normally. At 3, continue skips the printf but the loop goes on. At 4, printing resumes. At 5, break ends the loop, so 5 through 10 never happen.

The practical use of break is stopping a search once it has succeeded:

awk '{ for (i = 2; i <= NF; i++) if ($i > 100) { print $1, "has a value over 100 in field", i; break } }' wide.txt

u_alu has a value over 100 in field 2

Only u_alu qualifies, and the loop stopped at the first match rather than reporting fields 3 and 4 as well.

The loop starts at 2 for a reason that is not just tidiness. Start it at 1 and the name column is compared too:

awk '{ for (i = 1; i <= NF; i++) if ($i > 100) { print $1, "has a value over 100 in field", i; break } }' wide.txt

u_alu has a value over 100 in field 1
u_fifo has a value over 100 in field 1

Both records now "match", on field 1 — the instance name. $1 is u_alu, which is not a numeric string, so Chapter 5.4 section 9's rule applies and awk compares as text: u is 0x75, 1 is 0x31, so "u_alu" > "100" is true.

The break then makes it worse by hiding the evidence. Having matched field 1, the loop stops, so the real numeric fields are never examined and the output names the wrong column with complete confidence.

A loop over fields is a loop over a mixture of types. Start it past the label columns, or validate each field's shape before comparing it.

Both keywords apply to the innermost loop only. awk has no labelled break; escaping two levels means a flag variable or restructuring.

Note that break and continue affect the loop, while next and exit affect record processing — a distinction Chapter 5.6 makes properly, and a real source of confusion because all four look like control flow of the same kind.

10. Braces, Again

Chapter 5.4 section 5 showed that an unbraced if governs exactly one statement. Loops behave identically, and the symptom is different enough to be worth seeing:

awk 'BEGIN { for (i = 1; i <= 3; i++) printf "%s%s", sep, i; sep = " "; print "" }' | cat -e

123$

No spaces. The sep = " " is outside the loop, so it ran once after the loop finished and never primed the separator. The loop body was the printf alone.

With braces:

awk 'BEGIN { for (i = 1; i <= 3; i++) { printf "%s%s", sep, i; sep = " " }; print "" }' | cat -e

1 2 3$

The same lesson as 5.4 and worth the repetition: brace every multi-statement body. Indentation is invisible to awk, so a sep = " " lined up under the printf is still outside the loop if there are no braces.

11. Nested Loops and Column Aggregation

A loop inside the rule action, accumulating per column across records, aggregates a table:

awk '{ for (i = 2; i <= NF; i++) col[i] += $i; n++ } END { for (i = 2; i <= 4; i++) printf "col%d total %d mean %.1f\n", i, col[i], col[i]/n }' wide.txt

col2 total 195 mean 97.5
col3 total 155 mean 77.5
col4 total 133 mean 66.5

Column 2 totals 120 + 75 = 195, mean 97.5 over two records. The col[] array is indexed by column number — arrays are Module 6's subject, and this is the one use that needs no more than an index.

The END loop hardcodes 2 to 4, which is a limitation worth naming: it assumes the table's width. A version that does not assume it has to remember the widest record it saw, because NF in END is only the last record's:

awk '{ for (i = 2; i <= NF; i++) col[i] += $i; if (NF > maxnf) maxnf = NF; n++ } END { for (i = 2; i <= maxnf; i++) printf "col%d total %d mean %.1f\n", i, col[i], col[i]/n }' wide.txt

col2 total 195 mean 97.5
col3 total 155 mean 77.5
col4 total 133 mean 66.5

Same answer, and now correct for a table of any width — including a ragged one, where maxnf is the widest record rather than the last. Chapter 5.1's maximum-tracking pattern, applied to a field count.

A loop can also search within a record for the widest field, which is how you size a column for aligned output:

awk '{ for (i = 1; i <= NF; i++) if (length($i) > max) { max = length($i); which = $i } } END { print "widest field:", which, "at", max, "chars" }' wide.txt

widest field: u_fifo at 6 chars

length() is Module 8's, used here only to have something to compare. Note this tracker has the seeding weakness Chapter 5.1 section 9 covered — it works because field lengths are always positive, so the unset-0 seed is below every real value. On a quantity that could be negative it would fail silently.

12. The Loop That Never Ends

The fourth hazard. while has no advance clause, so forgetting to change the variable gives a loop that runs forever:

awk 'BEGIN { i = 1; while (i < 5) { print "this never stops" } }'

That command is not run here, because it does not terminate. awk will print until the disk fills or you interrupt it.

The same shape with a bail-out counter is safe, and is worth knowing as a debugging technique:

awk 'BEGIN { i = 1; guard = 0; while (i < 5) { guard++; if (guard > 100) { print "BAIL OUT after", guard, "iterations - i never advanced"; break } }; print "exited with i =", i }'

BAIL OUT after 101 iterations - i never advanced
exited with i = 1

i is still 1, which is the diagnosis: the loop variable never moved. The guard turned a hang into a one-line report naming the cause.

This is not something to ship in every loop. It is what you add when a script hangs and you need to know which loop and why — and the exited with i = 1 is the part that tells you, because a loop whose variable advanced but whose condition was wrong would show a different number.

A for loop is harder to get wrong this way because the advance clause is part of the syntax, which is the main reason to prefer for whenever a count is involved.

13. Common Mistakes

i < NF instead of i <= NF. Drops the last field of every record — section 5.

Starting a field loop at 0. $0 is the whole record, and its numeric prefix enters your total — section 6.

Forgetting to reset an accumulator per record. t = 0 at the top of the action. Variables persist across records by design.

Forgetting to reset a primed separator per record. Leaves a leading separator on every record after the first — section 4.

An unbraced multi-statement body. Only the first statement is in the loop — section 10.

Using do…while over data. Processes one item when there are none — section 8.

Forgetting the advance in a while. An infinite loop. Prefer for for counts.

Reusing a loop variable. There is no loop-local scope; i after the loop holds the value that failed the test.

Assuming NF in END is the table width. It is the last record's field count — section 11.

Confusing break with next. break leaves the loop; next abandons the record. Chapter 5.6.

14. Exercises

Exercise 1 — Predict the output

awk 'BEGIN { for (i = 3; i >= 1; i--) print i; print "after:", i }'

3
2
1
after: 0

Counting down. The loop ended when i >= 1 failed, which first happened at 0 — so i is 0 afterwards, one past the last value the body saw.

Exercise 2 — Walk the fields

Print every field of every record, tagged with its record and field number.

a b c
d e

awk '{ for (i = 1; i <= NF; i++) print NR, i, $i }' fields.txt

1 1 a
1 2 b
1 3 c
2 1 d
2 2 e

Five lines for five fields across two records of different widths.

Exercise 3 — Debug a dropped field

A script that sums a row reports totals that are consistently a little low. The command and data:

u_alu 120 95 88
u_fifo 75 60 45

awk '{ t = 0; for (i = 2; i < NF; i++) t += $i; print $1, t }' wide.txt

u_alu 215
u_fifo 135

215 against a true 303, and 135 against 180. In both cases the last column is missing: 215 + 88 = 303, and 135 + 45 = 180.

i < NF never visits field NF. The fix is one character:

awk '{ t = 0; for (i = 2; i <= NF; i++) t += $i; print $1, t }' wide.txt

u_alu 303
u_fifo 180

Note how plausible the wrong answer was. Nothing about 215 announces itself as a sum missing a term.

Exercise 4 — Debug a loop from zero

The same row sum, this time on a file with no name column, reports a total that is too high.

120 95 88

awk '{ t = 0; for (i = 0; i <= NF; i++) t += $i; print "total:", t }' numfirst.txt

total: 423

423 against a true 303 — an excess of exactly 120, which is the first column.

$0 is the whole record, 120 95 88. Chapter 5.3's conversion takes its numeric prefix, 120, and the loop added it as though it were a field.

awk '{ t = 0; for (i = 1; i <= NF; i++) t += $i; print "total:", t }' numfirst.txt

total: 303

Seeing the extra term directly makes it unambiguous:

awk '{ for (i = 0; i <= NF; i++) print i, "[" $i "]", "contributes", $i + 0 }' numfirst.txt

0 [120 95 88] contributes 120
1 [120] contributes 120
2 [95] contributes 95
3 [88] contributes 88

Index 0 and index 1 both contribute 120. Printing the intermediate values is faster than reasoning about them.

Exercise 5 — break on first match

Report the first field in each record that exceeds 100, and stop looking at that record once found.

awk '{ for (i = 2; i <= NF; i++) if ($i > 100) { print $1, "field", i, "=", $i; break } }' wide.txt

u_alu field 2 = 120

Only u_alu has such a field. Without break the loop would keep testing fields 3 and 4 — harmless here, and wasteful on a wide record where the answer was already found.

Exercise 6 — Engineering application

Aggregate a per-corner score table: total and mean per column, correct for a table of any width, with the denominator visible.

awk '{ for (i = 2; i <= NF; i++) { col[i] += $i; cnt[i]++ }; if (NF > maxnf) maxnf = NF } END { for (i = 2; i <= maxnf; i++) printf "col%d: total %d, mean %.1f over %d values\n", i, col[i], col[i]/cnt[i], cnt[i] }' wide.txt

col2: total 195, mean 97.5 over 2 values
col3: total 155, mean 77.5 over 2 values
col4: total 133, mean 66.5 over 2 values

Three improvements over the naive version. maxnf means the output adapts to the table's width instead of hardcoding it. A per-column count cnt[i] rather than a single n means a ragged table divides each column by the number of values that column actually had — which is Chapter 5.3's denominator lesson applied per column. And the count is printed, so a column with fewer values than its neighbours is visible rather than silently averaged over the wrong denominator.

The engineering caveat: these are means of recorded scores per corner, which is a different quantity from a design's coverage. A corner that was not run contributes no column and no value, so a table of three corners does not establish that only three were intended.

Exercise 7 — Challenge: prove the loop visited everything

Write a check that confirms a field loop visited every field in the file, independently of the loop being audited.

awk '{ for (i = 1; i <= NF; i++) visited++; total += NF } END { printf "visited %d, expected %d -> %s\n", visited, total, (visited == total ? "OK" : "MISMATCH") }' fields.txt

visited 5, expected 5 -> OK

The check works because the two numbers come from different places: visited is incremented by the loop under test, and total accumulates NF, which awk computed. If the loop's bound is wrong, they disagree.

Running the same check against the broken bound:

awk '{ for (i = 1; i < NF; i++) visited++; total += NF } END { printf "visited %d, expected %d -> %s\n", visited, total, (visited == total ? "OK" : "MISMATCH") }' fields.txt

visited 3, expected 5 -> MISMATCH

Caught, with the size of the discrepancy stated.

This is the general technique and it is worth more than the specific check. A loop that reports how many times it ran, against a count derived some other way, cannot have an undetected off-by-one. The same idea appeared in Chapter 5.1 as printing a tracker's denominator and in Chapter 5.3 as counting the rejected rows — in each case a second, independent number turns a silent error into a visible disagreement.

15. Summary

• for (init; test; advance) keeps all three clauses in one place. Prefer it whenever you are counting.
• The condition is tested one more time than the body runs, and the loop variable survives holding the value that failed.
• for (i = 1; i <= NF; i++) is the field-walking idiom, and NF is re-evaluated per record so it adapts to any width.
• $i takes an expression, which is what makes field loops possible.
• i < NF silently drops the last field of every record — 2% wrong on a wide table, 50% on a two-field one, and the last column is usually the important one.
• Index 0 is the whole record, not a missing field. A loop from 0 adds $0's numeric prefix to your total.
• Reset accumulators and primed separators per record. Variables persist by design.
• do…while runs its body before testing, so it processes one item over an empty collection.
• break leaves the loop; continue skips an iteration. Both affect only the innermost loop.
• An unbraced body is one statement, whatever the indentation suggests.
• NF in END is the last record's count, not the table's width. Track the maximum if you need it.
• Report how many times a loop ran against an independently derived count, and an off-by-one cannot hide.

You can now process a record of unknown width, and prove the loop saw all of it.

What none of this can do is stop processing. Every loop here ran to completion inside one record's action, and the rule itself always finished. Chapter 5.6 — next and exit covers abandoning a record part-way and stopping the whole run early, including why exit still runs END, and why printing the word FAIL does nothing whatsoever to the process's exit status.