Rebuilding Records

Part I · Module 3 · Chapter 3.6 · Rebuilding Records

1. Why This Matters

Chapter 3.3 section 6 demonstrated that setting OFS and printing $0 changes nothing, and explained why: $0 is the record as it was read.

That explanation was accurate and incomplete. Here is the missing half:

Assigning to any field makes awk rebuild $0 from the fields, joined with OFS.

One assignment turns OFS from a setting that only affects your print lists into a setting that rewrites the record itself. That is the mechanism behind converting a whitespace report to CSV, reordering columns, and normalising the spacing of a human-formatted report — all in a single pass, with no regular expressions.

2. Assigning to a Field Rebuilds the Record

Start with the baseline. Setting OFS and printing the record does nothing, exactly as Chapter 3.3 said:

DFF 120 1820
AND 45 180
MUX 18 144

awk 'BEGIN { OFS = "," } { print }' cells.rpt

DFF 120 1820
AND 45 180
MUX 18 144

Now change a field — any field — and print the record again:

awk 'BEGIN { OFS = "," } { $2 = "X"; print }' cells.rpt

DFF,X,1820
AND,X,180
MUX,X,144

Two things happened from one assignment. The second field became X, which you asked for. And every separator in the record became a comma, which you did not. awk rebuilt $0 by joining all the fields with OFS.

That second effect is the useful one.

3. The $1 = $1 Idiom

If any assignment triggers the rebuild, the cheapest possible assignment is a field to itself:

awk 'BEGIN { OFS = "," } { $1 = $1; print }' cells.rpt

DFF,120,1820
AND,45,180
MUX,18,144

$1 = $1 changes no data. It exists purely to mark the record as modified so that awk rebuilds it. The whole whitespace report became CSV, and nothing else in the command had to know the file's shape — no field count, no positions, no regex.

4. OFS Is Read at Rebuild Time

This is the subtlest behaviour in the module, and it produces a genuinely confusing bug.

The rebuild uses whatever OFS holds at the moment the record is rebuilt — not at the moment you print. Put the assignment before the OFS setting and watch what happens:

awk '{ $1 = $1; OFS = ","; print }' cells.rpt

DFF 120 1820
AND,45,180
MUX,18,144

Record 1 came out with spaces and records 2 and 3 with commas.

The trace. On record 1, $1 = $1 ran while OFS was still its default single space, so the rebuild used spaces; OFS was then set to a comma, too late for that record. But OFS is an ordinary variable and persists, so on record 2 it was already a comma when the rebuild happened.

Put the setting first and every record is consistent:

awk '{ OFS = ","; $1 = $1; print }' cells.rpt

DFF,120,1820
AND,45,180
MUX,18,144

5. Normalising Messy Spacing

The rebuild is also how you clean up inconsistent spacing, because joining with OFS discards whatever the original separators were.

printf 'DFF    120\t\t1820\n' > messy.rpt
awk '{ $1 = $1; print }' messy.rpt

DFF 120 1820

The input had a four-space run and two tabs. The output has single spaces, because OFS defaults to a single space and the rebuild used it. No OFS was set at all — the default did the work.

This is a genuinely useful one-liner for a human-formatted report that you want to feed to something stricter, and it is also a compact illustration of the whole chapter: nothing was edited, the record was reassembled.

6. Format Conversion in One Step

Combine sections 3 and 5 and the whole conversion is one command:

printf 'DFF    120\t\t1820\n' > messy.rpt
awk 'BEGIN { OFS = "," } { $1 = $1; print }' messy.rpt

DFF,120,1820

Irregular whitespace in, clean CSV out, with FS left at its generous default so runs of spaces and tabs all collapse correctly on the way in. Chapter 3.3's Figure 1 is doing both jobs at once here: the default FS splitting on the way in, and OFS joining on the way out.

Reordering is the same mechanism with real assignments:

awk 'BEGIN { OFS = "," } { t = $1; $1 = $3; $3 = t; print }' cells.rpt

1820,120,DFF
180,45,AND
144,18,MUX

The first and third fields are swapped using a temporary variable, and the record was rebuilt because fields were written. Note this is different from Chapter 3.1's reordering, which only changed the print order and left the record alone. Here the record itself is different.

7. Blanking a Field Is Not Dropping It

Now the first of two things the rebuild does not do, and it is worth being blunt because the expectation is so natural.

Setting a field to the empty string leaves the field in place, empty. It does not remove the column:

awk '{ $2 = ""; print }' cells.rpt

DFF  1820
AND  180
MUX  144

Look at the gap between DFF and 1820: that is two spaces. The record still has three fields, the middle one is empty, and both separators around it are still there. NF is still 3.

You will see this written as { $2 = ""; NF = NF; print }. The NF = NF is redundant — the assignment to $2 has already flagged the record for rebuilding, and adding a self-assignment to NF changes nothing:

awk '{ $2 = ""; NF = NF; print }' cells.rpt

DFF  1820
AND  180
MUX  144

Identical output. If you meet that idiom in someone else's script, it is not doing what its author probably thought.

8. How to Actually Drop a Column

Two honest ways, and the simpler one is better almost always.

Print the fields you want. This needs nothing from this chapter at all:

awk 'BEGIN { OFS = "," } { print $1, $3 }' cells.rpt

DFF,1820
AND,180
MUX,144

Two fields, one separator, no empty column. When you know which columns you want, say so — it is clearer than any rebuild trick and it cannot leave a stray separator behind.

Shift the later fields down and shorten the record. This genuinely removes a column from the record, which matters when you want to keep working with $0 or NF afterwards:

awk 'BEGIN { OFS = "," } { for (i = 2; i < NF; i++) $i = $(i+1); NF--; print }' cells.rpt

DFF,1820
AND,180
MUX,144

Each field from position 2 onwards is overwritten by the one after it, then NF-- shortens the record by one. The for loop is Chapter 5.5's subject and is used here only to show that the honest version exists; you do not need to write it yet.

9. Assigning to NF

NF is assignable, and assigning to it changes the record's length.

Shortening truncates:

awk 'BEGIN { OFS = "," } { NF = 2; print }' cells.rpt

DFF,120
AND,45
MUX,18

The area column is gone — genuinely gone, not blanked. This is the cleanest way to keep the first N columns of a report whose width you do not know.

Assigning a field beyond the current NF extends the record, filling the gap with empty fields:

awk 'BEGIN { OFS = "," } { $5 = "new"; print; print "NF now", NF }' cells.rpt

DFF,120,1820,,new
NF now,5
AND,45,180,,new
NF now,5
MUX,18,144,,new
NF now,5

$4 was never assigned, so it is empty, and the output shows two adjacent commas where it sits. NF became 5.

(The NF now,5 lines have a comma in them because OFS is a comma and print "NF now", NF is a two-value list. That is Chapter 3.3 section 5 being consistent, not a bug.)

10. Assigning to $0

The last case: writing to the whole record makes awk re-split it using the current FS.

awk '{ $0 = "p q r"; print NF, $2 }' cells.rpt

3 q
3 q
3 q

The record was replaced wholesale, and awk immediately re-derived the fields from the new text — NF is 3 and $2 is q, from the string just assigned rather than from the file.

This is the inverse of the rest of the chapter. Writing a field rebuilds $0 from the fields; writing $0 rebuilds the fields from $0. Both directions keep the two views consistent, which is why you can mix reading and writing freely without tracking which is current.

11. An Engineering Example

A human-formatted synthetic area report with inconsistent padding, which needs to become a clean CSV with the category column removed and a header.

printf 'u_alu    comb\t1820\nu_regfile  seq   2410\nu_io  pad\t\t640\n' > area_messy.rpt
awk 'BEGIN { OFS = ","; print "instance,area" } { print $1, $3 }' area_messy.rpt

instance,area
u_alu,1820
u_regfile,2410
u_io,640

The default FS absorbed every mixture of spaces and tabs on the way in, print $1, $3 selected and dropped, OFS joined with commas, and BEGIN supplied the header. No rebuild was needed at all, because section 8's advice applies: printing a field list is the right tool when you know which columns you want.

The rebuild earns its place when you want the record preserved but normalised — every column kept, spacing regularised:

printf 'u_alu    comb\t1820\nu_regfile  seq   2410\nu_io  pad\t\t640\n' > area_messy.rpt
awk 'BEGIN { OFS = "," } { $1 = $1; print }' area_messy.rpt

u_alu,comb,1820
u_regfile,seq,2410
u_io,pad,640

All three columns, cleanly delimited, from a file whose original spacing was inconsistent across every row.

The usual caveat applies to what these numbers mean. The third column is an area in whatever units the tool used, and Chapter 1.2 established that summing every row of a hierarchical report double-counts a parent with its children. This chapter changes the record's format, which does not make its contents mean anything more than they did.

12. Common Mistakes

Expecting OFS to affect print $0 with no assignment. Section 2. You need to write to a field first.

Setting OFS after the assignment. Section 4. Record 1 uses the old value and everything after it the new one — one wrong row near the top.

Expecting $2 = "" to drop a column. Section 7. It leaves an empty field with both separators, giving a doubled delimiter.

Writing NF = NF to force a rebuild after a field assignment. Section 7. Already redundant.

Reaching for the shift-down loop when a field list would do. Section 8. Prefer print $1, $3.

Forgetting that NF = 2 discards data. Section 9. It truncates the record, which is usually what you want and is irreversible within that record.

Being surprised that $5 = "x" creates an empty $4. Section 9. Extending a record fills the gap.

Assuming $1 = $1 is dead code and removing it. Section 3. It is the trigger, and deleting it silently restores the original spacing.

13. Exercises

Exercise 1 — Predict the output

DFF 120 1820
AND 45 180
MUX 18 144

awk 'BEGIN { OFS = "," } { print }' cells.rpt

DFF 120 1820
AND 45 180
MUX 18 144

No commas. OFS was set but no field was assigned, so $0 is still the original text. If you expected CSV, section 2 is the correction — and this is the single most common disappointment with OFS.

Exercise 2 — Modify a command

Make the previous command actually produce CSV, changing as little as possible.

awk 'BEGIN { OFS = "," } { $1 = $1; print }' cells.rpt

DFF,120,1820
AND,45,180
MUX,18,144

Three characters added. $1 = $1 flags the record so the rebuild happens.

Exercise 3 — Trace the timing bug

Predict the output of this before running it, paying attention to which record is which.

awk '{ $1 = $1; OFS = "-"; print }' cells.rpt

DFF 120 1820
AND-45-180
MUX-18-144

Record 1 is rebuilt with the default space because OFS had not yet been changed when the assignment ran. From record 2 the variable already holds -. One row differs from the rest, which is section 4's bug in miniature. Moving the OFS assignment into BEGIN fixes all three.

Exercise 4 — Debug a doubled delimiter

An engineer wants to remove the count column and writes the command below. They report "an extra space in the middle". Explain it and give them a version that works.

awk '{ $2 = ""; print }' cells.rpt

DFF  1820
AND  180
MUX  144

Blanking a field does not remove it. The record still has three fields with the middle one empty, so both of its separators remain and the output has two spaces. NF is still 3.

awk '{ print $1, $3 }' cells.rpt

DFF 1820
AND 180
MUX 144

Printing the two fields they want avoids the problem entirely, and is shorter than what they wrote.

Exercise 5 — Engineering application

Convert the messy area report to a tab-separated file with all three columns, spacing normalised, and a header row.

printf 'u_alu    comb\t1820\nu_regfile  seq   2410\nu_io  pad\t\t640\n' > area_messy.rpt
awk 'BEGIN { OFS = "\t"; print "instance\tcategory\tarea" } { $1 = $1; print }' area_messy.rpt

instance	category	area
u_alu	comb	1820
u_regfile	seq	2410
u_io	pad	640

OFS set to a tab in BEGIN — before any record, per section 4 — and $1 = $1 to trigger the rebuild on each one. The header is written with explicit tabs because it is a single string rather than a field list, so OFS does not apply to it.

Exercise 6 — Challenge: truncate, extend, and re-split

Predict each of these three outputs before running them, then explain which direction of rebuilding each one uses.

awk 'BEGIN { OFS = "," } { NF = 2; print NF, $0 }' cells.rpt

2,DFF,120
2,AND,45
2,MUX,18

awk 'BEGIN { OFS = "," } { $4 = "extra"; print NF, $0 }' cells.rpt

4,DFF,120,1820,extra
4,AND,45,180,extra
4,MUX,18,144,extra

awk 'BEGIN { FS = ":" } { $0 = "a:b:c"; print NF, $2 }' cells.rpt

3 b
3 b
3 b

The first two write to the record's fields — NF and $4 — so awk rebuilds $0 using OFS, and the comma output proves it. Note the second one extended by exactly one, so no empty gap appeared, unlike section 9's $5 case.

Notice also the comma straight after the field count: 2,DFF,120. That one is not part of the rebuilt record at all — print NF, $0 is a two-value list, so OFS separates the count from the record just as section 9's parenthetical described. Two different uses of the same variable in one line of output, which is worth seeing once.

The third writes to $0 itself, so awk goes the other way and re-derives the fields using FS — which is a colon here, giving three fields with $2 as b. The original file's contents are irrelevant; only the number of records came from it.

That is the table at the end of section 10, exercised in both directions.

14. Summary

• Assigning to any field makes awk rebuild $0 from the fields, joined with OFS. That is the half of the story Chapter 3.3 left out.
• $1 = $1 is the idiom. It changes no data and exists to flag the record as modified. Deleting it as dead code silently restores the original spacing.
• OFS is read at rebuild time, not print time. Setting it after the assignment makes record 1 differ from every other record — the same shape of bug as setting FS outside BEGIN, and the same cure.
• The rebuild normalises spacing, because joining with OFS discards whatever the original separators were. That alone makes a messy human-formatted report machine-readable.
• Blanking a field does not drop a column. $2 = "" leaves an empty field and a doubled separator, and the NF = NF often written with it is redundant.
• To drop a column, print the fields you want. The shift-down loop genuinely removes one, and is worth it only when a later rule needs the modified record.
• NF = 2 truncates the record; assigning beyond NF extends it and fills the gap with empty fields.
• Writing $0 re-splits it using FS. Fields rebuild the record with OFS; the record rebuilds the fields with FS.

That completes Module 3, and with it the whole of awk's input and output machinery. You can decide what a record is, how it is cut into fields, how to reach any of them from either end, how to detect that a report has changed shape, and how to emit the result in whatever format the next stage needs.

Every pattern in this module has been a position test, a text comparison, or a plain-text match between slashes. That last one has been doing far more work than it has been credited for: /ERROR/ is a regular expression, and so far it has only ever been asked to mean "contains this exact text". Module 4, Patterns, Conditions, and Regular Expressions, is where that becomes a language of its own — which is what the durable report parsers in Part III are built from, and what Chapter 3.4 pointed at when it said that matching the slack: label beats counting to six.