Field Separators: FS and OFS

Part I · Module 3 · Chapter 3.3 · Field Separators: FS and OFS

1. Why This Matters

awk's default splitting is generous and it carried Modules 1 and 2 entirely. Give it a report padded out so the columns line up and it finds exactly the fields you wanted.

Give it this and the default fails completely:

alu:120:1820
regfile:45:180
io:18:144

There is no whitespace on those lines at all. The default rule finds one field per record — the whole line — and $2 is empty. Every command you have learned returns nothing useful.

The fix is one setting, and the two settings this chapter covers are the most common cure for an awk command that runs without error and tells you nothing:

FS says how to split the input. OFS says how to join your output. They act in opposite directions and are not a matched pair.

2. FS: Telling awk How to Split

FS is the field separator for input. Set it to a colon and the file above parses:

awk -F: '{ print $1 }' cells.colon

alu
regfile
io

-F: is a command-line option meaning "set FS to a colon". awk now cuts each record at every colon, so $1 is the name, $2 the count and $3 the area.

A tab-separated coverage file, which is extremely common as a tool export:

alu_decode	184	200
alu_flags	96	96
fifo_ctrl	41	75

awk -F'\t' '{ print $1, $3 }' cov.tsv

alu_decode 200
alu_flags 96
fifo_ctrl 75

The \t is a tab, and the single quotes keep the shell from touching it. Note that the default rule would also have worked on this particular file, because a tab is whitespace — but only by luck. If any scope name contained a space, the default would split it and every column after it would shift. Saying -F'\t' states that the file is tab-delimited, which is both more precise and self-documenting.

3. Two Ways to Set FS

-F on the command line and FS in a BEGIN block do the same thing:

awk 'BEGIN { FS = ":" } { print $1 }' cells.colon

alu
regfile
io

Identical to section 2's output. Use whichever reads better: -F: is shorter for a one-liner, and BEGIN { FS = ":" } is clearer in a script file where it sits beside the other setup. Section 10 explains why BEGIN specifically, and not just anywhere.

4. OFS: How Your Output Is Joined

OFS is the output field separator. It is the character awk puts between values when you separate them with a comma in print.

Read a tab-separated file and emit comma-separated output — a format conversion in one command:

awk 'BEGIN { FS = "\t"; OFS = "," } { print $1, $2 }' cov.tsv

alu_decode,184
alu_flags,96
fifo_ctrl,41

FS did the reading, OFS did the writing, and the two are set independently because they are doing independent jobs.

5. OFS Applies Only Where There Is a Comma

This is the first surprise, and it catches people who set OFS and see no effect.

With a comma, OFS is used:

awk 'BEGIN { OFS = "," } { print $1, $2 }' cells.rpt

Given the whitespace-separated report:

DFF 120 1820
AND 45 180
MUX 18 144

DFF,120
AND,45
MUX,18

Without the comma, it is not:

awk 'BEGIN { OFS = "," } { print $1 $2 }' cells.rpt

DFF120
AND45
MUX18

No separator at all. print $1 $2 is not a list of two values — it is one value, the two fields concatenated. awk had nothing to put a separator between.

6. OFS Does Not Touch $0

The second surprise. Setting OFS has no effect on printing the whole record:

awk 'BEGIN { OFS = "," } { print $0 }' cells.rpt

DFF 120 1820
AND 45 180
MUX 18 144

Still spaces. $0 is the record as it was read — Chapter 2.2 established that it is the original text, not a re-join of the fields — so OFS never enters into it.

This is a frequent disappointment: someone sets OFS = ",", prints the record, and sees no commas. The record is not built from the fields, so there is nothing for OFS to do.

There is a way to make awk rebuild $0 using OFS, and it is the subject of Chapter 3.6 — Rebuilding Records. It is deliberately not shown here, because the useful thing to learn in this chapter is that printing a record and printing its fields are two different operations.

7. The Single-Character Rule

The third surprise, and it is the one Chapter 2.2 promised. The default whitespace rule collapses runs of separators. A single-character FS does not.

The default, on a line with three spaces between two values:

awk '{ print NF }' spaces.txt

a   b

2

Two fields, as Chapter 2.2 showed: a run of whitespace is one separator.

Now a single-character FS on adjacent delimiters:

awk -F, '{ print NF, "[" $2 "]" }' adjacent.csv

a,,c

3 []

Three fields, and the middle one is empty. The two adjacent commas did not collapse into one separator — they delimited an empty field between them.

That is the correct behaviour for delimited data. In a CSV, a,,c means three columns where the middle value is missing, and that is information you want preserved. It is only surprising if you learned the whitespace rule first and assumed it was universal.

8. FS = " " Is Special

The fourth surprise, and the sharpest. Setting FS to a single space does not mean "split on one space". It means "use the default whitespace rule".

awk 'BEGIN { FS = " " } { print NF }' spaces.txt

2

Two fields on a b — the generous default behaviour, not three separators producing four fields.

To get a genuine single-space separator you have to say so in a way that is not the literal single space, which means a regular expression:

awk 'BEGIN { FS = "[ ]" } { print NF }' spaces.txt

4

Four fields: a, empty, empty, b. Three spaces are now three separators, with two empty fields between them.

9. FS Can Be a Regular Expression

Any FS longer than one character is treated as a regular expression, which is occasionally exactly what you need:

awk 'BEGIN { FS = "[0-9]+" } { print $1, $2, $3 }' re.txt

ab12cd34ef

ab cd ef

[0-9]+ means "one or more digits", so the runs 12 and 34 each acted as a separator. Regular expressions are Module 4's subject; the point here is that FS is where they first become useful, and that this is why a two-character FS behaves differently from a one-character one.

A practical case: a report separated by a colon that may have spaces around it. FS = " *: *" would absorb the spaces. Chapter 3.5 covers the delimiters you actually meet.

10. The Gotcha: Setting FS in a Rule

The fifth surprise, and the reason section 3 said BEGIN rather than "anywhere".

Setting FS inside an ordinary rule does not re-split the record that has already been read:

awk 'NR == 1 { FS = ":" } { print $1 }' cells.colon

alu:120:1820
regfile
io

Look at the first line of output. Record 1 printed whole, because it had already been split — using the old FS, whitespace — before the rule ran. The new FS took effect from record 2 onwards, which is why records 2 and 3 print correctly.

The result is a program that is wrong on exactly one record. On a three-line file that is obvious. On a 50,000-line regression summary it is one bad row in the middle of a correct-looking report.

awk 'BEGIN { FS = ":" } { print $1 }' cells.colon

alu
regfile
io

BEGIN runs before any input is read — Chapter 2.4's phase model — so the setting is in place when the first record is split. That is the whole reason FS belongs there.

11. An Engineering Example

A synthetic coverage export, tab-separated, with explicit column meanings:

alu_decode	184	200
alu_flags	96	96
fifo_ctrl	41	75

Which scopes are not finished, reading tab-delimited input:

awk -F'\t' '$2 < $3 { print $1 }' cov.tsv

alu_decode
fifo_ctrl

The same data converted to CSV for a spreadsheet, with a header:

awk 'BEGIN { FS = "\t"; OFS = "," ; print "scope,covered,total" } { print $1, $2, $3 }' cov.tsv

scope,covered,total
alu_decode,184,200
alu_flags,96,96
fifo_ctrl,41,75

One command: FS reads tabs, OFS writes commas, BEGIN adds the header line. This is the most common real use of the two settings together.

12. Common Mistakes

Setting OFS and printing $0. Section 6. $0 is the original text; nothing for OFS to do. Rebuilding is Chapter 3.6.

Forgetting the comma in print. Section 5. print $1 $2 concatenates into one value, so OFS is never consulted.

Setting FS in an ordinary rule. Section 10. The first record is already split, so it comes out wrong and everything after it is right.

Expecting FS = " " to mean one space. Section 8. It means the default whitespace rule; FS = "[ ]" is the literal single space, and you rarely want it.

Assuming a single-character FS collapses runs. Section 7. It does not, and that is correct for delimited data.

Using -F'\t' without quotes. The shell may mangle the backslash. Single-quote it.

Relying on the default for tab-separated data. It usually works, because tabs are whitespace — right up until a value contains a space. Say -F'\t' and the file documents itself.

Assuming FS affects the output or OFS affects the input. They act in opposite directions, which is the point of Figure 1.

13. Exercises

Exercise 1 — Predict the output

alu:120:1820
regfile:45:180
io:18:144

awk '{ print NF }' cells.colon

1
1
1

One field per record. There is no whitespace on these lines, so the default rule finds nothing to split on and the whole record is $1. This is what "awk printed nothing useful" looks like from the inside.

Exercise 2 — Modify a command

Print the area column — the third — from the colon-separated file.

awk -F: '{ print $3 }' cells.colon

1820
180
144

One option changes the whole picture. Note $NF would also work here; which to prefer is Chapter 3.2's decision.

Exercise 3 — Trace the separator rules

Predict NF for each of these three commands on the same one-line file before running them.

a   b

awk '{ print NF }' spaces.txt

2

awk 'BEGIN { FS = " " } { print NF }' spaces.txt

2

awk 'BEGIN { FS = "[ ]" } { print NF }' spaces.txt

4

The default and FS = " " are the same thing — that is section 8's rule. FS = "[ ]" is a genuine single-space regular expression, so three spaces are three separators and the record has four fields, two of them empty. If you predicted 2, 4, 4 you had the special case backwards, which is the normal first guess.

Exercise 4 — Debug the one bad record

An engineer runs the command below on a colon-separated file of 2,000 rows and reports that "the first line is broken but the rest are fine". Explain the cause and fix it.

awk 'NR == 1 { FS = ":" } { print $1 }' cells.colon

alu:120:1820
regfile
io

Record 1 was split before the rule that set FS ran, so it was split on whitespace and the entire line became $1. From record 2 onwards the new FS was already in place. The bug is in the timing, not the separator.

awk 'BEGIN { FS = ":" } { print $1 }' cells.colon

alu
regfile
io

BEGIN runs before any record is read, so the setting applies to all of them. This is Chapter 2.4's phase model doing real work.

Exercise 5 — Engineering application

Convert the tab-separated coverage export to a pipe-separated file, keeping only the scope and the covered-bin count, with no header.

awk 'BEGIN { FS = "\t"; OFS = "|" } { print $1, $2 }' cov.tsv

alu_decode|184
alu_flags|96
fifo_ctrl|41

FS and OFS set independently in BEGIN, and the comma in print is what lets OFS apply. Changing the output delimiter needed no change to how the input is read, which is Figure 1's asymmetry being useful rather than confusing.

Exercise 6 — Challenge: two separators, one answer

A tool exports the same three columns in two formats on different machines — one colon-separated, one tab-separated. Write a single command that reads either and prints the first and third columns, then explain what it assumes.

alu:120:1820
regfile:45:180
io:18:144

awk -F'[:\t]' '{ print $1, $3 }' cells.colon

alu 1820
regfile 180
io 144

awk -F'[:\t]' '{ print $1, $3 }' cov.tsv

alu_decode 200
alu_flags 96
fifo_ctrl 75

One command, both files. [:\t] is a two-character-class regular expression meaning "a colon or a tab", and because it is longer than one character awk treats FS as a regex — section 9.

What it assumes is worth stating. It assumes neither a colon nor a tab ever appears inside a value, and that both formats put the same three columns in the same order. The first assumption is the dangerous one: a scope name containing a colon would split into two fields and shift everything after it. A single FS is always a claim about what cannot appear in the data, which is precisely the limitation Chapter 3.5 takes up.

14. Summary

• FS splits the input; OFS joins your output. They act in opposite directions and changing one does not affect the other.
• -F: and BEGIN { FS = ":" } are equivalent. Prefer BEGIN in a script where it sits with the other setup.
• OFS is used only where a comma appears in print. print $1 $2 concatenates into one value and OFS is never consulted.
• OFS does not affect $0. The record is the original text, so printing it shows the original separators. Rebuilding it is Chapter 3.6.
• A single-character FS does not collapse runs. a,,c is three fields with an empty middle, which is correct for delimited data and the opposite of the whitespace default.
• FS = " " means the default whitespace rule, not one space. FS = "[ ]" is the literal single space, and it will manufacture empty fields across any padded report.
• An FS longer than one character is a regular expression, which is how one command reads two delimiters.
• Setting FS in an ordinary rule leaves the first record split by the old one. It belongs in BEGIN, for the reason Chapter 2.4 explains.
• Any single FS is a claim about what cannot appear inside a value.

You can now read a file in whatever delimited shape a tool gave you, and emit it in whatever shape the next stage wants.

What has not changed in this chapter is what counts as a record: every example still read one line at a time. That is a separate setting, and it is the one that unlocks the report shape Chapter 1.2 kept deferring — a timing path whose startpoint, endpoint and slack sit on three consecutive lines, which no amount of field splitting can reach. Chapter 3.4 — Record Separators: RS and ORS is where a record stops being a line.