Custom Delimiters

Part I · Module 3 · Chapter 3.5 · Custom Delimiters

1. Why This Matters

Chapter 3.3 gave you FS and the rules it follows. What it did not give you is the judgement of what to set it to, and that turns out to carry a claim you should make deliberately:

Every single field separator is a claim about what cannot appear inside a value.

Setting FS to a comma says "no value in this file contains a comma". Usually that is true and the parsing is trivial. When it is false, awk does not complain — it just produces the wrong number of fields, which is section 9.

2. The Delimiters You Actually Meet

Five cover almost everything a semiconductor flow produces:

The rest of this chapter works through them, then shows where the table's optimism runs out.

3. Comma

A synthetic regression export:

alu_test,PASS,120
fifo_test,FAIL,180

awk -F, '{ print $2 }' results.csv

PASS
FAIL

The simplest case, and it works whenever no value contains a comma.

4. Tab

alu_decode	184	200
alu_flags	96	96
fifo_ctrl	41	75

awk -F'\t' '{ print $1, $3 }' cov.tsv

alu_decode 200
alu_flags 96
fifo_ctrl 75

Tab is the most robust of the delimited formats for engineering data, because a tab is far less likely to appear inside a scope name, a path or a message than a comma or a space is. If you get to choose the export format, choose this one.

Note the quotes around '\t'. Without them the shell may eat the backslash, and FS would become the literal letter t — which would split on every t in the file and is a memorably confusing failure.

5. Colon and Semicolon

alu:120:1820
regfile:45:180
io:18:144

awk -F: '{ print $3 }' cells.colon

1820
180
144

A semicolon needs quoting, because an unquoted ; is a command separator to the shell:

alu;120;1820

awk -F';' '{ print $2 }' semi.csv

120

6. Pipe, and the Bracket Idiom

u_alu|comb|1820
u_io|pad|640

The idiom you will see, and the one to prefer:

awk -F'[|]' '{ print $2 }' report.psv

comb
pad

The bare form also works:

awk -F'|' '{ print $2 }' report.psv

comb
pad

Both give the same answer, and it is worth knowing why rather than cargo-culting the brackets.

7. Several Delimiters at Once

A character class lets one command read a file that mixes delimiters, or two files that differ:

a,b;c

awk -F'[,;]' '{ print NF, $3 }' both.txt

3 c

Three fields, because both characters acted as separators. This is FS as a regular expression — the class is two characters long, so awk treats it as a pattern rather than a literal.

Useful when a tool changed its delimiter between versions and you need one script for both. It also widens the claim from section 1: now you are asserting that neither character appears inside a value.

8. Empty Fields: Leading, Trailing, Adjacent

Chapter 3.3 showed that adjacent delimiters produce an empty field. The same applies at the edges, and all three cases matter when reading real exports.

Adjacent:

awk -F, '{ print NF, "[" $2 "]" }' adjacent.csv

a,,c

3 []

Leading:

awk -F, '{ print NF, "[" $1 "]", "[" $2 "]" }' lead.csv

,b,c

3 [] [b]

$1 is empty and the real first value is in $2. A command written for a file without the leading delimiter reads every column one position to the left.

Trailing:

awk -F, '{ print NF }' trail.csv

a,b,

3

Three fields, the last one empty. A trailing delimiter is common in generated output and means the row has a final empty column, not that it has two columns.

9. The Limit: A Delimiter Inside a Field

Here is where a single FS provably stops working. CSV allows a value to contain the delimiter by wrapping the value in quotes:

alu,"mismatch, retry",FAIL

That row has three columns: alu, mismatch, retry, and FAIL. Here is what awk sees:

awk -F, '{ print "NF=" NF; for (i = 1; i <= NF; i++) print "  $" i " = [" $i "]" }' quoted.csv

NF=4
  $1 = [alu]
  $2 = ["mismatch]
  $3 = [ retry"]
  $4 = [FAIL]

Four fields instead of three. The quoted value was torn in half at its internal comma, leaving a stray opening quote on $2 and a stray closing quote with a leading space on $3. The verdict that should be in $3 is in $4.

(The for loop there is just to show every field at once; loops are Chapter 5.5 and nothing else in this chapter needs one.)

10. A Sanity Check for Delimited Data

Chapter 3.2's self-calibrating check applies directly here, and for delimited data it is more valuable than for whitespace data — because a delimiter problem changes the field count, which is exactly what the check watches.

a,b,c
d,e
f,g,h

awk -F, 'NR == 1 { want = NF } NF != want { print "row", NR, "has", NF, "fields, expected", want }' ragged.csv

row 2 has 2 fields, expected 3

Run against the quoted file, the same check catches the torn row:

awk -F, 'BEGIN { want = 3 } NF != want { print "row", NR, "has", NF, "fields, expected", want }' quoted.csv

row 1 has 4 fields, expected 3

Here the expected count is stated in BEGIN rather than learned from record 1, because record 1 is the broken one — a reminder from Chapter 3.2 that self-calibration assumes the first record is representative.

11. An Engineering Example

A synthetic coverage export arrives as a TSV and the dashboard wants pipe-separated rows with the scope and its two counts, only for scopes that are not finished.

awk 'BEGIN { FS = "\t"; OFS = "|" } $2 < $3 { print $1, $2, $3 }' cov.tsv

alu_decode|184|200
fifo_ctrl|41|75

FS reads tabs, the pattern compares two fields against each other, OFS writes pipes. The complete format conversion and filter is one command, which is the practical payoff of this chapter and 3.3 together.

The caveats, carried forward unchanged. The columns mean "covered bins" and "total bins" because the file says so, not because awk knows. And the two numbers must not be turned into an overall coverage figure by averaging percentages — Chapter 3.3 gave the arithmetic (82.22% averaged against 86.52% pooled on exactly this data) and Module 11 covers aggregation properly.

12. Common Mistakes

Using an unquoted -F;. The shell reads the semicolon as a command separator. Quote it.

Using -F\t without quotes. FS becomes the letter t, splitting on every t in the file.

Assuming a trailing delimiter means a shorter row. Section 8. a,b, is three fields, the last empty.

Assuming a leading delimiter is ignored. It produces an empty $1 and shifts everything right. This is the whitespace rule's generosity not applying, as Chapter 3.3 section 7 explained.

Treating an empty field as a missing field. Section 8's callout. NF tells them apart; the value does not.

Writing a regex to parse quoted CSV. Section 9. It works on your sample.

Putting a bare pipe in a multi-character FS. Section 6. It becomes alternation. Use a character class.

Choosing comma when you could have asked for tab. Section 9's first honest option, and the one most often forgotten.

13. Exercises

Exercise 1 — Predict the field count

a,b,

awk -F, '{ print NF }' trail.csv

3

Three, not two. The trailing comma delimits a third, empty field. If you predicted 2 you were applying the whitespace rule's habit of ignoring edges.

Exercise 2 — Modify a command

Read the pipe-separated report and print the area column, using the defensive bracketed form.

u_alu|comb|1820
u_io|pad|640

awk -F'[|]' '{ print $3 }' report.psv

1820
640

Exercise 3 — Trace the empty fields

For each row below, say what NF is and which fields are empty, before running it.

,a,b
a,,b
a,b,

awk -F, '{ print NR, "NF=" NF, "[" $1 "]", "[" $2 "]", "[" $3 "]" }' edges.csv

1 NF=3 [] [a] [b]
2 NF=3 [a] [] [b]
3 NF=3 [a] [b] []

All three rows have three fields; only the position of the empty one differs. Every edge case produces a field rather than removing one, which is the whole difference from the whitespace default.

Exercise 4 — Debug a torn row

A colleague's extractor reports a strange verdict for one test. Their command and file are below. Diagnose it and say what you would do.

alu,"mismatch, retry",FAIL

awk -F, '{ print $1, $3 }' quoted.csv

alu  retry"

They expected the verdict FAIL in $3. The quoted middle value contains a comma, so awk split it, the row has four fields, and $3 is the tail end of the message with its closing quote. The verdict is in $4.

The fix is not a cleverer FS. It is one of section 9's three options — re-export as TSV, parse it with a real CSV reader, or check NF and refuse:

awk -F, 'NF != 3 { print "REFUSING row", NR, "- got", NF, "fields, expected 3"; next } { print $1, $3 }' quoted.csv

REFUSING row 1 - got 4 fields, expected 3

Nothing wrong is reported, which is the best awk can honestly do with this file.

Exercise 5 — Engineering application

Convert the colon-separated cell export into comma-separated output, keeping only the name and the area, with a header.

alu:120:1820
regfile:45:180
io:18:144

awk 'BEGIN { FS = ":"; OFS = ","; print "cell,area" } { print $1, $3 }' cells.colon

cell,area
alu,1820
regfile,180
io,144

One FS in, one OFS out, a header from BEGIN. The count column was read and discarded, which costs nothing.

Exercise 6 — Challenge: one command, three formats

The same three columns arrive as colon-separated, tab-separated and pipe-separated files from three machines. Write one command that reads all three, then state precisely what it assumes.

awk -F'[:\t|]' '{ print $1, $3 }' cells.colon

alu 1820
regfile 180
io 144

awk -F'[:\t|]' '{ print $1, $3 }' cov.tsv

alu_decode 200
alu_flags 96
fifo_ctrl 75

awk -F'[:\t|]' '{ print $1, $3 }' report.psv

u_alu 1820
u_io 640

One character class, three formats. Because the class is longer than one character, FS is a regular expression — and the pipe is inside brackets, which is exactly the situation section 6 warned about: written bare in a three-character FS it would have been alternation.

What it assumes, stated plainly: that none of a colon, a tab or a pipe ever appears inside a value, in any of the three files. That is three claims where a single-delimiter command makes one, and each is a chance to be wrong. A path like clk:main or a message containing a pipe would split and shift every column after it.

So this is a convenience for files you control, not a general-purpose reader. The safer shape, when the formats are genuinely distinct, is to detect the format and use the right single FS for it — or to check NF as section 10 does, so that a wrong guess fails loudly instead of producing a plausible row.

14. Summary

• Every single field separator is a claim about what cannot appear inside a value. Make it deliberately.
• Comma, tab, colon, semicolon and pipe cover almost all tool output. Tab is the most robust; prefer it when you can choose.
• Quote the separator when the shell would otherwise eat it — -F';' and -F'\t' especially, since an unquoted \t becomes the letter t.
• A bare pipe works as a one-character FS because one-character separators are literal. Use [|] anyway, so the habit survives FS growing longer, where a bare pipe becomes alternation.
• A character class reads several delimiters at once, and widens the claim to all of them.
• Leading, trailing and adjacent delimiters all produce empty fields rather than being ignored. NF distinguishes an empty field from a missing one; the value cannot.
• A quoted field containing the delimiter cannot be parsed by any FS. Quoting is a nesting rule and FS is flat. Re-export as TSV, use a real CSV reader, or check NF and refuse.
• The self-calibrating NF check from Chapter 3.2 is more valuable on delimited data, because a delimiter problem shows up as a changed field count.

You can now read whatever delimited shape a tool hands you, and recognise the one shape you should refuse.

Every chapter in this module has read from a record and printed something derived from it, leaving the record itself untouched — Chapter 3.3 made a point of OFS being unable to change $0. That is only half the story. There is a way to make awk rebuild the record from its fields, and it is what turns all of this into a tool for cleaning a report on its way through rather than only reading one. Chapter 3.6 — Rebuilding Records is the last chapter of the module.