Turning a Line Into Fields

Part I · Module 5 · Chapter 5.8 · Turning a Line Into Fields

1. The Four Steps

Every line of a delimited file goes through the same sequence:

1. Strip the line ending — chomp, from Chapter 4.9.
2. Split on the separator — split, from Chapter 4.7, with -1 so trailing empty columns survive.
3. Check the field count — from Chapter 5.1 §5, because what comes next assumes a length.
4. Name the fields — list assignment, from Chapter 5.6 §6.

Step 3 is the one that gets skipped. Steps 1, 2 and 4 are the obvious work; step 3 is what makes the other three trustworthy.

use strict;
use warnings;

my $line = "alu_smoke_01,PASS,184,7\n";

chomp $line;

my @field = split /,/, $line, -1;

if (scalar(@field) != 4) {
    die "expected 4 fields, got " . scalar(@field) . "\n";
}

my ($test, $status, $total, $fails) = @field;

printf "%-14s %-6s total=%-5d fails=%d\n", $test, $status, $total, $fails;

alu_smoke_01   PASS   total=184   fails=7

Four statements of work and one guard. The die is Chapter 3.9's, and using it here means a malformed file stops the script rather than producing a quietly wrong report.

2. Two Ways to Walk Off a Short Row, Both Silent

This is the section the chapter exists for.

use strict;
use warnings;

my @field = split /,/, "alu,PASS", -1;

printf "got %d field(s)\n", scalar(@field);

my $total_by_index = $field[2];

my ($test, $status, $total) = @field;

printf "by index       : [%s]\n", (defined $total_by_index ? $total_by_index : "undef");
printf "by destructure : [%s]\n", (defined $total          ? $total          : "undef");

got 2 field(s)
by index       : [undef]
by destructure : [undef]

Two fields came back. Both attempts to read a third got undef, and STDERR is empty.

Those two lines look nothing alike and they are the same bug:

• $field[2] reads past the end of an array — Chapter 5.1 §5 measured that this is silent.
• my ($test, $status, $total) = @field is a list assignment with more variables than values — Chapter 5.6 §6 measured that this is silent too.

Perl has no bounds checking to switch on for either. The count check is the only thing that distinguishes a good row from a short one.

3. What a Real File Throws at You

A data file is rarely just rows. Here is one with every common complication:

# regression results, generated 2026-10-10
test,status,total,fails
alu_smoke_01,PASS,184,7

fifo_wr_02,FAIL,92,92
short_row,PASS
decoder_03,PASS,1240,

Six content lines and four problems: a comment, a header, a blank line, a short row, and a row whose last field is empty.

use strict;
use warnings;

open my $fh, "<", "results.csv" or die "cannot open results.csv: $!\n";

my $header_seen = 0;
my @good;
my @bad;

while (my $line = <$fh>) {
    chomp $line;

    next if $line =~ /^\s*$/;      # blank line
    next if $line =~ /^\s*#/;      # comment

    if (!$header_seen) {
        $header_seen = 1;
        next;
    }

    my @field = split /,/, $line, -1;

    if (scalar(@field) != 4) {
        push @bad, sprintf("%s (%d fields)", $line, scalar(@field));
        next;
    }

    push @good, \@field;
}

close $fh;

printf "accepted %d row(s), rejected %d\n", scalar(@good), scalar(@bad);

if (@bad) {
    print "rejected:\n";
    print "  $_\n" for @bad;
}

accepted 3 row(s), rejected 1
rejected:
  short_row,PASS (2 fields)

Three rows accepted, one rejected with its content and field count in the message — which is what makes it actionable. A reader of that output can find the offending line immediately.

Four things in that loop are worth naming:

• chomp first, before anything looks at the line. Chapter 4.9 §6's "chomp on input, once".
• Skip blanks and comments before counting fields, or a blank line becomes a one-field row and lands in @bad as noise. The two patterns are regular expressions and belong to Module 11; /^\s*$/ is "nothing but whitespace" and /^\s*#/ is "starts with a hash".
• Collect the failures rather than dying. Chapter 3.9's die is right when one bad row means the file is unusable; push @bad is right when you want to process what you can and report the rest. The accumulate pattern is Chapter 5.4 §5's.
• if (@bad) — Chapter 5.7 §3's idiom, so the rejected section appears only when there is something to report.

The \@field in push @good, \@field is a reference, which is Module 7's subject. It appears here because Chapter 5.12 is about to explain why push @good, @field would not work — and this chapter needed to store rows somehow. Treat it as a forward pointer rather than something to understand yet.

The row with an empty last field

Note decoder_03,PASS,1240, was accepted. It has four fields, the last one empty, and that is correct — an empty cell is data. It only works because of the -1 on the split:

use strict;
use warnings;

my $row = "decoder_03,PASS,1240,";

my @with    = split /,/, $row, -1;
my @without = split /,/, $row;

printf "with -1    : %d fields\n", scalar(@with);
printf "without -1 : %d fields\n", scalar(@without);

with -1    : 4 fields
without -1 : 3 fields

Without the -1 that row has three fields and fails the count check — rejected as malformed when it is perfectly good. Chapter 4.7 §2 measured why.

4. Three Ways to Name the Fields

use strict;
use warnings;

my @field = split /,/, "alu,smoke,PASS,184", -1;

my ($block, $test, $status, $total) = @field;
printf "destructured : %s / %s / %s / %s\n", $block, $test, $status, $total;

my ($st, $blk) = @field[2, 0];
printf "slice        : %s / %s\n", $st, $blk;

printf "by index     : %s / %s\n", $field[2], $field[0];

destructured : alu / smoke / PASS / 184
slice        : PASS / alu
by index     : PASS / alu

Prefer names. $status tells a reader what the comparison means; $field[2] makes them count commas in the header. The cost of the destructuring line is one line, and it also documents the file format at the top of the loop.

The one case for raw indexes is a single use in a single expression — a sort comparator, say, where naming would be noise.

5. Guarding by Minimum Rather Than Exactly

!= 4 is right when the format is fixed. Sometimes it is not:

use strict;
use warnings;

for my $row ("alu,PASS,184,7", "fifo,FAIL,92,92,extra,notes") {
    my @field = split /,/, $row, -1;

    if (scalar(@field) < 4) {
        printf "REJECT  %-32s (%d fields)\n", $row, scalar(@field);
        next;
    }

    my ($test, $status) = @field[0, 1];
    printf "accept  %-14s %-6s (%d fields)\n", $test, $status, scalar(@field);
}

accept  alu            PASS   (4 fields)
accept  fifo           FAIL   (6 fields)

Both accepted. A tool that gained two columns in an upgrade still parses, because the script asked for "at least four" rather than "exactly four".

Which guard to use is a real decision:

• != expected catches a format change immediately, which is what you want when a silent reinterpretation would be dangerous.
• < minimum survives added columns, which is what you want when the tool's output grows and you only ever read the first few fields.

Either is defensible; having neither is not.

6. Why This Matters in Engineering Work

This is the job. Regression summaries, timing reports, coverage tables, lint logs — all delimited text, all this pipeline. The module's earlier chapters are the parts; this is the assembly.

The short row is not hypothetical. A tool killed mid-write leaves a truncated final line. A field containing the separator shifts everything. A version change adds or drops a column. All three produce rows with the wrong field count, and all three are silent without the check.

A rejected row must say which row. push @bad, sprintf("%s (%d fields)", $line, scalar(@field)) costs nothing and turns "the report has gaps" into "line 7 of results.csv has 2 fields". That difference is the difference between a five-minute fix and an afternoon.

And printf "%d" on a missing field prints 0. Worth repeating because of what zero means in this trade: zero errors, zero violations, zero failures. Chapter 4.8 §6 measured it. A missing count rendered as a clean result is the worst available outcome, and the count check is what prevents it.

7. Common Mistakes

Skipping the count check. Both ways of reading a missing field are silent.

Splitting without -1. A legitimately empty last column makes the row look short — Chapter 4.7 §2.

Forgetting chomp. The last field carries a newline and no text comparison on it works — Chapter 4.9 §1.

Counting fields before skipping blanks and comments. A blank line becomes a one-field row and clutters the rejected list.

Destructuring more variables than the row has fields. The extras are undef, silently.

Using raw indexes throughout the loop. The reader has to count commas. Name the fields once.

Dying on the first bad row when you could report all of them. Collect and report unless one bad row genuinely invalidates the file.

8. Exercise

Exercise 1 — the four steps. Write a script that takes the single line "fifo_wr_02,FAIL,92,92\n" and produces a formatted report row, with all four steps from §1 present and the guard using != 4.

Exercise 2 — both silent failures. Split "alu,PASS" with -1, then read $field[3] and also destructure four variables from it. Print whether each result is defined and confirm STDERR is empty. Then say in one sentence why these two different-looking lines are the same bug.

Exercise 3 — the zero substitution. Take a short row's missing field and print it with printf "%d". Record what appears, capture STDERR, and state why this particular substitution is dangerous in a regression report.

Exercise 4 — build the fixture. Create a file containing a comment, a header, a blank line, two good rows, one short row and one row with an empty last field. Run §3's loop over it. Confirm the accepted and rejected counts, and that the rejected message names the line.

Exercise 5 — the limit matters. Run §3's loop again with the -1 removed from the split. Report which additional row is now rejected and why it should not have been.

Exercise 6 — name the fields. Take a six-field row and produce a report line using (a) full destructuring, (b) a slice of three fields, and (c) raw indexes. Say which you would commit for a loop that another engineer will maintain.

Exercise 7 — choose the guard. A tool's output has four columns today and the vendor's release notes say a fifth is coming. Write the guard you would use, and justify it in one sentence against the alternative.

9. Summary

• Four steps: chomp, split ... , -1, check the count, then name the fields.
• Two ways to walk off a short row — reading $field[n] past the end and destructuring more variables than there are values — and both are completely silent.
• undef fields interpolate as nothing and print as 0 under %d, so a short row produces a plausible report row with a fabricated zero.
• Split with -1 or a legitimately empty final column makes a good row look short.
• Skip blanks and comments before counting fields.
• Name the fields once with list assignment or a slice; raw indexes make the reader count commas.
• Report the rejected row's content and field count, so the failure is traceable to a line.
• Guard with != expected when the format is fixed, < minimum when it may grow. Having neither is the mistake.

Chapter 5.9 takes the rows this chapter accepted and puts them in order — including why the comparator variables are the one place a my declaration breaks the code.