Reading a File Line by Line

Part I · Module 13 · Chapter 13.2 · Reading a File Line by Line

1. The Loop

use strict;
use warnings;
use File::Temp qw(tempdir);

my $dir = tempdir(CLEANUP => 1);
open my $w, ">", "$dir/r.txt" or die $!;
print $w "u_alu/A -0.12\nu_fifo/B 0.44\nu_regf/C -0.05\n";
close $w;

open my $fh, "<", "$dir/r.txt" or die $!;
while (my $line = <$fh>) {
    chomp $line;
    printf "[%s]\n", $line;
}
close $fh;

[u_alu/A -0.12]
[u_fifo/B 0.44]
[u_regf/C -0.05]

Three lines in the file, three lines out. The loop is the part to memorise:

while (my $line = <$fh>) { chomp $line; ... }

<$fh> is the diamond operator. Used like this it reads one line and returns it, including the newline. At the end of the file it returns undef, which ends the loop.

The important thing about this shape is what it does not do: it never holds more than one line in memory. That is why it works on a file of any size, and it is the default you should reach for. Chapter 13.3 covers the alternative and when it is worth it.

2. Why chomp

use strict;
use warnings;
use File::Temp qw(tempdir);

my $dir = tempdir(CLEANUP => 1);
open my $w, ">", "$dir/r.txt" or die $!;
print $w "ns\nps\n";
close $w;

open my $fh, "<", "$dir/r.txt" or die $!;
while (my $line = <$fh>) {
    my $clean = $line;
    chomp $clean;
    printf "raw [%s] length %d  ->  clean [%s] length %d  |  raw ends in newline : %s\n",
        ($line =~ s/\n/\\n/r), length($line),
        $clean, length($clean),
        ($line =~ /\n\z/ ? "yes" : "no");
}
close $fh;
print "\nthe newline is part of the line until you chomp it\n";

raw [ns\n] length 3  ->  clean [ns] length 2  |  raw ends in newline : yes
raw [ps\n] length 3  ->  clean [ps] length 2  |  raw ends in newline : yes

the newline is part of the line until you chomp it

A two-character unit name arrives as a three-character string. The third character is the newline.

That matters because of what you do next. $line eq "ns" is false. $line + 0 happens to work, because Perl ignores trailing whitespace when making a number. A pattern anchored with $ still matches, because Chapter 11.8 measured that $ allows one trailing newline. So the newline breaks some tests and not others, which is exactly what makes it hard to spot.

chomp removes one trailing newline if there is one, and does nothing if there is not. It changes the variable in place and returns the number of characters removed, not the string — so my $clean = chomp $line; is a mistake that puts 1 in $clean.

Note the s/\n/\\n/r in that script: the /r modifier from Chapter 11.14, used to show the newline without changing the original.

3. All the Lines at Once

The diamond operator behaves differently depending on what you ask for.

use strict;
use warnings;
use File::Temp qw(tempdir);

my $dir = tempdir(CLEANUP => 1);
open my $w, ">", "$dir/r.txt" or die $!;
print $w "a\nb\nc\n";
close $w;

# SCALAR context: one line
open my $f1, "<", "$dir/r.txt" or die $!;
my $one = <$f1>;
close $f1;
chomp $one;
printf "scalar context : [%s]\n", $one;

# LIST context: every line, newlines still attached
open my $f2, "<", "$dir/r.txt" or die $!;
my @all = <$f2>;
close $f2;
printf "list context   : %d element(s)\n", scalar @all;
printf "element 0 raw  : [%s]\n", ($all[0] =~ s/\n/\\n/r);
chomp @all;
printf "chomp works on the whole array : %s\n", join(",", @all);

scalar context : [a]
list context   : 3 element(s)
element 0 raw  : [a\n]
chomp works on the whole array : a,b,c

Assigned to a scalar, <$fh> gives one line. Assigned to an array, it gives all of them — one element per line, each still carrying its newline.

This is Module 3's context rule applied to file reading, and the useful detail is on the last line: chomp @all chomps every element of the array in one call. You do not need a loop for it.

Reading all the lines is convenient and it costs memory proportional to the file. For a 200-line constraint file that is irrelevant. For a multi-gigabyte simulation log it is the difference between a script that runs and one that is killed. Chapter 13.3 measures that trade.

4. The Line That Breaks a Hand-Written Loop

Now the advanced point, and the reason the loop in section one is written exactly that way.

use strict;
use warnings;
use File::Temp qw(tempdir);

my $dir = tempdir(CLEANUP => 1);
# A file whose LAST line is just "0" with no trailing newline.
open my $w, ">", "$dir/z.txt" or die $!;
print $w "first\n0";
close $w;

# while (my $line = <$fh>) -- Perl adds defined() for you here.
my @got;
open my $fh, "<", "$dir/z.txt" or die $!;
while (my $line = <$fh>) { chomp $line; push @got, $line }
close $fh;
printf "while (my \$line = <\$fh>) read %d line(s): %s\n", scalar @got, join(",", @got);

# Written WITHOUT the implicit defined, the "0" line stops the loop early.
my @got2;
open my $fh2, "<", "$dir/z.txt" or die $!;
while (1) {
    my $line = <$fh2>;
    last if !$line;              # a plain truth test -- WRONG
    chomp $line; push @got2, $line;
}
close $fh2;
printf "a plain truth test read   %d line(s): %s\n", scalar @got2, join(",", @got2);
print "\nthe bare '0' line is FALSE, so the hand-written truth test stopped early\n";

while (my $line = <$fh>) read 2 line(s): first,0
a plain truth test read   1 line(s): first

the bare '0' line is FALSE, so the hand-written truth test stopped early

Same file, two loops, different numbers of lines. The idiomatic loop read both. The hand-written one read one and stopped.

The reason is Chapter 2's truth table. The file's last line is the single character 0, with no newline after it. As a string that is "0", and "0" is one of the few false values in Perl. So last if !$line fired on a perfectly good line of data.

The idiomatic loop does not have this problem because Perl rewrites it for you. When the condition of a while is exactly an assignment from a readline, Perl wraps it in defined() — so the real test is "did I get a line", not "is the line true". That is a special case in the language, added precisely because this bug was so common.

One more detail from that measurement worth keeping: the file's last line had no trailing newline, and the loop still returned it. The diamond operator gives you whatever is left, newline or not — which is why chomp is written to remove a newline if there is one rather than to remove the last character.

5. What To Remember

• The loop is while (my $line = <$fh>) { chomp $line; ... }. Write it that way.
• It holds one line at a time, so memory does not grow with the file.
• <$fh> returns the line including its newline, and undef at end of file.
• Every line carries a newline until you chomp it. Measured: a two-character name arrives as a three-character string.
• chomp edits in place and returns a count, not the string. my $x = chomp $line; puts 1 in $x.
• chomp @array chomps every element in one call.
• In list context <$fh> returns all the lines, one per element, newlines attached — convenient, and it costs memory proportional to the file.
• Perl adds defined() to while (my $x = <$fh>) for you. Measured: without it, a final line of just 0 ends the loop early and a line of data is lost.
• Outside that exact shape, write defined yourself.
• A last line with no trailing newline is still returned, which is why chomp removes a newline only if one is there.

Chapter 13.3 looks at the other way to read a file — all of it into one string — and measures when that is the right choice and when it is the reason a script gets killed on a compute node.