Reading a Whole File at Once, and When Not To

Part I · Module 13 · Chapter 13.3 · Reading a Whole File at Once, and When Not To

1. How To Slurp

use strict;
use warnings;
use File::Temp qw(tempdir);

my $dir = tempdir(CLEANUP => 1);
open my $w, ">", "$dir/r.txt" or die $!;
print $w "BEGIN\nu_alu/A -0.12\nu_fifo/B 0.44\nEND\n";
close $w;

# Slurp: read the whole file into ONE string.
my $whole = do {
    open my $fh, "<", "$dir/r.txt" or die $!;
    local $/;                 # undef -- no line separator
    <$fh>;
};

printf "characters      : %d\n", length $whole;
printf "newlines inside : %d\n", ($whole =~ tr/\n//);
printf "first 5 chars   : [%s]\n", substr($whole, 0, 5);
printf "last 4 chars    : [%s]\n", ($whole =~ s/\n/\\n/gr) =~ /(...n)$/ ? $1 : "?";

# Compare with the same file read as a LIST of lines.
open my $fh2, "<", "$dir/r.txt" or die $!;
my @lines = <$fh2>;
close $fh2;
printf "\nas one string   : 1 scalar, %d characters\n", length $whole;
printf "as a list       : %d elements\n", scalar @lines;

characters      : 38
newlines inside : 4
first 5 chars   : [BEGIN]
last 4 chars    : [ND\n]

as one string   : 1 scalar, 38 characters
as a list       : 4 elements

Four lines of text became one string of 38 characters, with the four newlines still inside it. Read as a list instead, the same file is four elements.

The slurp itself is three lines, and the middle one is the trick:

local $/;

$/ is the input record separator — the thing <$fh> uses to decide where a line ends. Setting it to undef means "there is no separator", so a single read takes everything. Chapter 12.8 covered local and why it is the right tool here: it puts $/ back when the block ends, so no other read in your program is affected.

The do { ... } block is just a way to keep the local and the filehandle together in one small scope. The value of the block is its last expression, which is the read.

2. When Slurping Is the Only Way

Some patterns cannot match one line at a time, because the thing you are looking for is spread over several.

use strict;
use warnings;
use File::Temp qw(tempdir);

my $dir = tempdir(CLEANUP => 1);
open my $w, ">", "$dir/r.txt" or die $!;
print $w "BEGIN\nu_alu/A -0.12\nu_fifo/B 0.44\nEND\n";
close $w;

# A pattern that must span SEVERAL lines cannot work line by line.
my $whole = do { open my $fh, "<", "$dir/r.txt" or die $!; local $/; <$fh> };

my ($body) = $whole =~ /^BEGIN\n(.*?)^END\n/ms;
printf "block body found : %s\n", (defined $body ? "yes" : "no");
printf "body lines       : %d\n", ($body =~ tr/\n//);

# Line by line, no single line ever contains both markers.
open my $fh2, "<", "$dir/r.txt" or die $!;
my $hits = 0;
while (my $l = <$fh2>) { $hits++ if $l =~ /^BEGIN\n(.*?)^END\n/ms }
close $fh2;
printf "same pattern per line matched : %d time(s)\n", $hits;

block body found : yes
body lines       : 2
same pattern per line matched : 0 time(s)

The same pattern found the block once in the slurped string and zero times reading line by line.

That is not a flaw in the loop. It is arithmetic: the pattern needs BEGIN and END in the same string, and the loop never puts them there. No amount of modifier-tweaking fixes it, because the data the pattern needs is not present.

So the clear cases for slurping are:

• A pattern that spans lines — pulling the body out of a BEGIN/END block, a stanza from a configuration file, a multi-line error report.
• A whole-file substitution — one s///g over the entire text rather than per line.
• A file you need twice — slurp once, then run several patterns over the string, instead of reading the file repeatedly.
• A small file where it is simply clearer. A 40-line configuration file is not a memory problem.

The pattern in that script uses /m so ^ anchors at each line and /s so .*? can cross newlines, with the lazy quantifier so it stops at the first END. All three come from Chapter 11.16, and this is the situation those modifiers exist for.

3. When Slurping Is the Problem

use strict;
use warnings;
use File::Temp qw(tempdir);

my $dir = tempdir(CLEANUP => 1);
my $file = "$dir/big.txt";

# 200,000 lines of about 25 characters each.
open my $w, ">", $file or die $!;
print $w sprintf("u_core/u_alu/bit%06d 0.42 ns\n", $_) for 1 .. 200_000;
close $w;

my $size = -s $file;
printf "file size on disk        : %d bytes\n", $size;

# SLURP: the whole file is in memory at once.
my $whole = do { open my $fh, "<", $file or die $!; local $/; <$fh> };
printf "slurped string length    : %d characters\n", length $whole;
printf "held in memory at once   : the whole %d bytes\n", length $whole;
undef $whole;

# LINE BY LINE: only the longest single line is ever held.
open my $fh, "<", $file or die $!;
my ($lines, $longest) = (0, 0);
while (my $l = <$fh>) {
    $lines++;
    $longest = length $l if length $l > $longest;
}
close $fh;
printf "\nlines read               : %d\n", $lines;
printf "longest single line      : %d characters\n", $longest;
printf "held in memory at once   : one line, %d bytes\n", $longest;
printf "ratio                    : %d x less\n", int($size / $longest);

file size on disk        : 6200000 bytes
slurped string length    : 6200000 characters
held in memory at once   : the whole 6200000 bytes

lines read               : 200000
longest single line      : 31 characters
held in memory at once   : one line, 31 bytes
ratio                    : 200000 x less

A 6.2 megabyte file. Slurping it holds 6,200,000 bytes. The read loop never holds more than 31 bytes — the longest single line.

Both approaches read all 200,000 lines and would compute the same answer. The difference is entirely in what is resident at once.

A practical middle path when a file is large and you need multi-line matching: read in chunks that respect record boundaries. Set $/ to the record separator rather than undef — if a report separates path records with a blank line, local $/ = "\n\n"; makes <$fh> return one whole record per read. You get multi-line matching inside a record while holding only one record in memory. That works whenever the file has a repeating structure, which most generated reports do.

4. The Middle Path: Read One Record at a Time

There is a third option between one line and the whole file, and it is the right answer more often than either.

use strict;
use warnings;
use File::Temp qw(tempdir);

my $dir = tempdir(CLEANUP => 1);
# A report of multi-line records separated by a BLANK line.
open my $w, ">", "$dir/paths.rpt" or die $!;
print $w "Startpoint: u_alu/A\nEndpoint: u_regf/D\nslack -0.12\n\n";
print $w "Startpoint: u_fifo/B\nEndpoint: u_mem/Q\nslack 0.44\n\n";
print $w "Startpoint: u_ctl/C\nEndpoint: u_regf/E\nslack -0.95\n\n";
close $w;

# Set $/ to the record separator: one READ gives one whole record.
my @worst;
{
    open my $fh, "<", "$dir/paths.rpt" or die $!;
    local $/ = "\n\n";
    while (my $record = <$fh>) {
        my ($start) = $record =~ /^Startpoint: (\S+)/m;
        my ($slack) = $record =~ /^slack\s+(-?[\d.]+)/m;
        next unless defined $slack;
        push @worst, [$start, $slack];
        printf "record %d : %-10s slack %s  (%d lines)\n",
            scalar @worst, $start, $slack, ($record =~ tr/\n//);
    }
    close $fh;
}

my ($w2) = sort { $a->[1] <=> $b->[1] } @worst;
printf "\nrecords read      : %d\n", scalar @worst;
printf "worst path        : %s at %s\n", $w2->[0], $w2->[1];
printf "memory held       : one record, not the whole file\n";

record 1 : u_alu/A    slack -0.12  (4 lines)
record 2 : u_fifo/B   slack 0.44  (4 lines)
record 3 : u_ctl/C    slack -0.95  (4 lines)

records read      : 3
worst path        : u_ctl/C at -0.95
memory held       : one record, not the whole file

Three records, four lines each, and each one arrived in a single read.

The trick is the same $/ from section one, set to the record separator rather than to undef:

local $/ = "\n\n";

A blank line is two newlines in a row, so that makes <$fh> return everything up to the next blank line. The variable is called the input record separator for exactly this reason — a "line" is just the default record.

What you get is both things at once:

• Multi-line patterns work, because the record you are holding contains all its lines. The two patterns in that loop use /m so ^ anchors inside the record, which is Chapter 11.16's modifier.
• Memory stays bounded, because you hold one record rather than the whole file. A report with ten thousand paths costs the same as one with ten.

This is the shape to reach for whenever a large file has a repeating structure, which generated reports almost always do. Timing reports separate paths with a blank line; log files often separate events with a marker line; netlists separate instances with a terminator. In every one of those cases the record is the natural unit, and reading it directly is both clearer and cheaper than slurping.

Two practical notes. $/ can be any string, so local $/ = "ENDPATH\n"; works if the format uses a marker instead of a blank line. And the separator stays attached to the record you get back, just as a newline stays attached to a line — so chomp still applies, and a pattern anchored with \z needs the same care Chapter 11.8 described.

5. What To Remember

• Slurping is local $/; then one read. Keep it in a small block so $/ is restored.
• my $whole = <$fh>; without local $/ reads only the FIRST line, silently.
• Slurp when a pattern must span lines. Measured: the BEGIN/END pattern matched once on the slurped string and zero times line by line.
• Also slurp for a whole-file substitution, for a file you need to scan several times, or when the file is small and it is clearer.
• Slurping costs memory equal to the file. Measured: 6,200,000 bytes held, against 31 bytes for the read loop — a ratio of 200,000 to 1 on this file.
• On a compute node that means being killed, not an error message. Make line by line your default.
• For a large file with multi-line records, set $/ to the record separator instead of undef — multi-line matching, one record in memory.

Chapter 13.4 turns to the other direction: writing. It covers print and printf to a filehandle, the difference three runs of > and >> make to a log file, and why close is the call that decides whether your data reached the disk.