Encoding, Binary Mode and Very Large Files

Part I · Module 13 · Chapter 13.12 · Encoding, Binary Mode and Very Large Files

1. Bytes or Characters

use strict;
use warnings;
use File::Temp qw(tempdir);

my $dir = tempdir(CLEANUP => 1);
my $file = "$dir/u.txt";

# WRITE with an explicit encoding layer.
open my $w, ">:encoding(UTF-8)", $file or die $!;
print $w "ohm \x{03A9} and micro \x{00B5}\n";
close $w;

printf "bytes on disk      : %d\n", -s $file;

# READ WITHOUT the layer: bytes, not characters.
open my $raw, "<", $file or die $!;
my $as_bytes = <$raw>; close $raw;
chomp $as_bytes;
printf "read as bytes      : length %d\n", length $as_bytes;

# READ WITH the layer: characters.
open my $chr, "<:encoding(UTF-8)", $file or die $!;
my $as_chars = <$chr>; close $chr;
chomp $as_chars;
printf "read as characters : length %d\n", length $as_chars;
printf "they differ        : %s\n", (length($as_bytes) == length($as_chars) ? "no" : "YES");

bytes on disk      : 20
read as bytes      : length 19
read as characters : length 17
they differ        : YES

The same file, read two ways, gives two different lengths — 19 and 17.

A file on disk is bytes. Nothing in the file says what those bytes mean. The mode string is where you tell Perl:

open my $fh, "<:encoding(UTF-8)", $file or die ...;

With that layer, Perl decodes the bytes into characters as it reads, so length counts characters and a pattern like /^\w+$/ works on the real letters. Without it, the Greek omega arrives as two separate bytes, which is why the byte count is higher.

The practical consequences of getting this wrong, in the order you will meet them:

• length and substr give wrong answers. A fixed-width column read with substr splits a multi-byte character in half.
• A pattern can match the middle of a character, producing text that is no longer valid.
• Sorting and case-folding misbehave on anything outside plain English.
• Printing produces the Wide character warning if you have decoded characters and send them to a handle with no encoding layer — the fix is a layer on the output handle too.

For the files most of this industry produces — tool logs, netlists, constraint files — plain ASCII is overwhelmingly the norm and none of this matters. It starts mattering as soon as a human types into one: a comment, an author name, an instance name copied from a document. Being explicit costs nine characters and removes the question.

2. When Any Interpretation Is Wrong

Some files are not text. A compiled database, an image, a tool's internal checkpoint. For those you want Perl to leave the bytes completely alone:

open my $fh, "<:raw", $file or die ...;

:raw means no decoding and no line-ending translation. The older spelling is binmode $fh; after opening, which does the same thing and is what you will see in existing code. Both are worth recognising.

The reason this matters is line-ending translation. On some platforms a text-mode read turns a carriage-return-newline pair into a single newline, which for text is helpful and for binary data is corruption — it changes the bytes, and the file you write back out is not the file you read. :raw turns that off.

A practical note for this field, since it is the most common real encounter with binary data: reading a file to compute a checksum, or copying one, must use :raw on both handles. Anything else risks a file that is subtly different from its source, and the difference will not show up until something else refuses to parse it.

3. When a Line Is Not a Useful Unit

use strict;
use warnings;
use File::Temp qw(tempdir);

my $dir = tempdir(CLEANUP => 1);
my $file = "$dir/blob.bin";

# Write 10,000 bytes with no newlines at all.
open my $w, ">:raw", $file or die $!;
print $w "A" x 10_000;
close $w;

printf "file size      : %d bytes\n", -s $file;

# A line-based read would get ONE enormous "line".
open my $l, "<:raw", $file or die $!;
my $one = <$l>; close $l;
printf "one readline   : %d bytes in a single scalar\n", length $one;

# read() takes a fixed number of bytes at a time.
open my $r, "<:raw", $file or die $!;
my ($chunks, $total, $buf) = (0, 0, "");
while (my $got = read($r, $buf, 4096)) {
    $chunks++; $total += $got;
}
close $r;
printf "read() in 4096 : %d chunk(s), %d bytes total\n", $chunks, $total;

file size      : 10000 bytes
one readline   : 10000 bytes in a single scalar
read() in 4096 : 3 chunk(s), 10000 bytes total

A 10,000-byte file with no newlines. One readline returned all 10,000 bytes in a single scalar.

This is Chapter 13.3's memory problem arriving through a loop that looked safe. The read loop is supposed to bound memory to one line — and it does exactly that, which is useless when the file is one line. The protection you thought you had depends on the data.

read removes the dependency. It takes a handle, a buffer and a count, and gives you at most that many bytes:

while (my $got = read($fh, $buf, 4096)) { ... }

Three chunks for 10,000 bytes — 4096, 4096, and 1808 — and the loop ends when read returns 0. Memory is bounded by the number you chose, whatever the file looks like.

Two related tools for large files, both worth knowing by name:

seek and tell. tell reports your current position in bytes and seek moves it. Together they let you read the end of a huge log without reading the start — seek $fh, -4096, 2 positions 4096 bytes before the end, which is how you implement something like tail cheaply.

Setting $/ to a record separator, which Chapter 13.3 mentioned. If a large file has repeating multi-line records, local $/ = "\n\n"; gives you one record per read — bounded memory and multi-line matching, which is usually the best of both.

And the honest default, as Chapter 13.3 argued: for text in lines, which is nearly everything here, the plain read loop is correct and simplest. Reach for this chapter's tools when you have measured that you need them.

4. What To Remember

• A file is bytes. The mode string is where you say what they mean: "<:encoding(UTF-8)".
• Bytes and characters give different lengths. Measured: 19 against 17 for the same file.
• Without a layer, a multi-byte character arrives as separate bytes, so length, substr and patterns all misbehave.
• Put a layer on the output handle too, or you get Wide character warnings.
• use open qw(:std :encoding(UTF-8)); is a reasonable one-line default for a new script.
• :raw (or binmode) for data that is not text — no decoding and no line-ending translation, which would otherwise corrupt it.
• Use :raw on both handles when copying or checksumming.
• The read loop bounds memory to one line, which is useless if the file is one line. Measured: 10,000 bytes in a single scalar.
• read($fh, $buf, 4096) bounds it by a number you choose. Measured: 3 chunks for 10,000 bytes.
• read may return fewer bytes than asked — use the returned count. It returns 0 at EOF and undef on error.
• seek and tell let you read the end of a huge file without the start.
• For text in lines, the plain loop is still the right default.

Chapter 13.13 covers the two things a script that writes files should almost always do: use a temporary file with a name nothing else can collide with, and put it in place in a way that never leaves a half-written file where a reader can see it.