-n and -p: The Implicit Loop

Part I · Module 15 · Chapter 15.7 · -n and -p: The Implicit Loop

1. The Loop You Do Not Write

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
my $f   = File::Spec->catfile($dir, "slack.txt");
open my $w, ">", $f or die "cannot write: $!\n";
print $w "U1/Q 0.44 setup\nU2/D -0.03 hold\nU3/CK 0.12 setup\n";
close $w;

my @demo = (
    [ q{perl -ne 'print if /setup/'},        "-n: you decide what prints" ],
    [ q{perl -pe 's/setup/SETUP/'},          "-p: every line prints, altered" ],
    [ q{perl -ne '1'},                       "-n with no print: nothing" ],
    [ q{perl -ne 'print "$. $_"'},           "-n: \$. is the line number" ],
    [ q{perl -pe '$_ = uc $_ if /hold/'},    "-p: change \$_ to change output" ],
);

for my $d (@demo) {
    my ($cmd, $why) = @$d;
    my $out = `cd '$dir' && $cmd slack.txt 2>&1`;
    $out =~ s/\s+\z//;
    $out = "(no output at all)" if $out eq "";
    $out =~ s/\n/ | /g;
    printf "%-34s %s\n     -> %s\n", $cmd, "($why)", $out;
}

perl -ne 'print if /setup/'        (-n: you decide what prints)
     -> U1/Q 0.44 setup | U3/CK 0.12 setup
perl -pe 's/setup/SETUP/'          (-p: every line prints, altered)
     -> U1/Q 0.44 SETUP | U2/D -0.03 hold | U3/CK 0.12 SETUP
perl -ne '1'                       (-n with no print: nothing)
     -> (no output at all)
perl -ne 'print "$. $_"'           (-n: $. is the line number)
     -> 1 U1/Q 0.44 setup | 2 U2/D -0.03 hold | 3 U3/CK 0.12 setup
perl -pe '$_ = uc $_ if /hold/'    (-p: change $_ to change output)
     -> U1/Q 0.44 setup | U2/D -0.03 HOLD | U3/CK 0.12 setup

The difference in one sentence: -n gives you the loop, -p gives you the loop and a print.

• -ne 'print if /setup/' printed two of three lines. The filter is yours.
• -pe 's/setup/SETUP/' printed all three, with the substitution applied. Nothing was filtered out, because -p prints every line whether you changed it or not.
• -ne '1' printed nothing. The loop ran three times and did nothing each time.

And $_ is the line. Everything from Module 4 about the default variable applies: /setup/ matches against $_, s/// substitutes in $_, print with no argument prints $_.

2. What They Actually Expand To

You do not have to believe any of that. -MO=Deparse prints the program Perl compiled, including the parts it added.

use strict;
use warnings;

for my $flag (qw(-n -p)) {
    my $out = `perl -MO=Deparse $flag -e 's/old/new/' 2>&1`;
    $out =~ s/^-e syntax OK\n//m;
    $out =~ s/\s+\z//;
    print "perl $flag -e 's/old/new/'  expands to:\n";
    print "    $_\n" for split /\n/, $out;
    print "\n";
}

perl -n -e 's/old/new/'  expands to:
    LINE: while (defined($_ = readline ARGV)) {
        s/old/new/;
    }

perl -p -e 's/old/new/'  expands to:
    LINE: while (defined($_ = readline ARGV)) {
        s/old/new/;
    }
    continue {
        die "-p destination: $!\n" unless print $_;
    }

That is Perl's own account of itself, and three things in it are worth reading carefully.

while (defined($_ = readline ARGV)) is the loop. ARGV is the magic handle from Chapter 15.1 that reads every file named in @ARGV, or standard input when there are none. The defined test is what makes a final line of 0 with no newline work correctly.

LINE: is a real label. So last LINE and next LINE work inside your program text, which is how a one-liner stops early:

perl -ne 'last LINE if $. > 100; print' big.rpt     # first 100 lines

-p's print is in a continue block, not at the end of the body. That is not a detail — it changes behaviour:

perl -pe 'next if /^#/' file      # does NOT drop comment lines

next skips the rest of the body and goes to continue, which prints. So the line is printed anyway. To actually drop lines you need -n and an explicit print, or you set $_ = "":

perl -ne 'print unless /^#/' file     # correct: drops comments

3. Choosing Between Them

A short rule: if you are filtering, use -n. If you are transforming, use -p.

Two notes on that table.

-p transforms by changing $_. Either a substitution, which edits $_ in place, or an outright assignment as in the numbering row. If you print something yourself inside -p, it is printed in addition to the line, which is a common way to get doubled output.

-n with an END block is the counting idiom. The END runs once after the loop, exactly as Chapter 15.4 measured, so it is where a total belongs:

perl -lane '$sum += $F[1]; $n++; END { printf "%d rows, mean %.3f\n", $n, $sum/$n }' slack.txt

BEGIN works the same way at the other end, which is where you set things up:

perl -ne 'BEGIN { $limit = 0.1 } print if (split)[1] < $limit' slack.txt

4. The Variables That Come With The Loop

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
for my $pair (["a.txt", "alpha\nbravo\n"], ["b.txt", "charlie\n"]) {
    open my $w, ">", File::Spec->catfile($dir, $pair->[0]) or die "cannot write: $!\n";
    print $w $pair->[1];
    close $w;
}

my @demo = (
    [ q{perl -ne 'print "$ARGV:$.: $_"'},                 "\$ARGV is the file, \$. the line number" ],
    [ q{perl -ne 'print "$.\n" if eof'},                  "eof is true on the last line of EACH file" ],
    [ q{perl -ne 'print "$.\n" if eof()'},                "eof() with parens: the very last line of ALL" ],
    [ q{perl -ne 'print "$ARGV:$.\n"'},                   "no close: \$. runs on across files" ],
    [ q{perl -ne 'print "$ARGV:$.\n"; close ARGV if eof'}, "close AFTER print: per-file numbers" ],
    [ q{perl -ne 'close ARGV if eof; print "$ARGV:$.\n"'}, "close BEFORE print: the trap" ],
);

for my $d (@demo) {
    my ($cmd, $why) = @$d;
    my $out = `cd '$dir' && $cmd a.txt b.txt 2>&1`;
    $out =~ s/\s+\z//;
    $out =~ s/\n/ | /g;
    printf "%s\n   %s\n   -> %s\n\n", $cmd, "($why)", $out;
}

perl -ne 'print "$ARGV:$.: $_"'
   ($ARGV is the file, $. the line number)
   -> a.txt:1: alpha | a.txt:2: bravo | b.txt:3: charlie

perl -ne 'print "$.\n" if eof'
   (eof is true on the last line of EACH file)
   -> 2 | 3

perl -ne 'print "$.\n" if eof()'
   (eof() with parens: the very last line of ALL)
   -> 3

perl -ne 'print "$ARGV:$.\n"'
   (no close: $. runs on across files)
   -> a.txt:1 | a.txt:2 | b.txt:3

perl -ne 'print "$ARGV:$.\n"; close ARGV if eof'
   (close AFTER print: per-file numbers)
   -> a.txt:1 | a.txt:2 | b.txt:1

perl -ne 'close ARGV if eof; print "$ARGV:$.\n"'
   (close BEFORE print: the trap)
   -> a.txt:1 | a.txt:0 | b.txt:0

Four variables and one function, measured together:

$_ is the current line, including its newline unless you used -l.

$. is the line number, and it does not reset between files by default — the second file continues the count. That surprises people reading several reports at once.

$ARGV is the current filename, which is what you put in a message about a bad line.

eof without parentheses is true at the end of each file; eof() with parentheses is true only at the end of the last one. That is a genuine difference and the measured rows show it: one prints a number per file, the other prints one number.

close ARGV if eof; resets $. for the next file — but the order matters, and getting it wrong is silent. The last three rows measure all three possibilities:

• No close gives 1, 2, 3 — the count runs straight on into the second file.
• close after the print gives 1, 2, 1 — correct per-file numbering.
• close before the print gives 1, 0, 0 — because on the last line of each file the reset happens first, so the line that is printed reports zero.

The middle form is the idiom. The third is the one to watch for: it produces plausible small numbers and no error.

5. What To Remember

• -n wraps your code in a loop over every input line. You decide what to print.
• -p is the same loop plus a print, so every line comes out whether you changed it or not.
• Measured: -ne '1' printed nothing — the loop ran and did nothing.
• $_ is the line, so /match/, s/// and bare print all work on it.
• -MO=Deparse prints the loop Perl generated, so none of this has to be taken on trust.
• The loop is while (defined($_ = readline ARGV)), reading every file in @ARGV or standard input.
• LINE: is a real label — last LINE if $. > 100 stops early.
• -p prints from a continue block, so next still prints the line. Use -n with an explicit print to drop lines.
• -p also checks that its print succeeded and dies if not.
• Filter with -n, transform with -p.
• -p transforms by changing $_; printing inside -p adds output rather than replacing it.
• -n with END { } is the counting idiom, and BEGIN { } is where setup goes.
• $. is the line number and does NOT reset between files — measured 1, 2, 3 across two files.
• close ARGV if eof; resets it, but only if it comes AFTER the print — measured 1, 2, 1 after, against 1, 0, 0 before.
• eof is per-file; eof() is the very end of all input. Measured, and the parentheses are the whole difference.

Chapter 15.8 adds the flag that makes these loops genuinely competitive with awk. -a splits each line into fields for you, and -F chooses what counts as a separator.