@INC and Finding a Module

Part I · Module 12 · Chapter 12.4 · @INC and Finding a Module

1. Reading the Error

use strict;
use warnings;

# The error every Perl user meets. Provoked on purpose, and parsed rather than
# printed raw, because the raw form ends in a list of absolute machine paths.
my $ok = eval { require VLSI::NotInstalled; 1 };
if (!$ok) {
    my ($first) = split /\n/, $@;
    my ($msg, $inc) = $first =~ m{^(Can't locate \S+ in \@INC).*\(\@INC contains: ([^)]*)\)};
    printf "message            : %s\n", $msg;
    my @dirs = split ' ', $inc;
    printf "directories searched: %d\n", scalar @dirs;
    printf "the path it wanted : VLSI/NotInstalled.pm\n";
    printf "so :: became /     : yes\n";
}

message            : Can't locate VLSI/NotInstalled.pm in @INC
directories searched: 9
the path it wanted : VLSI/NotInstalled.pm
so :: became /     : yes

The message has three pieces of information in it, and two of them are usually ignored.

The file it wanted. VLSI::NotInstalled became VLSI/NotInstalled.pm. Perl translates a module name into a relative path by replacing each :: with a directory separator and appending .pm. So a module named A::B::C is the file A/B/C.pm, and the directory structure under your library root has to match the name exactly — including its capitalisation, on a case-sensitive filesystem.

Every directory it searched. The real message ends with (@INC contains: ...) and a space-separated list. That list is the answer to "where should I have put it", and reading it is usually faster than guessing.

That it was require, not something else. Chapter 12.3 established that this error from a use happens at compile time and cannot be caught; from a require it is an ordinary exception, which is why this script can catch it with eval and examine it.

The script parses the message rather than printing it, because the directory list is a set of absolute paths specific to one machine — printing it raw would make this page's output impossible to reproduce anywhere else. The count is the reproducible part.

2. %INC: Which File Actually Answered

The companion to @INC is %INC, and it answers a different question: not where Perl looked, but what it found.

use strict;
use warnings;
use lib 'lib';
use VLSI::Log ();

# %INC records WHICH FILE satisfied each module name.
printf "loaded VLSI::Log from : %s\n", $INC{"VLSI/Log.pm"};
printf "is it under lib/      : %s\n", ($INC{"VLSI/Log.pm"} =~ m{(^|/)lib/VLSI/Log\.pm$} ? "yes" : "no");

# Loading it again is a no-op
my $again = require VLSI::Log;
printf "require a second time : returned %s (already in %%INC)\n", $again;

loaded VLSI::Log from : lib/VLSI/Log.pm
is it under lib/      : yes
require a second time : returned 1 (already in %INC)

%INC is keyed by the relative path and holds the actual file that was loaded. That makes it the tool for the second-most-common module problem: not "it cannot be found" but "it found the wrong one". When a module behaves like an older version than the one you just edited, print $INC{"Your/Module.pm"} and you will usually find it loaded from a system directory rather than your lib.

The last line shows the other half of %INC's job. Perl records what it has loaded, so a second require of the same module does no work and simply returns true. That is why loading a module from twenty different files costs nothing after the first, and why a module's top-level code runs exactly once.

Note the use VLSI::Log () in that script. The empty parentheses mean "load it but import nothing" — useful when you only want the module's side effects or intend to use fully-qualified names, and the one form of use that behaves like require as far as names are concerned.

3. Adding a Directory: use lib

use strict;
use warnings;

# use lib runs at COMPILE time, so a run-time count cannot see "before".
# BEGIN blocks run at compile time, in order, so they can.
my ($before, $after);
BEGIN { $before = scalar @INC }
use lib 'lib';
BEGIN { $after  = scalar @INC }

printf "\@INC before use lib : %d entries\n", $before;
printf "\@INC after  use lib : %d entries\n", $after;
printf "the new first entry : %s\n", $INC[0];
print  "use lib PREPENDS, so your project wins over an installed copy\n";
print  "and it does it at COMPILE time, which is why a run-time count sees no change\n";

@INC before use lib : 9 entries
@INC after  use lib : 10 entries
the new first entry : lib
use lib PREPENDS, so your project wins over an installed copy
and it does it at COMPILE time, which is why a run-time count sees no change

One directory added, and added at the front — which matters, because it means a module in your project takes precedence over an installed copy of the same name. That is usually what you want while developing, and it is worth knowing when a change you made is unexpectedly taking effect.

The shape of the measurement needs explaining, and it is a direct application of the previous chapter. My first version of this script counted @INC at run time either side of the use lib line and printed 10 and 10 — identical, with no change to show. That was not a bug in use lib; it was the compile-time rule. use lib is a use, so it had already run before any ordinary statement executed, which means a run-time "before" count is taken after the fact. Two BEGIN blocks run at compile time in source order, so they can bracket it honestly.

The two other ways to add a directory each need their own process, so they are quoted rather than shown as blocks. Both also prepend:

• PERL5LIB=/path perl script.pl — an environment variable, which puts /path first in @INC. This is the right tool when the script should not know about the layout, and the standard way to point a whole shell session or a job submission at a project library.
• perl -I/path script.pl — a command-line flag, same effect, for a one-off run.

So there are three mechanisms and one rule: they all prepend, so the thing you add wins.

4. The Trap: a Relative Path Is Relative To Your Shell

This is the measurement worth the chapter.

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

# A project laid out as bin/report.pl plus lib/VLSI/Units.pm
my $proj = tempdir(CLEANUP => 1);
mkdir File::Spec->catdir($proj, $_) for qw(bin lib);
mkdir File::Spec->catdir($proj, "lib", "VLSI");

open my $pm, ">", File::Spec->catfile($proj, "lib", "VLSI", "Units.pm") or die $!;
print $pm "package VLSI::Units;\nsub tag { return 'loaded' }\n1;\n";
close $pm;

# The script says `use lib "lib"` -- a path relative to the CURRENT DIRECTORY
my $script = File::Spec->catfile($proj, "bin", "report.pl");
open my $sc, ">", $script or die $!;
print $sc qq{use lib "lib";\nuse VLSI::Units;\nprint "ok: ", VLSI::Units::tag(), "\\n";\n};
close $sc;

# Only the CURRENT DIRECTORY changes; the script path is absolute both times.
for my $where ($proj, File::Spec->catdir($proj, "bin")) {
    my $label = ($where eq $proj) ? "cwd = project root" : "cwd = bin/";
    my $out = `cd "$where" && perl "$script" 2>&1`;
    $out =~ s/ \(\@INC contains:.*//s;
    $out = (split /\n/, $out)[0] // "";
    printf "%-20s %s\n", $label, $out;
}
print "\nsame script, same absolute path -- only the current directory differed\n";
print "use lib 'lib' resolves against the CWD, not against the script\n";

cwd = project root   ok: loaded
cwd = bin/           Can't locate VLSI/Units.pm in @INC (you may need to install the VLSI::Units module)

same script, same absolute path -- only the current directory differed
use lib 'lib' resolves against the CWD, not against the script

The same script, invoked by the same absolute path, worked from one directory and failed from another.

use lib 'lib' does not mean "the lib next to this script". It means "the directory called lib, resolved from wherever the process happens to be standing". Run from the project root that is the right directory; run from bin/ it is bin/lib, which does not exist.

This is why a script works when you test it and fails under a scheduler, a Makefile recipe, or a colleague's shell alias — all of which may run it from a different directory. The error you get is Can't locate ... in @INC, which sends people looking for a missing file rather than a wrong working directory.

5. The Fix: Anchor the Path to the Script

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $proj = tempdir(CLEANUP => 1);
mkdir File::Spec->catdir($proj, $_) for qw(bin lib);
mkdir File::Spec->catdir($proj, "lib", "VLSI");
open my $pm, ">", File::Spec->catfile($proj, "lib", "VLSI", "Units.pm") or die $!;
print $pm "package VLSI::Units;\nsub tag { return 'loaded' }\n1;\n";
close $pm;

# FindBin gives the directory of the SCRIPT, so the path no longer depends on cwd
open my $sc, ">", File::Spec->catfile($proj, "bin", "report.pl") or die $!;
print $sc <<'SCRIPT';
use strict;
use warnings;
use FindBin;
use lib "$FindBin::Bin/../lib";
use VLSI::Units;
print "ok: ", VLSI::Units::tag(), "\n";
SCRIPT
close $sc;

for my $where ($proj, File::Spec->catdir($proj, "bin"), "/tmp") {
    my $script = File::Spec->catfile($proj, "bin", "report.pl");
    my $out = `cd "$where" && perl "$script" 2>&1`;
    $out =~ s/\s+\z//; $out = (split /\n/, $out)[0] // "";
    my $label = $where eq "/tmp" ? "from /tmp" : ($where eq $proj ? "from the project root" : "from bin/");
    printf "%-22s %s\n", $label, $out;
}
print "\nwith FindBin the library path is anchored to the script, so cwd stops mattering\n";

from the project root  ok: loaded
from bin/              ok: loaded
from /tmp              ok: loaded

with FindBin the library path is anchored to the script, so cwd stops mattering

Three different working directories, including one entirely outside the project, and all three worked.

FindBin is a core module that works out the directory of the running script and puts it in $FindBin::Bin. Building the library path from that makes it independent of where the process started:

use lib "$FindBin::Bin/../lib";

That is the line to put in every script in a project with its own lib. It is slightly ugly, it is one line, and it removes an entire class of failure.

Two details worth knowing. FindBin must be loaded before the use lib that references it, which is why the order in that script matters — and it works because both are compile-time, running in source order, exactly as Chapter 12.3's measurement showed. And the variable is interpolated into a string, so the path is computed when the use lib runs rather than being a literal.

6. What To Remember

• The error tells you three things: the file it wanted, every directory it searched, and (by whether it was catchable) which keyword asked for it.
• A module name becomes a path: A::B::C is A/B/C.pm, case-sensitive, under some directory in @INC.
• Read the (@INC contains: ...) list before guessing. It is the answer to "where should this have been".
• Never install into the system library directories. They belong to the Perl installation, and on a shared farm you cannot write there anyway.
• %INC tells you which file actually answered, which diagnoses "it found the wrong copy". A second require of the same module is a no-op.
• use VLSI::Log () loads without importing.
• Three ways to add a directory, all prepending: use lib, PERL5LIB, and perl -I. What you add wins over an installed copy.
• use lib runs at compile time, so a run-time count either side of it sees no change. Probe it with BEGIN.
• A relative use lib resolves against the working directory, not the script. Measured: the same absolute script path worked from the project root and failed from bin/.
• use lib "$FindBin::Bin/../lib"; is the fix, and it worked from all three directories including one outside the project.

Chapter 12.5 builds the module file itself: the package line, strict and warnings, the subroutines it offers, and the trailing true value that is easy to forget and produces an error that does not obviously point at it.