CPAN, cpanm and the Site-Library Problem

Part I · Module 12 · Chapter 12.10 · CPAN, cpanm and the Site-Library Problem

1. Where a Plain Install Would Write

use strict;
use warnings;
use Config;

# The directories a system-wide install would write to.
for my $k (qw(installsitelib installsitearch installprivlib)) {
    printf "%-16s %s\n", $k, $Config{$k};
}

# Can this user actually write there?
printf "\nsitelib writable by me : %s\n", (-w $Config{installsitelib} ? "yes" : "NO");
printf "running as root        : %s\n", ($> == 0 ? "yes" : "no");

installsitelib   /Library/Perl/5.34
installsitearch  /Library/Perl/5.34/darwin-thread-multi-2level
installprivlib   /Library/Perl/Updates/5.34.1

sitelib writable by me : NO
running as root        : no

The last two lines are the chapter. The directory this Perl would install a module into is not writable by the user running it, and that user is not root.

That is not a quirk of this machine; it is the normal and correct arrangement, and it is exactly the situation on a shared compute farm. The Perl installation belongs to whoever administers the machine. It is shared by every user and every project, so a module you add to it is added for all of them, at whatever version you happened to install, and a module you upgrade in it may break somebody's working flow with no warning and no obvious cause.

The Config hash is worth knowing for its own sake: it holds the settings this Perl was built with, and perl -V:installsitelib prints any one of them from the command line without writing a script.

2. The Alternative: a Library That Belongs to the Project

The fix is to install into a directory you own and point Perl at it — which is the mechanism Chapter 12.4 already established, now used for third-party code rather than your own.

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Path qw(make_path);
use File::Spec;

# This is the LAYOUT that `cpanm -l extlib Some::Module` produces.
# Nothing is installed here: the tree is built by hand so the mechanism is
# visible and the page stays reproducible offline.
my $proj = tempdir(CLEANUP => 1);
my $libdir = File::Spec->catdir($proj, "extlib", "lib", "perl5");
make_path(File::Spec->catdir($libdir, "Vendor"));

open my $fh, ">", File::Spec->catfile($libdir, "Vendor", "Thing.pm") or die $!;
print $fh "package Vendor::Thing;\nsub version { '1.00' }\n1;\n";
close $fh;

printf "layout      : extlib/lib/perl5/Vendor/Thing.pm\n";

# A script finds it the same way 12.4 taught: PERL5LIB, or use lib.
my $script = File::Spec->catfile($proj, "report.pl");
open my $s, ">", $script or die $!;
print $s qq{use Vendor::Thing;\nprint "loaded version ", Vendor::Thing::version(), "\\n";\n};
close $s;

my $without = `perl "$script" 2>&1`;
$without =~ s/ \(\@INC contains:.*//s; $without = (split /\n/, $without)[0] // "";
printf "without PERL5LIB : %s\n", $without;

my $with = `PERL5LIB="$libdir" perl "$script" 2>&1`;
$with =~ s/\s+\z//;
printf "with PERL5LIB    : %s\n", $with;

layout      : extlib/lib/perl5/Vendor/Thing.pm
without PERL5LIB : Can't locate Vendor/Thing.pm in @INC (you may need to install the Vendor::Thing module)
with PERL5LIB    : loaded version 1.00

The same script, the same module on disk, and the only difference is one environment variable: without it, the familiar Can't locate; with it, the module loads.

Be clear about what that script did and did not do. It installed nothing. It created the directory tree by hand, in the exact shape an installer produces, because installing a module is outside what this page is allowed to do and would also make the page depend on a network and a particular module version. What is measured is the mechanism — that a module in a project-local tree is found through PERL5LIB — and that is the part you need.

The layout matters and is worth recognising: extlib/lib/perl5/ with the module tree beneath it. That is what cpanm -l extlib produces, and the lib/perl5 nesting is why PERL5LIB must point at extlib/lib/perl5 rather than at extlib.

3. The Commands

With the constraint and the mechanism established, the commands are short. None of them were run to build this page.

cpanm -l extlib Some::Module installs into ./extlib, creating the layout above. cpanm — App::cpanminus — is the modern client and is the one to ask for if it is missing; on the machine this page was built on, cpanm is not installed while the older cpan is. That is worth checking before planning around it.

cpanm -L extlib Some::Module is the capital-L variant, which installs the module and its dependencies into extlib while ignoring anything already installed system-wide. Use it when you want the local tree to be self-contained.

PERL5LIB=$PWD/extlib/lib/perl5 perl bin/report.pl runs a script against it. In practice you set PERL5LIB once in the job submission or the wrapper script rather than on every invocation.

use local::lib '--self-contained', 'extlib'; is the module-based alternative to setting the variable, and eval 'use local::lib ...' lets a script configure its own path. It is one more dependency to install, which is a slightly awkward position for a module whose job is installing dependencies.

cpanm --installdeps . reads a cpanfile in the current directory and installs what it lists. If a project is going to depend on third-party modules at all, a cpanfile listing them by name and minimum version is the thing that makes the dependency reproducible — otherwise the knowledge lives in whoever set the machine up.

4. Before You Install Anything

This chapter is rated Medium rather than High, and the reason is that the correct answer is usually not to install.

Check whether it is core first. Chapter 12.9's nine cover a large fraction of what scripts in this field actually need, and the list of core modules is much longer than those nine. perl -M<Module> -e 1 answers the question in one command — no output means it is there.

Check whether the farm already has it. A shared Perl in a semiconductor company usually has a curated set of extra modules installed deliberately, precisely so that users do not each build their own. Ask before assuming; the module may already be two directories away in @INC.

Count the dependencies. A module that pulls in thirty others is thirty things to install, thirty versions to pin, and thirty opportunities for a compiler to be missing — and modules with XS components need a working compiler toolchain, which a compute node may not have.

Consider whether forty lines of Perl is cheaper. This is the Chapter 11.22 argument in a different costume: the fact that a module exists is not a reason to take on a dependency. For something genuinely hard — CSV with quoting, JSON, date arithmetic — take the module. For something you could write and test in an afternoon, writing it may cost less than getting it approved and deployed.

If you do install, write it down. A cpanfile and a one-line note in the project README beats institutional memory. The failure mode otherwise is a script that works on one machine for one person, which is the problem Chapter 12.1 opened the module by measuring.

5. What To Remember

• A plain install writes into the Perl installation, and $Config{installsitelib} tells you where. Measured here: that directory is not writable by an ordinary user.
• That is the normal arrangement, not a problem to work around. The installation is shared, so changing it changes everyone's runs.
• sudo cpan is wrong even where it works. It silently makes your script depend on a machine-specific change nobody else has.
• Install into a project-local tree instead: cpanm -l extlib Some::Module, which produces extlib/lib/perl5/....
• Point PERL5LIB at extlib/lib/perl5, not at extlib. Measured: without the variable the module is not found; with it, it loads.
• cpanm -L makes the local tree self-contained; cpanm --installdeps . reads a cpanfile.
• cpanm may not be present — it is not on this machine, though cpan is. Check before planning around it.
• Check core first with perl -M<Module> -e 1, and ask whether the farm already provides it.
• Count the dependencies, and prefer core or your own code for anything you could write in an afternoon.
• Record what you installed in a cpanfile, or the knowledge lives in one person's shell history.

Chapter 12.11 returns to writing your own. A module can hold private state — a cache of results it has already computed — in a variable only its own subroutines can see, which is Chapter 12.7's my-at-file-scope technique put to a real purpose.