Three Kinds of Failure: Compile, Runtime, Logic

Part I · Module 14 · Chapter 14.1 · Three Kinds of Failure: Compile, Runtime, Logic

1. The Three, Side by Side

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);

# Three scripts, one for each kind of failure.
my %src = (
  "compile.pl" => <<'S',
print "A: started\n";
my $x = ;                      # a syntax error
print "B: finished\n";
S
  "runtime.pl" => <<'S',
print "A: started\n";
my @a = (1,2,3);
my $bad = 1 / 0;               # dies while running
print "B: finished\n";
S
  "logic.pl"   => <<'S',
print "A: started\n";
my @slack = (-0.12, 0.44, -0.95);
my $worst = $slack[0];
for my $s (@slack) { $worst = $s if $s > $worst }   # > should be <
printf "worst slack: %s\n", $worst;
print "B: finished\n";
S
);

for my $name (sort keys %src) {
    my $p = File::Spec->catfile($dir, $name);
    open my $w, ">", $p or die $!; print $w $src{$name}; close $w;
    my $out = `perl "$p" 2>&1`;
    my $rc  = $? >> 8;
    $out =~ s/\Q$dir\E/TMPDIR/g;          # the temp path varies per run
    $out =~ s/ at \S+ line/ at line/g;
    $out =~ s/\s+\z//; $out =~ s/\n/ | /g;
    printf "%-11s exit %-3d %s\n", $name, $rc, ($out eq "" ? "(no output)" : $out);
}

compile.pl  exit 255 syntax error at line 2, near "= ;" | Execution of TMPDIR/compile.pl aborted due to compilation errors.
logic.pl    exit 0   A: started | worst slack: 0.44 | B: finished
runtime.pl  exit 255 Illegal division by zero at line 3. | A: started

Read the three rows carefully, because each one behaves differently.

compile.pl — exit 255, and none of its own output. Line 1 was a perfectly good print and it never ran. Perl reads the whole file before executing any of it, so a syntax error anywhere stops everything. The message names the line and quotes the text near the problem.

runtime.pl — exit 255, and A: started did print. The script compiled fine. It ran, divided by zero, and died at that line. One line of work had already happened.

logic.pl — exit 0, both messages printed, and the answer is wrong. The list is -0.12, 0.44, -0.95 and it reported the worst slack as 0.44. The real worst is -0.95. The comparison uses > where it should use <.

Nothing complained. The exit status is 0, which every scheduler and every wrapper script reads as success.

2. Reading the Exit Status

The first thing a surrounding flow sees is the exit status, so it is worth knowing what it means.

0 means success by convention, and it is only a convention — a script that finishes is 0 unless it says otherwise.

A non-zero value means failure, and the specific number is rarely meaningful. Perl uses 255 for a die and for a compile error, as both rows above show. A script can choose its own with exit 2.

So a wrapper that checks the exit status catches the first two kinds of failure and is blind to the third. That is worth stating plainly, because an enormous amount of flow automation is built on exactly that check. "The job succeeded" means "the script did not crash", which is a much weaker statement than it sounds.

Two habits follow:

Make your script exit non-zero when it finds a problem, even if nothing crashed. A report script that found 400 violations should probably still exit 0 — that is data, not an error — but one that could not parse 40 of its 400 input lines should not pretend everything was fine.

Print a count of what you did. processed 360 of 400 rows turns a silent logic failure into a visible one, because the numbers stop matching what you expected. Chapter 14.8 develops this.

3. Why the Third Kind Is the Expensive One

The first two failures have something valuable in common: they tell you where they happened. A line number and a message is most of a diagnosis.

A logic error gives you none of that. There is no line number, because no line was wrong in a way Perl can see — $s > $worst is valid Perl doing exactly what it says. The only evidence is that the answer is wrong, and that requires somebody to know what the right answer was.

In this field the situation is worse than in most, for one specific reason: the output is plausible. worst slack: 0.44 is a real number that came from the real input. It is not a crash, not a nan, not an empty field. Somebody reading a report with that number in it has no reason to question it, and the design goes forward on it.

The failures measured elsewhere in this track are all of this kind:

• Chapter 11.8's /ERROR/ counted 4 error lines where there were 2 — a believable number.
• Chapter 11.10's unguarded $1 produced 4 report rows from 3 real data rows, with a duplicated path and a slack total of 0.71 where the guarded total is 0.27.71 instead of 0.27.
• Chapter 12.1's two copies of one rule reported different worst paths from the same input, both exiting 0.
• Chapter 13.5's unchecked open read a missing file as an empty one and reported a clean run.

None of those produced an error. All of them produced a number.

One more practical point about the three kinds. They are found at different times, and that ordering is useful:

• perl -c script.pl compiles without running, so it finds the first kind in a second. Run it before you run anything slow.
• use strict and use warnings convert a large family of would-be logic errors into compile errors or warnings. A mistyped variable name becomes a compile error instead of a silent undef, which is the single biggest reduction in the third category available to you.
• The third kind needs a test — a known input with a known answer. That is the only tool that works.

4. What To Remember

• Three kinds of failure: will not compile, dies while running, or finishes with a wrong answer.
• A compile error runs nothing. Measured: A: started never printed, exit 255.
• A runtime error stops part way. Measured: A: started printed, then it died, exit 255.
• A logic error exits 0. Measured: it printed worst slack: 0.44 when the worst value was -0.95.
• Exit 0 means "did not crash", not "was right". A flow that only checks the exit status is blind to the third kind.
• Exit non-zero when your script finds a problem, and print a count of what you processed.
• The third kind is expensive because the output is plausible — a real number from real input, with no line number and no message.
• Error handling does not help with logic errors. die, eval and Carp are about the first two kinds.
• perl -c finds the first kind in a second. Run it first.
• use strict and use warnings convert many would-be logic errors into visible ones — the biggest single reduction available.
• A logic error needs a test: a known input with a known answer.

Chapter 14.2 starts on the tools. warn reports a problem and lets the script carry on; die stops it. Both have one formatting rule that decides whether your message is useful to a person or cluttered with a line number they cannot use.