Classifying a Verdict: PASS, FAIL or UNKNOWN

Part I · Module 8 · Chapter 8.6 · Classifying a Verdict: PASS, FAIL or UNKNOWN

1. Six Reports, One of Them Absent

Everything in this chapter runs against the same six fixtures, written to disk so nothing is hypothetical.

reports/fail.rpt         block alu  / slack -0.120 / paths 1840
reports/pass.rpt         block fifo / slack  0.340 / paths  920
reports/nofield.rpt      block regf / paths 2310            <- no slack line
reports/nonnumeric.rpt   block pad  / slack N/A   / paths 0 <- slack is text
reports/empty.rpt        zero bytes
reports/missing.rpt      does not exist

Two are ordinary: one failing block and one passing block. The other four are the shapes real tool output actually takes when something upstream went wrong — a run that died before writing the field, a field the tool filled with N/A, a file created but never written, and a file that was never created because the tool crashed.

None of those four is a pass or a fail. They are all "I do not know".

2. The Two-State Check

Here is the check almost everybody writes first.

use strict;
use warnings;

# TWO-STATE: the check can only answer pass or fail.
sub verdict_two {
    my ($path) = @_;
    my $slack;
    if (open my $fh, "<", $path) {
        while (my $line = <$fh>) {
            $slack = $1 if $line =~ /^slack\s+(\S+)/;
        }
        close $fh;
    }
    return ($slack && $slack < 0) ? "FAIL" : "PASS";
}

for my $f (qw(fail pass nofield nonnumeric empty missing)) {
    my $p = "reports/$f.rpt";
    printf "%-11s %s\n", $f, verdict_two($p);
}

fail        FAIL
pass        PASS
nofield     PASS
nonnumeric  PASS
empty       PASS
missing     PASS

Argument "N/A" isn't numeric in numeric lt (<) at verdict2.pl line 14.

0

Four of the six unusable inputs reported PASS. The report that does not exist reported PASS. The empty file reported PASS. The report with no slack field reported PASS.

And look at what reached standard error: one line, for one of the four. The non-numeric case warned because "N/A" < 0 had to numify. The missing file, the empty file and the absent field produced no diagnostic at all — open failed and the if simply did not run its body, which is exactly what you asked it to do.

The script exited 0. A regression harness reading that exit status, or a human reading that column of PASS, learns nothing about four of the six runs.

3. The Ordering Problem

Before fixing the return value, there is a second trap worth settling, because the obvious fix runs straight into it.

The natural instinct is to add more comparison branches. Chapter 8.1 §5 measured why that cannot work:

undef  : < -0.1 = false  < 0 = false  >= 0 = true 
N/A    : < -0.1 = false  < 0 = false  >= 0 = true 
NaN    : < -0.1 = false  < 0 = false  >= 0 = false

undef and "N/A" both numify to 0, and 0 >= 0 is true — so a comparison chain classifies both as a perfectly good non-negative slack. Of nine probe values only NaN made every comparison false.

So bad data does not announce itself by failing every test. It announces itself by quietly passing one. No number of extra comparison branches will catch it, because the comparison operators have already converted the problem away before the branch is reached.

The consequence for how you write the decision is strict:

return "UNKNOWN"  unless defined $slack;                        # 1. is there a value
return "UNKNOWN"  unless $slack =~ /^[-+]?\d+(?:\.\d+)?$/;      # 2. is it a number
return $slack < 0 ? "FAIL" : "PASS";                            # 3. only now compare

0.34      -> ok
-0.5      -> critical
-0.05     -> marginal
undef     -> UNKNOWN
N/A       -> UNKNOWN
NaN       -> UNKNOWN

Usability first, then value. defined answers "is there anything here", the pattern match answers "is it a number", and only code that has passed both is allowed to compare. NaN is caught by the pattern because "NaN" does not match a digit pattern — so the one case comparisons could catch is handled by the same guard as the two they could not.

One honest caveat about that pattern, measured:

-0.0730   accepted
0.340     accepted
-0.120    accepted
+1.5      accepted
1.2e-3    REJECTED -> UNKNOWN
.5        REJECTED -> UNKNOWN
5.        REJECTED -> UNKNOWN
N/A       REJECTED -> UNKNOWN

It rejects scientific notation, and EDA tools emit it — a path count or a capacitance can easily arrive as 7.4231e+05. It also rejects .5 and 5., which are valid Perl numbers.

That is a deliberate trade rather than an oversight, and you have to pick a side knowingly:

• A strict pattern is safer when the format is known. If slack is always fixed-point, rejecting 1.2e-3 catches a tool whose output format changed — which is a real thing worth knowing about.
• A permissive pattern is right when the format varies. Widen it to /^[-+]?(?:\d+\.?\d*|\.\d+)(?:[eE][-+]?\d+)?$/, which accepts all four rejected forms above including 7.4231e+05.

What is not acceptable is leaving the question unasked, because a rejected-but-valid number becomes an UNKNOWN — and an UNKNOWN you did not intend is noise that trains people to ignore the real ones.

4. The Three-State Check

use strict;
use warnings;

# THREE-STATE: absent or unusable input is reported as UNKNOWN.
sub verdict_three {
    my ($path) = @_;

    return ("UNKNOWN", "report not found")        unless -e $path;
    open my $fh, "<", $path or return ("UNKNOWN", "cannot open: $!");

    my $slack;
    my $lines = 0;
    while (my $line = <$fh>) {
        $lines++;
        $slack = $1 if $line =~ /^slack\s+(\S+)/;
    }
    close $fh;

    return ("UNKNOWN", "report is empty")         if $lines == 0;
    return ("UNKNOWN", "no slack field")          unless defined $slack;
    return ("UNKNOWN", "slack not numeric: $slack")
        unless $slack =~ /^[-+]?\d+(?:\.\d+)?$/;

    return ($slack < 0 ? "FAIL" : "PASS", sprintf("slack %+.3f", $slack));
}

my %tally;
for my $f (qw(fail pass nofield nonnumeric empty missing)) {
    my ($v, $why) = verdict_three("reports/$f.rpt");
    $tally{$v}++;
    printf "%-11s %-8s %s\n", $f, $v, $why;
}
print "\n";
printf "%-8s %d\n", $_, $tally{$_} for sort keys %tally;

my $code = $tally{UNKNOWN} ? 2 : $tally{FAIL} ? 1 : 0;
printf "exit code would be %d\n", $code;

fail        FAIL     slack -0.120
pass        PASS     slack +0.340
nofield     UNKNOWN  no slack field
nonnumeric  UNKNOWN  slack not numeric: N/A
empty       UNKNOWN  report is empty
missing     UNKNOWN  report not found

FAIL     1
PASS     1
UNKNOWN  4
exit code would be 2

One FAIL, one PASS, four UNKNOWN — and every UNKNOWN carries its reason.

Compare the two tables directly:

Four rows changed, and so did the exit status. Note also that STDERR is now empty — the warning the two-state version produced is gone, because the non-numeric value is rejected by the pattern match before anything tries to compare it.

The decision, drawn

Read the captions: the first five steps have exactly one failure answer, and it is UNKNOWN. Only step 6 can produce PASS or FAIL, and it is reached only by input that survived all five gates. That is the structural property worth copying — the comparison is the last thing in the function, not the first.

5. Returning a Reason

The second return value is doing more work than it looks.

return ("UNKNOWN", "slack not numeric: $slack");

An UNKNOWN with no reason is only marginally better than a wrong PASS — it tells you to go and look, but not where. The reason string turns a verdict into a diagnosis, and it costs one extra list element.

Three things make a reason useful:

• Name the specific gate that failed, not the general situation. no slack field and report is empty lead to different investigations; bad report leads to neither.
• Include the offending value when there is one. slack not numeric: N/A tells you the tool wrote N/A on purpose, which is a different problem from the field holding garbage.
• Use $! for system failures. cannot open: $! carries the operating system's own explanation — Permission denied and No such file or directory are different bugs, and Chapter 3.9 §2 covers $! with die.

Note the -e test on the first line and the separate open ... or return. Those look redundant — if the file is missing, open would fail anyway. They are separate because they produce different reasons: a missing file is usually an upstream tool that did not run, while a file that exists but will not open is usually a permissions or filesystem problem. Collapsing them into one gate saves a line and loses the distinction. File test operators are Module 13's subject; -e for existence and -s for non-empty are the two worth knowing now.

6. Three States Need Three Exit Codes

my $code = $tally{UNKNOWN} ? 2 : $tally{FAIL} ? 1 : 0;

exit code would be 2

A verdict that only a human reads is half a verdict. The script's exit status is what a regression harness, a Makefile or a CI job actually branches on, and it needs the same three states:

The distinction between 1 and 2 is the whole point. A real failure and an unusable input demand different responses from different people. A FAIL goes to the designer who owns the block; an UNKNOWN goes to whoever owns the tool that did not produce a report. Collapsing them into one non-zero code sends every problem to the wrong person half the time.

And the ordering in that ternary chain is deliberate: UNKNOWN wins over FAIL. If any input was unreadable, the overall answer is "this run is not trustworthy" — because an unexamined block might also have been failing. Reporting 1 when four of six reports were unreadable would claim more knowledge than the run actually has.

7. Why This Matters in Engineering Work

A clean regression report is the most expensive wrong answer in the trade. A nightly run over two hundred blocks where eleven tools crashed will report two hundred passes from a two-state check, and a green dashboard is precisely the signal nobody investigates. The three-state version reports 189 checked and 11 unknown, and somebody goes and looks.

Sign-off asks a three-state question by nature. "Is this block clean?" has the answers yes, no, and we have not been able to measure it. The third is the one that holds up a tapeout, and a flow that cannot represent it will report the tapeout as ready.

And the distinction routes the work. FAIL is a design problem for the block owner. UNKNOWN is a flow problem for the CAD or methodology owner. One exit code for both means half of all escalations go to the wrong team first.

The cost of all this is about eight lines. Five guards, a reason string, and a three-way exit mapping — against a check that silently reported four of six broken runs as healthy.

8. Common Mistakes

A check whose return values are only PASS and FAIL. Unusable input lands in one of them, and it will be PASS.

Comparing before validating. undef and non-numeric text numify to 0, so >= 0 is true for both. Test defined and the pattern first.

Adding more comparison branches to catch bad data. They cannot; the conversion has already happened. Of nine probe values only NaN fails every comparison.

An UNKNOWN with no reason. It says go and look without saying where. Return a reason string.

Collapsing -e and a failed open into one gate. A missing file and an unreadable file are different problems.

One non-zero exit code for both FAIL and UNKNOWN. They need different people to act.

Letting FAIL outrank UNKNOWN in the summary. If anything was unreadable, the run is untrustworthy — UNKNOWN must win.

Treating an empty file as empty data. A zero-byte report usually means the tool died, not that there was nothing to report.

9. Exercise

Exercise 1 — build the fixtures. Create the six reports from §1 in a scratch directory, including the empty one, and deliberately do not create missing.rpt. Confirm with a listing that five files exist.

Exercise 2 — the two-state check. Write verdict_two from §2 and run it over all six. Record the six verdicts, every line of STDERR, and the exit status. State how many unusable inputs were reported as PASS and how many of those produced any diagnostic.

Exercise 3 — why it is not a bug. For each of the five lines listed in §2's callout, confirm the behaviour yourself with a one-line test. Then write one sentence naming where the defect actually is.

Exercise 4 — the ordering problem. For undef, "N/A" and "nan" + 0, print the three comparisons < -0.1, < 0 and >= 0 separately. Record which values pass >= 0 and explain why in terms of numification.

Exercise 5 — the three-state check. Write verdict_three from §4 and run it over all six fixtures. Record the verdicts with reasons, the tally, the exit code, and confirm STDERR is empty. Then explain why the N/A warning disappeared.

Exercise 6 — a seventh fixture. Add a report whose slack line reads slack -0.0730 and one whose slack line reads slack 1.2e-3. Run the three-state check on both and record what happens. If either is reported as UNKNOWN, decide whether the pattern or the requirement should change, and say why.

Exercise 7 — the reasons. Make one fixture unreadable with chmod 000 and run the check. Record the reason string, including what $! contributed. Then restore the permissions.

Exercise 8 — the exit mapping. Run the check over three sets: all passing, one failing and the rest passing, and one unknown among passing ones. Record the exit code for each and confirm UNKNOWN outranks FAIL.

Exercise 9 — the habit. Take any conditional you have written in this module's earlier chapters and ask §6's question of it: what does it do when the input is unusable, and is that distinguishable from a real result? Write down the answer, and widen the result set if it needs it.

10. Summary

• A two-state check has nowhere to put an unusable input, so it puts it in the passing state. Measured: four of six broken inputs reported PASS, three of them with no diagnostic at all, and the script exited 0.
• Every line of that check behaves exactly as documented. The defect is the shape of the return value, which is a design decision rather than a coding mistake — no linter and neither pragma can find it.
• You cannot catch bad data with more comparisons. undef and "N/A" numify to 0, so >= 0 is true for both; of nine probe values only NaN failed every comparison.
• Order the decision usability-first: defined, then a numeric pattern, then — and only then — the comparison.
• Five gates, each answering only UNKNOWN or continue, with the comparison last. Measured result: 1 FAIL, 1 PASS, 4 UNKNOWN, empty STDERR.
• Return a reason with every UNKNOWN — the specific gate, the offending value, and $! for system failures.
• Keep -e and a failed open as separate gates; a missing file and an unreadable one are different problems.
• Use three exit codes: 0 pass, 1 real failure, 2 could not check. They route to different owners.
• UNKNOWN outranks FAIL in a summary, because an unexamined input might also have been failing.
• The general habit: ask what a check returns when its input is unusable, and whether that is distinguishable from a real result. If not, widen the result set.

Module 8 is complete. Every form in it — the if chain, the negated guard, the statement modifier, the ternary, and the short-circuit check — exists to express a decision, and this chapter is the decision they are usually assembled to make. Module 9 turns to loops: the for and while forms, what $_ really is, and why a loop that finishes is not the same as a loop that did its work.