Perl Produces the Report, Another Tool Consumes It

Part II · Module 26 · Chapter 26.3 · Perl Produces the Report, Another Tool Consumes It

1. A Summary Written for People, Parsed by a Program

Here is how the problem usually starts. A Perl script at the end of a regression prints a summary for the engineers. It is clear and pleasant to read. Then someone writes a dashboard that needs the same numbers, and the quickest way to get them is to parse the summary.

The producer below prints one line per test. It takes a style argument so that one wording change can be measured; in real life that change is an edit somebody makes six months later to make the summary read better.

use strict;
use warnings;

my $style = shift // "v1";
my @results = (
    [ alu_smoke   => "PASS", 0 ],
    [ fifo_full   => "FAIL", 3 ],
    [ uart_parity => "PASS", 0 ],
);
for my $r (@results) {
    my ($test, $verdict, $errors) = @$r;
    if ($style eq "v1") {
        printf "Test %s %s with %d errors\n", $test,
            $verdict eq "PASS" ? "PASSED" : "FAILED", $errors;
    } else {
        # v2: someone made the summary friendlier
        printf "%-12s %s (%d error%s)\n", $test,
            $verdict eq "PASS" ? "passed" : "FAILED", $errors, $errors == 1 ? "" : "s";
    }
}

The consumer is a short Python script — the kind a colleague writes in ten minutes — that counts passes with a regular expression:

import re, sys

text = sys.stdin.read()
passes = len(re.findall(r"PASSED", text))
fails = len(re.findall(r"FAILED", text))
print(f"dashboard: {passes} passed, {fails} failed")

#!/bin/bash
for style in v1 v2; do
    echo "--- producer style $style"
    perl summary_prose.pl "$style" | sed 's/^/  /'
    perl summary_prose.pl "$style" | python3 scrape.py
    echo "  scraper exit status: $?"
done

--- producer style v1
  Test alu_smoke PASSED with 0 errors
  Test fifo_full FAILED with 3 errors
  Test uart_parity PASSED with 0 errors
dashboard: 2 passed, 1 failed
  scraper exit status: 0
--- producer style v2
  alu_smoke    passed (0 errors)
  fifo_full    FAILED (3 errors)
  uart_parity  passed (0 errors)
dashboard: 0 passed, 1 failed
  scraper exit status: 0

The friendlier summary made the dashboard report zero passes, and the scraper exited 0. Nothing crashed. Nothing logged a warning. The next morning's trend chart shows the pass rate falling from 67% to 0%, and someone spends an hour finding out that no test changed at all.

Neither author did anything unreasonable. The producer improved a message meant for people. The consumer parsed what it was given. The defect is that the hand-off was never defined, so a change to one side could not be checked against anything on the other.

2. The Schema Is the Contract

The fix is to produce a second output that exists only to be read by programs, and to write down what it contains. For a regression, that is a results file with one row per test:

Rows are sorted by test, the encoding is UTF-8, and lines end in a single newline.

Two of those rules carry more weight than they look.

"Empty when not known" is different from zero. A run that crashed (Chapter 21.2's NO_FINISH) has no error count — the log stopped before anything could be counted. Writing 0 would claim the run was checked and found clean. Writing nothing says, truthfully, that there is no number. A consumer that adds up the errors column must then decide what to do with a missing value, which is precisely the decision it should be forced to make.

The verdict is an enumeration, not free text. A consumer can only rely on values it has been told about. PASS, FAIL, NO_FINISH — three words, fixed spelling, fixed case. A producer that one day writes Pass is now detectably wrong, rather than quietly uncounted.

The prose summary from section 1 can stay. It is still the right thing to print for people. It just stops being an interface.

3. CSV Quoting: One Comma in a Message

CSV looks like the simplest possible format — values joined with commas — and that is exactly the trap. The first_error column holds a real error message, and real error messages contain commas.

The producer below writes the same four results in two ways: joined naively, and quoted by the standard CSV rule (RFC 4180): a field that contains a comma, a double quote, or a line break is wrapped in double quotes, and any double quote inside it is doubled.

use strict;
use warnings;
use File::Temp qw(tempfile);

my $mode = shift // "quoted";          # "naive" or "quoted"
my $out  = shift // "results.csv";

my @COLUMNS = qw(test verdict errors seed first_error);
my %VERDICT = map { $_ => 1 } qw(PASS FAIL NO_FINISH);

my @results = (
    { test => "uart_parity", verdict => "FAIL",      errors => 1,  seed => 42,
      first_error => 'parity mismatch: expected "odd"' },
    { test => "alu_smoke",   verdict => "PASS",      errors => 0,  seed => 7,
      first_error => "" },
    { test => "fifo_full",   verdict => "FAIL",      errors => 3,  seed => 1234,
      first_error => "SCBD mismatch: exp=0x1c, got=0x00" },
    { test => "dma_burst",   verdict => "NO_FINISH", errors => undef, seed => 99,
      first_error => "" },
);

sub csv_field {
    my ($v) = @_;
    return "" unless defined $v;          # empty means "not known"
    return $v unless $v =~ /[",\r\n]/;
    $v =~ s/"/""/g;
    return qq{"$v"};
}

for my $r (@results) {
    die "unknown verdict '$r->{verdict}' for $r->{test}\n" unless $VERDICT{$r->{verdict}};
}

my ($fh, $tmp) = tempfile("$out.XXXXXX", DIR => ".");
print {$fh} join(",", @COLUMNS), "\n";
for my $r (sort { $a->{test} cmp $b->{test} } @results) {
    my @f = map { $mode eq "naive" ? ($r->{$_} // "") : csv_field($r->{$_}) } @COLUMNS;
    print {$fh} join(",", @f), "\n";
}
close $fh or die "cannot finish $tmp: $!\n";
rename $tmp, $out or die "cannot rename $tmp to $out: $!\n";
print "wrote $out ($mode): ", scalar(@results), " rows\n";

The consumer is the obvious Python reader, using the standard library's csv module and reading columns by name:

import csv, sys

with open(sys.argv[1], newline="") as f:
    for row in csv.DictReader(f):
        print(f"  {row['test']:<12} first_error=[{row['first_error']}]"
              + (f"  extra={row[None]}" if None in row else ""))

#!/bin/bash
for mode in naive quoted; do
    perl write_results.pl "$mode" "results_$mode.csv"
    sed 's/^/  | /' "results_$mode.csv"
    python3 lenient_reader.py "results_$mode.csv"
    echo
done

wrote results_naive.csv (naive): 4 rows
  | test,verdict,errors,seed,first_error
  | alu_smoke,PASS,0,7,
  | dma_burst,NO_FINISH,,99,
  | fifo_full,FAIL,3,1234,SCBD mismatch: exp=0x1c, got=0x00
  | uart_parity,FAIL,1,42,parity mismatch: expected "odd"
  alu_smoke    first_error=[]
  dma_burst    first_error=[]
  fifo_full    first_error=[SCBD mismatch: exp=0x1c]  extra=[' got=0x00']
  uart_parity  first_error=[parity mismatch: expected "odd"]

wrote results_quoted.csv (quoted): 4 rows
  | test,verdict,errors,seed,first_error
  | alu_smoke,PASS,0,7,
  | dma_burst,NO_FINISH,,99,
  | fifo_full,FAIL,3,1234,"SCBD mismatch: exp=0x1c, got=0x00"
  | uart_parity,FAIL,1,42,"parity mismatch: expected ""odd"""
  alu_smoke    first_error=[]
  dma_burst    first_error=[]
  fifo_full    first_error=[SCBD mismatch: exp=0x1c, got=0x00]
  uart_parity  first_error=[parity mismatch: expected "odd"]

Read the fifo_full row of the naive file. The error message SCBD mismatch: exp=0x1c, got=0x00 contains one comma, so that line has six fields under a five-column header. Python's DictReader did not complain. It filled the five named columns from the first five fields — so first_error became the first half of the message — and put the rest under the key None. A dashboard that displays row['first_error'] shows a truncated message and gives no sign that anything was lost.

The quoted file reads back exactly, including the double quotes inside the uart_parity message. Note also the two properties the producer already has: rows come out sorted by test even though they were listed in a different order (Chapter 17.8), and dma_burst's unknown error count is an empty field, not 0.

4. A Consumer That Refuses Instead of Guessing

The lenient reader in section 3 is how most consumers are written, and it is why the naive file's defect was silent. A consumer of a defined schema can do something better: check the input against the schema and refuse it if it does not match, with a message that says exactly what is wrong.

import csv, sys

SCHEMA = ["test", "verdict", "errors", "seed", "first_error"]
VERDICTS = {"PASS", "FAIL", "NO_FINISH"}

def load(path):
    with open(path, newline="") as f:
        rows = list(csv.reader(f))
    if not rows or rows[0] != SCHEMA:
        raise SystemExit(f"dashboard: {path}: header is not the agreed schema")
    problems, data = [], []
    for n, r in enumerate(rows[1:], start=2):
        if len(r) != len(SCHEMA):
            problems.append(f"line {n}: {len(r)} fields, expected {len(SCHEMA)}")
            continue
        rec = dict(zip(SCHEMA, r))
        if rec["verdict"] not in VERDICTS:
            problems.append(f"line {n}: unknown verdict {rec['verdict']!r}")
        if rec["errors"] != "" and not rec["errors"].isdigit():
            problems.append(f"line {n}: errors {rec['errors']!r} is not a count")
        data.append(rec)
    if not data:
        problems.append("no tests in file -- a regression with zero tests is broken")
    if problems:
        raise SystemExit(f"dashboard: {path}: " + "; ".join(problems))
    return data

data = load(sys.argv[1])
by = {v: sum(1 for d in data if d["verdict"] == v) for v in sorted(VERDICTS)}
unknown = sum(1 for d in data if d["errors"] == "")
print(f"dashboard: {len(data)} tests  " + "  ".join(f"{k}={v}" for k, v in by.items())
      + f"  (error count unknown for {unknown})")

#!/bin/bash
perl write_results.pl naive  naive.csv  >/dev/null
perl write_results.pl quoted good.csv   >/dev/null
printf 'test,verdict,errors,seed,first_error\n' > empty.csv
sed 's/,PASS,/,Pass,/' good.csv > badverdict.csv

for f in good.csv naive.csv empty.csv badverdict.csv; do
    echo "--- $f"
    python3 dashboard.py "$f" 2>&1 | sed 's/^/  /'
    echo "  exit ${PIPESTATUS[0]}"
done

--- good.csv
  dashboard: 4 tests  FAIL=2  NO_FINISH=1  PASS=1  (error count unknown for 1)
  exit 0
--- naive.csv
  dashboard: naive.csv: line 4: 6 fields, expected 5
  exit 1
--- empty.csv
  dashboard: empty.csv: no tests in file -- a regression with zero tests is broken
  exit 1
--- badverdict.csv
  dashboard: badverdict.csv: line 2: unknown verdict 'Pass'
  exit 1

Four inputs, four precise answers, and three refusals with a non-zero status:

• The naive file is rejected at the exact line with the extra field. The same defect the lenient reader passed silently is now a one-line message that points at the producer.
• A header-only file is rejected because a regression with no tests in it is not a clean regression — it is a regression that did not run. Treating an empty result as success is the same mistake as Chapter 21.2's crashed log with zero errors.
• Pass instead of PASS is caught by the enumeration. In the scraper of section 1, a spelling change made results disappear; here it makes the consumer stop.

Because the dashboard exits non-zero when it refuses, the job that runs it fails, and the failure is visible at the point where it can be fixed — the same propagation rule Chapter 26.1 measured for the Perl side.

5. JSON: Key Order, and a Number That Arrives as a String

CSV suits flat tables. When a result has nested parts — per-corner slacks under each test, a list of error signatures — JSON is the usual choice, and Perl has a JSON encoder in its core distribution: JSON::PP, shipped with Perl since 5.14.

Two of its defaults matter at a hand-off.

Key order. A Perl hash has no defined order (Module 6), so by default JSON::PP writes keys in whatever order the hash yields them, and Perl randomises that order per process. Encoding the same five-key hash in five separate runs on this machine produced five different key orders. That is harmless to a program that reads JSON properly, and it is ruinous to anything that compares files — a golden-file test, a diff between two nights' results, a version-control history of a results file. ->canonical(1) sorts the keys, and the output becomes byte-identical from run to run.

Numbers and strings. JSON distinguishes -0.043 (a number) from "-0.043" (a string). Perl does not distinguish them the same way: a scalar can hold both forms, and JSON::PP decides which to write by looking at how the variable has been used so far.

use strict;
use warnings;
use JSON::PP;

my $json = JSON::PP->new->canonical(1);
my $line = "  slack (VIOLATED)   -0.043";

my ($slack) = $line =~ /(-?\d+\.\d+)/;      # captured from text
print "freshly captured:      ", $json->encode({ slack => $slack }), "\n";

my $is_violation = $slack < 0;               # any numeric use...
print "after a comparison:    ", $json->encode({ slack => $slack }), "\n";

my ($other) = $line =~ /(-?\d+\.\d+)/;
print "explicit 0 + \$other:   ", $json->encode({ slack => 0 + $other }), "\n";
print "and \$other itself:     ", $json->encode({ slack => $other }), "\n";

freshly captured:      {"slack":"-0.043"}
after a comparison:    {"slack":-0.043}
explicit 0 + $other:   {"slack":-0.043}
and $other itself:     {"slack":-0.043}

The same variable encoded as a string first and a number second. Nothing assigned to it in between; a comparison read it as a number, Perl cached the numeric value alongside the string, and from then on JSON::PP treated it as numeric.

The last two lines repeat the effect from a different direction. 0 + $other produced a number, as intended — and in doing so it read $other numerically, so $other itself now encodes as a number too. Evaluating an expression changed the type of a variable it only looked at.

So whether a field arrives downstream as a number depends on whether some unrelated line of the producer happened to do arithmetic with it — which can change when someone deletes a debug print.

The consumer feels this in two different ways, and only one of them is loud:

import json

as_string = json.loads('{"slack": "-0.043"}')
as_number = json.loads('{"slack": -0.043}')

for name, d in (("number", as_number), ("string", as_string)):
    try:
        print(f"{name:<6}: slack < 0 is {d['slack'] < 0}")
    except TypeError as e:
        print(f"{name:<6}: TypeError: {e}")

worst_num = min([-0.043, -0.5, 0.12])
worst_str = min(["-0.043", "-0.5", "0.12"])
print(f"worst of three slacks as numbers: {worst_num}")
print(f"worst of three slacks as strings: {worst_str}   <- wrong, and no error")

number: slack < 0 is True
string: TypeError: '<' not supported between instances of 'str' and 'int'
worst of three slacks as numbers: -0.5
worst of three slacks as strings: -0.043   <- wrong, and no error

Comparing a string with a number raised an error — loud, and easy to trace. Finding the minimum of strings raised nothing and returned -0.043, because strings compare character by character and "-0.0" sorts before "-0.5". The worst slack in the design is -0.5. A dashboard that ranks paths by slack would put the wrong path at the top, and nothing would ever say so.

The rule that follows is simple and has to be applied in the producer: at the point of output, convert every numeric field explicitly — 0 + $value for a number, and leave text fields alone. Do it in the one function that builds the output record, so the type of every field is decided in one place and not by the history of each variable.

6. Never Let a Consumer See Half a File

A consumer that polls a directory — a dashboard refreshing every minute, a CI step that starts when a file appears — can open the results file while the producer is still writing it. Half a CSV is a valid-looking CSV with fewer rows. Half a JSON document fails to parse, which is better, but only if the consumer treats a parse error as an error.

write_results.pl already avoids this, using Chapter 13.13's technique: write to a temporary file in the same directory, close it, check that the close succeeded, and rename it into place. On a local filesystem, rename within one directory replaces the old file in one step, so a reader sees either the complete old file or the complete new one. The temporary file must be in the same directory because a rename across filesystems is not a rename — it becomes a copy, and the guarantee is gone.

Two further habits make the hand-off robust to the producer failing:

Do not write the output at all when the input was unusable. A results file is a claim that results exist. If the Perl stage could not read its inputs, it should exit with its "unusable" status (Chapter 26.1's contract) and leave the previous file — or no file — rather than writing a header-only file that a careless consumer would read as "no tests failed".

Put the producer's identity in the file when the format allows it. In JSON, a top-level "schema": "results/1" and "producer": "write_results.pl" cost nothing and answer the first question anyone asks about a strange file. CSV has no standard place for that, which is one reason to choose JSON when a file will outlive the job that wrote it.

7. Changing the Schema Without Breaking Anyone

Schemas change. A new column is the common case — say a runtime_s column so the dashboard can show slow tests. Whether that change breaks the consumers depends almost entirely on how they read columns.

import csv, io

v1 = "test,verdict,errors,seed\nfifo_full,FAIL,3,1234\n"
v2 = "test,verdict,runtime_s,errors,seed\nfifo_full,FAIL,812,3,1234\n"

for label, text in (("v1 file", v1), ("v2 file", v2)):
    rows = list(csv.reader(io.StringIO(text)))
    positional = rows[1][2]                                   # "errors is column 3"
    named = dict(zip(rows[0], rows[1]))["errors"]             # "the column called errors"
    print(f"{label}: positional errors={positional:<4}  named errors={named}")

v1 file: positional errors=3     named errors=3
v2 file: positional errors=812   named errors=3

The v2 producer inserted the new column in the middle. The positional consumer now reads the runtime as the error count — 812 errors for a test that had 3 — and nothing fails. The named consumer is unaffected.

Three rules keep schema changes safe:

• Consumers read by column name, never by position. That one habit makes most additions harmless.
• Producers add new columns at the end, so that a positional consumer somebody else wrote also survives.
• Removing or renaming a column, or changing what a value means, is a breaking change. Give the file a new name or a new schema version, and tell the consumers. The strict dashboard.py from section 4 will refuse a changed header, which is the behaviour you want: a consumer that has not been updated stops loudly instead of producing numbers from a contract it does not understand.

8. Exercises

1. Write the JSON producer. Extend write_results.pl with a json mode that writes results.json containing a top-level schema, producer and tests array. Every numeric field must be a JSON number regardless of how the producer used it, and the file must be byte-identical across ten runs. Write the check that proves the second property, and make it fail by removing canonical(1).

2. The empty-errors decision. The dashboard reports "error count unknown for 1". A manager asks for a single "total errors" number on the front page. Write the two possible definitions — one that treats unknown as zero and one that does not — and explain, using Chapter 21.2's crashed run, which one can report a regression as healthier than it is.

3. Make the lenient reader strict. Modify lenient_reader.py so it rejects the naive file, using only the information DictReader already gives it. Why is checking None in row sufficient for an extra field but not for a missing one, and what does DictReader put in a short row?

4. A newline in a message. A test's first error message contains a line break. Predict what the quoted CSV row looks like, whether dashboard.py reads it correctly, and whether wc -l results.csv still equals the number of tests plus one. Run it and explain any difference from your prediction. What does that imply for a shell script that counts tests with wc -l?

5. Find the prose interface. In a flow you have access to, find one place where a program parses another program's human-readable output. Write the schema for the file that should replace it — columns, meanings, allowed values, what empty means — and identify the first field whose meaning you had to guess.

9. What To Remember

• A summary written for people is not an interface. Measured: a one-word wording change made a scraper count 0 passes of 2 and exit 0.
• Write a second output that only programs read, and document its schema: column names, order, meanings, allowed values.
• Empty means "not known"; zero means "counted and none". A crashed run has no error count (21.2).
• Make enumerated fields enumerations — fixed spelling, fixed case — so a wrong value is detectable.
• Quote CSV by the rule: wrap fields containing comma, quote or newline in quotes; double inner quotes.
• Measured: one unquoted comma made Python's DictReader silently truncate a message and park the rest under None.
• Text::CSV is CPAN, not core, and may be missing on a compute farm; a small, correct writer is enough for a fixed schema.
• The consumer should refuse, not guess: check the header, the field count, enumerations, and refuse an empty result.
• JSON::PP is core; use ->canonical(1) or key order changes every run.
• JSON::PP writes a scalar as a number or a string depending on its history. Measured: the same variable encoded as "-0.043", then -0.043. Convert with 0 + at the output boundary.
• Strings compare silently wrong: the minimum of "-0.043", "-0.5" came out as -0.043.
• Write through a temp file in the same directory and rename (13.13), and write nothing when the inputs were unusable.
• Consumers read columns by name; producers add columns at the end; anything else is a breaking change.

Chapter 26.4 crosses to the other half of a signoff flow — the half Perl does not own. Before Perl can parse a per-corner timing report, a Tcl script running inside the timing tool has to produce it, and the way that script is written decides what Perl will find.