Reading a Test List

Part II · Module 25 · Chapter 25.2 · Reading a Test List

1. Why the List Lives in a File

In Chapter 25.1 the runner took its tests from the command line. That is fine for five runs. A real regression has hundreds of tests, with different options, owned by different people. Those choices need to live somewhere that:

• people can edit without reading Perl — a designer adding a directed test should not have to touch the runner;
• shows up in code review as a readable diff — "added fifo_full, count 8" is a one-line change;
• can be split and shared — a short list for every merge, a long one for nightly, both reusing the same core tests.

A plain text file does all three. The runner reads it and turns it into data. That reading step is where this chapter lives.

There is one thing to be honest about first. There is no standard tests.list format. Every company, and often every project, has its own. The format below is this module's own, chosen to show the problems every such format meets. The techniques transfer; the exact syntax is yours to choose.

2. Write the Format Down First

The most useful thing you can do before writing a parser is to write down what it accepts. Put it at the top of the list itself, where the people editing the file will see it:

# tests.list -- the regression for sat_add.
# One test per line:  NAME  [key=value ...]  [+plusarg ...]
#   count=N    run N seeds of this test (default 1)
#   seed=N     use exactly this seed (default: the runner chooses)
#   timeout=S  per-run time limit in seconds

smoke
corner                         # known to fail until the fix lands
rand     count=4  +n=60        # four seeds, 60 vectors each
include lists/long.list

# lists/long.list -- slower tests, nightly only
stall    timeout=5
rand_big count=2  +n=4000  +msg_tag=RB#1

As rules, that format says:

The last row is the important one. A format that has no "anything else is an error" rule is not really a format: it is whatever the parser happens to do with unexpected input. The next section shows what that looks like.

The +msg_tag=RB#1 plusarg is there on purpose. It contains a # that is not a comment, because it is not preceded by a space. Real plusargs can contain almost any character, and a list format must not damage them.

3. The Four-Line Reader, Measured

This is how most test-list readers begin. It is the obvious code, and every line of it looks correct:

#!/usr/bin/perl
# first_reader.pl -- the four-line reader, exactly as first written.
use strict;
use warnings;

open my $fh, "<", "tests.list" or die "tests.list: $!\n";
my @tests;
while (my $l = <$fh>) {
    next if $l =~ /^\s*(?:#|$)/;
    my ($name, @opts) = split ' ', $l;
    push @tests, { name => $name, opts => \@opts };
}

for my $t (@tests) {
    printf "%-8s %d option(s): %s\n", $t->{name}, scalar @{ $t->{opts} }, "@{ $t->{opts} }";
}
print scalar(@tests), " test(s)\n";

smoke    0 option(s): 
corner   8 option(s): # known to fail until the fix lands
rand     8 option(s): count=4 +n=60 # four seeds, 60 vectors each
include  1 option(s): lists/long.list
4 test(s)

The file is correct. The reader ran with use warnings and printed no warning at all. And the result is wrong in three ways:

1. A test called include. The reader does not know about include lines, so include becomes a test name with one option. A runner would try to simulate a test called include and get a $fatal from the bench.
2. Comments became options. The skip test only catches lines that start with #. A comment at the end of a line is split into words like any other text, so corner has eight "options": #, known, to, fail, and so on. A later stage would try to turn known into a simulator argument.
3. Two tests vanished. stall and rand_big live in the included file, which was never read. Nothing in the output says that anything is missing.

The third one is the worst, because a missing test produces no failure. The regression simply gets smaller, and smaller regressions pass more often. Nobody investigates a pass.

4. split ' ' Is Special

One part of the four-line reader is right, and it is worth knowing why, because the alternatives that look the same are not. Perl treats a single space string as the pattern in split specially: it splits on runs of any whitespace (spaces, tabs, and also \r and \n) and ignores leading whitespace. Any other pattern does not:

#!/usr/bin/perl
# split_demo.pl -- three ways to split the same line.
use strict;
use warnings;

my $line = "  rand\tcount=4  +n=60\n";          # indented, a tab, a newline
chomp(my $l = $line);

for my $how ("split ' '", "split / /", "split /\\s+/") {
    my @f = $how eq "split ' '" ? split(' ', $l)
          : $how eq "split / /" ? split(/ /, $l)
          :                       split(/\s+/, $l);
    printf "%-14s %d fields: %s\n", $how, scalar @f, join(" ", map { "[" . s/\t/\\t/gr . "]" } @f);
}

split ' '      3 fields: [rand] [count=4] [+n=60]
split / /      5 fields: [] [] [rand\tcount=4] [] [+n=60]
split /\s+/    4 fields: [] [rand] [count=4] [+n=60]

split / / splits on each single space, so two spaces make an empty field, and the tab is not a separator at all: rand and count=4 stay glued together. split /\s+/ handles the tab, but leading indentation gives an empty first field, so the test name becomes the empty string. Only split ' ' gives the three fields a person sees.

People indent list entries to line them up, and editors insert tabs. Use split ' ', and write it with the quotes so the next reader knows it is deliberate.

5. Comments That Are Not Comments

The obvious way to remove a comment is to delete everything from # to the end of the line. Here is that rule and two others, applied to the rand_big line and to an ordinary whole-line comment:

#!/usr/bin/perl
# comment_demo.pl -- three comment rules, two lines.
use strict;
use warnings;

my @lines = ("rand_big count=2  +n=4000  +msg_tag=RB#1   # nightly only",
             "# a whole-line comment");
my @rules = (
    [ "s/#.*//",           sub { $_[0] =~ s/#.*//r } ],
    [ "s/\\s+#.*//",       sub { $_[0] =~ s/\s+#.*//r } ],
    [ "s/(?:^|\\s+)#.*//", sub { $_[0] =~ s/(?:^|\s+)#.*//r } ],
);
for my $l (@lines) {
    print "line: $l\n";
    printf "  %-18s [%s]\n", $_->[0], $_->[1]->($l) for @rules;
}

line: rand_big count=2  +n=4000  +msg_tag=RB#1   # nightly only
  s/#.*//            [rand_big count=2  +n=4000  +msg_tag=RB]
  s/\s+#.*//         [rand_big count=2  +n=4000  +msg_tag=RB#1]
  s/(?:^|\s+)#.*//   [rand_big count=2  +n=4000  +msg_tag=RB#1]
line: # a whole-line comment
  s/#.*//            []
  s/\s+#.*//         [# a whole-line comment]
  s/(?:^|\s+)#.*//   []

s/#.*// cuts the plusarg to +msg_tag=RB. The run still launches, with a different argument than the list says, and nothing reports it. s/\s+#.*// protects the plusarg but leaves the whole-line comment untouched, because there is no space before its #. The third rule says exactly what the format table says: a # at the start of the line, or after whitespace. It handles both.

This is the general lesson of the chapter in miniature. The rule in the code must be the rule in the format table, word for word. When they differ, the difference is a silent change to somebody's test.

6. Options: Strict Beats Lenient

Options are key=value words. A lenient reader stores whatever it finds, and lets later code look up the keys it knows about. That sounds flexible. Here is what it means in practice: if someone types cuont=4, the lenient reader stores cuont => 4, the runner looks up count, finds nothing, uses the default of 1, and the test runs one seed instead of four. No error, no warning — three quarters of that test's coverage disappears.

A strict reader holds a table of the options that exist, and a pattern for each one's value:

my %KNOWN = (count => qr/^[1-9]\d*$/, seed => qr/^\d+$/, timeout => qr/^\d+$/);

Then every option is checked twice: is the key in the table, and does the value match its pattern? cuont=4 fails the first check. count=4x fails the second. count=0 fails too, because a test that runs zero times is almost certainly a mistake, and the pattern says so.

The cost of strictness is that adding a new option means adding a row to %KNOWN. That is a feature: the table is now the documentation of every option the runner supports, in one place.

7. Includes: Whose Directory, and Cycles

An include line has two problems that are easy to get wrong.

Relative to what?

include lists/long.list means "the file lists/long.list next to this list". It does not mean "relative to wherever the person happened to be when they started the runner". If the reader passes the path straight to open, it is resolved against the current directory, and the list only works when the runner is started from one particular place:

#!/usr/bin/perl
# include_where.pl -- one include line, resolved two ways, from another directory.
use strict;
use warnings;
use File::Basename qw(dirname);
use File::Spec;

chdir "lists" or die "lists: $!\n";               # start from somewhere else
my $list = "../tests.list";
my $inc  = "lists/long.list";                      # as written in tests.list

my $by_cwd  = $inc;
my $by_list = File::Spec->catfile(dirname($list), $inc);
printf "relative to the current directory: %-9s (%s)\n", (-f $by_cwd  ? "found" : "NOT FOUND"), $by_cwd;
printf "relative to the including list:    %-9s (%s)\n", (-f $by_list ? "found" : "NOT FOUND"), $by_list;

relative to the current directory: NOT FOUND (lists/long.list)
relative to the including list:    found     (../lists/long.list)

Resolve an include against the directory of the file that contains it. The full reader below does this with File::Spec->rel2abs($path, dirname($this_file)). Chapter 17.1 applies the same idea to whole scripts: never depend on the directory a user happens to start from.

A list that includes itself

Lists that include each other will, sooner or later, form a loop: a.list includes b.list, and someone adds include a.list to b.list. A reader with no guard follows the loop until something breaks. What breaks, and when, is worth seeing once:

#!/usr/bin/perl
# cycle_demo.pl -- an include reader with no cycle guard, given a cycle.
use strict;
use warnings;

mkdir "cyc";
open my $fa, ">", "cyc/a.list" or die; print {$fa} "include cyc/b.list\nsmoke\n";  close $fa;
open my $fb, ">", "cyc/b.list" or die; print {$fb} "include cyc/a.list\ncorner\n"; close $fb;

my $depth = 0;
sub read_list {
    my ($file) = @_;
    $depth++;
    open my $fh, "<", $file or die "$file: $!\n";
    my @t;
    while (my $l = <$fh>) {
        my ($name, @rest) = split ' ', $l;
        if ($name eq "include") { push @t, read_list($rest[0]); next }
        push @t, $name;
    }
    return @t;
}

my @warnings;
local $SIG{__WARN__} = sub { push @warnings, $_[0] };
eval { read_list("cyc/a.list") };
(my $err = $@) =~ s/\n//;
my ($why) = $err =~ /: (.*)$/;
print "warnings: ", scalar @warnings, " (", $warnings[0] =~ /^(Deep recursion)/, ")\n";
print "died with: $why\n";
print "open files when it died: ", ($depth > 1000 ? "more than 1000" : $depth), "\n";

warnings: 1 (Deep recursion)
died with: Too many open files
open files when it died: more than 1000

Perl warned about deep recursion once, then kept going, holding one open file for each level, until the operating system refused to open any more. On this machine that took thousands of levels — the exact number depends on the system's open-file limit, which is why the script prints only "more than 1000". In a runner that does not trap the error, this is a crash with a message about open files, which points everywhere except at the one-line mistake in b.list.

The guard is a hash of files currently being read. Before opening a file, check whether it is already in the hash; if it is, that is a cycle. Use local so each file is removed from the hash when its reading finishes:

if ($open_now{$abs}) { push @errors, "$from: include '$path' would read it again (a cycle)"; return }
local $open_now{$abs} = 1;

Two details matter. The key is the absolute path (Cwd::abs_path), because lists/../tests.list and tests.list are the same file under different names. And it is "currently being read", not "ever read": including a shared file from two different lists is not a cycle, and a guard that rejected it would forbid the main reason includes exist. (Whether the tests inside it are then duplicated is a separate check — see the next section.)

8. The Full Reader

Here is the reader that implements the format table exactly. Notice that it never dies on a problem in the list. It collects every problem, prints them all with their locations, and exits with status 2 before any simulation is launched:

#!/usr/bin/perl
# read_tests.pl -- read a test list into records, or report every problem.
# Usage: perl read_tests.pl [LIST]          (default: tests.list)
use strict;
use warnings;
use Cwd qw(abs_path);
use File::Basename qw(dirname);
use File::Spec;

my %KNOWN = (count => qr/^[1-9]\d*$/, seed => qr/^\d+$/, timeout => qr/^\d+$/);
my (@tests, @errors, %where, %open_now, $TOP);

sub read_list {
    my ($file, $from, $as_written) = @_;
    $as_written //= $file;
    my $abs = abs_path($file);
    if (!defined $abs || !-f $abs) {
        push @errors, "$from: '$as_written': no such file";
        return;
    }
    if ($open_now{$abs}) {                         # already being read: a cycle
        push @errors, "$from: include '$as_written' would read it again (a cycle)";
        return;
    }
    local $open_now{$abs} = 1;
    $TOP //= dirname($abs);                        # locations print relative to it
    my $show = File::Spec->abs2rel($abs, $TOP);
    open my $fh, "<", $abs or do { push @errors, "$from: $show: $!"; return };

    while (my $l = <$fh>) {
        my $at = "$show:$.";
        $l =~ s/\s+\z//;                           # newline, \r and trailing blanks
        $l =~ s/(?:^|\s+)#.*//;                    # comment: '#' at start or after space
        next unless $l =~ /\S/;

        my ($name, @words) = split " ", $l;
        if ($name eq "include") {
            if (@words != 1) { push @errors, "$at: include needs exactly one file"; next }
            my $inc = File::Spec->rel2abs($words[0], dirname($abs));
            read_list($inc, $at, $words[0]);
            next;
        }
        if ($name !~ /^\w+$/) { push @errors, "$at: bad test name '$name'"; next }

        my %t = (name => $name, opts => {}, plusargs => [], at => $at);
        for my $w (@words) {
            if ($w =~ /^\+/) { push @{ $t{plusargs} }, $w; next }
            my ($k, $v) = $w =~ /^(\w+)=(.*)$/
                or do { push @errors, "$at: '$w' is neither key=value nor +plusarg"; next };
            if (!$KNOWN{$k})        { push @errors, "$at: unknown option '$k'"; next }
            if ($v !~ $KNOWN{$k})   { push @errors, "$at: bad value '$v' for '$k'"; next }
            $t{opts}{$k} = $v;
        }
        if (my $first = $where{$name}) {
            push @errors, "$at: test '$name' already listed at $first";
            next;
        }
        $where{$name} = $at;
        push @tests, \%t;
    }
}

my $list = shift // "tests.list";
read_list($list, "command line");

if (@errors) {
    print STDERR "error: $_\n" for @errors;
    print STDERR scalar(@errors), " problem(s) in $list; nothing will be run\n";
    exit 2;
}
for my $t (@tests) {
    my $o = join " ", map { "$_=$t->{opts}{$_}" } sort keys %{ $t->{opts} };
    printf "%-8s %-18s %-12s %s\n", $t->{name}, $o || "-", "@{ $t->{plusargs} }" || "-", $t->{at};
}
print scalar(@tests), " test(s)\n";

smoke    -                  -            tests.list:7
corner   -                  -            tests.list:8
rand     count=4            +n=60        tests.list:9
stall    timeout=5          -            lists/long.list:2
rand_big count=2            +n=4000 +msg_tag=RB#1 lists/long.list:3
5 test(s)

All five tests, in list order, each with its options separated from its plusargs, and each with the file and line it came from. That last column costs one variable and pays for itself the first time somebody asks "where is this test coming from?" in a list with six levels of includes. Locations are printed relative to the top list's directory, so they are short and do not depend on where the runner was started.

Walk through the parts that do the work:

• $l =~ s/\s+\z// removes the newline and any \r or trailing spaces in one step. chomp removes only the newline. Section 10 shows why that difference matters.
• The comment rule is the one from Section 5. The plusarg +msg_tag=RB#1 arrives intact.
• $. is the line number of the file being read. It stays correct after an include returns, because reading the next line from the outer file makes the outer file's counter current again. The error test below checks this: the errors after the include line have the right numbers.
• Duplicates. %where remembers where each name was first seen. A second smoke is an error that names both places, because "already listed" without saying where sends someone searching through every included file.
• Records, not strings. Each test is a hash with named fields — name, opts, plusargs, at. Chapter 25.3 expands each record into runs, and it reads these fields by name.

9. Every Mistake at Once

Here is a list with six mistakes that people really make, including an include file that includes the list back:

# bad/tests.list -- six mistakes people really make
smoke
rand     cuont=4  +n=60
corner   count=4x
include  common.list
fifo-2
smoke    count=2
include  missing.list

# bad/common.list -- includes the file that includes it
include tests.list
stall

#!/usr/bin/perl
# check_bad.pl -- run the reader on the bad list and show what it says.
use strict;
use warnings;

open STDERR, ">&", \*STDOUT or die;               # show errors in order
$| = 1;
system($^X, "read_tests.pl", "bad/tests.list");
print "exit status: ", $? >> 8, "\n";

error: tests.list:3: unknown option 'cuont'
error: tests.list:4: bad value '4x' for 'count'
error: common.list:2: include 'tests.list' would read it again (a cycle)
error: tests.list:6: bad test name 'fifo-2'
error: tests.list:7: test 'smoke' already listed at tests.list:2
error: tests.list:8: 'missing.list': no such file
6 problem(s) in bad/tests.list; nothing will be run
exit status: 2

Six problems, six lines, each with a location, and exit status 2 — and nothing was launched. Compare that with the alternative: a reader that dies at the first problem makes the person fix one mistake, rerun, fix the next, rerun, six times. Collecting every error before stopping is a small change in the code and a large change in how it feels to use.

Notice also what is not in the output: there is no row for stall, although common.list lists it. When a file contains an error, the reader does not run anything, so whether stall was read correctly no longer matters. The point of exit status 2 is that the person fixes the list first.

10. Invisible Characters: a Debugging Walkthrough

Here is a bug report you will receive one day: "the list is fine, I can see the file is there, but the runner says the include is missing". The list was edited on Windows, so each line ends in \r\n instead of \n. A reader that uses chomp and a regex for the include line keeps the \r:

#!/usr/bin/perl
# crlf_demo.pl -- a list saved with Windows line endings.
use strict;
use warnings;

open my $w, ">", "win.list" or die;
print {$w} "smoke\r\ninclude lists/long.list\r\nrand count=4\r\n";
close $w;

open my $fh, "<", "win.list" or die;
while (my $l = <$fh>) {
    chomp $l;                                      # removes \n, keeps \r
    if ($l =~ /^include\s+(.+)$/) {
        my $path = $1;
        print "include: ", (-f $path ? "found" : "NOT FOUND"), "\n";
        (my $shown = $path) =~ s/\r/\\r/g;         # make the invisible visible
        print "  the path is really: '$shown'\n";
    }
    if ($l =~ /count=(\S+)/) {
        print "count times 2 = ", $1 * 2, "  (no warning: '4\\r' numifies to 4)\n";
    }
}

open STDERR, ">&", \*STDOUT or die; $| = 1;
system($^X, "read_tests.pl", "win.list");

include: NOT FOUND
  the path is really: 'lists/long.list\r'
count times 2 = 8  (no warning: '4\r' numifies to 4)
smoke    -                  -            win.list:1
stall    timeout=5          -            lists/long.list:2
rand_big count=2            +n=4000 +msg_tag=RB#1 lists/long.list:3
rand     count=4            -            win.list:3
4 test(s)

Three things to take from this:

The error message lies by omission. Printed normally, the path looks exactly like lists/long.list, because a carriage return is invisible on a terminal. The debugging step that solves it is to make invisible characters visible: substitute them with escapes before printing, as the script does, or look at the raw bytes with od -c win.list, which shows \r \n at the end of each line.

The damage is selective. The number survived: Perl converts "4\r" to 4 with no warning, because trailing whitespace is allowed in a number. The path did not survive, because to the file system a \r is just another character in a name. So a CRLF list fails on some lines and works on others, which is exactly the kind of bug that makes people doubt their own eyes.

read_tests.pl handled it, because it strips with s/\s+\z//, and \s includes \r. That one line is the fix. Strip all trailing whitespace from every line you read from a file that humans edit.

11. Values With Spaces

The format so far cannot express a plusarg whose value contains a space, such as a message filter +msg="link down". If your project needs that, the standard tool is shellwords from Text::ParseWords, which ships with Perl and splits a line the way a shell does, honouring quotes. It has one behaviour you must guard against:

#!/usr/bin/perl
# quotes_demo.pl -- shellwords on a good line and on an unclosed quote.
use strict;
use warnings;
use Text::ParseWords qw(shellwords);

for my $l ('rand +msg="link down" count=2', 'rand +msg="link down count=2') {
    my @w = shellwords($l);
    printf "%-32s -> %d word(s) %s\n", $l, scalar @w, join(" ", map { "[$_]" } @w);
}

rand +msg="link down" count=2    -> 3 word(s) [rand] [+msg=link down] [count=2]
rand +msg="link down count=2     -> 0 word(s)

With a missing closing quote, shellwords returns an empty list — no error, no warning. A reader that does my ($name, @words) = shellwords($l) gets an undefined name and, depending on the next line of code, either skips the test or crashes somewhere unrelated. Either way, a test silently leaves the regression. If you use shellwords, treat an empty result from a non-blank line as an error with the line's location:

my @w = shellwords($l);
push @errors, "$at: unbalanced quote" and next if !@w;

Also note that shellwords does not know about comments. Strip the comment first, with the rule from Section 5 — and if a quoted value may contain #, you now need a small tokenizer rather than a regex. That is the point at which to ask whether the list format should stay this simple. Often the better answer is to keep values free of spaces and let the bench accept a different spelling.

12. Exercises

1. Line numbers inside an include. In bad/tests.list, the cycle error was reported as common.list:2. Change the reader so that every error inside an included file also shows how it was reached, like common.list:2 (included from tests.list:5). Test it with a three-level include.

2. A shared file is not a cycle. Create a.list and b.list that both include common.list, and a top list that includes a.list and b.list. Run read_tests.pl. Is the result a cycle error, a duplicate-test error, or neither? Decide what the right behaviour is for your team, and justify it.

3. Prove the strict checks can fail. For each of the six error messages the reader can produce, write the smallest list line that triggers it. Then delete each check from the reader in turn and confirm that the corresponding line now changes the output silently. A check that never fires has not been tested.

4. A -tag filter. Add tags to the format — for example tags=fast,alu — and a command-line option that selects only tests with a given tag. Where must the tag check happen, before or after duplicate detection? What should happen if no test matches the tag?

5. Your team's format. Find a test list from a real project, or design one. Write its format table in the style of Section 2, including the "anything else is an error" row. Then list three lines a person might type that your current reader would accept silently but interpret wrongly.

13. What To Remember

• Write the format down first, at the top of the list, and make "anything else is an error" one of its rules.
• Measured: the usual four-line reader made a test called include, gave corner eight options from its comment, and lost two tests — with use warnings on and nothing printed.
• A lost test produces no failure. The regression just gets smaller, and smaller regressions pass more often.
• Use split ' '. split / / makes empty fields and ignores tabs; split /\s+/ makes an empty first field on indented lines.
• A comment is # at the start or after whitespace. s/#.*// damaged a plusarg that contained #.
• Strict options: a table of known keys and a pattern per value. A typo like cuont=4 otherwise runs one seed instead of four.
• Resolve includes against the including file's directory, and guard cycles with a hash of files currently being read, keyed by absolute path.
• Collect every error with its file and line, then exit 2 before launching anything.
• Strip trailing whitespace, not just the newline. A Windows \r broke a path and left a number unharmed.
• shellwords returns an empty list on an unclosed quote. Treat that as an error.

Chapter 25.3 takes the records this reader produces and expands each one into the runs a regression actually launches: how many, with which seeds, and with which plusargs — chosen so that any failure can be reproduced from the report alone.