Perl Part II · Module 29 · Practical VLSI Automation Projects
Project: Coverage Report Parser
Every verification team reads a coverage roll-up each morning, and most of the ways it goes wrong are invisible in the numbers it prints: an average of runs instead of a merge, a bin that only the sum of several runs can close, a run from last month's coverage model mixed into this month's, a crashed run whose report stopped half-way, a seed counted twice, and a percentage that rounds up to 100. This project builds the merger that gets each of those right, runs it on real simulation results, and proves its tests can tell.
Advanced24 min readPerlPart IIProjectsCoverage
Part II · Module 29 · Chapter 29.4 · Project: Coverage Report Parser
1. The Brief
| Command | parse_coverage.pl REPORT... [--holes] [--csv FILE] [--goal PCT] |
| Reads | one functional coverage report per run |
| Writes | STDOUT: per coverpoint, the merged result beside the average and best single run; bins covered only by merging; with --holes, every unhit bin by name. --csv: one row per bin with its merged hits |
| Exit 0 | the roll-up was written |
| Exit 1 | written, but below --goal |
| Exit 2 | an input problem — a report cut off, a different coverage model, a run counted twice, a report whose own summary disagrees with its bins, a line the tool does not know — nothing written |
| Exit 3 | usage error |
Exit 1 is the one a nightly flow gates closure on, and it is judged on exact counts. Exit 2 outranks it, for the reason Chapter 29.3 gave: a roll-up built from inputs that cannot be trusted cannot certify anything.
The tool is built from pieces earlier modules measured:
- Counting with a hash — merging is
$hits{$bin} += $nacross runs (6.8). - Truncated, empty and missing logs — a report that stops half-way is its own state, not a short report (18.9).
- Did the test even finish? — the end marker decides whether anything else can be trusted (21.2).
- Parsing simulator coverage output and coverage closure roll-ups — Module 24's treatment of the same data, which this project assembles into one tool (24.3, 24.5).
The tool states its definitions rather than borrowing them. A bin is covered when its merged hits reach its at_least. A coverpoint's figure is covered bins over bins; the total is weighted by bins. Commercial tools may weight coverpoints differently, so the CSV carries the raw counts and any score can be recomputed from it.
2. Real Runs, In A Labelled Layout
Icarus Verilog does not support covergroups. So the testbench below does what a covergroup does, by hand: while it drives random operations into a small ALU, it counts every bin itself, and at the end it prints a report shaped like a covergroup report. The hit counts are real — each seed is a genuine simulation of the ALU. The layout was written for this chapter. It is not the layout of any coverage tool, and real tools' layouts differ from it and from each other; Chapter 21.1 treated simulator logs the same way.
// tb_alu_cov.v -- random stimulus for a small ALU, with FUNCTIONAL COVERAGE
// counted procedurally and printed as a covergroup-style report.
//
// Icarus Verilog does not support covergroups, so this testbench counts each
// bin itself. The hit counts are therefore REAL results of simulating the
// ALU with each seed. The report LAYOUT is written for this chapter; it is
// shaped like a covergroup report but it is not any vendor's format.
//
// +SEED=n stimulus seed +N=n number of operations
// +CRASH stop half-way through printing the report
// compile with -DOLD_MODEL for last month's model, which had no SHL bin
`timescale 1ns/1ps
module alu (input [2:0] op, input [7:0] a, b, output reg [7:0] y, output reg carry);
always @* begin
carry = 1'b0;
case (op)
3'd0: {carry, y} = a + b;
3'd1: y = a - b;
3'd2: y = a & b;
3'd3: y = a | b;
3'd4: y = a ^ b;
3'd5: y = a << 1;
default: y = 8'h00;
endcase
end
endmodule
module tb_alu_cov;
reg [2:0] op;
reg [7:0] a, b;
wire [7:0] y;
wire carry;
integer seed, seed0, n, i, r;
// cp_op: ADD SUB AND OR XOR SHL, and RESERVED (6,7) as an ignore bin
integer op_hits [0:7];
// cp_a: zero, low 1-15, mid 16-239, high 240-254, max 255 (max needs 3 hits)
integer a_hits [0:4];
// cp_carry: no / yes, sampled on ADD only
integer c_hits [0:1];
alu dut (.op(op), .a(a), .b(b), .y(y), .carry(carry));
task bin_line(input [8*10-1:0] name, input integer hits, input integer need);
$display(" bin %-10s hits %4d at_least %0d", name, hits, need);
endtask
task cp_header(input [8*10-1:0] name, input integer covered, input integer total);
$display(" coverpoint %0s %0d/%0d bins %0.2f%%", name, covered, total,
100.0 * covered / total);
endtask
integer cov, tot;
initial begin
if (!$value$plusargs("SEED=%d", seed)) seed = 1;
if (!$value$plusargs("N=%d", n)) n = 24;
seed0 = seed; // $random(seed) updates seed as it goes
for (i = 0; i < 8; i = i + 1) op_hits[i] = 0;
for (i = 0; i < 5; i = i + 1) a_hits[i] = 0;
c_hits[0] = 0; c_hits[1] = 0;
// The first draws from a small seed have near-zero high bits, so the
// first operation would be SHL in every run. Discard a few first.
for (i = 0; i < 8; i = i + 1) r = $random(seed);
for (i = 0; i < n; i = i + 1) begin
r = $random(seed);
// SHL is rare: 1 operation in 64 asks for it; RESERVED codes 1 in 64
case (r[30:25]) // high bits: $random's low bits are weak
6'd0: op = 3'd5;
6'd1: op = 3'd6;
default: op = r[24:22] % 5;
endcase
r = $random(seed);
case (r[30:25]) // operand a: extremes are rare
6'd0: a = 8'd0;
6'd1: a = 8'd255;
6'd2: a = 8'd240 + r[24:21];
6'd3: a = 8'd1 + r[24:21];
default: a = 8'd16 + (r[20:13] % 224);
endcase
b = $random(seed);
#1;
op_hits[op] = op_hits[op] + 1;
if (a == 0) a_hits[0] = a_hits[0] + 1;
else if (a < 16) a_hits[1] = a_hits[1] + 1;
else if (a < 240) a_hits[2] = a_hits[2] + 1;
else if (a < 255) a_hits[3] = a_hits[3] + 1;
else a_hits[4] = a_hits[4] + 1;
if (op == 3'd0) c_hits[carry] = c_hits[carry] + 1;
end
$display("=== FUNCTIONAL COVERAGE ===");
$display("test: alu_random seed: %0d operations: %0d", seed0, n);
$display("covergroup alu_cg");
`ifdef OLD_MODEL
cov = (op_hits[0]>0) + (op_hits[1]>0) + (op_hits[2]>0) + (op_hits[3]>0) + (op_hits[4]>0);
cp_header("cp_op", cov, 5);
`else
cov = (op_hits[0]>0) + (op_hits[1]>0) + (op_hits[2]>0) + (op_hits[3]>0) + (op_hits[4]>0) + (op_hits[5]>0);
cp_header("cp_op", cov, 6);
`endif
bin_line("ADD", op_hits[0], 1);
bin_line("SUB", op_hits[1], 1);
bin_line("AND", op_hits[2], 1);
bin_line("OR", op_hits[3], 1);
bin_line("XOR", op_hits[4], 1);
`ifndef OLD_MODEL
bin_line("SHL", op_hits[5], 1);
`endif
$display(" ignore %-7s hits %4d", "RESERVED", op_hits[6] + op_hits[7]);
if ($test$plusargs("CRASH")) $finish;
cov = (a_hits[0]>0) + (a_hits[1]>0) + (a_hits[2]>0) + (a_hits[3]>0) + (a_hits[4]>=3);
cp_header("cp_a", cov, 5);
bin_line("zero", a_hits[0], 1);
bin_line("low", a_hits[1], 1);
bin_line("mid", a_hits[2], 1);
bin_line("high", a_hits[3], 1);
bin_line("max", a_hits[4], 3);
cov = (c_hits[0]>0) + (c_hits[1]>0);
cp_header("cp_carry", cov, 2);
bin_line("no", c_hits[0], 1);
bin_line("yes", c_hits[1], 1);
$display("=== END COVERAGE ===");
$finish;
end
endmoduleThe model has three coverpoints and thirteen bins. Three details are there to test the merger:
SHLand the extreme values ofaare rare — about one operation in 64 each — so single runs miss them and a merge is needed.maxneeds three hits (at_least 3). No single one of the six seeds reaches that, but two of them together do.- About one operation in 64 uses a reserved opcode, recorded as an
ignorebin: reported, never counted.
The testbench comment records a detour worth knowing. The first draws from a small $random seed have high bits near zero, so in the first version every run's first operation was SHL — measured on 12 seeds, 12 of 12 — and SHL appeared in every run, where about 31% of runs were expected to contain it. The long-run rate was correct (88 in 6,400). Discarding the first eight draws fixed it, and the lesson is general: check that random stimulus is random at the start of a run, not only on average.
make_runs.sh simulates six seeds, keeping each report where a regression would, and builds the two damaged runs used in section 7: one compiled with -DOLD_MODEL, last month's model without the SHL bin, and one stopped half-way through printing its report.
#!/bin/bash
# make_runs.sh DIR -- simulate the ALU once per seed and keep each run's
# coverage report where a regression would: DIR/seed_N/cov.rpt.
# Also writes two damaged runs under DIR-bad/ for section 7.
set -u
out=${1:?usage: make_runs.sh DIR}
iverilog -o cov.vvp tb/tb_alu_cov.v || exit 1
iverilog -DOLD_MODEL -o cov_old.vvp tb/tb_alu_cov.v || exit 1
for seed in 1 2 3 4 5 6; do
mkdir -p "$out/seed_$seed"
vvp -n cov.vvp +SEED=$seed > "$out/seed_$seed/cov.rpt"
done
mkdir -p "$out-bad/old_model" "$out-bad/crashed"
vvp -n cov_old.vvp +SEED=7 > "$out-bad/old_model/cov.rpt"
vvp -n cov.vvp +SEED=8 +CRASH > "$out-bad/crashed/cov.rpt"
rm -f cov.vvp cov_old.vvp3. The Tool
#!/usr/bin/perl
# parse_coverage.pl -- merge per-run functional coverage reports into one
# closure roll-up: per coverpoint, the merged result beside what single runs
# achieved, the bins still unhit by name, and an optional CSV.
use strict;
use warnings;
use Getopt::Long qw(GetOptions);
use POSIX qw(floor);
use constant { OK => 0, BELOW_GOAL => 1, INPUT => 2, USAGE => 3 };
my %opt;
GetOptions(\%opt, "holes", "csv=s", "goal=f", "help") or usage("bad option");
usage() if $opt{help};
usage("give at least one coverage report") unless @ARGV;
usage("--goal must be between 0 and 100")
if defined $opt{goal} && ($opt{goal} < 0 || $opt{goal} > 100);
# ---- read every run; a run is a list of bins plus who it was
my (@runs, @problems);
for my $file (@ARGV) {
my $run = read_report($file);
push @runs, $run if $run;
}
# ---- every run must describe the same coverage model
if (@runs) {
my $ref = $runs[0];
for my $run (@runs[1 .. $#runs]) {
my @missing = grep { !exists $run->{bins}{$_} } sort keys %{ $ref->{bins} };
my @extra = grep { !exists $ref->{bins}{$_} } sort keys %{ $run->{bins} };
my @need = grep { exists $ref->{bins}{$_}
&& $run->{bins}{$_}{at_least} != $ref->{bins}{$_}{at_least} }
sort keys %{ $run->{bins} };
push @problems, "$run->{file}: different coverage model from $ref->{file}: "
. join("; ", (@missing ? "no bin " . join(", ", @missing) : ()),
(@extra ? "extra bin " . join(", ", @extra) : ()),
(@need ? "different at_least for " . join(", ", @need) : ()))
if @missing || @extra || @need;
}
}
# ---- and no run may be counted twice
my %seen_run;
for my $run (@runs) {
my $id = "$run->{test} seed $run->{seed}";
if (my $first = $seen_run{$id}) { push @problems, "$run->{file}: $id was already read from $first" }
else { $seen_run{$id} = $run->{file} }
}
push @problems, "no readable coverage report" if !@runs && !@problems;
if (@problems) {
print STDERR "parse_coverage.pl: $_\n" for @problems;
fail(INPUT, scalar(@problems) . " input problem(s); nothing written");
}
# ---- merge: hits add up across runs; a bin is covered when the SUM reaches
# ---- at_least, which can happen although no single run got there
my (%merged, @order);
for my $key (@{ $runs[0]{order} }) {
my %m = (%{ $runs[0]{bins}{$key} }, hits => 0, runs_hitting => 0, best_run => 0);
for my $run (@runs) {
my $h = $run->{bins}{$key}{hits};
$m{hits} += $h;
$m{runs_hitting}++ if $h > 0;
$m{best_run} = $h if $h > $m{best_run};
}
$m{covered} = $m{hits} >= $m{at_least} ? 1 : 0;
$merged{$key} = \%m;
push @order, $key;
}
# ---- the roll-up, per coverpoint
my (%cp_bins, %cp_cov, @cps);
for my $key (@order) {
my $cp = $merged{$key}{cp};
push @cps, $cp unless $cp_bins{$cp}++;
$cp_cov{$cp} += $merged{$key}{covered};
}
my @seeds = sort { $a <=> $b } map { $_->{seed} } @runs;
printf "merged %d run(s) of %s (seeds %s): %d coverpoint(s), %d bin(s)\n",
scalar @runs, $runs[0]{test}, join(",", @seeds), scalar @cps, scalar @order;
printf "\n%-18s %5s %8s %8s %9s %9s\n", "COVERPOINT", "BINS", "COVERED", "MERGED", "AVG RUN", "BEST RUN";
my ($all_bins, $all_cov) = (0, 0);
for my $cp (@cps) {
my @per_run = map { run_pct($_, $cp) } @runs;
my $avg = 0;
$avg += $_ / @per_run for @per_run;
my ($best) = sort { $b <=> $a } @per_run;
printf "%-18s %5d %8d %8s %9s %9s\n", $cp, $cp_bins{$cp}, $cp_cov{$cp},
pct($cp_cov{$cp}, $cp_bins{$cp}), pct1($avg), pct1($best);
$all_bins += $cp_bins{$cp};
$all_cov += $cp_cov{$cp};
}
printf "%-18s %5d %8d %8s\n", "TOTAL (bins)", $all_bins, $all_cov, pct($all_cov, $all_bins);
my @by_merge = grep { $merged{$_}{covered} && $merged{$_}{best_run} < $merged{$_}{at_least} } @order;
if (@by_merge) {
print "\ncovered only by merging (no single run reached at_least):\n";
printf " %-22s %d hits over %d run(s), best single run %d, at_least %d\n",
$_, @{ $merged{$_} }{qw(hits runs_hitting best_run at_least)} for @by_merge;
}
if ($opt{holes}) {
my @holes = grep { !$merged{$_}{covered} } @order;
printf "\nholes: %d bin(s) below at_least after merging\n", scalar @holes;
printf " %-22s hits %d of %d needed, hit in %d of %d run(s)\n",
$_, @{ $merged{$_} }{qw(hits at_least runs_hitting)}, scalar @runs for @holes;
}
if ($opt{csv}) {
my $tmp = "$opt{csv}.tmp.$$";
open my $out, ">", $tmp or fail(USAGE, "cannot write $tmp: $!");
print {$out} "group,coverpoint,bin,at_least,hits,runs_hitting,covered\n";
for my $key (@order) {
my $m = $merged{$key};
print {$out} join(",", @{$m}{qw(group point bin at_least hits runs_hitting)},
$m->{covered} ? "yes" : "no"), "\n";
}
close $out or fail(USAGE, "cannot write $tmp: $!");
rename $tmp, $opt{csv} or fail(USAGE, "cannot rename $tmp: $!");
}
# the goal is judged on exact counts, never on a rounded percentage
if (defined $opt{goal}) {
my $met = $all_cov * 100 >= $opt{goal} * $all_bins;
printf "\ngoal %s%%: %s\n", $opt{goal}, $met ? "met" : "NOT met";
exit BELOW_GOAL unless $met;
}
exit OK;
# ---------------------------------------------------------------- helpers
# A percentage for a closure report: rounded DOWN, so that 1999 of 2000
# bins can never be printed as 100.0%. From the counts, in whole tenths.
sub pct {
my ($n, $d) = @_;
my $tenths = int(1000 * $n / $d);
return sprintf "%d.%d%%", int($tenths / 10), $tenths % 10;
}
# The same rule for a value that is already a percentage (an average).
sub pct1 { return sprintf "%.1f%%", floor($_[0] * 10) / 10 }
# One run's percentage for one coverpoint, computed from its own bins.
sub run_pct {
my ($run, $cp) = @_;
my @keys = grep { $run->{bins}{$_}{cp} eq $cp } @{ $run->{order} };
my $cov = grep { $run->{bins}{$_}{hits} >= $run->{bins}{$_}{at_least} } @keys;
return 100 * $cov / @keys;
}
sub read_report {
my ($file) = @_;
open my $fh, "<", $file or do { push @problems, "$file: $!"; return };
my %run = (file => $file, bins => {}, order => []);
my ($in, $ended, $group, $cp, $point, %printed) = (0, 0);
while (my $line = <$fh>) {
if ($line =~ /^=== FUNCTIONAL COVERAGE ===/) { $in = 1; next }
next unless $in;
if ($line =~ /^=== END COVERAGE ===/) { $ended = 1; last }
if ($line =~ /^test: (\S+)\s+seed: (-?\d+)/) { @run{qw(test seed)} = ($1, $2) }
elsif ($line =~ /^covergroup (\w+)/) { $group = $1 }
elsif ($line =~ /^\s+coverpoint (\w+)\s+(\d+)\/(\d+) bins/) {
($point, $cp) = ($1, "$group.$1");
$printed{$cp} = [ $2, $3 ];
}
elsif ($line =~ /^\s+bin (\w+)\s+hits\s+(\d+)\s+at_least (\d+)/) {
my $key = "$cp.$1";
$run{bins}{$key} = { group => $group, cp => $cp, point => $point,
bin => $1, hits => $2, at_least => $3 };
push @{ $run{order} }, $key;
}
elsif ($line =~ /^\s+ignore \w+/) { } # excluded from coverage by definition
else { push @problems, "$file line $.: unrecognised line in the coverage section" }
}
close $fh;
unless ($in) { push @problems, "$file: not a coverage report (no start marker)"; return }
unless ($ended) { push @problems, "$file: report cut off (no END COVERAGE line)"; return }
unless (defined $run{test}) { push @problems, "$file: no test/seed line"; return }
# the report's own summary must agree with its bins
for my $c (sort keys %printed) {
my @keys = grep { $run{bins}{$_}{cp} eq $c } @{ $run{order} };
my $cov = grep { $run{bins}{$_}{hits} >= $run{bins}{$_}{at_least} } @keys;
my ($p_cov, $p_tot) = @{ $printed{$c} };
push @problems, "$file: $c says $p_cov/$p_tot bins, its bins give $cov/" . scalar @keys
if $cov != $p_cov || @keys != $p_tot;
}
return \%run;
}
sub usage {
my ($why) = @_;
print STDERR "parse_coverage.pl: $why\n" if $why;
print STDERR <<'END';
usage: parse_coverage.pl REPORT... [--holes] [--csv FILE] [--goal PCT]
exit: 0 written; 1 written, but --goal not met; 2 input problem
(nothing written); 3 usage
END
exit($why ? USAGE : OK);
}
sub fail {
my ($code, $msg) = @_;
unlink $opt{csv} if defined $opt{csv};
print STDERR "parse_coverage.pl: $msg\n";
exit $code;
}A merge sums hits; it does not average percentages
$m{covered} = $m{hits} >= $m{at_least} ? 1 : 0;The merged hit count is the sum over runs, and coverage is decided on the sum. That is the only definition that answers the question the team is asking — has this behaviour been exercised enough, anywhere? — and it has a consequence an average can never show: a bin with at_least 3 can be covered by two runs with 2 and 1 hits, although neither run covered it. The tool lists those bins separately, because a closure that depends on a merge is a closure nobody should lose by trimming the regression.
Every run must describe the same model
The first run's bin list is the reference, and every other run must have the same bins with the same at_least. A run from last month's model, missing a bin, would otherwise shrink or grow the denominator without any error — and the merged figure would describe neither model. The message names the bins that differ.
A run may be counted once
The test and seed line identifies a run. A second report with the same identity is refused, because a copied rerun adds its hits again, and for an at_least bin that can manufacture coverage. Section 7 measures exactly that.
The report's own summary is checked against its bins
Each coverpoint header says how many bins it covers. The tool recomputes that from the bins it read and refuses the report if the two disagree. That catches a report edited by hand, and — more often — a parser bug: if the tool ever misreads a line, its own count stops matching the tool's.
Lines it does not recognise are refused
Inside the coverage section, every line must be a known shape. A cross line from a newer tool version is refused rather than skipped, which is Chapter 29.2's whitelist again: a skipped line is a silently smaller report.
A closure percentage is rounded down, from the counts
sub pct {
my ($n, $d) = @_;
my $tenths = int(1000 * $n / $d);
return sprintf "%d.%d%%", int($tenths / 10), $tenths % 10;
}The next section measures why. And the goal is never judged on a printed figure at all — $all_cov * 100 >= $opt{goal} * $all_bins compares the counts directly.
4. Running It
#!/bin/bash
# demo.sh -- simulate six seeds, show one run's report, then merge all six.
bash make_runs.sh runs || exit 1
ls runs
echo
echo "== runs/seed_4/cov.rpt"
cat runs/seed_4/cov.rpt
echo
perl parse_coverage.pl runs/*/cov.rpt --holes --csv coverage.csv
echo "exit $?"
echo
cat coverage.csvseed_1
seed_2
seed_3
seed_4
seed_5
seed_6
== runs/seed_4/cov.rpt
=== FUNCTIONAL COVERAGE ===
test: alu_random seed: 4 operations: 24
covergroup alu_cg
coverpoint cp_op 5/6 bins 83.33%
bin ADD hits 4 at_least 1
bin SUB hits 6 at_least 1
bin AND hits 7 at_least 1
bin OR hits 3 at_least 1
bin XOR hits 4 at_least 1
bin SHL hits 0 at_least 1
ignore RESERVED hits 0
coverpoint cp_a 3/5 bins 60.00%
bin zero hits 1 at_least 1
bin low hits 0 at_least 1
bin mid hits 20 at_least 1
bin high hits 1 at_least 1
bin max hits 2 at_least 3
coverpoint cp_carry 2/2 bins 100.00%
bin no hits 1 at_least 1
bin yes hits 3 at_least 1
=== END COVERAGE ===
tb/tb_alu_cov.v:124: $finish called at 24000 (1ps)
merged 6 run(s) of alu_random (seeds 1,2,3,4,5,6): 3 coverpoint(s), 13 bin(s)
COVERPOINT BINS COVERED MERGED AVG RUN BEST RUN
alu_cg.cp_op 6 6 100.0% 86.1% 100.0%
alu_cg.cp_a 5 4 80.0% 26.6% 60.0%
alu_cg.cp_carry 2 2 100.0% 100.0% 100.0%
TOTAL (bins) 13 12 92.3%
covered only by merging (no single run reached at_least):
alu_cg.cp_a.max 3 hits over 2 run(s), best single run 2, at_least 3
holes: 1 bin(s) below at_least after merging
alu_cg.cp_a.low hits 0 of 1 needed, hit in 0 of 6 run(s)
exit 0
group,coverpoint,bin,at_least,hits,runs_hitting,covered
alu_cg,cp_op,ADD,1,38,6,yes
alu_cg,cp_op,SUB,1,35,6,yes
alu_cg,cp_op,AND,1,32,6,yes
alu_cg,cp_op,OR,1,23,6,yes
alu_cg,cp_op,XOR,1,12,5,yes
alu_cg,cp_op,SHL,1,3,2,yes
alu_cg,cp_a,zero,1,1,1,yes
alu_cg,cp_a,low,1,0,0,no
alu_cg,cp_a,mid,1,139,6,yes
alu_cg,cp_a,high,1,1,1,yes
alu_cg,cp_a,max,3,3,2,yes
alu_cg,cp_carry,no,1,19,6,yes
alu_cg,cp_carry,yes,1,19,6,yesSeed 4's own report shows 10 of 13 bins. Merged, the six seeds cover 12 of 13, and the table puts the merge beside what single runs achieved. For cp_a the merge is 80.0%, the best single run 60.0%, and the average of the six runs 26.6%.
Those three numbers answer different questions, and confusing them is the commonest coverage-reporting error. The average says how much a typical run covers — useful for tuning stimulus, useless for closure. The best run says what one seed can do alone. Only the merge says what the regression as a whole has exercised. A dashboard that averages per-run percentages reports cp_a at a third of its real closure.
max is listed as covered only by merging: three hits over two runs, where the best single run had two. And the one hole is named — cp_a.low, hit in none of the six runs — which is the line a verification engineer acts on: write a test, or a constraint, that drives a into 1 to 15.
5. A Percentage Must Never Round Up
#!/bin/bash
# rounding.sh -- how a closure percentage can claim more than it has.
perl -e '
use strict; use warnings;
for my $c ([12, 13], [1999, 2000], [19999, 20000], [2000, 2000]) {
my ($covered, $bins) = @$c;
my $tenths = int(1000 * $covered / $bins); # rounded down
printf "%5d of %5d bins: %%.1f prints %-7s rounded down: %d.%d%%\n",
$covered, $bins, sprintf("%.1f%%", 100 * $covered / $bins),
int($tenths / 10), $tenths % 10;
}
' 12 of 13 bins: %.1f prints 92.3% rounded down: 92.3%
1999 of 2000 bins: %.1f prints 100.0% rounded down: 99.9%
19999 of 20000 bins: %.1f prints 100.0% rounded down: 99.9%
2000 of 2000 bins: %.1f prints 100.0% rounded down: 100.0%%.1f rounds to the nearest tenth, so 1,999 of 2,000 bins prints as 100.0%, and so does 19,999 of 20,000. On a closure report, 100.0% is read as "done", and one unhit bin is exactly the thing the report exists to show. Rounding down can only understate, and an understatement sends someone to look. The tool rounds down and also judges --goal on counts, so even a goal of 100 cannot be met by a rounded figure; test 9 checks both on a 2,000-bin report.
6. Why Averages And Single Runs Mislead
The table makes the point numerically; the reasoning is worth stating once:
- An average treats a run that covered nothing new the same as one that closed the last hole. It measures stimulus quality, not closure.
- A single run's figure ignores everything the others did. It is the right number for "which seed should I rerun to reproduce this bin", and nothing else.
- A merge is the union of what was exercised, with hits summed so that
at_leastmeans "at least this many times across the regression".
The CSV keeps runs_hitting beside hits for the case a merge hides: a bin hit 19 times by one run and never by the other five is covered, but its coverage depends on a single seed. That is a fact a reviewer may want, so the tool writes it down.
7. Every Input Problem
#!/bin/bash
# inputs.sh -- the goal, then one damaged input per case.
bash make_runs.sh runs > /dev/null || exit 1
mkdir -p extra
try() { # LABEL ARGS...
local label=$1; shift
perl parse_coverage.pl "$@" --csv out.csv > /dev/null 2> err.txt
local rc=$?
printf '== %s\n exit %d, csv written: %s\n' "$label" "$rc" "$([ -e out.csv ] && echo yes || echo no)"
grep -v -E 'nothing written|^usage:|^exit:|^ ' err.txt | sed 's/^/ /'
rm -f out.csv
}
try "six runs, no goal" runs/*/cov.rpt
try "six runs, --goal 90" runs/*/cov.rpt --goal 90
try "six runs, --goal 95" runs/*/cov.rpt --goal 95
try "plus last month's model" runs/*/cov.rpt runs-bad/old_model/cov.rpt
try "plus a crashed run" runs/*/cov.rpt runs-bad/crashed/cov.rpt
cp runs/seed_4/cov.rpt extra/seed_4_rerun.rpt
try "seed 4 copied as a rerun" runs/*/cov.rpt extra/seed_4_rerun.rpt
sed 's/bin SHL hits 2/bin SHL hits 0/' runs/seed_1/cov.rpt > extra/edited.rpt
try "a hit count edited by hand" extra/edited.rpt runs/seed_[2-6]/cov.rpt
printf 'UVM_INFO @ 0: reporter [RNTST] Running test\n' > extra/sim.log
try "a simulation log" runs/*/cov.rpt extra/sim.log
sed 's/^ coverpoint cp_carry/ cross cp_op_x_cp_a 0\/30 bins\n&/' runs/seed_1/cov.rpt > extra/cross.rpt
try "a line the tool does not know" extra/cross.rpt runs/seed_[2-6]/cov.rpt
try "no reports at all"
echo
echo "why a run must not be counted twice -- the max bin needs 3 hits:"
for f in runs/seed_2/cov.rpt runs/seed_3/cov.rpt runs/seed_4/cov.rpt; do
printf ' %-20s max hits %s\n' "$f" "$(awk '/bin max/ {print $4}' "$f")"
done
perl parse_coverage.pl runs/seed_[234]/cov.rpt --holes | grep 'cp_a.max'
echo " counting seed 4 twice would make it 2 + 2 = 4 hits: covered, by a copy"== six runs, no goal
exit 0, csv written: yes
== six runs, --goal 90
exit 0, csv written: yes
== six runs, --goal 95
exit 1, csv written: yes
== plus last month's model
exit 2, csv written: no
parse_coverage.pl: runs-bad/old_model/cov.rpt: different coverage model from runs/seed_1/cov.rpt: no bin alu_cg.cp_op.SHL
== plus a crashed run
exit 2, csv written: no
parse_coverage.pl: runs-bad/crashed/cov.rpt line 12: unrecognised line in the coverage section
parse_coverage.pl: runs-bad/crashed/cov.rpt: report cut off (no END COVERAGE line)
== seed 4 copied as a rerun
exit 2, csv written: no
parse_coverage.pl: extra/seed_4_rerun.rpt: alu_random seed 4 was already read from runs/seed_4/cov.rpt
== a hit count edited by hand
exit 2, csv written: no
parse_coverage.pl: extra/edited.rpt: alu_cg.cp_op says 6/6 bins, its bins give 5/6
== a simulation log
exit 2, csv written: no
parse_coverage.pl: extra/sim.log: not a coverage report (no start marker)
== a line the tool does not know
exit 2, csv written: no
parse_coverage.pl: extra/cross.rpt line 18: unrecognised line in the coverage section
== no reports at all
exit 3, csv written: no
parse_coverage.pl: give at least one coverage report
why a run must not be counted twice -- the max bin needs 3 hits:
runs/seed_2/cov.rpt max hits 0
runs/seed_3/cov.rpt max hits 0
runs/seed_4/cov.rpt max hits 2
alu_cg.cp_a.max hits 2 of 3 needed, hit in 1 of 3 run(s)
counting seed 4 twice would make it 2 + 2 = 4 hits: covered, by a copyTen cases. The goal works in both directions — 92.3% meets 90 and misses 95, and the CSV is still written in both — and the six input problems each stop the run with a reason and no CSV.
The crashed run produces two messages, and the order matters for reading them. The first is a line the tool did not recognise: it is the simulator's own $finish called line, appearing where a coverpoint was expected. The second is the cause: no END COVERAGE line. A report that stops half-way is missing coverpoints, and merging it would either shrink the denominator or — if its coverpoints were simply absent — hide the fact that the run contributed nothing to them.
The edited report changed one hit count from 2 to 0 and nothing else. Its own header still says 6/6 bins, its bins now give 5/6, and the tool refuses it. The same check is what catches the tool's own misreadings.
The last block measures what the duplicate check prevents. Over seeds 2, 3 and 4, max has two hits — a genuine hole, two of three. Seed 4 is the only run that hit it. Counted twice, it would contribute four hits, and the bin would be reported covered by nothing but a copied file.
8. A Test Suite For The Tool
#!/usr/bin/perl
# t/parse_coverage.t -- black-box tests over real simulation runs.
use strict;
use warnings;
use Test::More;
use File::Temp qw(tempdir);
use FindBin qw($Bin);
my $TOOL = "$Bin/../parse_coverage.pl";
my $base = tempdir(CLEANUP => 1);
chdir "$Bin/.." or die;
system("bash", "make_runs.sh", "$base/runs") == 0 or die "make_runs.sh failed\n";
my @SIX = sort glob("$base/runs/seed_*/cov.rpt");
sub slurp { open my $fh, "<", $_[0] or return undef; local $/; return scalar <$fh> }
sub spew { open my $fh, ">", $_[0] or die "$_[0]: $!"; print {$fh} $_[1]; close $fh }
sub run_tool { # returns (status, stdout, stderr); the CSV goes to $base/out.csv
my @args = @_;
unlink "$base/out.csv";
my $pid = fork // die "fork: $!";
if ($pid == 0) {
open STDOUT, ">", "$base/stdout" or die;
open STDERR, ">", "$base/stderr" or die;
exec $^X, $TOOL, @args, "--csv", "$base/out.csv";
die "exec: $!";
}
waitpid $pid, 0;
return ($? >> 8, slurp("$base/stdout") // "", slurp("$base/stderr") // "");
}
# ---- six real runs
my ($rc, $out) = run_tool(@SIX, "--holes");
is($rc, 0, "six runs, no goal: exit 0");
like($out, qr/^TOTAL \(bins\)\s+13\s+12\s+92\.3%$/m, "12 of 13 bins covered after merging");
# merged hits must equal the sum over the six reports, counted here
# independently with the test's own pattern
my %sum;
for my $f (@SIX) {
my $cp;
for (split /\n/, slurp($f)) {
$cp = $1 if /^\s+coverpoint (\w+)/;
$sum{"$cp,$1"} += $2 if /^\s+bin (\w+)\s+hits\s+(\d+)/;
}
}
my %csv = map { my @f = split /,/; ("$f[1],$f[2]" => $f[4]) }
grep { !/^group,/ } split /\n/, slurp("$base/out.csv");
is_deeply(\%csv, \%sum, "every merged hit count equals the sum over the six reports");
like($out, qr/^covered only by merging.*\n alu_cg\.cp_a\.max\s+3 hits over 2 run\(s\), best single run 2/m,
"max (at_least 3) is covered only by adding runs together");
my ($holes) = $out =~ /^holes: (\d+) bin/m;
ok($holes == 1 && $out =~ /^ alu_cg\.cp_a\.low\s+hits 0 of 1 needed/m, "the one hole, by name: cp_a.low");
like($out, qr/^alu_cg\.cp_a\s+5\s+4\s+80\.0%\s+26\.6%\s+60\.0%$/m,
"cp_a: merged 80%, while the average run is 26.6% and the best 60%");
# ---- the goal decides the exit status, and the report is still written
($rc) = run_tool(@SIX, "--goal", 90);
is($rc, 0, "--goal 90 with 92.3%: exit 0");
($rc) = run_tool(@SIX, "--goal", 95);
ok($rc == 1 && -e "$base/out.csv", "--goal 95: exit 1, and the CSV is still written");
# ---- a closure figure is never rounded up
my $big = "=== FUNCTIONAL COVERAGE ===\ntest: big seed: 1 operations: 1\ncovergroup g\n"
. " coverpoint cp 1999/2000 bins 99.95%\n"
. join("", map { sprintf " bin b%04d hits %4d at_least 1\n", $_, $_ ? 1 : 0 } 0 .. 1999)
. "=== END COVERAGE ===\n";
spew("$base/big.rpt", $big);
($rc, $out) = run_tool("$base/big.rpt", "--goal", 100);
ok($out =~ /^TOTAL \(bins\)\s+2000\s+1999\s+99\.9%$/m && $rc == 1,
"1999 of 2000 bins prints 99.9%, and --goal 100 is not met");
# ---- every input problem: exit 2, a reason, no CSV
my @cases = (
[ "a different coverage model", [ @SIX, "$base/runs-bad/old_model/cov.rpt" ], qr/no bin alu_cg\.cp_op\.SHL/ ],
[ "a crashed run", [ @SIX, "$base/runs-bad/crashed/cov.rpt" ], qr/report cut off/ ],
);
spew("$base/rerun.rpt", slurp($SIX[3]));
push @cases, [ "the same run twice", [ @SIX, "$base/rerun.rpt" ], qr/seed 4 was already read/ ];
(my $edited = slurp($SIX[0])) =~ s/(bin SHL\s+hits\s+)2/${1}0/ or die "seed 1 has no SHL hits to edit";
spew("$base/edited.rpt", $edited);
push @cases, [ "a summary that disagrees with its bins", [ "$base/edited.rpt", @SIX[1 .. 5] ], qr/says 6\/6 bins, its bins give 5\/6/ ];
(my $cross = slurp($SIX[0])) =~ s/^( coverpoint cp_carry)/ cross a_x_b 0\/30 bins\n$1/m;
spew("$base/cross.rpt", $cross);
push @cases, [ "a line the tool does not know", [ "$base/cross.rpt", @SIX[1 .. 5] ], qr/unrecognised line/ ];
for my $c (@cases) {
my ($label, $args, $why) = @$c;
my ($crc, undef, $err) = run_tool(@$args);
ok($crc == 2 && $err =~ $why && !-e "$base/out.csv", "$label: exit 2, explained, no CSV");
}
($rc) = run_tool();
is($rc, 3, "no reports: exit 3");
done_testing();Test 3 is the merge, checked independently. It re-reads the six reports with its own pattern, sums every bin's hits, and requires the CSV to agree on every one — thirteen numbers, not a total.
Test 6 pins the three different figures for cp_a, so a tool that drifts into averaging fails a test with a readable name. Test 9 builds a 2,000-bin report in the test itself, because no amount of real simulation of a 13-bin model would ever produce 1,999 of 2,000.
#!/bin/bash
# tests.sh -- run the suite through prove, keeping prove's own exit status.
prove -v t/ > prove.log 2>&1
status=$?
grep -v 'wallclock' prove.log
echo "prove exit $status"t/parse_coverage.t ..
ok 1 - six runs, no goal: exit 0
ok 2 - 12 of 13 bins covered after merging
ok 3 - every merged hit count equals the sum over the six reports
ok 4 - max (at_least 3) is covered only by adding runs together
ok 5 - the one hole, by name: cp_a.low
ok 6 - cp_a: merged 80%, while the average run is 26.6% and the best 60%
ok 7 - --goal 90 with 92.3%: exit 0
ok 8 - --goal 95: exit 1, and the CSV is still written
ok 9 - 1999 of 2000 bins prints 99.9%, and --goal 100 is not met
ok 10 - a different coverage model: exit 2, explained, no CSV
ok 11 - a crashed run: exit 2, explained, no CSV
ok 12 - the same run twice: exit 2, explained, no CSV
ok 13 - a summary that disagrees with its bins: exit 2, explained, no CSV
ok 14 - a line the tool does not know: exit 2, explained, no CSV
ok 15 - no reports: exit 3
1..15
ok
All tests successful.
Result: PASS
prove exit 09. Proving The Suite Can Fail
#!/bin/bash
# mutate.sh -- plant one defect at a time in a COPY of the tool; does the
# suite notice?
mutant() { # LABEL PERL-SUBSTITUTION (applied to the whole file)
local label=$1 edit=$2 d failed
d=$(mktemp -d)
mkdir "$d/t"
cp -r make_runs.sh tb "$d/"
cp t/parse_coverage.t "$d/t/"
perl -0pe "$edit" parse_coverage.pl > "$d/parse_coverage.pl"
if cmp -s parse_coverage.pl "$d/parse_coverage.pl"; then
printf '%-44s NOT PLANTED (edit matched nothing)\n' "$label"
elif ! perl -c "$d/parse_coverage.pl" > /dev/null 2>&1; then
printf '%-44s INVALID (the mutant does not compile)\n' "$label"
else
failed=$(perl "$d/t/parse_coverage.t" 2>/dev/null | grep -c '^not ok')
printf '%-44s %s (%s failing test(s))\n' "$label" \
"$([ "$failed" -gt 0 ] && echo CAUGHT || echo SURVIVED)" "$failed"
fi
rm -rf "$d"
}
mutant "1 covered = the best single run" 's/\$m\{covered\} = \$m\{hits\} >=/\$m{covered} = \$m{best_run} >=/'
mutant "2 round percentages to nearest" 's/int\(1000 \* \$n \/ \$d\)/int(1000 * \$n \/ \$d + 0.5)/'
mutant "3 merge different models" 's/\n if \@missing \|\| \@extra \|\| \@need;/\n if 0;/'
mutant "4 accept a report with no END line" 's/unless \(\$ended\) \{ push \@problems, "\$file: report cut off \(no END COVERAGE line\)"; return \}//'
mutant "5 count a run twice" 's/if \(my \$first = \$seen_run\{\$id\}\)/if (my \$first = \$seen_run{"never"})/'
mutant "6 trust the summary, skip the check" 's/if \$cov != \$p_cov \|\| \@keys != \$p_tot;/if 0;/'
mutant "7 skip lines the tool does not know" 's/else \{ push \@problems, "\$file line \$\.: unrecognised line in the coverage section" \}//'
mutant "8 count an ignore bin as a bin" 's/\\s\+bin \(\\w\+\)\\s\+hits\\s\+\(\\d\+\)\\s\+at_least \(\\d\+\)\//\\s+(?:bin|ignore) (\\w+)\\s+hits\\s+(\\d+)(?:\\s+at_least (\\d+))?\//'1 covered = the best single run CAUGHT (5 failing test(s))
2 round percentages to nearest CAUGHT (1 failing test(s))
3 merge different models CAUGHT (1 failing test(s))
4 accept a report with no END line CAUGHT (1 failing test(s))
5 count a run twice CAUGHT (1 failing test(s))
6 trust the summary, skip the check CAUGHT (1 failing test(s))
7 skip lines the tool does not know CAUGHT (1 failing test(s))
8 count an ignore bin as a bin CAUGHT (9 failing test(s))All eight are caught. Getting to eight valid mutants taught something about mutation testing itself. The first list had two mutants that proved nothing:
- One did not compile. Replacing
if (my $first = $seen_run{$id})withif (0)removed the declaration of$first, which the next line still used, so underuse strictthe whole copy failed to compile — and "failed every test" looked like a strong result. It is no result at all. The script now runsperl -con each copy and reports a non-compiling one asINVALID. - One changed nothing. The first "count an ignore bin as a bin" edited a pattern that an
ignoreline never matched, becauseignorelines have noat_least. A mutant that changes no behaviour cannot be caught by any test. It is called an equivalent mutant, and a survivor should always be checked for being one before a test is written for it.
Adding the compile check to the earlier projects' scripts found one more invalid mutant, in Chapter 29.3, which was rewritten to keep its variable declared.
10. What The Tool Does Not Do
- It does not read cross coverage. A
crossline is refused, so a model that uses crosses needs the parser extended — a new line shape, a new bin key, and new samples — before it can be merged. - It does not weight coverpoints. The total is bin-weighted. A team whose sign-off uses per-coverpoint weights computes that from the CSV.
- It requires identical models. Merging across a model change — mapping renamed bins, retiring old ones — is a migration with its own review, not something a nightly tool should do silently.
- It reads one layout, written for this chapter. Pointing it at a real tool's report means a new reader and new samples first, exactly as Chapter 29.3's format table does for timing reports.
- It trusts the
testandseedline for identity. Two genuinely different runs that report the same identity are refused as duplicates; fixing that belongs in the regression runner that wrote them.
11. Exercises
1. Who hit it? Add a column to --holes output for bins covered by a single run: the seed that hit it. Why is that the most useful seed to know?
2. Weighted totals. Add --weights FILE (coverpoint and weight per line) and a weighted total beside the bin-weighted one. Write the test that shows the two differ on the six runs.
3. A model change, handled. Last month's model has no SHL bin. Write a separate script that merges an old-model run into the new model by treating the missing bin as unsampled, and argue in its usage text why that is or is not safe.
4. The early-run bias. Remove the warm-up loop from the testbench, rerun twelve seeds, and measure how often SHL appears in the first operation. Then write the check that would have caught it.
5. A daily trend. Keep yesterday's coverage.csv and write a ten-line script that lists bins that were covered yesterday and are not today. What change in the regression could cause that, if the design did not change?
12. What To Remember
- Closure is a merge, not an average. Measured:
cp_amerged 80.0%, best run 60.0%, average run 26.6%. - Sum the hits; decide coverage on the sum.
at_leastbins can close only across runs —maxdid, with 2 + 1. - Every run must describe the same model, and the message must name the bins that differ.
- A run may be counted once. A copied rerun covered a real hole with nothing but a copy.
- A report that stops half-way is refused, not merged as a smaller report.
- Check each report's summary against its own bins. The same check catches hand edits and parser bugs.
- Round a closure percentage down, and judge goals on counts. Measured:
%.1fprints 1,999 of 2,000 as 100.0%. - A mutant must compile and must change behaviour, or it proves nothing; check with
perl -c, and check survivors for equivalence.
Where this fits
Part of the Perl curriculum.