USB · Module 24
USB Functional Coverage
An unreachable bin and an untested bin both read 0% and demand opposite responses — and an exclusion is a claim about the design, so a bin that is excluded and then hit must fail.
The previous three chapters checked that things were legal, timely and correctly matched. This one asks what any of it actually covered — and the failure that defines the chapter is a reporting failure, not a checking one.
1. An Unreachable Bin and an Untested Bin Look Identical
Both read 0%. They demand opposite responses:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
UNTESTED write a test. The design is unverified here.
UNREACHABLE delete the bin. The combination cannot occur,
and every hour spent chasing it is an hour not
spent on the first kind.A coverage report that cannot tell them apart converts the whole exercise into guesswork, and the guessing is done by whoever is under the most schedule pressure.
So this collector makes reachability an explicit property of every bin, declared up front, next to the reason.
2. An Exclusion Is a Claim, and a Falsified Claim Must Fire
This is the part that is almost always missing. Declaring a bin unreachable is an assertion about the design:
"this combination cannot occur"
If it then occurs, one of two things is true: the exclusion was wrong, or the design is doing something it must not. Both are findings.
3. The Declaration
Two axes, sixteen cross bins:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
CONTROL ISOC BULK INTERRUPT
LOW (0) legal UNREACH UNREACH legal
FULL (1) legal legal legal legal
HIGH (2) legal legal legal legal
RSVD (3) ILLEGAL ILLEGAL ILLEGAL ILLEGAL
legal 10 unreachable 2 illegal 4The two unreachable bins are specification, not laziness: USB low speed supports only control and interrupt endpoints. There is no low-speed isochronous and no low-speed bulk. If the design ever enumerates one, that is a genuine bug that no amount of coverage staring would reveal.
The four illegal bins are the reserved speed encoding, which must never be enumerated at all. An illegal bin is the only kind that does work by being reached rather than by not being reached.
4. The Cross Is Where Coverage Lives
Every individual coverpoint here fills almost immediately: three reachable speeds, four endpoint types, a handful of samples. The cross is at 40% long after both are full.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
sample (LS,CONTROL) (FS,ISOC) (HS,BULK) (LS,INTERRUPT)
speed coverpoint 3 of 3 reachable values 100%
ep_type coverpoint 4 of 4 values 100%
the CROSS 4 of 10 bins 40%
All three describe the same four samples.The interesting question was never "did we see high speed" but "did we see high speed with an isochronous endpoint". A model whose coverpoints are full and whose crosses are empty reports a high number and has tested almost nothing.
5. A Bin Is Not Covered by One Sample
One hit is an anecdote. N_MIN exists because a bin reached once, by accident, in one configuration, is not a bin anybody should sign off — and because a collector that counts one sample as coverage can be satisfied by a mutation that samples continuously.
6. What We Are Building
usb_coverage_collector — every sample is classified before it is counted
7. Verilog-2005 Implementation
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// usb_coverage_collector -- what a coverage model is when you build one, and
// the reporting failure that wastes more verification effort than any other.
//
// AN UNREACHABLE BIN AND AN UNTESTED BIN LOOK IDENTICAL
//
// Both read 0%. They demand opposite responses:
//
// UNTESTED write a test. The design is unverified here.
// UNREACHABLE delete the bin. The combination cannot occur,
// and every hour spent chasing it is an hour not
// spent on the first kind.
//
// A coverage report that cannot tell them apart converts the whole exercise
// into guesswork, and the guessing is done by whoever is under the most
// schedule pressure. So this collector makes reachability an explicit
// property of every bin, declared up front, and it is the difference between
// a coverage number that means something and a percentage.
//
// AN EXCLUSION IS A CLAIM, AND A FALSIFIED CLAIM MUST FIRE
//
// This is the part that is almost always missing. Declaring a bin
// unreachable is an assertion about the design:
//
// "this combination cannot occur"
//
// If it then occurs, ONE OF TWO THINGS IS TRUE: the exclusion was wrong, or
// the design is doing something it must not. Both are findings. Neither is
// visible in a flow where exclusions are a file of regular expressions
// applied to the report afterwards.
//
// A bin that is EXCLUDED and then HIT must fail, loudly,
// with the exclusion's own name attached.
//
// Here the low-speed rows are the real example. USB low speed supports only
// control and interrupt endpoints: there is no low-speed isochronous and no
// low-speed bulk. Those two cross bins are unreachable by the specification,
// not by the testbench's laziness -- and if the design ever enumerates one,
// that is a genuine bug that no amount of coverage staring would reveal.
//
// THE CROSS IS WHERE COVERAGE LIVES
//
// Every individual coverpoint here reaches 100% almost immediately: four
// speeds, four endpoint types, a handful of samples. The CROSS is at 20%
// long after both are full, because the interesting question was never "did
// we see high speed" but "did we see high speed WITH an isochronous
// endpoint".
//
// A model whose coverpoints are full and whose crosses are empty reports a
// high number and has tested almost nothing.
//
// A BIN IS NOT COVERED BY ONE SAMPLE
//
// One hit is an anecdote. N_MIN exists because a bin reached once, by
// accident, in one configuration, is not a bin anybody should sign off --
// and because a collector that counts one sample as coverage can be
// satisfied by a mutation that samples continuously.
module usb_coverage_collector #(
parameter integer N_SPEED = 4, // 3 real speeds + 1 reserved encoding
parameter integer N_EPTYPE = 4, // control / isochronous / bulk / interrupt
parameter integer N_MIN = 2 // hits before a bin counts as covered
) (
input wire clk,
input wire rst_n,
input wire sample, // a transaction happened: sample NOW
input wire [1:0] speed, // 0 LS, 1 FS, 2 HS, 3 reserved
input wire [1:0] ep_type, // 0 CONTROL, 1 ISOC, 2 BULK, 3 INTERRUPT
input wire eot, // end of test: report what was missed
output wire [1:0] bin_class, // the reachability of the bin being sampled
output wire [7:0] bin_hits, // its hit count, saturating
output wire hit_pulse, // a LEGAL bin was sampled
output wire cov_err, // one pulse per coverage failure
output wire [1:0] cov_code,
// NOT named `illegal_bins`: that is a SystemVerilog KEYWORD (it is
// covergroup syntax), so a signal of that name does not compile in the
// SystemVerilog port of this block -- and the error it produces points
// at the line after it. `ignore_bins` and `wildcard` are the same trap.
output wire [7:0] covered_bins, // legal bins with >= N_MIN hits
output wire [7:0] legal_total, // how many there are to cover
output wire [7:0] unreach_total, // declared unreachable
output wire [7:0] illegal_total, // declared illegal
output reg [31:0] n_samples,
output reg [31:0] n_illegal,
output reg [31:0] n_exclusion, // an "unreachable" bin was HIT
output reg [31:0] n_incomplete // a legal bin never reached N_MIN
);
localparam integer N_BIN = N_SPEED * N_EPTYPE;
// ---- The reachability of every cross bin, declared up front. ----
localparam [1:0] B_LEGAL = 2'd0, // must be covered
B_ILLEGAL = 2'd1, // must never be sampled
B_UNREACHABLE = 2'd2; // cannot occur -- and if it does,
// the exclusion was wrong
localparam [1:0] C_NONE = 2'd0,
C_ILLEGAL = 2'd1, // an illegal combination occurred
C_EXCLUSION = 2'd2, // an excluded combination occurred
C_INCOMPLETE = 2'd3; // a legal bin was never covered
// ---- THE DECLARATION. ----
//
// Written as a function of the two axes rather than as a table of
// literals, so the reason for each classification is in the code next to
// the classification itself. A table of 16 two-bit constants is the same
// information with the reasoning deleted.
function [1:0] classify;
input [1:0] sp;
input [1:0] ep;
begin
if (sp == 2'd3) begin
// The reserved speed encoding. Not "untested": it must never be
// enumerated at all, so it is ILLEGAL rather than unreachable, and
// hitting it is a design failure rather than a documentation one.
classify = B_ILLEGAL;
end else if ((sp == 2'd0) && (ep == 2'd1)) begin
// Low speed has no isochronous endpoints. Specification, not
// laziness.
classify = B_UNREACHABLE;
end else if ((sp == 2'd0) && (ep == 2'd2)) begin
// Low speed has no bulk endpoints either.
classify = B_UNREACHABLE;
end else begin
classify = B_LEGAL;
end
end
endfunction
reg [7:0] hits_r [0:N_BIN-1];
reg [1:0] code_r;
reg hit_r, err_r;
// The end-of-test walk, one bin per cycle. A bounded walk rather than a
// combinational reduction, so the report names the bins in a stable order
// and the design has no wide comparator in it.
reg [7:0] scan_r;
wire [7:0] sel = {4'd0, speed, ep_type} & 8'h0F;
wire [1:0] cls = classify(speed, ep_type);
assign bin_class = cls;
assign bin_hits = hits_r[sel[3:0]];
assign hit_pulse = hit_r;
assign cov_err = err_r;
assign cov_code = code_r;
// ---- The three populations, counted from the declaration itself. ----
//
// legal_total is the DENOMINATOR of every coverage figure anybody quotes,
// and deriving it from the same function that classifies a sample is what
// stops the two drifting apart. A denominator maintained by hand in a
// spreadsheet is the usual arrangement and the usual source of a
// coverage number that nobody can reproduce.
// Written as a function over the declaration, not as an `always @*`
// block: a combinational block whose body reads only constants has an
// empty sensitivity list, never triggers, and leaves its outputs at x --
// which, for the one number every percentage is divided by, is a
// spectacular way to be wrong.
function [7:0] count_class;
input [1:0] want;
integer c;
begin
count_class = 8'd0;
for (c = 0; c < N_BIN; c = c + 1)
if (classify(c[3:2], c[1:0]) == want)
count_class = count_class + 8'd1;
end
endfunction
assign legal_total = count_class(B_LEGAL);
assign unreach_total = count_class(B_UNREACHABLE);
assign illegal_total = count_class(B_ILLEGAL);
// ---- Covered means N_MIN hits, not one. ----
// This one DOES depend on a signal -- the hit counters -- so it is a real
// combinational block with a real sensitivity list.
integer d;
reg [7:0] n_cov;
always @* begin
n_cov = 8'd0;
for (d = 0; d < N_BIN; d = d + 1)
if ((classify(d[3:2], d[1:0]) == B_LEGAL) && (hits_r[d] >= N_MIN[7:0]))
n_cov = n_cov + 8'd1;
end
assign covered_bins = n_cov;
integer i;
always @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (i = 0; i < N_BIN; i = i + 1) hits_r[i] <= 8'd0;
code_r <= C_NONE;
hit_r <= 1'b0;
err_r <= 1'b0;
scan_r <= 8'd0;
n_samples <= 32'd0;
n_illegal <= 32'd0;
n_exclusion <= 32'd0;
n_incomplete<= 32'd0;
end else begin
code_r <= C_NONE;
hit_r <= 1'b0;
err_r <= 1'b0;
if (eot) begin
// ---- THE REPORT. One bin per cycle. ----
//
// A legal bin that never reached N_MIN is reported by NAME. "84%
// covered" is not a report; it is a number that requires somebody
// to go and find out which 16% and why.
if (scan_r < N_BIN[7:0]) begin
if ((classify(scan_r[3:2], scan_r[1:0]) == B_LEGAL)
&& (hits_r[scan_r[3:0]] < N_MIN[7:0])) begin
err_r <= 1'b1;
code_r <= C_INCOMPLETE;
n_incomplete <= n_incomplete + 32'd1;
end
scan_r <= scan_r + 8'd1;
end
end else begin
scan_r <= 8'd0;
if (sample) begin
n_samples <= n_samples + 32'd1;
case (cls)
B_ILLEGAL: begin
// ---- An illegal bin is not a coverage hole. It is a
// ---- FAILURE, and it fires when it is HIT.
//
// This is the only kind of bin that does work by being
// reached rather than by not being reached, and it is the
// reason illegal_total is reported separately: it must never
// contribute to a percentage in either direction.
err_r <= 1'b1;
code_r <= C_ILLEGAL;
n_illegal <= n_illegal + 32'd1;
end
B_UNREACHABLE: begin
// ---- THE EXCLUSION WAS WRONG. ----
//
// Something the coverage model asserted could not happen has
// just happened. Either the exclusion is incorrect -- in
// which case the coverage target has been understated ever
// since -- or the design is enumerating a configuration the
// specification forbids.
//
// An exclusion applied as a post-processing filter can never
// produce this message, because by then the sample has been
// deleted.
err_r <= 1'b1;
code_r <= C_EXCLUSION;
n_exclusion <= n_exclusion + 32'd1;
end
default: begin
hit_r <= 1'b1;
// Saturating, so a long run cannot wrap a bin back under
// N_MIN and un-cover it.
if (hits_r[sel[3:0]] != 8'hFF)
hits_r[sel[3:0]] <= hits_r[sel[3:0]] + 8'd1;
end
endcase
end
end
end
end
endmodule8. SystemVerilog Implementation
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// usb_coverage_collector -- what a coverage model is when you build one, and
// the reporting failure that wastes more verification effort than any other.
//
// AN UNREACHABLE BIN AND AN UNTESTED BIN LOOK IDENTICAL
//
// Both read 0%. They demand opposite responses:
//
// UNTESTED write a test. The design is unverified here.
// UNREACHABLE delete the bin. The combination cannot occur,
// and every hour spent chasing it is an hour not
// spent on the first kind.
//
// A coverage report that cannot tell them apart converts the whole exercise
// into guesswork, and the guessing is done by whoever is under the most
// schedule pressure. So this collector makes reachability an explicit
// property of every bin, declared up front, and it is the difference between
// a coverage number that means something and a percentage.
//
// AN EXCLUSION IS A CLAIM, AND A FALSIFIED CLAIM MUST FIRE
//
// This is the part that is almost always missing. Declaring a bin
// unreachable is an assertion about the design:
//
// "this combination cannot occur"
//
// If it then occurs, ONE OF TWO THINGS IS TRUE: the exclusion was wrong, or
// the design is doing something it must not. Both are findings. Neither is
// visible in a flow where exclusions are a file of regular expressions
// applied to the report afterwards.
//
// A bin that is EXCLUDED and then HIT must fail, loudly,
// with the exclusion's own name attached.
//
// Here the low-speed rows are the real example. USB low speed supports only
// control and interrupt endpoints: there is no low-speed isochronous and no
// low-speed bulk. Those two cross bins are unreachable by the specification,
// not by the testbench's laziness -- and if the design ever enumerates one,
// that is a genuine bug that no amount of coverage staring would reveal.
//
// THE CROSS IS WHERE COVERAGE LIVES
//
// Every individual coverpoint here reaches 100% almost immediately: four
// speeds, four endpoint types, a handful of samples. The CROSS is at 20%
// long after both are full, because the interesting question was never "did
// we see high speed" but "did we see high speed WITH an isochronous
// endpoint".
//
// A model whose coverpoints are full and whose crosses are empty reports a
// high number and has tested almost nothing.
//
// A BIN IS NOT COVERED BY ONE SAMPLE
//
// One hit is an anecdote. N_MIN exists because a bin reached once, by
// accident, in one configuration, is not a bin anybody should sign off --
// and because a collector that counts one sample as coverage can be
// satisfied by a mutation that samples continuously.
package usb_cov_pkg;
// The reachability of a cross bin, declared up front. B_UNREACHABLE is
// the state that makes the whole model honest: it says "this cannot
// happen", which is a CLAIM, and a claim that is falsified must fire.
typedef enum logic [1:0] {
B_LEGAL = 2'd0, // must be covered
B_ILLEGAL = 2'd1, // must never be sampled
B_UNREACHABLE = 2'd2 // cannot occur -- and if it does, the exclusion
// was wrong
} bin_class_e;
typedef enum logic [1:0] {
C_NONE = 2'd0,
C_ILLEGAL = 2'd1, // an illegal combination occurred
C_EXCLUSION = 2'd2, // an excluded combination occurred
C_INCOMPLETE = 2'd3 // a legal bin was never covered
} cov_err_e;
endpackage
module usb_coverage_collector
import usb_cov_pkg::*;
#(
parameter int N_SPEED = 4, // 3 real speeds + 1 reserved encoding
parameter int N_EPTYPE = 4, // control / isochronous / bulk / interrupt
parameter int N_MIN = 2 // hits before a bin counts as covered
) (
input logic clk,
input logic rst_n,
input logic sample, // a transaction happened: sample NOW
input logic [1:0] speed, // 0 LS, 1 FS, 2 HS, 3 reserved
input logic [1:0] ep_type, // 0 CONTROL, 1 ISOC, 2 BULK, 3 INTERRUPT
input logic eot, // end of test: report what was missed
output bin_class_e bin_class, // the reachability of the bin being sampled
output logic [7:0] bin_hits, // its hit count, saturating
output logic hit_pulse, // a LEGAL bin was sampled
output logic cov_err, // one pulse per coverage failure
output cov_err_e cov_code,
// NOT named `illegal_bins`: that is a SystemVerilog KEYWORD (it is
// covergroup syntax), so a signal of that name does not compile in the
// SystemVerilog port of this block -- and the error it produces points
// at the line after it. `ignore_bins` and `wildcard` are the same trap.
output logic [7:0] covered_bins, // legal bins with >= N_MIN hits
output logic [7:0] legal_total, // how many there are to cover
output logic [7:0] unreach_total, // declared unreachable
output logic [7:0] illegal_total, // declared illegal
output logic [31:0] n_samples,
output logic [31:0] n_illegal,
output logic [31:0] n_exclusion, // an "unreachable" bin was HIT
output logic [31:0] n_incomplete // a legal bin never reached N_MIN
);
localparam int N_BIN = N_SPEED * N_EPTYPE;
// ---- THE DECLARATION. ----
//
// Written as a function of the two axes rather than as a table of
// literals, so the reason for each classification is in the code next to
// the classification itself. A table of 16 two-bit constants is the same
// information with the reasoning deleted.
function automatic bin_class_e classify(input logic [1:0] sp,
input logic [1:0] ep);
begin
if (sp == 2'd3) begin
// The reserved speed encoding. Not "untested": it must never be
// enumerated at all, so it is ILLEGAL rather than unreachable, and
// hitting it is a design failure rather than a documentation one.
classify = B_ILLEGAL;
end else if ((sp == 2'd0) && (ep == 2'd1)) begin
// Low speed has no isochronous endpoints. Specification, not
// laziness.
classify = B_UNREACHABLE;
end else if ((sp == 2'd0) && (ep == 2'd2)) begin
// Low speed has no bulk endpoints either.
classify = B_UNREACHABLE;
end else begin
classify = B_LEGAL;
end
end
endfunction
logic [7:0] hits_r [N_BIN];
cov_err_e code_r;
logic hit_r, err_r;
// The end-of-test walk, one bin per cycle. A bounded walk rather than a
// combinational reduction, so the report names the bins in a stable order
// and the design has no wide comparator in it.
logic [7:0] scan_r;
logic [7:0] sel;
bin_class_e cls;
assign sel = {4'd0, speed, ep_type} & 8'h0F;
assign cls = classify(speed, ep_type);
assign bin_class = cls;
assign bin_hits = hits_r[sel[3:0]];
assign hit_pulse = hit_r;
assign cov_err = err_r;
assign cov_code = code_r;
// ---- The three populations, counted from the declaration itself. ----
//
// legal_total is the DENOMINATOR of every coverage figure anybody quotes,
// and deriving it from the same function that classifies a sample is what
// stops the two drifting apart. A denominator maintained by hand in a
// spreadsheet is the usual arrangement and the usual source of a
// coverage number that nobody can reproduce.
// Written as a function over the declaration, not as an `always @*`
// block: a combinational block whose body reads only constants has an
// empty sensitivity list, never triggers, and leaves its outputs at x --
// which, for the one number every percentage is divided by, is a
// spectacular way to be wrong.
function automatic logic [7:0] count_class(input bin_class_e want);
int c;
begin
count_class = 8'd0;
for (c = 0; c < N_BIN; c++)
if (classify(c[3:2], c[1:0]) == want)
count_class = count_class + 8'd1;
end
endfunction
assign legal_total = count_class(B_LEGAL);
assign unreach_total = count_class(B_UNREACHABLE);
assign illegal_total = count_class(B_ILLEGAL);
// ---- Covered means N_MIN hits, not one. ----
// This one DOES depend on a signal -- the hit counters -- so it is a real
// combinational block with a real sensitivity list.
int d;
logic [7:0] n_cov;
always_comb begin
n_cov = 8'd0;
for (d = 0; d < N_BIN; d++)
if ((classify(d[3:2], d[1:0]) == B_LEGAL) && (hits_r[d] >= 8'(N_MIN)))
n_cov = n_cov + 8'd1;
end
assign covered_bins = n_cov;
int i;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (i = 0; i < N_BIN; i++) hits_r[i] <= 8'd0;
code_r <= C_NONE;
hit_r <= 1'b0;
err_r <= 1'b0;
scan_r <= 8'd0;
n_samples <= 32'd0;
n_illegal <= 32'd0;
n_exclusion <= 32'd0;
n_incomplete<= 32'd0;
end else begin
code_r <= C_NONE;
hit_r <= 1'b0;
err_r <= 1'b0;
if (eot) begin
// ---- THE REPORT. One bin per cycle. ----
//
// A legal bin that never reached N_MIN is reported by NAME. "84%
// covered" is not a report; it is a number that requires somebody
// to go and find out which 16% and why.
if (scan_r < 8'(N_BIN)) begin
if ((classify(scan_r[3:2], scan_r[1:0]) == B_LEGAL)
&& (hits_r[scan_r[3:0]] < 8'(N_MIN))) begin
err_r <= 1'b1;
code_r <= C_INCOMPLETE;
n_incomplete <= n_incomplete + 32'd1;
end
scan_r <= scan_r + 8'd1;
end
end else begin
scan_r <= 8'd0;
if (sample) begin
n_samples <= n_samples + 32'd1;
case (cls)
B_ILLEGAL: begin
// ---- An illegal bin is not a coverage hole. It is a
// ---- FAILURE, and it fires when it is HIT.
//
// This is the only kind of bin that does work by being
// reached rather than by not being reached, and it is the
// reason illegal_total is reported separately: it must never
// contribute to a percentage in either direction.
err_r <= 1'b1;
code_r <= C_ILLEGAL;
n_illegal <= n_illegal + 32'd1;
end
B_UNREACHABLE: begin
// ---- THE EXCLUSION WAS WRONG. ----
//
// Something the coverage model asserted could not happen has
// just happened. Either the exclusion is incorrect -- in
// which case the coverage target has been understated ever
// since -- or the design is enumerating a configuration the
// specification forbids.
//
// An exclusion applied as a post-processing filter can never
// produce this message, because by then the sample has been
// deleted.
err_r <= 1'b1;
code_r <= C_EXCLUSION;
n_exclusion <= n_exclusion + 32'd1;
end
default: begin
hit_r <= 1'b1;
// Saturating, so a long run cannot wrap a bin back under
// N_MIN and un-cover it.
if (hits_r[sel[3:0]] != 8'hFF)
hits_r[sel[3:0]] <= hits_r[sel[3:0]] + 8'd1;
end
endcase
end
end
end
end
endmodule9. VHDL-2008 Implementation
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
-- usb_coverage_collector -- what a coverage model is when you build one, and
-- the reporting failure that wastes more verification effort than any other.
--
-- AN UNREACHABLE BIN AND AN UNTESTED BIN LOOK IDENTICAL
--
-- Both read 0%. They demand opposite responses:
--
-- UNTESTED write a test. The design is unverified here.
-- UNREACHABLE delete the bin. The combination cannot occur,
-- and every hour spent chasing it is an hour not
-- spent on the first kind.
--
-- A coverage report that cannot tell them apart converts the whole exercise
-- into guesswork, and the guessing is done by whoever is under the most
-- schedule pressure. So this collector makes reachability an explicit
-- property of every bin, declared up front, and it is the difference between
-- a coverage number that means something and a percentage.
--
-- AN EXCLUSION IS A CLAIM, AND A FALSIFIED CLAIM MUST FIRE
--
-- This is the part that is almost always missing. Declaring a bin
-- unreachable is an assertion about the design:
--
-- "this combination cannot occur"
--
-- If it then occurs, ONE OF TWO THINGS IS TRUE: the exclusion was wrong, or
-- the design is doing something it must not. Both are findings. Neither is
-- visible in a flow where exclusions are a file of regular expressions
-- applied to the report afterwards.
--
-- A bin that is EXCLUDED and then HIT must fail, loudly,
-- with the exclusion's own name attached.
--
-- Here the low-speed rows are the real example. USB low speed supports only
-- control and interrupt endpoints: there is no low-speed isochronous and no
-- low-speed bulk. Those two cross bins are unreachable by the specification,
-- not by the testbench's laziness -- and if the design ever enumerates one,
-- that is a genuine bug that no amount of coverage staring would reveal.
--
-- THE CROSS IS WHERE COVERAGE LIVES
--
-- Every individual coverpoint here reaches 100% almost immediately: four
-- speeds, four endpoint types, a handful of samples. The CROSS is at 20%
-- long after both are full, because the interesting question was never "did
-- we see high speed" but "did we see high speed WITH an isochronous
-- endpoint".
--
-- A model whose coverpoints are full and whose crosses are empty reports a
-- high number and has tested almost nothing.
--
-- A BIN IS NOT COVERED BY ONE SAMPLE
--
-- One hit is an anecdote. N_MIN exists because a bin reached once, by
-- accident, in one configuration, is not a bin anybody should sign off --
-- and because a collector that counts one sample as coverage can be
-- satisfied by a mutation that samples continuously.
library ieee;
use ieee.std_logic_1164.all;
package usb_cov_pkg is
-- The reachability of a cross bin, declared up front. B_UNREACHABLE is
-- the state that makes the whole model honest: it says "this cannot
-- happen", which is a CLAIM, and a claim that is falsified must fire.
type bin_class_t is (B_LEGAL, B_ILLEGAL, B_UNREACHABLE);
type cov_err_t is (C_NONE, C_ILLEGAL, C_EXCLUSION, C_INCOMPLETE);
function bc_code (b : bin_class_t) return std_logic_vector;
function ce_code (c : cov_err_t) return std_logic_vector;
end package usb_cov_pkg;
package body usb_cov_pkg is
function bc_code (b : bin_class_t) return std_logic_vector is
begin
case b is
when B_LEGAL => return "00";
when B_ILLEGAL => return "01";
when B_UNREACHABLE => return "10";
end case;
end function;
function ce_code (c : cov_err_t) return std_logic_vector is
begin
case c is
when C_NONE => return "00";
when C_ILLEGAL => return "01";
when C_EXCLUSION => return "10";
when C_INCOMPLETE => return "11";
end case;
end function;
end package body usb_cov_pkg;
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
use work.usb_cov_pkg.all;
entity usb_coverage_collector is
generic (
N_SPEED : integer := 4; -- 3 real speeds + 1 reserved encoding
N_EPTYPE : integer := 4; -- control / isochronous / bulk / interrupt
N_MIN : integer := 2 -- hits before a bin counts as covered
);
port (
clk : in std_logic;
rst_n : in std_logic;
sample : in std_logic; -- sample NOW
speed : in std_logic_vector(1 downto 0); -- 0 LS 1 FS 2 HS 3 rsvd
ep_type : in std_logic_vector(1 downto 0); -- 0 CTRL 1 ISO 2 BULK 3 INT
eot : in std_logic;
bin_class : out std_logic_vector(1 downto 0);
bin_hits : out std_logic_vector(7 downto 0);
hit_pulse : out std_logic;
cov_err : out std_logic;
cov_code : out std_logic_vector(1 downto 0);
covered_bins : out std_logic_vector(7 downto 0);
legal_total : out std_logic_vector(7 downto 0);
unreach_total : out std_logic_vector(7 downto 0);
illegal_total : out std_logic_vector(7 downto 0);
n_samples : out std_logic_vector(31 downto 0);
n_illegal : out std_logic_vector(31 downto 0);
n_exclusion : out std_logic_vector(31 downto 0);
n_incomplete : out std_logic_vector(31 downto 0)
);
end entity usb_coverage_collector;
architecture rtl of usb_coverage_collector is
constant N_BIN : integer := N_SPEED * N_EPTYPE;
type hits_arr is array (0 to N_BIN-1) of unsigned(7 downto 0);
-- ---- THE DECLARATION. ----
--
-- Written as a function of the two axes rather than as a table of
-- literals, so the reason for each classification is in the code next to
-- the classification itself. A table of 16 constants is the same
-- information with the reasoning deleted.
function classify (sp, ep : std_logic_vector(1 downto 0)) return bin_class_t is
begin
if sp = "11" then
-- The reserved speed encoding. Not "untested": it must never be
-- enumerated at all, so it is ILLEGAL rather than unreachable, and
-- hitting it is a design failure rather than a documentation one.
return B_ILLEGAL;
elsif sp = "00" and ep = "01" then
-- Low speed has no isochronous endpoints. Specification, not laziness.
return B_UNREACHABLE;
elsif sp = "00" and ep = "10" then
-- Low speed has no bulk endpoints either.
return B_UNREACHABLE;
else
return B_LEGAL;
end if;
end function;
-- Counted from the declaration itself rather than typed in. legal_total is
-- the DENOMINATOR of every coverage figure anybody quotes, and deriving it
-- from the same function that classifies a sample is what stops the two
-- drifting apart.
function count_class (want : bin_class_t) return unsigned is
variable n : unsigned(7 downto 0) := (others => '0');
begin
for c in 0 to N_BIN - 1 loop
if classify(std_logic_vector(to_unsigned(c / N_EPTYPE, 2)),
std_logic_vector(to_unsigned(c mod N_EPTYPE, 2))) = want then
n := n + 1;
end if;
end loop;
return n;
end function;
signal hits_r : hits_arr := (others => (others => '0'));
signal code_r : cov_err_t := C_NONE;
signal hit_r, err_r : std_logic := '0';
-- The end-of-test walk, one bin per cycle. A bounded walk rather than a
-- combinational reduction, so the report names the bins in a stable order
-- and the design has no wide comparator in it.
signal scan_r : unsigned(7 downto 0) := (others => '0');
signal sel : integer range 0 to N_BIN-1;
signal cls : bin_class_t;
signal n_cov : unsigned(7 downto 0);
-- Accumulators are held as unsigned rather than as range-constrained
-- integers: a constrained integer aborts simulation on overflow, which
-- turns a mutation into a crash instead of a measured kill.
signal c_smp, c_ill, c_exc, c_inc : unsigned(31 downto 0) := (others => '0');
begin
sel <= to_integer(unsigned(speed)) * N_EPTYPE + to_integer(unsigned(ep_type));
cls <= classify(speed, ep_type);
bin_class <= bc_code(cls);
bin_hits <= std_logic_vector(hits_r(sel));
hit_pulse <= hit_r;
cov_err <= err_r;
cov_code <= ce_code(code_r);
legal_total <= std_logic_vector(count_class(B_LEGAL));
unreach_total <= std_logic_vector(count_class(B_UNREACHABLE));
illegal_total <= std_logic_vector(count_class(B_ILLEGAL));
-- ---- Covered means N_MIN hits, not one. ----
cov_count : process (hits_r)
variable n : unsigned(7 downto 0);
begin
n := (others => '0');
for d in 0 to N_BIN - 1 loop
if classify(std_logic_vector(to_unsigned(d / N_EPTYPE, 2)),
std_logic_vector(to_unsigned(d mod N_EPTYPE, 2))) = B_LEGAL
and hits_r(d) >= to_unsigned(N_MIN, 8) then
n := n + 1;
end if;
end loop;
n_cov <= n;
end process;
covered_bins <= std_logic_vector(n_cov);
n_samples <= std_logic_vector(c_smp);
n_illegal <= std_logic_vector(c_ill);
n_exclusion <= std_logic_vector(c_exc);
n_incomplete <= std_logic_vector(c_inc);
process (clk, rst_n)
variable si : integer;
begin
if rst_n = '0' then
hits_r <= (others => (others => '0'));
code_r <= C_NONE;
hit_r <= '0';
err_r <= '0';
scan_r <= (others => '0');
c_smp <= (others => '0');
c_ill <= (others => '0');
c_exc <= (others => '0');
c_inc <= (others => '0');
elsif rising_edge(clk) then
code_r <= C_NONE;
hit_r <= '0';
err_r <= '0';
if eot = '1' then
-- ---- THE REPORT. One bin per cycle. ----
--
-- A legal bin that never reached N_MIN is reported by NAME. "84%
-- covered" is not a report; it is a number that requires somebody
-- to go and find out which 16% and why.
if scan_r < to_unsigned(N_BIN, 8) then
si := to_integer(scan_r);
if classify(std_logic_vector(to_unsigned(si / N_EPTYPE, 2)),
std_logic_vector(to_unsigned(si mod N_EPTYPE, 2))) = B_LEGAL
and hits_r(si) < to_unsigned(N_MIN, 8) then
err_r <= '1';
code_r <= C_INCOMPLETE;
c_inc <= c_inc + 1;
end if;
scan_r <= scan_r + 1;
end if;
else
scan_r <= (others => '0');
if sample = '1' then
c_smp <= c_smp + 1;
case cls is
when B_ILLEGAL =>
-- ---- An illegal bin is not a coverage hole. It is a
-- ---- FAILURE, and it fires when it is HIT.
--
-- This is the only kind of bin that does work by being
-- reached rather than by not being reached, and it is the
-- reason illegal_total is reported separately: it must never
-- contribute to a percentage in either direction.
err_r <= '1';
code_r <= C_ILLEGAL;
c_ill <= c_ill + 1;
when B_UNREACHABLE =>
-- ---- THE EXCLUSION WAS WRONG. ----
--
-- Something the coverage model asserted could not happen has
-- just happened. Either the exclusion is incorrect -- in
-- which case the coverage target has been understated ever
-- since -- or the design is enumerating a configuration the
-- specification forbids.
--
-- An exclusion applied as a post-processing filter can never
-- produce this message, because by then the sample has been
-- deleted.
err_r <= '1';
code_r <= C_EXCLUSION;
c_exc <= c_exc + 1;
when others =>
hit_r <= '1';
-- Saturating, so a long run cannot wrap a bin back under
-- N_MIN and un-cover it.
if hits_r(sel) /= x"FF" then
hits_r(sel) <= hits_r(sel) + 1;
end if;
end case;
end if;
end if;
end if;
end process;
end architecture rtl;10. Seeing a Bin Covered, an Illegal Hit, and a Falsified Exclusion
Three samples, three different outcomes
usb_coverage_collector — legal, illegal, excluded, incomplete
10 cyclesCycles 4 and 6 are both failures and they are different failures. One says the design did something forbidden; the other says the coverage model was wrong about the design. Collapsing them into "an error" loses the only information that decides who fixes it.
11. The Testbenches
The exhaustive sweep is every one of the 16 cross bins crossed with all four combinations of {sample, eot} — 64 pairs. But the checks that carry the chapter are three specific numbers and two specific events.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
check(legal_total === 8'd10,
"the coverage denominator is not 10 -- quoting a percentage against 16 understates coverage by 37%, and against 12 by 17%, which looks entirely plausible");
check(unreach_total === 8'd2,
"the two low-speed exclusions are not declared");
check(illegal_total === 8'd4,
"the reserved speed encoding is not declared illegal for every endpoint type");
check(legal_total + unreach_total + illegal_total === 8'd16,
"the three populations do not account for every cross bin");Both edges of N_MIN, on every legal bin:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
b_i = covered_bins;
hit(sp2[1:0], ep2[1:0]);
check(covered_bins == b_i,
"a bin was counted as covered after ONE sample -- one hit is an anecdote, and a collector satisfied by one sample is satisfied by a mutation that samples continuously");
for (k = 1; k < N_MIN; k = k + 1) hit(sp2[1:0], ep2[1:0]);
check(covered_bins == b_i + 1,
"a bin was not counted as covered after N_MIN samples");And the report, checked in both directions — an empty model names every legal bin, a full model names none:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
report_run;
check(rep_n == 10,
"the end-of-test report did not name every uncovered legal bin -- a percentage requires somebody to go and find out WHICH bins, and that is the work the report exists to do");11.1 Verilog testbench
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Testbench for usb_coverage_collector (Verilog-2005).
//
// THE CHECK THIS SUITE EXISTS FOR
//
// The DENOMINATOR. Sixteen cross bins; four are illegal, two are
// unreachable, ten are legal. A coverage figure quoted against 16 is wrong
// by 37%, and a coverage figure quoted against 12 -- the usual mistake, in
// which unreachable bins are counted but illegal ones are not -- is wrong by
// 17% and looks entirely plausible.
//
// legal_total must be 10
// unreach_total must be 2
// illegal_total must be 4
//
// Those three numbers are checked directly, because they are the numbers
// every percentage in the project is divided by.
//
// AND THE ONE THAT IS ALWAYS MISSING
//
// An excluded bin that is HIT. The exclusion said it could not happen; it
// happened. That must fail with the exclusion's name attached, and in a flow
// where exclusions are a post-processing filter it cannot, because the
// sample has already been deleted by the time anybody looks.
//
// WHAT IS EXHAUSTIVE HERE
//
// Every one of the 16 cross bins crossed with all four combinations of
// {sample, eot} = 64 pairs, each driven directly.
`timescale 1ns/1ps
module tb_cv_v;
localparam integer N_SPEED = 4;
localparam integer N_EPTYPE = 4;
localparam integer N_MIN = 2;
localparam integer N_BIN = N_SPEED * N_EPTYPE;
localparam [1:0] B_LEGAL=2'd0, B_ILLEGAL=2'd1, B_UNREACHABLE=2'd2;
localparam [1:0] C_NONE=2'd0, C_ILLEGAL=2'd1, C_EXCLUSION=2'd2,
C_INCOMPLETE=2'd3;
reg clk = 1'b0, rst_n = 1'b0;
reg sample = 1'b0, eot = 1'b0;
reg [1:0] speed = 2'd0, ep_type = 2'd0;
wire [1:0] bin_class, cov_code;
wire [7:0] bin_hits, covered_bins, legal_total, unreach_total, illegal_total;
wire hit_pulse, cov_err;
wire [31:0] n_samples, n_illegal, n_exclusion, n_incomplete;
usb_coverage_collector #(.N_SPEED(N_SPEED), .N_EPTYPE(N_EPTYPE),
.N_MIN(N_MIN)) dut (
.clk(clk), .rst_n(rst_n),
.sample(sample), .speed(speed), .ep_type(ep_type), .eot(eot),
.bin_class(bin_class), .bin_hits(bin_hits), .hit_pulse(hit_pulse),
.cov_err(cov_err), .cov_code(cov_code),
.covered_bins(covered_bins), .legal_total(legal_total),
.unreach_total(unreach_total), .illegal_total(illegal_total),
.n_samples(n_samples), .n_illegal(n_illegal),
.n_exclusion(n_exclusion), .n_incomplete(n_incomplete)
);
always #5 clk = ~clk;
integer errors = 0, checks = 0;
task check(input cond, input [1023:0] msg);
begin
checks = checks + 1;
if (!cond) begin
errors = errors + 1;
if (errors <= 25)
$display("FAIL @%0t: %0s | sp=%0d ep=%0d cls=%0d hits=%0d cov=%0d err=%b(%0d)",
$time, msg, speed, ep_type, bin_class, bin_hits,
covered_bins, cov_err, cov_code);
end
end
endtask
// ------------------------------------------------------------------
// The shadow model. Its own classification, written from the
// specification rather than copied from the design's function.
// ------------------------------------------------------------------
function [1:0] m_class;
input [1:0] sp;
input [1:0] ep;
begin
if (sp == 2'd3) m_class = B_ILLEGAL;
else if ((sp == 2'd0) && (ep == 2'd1)) m_class = B_UNREACHABLE;
else if ((sp == 2'd0) && (ep == 2'd2)) m_class = B_UNREACHABLE;
else m_class = B_LEGAL;
end
endfunction
reg [7:0] m_hits [0:N_BIN-1];
reg [7:0] m_scan;
reg [1:0] m_code;
reg m_hit, m_err;
integer m_smp, m_ill, m_exc, m_inc;
integer seen [0:63]; // 16 bins x 4 combinations of {sample, eot}
integer n_seen, n_steps;
task model_reset;
integer j;
begin
for (j = 0; j < N_BIN; j = j + 1) m_hits[j] = 8'd0;
m_scan = 8'd0; m_code = C_NONE; m_hit = 1'b0; m_err = 1'b0;
m_smp = 0; m_ill = 0; m_exc = 0; m_inc = 0;
for (j = 0; j < 64; j = j + 1) seen[j] = 0;
n_seen = 0; n_steps = 0;
end
endtask
integer q, idx, exp_cov, exp_leg, exp_unr, exp_ill;
task step(input sm, input [1:0] sp, input [1:0] ep, input eo);
reg [3:0] s;
reg [1:0] cl;
begin
sample = sm; speed = sp; ep_type = ep; eot = eo;
#1;
s = {sp, ep};
cl = m_class(sp, ep);
check(bin_class === cl,
"bin_class disagrees -- the collector's reachability declaration is not the specification's");
check(bin_hits === m_hits[s], "bin_hits disagrees with the shadow model");
check(hit_pulse === m_hit, "the hit pulse disagrees");
check(cov_err === m_err, "the coverage-error pulse disagrees");
check(cov_code === m_code, "cov_code disagrees");
check(!(hit_pulse && cov_err),
"a sample was reported as both a coverage hit and a coverage failure");
// ---- THE DENOMINATOR. Checked every cycle, because every
// ---- percentage in the project is divided by it.
exp_leg = 0; exp_unr = 0; exp_ill = 0; exp_cov = 0;
for (q = 0; q < N_BIN; q = q + 1) begin
case (m_class(q[3:2], q[1:0]))
B_LEGAL: begin
exp_leg = exp_leg + 1;
if (m_hits[q] >= N_MIN[7:0]) exp_cov = exp_cov + 1;
end
B_UNREACHABLE: exp_unr = exp_unr + 1;
default: exp_ill = exp_ill + 1;
endcase
end
check(legal_total === exp_leg[7:0],
"legal_total disagrees -- the coverage denominator is wrong, and so is every percentage quoted from it");
check(unreach_total === exp_unr[7:0], "unreach_total disagrees");
check(illegal_total === exp_ill[7:0], "illegal_total disagrees");
check(covered_bins === exp_cov[7:0], "covered_bins disagrees");
check(covered_bins <= legal_total,
"more bins are covered than there are legal bins to cover");
idx = s * 4 + (sm ? 2 : 0) + (eo ? 1 : 0);
if (seen[idx] == 0) begin seen[idx] = 1; n_seen = n_seen + 1; end
n_steps = n_steps + 1;
// ---- advance the shadow model ----
m_code = C_NONE; m_hit = 1'b0; m_err = 1'b0;
if (eo) begin
if (m_scan < N_BIN[7:0]) begin
if ((m_class(m_scan[3:2], m_scan[1:0]) == B_LEGAL)
&& (m_hits[m_scan[3:0]] < N_MIN[7:0])) begin
m_err = 1'b1; m_code = C_INCOMPLETE; m_inc = m_inc + 1;
end
m_scan = m_scan + 8'd1;
end
end else begin
m_scan = 8'd0;
if (sm) begin
m_smp = m_smp + 1;
case (cl)
B_ILLEGAL: begin
m_err = 1'b1; m_code = C_ILLEGAL; m_ill = m_ill + 1;
end
B_UNREACHABLE: begin
m_err = 1'b1; m_code = C_EXCLUSION; m_exc = m_exc + 1;
end
default: begin
m_hit = 1'b1;
if (m_hits[s] != 8'hFF) m_hits[s] = m_hits[s] + 8'd1;
end
endcase
end
end
@(posedge clk); #1;
sample = 1'b0; eot = 1'b0;
end
endtask
task idle(input integer n);
integer j;
begin for (j = 0; j < n; j = j + 1) step(1'b0, 2'd0, 2'd0, 1'b0); end
endtask
task hit(input [1:0] sp, input [1:0] ep);
begin step(1'b1, sp, ep, 1'b0); end
endtask
// Walk the end-of-test report to completion, one bin per cycle.
integer rep_n;
task report_run;
integer j;
begin
rep_n = n_incomplete;
for (j = 0; j < N_BIN + 2; j = j + 1) step(1'b0, 2'd0, 2'd0, 1'b1);
rep_n = n_incomplete - rep_n;
idle(1);
end
endtask
integer k, sp2, ep2, cb, b_i;
initial begin
model_reset;
repeat (3) @(posedge clk);
rst_n = 1'b1;
@(posedge clk); #1;
// ---- Phase A: the state after reset ----
check(covered_bins === 8'd0, "reset left bins covered");
check(cov_err === 1'b0, "reset asserted a coverage failure");
// ---- Phase B: THE DENOMINATOR, stated as three explicit numbers. ----
//
// Sixteen cross bins. Four illegal (the reserved speed encoding, any
// endpoint type). Two unreachable (low speed has no isochronous and no
// bulk endpoints). Ten legal -- and ten is what every percentage in the
// project must be divided by.
check(legal_total === 8'd10,
"the coverage denominator is not 10 -- quoting a percentage against 16 understates coverage by 37%, and against 12 by 17%, which looks entirely plausible");
check(unreach_total === 8'd2,
"the two low-speed exclusions are not declared");
check(illegal_total === 8'd4,
"the reserved speed encoding is not declared illegal for every endpoint type");
check(legal_total + unreach_total + illegal_total === 8'd16,
"the three populations do not account for every cross bin");
// ---- Phase B2: an EMPTY model names every legal bin it is missing. ----
//
// "0% covered" is a number, not a report. What is wanted is the list,
// and the list is exactly `legal_total` long when nothing has been
// sampled -- which is also the cheapest possible check that the
// reporting walk visits every bin rather than stopping early.
report_run;
check(rep_n == 10,
"the end-of-test report did not name every uncovered legal bin -- a percentage requires somebody to go and find out WHICH bins, and that is the work the report exists to do");
// ---- Phase C: N_MIN. Both edges. One hit is an anecdote. ----
for (sp2 = 0; sp2 < 3; sp2 = sp2 + 1) begin
for (ep2 = 0; ep2 < 4; ep2 = ep2 + 1) begin
if (m_class(sp2[1:0], ep2[1:0]) == B_LEGAL) begin
b_i = covered_bins;
hit(sp2[1:0], ep2[1:0]);
check(covered_bins == b_i,
"a bin was counted as covered after ONE sample -- one hit is an anecdote, and a collector satisfied by one sample is satisfied by a mutation that samples continuously");
for (k = 1; k < N_MIN; k = k + 1) hit(sp2[1:0], ep2[1:0]);
check(covered_bins == b_i + 1,
"a bin was not counted as covered after N_MIN samples");
end
end
end
check(covered_bins === legal_total,
"covering every legal bin did not fill the coverage model");
// ---- ...and a full model reports NOTHING at end of test. ----
report_run;
check(rep_n == 0,
"a fully covered model still reported incomplete bins");
// ---- Phase D: an ILLEGAL bin fires by being HIT. ----
for (ep2 = 0; ep2 < 4; ep2 = ep2 + 1) begin
b_i = n_illegal;
hit(2'd3, ep2[1:0]);
check(n_illegal == b_i + 1,
"the reserved speed encoding was sampled and not reported -- an illegal bin is the only kind that does work by being reached");
check(covered_bins === legal_total,
"an illegal sample changed the coverage figure");
end
// ---- Phase E: AN EXCLUSION WAS WRONG. ----
//
// The two low-speed bins are declared unreachable. Hitting one falsifies
// that claim, and the falsification must be reported by name -- which a
// post-processing exclusion filter can never do, because by then the
// sample has been deleted.
b_i = n_exclusion;
hit(2'd0, 2'd1); // low speed + isochronous
check(n_exclusion == b_i + 1,
"a bin declared unreachable was sampled and nothing was reported -- either the exclusion is wrong or the design is enumerating a forbidden configuration, and both are findings");
hit(2'd0, 2'd2); // low speed + bulk
check(n_exclusion == b_i + 2, "the second exclusion was not checked");
check(covered_bins === legal_total,
"an excluded sample was counted toward coverage");
// ---- Phase F: THE CROSS IS WHERE COVERAGE LIVES. ----
//
// Reset the model by re-running from scratch is not possible here, so
// this phase is measured as a DELTA against a fresh instance in the
// mutation runs; what is checked here is the structural claim: three
// speeds and four endpoint types, every axis value exercised, and only
// four of the ten cross bins reached.
//
// 3/3 speeds and 4/4 endpoint types is 100% on both coverpoints.
// 4/10 cross bins is 40%. Both statements describe the same stimulus.
check(legal_total === 8'd10,
"the cross has stopped being ten bins");
// ---- Phase G: EXHAUSTIVE. Every bin x every input combination. ----
for (cb = 0; cb < N_BIN; cb = cb + 1) begin
for (k = 0; k < 4; k = k + 1) begin
step(k[1], cb[3:2], cb[1:0], k[0]);
step(k[1], cb[3:2], cb[1:0], k[0]);
end
idle(1);
end
// ---- Phase H: random ----
for (k = 0; k < 30000; k = k + 1)
step(($unsigned($random) % 100) < 55,
$random, $random,
($unsigned($random) % 1000) < 9);
// ---- Phase I: the report, after everything. Every legal bin has been
// ---- hit many times by now, so nothing may be reported incomplete.
idle(2);
report_run;
check(rep_n == 0,
"the end-of-test report named a bin as incomplete although every legal bin had been covered");
check(covered_bins === legal_total,
"the model is not fully covered after the random phase");
// ---- Final agreement ----
check(n_samples === m_smp[31:0], "n_samples disagrees with the model");
check(n_illegal === m_ill[31:0], "n_illegal disagrees");
check(n_exclusion === m_exc[31:0], "n_exclusion disagrees");
check(n_incomplete === m_inc[31:0], "n_incomplete disagrees");
check(n_seen == 64, "not every cross bin was driven with every input combination");
check(n_illegal > 32'd0, "an illegal combination was never sampled");
check(n_exclusion > 32'd0, "an excluded combination was never sampled");
check(n_incomplete > 32'd0,
"the end-of-test report never named an incomplete bin, so its message was never exercised");
check(n_samples > 32'd1000, "too few samples to mean anything");
$display("REACH bin-x-input=%0d/64 steps=%0d", n_seen, n_steps);
$display("BINS legal=%0d unreachable=%0d illegal=%0d covered=%0d",
legal_total, unreach_total, illegal_total, covered_bins);
$display("COUNTERS samples=%0d illegal=%0d exclusion=%0d incomplete=%0d",
n_samples, n_illegal, n_exclusion, n_incomplete);
$display("%0s: %0d errors in %0d checks", (errors==0)?"PASS":"FAIL", errors, checks);
$finish;
end
endmodule11.2 SystemVerilog testbench
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Testbench for usb_coverage_collector (SystemVerilog).
//
// THE CHECK THIS SUITE EXISTS FOR
//
// The DENOMINATOR. Sixteen cross bins; four are illegal, two are
// unreachable, ten are legal. A coverage figure quoted against 16 is wrong
// by 37%, and a coverage figure quoted against 12 -- the usual mistake, in
// which unreachable bins are counted but illegal ones are not -- is wrong by
// 17% and looks entirely plausible.
//
// legal_total must be 10
// unreach_total must be 2
// illegal_total must be 4
//
// Those three numbers are checked directly, because they are the numbers
// every percentage in the project is divided by.
//
// AND THE ONE THAT IS ALWAYS MISSING
//
// An excluded bin that is HIT. The exclusion said it could not happen; it
// happened. That must fail with the exclusion's name attached, and in a flow
// where exclusions are a post-processing filter it cannot, because the
// sample has already been deleted by the time anybody looks.
//
// WHAT IS EXHAUSTIVE HERE
//
// Every one of the 16 cross bins crossed with all four combinations of
// {sample, eot} = 64 pairs, each driven directly.
`timescale 1ns/1ps
module tb_cv_sv;
import usb_cov_pkg::*;
localparam int N_SPEED = 4;
localparam int N_EPTYPE = 4;
localparam int N_MIN = 2;
localparam int N_BIN = N_SPEED * N_EPTYPE;
logic clk = 1'b0, rst_n = 1'b0;
logic sample = 1'b0, eot = 1'b0;
logic [1:0] speed = 2'd0, ep_type = 2'd0;
bin_class_e bin_class;
cov_err_e cov_code;
logic [7:0] bin_hits, covered_bins, legal_total, unreach_total, illegal_total;
logic hit_pulse, cov_err;
logic [31:0] n_samples, n_illegal, n_exclusion, n_incomplete;
usb_coverage_collector #(.N_SPEED(N_SPEED), .N_EPTYPE(N_EPTYPE),
.N_MIN(N_MIN)) dut (.*);
always #5 clk = ~clk;
int errors = 0, checks = 0;
task automatic check(input logic cond, input string msg);
begin
checks++;
if (!cond) begin
errors++;
if (errors <= 25)
$display("FAIL @%0t: %0s | sp=%0d ep=%0d cls=%0d hits=%0d cov=%0d err=%b(%0d)",
$time, msg, speed, ep_type, bin_class, bin_hits,
covered_bins, cov_err, cov_code);
end
end
endtask
// ------------------------------------------------------------------
// The shadow model. Its own classification, written from the
// specification rather than copied from the design's function.
// ------------------------------------------------------------------
function automatic bin_class_e m_class(input logic [1:0] sp,
input logic [1:0] ep);
begin
if (sp == 2'd3) m_class = B_ILLEGAL;
else if ((sp == 2'd0) && (ep == 2'd1)) m_class = B_UNREACHABLE;
else if ((sp == 2'd0) && (ep == 2'd2)) m_class = B_UNREACHABLE;
else m_class = B_LEGAL;
end
endfunction
logic [7:0] m_hits [N_BIN];
logic [7:0] m_scan;
cov_err_e m_code;
logic m_hit, m_err;
int m_smp, m_ill, m_exc, m_inc;
int seen [64]; // 16 bins x 4 combinations of {sample, eot}
int n_seen, n_steps;
task automatic model_reset();
int j;
begin
for (j = 0; j < N_BIN; j++) m_hits[j] = 8'd0;
m_scan = 8'd0; m_code = C_NONE; m_hit = 1'b0; m_err = 1'b0;
m_smp = 0; m_ill = 0; m_exc = 0; m_inc = 0;
for (j = 0; j < 64; j++) seen[j] = 0;
n_seen = 0; n_steps = 0;
end
endtask
int q, idx, exp_cov, exp_leg, exp_unr, exp_ill;
task automatic step(input logic sm, input logic [1:0] sp,
input logic [1:0] ep, input logic eo);
logic [3:0] s;
bin_class_e cl;
begin
sample = sm; speed = sp; ep_type = ep; eot = eo;
#1;
s = {sp, ep};
cl = m_class(sp, ep);
check(bin_class === cl,
"bin_class disagrees -- the collector's reachability declaration is not the specification's");
check(bin_hits === m_hits[s], "bin_hits disagrees with the shadow model");
check(hit_pulse === m_hit, "the hit pulse disagrees");
check(cov_err === m_err, "the coverage-error pulse disagrees");
check(cov_code === m_code, "cov_code disagrees");
check(!(hit_pulse && cov_err),
"a sample was reported as both a coverage hit and a coverage failure");
// ---- THE DENOMINATOR. Checked every cycle, because every
// ---- percentage in the project is divided by it.
exp_leg = 0; exp_unr = 0; exp_ill = 0; exp_cov = 0;
for (q = 0; q < N_BIN; q++) begin
case (m_class(q[3:2], q[1:0]))
B_LEGAL: begin
exp_leg = exp_leg + 1;
if (m_hits[q] >= 8'(N_MIN)) exp_cov = exp_cov + 1;
end
B_UNREACHABLE: exp_unr = exp_unr + 1;
default: exp_ill = exp_ill + 1;
endcase
end
check(legal_total === 8'(exp_leg),
"legal_total disagrees -- the coverage denominator is wrong, and so is every percentage quoted from it");
check(unreach_total === 8'(exp_unr), "unreach_total disagrees");
check(illegal_total === 8'(exp_ill), "illegal_total disagrees");
check(covered_bins === 8'(exp_cov), "covered_bins disagrees");
check(covered_bins <= legal_total,
"more bins are covered than there are legal bins to cover");
idx = s * 4 + (sm ? 2 : 0) + (eo ? 1 : 0);
if (seen[idx] == 0) begin seen[idx] = 1; n_seen = n_seen + 1; end
n_steps++;
// ---- advance the shadow model ----
m_code = C_NONE; m_hit = 1'b0; m_err = 1'b0;
if (eo) begin
if (m_scan < 8'(N_BIN)) begin
if ((m_class(m_scan[3:2], m_scan[1:0]) == B_LEGAL)
&& (m_hits[m_scan[3:0]] < 8'(N_MIN))) begin
m_err = 1'b1; m_code = C_INCOMPLETE; m_inc = m_inc + 1;
end
m_scan = m_scan + 8'd1;
end
end else begin
m_scan = 8'd0;
if (sm) begin
m_smp++;
case (cl)
B_ILLEGAL: begin
m_err = 1'b1; m_code = C_ILLEGAL; m_ill = m_ill + 1;
end
B_UNREACHABLE: begin
m_err = 1'b1; m_code = C_EXCLUSION; m_exc = m_exc + 1;
end
default: begin
m_hit = 1'b1;
if (m_hits[s] != 8'hFF) m_hits[s] = m_hits[s] + 8'd1;
end
endcase
end
end
@(posedge clk); #1;
sample = 1'b0; eot = 1'b0;
end
endtask
task automatic idle(input int n);
repeat (n) step(1'b0, 2'd0, 2'd0, 1'b0);
endtask
task automatic hit(input logic [1:0] sp, input logic [1:0] ep);
step(1'b1, sp, ep, 1'b0);
endtask
// Walk the end-of-test report to completion, one bin per cycle.
int rep_n;
task automatic report_run();
rep_n = n_incomplete;
repeat (N_BIN + 2) step(1'b0, 2'd0, 2'd0, 1'b1);
rep_n = n_incomplete - rep_n;
idle(1);
endtask
int k, sp2, ep2, cb, b_i;
initial begin
model_reset();
repeat (3) @(posedge clk);
rst_n = 1'b1;
@(posedge clk); #1;
// ---- Phase A: the state after reset ----
check(covered_bins === 8'd0, "reset left bins covered");
check(cov_err === 1'b0, "reset asserted a coverage failure");
// ---- Phase B: THE DENOMINATOR, stated as three explicit numbers. ----
//
// Sixteen cross bins. Four illegal (the reserved speed encoding, any
// endpoint type). Two unreachable (low speed has no isochronous and no
// bulk endpoints). Ten legal -- and ten is what every percentage in the
// project must be divided by.
check(legal_total === 8'd10,
"the coverage denominator is not 10 -- quoting a percentage against 16 understates coverage by 37%, and against 12 by 17%, which looks entirely plausible");
check(unreach_total === 8'd2,
"the two low-speed exclusions are not declared");
check(illegal_total === 8'd4,
"the reserved speed encoding is not declared illegal for every endpoint type");
check(legal_total + unreach_total + illegal_total === 8'd16,
"the three populations do not account for every cross bin");
// ---- Phase B2: an EMPTY model names every legal bin it is missing. ----
//
// "0% covered" is a number, not a report. What is wanted is the list,
// and the list is exactly `legal_total` long when nothing has been
// sampled -- which is also the cheapest possible check that the
// reporting walk visits every bin rather than stopping early.
report_run();
check(rep_n == 10,
"the end-of-test report did not name every uncovered legal bin -- a percentage requires somebody to go and find out WHICH bins, and that is the work the report exists to do");
// ---- Phase C: N_MIN. Both edges. One hit is an anecdote. ----
for (sp2 = 0; sp2 < 3; sp2++) begin
for (ep2 = 0; ep2 < 4; ep2++) begin
if (m_class(2'(sp2), 2'(ep2)) == B_LEGAL) begin
b_i = covered_bins;
hit(2'(sp2), 2'(ep2));
check(covered_bins == b_i,
"a bin was counted as covered after ONE sample -- one hit is an anecdote, and a collector satisfied by one sample is satisfied by a mutation that samples continuously");
for (k = 1; k < N_MIN; k++) hit(2'(sp2), 2'(ep2));
check(covered_bins == b_i + 1,
"a bin was not counted as covered after N_MIN samples");
end
end
end
check(covered_bins === legal_total,
"covering every legal bin did not fill the coverage model");
// ---- ...and a full model reports NOTHING at end of test. ----
report_run();
check(rep_n == 0,
"a fully covered model still reported incomplete bins");
// ---- Phase D: an ILLEGAL bin fires by being HIT. ----
for (ep2 = 0; ep2 < 4; ep2++) begin
b_i = n_illegal;
hit(2'd3, 2'(ep2));
check(n_illegal == b_i + 1,
"the reserved speed encoding was sampled and not reported -- an illegal bin is the only kind that does work by being reached");
check(covered_bins === legal_total,
"an illegal sample changed the coverage figure");
end
// ---- Phase E: AN EXCLUSION WAS WRONG. ----
//
// The two low-speed bins are declared unreachable. Hitting one falsifies
// that claim, and the falsification must be reported by name -- which a
// post-processing exclusion filter can never do, because by then the
// sample has been deleted.
b_i = n_exclusion;
hit(2'd0, 2'd1); // low speed + isochronous
check(n_exclusion == b_i + 1,
"a bin declared unreachable was sampled and nothing was reported -- either the exclusion is wrong or the design is enumerating a forbidden configuration, and both are findings");
hit(2'd0, 2'd2); // low speed + bulk
check(n_exclusion == b_i + 2, "the second exclusion was not checked");
check(covered_bins === legal_total,
"an excluded sample was counted toward coverage");
// ---- Phase F: THE CROSS IS WHERE COVERAGE LIVES. ----
//
// Reset the model by re-running from scratch is not possible here, so
// this phase is measured as a DELTA against a fresh instance in the
// mutation runs; what is checked here is the structural claim: three
// speeds and four endpoint types, every axis value exercised, and only
// four of the ten cross bins reached.
//
// 3/3 speeds and 4/4 endpoint types is 100% on both coverpoints.
// 4/10 cross bins is 40%. Both statements describe the same stimulus.
check(legal_total === 8'd10,
"the cross has stopped being ten bins");
// ---- Phase G: EXHAUSTIVE. Every bin x every input combination. ----
for (cb = 0; cb < N_BIN; cb++) begin
for (k = 0; k < 4; k++) begin
step(1'(k[1]), 2'(cb[3:2]), 2'(cb[1:0]), 1'(k[0]));
step(1'(k[1]), 2'(cb[3:2]), 2'(cb[1:0]), 1'(k[0]));
end
idle(1);
end
// ---- Phase H: random ----
for (k = 0; k < 30000; k++)
step($urandom_range(0,99) < 55,
2'($urandom()), 2'($urandom()),
$urandom_range(0,999) < 9);
// ---- Phase I: the report, after everything. Every legal bin has been
// ---- hit many times by now, so nothing may be reported incomplete.
idle(2);
report_run();
check(rep_n == 0,
"the end-of-test report named a bin as incomplete although every legal bin had been covered");
check(covered_bins === legal_total,
"the model is not fully covered after the random phase");
// ---- Final agreement ----
check(n_samples === 32'(m_smp), "n_samples disagrees with the model");
check(n_illegal === 32'(m_ill), "n_illegal disagrees");
check(n_exclusion === 32'(m_exc), "n_exclusion disagrees");
check(n_incomplete === 32'(m_inc), "n_incomplete disagrees");
check(n_seen == 64, "not every cross bin was driven with every input combination");
check(n_illegal > 32'd0, "an illegal combination was never sampled");
check(n_exclusion > 32'd0, "an excluded combination was never sampled");
check(n_incomplete > 32'd0,
"the end-of-test report never named an incomplete bin, so its message was never exercised");
check(n_samples > 32'd1000, "too few samples to mean anything");
$display("REACH bin-x-input=%0d/64 steps=%0d", n_seen, n_steps);
$display("BINS legal=%0d unreachable=%0d illegal=%0d covered=%0d",
legal_total, unreach_total, illegal_total, covered_bins);
$display("COUNTERS samples=%0d illegal=%0d exclusion=%0d incomplete=%0d",
n_samples, n_illegal, n_exclusion, n_incomplete);
$display("%0s: %0d errors in %0d checks", (errors==0)?"PASS":"FAIL", errors, checks);
$finish;
end
endmodule11.3 VHDL testbench
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
-- Testbench for usb_coverage_collector (VHDL-2008).
--
-- THE CHECK THIS SUITE EXISTS FOR
--
-- The DENOMINATOR. Sixteen cross bins; four are illegal, two are
-- unreachable, ten are legal. A coverage figure quoted against 16 is wrong
-- by 37%, and a coverage figure quoted against 12 -- the usual mistake, in
-- which unreachable bins are counted but illegal ones are not -- is wrong by
-- 17% and looks entirely plausible.
--
-- legal_total must be 10
-- unreach_total must be 2
-- illegal_total must be 4
--
-- Those three numbers are checked directly, because they are the numbers
-- every percentage in the project is divided by.
--
-- AND THE ONE THAT IS ALWAYS MISSING
--
-- An excluded bin that is HIT. The exclusion said it could not happen; it
-- happened. That must fail with the exclusion's name attached, and in a flow
-- where exclusions are a post-processing filter it cannot, because the
-- sample has already been deleted by the time anybody looks.
--
-- WHAT IS EXHAUSTIVE HERE
--
-- Every one of the 16 cross bins crossed with all four combinations of
-- (sample, eot) = 64 pairs, each driven directly.
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
use std.textio.all;
use work.usb_cov_pkg.all;
entity tb_cv_vhdl is
end entity tb_cv_vhdl;
architecture sim of tb_cv_vhdl is
constant N_SPEED : integer := 4;
constant N_EPTYPE : integer := 4;
constant N_MIN : integer := 2;
constant N_BIN : integer := N_SPEED * N_EPTYPE;
signal clk : std_logic := '0';
signal rst_n : std_logic := '0';
signal done : boolean := false;
signal sample, eot : std_logic := '0';
signal speed, ep_type : std_logic_vector(1 downto 0) := "00";
signal bin_class, cov_code : std_logic_vector(1 downto 0);
signal bin_hits, covered_bins : std_logic_vector(7 downto 0);
signal legal_total, unreach_total, illegal_total : std_logic_vector(7 downto 0);
signal hit_pulse, cov_err : std_logic;
signal n_samples, n_illegal, n_exclusion, n_incomplete : std_logic_vector(31 downto 0);
begin
dut : entity work.usb_coverage_collector
generic map (N_SPEED => N_SPEED, N_EPTYPE => N_EPTYPE, N_MIN => N_MIN)
port map (
clk => clk, rst_n => rst_n,
sample => sample, speed => speed, ep_type => ep_type, eot => eot,
bin_class => bin_class, bin_hits => bin_hits, hit_pulse => hit_pulse,
cov_err => cov_err, cov_code => cov_code,
covered_bins => covered_bins, legal_total => legal_total,
unreach_total => unreach_total, illegal_total => illegal_total,
n_samples => n_samples, n_illegal => n_illegal,
n_exclusion => n_exclusion, n_incomplete => n_incomplete
);
clk <= (not clk) after 5 ns when not done else '0';
stim : process
type hits_arr is array (0 to N_BIN-1) of unsigned(7 downto 0);
type seen_arr is array (0 to 63) of integer;
variable errors, checks : integer := 0;
-- ---- The shadow model. Its own classification, written from the
-- ---- specification rather than copied from the design's function.
impure function m_class (sp, ep : std_logic_vector(1 downto 0))
return bin_class_t is
begin
if sp = "11" then return B_ILLEGAL;
elsif sp = "00" and ep = "01" then return B_UNREACHABLE;
elsif sp = "00" and ep = "10" then return B_UNREACHABLE;
else return B_LEGAL;
end if;
end function;
variable m_hits : hits_arr := (others => (others => '0'));
variable m_scan : unsigned(7 downto 0) := (others => '0');
variable m_code : cov_err_t := C_NONE;
variable m_hit, m_err : std_logic := '0';
variable m_smp, m_ill, m_exc, m_inc : integer := 0;
variable seen : seen_arr := (others => 0);
variable n_seen, n_steps : integer := 0;
-- A deterministic LFSR, so a rerun reproduces exactly the same traffic.
variable lfsr : unsigned(31 downto 0) := x"C0FFEE11";
impure function rnd32 return unsigned is
begin
lfsr := lfsr(30 downto 0) &
(lfsr(31) xor lfsr(21) xor lfsr(1) xor lfsr(0));
return lfsr;
end function;
-- Only the low 30 bits are converted: a full 32-bit unsigned does not
-- fit in VHDL's INTEGER, and to_integer aborts the run rather than
-- wrapping.
impure function rnd_nat return integer is
variable u : unsigned(31 downto 0);
begin
u := rnd32;
return to_integer(u(29 downto 0));
end function;
impure function rnd2 return std_logic_vector is
variable u : unsigned(31 downto 0);
begin
u := rnd32;
return std_logic_vector(u(1 downto 0));
end function;
impure function rnd_lt (pct, base : integer) return std_logic is
begin
if (rnd_nat mod base) < pct then return '1'; else return '0'; end if;
end function;
procedure chk (cond : boolean; msg : string) is
begin
checks := checks + 1;
if not cond then
errors := errors + 1;
if errors <= 25 then
report "FAIL: " & msg &
" | cls=" & integer'image(bin_class_t'pos(m_class(speed, ep_type))) &
" cov=" & integer'image(to_integer(unsigned(covered_bins)))
severity note;
end if;
end if;
end procedure;
procedure step (sm : std_logic; sp, ep : std_logic_vector(1 downto 0);
eo : std_logic) is
variable s, idx, si : integer;
variable cl : bin_class_t;
variable exp_cov, exp_leg, exp_unr, exp_ill : integer;
begin
sample <= sm; speed <= sp; ep_type <= ep; eot <= eo;
wait for 1 ns;
s := to_integer(unsigned(sp)) * N_EPTYPE + to_integer(unsigned(ep));
cl := m_class(sp, ep);
chk(bin_class = bc_code(cl),
"bin_class disagrees -- the collector's reachability declaration is not the specification's");
chk(bin_hits = std_logic_vector(m_hits(s)),
"bin_hits disagrees with the shadow model");
chk(hit_pulse = m_hit, "the hit pulse disagrees");
chk(cov_err = m_err, "the coverage-error pulse disagrees");
chk(cov_code = ce_code(m_code), "cov_code disagrees");
chk(not (hit_pulse = '1' and cov_err = '1'),
"a sample was reported as both a coverage hit and a coverage failure");
-- ---- THE DENOMINATOR. Checked every cycle, because every
-- ---- percentage in the project is divided by it.
exp_leg := 0; exp_unr := 0; exp_ill := 0; exp_cov := 0;
for q in 0 to N_BIN - 1 loop
case m_class(std_logic_vector(to_unsigned(q / N_EPTYPE, 2)),
std_logic_vector(to_unsigned(q mod N_EPTYPE, 2))) is
when B_LEGAL =>
exp_leg := exp_leg + 1;
if m_hits(q) >= to_unsigned(N_MIN, 8) then
exp_cov := exp_cov + 1;
end if;
when B_UNREACHABLE => exp_unr := exp_unr + 1;
when others => exp_ill := exp_ill + 1;
end case;
end loop;
chk(unsigned(legal_total) = to_unsigned(exp_leg, 8),
"legal_total disagrees -- the coverage denominator is wrong, and so is every percentage quoted from it");
chk(unsigned(unreach_total) = to_unsigned(exp_unr, 8), "unreach_total disagrees");
chk(unsigned(illegal_total) = to_unsigned(exp_ill, 8), "illegal_total disagrees");
chk(unsigned(covered_bins) = to_unsigned(exp_cov, 8), "covered_bins disagrees");
chk(unsigned(covered_bins) <= unsigned(legal_total),
"more bins are covered than there are legal bins to cover");
idx := s * 4;
if sm = '1' then idx := idx + 2; end if;
if eo = '1' then idx := idx + 1; end if;
if seen(idx) = 0 then seen(idx) := 1; n_seen := n_seen + 1; end if;
n_steps := n_steps + 1;
-- ---- advance the shadow model ----
m_code := C_NONE; m_hit := '0'; m_err := '0';
if eo = '1' then
if m_scan < to_unsigned(N_BIN, 8) then
si := to_integer(m_scan);
if m_class(std_logic_vector(to_unsigned(si / N_EPTYPE, 2)),
std_logic_vector(to_unsigned(si mod N_EPTYPE, 2))) = B_LEGAL
and m_hits(si) < to_unsigned(N_MIN, 8) then
m_err := '1'; m_code := C_INCOMPLETE; m_inc := m_inc + 1;
end if;
m_scan := m_scan + 1;
end if;
else
m_scan := (others => '0');
if sm = '1' then
m_smp := m_smp + 1;
case cl is
when B_ILLEGAL =>
m_err := '1'; m_code := C_ILLEGAL; m_ill := m_ill + 1;
when B_UNREACHABLE =>
m_err := '1'; m_code := C_EXCLUSION; m_exc := m_exc + 1;
when others =>
m_hit := '1';
if m_hits(s) /= x"FF" then m_hits(s) := m_hits(s) + 1; end if;
end case;
end if;
end if;
wait until rising_edge(clk);
wait for 1 ns;
sample <= '0'; eot <= '0';
end procedure;
procedure idle (n : integer) is
begin
for j in 1 to n loop
step('0', "00", "00", '0');
end loop;
end procedure;
procedure hit (sp, ep : std_logic_vector(1 downto 0)) is
begin
step('1', sp, ep, '0');
end procedure;
variable rep_n : integer := 0;
-- Walk the end-of-test report to completion, one bin per cycle.
procedure report_run is
begin
rep_n := to_integer(unsigned(n_incomplete));
for j in 1 to N_BIN + 2 loop
step('0', "00", "00", '1');
end loop;
rep_n := to_integer(unsigned(n_incomplete)) - rep_n;
idle(1);
end procedure;
function sv (n : integer) return std_logic_vector is
begin
return std_logic_vector(to_unsigned(n, 2));
end function;
variable b_i : integer := 0;
variable sm_v, eo_v : std_logic;
variable ln : line;
begin
wait for 33 ns;
rst_n <= '1';
wait until rising_edge(clk);
wait for 1 ns;
-- ---- Phase A: the state after reset ----
chk(unsigned(covered_bins) = 0, "reset left bins covered");
chk(cov_err = '0', "reset asserted a coverage failure");
-- ---- Phase B: THE DENOMINATOR, stated as three explicit numbers. ----
chk(unsigned(legal_total) = 10,
"the coverage denominator is not 10 -- quoting a percentage against 16 understates coverage by 37%, and against 12 by 17%, which looks entirely plausible");
chk(unsigned(unreach_total) = 2,
"the two low-speed exclusions are not declared");
chk(unsigned(illegal_total) = 4,
"the reserved speed encoding is not declared illegal for every endpoint type");
chk(unsigned(legal_total) + unsigned(unreach_total)
+ unsigned(illegal_total) = 16,
"the three populations do not account for every cross bin");
-- ---- Phase B2: an EMPTY model names every legal bin it is missing. ----
report_run;
chk(rep_n = 10,
"the end-of-test report did not name every uncovered legal bin -- a percentage requires somebody to go and find out WHICH bins, and that is the work the report exists to do");
-- ---- Phase C: N_MIN. Both edges. One hit is an anecdote. ----
for sp2 in 0 to 2 loop
for ep2 in 0 to 3 loop
if m_class(sv(sp2), sv(ep2)) = B_LEGAL then
b_i := to_integer(unsigned(covered_bins));
hit(sv(sp2), sv(ep2));
chk(to_integer(unsigned(covered_bins)) = b_i,
"a bin was counted as covered after ONE sample -- one hit is an anecdote, and a collector satisfied by one sample is satisfied by a mutation that samples continuously");
for k in 1 to N_MIN - 1 loop
hit(sv(sp2), sv(ep2));
end loop;
chk(to_integer(unsigned(covered_bins)) = b_i + 1,
"a bin was not counted as covered after N_MIN samples");
end if;
end loop;
end loop;
chk(covered_bins = legal_total,
"covering every legal bin did not fill the coverage model");
-- ---- ...and a full model reports NOTHING at end of test. ----
report_run;
chk(rep_n = 0, "a fully covered model still reported incomplete bins");
-- ---- Phase D: an ILLEGAL bin fires by being HIT. ----
for ep2 in 0 to 3 loop
b_i := to_integer(unsigned(n_illegal));
hit("11", sv(ep2));
chk(to_integer(unsigned(n_illegal)) = b_i + 1,
"the reserved speed encoding was sampled and not reported -- an illegal bin is the only kind that does work by being reached");
chk(covered_bins = legal_total,
"an illegal sample changed the coverage figure");
end loop;
-- ---- Phase E: AN EXCLUSION WAS WRONG. ----
b_i := to_integer(unsigned(n_exclusion));
hit("00", "01"); -- low speed + isochronous
chk(to_integer(unsigned(n_exclusion)) = b_i + 1,
"a bin declared unreachable was sampled and nothing was reported -- either the exclusion is wrong or the design is enumerating a forbidden configuration, and both are findings");
hit("00", "10"); -- low speed + bulk
chk(to_integer(unsigned(n_exclusion)) = b_i + 2,
"the second exclusion was not checked");
chk(covered_bins = legal_total,
"an excluded sample was counted toward coverage");
-- ---- Phase F: THE CROSS IS WHERE COVERAGE LIVES. ----
chk(unsigned(legal_total) = 10, "the cross has stopped being ten bins");
-- ---- Phase G: EXHAUSTIVE. Every bin x every input combination. ----
for cb in 0 to N_BIN - 1 loop
for k in 0 to 3 loop
if (k / 2) mod 2 = 1 then sm_v := '1'; else sm_v := '0'; end if;
if k mod 2 = 1 then eo_v := '1'; else eo_v := '0'; end if;
step(sm_v, sv(cb / N_EPTYPE), sv(cb mod N_EPTYPE), eo_v);
step(sm_v, sv(cb / N_EPTYPE), sv(cb mod N_EPTYPE), eo_v);
end loop;
idle(1);
end loop;
-- ---- Phase H: random ----
for k in 0 to 29999 loop
sm_v := rnd_lt(55, 100);
eo_v := rnd_lt(9, 1000);
step(sm_v, rnd2, rnd2, eo_v);
end loop;
-- ---- Phase I: the report, after everything. ----
idle(2);
report_run;
chk(rep_n = 0,
"the end-of-test report named a bin as incomplete although every legal bin had been covered");
chk(covered_bins = legal_total,
"the model is not fully covered after the random phase");
-- ---- Final agreement ----
chk(to_integer(unsigned(n_samples)) = m_smp, "n_samples disagrees with the model");
chk(to_integer(unsigned(n_illegal)) = m_ill, "n_illegal disagrees");
chk(to_integer(unsigned(n_exclusion)) = m_exc, "n_exclusion disagrees");
chk(to_integer(unsigned(n_incomplete)) = m_inc, "n_incomplete disagrees");
chk(n_seen = 64, "not every cross bin was driven with every input combination");
chk(to_integer(unsigned(n_illegal)) > 0, "an illegal combination was never sampled");
chk(to_integer(unsigned(n_exclusion)) > 0, "an excluded combination was never sampled");
chk(to_integer(unsigned(n_incomplete)) > 0,
"the end-of-test report never named an incomplete bin, so its message was never exercised");
chk(to_integer(unsigned(n_samples)) > 1000, "too few samples to mean anything");
write(ln, string'("REACH bin-x-input=") & integer'image(n_seen) &
"/64 steps=" & integer'image(n_steps));
writeline(output, ln);
write(ln, string'("BINS legal=") & integer'image(to_integer(unsigned(legal_total))) &
" unreachable=" & integer'image(to_integer(unsigned(unreach_total))) &
" illegal=" & integer'image(to_integer(unsigned(illegal_total))) &
" covered=" & integer'image(to_integer(unsigned(covered_bins))));
writeline(output, ln);
write(ln, string'("COUNTERS samples=") & integer'image(to_integer(unsigned(n_samples))) &
" illegal=" & integer'image(to_integer(unsigned(n_illegal))) &
" exclusion=" & integer'image(to_integer(unsigned(n_exclusion))) &
" incomplete=" & integer'image(to_integer(unsigned(n_incomplete))));
writeline(output, ln);
if errors = 0 then
write(ln, string'("PASS: 0 errors in ") & integer'image(checks) & " checks");
else
write(ln, string'("FAIL: ") & integer'image(errors) & " errors in " &
integer'image(checks) & " checks");
end if;
writeline(output, ln);
done <= true;
wait;
end process;
end architecture sim;12. Exhaustive Verification
| Measure | Verilog | SystemVerilog | VHDL |
|---|---|---|---|
| bin x input | 64 / 64 | 64 / 64 | 64 / 64 |
| Steps | 30229 | 30229 | 30229 |
| Checks executed | 332571 | 332571 | 332571 |
| legal bins (the denominator) | 10 | 10 | 10 |
| unreachable bins | 2 | 2 | 2 |
| illegal bins | 4 | 4 | 4 |
| bins covered at the end | 10 / 10 | 10 / 10 | 10 / 10 |
| samples taken | 16376 | 16482 | 16487 |
| illegal combinations sampled | 4065 | 4179 | 4105 |
| exclusions falsified | 2085 | 2040 | 2131 |
| incomplete bins named | 10 | 10 | 10 |
| Result | PASS | PASS | PASS |
The three population numbers are identical in all three languages because they are derived from the declaration rather than typed in — which is the property the chapter is about.
The exclusions falsified row is large only because the random phase drives every combination uniformly, including the two the specification forbids. In a real environment that number should be zero, and any non-zero value is the most interesting line in the report.
13. Mutation Testing
| # | Mutation | Verilog | SysVer | VHDL |
|---|---|---|---|---|
| W2 | illegal bins counted as coverable — denominator 16 | 117795 | 118137 | 117915 |
| W1 | the exclusions are not declared — denominator 12 | 104386 | 104251 | 104760 |
| W7 | the sample qualifier is ignored | 23188 | 23007 | 20832 |
| W3 | an illegal combination is sampled and not reported | 8136 | 8364 | 8216 |
| W4 | a falsified exclusion is not reported | 4174 | 4084 | 4266 |
| W6 | the end-of-test report names nothing | 23 | 23 | 23 |
| W5 | one hit counts as covered | 20 | 20 | 20 |
| — | unmutated baseline | 0 | 0 | 0 |
All seven die in all three languages, all counts distinct.
W1 and W2 are the two denominator mutations and they are the two largest, which is the right shape: the denominator is checked on every single step, so a wrong one fails continuously. That is deliberate. A number that every percentage depends on is worth checking more often than anything else in the suite.
W4 is the mutation that most environments would score zero. A falsified exclusion produces no wrong data — the sample is simply not counted, which is what an exclusion is for. It is caught only because the collector treats the exclusion as a claim and checks it.
14. Debugging Walkthrough: The 98% That Meant Nothing
The report. A USB device project reports 98% functional coverage and signs off. Three escapes are found in the first month of silicon, all in isochronous transfers at full speed.
Step 1 — was that combination covered? Open the report. speed x ep_type cross: 100%.
Step 2 — how many bins does the cross have? Twelve. The tool generated 16 and an exclusion file removed four.
Step 3 — which four? The exclusion file removes the entire ISOC column with a comment reading # not supported yet. It was written eighteen months earlier, when isochronous support genuinely was not implemented.
Step 4 — so the cross is 12 of 12. Every isochronous bin is excluded, isochronous support shipped a year ago, and the coverage model has been reporting 100% on a column it deletes.
Step 5 — why did nothing catch it? Because the exclusion is a post-processing filter. Isochronous transfers were being driven — thousands of them — and every sample was deleted before anybody looked. The evidence that the exclusion was wrong was generated on every single run and discarded on every single run.
Step 6 — what would have caught it. An exclusion implemented as a bin class rather than a filter: the first isochronous sample falsifies the claim and fails the run, eighteen months earlier, with the exclusion's own comment in the message.
15. The SystemVerilog Covergroup This Block Models
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// The hardware collector above is what a covergroup DOES. This is the same
// model in the language that has the feature, and the differences are worth
// reading carefully -- particularly what SystemVerilog gives you and what it
// still leaves to you.
covergroup cg_usb_config @(posedge clk iff sample);
option.per_instance = 1;
// ---- at_least is N_MIN. It defaults to 1, which is the default that
// ---- makes a coverage report an anecdote.
option.at_least = 2;
cp_speed : coverpoint speed {
bins low = {2'd0};
bins full = {2'd1};
bins high = {2'd2};
// NOT ignore_bins. The reserved encoding must never be enumerated, so
// hitting it is a FAILURE -- and an illegal_bins hit is an error, which
// is exactly the semantics wanted. ignore_bins would delete it silently.
illegal_bins reserved = {2'd3};
}
cp_eptype : coverpoint ep_type {
bins control = {2'd0};
bins isoc = {2'd1};
bins bulk = {2'd2};
bins interrupt = {2'd3};
}
// ---- THE CROSS. Both coverpoints above fill in a handful of samples;
// ---- this is the thing that takes a campaign to fill.
x_speed_eptype : cross cp_speed, cp_eptype {
// ---- The two specification exclusions, written HERE, next to the
// ---- bins they exclude, with the reason attached.
//
// USB low speed supports only control and interrupt endpoints. This is
// not "not implemented yet" and it is not "hard to reach" -- it is the
// specification, and the distinction matters because the first two
// expire and this one does not.
ignore_bins ls_has_no_isoc =
binsof(cp_speed.low) && binsof(cp_eptype.isoc);
ignore_bins ls_has_no_bulk =
binsof(cp_speed.low) && binsof(cp_eptype.bulk);
}
endgroup15.1 Sampling, and the thing iff is doing
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// `@(posedge clk iff sample)` is not decoration. Without the iff the
// covergroup samples on EVERY clock edge, which is mutation W7:
//
// - every bin fills almost immediately, from idle cycles
// - at_least is satisfied by repetition of nothing
// - the report reaches 100% and means nothing at all
//
// The qualifier is what makes a sample correspond to an EVENT. A coverage
// model sampled on a free-running clock measures the clock.
covergroup cg_usb_config @(posedge clk iff sample);
// ...and the corresponding trap on the other side: sampling from a task
// called by the driver rather than from the monitor.
//
// the DRIVER knows what it INTENDED to send
// the MONITOR knows what actually appeared on the bus
//
// Coverage sampled in the driver records the stimulus, not the design's
// behaviour, and will read 100% on a design that ignores its inputs
// entirely. Sample in the monitor, always -- for the same reason chapter
// 24.1's checker may not read the DUT's own state.15.2 Reporting, which is the part nobody writes
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
class usb_coverage_report extends uvm_component;
`uvm_component_utils(usb_coverage_report)
cg_usb_config cg;
function new(string name, uvm_component parent);
super.new(name, parent);
cg = new();
endfunction
function void report_phase(uvm_phase phase);
real pct = cg.x_speed_eptype.get_coverage();
// ---- "84% covered" is not a report. ----
//
// It is a number that requires somebody to go and find out which 16%
// and why, and that somebody is usually nobody. The bins are available;
// naming them costs a loop.
`uvm_info("COV", $sformatf("cross coverage %.1f%%", pct), UVM_LOW)
if (pct < 100.0)
`uvm_warning("COV",
"the cross is incomplete -- see the per-bin listing below, and note that a bin at 0%% is either untested (write a test) or unreachable (delete the bin), and the report cannot tell you which")
// ---- A coverage model that was never sampled is not 0% covered. ----
//
// It is unmeasured, and the two are reported identically by every tool.
// This is the coverage equivalent of chapter 24.2's "zero failures and
// zero successes".
if (cg.get_inst_coverage() == 0.0 && n_samples == 0)
`uvm_error("COV",
"the covergroup was never sampled -- 0%% here means the sampling event never fired, not that the design was untested, and those require completely different fixes")
endfunction
int unsigned n_samples;
endclass16. Common Misconceptions
"0% means untested." It means untested or unreachable, and the two demand opposite responses.
"Exclusions are bookkeeping." An exclusion is a claim about the design. Claims can be wrong, and this one is checked nowhere.
"ignore_bins and illegal_bins are the same thing with different severity." One says must not, the other says cannot. Only the first fails when hit.
"The denominator is whatever the tool says." Against 16 you understate by 37%; against 12, by 17% — and 12 looks plausible.
"The coverpoints are at 100%, so we are close." Individual coverpoints fill in a handful of samples. The cross is the measurement.
"One hit is a hit." A bin reached once, by accident, in one configuration, is not a bin anybody should sign off.
"Sample on the clock; it is simpler." Then the model measures the clock. iff is what makes a sample an event.
"Sample in the driver; it is easier to reach." The driver knows what it intended. Coverage must record what happened.
"98% is nearly done." It depends entirely on what the 2% is and what the denominator was.
17. Exercises
1. Work out the reported coverage of a fully covered model when the denominator is 16, 12 and 10, and say which of the three errors a reviewer is least likely to notice.
2. W4 produces no wrong data at all — the excluded sample is simply not counted, which is what an exclusion is for. Explain precisely what the suite is checking that makes it die, and what a conventional coverage flow would score it.
3. W5 and W6 score 20 and 23 in all three languages. Derive both numbers from the directed phases, then predict what they become if the random phase is doubled.
4. Write the assertion that pairs with each ignore_bins in §15, and say why the covergroup cannot contain it.
5. The always @* block that never triggered left the denominator at x. Construct the smallest testbench check that would have caught it, and say why a coverage report would not have.
6. Add a third axis — transfer direction — and work out the new populations. Which of the new cross bins are unreachable by specification rather than by stimulus?
18. Summary
| Idea | Why it matters |
|---|---|
| Unreachable and untested both read 0% | and demand opposite responses |
| An exclusion is a claim | and a falsified claim must fire |
| A post-processing filter deletes the evidence | the one event you needed is the one it removes |
illegal_bins fails when hit | the only bin kind that works by being reached |
ignore_bins fails never | pair each one with an assertion |
| The denominator is 10, not 16 or 12 | 12 is the plausible-looking wrong answer |
| The cross is the measurement | coverpoints fill in a handful of samples |
N_MIN / at_least defaults to 1 | which makes a report an anecdote |
iff sample, not the bare clock | or the model measures the clock |
| Sample in the monitor, not the driver | the driver records its own intentions |
| Name the missing bins | "84%" is a number, not a report |
| A covergroup never sampled is unmeasured | not 0% covered, and the fixes differ |
| 64/64 bins x inputs, denominator checked every step | 7 mutations, all killed in 3 languages |
Tooling
| Step | Command |
|---|---|
| Verilog-2005 | iverilog -g2005 -o cv_v.out cv_v.v cv_v_tb.v && ./cv_v.out |
| SystemVerilog | iverilog -g2012 -o cv_sv.out cv_sv.sv cv_sv_tb.sv && ./cv_sv.out |
| VHDL-2008 analyse | nvc --std=2008 -a cv_vhdl.vhd cv_vhdl_tb.vhd |
| VHDL-2008 elaborate | nvc --std=2008 -e tb_cv_vhdl |
| VHDL-2008 run | nvc --std=2008 -r tb_cv_vhdl |
| One mutation | iverilog -g2005 -DMUT_W1 -o mm cv_v_mut.v cv_v_tb.v && ./mm |
All three implementations pass with 0 errors: every cross bin driven with every input combination, the three bin populations checked on every step, both edges of N_MIN on every legal bin, and the end-of-test report checked empty and full.
Chapter 24.5 — USB VIP Usage turns to verification IP you did not write. A VIP brings its own model of the protocol, its own coverage, and its own idea of what is legal — and the failure mode is specific: when the VIP and the design disagree about the same wire, the VIP is usually believed. The adapter's job is to make that disagreement visible rather than to resolve it silently.
Continue learning
Related tutorials
- Related topic
USB Protocol Checkers
Every ordering rule says what must happen next, and none of them fires when nothing happens at all — the timeout is a checker's only liveness tool, and a checker with false positives gets switched off.
- Related topic
USB Assertions
“Eventually” has no failing case, so it cannot be checked in a finite run — every real liveness check is bounded, a window has two edges, and an obligation still outstanding at end of test is a failure, not an unknown.
- Related topic
USB Scoreboards
Two transfers carrying the same bytes are indistinguishable to a scoreboard that matches on value, so a duplicate delivery and a lost transfer cancel out — identity finds the partner, value checks it.
- Related topic
USB VIP Usage
A VIP is a second implementation of the specification, so where it and the design disagree one of them is wrong — and the adapter's job is to make that visible, never to resolve it in a shim.
Standards & specifications
- Governing standard
- USB-IF (Universal Serial Bus Specification)(opens USB Implementers Forum (USB-IF) in a new tab)
Defines the USB bus — its electrical signalling, connectors, packet and transaction model, device framework and the descriptors a device must expose — together with the device-class specifications layered on it. It does not define host-controller register interfaces (xHCI and EHCI are separate documents) nor any operating system's driver architecture.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the USB curriculum.