USB · Module 30
Verification Review Checklist
A green testbench is a claim, not a proof. Eight questions that audit the claim — did the test execute, is the expectation independent, is the denominator real, can the stimulus reach the condition, has the checker ever been shown to fail — worked on a bench that passes a design with two blockers.
30.1 reviewed a design and found six things wrong with it. Every one of those findings was settled by running something — and the thing that was run is itself engineered software, written by people under a deadline, which can be wrong in ways that no amount of running it will ever reveal.
1. The Question
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
A DESIGN REVIEW ASKS is this design right?
A VERIFICATION REVIEW ASKS is the EVIDENCE that it is right worth
anything?Those are different questions with different failure modes, and the second one is harder because its failures are silent by construction. A design bug produces a wrong value somewhere. A verification bug produces a pass.
Here is the artefact this chapter reviews. It arrived with the specimen from 30.1 — the version with the six findings — and this is what it prints:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
DIRECTED-ONLY : 10 checks, 0 errors
coverage
short-packet completions . 1
limit completions ........ 1
halt sequences ........... 1
collisions tested ........ 0
TOTAL : 3,610 checks, 0 errors
PASSIt is 180 lines. It has a directed phase, a random phase and a coverage summary. It is not lazy or obviously bad. And it passes on a design with two BLOCKER findings and two HIGH ones.
2. The Order
The order of a verification review
3. The Bench Under Review
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// =====================================================================
// tb_usb_ep_xfer_submitted -- THE BENCH THAT CAME WITH THE DESIGN.
//
// This is the artefact chapter 30.2 reviews. It is 180 lines, it is
// not lazy, it has a reference model and a random phase and a coverage
// summary, and it reports:
//
// DIRECTED-ONLY : 316 checks, 0 errors
// TOTAL : 4,096 checks, 0 errors
// PASS
//
// It passes on a design with two BLOCKER findings and two HIGH ones.
// Every defect in it is one a reviewer can find by reading, and none of
// them is visible in the number it prints.
//
// DELIBERATELY DEFECTIVE. Do not copy it.
// =====================================================================
`timescale 1ns/1ps
module tb_usb_ep_xfer_submitted;
reg clk = 1'b0;
reg rst_n, usb_reset, cfg_we, pkt_valid, halt_set, halt_clr, fw_ack;
reg [6:0] cfg_maxpkt, pkt_len;
reg [15:0] cfg_limit;
wire halted, ep_busy, xfer_done, short_pkt;
wire [15:0] xfer_bytes, n_xfer, n_halt;
usb_ep_xfer_before dut (
.clk(clk), .rst_n(rst_n), .usb_reset(usb_reset),
.cfg_we(cfg_we), .cfg_maxpkt(cfg_maxpkt), .cfg_limit(cfg_limit),
.pkt_valid(pkt_valid), .pkt_len(pkt_len),
.halt_set(halt_set), .halt_clr(halt_clr), .fw_ack(fw_ack),
.halted(halted), .ep_busy(ep_busy), .xfer_done(xfer_done),
.xfer_bytes(xfer_bytes), .short_pkt(short_pkt),
.n_xfer(n_xfer), .n_halt(n_halt)
);
always #5 clk = ~clk;
integer checks, errors, rnd_checks;
integer i, n, collisions_tested;
integer cov_short, cov_limit, cov_halt;
reg [15:0] expected_bytes;
task ck;
input [255:0] what;
input [31:0] got;
input [31:0] exp;
begin
checks = checks + 1;
if (got !== exp) begin
errors = errors + 1;
$display(" ** %0s: got %0d expected %0d", what, got, exp);
end
end
endtask
task step; begin
@(posedge clk); #1;
usb_reset = 0; cfg_we = 0; pkt_valid = 0;
halt_set = 0; halt_clr = 0; fw_ack = 0; pkt_len = 0;
end endtask
task reset_dut; begin
rst_n = 0; usb_reset = 0; cfg_we = 0; cfg_maxpkt = 7'd64;
cfg_limit = 16'd0; pkt_valid = 0; pkt_len = 0;
halt_set = 0; halt_clr = 0; fw_ack = 0;
repeat (3) @(posedge clk); #1; rst_n = 1; step;
end endtask
task cfg; input [6:0] mp; input [15:0] lim;
begin cfg_we = 1; cfg_maxpkt = mp; cfg_limit = lim; step; end
endtask
task pkt; input [6:0] len;
begin pkt_valid = 1; pkt_len = len; step; end
endtask
task ack; begin fw_ack = 1; step; end endtask
// ---- a short transfer completes, and the byte count is right ----
task test_short_transfer;
begin
reset_dut; cfg(7'd8, 16'd1024);
pkt(7'd8); pkt(7'd3);
// the expected value, read from the design
expected_bytes = xfer_bytes;
ck("short transfer completes", {31'd0, xfer_done}, 32'd1);
ck("byte count", {16'd0, xfer_bytes}, {16'd0, expected_bytes});
ck("flagged short", {31'd0, short_pkt}, 32'd1);
cov_short = cov_short + 1;
ack;
end
endtask
// ---- a transfer that ends on the configured limit ----
task test_limit_transfer;
begin
reset_dut; cfg(7'd8, 16'd16);
pkt(7'd8); pkt(7'd8);
ck("limit transfer completes", {31'd0, xfer_done}, 32'd1);
ck("byte count", {16'd0, xfer_bytes}, 32'd16);
cov_limit = cov_limit + 1;
ack;
end
endtask
// ---- halting and unhalting ----
task test_halt;
begin
reset_dut; cfg(7'd8, 16'd1024);
halt_set = 1; step;
ck("halted", {31'd0, halted}, 32'd1);
pkt(7'd4);
ck("no packets while halted", {16'd0, n_xfer}, 32'd0);
halt_clr = 1; step;
ck("unhalted", {31'd0, halted}, 32'd0);
cov_halt = cov_halt + 1;
end
endtask
// ---- a transfer that is an exact multiple of maxpkt, terminated by
// a zero-length packet ----
task test_zlp_termination;
begin
reset_dut; cfg(7'd8, 16'd1024);
pkt(7'd8); pkt(7'd8);
pkt(7'd0);
ck("the ZLP completes it", {31'd0, xfer_done}, 32'd1);
ck("byte count", {16'd0, xfer_bytes}, 32'd16);
ack;
end
endtask
// ---- halt_set and halt_clr in the same cycle ----
task test_halt_collision;
begin
reset_dut; cfg(7'd8, 16'd1024);
for (i = 0; i < collisions_tested; i = i + 1) begin
halt_set = 1; halt_clr = 1; step;
ck("collision resolved", {31'd0, halted}, 32'd0);
end
end
endtask
// ---- a bus reset must not disturb the configuration ----
task test_bus_reset;
begin
reset_dut;
usb_reset = 1; step;
ck("bus reset leaves us idle", {31'd0, xfer_done}, 32'd0);
ck("and unhalted", {31'd0, halted}, 32'd0);
end
endtask
task random_phase;
integer r;
begin
reset_dut; cfg(7'd8, 16'd256);
for (n = 0; n < 1200; n = n + 1) begin
r = {$random} % 100;
if (r < 55) pkt(({$random} % 8) + 7'd1);
else if (r < 80) ack;
else if (r < 88) begin halt_set = 1; step; end
else if (r < 96) begin halt_clr = 1; step; end
else step;
rnd_checks = rnd_checks + 3;
if (xfer_done && (xfer_bytes > 16'd2048)) begin
errors = errors + 1;
$display(" ** random: implausible byte count");
end
end
end
endtask
initial begin
checks = 0; errors = 0; rnd_checks = 0;
cov_short = 0; cov_limit = 0; cov_halt = 0;
collisions_tested = 0;
test_short_transfer;
test_limit_transfer;
test_halt;
test_halt_collision;
test_bus_reset;
$display(" DIRECTED-ONLY : %0d checks, %0d errors", checks, errors);
random_phase;
$display("");
$display(" coverage");
$display(" short-packet completions . %0d", cov_short);
$display(" limit completions ........ %0d", cov_limit);
$display(" halt sequences ........... %0d", cov_halt);
$display(" collisions tested ........ %0d", collisions_tested);
$display("");
$display(" TOTAL : %0d checks, %0d errors", checks + rnd_checks, errors);
if (errors == 0) $display(" PASS"); else $display(" FAIL");
$finish;
end
endmoduleRead it before the next section. Every defect is visible from the source, and none of them is visible from the result.
4. Question 2 — Did The Test Execute?
Was the task called? Did the loop iterate? Did the scenario occur?
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
INVARIANT every test that exists in the source runs, and every
loop that exists executes at least once
EVIDENCE a counter per scenario, printed, and READ. Not a log
line saying the test started -- a count of the thing
the test was supposed to produce.
FAILURE SIGNATURE a coverage hole that nobody can explain, because the
test for it is right there in the file
FALSE CONFIDENCE "the test is in the suite" -- being in the file and
being in the run are different properties
NEXT if it did not run, find out whether it ever did, and
when it stoppedThe submitted bench fails this twice, and the two failures have different shapes.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
task test_zlp_termination;
begin
reset_dut; cfg(7'd8, 16'd1024);
pkt(7'd8); pkt(7'd8);
pkt(7'd0);
ck("the ZLP completes it", {31'd0, xfer_done}, 32'd1);
ck("byte count", {16'd0, xfer_bytes}, 32'd16);
ack;
end
endtaskThat task is correct. It tests exactly the right thing, it would fail on the
submitted design, and it is never called. The initial block invokes five
tasks and this is not one of them. Finding F2 — a BLOCKER — escapes because of
a missing line, and no tool reports an unused task in Verilog.
The second is worse, because it does run:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
task test_halt_collision;
begin
reset_dut; cfg(7'd8, 16'd1024);
for (i = 0; i < collisions_tested; i = i + 1) begin
halt_set = 1; halt_clr = 1; step;
ck("collision resolved", {31'd0, halted}, 32'd0);
end
end
endtaskcollisions_tested is initialised to zero and never incremented. The loop
executes zero times. The task is called, it returns, it reports nothing, and
the variable that should have been its input is used as its bound. Finding
F5 escapes.
5. Question 3 — Is The Expectation Independent?
Could the checker reproduce the design's bug, because it gets its answer from the design?
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
INVARIANT the expected value is computed from the INPUTS and
the model's own prior state, never from a DUT output
EVIDENCE read every expected-value expression and find where
each term comes from
FAILURE SIGNATURE a check that has never failed and never will, sitting
in the suite looking like coverage
FALSE CONFIDENCE "we have a reference model" -- a model built the same
way as the design reproduces the design's misreading
of the specification and then agrees with it
NEXT ask whether the model could have been written by
somebody who had not seen the RTLThe submitted bench contains the purest form of the defect:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
expected_bytes = xfer_bytes;
ck("byte count", {16'd0, xfer_bytes}, {16'd0, expected_bytes});A check that compares a value to itself. It increments the check counter, it appears in the total, it will never fail, and reading quickly it looks like the most important assertion in the file.
There is a subtler version, and 30.4's specimen produced it for real:
6. Question 4 — Is The Denominator Real?
Before believing
covered / total, derivetotal.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
INVARIANT every combination counted in the denominator is
reachable, and every dimension that is supposed to
vary actually varies
EVIDENCE the arithmetic, written out, with the excluded
combinations named and justified
FAILURE SIGNATURE a coverage number that plateaus and a team that
spends a month chasing bins that cannot exist
FALSE CONFIDENCE "we are at 94%" -- of what?
NEXT if a bin is unreachable, decide whether that is a
design property or a design bug30.1's reviewed bench claims 48 combinations. Here is the derivation it prints, and the shape a reviewer should expect to see:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
maxpkt 8, 16, 32, 64 the four full-speed bulk maxima 4
stimulus ZLP / short / exactly-maxpkt 3
state idle / mid-transfer / completion pending / halted 4
--------------------------------------------------------------------
4 x 3 x 4 = 48and, as importantly, what is not an axis and why:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
packet lengths 1 .. maxpkt-1 are not a separate dimension: the design's
only length-dependent decision is "strictly shorter than maxpkt", so
every value in that range is the same case and 1 is its representative.That second paragraph is what distinguishes a derived denominator from a chosen one. A reviewer should be able to disagree with it — and if the design later grows a decision that depends on the actual length, the paragraph is what tells the next person the denominator is now wrong.
The submitted bench has no denominator at all. It has three coverage counters, all of which read 1, and a fourth that reads 0.
7. Question 5 — Can The Stimulus Reach It?
Value reachability is not enough. Can the generator produce the ordering, the duration, the simultaneity?
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
INVARIANT for every behaviour the suite claims to test, the
stimulus can construct the PRECONDITION -- including
temporal preconditions
EVIDENCE a measured count of the OUTCOME, not of the attempts
FAILURE SIGNATURE a random phase that has run for a billion cycles and
entered the interesting state zero times
FALSE CONFIDENCE iteration count. "We ran four thousand events" says
nothing about what any of them reached.
NEXT steer the generator at the window, then re-measureThe submitted bench's random phase draws packet lengths from 1 to 8:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
if (r < 55) pkt(({$random} % 8) + 7'd1);+ 1. A zero-length packet — the boundary that finding F2 is about — has
probability zero, in every run, forever. The phase is not weak here; it is
incapable, and no amount of running it changes that.
8. Question 6 — Can The Checker Fail?
Has this checker ever been run against a design known to be wrong?
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
INVARIANT for every checker that matters: a correct design
produces zero failures, and a specific broken design
produces a specific failure
EVIDENCE both runs, both outputs
FAILURE SIGNATURE none, ever -- which is the problem
FALSE CONFIDENCE "it passes" -- a checker that cannot fail also passes
NEXT if it does not fire, find out whether the stimulus
reached it or the check is inertThis is the cheapest high-value item in the chapter and the most often skipped. 30.1's bench was run against the submitted design specifically to produce this:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
phase 1 findings : 241 checks, 39 errors
DIRECTED-ONLY : 2,433 checks, 160 errors
TOTAL : 34,449 checks, 19,199 errorswith every finding firing by name. That single run validates twelve directed scenarios at once, and it takes as long as a compile.
The negative control also catches the failure mode that nothing else does: a scenario that passes on both designs. Such a scenario is present, runs, reports nothing, and is load-bearing in nobody's mind but its author's — and it is indistinguishable from a working one until the day somebody relies on it.
9. Question 7 — What Does A Surviving Mutation Mean?
Four answers. Only one of them is "the testbench is weak".
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
A MUTATION SURVIVED. IT MEANS ONE OF:
1 EQUIVALENT the mutated expression cannot change behaviour.
Often because the code it changed is DEAD.
2 UNREACHABLE the mutated line is reachable in principle but not
under any stimulus the suite produces.
3 STIMULUS HOLE the condition is reachable and nothing constructs it.
4 CHECKER HOLE the condition occurs and nothing looks at the result.
THE PROCEDURE, and it is short:
instrument the changed expression
|
does it ever become DECISIVE -- that is, does it ever determine
the outcome rather than agreeing with something else?
|
NO -> equivalent or dead logic. Fix the DESIGN, not the bench.
YES -> reachable. Now ask whether the stimulus creates it (3) or
whether it creates it and nobody checks (4).10. The Scenario-Removal Experiment
The most useful single technique in this chapter, and it makes a claim falsifiable that is otherwise a matter of opinion.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
CLAIM "this scenario is important"
EXPERIMENT 1 run the mutant with the scenario -> record
2 remove the scenario
3 run the mutant again -> record
4 run the BASE without the scenario -> must be clean
5 restore
RESULT a number, or the claim was wrongRun on 30.1's collision finding:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
BASE, full bench directed 0 total 0
A-M5, full bench directed 3 total 14,171
A-M5, F5 scenario REMOVED directed 0 total 14,168
BASE, F5 scenario removed directed 0 total 0and on 30.4's interrupt collision, where the result is sharper still:
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
BASE, full bench directed 0 total 0
C-M1, full bench directed 30 total 18,173
C-M1, collision phase REMOVED directed 0 total 18,143
BASE, collision phase removed directed 0 total 011. The Reviewed Bench (Verilog)
Every defect from sections 4 to 8 is closed, and the additions that close them are mostly counters rather than checks.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// =====================================================================
// tb_usb_ep_xfer -- Verilog-2005 testbench for usb_ep_xfer.
//
// This is the bench AFTER the review in 30.2. It is written to be
// pointed at either version of the design:
//
// iverilog -g2005 -o good usb_ep_xfer.v tb_usb_ep_xfer.v
// iverilog -g2005 -DUSE_BEFORE -o bad \
// usb_ep_xfer_before.v usb_ep_xfer.v tb_usb_ep_xfer.v
//
// The second command is the NEGATIVE CONTROL. A checker that has never
// been shown to fail has not been validated, and the number it reports
// against the submitted design is the evidence that each of the six
// findings in 30.1 is real rather than a matter of taste.
//
// PHASES
// 1 FINDINGS one scenario per review finding, F1..F6
// 2 EXHAUSTIVE 48 (configuration x stimulus x state) combinations
// 3 STREAM whole transfers, checked at their completion points
// 4 RANDOM supplementary, and audited for what it reaches
// =====================================================================
`timescale 1ns/1ps
module tb_usb_ep_xfer;
reg clk = 1'b0;
reg rst_n;
reg usb_reset;
reg cfg_we;
reg [6:0] cfg_maxpkt;
reg [15:0] cfg_limit;
reg pkt_valid;
reg [6:0] pkt_len;
reg halt_set, halt_clr, fw_ack;
wire halted, ep_busy, xfer_done, short_pkt;
wire [15:0] xfer_bytes, n_xfer, n_halt;
wire [15:0] n_lost;
`ifdef USE_BEFORE
// The submitted design has no lost-completion counter -- its absence
// is finding F3. Tie it off so the same bench can drive both.
usb_ep_xfer_before dut (
`else
usb_ep_xfer dut (
.n_lost(n_lost),
`endif
.clk(clk), .rst_n(rst_n), .usb_reset(usb_reset),
.cfg_we(cfg_we), .cfg_maxpkt(cfg_maxpkt), .cfg_limit(cfg_limit),
.pkt_valid(pkt_valid), .pkt_len(pkt_len),
.halt_set(halt_set), .halt_clr(halt_clr), .fw_ack(fw_ack),
.halted(halted), .ep_busy(ep_busy), .xfer_done(xfer_done),
.xfer_bytes(xfer_bytes), .short_pkt(short_pkt),
.n_xfer(n_xfer), .n_halt(n_halt)
);
`ifdef USE_BEFORE
assign n_lost = 16'd0;
`endif
always #5 clk = ~clk;
// ---- the independent reference model ----------------------------
// Written from the contract in the module header, not from the RTL.
// It never reads a DUT output, so it is capable of disagreeing.
reg [6:0] rm_maxpkt;
reg [15:0] rm_limit, rm_acc, rm_bytes;
reg rm_halt, rm_done, rm_short;
reg [15:0] rm_xfer, rm_haltc, rm_lost;
integer chk_dir, chk_rnd, err, in_random;
integer m_accept, m_zlp, m_maxpkt_pkt, m_short, m_limit, m_drop,
m_halt_coll, m_busy, m_usbrst, m_setupfail;
integer i, k;
task bump; begin
if (in_random) chk_rnd = chk_rnd + 1; else chk_dir = chk_dir + 1;
end endtask
task ck;
input [255:0] what;
input [31:0] got;
input [31:0] exp;
begin
bump;
if (got !== exp) begin
err = err + 1;
if (!in_random && err <= 40)
$display(" ** %0s: got %0d expected %0d (t=%0t)", what, got, exp, $time);
end
end
endtask
task ref_step;
reg offered, accept, dropped, is_short, hit_lim, complete;
reg [15:0] acc_next;
begin
if (!rst_n) begin
rm_maxpkt = 7'd64; rm_limit = 0; rm_acc = 0; rm_halt = 0;
rm_done = 0; rm_short = 0; rm_bytes = 0;
rm_xfer = 0; rm_haltc = 0; rm_lost = 0;
end else if (usb_reset) begin
rm_acc = 0; rm_halt = 0; rm_done = 0; rm_short = 0; rm_bytes = 0;
// maxpkt, limit and the counters deliberately survive
end else begin
if (cfg_we) begin rm_maxpkt = cfg_maxpkt; rm_limit = cfg_limit; end
if (halt_clr) begin
rm_halt = 0; rm_acc = 0; rm_done = 0; rm_short = 0; rm_bytes = 0;
end else if (halt_set) begin
if (!rm_halt) m_halt_coll = m_halt_coll; // tally is below
rm_halt = 1;
rm_haltc = rm_haltc + 1;
end else begin
offered = pkt_valid && !rm_halt;
accept = offered && !rm_done;
dropped = offered && rm_done;
acc_next = rm_acc + {9'd0, pkt_len};
is_short = (pkt_len < rm_maxpkt);
hit_lim = (acc_next >= rm_limit);
complete = accept && (is_short || hit_lim);
if (accept) begin
if (complete) begin
rm_bytes = acc_next; rm_short = is_short; rm_done = 1;
rm_acc = 0; rm_xfer = rm_xfer + 1;
m_short = m_short + (is_short ? 1 : 0);
m_limit = m_limit + ((!is_short && hit_lim) ? 1 : 0);
end else rm_acc = acc_next;
m_accept = m_accept + 1;
if (pkt_len == 0) m_zlp = m_zlp + 1;
if (pkt_len == rm_maxpkt) m_maxpkt_pkt = m_maxpkt_pkt + 1;
end
if (fw_ack) rm_done = 0;
if (dropped) begin rm_lost = rm_lost + 1; m_drop = m_drop + 1; end
end
if (halt_set && halt_clr) m_halt_coll = m_halt_coll + 1;
if (rm_done) m_busy = m_busy + 1;
end
if (usb_reset) m_usbrst = m_usbrst + 1;
end
endtask
task cmp; begin
ck("halted", {31'd0, halted}, {31'd0, rm_halt});
ck("ep_busy", {31'd0, ep_busy}, {31'd0, rm_done});
ck("xfer_done", {31'd0, xfer_done}, {31'd0, rm_done});
ck("xfer_bytes", {16'd0, xfer_bytes}, {16'd0, rm_bytes});
ck("short_pkt", {31'd0, short_pkt}, {31'd0, rm_short});
ck("n_xfer", {16'd0, n_xfer}, {16'd0, rm_xfer});
ck("n_halt", {16'd0, n_halt}, {16'd0, rm_haltc});
ck("n_lost", {16'd0, n_lost}, {16'd0, rm_lost});
end endtask
task step; begin
#1;
@(posedge clk);
ref_step;
#1;
cmp;
usb_reset = 0; cfg_we = 0; pkt_valid = 0;
halt_set = 0; halt_clr = 0; fw_ack = 0;
pkt_len = 0;
end endtask
task idle; begin step; end endtask
task hard_reset; begin
rst_n = 0; usb_reset = 0; cfg_we = 0; cfg_maxpkt = 7'd64;
cfg_limit = 16'd0; pkt_valid = 0; pkt_len = 0;
halt_set = 0; halt_clr = 0; fw_ack = 0;
repeat (3) begin @(posedge clk); ref_step; end
#1; rst_n = 1;
@(posedge clk); ref_step; #1; cmp;
end endtask
task cfg;
input [6:0] mp;
input [15:0] lim;
begin cfg_we = 1; cfg_maxpkt = mp; cfg_limit = lim; step; end
endtask
task pkt;
input [6:0] len;
begin pkt_valid = 1; pkt_len = len; step; end
endtask
task do_ack; begin fw_ack = 1; step; end endtask
task do_halt; begin halt_set = 1; step; end endtask
task do_unhalt; begin halt_clr = 1; step; end endtask
task do_busreset; begin usb_reset = 1; step; end endtask
// -----------------------------------------------------------------
// PHASE 1 -- one scenario per review finding.
// -----------------------------------------------------------------
task phase_findings;
begin
// F1 a USB bus reset must not un-configure the endpoint. With
// maxpkt back at its power-on 64 and limit back at 0, a packet
// that should have continued the transfer completes instead.
hard_reset; cfg(7'd8, 16'd64); do_busreset;
pkt(7'd8);
ck("F1 config survives the bus reset", {31'd0, xfer_done}, 32'd0);
// F2 a transfer that is an exact multiple of maxpkt is terminated
// by a ZERO-LENGTH packet. Drop it and the transfer hangs.
hard_reset; cfg(7'd8, 16'd64);
pkt(7'd8); pkt(7'd8); pkt(7'd8); pkt(7'd8);
ck("F2 not done after 32 bytes", {31'd0, xfer_done}, 32'd0);
pkt(7'd0);
ck("F2 the ZLP completes it", {31'd0, xfer_done}, 32'd1);
ck("F2 byte count", {16'd0, xfer_bytes}, 32'd32);
ck("F2 reported as short", {31'd0, short_pkt}, 32'd1);
// F3 a completion is never overwritten. The packet layer must not
// offer while ep_busy; one offered anyway is dropped and
// counted, and the FIRST completion is still there.
hard_reset; cfg(7'd8, 16'd1024);
pkt(7'd4);
ck("F3 first completion", {16'd0, xfer_bytes}, 32'd4);
pkt(7'd6);
ck("F3 first is still there", {16'd0, xfer_bytes}, 32'd4);
ck("F3 the offer was recorded", {16'd0, n_lost}, 32'd1);
do_ack;
ck("F3 slot released", {31'd0, ep_busy}, 32'd0);
// F4 clearing a halt abandons the WHOLE transfer, pending
// completion included. A leftover completion lets the next
// transfer report the abandoned one's length.
hard_reset; cfg(7'd8, 16'd1024);
pkt(7'd5);
ck("F4 a completion is pending", {16'd0, xfer_bytes}, 32'd5);
do_halt; do_unhalt;
ck("F4 completion abandoned too", {31'd0, xfer_done}, 32'd0);
ck("F4 byte count cleared", {16'd0, xfer_bytes}, 32'd0);
// F5 halt_set and halt_clr in the same cycle: the clear is the
// host's explicit recovery action and wins.
hard_reset; cfg(7'd8, 16'd1024);
do_halt;
ck("F5 halted first", {31'd0, halted}, 32'd1);
halt_set = 1; halt_clr = 1; step;
ck("F5 the clear wins", {31'd0, halted}, 32'd0);
// F6 ep_busy is the completion flag itself. A transfer that
// completes with ZERO bytes -- legal once F2 is fixed -- must
// still look busy.
hard_reset; cfg(7'd8, 16'd1024);
pkt(7'd0);
ck("F6 zero-byte transfer completed", {31'd0, xfer_done}, 32'd1);
ck("F6 and reports zero bytes", {16'd0, xfer_bytes}, 32'd0);
ck("F6 and still looks busy", {31'd0, ep_busy}, 32'd1);
end
endtask
// -----------------------------------------------------------------
// PHASE 2 -- the exhaustive sweep.
//
// DENOMINATOR, derived rather than chosen:
// maxpkt 8, 16, 32, 64 the four full-speed bulk maxima 4
// stimulus ZLP / short / exactly-maxpkt 3
// state idle / mid-transfer / completion pending / halted 4
// ------------------------------------------------------------------
// 4 x 3 x 4 = 48
// Packet lengths between 1 and maxpkt-1 are not a separate axis:
// the design's only length-dependent decision is "strictly shorter
// than maxpkt", so every value in that range is the same case, and
// 1 is its representative. The random phase covers the rest.
// -----------------------------------------------------------------
integer mp_i, st_i, ev_i;
reg [6:0] mps [0:3];
task setup_state;
input integer which; // 0 idle, 1 mid-transfer, 2 pending, 3 halted
input [6:0] mp;
reg [15:0] stat;
begin
hard_reset; cfg(mp, 16'd4096);
case (which)
1: pkt(mp); // accepted, not short, not at limit
2: pkt(7'd1); // short -> a completion is pending
3: do_halt;
default: ;
endcase
// Prove the state was built, rather than assuming it.
bump;
if (((which == 1) && (xfer_done !== 1'b0 || halted !== 1'b0)) ||
((which == 2) && (xfer_done !== 1'b1)) ||
((which == 3) && (halted !== 1'b1)) ||
((which == 0) && (xfer_done !== 1'b0 || halted !== 1'b0))) begin
err = err + 1; m_setupfail = m_setupfail + 1;
$display(" ** setup: state %0d not reached (maxpkt=%0d)", which, mp);
end
end
endtask
task phase_sweep;
begin
mps[0] = 7'd8; mps[1] = 7'd16; mps[2] = 7'd32; mps[3] = 7'd64;
for (mp_i = 0; mp_i < 4; mp_i = mp_i + 1)
for (st_i = 0; st_i < 4; st_i = st_i + 1)
for (ev_i = 0; ev_i < 3; ev_i = ev_i + 1) begin
setup_state(st_i, mps[mp_i]);
case (ev_i)
0: pkt(7'd0); // the zero-length packet
1: pkt(7'd1); // a short packet
2: pkt(mps[mp_i]); // exactly the maximum
endcase
idle;
end
end
endtask
// -----------------------------------------------------------------
// PHASE 3 -- whole transfers, checked at their completion points.
// -----------------------------------------------------------------
task phase_stream;
integer n;
begin
// 3a: a transfer that ends on the limit, exactly.
hard_reset; cfg(7'd8, 16'd24);
pkt(7'd8); pkt(7'd8);
ck("3a not yet", {31'd0, xfer_done}, 32'd0);
pkt(7'd8);
ck("3a limit reached", {31'd0, xfer_done}, 32'd1);
ck("3a byte count", {16'd0, xfer_bytes}, 32'd24);
ck("3a not short", {31'd0, short_pkt}, 32'd0);
do_ack;
// 3b: a transfer that ends one byte short of the limit.
hard_reset; cfg(7'd8, 16'd24);
pkt(7'd8); pkt(7'd8); pkt(7'd7);
ck("3b short wins", {31'd0, short_pkt}, 32'd1);
ck("3b byte count", {16'd0, xfer_bytes}, 32'd23);
do_ack;
// 3c: eight back-to-back transfers, each acknowledged.
hard_reset; cfg(7'd16, 16'd4096);
for (n = 1; n <= 8; n = n + 1) begin
pkt(7'd16); pkt(n[6:0]);
ck("3c byte count", {16'd0, xfer_bytes}, 16 + n);
do_ack;
end
ck("3c eight transfers", {16'd0, n_xfer}, 32'd8);
ck("3c none lost", {16'd0, n_lost}, 32'd0);
end
endtask
// -----------------------------------------------------------------
// PHASE 4 -- random, audited.
// -----------------------------------------------------------------
task phase_random;
integer r;
begin
in_random = 1;
hard_reset; cfg(7'd8, 16'd256);
for (k = 0; k < 4000; k = k + 1) begin
r = {$random} % 100;
if (r < 48) begin
// lengths biased at the two boundaries, because uniform over
// 0..maxpkt spends almost all of its time on the one case the
// design treats identically
if (({$random} % 100) < 22) pkt(7'd0);
else if (({$random} % 100) < 30) pkt(rm_maxpkt);
else pkt(({$random} % (rm_maxpkt + 1)));
end else if (r < 74) begin
do_ack;
end else if (r < 80) begin
do_halt;
end else if (r < 86) begin
do_unhalt;
end else if (r < 89) begin
// the same-cycle halt collision, steered rather than hoped for
halt_set = 1; halt_clr = 1; step;
end else if (r < 92) begin
do_busreset;
end else if (r < 95) begin
// Half the reconfigurations pick a limit that is a SMALL
// MULTIPLE of the maximum packet size, so that the limit path
// is reachable before a short packet ends the transfer. With
// a uniform limit the design completes on a short packet
// almost every time and the other completion condition is
// exercised twice in four thousand events.
if (({$random} % 100) < 50)
cfg(mps[{$random} % 4], ((({$random} % 6) + 2) * rm_maxpkt));
else
cfg(mps[{$random} % 4], ({$random} % 512) + 16'd8);
end else begin
idle;
end
end
in_random = 0;
end
endtask
initial begin
chk_dir = 0; chk_rnd = 0; err = 0; in_random = 0;
m_accept=0; m_zlp=0; m_maxpkt_pkt=0; m_short=0; m_limit=0; m_drop=0;
m_halt_coll=0; m_busy=0; m_usbrst=0; m_setupfail=0;
mps[0] = 7'd8; mps[1] = 7'd16; mps[2] = 7'd32; mps[3] = 7'd64;
phase_findings;
$display(" phase 1 findings : %0d checks, %0d errors", chk_dir, err);
phase_sweep;
$display(" phase 2 exhaustive : %0d checks, %0d errors (48 combinations)",
chk_dir, err);
phase_stream;
$display(" phase 3 stream : %0d checks, %0d errors", chk_dir, err);
$display(" ---- DIRECTED-ONLY : %0d checks, %0d errors ----", chk_dir, err);
phase_random;
$display("");
$display(" measured reachability (all phases)");
$display(" packets accepted ....... %0d", m_accept);
$display(" of which zero-length . %0d", m_zlp);
$display(" of which exactly max . %0d", m_maxpkt_pkt);
$display(" completions on short ... %0d", m_short);
$display(" completions on limit ... %0d", m_limit);
$display(" packets dropped busy ... %0d", m_drop);
$display(" halt set+clear same cyc %0d", m_halt_coll);
$display(" pending-completion cyc . %0d", m_busy);
$display(" USB bus resets ......... %0d", m_usbrst);
$display(" setup failures ......... %0d", m_setupfail);
$display("");
$display(" directed checks ........ %0d", chk_dir);
$display(" random checks .......... %0d", chk_rnd);
$display(" TOTAL checks ........... %0d", chk_dir + chk_rnd);
$display(" ERRORS ................. %0d", err);
if (err == 0) $display(" PASS"); else $display(" FAIL");
$finish;
end
endmodule12. SystemVerilog And VHDL
The same bench in the other two languages, presenting the same directed stimulus in the same order. That property is not decoration: the directed counts must agree to the digit, and when they do not, something is different that should not be.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
VERILOG SYSTEMVERILOG VHDL-2008
phase 1 findings 241 241 241
phase 2 exhaustive 2,113 2,113 2,113
phase 3 stream 2,433 2,433 2,433
---- DIRECTED 2,433 2,433 2,433
errors 0 0 0
TOTAL 34,449 34,449 34,449Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// =====================================================================
// tb_usb_ep_xfer_sv -- SystemVerilog testbench for usb_ep_xfer_sv.
//
// Phases 1-3 present the SAME directed stimulus, in the same order, as
// the Verilog-2005 bench, so their directed check counts must agree to
// the digit. Phase 4 uses its own generator and is not expected to.
//
// The negative control against the submitted design is run from the
// Verilog bench, which carries the `USE_BEFORE` switch; the submitted
// design exists in one language only, because reading a defect once is
// enough.
//
// PHASES
// 1 FINDINGS one scenario per review finding, F1..F6
// 2 EXHAUSTIVE 48 (configuration x stimulus x state) combinations
// 3 STREAM whole transfers, checked at their completion points
// 4 RANDOM supplementary, and audited for what it reaches
// =====================================================================
`timescale 1ns/1ps
module tb_usb_ep_xfer_sv;
logic clk = 1'b0;
logic rst_n;
logic usb_reset;
logic cfg_we;
logic [6:0] cfg_maxpkt;
logic [15:0] cfg_limit;
logic pkt_valid;
logic [6:0] pkt_len;
logic halt_set, halt_clr, fw_ack;
wire halted, ep_busy, xfer_done, short_pkt;
wire [15:0] xfer_bytes, n_xfer, n_halt;
wire [15:0] n_lost;
usb_ep_xfer_sv dut (
.n_lost(n_lost),
.clk(clk), .rst_n(rst_n), .usb_reset(usb_reset),
.cfg_we(cfg_we), .cfg_maxpkt(cfg_maxpkt), .cfg_limit(cfg_limit),
.pkt_valid(pkt_valid), .pkt_len(pkt_len),
.halt_set(halt_set), .halt_clr(halt_clr), .fw_ack(fw_ack),
.halted(halted), .ep_busy(ep_busy), .xfer_done(xfer_done),
.xfer_bytes(xfer_bytes), .short_pkt(short_pkt),
.n_xfer(n_xfer), .n_halt(n_halt)
);
always #5 clk = ~clk;
// ---- the independent reference model ----------------------------
// Written from the contract in the module header, not from the RTL.
// It never reads a DUT output, so it is capable of disagreeing.
logic [6:0] rm_maxpkt;
logic [15:0] rm_limit, rm_acc, rm_bytes;
logic rm_halt, rm_done, rm_short;
logic [15:0] rm_xfer, rm_haltc, rm_lost;
int chk_dir, chk_rnd, err;
bit in_random;
int m_accept, m_zlp, m_maxpkt_pkt, m_short, m_limit, m_drop,
m_halt_coll, m_busy, m_usbrst, m_setupfail;
int i, k;
task bump; begin
if (in_random) chk_rnd = chk_rnd + 1; else chk_dir = chk_dir + 1;
end endtask
task ck(string what, logic [31:0] got, logic [31:0] exp);
begin
bump;
if (got !== exp) begin
err = err + 1;
if (!in_random && err <= 40)
$display(" ** %s: got %0d expected %0d (t=%0t)", what, got, exp, $time);
end
end
endtask
task ref_step;
logic offered, accept, dropped, is_short, hit_lim, complete;
logic [15:0] acc_next;
begin
if (!rst_n) begin
rm_maxpkt = 7'd64; rm_limit = 0; rm_acc = 0; rm_halt = 0;
rm_done = 0; rm_short = 0; rm_bytes = 0;
rm_xfer = 0; rm_haltc = 0; rm_lost = 0;
end else if (usb_reset) begin
rm_acc = 0; rm_halt = 0; rm_done = 0; rm_short = 0; rm_bytes = 0;
// maxpkt, limit and the counters deliberately survive
end else begin
if (cfg_we) begin rm_maxpkt = cfg_maxpkt; rm_limit = cfg_limit; end
if (halt_clr) begin
rm_halt = 0; rm_acc = 0; rm_done = 0; rm_short = 0; rm_bytes = 0;
end else if (halt_set) begin
if (!rm_halt) m_halt_coll = m_halt_coll; // tally is below
rm_halt = 1;
rm_haltc = rm_haltc + 1;
end else begin
offered = pkt_valid && !rm_halt;
accept = offered && !rm_done;
dropped = offered && rm_done;
acc_next = rm_acc + {9'd0, pkt_len};
is_short = (pkt_len < rm_maxpkt);
hit_lim = (acc_next >= rm_limit);
complete = accept && (is_short || hit_lim);
if (accept) begin
if (complete) begin
rm_bytes = acc_next; rm_short = is_short; rm_done = 1;
rm_acc = 0; rm_xfer = rm_xfer + 1;
m_short = m_short + (is_short ? 1 : 0);
m_limit = m_limit + ((!is_short && hit_lim) ? 1 : 0);
end else rm_acc = acc_next;
m_accept = m_accept + 1;
if (pkt_len == 0) m_zlp = m_zlp + 1;
if (pkt_len == rm_maxpkt) m_maxpkt_pkt = m_maxpkt_pkt + 1;
end
if (fw_ack) rm_done = 0;
if (dropped) begin rm_lost = rm_lost + 1; m_drop = m_drop + 1; end
end
if (halt_set && halt_clr) m_halt_coll = m_halt_coll + 1;
if (rm_done) m_busy = m_busy + 1;
end
if (usb_reset) m_usbrst = m_usbrst + 1;
end
endtask
task cmp; begin
ck("halted", {31'd0, halted}, {31'd0, rm_halt});
ck("ep_busy", {31'd0, ep_busy}, {31'd0, rm_done});
ck("xfer_done", {31'd0, xfer_done}, {31'd0, rm_done});
ck("xfer_bytes", {16'd0, xfer_bytes}, {16'd0, rm_bytes});
ck("short_pkt", {31'd0, short_pkt}, {31'd0, rm_short});
ck("n_xfer", {16'd0, n_xfer}, {16'd0, rm_xfer});
ck("n_halt", {16'd0, n_halt}, {16'd0, rm_haltc});
ck("n_lost", {16'd0, n_lost}, {16'd0, rm_lost});
end endtask
task step; begin
#1;
@(posedge clk);
ref_step;
#1;
cmp;
usb_reset = 0; cfg_we = 0; pkt_valid = 0;
halt_set = 0; halt_clr = 0; fw_ack = 0;
pkt_len = 0;
end endtask
task idle; step; endtask
task hard_reset; begin
rst_n = 0; usb_reset = 0; cfg_we = 0; cfg_maxpkt = 7'd64;
cfg_limit = 16'd0; pkt_valid = 0; pkt_len = 0;
halt_set = 0; halt_clr = 0; fw_ack = 0;
repeat (3) begin @(posedge clk); ref_step; end
#1; rst_n = 1;
@(posedge clk); ref_step; #1; cmp;
end endtask
task cfg(logic [6:0] mp, logic [15:0] lim);
cfg_we = 1; cfg_maxpkt = mp; cfg_limit = lim; step;
endtask
task pkt(logic [6:0] len);
pkt_valid = 1; pkt_len = len; step;
endtask
task do_ack; fw_ack = 1; step; endtask
task do_halt; halt_set = 1; step; endtask
task do_unhalt; halt_clr = 1; step; endtask
task do_busreset; usb_reset = 1; step; endtask
// -----------------------------------------------------------------
// PHASE 1 -- one scenario per review finding.
// -----------------------------------------------------------------
task phase_findings;
begin
// F1 a USB bus reset must not un-configure the endpoint. With
// maxpkt back at its power-on 64 and limit back at 0, a packet
// that should have continued the transfer completes instead.
hard_reset; cfg(7'd8, 16'd64); do_busreset;
pkt(7'd8);
ck("F1 config survives the bus reset", {31'd0, xfer_done}, 32'd0);
// F2 a transfer that is an exact multiple of maxpkt is terminated
// by a ZERO-LENGTH packet. Drop it and the transfer hangs.
hard_reset; cfg(7'd8, 16'd64);
pkt(7'd8); pkt(7'd8); pkt(7'd8); pkt(7'd8);
ck("F2 not done after 32 bytes", {31'd0, xfer_done}, 32'd0);
pkt(7'd0);
ck("F2 the ZLP completes it", {31'd0, xfer_done}, 32'd1);
ck("F2 byte count", {16'd0, xfer_bytes}, 32'd32);
ck("F2 reported as short", {31'd0, short_pkt}, 32'd1);
// F3 a completion is never overwritten. The packet layer must not
// offer while ep_busy; one offered anyway is dropped and
// counted, and the FIRST completion is still there.
hard_reset; cfg(7'd8, 16'd1024);
pkt(7'd4);
ck("F3 first completion", {16'd0, xfer_bytes}, 32'd4);
pkt(7'd6);
ck("F3 first is still there", {16'd0, xfer_bytes}, 32'd4);
ck("F3 the offer was recorded", {16'd0, n_lost}, 32'd1);
do_ack;
ck("F3 slot released", {31'd0, ep_busy}, 32'd0);
// F4 clearing a halt abandons the WHOLE transfer, pending
// completion included. A leftover completion lets the next
// transfer report the abandoned one's length.
hard_reset; cfg(7'd8, 16'd1024);
pkt(7'd5);
ck("F4 a completion is pending", {16'd0, xfer_bytes}, 32'd5);
do_halt; do_unhalt;
ck("F4 completion abandoned too", {31'd0, xfer_done}, 32'd0);
ck("F4 byte count cleared", {16'd0, xfer_bytes}, 32'd0);
// F5 halt_set and halt_clr in the same cycle: the clear is the
// host's explicit recovery action and wins.
hard_reset; cfg(7'd8, 16'd1024);
do_halt;
ck("F5 halted first", {31'd0, halted}, 32'd1);
halt_set = 1; halt_clr = 1; step;
ck("F5 the clear wins", {31'd0, halted}, 32'd0);
// F6 ep_busy is the completion flag itself. A transfer that
// completes with ZERO bytes -- legal once F2 is fixed -- must
// still look busy.
hard_reset; cfg(7'd8, 16'd1024);
pkt(7'd0);
ck("F6 zero-byte transfer completed", {31'd0, xfer_done}, 32'd1);
ck("F6 and reports zero bytes", {16'd0, xfer_bytes}, 32'd0);
ck("F6 and still looks busy", {31'd0, ep_busy}, 32'd1);
end
endtask
// -----------------------------------------------------------------
// PHASE 2 -- the exhaustive sweep.
//
// DENOMINATOR, derived rather than chosen:
// maxpkt 8, 16, 32, 64 the four full-speed bulk maxima 4
// stimulus ZLP / short / exactly-maxpkt 3
// state idle / mid-transfer / completion pending / halted 4
// ------------------------------------------------------------------
// 4 x 3 x 4 = 48
// Packet lengths between 1 and maxpkt-1 are not a separate axis:
// the design's only length-dependent decision is "strictly shorter
// than maxpkt", so every value in that range is the same case, and
// 1 is its representative. The random phase covers the rest.
// -----------------------------------------------------------------
int mp_i, st_i, ev_i;
logic [6:0] mps [4];
task setup_state(int which, logic [6:0] mp);
begin
hard_reset; cfg(mp, 16'd4096);
case (which)
1: pkt(mp); // accepted, not short, not at limit
2: pkt(7'd1); // short -> a completion is pending
3: do_halt;
default: ;
endcase
// Prove the state was built, rather than assuming it.
bump;
if (((which == 1) && (xfer_done !== 1'b0 || halted !== 1'b0)) ||
((which == 2) && (xfer_done !== 1'b1)) ||
((which == 3) && (halted !== 1'b1)) ||
((which == 0) && (xfer_done !== 1'b0 || halted !== 1'b0))) begin
err = err + 1; m_setupfail = m_setupfail + 1;
$display(" ** setup: state %0d not reached (maxpkt=%0d)", which, mp);
end
end
endtask
task phase_sweep;
begin
mps[0] = 7'd8; mps[1] = 7'd16; mps[2] = 7'd32; mps[3] = 7'd64;
for (mp_i = 0; mp_i < 4; mp_i = mp_i + 1)
for (st_i = 0; st_i < 4; st_i = st_i + 1)
for (ev_i = 0; ev_i < 3; ev_i = ev_i + 1) begin
setup_state(st_i, mps[mp_i]);
case (ev_i)
0: pkt(7'd0); // the zero-length packet
1: pkt(7'd1); // a short packet
2: pkt(mps[mp_i]); // exactly the maximum
endcase
idle;
end
end
endtask
// -----------------------------------------------------------------
// PHASE 3 -- whole transfers, checked at their completion points.
// -----------------------------------------------------------------
task phase_stream;
int n;
begin
// 3a: a transfer that ends on the limit, exactly.
hard_reset; cfg(7'd8, 16'd24);
pkt(7'd8); pkt(7'd8);
ck("3a not yet", {31'd0, xfer_done}, 32'd0);
pkt(7'd8);
ck("3a limit reached", {31'd0, xfer_done}, 32'd1);
ck("3a byte count", {16'd0, xfer_bytes}, 32'd24);
ck("3a not short", {31'd0, short_pkt}, 32'd0);
do_ack;
// 3b: a transfer that ends one byte short of the limit.
hard_reset; cfg(7'd8, 16'd24);
pkt(7'd8); pkt(7'd8); pkt(7'd7);
ck("3b short wins", {31'd0, short_pkt}, 32'd1);
ck("3b byte count", {16'd0, xfer_bytes}, 32'd23);
do_ack;
// 3c: eight back-to-back transfers, each acknowledged.
hard_reset; cfg(7'd16, 16'd4096);
for (n = 1; n <= 8; n = n + 1) begin
pkt(7'd16); pkt(7'(n));
ck("3c byte count", {16'd0, xfer_bytes}, 16 + n);
do_ack;
end
ck("3c eight transfers", {16'd0, n_xfer}, 32'd8);
ck("3c none lost", {16'd0, n_lost}, 32'd0);
end
endtask
// -----------------------------------------------------------------
// PHASE 4 -- random, audited.
// -----------------------------------------------------------------
task phase_random;
int r;
begin
in_random = 1;
hard_reset; cfg(7'd8, 16'd256);
for (k = 0; k < 4000; k = k + 1) begin
r = $urandom_range(99);
if (r < 48) begin
// lengths biased at the two boundaries, because uniform over
// 0..maxpkt spends almost all of its time on the one case the
// design treats identically
if (($urandom_range(99)) < 22) pkt(7'd0);
else if (($urandom_range(99)) < 30) pkt(rm_maxpkt);
else pkt(7'($urandom_range(rm_maxpkt)));
end else if (r < 74) begin
do_ack;
end else if (r < 80) begin
do_halt;
end else if (r < 86) begin
do_unhalt;
end else if (r < 89) begin
// the same-cycle halt collision, steered rather than hoped for
halt_set = 1; halt_clr = 1; step;
end else if (r < 92) begin
do_busreset;
end else if (r < 95) begin
// Half the reconfigurations pick a limit that is a SMALL
// MULTIPLE of the maximum packet size, so that the limit path
// is reachable before a short packet ends the transfer. With
// a uniform limit the design completes on a short packet
// almost every time and the other completion condition is
// exercised twice in four thousand events.
if (($urandom_range(99)) < 50)
cfg(mps[$urandom_range(3)], ((($urandom_range(5)) + 2) * rm_maxpkt));
else
cfg(mps[$urandom_range(3)], 16'($urandom_range(511)) + 16'd8);
end else begin
idle;
end
end
in_random = 0;
end
endtask
initial begin
chk_dir = 0; chk_rnd = 0; err = 0; in_random = 0;
m_accept=0; m_zlp=0; m_maxpkt_pkt=0; m_short=0; m_limit=0; m_drop=0;
m_halt_coll=0; m_busy=0; m_usbrst=0; m_setupfail=0;
mps[0] = 7'd8; mps[1] = 7'd16; mps[2] = 7'd32; mps[3] = 7'd64;
phase_findings;
$display(" phase 1 findings : %0d checks, %0d errors", chk_dir, err);
phase_sweep;
$display(" phase 2 exhaustive : %0d checks, %0d errors (48 combinations)",
chk_dir, err);
phase_stream;
$display(" phase 3 stream : %0d checks, %0d errors", chk_dir, err);
$display(" ---- DIRECTED-ONLY : %0d checks, %0d errors ----", chk_dir, err);
phase_random;
$display("");
$display(" measured reachability (all phases)");
$display(" packets accepted ....... %0d", m_accept);
$display(" of which zero-length . %0d", m_zlp);
$display(" of which exactly max . %0d", m_maxpkt_pkt);
$display(" completions on short ... %0d", m_short);
$display(" completions on limit ... %0d", m_limit);
$display(" packets dropped busy ... %0d", m_drop);
$display(" halt set+clear same cyc %0d", m_halt_coll);
$display(" pending-completion cyc . %0d", m_busy);
$display(" USB bus resets ......... %0d", m_usbrst);
$display(" setup failures ......... %0d", m_setupfail);
$display("");
$display(" directed checks ........ %0d", chk_dir);
$display(" random checks .......... %0d", chk_rnd);
$display(" TOTAL checks ........... %0d", chk_dir + chk_rnd);
$display(" ERRORS ................. %0d", err);
if (err == 0) $display(" PASS"); else $display(" FAIL");
$finish;
end
endmoduleAzvya Education Pvt. Ltd.VLSI Mentor
Snippet
-- =====================================================================
-- tb_usb_ep_xfer -- VHDL-2008 testbench for usb_ep_xfer.
--
-- Phases 1-3 present the SAME directed stimulus, in the same order, as
-- the Verilog-2005 and SystemVerilog benches, so their directed check
-- counts must agree to the digit. Phase 4 uses its own generator.
--
-- The reference model lives in process VARIABLES: a variable updates
-- immediately, which is what a model stepped inside the stimulus
-- process needs, and it makes a second driver impossible.
-- =====================================================================
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
use ieee.math_real.all;
entity tb_usb_ep_xfer is
end entity tb_usb_ep_xfer;
architecture sim of tb_usb_ep_xfer is
constant HALF : time := 10 ns;
signal clk : std_logic := '0';
signal rst_n : std_logic := '0';
signal usb_reset : std_logic := '0';
signal cfg_we : std_logic := '0';
signal cfg_maxpkt : unsigned(6 downto 0) := to_unsigned(64, 7);
signal cfg_limit : unsigned(15 downto 0) := (others => '0');
signal pkt_valid : std_logic := '0';
signal pkt_len : unsigned(6 downto 0) := (others => '0');
signal halt_set : std_logic := '0';
signal halt_clr : std_logic := '0';
signal fw_ack : std_logic := '0';
signal halted, ep_busy, xfer_done, short_pkt : std_logic;
signal xfer_bytes, n_xfer, n_halt, n_lost : unsigned(15 downto 0);
signal done_flag : boolean := false;
function b2i (s : std_logic) return integer is
begin
if s = '1' then return 1; else return 0; end if;
end function b2i;
begin
dut : entity work.usb_ep_xfer
port map (
clk => clk, rst_n => rst_n, usb_reset => usb_reset,
cfg_we => cfg_we, cfg_maxpkt => cfg_maxpkt, cfg_limit => cfg_limit,
pkt_valid => pkt_valid, pkt_len => pkt_len,
halt_set => halt_set, halt_clr => halt_clr, fw_ack => fw_ack,
halted => halted, ep_busy => ep_busy, xfer_done => xfer_done,
xfer_bytes => xfer_bytes, short_pkt => short_pkt,
n_xfer => n_xfer, n_halt => n_halt, n_lost => n_lost
);
clkgen : process
begin
while not done_flag loop
clk <= '0'; wait for HALF;
clk <= '1'; wait for HALF;
end loop;
wait;
end process clkgen;
stim : process
variable chk_dir, chk_rnd, errs : natural := 0;
variable in_random : boolean := false;
variable shown : natural := 0;
-- the independent reference model
variable rm_maxpkt : unsigned(6 downto 0) := to_unsigned(64, 7);
variable rm_limit : unsigned(15 downto 0) := (others => '0');
variable rm_acc : unsigned(15 downto 0) := (others => '0');
variable rm_bytes : unsigned(15 downto 0) := (others => '0');
variable rm_halt, rm_done, rm_short : std_logic := '0';
variable rm_xfer, rm_haltc, rm_lost : natural := 0;
variable m_accept, m_zlp, m_maxpkt_pkt, m_short, m_limit : natural := 0;
variable m_drop, m_halt_coll, m_busy, m_usbrst, m_setupfail : natural := 0;
type mp_arr is array (0 to 3) of natural;
constant mps : mp_arr := (8, 16, 32, 64);
variable seed1 : positive := 461_003;
variable seed2 : positive := 92_137;
procedure bump is
begin
if in_random then chk_rnd := chk_rnd + 1;
else chk_dir := chk_dir + 1;
end if;
end procedure bump;
procedure ck (what : string; got : integer; exp : integer) is
begin
bump;
if got /= exp then
errs := errs + 1;
if (not in_random) and shown < 40 then
shown := shown + 1;
report " ** " & what & ": got " & integer'image(got) &
" expected " & integer'image(exp) severity warning;
end if;
end if;
end procedure ck;
procedure ref_step is
variable offered, acc_ok, dropped_v : boolean;
variable is_short, hit_lim, complete_v : boolean;
variable acc_next : unsigned(16 downto 0);
begin
if rst_n = '0' then
rm_maxpkt := to_unsigned(64, 7);
rm_limit := (others => '0'); rm_acc := (others => '0');
rm_bytes := (others => '0');
rm_halt := '0'; rm_done := '0'; rm_short := '0';
rm_xfer := 0; rm_haltc := 0; rm_lost := 0;
elsif usb_reset = '1' then
rm_acc := (others => '0'); rm_bytes := (others => '0');
rm_halt := '0'; rm_done := '0'; rm_short := '0';
else
if cfg_we = '1' then
rm_maxpkt := cfg_maxpkt;
rm_limit := cfg_limit;
end if;
if halt_clr = '1' then
rm_halt := '0'; rm_acc := (others => '0');
rm_done := '0'; rm_short := '0'; rm_bytes := (others => '0');
elsif halt_set = '1' then
rm_halt := '1';
rm_haltc := rm_haltc + 1;
else
offered := (pkt_valid = '1') and (rm_halt = '0');
acc_ok := offered and (rm_done = '0');
dropped_v := offered and (rm_done = '1');
acc_next := ('0' & rm_acc) + resize(pkt_len, 17);
is_short := pkt_len < rm_maxpkt;
hit_lim := acc_next >= resize(rm_limit, 17);
complete_v := acc_ok and (is_short or hit_lim);
if acc_ok then
if complete_v then
rm_bytes := acc_next(15 downto 0);
if is_short then rm_short := '1'; else rm_short := '0'; end if;
rm_done := '1';
rm_acc := (others => '0');
rm_xfer := rm_xfer + 1;
if is_short then m_short := m_short + 1;
else m_limit := m_limit + 1;
end if;
else
rm_acc := acc_next(15 downto 0);
end if;
m_accept := m_accept + 1;
if pkt_len = 0 then m_zlp := m_zlp + 1; end if;
if pkt_len = rm_maxpkt then m_maxpkt_pkt := m_maxpkt_pkt + 1; end if;
end if;
if fw_ack = '1' then rm_done := '0'; end if;
if dropped_v then
rm_lost := rm_lost + 1;
m_drop := m_drop + 1;
end if;
end if;
if halt_set = '1' and halt_clr = '1' then
m_halt_coll := m_halt_coll + 1;
end if;
if rm_done = '1' then m_busy := m_busy + 1; end if;
end if;
if usb_reset = '1' then m_usbrst := m_usbrst + 1; end if;
end procedure ref_step;
procedure cmp is
begin
ck("halted", b2i(halted), b2i(rm_halt));
ck("ep_busy", b2i(ep_busy), b2i(rm_done));
ck("xfer_done", b2i(xfer_done), b2i(rm_done));
ck("xfer_bytes", to_integer(xfer_bytes), to_integer(rm_bytes));
ck("short_pkt", b2i(short_pkt), b2i(rm_short));
ck("n_xfer", to_integer(n_xfer), rm_xfer);
ck("n_halt", to_integer(n_halt), rm_haltc);
ck("n_lost", to_integer(n_lost), rm_lost);
end procedure cmp;
procedure step is
begin
wait for 1 ns;
wait until rising_edge(clk);
ref_step;
wait for 1 ns;
cmp;
usb_reset <= '0'; cfg_we <= '0'; pkt_valid <= '0';
halt_set <= '0'; halt_clr <= '0'; fw_ack <= '0';
pkt_len <= (others => '0');
end procedure step;
procedure idle is begin step; end procedure idle;
procedure hard_reset is
begin
rst_n <= '0'; usb_reset <= '0'; cfg_we <= '0';
cfg_maxpkt <= to_unsigned(64, 7); cfg_limit <= (others => '0');
pkt_valid <= '0'; pkt_len <= (others => '0');
halt_set <= '0'; halt_clr <= '0'; fw_ack <= '0';
for i in 0 to 2 loop wait until rising_edge(clk); ref_step; end loop;
wait for 1 ns;
rst_n <= '1';
wait until rising_edge(clk);
ref_step;
wait for 1 ns;
cmp;
end procedure hard_reset;
procedure cfg (mp : natural; lim : natural) is
begin
cfg_we <= '1';
cfg_maxpkt <= to_unsigned(mp, 7);
cfg_limit <= to_unsigned(lim, 16);
step;
end procedure cfg;
procedure pkt (len : natural) is
begin
pkt_valid <= '1';
pkt_len <= to_unsigned(len, 7);
step;
end procedure pkt;
procedure do_ack is begin fw_ack <= '1'; step; end procedure;
procedure do_halt is begin halt_set <= '1'; step; end procedure;
procedure do_unhalt is begin halt_clr <= '1'; step; end procedure;
procedure do_busreset is begin usb_reset <= '1'; step; end procedure;
-- ---- PHASE 1 : one scenario per review finding ----
procedure phase_findings is
begin
hard_reset; cfg(8, 64); do_busreset;
pkt(8);
ck("F1 config survives the bus reset", b2i(xfer_done), 0);
hard_reset; cfg(8, 64);
pkt(8); pkt(8); pkt(8); pkt(8);
ck("F2 not done after 32 bytes", b2i(xfer_done), 0);
pkt(0);
ck("F2 the ZLP completes it", b2i(xfer_done), 1);
ck("F2 byte count", to_integer(xfer_bytes), 32);
ck("F2 reported as short", b2i(short_pkt), 1);
hard_reset; cfg(8, 1024);
pkt(4);
ck("F3 first completion", to_integer(xfer_bytes), 4);
pkt(6);
ck("F3 first is still there", to_integer(xfer_bytes), 4);
ck("F3 the offer was recorded", to_integer(n_lost), 1);
do_ack;
ck("F3 slot released", b2i(ep_busy), 0);
hard_reset; cfg(8, 1024);
pkt(5);
ck("F4 a completion is pending", to_integer(xfer_bytes), 5);
do_halt; do_unhalt;
ck("F4 completion abandoned too", b2i(xfer_done), 0);
ck("F4 byte count cleared", to_integer(xfer_bytes), 0);
hard_reset; cfg(8, 1024);
do_halt;
ck("F5 halted first", b2i(halted), 1);
halt_set <= '1'; halt_clr <= '1'; step;
ck("F5 the clear wins", b2i(halted), 0);
hard_reset; cfg(8, 1024);
pkt(0);
ck("F6 zero-byte transfer completed", b2i(xfer_done), 1);
ck("F6 and reports zero bytes", to_integer(xfer_bytes), 0);
ck("F6 and still looks busy", b2i(ep_busy), 1);
end procedure phase_findings;
-- ---- PHASE 2 : the exhaustive sweep, 4 x 4 x 3 = 48 ----
procedure setup_state (which : natural; mp : natural) is
variable ok : boolean;
begin
hard_reset; cfg(mp, 4096);
case which is
when 1 => pkt(mp);
when 2 => pkt(1);
when 3 => do_halt;
when others => null;
end case;
bump;
ok := true;
if which = 1 and (xfer_done /= '0' or halted /= '0') then ok := false; end if;
if which = 2 and xfer_done /= '1' then ok := false; end if;
if which = 3 and halted /= '1' then ok := false; end if;
if which = 0 and (xfer_done /= '0' or halted /= '0') then ok := false; end if;
if not ok then
errs := errs + 1; m_setupfail := m_setupfail + 1;
report " ** setup: state not reached" severity warning;
end if;
end procedure setup_state;
procedure phase_sweep is
begin
for mp_i in 0 to 3 loop
for st_i in 0 to 3 loop
for ev_i in 0 to 2 loop
setup_state(st_i, mps(mp_i));
case ev_i is
when 0 => pkt(0);
when 1 => pkt(1);
when others => pkt(mps(mp_i));
end case;
idle;
end loop;
end loop;
end loop;
end procedure phase_sweep;
-- ---- PHASE 3 : whole transfers ----
procedure phase_stream is
begin
hard_reset; cfg(8, 24);
pkt(8); pkt(8);
ck("3a not yet", b2i(xfer_done), 0);
pkt(8);
ck("3a limit reached", b2i(xfer_done), 1);
ck("3a byte count", to_integer(xfer_bytes), 24);
ck("3a not short", b2i(short_pkt), 0);
do_ack;
hard_reset; cfg(8, 24);
pkt(8); pkt(8); pkt(7);
ck("3b short wins", b2i(short_pkt), 1);
ck("3b byte count", to_integer(xfer_bytes), 23);
do_ack;
hard_reset; cfg(16, 4096);
for n in 1 to 8 loop
pkt(16); pkt(n);
ck("3c byte count", to_integer(xfer_bytes), 16 + n);
do_ack;
end loop;
ck("3c eight transfers", to_integer(n_xfer), 8);
ck("3c none lost", to_integer(n_lost), 0);
end procedure phase_stream;
-- ---- PHASE 4 : random, audited ----
impure function rnd (n : positive) return natural is
variable x : real;
begin
uniform(seed1, seed2, x);
return natural(real(n - 1) * x);
end function rnd;
procedure phase_random is
variable r : natural;
begin
in_random := true;
hard_reset; cfg(8, 256);
for k in 0 to 3999 loop
r := rnd(100);
if r < 48 then
if rnd(100) < 22 then pkt(0);
elsif rnd(100) < 30 then pkt(to_integer(rm_maxpkt));
else pkt(rnd(to_integer(rm_maxpkt) + 1));
end if;
elsif r < 74 then do_ack;
elsif r < 80 then do_halt;
elsif r < 86 then do_unhalt;
elsif r < 89 then
halt_set <= '1'; halt_clr <= '1'; step;
elsif r < 92 then do_busreset;
elsif r < 95 then
if rnd(100) < 50 then
cfg(mps(rnd(4)), (rnd(6) + 2) * to_integer(rm_maxpkt));
else
cfg(mps(rnd(4)), rnd(512) + 8);
end if;
else idle;
end if;
end loop;
in_random := false;
end procedure phase_random;
begin
phase_findings;
report " phase 1 findings : " & integer'image(chk_dir) &
" checks, " & integer'image(errs) & " errors";
phase_sweep;
report " phase 2 exhaustive : " & integer'image(chk_dir) &
" checks, " & integer'image(errs) & " errors (48 combinations)";
phase_stream;
report " phase 3 stream : " & integer'image(chk_dir) &
" checks, " & integer'image(errs) & " errors";
report " ---- DIRECTED-ONLY : " & integer'image(chk_dir) &
" checks, " & integer'image(errs) & " errors ----";
phase_random;
report " measured reachability (all phases)";
report " packets accepted ....... " & integer'image(m_accept);
report " of which zero-length . " & integer'image(m_zlp);
report " of which exactly max . " & integer'image(m_maxpkt_pkt);
report " completions on short ... " & integer'image(m_short);
report " completions on limit ... " & integer'image(m_limit);
report " packets dropped busy ... " & integer'image(m_drop);
report " halt set+clear same cyc " & integer'image(m_halt_coll);
report " pending-completion cyc . " & integer'image(m_busy);
report " USB bus resets ......... " & integer'image(m_usbrst);
report " setup failures ......... " & integer'image(m_setupfail);
report " directed checks ........ " & integer'image(chk_dir);
report " random checks .......... " & integer'image(chk_rnd);
report " TOTAL checks ........... " & integer'image(chk_dir + chk_rnd);
report " ERRORS ................. " & integer'image(errs);
if errs = 0 then report " PASS"; else report " FAIL" severity failure; end if;
done_flag <= true;
wait;
end process stim;
end architecture sim;13. Where UVM Fits, And What To Review In It
A transaction-level environment moves the defects in this chapter rather than removing them. Every question from sections 4 to 9 still applies; what changes is where each one is answered, and a UVM review is largely a review of responsibility boundaries.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
COMPONENT ITS ONE JOB WHAT IT MUST NOT DO
----------------- ------------------------- -----------------------
sequence item describe an ARCHITECTURAL mirror RTL pin names
event
sequence construct a scenario, compute what the answer
including its temporal should be
preconditions
driver turn an item into pin decide anything
activity
monitor OBSERVE and publish predict
reference model PREDICT from inputs and read a DUT output
its own state
scoreboard COMPARE the two reconstruct stimulus
coverage measure the scenario count bins that cannot
space existThe review question that subsumes most of them is one sentence:
Point at the component that would notice if the design were wrong, and explain where its opinion comes from.
If the answer traces back to a DUT output, or to the same person's reading of the same paragraph that produced the RTL, the environment is large and the evidence is thin.
14. What This Module's Own Benches Got Wrong
Three defects were found in the verification written for this module, and they are reported here rather than quietly fixed because each is a standing item.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
DEFECT FOUND BY ITEM
------------------------------ -------------------------- ------
the interrupt specimen's an independently-stated Q3
reference model contained the directed check disagreeing
design's own wrong expression with both
and agreed with it for 39,108
cycles
the three implementations of the cross-language Q6
specimen A were not directed counts differing
structurally equivalent, so 106 vs 29 on one mutation
one mutation meant two
different things
a VHDL mutation silently the driver asserting that --
failed to generate, which every replacement applied
would have read as a survivorThe third is the cheapest and it is worth stating as a rule on its own: a
mutation script must assert that its replacement applied. A mutation that fails
to generate produces a score of zero, and a score of zero is indistinguishable
from a survivor. One assert count == 1 per edit.
15. The Checklist
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
1 WHAT ARE WE PROVING?
[ ] there is a written obligation, not just a set of runs
[ ] every "must not" in the design's contract has a check
[ ] the directed suite passes WITHOUT the random phase
2 DID THE TEST EXECUTE?
[ ] every task defined is called
[ ] every loop bound is non-zero, and is not a variable that was
meant to be a parameter
[ ] every scenario has a counter, printed
[ ] a counter reading zero FAILS the run rather than appearing in it
3 IS THE EXPECTATION INDEPENDENT?
[ ] no expected value is read from a DUT output
[ ] the reference model does not share the design's structure
[ ] at least some expectations are stated as INTENT rather than
computed, because a model and a design from one author are one
artefact
4 IS THE DENOMINATOR REAL?
[ ] the total is derived in writing, with the arithmetic shown
[ ] excluded combinations are named and justified
[ ] every dimension that should vary actually varies
[ ] the axes are listed -- and SIMULTANEITY is one of them, or its
absence is deliberate
5 CAN THE STIMULUS REACH IT?
[ ] every boundary value is producible by the generator
[ ] every temporal precondition is constructible
[ ] the OUTCOME is counted, not the attempt
[ ] distributions are steered at the windows, not uniform
6 CAN THE CHECKER FAIL?
[ ] the suite has been run against a design known to be wrong
[ ] every scenario fired on that run
[ ] a scenario that passed on BOTH is investigated
7 MUTATIONS
[ ] every replacement is asserted to have applied
[ ] every survivor is classified: equivalent / unreachable /
stimulus hole / checker hole
[ ] equivalence is MEASURED with a probe, not argued
[ ] a scenario that is the sole detector of a finding is marked
load-bearing, by removing it and re-running
8 BLIND SPOTS
[ ] what this suite cannot establish is written down
[ ] metastability, analog behaviour and anything across a clock
boundary are named explicitly as outside it16. Exercises
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
1 READ THE BENCH
Find all seven defects in the submitted bench in section 3. Five are
named in this chapter. For the other two, say which review item
would have caught them.
2 THE ZERO
"collisions tested 0" was printed and ignored. Write the three lines
that would have turned it into a failure, and say where they go.
3 DENOMINATOR
Derive the denominator for a sweep of the 30.4 interrupt specimen
that INCLUDES simultaneity as an axis. How large does it get, and is
it still exhaustible?
4 REACHABILITY
The submitted bench draws packet lengths from 1 to 8. List every
finding from 30.1 that this alone makes unreachable, and say which
ones survive the fix.
5 NEGATIVE CONTROL
Write the negative control for the 30.3 specimen: a deliberately
wrong legality table, and the expected failure signature.
6 SURVIVOR TRIAGE
A mutation survives. Write the probe you would add to distinguish
"equivalent" from "stimulus hole", for a mutation in a comparison
rather than in an assignment.
7 REMOVAL EXPERIMENT
Pick any directed scenario in the reviewed bench and run the
experiment from section 10 against the mutation you think it
catches. Report the four numbers. If the third is not zero, the
scenario is not the sole detector -- find the other one.
8 UVM REVIEW
Given an environment where the monitor publishes an expected value,
describe the smallest change that restores the boundary, and what it
costs.
9 THE BLIND SPOT
Write the "what this suite cannot establish" paragraph for 30.1's
reviewed bench. Three items minimum. One of them is not about USB.17. The Interview Answer
"Your regression is green and coverage is at 96%. What would still worry you?"
Both of those numbers are claims about the testbench, and neither is a claim about the design.
Green means every check that ran and could fail, did not. It says nothing about
checks that did not run — a task nobody called, a loop bounded by a variable that
was meant to be a parameter — and nothing about checks that cannot fail, like an
expected value read from a DUT output. The first thing I would do is look for a
measured zero: a scenario counter, a coverage bin, an event tally that reads
zero and is sitting in the log above a PASS. In this module's own specimen the
line collisions tested 0 was printed on every run of a bench that passed a
design with two blockers.
96% means 96% of a denominator, and the denominator is the interesting half. I would ask for the derivation — written out, with the excluded combinations named. Very often a plateau is bins that cannot exist, and a month gets spent on them. Equally often the opposite: the denominator is honest but its axes are incomplete. An exhaustive sweep over state and stimulus does not contain simultaneity, so a set-and-clear collision can be missed by a sweep that is genuinely exhaustive over everything it covers. I measured that on a 96-bin sweep: the collision mutation scored zero against it and thirty against one eleven-line scenario.
Then I would ask whether any of it has ever been shown to fail. A checker that has never fired has not been validated, and running the suite against a design you know is broken validates the whole directed set in one compile. It also catches the scenario that passes on both, which is invisible by any other means.
And I would ask what a surviving mutation was concluded to mean — because "surviving mutant, therefore weak testbench" is wrong about a quarter of the time. It can be an equivalent mutant, and the commonest cause of that is dead code in the design. I would want the probe rather than the argument: instrument the expression and count the cycles in which it is decisive. If the answer is zero, the finding is in the RTL, not in the bench.
The last thing, which is not a number: what does this suite not establish? If nobody can answer, the honest state is that the blind spots are unknown rather than absent. Metastability is always on that list — no simulator models it, at any seed, for any number of cycles.
18. What Carries Forward
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
THE PROCEDURE
o eight questions, ordered: obligation, execution, independence,
denominator, reachability, falsifiability, mutation meaning, blind
spots
o a design bug produces a wrong value; a verification bug produces a
PASS
THE ITEMS
o count the OUTCOME and print it; a measured zero is the cheapest bug
report and the easiest to walk past
o a task defined and not called, and a loop bounded by a variable that
was meant to be a parameter, are what merges and refactors produce --
so the bench must assert its own scenarios happened
o an expected value read from a DUT output is a check that cannot fail
o a reference model written by the design's author from the design's
sentence is ONE artefact wearing two hats; break the symmetry with
expectations stated as INTENT
o derive the denominator, name the exclusions, and LIST THE AXES --
simultaneity is an axis and it is usually missing
o run the suite against a design known to be wrong; it validates every
directed scenario in one compile and catches the scenario that
passes on both
o a survivor has four meanings and only one blames the bench; measure
equivalence with a probe rather than arguing it
o a mutation script must assert its replacement applied, or a
generation failure reads as a survivor
o mark the scenarios that are sole detectors, by removing them
WHAT THIS MODULE'S OWN BENCHES GOT WRONG
o a reference model containing the design's own wrong expression,
agreeing with it for 39,108 cycles
o three implementations believed equivalent that were not, surfaced
only by the cross-language directed counts differing 106 vs 29The next chapter changes the question from "is it right" to "is it conformant" — which is not the same thing, and a device can ship without either party noticing the difference.
Continue learning
Related tutorials
- Related topic
USB vs UART
UART spends zero wires on synchronisation and pays a tolerance budget that shrinks as the frame grows; USB spends a SYNC field, an encoding rule and a PLL to buy that budget away — measured across 5376 exhaustive points, not quoted.
- Related topic
USB vs SPI
SPI selects a peripheral with a wire routed at layout time and USB with an address the host assigned — so a chip-select contention is invisible to every slave (0 of 11) while a duplicate USB address is detected every time (274 of 274).
- Related topic
USB vs Ethernet
USB has one authority that assigns every address; Ethernet has none, so a switch infers the topology from traffic — and an inferred table is wrong 294 times out of 1065 where an assigned one is wrong 0 times out of 130.
- Related topic
USB vs PCIe
USB holds one transaction outstanding per endpoint so its throughput is exactly 1/(latency+1) whatever the wire carries; PCIe tags many at once and needs exactly latency+1 tags to saturate — both measured as closed forms over 64 points.
Standards & specifications
- Governing standard
- USB-IF (Universal Serial Bus Specification)(opens USB Implementers Forum (USB-IF) in a new tab)
Defines the USB bus — its electrical signalling, connectors, packet and transaction model, device framework and the descriptors a device must expose — together with the device-class specifications layered on it. It does not define host-controller register interfaces (xHCI and EHCI are separate documents) nor any operating system's driver architecture.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the USB curriculum.