Verification Completeness Review

Chapter 24.1 asked what a test would still pass with. Turn that question on the environment itself and it becomes sharper, and much less comfortable:

Is this environment capable of failing, and for which defects?

A verification review is not an audit of effort. The suite is large, the coverage report is green, the engineer who built it is good at their job, and none of that is evidence about the design. What a review has to establish is narrower and checkable: which specific claims about this I²C block are supported by which specific observations, and what a defect would have to look like to slip past all of them.

1. What a Passing Regression Is Consistent With

Before looking at any particular environment, it helps to hold the full list of things "everything passes" does not rule out.

Every row is a real failure mode from this curriculum's own work: 20.1 reports the scoreboard that agreed with itself, 20.8 the oracle discipline that fixes it, and 20.9 an injected corruption that a protocol-level checker reports as a flawless transfer.

The rest of this chapter takes the first two rows and measures them, because they are the two that a review can settle in an afternoon and the two that most often turn out to be true.

2. The Experiment That Takes One Line: Remove the Device

Here is a register target with the contract Module 18 wrote down — a pointer that auto-increments modulo the register count, and a read-only mask whose bits refuse a write without advancing the pointer.

   module reg_target #(parameter integer N_REG = 8,
                       parameter [7:0]   RO_MASK = 8'h04,
                       parameter integer PRESENT = 1,
                       parameter integer BUG_RO_POINTER = 0)
     (input wire clk,
      input wire        do_write, do_read, set_ptr,
      input wire [7:0]  ptr_in, wdata,
      output reg [7:0]  rdata,
      output reg        acked);

      reg [7:0] mem [0:255];
      reg [7:0] ptr;
      integer i;
      initial begin
         for (i = 0; i < 256; i = i + 1) mem[i] = 8'h00;
         ptr = 8'h00; rdata = 8'hFF; acked = 1'b0;
      end

      always @(posedge clk) begin
         if (!PRESENT) begin
            rdata <= 8'hFF;       // nobody pulls the bus low
            acked <= 1'b0;        // and nobody acknowledges
         end else begin
            acked <= 1'b0;
            if (set_ptr) begin
               ptr   <= ptr_in % N_REG;
               acked <= 1'b1;
            end else if (do_write) begin
               if (RO_MASK[ptr % N_REG]) begin
                  // refused. The contract says the pointer does NOT advance.
                  acked <= 1'b0;
                  if (BUG_RO_POINTER) ptr <= (ptr + 1) % N_REG;   // the planted defect
               end else begin
                  mem[ptr % N_REG] <= wdata;
                  ptr              <= (ptr + 1) % N_REG;
                  acked            <= 1'b1;
               end
            end else if (do_read) begin
               rdata <= mem[ptr % N_REG];
               ptr   <= (ptr + 1) % N_REG;
               acked <= 1'b1;
            end
         end
      end
   endmodule

PRESENT = 0 is the whole experiment. It does not comment out the instance — it models the device being absent from the bus, which on an open-drain bus means reads return 0xFF and nothing acknowledges. That is what a real bus does with nobody answering, and it makes the experiment physical rather than a build trick.

Two scoreboards run over the same stimulus. They differ in where the expected value comes from.

   task op_write(input [7:0] d);
      begin
         @(negedge clk); wdata = d; do_write = 1;
         @(negedge clk); do_write = 0;
         obs_wdata = d;                       // the monitor reads the wire
         obs_ack   = acked;

         // SB-A: what the monitor saw transmitted vs what the sequence meant.
         cmp_a = cmp_a + 1;
         if (obs_wdata !== d) mism_a = mism_a + 1;

         if (!RO_MASK[mptr]) begin            // the predictor follows the CONTRACT
            model[mptr] = d;
            mptr = (mptr + 1) % N_REG;
         end
      end
   endtask

   task op_read;
      reg [7:0] expect_at;
      begin
         expect_at = mptr;
         @(negedge clk); do_read = 1;
         @(negedge clk); do_read = 0;
         @(negedge clk);
         obs_rdata = rdata;                   // the monitor reads the wire

         cmp_b = cmp_b + 1;
         if (obs_rdata !== model[expect_at]) begin
            mism_b = mism_b + 1;
            $display("    SB-B mismatch at reg %0d: bus=0x%02h contract=0x%02h",
                     expect_at, obs_rdata, model[expect_at]);
         end
         mptr = (mptr + 1) % N_REG;
      end
   endtask

Read the upper row's dashed links as the part the comparison does not depend on. SB-A's expected value reaches it from the sequence, not from the diagram's left-to-right path, so the device and monitor between them are decorative as far as the verdict is concerned. SB-B has no such shortcut: its expected value comes from a predictor that has never seen the bus, so the only way the two ends can agree is if the device produced what the contract says it should.

Run both with the device fitted, then with it gone.

PRESENT=1 BUG=0 | SB-A circular: 7 comparisons, 0 mismatches -> PASS
PRESENT=1 BUG=0 | SB-B contract: 7 comparisons, 0 mismatches -> PASS

PRESENT=0 BUG=0 | SB-A circular: 7 comparisons, 0 mismatches -> PASS
PRESENT=0 BUG=0 | SB-B contract: 7 comparisons, 7 mismatches -> FAIL

SB-A performs seven comparisons and reports zero mismatches with no device on the bus at all. It is not weak, or incomplete, or in need of more tests. It is measuring the path from the sequence through the driver to the monitor and back, and the design is not on that path. Adding a thousand more transactions changes nothing, because the quantity that is wrong is not the number of comparisons.

3. The Experiment That Separates Coverage From Checking

Now the second row of §1's table, with the planted defect switched on. The defect is 20.1's P07: a refused write advances the pointer when the contract says it must not.

Six coverage bins, written as plain counters so the claim is visible and so the run needs no particular tool. Each bin is named by the risk it stands for, which is the only thing that makes a bin reviewable:

   integer b_write = 0;      // a write was performed
   integer b_read  = 0;      // a read was performed
   integer b_ro    = 0;      // a write to a read-only register was refused
   integer b_lo    = 0;      // an access in the low address partition  (0-3)
   integer b_hi    = 0;      // an access in the high address partition (4-7)
   integer b_wrap  = 0;      // the pointer wrapped modulo N_REG

The suite writes and reads back every writable register, then performs the read-only refusal and confirms the register is unchanged — which is the test everybody writes, and it is a correct test. FOLLOW_UP adds one thing and only one: a write after a refusal.

      // the read-only refusal -- this is the bin everybody writes
      op_set_ptr(8'd2);
      op_write(8'h55);
      op_set_ptr(8'd2);
      op_read_and_check;            // register 2 must be unchanged

      // the transfer the plan usually does not contain
      if (FOLLOW_UP) begin
         op_set_ptr(8'd2);
         op_write(8'h55);           // refused again -- pointer must NOT move
         op_write(8'h99);           // so THIS must land in register 2 as well...
         op_set_ptr(8'd3);
         op_read_and_check;         // ...and register 3 must still hold 0xA3
      end

Four runs — defect off and on, follow-up off and on:

BUG=0 FOLLOW_UP=0 | bins hit 6/6  (write=8 read=8 ro=1 lo=8 hi=8 wrap=2)
BUG=0 FOLLOW_UP=0 | coverage 100%  checks 8  mismatches 0 -> PASS

BUG=0 FOLLOW_UP=1 | bins hit 6/6  (write=10 read=9 ro=3 lo=10 hi=8 wrap=2)
BUG=0 FOLLOW_UP=1 | coverage 100%  checks 9  mismatches 0 -> PASS

BUG=1 FOLLOW_UP=0 | bins hit 6/6  (write=8 read=8 ro=1 lo=8 hi=8 wrap=2)
BUG=1 FOLLOW_UP=0 | coverage 100%  checks 8  mismatches 0 -> PASS

    mismatch at reg 3: bus=0x99 contract=0xa3
BUG=1 FOLLOW_UP=1 | bins hit 6/6  (write=10 read=9 ro=3 lo=10 hi=8 wrap=2)
BUG=1 FOLLOW_UP=1 | coverage 100%  checks 9  mismatches 1 -> FAIL

Read the third block. The defect is present, the read-only bin is hit, coverage is 100 %, and the environment passes. The fourth block catches it, and the difference between the two is one extra transfer — a transfer that changes the coverage report not at all. Coverage is identical in all four runs.

That is the whole argument against treating a coverage percentage as a completeness measure, and it is worth stating as a rule with three parts rather than a slogan:

A behavior is verified when it is stimulated, observed, and checked. Coverage measures the first. A monitor provides the second. Only a comparison against something that did not come from the stimulus provides the third — and a bin can be green with the last two entirely absent.

The b_ro bin is worth dwelling on because it is a good bin, not a straw man. "A write to a read-only register was refused" names a real risk, and hitting it means real stimulus reached a real corner. What it does not say is which consequences of the refusal were observed. Refusal has two: the write does not land, and the pointer does not move. The suite checked the first. The bin is satisfied by either.

4. The Evidence Hierarchy

Different claims need different evidence, and the commonest review failure is accepting evidence of the wrong kind because the number attached to it is large.

Two of those rows are worth defending explicitly because they are the ones most often argued in reviews.

"Simulation passed" cannot become "hardware is correct." Chapter 19.7 catalogues the divergences, and the shortest summary is that a simulator's bus is an idealization with no rise time, no threshold, no coupling and no supply. A design can be functionally perfect in every simulation and fail on a board because 300 ns of rise time ate the sampling margin.

"The assertion is in the file" cannot become "the property holds." A property whose antecedent never occurred passes every cycle of every test. The reviewable quantity is not how many assertions exist but how many of them ever evaluated their consequent at all — and for I²C specifically, the antecedents most likely to be unreachable are the interesting ones: arbitration loss, clock stretching at an unusual point, a repeated START into a state that was mid-transfer.

5. Reviewing a Coverage Model

Four questions, in order, per covergroup. The first one disposes of most bins.

1. What risk does this bin stand for? A bin that cannot be finished with a sentence beginning "a design could get this wrong by…" is measuring the stimulus generator, not the design.
2. Which check consumes the transactions that hit it? If the answer is "the scoreboard compares everything", ask what it compares to — §2's experiment applies.
3. Which consequence of the covered behavior was observed? b_ro above is the worked example: one bin, two consequences, one checked.
4. Is this cross a risk, or a product? A cross of address × direction × length has an attractive number of bins and most of them represent nothing. A cross of stretching × phase of the transfer is small and every cell is a real hazard, because the interesting thing about stretching is exactly where in the transfer it lands (12.3).

The corresponding question about holes is the more productive one, and it is not "which bins are empty". It is: which behavior appears in no bin at all? An empty bin is visible and someone is already working on it. A behavior with no bin is invisible, and 20.2's feature matrix exists precisely to make that class of gap findable by construction rather than by luck.

6. Reading a Mutation Result Honestly

Mutation testing measures what a coverage report cannot: whether the environment notices a broken design. It is the strongest evidence available here, and it is also the easiest to misread.

A survivor is a question, not a verdict. Classify before concluding:

• Equivalent. The mutated design behaves identically on every legal input. The environment is fine and the design has a redundancy — a bound written twice, a condition that cannot occur. Worth a comment in the RTL, not a test.
• Outside the contract. The mutation changes behavior the published contract does not constrain. The plan is the artifact to fix, by deciding whether the contract should say something.
• Invalid. The mutation did not compile, or compiled into a file the build never consumed. This is the dangerous one, discussed below.
• A genuine hole. The environment should have caught it and did not. Only here does a test get written — and the test to write is the one that checks the property, not the one that kills the mutant.

That last distinction matters more than it sounds. A test written to kill a specific mutant tends to check the mutated expression; a test written for the property it violated checks a claim, and claims generalize to defects nobody injected.

7. A Completeness Review Checklist

Each line is a question with an observable answer.

Non-circularity

• Does the environment pass with the device removed? (Run it.)
• For each comparison: where did the expected value come from — a protocol rule, a written contract, or physical law? Anything else is the stimulus in disguise. (20.8)
• Does any monitor take an input from inside the design? Every claim downstream of that signal is unproven.

Capability

• Has the scoreboard ever reported a mismatch? If not, make it — by injecting a defect, not by editing the check.
• For each assertion: has its antecedent occurred in any run, and how many times?
• For each error path — NACK, arbitration loss, timeout, stuck line — is there a test that causes it, as opposed to a check that would notice it?

Plan

• Is there a written claim behind each test, with a falsifying observation? (20.1)
• Which objectives are marked unprovable with the available tools, and is that list non-empty? An empty list usually means the plan describes the toolchain rather than the design.
• Which parameter values were elaborated, as opposed to declared legal? (19.9 found a legal-looking value that does not elaborate.)

Coverage

• For each bin: the risk it stands for, and the check that consumes it.
• Which protocol behavior appears in no bin at all?
• Which consequences of a covered behavior went unobserved?

Mutation

• Did the harness prove the mutated file was compiled?
• Is every survivor classified, and is the classification argued rather than assumed?

8. What This Chapter's Evidence Does Not Establish

The target in §2 is a transaction-level model, not an I²C bus. That is deliberate — the claims are about environment structure and hold at any level of bus detail — but it means none of these runs say anything about framing, timing, stretching or arbitration. Nor is BUG_RO_POINTER a discovered defect: it is a planted one, chosen because it is the defect 20.1 reports surviving this curriculum's own first scoreboard run. The 2×2 measures the environment's response to a known defect, which is the only thing a mutation-style experiment ever measures.

And the DUT-removal experiment proves a negative about one scoreboard, not a positive about the other. SB-B failing when the device is absent shows the device is on its comparison path. It does not show SB-B's oracle is right — that is a separate claim, supported by the contract being written down independently and by nothing else.

9. Common Misconceptions

"100 % coverage means verification is done." §3 runs a defect at 100 % coverage with zero mismatches. Coverage measures stimulus. Checking is a different mechanism and there is no arithmetic relationship between them.

"A large suite is strong evidence." Size is consistent with every row of §1's table. The seven-comparison SB-A would have been just as wrong with seven thousand.

"If a mutant survives, the RTL must be wrong." A survivor says the environment did not detect that change. Three of the four classifications in §6 end with no test being written, and one of them ends with a correction to the harness.

"Assertions prove behavior." An assertion whose antecedent never occurs passes forever. Existence is not evaluation, and evaluation is not non-vacuous evaluation.

"The monitor can use the design's internal state — it is much simpler." It is, and it converts every claim that depends on that signal into an assumption. Simplicity bought there is paid for in the strength of the entire result.

"We tested the error paths; there are checks for NACK and timeout." A check for an event is not a test that causes it. Ask which test produces the NACK, and count how many times the check evaluated.

// The scoreboard compares every transaction and has never reported a
// mismatch. Reviewed, approved, reused on three projects.
//
//    txn      = seq.next();        // the sequence decides: write 0xA5 to reg 3
//    driver.send(txn);             // the driver puts it on the bus
//    sb.expect(txn);               // <-- the SAME object
//    observed = mon.get();         // the monitor reads it back off the wire
//    sb.compare(observed);         // 0xA5 == 0xA5. PASS.
//
// It measures whether the driver transmits what it was told and whether the
// monitor reads what was transmitted. Both are worth knowing. Neither is
// verification of the design, which is not on the path between them.

Two changes, and the first is what makes the second stick.

1. MAKE THE TYPES DIFFER. An intent is what was asked for; an observation is
 what the wire carried. They are not the same shape -- an observation knows
 whether the address was acknowledged and no intent can, because that is the
 target's decision. Once they are separate types, sb.expect(intent) does not
 compile.

2. COMPUTE THE EXPECTED VALUE FROM THE CONTRACT. For a read, what should come
 back is what the device contract says is in the selected register, produced
 by a model that has never seen the bus -- SB-B in this chapter.

Then keep the removal test in the regression permanently, as a test that is
REQUIRED TO FAIL. It costs one run and it is the only direct evidence that the
environment is still measuring something.

// Sign-off pack:
//    functional coverage   100%   (six bins, all hit)
//    line coverage          98%
//    assertions             0 failures
//    regression           412/412 pass
//
// The read-only bin -- "a write to a read-only register was refused" -- is
// green. The test behind it is correct:
//
//    write(RO_REG, 0x55);            // refused
//    check(read(RO_REG) == old);     // unchanged. PASS.
//
// Refusal has TWO consequences: the write does not land, and the pointer does
// not move. The bin is satisfied by either. Only the first was observed.

Review each bin by asking which CONSEQUENCES of the covered behavior are
observed, and by whom:

  bin  b_ro  "a write to a read-only register was refused"
    consequence 1  the register is unchanged     -> checked by a read-back
    consequence 2  the pointer did not advance   -> checked by NOTHING

Consequence 2's falsifying observation is a LATER write landing one register
along -- two transfers away from the refusal that caused it. Writing the
falsifier down is what surfaces the missing transfer, which is why an objective
without one is not an objective (20.1).

Then close the loop the other way: mutate the pointer behaviour and confirm the
new test fails. A test added to cover a hole that has never been seen failing
is a hole with a test next to it.

10. Reason It Through

A. An engineer shows you a regression: 900 tests, 100 % functional coverage across 40 bins, zero assertion failures, and a mutation campaign reporting 96 % kill. Which single number do you ask about first, and why that one?

The mutation campaign's denominator and its compile evidence — in that order. 96 % kill is by far the strongest claim in the list, which is exactly why it needs the most scrutiny: a harness that mutated files the build never consumed produces a high kill rate made of fictional KILLEDs, and nobody audits a KILLED. So ask how the harness proved the mutated file was the compiled file. Then ask for the survivor list and its classifications, because the four surviving percent is where the information is. The coverage and assertion numbers are consistent with almost anything and can wait.

B. A reviewer asks for the device-removal test and the engineer objects that the environment will obviously fail, so the run is a waste of a simulation. How do you respond?

By agreeing about the expectation and running it anyway, because the value of the experiment is entirely in the case where the expectation is wrong. It costs one simulation and one parameter. More usefully, propose keeping it in the regression as a test that is required to fail: a check that has never been observed failing is indistinguishable from a check that cannot fail, and this one degrades silently — the day somebody refactors an expected value to come from the sequence, every other test stays green and only this one notices.

C. An I²C environment has an assertion that SDA is stable while SCL is high, and a coverage bin for clock stretching. Both are green across a 4-hour random regression. What would you need to see before treating either as evidence?

For the assertion: the number of times it evaluated non-vacuously, and whether any of those evaluations occurred during a stretch. The rule's antecedent is "inside a byte, SCL high" — and if the environment's framing tracker is itself wrong about where bytes begin, the assertion is green because it is looking at nothing. For the coverage bin: which check consumed the stretched transactions. Stretching is a case where stimulus and checking pull apart very easily, because a stretch that the controller handles correctly produces a transfer identical to one with no stretch at all — so the bin can be hit thousands of times by traffic no check distinguishes from the unstretched case. The observation that would make the bin mean something is a transfer where the stretch changed something checkable: a byte that arrives late, a timeout that did not fire, a sample taken after the release rather than during the hold.

D. The plan contains the objective "the synchronizer prevents metastability from propagating into the protocol FSM," and the engineer has closed it with a passing directed test. What is wrong, and what should the plan say instead?

No RTL simulation can produce evidence for that claim, so the test that closed it proved something else — most likely that the chain has the right number of stages and passes a level through, which is a structural property and worth checking, but not the claim written down. The plan should split it: a structural objective, closable by simulation and inspection, that the chain is N stages deep with no intermediate tap; and a reliability objective about settling time, recorded as not provable with the available tools, with what would be needed (a library characterization, an MTBF calculation from the flop's metastability window and the input's transition rate). Deleting the second one turns the plan into a description of the toolchain, and the claim does not stop mattering because nothing here can address it.

11. Understanding Check

12. What 24.2 Settled

The reviewable question is whether the environment can fail. Not how many tests, not what percentage, not how long the regression runs. Two experiments settle most of it, and both are cheap: remove the device, and inject a defect the plan claims to cover.

An environment can pass with no device attached. Measured: seven comparisons, zero mismatches, nothing on the bus. That is not an incomplete environment; it is one whose comparisons close a loop the design is not on, and more stimulus cannot help.

Coverage and checking are independent, and the gap has a size. Measured: an active defect, 100 % of bins hit, zero mismatches. One added transfer catches it and leaves the coverage report byte-for-byte identical. A bin names a risk; only a comparison against something outside the stimulus retires one.

Every claim has a kind of evidence that can support it, and several that cannot. Simulation does not reach hardware, an assertion's existence does not reach its evaluation, and a coverage bin does not reach a check. A review that keeps those separate produces decisions; one that lets a large number stand for the wrong claim produces confidence.

The next chapter takes the claims that this one ruled out of simulation's reach — rise time, capacitance, synchronization, constraints — and asks what evidence is available for them before a board is powered for the first time. Chapter 24.3 — Electrical, Timing and Bring-Up Readiness Review.