Skip to content
VLSI Mentor

UART · Module 15

Functional Coverage and Configuration Crosses

A coverage model built from requirements rather than convenience, why the cross matters and the marginals do not, reporting holes instead of percentages, and a real defect that survived a full suite because one bin was empty.

Coverage answers exactly one question: was this situation ever reached? It does not say whether anything checked the outcome, and the two get conflated constantly — usually by a number on a slide.

This chapter builds the model, shows why the cross is the only part worth tracking, replaces the percentage with a list of holes, and ends with a defect that survived sixty-two passing checks and four thousand clocks of random traffic because one bin was never hit.

1. Bins Come From Requirements, Not From Convenience

The easy way to build a coverage model is to bin whatever is already a variable. That produces a model that measures the testbench rather than the design.

Every axis here is one that Chapter 14.1's requirement list makes behaviour depend on:

AxisBinsThe requirement it serves
data width5, 6, 7, 8, 9R1 — the frame is defined per width
parity modenone, even, odd, markR2 — parity is defined per mode
stop length1, 1.5, 2 bitsR1 — a legal configuration axis
injected faultnone, parity, stop, short stop, glitch, breakR4, R5, R7 — each needs its own injection
baud offset−10% … +10% in 1% stepsthe one continuous axis with a budget
FIFO levelempty, 1, middle, DEPTH−1, fullthe boundaries, not the interior

The FIFO level row is the model in miniature. Binning the level 0–16 as sixteen bins would say almost nothing: the interesting values are the ones where behaviour changes, and everything from 2 to 14 behaves identically. Five bins, four of which are boundaries.

2. The Cross Is the Model; the Marginals Are Decoration

Five widths, four parity modes and three stop lengths is sixty distinct frame formats. Hitting every width and every parity mode separately takes five frames and tells you nothing about whether nine-bit-with-mark-parity was ever sent.

The suite demonstrates this deliberately:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Six samples. Every marginal axis is fully covered.
cov.cov_sample(5, 0, 2, 0, 10);
cov.cov_sample(6, 1, 3, 0, 10);
cov.cov_sample(7, 2, 4, 0, 10);
cov.cov_sample(8, 3, 2, 0, 10);
cov.cov_sample(9, 0, 3, 0, 10);
cov.cov_sample(9, 1, 4, 0, 10);
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  PASS every width was exercised
  PASS and every parity mode
  PASS and every stop length
  PASS yet 54 of the 60 CROSS bins were never reached

Every marginal bin full. Ninety percent of the cross empty. A report tracking only the marginals would call that closed.

3. The Model

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
//===========================================================================
//  uart_cov_v — the functional coverage model of Chapter 15.3,
//               in Verilog-2001
//
//  NOT SYNTHESIZABLE. A coverage collector records WHAT WAS EXERCISED. It
//  checks nothing, and confusing the two is the most common mistake in this
//  area: a full bin says the condition was reached, not that anything
//  verified the outcome.
//
//  VERILOG-2001 HAS NO COVERGROUPS. SystemVerilog writes
//
//      covergroup cg @(posedge clk);
//          cp_parity : coverpoint parity { bins none = {0}; ... }
//          x_fmt     : cross cp_width, cp_parity, cp_stop;
//      endgroup
//
//  and gets bin management, crosses and a percentage for free. Here the bins
//  are arrays and the cross is a three-dimensional one. The arithmetic is the
//  same; what is lost is the tool's report, which is why the report task
//  below has to be written by hand -- and, as it turns out, why it ends up
//  better. See the note on holes.
//
//  THE BINS ARE CHOSEN FROM REQUIREMENTS, NOT FROM CONVENIENCE. Every axis
//  here is one that Chapter 14.1's requirement list makes behaviour depend
//  on: the frame format (R1, R2), the injected fault (R4, R5, R7), and the
//  baud offset, which is the one continuous axis the design has a budget for.
//===========================================================================
`timescale 1ns/1ps

module uart_cov_v #(
    parameter NBAUD = 21,               // baud-offset bins, -10% .. +10%
    parameter DEPTH = 16                // FIFO depth, for the level bins
) ();

    //---- the axes ---------------------------------------------------------
    integer cov_width  [5:9];           // data bits
    integer cov_parity [0:3];           // none / even / odd / mark
    integer cov_stop   [2:4];           // stop length in half-bits
    integer cov_err    [0:5];           // ERR_NONE .. ERR_BREAK
    integer cov_baud   [0:NBAUD-1];
    integer cov_lvl    [0:4];           // empty / 1 / mid / DEPTH-1 / full

    //---- the cross that actually matters ----------------------------------
    // Marginal coverage is nearly free and nearly useless: hitting every
    // width and every parity mode separately says nothing about whether
    // 9-bit-with-mark-parity was ever sent. Only the cross does.
    integer cov_fmt [5:9][0:3][2:4];

    integer n_samples;

    task cov_reset;
        integer a, b, c;
        begin
            for (a = 5; a <= 9; a = a + 1) cov_width[a]  = 0;
            for (a = 0; a <= 3; a = a + 1) cov_parity[a] = 0;
            for (a = 2; a <= 4; a = a + 1) cov_stop[a]   = 0;
            for (a = 0; a <= 5; a = a + 1) cov_err[a]    = 0;
            for (a = 0; a < NBAUD; a = a + 1) cov_baud[a] = 0;
            for (a = 0; a <= 4; a = a + 1) cov_lvl[a]    = 0;
            for (a = 5; a <= 9; a = a + 1)
              for (b = 0; b <= 3; b = b + 1)
                for (c = 2; c <= 4; c = c + 1) cov_fmt[a][b][c] = 0;
            n_samples = 0;
        end
    endtask

    // Sample one frame. Called from the testbench with the description the
    // stimulus used -- NOT with anything read back out of the design. A
    // coverage model fed from the DUT records what the DUT did, which is a
    // different and much less useful thing.
    task cov_sample;
        input integer nb;               // 5..9
        input integer pm;               // 0..3
        input integer sh;               // 2..4
        input integer er;               // 0..5
        input integer baud_idx;         // 0..NBAUD-1
        begin
            cov_width[nb]  = cov_width[nb]  + 1;
            cov_parity[pm] = cov_parity[pm] + 1;
            cov_stop[sh]   = cov_stop[sh]   + 1;
            cov_err[er]    = cov_err[er]    + 1;
            if (baud_idx >= 0 && baud_idx < NBAUD)
                cov_baud[baud_idx] = cov_baud[baud_idx] + 1;
            cov_fmt[nb][pm][sh] = cov_fmt[nb][pm][sh] + 1;
            n_samples = n_samples + 1;
        end
    endtask

    // FIFO occupancy, sampled separately because it is not a per-frame fact.
    task cov_sample_level;
        input integer lvl;
        begin
            if (lvl == 0)            cov_lvl[0] = cov_lvl[0] + 1;
            else if (lvl == 1)       cov_lvl[1] = cov_lvl[1] + 1;
            else if (lvl == DEPTH)   cov_lvl[4] = cov_lvl[4] + 1;
            else if (lvl == DEPTH-1) cov_lvl[3] = cov_lvl[3] + 1;
            else                     cov_lvl[2] = cov_lvl[2] + 1;
        end
    endtask

    //---- the report -------------------------------------------------------
    // It prints HOLES, not a percentage, and the difference is not cosmetic.
    // "87% format coverage" is a number for a slide. "width=9 parity=MARK
    // stop=1.5 never driven" is a line of stimulus somebody can write this
    // afternoon.
    integer fmt_holes, axis_holes;

    task cov_report;
        integer a, b, c;
        begin
            fmt_holes  = 0;
            axis_holes = 0;
            for (a = 5; a <= 9; a = a + 1)
                if (cov_width[a] == 0) begin
                    axis_holes = axis_holes + 1;
                    $display("  HOLE: data width %0d never driven", a);
                end
            for (a = 0; a <= 3; a = a + 1)
                if (cov_parity[a] == 0) begin
                    axis_holes = axis_holes + 1;
                    $display("  HOLE: parity mode %0d never driven", a);
                end
            for (a = 2; a <= 4; a = a + 1)
                if (cov_stop[a] == 0) begin
                    axis_holes = axis_holes + 1;
                    $display("  HOLE: stop length %0d half-bits never driven", a);
                end
            for (a = 0; a <= 5; a = a + 1)
                if (cov_err[a] == 0) begin
                    axis_holes = axis_holes + 1;
                    $display("  HOLE: error kind %0d never injected", a);
                end
            for (a = 0; a <= 4; a = a + 1)
                if (cov_lvl[a] == 0) begin
                    axis_holes = axis_holes + 1;
                    $display("  HOLE: FIFO level bin %0d never reached", a);
                end
            for (a = 5; a <= 9; a = a + 1)
              for (b = 0; b <= 3; b = b + 1)
                for (c = 2; c <= 4; c = c + 1)
                  if (cov_fmt[a][b][c] == 0) begin
                      fmt_holes = fmt_holes + 1;
                      $display("  HOLE: width=%0d parity=%0d stop=%0d never driven",
                               a, b, c);
                  end
            $display("  coverage: %0d samples, format cross %0d of 60 bins hit, %0d holes",
                     n_samples, 60 - fmt_holes, fmt_holes);
        end
    endtask

    initial cov_reset;
endmodule

SystemVerilog would express all of this as a covergroup with coverpoint and cross, and get bin management and a percentage for free. Icarus Verilog 13.0 does not implement covergroups, so the SystemVerilog model is the same arrays with int and logic — confirmed by trying, not assumed.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
//===========================================================================
//  uart_cov — the functional coverage model of Chapter 15.3,
//               in SystemVerilog
//
//  NOT SYNTHESIZABLE. A coverage collector records WHAT WAS EXERCISED. It
//  checks nothing, and confusing the two is the most common mistake in this
//  area: a full bin says the condition was reached, not that anything
//  verified the outcome.
//
//  SYSTEMSYSTEMVERILOG HAS COVERGROUPS, AND THIS FILE CANNOT USE THEM. SystemVerilog writes
//
//      covergroup cg @(posedge clk);
//          cp_parity : coverpoint parity { bins none = {0}; ... }
//          x_fmt     : cross cp_width, cp_parity, cp_stop;
//      endgroup
//
//  and gets bin management, crosses and a percentage for free. Here the bins
//  are arrays and the cross is a three-dimensional one. The arithmetic is the
//  same; what is lost is the tool's report, which is why the report task
//  below has to be written by hand -- and, as it turns out, why it ends up
//  better. See the note on holes.
//
//  THE BINS ARE CHOSEN FROM REQUIREMENTS, NOT FROM CONVENIENCE. Every axis
//  here is one that Chapter 14.1's requirement list makes behaviour depend
//  on: the frame format (R1, R2), the injected fault (R4, R5, R7), and the
//  baud offset, which is the one continuous axis the design has a budget for.
//===========================================================================
`timescale 1ns/1ps

module uart_cov #(
    parameter NBAUD = 21,               // baud-offset bins, -10% .. +10%
    parameter DEPTH = 16                // FIFO depth, for the level bins
) ();

    //---- the axes ---------------------------------------------------------
    int cov_width  [5:9];           // data bits
    int cov_parity [0:3];           // none / even / odd / mark
    int cov_stop   [2:4];           // stop length in half-bits
    int cov_err    [0:5];           // ERR_NONE .. ERR_BREAK
    int cov_baud   [0:NBAUD-1];
    int cov_lvl    [0:4];           // empty / 1 / mid / DEPTH-1 / full

    //---- the cross that actually matters ----------------------------------
    // Marginal coverage is nearly free and nearly useless: hitting every
    // width and every parity mode separately says nothing about whether
    // 9-bit-with-mark-parity was ever sent. Only the cross does.
    int cov_fmt [5:9][0:3][2:4];

    int n_samples;

    task cov_reset;
        int a, b, c;
        begin
            for (a = 5; a <= 9; a = a + 1) cov_width[a]  = 0;
            for (a = 0; a <= 3; a = a + 1) cov_parity[a] = 0;
            for (a = 2; a <= 4; a = a + 1) cov_stop[a]   = 0;
            for (a = 0; a <= 5; a = a + 1) cov_err[a]    = 0;
            for (a = 0; a < NBAUD; a = a + 1) cov_baud[a] = 0;
            for (a = 0; a <= 4; a = a + 1) cov_lvl[a]    = 0;
            for (a = 5; a <= 9; a = a + 1)
              for (b = 0; b <= 3; b = b + 1)
                for (c = 2; c <= 4; c = c + 1) cov_fmt[a][b][c] = 0;
            n_samples = 0;
        end
    endtask

    // Sample one frame. Called from the testbench with the description the
    // stimulus used -- NOT with anything read back out of the design. A
    // coverage model fed from the DUT records what the DUT did, which is a
    // different and much less useful thing.
    task cov_sample;
        input int nb;               // 5..9
        input int pm;               // 0..3
        input int sh;               // 2..4
        input int er;               // 0..5
        input int baud_idx;         // 0..NBAUD-1
        begin
            cov_width[nb]  = cov_width[nb]  + 1;
            cov_parity[pm] = cov_parity[pm] + 1;
            cov_stop[sh]   = cov_stop[sh]   + 1;
            cov_err[er]    = cov_err[er]    + 1;
            if (baud_idx >= 0 && baud_idx < NBAUD)
                cov_baud[baud_idx] = cov_baud[baud_idx] + 1;
            cov_fmt[nb][pm][sh] = cov_fmt[nb][pm][sh] + 1;
            n_samples++;
        end
    endtask

    // FIFO occupancy, sampled separately because it is not a per-frame fact.
    task cov_sample_level;
        input int lvl;
        begin
            if (lvl == 0)            cov_lvl[0] = cov_lvl[0] + 1;
            else if (lvl == 1)       cov_lvl[1] = cov_lvl[1] + 1;
            else if (lvl == DEPTH)   cov_lvl[4] = cov_lvl[4] + 1;
            else if (lvl == DEPTH-1) cov_lvl[3] = cov_lvl[3] + 1;
            else                     cov_lvl[2] = cov_lvl[2] + 1;
        end
    endtask

    //---- the report -------------------------------------------------------
    // It prints HOLES, not a percentage, and the difference is not cosmetic.
    // "87% format coverage" is a number for a slide. "width=9 parity=MARK
    // stop=1.5 never driven" is a line of stimulus somebody can write this
    // afternoon.
    int fmt_holes, axis_holes;

    task cov_report;
        int a, b, c;
        begin
            fmt_holes  = 0;
            axis_holes = 0;
            for (a = 5; a <= 9; a = a + 1)
                if (cov_width[a] == 0) begin
                    axis_holes++;
                    $display("  HOLE: data width %0d never driven", a);
                end
            for (a = 0; a <= 3; a = a + 1)
                if (cov_parity[a] == 0) begin
                    axis_holes++;
                    $display("  HOLE: parity mode %0d never driven", a);
                end
            for (a = 2; a <= 4; a = a + 1)
                if (cov_stop[a] == 0) begin
                    axis_holes++;
                    $display("  HOLE: stop length %0d half-bits never driven", a);
                end
            for (a = 0; a <= 5; a = a + 1)
                if (cov_err[a] == 0) begin
                    axis_holes++;
                    $display("  HOLE: error kind %0d never injected", a);
                end
            for (a = 0; a <= 4; a = a + 1)
                if (cov_lvl[a] == 0) begin
                    axis_holes++;
                    $display("  HOLE: FIFO level bin %0d never reached", a);
                end
            for (a = 5; a <= 9; a = a + 1)
              for (b = 0; b <= 3; b = b + 1)
                for (c = 2; c <= 4; c = c + 1)
                  if (cov_fmt[a][b][c] == 0) begin
                      fmt_holes++;
                      $display("  HOLE: width=%0d parity=%0d stop=%0d never driven",
                               a, b, c);
                  end
            $display("  coverage: %0d samples, format cross %0d of 60 bins hit, %0d holes",
                     n_samples, 60 - fmt_holes, fmt_holes);
        end
    endtask

    initial cov_reset;
endmodule

VHDL has no covergroups either, so the bins are arrays again — but VHDL is the one language here that gets a second, complementary mechanism, and it is not in the model at all. PSL cover directives, which NVC runs, cover sequences:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
-- In uart_fifo_assert.vhd. This is not a value being binned -- it is a
-- temporal shape: "the FIFO went from full to empty at least once".
-- No array of counters expresses it.
-- psl c_fill_drain : cover {full_i = '1'; (full_i = '0')[*]; empty_i = '1'}
--       report "COVER: the FIFO went from full to empty";

The two are complementary: arrays for the configuration cross, PSL cover for the temporal shapes. A model built only from arrays cannot say whether a fill-then-drain ever happened, only how many times each level was observed.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
--===========================================================================
--  uart_cov_pkg — the functional coverage model of Chapter 15.3,
--                 in VHDL-2008
--
--  NOT SYNTHESIZABLE. A coverage collector records WHAT WAS EXERCISED. It
--  checks nothing, and confusing the two is the commonest mistake in this
--  area: a full bin says the condition was reached, not that anything
--  verified the outcome.
--
--  VHDL HAS NO COVERGROUPS EITHER, so the bins are arrays -- the same shape
--  as the Verilog-2001 model. What VHDL does have, and neither Verilog file
--  can use on this toolchain, is PSL `cover`:
--
--      -- psl c_fill_drain : cover {full = '1'; (full = '0')[*]; empty = '1'};
--
--  That is a SEQUENCE being covered, not a value -- "the FIFO went from full
--  to empty at least once" -- and no array of counters expresses it. The two
--  mechanisms are complementary: arrays for the configuration cross, PSL
--  cover for the temporal shapes. See uart_fifo_assert.vhd for the latter.
--
--  THE BINS ARE CHOSEN FROM REQUIREMENTS, NOT FROM CONVENIENCE. Every axis
--  is one that Chapter 14.1's requirement list makes behaviour depend on.
--
--  It is a PACKAGE with a shared record rather than an entity, because a
--  coverage model is called from wherever stimulus is generated -- VHDL has
--  no cross-entity subprogram call, and an entity would force every sample
--  through a port.
--===========================================================================
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

package uart_cov_pkg is

    constant NBAUD : positive := 21;    -- baud-offset bins, -10% .. +10%
    constant DEPTH : positive := 16;

    type width_arr  is array (5 to 9) of natural;
    type parity_arr is array (0 to 3) of natural;
    type stop_arr   is array (2 to 4) of natural;
    type err_arr    is array (0 to 5) of natural;
    type baud_arr   is array (0 to NBAUD-1) of natural;
    type lvl_arr    is array (0 to 4) of natural;

    -- The cross that actually matters. Marginal coverage is nearly free and
    -- nearly useless: hitting every width and every parity mode separately
    -- says nothing about whether 9-bit-with-mark-parity was ever sent.
    type fmt_stop_arr   is array (2 to 4) of natural;
    type fmt_parity_arr is array (0 to 3) of fmt_stop_arr;
    type fmt_arr        is array (5 to 9) of fmt_parity_arr;

    type cov_t is record
        width   : width_arr;
        parity  : parity_arr;
        stop    : stop_arr;
        err     : err_arr;
        baud    : baud_arr;
        lvl     : lvl_arr;
        fmt     : fmt_arr;
        samples : natural;
    end record cov_t;

    procedure cov_reset(variable c : inout cov_t);

    -- Sample one frame, with the description the STIMULUS used -- never with
    -- anything read back out of the design. A coverage model fed from the DUT
    -- records what the DUT did, which is a different and much less useful
    -- thing.
    procedure cov_sample(variable c : inout cov_t;
                         constant nb, pm, sh, er, baud_idx : in natural);

    procedure cov_sample_level(variable c : inout cov_t; constant lvl : in natural);

    -- Prints HOLES, not a percentage. "87% format coverage" is a number for a
    -- slide; "width=9 parity=MARK stop=1.5 never driven" is a line of
    -- stimulus somebody can write this afternoon.
    procedure cov_report(variable c : inout cov_t;
                         variable fmt_holes  : out natural;
                         variable axis_holes : out natural);

end package uart_cov_pkg;

package body uart_cov_pkg is

    procedure cov_reset(variable c : inout cov_t) is
    begin
        c.width   := (others => 0);
        c.parity  := (others => 0);
        c.stop    := (others => 0);
        c.err     := (others => 0);
        c.baud    := (others => 0);
        c.lvl     := (others => 0);
        c.fmt     := (others => (others => (others => 0)));
        c.samples := 0;
    end procedure cov_reset;

    procedure cov_sample(variable c : inout cov_t;
                         constant nb, pm, sh, er, baud_idx : in natural) is
    begin
        c.width(nb)  := c.width(nb)  + 1;
        c.parity(pm) := c.parity(pm) + 1;
        c.stop(sh)   := c.stop(sh)   + 1;
        c.err(er)    := c.err(er)    + 1;
        if baud_idx < NBAUD then
            c.baud(baud_idx) := c.baud(baud_idx) + 1;
        end if;
        c.fmt(nb)(pm)(sh) := c.fmt(nb)(pm)(sh) + 1;
        c.samples := c.samples + 1;
    end procedure cov_sample;

    procedure cov_sample_level(variable c : inout cov_t; constant lvl : in natural) is
    begin
        if    lvl = 0         then c.lvl(0) := c.lvl(0) + 1;
        elsif lvl = 1         then c.lvl(1) := c.lvl(1) + 1;
        elsif lvl = DEPTH     then c.lvl(4) := c.lvl(4) + 1;
        elsif lvl = DEPTH - 1 then c.lvl(3) := c.lvl(3) + 1;
        else                       c.lvl(2) := c.lvl(2) + 1;
        end if;
    end procedure cov_sample_level;

    procedure cov_report(variable c : inout cov_t;
                         variable fmt_holes  : out natural;
                         variable axis_holes : out natural) is
        variable fh, ah : natural := 0;
    begin
        fh := 0; ah := 0;
        for a in 5 to 9 loop
            if c.width(a) = 0 then
                ah := ah + 1;
                report "  HOLE: data width " & integer'image(a) & " never driven"
                    severity note;
            end if;
        end loop;
        for a in 0 to 3 loop
            if c.parity(a) = 0 then
                ah := ah + 1;
                report "  HOLE: parity mode " & integer'image(a) & " never driven"
                    severity note;
            end if;
        end loop;
        for a in 2 to 4 loop
            if c.stop(a) = 0 then
                ah := ah + 1;
                report "  HOLE: stop length " & integer'image(a)
                     & " half-bits never driven" severity note;
            end if;
        end loop;
        for a in 0 to 5 loop
            if c.err(a) = 0 then
                ah := ah + 1;
                report "  HOLE: error kind " & integer'image(a) & " never injected"
                    severity note;
            end if;
        end loop;
        for a in 0 to 4 loop
            if c.lvl(a) = 0 then
                ah := ah + 1;
                report "  HOLE: FIFO level bin " & integer'image(a) & " never reached"
                    severity note;
            end if;
        end loop;
        for a in 5 to 9 loop
          for b in 0 to 3 loop
            for d in 2 to 4 loop
              if c.fmt(a)(b)(d) = 0 then
                  fh := fh + 1;
                  report "  HOLE: width=" & integer'image(a)
                       & " parity=" & integer'image(b)
                       & " stop="   & integer'image(d) & " never driven"
                      severity note;
              end if;
            end loop;
          end loop;
        end loop;
        report "  coverage: " & integer'image(c.samples) & " samples, format cross "
             & integer'image(60 - fh) & " of 60 bins hit, "
             & integer'image(fh) & " holes" severity note;
        fmt_holes  := fh;
        axis_holes := ah;
    end procedure cov_report;

end package body uart_cov_pkg;

4. Report Holes, Not a Percentage

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  HOLE: width=9 parity=3 stop=3 never driven
  HOLE: width=9 parity=3 stop=4 never driven
  HOLE: error kind 5 never injected
  coverage: 6 samples, format cross 6 of 60 bins hit, 54 holes

"Ninety percent format coverage" is a number a manager can put on a slide and nobody can act on. "width=9 parity=MARK stop=1.5 never driven" is a line of stimulus somebody can write this afternoon.

The report is the deliverable. The counters are how it is produced.

5. Calibrating the Instrument

A coverage model is a measuring instrument, and one that has never been calibrated produces numbers nobody should act on. The suite checks three things:

  • it counts what happened — a sample lands in the bins it should, and in no others;
  • it finds what did not — with a bin unreached, the report names it;
  • it is not fooled by the marginals — §2's case, constructed on purpose.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
//===========================================================================
//  tb_uart_cov_v — self-checking Verilog-2001 testbench
//
//  A coverage model is a measuring instrument, and an instrument that has
//  never been calibrated reports numbers nobody should act on.
//
//  The checks below are of three kinds:
//
//    IT COUNTS WHAT HAPPENED.  A sample lands in the bins it should and in
//                              no others.
//    IT FINDS WHAT DID NOT.    With a bin unreached, the report NAMES it.
//                              This is the half people skip, and it is the
//                              half the report exists for.
//    IT IS NOT FOOLED BY THE   Every marginal bin can be full while the
//    MARGINALS.                cross is full of holes. The suite constructs
//                              exactly that case, because it is the one that
//                              makes a coverage number lie.
//===========================================================================
`timescale 1ns/1ps

module tb_uart_cov_v;

    localparam NBAUD = 21;
    localparam DEPTH = 16;

    uart_cov_v #(.NBAUD(NBAUD), .DEPTH(DEPTH)) cov ();

    integer checks = 0, failures = 0;
    task check;
        input cond;
        input [8*80-1:0] name;
        begin
            checks = checks + 1;
            if (cond) $display("  PASS %0s", name);
            else begin failures = failures + 1; $display("  FAIL %0s", name); end
        end
    endtask

    integer nb, pm, sh, i, tot;

    initial begin
        #10_000_000;
        $display("  FAIL watchdog: simulation did not finish");
        $display("== %0d checks, %0d failures ==", checks+1, failures+1);
        $display("   RESULT: VERILOG COVERAGE TESTS FAILED (timeout)");
        $finish;
    end

    initial begin
        $display("== uart_cov_v : self-checking Verilog testbench ==");

        //=== it starts empty ================================================
        cov.cov_reset;
        check(cov.n_samples == 0, "reset leaves the model with no samples");
        tot = 0;
        for (nb = 5; nb <= 9; nb = nb + 1) tot = tot + cov.cov_width[nb];
        check(tot == 0, "and every width bin at zero");

        //=== one sample lands where it should ===============================
        cov.cov_sample(8, 1, 2, 0, 10);         // 8 bits, even, 1 stop, clean, nominal
        check(cov.n_samples == 1,        "one sample is counted once");
        check(cov.cov_width[8]  == 1,    "and lands in the width-8 bin");
        check(cov.cov_parity[1] == 1,    "and the parity-EVEN bin");
        check(cov.cov_stop[2]   == 1,    "and the 1-stop-bit bin");
        check(cov.cov_err[0]    == 1,    "and the no-error bin");
        check(cov.cov_baud[10]  == 1,    "and the nominal-baud bin");
        check(cov.cov_fmt[8][1][2] == 1, "and the CROSS bin for that format");
        check(cov.cov_width[7]  == 0 && cov.cov_parity[0] == 0,
              "and nowhere else");

        //=== an out-of-range baud index is ignored, not a crash =============
        cov.cov_sample(8, 0, 2, 0, 99);
        check(cov.n_samples == 2, "an out-of-range baud index still counts the sample");
        check(cov.cov_baud[10] == 1, "but does not land in a baud bin");

        //=== the FIFO level bins, at their boundaries =======================
        cov.cov_reset;
        cov.cov_sample_level(0);
        cov.cov_sample_level(1);
        cov.cov_sample_level(8);
        cov.cov_sample_level(DEPTH-1);
        cov.cov_sample_level(DEPTH);
        check(cov.cov_lvl[0] == 1 && cov.cov_lvl[1] == 1 && cov.cov_lvl[2] == 1
              && cov.cov_lvl[3] == 1 && cov.cov_lvl[4] == 1,
              "the five FIFO level bins each take their boundary value");
        cov.cov_sample_level(2);
        cov.cov_sample_level(DEPTH-2);
        check(cov.cov_lvl[2] == 3,
              "and everything between the boundaries falls in the middle bin");

        //=== an empty model is ALL holes ====================================
        cov.cov_reset;
        cov.cov_report;
        check(cov.fmt_holes == 60, "an unexercised model reports all 60 cross holes");
        check(cov.axis_holes == 5 + 4 + 3 + 6 + 5,
              "and every marginal bin as a hole too");

        //=== the full cross closes it =======================================
        cov.cov_reset;
        for (nb = 5; nb <= 9; nb = nb + 1)
          for (pm = 0; pm <= 3; pm = pm + 1)
            for (sh = 2; sh <= 4; sh = sh + 1)
              cov.cov_sample(nb, pm, sh, 0, 10);
        cov.cov_report;
        check(cov.n_samples == 60, "the full cross is 60 samples");
        check(cov.fmt_holes == 0,  "and leaves no cross holes");

        //=== THE case that makes a coverage number lie ======================
        // Every marginal axis fully covered, and the cross barely touched.
        // A report that only tracked the marginals would call this closed.
        cov.cov_reset;
        cov.cov_sample(5, 0, 2, 0, 10);
        cov.cov_sample(6, 1, 3, 0, 10);
        cov.cov_sample(7, 2, 4, 0, 10);
        cov.cov_sample(8, 3, 2, 0, 10);
        cov.cov_sample(9, 0, 3, 0, 10);
        cov.cov_sample(9, 1, 4, 0, 10);
        cov.cov_report;
        tot = 0;
        for (nb = 5; nb <= 9; nb = nb + 1) if (cov.cov_width[nb]  > 0) tot = tot + 1;
        check(tot == 5, "every width was exercised");
        tot = 0;
        for (pm = 0; pm <= 3; pm = pm + 1) if (cov.cov_parity[pm] > 0) tot = tot + 1;
        check(tot == 4, "and every parity mode");
        tot = 0;
        for (sh = 2; sh <= 4; sh = sh + 1) if (cov.cov_stop[sh]   > 0) tot = tot + 1;
        check(tot == 3, "and every stop length");
        check(cov.fmt_holes == 54,
              "yet 54 of the 60 CROSS bins were never reached");

        //=== the error axis =================================================
        cov.cov_reset;
        for (i = 0; i <= 5; i = i + 1) cov.cov_sample(8, 0, 2, i, 10);
        cov.cov_report;
        tot = 0;
        for (i = 0; i <= 5; i = i + 1) if (cov.cov_err[i] > 0) tot = tot + 1;
        check(tot == 6, "all six error kinds recorded when all six are injected");
        cov.cov_reset;
        for (i = 0; i <= 4; i = i + 1) cov.cov_sample(8, 0, 2, i, 10);
        cov.cov_report;
        check(cov.cov_err[5] == 0,
              "and the one that was skipped is reported as a hole by name");

        $display("== %0d checks, %0d failures ==", checks, failures);
        if (failures == 0) $display("   RESULT: ALL VERILOG COVERAGE TESTS PASSED");
        else               $display("   RESULT: VERILOG COVERAGE TESTS FAILED");
        $finish;
    end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
//===========================================================================
//  tb_uart_cov — self-checking SystemVerilog testbench
//
//  A coverage model is a measuring instrument, and an instrument that has
//  never been calibrated reports numbers nobody should act on.
//
//  The checks below are of three kinds:
//
//    IT COUNTS WHAT HAPPENED.  A sample lands in the bins it should and in
//                              no others.
//    IT FINDS WHAT DID NOT.    With a bin unreached, the report NAMES it.
//                              This is the half people skip, and it is the
//                              half the report exists for.
//    IT IS NOT FOOLED BY THE   Every marginal bin can be full while the
//    MARGINALS.                cross is full of holes. The suite constructs
//                              exactly that case, because it is the one that
//                              makes a coverage number lie.
//===========================================================================
`timescale 1ns/1ps

module tb_uart_cov;

    localparam NBAUD = 21;
    localparam DEPTH = 16;

    uart_cov #(.NBAUD(NBAUD), .DEPTH(DEPTH)) cov ();

    int checks = 0, failures = 0;
    task automatic check(input logic cond, input string name);
        checks++;
        if (cond) $display("  PASS %0s", name);
        else begin failures++; $display("  FAIL %0s", name); end
    endtask

    int nb, pm, sh, i, tot;

    initial begin
        #10_000_000;
        $display("  FAIL watchdog: simulation did not finish");
        $display("== %0d checks, %0d failures ==", checks+1, failures+1);
        $display("   RESULT: SYSTEMVERILOG COVERAGE TESTS FAILED (timeout)");
        $finish;
    end

    initial begin
        $display("== uart_cov : self-checking Verilog testbench ==");

        //=== it starts empty ================================================
        cov.cov_reset;
        check(cov.n_samples == 0, "reset leaves the model with no samples");
        tot = 0;
        for (nb = 5; nb <= 9; nb = nb + 1) tot = tot + cov.cov_width[nb];
        check(tot == 0, "and every width bin at zero");

        //=== one sample lands where it should ===============================
        cov.cov_sample(8, 1, 2, 0, 10);         // 8 bits, even, 1 stop, clean, nominal
        check(cov.n_samples == 1,        "one sample is counted once");
        check(cov.cov_width[8]  == 1,    "and lands in the width-8 bin");
        check(cov.cov_parity[1] == 1,    "and the parity-EVEN bin");
        check(cov.cov_stop[2]   == 1,    "and the 1-stop-bit bin");
        check(cov.cov_err[0]    == 1,    "and the no-error bin");
        check(cov.cov_baud[10]  == 1,    "and the nominal-baud bin");
        check(cov.cov_fmt[8][1][2] == 1, "and the CROSS bin for that format");
        check(cov.cov_width[7]  == 0 && cov.cov_parity[0] == 0,
              "and nowhere else");

        //=== an out-of-range baud index is ignored, not a crash =============
        cov.cov_sample(8, 0, 2, 0, 99);
        check(cov.n_samples == 2, "an out-of-range baud index still counts the sample");
        check(cov.cov_baud[10] == 1, "but does not land in a baud bin");

        //=== the FIFO level bins, at their boundaries =======================
        cov.cov_reset;
        cov.cov_sample_level(0);
        cov.cov_sample_level(1);
        cov.cov_sample_level(8);
        cov.cov_sample_level(DEPTH-1);
        cov.cov_sample_level(DEPTH);
        check(cov.cov_lvl[0] == 1 && cov.cov_lvl[1] == 1 && cov.cov_lvl[2] == 1
              && cov.cov_lvl[3] == 1 && cov.cov_lvl[4] == 1,
              "the five FIFO level bins each take their boundary value");
        cov.cov_sample_level(2);
        cov.cov_sample_level(DEPTH-2);
        check(cov.cov_lvl[2] == 3,
              "and everything between the boundaries falls in the middle bin");

        //=== an empty model is ALL holes ====================================
        cov.cov_reset;
        cov.cov_report;
        check(cov.fmt_holes == 60, "an unexercised model reports all 60 cross holes");
        check(cov.axis_holes == 5 + 4 + 3 + 6 + 5,
              "and every marginal bin as a hole too");

        //=== the full cross closes it =======================================
        cov.cov_reset;
        for (nb = 5; nb <= 9; nb = nb + 1)
          for (pm = 0; pm <= 3; pm = pm + 1)
            for (sh = 2; sh <= 4; sh = sh + 1)
              cov.cov_sample(nb, pm, sh, 0, 10);
        cov.cov_report;
        check(cov.n_samples == 60, "the full cross is 60 samples");
        check(cov.fmt_holes == 0,  "and leaves no cross holes");

        //=== THE case that makes a coverage number lie ======================
        // Every marginal axis fully covered, and the cross barely touched.
        // A report that only tracked the marginals would call this closed.
        cov.cov_reset;
        cov.cov_sample(5, 0, 2, 0, 10);
        cov.cov_sample(6, 1, 3, 0, 10);
        cov.cov_sample(7, 2, 4, 0, 10);
        cov.cov_sample(8, 3, 2, 0, 10);
        cov.cov_sample(9, 0, 3, 0, 10);
        cov.cov_sample(9, 1, 4, 0, 10);
        cov.cov_report;
        tot = 0;
        for (nb = 5; nb <= 9; nb = nb + 1) if (cov.cov_width[nb]  > 0) tot++;
        check(tot == 5, "every width was exercised");
        tot = 0;
        for (pm = 0; pm <= 3; pm = pm + 1) if (cov.cov_parity[pm] > 0) tot++;
        check(tot == 4, "and every parity mode");
        tot = 0;
        for (sh = 2; sh <= 4; sh = sh + 1) if (cov.cov_stop[sh]   > 0) tot++;
        check(tot == 3, "and every stop length");
        check(cov.fmt_holes == 54,
              "yet 54 of the 60 CROSS bins were never reached");

        //=== the error axis =================================================
        cov.cov_reset;
        for (i = 0; i <= 5; i = i + 1) cov.cov_sample(8, 0, 2, i, 10);
        cov.cov_report;
        tot = 0;
        for (i = 0; i <= 5; i = i + 1) if (cov.cov_err[i] > 0) tot++;
        check(tot == 6, "all six error kinds recorded when all six are injected");
        cov.cov_reset;
        for (i = 0; i <= 4; i = i + 1) cov.cov_sample(8, 0, 2, i, 10);
        cov.cov_report;
        check(cov.cov_err[5] == 0,
              "and the one that was skipped is reported as a hole by name");

        $display("== %0d checks, %0d failures ==", checks, failures);
        if (failures == 0) $display("   RESULT: ALL SYSTEMVERILOG COVERAGE TESTS PASSED");
        else               $display("   RESULT: SYSTEMVERILOG COVERAGE TESTS FAILED");
        $finish;
    end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
--===========================================================================
--  tb_uart_cov — self-checking VHDL-2008 testbench
--
--  A coverage model is a measuring instrument, and an instrument that has
--  never been calibrated reports numbers nobody should act on.
--
--  The checks are of three kinds:
--    IT COUNTS WHAT HAPPENED.  A sample lands in the bins it should and in
--                              no others.
--    IT FINDS WHAT DID NOT.    With a bin unreached, the report NAMES it.
--    IT IS NOT FOOLED BY THE   Every marginal bin can be full while the
--    MARGINALS.                cross is full of holes. The suite constructs
--                              exactly that case, because it is the one that
--                              makes a coverage number lie.
--
--  Same 24 counted checks as the Verilog and SystemVerilog twins.
--===========================================================================
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

use work.uart_cov_pkg.all;

entity tb_uart_cov is
end entity tb_uart_cov;

architecture sim of tb_uart_cov is
begin

    stim : process
        variable checks, failures : natural := 0;
        variable c : cov_t;
        variable fh, ah, tot : natural;

        procedure check(cond : boolean; name : string) is
        begin
            checks := checks + 1;
            if cond then
                report "  PASS " & name severity note;
            else
                failures := failures + 1;
                report "  FAIL " & name severity error;
            end if;
        end procedure check;
    begin
        report "== uart_cov : self-checking VHDL testbench ==" severity note;

        --=== it starts empty ================================================
        cov_reset(c);
        check(c.samples = 0, "reset leaves the model with no samples");
        tot := 0;
        for a in 5 to 9 loop tot := tot + c.width(a); end loop;
        check(tot = 0, "and every width bin at zero");

        --=== one sample lands where it should ===============================
        cov_sample(c, 8, 1, 2, 0, 10);
        check(c.samples = 1,     "one sample is counted once");
        check(c.width(8)  = 1,   "and lands in the width-8 bin");
        check(c.parity(1) = 1,   "and the parity-EVEN bin");
        check(c.stop(2)   = 1,   "and the 1-stop-bit bin");
        check(c.err(0)    = 1,   "and the no-error bin");
        check(c.baud(10)  = 1,   "and the nominal-baud bin");
        check(c.fmt(8)(1)(2) = 1,"and the CROSS bin for that format");
        check(c.width(7) = 0 and c.parity(0) = 0, "and nowhere else");

        --=== an out-of-range baud index is ignored, not a crash =============
        cov_sample(c, 8, 0, 2, 0, 99);
        check(c.samples = 2, "an out-of-range baud index still counts the sample");
        check(c.baud(10) = 1, "but does not land in a baud bin");

        --=== the FIFO level bins, at their boundaries =======================
        cov_reset(c);
        cov_sample_level(c, 0);
        cov_sample_level(c, 1);
        cov_sample_level(c, 8);
        cov_sample_level(c, DEPTH-1);
        cov_sample_level(c, DEPTH);
        check(c.lvl(0) = 1 and c.lvl(1) = 1 and c.lvl(2) = 1
              and c.lvl(3) = 1 and c.lvl(4) = 1,
              "the five FIFO level bins each take their boundary value");
        cov_sample_level(c, 2);
        cov_sample_level(c, DEPTH-2);
        check(c.lvl(2) = 3,
              "and everything between the boundaries falls in the middle bin");

        --=== an empty model is ALL holes ====================================
        cov_reset(c);
        cov_report(c, fh, ah);
        check(fh = 60, "an unexercised model reports all 60 cross holes");
        check(ah = 5 + 4 + 3 + 6 + 5, "and every marginal bin as a hole too");

        --=== the full cross closes it =======================================
        cov_reset(c);
        for nb in 5 to 9 loop
          for pm in 0 to 3 loop
            for sh in 2 to 4 loop
              cov_sample(c, nb, pm, sh, 0, 10);
            end loop;
          end loop;
        end loop;
        cov_report(c, fh, ah);
        check(c.samples = 60, "the full cross is 60 samples");
        check(fh = 0,         "and leaves no cross holes");

        --=== THE case that makes a coverage number lie ======================
        cov_reset(c);
        cov_sample(c, 5, 0, 2, 0, 10);
        cov_sample(c, 6, 1, 3, 0, 10);
        cov_sample(c, 7, 2, 4, 0, 10);
        cov_sample(c, 8, 3, 2, 0, 10);
        cov_sample(c, 9, 0, 3, 0, 10);
        cov_sample(c, 9, 1, 4, 0, 10);
        cov_report(c, fh, ah);
        tot := 0;
        for nb in 5 to 9 loop if c.width(nb)  > 0 then tot := tot + 1; end if; end loop;
        check(tot = 5, "every width was exercised");
        tot := 0;
        for pm in 0 to 3 loop if c.parity(pm) > 0 then tot := tot + 1; end if; end loop;
        check(tot = 4, "and every parity mode");
        tot := 0;
        for sh in 2 to 4 loop if c.stop(sh)   > 0 then tot := tot + 1; end if; end loop;
        check(tot = 3, "and every stop length");
        check(fh = 54, "yet 54 of the 60 CROSS bins were never reached");

        --=== the error axis =================================================
        cov_reset(c);
        for i in 0 to 5 loop cov_sample(c, 8, 0, 2, i, 10); end loop;
        cov_report(c, fh, ah);
        tot := 0;
        for i in 0 to 5 loop if c.err(i) > 0 then tot := tot + 1; end if; end loop;
        check(tot = 6, "all six error kinds recorded when all six are injected");
        cov_reset(c);
        for i in 0 to 4 loop cov_sample(c, 8, 0, 2, i, 10); end loop;
        cov_report(c, fh, ah);
        check(c.err(5) = 0,
              "and the one that was skipped is reported as a hole by name");

        report "== " & integer'image(checks) & " checks, "
                     & integer'image(failures) & " failures ==" severity note;
        if failures = 0 then
            report "   RESULT: ALL VHDL COVERAGE TESTS PASSED" severity note;
        else
            report "   RESULT: VHDL COVERAGE TESTS FAILED" severity error;
        end if;
        wait;
    end process stim;

end architecture sim;

Twenty-four checks, identical in all three languages:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  PASS one sample is counted once
  PASS and lands in the width-8 bin
  PASS and the CROSS bin for that format
  PASS and nowhere else
  PASS the five FIFO level bins each take their boundary value
  PASS an unexercised model reports all 60 cross holes
  PASS the full cross is 60 samples
  PASS every width was exercised
  PASS yet 54 of the 60 CROSS bins were never reached
  PASS and the one that was skipped is reported as a hole by name
== 24 checks, 0 failures ==

Verilog-2001    : 24 checks, 0 failures
SystemVerilog   : 24 checks, 0 failures
VHDL-2008       : 24 checks, 0 failures

6. The Defect That Survived Everything

This is the result the chapter exists for.

The FIFO property checker of Chapter 15.2 was mutated twelve ways. Eleven mutants died. One did not:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
M4  fifo  acceptance rule simplified to push && !full        *** SURVIVED ***

That mutation replaces the FIFO's real acceptance rule — a push into a full FIFO is accepted when a pop frees an entry on the same cycle — with the naive one. It is a genuine defect, it is the exact bug that produced 41 false failures earlier in this module's work, and it survived:

  • 62 passing checks across three languages,
  • 4,079 clocks of random push/pop traffic,
  • the fill test, the drain test, the overflow test and the underflow test.

Why? One line of instrumentation answers it:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  clocks=4079   push+pop while FULL occurred 0 times

Zero. The single cycle on which the two rules disagree never happened. Not rarely — never.

7. Verification

Calibrate the model before quoting it. §5. A coverage model is a program and nobody writes a correct program first time — including the program whose job is to tell you what you missed.

Track the cross, not the marginals. §2's six samples fill every marginal axis and 10% of the cross.

Report holes by name. A percentage cannot be acted on; width=9 parity=MARK stop=1.5 can.

Assert that the sample count is non-zero. A coverage model that was never connected reports no holes at all in some implementations and all holes in others; neither is a useful signal on its own.

Treat an empty bin as a task, not a statistic. §6's bin was empty for a reason — the stimulus could not reach it — and the fix was five lines of directed traffic.

And do not let coverage stand in for checking. A full bin with no checker behind it is worse than an empty one, because it looks like progress.

8. Debugging

9. Understanding Check

10. Summary

Coverage answers one question — was this reached — and never says whether anything checked it.

Bins come from requirements. Every axis here is one Chapter 14.1's requirement list makes behaviour depend on; the data value gets no axis because the design does not distinguish values.

The cross is the model and the marginals are decoration: six frames fill every marginal axis and leave 54 of 60 cross bins empty.

Report holes by name, not a percentage. One can be acted on this afternoon; the other goes on a slide.

The FIFO level gets five bins, four of them boundaries, because the interior behaves identically.

VHDL gets a second mechanism the others do not: PSL cover on sequences, which bins a temporal shape rather than a value.

And the headline: a real defect survived 62 checks and 4,079 clocks of random traffic because one bin was empty — zero occurrences, not rare ones. Five lines of directed stimulus killed it and took the campaign to twelve of twelve.

11. What Comes Next

Chapter 15.4 drives the error axis of this model for real — six injected faults, and what detects each one.

Browse the full path on the UART tutorials index. For the properties whose coverage this measures, read back to Chapter 15.2.

Continue learning

Where this fits

Part of the UART curriculum.