Random Testing

Verification Foundations · Module 1 · Page 1.7

The Engineering Problem

Directed testing hit a wall: its effort grows by addition while the design's corner space grows by multiplication, so the escaping bugs hide in the corners nobody thought to hand-write. The obvious fix is to stop enumerating scenarios one by one and instead generate them — let a machine produce stimulus you never imagined.

But naïve randomness fails immediately. If you randomize every input bit independently, almost everything you generate is illegal — misaligned addresses, impossible opcodes, protocol-violating handshakes — and the DUT rejects it before any interesting behaviour happens. Pure random is mostly noise.

The methodology that actually works is constrained-random (CRV): you don't pick the values, and you don't let chaos pick them either — you describe the legal space with constraints and let a solver draw legal-but-unpredictable samples from it. That buys enormous breadth, but it introduces two new problems directed testing never had:

If a machine chose the stimulus, (1) how do you reproduce the exact run that found a bug, and (2) how do you know what the random flow actually exercised?

Motivation — why generate stimulus instead of writing it

The case for constrained-random is an asymmetry of effort against reach:

• One generator, millions of scenarios. A single constrained transaction class, written once, produces a different legal stimulus on every call to randomize(). Ten thousand iterations is ten thousand scenarios you did not type.
• It finds the corners you'd never enumerate. The bugs that escape directed suites live in combinations — feature A in mode X while feature B back-pressures during a wrap burst. Randomisation stumbles into these interactions naturally; a human almost never writes them.
• The surprise is the point. A directed test can only confirm what you already suspected. A constrained-random run can surprise you — fail in a way you didn't predict — which is exactly the bug class that matters most, because it is the one you would otherwise ship.
• It scales with compute, not headcount. Adding verification means adding cycles and seeds on a regression farm, not adding engineers to write more cases. Effort moves from enumerating scenarios to describing the legal space and measuring the result.

The motivation is not that random replaces directed — the previous lesson showed directed owns the specific, known, high-value scenarios and regression — but that for breadth, generating legal stimulus beats hand-writing it by orders of magnitude.

Mental Model

Hold this picture:

Directed testing is writing the script; constrained-random is writing the rules of the game and letting a solver play it a million times. You declare which fields are random (rand), you write constraints that fence off the legal region of the value space, and on each randomize() a constraint solver picks one legal-but-unpredictable point inside that fence. You never name the values — you name the boundaries, and the solver explores within them. A single integer, the seed, determines the entire sequence of draws, so "random" is fully replayable: same seed, same run, every time.

The skill shifts accordingly. In directed testing you craft cases; in constrained-random you craft the space — tight enough that every sample is legal, loose enough that the interesting corners remain reachable. Over-constrain and the solver can never reach the bug; under-constrain and it wastes cycles generating illegal noise.

Visual Explanation — the constrained-random generator

A constrained-random environment is a small machine: constraints plus a seed feed a solver, the solver produces one legal transaction per call, and that stream of transactions drives the DUT. The seed is the single knob that makes the whole stream reproducible.

The two inputs to the solver carry the whole idea. Constraints are how you keep every sample legal without naming any value — they replace ten thousand hand-written cases with one description of the space. The seed is how you make a pseudo-random process deterministic: the same seed reproduces the same draws, so a failure found at run-time can be replayed bit-for-bit at debug-time. Lose either and the methodology breaks — no constraints means illegal noise, no recorded seed means an unreproducible failure.

RTL / Simulation Perspective — write the rules, let the solver play

Take a bus transaction. Instead of hand-writing addresses and lengths, declare the random fields and fence the legal space with constraints; then loop randomize() to get a fresh legal scenario every iteration.

class bus_txn;
  rand bit [31:0] addr;
  rand bit [7:0]  len;        // burst length, in beats
  rand bit [1:0]  burst;      // 00=FIXED, 01=INCR, 10=WRAP
 
  // Constraints define the LEGAL space — never specific values.
  constraint c_addr { addr % 4 == 0; }                      // word-aligned
  constraint c_len  { len inside {[1:16]}; }                // 1..16 beats
  constraint c_wrap { burst == 2'b10 -> len inside {2,4,8,16}; } // WRAP ⇒ power-of-two
endclass
 
initial begin
  bus_txn t = new();
  repeat (10_000) begin
    if (!t.randomize())                  // solver draws one legal-but-unpredictable txn
      $fatal(1, "constraints unsatisfiable — the legal space is empty");
    drive(t);                            // each iteration: a scenario no one hand-wrote
  end
end

Read what changed from directed testing. There is no list of cases — there is a description of legality (c_addr, c_len, c_wrap) and a loop. The solver guarantees every one of the ten thousand transactions is legal (word-aligned, in-range, WRAP bursts power-of-two), yet no two runs of different seed produce the same stream. The engineer's effort moved from typing scenarios to bounding the space — and that single c_wrap implication is doing the work of excluding a whole class of illegal bursts that a directed author would have had to remember to avoid in every case.

Verification Perspective — breadth's two blind spots

Constrained-random buys breadth, but breadth alone is not verification. Two blind spots come with it, and each is closed by a different companion methodology:

• You don't know what it hit. Ten million legal transactions is an unknown amount of verification until you measure it. Without functional coverage, you cannot tell a thorough run from a lucky one, and the solver may pound the same easy states forever while a hard corner goes untouched. The fix is coverage — the entire subject of the next lesson, and the reason coverage and constrained-random are always taught together.
• It judges nothing. A random generator produces stimulus; it does not know the right answer. Something else has to check — assertions for contracts, a scoreboard for end-to-end intent. Random stimulus without checking just exercises the design quietly.

There is also a tuning hazard between the two: the legal space can be mis-fenced. Constrain too loosely and the solver wastes cycles on illegal or uninteresting stimulus; constrain too tightly and it can never reach the corner where the bug lives — and with no coverage, nobody notices the corner was excluded. This is why constrained-random is never run "open-loop": it is steered by coverage and judged by checkers. On its own it is a powerful engine with no map and no referee.

Runtime / Execution Flow — the seed makes a random failure reproducible

The operational fact that makes constrained-random engineering is reproducibility. A regression runs thousands of seeds; when one fails, the recorded seed lets you replay that exact run deterministically and debug it like any directed test.

This loop is why "random" is not a synonym for "irreproducible." A pseudo-random generator seeded with a known integer is fully deterministic: same seed, same constraints, same RTL → identical run. So the discipline is simple and absolute — always log the seed. A failing seed is a perfect, replayable bug report; an unlogged failing seed is a bug you may never reproduce. The best practice is to keep the seed that found each bug as a permanent directed-style regression, marrying random's discovery to directed's permanence.

Waveform Perspective — an unplanned corner the solver found for free

Here is breadth made concrete. A constrained-random run drives a multi-beat burst, and because ready back-pressure is also randomised, the solver produces a mid-burst stall — valid held high while ready drops in the middle of the burst — a corner a directed author rarely writes by hand.

Nothing on this waveform was hand-placed. The burst length, the moment ready drops, the stall landing on beat 2 rather than at the start — all fell out of the solver sampling a legal space. That is the breadth directed testing cannot reach by addition. The two things still missing from the picture are exactly the two blind spots: a coverage bin to record that "mid-burst stall" was finally hit, and an assertion to judge that the handshake stayed legal through the stall. Random created the corner; measurement and checking give it meaning.

DebugLab — 50 million cycles, and the bug was never given a chance

constraint c_len { len inside {[1:16]}; }   // author's "legal" space
spec-legal lengths:  0, 1, 2, ... 16        // ← 0 is legal and must be handled
the solver could draw:  1..16 only          // len == 0 was UNREACHABLE
zero-length transfers actually generated:  0  (in 50,000,000 cycles)

Common Mistakes

• Equating cycle count with coverage. "We ran 50 million cycles" measures effort, not reach. Without functional coverage you have no idea what those cycles exercised — they may have pounded the same easy states forever.
• Over-constraining the legal space. A too-tight constraint silently excludes the corner where the bug lives, and with no coverage nobody notices. Constrain to what the spec calls legal, not to what feels typical — and include the boundary values explicitly.
• Under-constraining into illegal noise. The opposite failure: randomise too freely and the solver burns cycles generating stimulus the DUT legally rejects, exercising nothing. Legality is the floor; the solver should never produce an illegal transaction.
• Not logging the seed. A random failure whose seed wasn't recorded may never be reproduced. The seed is the entire bridge from "random" to "debuggable" — log it on every run.
• Running random without a checker. A generator produces stimulus, not verdicts. Constrained-random must be paired with assertions and/or a scoreboard, or it exercises the design without ever judging it.
• Treating random as a replacement for directed. They are complementary: random for breadth, directed for the specific corners and for locking found bugs against regression. Drop directed entirely and you lose the deterministic regression net.

Senior Design Review Notes

Interview Insights

Exercises

1. Constraint as a legality claim. For the bus_txn class in this lesson, state in plain English what each of c_addr, c_len, and c_wrap declares to be illegal. Then identify one spec-legal value of len that c_len excludes, and the bug class that exclusion would hide.
2. Diagnose the green campaign. A team reports "8 million cycles, all passing, zero failures" on a packet router. Write the single question that exposes whether this is thorough or merely lucky, and name the artifact you would ask to see.
3. Find vs lock. A constrained-random run fails on seed 0xC0FFEE. Describe exactly how you reproduce it, and then explain what you would add to the suite so this corner is protected even after the constraints later change.
4. Fence the space correctly. You must verify a FIFO whose legal depths are 0–16 (zero is legal and special). Write the len constraint two ways — one that over-constrains (and hides the zero bug) and one that is correct — and add the one coverage bin that would have caught the difference.

Summary

• Random testing climbs directed testing's wall by generating stimulus instead of enumerating it. Its usable form is constrained-random: declare rand fields, fence the legal space with constraints, and let a solver draw legal-but-unpredictable transactions — reaching interaction corners no one would hand-write.
• The effort shifts from writing cases to describing the legal space: one generator yields millions of scenarios, scaling with compute rather than headcount.
• Two mechanisms make it engineering rather than gambling. The seed makes any run fully reproducible — same seed, same run — so a random failure is as debuggable as a directed one; always log the seed.
• Breadth has two blind spots: random doesn't know what it hit (closed by coverage — the next lesson) and doesn't judge anything (closed by assertions and a scoreboard). And the legal space can be mis-fenced — over-constraint silently excludes the bug, invisible without coverage.
• The durable rule of thumb: constrain to the spec's legal space, log every seed, and never trust cycle count over coverage — random discovers the corner, but only measurement proves you reached it.

Next — Coverage-Driven Verification: constrained-random's breadth is unaccountable until you can measure it. The next lesson takes up functional coverage — how you define "done," reveal the corners random never hit, and steer stimulus to close them.