Generating Stimulus and Test Vectors in C++

Part II · Module 27 · Chapter 27.5 · Generating Stimulus and Test Vectors in C++

1. The Requirement That Comes First

A stimulus generator has four jobs, and they are not equally important.

Reproducibility. Given the same seed, the same stream. Without this, a failure is an anecdote. Coverage. The vectors reach the cases that matter, including the boundaries. Legality. Every vector is a transaction the protocol permits — Chapter 27.2's invariant does this for free if the generator constructs real transaction objects. Shape. The distribution hits interesting values more often than uniform randomness would.

Reproducibility outranks the rest because it is the only one you cannot add later. A generator with poor coverage can be improved next week against the same seeds; a generator whose streams cannot be replayed means every failure found this quarter is unreproducible, and no amount of later work recovers them.

So the design rule is short: the entire stream must be a pure function of one recorded number.

2. The Measurement

Two thousand draws, reduced to one FNV-1a digest so that an entire stream can be compared at a glance. Three arms:

A. std::mt19937 with an EXPLICIT seed (seed=12345)
   first 8:  226 229  29 129 164 169 254 126
   digest : 6d46fe07f87757ac

B. the SAME seed, but one extra draw taken first
   first 8:  229  29 129 164 169 254 126 101
   digest : 490360ca754c931b   <-- a DIFFERENT stream

C. std::random_device -- no seed recorded anywhere
   digest : …   <-- changes every run, unreplayable

Three readings, and the middle one is the chapter's real subject.

Arm A is the whole requirement satisfied. A seeded engine is a deterministic function; the two-thousand-value stream came out byte-identical every time. Record the seed and the stream is recoverable forever.

Arm B is the failure that actually happens. Look at the first eight values: arm A starts 226 229 29 129… and arm B starts 229 29 129 164… — the same sequence shifted by one. Nothing was reseeded and nothing is broken; one extra consumer drew one number before the loop, and every subsequent vector moved. In a real environment that extra consumer is a new feature — a randomised delay, an added field, a coin flip for a new mode — inserted by someone else. The seed is still recorded and the replay no longer works, which is the worst version of this problem because the mechanism looks correct.

Arm C is what "I want real randomness" costs. Five runs, five streams, and the number that selected them exists nowhere. A failure found in one of those runs cannot be reproduced even in principle.

3. The Engine Is Portable; the Distribution Is Not

This distinction matters more than it sounds, and C++ handles the two halves very differently.

The engine's output is pinned down by the standard. std::mt19937 is a named algorithm with a specified state transition, and the standard requires a specific value as a conformance check: default-seeded, the 10,000th output must be 4123659995. Measured:

D. the raw engine, which IS specified by the standard
   mt19937 default-seeded, 10000th value = 4123659995
   the standard REQUIRES 4123659995 here: yes

So mt19937 seeded with N produces the same 32-bit sequence on any conforming implementation — a genuinely useful portability guarantee, and the reason a stored seed is meaningful across machines and compilers.

The distributions are deliberately not pinned down. std::uniform_int_distribution<unsigned> d(0, 255) is specified by the distribution it produces, not by the mapping from engine output to result. Two conforming libraries may consume different numbers of engine values and return different sequences from the same seeded engine. I cannot demonstrate a divergence here — libc++ is the only standard library on this machine — so I am stating this as a conformance fact rather than a measurement, which is the same care Chapter 26.8 §9 took about x86 memory ordering.

The consequence for a verification flow is concrete. A seed stored in a bug report today may not reproduce after a toolchain upgrade, even though the engine is identical, because the distribution's internal consumption changed. If seeds must survive that, do the reduction yourself:

std::mt19937 gen{seed};
// portable: the engine is standardised, and this arithmetic is yours
const unsigned byteValue = gen() % 256u;          // biased, and reproducible

% 256 on a 32-bit engine output is unbiased in this specific case because 2^32 is a multiple of 256; for a range that does not divide evenly, rejection sampling in your own code keeps both the uniformity and the portability:

unsigned below(std::mt19937& g, unsigned n)       // uniform in [0, n), portable
{
    const unsigned limit = std::mt19937::max() - std::mt19937::max() % n;
    unsigned v;
    do { v = g(); } while (v >= limit);
    return v % n;
}

That is six lines to own a property the library will not promise you. Whether it is worth it depends on whether your seeds are expected to outlive a compiler upgrade — for a long-lived regression suite, they usually are.

4. One Generator, One Seed, One Stream

Arm B is a design problem, not a coding mistake, and there are two structural fixes.

Make the stream a function of the vector index rather than of history. If each vector is derived from its own index, inserting a new consumer cannot shift anything:

// vector i depends on the seed and on i -- and on nothing else
std::mt19937 forVector(unsigned seed, unsigned i)
{
    return std::mt19937{seed ^ (0x9E3779B9u * (i + 1u))};
}

Now vector 500 is the same whatever happened before it, and a test that replays only vector 500 gets the right one. The cost is a fresh engine construction per vector, which for mt19937 means initialising 624 words of state — not free, and irrelevant beside a simulation step, since Chapter 27.10 §3 measured one interpreted Verilog step at thousands of nanoseconds.

Or give each consumer its own engine. One for addresses, one for lengths, one for delays, each seeded from the master seed with a fixed offset. Adding a randomised delay then perturbs only the delay stream:

std::mt19937 addrGen{seed + 1}, lenGen{seed + 2}, delayGen{seed + 3};

Both approaches implement the same rule: a vector's value must not depend on how many random numbers something else happened to consume. The second is usually more practical in an existing environment, because it can be adopted one consumer at a time.

5. Directed and Random Are Not Alternatives

Both are necessary and they do different jobs.

Directed vectors encode knowledge. Zero length, maximum length, the exact saturation boundary, an address that wraps, the first and last entry of a table, every value that has ever broken this block. They are cheap, they run every time, they never regress, and they are the only way to guarantee a specific case is covered. Chapter 27.4 §2's bug lives at exactly one boundary — a + b == 256 — and a directed vector at that boundary finds it in one comparison.

Random vectors find what you did not think of. That is their only advantage and it is a decisive one.

The practical structure is a list of directed vectors first, then randomisation to fill the space:

std::vector<Txn> buildStimulus(unsigned seed, int randomCount)
{
    std::vector<Txn> v = directedCorners();      // always, and always first
    std::mt19937 gen{seed};
    for (int i = 0; i < randomCount; ++i) v.push_back(randomTxn(gen));
    return v;
}

Two reasons directed vectors come first. They run even if the random section is cut short by a time limit or an early exit. And a failure in the directed section is unambiguous — it names a known case rather than requiring a seed replay.

And the boundary values are worth generating deliberately rather than hoping uniform randomness finds them. The chance of a uniform 8-bit pair summing to exactly 256 is 255 in 65,536 — about 0.4%, so a few hundred random pairs will probably hit it and a dozen will not. Shaping the distribution towards boundaries is more valuable than increasing the count: draw from the set {0, 1, 2, max-1, max, max/2} some fraction of the time, which is what a SystemVerilog constraint with weighted distributions does and what you have to write by hand in C++.

6. Writing Vectors Down

Chapter 23.8 built the file format; the verification-specific points are these.

A stimulus file is a better artefact than a seed for anything you want to keep. It is readable, diffable, reviewable, independent of the generator's code, and it reproduces after a refactor. The seed is a generator; the file is the vectors.

The file must say what produced it. A header recording the seed, the generator version and the configuration makes the file self-describing, so a vector set found in a directory two years later can be explained.

And the consumer must refuse an input it does not understand. Chapter 23.8 §1's rule — a parser that silently skips a malformed line turns a corrupt stimulus file into a short clean run. Chapter 23.8 §2's floor condition is the other half: assert how many vectors were read before reporting any verdict.

7. Legality for Free

If the generator produces Txn objects rather than tuples of integers, Chapter 27.2 §4's constructor has already rejected every illegal combination, and there is nothing further to do — an illegal vector cannot be in the stream because it cannot be constructed.

What remains is the generator's own obligation: do not generate values the constructor will reject. A rejection loop is the simple answer, and it needs a bound:

Txn randomTxn(std::mt19937& gen)
{
    for (int attempt = 0; attempt < 100; ++attempt) {
        const auto size = Size{gen() % 4};
        const unsigned bytes = 1u << (unsigned)size;
        const auto addr = Addr{(gen() % 0x10000u) & ~(bytes - 1u)};   // aligned by construction
        const auto len  = Len{1u + gen() % 256u};                     // 1..256 by construction
        try { return Txn{addr, len, size}; } catch (const std::invalid_argument&) { }
    }
    throw std::runtime_error("could not construct a legal transaction in 100 attempts");
}

Two deliberate choices there. The address is masked and the length biased into range, so the constructor is a safety net rather than the mechanism — a generator that relies on rejection for most of its draws is slow and, worse, has a distribution nobody can describe. And the loop is bounded and throws on exhaustion, because an unbounded rejection loop with contradictory constraints hangs the regression, which is Chapter 23.3's measured hang in a different costume.

8. The Table

9. What to Actually Do

• Make the whole stream a pure function of one recorded seed, and record it in the run's output, not just the log.
• Never use std::random_device to seed a regression run unless the value it returns is immediately printed and stored.
• Seed per vector index, or give each consumer its own engine, so a new random consumer cannot re-label every vector.
• Treat the engine as portable and the distributions as not. Do the reduction yourself if seeds must survive a toolchain upgrade.
• Record a manifest, not a seed: seed, configuration, commit, tool versions. Then test the replay in CI.
• Convert a reproduced failure into a directed vector as soon as you understand it — the seed found it, the directed test keeps it.
• Put directed vectors first so they run even when the random section is cut short.
• Shape the distribution towards boundaries rather than just increasing the count.
• Generate legal values by construction, with the transaction constructor as a net, and bound every rejection loop.
• Write vectors to a file for anything you want to keep, with a header saying what produced it, and assert how many were read before reporting a verdict.

10. Mistakes Worth Making Once

std::random_device seeding a nightly run. Five runs, five streams, no record of any of them.

Adding a randomised delay to an existing generator. Every stored seed now replays a different test.

Assuming a logged seed makes a failure reproducible. It reproduces a stream, against that version of the code.

Replaying a seed into a changed configuration. Same seed, different vector count, different run.

Assuming uniform_int_distribution is portable. The engine is; the distribution is not.

A regression suite made only of seeds. One refactor from testing nothing in particular.

Relying on randomness to hit a boundary. 0.4% for the one in Chapter 27.4.

An unbounded rejection loop. Contradictory constraints hang the run instead of failing it.

Generating integers instead of transaction objects. You have moved the legality check to nowhere.

A stimulus file with no header. Found in a directory two years later, unexplainable.

11. Interview Relevance

The frozen curriculum marks this High, and the question is usually how do you make a random test reproducible? The expected answer is "log the seed". The answer that stands out explains what a seed does and does not promise.

Start with the mechanism: a seeded engine is a deterministic function, measured here as a 2000-value stream with an identical digest across five runs, while std::random_device gave five different streams in five runs with the selecting value recorded nowhere.

Then the failure that makes the naive answer wrong, because this is the part that shows experience. One extra draw taken upstream — a new randomised delay, a new field — shifts the entire stream: the measured streams differ by exactly a one-position shift, so the seed is still logged and the replay no longer reproduces the failure. The fix is structural: derive each vector from the seed and its index, or give each consumer its own engine.

Then what else has to be recorded: the code commit, the configuration, and the tool versions, because a seed reproduces a stream rather than a run. If there is room, two good details are that the engine is portable by standard mandate — mt19937 default-seeded must give 4123659995 as its 10,000th value, which is checkable — while the distributions are explicitly not required to match across implementations; and that a reproduced failure should be promoted to a directed vector, since the seed's value decays with every change to the generator.

12. What You Should Be Able to Say Now

• That reproducibility outranks coverage, legality and distribution shape, because it is the only one that cannot be added later.
• That a seeded mt19937 reproduced a 2000-value stream identically in 5 of 5 runs, and std::random_device produced 5 different streams in 5 runs.
• That one extra draw upstream shifted the entire stream — the measured sequences differ by a one-position shift — so a logged seed stops reproducing when the generator's call pattern changes.
• That the fixes are structural: seed per vector index, or one engine per consumer.
• That the engine is portable by standard mandate (default-seeded mt19937, 10,000th value 4123659995, verified) while the distributions are not, and that the workaround is to do the reduction yourself.
• That a seed promises a stream, not a run, and that reproducing a failure needs a manifest: seed, code commit, configuration, tool versions.
• That the replay itself should be tested in CI, since an unreplayed seed is only a hypothesis.
• That a reproduced failure should be converted into a directed vector, because a seed's value decays as the generator changes.
• That directed vectors come first — they survive a truncated run and they name the case — and random vectors exist to find what you did not think of.
• That the chance of random 8-bit pairs hitting a + b == 256 is 255 in 65,536, so shaping towards boundaries beats raising the count.
• That generating transaction objects rather than integers puts legality in the constructor, and that every rejection loop needs a bound and a throw.

13. Exercises

Exercise 1 — the digest. Generate 2000 values from a seeded engine, reduce the stream to one digest, and confirm it is identical across five runs.

Exercise 2 — break the replay. Add one extra draw before the loop. Confirm the digest changes and that the first values are the old ones shifted by one.

Exercise 3 — the unrecorded seed. Switch to std::random_device and run five times. Then make it print and store the value it produced, and confirm you can replay.

Exercise 4 — the engine's mandated value. Default-seed mt19937, draw 10,000 values, and check the last one is 4123659995.

Exercise 5 — portable reduction. Replace uniform_int_distribution with your own bounded rejection sampler and confirm the stream still looks uniform over a large sample.

Exercise 6 — index-seeded vectors. Derive vector i from the seed and i. Insert a new random consumer and confirm vector 500 is unchanged.

Exercise 7 — the manifest. Write a run manifest with the seed, configuration, commit and tool versions, then replay a passing run from it in a fresh checkout and diff the output.

14. Summary

• Reproducibility is the first requirement, ahead of coverage, legality and distribution shape, because it is the only one that cannot be retrofitted.
• Measured: a seeded mt19937 reproduced its 2000-value stream identically in 5 of 5 runs; std::random_device gave 5 different streams in 5 runs with nothing recording which.
• And one extra draw upstream shifted the whole stream. The two digests differ, and the values are the same sequence offset by one — so a correctly logged seed stops reproducing the moment someone adds a random consumer.
• The engine is portable by standard mandate — default-seeded mt19937's 10,000th value must be 4123659995, and is — while the distributions are explicitly not required to match across implementations, which I state as conformance rather than measurement because only one standard library exists here.
• Fix the shift structurally: derive each vector from the seed and its index, or give each consumer a separate engine.
• A seed promises a stream, not a run. Reproducing a failure also needs the code, the configuration and the tool versions — record a manifest and test the replay in CI.
• Promote every reproduced failure to a directed vector, because the seed's value decays with each change to the generator.
• Directed first, then random. Directed vectors survive a truncated run and name their case; random ones find what you did not think of. Random 8-bit pairs hit a + b == 256 only 0.4% of the time, so shape the distribution towards boundaries.
• Generate transaction objects, not integers, so legality lives in the constructor — and bound every rejection loop so contradictory constraints fail rather than hang.

Chapter 27.6 takes the two streams this chapter and the model produce, and compares them in a way you can debug from.