Reference Models: What They Must Guarantee

Part II · Module 27 · Chapter 27.3 · Reference Models: What They Must Guarantee

1. The Three Obligations

A reference model is any code whose output is treated as the expected answer. The moment you write if (model != dut) report_error(), you have given it authority, and three properties decide whether that authority is earned.

Determinism, wherever the specification is deterministic. Given the same inputs and the same starting state, the model must produce the same output. Not usually — always. An oracle that differs between runs makes every mismatch ambiguous: was the design wrong, or was the model?

The specification as its authority. The model's answers must be derivable from the protocol document, the architecture spec, the standard. Not from reading the RTL, which reproduces its mistakes — Chapter 27.4 measures that failure and it is the worst one available in this module.

An explicit statement of what it does not model. Timing, arbitration, X-propagation, ECC, power states. A comparison in an area the model does not cover does not fail; it passes, which is worse than failing.

The rest of this chapter is mostly about the first obligation, because it is the one that can be measured directly, and because the measurements turned out to contradict most of what I expected to find.

2. Why Determinism Is Not Optional

Consider what a non-deterministic model does to a regression. A nightly run reports forty mismatches. You re-run it to collect waveforms, and there are thirty-eight. Nothing has been fixed.

You cannot bisect, because the failure does not reproduce. You cannot triage, because a mismatch no longer implies a design bug. You cannot sign off, because a clean run is consistent with a model that happened to agree this time. And the fix is not more runs — Chapter 26.9 §3 established the shape of that trap for genuinely schedule-dependent bugs, and the point here is sharper: a reference model has no excuse for being schedule-dependent, because it is a function from inputs to outputs with no concurrency in it.

So determinism is a property to engineer, and that means knowing which constructs threaten it. I wrote four candidates, ran each three times, and expected a clean sweep of hazards.

3. One Hazard of Four

A. an uninitialised accumulator
   total after adding 0..9 (should be 45) = …

B. ordering results by POINTER, with ASLR active
   report order: alpha beta gamma delta

C. iterating an unordered_map to build the report
   unordered order: rd_single wr_single idle rd_burst wr_burst
   ordered   order: idle rd_burst rd_single wr_burst wr_single

D. floating point, summed in two orders
   forward = 2.9289682539682538
   reverse = 2.9289682539682538
   equal   = yes

A is the real one, and it is as bad as it looks. Counter c; with no initialiser, then c.n += i for i in 0 to 9. The answer should be 45. It was a different nineteen-digit number every run, because += read an indeterminate value — Chapter 25.2 §1's defect, in the one place where it does maximum damage. A model that reads uninitialised state is not slightly wrong; it is a different model each time you run it.

B did not reproduce, and the reason is worth knowing. Sorting results by the address of a heap object, with address-space randomisation active, looks like textbook nondeterminism. It printed the same order three times out of three. ASLR shifts the heap base, but four sequential make_unique calls return increasing addresses regardless of where the base landed, so sorting by address reproduced the insertion order exactly. The hazard is real in principle and did not fire here, and I am reporting that rather than dressing up a failed demonstration — it is the Chapter 24.8 §3 discipline, where a textbook claim about build dependencies also failed to reproduce over five repetitions and had to be retracted.

C did not reproduce either, and it is still a defect. std::unordered_map iteration gave the identical order three times, because libc++ does not randomise hashing for these keys. But look at what the order is: rd_single wr_single idle rd_burst wr_burst, against the sorted order idle rd_burst rd_single wr_burst wr_single. It is stable and arbitrary. Nothing in the standard promises it, a library upgrade may change it, and a rehash after an insertion can change it within a single run. So a report built by iterating an unordered_map is reproducible today by luck and is not a thing to rely on; use std::map, or sort before reporting.

D needed better data, which §4 is about.

4. Floating Point: The Hazard That Needed the Right Input

Summation order is the classic determinism trap, and on ten reciprocals it did nothing at all. With data chosen to expose it:

  ten reciprocals 1/1..1/10  forward=2.9289682539682538  reverse=2.9289682539682538  equal
  1e16 then one hundred 1.0  forward=10000000000000000  reverse=10000000000000100  DIFFERENT
  one thousand 0.1           forward=99.999999999998593  reverse=99.999999999998593  equal

The middle row is the finding: summing forward produced 10000000000000000 — the forward pass lost all one hundred additions. Adding 1.0 to 10^16 in double cannot change it, because the gap between representable neighbours at that magnitude is 2. Summing in reverse accumulates the hundred 1.0s first, reaching 100, and then adds 10^16, giving 10000000000000100.

Two conclusions, and the second is the one that matters for a model.

Order matters only for data with a wide dynamic range. Two of the three cases were unaffected. So "floating-point addition is not associative" is true and is not by itself a reason to panic; it becomes a reason when the model sums quantities of very different magnitudes, which in practice means power, error accumulation, and anything with a large offset.

And the build flag changes the answer. The same source, compiled with -ffast-math:

  the same source built with -ffast-math:
  ten reciprocals 1/1..1/10  forward=2.9289682539682538  reverse=2.9289682539682538  equal
  1e16 then one hundred 1.0  forward=10000000000000084  reverse=10000000000000100  DIFFERENT
  one thousand 0.1           forward=99.999999999999773  reverse=99.999999999999773  equal

Two of the three cases moved. The 10^16 case went from 10000000000000000 to 10000000000000084 — the compiler reassociated the sum, which -ffast-math explicitly licenses — and the thousand-tenths case changed in its last three digits. For an oracle this is disqualifying: the expected answer now depends on the optimisation flags of the build, so a model compiled one way and a stored set of golden vectors generated the other way disagree for no reason connected to the design.

The rules that follow are short. Do not build a reference model with -ffast-math. Where the model sums many values, fix the order explicitly and document it. Where the specification defines a fixed-point or integer computation, model it in integers — most hardware arithmetic is exact, and a model using double for something the hardware does in 18-bit fixed point has invented an error source that the DUT does not have.

5. The Other Determinism Sources Worth Naming

Beyond the four measured, these are the ones that actually bite in verification code:

The last line of defence is cheap and worth building early: run the model twice on the same input and compare the outputs byte for byte, as part of the test suite. That one test catches every row in this table, costs the runtime of the model twice, and is the test most reference models do not have. Chapter 27.11 §5 wires it into the end-to-end flow.

6. Determinism Is Not the Same as Predicting One Answer

An important refinement, because the obligation is "deterministic where the specification is deterministic", and sometimes it is not.

Hardware legitimately has freedom. Which of two requesters wins an arbitration, how many cycles a cache fill takes, the order in which two independent completions return — these may be unspecified by design, and a model that insists on one answer will report mismatches that are not bugs.

The wrong fix is to make the model guess. The two right ones are:

Model the set of acceptable answers, and have the checker accept membership rather than equality. For an arbiter, the model says "one of these two requesters must win, and the loser must win next" — a property, not a value.

Or move the check. If the ordering is genuinely free, compare after re-ordering — sort both streams by transaction id, or compare multisets. Chapter 27.6 §7 is about exactly this, because a scoreboard that enforces an order the design does not guarantee produces a flood of false mismatches, which is how teams learn to ignore their scoreboards.

What must not happen is the model becoming non-deterministic to match the hardware's freedom. The model should be a deterministic description of a non-deterministic allowance.

7. Testing the Model Itself

A reference model is production code that happens to judge other code, and it needs its own tests. Four, in increasing value.

Known vectors from the specification. Published check values, worked examples from the standard, vectors from a conformance suite. This is the strongest test available because the authority is external — Chapter 27.11 §2 validates a CRC model against a published check value, and that single check is what makes the model trustworthy.

Determinism: §5's run-it-twice comparison.

Boundaries. Zero length, maximum length, the exact value where saturation starts, address wrap, the first and last entry of a table. Chapter 21.2's register-mask example is the cautionary tale — the bug lived at width == 32 and three middle-value assertions found nothing.

Properties rather than values, where you have them. A CRC of a message with its own CRC appended is zero. An encoder followed by a decoder is the identity. A transfer conserves total bytes. These catch classes of error that a handful of vectors miss, and they do not require knowing the right answer.

What does not count as testing the model: running it against the DUT. That is the comparison it exists to perform, and agreement proves only that two things agree — which is the whole subject of the next chapter.

8. The Table

9. What to Actually Do

• Initialise every member at its declaration. It is the one hazard here that actually fired, and the easiest to prevent.
• Iterate std::map, or sort before reporting. Never build output by iterating an unordered_map.
• Do not order results by pointer address, even though it reproduced here.
• Fix the summation order explicitly where magnitudes differ, and prefer integer or fixed-point arithmetic where the hardware is exact.
• Never build a reference model with -ffast-math. It changed two of three answers.
• Add a run-it-twice-and-diff test to the suite. It catches every determinism source at once.
• No randomness without a recorded seed, no wall-clock time, no pids, no unsorted directory listings, no threads.
• Validate against vectors from outside the project before trusting the model at all.
• Test the boundaries, not the middle: zero, maximum, the saturation point, the wrap.
• Where the hardware is genuinely free, model the acceptable set or compare after canonical re-ordering — do not make the model non-deterministic to match.
• Write down what the model does not model, and make sure nobody believes those comparisons mean anything.

10. Mistakes Worth Making Once

An uninitialised accumulator in the model. Three runs, three nineteen-digit answers, none of them 45.

Building the mismatch report by iterating an unordered_map. Stable by luck, arbitrary by design.

A model built with -ffast-math and golden vectors generated without it. Two of three results move.

Using double for arithmetic the hardware does exactly. You invented an error source the DUT does not have.

Summing a large offset and many small terms forward. All one hundred additions vanished.

A model that seeds from the clock. Every run is a new oracle.

Insisting on one answer where the arbitration is free. A flood of false mismatches, and then nobody reads the scoreboard.

Making the model non-deterministic so it matches the hardware's freedom. Now neither side is an oracle.

Testing the model by running it against the DUT. Agreement between two things is not correctness.

Testing only middle values. The bug is at the boundary — zero, maximum, the wrap.

11. Interview Relevance

The frozen curriculum marks this Critical. The question is often phrased as what makes a reference model trustworthy? and the three-part answer — deterministic, specification-derived, explicit about what it excludes — is a complete one that most candidates do not give in full.

The part that shows experience is determinism, and specifically being concrete about the sources. Naming uninitialised members, unseeded randomness, wall-clock time, unordered_map iteration order and float summation order covers the realistic list. Volunteering the measurement makes it stick: an uninitialised accumulator produced three different nineteen-digit answers in three runs where the correct total was 45, and a forward float sum of 10^16 plus one hundred 1.0s lost all one hundred additions. Then the consequence, which is the actual point: without determinism a mismatch does not imply a design bug, so you cannot triage, bisect or sign off.

Two further moves are worth having ready. -ffast-math changed two of three results on identical source, so the expected answer became a property of the build — a good concrete reason the model's build flags are part of its contract. And the cheapest test of all is to run the model twice and diff the output, which catches every source in the list and is the test most models lack. If there is time, the refinement in §6 is the senior answer: where the hardware is genuinely free, the model should describe the set of acceptable outcomes rather than becoming non-deterministic itself.

12. What You Should Be Able to Say Now

• The three obligations: determinism where the specification is deterministic, the specification as authority, and a written list of exclusions.
• Why determinism comes first: without it a mismatch does not imply a design bug, so triage, bisection and sign-off all stop working.
• That the measured hazard was an uninitialised accumulator — three runs, three different nineteen-digit values, correct answer 45.
• That pointer ordering under ASLR and unordered_map iteration both reproduced 3 of 3 here, so they are theoretical hazards on this toolchain today — and that this is not a reason to rely on either.
• That float summation order was harmless on two datasets and on the third the forward sum lost all one hundred additions to a 10^16 offset.
• That -ffast-math changed two of three results, making the oracle's answer a property of the build.
• The rest of the list: unseeded randomness, wall-clock time, pids, unsorted directory listings, environment variables, threads.
• That the cheapest universal test is run it twice and diff, and the strongest is known vectors from outside the project.
• That boundary tests, not middle values, find model bugs.
• That where the hardware is genuinely free you model the acceptable set or compare after re-ordering, rather than making the model non-deterministic.
• That comparing the model against the DUT does not test the model.

13. Exercises

Exercise 1 — the hazard that fires. Write a model with one uninitialised accumulator and run it three times. Then initialise it and confirm the answer is stable and correct.

Exercise 2 — the ones that do not. Reproduce §3's pointer-order and unordered_map arms three times each. Then break the pointer one deliberately by freeing an allocation in the middle, and break the map one by forcing a rehash.

Exercise 3 — find the data that exposes summation order. Start from ten reciprocals and confirm forward equals reverse. Then construct an input where it does not, and explain the gap between representable neighbours.

Exercise 4 — the build flag. Build the same model with and without -ffast-math and diff the outputs.

Exercise 5 — the two-run test. Add a test that runs the model twice on the same input and compares the output byte for byte. Then introduce each hazard from §5 in turn and confirm the test catches it.

Exercise 6 — the acceptable set. Model a two-requester arbiter whose winner is unspecified, and write a checker that accepts either outcome while still rejecting starvation.

14. Summary

• A reference model has three obligations: determinism where the specification is deterministic, authority derived from the specification rather than the implementation, and an explicit statement of what it does not model.
• Determinism comes first because without it a mismatch no longer implies a design bug — you cannot triage, cannot bisect, cannot sign off.
• Of four determinism hazards, one fired. An uninitialised accumulator produced three different nineteen-digit values in three runs where the answer should have been 45.
• Pointer ordering under ASLR and unordered_map iteration both reproduced 3 of 3 — reported as negative results. Neither is safe to rely on: the first held because allocation was sequential, the second because this library hashes these keys this way.
• Float summation order was harmless on two datasets and decisive on the third, where the forward sum of 10^16 plus one hundred 1.0s lost all one hundred additions.
• -ffast-math changed two of the three results on identical source, which makes the oracle's answer depend on build flags. Do not build a reference model with it, and use integers where the hardware is exact.
• The cheapest universal test is to run the model twice and diff the output; it catches every source at once and most models do not have it.
• The strongest test is known vectors from outside the project, then determinism, then boundaries, then algebraic properties. Comparing against the DUT tests nothing about the model.
• Where the hardware is genuinely free, describe the acceptable set rather than making the model non-deterministic to match it.
• And when avoidance is nearly free while the failure is unbounded, you act on a hazard without waiting for it to reproduce — which is why all four constructs are on the avoid list though only one misbehaved.

Chapter 27.4 is the second obligation, and it is the one that can invalidate an entire regression: where the model's answers actually came from.