When the Model Is Wrong, Not the DUT

Part II · Module 24 · Chapter 24.9 · When the Model Is Wrong, Not the DUT

1. The Meeting

The scoreboard fires. The verification engineer has an independent model and a mismatch report. The design engineer has read the RTL and believes it is right. Both are competent, both are confident, and the discussion is about to be settled by whoever is more senior or more tired.

This chapter is the alternative: an order of questions, each with evidence attached, so that the answer comes from something checkable. Every case in it is one this module actually produced.

And the premise to get out of the way first: the model being wrong is a normal outcome, not a failure of professionalism. A model is written from a document by a person, early, often before the design settles. It will be wrong sometimes. The danger is not that it is wrong; §6 is what the danger actually is.

2. First: Is the Comparison Sound?

The cheapest question, and the one most often skipped because it feels like plumbing rather than logic.

Measured in Chapter 24.4, on a design that implements its specification exactly:

  compared at the drive cycle  : 30 checks, 16 mismatch(es)
  compared when the DUT says so: 30 checks,  0 mismatch(es)

Sixteen mismatches from a correct design and a correct model. Neither implementation was wrong; the comparison was.

Four things to clear, none of which requires opening either implementation:

• Pairing — are the two values from the same transaction? (24.4 §2)
• Ordering — was the model stepped once per transaction, in the design's order? (24.1 §4: 2 of 40 reversed)
• Reset — did both sides see the same reset? (24.1 §4: 0 of 40 without one)
• Normalisation — same signedness, width and order before comparing? (24.8 §4: -1 against 0xFF)

This takes minutes and eliminates a whole category. Do it before anyone opens a file.

3. Second: Does the Model Satisfy a Hand-Derived Table?

The question that can actually break a deadlock, because it is evidence about the model that does not involve the design at all.

Chapter 24.6's callout asked for a boundary table derived by hand from the clause. This is what it is for. Measured, the model against seven rows derived from clause S4:

  7 of 7 hand-derived rows pass -> the model is consistent with the clause

Notice what that does and does not establish. It does not say the model is right about everything. It says the model agrees with a reading of the clause that was written down before the dispute, by someone working from the specification, and it does so without the design being in the room.

If this check fails, the conversation is over and the model is wrong. If it passes, the dispute has moved: it is now about what the clause means, which is a question for the specification's owner rather than for either engineer.

That is the single most useful move in the chapter. A disagreement about an implementation is hard to settle; a disagreement about a sentence is a thing a third person can read.

4. Third and Fourth: Position, Then Fraction

Where the first disagreement falls localises better than anything else in the report.

Measured across this module's two design bugs:

  the saturation-bound case first disagreed at check 28, driving toward the
  negative bound -- which is exactly where clause S3 applies
  the unsigned-concat case first disagreed at check 2, nowhere near a bound

A first disagreement at a clause boundary points at that clause — a bound, a corner, a rounding tie. A first disagreement immediately, on ordinary values, points at something structural: an operator, a width, a signedness. Chapter 24.5's polynomial bug disagreed at byte 1, which is the same signature.

What fraction disagrees comes last, and is labelled weak deliberately. Measured across this module:

  2 of 70   -> a narrow condition: a boundary, a corner, one code
  51 of 70  -> a wholesale difference: an operator, a width, a signedness
  16 of 30  -> measured from a CORRECT design with a mis-paired comparison
  8 of 9    -> a wholly wrong algorithm (24.5)

Four rates spanning 3% to 89%, and they do not order by severity. A correct design produced 53%; a real bug produced 3%. The fraction is a hint about where to look, never a verdict.

5. The Question That Must Not Be Asked

"Let's check it against the other model."

It is the most natural suggestion in the room and it settles nothing.

Measured, the two models built in this module — one from the specification, one transcribed from the RTL — on every possible single sample from reset:

  spec model vs RTL-derived model, one sample from reset: 256 of 256 agree

All of them. And on the design that mattered, Chapter 24.3 measured those same two models at 51 mismatches against 0.

So two models agreeing is compatible with one of them certifying a broken design. Agreement between two things proves neither — Chapter 18.2's point, and the sentence Module 24's own blurb is built on: a model written in C is not golden because it is written in C.

A second model is useful for exactly one thing: finding transcription errors, where two independent authors working from the same clause disagree. Chapter 22.5 §6 measured its limit — two people reading one ambiguous paragraph the same way produce a symmetric bug that nothing in either implementation reveals.

6. What the Danger Actually Is

The chapter is titled for the case where the model is wrong, and the important thing about that case is not how to detect it. It is what happens next.

Adjusting a model is cheap, invisible and permanent. Nobody reviews it, nothing records that it happened, and the model keeps its status as an oracle. Chapter 24.3 §5 named "debugging the model until it agrees" as the commonest way independence leaks, and this chapter is the moment that pressure is applied — a mismatch, a confident design engineer, and a deadline.

The asymmetry is worth stating plainly:

• Changing the design is visible: a commit, a review, a regression.
• Changing the specification is visible: a document, an agreement, a notification.
• Changing the model is a one-line edit nobody asked to see.

So the rule the measurements support:

Change the model only when a clause says it is wrong. If no clause says so, the answer is elsewhere: the specification is ambiguous and needs fixing (24.3 §6), the design is wrong, or the comparison is (§2).

The practical form is a sentence in the commit message: which clause, and what it says. A model change that cannot name one is the leak, happening.

And the honest other half: sometimes the model simply is wrong, the clause is clear, and the fix takes thirty seconds. That is a good day — the oracle was checked and corrected against the specification, which is the process working. The distinction is not whether the model changes; it is what authorised the change.

7. The Misconception: The Design Engineer Knows the Block Best

"They wrote the RTL. If they say the behaviour is correct, the model is wrong."

Expertise is real and it is about the wrong question.

The design engineer knows what the implementation does, in far more detail than anyone else. The dispute is not about what it does — the scoreboard already reported that precisely. The dispute is about what the specification requires, and on that question the person who wrote the implementation has exactly the same standing as the person who wrote the model: both read the document, and one of them read it differently.

Worse, there is a specific reason to be careful with the implementer's reading: they have had the clause's ambiguity resolved for them by the act of implementing it. Once you have written acc >> 4, "rounded" means floor, and it is genuinely hard to see that it ever meant anything else. Chapter 24.6 measured four other readings that all deserve the word.

So the replacement for seniority is §3: take the question out of both people's hands and put it on a clause. If the clause is clear, it decides. If it is not — which is the common case — then neither engineer was wrong, the specification was, and the fix is a sentence that makes it unambiguous, which is worth more than either outcome of the argument.

The accurate position:

The question is never "who is right", it is "what does the clause say". If nobody can point at one, the specification is the defect, and resolving it that way is the only outcome that prevents the same meeting happening again.

8. What You Should Be Able to Say Now

1. The model being wrong is a normal outcome — written early, from a document, by a person.
2. Triage in order, cheapest first, with evidence attached to each step.
3. First, the comparison — measured, 16 of 30 from a design meeting its specification exactly. Pairing · ordering · reset · normalisation.
4. Second, a hand-derived table — measured, 7 of 7 rows pass. Evidence about the model with the design absent.
5. That move converts an argument about implementations into a question about a sentence, which a third person can read.
6. Third, where the first disagreement falls — check 28 at a bound points at a clause; check 2 at nothing points at an operator or a width.
7. Fourth, the fraction, and it is weak — 2 of 70, 51 of 70, 16 of 30, 8 of 9 span 3% to 89% and do not order by severity.
8. Never ask whether a second model agrees — measured, 256 of 256 agreement between two models that were 51 against 0 where it mattered.
9. A second model finds transcription errors only — not shared misreadings (22.5 §6).
10. The danger is adjustment, not error: changing the design or the spec is visible; changing the model is a one-line edit nobody reviews.
11. Change the model only when a clause says it is wrong, and name the clause in the commit message.
12. Expertise is about the implementation; the dispute is about the specification — and the implementer has had the ambiguity resolved by the act of implementing.

9. Exercises

1. A scoreboard fires. List, in order, the four things you would check before opening either implementation, and what each would cost.
2. Build a hand-derived table for a clause you work with and run your model against it. What does a pass establish, exactly?
3. For two bugs you have seen, record where the first disagreement fell. Did the position point at the cause?
4. Collect four mismatch rates from your own history with their causes. Do they order by severity?
5. §5 says never to ask whether a second model agrees. Construct the case where a second model is worth having, and state precisely what it detects.
6. Find a model change in your project's history. Can you tell from the commit message which clause authorised it?
7. Write the commit-message sentence that §6 requires, for a model change you would make today.
8. Your specification turns out to be ambiguous. Write down what changes, and who has to agree to it, for the dispute not to recur.

10. Module 24 in One Page

Nine chapters from the sentence in the module's own blurb: a model written in C is not golden because it is written in C.

• Three obligations make a model an oracle — independence, determinism where required, and a specification rather than an implementation as its authority. Measured: 51 mismatches against 0 on one buggy design from two C models, and 0 of 40 outputs matching when a stateful model was not reset (24.1).
• One picture, with two load-bearing features: the specification carries no arrow, and there is exactly one comparison (24.2).
• Independence fails silently. Measured exhaustively: across all 256 single samples from reset the spec-derived and RTL-derived models agree on every one, so the cheapest test is provably unable to tell them apart — and one transaction later 32,768 of 65,536 sequences separate them (24.3).
• A correct model and a correct design still need a correct comparison — 16 of 30 mismatches from a design that meets its specification (24.4).
• Validate the oracle, then judge — the CRC reaches the published 0xF4 before it is allowed to say anything about a design (24.5).
• One word can hide five implementations, and the two readings closest to correct differ on 3.1% of the range while the two you get by accident differ on half (24.6).
• A round trip over 65,536 frames separated a correct codec pair from a byte-swapped one on nothing at all; one hand-derived frame separated them immediately (24.7).
• The comparison is code: stopping early loses the count, the exclusive-or names the fault class, and -1 against 0xFF is a false mismatch (24.8).
• And when it fires, triage by evidence — with the question about a second model struck off the list (24.9).

The thread running through all nine is one sentence, and it is the same one Modules 21, 22 and 23 each arrived at by a different route: the check that confirms a thing is usually the check that cannot distinguish it from its most likely failure. One sample from reset. One stream. A round trip. A matching rate. Every one of them passes for reasons unrelated to the property being claimed, and the only reliable escape is an answer that came from outside the thing being checked — a published value, a hand derivation, a numbered clause.

Everything in Modules 19 to 24 assumed the pieces fit together: the declaration matches, the library loads, the types line up, the simulator stays running. The closing module is what to do when they do not — taught, as its blurb says, as a chain from a wrong mental model to an observable failure to the habit that stops it returning.