Comparing Results and Reporting Mismatches

Part II · Module 24 · Chapter 24.8 · Comparing Results and Reporting Mismatches

1. The Comparison Is Code Too

Every chapter since 24.4 has quoted a count — 0 of 70, 2 of 70, 16 of 30, 8 of 9, 51 against 0. Those numbers exist because something compared two values and wrote down what it found, and that something is a piece of code with decisions in it.

Three decisions, and the purpose names all three: compare every element, normalise before comparing, report enough to debug from. This chapter measures each on the same data so the costs are comparable.

The data: a sixteen-byte expected buffer and an actual one with four corrupted bytes.

  expected    10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F
  actual      10 11 12 12 14 15 16 97 18 19 1A 0B 1E 1D 1E 1F

2. Stopping at the First Difference

The default, because it is what a memcmp does and what a hand-written loop does if nobody thinks about it. Measured:

  reports: byte 3 differs (0x13 vs 0x12)
  bytes actually wrong: unknown from this report

One fact, and the fact that is missing is the important one: how many.

From that report, these three situations are indistinguishable:

• one byte wrong — a single event, probably a specific corner
• four bytes wrong, scattered — which is what actually happened
• all sixteen wrong — a wholesale failure, probably an offset or a length

Those are three different investigations, and the report cannot tell them apart.

3. Comparing Every Element

One more loop iteration. Measured:

  differing bytes: 3 7 11 12   (4 of 16)

One run, and now the shape is visible: four scattered bytes, not one and not all of them. The count is a diagnosis in itself.

And the third instruction — report the difference, not just the fact of one. The exclusive-or of each pair costs nothing:

  byte  exp  act  xor
     3  13   12   01
     7  17   97   80
    11  1B   0B   10
    12  1C   1E   02
  single-bit errors: 4 of 4

All four are single-bit flips. That is a completely different diagnosis from four wholly wrong bytes: it points at something bit-level — a mask, a shift, a corrupted path — rather than at an indexing or ordering error.

Chapter 21.8 made the same point about arrays and this is its general form: a comparison that reports only "they differ" has thrown away the information that identifies the class of fault.

4. Normalising Before Comparing

The second instruction, and the one that produces false mismatches rather than uninformative ones.

The model works in int32_t. The design's result arrives from an 8-bit port. Measured:

  model says -1, DUT port reads 0xFF (255 unsigned)
  compared raw            : DIFFER
  compared after widening : equal   (DUT normalised to -1)

The values were never different. 0xFF read as a signed 8-bit quantity is -1; the two sides were holding the same number in two representations, and the comparison was asking the wrong question.

This is Chapter 22.2 §5's finding — signedness is decided on landing, and the same bits are 255 or -1 depending on how the destination is declared — arriving in the one place where it causes the most damage. A scoreboard that reports this is reporting a correct design as broken, which is the same class of harm Chapter 24.4 §5 measured when a pairing error produced 16 false mismatches in 30 checks.

So normalisation is not tidiness. Both sides must be brought into one representation before any comparison happens, and the three that matter here have all been measured earlier in the track:

• Signedness — 22.2 §5, and this section.
• Width — 22.2 §3 measured a declared width doing nothing; the destination decides, so normalise to the width the specification names.
• Order — 21.6 §4 measured that a permuted array has an identical sum; if either side can reorder, normalise before comparing or compare position by position.

5. The Misconception: The First Difference Is Enough

"Report the first mismatch and stop. Fix it, re-run, and the next one shows up. You get there in the end."

You do get there, and the cost is measurable.

It is one run per differing element. Four corrupted bytes is four runs where one would have done, and in a regression where the failing test takes an hour, that is the difference between a morning and a week.

The shape is lost, and the shape is the diagnosis. Measured, bytes 3, 7, 11, 12 — scattered. Had they been 0..3, that would say "the first word"; had they been every fourth byte, "a stride"; had they been all sixteen, "an offset or a length". Stopping at byte 3 produces the same message in all four cases.

And the fix-and-re-run loop is where models get adjusted. Each cycle presents a single isolated difference with no context, which is exactly the situation in which someone decides the model must be wrong about this one value and patches it. Chapter 24.3 §5 named that as one of the four ways independence leaks, and Chapter 24.9 is about resisting it.

There is one real exception, and it is worth stating so the rule does not get applied blindly: when the two sides have diverged structurally — different lengths, a stream that has lost synchronisation — every element after the first differs for the same reason, and listing ten thousand of them is noise. The answer there is not to stop at the first, but to report the first, the count, and a bound: first difference at element 3; 9,997 of 10,000 differ. That one line distinguishes scattered corruption from a lost sync, which is the distinction that matters.

The accurate position:

Compare everything, report the set and its shape, and cap the output rather than the comparison. The loop costs microseconds; a re-run costs an hour, and an isolated difference with no context is how a correct model gets quietly adjusted.

6. What You Should Be Able to Say Now

1. The comparison is code with decisions in it — compare every element, normalise first, report enough to debug from.
2. Stopping early destroys the count — measured, byte 3 reported, and 1, 4 or 16 wrong bytes all produce that same message.
3. Comparing everything costs one loop iteration — measured, bytes 3, 7, 11, 12 — 4 of 16, in the same run.
4. The shape is the diagnosis: scattered, contiguous, every-fourth, or all.
5. The exclusive-or is free and names the fault class — measured, 01 80 10 02, 4 of 4 single-bit.
6. Normalisation prevents false mismatches — measured, -1 against 0xFF differs raw and is equal once widened as signed.
7. That is 22.2 §5 in a scoreboard — the values were never different, only the representations.
8. Three things to normalise: signedness (22.2) · width (22.2 §3) · order (21.6 §4).
9. A report carries seven things: which check of how many · time · stimulus · both values · the difference · how many and which elements · both transaction indices.
10. The test for a report: can a reader tell a design bug from a harness bug without re-running?
11. Stopping costs one run per difference and is where models get quietly adjusted — cap the output, not the comparison.

7. Exercises

1. Corrupt four scattered bytes of a buffer and compare it two ways. What does each report, and how many runs would each need to find all four?
2. Print the exclusive-or of every differing pair. What does an all-single-bit result suggest that four whole wrong bytes would not?
3. Construct four corruptions with a shape — contiguous, or every fourth byte — and check that your report makes the shape visible.
4. Compare a signed model result against an unsigned port reading without normalising. At what value does the first false mismatch appear?
5. Write the normalisation for signedness, width and order for a comparison in your own project. Which of the three was missing?
6. Take a mismatch report your scoreboard produces today and score it against §4's seven fields. Which missing field would have saved the most time recently?
7. §5 names one real exception. Write the report line for a lost-sync case that distinguishes it from scattered corruption.
8. Describe how a fix-and-re-run loop on isolated differences leads to a model being adjusted, and which chapter's failure that produces.

8. Summary

• The comparison is code: compare every element · normalise first · report enough to debug from.
• Measured: stopping early reports byte 3 and loses the count — 1, 4 and 16 look identical.
• Comparing all of it: bytes 3, 7, 11, 12 — 4 of 16, one run, one extra iteration.
• The exclusive-or is free: 01 80 10 02, 4 of 4 single-bit — a different fault class entirely.
• Normalise: -1 against 0xFF differs raw, equal once widened — 22.2 §5 in a scoreboard, and a false mismatch against a correct design.
• Three normalisations: signedness · width · order.
• Seven fields in a report, and the test is whether a reader can tell a design bug from a harness bug without re-running.
• Stopping costs a run per difference and invites quiet model adjustment — cap the output, not the comparison.

The report is now good enough to act on. What it still cannot tell you is the thing everyone wants it to: when the model and the design disagree and both authors are confident, which of the two is wrong.