Scoreboard-Style Comparison and Mismatch Reporting

Part II · Module 27 · Chapter 27.6 · Scoreboard-Style Comparison and Mismatch Reporting

1. What a Scoreboard Is

A scoreboard holds two streams and decides whether they agree: what the model said should happen, and what the design did. Everything else about it — queues, ordering, timeouts, reporting — follows from the fact that the two streams arrive at different times and in possibly different orders.

The two jobs are deciding and explaining, and they have very different difficulty. Deciding is one ==. Explaining is the subject of this chapter, and it is where a scoreboard either saves the team time or wastes it.

2. The Report Everybody Writes First

for (std::size_t i = 0; i < e.size(); ++i)
    if (e[i] != a[i]) {
        std::printf("FAIL: first mismatch at index %zu: expected 0x%02X got 0x%02X\n",
                    i, e[i], a[i]);
        return;
    }

It is correct. It detects every failure that a full comparison would detect — if any element differs, this finds one. The verdict is right.

And it throws away almost all of the evidence. The run produced two hundred comparisons; the report describes one of them. Everything that distinguishes one kind of bug from another lives in the two hundred, and §3 measures how much that matters.

3. Two Faults, One Message

The same two-hundred-element expected stream, with two different faults injected into the actual stream.

Fault A: output bit 3 is stuck high. A plausible hardware failure — a tied net, a bad mask, an inverted enable. Fault B: the output stream is delayed by one element. Equally plausible — a pipeline stage added without adjusting the monitor, a missing first-cycle handshake.

First, the report that stops at the first mismatch:

A. a report that stops at the first mismatch
  fault A    FAIL: first mismatch at index 1: expected 0x30 got 0x38
  fault B    FAIL: first mismatch at index 0: expected 0x0B got 0x00

Two sentences of the same shape, differing in an index and two hex values. Nothing there tells you that one of these is a stuck bit and the other is a timing offset. Both messages lead to the same next action — open the waveform, find that index, start reading — and only one of them needed to.

Now the same two runs, comparing every element and deriving two statistics:

B. the same two runs, comparing every element
  fault A    99 of 200 elements mismatch
             bits that ever differed: b3(99)
             mismatches whose actual == previous expected: 0 of 99
  fault B    200 of 200 elements mismatch
             bits that ever differed: b0(200) b1(101) b2(149) b3(125) b4(62) b5(168) b6(115) b7(57)
             mismatches whose actual == previous expected: 199 of 200

Each fault now names itself. Fault A differs in exactly one bit position, in every single mismatch, and never looks like a shifted stream — that is a stuck or inverted bit, and the bit number is printed. Fault B differs in all eight bits, and its actual value equals the previous expected value 199 times out of 200 — that is an off-by-one, and no waveform is needed to know it.

Three details about these numbers.

The 99 is a property of the data, not a coincidence. Bit 3 stuck high can only cause a mismatch where the expected value's bit 3 is already low. Of the 200 values (i*37 + 11) & 0xFF, exactly 99 have bit 3 clear. So the count is derivable, which is what makes it trustworthy — and the whole transcript is deterministic and reproduces byte for byte.

The two derived statistics are the entire mechanism, and they are cheap: accumulate expected ^ actual into a per-bit histogram, and count how often actual[i] == expected[i-1]. That is about ten lines, and between them they separate the two commonest classes of stream bug.

And the mismatch count alone is informative before any of that. 99 of 200 is a data-dependent bug; 200 of 200 is something structural, usually an offset or a wholesale misconfiguration; 1 of 200 is a specific input, which is the case where the first-mismatch report was adequate all along.

4. What the Report Should Contain

From the measurement, in order of value per line:

Two things that should not be in it. Every mismatch, when there are two hundred thousand — print the first few and the statistics, and write the rest to a file. And a bare FAIL, which Chapter 22.9 §1's rule rejects: a diagnostic has to be actionable, and the actionable content here is the pattern.

The shape worth standardising on, emitted by the same program so that this is captured output rather than a proposal:

  fault A -- one output bit stuck high
  compared      : 200
  mismatches    : 99  (49.5%)
  first 3       : idx   1 in=0x01 model=0x30 dut=0x38
                 idx   2 in=0x02 model=0x55 dut=0x5D
                 idx   5 in=0x05 model=0xC4 dut=0xCC
  bit histogram : b3(99)
  shifted       : 0 of 99
  VERDICT       : FAIL

The input column is the one people leave out, and it is what makes the line actionable: an expected and an actual value tell you what went wrong, and only the input tells you what to do next.

5. Ordering, and the False Mismatches It Causes

The hardest real-world part of a scoreboard is not comparing — it is deciding what to compare with what.

In-order streams compare element by element, which is §3's case.

Out-of-order completions are the common complication: a design may legitimately return transactions in a different order from the one they were issued in. Comparing positionally then produces a flood of mismatches with no bugs behind them, and Chapter 27.3 §6's rule applies — the model must not be made non-deterministic to compensate. The fix is to compare on identity:

// expected and actual, keyed by transaction id rather than position
std::map<unsigned, Txn> expectedById, actualById;

Then a mismatch is a real disagreement, and two further conditions become visible and checkable: an id in expected with no actual is a dropped transaction, and an id in actual with no expected is an unexpected one. Positional comparison cannot distinguish those two from a data error, and both are serious.

Partial ordering is the awkward middle: order matters within a stream but not between streams. Compare per stream, not globally.

And an unmatched entry needs a timeout, or the scoreboard reports zero mismatches on a design that stopped responding — the same silent-pass shape as Chapter 23.8 §2's empty run. At the end of the test, anything left in a queue is a failure:

if (!expectedById.empty())
    report("%zu expected transactions never arrived", expectedById.size());
if (!actualById.empty())
    report("%zu unexpected transactions arrived", actualById.size());

6. Comparing Transactions Rather Than Bytes

Everything above used bytes. Real scoreboards compare transactions, and two decisions matter.

Use a defaulted operator== so a field added later is compared automatically — Chapter 27.2 §6's argument. The failure mode of a hand-written comparison is that it silently stops checking a new field, which is the scoreboard equivalent of Chapter 27.4's non-independent model: a comparison that cannot fail.

And report field by field, not object by object. expected != actual on a ten-field transaction tells you nothing; which field differed is the whole content:

void reportDiff(const Txn& e, const Txn& a)
{
    if (e.addr() != a.addr()) report("addr: model 0x%X dut 0x%X", e.addr(), a.addr());
    if (e.len()  != a.len())  report("len : model %u dut %u",     e.len(),  a.len());
    // ... one line per field
}

The per-field version also aggregates usefully: counting mismatches per field across the run is the transaction-level analogue of §3's bit histogram. All mismatches in addr and none in len points at address decoding; mismatches spread evenly across fields points at an ordering or indexing problem.

7. Making the Scoreboard Prove It Can Fail

The scoreboard decides whether the run passed, so it is exactly the component whose own failure is invisible. Chapter 27.4 §8's habit applies with full force, and there are three cheap controls.

Inject a fault and confirm the report. Not just that it fails — that the diagnosis is right. §3's two faults are a good pair to keep as self-tests, since each exercises a different statistic.

Unit-test the comparator with no simulation at all. It is a pure function of two vectors, so it can be tested in milliseconds — including the empty case, the all-matching case, the off-by-one case, and the case where one stream is shorter.

Assert the work count. compared : 200 beside the verdict, and a floor on it. A scoreboard that compared nothing reports a pass, which is Chapter 23.8 §2's measured defect and the easiest silent failure in any flow.

The one that catches the most in practice is the shortest: a test in which the actual stream is deliberately correct except for one element, asserting that the scoreboard reports exactly one mismatch at exactly that index. If that test is missing, nothing anywhere establishes that the comparison runs.

8. The Table

9. What to Actually Do

• Compare every element, not just until the first mismatch.
• Print how many were compared next to the verdict, and assert a floor on it.
• Accumulate a per-bit XOR histogram over all mismatches — it names a stuck or inverted bit for free.
• Count how often actual equals the previous expected — it names an off-by-one for free.
• Print the first few mismatches in full, with the input, and write the rest to a file.
• Compare by transaction id wherever completions may be reordered, and report dropped and unexpected ids separately.
• Time out unmatched entries and fail on anything left in a queue at the end of the test.
• Default operator== on the transaction, and report field by field; count mismatches per field across the run.
• Unit-test the comparator with no simulation: empty, all-matching, one-wrong, off-by-one, short stream.
• Keep a one-element-wrong self-test that asserts exactly one mismatch at exactly that index.
• Never print a bare FAIL.

10. Mistakes Worth Making Once

Stopping at the first mismatch. Two unrelated faults, two nearly identical messages.

Reporting a verdict with no comparison count. A scoreboard that compared nothing passes.

Printing all two hundred thousand mismatches. The log is now the problem.

Omitting the input from the mismatch line. You know what went wrong and not what to run.

Comparing positionally when completions may reorder. A flood of mismatches with no bug behind them.

Making the model non-deterministic to match a reordering design. Now neither side is an oracle.

Never timing out an unmatched expected transaction. A design that stopped responding reports zero mismatches.

A hand-written operator== that someone's new field is not in.

Comparing whole objects and reporting expected != actual. Which field?

Never injecting a fault to check the scoreboard. The one component whose failure is invisible.

11. Interview Relevance

The frozen curriculum marks this Critical. The question is usually how would you write a scoreboard? and the structural answer — two queues, compare on arrival, fail on anything unmatched at the end — is the expected one.

The point that distinguishes a strong answer is what the report contains, because that is where a scoreboard's value actually sits. Say that you compare every element rather than stopping at the first mismatch, and justify it with the measurement: a stuck output bit and a one-element stream delay produced nearly identical first-mismatch messages, while the full comparison showed one fault differing in exactly one bit in all 99 of its mismatches and the other equal to the previous expected value in 199 of its 200 — a stuck bit and an off-by-one, each diagnosed without opening a waveform.

Then the two statistics, because they are concrete and cheap: a per-bit histogram of expected XOR actual, and a count of how many mismatches have actual[i] == expected[i-1]. Together with the mismatch fraction — one is a specific input, half is data-dependent, all is structural — they classify most stream bugs in about twenty lines.

Then ordering, which is where real scoreboards get hard: compare by transaction id rather than position where completions may reorder, which also makes dropped and unexpected transactions visible as distinct conditions, and time out anything unmatched so a design that stopped responding cannot report a clean run. Finishing with "and I keep a self-test that corrupts exactly one element and asserts exactly one mismatch at that index" shows you have thought about the scoreboard being wrong.

12. What You Should Be Able to Say Now

• That a scoreboard decides and explains, and that deciding is one == while explaining is where the value is.
• That first-mismatch reporting gives the right verdict and almost no evidence.
• That two genuinely different faults produced nearly identical first-mismatch messages.
• That the full comparison diagnosed both: fault A 99 of 200, bit b3 only, 0 shifted — a stuck bit; fault B 200 of 200, all eight bits, 199 of 200 equal to the previous expected — an off-by-one.
• That 99 is derivable from the data (the values whose bit 3 is already clear), which is why it is trustworthy.
• The two aggregate statistics: a per-bit XOR histogram and a shift count.
• That the mismatch fraction classifies before anything else: one, half, or all.
• That the report must carry the comparison count and the input, and that a bare FAIL is not a diagnostic.
• That reordered completions require comparison by id, which also exposes dropped and unexpected transactions.
• That unmatched entries need a timeout, or a dead design reports a clean run.
• That transactions should use a defaulted operator== and be reported field by field, with per-field counts across the run.
• That the scoreboard needs its own tests — a fault injection, a unit-tested comparator, and a one-element-wrong self-test.

13. Exercises

Exercise 1 — the two faults. Build a 200-element expected stream, inject a stuck bit and a one-element delay, and write a first-mismatch report for both. Satisfy yourself that the two messages do not distinguish the faults.

Exercise 2 — the two statistics. Add a per-bit XOR histogram and a shift count. Confirm each fault is now named by the report alone.

Exercise 3 — derive the 99. Predict the mismatch count for the stuck-bit fault from the expected data before running it.

Exercise 4 — a third fault. Inject a saturation bug that only affects values above a threshold, and work out which statistic reveals it. Add the one that does.

Exercise 5 — out of order. Make the actual stream arrive in a permuted order and watch a positional comparator produce a flood of false mismatches. Then key both sides by id.

Exercise 6 — the dead design. Stop producing actual transactions halfway through and confirm your scoreboard fails rather than passing. Add the end-of-test queue check if it did not.

Exercise 7 — the self-test. Write the test that corrupts exactly one element and asserts exactly one mismatch at that index. Then break the comparator deliberately and confirm the test catches it.

14. Summary

• A scoreboard decides and explains. Deciding is one comparison; explaining is where almost all of its value lies.
• Stopping at the first mismatch gives a correct verdict and throws the evidence away. Measured: a stuck output bit and a one-element stream delay produced nearly identical first-mismatch messages.
• Comparing every element diagnosed both. Fault A: 99 of 200 mismatches, bit 3 the only bit that ever differed, 0 shifted — a stuck bit. Fault B: 200 of 200, all eight bits, and 199 of 200 actual values equal to the previous expected — an off-by-one.
• The 99 is derivable from the data rather than coincidental, and the whole transcript reproduces byte for byte.
• Two aggregate statistics do the work, in about twenty lines: a per-bit histogram of expected XOR actual, and a count of actual[i] == expected[i-1].
• The mismatch fraction classifies before anything else: one is a specific input, half is data-dependent, all is structural.
• A report needs the comparison count, the mismatch count, the pattern, the first few mismatches in full, and the input that produced them. A bare FAIL is not a diagnostic.
• Where completions may reorder, compare by transaction id — which also turns dropped and unexpected transactions into distinct, reportable conditions — and never make the model non-deterministic to accommodate the design's freedom.
• Time out unmatched entries and fail on anything left in a queue, or a design that stopped responding reports zero mismatches.
• Compare transactions with a defaulted operator==, report field by field, and count mismatches per field across the run.
• And test the scoreboard: inject a fault and check the diagnosis, unit-test the comparator with no simulator, assert the work count, and keep a one-element-wrong self-test.

Chapter 27.7 moves from streams to state: a register map whose software-visible behaviour has to be predicted rather than assumed.