Test Vectors and Machine-Readable Results

Part I · Module 23 · Chapter 23.8 · Test Vectors and Machine-Readable Results

1. The Stimulus File, and Refusing to Run on What You Did Not Understand

The file format is the simplest thing that works: a comment header, then one vector per line.

# id  a           b           expected
1    0x00000001  0x00000002  0x00000003
2    0x7FFFFFFF  0x00000001  0x80000000
3    0xFFFFFFFF  0x00000001  0xFFFFFFFF
4    0xFFFFFFFE  0x00000001  0xFFFFFFFF
5    0x0000000A  0x0000000A  0x00000014
6    0x00000001  0xZZZZZZZZ  0x00000002
7    0x00000001  0x00000002

Line 6 has invalid hexadecimal and line 7 is missing its expected value. Reading it with Chapter 23.3's discipline — per-line istringstream, whole-field validation with std::from_chars, and the whole-line check:

std::istringstream ls(line);
Vector v{};
std::string ha, hb, he;
if ((ls >> v.id >> ha >> hb >> he) && (ls >> std::ws).eof()) {
    auto hex = [](const std::string& s, std::uint32_t& out) {
        const char* b = s.data();
        const char* e = s.data() + s.size();
        if (s.rfind("0x", 0) == 0) b += 2;
        const auto r = std::from_chars(b, e, out, 16);
        return r.ec == std::errc{} && r.ptr == e;
    };
    if (hex(ha, v.a) && hex(hb, v.b) && hex(he, v.expected)) { keep(v); continue; }
}
badLines.push_back(lineNo);

  opened=1  lines=8  vectors=5  malformed lines: 7 8
  -> 5 usable vectors out of 7 data lines. A run that proceeds on 5
     and reports "5 of 5 passed" has hidden two vectors.

Three details in that reader are the module's lessons applied.

Base 16 through std::from_chars, not >> std::hex. Chapter 23.3 §4 measured "0x1F" read with >> int giving 0 with a clean stream — every address in this file would have become zero and the run would have looked fine. The 0x prefix is stripped explicitly and the digits are parsed with an explicit base, and r.ptr == e is what rejects 0xZZZZZZZZ.

The whole-line check catches the short line. Line 7's three successful extractions would otherwise have left expected holding whatever the previous iteration left, or zero.

The malformed lines are retained by number, not counted. malformed lines: 7 8 is actionable; malformed: 2 is not. Chapter 23.7 §3 is the same rule and Chapter 22.9 §1 is where it comes from.

And then the decision that matters: five usable vectors out of seven data lines is not a run. A model that proceeds on the five and reports "5 of 5 passed" has produced a true statement and a misleading report. For a stimulus file the right policy is to reject the whole file, because the file is an input you own and a malformed line in it is a bug in whatever generated it — which is Chapter 23.3 §1's table, choosing "stop and reject" over "skip and count".

2. The Gate That Passes Because Nothing Ran

Now the same program with the stimulus file missing entirely:

  stimulus file absent: opened=0 vectors=0 failures=0
  a gate written as `if (failures == 0) PASS` -> *** PASS ***
  the same gate with a floor on the work done:
    `opened && vectors > 0 && failures == 0` -> *** FAIL (correctly) ***
  -> zero failures is only good news if something was checked

Zero vectors, zero failures, and the gate passes. This is Chapter 23.4 §1's unchecked open and Chapter 23.7 §2's lenient parser arriving at the same place from a third direction, and it is the most expensive defect in this module because it sits at the point where a human stops looking.

The failure chain is short and entirely plausible: a previous flow step fails to generate the stimulus, or writes it to a different directory, or the path contains a variable that expanded to nothing. The model runs, opens nothing, tests nothing, reports no failures, and exits 0. The regression dashboard turns green. Nobody investigates a green run.

The fix is to make the verdict depend on the work as well as the findings:

struct Verdict { bool pass; std::string why; };

Verdict decide(const Load& L, int failures)
{
    if (!L.opened)            return {false, "stimulus file could not be opened"};
    if (!L.badLines.empty())  return {false, "stimulus file has malformed lines"};
    if (L.v.empty())          return {false, "stimulus file contained no vectors"};
    if (failures != 0)        return {false, std::to_string(failures) + " vector(s) failed"};
    return {true, std::to_string(L.v.size()) + " vector(s) passed"};
}

Four floor conditions before the interesting one, each returning a reason rather than a bool. The reason is what makes a red run diagnosable without re-running it, and it is the difference between "FAIL" and "stimulus file contained no vectors".

And the ordering matters. Checking failures != 0 first would report "0 vectors failed" for an empty run, which is true and useless. The floor conditions come first because they describe the premise, and a verdict on a false premise is not a verdict.

3. The Model, and the Vector That Separates Right from Wrong

The thing under test is a saturating 32-bit add, implemented twice:

// wrong: saturates to one less than the maximum
std::uint32_t satAddBuggy(std::uint32_t a, std::uint32_t b)
{
    const std::uint64_t s = (std::uint64_t)a + b;
    return (s > 0xFFFFFFFFull) ? 0xFFFFFFFEu : (std::uint32_t)s;
}

// right
std::uint32_t satAddFixed(std::uint32_t a, std::uint32_t b)
{
    const std::uint64_t s = (std::uint64_t)a + b;
    return (s > 0xFFFFFFFFull) ? 0xFFFFFFFFu : (std::uint32_t)s;
}

  buggy model:       5 vectors, 4 pass, 1 fail   [id 3 got 0xFFFFFFFE want 0xFFFFFFFF]
  fixed model:       5 vectors, 5 pass, 0 fail
  -> the only vector that separates them is the one AT the boundary (id 3)

Four of the five vectors pass against the broken model. Vectors 1, 2, 4 and 5 — a small add, a signed-boundary add, an add that reaches exactly the maximum without overflowing, and a middling add — are all correct in both implementations, so they discriminate nothing. Only vector 3, which actually overflows, tells the two apart.

That ratio is the normal case, not a contrived one, and it is why a vector file needs designing rather than generating. Chapter 21.2 §3 measured the same thing from the assertion side: three static_asserts at comfortable widths found nothing and the fourth, at the boundary, found real undefined behaviour in this curriculum's own example. Assert and stimulate the boundaries: zero, one, the maximum, the maximum plus one, and whatever value makes the branch you are worried about actually execute.

A practical corollary for the results file: a run reporting "4 of 5 passed" is far more informative than "80% pass rate", because the identity of the failing vector is the whole finding. §4's format puts the id, the value obtained and the value expected on the failing row for exactly that reason.

4. The Results File

Here is the output, which is the chapter's deliverable:

    # vlsi-mentor model results  schema=1
    id,got,expected,status
    1,0x00000003,0x00000003,PASS
    2,0x80000000,0x80000000,PASS
    3,0xFFFFFFFE,0xFFFFFFFF,FAIL
    4,0xFFFFFFFF,0xFFFFFFFF,PASS
    5,0x00000014,0x00000014,PASS
    # summary vectors=5 pass=4 fail=1 elapsed_us=1234.5678901234567

Six design decisions, each of which is there because of something measured earlier in this module.

A schema line with a version. Chapter 23.6 §5's five drift scenarios are the general problem, and a consumer that checks schema=1 can refuse a file it was not written for instead of misreading it. This is the one line that makes the format maintainable.

A header row naming the fields. A consumer can find a column by name rather than by position, which means adding a column later does not break it.

Fixed-width uppercase hex for values. Chapter 23.2 §4 — the padding is what makes a text sort an address sort and a diff align, and it is why 0x00000003 is better than 0x3.

PASS and FAIL as literal words. Not 1 and 0, and not the exit status. A human skimming the file and a grep -c FAIL both work.

Comments marked with #. So the data rows are exactly the lines that are not comments and not the header, which is a rule a consumer can implement in one line.

A summary line that duplicates information already in the rows. This looks like redundancy and is the most useful line in the file — §5 is why.

The timing field is written at full precision, per Chapter 23.2 §5:

o << "# summary vectors=" << r.size() << " pass=" << pass << " fail=" << fail
  << " elapsed_us=" << std::setprecision(std::numeric_limits<double>::max_digits10)
  << elapsedUs << "\n";

Without max_digits10 the elapsed time would print as 1234.57 and comparing two runs' timings for equality would fail against the same run.

5. Reading Your Own Results Back

The test that the format works is that your own reader agrees with it:

  data rows 5 (pass 4, fail 1)
  summary says vectors=5 pass=4 fail=1
  rows agree with the summary: YES
  elapsed_us round-trip: wrote 1234.5678901234567, read 1234.5678901234567 -> EXACT

The rows and the summary are computed from different things and must agree. The rows are counted by the reader; the summary was written by the producer. A mismatch means one of a small set of specific faults: the writer was interrupted between the rows and the summary, two runs' output got concatenated, a row was dropped by a filter in between, or the summary was computed from a different collection than the rows were written from.

This is Chapter 23.7 §5's balance check applied within one file, and the reason to deliberately include redundant data in a machine-readable format: a file whose internal numbers cannot disagree is a file you can trust a little. Chapter 20.10 computed one important total by two independent routes for the same reason, and Chapter 22.9 §4's ring buffer is the same instinct about diagnostics.

The elapsed_us row is the Chapter 23.2 §5 round trip confirmed end to end: written at max_digits10 and read back byte-exact, so a timing regression check can compare numbers rather than tolerate them.

One further check worth adding for a file that may be truncated: compare the number of data rows against the summary's vectors field before believing either. A results file cut off mid-write has rows and no summary, which the agreement check catches as a missing summary rather than as agreement.

6. What the Process Returns

The exit status is the only part of this a flow script reads without parsing anything, so it has to be right:

  opened=1 vectors=5 malformed=2 failures=1
  exit 0  only if: the file opened, every line parsed, at least one
  vector ran, and no vector failed.
  here: conditions NOT met -> this run must exit NON-ZERO
  (and the results file exists either way: yes)

Two rules, and the second is less obvious.

Exit 0 means every floor condition held and nothing failed. The four conditions from §2, in that order, so the status agrees with the verdict in the file rather than being computed separately — otherwise you have two sources of truth and they will eventually disagree.

The results file must be written whether the run passed or failed. A tool that writes its results only on success is useless for debugging, because the failing run is the one whose output you need. Chapter 23.4 §6's write-to-temp-then-rename is how you get a complete file even when the run ends badly, and it also means a consumer never sees a half-written file.

The complementary rule, from Chapter 23.7 §7: a flow step must distinguish failed from did not run. Both are non-zero exit statuses and they need different human responses, which is why the verdict carries a reason string.

7. Making the Model Testable on Both Sides

The whole structure follows Chapter 23.5 §6: keep the file handling thin and put everything else behind the stream abstraction.

// pure logic -- no files, no streams
std::vector<Result> runVectors(const std::vector<Vector>&,
                               std::uint32_t (*model)(std::uint32_t, std::uint32_t));

// stream-level -- testable with a string literal
Load        readVectors(std::istream&);
void        writeResults(std::ostream&, const std::vector<Result>&, double elapsedUs);

// file-level -- four lines, and where Chapter 23.4 §1's defect would live
int main(int argc, char** argv);

The payoff is that the interesting tests need no filesystem at all:

// a malformed stimulus must be rejected, not silently reduced
std::istringstream bad("1 0x1 0xZZ 0x2\n");
assert(!readVectors(bad).badLines.empty());

// an empty stimulus must not pass
std::istringstream empty("# header only\n");
assert(!decide(readVectors(empty), 0).pass);

// the rows and the summary must agree
std::ostringstream out;
writeResults(out, results, 1.0);
assert(summaryAgreesWithRows(out.str()));

The second of those is the test that would have caught §2's defect, and it is a negative test — it asserts that something fails. A suite containing only well-formed stimulus files proves the model works and proves nothing about the gate, which is Chapter 22.8's point and the reason the defect survives in real flows: everyone tests the happy path, and the happy path is where these four defects are invisible by construction.

8. Stimulus Formats Worth Knowing

The flat text format in §1 is right for a small, hand-maintained vector set. Three alternatives come up:

• CSV with a header row. Good when the vector set is generated by a script or a spreadsheet, and the header lets you add a column without breaking consumers. Chapter 23.3 §7's warning applies: getline-splitting is not an RFC 4180 reader, and its failure mode is misaligned columns rather than an error.
• One vector per line with named fields — id=3 a=0xFFFFFFFF b=0x1 exp=0xFFFFFFFF. More verbose, order-independent, and self-describing, which makes it the most robust against format drift. Worth the characters when the file is long-lived.
• Binary, per Chapter 23.6, when the vector count is in the millions. Then the header with a magic number, a version and a record count stops being optional.

Whichever you choose, two properties are not negotiable. The file must say what it is — a schema or version line — and a reader must reject what it does not understand rather than skipping it. Everything else is convenience.

9. The Table

10. What to Actually Do

• Reject a stimulus file with any malformed line, and name the line numbers. Five usable vectors out of seven data lines is not a run.
• Parse hex with std::from_chars and an explicit base, checking r.ec and r.ptr == e. >> std::hex on "0x1F" gives 0 and reports success.
• Put a floor on the work in every gate — opened, fully parsed, at least one vector — and check it before the failure count. Measured: without it, a missing stimulus file passes.
• Make the floor as strong as the flow allows. vectors == expectedCount catches a truncated file that vectors > 0 passes.
• Return a reason, not a bool. "stimulus file contained no vectors" is diagnosable without re-running.
• Design vectors at the boundaries. Measured: 1 of 5 separated a correct saturating add from a wrong one.
• Put the failing vector's id, got and expected in the file. The identity of the failure is the finding.
• Give the results file a schema line, a header row, fixed-width hex, literal PASS/FAIL, and a summary that duplicates the row counts.
• Write timings at max_digits10 so two runs can be compared exactly.
• Write the results file on failure too, via write-to-temp-then-rename.
• Keep the file handling to four lines and put the rest behind std::istream& / std::ostream&.
• Write the negative tests — a malformed stimulus must be rejected, an empty one must not pass.

11. Mistakes Worth Making Once

if (failures == 0) pass();. Measured: passes a run whose stimulus file does not exist.

Checking the failure count before the floor conditions. "0 vectors failed" is true, passes, and means nothing.

Proceeding on the lines that parsed. "5 of 5 passed" is a true and misleading sentence.

>> std::hex for a vector file. Every value becomes 0 and the run looks clean.

Vectors only at comfortable values. Four of five passed the broken model.

A pass rate instead of the failing ids. 80% is not a finding; id 3 got 0xFFFFFFFE want 0xFFFFFFFF is.

Writing the results file only on success. You have discarded the output of the run you need.

A summary computed from a different collection than the rows. The agreement check exists to catch this, so do not defeat it by computing both from one variable at the last moment.

Timings at the default precision. The comparison fails against the same run.

No schema line. Chapter 23.6 §5's drift, in text.

Only testing with good stimulus. Every defect in this chapter is invisible on the happy path.

12. Interview Relevance

The frozen curriculum marks this chapter Advanced and High, and it is the one most directly about the job: "how would you make your reference model part of the regression flow?"

The answer has a structure. Stimulus in through a validated reader that rejects a file it does not fully understand; the model as pure logic with no I/O; results out in a versioned, machine-readable format with a cross-checkable summary; an exit status derived from the same verdict that went into the file; and the results file written whether the run passed or not.

The part that distinguishes a strong answer is the gate. Saying that a gate must assert a floor on the work done — the file opened, every line parsed, at least one vector ran — before it looks at the failure count, and being able to say that without it a missing stimulus file produces zero vectors, zero failures and a green run, shows you have seen a flow report success for doing nothing. Adding that the stronger floor is vectors == expectedCount rather than vectors > 0 shows you have thought about the truncated-file case too.

If vector design comes up, the measurement is a good one to have: one of five vectors separated a correct saturating add from a wrong one, and it was the one at the boundary.

13. What You Should Be Able to Say Now

• That a stimulus file with malformed lines should be rejected, with the line numbers named, rather than reduced to the lines that parsed.
• That hex must be parsed with an explicit base and a boundary check, because >> std::hex on "0x1F" gives 0 and reports success.
• That a gate written as zero-failures-means-pass passed a run with no stimulus file — 0 vectors, 0 failures — and that the fix is a floor on the work, checked first.
• Why the floor is the fix rather than more checking: the absence of a finding and the absence of a search produce identical output, in all four places this module measured it.
• That a verdict should carry a reason, so a red run is diagnosable without re-running it.
• That 1 of 5 vectors separated a correct saturating add from a broken one, and that it was the boundary vector — the same lesson as Chapter 21.2 §3's fourth static_assert.
• What goes in a machine-readable results file and why each part is there, including the deliberately redundant summary line.
• That timings written at max_digits10 round-trip exactly, and that the default precision does not.
• That the results file must exist after a failing run, and that a negative test is what catches the empty-run gate.

14. Exercises

Exercise 1 — the empty run. Build the model and gate with if (failures == 0) pass(), run it with the stimulus file renamed, and watch it pass. Then add the floor conditions and confirm it fails with a reason. Then truncate the stimulus to its header only and check that vectors > 0 catches it.

Exercise 2 — the boundary vector. Run the five vectors against both models and confirm only id 3 separates them. Then write three more vectors that also separate them, and three that cannot, and say what distinguishes the two groups.

Exercise 3 — the hex trap. Read the stimulus with >> std::hex instead of from_chars and record what the vectors become. Confirm the run reports a clean parse.

Exercise 4 — the summary check. Write the results file, then hand-edit one data row's status from PASS to FAIL and run the reader. Confirm the rows and summary disagree. Then delete the summary line and confirm the reader reports a missing summary rather than agreement.

Exercise 5 — the negative tests. Write the three tests from §7 against std::istringstream and std::ostringstream. Then delete the floor conditions from decide and confirm exactly one test fails.

15. Summary

• A model that reads a stimulus file and writes a results file is a flow component; one that prints to the console is a program somebody has to watch. The difference is almost entirely I/O.
• Reject a stimulus file you did not fully understand. Measured: 8 lines gave 5 usable vectors and 2 malformed lines, and proceeding on the 5 produces the true, misleading sentence "5 of 5 passed".
• Parse hex with from_chars and an explicit base. >> std::hex on "0x1F" yields 0 with a clean stream, which would have turned every vector into zero.
• A gate written as zero-failures-means-pass passed a run with no stimulus file — opened=0 vectors=0 failures=0. The fix is a floor on the work done — opened, every line parsed, at least one vector ran — checked before the failure count, because "0 vectors failed" is true and useless.
• This is the module's recurring defect, measured four ways: an unchecked open (0 records, exit 0), a lenient parser (FATAL=0 on a log with a fatal), a wrong binary layout (numbers instead of an error), and this gate. In each, the absence of a finding and the absence of a search produce the same output, which is why checking harder never fixes it.
• Design vectors at boundaries. Only 1 of 5 separated a correct saturating add from one that saturates to 0xFFFFFFFE.
• A results file needs a schema line, a header row, fixed-width hex, literal PASS/FAIL, and a summary that duplicates the row counts — the redundancy is what proves the rows and the totals came from one run.
• Write timings at max_digits10: measured, 1234.5678901234567 round-trips exactly, and the default precision does not.
• The results file must exist after a failing run, written atomically, because that is the run you need to debug.
• Write the negative tests. A suite of well-formed inputs exercises none of this.

This closes Module 23. Streams turned out to be one model — a sequence of characters with four state bits — and every chapter was a different way of being honest about what that model does and does not tell you: that a failed read consumes nothing, that a successful read is not a validated field, that written is not saved, that bytes do not describe themselves, and that a clean report is only evidence if something was examined.