Parsing Simulation Logs and Reports

Part I · Module 23 · Chapter 23.7 · Parsing Simulation Logs and Reports

1. Two Parsers, One Log, Opposite Verdicts

Here is a realistic simulator log. It has a header comment, two message formats for events — because the tool's UVM layer prints differently from its testbench layer — and a self-reported summary at the end:

# sim v2.1  seed=4821  design=soc_top
[           0] NOTE  tb      reset asserted
[         120] NOTE  tb      reset released
[     4823991] WARN  axi0    latency 412 cycles exceeds budget 200
[     4823994] ERROR axi0    addr 0x4000A010 resp=SLVERR master=2
[     4824010] ERROR lsu     addr 0x400000C8 resp=DECERR master=0
UVM_ERROR @ 4824100: uvm_test_top.env.sb [SB] mismatch exp=0x12 got=0x34
[     4824200] WARN  dma     descriptor ring wrapped
UVM_FATAL @ 4824300: uvm_test_top.env.sb [SB] too many mismatches
[     4824400] NOTE  tb      test finished
Simulation complete: 3 errors, 1 fatal

The first parser handles the bracket format and skips anything else — which is what almost everybody writes first:

while (std::getline(in, line)) {
    if (line.empty() || line[0] == '#') continue;
    if (line[0] != '[') { ++skipped; continue; }      // <-- the defect
    ...
}

The second insists on classifying every line, and records the line numbers it cannot:

if (line[0] == '[')              { parseBracket(line); }
else if (line.rfind("UVM_", 0) == 0) { parseUvm(line); }
else if (line.rfind("Simulation complete:", 0) == 0) { /* known, ignored */ }
else                             { unknown.push_back(lineno); }

  lenient: 7 events, 3 lines skipped SILENTLY
           ERROR=2  FATAL=0
  strict : 9 events, 0 line(s) it could not classify
           ERROR=2  UVM_ERROR=1  UVM_FATAL=1
  the log's own last line says: "3 errors, 1 fatal"
  -> the lenient parser MISSED the fatal and under-counted errors,
     and said nothing about it.

The lenient parser reports zero fatals for a log containing a UVM_FATAL. It also reports two errors where the tool itself says three. It did not fail, did not warn, and produced a complete-looking report.

The three skipped lines were the two UVM_ events and the summary. Nothing in the parser's output distinguishes "there were no fatals" from "I did not look at the lines where fatals appear".

2. The Gate Each Parser Produces

A parser's output is usually not read by a person. It feeds a pass/fail decision in a regression flow:

  lenient gate: fatal count 0 -> *** PASS ***
  strict  gate: fatal count 1, unclassified 0 -> *** FAIL ***
  -> the run had a UVM_FATAL. One of these gates is a liability.

The lenient gate passes a run that contained a fatal error. That is the whole chapter in one line. Every property that makes the lenient parser pleasant — it never crashes, it never complains, it handles whatever you throw at it — is what makes this outcome possible.

Notice the asymmetry in the consequences. A parser that is too strict fails a good run, which is annoying, visible within minutes, and fixed by adding a line format. A parser that is too lenient passes a bad run, which is invisible, may stay invisible for months, and is discovered when the silicon does not work. The two errors are not comparable, so the default must be strict.

The strict gate above also has a second condition, and it is the more important one: unclassified == 0. Counting fatals correctly today does not help if the tool renames UVM_FATAL tomorrow. The unclassified count is what notices that.

3. Format Drift, and Why the Lenient Parser Cannot Report It

Now add two lines of the kind a tool upgrade actually produces — one with a typo'd severity, one that lost its brackets:

[     4824500] ERORR axi0    addr 0x40000010 resp=SLVERR master=1
4824600 ERROR axi0 no brackets at all

  added one typo'd severity and one format-drifted line
  lenient: 8 events, 4 skipped   ERROR=2 (unchanged)
  strict : 10 events, unclassified lines: 13
  -> only the strict parser can tell you the log format changed

The lenient parser's error count did not move, and its skip count went from 3 to 4 — a number it never prints. From the outside, nothing changed. The strict parser names line 13.

Then there is the subtler half:

  NOTE: "ERORR" parses FINE as a severity token -- a free-text severity
        field cannot be validated by the parse. Check it against a SET.
        unknown severity token: "ERORR"
        severities outside the known set: 1

ERORR is a perfectly good whitespace-delimited token. Both parsers extract it successfully as the severity field, because the parse only asks whether a word is there. The line is structurally valid and semantically meaningless, and an error that should have been counted was not — by either parser, until the severity is checked against a known set:

constexpr std::string_view KNOWN[] = { "NOTE", "WARN", "ERROR", "FATAL" };

bool knownSeverity(std::string_view s)
{
    if (s.starts_with("UVM_")) return true;
    return std::ranges::find(KNOWN, s) != std::ranges::end(KNOWN);
}

This is Chapter 23.3 §4's lesson in a new costume. There, operator>> reported success for "12abc" because it read a valid prefix. Here the extraction reports success for "ERORR" because it read a valid token. In both cases the stream answered the question it was asked — is something there — and the question that mattered was whether the something was one of the values this field is allowed to hold. A structural parse cannot validate an enumerated field; only a membership test can.

4. std::regex Against Hand Parsing

Regular expressions are the obvious tool for this and are worth measuring before you reach for them. The same twenty thousand bracket-format lines, parsed both ways:

// regex
const std::regex re(R"(^\[\s*(\d+)\]\s+(\w+)\s+(\w+)\s+(.*)$)");
std::smatch m;
if (std::regex_match(line, m, re)) total += std::stoll(m[1].str());

// hand
if (line.size() < 2 || line[0] != '[') continue;
const std::size_t rb = line.find(']');
if (rb == std::string::npos) continue;
std::size_t s = 1;
while (s < rb && line[s] == ' ') ++s;
long long cyc = 0;
if (std::from_chars(line.data() + s, line.data() + rb, cyc).ec == std::errc{})
    total += cyc;

Two orders of magnitude. Each figure is already a minimum of fifteen repetitions within a run, reported as a median over seven runs. A single run of this experiment reported 155× and 7,442 ns per line — the per-line figure being well below everything the seven runs produced — which is why the table carries ranges.

std::regex is slow for specific, known reasons: it compiles to an interpreted state machine rather than native code, it allocates for the match results, and libstdc++ and libc++ have both been criticised for this for a decade. Twelve microseconds per line means a hundred-thousand-line log takes over a second in the regex alone.

That said, speed is usually not the deciding factor:

• For a log you parse once in a script-like tool, use the regex. A second is nothing and the regex is far easier to read and to change.
• For a parser in a flow that processes thousands of logs, hand-parse the hot format and keep regexes for the rare ones.
• For correctness, neither is better. Both parsers in §1 can be written either way; the regex version of the lenient parser is just as capable of silently skipping three lines. The leniency is in the continue, not in the matching technique.

One real advantage of the regex is that the format is visible as a string, which makes drift easier to accommodate — and one real hazard is that .* makes a pattern match more than you intended, so anchoring with ^ and $ and avoiding greedy wildcards in the middle matters. Chapter 20's algorithm chapters cover std::ranges alternatives for the scanning parts.

5. Two Groupings That Must Agree

A cheap, powerful check on a parse: group the same events two independent ways and require the totals to match.

  by severity: ERROR=2 NOTE=3 UVM_ERROR=1 UVM_FATAL=1 WARN=2
  by unit    : axi0=2 dma=1 lsu=1 tb=3 uvm=2
  severity total 9, unit total 9, events 9 -> BALANCED

Every event has exactly one severity and exactly one unit, so both groupings must sum to the event count. If they do not, something was dropped — a line counted into no bucket, a field left empty and silently excluded, a map key that collided.

This costs four lines and catches a class of defect that is otherwise invisible, because a report with one missing row looks exactly like a report. It is the same device as Chapter 23.3 §6's whole-line check: not validating each item, but asserting that the parts sum to the whole. Chapter 23.8 §5 uses it again between the data rows and the summary line of a results file, and Chapter 20.10 computed one important total by two independent routes for the same reason.

The generalisation is worth stating: print the totals that must balance, and check them. A report whose internal numbers cannot disagree is a report you can trust a little; one whose numbers are each computed once and never cross-checked is a report you cannot.

6. Structuring a Parser That Will Outlive the Format

The tool's format will change. Design for that rather than against it:

struct Event { long long cycle; std::string sev, unit, text; };

class LogParser {
public:
    // one handler per known line shape, tried in order
    Parsed parse(std::istream& in);
private:
    static std::optional<Event> bracket(std::string_view);
    static std::optional<Event> uvm(std::string_view);
    static bool                 knownNoise(std::string_view);
};

Five properties make this survivable.

Each line shape is a separate function returning std::optional<Event>. Chapter 21.5 is the type: absence here means "this is not my shape", which the dispatcher distinguishes from an error.

Known-but-uninteresting lines are enumerated, not skipped. The Simulation complete: summary, a banner, a progress line — list them as knownNoise. The point is that the else branch at the bottom then means unknown, and only unknown.

Unknown lines are collected with their line numbers and surfaced. Not counted — named. Chapter 22.9 §1's rule about actionable diagnostics.

Enumerated fields are checked against a set, per §3, so ERORR is reported rather than absorbed.

The parser takes std::istream&, so every one of these behaviours is testable with a string literal, per Chapter 23.5 §6. The tests that matter here are the negative ones: a log with an unknown line must produce a non-empty unknown, and a log with a UVM_FATAL must fail the gate. A test suite that only feeds the parser well-formed logs proves nothing about the behaviour this chapter is about — which is Chapter 22.8's point that a check needs a test proving it can fail.

7. What the Tool Tells You About Itself

The log in §1 ends with Simulation complete: 3 errors, 1 fatal, and that line is a gift: it is the tool's own count, so you can compare it against yours.

if (summary && (summary->errors != counted.errors || summary->fatals != counted.fatals))
    report("parser and tool disagree: tool says ...; we counted ...");

A disagreement means either the parser missed events or the tool counts differently from how you assumed — and either way you need to know before trusting the number. In §1 the lenient parser counted 2 errors and 0 fatals against the tool's 3 and 1; that comparison alone would have caught the defect, without anybody thinking about line formats.

Two cautions. The tool's own count may be defined differently — some tools count demoted errors, some count per-instance rather than per-message — so the first run of this comparison usually teaches you something about the tool rather than about your parser. And the summary line may be absent, because the run was killed; a missing summary is itself a finding, and treating "no summary line" as "nothing to compare" is how a timed-out run gets reported as clean.

Also check that the run finished. A log with no final summary, or no test finished, is an incomplete run. Chapter 23.8 §2 is the same floor condition.

8. The Table

9. What to Actually Do

• Make the unknown lines part of the return value, with their line numbers, and mark the function [[nodiscard]]. This is the fix; everything else is detail.
• Make the gate require unknown.empty(), not just a zero fatal count. Measured: the fatal count alone passed a failing run.
• Enumerate the lines you deliberately ignore, so the final else means only unknown.
• Check every enumerated field against a known set. ERORR is a valid token and an invalid severity.
• Compare your counts against the tool's own summary, and treat a missing summary as a finding rather than as nothing to compare.
• Assert a floor on the work done — the run finished, at least one event was parsed — before believing a clean verdict.
• Group the events two ways and check the totals balance. Four lines, and it catches a dropped row.
• Use a regex for a one-off tool and hand-parse the hot format in a flow. Measured 144×, but leniency is in the continue, not in the technique.
• Test with malformed logs. A suite of well-formed logs cannot exercise any of this.

10. Mistakes Worth Making Once

if (not my format) continue;. The defect of the chapter. It passed a run with a UVM_FATAL.

Counting skipped lines and not printing the count. The information existed and was discarded.

Reporting skipped: 4 instead of unknown line 13. A count tells the user something is wrong; a line number tells them where.

Trusting a severity you extracted. ERORR is a word, and the extraction succeeded.

One format when the log has two. The UVM layer and the testbench layer print differently, and the second format is where the fatals were.

A gate on the fatal count alone. It cannot notice that UVM_FATAL was renamed.

Treating a missing summary line as nothing to compare. That is how a killed run reports clean.

Only testing with good logs. Every behaviour in this chapter is about bad ones.

Reaching for std::regex in a per-log hot path without measuring. 12 microseconds a line.

11. Interview Relevance

The frozen curriculum marks this chapter Advanced and High, and it is a favourite because it is a real job rather than a language quiz. The question is usually "you need to parse simulator logs and decide pass or fail — how do you do it?"

The answer that stands out leads with the failure mode rather than with the parsing: a parser that skips lines it does not recognise will eventually report zero errors for a failing run, so the unknown lines have to be part of the result and the gate has to require that there are none. Then the supporting points: two message formats in one log because different layers print differently; enumerated fields checked against a set because ERORR parses as a valid token; a floor condition so that zero findings is distinguishable from zero searching; and a comparison against the tool's own summary count.

If performance comes up, std::regex at about 144× a hand parse is a good figure to have, together with the judgement that it is usually the right choice anyway and that leniency is a design decision rather than a consequence of the matching technique.

12. What You Should Be Able to Say Now

• That a lenient parser reported FATAL=0 for a log containing a UVM_FATAL, and that its gate passed, while a strict parser over the same log failed it.
• That the two errors are not symmetric: too strict fails a good run visibly, too lenient passes a bad run invisibly — so strict is the default.
• That two drifted lines left the lenient parser's error count unchanged, so there was no signal at all.
• That ERORR parses as a valid severity token, and that an enumerated field needs a membership test, not a structural parse — the same lesson as "12abc" in Chapter 23.3 §4.
• That the fix is in the interface: return the events and the unknown line numbers, and gate on both.
• That the absence of a finding and the absence of a search produce identical output, which is why a gate needs a floor on the work done.
• That std::regex_match costs about 144× a hand parse (~12,300 ns against ~88 ns per line), and that this is usually not the deciding factor.
• That two independent groupings of the same events must sum to the same total, and that this is a four-line check for a dropped row.

13. Exercises

Exercise 1 — the two verdicts. Write both parsers over §1's log and reproduce the two gates. Then delete the UVM_FATAL line and confirm both now pass — which is the point: on a clean log the parsers are indistinguishable.

Exercise 2 — drift. Add §3's two lines and confirm the lenient error count does not move. Then rename UVM_FATAL to UVM_FATAL_ERROR throughout and show which gate notices.

Exercise 3 — the membership test. Add the severity-set check to the strict parser and confirm it reports ERORR. Then find a severity the real tool emits that is not in your set, and decide whether to add it or to report it.

Exercise 4 — the balance. Group the parsed events by severity and by unit and assert the totals match. Then deliberately drop every event whose unit is empty and watch the assertion fire.

Exercise 5 — the floor. Make the gate require that the log contains a completion line and at least one parsed event. Run it on a truncated log and confirm it fails.

14. Summary

• Tool output has no schema and will change, so the question is not how to parse it but what to do with a line you did not expect.
• Measured, on one log: the lenient parser reported 7 events, 3 silently skipped lines, ERROR=2 and FATAL=0; the strict parser reported 9 events, 0 unclassified, and UVM_FATAL=1. The tool's own summary says 3 errors, 1 fatal. The lenient gate passed a run containing a fatal error.
• Too strict fails a good run visibly; too lenient passes a bad run invisibly. The errors are not comparable, so strict is the default.
• An error count alone is not a signal. Two drifted lines left it unchanged; only the unclassified-line count moved.
• A structural parse cannot validate an enumerated field. ERORR extracted successfully as a severity — check against a known set, exactly as "12abc" needed a boundary check in Chapter 23.3 §4.
• The real fix is the interface. Return the events and the unknown line numbers, mark it [[nodiscard]], and gate on unknown.empty() as well as the counts — because the absence of a finding and the absence of a search produce the same output.
• Group two independent ways and require the totals to balance — 9, 9 and 9 here. Four lines, and it catches a dropped row.
• std::regex_match costs about 144× a hand parse (124–176× over seven runs; ~12,300 ns against ~88 ns per line). Usually worth it for readability — and leniency lives in the continue, not in the regex.
• Test with malformed logs, because a suite of well-formed ones exercises none of this.

Chapter 23.8 closes the module by turning this discipline around: reading a stimulus file, running a model, and emitting results another tool can consume — where the same "zero findings because nothing ran" defect shows up as a gate that passes an empty run.