Designing a Result Record

Part II · Module 21 · Chapter 21.1 · Designing a Result Record

1. Two Different Records

Chapter 20.16's parser returns seventeen fields. That record is exactly right for what it is: everything one parser learned about one log, including things it learned about itself — whether its counts agreed with the simulator's, how many lines it did not recognise.

A result record is a different object with a different job. It is what gets written to a file, read by a script somebody else wrote, loaded into a dashboard, and compared against the same field from three months ago.

Nothing in that picture shares code. The parser does not know what the dashboard needs; the dashboard cannot ask the parser. The field list is the only thing holding them together, which is why it is worth designing before any of them is written.

2. Eight Fields

# every result carries: test, seed, status, runtime_s,
# errors, first_failure, log_path, run_dir

That is the canonical list, and each field answers a question somebody actually asks of a regression:

Note the phrasing of the comment: every result carries. That is a floor, not a ceiling. A project whose signoff is assertion-based adds assertions; one with a warning budget adds warnings; one running on a farm adds host and job_id. What the eight guarantee is that any consumer may rely on them being present, in every record, from every producer — which is the property that lets a dashboard be written against a sweep it has never seen.

3. What Was Dropped, and Why

Nine of Chapter 20.16's fields do not cross the boundary, for three distinct reasons:

PARSE_RECORD = {
    "log": "runs/smoke_1839274/sim.log", "test": "axi_smoke_test", "seed": 1839274,
    "errors": 2, "fatals": 0, "warnings": 1, "assertions": 0,
    "first_failure": "UVM_ERROR drv.sv(88) @ 141900: bad handshake",
    "first_failure_time": 141900,
    "context": ["a", "b", "UVM_ERROR ...", "c"], "context_marker": 2,
    "started": True, "finished": True, "unparsed": 4, "counts_agree": True,
    "status": "FAIL", "runtime_s": None,
}

RESULT_FIELDS = ["test", "seed", "status", "runtime_s",
                 "errors", "first_failure", "log_path", "run_dir"]

kept = [f for f in RESULT_FIELDS if f in PARSE_RECORD]
dropped = sorted(set(PARSE_RECORD) - set(RESULT_FIELDS))

print("parse record :", len(PARSE_RECORD), "fields")
print("result record:", len(RESULT_FIELDS), "fields")
print("carried over :", len(kept))
print("dropped      :", dropped)

parse record : 17 fields
result record: 8 fields
carried over : 6
dropped      : ['assertions', 'context', 'context_marker', 'counts_agree', 'fatals', 'finished', 'first_failure_time', 'log', 'started', 'unparsed', 'warnings']

Unbounded or structured. context is a list of lines and context_marker is an index into it. Section 5 shows what a list does to a CSV cell; the short version is that it cannot cross that boundary intact, and a field that survives in one format and not another is not a field in the record.

About the parser, not the result. counts_agree and unparsed describe how well the parse went. They belong in the parser's log or its exception, not in a table of test results — a dashboard column headed unparsed is a question nobody reading a regression wants answered.

Derivable from what is kept. started and finished were the inputs to status, and status already encodes them: NO_RUN means not started, NO_FINISH means started and not finished. Storing all three invites them to disagree after somebody edits one — the same redundancy Chapter 20.15 Section 4 removed from the parser's state.

fatals, warnings and assertions are the interesting case: they are none of the three. They are bounded, they are about the result, and they are not derivable. They are absent because the canonical eight are a floor, and a project that gates on them should add them — in which case they must be added everywhere, by every producer, or the guarantee in Section 2 is gone.

4. What Was Added

Two fields the parser could not have known:

def to_result(parse_record, run_dir):
    """The subset of a parse record that crosses a file boundary."""
    return {
        "test": parse_record["test"],
        "seed": parse_record["seed"],
        "status": parse_record["status"],
        "runtime_s": parse_record["runtime_s"],
        "errors": parse_record["errors"],
        "first_failure": parse_record["first_failure"],
        "log_path": parse_record["log"],
        "run_dir": run_dir,
    }


PARSE_RECORD = {
    "log": "runs/smoke_1839274/sim.log", "test": "axi_smoke_test", "seed": 1839274,
    "errors": 2, "first_failure": "UVM_ERROR drv.sv(88) @ 141900: bad handshake",
    "status": "FAIL", "runtime_s": None,
}

result = to_result(PARSE_RECORD, "runs/smoke_1839274")
for key in ["test", "seed", "status", "runtime_s", "errors", "log_path", "run_dir"]:
    print(f"  {key:<14} {result[key]!r}")

  test           'axi_smoke_test'
  seed           1839274
  status         'FAIL'
  runtime_s      None
  errors         2
  log_path       'runs/smoke_1839274/sim.log'
  run_dir        'runs/smoke_1839274'

run_dir is the sweep's knowledge. The parser was handed a log path and never knew what else sat beside it — the waveform, the coverage database, the command that produced it. A failing result whose record names only the log sends an engineer looking for the rest.

log_path is a rename of the parser's log, and the rename is deliberate: log in a table of results is ambiguous — is it the path, the contents, the logger? A field name that crosses a boundary is read by people who have not read the parser.

runtime_s is still None here, exactly as Chapter 20.16 left it. The _s suffix is doing work too: a bare runtime in a CSV column invites somebody to put minutes in it, and the first time that happens the dashboard is wrong by sixty with no error anywhere. Units belong in the field name.

5. First Test: a Field Must Survive the Format

import csv
import io

CONTEXT = ["UVM_INFO drv.sv(60) @ 141700: issuing write",
           "UVM_ERROR drv.sv(88) @ 141900: bad handshake",
           "UVM_INFO drv.sv(92) @ 142000: retrying"]

buffer = io.StringIO()
writer = csv.DictWriter(buffer, fieldnames=["test", "context"])
writer.writeheader()
writer.writerow({"test": "axi_smoke_test", "context": CONTEXT})

text = buffer.getvalue()
print(text)

back = next(csv.DictReader(io.StringIO(text)))
print("read back as:", type(back["context"]).__name__)
print("first 44     :", back["context"][:44])
print("is it a list?:", isinstance(back["context"], list))

test,context
axi_smoke_test,"['UVM_INFO drv.sv(60) @ 141700: issuing write', 'UVM_ERROR drv.sv(88) @ 141900: bad handshake', 'UVM_INFO drv.sv(92) @ 142000: retrying']"

read back as: str
first 44     : ['UVM_INFO drv.sv(60) @ 141700: issuing writ
is it a list?: False

The list became the Python repr of a list, as a string. Nothing is lost in the sense that the characters are all there; everything is lost in the sense that no tool outside Python can read it back, and inside Python the only way to recover it is eval, which is not an option on a file anyone can write to.

That is the first test a field has to pass: can this format represent it? A CSV cell holds one scalar. A list, a dict or a nested record does not fit, and forcing one in produces a cell that looks like data and is not.

JSON passes this test, which is Chapter 21.4's point and part of why both formats exist in a flow. The record designed here is deliberately CSV-compatible — every field a scalar — so that the same record works in both, and that is a constraint accepted on purpose rather than a limitation discovered later.

6. Second Test: a Field Must Survive the Round Trip

import csv
import io
import json

ROW = {"test": "axi_smoke_test", "seed": 1839274, "status": "FAIL",
       "runtime_s": None, "errors": 2, "first_failure": "bad, handshake",
       "log_path": "runs/a/sim.log", "run_dir": "runs/a", "finished": True}

FIELDS = list(ROW)

buffer = io.StringIO()
writer = csv.DictWriter(buffer, fieldnames=FIELDS)
writer.writeheader()
writer.writerow(ROW)
csv_back = next(csv.DictReader(io.StringIO(buffer.getvalue())))

json_back = json.loads(json.dumps(ROW))

print(f"{'field':<14} {'wrote':<22} {'csv read back':<22} same")
changed = 0
for key in FIELDS:
    same = ROW[key] == csv_back[key]
    changed += 0 if same else 1
    print(f"{key:<14} {repr(ROW[key]):<22} {repr(csv_back[key]):<22} {same}")

print()
print("csv  fields changed:", changed, "of", len(FIELDS))
print("json identical     :", json_back == ROW)

field          wrote                  csv read back          same
test           'axi_smoke_test'       'axi_smoke_test'       True
seed           1839274                '1839274'              False
status         'FAIL'                 'FAIL'                 True
runtime_s      None                   ''                     False
errors         2                      '2'                    False
first_failure  'bad, handshake'       'bad, handshake'       True
log_path       'runs/a/sim.log'       'runs/a/sim.log'       True
run_dir        'runs/a'               'runs/a'               True
finished       True                   'True'                 False

csv  fields changed: 4 of 9
json identical     : True

Four of nine fields come back as a different value, and none of them raised anything.

seed and errors are integers that return as strings — Chapter 19.8 Section 3's problem, now arriving from a file instead of a regex. runtime_s was None and returns as ''. And finished was True and returns as the string 'True', which is the worst of the four, because bool('False') is also True and a boolean written to a CSV and read back can never be false again unless somebody converts it deliberately.

JSON returns all nine unchanged, including the None and the True.

Two conclusions, and they shape the rest of this module. CSV is a text format with no types, so every consumer must convert — which is why Chapter 21.2's annotation is float(row["runtime_s"] or 0) and not row["runtime_s"]. And a boolean does not belong in a result record that may be written to CSV; the eight canonical fields contain none, and status carries what two or three booleans would have.

first_failure passing is worth noticing too: it contains a comma, and the writer quoted it correctly. Chapter 21.3 is about the other things the writer has to get right.

7. Who Fills Each Field

A record filled by three different producers needs each field to have exactly one owner:

def new_result():
    """Every key present, every value None, before anybody fills anything."""
    return {"test": None, "seed": None, "status": None, "runtime_s": None,
            "errors": None, "first_failure": None, "log_path": None, "run_dir": None}


record = new_result()
print("empty      :", sum(1 for v in record.values() if v is None), "of", len(record), "unfilled")

record.update({"test": "axi_smoke_test", "seed": 1839274, "status": "FAIL",
               "errors": 2, "first_failure": "bad handshake",
               "log_path": "runs/a/sim.log"})
print("after parse:", sum(1 for v in record.values() if v is None), "unfilled")

record["run_dir"] = "runs/a"
record["runtime_s"] = 143.2
print("after sweep:", sum(1 for v in record.values() if v is None), "unfilled")
print("complete   :", all(v is not None for v in record.values()))

empty      : 8 of 8 unfilled
after parse: 2 unfilled
after sweep: 0 unfilled
complete   : True

new_result creates every key up front, which is Chapter 20.6's always-present rule applied one level up — and here it buys something extra: an unfilled field is visibly None rather than absent, so sum(1 for v in record.values() if v is None) is a real completeness check that a missing key could not give you.

That check is worth running before a record is written. A sweep that forgot to set run_dir produces a table with an empty column, which looks like missing data about the tests rather than a bug in the sweep.

8. The Record Is a Contract

Everything in this chapter is the same argument Chapter 20.6 made about a parser, moved up a level: the field list is what consumers rely on without reading the producer.

Three properties make it usable, and all three are decisions rather than consequences:

Every field, every record. A consumer that has to test if "runtime_s" in row for every field is a consumer with no contract.

Every field a scalar. It costs the context block, and it buys a record that is the same shape in CSV and in JSON. Chapter 21.4 shows what JSON can do with the richer shape when a flow genuinely needs it.

Every field named for what it is, with units. runtime_s, not runtime. log_path, not log.

What it buys is the thing Module 20 could not do: a comparison. Two records from two nights, with the same fields and the same meanings, can be diffed — which is Chapter 30.3's new-and-fixed-failures report, and it is impossible if last night's sweep wrote runtime in minutes.

9. Predict, Run, Observe

import csv
import io

ROW = {"a": 1, "b": None, "c": True, "d": "x,y", "e": ""}
buffer = io.StringIO()
writer = csv.DictWriter(buffer, fieldnames=list(ROW))
writer.writeheader()
writer.writerow(ROW)
back = next(csv.DictReader(io.StringIO(buffer.getvalue())))

# A
print("A:", repr(back["a"]), repr(back["b"]))

# B
print("B:", repr(back["c"]), bool(back["c"]))

# C
print("C:", bool("False"), bool(""))

# D
print("D:", repr(back["d"]))

# E
print("E:", back["b"] == back["e"])

# F
print("F:", int(back["a"]) + 1)

# G
print("G:", sorted(back) == sorted(ROW))

The verified output:

A: '1' ''
B: 'True' True
C: True False
D: 'x,y'
E: True
F: 2
G: True

A — the integer came back as a string, and the None came back as ''. The reader never produces None: a missing value and an empty one are the same empty field.

B — the boolean is the string 'True', which is truthy.

C — and so is the string 'False'. A boolean survives a CSV round trip only as a value that is always true.

D — the comma inside the value was quoted on write and unquoted on read, correctly. The reader is not a split(",").

E — True, because the None and the empty string both wrote an empty field. They are indistinguishable on the way back, which is why a record should not rely on the difference.

F — conversion is the consumer's job, every time.

G — the field names survive, which is the one thing CSV does keep.

10. Exercises

Exercise 1 — two records. List the fields your own parser produces and the fields a report needs. Say how many are in both.

Exercise 2 — three reasons. For every field you drop, say whether it is unbounded, about the parser, or derivable. Flag any that is none of the three.

Exercise 3 — the floor. Name one field your project would add to the canonical eight and say what would break if only half your producers added it.

Exercise 4 — what the parser cannot know. Identify two fields in your flow that no parser can fill, and name the component that can.

Exercise 5 — the rename. Find a field whose name is clear inside the producer and ambiguous in a table. Rename it and say what the new name tells a stranger.

Exercise 6 — units in the name. Find a numeric field with no unit in its name. Say what a consumer could reasonably assume and what that would cost.

Exercise 7 — the format test. Write a list-valued field to a CSV and read it back. Report its type and say what it would take to recover the list.

Exercise 8 — the round-trip table. Build Section 6's table for your own record. Report how many fields changed.

Exercise 9 — the boolean. Write False to a CSV, read it back, and evaluate its truth. Say what you would store instead.

Exercise 10 — completeness. Write new_result and count unfilled fields after each producer runs. Say which field is unowned in your flow.

Exercise 11 — the diff. Take two records from two runs and write the comparison a nightly report would do. Say what would break if one of them had been written by a different version of the sweep.

11. Summary

• The record that crosses a file boundary is not the record in memory. Chapter 20.16's parser produces seventeen fields; the result record carries eight.
• Nothing in the chain shares code, so the field list is the only thing holding producers and consumers together.
• The eight canonical fields are a floor, not a ceiling — what every consumer may rely on being present in every record.
• Three reasons to drop a field: unbounded or structured, about the parser rather than the result, or derivable from what is kept.
• status already encodes started and finished, and storing all three invites them to disagree.
• run_dir is the sweep's knowledge and runtime_s is the runner's — the parser cannot fill either.
• Rename for the reader: log becomes log_path, because a field name crossing a boundary is read by people who never saw the producer.
• Units belong in the field name. runtime_s, not runtime.
• First test: can the format represent it? A list in a CSV cell becomes a Python repr that nothing outside Python can read back.
• Second test: does it survive the round trip? Four of nine fields come back from a CSV as a different value, with nothing raised.
• CSV has no types — integers return as strings and None returns as '', which is why every consumer converts.
• A boolean survives a CSV round trip only as a value that is always true, because bool('False') is True. Keep booleans out of the record.
• Build the record with every key present and None, so an unfilled field is visible and completeness is checkable.
• Give every field exactly one owner. An unowned field is a blank column that looks like missing data rather than a bug.
• A contract is what makes two nights comparable, and it is impossible if one sweep wrote minutes where the other wrote seconds.

The next chapter reads a table of these records back — and deals with the header row, the blank line and the field that is sometimes empty.