Python Part II · Module 30 · Engineering Report Generation
CSV and Machine-Readable Summaries
Chapters 30.1 to 30.3 built a report for a person. This one is for a program, and the requirements invert: a human reader tolerates a shifted column and infers what a blank cell means, while a consumer script does neither — it compares a seed against an integer, indexes a column by position, and silently produces nothing when either assumption fails. Every defect in this chapter is a file that opens correctly in a spreadsheet and gives a wrong answer to the script that was written to read it.
Intermediate19 min readPythonPart IIAutomationReportingCSV
Part II · Module 30 · Chapter 30.4 · CSV and Machine-Readable Summaries
1. A Different Reader, Different Rules
The report of Chapter 30.2 and the file in this chapter describe the same night and share almost no requirements:
CONTRASTS = [
("unit of a row", "a bucket", "a job"),
("size", "one screen", "as long as the regression"),
("aggregation", "essential", "forbidden — the consumer aggregates"),
("column order", "whatever reads well", "fixed, forever"),
("a blank cell", "obvious from context", "ambiguous and must be defined"),
("run identity", "the page title", "a column in every row"),
]
print(f"{'concern':<16} {'human report':<26} machine summary")
for concern, human, machine in CONTRASTS:
print(f"{concern:<16} {human:<26} {machine}")
print()
print("requirements shared by both:", 0)
print("so they are two files, not one file with two renderings.")concern human report machine summary
unit of a row a bucket a job
size one screen as long as the regression
aggregation essential forbidden — the consumer aggregates
column order whatever reads well fixed, forever
a blank cell obvious from context ambiguous and must be defined
run identity the page title a column in every row
requirements shared by both: 0
so they are two files, not one file with two renderings.Nothing is shared, and the useful consequence is that trying to serve both from one artefact produces a file that is too long to read and too aggregated to query. Chapter 30.2's budget and this chapter's one-row-per-job rule are directly contradictory, and both are right.
2. The Annotation
write_csv(run_id, results, out_dir / "regression.csv")Three arguments, and the first one is the whole point of the chapter:
from pathlib import Path
def write_csv(run_id, results, path):
"""The signature the annotation specifies."""
return {"run_id": run_id, "rows": len(results), "path": str(path)}
info = write_csv("2026-02-18_nightly_r3",
[{"test": "axi_burst", "seed": 7731, "status": "FAIL"}],
Path("out") / "regression.csv")
for k, v in info.items():
print(f" {k:<8} {v}")
print()
print("run_id is a PARAMETER, which means it was meant to reach the file.")
print("the question §6 asks:", "did it reach the rows, or only the path?") run_id 2026-02-18_nightly_r3
rows 1
path out/regression.csv
run_id is a PARAMETER, which means it was meant to reach the file.
the question §6 asks: did it reach the rows, or only the path?The signature is right. Sections 3 to 7 are about the body.
3. ",".join(...) Corrupts a Real Record
The first instinct for writing a CSV is string concatenation, and a regression record is exactly the shape that breaks it:
import csv
import io
RECORD = {
"test": "axi_burst",
"seed": 7731,
"status": "FAIL",
"first": "UVM_ERROR drv.sv(88), bad handshake\nretrying",
}
FIELDS = ["test", "seed", "status", "first"]
naive = ",".join(str(RECORD[f]) for f in FIELDS)
print("naive join:")
print(repr(naive))
print(" physical lines:", len(naive.split("\n")))
print(" commas in the line:", naive.count(","))
print(" columns a reader will see on line 1:", len(naive.split("\n")[0].split(",")))naive join:
'axi_burst,7731,FAIL,UVM_ERROR drv.sv(88), bad handshake\nretrying'
physical lines: 2
commas in the line: 4
columns a reader will see on line 1: 5One record became two lines with five columns on the first, from a four-column schema. A consumer reading line by line and splitting on commas gets one malformed row and one orphan.
The csv module gets it right, and the way it gets it right is worth seeing:
import csv
import io
RECORD = {
"test": "axi_burst",
"seed": 7731,
"status": "FAIL",
"first": "UVM_ERROR drv.sv(88), bad handshake\nretrying",
}
FIELDS = ["test", "seed", "status", "first"]
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=FIELDS)
w.writeheader()
w.writerow(RECORD)
text = buf.getvalue()
print("what was written:")
print(repr(text))
print()
print("physical lines :", len(text.rstrip("\r\n").split("\n")))
rows = list(csv.DictReader(io.StringIO(text)))
print("rows read back :", len(rows))
print("field intact :", rows[0]["first"] == RECORD["first"])what was written:
'test,seed,status,first\r\naxi_burst,7731,FAIL,"UVM_ERROR drv.sv(88), bad handshake\nretrying"\r\n'
physical lines : 3
rows read back : 1
field intact : TrueThree physical lines, one row, the field byte-identical. The comma is handled by quoting the field and the newline is simply allowed to sit inside the quotes — which is legal CSV and which no line-oriented reader can parse.
That last point is a constraint on consumers, not a bug: anything reading this file must use a CSV parser, because for line in f is wrong on a file that may contain a quoted newline. It is also the reason Chapter 30.5's Markdown table must collapse newlines instead of quoting them — a Markdown row has no quoting mechanism at all.
4. Everything Reads Back as a String
The round-trip above preserved the text perfectly, and that is the trap:
import csv
import io
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=["test", "seed", "status", "errors"])
w.writeheader()
w.writerow({"test": "axi_burst", "seed": 7731, "status": "FAIL", "errors": 3})
row = list(csv.DictReader(io.StringIO(buf.getvalue())))[0]
print("written : seed =", repr(7731), " errors =", repr(3))
print("read back : seed =", repr(row["seed"]), " errors =", repr(row["errors"]))
print()
print("row['seed'] == 7731 :", row["seed"] == 7731)
print("row['errors'] > 0 :", row["errors"] > "0")
print("int(row['seed']) == 7731 :", int(row["seed"]) == 7731)
print("int(row['errors']) > 0 :", int(row["errors"]) > 0)written : seed = 7731 errors = 3
read back : seed = '7731' errors = '3'
row['seed'] == 7731 : False
row['errors'] > 0 : True
int(row['seed']) == 7731 : True
int(row['errors']) > 0 : Truerow["seed"] == 7731 is False. A consumer filtering for a particular seed finds nothing, reports nothing, and looks like it worked.
The errors comparison is the more insidious one. row["errors"] > "0" is a string comparison that happens to give the right answer here and will not in general — "10" > "0" is True but "10" > "9" is False, so a filter for "more than 9 errors" silently excludes every row with 10 or more. The fix is to convert at the boundary, once, with a declared schema:
import csv
import io
SCHEMA = {"test": str, "seed": int, "status": str, "errors": int, "runtime_s": float}
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=list(SCHEMA))
w.writeheader()
w.writerow({"test": "axi_burst", "seed": 7731, "status": "FAIL",
"errors": 10, "runtime_s": 142.5})
w.writerow({"test": "axi_wrap", "seed": 1190, "status": "FAIL",
"errors": 9, "runtime_s": 88.25})
def load(text, schema):
out = []
for raw in csv.DictReader(io.StringIO(text)):
out.append({k: fn(raw[k]) for k, fn in schema.items()})
return out
rows = load(buf.getvalue(), SCHEMA)
for r in rows:
print(f" {r['test']:<10} seed={r['seed']!r:<6} errors={r['errors']!r:<4} "
f"runtime={r['runtime_s']!r}")
print()
print("string comparison '10' > '9' :", "10" > "9")
print("typed comparison 10 > 9 :", 10 > 9)
print("rows with more than 9 errors, typed:",
[r["test"] for r in rows if r["errors"] > 9]) axi_burst seed=7731 errors=10 runtime=142.5
axi_wrap seed=1190 errors=9 runtime=88.25
string comparison '10' > '9' : False
typed comparison 10 > 9 : True
rows with more than 9 errors, typed: ['axi_burst']"10" > "9" is False, so the untyped filter would have dropped the row with the most errors. One dict comprehension at the read boundary removes the whole class, and the schema doubles as the fieldnames list — which is Section 5.
5. Columns Drift Unless You Name Them
A row built from a dict's values inherits that dict's insertion order, and two records assembled by different code paths do not share one:
r1 = {"test": "axi_burst", "seed": 7731, "status": "PASS"}
r2 = {"status": "FAIL", "test": "axi_wrap", "seed": 1190}
print("rendered from .values():")
for r in (r1, r2):
print(" " + ",".join(str(v) for v in r.values()))
print()
print("same three fields, same two records, different column order.")
print("errors raised:", 0)rendered from .values():
axi_burst,7731,PASS
FAIL,axi_wrap,1190
same three fields, same two records, different column order.
errors raised: 0A consumer indexing fields[2] for the status reads PASS from the first row and 1190 from the second. Explicit fieldnames makes the order a property of the file rather than of whichever dict arrived:
import csv
import io
FIELDS = ["test", "seed", "status"]
RECORDS = [
{"test": "axi_burst", "seed": 7731, "status": "PASS"},
{"status": "FAIL", "test": "axi_wrap", "seed": 1190},
]
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=FIELDS)
w.writeheader()
for r in RECORDS:
w.writerow(r)
print(buf.getvalue().replace("\r\n", "\n"), end="")
print()
print("column order is now the file's, not the record's.")test,seed,status
axi_burst,7731,PASS
axi_wrap,1190,FAIL
column order is now the file's, not the record's.Now the two cases that DictWriter makes decisions about, and which must be decided deliberately:
import csv
import io
FIELDS = ["test", "seed", "status"]
print("a record with an EXTRA key:")
try:
w = csv.DictWriter(io.StringIO(), fieldnames=FIELDS, extrasaction="raise")
w.writerow({"test": "a", "seed": 1, "status": "PASS", "wave": "a.fsdb"})
except ValueError as exc:
print(" extrasaction='raise' ->", exc)
sb = io.StringIO()
w = csv.DictWriter(sb, fieldnames=FIELDS, extrasaction="ignore")
w.writerow({"test": "a", "seed": 1, "status": "PASS", "wave": "a.fsdb"})
print(" extrasaction='ignore' ->", repr(sb.getvalue()))
print()
print("a record MISSING a key:")
sb2 = io.StringIO()
w2 = csv.DictWriter(sb2, fieldnames=FIELDS, restval="")
w2.writerow({"test": "a", "seed": 1})
print(" restval='' ->", repr(sb2.getvalue()))
sb3 = io.StringIO()
w3 = csv.DictWriter(sb3, fieldnames=FIELDS, restval="UNKNOWN")
w3.writerow({"test": "a", "seed": 1})
print(" restval='UNKNOWN' ->", repr(sb3.getvalue()))a record with an EXTRA key:
extrasaction='raise' -> dict contains fields not in fieldnames: 'wave'
extrasaction='ignore' -> 'a,1,PASS\r\n'
a record MISSING a key:
restval='' -> 'a,1,\r\n'
restval='UNKNOWN' -> 'a,1,UNKNOWN\r\n'extrasaction="raise" is the right default for a report generator. The day somebody adds a wave field to the result record, the writer stops rather than silently dropping a column that a consumer may already be looking for — which is Chapter 29.2's refuse-the-ungoverned-metric argument applied to a schema.
And restval makes the missing-versus-empty distinction into a decision. With restval="" a job whose status was never determined is indistinguishable from one whose status is the empty string; with "UNKNOWN" it is a value a consumer can filter on. Chapter 30.1 Section 6's None-is-not-empty rule, in a file format with no None.
6. The Run Identity Has to Be Inside the File
Here is why the annotation passes run_id as a parameter:
import csv
import io
ROWS = [
{"test": "axi_burst", "seed": 7731, "status": "FAIL"},
{"test": "axi_wrap", "seed": 1190, "status": "PASS"},
]
FIELDS = ["test", "seed", "status"]
def render(rows):
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=FIELDS)
w.writeheader()
for r in rows:
w.writerow(r)
return buf.getvalue()
monday = render(ROWS)
friday = render(ROWS)
print("monday path : runs/2026-02-16_nightly/regression.csv")
print("friday path : runs/2026-02-20_nightly/regression.csv")
print()
print("file contents identical:", monday == friday)
print("bytes :", len(monday.encode()))
print()
print("after somebody attaches one to a ticket, the filename is:",
"regression.csv")
print("and the run it came from is:", "unrecoverable")monday path : runs/2026-02-16_nightly/regression.csv
friday path : runs/2026-02-20_nightly/regression.csv
file contents identical: True
bytes : 59
after somebody attaches one to a ticket, the filename is: regression.csv
and the run it came from is: unrecoverableTwo byte-identical files from runs four days apart. The identity lived in the directory name, and a directory name survives exactly until somebody copies, downloads, emails or attaches the file — which is the entire purpose of a machine-readable summary.
Putting the identity in every row costs one column and makes the file self-describing:
import csv
import io
FIELDS = ["run_id", "started_utc", "tool", "test", "seed", "status"]
RUN = {"run_id": "2026-02-18_nightly_r3",
"started_utc": "2026-02-18T22:05:00Z",
"tool": "simtool-1.2.3"}
ROWS = [
{"test": "axi_burst", "seed": 7731, "status": "FAIL"},
{"test": "axi_wrap", "seed": 1190, "status": "PASS"},
]
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=FIELDS, extrasaction="raise")
w.writeheader()
for r in ROWS:
w.writerow({**RUN, **r})
text = buf.getvalue()
print(text.replace("\r\n", "\n"), end="")
back = list(csv.DictReader(io.StringIO(text)))
print()
print("rows :", len(back))
print("distinct run_ids:", sorted({r["run_id"] for r in back}))
print("file identifies its own run:", len({r["run_id"] for r in back}) == 1)
print("two runs concatenated would show:", "2 distinct run_ids")run_id,started_utc,tool,test,seed,status
2026-02-18_nightly_r3,2026-02-18T22:05:00Z,simtool-1.2.3,axi_burst,7731,FAIL
2026-02-18_nightly_r3,2026-02-18T22:05:00Z,simtool-1.2.3,axi_wrap,1190,PASS
rows : 2
distinct run_ids: ['2026-02-18_nightly_r3']
file identifies its own run: True
two runs concatenated would show: 2 distinct run_idsRepeating the run identity on every row looks wasteful and is the reason this format is useful: two of these files concatenate into a valid dataset, and the consumer can still tell the runs apart. A header-block format with the identity written once cannot be concatenated, cannot be loaded by a plain CSV reader, and cannot answer a question that spans two nights.
The tool column earns its place for the reason Chapter 29.1 Section 8 gave: when a number looks wrong, the first question is which version produced it.
7. A List Cannot Cross a Cell
Chapter 21.1 Section 5 tested each field of a result record against the question "does it survive the format?", and one field failed:
import csv
import io
RECORD = {
"test": "axi_burst",
"seed": 7731,
"errors": ["UVM_ERROR drv.sv(88): bad handshake",
"UVM_ERROR drv.sv(91): bad handshake"],
}
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=["test", "seed", "errors"])
w.writeheader()
w.writerow(RECORD)
text = buf.getvalue()
back = list(csv.DictReader(io.StringIO(text)))[0]
print("written errors type:", type(RECORD["errors"]).__name__)
print("read back type :", type(back["errors"]).__name__)
print("read back value :", repr(back["errors"])[:60])
print()
print("is it a list? ", isinstance(back["errors"], list))
print("round trip intact? ", back["errors"] == RECORD["errors"])
print("usable by a consumer:", False)written errors type: list
read back type : str
read back value : "['UVM_ERROR drv.sv(88): bad handshake', 'UVM_ERROR drv.sv(9
is it a list? False
round trip intact? False
usable by a consumer: FalseA list written to a CSV cell becomes its Python repr, which round-trips as a string that happens to look like a list and is not one. A consumer either gets it wrong or reaches for ast.literal_eval, which makes the file's schema "Python source" rather than CSV.
The answer is the one Chapter 21.1 reached: store what a cell can hold, and point at the rest.
import csv
import io
RECORD = {
"test": "axi_burst",
"seed": 7731,
"errors": ["UVM_ERROR drv.sv(88): bad handshake",
"UVM_ERROR drv.sv(91): bad handshake"],
"log": "runs/axi_burst_7731/sim.log",
}
FIELDS = ["test", "seed", "error_count", "first_error", "log"]
flat = {
"test": RECORD["test"],
"seed": RECORD["seed"],
"error_count": len(RECORD["errors"]),
"first_error": RECORD["errors"][0] if RECORD["errors"] else "",
"log": RECORD["log"],
}
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=FIELDS, extrasaction="raise")
w.writeheader()
w.writerow(flat)
back = list(csv.DictReader(io.StringIO(buf.getvalue())))[0]
for k in FIELDS:
print(f" {k:<12} {back[k]!r}")
print()
print("errors retrievable in full from:", back["log"])
print("every cell is a scalar:", all(not isinstance(v, (list, dict)) for v in flat.values()))
print("count is queryable, text is readable, nothing was invented.") test 'axi_burst'
seed '7731'
error_count '2'
first_error 'UVM_ERROR drv.sv(88): bad handshake'
log 'runs/axi_burst_7731/sim.log'
errors retrievable in full from: runs/axi_burst_7731/sim.log
every cell is a scalar: True
count is queryable, text is readable, nothing was invented.A count a consumer can filter on, one error a human can read, and a path to all of them. This is Chapter 28.1 Section 12's rule for the third time in Part II: extract what a program compares, point at what a person reads.
8. The Writer, Assembled
import csv
import io
from dataclasses import dataclass
from pathlib import Path
from typing import List, Optional
SCHEMA = {
"run_id": str, "started_utc": str, "tool": str,
"test": str, "seed": int, "status": str,
"error_count": int, "first_error": str,
"runtime_s": float, "log": str, "run_dir": str,
}
FIELDS = list(SCHEMA)
@dataclass(frozen=True)
class JobResult:
test: str
seed: int
status: str
errors: List[str]
runtime_s: float
log: str
run_dir: str
def write_csv(run_id, run_meta, results, path):
"""One row per job, fixed columns, run identity on every row."""
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=FIELDS, extrasaction="raise",
restval="UNKNOWN")
w.writeheader()
for r in results:
w.writerow({
"run_id": run_id,
"started_utc": run_meta["started_utc"],
"tool": run_meta["tool"],
"test": r.test, "seed": r.seed, "status": r.status,
"error_count": len(r.errors),
"first_error": r.errors[0] if r.errors else "",
"runtime_s": f"{r.runtime_s:.2f}",
"log": r.log, "run_dir": r.run_dir,
})
Path(path).write_text(buf.getvalue(), encoding="utf-8", newline="")
return len(results)
def load_csv(path, schema):
text = Path(path).read_text(encoding="utf-8")
rows = []
for raw in csv.DictReader(io.StringIO(text)):
rows.append({k: fn(raw[k]) for k, fn in schema.items()})
return rows
RESULTS = [
JobResult("axi_burst", 7731, "FAIL",
["UVM_ERROR drv.sv(88), bad handshake\nretrying"],
142.5, "runs/axi_burst_7731/sim.log", "runs/axi_burst_7731"),
JobResult("axi_wrap", 1190, "PASS", [], 88.25,
"runs/axi_wrap_1190/sim.log", "runs/axi_wrap_1190"),
JobResult("dma_long", 2048, "TIMEOUT", [], 1800.0,
"runs/dma_long_2048/sim.log", "runs/dma_long_2048"),
]
n = write_csv("2026-02-18_nightly_r3",
{"started_utc": "2026-02-18T22:05:00Z", "tool": "simtool-1.2.3"},
RESULTS, "regression.csv")
rows = load_csv("regression.csv", SCHEMA)
print(f"rows written: {n} rows read: {len(rows)}")
print(f"columns : {len(FIELDS)}")
print()
print(f"{'test':<10} {'seed':>6} {'status':<8} {'errs':>5} {'runtime':>9}")
for r in rows:
print(f"{r['test']:<10} {r['seed']:>6} {r['status']:<8} "
f"{r['error_count']:>5} {r['runtime_s']:>9.2f}")
print()
print("seed is an int :", all(isinstance(r["seed"], int) for r in rows))
print("runtime is a float :", all(isinstance(r["runtime_s"], float) for r in rows))
print("embedded newline kept :",
"\n" in [r["first_error"] for r in rows if r["test"] == "axi_burst"][0])
print("one run_id :", sorted({r["run_id"] for r in rows}))
print("physical lines in file:",
len(Path("regression.csv").read_text(encoding="utf-8").rstrip("\r\n").split("\n")))rows written: 3 rows read: 3
columns : 11
test seed status errs runtime
axi_burst 7731 FAIL 1 142.50
axi_wrap 1190 PASS 0 88.25
dma_long 2048 TIMEOUT 0 1800.00
seed is an int : True
runtime is a float : True
embedded newline kept : True
one run_id : ['2026-02-18_nightly_r3']
physical lines in file: 5Three rows across five physical lines, with the embedded newline preserved and every value correctly typed on the way back in. The physical lines: 5 figure is the load-bearing one: it proves the quoted newline is really in the file, and it is why wc -l is the wrong tool for counting jobs in a regression CSV.
Note newline="" on the write. Without it, Python's text layer translates the writer's \r\n again on some platforms and the file gains blank rows — the one platform-dependent detail in this chapter, and the reason the csv documentation specifies it.
9. Row Order Decides Whether Two Files Diff
A machine-readable summary is read by scripts and also by diff, and that second reader is the one nobody designs for.
RESULTS = [("dma_long", 2048), ("axi_burst", 7731), ("axi_wrap", 1190)]
print("as the scheduler completed them (one possible order):")
for t, s in RESULTS:
print(f" {t},{s}")
print()
print("the same run, a different night, same results:")
for t, s in [("axi_wrap", 1190), ("dma_long", 2048), ("axi_burst", 7731)]:
print(f" {t},{s}")
print()
print("rows differing:", 3)
print("results differing:", 0)as the scheduler completed them (one possible order):
dma_long,2048
axi_burst,7731
axi_wrap,1190
the same run, a different night, same results:
axi_wrap,1190
dma_long,2048
axi_burst,7731
rows differing: 3
results differing: 0Every row differs and nothing changed. Writing rows in completion order makes the file a record of the scheduler rather than of the regression, and it destroys the cheapest comparison anybody has: diff last_night.csv tonight.csv.
It is also a determinism violation of exactly the kind Module 27 warned about, arriving in a file instead of in a printed value. Sorting by a key the run owns fixes it:
import csv
import io
RESULTS = [("dma_long", 2048, "TIMEOUT"), ("axi_burst", 7731, "FAIL"),
("axi_wrap", 1190, "PASS")]
def render(results, sort=True):
rows = sorted(results, key=lambda r: (r[0], r[1])) if sort else results
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=["test", "seed", "status"])
w.writeheader()
for t, s, st in rows:
w.writerow({"test": t, "seed": s, "status": st})
return buf.getvalue()
shuffled = [RESULTS[2], RESULTS[0], RESULTS[1]]
print("unsorted, two orders identical:", render(RESULTS, False) == render(shuffled, False))
print("sorted, two orders identical :", render(RESULTS, True) == render(shuffled, True))
print()
print(render(RESULTS, True).replace("\r\n", "\n"), end="")
print()
print("sort key:", "(test, seed) — both are the caller's, neither is measured")unsorted, two orders identical: False
sorted, two orders identical : True
test,seed,status
axi_burst,7731,FAIL
axi_wrap,1190,PASS
dma_long,2048,TIMEOUT
sort key: (test, seed) — both are the caller's, neither is measuredThe sort key matters as much as the sorting. (test, seed) are values the caller chose before the run started, so they are stable across nights; sorting by runtime or by status would reintroduce exactly the instability the sort was meant to remove, because both are outcomes.
One consequence is worth stating because it looks like a drawback and is not: a sorted file no longer tells you what ran first. That information belongs in a column — a start timestamp — where a consumer can sort by it on demand, rather than in the row order where it silently prevents every other comparison.
10. Predict, Run, Observe
import csv
import io
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=["seed", "errors"])
w.writeheader()
w.writerow({"seed": 7731, "errors": 10})
row = list(csv.DictReader(io.StringIO(buf.getvalue())))[0]
# A
print("A:", row["seed"] == 7731, int(row["seed"]) == 7731)
# B
print("B:", "10" > "9", 10 > 9)
# C
r1 = {"a": 1, "b": 2}
r2 = {"b": 2, "a": 1}
print("C:", list(r1.values()), list(r2.values()))
# D
print("D:", repr(",".join(["x", "a,b"])))The verified output:
A: False True
B: False True
C: [1, 2] [2, 1]
D: 'x,a,b'A — the comparison a consumer writes first is false. Convert at the boundary.
B — the string comparison that silently drops every row with ten or more errors.
C — two records with the same fields and opposite column orders, from .values().
D — three columns from two fields, with no quoting and no error.
11. Exercises
Exercise 1 — two readers. List the requirements your human report and your machine summary share. Say what that implies.
Exercise 2 — contradictory rules. Explain why Chapter 30.2's line budget and this chapter's one-row-per-job rule cannot both be satisfied by one file.
Exercise 3 — break it with a comma. Put a comma in a first-failure field and run your current writer. Report what the file contains.
Exercise 4 — break it with a newline. Do the same with a newline and count the physical lines.
Exercise 5 — wc -l is wrong. Explain why counting lines does not count jobs, and what does.
Exercise 6 — the free-form column. Name every column in your CSV that can contain arbitrary log text.
Exercise 7 — truncate or quote. Decide for your flow whether the first-failure column is truncated or fully quoted, and say why.
Exercise 8 — the false comparison. Find a consumer script that compares a CSV value against an integer, and run it.
Exercise 9 — the string filter. Construct a "more than 9" filter on untyped values and show which row it drops.
Exercise 10 — a schema at the boundary. Add a type map to one of your loaders and convert every column once.
Exercise 11 — column drift. Build two records in different key orders and render both from .values().
Exercise 12 — index by position. Say what a consumer reading fields[2] gets from each of those two rows.
Exercise 13 — name your fields. Convert one writer to explicit fieldnames and confirm the header matches.
Exercise 14 — raise on extras. Set extrasaction="raise" and add a field to your record. Report the exception.
Exercise 15 — missing versus empty. Choose a restval for your flow and justify it in one sentence.
Exercise 16 — two identical files. Produce byte-identical CSVs from two different runs of your flow.
Exercise 17 — identity in the row. Add run_id to every row and say what it costs in bytes for a 400-job run.
Exercise 18 — concatenate. Join two nights' CSVs and confirm a plain reader can still separate the runs.
Exercise 19 — why not a header block. Explain what a header-block format cannot do that a repeated column can.
Exercise 20 — the tool column. Say what question the tool column answers and when you would need it.
Exercise 21 — a list in a cell. Write a list to a CSV cell, read it back, and report its type.
Exercise 22 — flatten it. Replace one list column with a count, a first value and a path.
Exercise 23 — literal_eval is a schema. Explain why using ast.literal_eval on a column makes the file's format Python rather than CSV.
Exercise 24 — newline="". Remove it from a writer and describe what changes.
12. Summary
- A human report and a machine summary share no requirements — the bucket is the unit of one and the job is the unit of the other, and both rules are right.
",".join(...)corrupts a real record. One four-column row became two lines with five columns on the first.- The
csvmodule quotes the comma and allows the newline inside the quotes: three physical lines, one row, the field byte-identical. - A quoted newline is legal CSV, so a consumer must use a CSV parser and
wc -ldoes not count jobs. - The column that breaks a CSV is always the error message, because it is the only free-form one and the one consumers most want.
- Every value reads back as a string.
row["seed"] == 7731is False. "10" > "9"is False, so an untyped "more than 9 errors" filter drops exactly the worst rows. Convert at the boundary with a declared schema.- Columns drift unless you name them. Two records with the same fields rendered in opposite orders with no error.
extrasaction="raise"is the right default so a new field stops the writer instead of silently vanishing.restvalturns missing-versus-empty into a decision in a format that has noNone.- Two byte-identical
regression.csvfiles came from runs four days apart. A directory name survives until somebody copies the file. - Repeat the run identity on every row: it costs one column and it makes two files concatenate into a valid dataset.
- A list written to a cell becomes its Python
repr— store a count, a first value and a path instead. - Pass
newline=""to the writer, the one platform-dependent detail in the chapter. - Row order decides whether two files diff. Completion order made every row differ on a night when nothing changed.
- Sort by keys the caller chose —
(test, seed)— never by an outcome like runtime or status, which reintroduces the instability. - Put start time in a column, not in the row order, so a consumer can sort by it without losing every other comparison.
The next chapter renders the same facts for a human to paste into a review, where the escaping problem is the same and the available tools are weaker.
Where this fits
Part of the Python curriculum.