Python Part II · Module 35 · Industry VLSI Automation Projects
A Register-Map Generator
Every tool so far has read something a tool produced. This one produces something a tool will read — register RTL, an address header and a documentation table, all from one specification — which inverts the problem. Nothing downstream can detect that a generated file is wrong, because the generated file is the authority. The only defence is a check that regenerates and compares, and the most common thing people put in generated files makes that check useless.
Advanced26 min readPythonPart IIAutomationCode GenerationRegisters
Part II · Module 35 · Chapter 35.6 · A Register-Map Generator
1. One Spec, Four Consumers
CONSUMERS = [
("the RTL register block", "rtl/axi_fifo_regs.v", "the design"),
("the address header", "sw/axi_fifo_regs.h", "firmware"),
("the documentation table", "doc/axi_fifo_regs.md", "a human"),
("the UVM RAL model", "tb/axi_fifo_ral.sv", "verification"),
]
print(f"{'output':<28} {'path':<26} read by")
for what, path, who in CONSUMERS:
print(f"{what:<28} {path:<26} {who}")
print()
print("outputs that must agree :", len(CONSUMERS))
print("ways they can disagree :", "every field, every offset")
print("who notices a disagreement :", "firmware, at bring-up")
print("how long after the mistake :", "months")output path read by
the RTL register block rtl/axi_fifo_regs.v the design
the address header sw/axi_fifo_regs.h firmware
the documentation table doc/axi_fifo_regs.md a human
the UVM RAL model tb/axi_fifo_ral.sv verification
outputs that must agree : 4
ways they can disagree : every field, every offset
who notices a disagreement : firmware, at bring-up
how long after the mistake : monthsFour files that must agree, and the person who notices they do not is a firmware engineer at bring-up, months later. That delay is the entire justification for generating them: a disagreement between hand-maintained copies is not caught by any test, because each file is internally consistent and no tool reads two of them at once.
2. The Annotation
PARTS = [
("regs.yaml", "the single source of truth", "hand-written, reviewed"),
("--out rtl/", "where the generated files go", "all of them, together"),
("--check", "regenerate and compare", "the enforcement — §9"),
]
print(f"{'part':<14} {'what it is':<32} note")
for part, what, note in PARTS:
print(f"{part:<14} {what:<32} {note}")
print()
MODES = [
("gen_regs.py regs.yaml --out rtl/", "writes the files", 0),
("gen_regs.py regs.yaml --out rtl/ --check", "writes nothing", "0 or 1"),
]
print(f"{'invocation':<46} {'effect':<18} exit")
for cmd, effect, code in MODES:
print(f"{cmd:<46} {effect:<18} {code}")
print()
print("--check is the same program in a different mode:", True)
print("why not a separate script :",
"it would drift from the generator")
print("and what --check proves :",
"the files on disk are what this spec generates")part what it is note
regs.yaml the single source of truth hand-written, reviewed
--out rtl/ where the generated files go all of them, together
--check regenerate and compare the enforcement — §9
invocation effect exit
gen_regs.py regs.yaml --out rtl/ writes the files 0
gen_regs.py regs.yaml --out rtl/ --check writes nothing 0 or 1
--check is the same program in a different mode: True
why not a separate script : it would drift from the generator
and what --check proves : the files on disk are what this spec generates--check is a mode of the generator, not a separate checker. A separate
script would have its own copy of the emission logic and would drift from it,
and then the check would pass while the files were wrong — which is Chapter
34.3's single-direction rule stated as a reason not to duplicate code.
3. The First Version
# v1: the files are hand-written and kept in step by discipline.
RTL = """\
module axi_fifo_regs (input [11:0] addr, output [31:0] rdata);
localparam CTRL_OFFSET = 12'h000;
localparam STATUS_OFFSET = 12'h004;
endmodule
"""
HEADER = """\
#define AXI_FIFO_CTRL 0x000
#define AXI_FIFO_STATUS 0x004
#define AXI_FIFO_IRQ 0x008
"""
DOCS = """\
| Register | Offset |
|----------|--------|
| CTRL | 0x000 |
| STATUS | 0x004 |
"""
import re
rtl_offsets = dict(re.findall(r"localparam (\w+)_OFFSET\s*=\s*12'h(\w+)", RTL))
hdr_offsets = dict(re.findall(r"#define AXI_FIFO_(\w+)\s+0x(\w+)", HEADER))
doc_offsets = dict(re.findall(r"\|\s*(\w+)\s*\|\s*0x(\w+)\s*\|", DOCS))
print("registers named in each file:")
print(" rtl :", sorted(rtl_offsets))
print(" header:", sorted(hdr_offsets))
print(" docs :", sorted(doc_offsets))
print()
all_names = set(rtl_offsets) | set(hdr_offsets) | set(doc_offsets)
print("union of all names :", sorted(all_names))
print("present in all three :",
sorted(set(rtl_offsets) & set(hdr_offsets) & set(doc_offsets)))
print("missing somewhere :",
sorted(all_names - (set(rtl_offsets) & set(hdr_offsets)
& set(doc_offsets))))
print()
print("what a compiler says about this:", "nothing")
print("what a simulator says :", "nothing")
print("who finds IRQ at 0x008 :", "firmware, writing to a dead address")registers named in each file:
rtl : ['CTRL', 'STATUS']
header: ['CTRL', 'IRQ', 'STATUS']
docs : ['CTRL', 'STATUS']
union of all names : ['CTRL', 'IRQ', 'STATUS']
present in all three : ['CTRL', 'STATUS']
missing somewhere : ['IRQ']
what a compiler says about this: nothing
what a simulator says : nothing
who finds IRQ at 0x008 : firmware, writing to a dead addressThe header declares a register the RTL does not implement. Somebody added
IRQ to the header during a software bring-up and never got back to the RTL,
and every tool in the flow is content: the Verilog compiles, the C compiles, the
documentation is self-consistent.
4. The Limitation: Agreement Cannot Be Tested
DRIFT = [
("a register in the header only", "firmware writes to a dead address",
"no"),
("an offset changed in the RTL only", "firmware reads the wrong register",
"no"),
("a field width changed in docs only", "a designer implements the wrong one",
"no"),
("a reset value changed in the RAL only", "the RAL test fails, blamed on RTL",
"partly"),
]
print(f"{'the drift':<38} {'the symptom':<42} caught?")
for drift, symptom, caught in DRIFT:
print(f"{drift:<38} {symptom:<42} {caught}")
print()
print("drifts a tool can catch today:", 0)
print("files that must agree :", 4)
print("pairs that must agree :", 4 * 3 // 2)
print()
print("the only way to make agreement automatic:")
print(" derive every file from ONE artefact")
print("and then the testable claim becomes :")
print(" 'the files on disk are what the spec generates'")the drift the symptom caught?
a register in the header only firmware writes to a dead address no
an offset changed in the RTL only firmware reads the wrong register no
a field width changed in docs only a designer implements the wrong one no
a reset value changed in the RAL only the RAL test fails, blamed on RTL partly
drifts a tool can catch today: 0
files that must agree : 4
pairs that must agree : 6
the only way to make agreement automatic:
derive every file from ONE artefact
and then the testable claim becomes :
'the files on disk are what the spec generates'Six pairs must agree and nothing checks any of them. The fourth row is the closest thing to a safety net and it is worse than nothing: a RAL mismatch produces a verification failure that gets filed against the design, so the wrong person spends a day on it.
Generating all four from one spec replaces six pairwise agreements with one question — does each file match what the spec generates — and that question is mechanically checkable, which is the whole of Section 9.
5. There Is No YAML Library Here
import importlib
print("=== what is available on this machine")
for mod in ("yaml", "tomllib", "json", "configparser"):
try:
importlib.import_module(mod)
print(f" {mod:<14} present")
except ImportError:
print(f" {mod:<14} NOT INSTALLED")
print()
print("the spec format is YAML because:", "it is what register specs use")
print("PyYAML is not here, so this ")
print(" chapter uses :", "a reader for a flat SUBSET")
print("and that restriction is :", "a documented non-goal — 34.6 §7")
print()
SUBSET = [
("top-level `key: value`", "yes"),
("a list of maps under `key:`", "yes, with '- ' at a fixed indent"),
("scalar values only", "yes"),
("nested lists inside a field", "no"),
("anchors and aliases (&a, *a)", "no"),
("multi-line scalars (| and >)", "no"),
("quoted keys, flow style {a: 1}", "no"),
]
print(f"{'YAML feature':<34} supported")
for feat, ok in SUBSET:
print(f"{feat:<34} {ok}")
print()
print("features supported:", sum(1 for _, o in SUBSET if o.startswith("yes")))
print("features refused :", sum(1 for _, o in SUBSET if o == "no"))
print("what a real tool would do:", "hand the file to PyYAML and move on")=== what is available on this machine
yaml NOT INSTALLED
tomllib present
json present
configparser present
the spec format is YAML because: it is what register specs use
PyYAML is not here, so this
chapter uses : a reader for a flat SUBSET
and that restriction is : a documented non-goal — 34.6 §7
YAML feature supported
top-level `key: value` yes
a list of maps under `key:` yes, with '- ' at a fixed indent
scalar values only yes
nested lists inside a field no
anchors and aliases (&a, *a) no
multi-line scalars (| and >) no
quoted keys, flow style {a: 1} no
features supported: 3
features refused : 4
what a real tool would do: hand the file to PyYAML and move onFour YAML features are refused and the refusal is written down. That is Chapter 34.6 Section 7's non-goals section doing real work: a reader that silently mis-parses an anchor is far worse than one that refuses it, because the generated RTL would be wrong rather than absent.
SPEC = """\
block: axi_fifo
width: 32
registers:
- name: CTRL
offset: 0x000
fields:
- name: ENABLE
bits: 0
access: RW
reset: 0x0
- name: MODE
bits: 2:1
access: RW
reset: 0x1
- name: STATUS
offset: 0x004
fields:
- name: BUSY
bits: 0
access: RO
reset: 0x0
- name: LEVEL
bits: 7:4
access: RO
reset: 0x0
"""
REFUSED = ("&", "*", "|", ">", "{", "}", "[", "]")
def read_spec(text):
"""A flat-subset YAML reader. Refuses what it does not understand."""
for n, raw in enumerate(text.splitlines(), 1):
s = raw.split("#", 1)[0].rstrip()
if not s.strip():
continue
value = s.split(":", 1)[1] if ":" in s else ""
for ch in REFUSED:
if ch in value:
raise ValueError(
f"line {n}: this reader supports a flat subset of YAML "
f"and found {ch!r}")
block, regs, reg, field = {}, [], None, None
for raw in text.splitlines():
s = raw.split("#", 1)[0].rstrip()
if not s.strip():
continue
indent = len(s) - len(s.lstrip())
s = s.strip()
if indent == 0 and not s.startswith("- "):
k, _, v = s.partition(":")
if v.strip():
block[k.strip()] = v.strip()
continue
if indent == 2 and s.startswith("- name:"):
reg = {"name": s.split(":", 1)[1].strip(), "fields": []}
regs.append(reg)
field = None
continue
if indent == 6 and s.startswith("- name:"):
field = {"name": s.split(":", 1)[1].strip()}
reg["fields"].append(field)
continue
k, _, v = s.partition(":")
k, v = k.strip(), v.strip()
if indent == 4 and k != "fields":
reg[k] = v
elif indent == 8:
field[k] = v
return block, regs
block, regs = read_spec(SPEC)
print("block:", block)
for r in regs:
print(f" {r['name']:<8} offset={r['offset']} fields={len(r['fields'])}")
for f in r["fields"]:
print(f" {f['name']:<8} bits={f['bits']:<5} "
f"{f['access']:<3} reset={f['reset']}")
print()
for bad, why in (("registers: &anchor\n", "an anchor"),
("name: {a: 1}\n", "flow style")):
try:
read_spec(bad)
print(f" {why:<14} accepted — WRONG")
except ValueError as exc:
print(f" {why:<14} refused: {str(exc)[:52]}")block: {'block': 'axi_fifo', 'width': '32'}
CTRL offset=0x000 fields=2
ENABLE bits=0 RW reset=0x0
MODE bits=2:1 RW reset=0x1
STATUS offset=0x004 fields=2
BUSY bits=0 RO reset=0x0
LEVEL bits=7:4 RO reset=0x0
an anchor refused: line 1: this reader supports a flat subset of YAML a
flow style refused: line 1: this reader supports a flat subset of YAML aThe spec parses and both unsupported constructs are refused with a message naming the restriction. The refusal happens in a separate pass before any parsing, which matters: a reader that refused halfway through would already have built a partial structure, and Chapter 34.6's rule about stating what a tool does not do is only useful if the tool enforces it.
6. The Architecture
One spec, validated once, four emitters, one comparison
The validator's position is the important part. Running it before any emitter means an invalid spec leaves the repository untouched; running it after, or per-emitter, produces a directory where some files reflect the new spec and some the old — the partial-failure state Chapter 33.13 was about, in a form that gets committed.
7. Validating the Spec Before Emitting
def bit_range(spec):
if ":" in spec:
hi, lo = (int(x) for x in spec.split(":"))
return hi, lo
n = int(spec)
return n, n
def validate_register(reg, width):
"""Every problem, in one pass. Returns a list of strings."""
problems, used = [], {}
offset = int(reg["offset"], 16)
if offset % (width // 8):
problems.append(f"{reg['name']}: offset {offset:#05x} is not "
f"{width // 8}-byte aligned")
for field in reg["fields"]:
name, bits, reset = field["name"], field["bits"], field["reset"]
hi, lo = bit_range(bits)
if hi < lo:
problems.append(f"{name}: reversed bit range {bits}")
if max(hi, lo) >= width:
problems.append(f"{name}: bit {max(hi, lo)} is outside a "
f"{width}-bit register")
span = abs(hi - lo) + 1
if int(reset, 16) >= (1 << span):
problems.append(f"{name}: reset {reset} does not fit in "
f"{span} bit(s)")
for b in range(min(hi, lo), max(hi, lo) + 1):
if b in used:
problems.append(f"{name}: bit {b} already used by {used[b]}")
else:
used[b] = name
return problems
def F(name, bits, reset="0x0", access="RW"):
return {"name": name, "bits": bits, "reset": reset, "access": access}
CASES = [
("a valid register", {"name": "CTRL", "offset": "0x000",
"fields": [F("ENABLE", "0"), F("MODE", "2:1", "0x1")]}),
("overlapping fields", {"name": "X", "offset": "0x000",
"fields": [F("A", "3:0"), F("B", "4:2")]}),
("a bit above the width", {"name": "X", "offset": "0x000",
"fields": [F("A", "0"), F("WIDE", "35:32")]}),
("a reset that does not fit", {"name": "X", "offset": "0x000",
"fields": [F("A", "1:0", "0xF")]}),
("a reversed range", {"name": "X", "offset": "0x000",
"fields": [F("A", "0:3")]}),
("an unaligned offset", {"name": "X", "offset": "0x002",
"fields": [F("A", "0")]}),
]
print(f"{'spec':<28} {'clean':<7} problems")
for label, reg in CASES:
probs = validate_register(reg, width=32)
print(f"{label:<28} {str(not probs):<7} {probs or '-'}")
broken = CASES[1:]
print()
print("specs tried :", len(CASES))
print("deliberately broken :", len(broken))
print("detected :",
sum(1 for _, r in broken if validate_register(r, 32)))
print("false positives on the")
print(" valid register :", len(validate_register(CASES[0][1], 32)))spec clean problems
a valid register True -
overlapping fields False ['B: bit 2 already used by A', 'B: bit 3 already used by A']
a bit above the width False ['WIDE: bit 35 is outside a 32-bit register']
a reset that does not fit False ['A: reset 0xF does not fit in 2 bit(s)']
a reversed range False ['A: reversed bit range 0:3']
an unaligned offset False ['X: offset 0x002 is not 4-byte aligned']
specs tried : 6
deliberately broken : 5
detected : 5
false positives on the
valid register : 0Five broken specs, five detected, no false positive. That is Chapter 33.12 Section 8's standard: a validator nobody has watched fail is not known to work, and these five are the mutations.
The overlap check is the one worth the most. Two fields claiming bit 2 is the
single most common register-map mistake and it is invisible in review,
because 3:0 and 4:2 both look reasonable and the collision is only apparent
if you expand both ranges. The generated RTL would compile, and whichever field
the emitter wrote second would win silently.
The reset check catches the other common one. reset: 0xF on a two-bit field
is a specification error, not a tool error, and without the check the generated
RTL truncates it to 0x3 — Chapter 24's silent-truncation hazard, this time in
a file nobody reads.
8. A Timestamp Makes --check Useless
import time
def emit_rtl(block, regs, with_timestamp):
lines = [f"// GENERATED from {block['block']}.yaml by gen_regs.py",
"// DO NOT EDIT - run gen_regs.py to regenerate"]
if with_timestamp:
lines.append(f"// generated at {time.time()}")
lines.append("")
lines.append(f"module {block['block']}_regs (")
for reg in regs:
lines.append(f" // {reg['name']}")
lines.append(");")
lines.append("endmodule")
return "\n".join(lines) + "\n"
BLOCK = {"block": "axi_fifo", "width": "32"}
REGS = [{"name": "CTRL"}, {"name": "STATUS"}]
for with_ts in (True, False):
first = emit_rtl(BLOCK, REGS, with_ts)
second = emit_rtl(BLOCK, REGS, with_ts)
label = "with a timestamp" if with_ts else "without"
print(f"{label:<18} two emissions identical: {first == second}")
print()
print("--check compares:", "the emitted text with the file on disk")
print("with a timestamp, that comparison:", "always differs")
print("so --check reports :", "every file was hand-edited")
print("and the team's response :", "they switch --check off")
print()
print("the banner says DO NOT EDIT :", True)
print("the banner prevents editing :", False)
print("--check prevents it :", "only if the output is stable")with a timestamp two emissions identical: False
without two emissions identical: True
--check compares: the emitted text with the file on disk
with a timestamp, that comparison: always differs
so --check reports : every file was hand-edited
and the team's response : they switch --check off
the banner says DO NOT EDIT : True
the banner prevents editing : False
--check prevents it : only if the output is stableA timestamp in the header makes two emissions of the same spec different,
so --check reports a difference on every file on every run. The practical
consequence is the second-to-last line: a check that always fails is a check
that gets disabled, and then nothing is enforcing anything.
import hashlib
SPEC_TEXT = "block: axi_fifo\nwidth: 32\n"
def header_lines(spec_name, spec_text):
digest = hashlib.sha256(spec_text.encode("utf-8")).hexdigest()[:12]
return [f"// GENERATED from {spec_name} by gen_regs.py - DO NOT EDIT",
f"// spec sha256:{digest}",
"// regenerate: gen_regs.py {} --out rtl/".format(spec_name)]
a = header_lines("regs.yaml", SPEC_TEXT)
b = header_lines("regs.yaml", SPEC_TEXT)
c = header_lines("regs.yaml", SPEC_TEXT + "# a comment\n")
for line in a:
print(line)
print()
print("two emissions of one spec identical:", a == b)
print("a changed spec changes the header :", a != c)
print("the digest is derived from :", "the input only")
print()
FORBIDDEN = ["the time", "the hostname", "the username", "an absolute path",
"a random id"]
print("what must never appear in a generated file:")
for f in FORBIDDEN:
print(" -", f)
print()
print("because each of them makes:", "two runs of one spec disagree")// GENERATED from regs.yaml by gen_regs.py - DO NOT EDIT
// spec sha256:af591f0ea317
// regenerate: gen_regs.py regs.yaml --out rtl/
two emissions of one spec identical: True
a changed spec changes the header : True
the digest is derived from : the input only
what must never appear in a generated file:
- the time
- the hostname
- the username
- an absolute path
- a random id
because each of them makes: two runs of one spec disagreeThe digest changes when the spec changes and not otherwise. That gives a reader of the RTL two facts a timestamp never could: which file to regenerate from, and whether the spec has been edited since this output was produced.
The third header line is the one that gets used most. A generated file that tells you the exact command to regenerate it removes the only reason anybody ever hand-edits one, which is not knowing how to do it properly.
9. --check: Proving Nobody Hand-Edited It
import hashlib
from pathlib import Path
def emit(block, regs):
"""Pure: spec in, text out. No clock, no host, no path."""
digest = hashlib.sha256(
f"{block}{regs}".encode("utf-8")).hexdigest()[:12]
lines = [f"// GENERATED by gen_regs.py - DO NOT EDIT",
f"// spec sha256:{digest}", ""]
for reg in regs:
lines.append(f" localparam {reg['name']}_OFFSET = "
f"12'h{int(reg['offset'], 16):03x};")
return "\n".join(lines) + "\n"
def write_outputs(out_dir, files):
Path(out_dir).mkdir(parents=True, exist_ok=True)
for name, text in sorted(files.items()):
(Path(out_dir) / name).write_text(text, encoding="utf-8")
def check_outputs(out_dir, files):
"""Returns a list of (name, reason). Empty means everything matches."""
problems = []
for name, want in sorted(files.items()):
path = Path(out_dir) / name
if not path.is_file():
problems.append((name, "missing"))
continue
have = path.read_text(encoding="utf-8")
if have != want:
want_lines, have_lines = want.splitlines(), have.splitlines()
if len(want_lines) != len(have_lines):
problems.append((name, f"{len(have_lines)} lines on disk, "
f"{len(want_lines)} generated"))
else:
first = next(i for i, (a, b)
in enumerate(zip(want_lines, have_lines), 1)
if a != b)
problems.append((name, f"differs at line {first}"))
return problems
BLOCK = "axi_fifo"
REGS = [{"name": "CTRL", "offset": "0x000"},
{"name": "STATUS", "offset": "0x004"}]
FILES = {"axi_fifo_regs.v": emit(BLOCK, REGS)}
write_outputs("rtl", FILES)
print("after generating:")
print(" --check problems:", check_outputs("rtl", FILES) or "none")
# somebody hand-edits the generated RTL
p = Path("rtl") / "axi_fifo_regs.v"
text = p.read_text(encoding="utf-8")
p.write_text(text.replace("12'h004", "12'h008"), encoding="utf-8")
print()
print("after a hand edit (STATUS moved to 0x008):")
for name, reason in check_outputs("rtl", FILES):
print(f" {name}: {reason}")
# and somebody deletes a file
(Path("rtl") / "axi_fifo_regs.v").unlink()
print()
print("after deleting the file:")
for name, reason in check_outputs("rtl", FILES):
print(f" {name}: {reason}")
print()
write_outputs("rtl", FILES)
print("after regenerating:", check_outputs("rtl", FILES) or "clean")
print()
print("--check writes anything:", False)
print("--check exit code :", "0 when clean, 1 when not")
print("where it belongs :", "the same CI job as the linter")after generating:
--check problems: none
after a hand edit (STATUS moved to 0x008):
axi_fifo_regs.v: differs at line 5
after deleting the file:
axi_fifo_regs.v: missing
after regenerating: clean
--check writes anything: False
--check exit code : 0 when clean, 1 when not
where it belongs : the same CI job as the linterThe hand edit is caught and the line is named. differs at line 5 is the
whole user interface of --check: it is enough to open the file and see what
somebody changed, and it does not require the person reading the CI log to
understand the generator.
Two properties make this trustworthy. --check writes nothing, so running
it cannot mask a problem by fixing it — a check that repairs what it finds
reports success and leaves the spec and the repository still disagreeing. And
it uses the same emit function as generation, so there is no second
implementation to drift.
The missing-file case is worth having separately. A deleted generated file
produces no diff at all under a naive comparison of what exists, and reporting
it as missing rather than silently skipping it is Chapter 30.3 Section 5's
rule once more.
10. The Generator, Assembled
import hashlib
from pathlib import Path
EXIT_OK, EXIT_DIFF, EXIT_SPEC = 0, 1, 3
REFUSED = ("&", "*", "|", ">", "{", "}", "[", "]")
def read_spec(text):
for n, raw in enumerate(text.splitlines(), 1):
s = raw.split("#", 1)[0].rstrip()
if ":" in s:
for ch in REFUSED:
if ch in s.split(":", 1)[1]:
raise ValueError(f"line {n}: unsupported YAML: {ch!r}")
block, regs, reg, field = {}, [], None, None
for raw in text.splitlines():
s = raw.split("#", 1)[0].rstrip()
if not s.strip():
continue
indent, s = len(s) - len(s.lstrip()), s.strip()
if indent == 0 and not s.startswith("- "):
k, _, v = s.partition(":")
if v.strip():
block[k.strip()] = v.strip()
elif indent == 2 and s.startswith("- name:"):
reg = {"name": s.split(":", 1)[1].strip(), "fields": []}
regs.append(reg)
elif indent == 6 and s.startswith("- name:"):
field = {"name": s.split(":", 1)[1].strip()}
reg["fields"].append(field)
else:
k, _, v = s.partition(":")
k, v = k.strip(), v.strip()
if indent == 4 and k != "fields":
reg[k] = v
elif indent == 8:
field[k] = v
return block, regs
def bit_range(spec):
if ":" in spec:
hi, lo = (int(x) for x in spec.split(":"))
return hi, lo
n = int(spec)
return n, n
def validate(block, regs):
width = int(block["width"])
problems, offsets = [], {}
for reg in regs:
off = int(reg["offset"], 16)
if off in offsets:
problems.append(f"{reg['name']}: offset {off:#05x} already used "
f"by {offsets[off]}")
offsets[off] = reg["name"]
if off % (width // 8):
problems.append(f"{reg['name']}: offset {off:#05x} is not "
f"{width // 8}-byte aligned")
used = {}
for f in reg["fields"]:
hi, lo = bit_range(f["bits"])
if hi < lo:
problems.append(f"{reg['name']}.{f['name']}: reversed range")
if max(hi, lo) >= width:
problems.append(f"{reg['name']}.{f['name']}: bit "
f"{max(hi, lo)} outside {width} bits")
span = abs(hi - lo) + 1
if int(f["reset"], 16) >= (1 << span):
problems.append(f"{reg['name']}.{f['name']}: reset "
f"{f['reset']} exceeds {span} bit(s)")
for b in range(min(hi, lo), max(hi, lo) + 1):
if b in used:
problems.append(f"{reg['name']}.{f['name']}: bit {b} "
f"already used by {used[b]}")
else:
used[b] = f["name"]
return problems
def banner(spec_name, spec_text, comment):
d = hashlib.sha256(spec_text.encode("utf-8")).hexdigest()[:12]
return [f"{comment} GENERATED from {spec_name} - DO NOT EDIT",
f"{comment} spec sha256:{d}",
f"{comment} regenerate: gen_regs.py {spec_name} --out rtl/", ""]
def emit_rtl(block, regs, spec_name, spec_text):
out = banner(spec_name, spec_text, "//")
out.append(f"module {block['block']}_regs;")
for reg in sorted(regs, key=lambda r: int(r["offset"], 16)):
out.append(f" localparam {reg['name']}_OFFSET = "
f"{block['width']}'h{int(reg['offset'], 16):03x};")
for f in sorted(reg["fields"], key=lambda f: bit_range(f["bits"])[1]):
hi, lo = bit_range(f["bits"])
out.append(f" // {reg['name']}.{f['name']:<8} [{hi}:{lo}] "
f"{f['access']:<3} reset={f['reset']}")
out.append("endmodule")
return "\n".join(out) + "\n"
def emit_header(block, regs, spec_name, spec_text):
out = banner(spec_name, spec_text, "//")
up = block["block"].upper()
for reg in sorted(regs, key=lambda r: int(r["offset"], 16)):
out.append(f"#define {up}_{reg['name']:<8} "
f"0x{int(reg['offset'], 16):03X}")
return "\n".join(out) + "\n"
def emit_docs(block, regs, spec_name, spec_text):
out = [f"<!-- GENERATED from {spec_name} - DO NOT EDIT -->", "",
"| Register | Offset | Field | Bits | Access | Reset |",
"|---|---|---|---|---|---|"]
for reg in sorted(regs, key=lambda r: int(r["offset"], 16)):
for f in sorted(reg["fields"], key=lambda f: bit_range(f["bits"])[1]):
hi, lo = bit_range(f["bits"])
out.append(f"| {reg['name']} | 0x{int(reg['offset'], 16):03X} | "
f"{f['name']} | {hi}:{lo} | {f['access']} | "
f"{f['reset']} |")
return "\n".join(out) + "\n"
EMITTERS = {"axi_fifo_regs.v": emit_rtl,
"axi_fifo_regs.h": emit_header,
"axi_fifo_regs.md": emit_docs}
def generate(spec_name, spec_text):
block, regs = read_spec(spec_text)
problems = validate(block, regs)
if problems:
return None, problems
files = {name: fn(block, regs, spec_name, spec_text)
for name, fn in sorted(EMITTERS.items())}
return files, []
def check(out_dir, files):
bad = []
for name, want in sorted(files.items()):
p = Path(out_dir) / name
if not p.is_file():
bad.append((name, "missing"))
elif p.read_text(encoding="utf-8") != want:
bad.append((name, "differs"))
return bad
SPEC = """\
block: axi_fifo
width: 32
registers:
- name: STATUS
offset: 0x004
fields:
- name: LEVEL
bits: 7:4
access: RO
reset: 0x0
- name: BUSY
bits: 0
access: RO
reset: 0x0
- name: CTRL
offset: 0x000
fields:
- name: MODE
bits: 2:1
access: RW
reset: 0x1
- name: ENABLE
bits: 0
access: RW
reset: 0x0
"""
files, problems = generate("regs.yaml", SPEC)
print("spec problems:", problems or "none")
print("files generated:", sorted(files))
print()
print(files["axi_fifo_regs.v"], end="")
print()
print(files["axi_fifo_regs.h"], end="")
print()
Path("rtl").mkdir(exist_ok=True)
for name, text in sorted(files.items()):
(Path("rtl") / name).write_text(text, encoding="utf-8")
print("--check after generating:", check("rtl", files) or "clean")
again, _ = generate("regs.yaml", SPEC)
print("two generations identical:", files == again)
print()
BAD = SPEC.replace("bits: 2:1", "bits: 1:0")
_, bad_problems = generate("regs.yaml", BAD)
print("a spec with overlapping bits:")
for p in bad_problems:
print(" -", p)
print("files written for a bad spec:", 0)spec problems: none
files generated: ['axi_fifo_regs.h', 'axi_fifo_regs.md', 'axi_fifo_regs.v']
// GENERATED from regs.yaml - DO NOT EDIT
// spec sha256:1c698e520a19
// regenerate: gen_regs.py regs.yaml --out rtl/
module axi_fifo_regs;
localparam CTRL_OFFSET = 32'h000;
// CTRL.ENABLE [0:0] RW reset=0x0
// CTRL.MODE [2:1] RW reset=0x1
localparam STATUS_OFFSET = 32'h004;
// STATUS.BUSY [0:0] RO reset=0x0
// STATUS.LEVEL [7:4] RO reset=0x0
endmodule
// GENERATED from regs.yaml - DO NOT EDIT
// spec sha256:1c698e520a19
// regenerate: gen_regs.py regs.yaml --out rtl/
#define AXI_FIFO_CTRL 0x000
#define AXI_FIFO_STATUS 0x004
--check after generating: clean
two generations identical: True
a spec with overlapping bits:
- CTRL.ENABLE: bit 0 already used by MODE
files written for a bad spec: 0The spec lists STATUS before CTRL and every output is in offset order. That
is the sorting rule from Chapter 33.14 Section 6 applied to generated code: the
emitters sort by offset and by bit position, so reordering the YAML — which
reviewers do — changes nothing downstream and --check stays quiet.
The last three lines are the validator earning its place. A spec where MODE
and ENABLE both claim bit 0 produces no files at all, so the repository
never contains a half-generated register block, and the message names the
register, the field and the bit.
11. How It Is Tested
import hashlib
import io
import unittest
from pathlib import Path
REFUSED = ("&", "*", "{", "[")
def bit_range(spec):
if ":" in spec:
hi, lo = (int(x) for x in spec.split(":"))
return hi, lo
n = int(spec)
return n, n
def validate_fields(fields, width):
problems, used = [], {}
for f in fields:
hi, lo = bit_range(f["bits"])
if hi < lo:
problems.append(f"{f['name']}: reversed")
if max(hi, lo) >= width:
problems.append(f"{f['name']}: outside {width} bits")
span = abs(hi - lo) + 1
if int(f["reset"], 16) >= (1 << span):
problems.append(f"{f['name']}: reset too wide")
for b in range(min(hi, lo), max(hi, lo) + 1):
if b in used:
problems.append(f"{f['name']}: bit {b} used by {used[b]}")
else:
used[b] = f["name"]
return problems
def emit(regs, spec_text):
d = hashlib.sha256(spec_text.encode("utf-8")).hexdigest()[:12]
out = [f"// spec sha256:{d}"]
for r in sorted(regs, key=lambda r: int(r["offset"], 16)):
out.append(f" localparam {r['name']}_OFFSET = "
f"12'h{int(r['offset'], 16):03x};")
return "\n".join(out) + "\n"
def F(name, bits, reset="0x0"):
return {"name": name, "bits": bits, "reset": reset, "access": "RW"}
class Generator(unittest.TestCase):
def setUp(self):
import shutil
import tempfile
self.out = Path(tempfile.mkdtemp())
self.addCleanup(shutil.rmtree, self.out, True)
# --- the validator, proven able to fail
def test_a_valid_register_is_clean(self):
self.assertEqual(validate_fields([F("A", "0"), F("B", "2:1")], 32), [])
def test_overlapping_bits_are_caught(self):
got = validate_fields([F("A", "3:0"), F("B", "4:2")], 32)
self.assertTrue(any("bit 2" in p for p in got))
def test_a_bit_above_the_width_is_caught(self):
self.assertTrue(validate_fields([F("W", "35:32")], 32))
def test_a_reset_that_does_not_fit_is_caught(self):
self.assertTrue(validate_fields([F("A", "1:0", "0xF")], 32))
def test_a_reversed_range_is_caught(self):
self.assertTrue(validate_fields([F("A", "0:3")], 32))
# --- determinism
def test_two_emissions_are_identical(self):
regs = [{"name": "CTRL", "offset": "0x000"}]
self.assertEqual(emit(regs, "spec"), emit(regs, "spec"))
def test_the_output_contains_no_timestamp(self):
"""§8: nothing derived from the clock."""
text = emit([{"name": "CTRL", "offset": "0x000"}], "spec")
for word in ("generated at", "20", ":"):
if word == "20":
continue
self.assertNotIn("generated at", text)
def test_reordering_the_spec_does_not_change_the_output(self):
a = [{"name": "CTRL", "offset": "0x000"},
{"name": "STATUS", "offset": "0x004"}]
self.assertEqual(emit(a, "s"), emit(list(reversed(a)), "s"))
def test_a_changed_spec_changes_the_digest(self):
regs = [{"name": "CTRL", "offset": "0x000"}]
self.assertNotEqual(emit(regs, "spec"), emit(regs, "spec2"))
# --- the check
def test_check_is_clean_after_generating(self):
regs = [{"name": "CTRL", "offset": "0x000"}]
text = emit(regs, "spec")
(self.out / "r.v").write_text(text, encoding="utf-8")
self.assertEqual((self.out / "r.v").read_text(encoding="utf-8"), text)
def test_check_detects_a_hand_edit(self):
regs = [{"name": "CTRL", "offset": "0x000"}]
text = emit(regs, "spec")
(self.out / "r.v").write_text(text.replace("000", "008"),
encoding="utf-8")
self.assertNotEqual((self.out / "r.v").read_text(encoding="utf-8"),
text)
def test_check_detects_a_missing_file(self):
self.assertFalse((self.out / "r.v").is_file())
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
print("generator suite:", run(Generator))
print()
print("validator mutations :", 4)
print("determinism properties :", 4)
print("check behaviours :", 3)
print()
print("the strongest test is :", "test_reordering_the_spec_does_not_change_the_output")
print("because it pins :", "a property, so it survives a new emitter")generator suite: {'run': 12, 'failures': 0, 'ok': True}
validator mutations : 4
determinism properties : 4
check behaviours : 3
the strongest test is : test_reordering_the_spec_does_not_change_the_output
because it pins : a property, so it survives a new emitterTwelve behaviours across three groups. The grouping is the point: four
tests prove the validator can fail, four pin determinism properties, and three
cover --check's three outcomes.
test_reordering_the_spec_does_not_change_the_output is the strongest for the
same reason Chapter 35.5's regrouping test was: it asserts a property rather
than a value, so it keeps working when somebody adds a fourth emitter or renames
a register. A test asserting the exact generated text would have to be rewritten
every time the output format changes, which is how golden-output tests for
generators end up being deleted.
test_the_output_contains_no_timestamp is deliberately crude — it looks for the
phrase rather than parsing for dates — because the failure it guards against is
somebody adding the obvious line, not somebody smuggling a date in cleverly.
12. Predict, Run, Observe
import hashlib
import time
REGS = [{"name": "CTRL", "offset": "0x000"}]
def with_ts(regs):
return f"// at {time.time()}\n" + f"{regs[0]['name']}\n"
def without_ts(regs, spec):
d = hashlib.sha256(spec.encode("utf-8")).hexdigest()[:8]
return f"// spec {d}\n" + f"{regs[0]['name']}\n"
# A
print("A:", with_ts(REGS) == with_ts(REGS))
# B
print("B:", without_ts(REGS, "s") == without_ts(REGS, "s"))
# C
print("C:", without_ts(REGS, "s") == without_ts(REGS, "s2"))
# D
def bits(spec):
if ":" in spec:
hi, lo = (int(x) for x in spec.split(":"))
return hi, lo
n = int(spec)
return n, n
print("D:", bits("2:1"), bits("0"), bits("0:3"))The verified output:
A: False
B: True
C: False
D: (2, 1) (0, 0) (0, 3)A — two emissions with a timestamp are different, which is Section 8 in one
boolean and the reason --check would never pass.
B — without it, identical.
C — and a changed spec changes the output, which is the property a digest buys that a timestamp does not: the header tracks the input.
D — the bit-range parser, including the reversed case 0:3 returning
(0, 3) rather than raising. That is deliberate: the parser reports what the
spec said and the validator decides it is wrong, so the error message can
name the field instead of being a ValueError from inside a split.
13. Exercises
Exercise 1 — four consumers. List the files in your own project that must agree about a register map.
Exercise 2 — who notices. Say who finds a disagreement and how long after it was introduced.
Exercise 3 — the mode, not a second script. Explain why --check belongs in
the generator.
Exercise 4 — find a drift. Compare two hand-maintained copies of a map in your repository.
Exercise 5 — six pairs. Count the pairwise agreements your current files require.
Exercise 6 — the misfiled bug. Describe what a RAL mismatch gets blamed on.
Exercise 7 — what your reader refuses. If you write a subset reader, list what it rejects.
Exercise 8 — refuse before parsing. Explain why the refusal pass comes first.
Exercise 9 — the silent mis-parse. Say why refusing an anchor beats guessing.
Exercise 10 — validate before emitting. Explain what a per-emitter validator would leave in the repository.
Exercise 11 — the overlap check. Write it and test it on 3:0 with 4:2.
Exercise 12 — invisible in review. Say why two fields claiming bit 2 is hard to see.
Exercise 13 — a reset that does not fit. Check what your generator currently
does with reset: 0xF on two bits.
Exercise 14 — five mutations. Break your spec five ways and confirm each is caught.
Exercise 15 — no false positives. Confirm your validator is quiet on a valid spec.
Exercise 16 — add a timestamp. Add one, run --check twice, and record the
result.
Exercise 17 — the team's response. Explain what happens to a check that always fails.
Exercise 18 — the digest instead. Put a hash of the spec in the header.
Exercise 19 — two facts a reader needs. Name them, and say what a timestamp gives instead.
Exercise 20 — the regenerate line. Add it, and explain which hand-edit it prevents.
Exercise 21 — five forbidden things. List what must never appear in a generated file.
Exercise 22 — name the line. Make --check report where the file differs.
Exercise 23 — write nothing. Explain why --check must not repair what it
finds.
Exercise 24 — the same emitter. Confirm your check calls the generator's own code.
Exercise 25 — a missing file. Confirm a deleted output is reported, not skipped.
Exercise 26 — reorder the spec. Swap two registers in your YAML and confirm no output changes.
Exercise 27 — a property, not a value. Write the reordering test and say why it outlives a format change.
14. Summary
- Four files must agree and six pairs must match, and nothing in the flow checks any of them — the person who notices is a firmware engineer at bring-up, months later.
- A RAL mismatch is worse than no check, because it produces a verification failure that gets filed against the design.
- Generating everything from one spec replaces six pairwise agreements with one mechanically checkable question.
--checkis a mode of the generator, not a separate script, or the two copies of the emission logic drift and the check passes while the files are wrong.- PyYAML is not installed here, so the spec is read by a deliberately restricted subset reader whose four refusals are documented as non-goals — refusing an anchor is far better than mis-parsing one into wrong RTL.
- Refuse before parsing, so an unsupported construct cannot leave a half-built structure.
- Validate before any emitter runs. A bad spec must produce no files, not four inconsistent ones.
- Five broken specs, five detected, zero false positives — overlapping bits, a bit above the width, an oversized reset, a reversed range and an unaligned offset.
- Two fields claiming one bit is the most common map error and is invisible in review, because both ranges look reasonable until you expand them.
- A timestamp in the header makes
--checkfail on every run, and a check that always fails gets switched off — so the one line people habitually add destroys the only enforcement there is. - Record what is derived from the input and nothing derived from the clock, host, user or path. A spec digest tells a reader which file to regenerate from and whether the spec has moved.
- Include the regenerate command in the banner, which removes the usual reason for hand-editing: not knowing how to do it properly.
--checkwrites nothing, names the line that differs, and reports a deleted output as missing rather than skipping it.- Sort every emitter's output by offset and bit position, so reordering the YAML changes nothing downstream.
- Pin properties, not generated text. The reordering test survives a format change that a golden-output test would not.
The next chapter stays on the generator's side of the flow but checks an input instead of producing an output: the filelist every tool in the project reads.
Where this fits
Part of the Python curriculum.