JSON as the Machine-Readable Default

Part II · Module 21 · Chapter 21.4 · JSON as the Machine-Readable Default

1. The Thing a Table Could Not Hold

Chapter 21.1 Section 5 wrote a list into a CSV cell and got a Python repr back. The same value through JSON:

import json

RECORD = {
    "test": "axi_smoke", "seed": 1839274, "status": "FAIL",
    "runtime_s": None, "errors": 2,
    "context": ["UVM_INFO drv.sv(60) @ 141700: issuing write",
                "UVM_ERROR drv.sv(88) @ 141900: bad handshake"],
    "counts": {"errors": 2, "fatals": 0, "warnings": 1},
}

text = json.dumps(RECORD)
back = json.loads(text)

print("identical     :", back == RECORD)
print("context type  :", type(back["context"]).__name__, len(back["context"]))
print("nested lookup :", back["counts"]["fatals"])
print("None survived :", back["runtime_s"] is None)

identical     : True
context type  : list 2
nested lookup : 0
None survived : True

A list stayed a list, a nested dict stayed a dict, and None came back as None rather than Chapter 21.2's ''. That is the whole case for JSON in one output block, and it is why the canonical purpose calls it the right first choice for data one script writes and another reads.

The other half of the claim is that it needs nothing installed. json is in the standard library — Chapter 13.4 — so a script using it runs on a bare interpreter on a compute farm with no package manager and no virtual environment. For a flow that has to work on whatever the farm has, that is worth more than any feature.

2. What "Exactly" Covers

JSON has six types. Python has more, and the mapping is worth knowing before you rely on it:

import json

print(json.dumps({"a": None, "b": True, "c": False, "d": 1, "e": 1.5, "f": "x"}))

{"a": null, "b": true, "c": false, "d": 1, "e": 1.5, "f": "x"}

The first seven rows are exact. The last three are the chapter.

3. Two Silent Conversions and One Loud One

import json

print("=== a tuple")
pair = (1, 2)
back = json.loads(json.dumps({"pair": pair}))["pair"]
print("  wrote", type(pair).__name__, pair, "-> read", type(back).__name__, back)
print("  equal:", back == pair)

print()
print("=== an integer dict key")
numbered = {1: "axi_smoke", 2: "axi_reset"}
text = json.dumps(numbered)
restored = json.loads(text)
print("  json text:", text)
print("  keys back:", list(restored))
try:
    print(restored[1])
except KeyError as exc:
    print("  restored[1] -> KeyError:", exc)

print()
print("=== a set")
try:
    json.dumps({"seen": {1, 2}})
except TypeError as exc:
    print("  TypeError:", exc)

=== a tuple
  wrote tuple (1, 2) -> read list [1, 2]
  equal: False

=== an integer dict key
  json text: {"1": "axi_smoke", "2": "axi_reset"}
  keys back: ['1', '2']
  restored[1] -> KeyError: 1

=== a set
  TypeError: Object of type set is not JSON serializable

A tuple becomes a list. JSON has one sequence type. The values are intact and the type is not, so a record compared with == against its pre-write self fails — which is how this is usually discovered, in a test that round-trips a record and asserts equality.

An integer dict key becomes a string. This is the dangerous one, because nothing raises at any point: the write succeeds, the file is valid JSON, the read succeeds, and the resulting dict has '1' where the original had 1. The failure happens later and somewhere else, at cfg[seed] with an integer seed, and the traceback points at the lookup rather than at the write.

A set raises. Loud, immediate, and names the type. This is the good case — you find out at the write, with a stack pointing at the line that produced the set.

The practical rule that follows: a record intended for JSON should contain only JSON's types. Chapter 20.16's record already does, which was not an accident — its fields are strings, integers, floats, booleans, None and one list. Chapter 21.1's eight are all scalars. If you find yourself adding a set or an integer-keyed dict to a record that gets serialised, convert it at the boundary — sorted(the_set) and {str(k): v} — where the conversion is visible, rather than letting the library do it where it is not.

4. JSON Does Not Fix the Float Problem

Chapter 21.3 Section 6 showed csv writing 0.30000000000000004 and argued for formatting at the point of writing. It is worth being explicit that changing format does not change that:

import csv
import io
import json

VALUE = 0.1 + 0.2

buffer = io.StringIO()
csv.DictWriter(buffer, fieldnames=["runtime_s"]).writerow({"runtime_s": VALUE})

print("python repr:", repr(VALUE))
print("csv  writes:", buffer.getvalue().strip())
print("json writes:", json.dumps({"runtime_s": VALUE}))
print("round trips:", json.loads(json.dumps({"runtime_s": VALUE}))["runtime_s"] == VALUE)

python repr: 0.30000000000000004
csv  writes: 0.30000000000000004
json writes: {"runtime_s": 0.30000000000000004}
round trips: True

Identical digits. JSON's advantage here is that it round-trips the float exactly — the value you read back is the value you wrote, which csv cannot promise because it hands you a string to convert.

But the diff stability problem is unchanged: two runs that produce arithmetically-identical runtimes by different routes can still write different digits, and a nightly comparison shows a change. Chapter 21.3's answer applies without modification — decide the precision when you write, here by rounding the value rather than formatting it as text, because JSON should hold a number and not a string:

import json

MEASURED = 0.1 + 0.2

print("raw     :", json.dumps({"runtime_s": MEASURED}))
print("rounded :", json.dumps({"runtime_s": round(MEASURED, 3)}))
print("as text :", json.dumps({"runtime_s": f"{MEASURED:.3f}"}))

raw     : {"runtime_s": 0.30000000000000004}
rounded : {"runtime_s": 0.3}
as text : {"runtime_s": "0.300"}

round() keeps it a number, which every consumer can compare and sum. The string version is stable too and makes the consumer convert — the CSV problem, reintroduced into a format that did not have it.

5. Writing for a Diff

import json

CONFIG = {"sim": "xcelium", "timeout_s": 7200,
          "tests": {"axi_smoke": {"seeds": 4}, "axi_burst": {"seeds": 16}},
          "waves": False}

print("--- compact")
print(json.dumps(CONFIG))
print()
print("--- indent=2, sort_keys=True")
print(json.dumps(CONFIG, indent=2, sort_keys=True))

--- compact
{"sim": "xcelium", "timeout_s": 7200, "tests": {"axi_smoke": {"seeds": 4}, "axi_burst": {"seeds": 16}}, "waves": false}

--- indent=2, sort_keys=True
{
  "sim": "xcelium",
  "tests": {
    "axi_burst": {
      "seeds": 16
    },
    "axi_smoke": {
      "seeds": 4
    }
  },
  "timeout_s": 7200,
  "waves": false
}

indent=2 is the canonical annotation's argument and it does two things: it makes the file readable, and it puts one value per line so a line-based diff shows which value changed rather than that the single line changed.

sort_keys=True is not in the canonical annotation and is worth adding for the same reason Chapter 21.3 sorted its rows: dict order is insertion order, so a config assembled in a different order writes a different file with identical content. Sorting makes the file a function of the data alone.

Note "waves": false — lower case, because that is JSON's spelling. Reading it back gives Python's False, so this is a notation difference rather than a conversion.

For a file a machine writes and a machine reads, compact is fine and smaller. For anything a person will ever open, indent=2, sort_keys=True costs nothing that matters.

6. Reading

{
  "sim": "xcelium",
  "timeout_s": 7200,
  "tests": {
    "axi_smoke": {"seeds": 4},
    "axi_burst": {"seeds": 16}
  },
  "waves": false
}

import json
from pathlib import Path

cfg = json.loads(Path("sim.json").read_text(encoding="utf-8"))
print("loads :", cfg["sim"], cfg["timeout_s"], cfg["waves"])

with open("sim.json", encoding="utf-8") as handle:
    same = json.load(handle)
print("load  :", same == cfg)

print("tests :", sorted(cfg["tests"]))
print("seeds :", cfg["tests"]["axi_burst"]["seeds"])

loads : xcelium 7200 False
load  : True
tests : ['axi_burst', 'axi_smoke']
seeds : 16

Two functions, one letter apart, and the difference catches people once: json.load takes a file object; json.loads takes a string. The same pairing exists for writing — json.dump to a file, json.dumps to a string.

The canonical annotation uses the loads plus Path.read_text form:

cfg = json.loads(Path("sim.json").read_text())

which is one line instead of two and is the right choice for a config — a file small enough to read whole, where Chapter 18.3's Path is already in use. The with open(...) plus json.load form is better when the file is large, because it does not hold the text and the parsed structure at the same time.

encoding="utf-8" is specified on the read_text, for Chapter 15.3's reason. JSON is defined to be UTF-8, so this is not a guess.

7. JSON Is Strict, and That Is the Cost

Everything above is about a file one script writes and another reads. The moment a person edits it, strictness stops being free:

import json

ATTEMPTS = [
    ('{"sim": "xcelium", "timeout_s": 7200,}', "a trailing comma"),
    ('{"sim": "xcelium"} // the only simulator we have', "a comment"),
    ("{'sim': 'xcelium'}", "single quotes"),
    ('{"sim": "xcelium",\n "timeout_s": 7200\n "waves": false}', "a forgotten comma"),
]

for text, what in ATTEMPTS:
    try:
        json.loads(text)
        print(f"{what:<18} parsed")
    except json.JSONDecodeError as exc:
        print(f"{what:<18} line {exc.lineno} col {exc.colno}: {exc.msg}")

a trailing comma   line 1 col 37: Illegal trailing comma before end of object
a comment          line 1 col 20: Extra data
single quotes      line 1 col 2: Expecting property name enclosed in double quotes
a forgotten comma  line 3 col 2: Expecting ',' delimiter

Four edits a person makes, four parse failures. They are good errors — each names a line, a column and what was expected — and they arrive before anything runs, which is Chapter 21.7's whole argument.

They are also four reasons an engineer who wanted to disable waves for one night ends up with a regression that does not start. The trailing comma is the commonest by a distance, because it is what you leave behind after deleting the last entry of a list.

JSON has no comments. That is the restriction people feel most, because a config is exactly the kind of file that wants a note saying why a timeout is 7200. The usual workaround is a key nobody reads:

import json

CONFIG = {
    "_comment": "timeout raised to 7200 for the gate-level runs, see VLSI-4412",
    "sim": "xcelium",
    "timeout_s": 7200,
}

cfg = json.loads(json.dumps(CONFIG))
print("keys    :", sorted(cfg))
print("comment :", cfg["_comment"][:34])
print("ignored :", {k: v for k, v in cfg.items() if not k.startswith("_")})

keys    : ['_comment', 'sim', 'timeout_s']
comment : timeout raised to 7200 for the gat
ignored : {'sim': 'xcelium', 'timeout_s': 7200}

It works, and it is a convention rather than a feature: every consumer has to agree to skip keys starting with _, and a validator like Chapter 21.7's has to be told they are allowed.

This is the gap Chapter 21.5 is about. JSON is the right default and it is not the right format for a file engineers edit by hand, and the next chapter is about what the alternative costs.

8. The Shape of a Config Read

Putting the pieces together — read, check the keys exist, use them:

{
  "sim": "xcelium",
  "timeout_s": 7200,
  "waves": false,
  "tests": {
    "axi_smoke": {"seeds": 4},
    "axi_burst": {"seeds": 16}
  }
}

import json
from pathlib import Path

REQUIRED = ["sim", "timeout_s", "tests"]


def load_config(path):
    """Read a JSON config and confirm the top-level keys are present."""
    cfg = json.loads(Path(path).read_text(encoding="utf-8"))
    if not isinstance(cfg, dict):
        raise ValueError(f"{path}: top level must be an object, got {type(cfg).__name__}")
    missing = [key for key in REQUIRED if key not in cfg]
    if missing:
        raise ValueError(f"{path}: missing {missing}")
    return cfg


cfg = load_config("regress.json")
print("sim      :", cfg["sim"])
print("timeout_s:", cfg["timeout_s"])
print("waves    :", cfg["waves"])

jobs = 0
for test in sorted(cfg["tests"]):
    seeds = cfg["tests"][test]["seeds"]
    jobs += seeds
    print(f"  {test:<12} {seeds:>3} seeds")
print("total jobs:", jobs)

sim      : xcelium
timeout_s: 7200
waves    : False
  axi_burst     16 seeds
  axi_smoke      4 seeds
total jobs: 20

The isinstance check is worth the line: a JSON file whose top level is a list, or a bare number, parses perfectly and then fails at the first cfg["sim"] with a TypeError that names list indices. Checking the shape before using it turns that into a message naming the file.

The key check is the beginning of validation and deliberately not the end — a sim of "xcelum" passes it, and a timeout_s of -1 passes it. Chapter 21.7 is where that is finished.

9. Predict, Run, Observe

import json

# A
print("A:", json.dumps({"a": (1, 2)}))

# B
print("B:", json.loads(json.dumps({"a": (1, 2)}))["a"] == (1, 2))

# C
print("C:", json.dumps({3: "x"}))

# D
print("D:", list(json.loads(json.dumps({3: "x"}))))

# E
print("E:", json.dumps({"a": None, "b": True}))

# F
try:
    json.loads('{"a": 1,}')
except json.JSONDecodeError as exc:
    print("F:", exc.msg)

# G
print("G:", json.dumps({"b": 1, "a": 2}), json.dumps({"b": 1, "a": 2}, sort_keys=True))

# H
print("H:", type(json.loads("3")).__name__, type(json.loads('"3"')).__name__)

The verified output:

A: {"a": [1, 2]}
B: False
C: {"3": "x"}
D: ['3']
E: {"a": null, "b": true}
F: Illegal trailing comma before end of object
G: {"b": 1, "a": 2} {"a": 2, "b": 1}
H: int str

A and B — the tuple is written as an array and comes back as a list, so an equality check against the original fails.

C and D — the integer key is written and returned as a string, with nothing raised at either end.

E — null and true, JSON's spellings, which map back to None and True.

F — the trailing comma, named precisely.

G — insertion order by default, alphabetical with sort_keys. The second is the one that diffs.

H — a bare number and a bare string are both valid JSON documents, which is why Section 8 checks that the top level is an object.

10. Exercises

Exercise 1 — what a table could not hold. Round-trip a record containing a list and a nested dict through JSON. Confirm it comes back equal.

Exercise 2 — the type table. For each of the ten rows in Section 2, write the value, read it back, and record the type.

Exercise 3 — the tuple. Round-trip a tuple and compare with ==. Say where in a real codebase this would first be noticed.

Exercise 4 — the integer key. Build a config keyed by integers, round-trip it, and then look one up by integer. Record the error and say at which of the four steps it could have been caught.

Exercise 5 — the set. Serialise a set and record the exception. Say why this is the best of the three behaviours.

Exercise 6 — convert at the boundary. Take a record containing a set and an integer-keyed dict and write the conversion explicitly. Say what you gained over letting the library do it.

Exercise 7 — floats. Write 0.1 + 0.2 through CSV and through JSON. Compare the digits, then make the JSON stable without turning the number into a string.

Exercise 8 — diffable output. Write the same config twice with the keys inserted in different orders, with and without sort_keys. Diff the four files.

Exercise 9 — load and loads. Read the same file both ways and confirm the results are equal. Say which you would use for a 2 GB file and why.

Exercise 10 — four human edits. Reproduce all four parse failures from Section 7 and record the line, column and message for each.

Exercise 11 — the comment convention. Add an _comment key and write the consumer that ignores it. Say what else in your flow has to know about the convention.

Exercise 12 — the shape check. Write a JSON file whose top level is a list and load it with and without the isinstance check. Compare the two error messages.

11. Summary

• JSON holds what a table cannot — lists, nested objects, and None as None rather than ''.
• It needs nothing installed, so a script using it runs on a bare interpreter on any farm machine.
• "Round-trips exactly" means exactly, for the types JSON has — and Python has three it does not.
• A tuple comes back as a list, silently, so an equality check against the original fails.
• An integer dict key comes back as a string, silently, and the failure appears later at a lookup that finds nothing.
• A set raises TypeError at the write — the loud case, and the best of the three.
• A record intended for JSON should contain only JSON's types, with any conversion done explicitly at the boundary.
• Integer keys bite hardest in configs, where a per-priority or per-seed table silently stops matching.
• JSON does not fix the float problem. 0.1 + 0.2 writes the same digits as CSV; it does round-trip exactly, which CSV cannot.
• Round the number rather than formatting it as text, so the file holds a number a consumer can sum.
• indent=2 puts one value per line so a diff names the value that changed; sort_keys=True makes the file a function of the data rather than of insertion order.
• json.load takes a file object and json.loads takes a string — same for dump and dumps.
• json.loads(Path(p).read_text(encoding="utf-8")) is the right form for a config; json.load on an open file is better for a large one.
• JSON is strict, and a person editing it hits four parse errors: a trailing comma, a comment, single quotes and a forgotten comma.
• There are no comments, and the _comment key is a convention every consumer and validator must be told about.
• Check that the top level is an object before indexing it — a bare list or number is valid JSON and fails later with a confusing error.

The next chapter is about the format that fixes the human-editing problem, and what adding it to a flow actually costs.