Skip to content
VLSI Mentor

Python Part II · Module 27 · Parallel Regression

Establishing the Serial Baseline

Before changing how a regression runs, record what it produces. That sounds obvious and the obvious version of it is the wrong one: most teams record how long the serial run took, discover afterwards that the number was never actionable, and have nothing at all to compare the parallel results against. The baseline worth having is the set of verdicts, and comparing two of them turns out to be a single expression — because of a keyword argument chosen six chapters ago for what looked at the time like a different reason.

Advanced18 min readPythonPart IIAutomationRegressionVerification Flow

Part II · Module 27 · Chapter 27.2 · Establishing the Serial Baseline

1. Which Baseline

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
CANDIDATES = [
    ("how long it took", "varies with machine, load and day", "nobody acts on it"),
    ("the pass count", "one number", "says nothing about which jobs"),
    ("the summary", "five numbers", "two different nights can match by accident"),
    ("the set of verdicts", "one per job", "a difference names the job"),
]

print(f"{'candidate':<22} {'what it is':<36} {'what it is worth'}")
for name, what, worth in CANDIDATES:
    print(f"{name:<22} {what:<36} {worth}")

print()
print("which one answers 'did parallelising change anything?':", CANDIDATES[-1][0])
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
candidate              what it is                           what it is worth
how long it took       varies with machine, load and day    nobody acts on it
the pass count         one number                           says nothing about which jobs
the summary            five numbers                         two different nights can match by accident
the set of verdicts    one per job                          a difference names the job

which one answers 'did parallelising change anything?': the set of verdicts

A duration is the least durable thing about a serial run. It depends on which machine was free, what else was running, and whether the filesystem was busy — so a "2.8× speedup" measured once is not a fact anybody can act on, and this curriculum does not publish such figures for exactly that reason.

The third row is the subtle one. Chapter 26.1's summary is five numbers, and two genuinely different nights can produce identical fives — one job flipping from PASS to FAIL while another flips the other way leaves every count unchanged. A summary is the right thing to report and the wrong thing to diff.

What answers the question is one verdict per job, which Chapter 26.6 already collects.

2. The Annotation

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
serial = run_all(jobs, workers=1)
# every later parallel run is compared against this

The first line is the easy half: workers=1 is Chapter 27.3's pool with one worker, so the same code path produces the baseline and the thing being compared — which matters, because a baseline produced by different code compares two changes at once.

The comment is the chapter. Compared how, and on what?

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass, field
from typing import Optional


@dataclass(frozen=True)
class Result:
    test: str
    seed: int
    status: str
    reason: Optional[str] = None
    run_dir: Optional[str] = field(default=None, compare=False)
    runtime_s: float = field(default=0.0, compare=False)


SERIAL = [Result("a", 1, "PASS", None, "runs/a_1", 180.0),
          Result("b", 2, "FAIL", "UVM_ERROR", "runs/b_2", 200.0),
          Result("c", 3, "PASS", None, "runs/c_3", 150.0)]

PARALLEL = [Result("c", 3, "PASS", None, "par/c_3", 151.9),
            Result("a", 1, "PASS", None, "par/a_1", 402.0),
            Result("b", 2, "FAIL", "UVM_ERROR", "par/b_2", 205.3)]

print("as lists:", SERIAL == PARALLEL)
print("as sets :", set(SERIAL) == set(PARALLEL))
print()
print("they differ in order   :", [r.test for r in SERIAL], "vs", [r.test for r in PARALLEL])
print("they differ in runtime :", SERIAL[0].runtime_s, "vs", PARALLEL[1].runtime_s)
print("they differ in run_dir :", SERIAL[0].run_dir, "vs", PARALLEL[1].run_dir)
print("and they are the same result set anyway")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
as lists: False
as sets : True

they differ in order   : ['a', 'b', 'c'] vs ['c', 'a', 'b']
they differ in runtime : 180.0 vs 402.0
they differ in run_dir : runs/a_1 vs par/a_1
and they are the same result set anyway

One expression, and it already ignores everything that is supposed to differ.

The lists are unequal because results arrive in completion order — Chapter 27.1 Section 6's last row — and that is not a finding. The sets are equal because:

  • runtime_s carries compare=False, decided in Chapter 25.1 Section 7. Job a took more than twice as long under the parallel run, which is expected on a loaded machine and says nothing about the design.
  • run_dir carries it too, added here for the same reason: a parallel run under a different root produces a different path for every job, and comparing paths would report four hundred differences.
  • frozen=True supplies __hash__, without which set() raises.

Chapter 25.1 Section 7 justified compare=False on the grounds that two runs of one test should compare equal. This is the chapter it was built for, three modules later, and the payoff is that the central comparison of a parallelisation effort is set(a) == set(b).

3. What a Difference Looks Like

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass, field
from typing import Optional


@dataclass(frozen=True)
class Result:
    test: str
    seed: int
    status: str
    reason: Optional[str] = None
    run_dir: Optional[str] = field(default=None, compare=False)
    runtime_s: float = field(default=0.0, compare=False)


SERIAL = [Result("a", 1, "PASS"), Result("b", 2, "FAIL", "UVM_ERROR"), Result("c", 3, "PASS")]
PARALLEL = [Result("c", 3, "PASS"), Result("a", 1, "FAIL", "UVM_ERROR"),
            Result("b", 2, "FAIL", "UVM_ERROR")]

print("sets equal:", set(SERIAL) == set(PARALLEL))
print()
only_serial = sorted(set(SERIAL) - set(PARALLEL), key=lambda r: (r.test, r.seed))
only_parallel = sorted(set(PARALLEL) - set(SERIAL), key=lambda r: (r.test, r.seed))
print("only in serial  :", [(r.test, r.seed, r.status) for r in only_serial])
print("only in parallel:", [(r.test, r.seed, r.status) for r in only_parallel])
print()
print("what that tells you:", "job (a, 1) is involved — twice, as two members")
print("what it does not   :", "that it is one job whose status changed")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
sets equal: False

only in serial  : [('a', 1, 'PASS')]
only in parallel: [('a', 1, 'FAIL')]

what that tells you: job (a, 1) is involved — twice, as two members
what it does not   : that it is one job whose status changed

The set difference detects the change and reports it awkwardly: one job appears as two unrelated members, one in each direction. With four hundred jobs and a dozen differences, that is twenty-four rows to pair up by eye.

Keying by job identity gives the answer directly:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass, field
from typing import Optional


@dataclass(frozen=True)
class Result:
    test: str
    seed: int
    status: str
    reason: Optional[str] = None
    runtime_s: float = field(default=0.0, compare=False)


def diff(baseline, candidate):
    a = {(r.test, r.seed): r for r in baseline}
    b = {(r.test, r.seed): r for r in candidate}
    out = []
    for key in sorted(set(a) | set(b)):
        x, y = a.get(key), b.get(key)
        if x is None:
            out.append((key, "only in candidate", None, (y.status, y.reason)))
        elif y is None:
            out.append((key, "only in baseline", (x.status, x.reason), None))
        elif x != y:
            out.append((key, "changed", (x.status, x.reason), (y.status, y.reason)))
    return out


SERIAL = [Result("a", 1, "PASS"), Result("b", 2, "FAIL", "UVM_ERROR"),
          Result("c", 3, "PASS"), Result("d", 4, "PASS")]
PARALLEL = [Result("c", 3, "PASS"), Result("a", 1, "FAIL", "UVM_ERROR"),
            Result("b", 2, "FAIL", "UVM_ERROR"), Result("e", 5, "PASS")]

for key, kind, was, now in diff(SERIAL, PARALLEL):
    print(f"{str(key):<10} {kind:<18} {str(was):<22} -> {now}")

print()
print("differences:", len(diff(SERIAL, PARALLEL)))
print("identical runs give:", diff(SERIAL, SERIAL))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
('a', 1)   changed            ('PASS', None)         -> ('FAIL', 'UVM_ERROR')
('d', 4)   only in baseline   ('PASS', None)         -> None
('e', 5)   only in candidate  None                   -> ('PASS', None)

differences: 3
identical runs give: []

Three kinds of difference, each named, and the empty list for identical runs is what makes if diff(...) read correctly — Chapter 24.6 Section 4's argument about a record diff, applied to results.

The second and third rows matter as much as the first. A job present in one run and absent from the other is not a status change; it is Chapter 26.1's accounting failing, and the two must not be reported the same way. A parallel run that lost job d and gained job e has a job-list problem, not a design one.

4. What Must and Must Not Be Compared

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass, field, fields
from typing import Optional


@dataclass(frozen=True)
class Result:
    test: str
    seed: int
    status: str
    reason: Optional[str] = None
    error_count: int = 0
    run_dir: Optional[str] = field(default=None, compare=False)
    log_path: Optional[str] = field(default=None, compare=False)
    runtime_s: float = field(default=0.0, compare=False)
    worker: Optional[int] = field(default=None, compare=False)


base = Result("a", 1, "PASS", None, 0, "runs/a_1", "runs/a_1/sim.log", 180.0, None)
same = Result("a", 1, "PASS", None, 0, "elsewhere/a_1", "elsewhere/a_1/sim.log", 9999.0, 17)

print("compared    :", [f.name for f in fields(base) if f.compare])
print("not compared:", [f.name for f in fields(base) if not f.compare])
print()
print("equal despite a 55x runtime difference:", base == same)
print("equal despite a different directory   :", base.run_dir != same.run_dir and base == same)
print("equal despite running on worker 17    :", base == same)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
compared    : ['test', 'seed', 'status', 'reason', 'error_count']
not compared: ['run_dir', 'log_path', 'runtime_s', 'worker']

equal despite a 55x runtime difference: True
equal despite a different directory   : True
equal despite running on worker 17    : True

Five fields compared and four excluded, and each exclusion has to be justified by the same question: does this field differ between two correct runs of the same job?

runtime_s does, on any loaded machine. run_dir and log_path do, under a different root. worker is a new field this module introduces and is pure scheduling detail.

error_count is included, and that is a judgement worth defending. A job that failed with three errors serially and one error in parallel has produced a different result even though both are FAIL, and that difference is far more likely to be a flow problem — a truncated log, Chapter 26.4's interleaving — than a design one. Including it makes the comparison stricter, which is the right default for a baseline whose purpose is to catch exactly that.

5. Compare the Buckets Too

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from collections import Counter

SERIAL_BUCKETS = {"drv.sv(88) mismatch": 12, "mon.sv(31) transition": 3}
PARALLEL_BUCKETS = {"drv.sv(88) mismatch": 12, "mon.sv(31) transition": 3}
SUSPICIOUS = {"drv.sv(88) mismatch": 12, "mon.sv(31) transition": 3,
              "infrastructure: no licence": 4}

for label, buckets in (("identical", PARALLEL_BUCKETS), ("suspicious", SUSPICIOUS)):
    appeared = sorted(set(buckets) - set(SERIAL_BUCKETS))
    gone = sorted(set(SERIAL_BUCKETS) - set(buckets))
    print(f"{label:<12} new={appeared} gone={gone} "
          f"total={sum(buckets.values())} vs {sum(SERIAL_BUCKETS.values())}")

print()
print("the new bucket is:", "infrastructure")
print("what that means  :", "parallelising introduced a resource failure, not a design one")
print("which chapter    :", "27.5")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
identical    new=[] gone=[] total=15 vs 15
suspicious   new=['infrastructure: no licence'] gone=[] total=19 vs 15

the new bucket is: infrastructure
what that means  : parallelising introduced a resource failure, not a design one
which chapter    : 27.5

Chapter 26.7 Section 9's night-to-night comparison, pointed at serial versus parallel instead.

The per-job diff in Section 3 would report four changed jobs here and leave you to notice they share a cause. The bucket diff names the cause directly, and an entirely new infrastructure bucket appearing when the only change was the worker count is about as unambiguous a signal as this module produces.

That is the division of labour worth keeping: the per-job diff says which jobs moved, the bucket diff says why.

6. A Baseline of What?

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
BASELINE_META = {"commit": "9f668ff", "sim": "vcs", "sim_version": "U-2023.12",
                 "suite": "nightly", "tags": ["nightly"],
                 "jobs": 600, "seed_base": 1839274, "seed_mode": "consecutive",
                 "workers": 1}

CANDIDATE_META = dict(BASELINE_META, workers=32)
DRIFTED_META = dict(BASELINE_META, workers=32, commit="a1b2c3d", seed_base=2000000)


def comparable(a, b):
    ignored = {"workers"}
    differ = sorted(k for k in set(a) | set(b)
                    if k not in ignored and a.get(k) != b.get(k))
    return (not differ), differ


for label, meta in (("same experiment", CANDIDATE_META), ("drifted", DRIFTED_META)):
    ok, differ = comparable(BASELINE_META, meta)
    print(f"{label:<17} comparable={str(ok):<6} differing fields: {differ}")

print()
print("the only field allowed to differ:", "workers")
print("why                             :", "it is the thing under test")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
same experiment   comparable=True   differing fields: []
drifted           comparable=False  differing fields: ['commit', 'seed_base']

the only field allowed to differ: workers
why                             : it is the thing under test

A baseline that does not record what it was of is the same thing as no baseline.

Every difference between two result sets has two possible explanations — the change you made, or a change you did not notice — and the metadata is what eliminates the second. A comparison across two commits is not a parallelisation experiment; it is two experiments at once, and the differences are unattributable.

workers is the one field required to differ, because it is the independent variable. Everything else being pinned is what makes this an experiment rather than an observation. Chapter 24.6's environment fingerprint belongs in here too, and for the same reason it existed there: it worked last week has to become a diff.

7. The Baseline, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import json
from collections import Counter
from dataclasses import dataclass, field, asdict
from pathlib import Path
from typing import Optional


@dataclass(frozen=True)
class Result:
    test: str
    seed: int
    status: str
    reason: Optional[str] = None
    error_count: int = 0
    run_dir: Optional[str] = field(default=None, compare=False)
    runtime_s: float = field(default=0.0, compare=False)


def save_baseline(path, meta, results):
    payload = {"meta": meta,
               "results": sorted((asdict(r) for r in results),
                                 key=lambda d: (d["test"], d["seed"]))}
    Path(path).write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n",
                          encoding="utf-8")
    return len(payload["results"])


def load_baseline(path):
    payload = json.loads(Path(path).read_text(encoding="utf-8"))
    results = [Result(**d) for d in payload["results"]]
    return payload["meta"], results


def compare(baseline_meta, baseline, candidate_meta, candidate, ignore=("workers",)):
    meta_diff = sorted(k for k in set(baseline_meta) | set(candidate_meta)
                       if k not in ignore and baseline_meta.get(k) != candidate_meta.get(k))
    a = {(r.test, r.seed): r for r in baseline}
    b = {(r.test, r.seed): r for r in candidate}
    changed = [(k, a[k].status, b[k].status) for k in sorted(set(a) & set(b)) if a[k] != b[k]]
    return {"comparable": not meta_diff, "meta_diff": meta_diff,
            "missing": sorted(set(a) - set(b)), "extra": sorted(set(b) - set(a)),
            "changed": changed, "identical": not (meta_diff or changed)
            and set(a) == set(b)}


META = {"commit": "9f668ff", "sim": "vcs", "suite": "nightly", "workers": 1}
SERIAL = [Result("a", 1, "PASS"), Result("b", 2, "FAIL", "UVM_ERROR", 1),
          Result("c", 3, "PASS")]

n = save_baseline("baseline.json", META, SERIAL)
meta, loaded = load_baseline("baseline.json")
print("saved results      :", n)
print("round-trips        :", set(loaded) == set(SERIAL))
print()

GOOD = [Result("c", 3, "PASS", None, 0, "p/c", 99.0), Result("a", 1, "PASS", None, 0, "p/a", 1.0),
        Result("b", 2, "FAIL", "UVM_ERROR", 1, "p/b", 2.0)]
BAD = [Result("c", 3, "PASS"), Result("a", 1, "FAIL", "UVM_ERROR", 1), Result("b", 2, "PASS")]

for label, cand in (("parallel, clean", GOOD), ("parallel, changed", BAD)):
    r = compare(meta, loaded, dict(META, workers=32), cand)
    print(f"{label:<19} identical={str(r['identical']):<6} changed={r['changed']}")

r = compare(meta, loaded, dict(META, workers=32, commit="other"), GOOD)
print()
print("different commit   : comparable =", r["comparable"], "| meta_diff =", r["meta_diff"])
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
saved results      : 3
round-trips        : True

parallel, clean     identical=True   changed=[]
parallel, changed   identical=False  changed=[(('a', 1), 'PASS', 'FAIL'), (('b', 2), 'FAIL', 'PASS')]

different commit   : comparable = False | meta_diff = ['commit']

Five decisions, and the fourth is the one that keeps the experiment honest.

The results are saved sorted by (test, seed), so the file itself is diffable with ordinary tools and two baselines of the same regression produce byte-identical files when nothing changed — Chapter 24.6 Section 6's canonicalisation argument.

asdict and Result(**d) round-trip, confirmed rather than assumed. A baseline that does not reload into equal objects is a baseline that will report spurious differences forever.

changed reports the pair, so (('b', 2), 'FAIL', 'PASS') shows a job that got better — which is just as suspicious as one that got worse when the only change was the worker count.

comparable is separate from identical. A comparison across different commits is not "different", it is invalid, and collapsing the two would let somebody read a meaningless diff as a finding. The function reports both and the caller must look at comparable first.

missing and extra are carried, because a parallel run that lost a job is Chapter 26.1's accounting failure and must not be reported as a status change.

8. When They Differ

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
WORKERS = [1, 2, 4, 8, 16, 32]
DIFFERS = {1: False, 2: False, 4: False, 8: True, 16: True, 32: True}


def bisect(workers, differs):
    """Find the smallest worker count at which the results change."""
    good = [w for w in workers if not differs[w]]
    bad = [w for w in workers if differs[w]]
    return max(good) if good else None, min(bad) if bad else None


last_good, first_bad = bisect(WORKERS, DIFFERS)

for w in WORKERS:
    print(f"workers={w:>3}  matches the baseline: {not DIFFERS[w]}")

print()
print("last good   :", last_good)
print("first bad   :", first_bad)
print("the threshold is between:", (last_good, first_bad))
print()
print("what a threshold suggests:", "a resource limit, not a logic bug")
print("what no threshold suggests:", "it differs even at 2 — shared state")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
workers=  1  matches the baseline: True
workers=  2  matches the baseline: True
workers=  4  matches the baseline: True
workers=  8  matches the baseline: False
workers= 16  matches the baseline: False
workers= 32  matches the baseline: False

last good   : 4
first bad   : 8
the threshold is between: (4, 8)

what a threshold suggests: a resource limit, not a logic bug
what no threshold suggests: it differs even at 2 — shared state

A parallel run that does not match the baseline is a finding, and the first question is not what differs but at what worker count it starts. Halving until it stops is three or four runs and it discriminates between two completely different causes.

A clean threshold points at a resource. Results that match at four workers and differ at eight mean something ran out between the two — memory, licences, disk bandwidth — and Chapter 27.5 is the chapter. The threshold itself is the measurement: whatever the fifth worker needed, there were four of it.

No threshold points at shared state. If two workers already differ, nothing was exhausted; two jobs interfered, which is Chapter 27.4 Section 5's latent collision. The failing jobs then name the shared thing, because they are the ones that overlapped.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
BASELINE = {("a", 1): "PASS", ("b", 2): "PASS", ("c", 3): "FAIL", ("d", 4): "PASS"}
PARALLEL = {("a", 1): "PASS", ("b", 2): "FAIL", ("c", 3): "FAIL", ("d", 4): "ERROR"}

changed = [k for k in BASELINE if BASELINE[k] != PARALLEL[k]]
to_worse = [k for k in changed if BASELINE[k] == "PASS" and PARALLEL[k] != "PASS"]
to_infra = [k for k in changed if PARALLEL[k] == "ERROR"]

print("jobs            :", len(BASELINE))
print("changed         :", changed)
print("passed, now not :", to_worse)
print("now infra errors:", to_infra)
print()
print("the infra ones are:", "Chapter 27.5's territory — a resource ran out")
print("the others are    :", "Chapter 27.4's — something was shared")
print("investigate first :", "the infra ones, because they have a named cause")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
jobs            : 4
changed         : [('b', 2), ('d', 4)]
passed, now not : [('b', 2), ('d', 4)]
now infra errors: [('d', 4)]

the infra ones are: Chapter 27.5's territory — a resource ran out
the others are    : Chapter 27.4's — something was shared
investigate first : the infra ones, because they have a named cause

Split the differences by their new status before investigating any of them. A job that became ERROR has a named cause already — Chapter 26.7's bucket will say which resource — and a job that went from PASS to FAIL does not.

Investigating the infrastructure ones first is usually enough, because a resource shortage is frequently the cause of both: a job that could not get a licence is ERROR, and a job whose sibling could not get one may have been slowed into a timeout.

9. A Baseline Needs a Deterministic Flow

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
NON_DET = 0.02
JOBS = 400
expected_spurious = NON_DET * JOBS

print("non-determinism rate   :", f"{NON_DET:.0%}")
print("jobs                   :", JOBS)
print("spurious differences   :", f"{expected_spurious:.0f} per comparison")
print()
print("a serial baseline against a parallel run:")
print("   differences from parallelising :", "unknown")
print("   differences from non-determinism:", f"about {expected_spurious:.0f}")
print("   distinguishable                :", False)
print()
print("re-running the SERIAL regression twice would also differ by:",
      f"about {expected_spurious:.0f}")
print("so the baseline does not even match ITSELF:", True)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
non-determinism rate   : 2%
jobs                   : 400
spurious differences   : 8 per comparison

a serial baseline against a parallel run:
   differences from parallelising : unknown
   differences from non-determinism: about 8
   distinguishable                : False

re-running the SERIAL regression twice would also differ by: about 8
so the baseline does not even match ITSELF: True

A baseline is only meaningful for a flow that produces the same answers twice. At a two percent non-determinism rate, two serial runs of the identical regression differ by about eight jobs — so any comparison has eight differences before parallelising contributes anything, and the parallel run's own differences are indistinguishable noise.

This is Chapter 25.7's soak being a hard prerequisite rather than a recommendation, and Chapter 27.1 Section 7 listed it as a precondition for exactly this reason. The test is cheap: run the serial regression twice and diff it against itself.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
def baseline_is_trustworthy(serial_a, serial_b):
    """Two serial runs of the same regression must produce identical results."""
    a = {(r[0], r[1]): r[2] for r in serial_a}
    b = {(r[0], r[1]): r[2] for r in serial_b}
    differ = sorted(k for k in set(a) | set(b) if a.get(k) != b.get(k))
    return (not differ), differ


RUN_A = [("a", 1, "PASS"), ("b", 2, "FAIL"), ("c", 3, "PASS")]
RUN_B = [("a", 1, "PASS"), ("b", 2, "FAIL"), ("c", 3, "PASS")]
RUN_C = [("a", 1, "PASS"), ("b", 2, "PASS"), ("c", 3, "PASS")]

for label, other in (("identical", RUN_B), ("non-deterministic", RUN_C)):
    ok, differ = baseline_is_trustworthy(RUN_A, other)
    print(f"{label:<20} trustworthy={str(ok):<6} differences={differ}")

print()
print("run this before taking a baseline:", True)
print("cost                             :", "one extra serial run")
print("what it saves                    :", "every later comparison being unreadable")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
identical            trustworthy=True   differences=[]
non-deterministic    trustworthy=False  differences=[('b', 2)]

run this before taking a baseline: True
cost                             : one extra serial run
what it saves                    : every later comparison being unreadable

One extra serial run, and the baseline is either trustworthy or it is not.

The second row is what a two percent rate looks like at this scale: one job out of three, with nothing else changed. At four hundred jobs it is eight, and they are different eight each time — which is precisely the signature people attribute to the parallelisation they have not done yet.

10. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass, field, fields
from typing import Optional


@dataclass(frozen=True)
class R:
    test: str
    status: str
    run_dir: Optional[str] = field(default=None, compare=False)
    t: float = field(default=0.0, compare=False)


a = R("x", "PASS", "runs/x", 10.0)
b = R("x", "PASS", "other/x", 900.0)
c = R("x", "FAIL", "runs/x", 10.0)

# A
print("A:", a == b)

# B
print("B:", a == c)

# C
print("C:", [a, c] == [c, a], {a, c} == {c, a})

# D
print("D:", [f.name for f in fields(a) if f.compare])

# E
print("E:", len({a, b, c}))

# F
print("F:", sorted({a, c} - {a}, key=lambda r: r.status)[0].status)

# G
A = {("x", 1): "PASS"}
B = {("x", 1): "FAIL", ("y", 2): "PASS"}
print("G:", sorted(set(A) - set(B)), sorted(set(B) - set(A)))

# H
print("H:", [(k, A[k], B[k]) for k in sorted(set(A) & set(B)) if A[k] != B[k]])

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
A: True
B: False
C: False True
D: ['test', 'status']
E: 2
F: FAIL
G: [] [('y', 2)]
H: [(('x', 1), 'PASS', 'FAIL')]

A — a different directory and a 90× runtime difference do not make two results different.

B — a different status does.

C — order matters for lists and not for sets, which is why the comparison is a set comparison.

D — only the fields that identify an outcome take part.

E — three results, two distinct outcomes.

F — the set difference isolates the changed member, one direction at a time.

G — a job present in only one run shows up as extra, not as a change.

H — and a job in both with a different verdict is named with both values.

11. Exercises

Exercise 1 — four candidates. For each, say what it would and would not tell you after parallelising.

Exercise 2 — the summary trap. Construct two different nights whose five summary numbers are identical.

Exercise 3 — same code path. Say why the baseline should come from workers=1 rather than from separate serial code.

Exercise 4 — one expression. Compare two result sets that differ in order, runtime and directory, and show them equal.

Exercise 5 — the payoff. Name the chapter that made set(a) == set(b) possible and the argument it used at the time.

Exercise 6 — without frozen. Remove frozen=True and record what the comparison does.

Exercise 7 — the awkward report. Show a set difference reporting one changed job as two members.

Exercise 8 — key by identity. Write the per-job diff and show it naming the change.

Exercise 9 — three kinds. Produce a changed job, a missing job and an extra job in one comparison.

Exercise 10 — not a status change. Say why a missing job must be reported differently.

Exercise 11 — justify every exclusion. For each compare=False field in your result, answer "could a real bug show up only here?".

Exercise 12 — error_count. Argue for including it, and give a flow bug it would catch.

Exercise 13 — the widening move. Explain why adding compare=False to silence a noisy diff resembles Chapter 26.7's over-normalisation.

Exercise 14 — bucket diff. Compare serial and parallel bucket counts and construct a case where a new infrastructure bucket appears.

Exercise 15 — division of labour. Say what the per-job diff tells you and what the bucket diff adds.

Exercise 16 — metadata. List everything your baseline must record. Check which of them your current flow stores.

Exercise 17 — the independent variable. Say why workers is the only field allowed to differ.

Exercise 18 — two experiments at once. Describe what a comparison across two commits actually measures.

Exercise 19 — comparable before identical. Say what goes wrong if a caller checks only identical.

Exercise 20 — bisect. Halve the worker count until the results match. Report the threshold.

Exercise 21 — two causes. Say what a clean threshold suggests and what no threshold suggests.

Exercise 22 — split the differences. Separate jobs that became ERROR from those that became FAIL, and say which to investigate first.

Exercise 23 — against itself. Run your serial regression twice and diff it. Report the differences.

Exercise 24 — eight before you start. Compute how many spurious differences your non-determinism rate produces over your job count.

Exercise 25 — stale by design. Say how long your serial baseline is valid for and what survives it.

12. Summary

  • A duration is the least durable thing about a serial run, and a one-off speedup figure is not something anybody acts on.
  • A summary is the right thing to report and the wrong thing to diff — two different nights can produce identical counts.
  • The baseline that matters is one verdict per job, which Chapter 26.6 already collects.
  • Produce it with workers=1 through the same code path, so the comparison tests one change.
  • set(serial) == set(parallel) ignores order, runtime and directory — because of field(compare=False) from Chapter 25.1 Section 7 and frozen=True.
  • This is the chapter that decision was for, three modules downstream.
  • A set difference reports one changed job as two members; key by (test, seed) and the diff names the change.
  • A missing or extra job is an accounting failure, not a status change, and must be reported separately.
  • Every compare=False is a claim that no real bug can hide in that field — ask that, not "does it differ?".
  • Include error_count: three errors serially and one in parallel is a flow problem, most likely an interleaved log.
  • Diff the buckets as well as the jobs — the per-job diff says which moved, the bucket diff says why.
  • A new infrastructure bucket when only the worker count changed points straight at Chapter 27.5.
  • A baseline must record what it was of — commit, simulator and version, suite, job count, seed policy.
  • workers is the only field allowed to differ, because it is the independent variable.
  • Report comparable separately from identical, so a cross-commit comparison is invalid rather than merely different.
  • Save results sorted and round-trip them, so two unchanged baselines produce byte-identical files.
  • When a parallel run differs, bisect the worker count first — three or four runs locate the threshold.
  • A clean threshold points at a resource; no threshold points at shared state, and the failing jobs name the shared thing.
  • Split the differences by their new status: a job that became ERROR already has a named cause.
  • A baseline only means anything for a deterministic flow — at 2% over 400 jobs it differs from itself by about eight.
  • Diff the serial run against a second serial run before taking a baseline; one extra run decides whether any later comparison can be read.
  • A serial baseline is valid for one commit and then thrown away; Chapter 26.7's bucket history is what survives.

The next chapter runs the jobs concurrently, and compares what comes back against exactly this.

Where this fits

Part of the Python curriculum.