Skip to content
VLSI Mentor

Python Part II · Module 26 · Regression Automation

Test Lists and Regression Configuration

Chapter 26.1's whole contract rests on knowing how many jobs there are before any of them runs. That number comes from expanding a suite — a list of tests, each with tags and a seed policy — into concrete work, and the expansion is a one-line comprehension that encodes two decisions most people make by accident. One is the order the jobs come out in, which decides what a regression killed at the halfway mark is able to tell you. The other is what happens when a filter matches nothing.

Intermediate18 min readPythonPart IIAutomationRegressionVerification Flow

Part II · Module 26 · Chapter 26.2 · Test Lists and Regression Configuration

1. The Suite Is Data

Azvya Education Pvt. Ltd.VLSI Mentor
suite.json
{
  "tests": [
    {"name": "axi_smoke", "seeds": [1, 2], "tags": ["smoke", "axi"]},
    {"name": "axi_burst", "seeds": [1, 2, 3], "tags": ["nightly", "axi"]},
    {"name": "mem_walk", "seeds": [7], "tags": ["nightly", "mem"]}
  ]
}
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import json
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class Test:
    name: str
    seeds: tuple
    tags: tuple = ()


raw = json.loads(Path("suite.json").read_text(encoding="utf-8"))
tests = [Test(t["name"], tuple(t["seeds"]), tuple(t.get("tags", ()))) for t in raw["tests"]]

for t in tests:
    print(f"{t.name:<11} seeds={t.seeds} tags={t.tags}")

print()
print("tests loaded:", len(tests))
print("every seeds field is a tuple:", all(isinstance(t.seeds, tuple) for t in tests))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
axi_smoke   seeds=(1, 2) tags=('smoke', 'axi')
axi_burst   seeds=(1, 2, 3) tags=('nightly', 'axi')
mem_walk    seeds=(7,) tags=('nightly', 'mem')

tests loaded: 3
every seeds field is a tuple: True

JSON because Chapter 21.4 made it the machine-readable default and because a suite is read by a program far more often than it is edited. A YAML file with the same structure reads more pleasantly for the humans who maintain it, and Chapter 21.5 covered that trade — the loading code below is identical either way, which is the point of keeping the parse at the edge.

The lists become tuples on the way in. Chapter 25.1 Section 4's reason: a Test has to be hashable and immutable so a job built from it can be a dictionary key, and a list field makes a dataclass unhashable.

2. Filtering

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass


@dataclass(frozen=True)
class Test:
    name: str
    seeds: tuple
    tags: tuple = ()


TESTS = [Test("axi_smoke", (1, 2), ("smoke", "axi")),
         Test("axi_burst", (1, 2, 3), ("nightly", "axi")),
         Test("mem_walk", (7,), ("nightly", "mem"))]


def select(tests, *, tags=None, exclude=None):
    """Include if it carries ANY required tag; drop if it carries ANY excluded one."""
    chosen = []
    for t in tests:
        if tags and not set(tags) & set(t.tags):
            continue
        if exclude and set(exclude) & set(t.tags):
            continue
        chosen.append(t)
    return chosen


QUERIES = [{}, {"tags": ["smoke"]}, {"tags": ["nightly"]},
           {"tags": ["axi"], "exclude": ["smoke"]}, {"tags": ["gpu"]}]

for q in QUERIES:
    print(f"{str(q):<42} -> {[t.name for t in select(TESTS, **q)]}")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
{}                                         -> ['axi_smoke', 'axi_burst', 'mem_walk']
{'tags': ['smoke']}                        -> ['axi_smoke']
{'tags': ['nightly']}                      -> ['axi_burst', 'mem_walk']
{'tags': ['axi'], 'exclude': ['smoke']}    -> ['axi_burst']
{'tags': ['gpu']}                          -> []

Two decisions in that function, and both are the kind that get made silently and argued about later.

tags is ANY, not ALL. Asking for ["smoke", "nightly"] gives you both sets rather than their intersection, which is what people mean by a tag list almost every time. If you want ALL, that is a different parameter and it should be named — require_all_tags — not a different reading of the same one.

exclude wins over tags. A test carrying both axi and smoke is dropped by {"tags": ["axi"], "exclude": ["smoke"]}, because an exclusion is usually a safety rail — never run the destructive ones — and a safety rail that an inclusion can override is not one.

Both are written down here because neither is discoverable from a call site. Chapter 22.6's point about a command-line contract applies to a configuration contract identically.

3. The Suite Is Not the Job List

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass


@dataclass(frozen=True)
class Test:
    name: str
    seeds: tuple
    tags: tuple = ()


TESTS = [Test("axi_smoke", (1, 2), ("smoke", "axi")),
         Test("axi_burst", (1, 2, 3), ("nightly", "axi")),
         Test("mem_walk", (7,), ("nightly", "mem"))]

print("tests in the suite:", len(TESTS))
print("seeds per test    :", {t.name: len(t.seeds) for t in TESTS})
print("jobs to run       :", sum(len(t.seeds) for t in TESTS))
print()
print("Chapter 26.1's `total` is the second number:", sum(len(t.seeds) for t in TESTS))
print("a summary built on the first would be wrong by:",
      sum(len(t.seeds) for t in TESTS) - len(TESTS))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
tests in the suite: 3
seeds per test    : {'axi_smoke': 2, 'axi_burst': 3, 'mem_walk': 1}
jobs to run       : 6

Chapter 26.1's `total` is the second number: 6
a summary built on the first would be wrong by: 3

Three tests, six jobs. A test is a thing somebody wrote; a job is one execution of it at one seed, and Chapter 26.1's entire invariant counts the second. A regression that reports "3 tests" and runs six jobs has two numbers in circulation and will eventually print the wrong one.

Naming them differently throughout is the cheap defence — tests and jobs, never items or list.

4. The Expansion

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
jobs = [Job(t.name, s) for t in suite.tests for s in t.seeds]
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass


@dataclass(frozen=True)
class Test:
    name: str
    seeds: tuple
    tags: tuple = ()


@dataclass(frozen=True)
class Job:
    test: str
    seed: int


TESTS = [Test("axi_smoke", (1, 2), ("smoke", "axi")),
         Test("axi_burst", (1, 2, 3), ("nightly", "axi")),
         Test("mem_walk", (7,), ("nightly", "mem"))]

test_major = [Job(t.name, s) for t in TESTS for s in t.seeds]

all_seeds = sorted({s for t in TESTS for s in t.seeds})
seed_major = [Job(t.name, s) for s in all_seeds for t in TESTS if s in t.seeds]

print("test-major:", [(j.test, j.seed) for j in test_major])
print("seed-major:", [(j.test, j.seed) for j in seed_major])
print()
print("same jobs :", set(test_major) == set(seed_major))
print("same order:", test_major == seed_major)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
test-major: [('axi_smoke', 1), ('axi_smoke', 2), ('axi_burst', 1), ('axi_burst', 2), ('axi_burst', 3), ('mem_walk', 7)]
seed-major: [('axi_smoke', 1), ('axi_burst', 1), ('axi_smoke', 2), ('axi_burst', 2), ('axi_burst', 3), ('mem_walk', 7)]

same jobs : True
same order: False

The annotation is the first form, and it reads cleanly — for t ... for s in t.seeds is a nested comprehension of exactly the kind Chapter 12.6 endorses, where the second clause iterates something belonging to the first.

The second form is where that chapter's limit shows: for s in all_seeds for t in TESTS if s in t.seeds needs a precomputed seed set, an inner filter, and a reader who holds all three in mind. It is a plain loop's job.

But the order is not cosmetic, and the difference only matters at one moment:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
JOBS_TEST_MAJOR = [("axi_smoke", 1), ("axi_smoke", 2), ("axi_burst", 1),
                   ("axi_burst", 2), ("axi_burst", 3), ("mem_walk", 7)]
JOBS_SEED_MAJOR = [("axi_smoke", 1), ("axi_burst", 1), ("mem_walk", 7),
                   ("axi_smoke", 2), ("axi_burst", 2), ("axi_burst", 3)]

for label, jobs in (("test-major", JOBS_TEST_MAJOR), ("seed-major", JOBS_SEED_MAJOR)):
    done = jobs[:3]
    print(f"{label}: killed after 3 jobs")
    print(f"   ran            : {done}")
    print(f"   tests covered  : {sorted({t for t, _ in done})}")
    print(f"   tests untouched: {sorted({t for t, _ in jobs} - {t for t, _ in done})}")
    print()
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
test-major: killed after 3 jobs
   ran            : [('axi_smoke', 1), ('axi_smoke', 2), ('axi_burst', 1)]
   tests covered  : ['axi_burst', 'axi_smoke']
   tests untouched: ['mem_walk']

seed-major: killed after 3 jobs
   ran            : [('axi_smoke', 1), ('axi_burst', 1), ('mem_walk', 7)]
   tests covered  : ['axi_burst', 'axi_smoke', 'mem_walk']
   tests untouched: []

Killed at the halfway mark, seed-major has touched every test and test-major has not touched one at all.

That is the policy question the clause order answers without being asked. Seed-major gives breadth early — after one pass you know whether every test runs at all, which is what you want from a regression that might be cut short or whose early results trigger an alert. Test-major gives depth early — one test fully characterised across its seeds, which is what you want when chasing a specific flaky test.

Neither is right in general. What is wrong is choosing by accident, and Chapter 26.8's resume makes the consequence permanent: a regression that died and resumes will finish in whatever order the list was built in.

5. Nothing Matched

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass


@dataclass(frozen=True)
class Test:
    name: str
    seeds: tuple
    tags: tuple = ()


@dataclass(frozen=True)
class Job:
    test: str
    seed: int


TESTS = [Test("axi_smoke", (1, 2), ("smoke", "axi"))]
selected = [t for t in TESTS if "gpu" in t.tags]
jobs = [Job(t.name, s) for t in selected for s in t.seeds]

print("tests matching 'gpu':", len(selected))
print("jobs                :", jobs)
print("anything raised     :", "no")
print()
print("the regression then reports:")
print("   total   : 0")
print("   passed  : 0")
print("   failed  : 0")
print("   verdict : 'all tests passed'")
print()
print("which is indistinguishable from a clean nightly:", True)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
tests matching 'gpu': 0
jobs                : []
anything raised     : no

the regression then reports:
   total   : 0
   passed  : 0
   failed  : 0
   verdict : 'all tests passed'

which is indistinguishable from a clean nightly: True

Zero of zero passed. Technically true, catastrophically misleading, and produced by a typo in a tag name.

Chapter 26.1 Section 9 already refused to compute a pass rate from an empty regression, which stops the percentage being wrong. It does not stop a flow from running nothing and exiting zero. The guard belongs here, at expansion, where the message can name the cause:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass


@dataclass(frozen=True)
class Test:
    name: str
    seeds: tuple
    tags: tuple = ()


@dataclass(frozen=True)
class Job:
    test: str
    seed: int


class SuiteError(Exception):
    """The suite produced no work, or work that cannot be run."""


TESTS = [Test("axi_smoke", (1, 2), ("smoke", "axi")),
         Test("axi_burst", (1, 2, 3), ("nightly", "axi")),
         Test("no_seeds", (), ("nightly",))]


def expand(tests, *, tags=None, min_jobs=1):
    known = sorted({tag for t in tests for tag in t.tags})
    if tags:
        unknown = sorted(set(tags) - set(known))
        if unknown:
            raise SuiteError(f"unknown tag(s) {unknown} — the suite defines {known}")
    chosen = [t for t in tests if not tags or set(tags) & set(t.tags)]
    empty = [t.name for t in chosen if not t.seeds]
    if empty:
        raise SuiteError(f"test(s) {empty} have no seeds — give them one or remove them")
    jobs = [Job(t.name, s) for t in chosen for s in t.seeds]
    if len(jobs) < min_jobs:
        raise SuiteError(f"{len(jobs)} job(s) selected, expected at least {min_jobs}")
    return jobs


print("ok        :", len(expand(TESTS, tags=["axi"])), "jobs")
for q in ({"tags": ["gpu"]}, {"tags": ["nightly"]}, {"tags": ["smoke"], "min_jobs": 50}):
    try:
        expand(TESTS, **q)
    except SuiteError as exc:
        print(f"{str(q)[:28]:<28} SuiteError: {exc}")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
ok        : 5 jobs
{'tags': ['gpu']}            SuiteError: unknown tag(s) ['gpu'] — the suite defines ['axi', 'nightly', 'smoke']
{'tags': ['nightly']}        SuiteError: test(s) ['no_seeds'] have no seeds — give them one or remove them
{'tags': ['smoke'], 'min_job SuiteError: 2 job(s) selected, expected at least 50

Three distinct guards, three distinct messages, and the first is the one that earns its keep.

An unknown tag is caught before anything else, and the message lists the tags the suite does define. A typo'd --tag nitely then produces a usable error instead of a silent zero-job run, which is Chapter 22.3's argument about a CLI rejecting the unknown case, applied to a config value.

A test with no seeds is a suite bug, not an empty selection. It contributes nothing and nobody notices, so it is worth naming separately — Chapter 21.7's validate-at-the-boundary again.

min_jobs catches the plausible-but-wrong selection. A nightly that normally runs four hundred jobs and today selected two has a problem that zero-checking would miss entirely. It is Chapter 25.2 Section 6's min_files, in a different costume and for the same reason.

6. Duplicates

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from collections import Counter
from dataclasses import dataclass


@dataclass(frozen=True)
class Test:
    name: str
    seeds: tuple
    tags: tuple = ()


@dataclass(frozen=True)
class Job:
    test: str
    seed: int


TESTS = [Test("axi_smoke", (1, 2), ("smoke",)),
         Test("axi_burst", (1, 2, 3), ("nightly",)),
         Test("axi_smoke", (1,), ("regression",))]

jobs = [Job(t.name, s) for t in TESTS for s in t.seeds]
counts = Counter(jobs)
dupes = {j: n for j, n in counts.items() if n > 1}

print("jobs produced :", len(jobs))
print("unique jobs   :", len(set(jobs)))
print("duplicated    :", [(j.test, j.seed, n) for j, n in dupes.items()])
print()
print("if total comes from len(jobs):", len(jobs))
print("if results are keyed by job  :", len(set(jobs)))
print("the invariant then fails by  :", len(jobs) - len(set(jobs)))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
jobs produced : 6
unique jobs   : 5
duplicated    : [('axi_smoke', 1, 2)]

if total comes from len(jobs): 6
if results are keyed by job  : 5
the invariant then fails by  : 1

Two suite entries overlapped and (axi_smoke, 1) is in the list twice. It happens whenever a test appears in two groups — a smoke list and a regression list — which is normal suite maintenance rather than a mistake.

The consequence is specific and nasty. total would be 6; the results, keyed by job for Chapter 26.8's resume, would be 5; and Chapter 26.1's invariant fails by one with no job missing. An apparently lost job that was never there.

Two defensible resolutions, and the wrong move is to pick one silently:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass


@dataclass(frozen=True)
class Job:
    test: str
    seed: int


RAW = [Job("axi_smoke", 1), Job("axi_smoke", 2), Job("axi_burst", 1),
       Job("axi_smoke", 1)]


def dedupe(jobs):
    """Keep first occurrence, preserve order — dict keys are insertion-ordered."""
    return list(dict.fromkeys(jobs))


unique = dedupe(RAW)
print("raw    :", [(j.test, j.seed) for j in RAW])
print("deduped:", [(j.test, j.seed) for j in unique])
print("order preserved:", unique[0] == RAW[0] and unique[-1] == RAW[2])
print("dropped:", len(RAW) - len(unique))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
raw    : [('axi_smoke', 1), ('axi_smoke', 2), ('axi_burst', 1), ('axi_smoke', 1)]
deduped: [('axi_smoke', 1), ('axi_smoke', 2), ('axi_burst', 1)]
order preserved: True
dropped: 1

Deduplicate and report the count is the right default: running the same test at the same seed twice produces the same result twice, which wastes a slot and adds nothing. dict.fromkeys does it in one expression and preserves order, because dictionaries have been insertion-ordered since 3.7 and set() would not.

The alternative — treat a duplicate as a suite error — is defensible for a tightly curated suite and intolerable for one assembled from several group files. What is not defensible is dropping one silently, which is why dropped is reported.

7. A Suite Composed of Several Files

Azvya Education Pvt. Ltd.VLSI Mentor
groups.json
{
  "groups": {
    "smoke":     {"tests": ["axi_smoke", "mem_smoke"], "seeds": [1]},
    "axi_full":  {"tests": ["axi_smoke", "axi_burst"], "seeds": [1, 2, 3]},
    "overnight": {"include": ["axi_full"], "tests": ["mem_walk"], "seeds": [1, 2]}
  }
}
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import json
from pathlib import Path


class SuiteError(Exception):
    """A group definition that cannot be resolved."""


def resolve(groups, name, seen=None):
    """Flatten a group and everything it includes. Detects cycles."""
    seen = seen or []
    if name in seen:
        raise SuiteError(f"group cycle: {' -> '.join(seen + [name])}")
    if name not in groups:
        raise SuiteError(f"unknown group {name!r} — known: {sorted(groups)}")
    g = groups[name]
    out = {t: tuple(g.get("seeds", (1,))) for t in g.get("tests", ())}
    for included in g.get("include", ()):
        for test, seeds in resolve(groups, included, seen + [name]).items():
            out.setdefault(test, seeds)
    return out


GROUPS = json.loads(Path("groups.json").read_text(encoding="utf-8"))["groups"]

for name in ("smoke", "axi_full", "overnight"):
    flat = resolve(GROUPS, name)
    jobs = sum(len(s) for s in flat.values())
    print(f"{name:<11} {len(flat)} test(s), {jobs} job(s): "
          f"{ {t: len(s) for t, s in sorted(flat.items())} }")

print()
GROUPS["a"] = {"include": ["b"]}
GROUPS["b"] = {"include": ["a"]}
for bad in ("a", "nosuch"):
    try:
        resolve(GROUPS, bad)
    except SuiteError as exc:
        print("SuiteError:", exc)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
smoke       2 test(s), 2 job(s): {'axi_smoke': 1, 'mem_smoke': 1}
axi_full    2 test(s), 6 job(s): {'axi_burst': 3, 'axi_smoke': 3}
overnight   3 test(s), 8 job(s): {'axi_burst': 3, 'axi_smoke': 3, 'mem_walk': 2}

SuiteError: group cycle: a -> b -> a
SuiteError: unknown group 'nosuch' — known: ['a', 'axi_full', 'b', 'overnight', 'smoke']

Real suites are not one flat list. They are a handful of named groups — a smoke set, a per-block set, an overnight set — and the overnight set is usually the others plus a few more, which is where the interesting behaviour lives.

Three decisions in resolve, and the second is the one that decides what overnight actually means.

A cycle raises and names the path. a -> b -> a is immediately diagnosable; the alternative is a RecursionError whose traceback names no group at all. Group files are edited by many people and a cycle is a matter of time.

setdefault means the outer group's own seeds win. overnight declares seeds: [1, 2] and includes axi_full, which declares three — and axi_smoke keeps three, because it arrived from the include with seeds already attached. The outer [1, 2] applies only to mem_walk, the test overnight declares itself.

That is a defensible policy and it is not the only one: the alternative, where the including group's seeds override everything it pulls in, is equally defensible and gives overnight eight jobs of a different shape. What is not defensible is leaving it implicit, because the two differ by four jobs on a three-group file and by hundreds on a real one.

An unknown group lists the known ones, which is Chapter 26.2 Section 5's unknown-tag message applied one level up — a typo'd group name in a nightly's configuration otherwise produces a silent zero-job run.

8. The Expansion, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor
suite.json
{
  "defaults": {"seeds": [1]},
  "tests": [
    {"name": "axi_smoke", "seeds": [1, 2], "tags": ["smoke", "axi"]},
    {"name": "axi_burst", "seeds": [1, 2, 3], "tags": ["nightly", "axi"]},
    {"name": "mem_walk", "tags": ["nightly", "mem"]},
    {"name": "axi_smoke", "seeds": [1], "tags": ["regression", "axi"]}
  ]
}
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import json
from dataclasses import dataclass
from pathlib import Path


class SuiteError(Exception):
    """The suite produced no work, or work that cannot be run."""


@dataclass(frozen=True)
class Test:
    name: str
    seeds: tuple
    tags: tuple = ()


@dataclass(frozen=True)
class Job:
    test: str
    seed: int


def load_suite(path):
    raw = json.loads(Path(path).read_text(encoding="utf-8"))
    default_seeds = tuple(raw.get("defaults", {}).get("seeds", (1,)))
    tests = []
    for entry in raw["tests"]:
        if "name" not in entry:
            raise SuiteError(f"a test entry has no name: {entry}")
        tests.append(Test(entry["name"],
                          tuple(entry.get("seeds", default_seeds)),
                          tuple(entry.get("tags", ()))))
    return tests


def expand(tests, *, tags=None, exclude=None, min_jobs=1):
    known = sorted({tag for t in tests for tag in t.tags})
    for group in (tags, exclude):
        unknown = sorted(set(group or ()) - set(known))
        if unknown:
            raise SuiteError(f"unknown tag(s) {unknown} — the suite defines {known}")

    chosen = [t for t in tests
              if (not tags or set(tags) & set(t.tags))
              and not (exclude and set(exclude) & set(t.tags))]

    empty = sorted({t.name for t in chosen if not t.seeds})
    if empty:
        raise SuiteError(f"test(s) {empty} have no seeds")

    raw_jobs = [Job(t.name, s) for t in chosen for s in t.seeds]
    jobs = list(dict.fromkeys(raw_jobs))
    if len(jobs) < min_jobs:
        raise SuiteError(f"{len(jobs)} job(s) selected, expected at least {min_jobs}")
    return jobs, {"tests_selected": len(chosen), "jobs_raw": len(raw_jobs),
                  "duplicates_dropped": len(raw_jobs) - len(jobs)}


tests = load_suite("suite.json")
jobs, stats = expand(tests, tags=["axi"])

print("tests in file      :", len(tests))
print("stats              :", stats)
print("jobs (the total)   :", len(jobs))
for j in jobs:
    print(f"   {j.test:<11} seed {j.seed}")
print()
print("mem_walk used the default seed:", Test("mem_walk", (1,)) in
      [Test(t.name, t.seeds) for t in tests])
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
tests in file      : 4
stats              : {'tests_selected': 3, 'jobs_raw': 6, 'duplicates_dropped': 1}
jobs (the total)   : 5
   axi_smoke   seed 1
   axi_smoke   seed 2
   axi_burst   seed 1
   axi_burst   seed 2
   axi_burst   seed 3

mem_walk used the default seed: True

Four entries, three selected, six raw jobs, one duplicate dropped, five jobs to run — and that final 5 is Chapter 26.1's total, the number the entire invariant is reconciled against.

Three things carry forward.

defaults.seeds means a test can omit the field, which is Chapter 21.6's layering: the suite states the common case once and each test overrides only what differs. mem_walk has no seeds key and gets (1,).

stats is returned alongside the jobs. Chapter 24.4 Section 6's argument — a destructive or lossy step should report what it did — and duplicates_dropped: 1 is exactly the line somebody needs when the job count does not match their expectation.

The dedupe happens after the filter, not before. Filtering first means a test excluded by tag cannot contribute a duplicate, and the reported count then reflects the selection rather than the whole file.

9. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
from dataclasses import dataclass


@dataclass(frozen=True)
class T:
    name: str
    seeds: tuple
    tags: tuple = ()


@dataclass(frozen=True)
class J:
    test: str
    seed: int


TESTS = [T("a", (1, 2), ("x",)), T("b", (3,), ("y",)), T("a", (1,), ("y",))]

# A
print("A:", len(TESTS), sum(len(t.seeds) for t in TESTS))

# B
jobs = [J(t.name, s) for t in TESTS for s in t.seeds]
print("B:", [(j.test, j.seed) for j in jobs])

# C
print("C:", len(jobs), len(set(jobs)))

# D
print("D:", [(j.test, j.seed) for j in dict.fromkeys(jobs)])

# E
print("E:", [t.name for t in TESTS if {"y"} & set(t.tags)])

# F
print("F:", [t.name for t in TESTS if not ({"y"} & set(t.tags))])

# G
print("G:", sorted({tag for t in TESTS for tag in t.tags}))

# H
print("H:", {J("a", 1): "PASS"}[J("a", 1)])

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
A: 3 4
B: [('a', 1), ('a', 2), ('b', 3), ('a', 1)]
C: 4 3
D: [('a', 1), ('a', 2), ('b', 3)]
E: ['b', 'a']
F: ['a']
G: ['x', 'y']
H: PASS

A — three tests, four jobs. The suite is not the job list.

B — the expansion, in test-major order, with a duplicate at the end.

C — four produced, three unique.

D — dict.fromkeys deduplicates and preserves first-seen order.

E — an ANY-tag filter; both the b entry and the second a entry carry y.

F — and its complement is not the same set, because a appears twice with different tags.

G — the known tag set, which is what an unknown-tag message should list.

H — a frozen Job is hashable, which Chapter 26.8 depends on.

10. Exercises

Exercise 1 — load it. Write your suite as JSON and load it into frozen dataclasses. Confirm the sequence fields became tuples.

Exercise 2 — why tuples. Say what a list field would prevent, citing Chapter 26.8.

Exercise 3 — ANY or ALL. Decide which your tag filter means, and say how you would offer the other.

Exercise 4 — exclusion wins. Construct a test carrying both an included and an excluded tag. Justify the outcome.

Exercise 5 — two numbers. Report your suite's test count and job count. Say which is Chapter 26.1's total.

Exercise 6 — naming. Find a variable in your flow that could hold either, and rename it.

Exercise 7 — both orders. Produce test-major and seed-major job lists and show they are the same set.

Exercise 8 — killed halfway. For each order, report which tests a regression cut short has touched.

Exercise 9 — choose. Say which order your nightly should use and why, in two sentences.

Exercise 10 — zero of zero. Produce an empty selection and show the regression reporting success.

Exercise 11 — the unknown tag. Implement the check and make the message list the known tags.

Exercise 12 — no seeds. Add a test with an empty seed list and show it caught separately.

Exercise 13 — min_jobs. Pick a threshold for your nightly and say what it catches that zero-checking does not.

Exercise 14 — the overlap. Put one test in two groups and show the duplicate.

Exercise 15 — the phantom loss. Explain how a duplicate breaks Chapter 26.1's invariant with no job actually missing.

Exercise 16 — dedupe in order. Use dict.fromkeys and say why set() would not do.

Exercise 17 — report the drop. Say why duplicates_dropped belongs in the output.

Exercise 18 — the other policy. Argue for treating a duplicate as an error, and say for which kind of suite.

Exercise 19 — defaults. Add a defaults.seeds to your suite and a test that omits the field.

Exercise 20 — groups. Write your suite as named groups with one group including another.

Exercise 21 — the cycle. Create one and confirm your resolver names the path rather than raising RecursionError.

Exercise 22 — whose seeds win. Decide the policy for an including group's seeds, implement it, and count the jobs both ways.

Exercise 23 — flatten at the edge. Say what the rest of your flow should never have to know about.

Exercise 24 — filter then dedupe. Say what changes if you reverse those two steps.

11. Summary

  • Keep the suite as data — JSON by Chapter 21.4's default — and convert sequences to tuples on the way in so the records stay hashable.
  • A tag filter means ANY by default; offer ALL as a separately named option rather than a different reading.
  • Exclusion wins over inclusion, because an exclusion is usually a safety rail.
  • Three tests are six jobs. The suite is not the job list, and Chapter 26.1's total is the second number.
  • Name them tests and jobs throughout, or the wrong one eventually gets printed.
  • The comprehension's clause order decides the job order, and the two orders produce the same set.
  • Seed-major gives breadth early; test-major gives depth early — and the difference only shows when a regression is cut short.
  • An empty selection raises nothing and reports zero of zero passed, which is indistinguishable from a clean nightly.
  • Reject an unknown tag and list the known ones — a typo'd tag must not produce a silent zero-job run.
  • A test with no seeds is a suite bug, worth its own message.
  • min_jobs catches the plausible-but-wrong selection that zero-checking misses.
  • Two suite entries can produce the same job twice, which breaks the invariant with no job actually missing.
  • Deduplicate with dict.fromkeys to preserve order, and report how many were dropped.
  • Layer defaults.seeds under each test's own, so the common case is stated once.
  • Real suites are named groups, and one group usually includes another — flatten them once at the edge so nothing downstream sees a group.
  • A group cycle must name the path, or a RecursionError names nothing.
  • Decide explicitly whose seeds win when an including group declares its own; the two policies differ by hundreds of jobs on a real file.
  • An unknown group name lists the known ones, like an unknown tag.
  • Filter before deduplicating, so the reported count reflects the selection.

The next chapter decides what those seeds should be, and how to keep every one of them recoverable.

Where this fits

Part of the Python curriculum.