Python Part II · Module 34 · From Script to Maintainable Tool

From One File to Modules Others Can Import

Chapter 34.1 measured a script in which zero of three functions could be called from a test, and found the reason was structural: importing the file ran the regression. This chapter performs the first and most consequential refactoring — one file becomes a package — and the interesting part is not the mechanics but what the import system actually does when you ask for one piece of it. Two of the measurements here contradict what the usual advice implies.

Advanced23 min readPythonPart IIAutomationPackagesImports

Part II · Module 34 · Chapter 34.2 · From One File to Modules Others Can Import

1. What One File Costs

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

WANTS = [
    ("reuse the parser in a reporting script", "import the file",
     "runs the regression"),
    ("test the severity counting", "import the file",
     "runs the regression"),
    ("call the classifier from a notebook", "import the file",
     "runs the regression"),
    ("run the regression", "run the file", "works"),
]

print(f"{'what you want':<42} {'what you must do':<20} what happens")
for want, how, result in WANTS:
    print(f"{want:<42} {how:<20} {result}")

print()
print("uses the one file supports:", 1)
print("uses it blocks             :", 3)
print("and the blocker is         :", "six calls at module level — 34.1 §3")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

what you want                              what you must do     what happens
reuse the parser in a reporting script     import the file      runs the regression
test the severity counting                 import the file      runs the regression
call the classifier from a notebook        import the file      runs the regression
run the regression                         run the file         works

One supported use, three blocked ones, and all three are blocked by the same thing: the file has no entry point, so importing it is running it. Chapter 34.1 Section 9 measured that directly — the unguarded module printed three lines on import where the guarded one printed none, and both exposed the same function.

2. The Annotation

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

from flow.parse import parse_sim_log
from flow.tools import run_sim

Two lines, and they encode three decisions that are worth separating before building anything. There is a package called flow, so the names do not collide with anything else on the path. There are separate modules for parsing and for tool control, which is Chapter 34.3's three-layer split beginning. And the import is by name, not by file path, so the caller does not need to know where the package lives.

The decision the two lines do not state is the one Section 5 is about: which of parse and tools is allowed to import the other.

3. The Package

Here is the split. Four files, and the whole of Chapter 34.1's specimen logic is in them — with the print statements at the top of each module present on purpose, so Section 4 can measure what gets executed:

Azvya Education Pvt. Ltd.VLSI Mentor

flow/__init__.py

"""flow - the axi_fifo regression toolkit."""
print("[import] flow/__init__.py")
__version__ = "0.1.0"

Azvya Education Pvt. Ltd.VLSI Mentor

flow/parse.py

"""Text in, facts out. No files, no tools, no printing."""
print("[import] flow/parse.py")

import re

SEVERITIES = ("UVM_FATAL", "UVM_ERROR", "UVM_WARNING")
TESTNAME = re.compile(r"UVM_TESTNAME=(\S+)")
SEED = re.compile(r"ntb_random_seed=(\d+)")


def count_severities(text):
    """Count real severity lines, excluding the UVM_ERROR : summary line."""
    counts = dict.fromkeys(SEVERITIES, 0)
    for line in text.splitlines():
        for sev in SEVERITIES:
            if line.startswith(sev) and f"{sev} :" not in line:
                counts[sev] += 1
    return counts


def classify(counts, text):
    """The four-way verdict, as a pure function of what was counted."""
    if "TEST DONE" not in text:
        return "TIMEOUT"
    if counts["UVM_FATAL"]:
        return "FATAL"
    if counts["UVM_ERROR"]:
        return "FAIL"
    return "PASS"


def parse_sim_log(text):
    """The module's one public entry point."""
    counts = count_severities(text)
    m, s = TESTNAME.search(text), SEED.search(text)
    return {
        "test": m.group(1) if m else None,
        "seed": int(s.group(1)) if s else None,
        "errors": counts["UVM_ERROR"],
        "fatals": counts["UVM_FATAL"],
        "warnings": counts["UVM_WARNING"],
        "status": classify(counts, text),
    }

Azvya Education Pvt. Ltd.VLSI Mentor

flow/tools.py

"""Everything that touches the world: processes and the filesystem."""
print("[import] flow/tools.py")

import subprocess
import sys
from pathlib import Path


def sim_command(sim, filelist, test, seed, log):
    """Pure: build the argument vector. Returns a list, never a string."""
    return [sim, "-f", str(filelist), f"+UVM_TESTNAME={test}",
            f"+ntb_random_seed={seed}", "-l", str(log)]


def run_sim(sim, filelist, test, seed, run_dir):
    """Impure: launch the tool and return (returncode, log_path)."""
    run_dir = Path(run_dir)
    run_dir.mkdir(parents=True, exist_ok=True)
    log = run_dir / "sim.log"
    argv = sim_command(sim, filelist, test, seed, log)
    cp = subprocess.run(argv, cwd=run_dir, stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE, text=True)
    return cp.returncode, log


def read_log(log):
    """Impure: the only place the parser's input comes from."""
    p = Path(log)
    if not p.is_file():
        return None
    return p.read_text(encoding="utf-8", errors="replace")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import sys
from pathlib import Path

sys.path.insert(0, ".")

print("--- importing only the parser")
from flow.parse import parse_sim_log

LOG = ("UVM_INFO @0 UVM_TESTNAME=axi_burst_test ntb_random_seed=4121\n"
       "UVM_WARNING drv.sv(9) late\n"
       "UVM_ERROR drv.sv(88) data mismatch\n"
       "UVM_ERROR mon.sv(12) crc\n"
       "UVM_ERROR :    2\n"
       "TEST DONE\n")

print()
print("parse_sim_log(LOG):")
for k, v in parse_sim_log(LOG).items():
    print(f"  {k:<10} {v!r}")
print()
print("flow.tools in sys.modules:", "flow.tools" in sys.modules)
print("flow.parse in sys.modules:", "flow.parse" in sys.modules)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

--- importing only the parser
[import] flow/__init__.py
[import] flow/parse.py

parse_sim_log(LOG):
  test       'axi_burst_test'
  seed       4121
  errors     2
  fatals     0
  warnings   1
  status     'FAIL'

flow.tools in sys.modules: False
flow.parse in sys.modules: True

The two [import] lines appear exactly where the import statement sits. from flow.parse import ... is an ordinary executable statement, not a declaration hoisted to the top, so it runs after the print above it — which is worth seeing once, because it is the reason the usual convention of putting every import at the top of a file means every imported module's side effects happen before your program's first line. Chapter 34.1's specimen is what that costs when those side effects are a regression run.

The facts this section exists for are the last two lines, and Section 4 is about nothing else.

4. Importing the Parser Does Not Import the Simulator

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import sys
from pathlib import Path

for d, files in {
    "flow": {
        "__init__.py": '"""flow."""\nprint("[import] flow/__init__.py")\n',
        "parse.py": ('print("[import] flow/parse.py")\n'
                     'def parse_sim_log(text):\n'
                     '    return {"errors": text.count("UVM_ERROR")}\n'),
        "tools.py": ('print("[import] flow/tools.py")\n'
                     'import subprocess\n'
                     'def run_sim(*a):\n'
                     '    return 0\n'),
        "report.py": ('print("[import] flow/report.py")\n'
                      'def to_csv(rows):\n'
                      '    return "test,status\\n"\n'),
    },
}.items():
    Path(d).mkdir(exist_ok=True)
    for name, body in files.items():
        (Path(d) / name).write_text(body, encoding="utf-8")

sys.path.insert(0, ".")

from flow.parse import parse_sim_log

MODULES = ["flow", "flow.parse", "flow.tools", "flow.report"]
print()
print(f"{'module':<14} loaded by `from flow.parse import ...`")
for m in MODULES:
    print(f"{m:<14} {m in sys.modules}")

print()
loaded = sum(1 for m in MODULES if m in sys.modules)
print("modules in the package :", len(MODULES))
print("modules actually loaded:", loaded)
print("modules NOT loaded     :", len(MODULES) - loaded)
print("what that means        :",
      "a reporting script pays for the parser only")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[import] flow/__init__.py
[import] flow/parse.py

module         loaded by `from flow.parse import ...`
flow           True
flow.parse     True
flow.tools     False
flow.report    False

modules in the package : 4
modules actually loaded: 2
modules NOT loaded     : 2
what that means        : a reporting script pays for the parser only

Two of four modules load. flow/__init__.py runs because importing anything inside a package initialises the package first; flow/parse.py runs because that is what was asked for; and flow/tools.py and flow/report.py are never touched.

That is the property that makes the annotation's two lines worth writing as two lines. A script that only needs to re-parse yesterday's logs imports flow.parse and gets the parser — not subprocess, not the filesystem helpers, and crucially not any module-level work those files might do.

5. One Direction of Dependency

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

LAYERS = [
    ("flow.parse", "text -> facts", [], "pure"),
    ("flow.report", "facts -> text/files", ["flow.parse"], "mostly pure"),
    ("flow.tools", "world -> text, status", [], "impure"),
    ("run_regress.py", "the command line", ["flow.parse", "flow.tools",
                                            "flow.report"], "impure"),
]

print(f"{'module':<18} {'job':<22} {'imports':<42} purity")
for name, job, deps, purity in LAYERS:
    print(f"{name:<18} {job:<22} {', '.join(deps) or '-':<42} {purity}")

print()
print("modules importing flow.tools:", 1)
print("modules importing flow.parse:", 2)
print("modules flow.parse imports  :", 0)
print()
print("the rule:", "a lower layer never imports a higher one")
print("so the direction is:", "run_regress -> report -> parse, tools -> nothing")
print("and the bottom of the graph is:", "flow.parse, which imports nothing")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

module             job                    imports                                    purity
flow.parse         text -> facts          -                                          pure
flow.report        facts -> text/files    flow.parse                                 mostly pure
flow.tools         world -> text, status  -                                          impure
run_regress.py     the command line       flow.parse, flow.tools, flow.report        impure

modules importing flow.tools: 1
modules importing flow.parse: 2
modules flow.parse imports  : 0

the rule: a lower layer never imports a higher one
so the direction is: run_regress -> report -> parse, tools -> nothing
and the bottom of the graph is: flow.parse, which imports nothing

flow.parse imports nothing from the package. That is not an accident of this example, it is the design rule: the module holding the pure logic sits at the bottom and depends on nobody, so it can be imported by anything, tested without setup, and reused by a script that has never heard of a simulator.

flow.tools also imports nothing from the package, which is worth noticing because it is impure. Purity and position are different axes: tools is at the bottom because nothing in flow needs it, not because it is clean.

The direction is what Chapter 34.3 formalises with a diagram. Here the rule is enough: a module may import from a layer below it and never from one above. Break that and you get Section 6.

6. A Circular Import Blames the Victim

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import sys
from pathlib import Path

Path("circ").mkdir(exist_ok=True)
(Path("circ") / "__init__.py").write_text("", encoding="utf-8")
(Path("circ") / "parse.py").write_text(
    "from circ.report import format_row\n"
    "\n"
    "def parse_sim_log(text):\n"
    "    return {'errors': text.count('UVM_ERROR')}\n", encoding="utf-8")
(Path("circ") / "report.py").write_text(
    "from circ.parse import parse_sim_log\n"
    "\n"
    "def format_row(r):\n"
    "    return 'ok'\n", encoding="utf-8")

sys.path.insert(0, ".")


def try_import(statement):
    """Import in a fresh interpreter state so order is what we control."""
    for name in [k for k in sys.modules if k.startswith("circ")]:
        del sys.modules[name]
    try:
        exec(statement, {})
        return "OK", ""
    except ImportError as exc:
        msg = str(exc)
        return type(exc).__name__, msg[:msg.index(" (")] if " (" in msg else msg


for stmt in ("from circ.parse import parse_sim_log",
             "from circ.report import format_row"):
    kind, msg = try_import(stmt)
    print(f"  {stmt}")
    print(f"    -> {kind}: {msg}")

print()
print("the cycle is              :", "parse -> report -> parse")
print("the module named in the   ")
print("  error is always         :", "the one you imported first")
print("so the name in the message:", "is the victim, not the cause")
print("how you actually find it  :", "draw the direction — §5")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  from circ.parse import parse_sim_log
    -> ImportError: cannot import name 'parse_sim_log' from partially initialized module 'circ.parse'
  from circ.report import format_row
    -> ImportError: cannot import name 'format_row' from partially initialized module 'circ.report'

the cycle is              : parse -> report -> parse
the module named in the   
  error is always         : the one you imported first
so the name in the message: is the victim, not the cause
how you actually find it  : draw the direction — §5

The same cycle produces two different error messages, and each one names whichever module you asked for. Import circ.parse and the error is about parse_sim_log; import circ.report and it is about format_row — from the identical, unchanged pair of files.

That matters because of how people debug it. The message says "cannot import name parse_sim_log from partially initialized module circ.parse", which reads as an accusation against parse.py, and parse.py is fine. The offending line is the from circ.report import format_row at its top, and the only reliable way to find it is to look at the direction of every import in the package and find the pair that points both ways.

The phrase "partially initialized module" is the real clue. Python had started executing circ/parse.py, reached line 1, gone off to execute circ/report.py, which asked for a name from circ.parse — a module that is on the stack and not yet finished, so the name does not exist yet. Nothing is wrong with either file in isolation.

The fix is never "import it inside the function", although that works. It is to decide which layer is lower. Here report formats facts, so it depends on parse; parse has no business knowing how its output is formatted, and deleting its import removes the cycle and improves the design at the same time.

7. Does `init.py` Matter?

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import sys
from pathlib import Path

# with __init__.py
Path("withinit").mkdir(exist_ok=True)
(Path("withinit") / "__init__.py").write_text(
    '"""withinit."""\n', encoding="utf-8")
(Path("withinit") / "parse.py").write_text(
    "def parse_sim_log(t):\n    return t.count('UVM_ERROR')\n",
    encoding="utf-8")

# without __init__.py
Path("noinit").mkdir(exist_ok=True)
(Path("noinit") / "parse.py").write_text(
    "def parse_sim_log(t):\n    return t.count('UVM_ERROR')\n",
    encoding="utf-8")

sys.path.insert(0, ".")

for name in ("withinit", "noinit"):
    mod = __import__(f"{name}.parse", fromlist=["parse_sim_log"])
    pkg = sys.modules[name]
    print(f"  {name}")
    print(f"    the import worked      : {mod.parse_sim_log('UVM_ERROR x') == 1}")
    print(f"    pkg.__file__ is None   : {pkg.__file__ is None}")
    print(f"    type(pkg.__path__)     : {type(pkg.__path__).__name__}")

print()
print("so __init__.py is required for the import:", False)
print("what breaks without it:", "Path(pkg.__file__).parent")
print("which code does that  :", "anything locating a bundled data file")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  withinit
    the import worked      : True
    pkg.__file__ is None   : False
    type(pkg.__path__)     : list
  noinit
    the import worked      : True
    pkg.__file__ is None   : True
    type(pkg.__path__)     : _NamespacePath

so __init__.py is required for the import: False
what breaks without it: Path(pkg.__file__).parent
which code does that  : anything locating a bundled data file

The import works either way. A directory without __init__.py becomes a namespace package, which is a deliberate feature and not a mistake — so the common claim that a package "needs" __init__.py is false, and a team that believes it will waste an afternoon on an import problem that has a different cause.

The measured consequence is the one worth knowing. A namespace package's __file__ is None and its __path__ is a _NamespacePath rather than a list, so any code written as Path(flow.__file__).parent / "schema.json" — the normal way to find a data file shipped beside the code — raises TypeError on a None argument. That is the real argument for writing the file: not that the import requires it, but that half the ecosystem assumes __file__ is a string.

Two smaller reasons to write it anyway. It gives the package a docstring and a __version__, which Chapter 34.6 needs for handover. And it makes the directory's intent explicit to a reader who is deciding whether a stray folder is a package or a pile of scripts.

8. What `run_regress.py` Becomes

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import sys
from pathlib import Path

Path("flow").mkdir(exist_ok=True)
(Path("flow") / "__init__.py").write_text("", encoding="utf-8")
(Path("flow") / "parse.py").write_text(
    "def parse_sim_log(text):\n"
    "    errs = sum(1 for l in text.splitlines()\n"
    "               if l.startswith('UVM_ERROR') and 'UVM_ERROR :' not in l)\n"
    "    done = 'TEST DONE' in text\n"
    "    return {'errors': errs,\n"
    "            'status': 'TIMEOUT' if not done else\n"
    "                      ('FAIL' if errs else 'PASS')}\n", encoding="utf-8")
(Path("flow") / "report.py").write_text(
    "def to_csv(rows):\n"
    "    out = ['test,seed,status,errors']\n"
    "    for r in rows:\n"
    "        out.append(f\"{r['test']},{r['seed']},{r['status']},{r['errors']}\")\n"
    "    return '\\n'.join(out) + '\\n'\n", encoding="utf-8")

Path("run_regress.py").write_text(
    '"""run_regress.py - run a suite; exit 0 only if every job passed."""\n'
    "import sys\n"
    "from pathlib import Path\n"
    "\n"
    "from flow.parse import parse_sim_log\n"
    "from flow.report import to_csv\n"
    "\n"
    "TESTS = ['axi_burst_test', 'axi_lite_test']\n"
    "SEEDS = [4121, 7734]\n"
    "\n"
    "\n"
    "def collect(logs):\n"
    "    rows = []\n"
    "    for (test, seed), text in logs.items():\n"
    "        r = parse_sim_log(text)\n"
    "        r['test'], r['seed'] = test, seed\n"
    "        rows.append(r)\n"
    "    return rows\n"
    "\n"
    "\n"
    "def main(logs):\n"
    "    rows = collect(logs)\n"
    "    Path('summary.csv').write_text(to_csv(rows), encoding='utf-8')\n"
    "    failed = [r for r in rows if r['status'] != 'PASS']\n"
    "    for r in rows:\n"
    "        print(f\"{r['test']} seed={r['seed']} -> {r['status']}\")\n"
    "    print(f'{len(rows) - len(failed)}/{len(rows)} passed')\n"
    "    return 1 if failed else 0\n"
    "\n"
    "\n"
    "if __name__ == '__main__':\n"
    "    sys.exit(main({}))\n", encoding="utf-8")

sys.path.insert(0, ".")
import run_regress

GOOD = "UVM_INFO a\nTEST DONE\n"
BAD = "UVM_INFO a\nUVM_ERROR drv.sv(88) x\nUVM_ERROR :    1\nTEST DONE\n"
LOGS = {("axi_burst_test", 4121): BAD, ("axi_lite_test", 7734): GOOD}

rc = run_regress.main(LOGS)
print()
print("exit code main() returned :", rc)
print("summary.csv written       :", Path("summary.csv").is_file())
print()
print(Path("summary.csv").read_text(encoding="utf-8"), end="")
print()
print("rows reaching the report  :", len(run_regress.collect(LOGS)))
print("34.1's RESULTS bug is now :", "impossible — collect RETURNS the rows")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

axi_burst_test seed=4121 -> FAIL
axi_lite_test seed=7734 -> PASS
1/2 passed

exit code main() returned : 1
summary.csv written       : True

test,seed,status,errors
axi_burst_test,4121,FAIL,1
axi_lite_test,7734,PASS,0

rows reaching the report  : 2
34.1's RESULTS bug is now : impossible — collect RETURNS the rows

Two rows reach the report and the exit code is 1. Compare that with Chapter 34.1 Section 6, where twelve simulations produced zero rows and an exit code of 0.

The structural reason is in the last line and it is worth stating plainly: collect returns its rows. There is no global for the caller to forget to fill, because the data travels by return value, and main cannot write the report without having received something. The bug is not fixed — it is unrepresentable.

Note what else arrived with almost no effort. main takes an argument and returns an int, which is Chapter 33.8 Section 5's testable entry point; the if __name__ guard is present, so the module is importable; and the exit code is 1 when anything failed, which is the promise Chapter 34.6 will document.

9. The Second Consumer

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import sys
from pathlib import Path

Path("flow").mkdir(exist_ok=True)
(Path("flow") / "__init__.py").write_text("", encoding="utf-8")
(Path("flow") / "parse.py").write_text(
    'print("[import] flow/parse.py")\n'
    "def parse_sim_log(text):\n"
    "    errs = sum(1 for l in text.splitlines()\n"
    "               if l.startswith('UVM_ERROR') and 'UVM_ERROR :' not in l)\n"
    "    return {'errors': errs, 'status': 'FAIL' if errs else 'PASS'}\n",
    encoding="utf-8")
(Path("flow") / "tools.py").write_text(
    'print("[import] flow/tools.py")\n'
    "import subprocess\n"
    "def run_sim(*a):\n"
    "    raise RuntimeError('would launch a simulator')\n", encoding="utf-8")

# a completely different script, written by somebody else, months later
Path("triage_old_logs.py").write_text(
    '"""Re-parse yesterday\'s logs. Runs no simulations."""\n'
    "from pathlib import Path\n"
    "\n"
    "from flow.parse import parse_sim_log\n"
    "\n"
    "\n"
    "def worst(logs):\n"
    "    rows = [dict(parse_sim_log(t), name=n) for n, t in logs.items()]\n"
    "    bad = [r for r in rows if r['status'] != 'PASS']\n"
    "    return max(bad, key=lambda r: r['errors']) if bad else None\n",
    encoding="utf-8")

sys.path.insert(0, ".")
import triage_old_logs

LOGS = {
    "run_0001/sim.log": "UVM_ERROR a.sv(1) x\nUVM_ERROR :   1\nTEST DONE\n",
    "run_0002/sim.log": "UVM_INFO ok\nTEST DONE\n",
    "run_0003/sim.log": ("UVM_ERROR b.sv(2) y\nUVM_ERROR c.sv(3) z\n"
                         "UVM_ERROR :   2\nTEST DONE\n"),
}

print()
print("worst run:", triage_old_logs.worst(LOGS))
print()
print("flow.tools loaded:", "flow.tools" in sys.modules)
print("simulations run  :", 0)
print("lines of flow.parse reused:", "all of it")
print("lines of run_regress reused:", 0)
print()
print("this script was impossible before:", "34.1 §9 — the import ran the regression")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[import] flow/parse.py

worst run: {'errors': 2, 'status': 'FAIL', 'name': 'run_0003/sim.log'}

flow.tools loaded: False
simulations run  : 0
lines of flow.parse reused: all of it
lines of run_regress reused: 0

this script was impossible before: 34.1 §9 — the import ran the regression

A second script, written months later by somebody else, reusing the parser and running no simulations. flow.tools never loads — note the single [import] line — so run_sim's deliberate RuntimeError is never in reach, and the new script does not inherit run_regress.py's command line, its test list, or its exit-code policy.

That is the sentence in the canonical purpose, demonstrated: a second script can reuse the parser without inheriting the first script's command line.

10. What Moved and What Did Not

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

BEFORE = {
    "lines": 166, "functions": 3, "testable functions": 0,
    "modules": 1, "importable": False, "sys.exit": 0,
}
AFTER = {
    "lines": 166, "functions": 9, "testable functions": 5,
    "modules": 4, "importable": True, "sys.exit": 1,
}

print(f"{'property':<22} {'34.1':>10} {'34.2':>10}  changed")
for k in BEFORE:
    changed = BEFORE[k] != AFTER[k]
    print(f"{k:<22} {str(BEFORE[k]):>10} {str(AFTER[k]):>10}  {changed}")

print()
print("the LOGIC that changed:", "none of it")
print("what changed instead  :", "where each piece lives")
print()
UNFIXED = ["16 path-like literals are still hard-coded — 34.5",
           "no docstring says what the exit codes mean — 34.6",
           "no class, and it may not need one — 34.4",
           "parse and report are split but not yet LAYERED — 34.3"]
print("defects from 34.1 still present:", len(UNFIXED))
for u in UNFIXED:
    print("   -", u)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

property                     34.1       34.2  changed
lines                         166        166  False
functions                       3          9  True
testable functions              0          5  True
modules                         1          4  True
importable                  False       True  True
sys.exit                        0          1  True

the LOGIC that changed: none of it
what changed instead  : where each piece lives

defects from 34.1 still present: 4
   - 16 path-like literals are still hard-coded — 34.5
   - no docstring says what the exit codes mean — 34.6
   - no class, and it may not need one — 34.4
   - parse and report are split but not yet LAYERED — 34.3

Zero to five testable functions, with the same logic and about the same number of lines. The line count is the control in that experiment: nothing was rewritten, nothing was added, and the number of functions a test can call went from none to five, because the pure computations Chapter 34.1 Section 10 listed are now in a module that imports nothing and touches nothing.

Four of Chapter 34.1's defects are untouched, and the list names which chapter takes each one. The split was first because every one of those four needs the file to be importable before it can be attempted — which is why the diagram in Chapter 34.1 Section 11 put this stage second, immediately after measuring.

11. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import sys
from pathlib import Path

Path("pkg").mkdir(exist_ok=True)
(Path("pkg") / "__init__.py").write_text(
    'print("init")\n', encoding="utf-8")
(Path("pkg") / "a.py").write_text(
    'print("a")\nX = 1\n', encoding="utf-8")
(Path("pkg") / "b.py").write_text(
    'print("b")\nfrom pkg.a import X\nY = X + 1\n', encoding="utf-8")
sys.path.insert(0, ".")

# A
from pkg.a import X
print("A:", X, "pkg.b" in sys.modules)

# B
from pkg.b import Y
print("B:", Y, "pkg.a" in sys.modules)

# C
import pkg.a
print("C:", pkg.a is sys.modules["pkg.a"])

# D
Path("nop").mkdir(exist_ok=True)
(Path("nop") / "m.py").write_text("Z = 9\n", encoding="utf-8")
from nop.m import Z
print("D:", Z, sys.modules["nop"].__file__ is None)

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

init
a
A: 1 False
b
B: 2 True
C: True
D: 9 True

A — importing pkg.a runs pkg/__init__.py then pkg/a.py, and pkg.b is not loaded. Section 4 in two values.

B — importing pkg.b prints only b, because pkg.a was already in sys.modules from A and is not re-executed. That is why the [import] lines in Section 3 appear once each no matter how many modules ask for them.

C — pkg.a the name and sys.modules["pkg.a"] the entry are the same object, which is Chapter 13.3 Section 7's point and the reason a module-level global is shared by every importer rather than copied.

D — a directory with no __init__.py imported fine and its package __file__ is None. Section 7, as one boolean.

12. Exercises

Exercise 1 — one supported use. For your own script, list the things you cannot do with it because importing it runs it.

Exercise 2 — three decisions. Name the three decisions encoded in the annotation's two lines.

Exercise 3 — the unstated one. Say which decision the two lines do not make.

Exercise 4 — split your parser out. Create a package and move the pure text-to-facts logic into it, unchanged.

Exercise 5 — mark the imports. Put a temporary print at the top of each module and record which ones run.

Exercise 6 — two of four. Confirm that importing one module does not load its siblings.

Exercise 7 — the order surprise. Explain why the [import] lines appear before the first print in your own driver.

Exercise 8 — what __init__.py costs. Add an expensive line to it and say who pays.

Exercise 9 — keep it thin. Reduce an __init__.py to a docstring, a version and re-exports.

Exercise 10 — draw the direction. List every intra-package import in your code and mark its direction.

Exercise 11 — the bottom of the graph. Identify the module that imports nothing, and check it is the pure one.

Exercise 12 — purity versus position. Find a module that is impure and still at the bottom, and explain why both are true.

Exercise 13 — make a cycle. Build the two-module cycle and read the error.

Exercise 14 — both directions. Import each side first and record the two different messages.

Exercise 15 — the victim. Explain why the module named in the error is not the one to fix.

Exercise 16 — partially initialized. Describe, in your own words, the state Python was in when it raised.

Exercise 17 — the real fix. Resolve a cycle by deciding which layer is lower, not by moving the import into a function.

Exercise 18 — no __init__.py. Confirm the import still works.

Exercise 19 — __file__ is None. Write the Path(pkg.__file__).parent line and watch it fail.

Exercise 20 — who assumes a string. Find code in your own project that assumes __file__ is a path.

Exercise 21 — two smaller reasons. State the other two reasons to write the file.

Exercise 22 — collect returns. Rewrite one function that communicates by global so it returns instead.

Exercise 23 — unrepresentable. Explain why Chapter 34.1's RESULTS bug cannot exist in the new shape.

Exercise 24 — main(args) -> int. Give your entry point an argument and a return value.

Exercise 25 — the second consumer. Write a script that imports only your parser, and confirm the tool layer does not load.

Exercise 26 — what it did not inherit. List what the second script avoided by not importing the first.

Exercise 27 — the control variable. Count your lines before and after, and say what an unchanged count proves.

Exercise 28 — count the testable surface. Re-run Chapter 34.1 Section 10's measurement on the package.

Exercise 29 — what is still broken. List the defects the split did not fix and which chapter handles each.

Exercise 30 — why this stage first. Explain why the four remaining defects all needed the file to be importable.

13. Summary

One file supports one use. Running it works; importing it to reuse, test, or explore runs the regression — Chapter 34.1 Section 9's measurement.
The annotation's two lines encode three decisions: a package, separate modules for parsing and tool control, and import by name rather than path.
from flow.parse import ... runs flow/__init__.py and flow/parse.py only. Two of four modules load; flow.tools and flow.report are never executed.
So anything in __init__.py is paid by every consumer, including one that only wanted a pure function. Keep it to a docstring, a version and re-exports.
The pure module sits at the bottom and imports nothing from the package, which is what makes it importable by anything and testable without setup.
Purity and position are different axes — flow.tools is impure and also at the bottom, because nothing in the package needs it.
A circular import's error names whichever module you imported first, so the same unchanged cycle produces two different messages and both blame the victim rather than the cause.
"Partially initialized module" is the real clue: the named module is on the stack, unfinished, and nothing is wrong with either file alone.
Resolve a cycle by deciding which layer is lower, not by hiding the import inside a function.
A package with no __init__.py imports perfectly well — it becomes a namespace package, so "packages need __init__.py" is false.
Its __file__ is None and __path__ is a _NamespacePath, which breaks the normal Path(pkg.__file__).parent way of finding a bundled data file. That, not the import, is the reason to write the file.
collect returns its rows, so Chapter 34.1's empty-report bug is not fixed but unrepresentable — the caller cannot write a report without having received something.
A second script, months later, imports only the parser and runs no simulations, inheriting none of the first script's command line.
Zero to five testable functions with the same logic and the same line count. The unchanged line count is the control: nothing was rewritten, only relocated.
Four of Chapter 34.1's defects remain, and every one of them needed the file to be importable first — which is why this stage comes immediately after measuring.

The next chapter turns the three modules into three layers, with the dependency direction made explicit and enforced.

Where this fits

Part of the Python curriculum.

1. What One File Costs

2. The Annotation

3. The Package

4. Importing the Parser Does Not Import the Simulator

5. One Direction of Dependency

6. A Circular Import Blames the Victim

7. Does __init__.py Matter?

8. What run_regress.py Becomes

9. The Second Consumer

10. What Moved and What Did Not

11. Predict, Run, Observe

12. Exercises

13. Summary

Where this fits

7. Does `init.py` Matter?

8. What `run_regress.py` Becomes