Packages and Laying Out a Small Project

Part I · Module 13 · Chapter 13.7 · Packages and Laying Out a Small Project

1. When a Flat Directory Stops Helping

checker/
    parse.py
    report.py
    submit.py
    config.py
    run.py

Five files at one level. Nothing in that listing says which four are library code and which one you type python3 in front of, and nothing groups the two that belong together. Add six more and a newcomer has to open each to find out what it is.

2. A Package Is a Folder

project/
    flow/
        parse.py
        report.py
    run.py

flow is now a package — a directory of modules that Python can import as a unit. The dotted name follows the directory tree:

def count_errors(lines):
    """Return the number of UVM_ERROR lines."""
    n = 0
    for line in lines:
        if "UVM_ERROR" in line:
            n += 1
    return n

def summarise(count):
    """Return a one-line verdict for an error count."""
    if count == 0:
        return "PASS"
    return "FAIL (" + str(count) + " errors)"

from flow.parse import count_errors
from flow.report import summarise

log = ["UVM_ERROR a", "INFO b", "UVM_ERROR c"]

print(summarise(count_errors(log)))

FAIL (2 errors)

flow/parse.py is imported as flow.parse. A dot in an import is a slash in the filesystem — that one sentence covers most of what a package is.

All of Chapter 13.2's forms work on a package, with the dotted name in place of a plain one:

def count_errors(lines):
    """Return the number of UVM_ERROR lines."""
    return len([line for line in lines if "UVM_ERROR" in line])

import flow.parse
from flow.parse import count_errors
import flow.parse as parsing

log = ["UVM_ERROR a", "UVM_ERROR b"]

print(flow.parse.count_errors(log))
print(count_errors(log))
print(parsing.count_errors(log))
print(flow.parse.__name__)

2
2
2
flow.parse

Note the module's __name__: flow.parse, not parse. A module inside a package knows its full dotted path.

This is also where Chapter 13.3's aliasing earns its place — import flow.parse as parsing replaces a two-segment dotted name with one descriptive word, which is the case that chapter called genuinely worthwhile.

3. Deeper Nesting

The rule does not change with depth:

project/
    flow/
        parsing/
            uvm.py
    run.py

def severity(line):
    """Return the severity word at the start of a UVM log line."""
    return line.split()[0]

from flow.parsing.uvm import severity

print(severity("UVM_ERROR @120: crc mismatch"))
print(severity("UVM_INFO @1: ok"))

UVM_ERROR
UVM_INFO

Three segments, three directory levels. Past about three the dotted names get long enough that Chapter 13.3's aliasing is worth reaching for.

4. __init__.py

You will see an empty file named __init__.py in most package directories:

project/
    flow/
        __init__.py
        parse.py
    run.py

Here is the honest position: it is optional on modern Python. Section 2's examples had no __init__.py and imported fine, because since Python 3.3 a directory without one still works as a package.

It is still worth writing, for two reasons.

It makes the intent explicit. A directory containing __init__.py is unambiguously a package somebody meant to create, rather than a folder that happens to contain .py files. Tools and readers both benefit.

It is a place to put the package's public face. It runs when the package is first imported, so it can re-export the names you want callers to use:

from flow.parse import count_errors
from flow.report import summarise

def count_errors(lines):
    """Return the number of UVM_ERROR lines."""
    return len([line for line in lines if "UVM_ERROR" in line])

def summarise(count):
    """Return a one-line verdict for an error count."""
    if count == 0:
        return "PASS"
    return "FAIL (" + str(count) + " errors)"

from flow import count_errors, summarise

print(summarise(count_errors(["UVM_ERROR a", "UVM_ERROR b"])))

FAIL (2 errors)

run.py now imports from flow directly, without needing to know that count_errors lives in parse.py. That is a real benefit: the package's internal file layout can change without every caller changing with it.

5. Where the Entry Script Goes

Outside the package, at the project root. That is the layout the canonical annotation shows and it is not arbitrary:

project/
    flow/
        __init__.py
        parse.py
        report.py
    run.py          <- the thing you run

run.py is not library code. Nobody imports it; it is the arrangement — Chapter 11.15's edge, and Chapter 13.6's main() lives in it:

def count_errors(lines):
    """Return the number of UVM_ERROR lines."""
    return len([line for line in lines if "UVM_ERROR" in line])

def summarise(count):
    """Return a one-line verdict for an error count."""
    if count == 0:
        return "PASS"
    return "FAIL (" + str(count) + " errors)"

from flow.parse import count_errors
from flow.report import summarise


def main():
    """Check the sample logs and print a verdict for each."""
    logs = {
        "axi_smoke": ["INFO ok"],
        "axi_burst": ["UVM_ERROR a", "UVM_ERROR b"],
    }
    for name in sorted(logs):
        print(name, "->", summarise(count_errors(logs[name])))


if __name__ == "__main__":
    main()

axi_burst -> FAIL (2 errors)
axi_smoke -> PASS

Everything from the module converging: a package holding pure functions, an entry script holding the arrangement, a main(), and a guard.

6. The Failure That Actually Happens

Chapter 13.5 Section 6 said where you run from decides what is found. With a package that becomes the single most common real problem:

project/
    flow/
        parse.py
    run.py

$ cd project && python3 run.py          -> works
$ cd project/flow && python3 ../run.py  -> ModuleNotFoundError: No module named 'flow'

The second command runs the same file with the same code and fails, because Python looked for flow relative to the directory it started in — and from inside flow there is no flow.

Run from the project root, always. When a project that worked yesterday reports ModuleNotFoundError on one of its own packages, check the working directory before reading a line of code. In a nightly job that means the script is launched with the project root as its working directory; the cause is almost never the import line.

7. A Layout That Scales

For a project with a handful of modules:

project/
    flow/
        __init__.py
        parse.py          library: pure functions
        report.py         library: pure functions
        submit.py         library: touches tools
    run.py                entry point, main(), guard
    README.md             what this is and how to run it

The rule behind it is Chapter 11.15's, applied to directories: things that compute go in the package; the thing that decides what to compute goes at the root.

Tests eventually get a tests/ directory beside flow/, which Part II covers once there is a tool to run them with.

8. What Part I Leaves Out

Two things exist and are deliberately out of scope here, worth knowing by name so you recognise them:

Relative imports — from .parse import count_errors, with a leading dot, used inside a package to refer to its siblings. They work, they interact with how the package was launched in ways that surprise people, and the absolute form used throughout this chapter is always available and always clearer at this scale.

Installable packages — the machinery that lets pip install put your package somewhere importable from any directory, which removes Section 6's problem entirely. That is a packaging topic and belongs with the tooling in Part II.

9. Predict, Run, Observe

from tools.text import shout

def shout(s):
    """Return s uppercased."""
    return s.upper()

import tools.text
from tools import shout

print("A:", tools.text.shout("ok"))
print("B:", shout("ok"))
print("C:", tools.text.__name__)
print("D:", __name__)

A: OK
B: OK
C: tools.text
D: __main__

A — the full dotted path reaches the function.

B — the same function, re-exported by __init__.py, imported from the package directly.

C — a module inside a package knows its full dotted name.

D — the entry script is still "__main__", exactly as Chapter 13.6 said. Packages change nothing about that.

10. Exercises

Exercise 1 — make a package. Turn a flat two-file project into flow/parse.py plus run.py at the root. Confirm it runs and say what the module's __name__ became.

Exercise 2 — dot is slash. For from flow.parsing.uvm import severity, name the directory, the subdirectory and the file. Then write the path.

Exercise 3 — without __init__.py. Build a working package with no __init__.py. Say whether it imports, and then give two reasons to add one anyway.

Exercise 4 — re-export. Add an __init__.py that re-exports two functions from two modules, and import both from the package directly. Say what a caller no longer needs to know.

Exercise 5 — the launch directory. Build a package, confirm it runs from the root, then run the same entry script from inside the package directory. Record the error and explain it without mentioning the code.

Exercise 6 — place each file. For a project with a log parser, a report formatter, a tool launcher and a script that ties them together, say which go in the package and which at the root, and why.

Exercise 7 — alias a long name. Write a three-segment import and give it a good alias. Say why this is the case Chapter 13.3 called worthwhile.

Exercise 8 — read a layout. Given proj/, proj/flow/__init__.py, proj/flow/sta.py, proj/flow/synth.py and proj/run.py, say what you would expect each file to contain and which one you would launch.

11. Summary

• A package is a directory of modules importable as a unit.
• A dot in an import is a slash in the filesystem — the dotted name follows the directory tree exactly, to any depth.
• A module inside a package knows its full dotted __name__, such as flow.parse.
• All of Chapter 13.2's import forms work on packages, and Chapter 13.3's aliasing is genuinely worthwhile once names reach three segments.
• __init__.py is optional on modern Python — a directory without one still imports.
• Write it anyway: it makes the intent explicit and gives the package a public face through re-exports.
• Keep it small or empty, because it runs on every import of any part of the package.
• The entry script goes outside the package, at the project root. It is the arrangement, not library code.
• Run from the project root, always. ModuleNotFoundError on your own package is usually a working-directory problem.
• Things that compute go in the package; the thing that decides what to compute goes at the root — Chapter 11.15's boundary, applied to directories.

That closes Module 13. A script is no longer one file: you can reach the several hundred modules that ship with Python, split your own work across files along a boundary that makes the pieces reusable, make a file both runnable and importable, and lay out a project so that a newcomer can tell at a glance what to open and what to launch.

Everything so far has assumed the code works. The next module is about what happens when it does not — the errors Python raises, how to read one, and how to decide which ones your script should handle and which it should let stop everything.