Handing Off to Python for Data and Reports

Part II · Module 35 · Chapter 35.4 · Handing Off to Python

Chapter 35.3 gave the dependency graph to Make. This chapter gives the data handling to Python — but only after establishing where the handover belongs, because "when it gets complicated" is not a criterion anybody can apply.

1. The First Threshold Is Arithmetic, and It Is Lower Than Expected

A report needs a pass rate. 1,560 of 2,000:

$ /bin/bash -c 'p=1560; n=2000; echo "pass rate: $(( p / n * 100 ))%"'
pass rate: 0%

Zero. p / n is integer division, which truncates to 0 before the multiplication. Reordering to p * 100 / n happens to work here and overflows for large counts, which is a fix that fails later rather than now.

And the obvious repair is not available:

$ /bin/bash -c 'echo "$(( 1560.0 / 2000 ))"'
/bin/bash: 1560.0 / 2000 : syntax error: invalid arithmetic operator (error token is ".0 / 2000 ")

Bash has no floating-point arithmetic at all. Not imprecise — absent, with the error blaming the decimal point.

So the threshold looks like it is crossed the moment a report divides. It is not, because the shell's toolbox is bigger than bash:

  bc:     78.00
  awk:    78.00
  python: 78.00

bc and awk both compute it, and Chapter 13.7 already put awk in the flow. Arithmetic alone does not justify a language change — it justifies using awk rather than $(( )), which is a much smaller decision.

2. The Same Report, Twice

#!/bin/bash
# the same report in shell: per test, the count of each class, a pass rate,
# and the tests whose pass rate is below a threshold, sorted by it.
set -u
f=${1:?need results.tsv}
th=${2:-80}
printf '%-12s %5s %5s %5s %7s\n' TEST RUNS PASS FAIL 'RATE%'
for t in $(awk -F'\t' 'NR>1{print $2}' "$f" | sort -u); do
  runs=$(awk -F'\t' -v t="$t" 'NR>1 && $2==t' "$f" | /usr/bin/grep -c .)
  pass=$(awk -F'\t' -v t="$t" 'NR>1 && $2==t && $4=="PASS"' "$f" | /usr/bin/grep -c .)
  fail=$(awk -F'\t' -v t="$t" 'NR>1 && $2==t && $4=="FAIL"' "$f" | /usr/bin/grep -c .)
  rate=$(echo "scale=1; 100*$pass/$runs" | bc)
  printf '%-12s %5s %5s %5s %7s\n' "$t" "$runs" "$pass" "$fail" "$rate"
done | sort -k5,5n

#!/usr/bin/env python3
"""The same report in Python. Reads the TSV the shell produced; writes a table
and, with --json, a machine-readable form the shell can hand onward."""
import sys, csv, json, collections
def main(argv):
    path = argv[1] if len(argv) > 1 else None
    if not path:
        print("report.py: need results.tsv", file=sys.stderr); return 2
    want_json = "--json" in argv
    try:
        with open(path, newline="") as fh:
            rows = [r for r in csv.DictReader(
                (l for l in fh if not l.startswith("#") or l.startswith("#spec")),
                delimiter="\t") ]
    except OSError as e:
        print(f"report.py: {e}", file=sys.stderr); return 2
    # the header line starts with '#spec', so strip the '#' from the first key
    if rows and "#spec" in rows[0]:
        rows = [{(k[1:] if k.startswith("#") else k): v for k, v in r.items()} for r in rows]
    if not rows:
        print("report.py: no rows", file=sys.stderr); return 3
    by = collections.defaultdict(collections.Counter)
    for r in rows:
        by[r["test"]][r["verdict"]] += 1
    out = []
    for t, c in by.items():
        runs = sum(c.values())
        out.append({"test": t, "runs": runs, "pass": c["PASS"], "fail": c["FAIL"],
                    "rate": round(100.0 * c["PASS"] / runs, 1),
                    "classes": dict(c)})
    out.sort(key=lambda d: (d["rate"], d["test"]))
    if want_json:
        json.dump(out, sys.stdout, indent=1, sort_keys=True); print()
        return 0
    print(f'{"TEST":<12} {"RUNS":>5} {"PASS":>5} {"FAIL":>5} {"RATE%":>7}')
    for d in out:
        print(f'{d["test"]:<12} {d["runs"]:>5} {d["pass"]:>5} {d["fail"]:>5} {d["rate"]:>7}')
    return 0
sys.exit(main(sys.argv))

=== same output? ===
  identical (sorted)

And the comparison that most people assume they already know the answer to:

  lines of code: shell 14, python 39

The shell version is shorter — by a factor of 2.8. That is not an artifact of writing it carelessly; it is what the shell is good at. Fourteen lines that each compose existing tools against twenty-eight that build the same result from primitives.

  shell  rep 1: 1355ms    python rep 1: 712ms
  shell  rep 2: 1800ms    python rep 2: 408ms
  shell  rep 3: 1759ms    python rep 3: 616ms

Python is faster here, on data small enough that nobody cares. Note the spread already: Python's three reps range from 408 to 712 ms — 1.7× between repetitions of identical work, which is a warning worth taking seriously and which §3 returns to. On this evidence alone there is no case for the handover: the shell version is shorter, produces identical output, and takes under two seconds.

3. The Actual Threshold: How Many Times the Data Is Read

=== how many times does each version read the file? ===
  distinct tests: 10
  shell: 1 (the sort -u) + 10 x 3 awk passes = 31 full reads
  python: 1

Thirty-one against one. Look at the shell loop again:

for t in $(awk -F'\t' 'NR>1{print $2}' "$f" | sort -u); do
  runs=$(awk -F'\t' -v t="$t" 'NR>1 && $2==t' "$f" | /usr/bin/grep -c .)
  pass=$(awk -F'\t' -v t="$t" 'NR>1 && $2==t && $4=="PASS"' "$f" | /usr/bin/grep -c .)
  fail=$(awk -F'\t' -v t="$t" 'NR>1 && $2==t && $4=="FAIL"' "$f" | /usr/bin/grep -c .)

Three passes over the whole file per test, because the shell has no place to keep a running tally across a loop that spawns a process per metric. The cost is rows × groups × metrics, and nothing about the code says so.

Which predicts that the gap should widen with the data, and it does:

  results.tsv    2000 rows   shell   2014ms   python   736ms   ratio 2.7x
  big.tsv       20000 rows   shell   5213ms   python   721ms   ratio 7.2x

The ratio went from 2.7× to 7.2× on ten times the data — the shell paying for every extra row thirty-one times over.

That was one run, though, and §2 already showed Python's reps varying by 1.7×. Repeating the whole thing twice more, the second time with five repetitions per cell:

  run 2    2000 rows   shell   884ms   python    88ms   ratio 10.0x
          20000 rows   shell  2751ms   python   181ms   ratio 15.2x

  run 3    2000 rows   shell  [562, 1008, 777, 2019, 2461]  median 1008ms
                       python [610, 377, 176, 134, 142]     median  176ms   5.7x
          20000 rows   shell  [3387, 4732, 2660, 2938, 4516] median 3387ms
                       python [568, 272, 353, 235, 281]      median  281ms  12.1x

The spread within a single condition is 4.4× for the shell and 4.6× for Python — identical work, same file, same machine. This is a shared machine running other jobs, and Chapter 34.6 measured exactly this: a wall-clock number from a loaded machine is not a measurement of the thing you think you are measuring.

So two of the three numbers in that first table have to be given up, and one survives:

My first draft of this chapter made the middle claim. It is the kind of error a benchmark invites: two numbers that happened to land close together, in a quantity whose noise band is wider than the difference between them. The ratio doubling from 2,000 to 20,000 rows is the robust result, because it appeared in three independent runs — and the reason it is robust is that it is not really a timing fact at all.

So the criterion is:

Hand off when the task needs to group by a key and compute more than one thing per group.

That is a property you can see in the source. One pass with one accumulator is a shell job; k metrics across g groups is k × g passes in the shell and one in any language with a dictionary.

And it explains why the earlier chapters' pipelines were right to stay in the shell:

sort | uniq -c is the shell's dictionary, and it works precisely when there is one thing to count per key. The moment a second metric appears, the shell either re-reads or starts building parallel awk scripts — and a single awk program with arrays is the honest intermediate step, which is worth saying because it is often enough:

awk -F'\t' 'NR>1 {n[$2]++; if ($4=="PASS") p[$2]++; if ($4=="FAIL") f[$2]++}
             END {for (t in n) printf "%-12s %5d %5d %5d %7.1f\n", t, n[t], p[t], f[t], 100*p[t]/n[t]}' results.tsv

One pass, three metrics, arithmetic included. If that is where the task stops, it does not need Python.

4. What Python Is Actually For

The threshold above says when. This says what for, and it is not speed.

=== and the thing the shell version cannot produce at all ===
  [
   {
    "classes": {
     "CRASH": 6,
     "FAIL": 20,
     "INCONSISTENT": 10,
     "NO_VERDICT": 1,
     "PASS": 148,
     "TIMEOUT": 8,
     "VACUOUS": 7
    },
    "fail": 20,
    "pass": 148,
    "rate": 74.0,

A nested structure. Seven classes per test, per test, in a form something else can read. The awk version above handles three fixed metrics; it cannot produce a per-test map of a variable set of classes without naming them, and Chapter 32.3's class list is exactly the kind of thing that grows.

Four things that belong on the Python side, each for a structural reason rather than a performance one:

Nested or variable-shaped output. JSON for a dashboard, a per-test per-class table, anything a later tool parses.

Arithmetic that must be right. Not just possible — correct at the edges. rate on a test with zero runs is a division by zero that awk prints as inf and Python raises on, and the raise is what you want.

Anything stateful across the whole input. A trend across ninety nights, a comparison that needs both files in memory, a deduplication by a composite key.

And anything with a real test suite. Chapter 35.1 built one for shell functions and it is harder work than the Python equivalent; a reporting module with twenty cases is routine in Python and a project in shell.

5. The Interface in Both Directions

Python's side of the contract is narrower than it looks, and all four points were exercised by the script in Section 2.

Arguments, not environment. report.py results.tsv --json rather than reading $RESULTS. The shell already has Chapter 35.2's configuration and should resolve it before the call, so Python has one job and a reviewable signature.

Exit status, with the same contract as everything else:

if not path:
    print("report.py: need results.tsv", file=sys.stderr); return 2
…
except OSError as e:
    print(f"report.py: {e}", file=sys.stderr); return 2
…
if not rows:
    print("report.py: no rows", file=sys.stderr); return 3

2 for usage, 3 for no data — the same values Chapter 34.1's wrapper uses, and never 1. A Python tool whose uncaught exception reaches the top exits 1, which is the value reserved for a failing test, so the except is not politeness — it is the contract.

Diagnostics on stderr, data on stdout. print(..., file=sys.stderr) for every message, so the shell can do report.py f.tsv > report.txt and get only the report. Chapter 12.3's separation, and Chapter 35.1's library contract applied across a language boundary.

And the empty case is a refusal, not an empty report. if not rows: return 3. A report over zero rows prints a header and a clean-looking nothing, which is Chapter 32.7's measured false green — arithmetic over an empty set returning success.

Which Python version to assume is also part of the contract:

$ python3 --version

and the answer belongs in the flow's documentation (Chapter 35.6) rather than in a comment, because a flow that works on one engineer's machine and not another's usually differs here rather than in the code. The script above uses only the standard library — sys, csv, json, collections — deliberately: a reporting tool that needs a package installed is a reporting tool that fails on the build machine, and Chapter 32.8 measured what the build machine's environment actually looks like.

6. What Stays in the Shell, Permanently

The last row is the one that is easy to lose. Chapter 33.6 measured a trace showing the exact argv of every command — and a flow whose control lives in Python shows one line, python3 flow.py, and nothing about what it did. Python can log, and a log is a thing somebody wrote; a trace is a thing the shell produces whether or not anybody thought to.

So the division is not by difficulty. It is:

The shell runs things and moves bytes. Python computes over data. Make decides what needs doing.

Each tool doing the thing its model fits, with a file at every boundary.

Practice

1. Compute a pass rate in bash arithmetic and read the answer.
2. Try it with a decimal and read the error.
3. Compute it with bc, awk and Python and compare.
4. Say why arithmetic alone does not justify a language change.
5. Write the same report in shell and in Python and diff the output.
6. Count the lines of each and note which is shorter.
7. Time both on your real data.
8. State both wrong conclusions that measurement invites.
9. Count how many times each version reads its input.
10. Work out the shell version's cost in terms of rows, groups and metrics.
11. Run both on ten times the data and compare the ratios.
12. State the threshold as a property of the code.
13. Classify three earlier pipelines by groups and metrics, and say why they stayed in the shell.
14. Write the single-awk version with arrays, and decide whether it is enough.
15. Produce nested output and try to do the same in the shell.
16. Divide by a zero group size in awk and in Python, and compare.
17. Give four structural reasons for the Python side.
18. Define the boundary format and give its four properties.
19. State the one-direction rule.
20. Write the per-row python3 -c anti-pattern and time it.
21. Give a Python tool arguments rather than environment, and say why.
22. Make its exit statuses match the flow's contract, and say what an uncaught exception returns.
23. Send diagnostics to stderr and confirm a redirect gives only data.
24. Make the empty case a refusal.
25. Check your Python version and decide where that belongs.
26. List the standard-library imports and justify avoiding packages.
27. List what stays in the shell permanently, and defend the last entry.

Check Your Understanding

1. What does $(( 1560 / 2000 * 100 )) give, and why?
2. What does bash do with a decimal in arithmetic?
3. Which tools compute the rate correctly?
4. Why does that not justify a language change?
5. Did the two reports agree?
6. Which was shorter, and by how much?
7. What were the measured times?
8. Give the two wrong conclusions from that data, and what both lack.
9. How many times did each version read the file, and why?
10. What is the shell version's cost in terms of rows, groups and metrics?
11. What happened to the ratio at ten times the data — and what happened when you repeated the whole measurement?
12. State the threshold.
13. Why did Chapter 29.4's frequency pipeline stay in the shell?
14. What is sort \| uniq -c in this framing?
15. What is the honest intermediate step, and when is it enough?
16. Name four structural reasons for Python, and say why speed is not one.
17. What can the single-awk version not produce?
18. What is the boundary format, and why tabs?
19. Why a header, and what asymmetry do you accept?
20. Why one row per record always?
21. State the one-direction rule, and what both violations have in common.
22. What is the per-row anti-pattern, and what does it cost?
23. Why arguments rather than environment?
24. What does an uncaught Python exception exit with, and why does that matter?
25. Why must the empty case be a refusal?
26. Why avoid non-standard-library packages?
27. What stays in the shell, and why is the control structure the easiest to lose?
28. State the three-way division.

7. Remember

• Bash cannot compute a pass rate. $(( p / n * 100 )) gave 0, and a decimal is a syntax error — but bc and awk both work, so arithmetic alone is not a reason to change language.
• The shell version was shorter — 14 lines against 39 — and produced identical output. Both "keep it" and "rewrite it" from that data are judgements without a criterion.
• The threshold is file passes: 31 against 1. The shell's cost is rows × groups × metrics.
• The ratio roughly doubles from 2,000 to 20,000 rows — in all three runs (2.7→7.2, 10.0→15.2, 5.7→12.1). The absolute times do not reproduce: 4.4× spread for the shell and 4.6× for Python within one condition, so a wall-clock number from a shared machine is not a measurement. My first draft claimed Python was flat; that difference was inside its own noise band.
• Hand off when you must group by a key and compute more than one thing per group. That is visible in the source, before the data grows.
• sort | uniq -c is the shell's dictionary and suffices for one metric per key; a single awk with arrays is the honest next step.
• Python is for nested output, edge-correct arithmetic, whole-input state and a real test suite — not for speed.
• The boundary is a tab-separated file with a header, one row per record, flowing one way, produced by the shell.
• Never cross a language boundary per row. Move the loop, not the startup cost.
• Python's contract: arguments not environment, statuses matching the flow (never 1 — an uncaught exception is 1), diagnostics on stderr, and a refusal on no rows.
• Standard library only, because the build machine is barer than your desk.
• The shell runs things and moves bytes; Python computes; Make decides what needs doing — and the flow's control stays in the shell so that bash -x still shows what ran.

8. What Is Next

Two boundaries are now drawn with measurements behind them. Chapter 35.5 draws the third and most awkward one — the shell generating a script for a tool's own interpreter — where the thing being handed over is not data but instructions, and the usual implementation loses any record of what the tool was asked to do.