What a Regression Actually Has to Do

Part II · Module 32 · Chapter 32.1 · What a Regression Actually Has to Do

Module 31 built the mechanics: a matrix, bounded slots, isolated run directories, and per-run status files reconciled against the matrix. Every piece of it is invoked by hand.

A regression is what you get when those pieces become a system somebody can run while asleep. And the first thing that system needs is not code.

1. Four Obligations

"Run the tests and tell me if they passed" is not a specification, because it does not say what a test is, what passed means, what tell me consists of, or what happens when a run does neither. Written out properly, a regression has four obligations:

Those are four separate things, and the rest of this module is one chapter per obligation plus the mechanics around them. This chapter's job is to establish — by measurement, not assertion — that none of the four is automatic.

2. Thirteen Things a Run Can Do

The fixture for this module is one compiled image containing thirteen tests. Four of them are ordinary. The other nine exist because each one is a thing real runs do, and each one breaks a different naive assumption. Here is every one of them, run once:

  TEST          rc   t/out  verdict line   ERRORs  reported
  smoke         0    no     TEST PASSED    0       checks=3 errors=0
  fairness      0    no     TEST PASSED    0       checks=6 errors=0
  random        0    no     TEST PASSED    0       checks=52 errors=0
  b2b           0    no     TEST PASSED    0       checks=8 errors=0
  prio0         0    no     TEST FAILED    1       checks=1 errors=1
  inconsistent  0    no     TEST PASSED    1       checks=1 errors=0
  crash         1    no                    0
  silent        0    no                    0
  flaky         0    no     TEST FAILED    1       checks=1 errors=1
  hang          0    YES                   0
  bogus         0    no     TEST FAILED    1       checks=0 errors=1
  vacuous       0    no     TEST PASSED    0       checks=0 errors=0
  latecrash     1    no     TEST PASSED    0       checks=1 errors=0

Read the rc column on its own. Eleven of the thirteen exited zero, including a test that failed a design check (prio0), a test that hung until it was killed (hang), a test that checked nothing (vacuous), a test that printed no verdict at all (silent), and a test whose own log carries an error next to the word PASSED (inconsistent).

Now read the verdict column on its own. Four runs have no verdict line, one claims PASSED while its log contains an ERROR, one claims PASSED after making zero checks, and one claims PASSED and then exits non-zero.

Neither column is a verdict. That is Chapter 32.3's whole subject, measured there in full; the point here is that obligation 2 — decide a verdict — is real work and not a grep.

3. A Defined Set Is Not the Same as a Varied Set

Obligation 1 says a defined set. Chapter 31.3 made the set explicit — tests crossed with seeds — and twenty seeds certainly look like twenty different runs.

The flaky test is seed-dependent on purpose. Over seeds 1 to 20:

$ for s in $(seq 1 20); do ... ; done
1:PASS 2:FAIL 3:PASS 4:PASS 5:PASS 6:PASS 7:PASS 8:FAIL 9:PASS 10:PASS
11:PASS 12:PASS 13:PASS 14:FAIL 15:PASS 16:PASS 17:PASS 18:PASS 19:FAIL 20:FAIL
  15 passed, 5 failed of 20 seeds

Five of twenty, reproducibly — seed 2 run three times gave three byte-identical logs. That is a seed sweep doing its job.

Getting there took one correction, and the correction is the point of this section. The first version of flaky keyed its decision on the low two bits of the first $random(seed) draw, and it passed on all twenty seeds. Not sometimes — always. So I measured the draws:

  seed   first draw      draw&3  draw&1
     1    -2147414528       0       0
     2    -2147345408       0       0
     3    -2147276288       0       0
     4    -2147207168       0       0
     5    -2147138048       0       0
     6    -2147068928       0       0

The low two bits of the first draw are zero for every seed. The draws are not even scattered — they are evenly spaced, 69,120 apart. A test keyed on draw & 3 is not flaky; it is constant, and a twenty-seed sweep reports twenty passes while exercising exactly one branch.

A uniform histogram is not variety

The obvious repair is to use higher bits. Over the same twenty seeds:

  over 20 seeds, how often each value appears:
    first draw, low 2 bits : 0=20 1=0 2=0 3=0
    first draw, bits 17:16 : 0=5 1=6 2=5 3=4
    SECOND draw, low 2 bits: 0=6 1=5 2=5 3=4

Bits 17:16 give 5/6/5/4 — as even a distribution as twenty samples allow. It looks fixed. It is not:

  seed  draw1&3  (draw1>>16)&3  draw2&3
     1      0         1              0
     2      0         2              1
     3      0         3              2
     4      0         0              2
     5      0         1              3
     6      0         2              0
     7      0         3              0
     8      0         0              1

Bits 17:16 go 1, 2, 3, 0, 1, 2, 3, 0 — a strict cycle of period four. The histogram is uniform because the cycle is exact, which is the opposite of random. And it has a consequence:

  a sweep in steps of 4 -- seeds 1,5,9,...,37
  seed  (draw1>>16)&3   draw2&3
     1           1            0
     5           1            3
     9           1            2
    13           1            0
    17           1            3
    21           2            2

Five seeds at stride 4 give an identical outcome. A sweep of seq 1 37 4 would run five simulations, report five results, and have exercised one case. The same sweep at stride 10 looks perfectly varied — so whether the sweep covers anything depends on a stride nobody chose deliberately.

The second draw's low bits were the only source that varied without an obvious period, which is what flaky now uses.

4. What "Record Enough to Act On" Means

Obligation 3 is the one that gets written last and is needed first. The test is simple: can somebody act on the report without access to the machine that produced it?

For each run, that means recording more than a verdict:

Two rows of that table are there because of a single measured behaviour each, which is the clearest possible argument for keeping them. latecrash and smoke have identical verdict lines and differ only in rc. vacuous and smoke have identical verdict lines and rc, and differ only in checks.

And for the regression as a whole: the test list and seed list actually used, the compiled image's identity, the host, the tool version, and the time. Chapter 32.4 builds that file; Chapter 32.6 measures what happens when the image's identity is the one thing you did not keep.

5. What a Status Has to Distinguish

Obligation 4 looks like the easiest and has the subtlest requirement. A caller — a nightly job, a CI pipeline, a person's shell — needs to tell apart at least four situations:

  0  every run passed
  1  the regression finished; something is not a pass   -> a DESIGN question
  2  the regression ran but did not finish              -> an INFRASTRUCTURE question
  3  there is nothing to report on                      -> a SETUP question

Collapsing those is the commonest regression bug there is. "Non-zero means failures" merges 1 with 2 and 3, so a regression that never started reads as a regression that found bugs — and somebody spends a morning looking for a design problem that does not exist.

Chapter 32.7 measures that contract end to end, including the single most effective way to destroy it, which is to pipe the regression's output into a log file.

6. The Shape of the Rest of This Module

The system this module builds runs on this machine, in this fixture, with the thirteen behaviours above. One complete pass of it:

$ SLOTS=8 LIMIT_S=4 regress
regress: /…/m32/regr/20261009T093929Z
regress: 24 runs, 8 slots, 4s limit each
regress: dispatch finished in 6s (xargs rc=0)
regress: 24 result rows
==============================================================
 regression 20261009T093929Z
==============================================================
 24 runs: 9 PASS  3 FAIL  2 TIMEOUT  2 CRASH
          2 NO_VERDICT  4 INCONSISTENT  2 VACUOUS
          0 MISSING  0 NOSTATUS

Twenty-four runs, thirteen tests crossed with two seeds, six seconds. Nine passed. Fifteen did not, in six different ways — and a regression that reported "9 of 24 passed, 15 failed" would be wrong fifteen times, because only three of those fifteen are failures.

The chapters that follow take that apart:

Practice

1. Write the four obligations down for a flow you actually use, and name how each one could fail.
2. Run one test and read its exit status, then read its log, and say which you would trust.
3. Build a test that fails a real check and confirm what status it exits with.
4. Build a test that prints TEST PASSED while counting an error and run it.
5. Build a test that prints no verdict at all.
6. Build a test that hangs and establish how you would notice.
7. Build a test that passes and then dies, and compare its row with a clean pass.
8. Build a test that checks nothing and says it passed.
9. Tabulate all of them — exit status, verdict line, error lines, check count.
10. Count how many exited zero and how many of those actually passed.
11. Sweep a seed-dependent test over twenty seeds and record the split.
12. Run one failing seed three times and compare the logs byte for byte.
13. Print your generator's first draw for twenty consecutive seeds and look at the low bits.
14. Histogram a higher bit field, then print the sequence and look for a period.
15. Sweep at stride 4 and stride 10 and compare what each covers.
16. List what you would have to record for somebody else to act on a failure without your machine.
17. Find two of your run outcomes that differ in exactly one recorded field — and then try deleting that field.
18. Write the four-value status contract and say which real situation each value is for.

Check Your Understanding

1. Why is "run the tests and tell me if they passed" not a specification?
2. Name the four obligations and how each one fails.
3. Of thirteen measured behaviours, how many exited zero?
4. Name four non-passing behaviours that still exited zero.
5. What is wrong with inconsistent's log?
6. How do vacuous and smoke differ?
7. How do latecrash and smoke differ?
8. Why were vacuous and latecrash added to the fixture later?
9. What did the first version of flaky do, and why did it pass on every seed?
10. What are the low two bits of the first $random(seed) draw across seeds 1–20?
11. Bits 17:16 have a 5/6/5/4 histogram. Why is that not evidence of randomness?
12. What happens to a sweep at stride 4, and why is that frightening?
13. Why does the stride matter when nobody chooses it deliberately?
14. Give the three practical rules this section produces.
15. How do you tell a good all-passing sweep from one that could not fail?
16. What is the test for "recorded enough to act on"?
17. Why record the tool's exit status and the verdict?
18. Why record the check count?
19. What four situations must a status distinguish?
20. What goes wrong when 1, 2 and 3 are merged?
21. In the 24-run regression, why would "9 passed, 15 failed" be wrong?

7. Remember

• Four obligations: run a defined set, decide each verdict, record enough to act on, return a status a caller can gate on. None is automatic.
• Eleven of thirteen measured behaviours exited zero, including a design failure, a hang, a vacuous pass and a run with no verdict at all.
• Neither the exit status nor the verdict line is a verdict. Four runs printed no verdict; one printed PASSED next to an ERROR; one printed PASSED after zero checks.
• A sweep's size is not its coverage. The low two bits of $random's first draw were constant across twenty seeds, so a twenty-seed sweep exercised one branch and reported twenty passes.
• A uniform histogram is not variety — 5/6/5/4 came from a strict period-4 cycle.
• Five seeds at stride 4 gave one outcome. Coverage depended on a stride nobody chose.
• An all-passing sweep is a question. Break something deliberately and check that it notices.
• Record what distinguishes outcomes: latecrash differs from a clean pass only in the exit status, and vacuous only in the check count.
• A status must separate design, infrastructure and setup. "Non-zero means failures" sends people hunting bugs that do not exist.
• Fifteen of twenty-four runs were not a pass, in six different ways. Only three of them were failures.

8. What Is Next

The obligations are stated and the behaviours that make them necessary are measured. Chapter 32.2 lays out the driver that has to meet them — five stages, each writing a file the next one reads, so that any stage can be re-run on its own against the previous stage's output.