Reproducing a Run

Part II · Module 29 · Chapter 29.5 · Reproducing a Run

Chapter 29.2 established the identity of a run's inputs. This chapter is what that is for: a month later, somebody asks whether a result is still true, and the only honest answers are "yes, here it is again" or "I cannot tell".

1. The test is a replay

2. What goes in

namespace eval ::run { variable KEYS {design stage pattern sdc seed effort} }

proc ::run::record {dir args} {
    ...
    # 1. what was asked for
    foreach k $::run::KEYS { puts $fh [format "cfg\t%s\t%s" $k [dict get $cfg $k]] }
    # 2. what it ran on
    puts $fh [format "env\ttcl\t%s" [info patchlevel]]
    puts $fh [format "env\thasher\t%s" $::prov::HASHER]
    puts $fh [format "env\tscript\t%s" [file tail [info script]]]
    # 3. the identity of every input
    foreach f [dict get $cfg inputs] {
        puts $fh [format "input\t%s\t%s" [::prov::digest $f] $f]
    }
    close $fh
}

=== the record written beside it ===
cfg	design	core
cfg	stage	synth
cfg	pattern	u_core/*
cfg	sdc	constraints/c.sdc
cfg	seed	7
cfg	effort	high
env	tcl	8.5.9
env	hasher	tcllib
env	script	probe7.tcl
input	854A7178B03C	rtl/core.sv
input	1B1685A8BC30	constraints/c.sdc

Three sections, and the division is the design.

cfg — what was asked for. Every argument that changes the result. The declared KEYS list means a record missing one of them is rejected at write time, which is Chapter 27.4's contract applied to a record instead of to a session.

env — what it ran on. Interpreter version, which hasher was available, which script. A result is only comparable against another from the same tool and version, which is Chapter 24.2 §6's rule for the fourth time in Part II.

input — the identity of every input. Digests rather than timestamps, for Chapter 29.2's reasons. This is the section that makes "can this be replayed" answerable.

Note the seed is in cfg. A randomised step without its recorded seed is a run that cannot be repeated at all — Chapter 25.5 §1's point, and the field people most often leave out.

3. The replay

set d [::run::read $m]
set cfg [dict get $d cfg]
set replay [run_flow_checked [dict get $cfg pattern] [dict get $cfg sdc]]

Every argument comes from the record. Nothing is passed in.

=== replay, reading every argument from the record ===
    original : cells 4  sized 2  viol 2  FAIL
    replay   : cells 4  sized 2  viol 2  FAIL
    identical: yes

Identical, including the verdict and every count. That is the property worth having and it is the one that is usually assumed rather than checked.

And the check that precedes it:

=== is it replayable? ===
    inputs 2  missing 0  changed 0

replayable asks before the replay is attempted, which is the useful form. "Can this still be reproduced?" is asked far more often than "reproduce it" — in a review, in a bug triage, when deciding whether an old result still counts — and answering it costs two digests rather than a run.

4. The four ways a record stops working

=== an input is EDITED after the run ===
    inputs 2  changed 1
        changed: rtl/core.sv
    -> the recorded result cannot be reproduced from the current tree

=== an input is DELETED ===
    inputs 2  missing 1  changed 1
        missing: constraints/c.sdc

=== a record that is itself incomplete ===
    1 / work/bad.manifest: record is missing cfg 'pattern'

=== and a malformed line ===
    1 / work/bad2.manifest: malformed: cfg design core

Four distinct failures, each named and located.

An input was edited. The digest moved, so the recorded result describes a tree that no longer exists. Note this is not an error in the usual sense — it is the normal state of affairs a month later, and the right response is "that result is historical", not "something is broken".

An input was deleted. Reported separately from "changed" because the responses differ: a changed file can be recovered from version control, and a deleted one may not exist anywhere.

The record is incomplete. missing cfg 'pattern' — caught by the same KEYS list that guards the write, so a record written by an older version of the script is rejected rather than silently replayed with a default.

A line is malformed. malformed: cfg design core — the line itself is in the message, because a manifest is usually hand-edited at least once and the edit is usually a space where a tab should be. Chapter 28.3 §5's rule: name the file and show the line.

5. A Tcl subtlety in the record itself

env	script	probe7.tcl

::run::record is defined in record.tcl and records probe7.tcl — the file that called it.

=== what does info script report inside a proc? ===
    defined in : flow/record.tcl
    running in : probe8.tcl  <- the file being EVALUATED now
    from a proc at top level : 'probe8.tcl'

6. What a replay does not prove

The honest limits, because §3's identical: yes is persuasive.

A replay reproduces the computation, not the tool. Identical inputs through a different tool release can give a different answer — a changed optimisation heuristic, a fixed timing-calculation bug, a new default. That is why the version is in the record, and recording it makes a mismatch visible rather than impossible. The version is a precondition for comparison, not a guarantee of equality.

The environment is larger than the record. Licence server, available memory, thread count, locale, umask, every environment variable the tool reads. A record naming all of them is unmaintainable; a record naming none of them is a record that works until one of them matters. The practical line is to record the ones the flow sets (Chapter 28.1 §5) and accept that the rest is ambient.

Parallelism makes some results genuinely irreproducible. A tool that distributes work across threads and merges results in completion order can produce a different-but-equally-valid answer each run. No record fixes that; the honest handling is to know whether your tool is deterministic and to say so next to the result.

And a replay from the same inputs cannot tell you whether the inputs were right. Chapter 29.2 is upstream of this: reproducing an analysis of the wrong netlist reproduces it faithfully.

7. What this costs and when to pay it

Being practical, because "record everything always" is not a plan.

Hashing is per stage boundary, not per command. A 2 GB netlist takes real time to digest, so the record is written once when a stage completes — which is also exactly when the inputs are known and the outputs exist.

The record lives beside the results. Not in a database, not in a wiki — in the stage's own output directory, named for the run (Chapter 26.6 §1). A record somewhere else is a record that gets separated from what it describes.

And the one thing to record even when nothing else is affordable: the command line and the tool version. Those two cost nothing, go stale slowest, and answer the most common question — "what did you actually run?" Everything else in this chapter is an improvement on a baseline that most flows do not yet have.

8. Exercises

Exercise 1. Why is reading a record not a test of it?

Exercise 2. What are the three sections of the record, and what does each answer?

Exercise 3. Why is the seed in cfg, and what happens without it?

Exercise 4. What does the replay demonstrate that an inspection cannot?

Exercise 5. Why is replayable more useful than the replay itself?

Exercise 6. Why report a changed input separately from a missing one?

Exercise 7. Why is an edited input not really an error?

Exercise 8. Why does the malformed-line message include the line?

Exercise 9. What does info script return inside a procedure, and what did it record here?

Exercise 10. How do you capture a library's own identity instead?

Exercise 11. Why does a replay not prove the tool would agree?

Exercise 12. Name two things that make a result genuinely irreproducible regardless of the record.