Tcl · Module 27

Idempotent Helper Procedures

A flow script is re-run far more often than it is written, so running it twice must be harmless. Finding out whether it is cannot be done by reading the code.

Intermediate20 min readtcledapart-iiinfrastructureidempotencyhelpers

Part II · Module 27 · Chapter 27.7 · Idempotent Helper Procedures

A flow script gets written once and run hundreds of times — after a crash, from a partial state, at a prompt while debugging, and by a driver that retries. So the question "what does this do if it runs twice" is not hypothetical; it is the normal case.

1. The same setup, run twice

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc setup_naive {} {
    create_clock_naive core_clk 5.0
    create_clock_naive io_clk  10.0
    add_lib_path_naive libs/std
    add_lib_path_naive libs/io
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the naive setup, run once then twice ===
    after 1 run : clocks core_clk=5.0 io_clk=10.0  libdirs 2 -> libs/std libs/io
    after 2 runs: clocks core_clk=5.0 io_clk=10.0  libdirs 4 -> libs/std libs/io libs/std libs/io

One function, one second call, two different outcomes — and the difference is the data structure.

The clocks are unchanged: dict set on an existing key overwrites it, so writing core_clk = 5.0 twice leaves one entry with the same value. That operation is accidentally idempotent — nobody designed it to be.

The library paths are duplicated: lappend appends, so the list goes from two entries to four. That operation is not idempotent, and nothing in the calling code distinguishes it from the one that is.

So "is this helper safe to run twice" is not answerable by reading the helper. It depends on what each line does to its container:

Operation	Second call	Idempotent?
`dict set d k v`	overwrites	yes, accidentally
`set ::X $v`	overwrites	yes, accidentally
`lappend l $v`	appends	no
`incr n`	increments	no
`append s $text`	concatenates	no
`file mkdir $d`	succeeds (§4)	yes, on a directory
`proc name {} {}`	replaces	yes

The accidental cases are the dangerous ones, because they make a script look idempotent in testing. A setup that only sets variables and defines procedures survives a double run, so nobody checks — and then one lappend is added for a library path and the property is gone with no visible change.

2. Making it deliberate

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# Returns what it DID, so a caller and a log can tell "created" from
# "already there". Returning nothing would make the two indistinguishable.
proc create_clock_if_missing {name period} {
    if {[dict exists $::flow::CLOCKS $name]} {
        set have [dict get $::flow::CLOCKS $name]
        if {$have != $period} {
            error "clock $name already defined with period $have, not $period" \
                  {} [list FLOW CLOCKCONFLICT $name $have $period]
        }
        return "unchanged $name"
    }
    dict set ::flow::CLOCKS $name $period
    return "created $name"
}

proc add_lib_path_once {dir} {
    if {[lsearch -exact $::flow::LIB_DIRS $dir] >= 0} { return "unchanged $dir" }
    lappend ::flow::LIB_DIRS $dir
    return "added $dir"
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the idempotent setup, from a clean state ===
    run 1:
      created core_clk
      created io_clk
      added libs/std
      added libs/io
    after 1 run : clocks core_clk=5.0 io_clk=10.0  libdirs 2
    run 2:
      unchanged core_clk
      unchanged io_clk
      unchanged libs/std
      unchanged libs/io
    after 2 runs: clocks core_clk=5.0 io_clk=10.0  libdirs 2

2 library paths after two runs, and the second run says unchanged four times.

The return values are the part people leave out. A helper that returns nothing makes "I just created it" and "it was already there" indistinguishable — and that distinction is exactly what you need when a re-run was supposed to be a no-op and something changed anyway. Printing them turns a silent re-run into a one-line statement of what it actually did.

3. A conflict is not a no-op

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== and a conflict is not a no-op ===
    1 / clock core_clk already defined with period 5.0, not 4.0

There are three outcomes, not two: created, unchanged, and conflict.

create_clock_if_missing core_clk 4.0 when core_clk already exists at 5.0 is not "already done". Somebody asked for a 4 ns clock and the session has a 5 ns one — and the helper has no way to know which is right.

A helper that treated that as unchanged would silently keep the wrong period, and every timing number afterwards would be computed against a constraint nobody asked for. That is the Part II failure shape once more: a correct-looking operation producing a confident wrong result.

So if_missing must mean "create it if absent, confirm it if present, refuse if it disagrees". The name is a slight lie and the behaviour is right; create_clock_or_confirm would be more honest and nobody would type it.

Note $have != $period rather than ne: periods are numbers, so 5.0 and 5.00 must compare equal, and ne would call them different (Chapter 4.3). Choosing the numeric comparison here is a small decision that prevents a spurious conflict every time somebody writes a period with a different number of decimal places.

4. `file mkdir` is already idempotent — but only on a directory

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== what plain file mkdir does in the same two cases ===
    on an existing dir  : 0 / no error
    on an existing file : 1 / can't create directory "work/notadir": file already exists

file mkdir on an existing directory succeeds silently, so the usual if {![file exists $d]} {file mkdir $d} guard is unnecessary. On an existing file it fails.

So the check that matters is the type, not the existence:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc ensure_dir {path} {
    if {[file isdirectory $path]} { return "unchanged $path" }
    if {[file exists $path]} {
        error "$path exists and is not a directory" {} [list FLOW NOTADIR $path]
    }
    file mkdir $path
    return "created $path"
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== ensure_dir: directory, twice, then a file in the way ===
    created work/idem
    unchanged work/idem
    1 / work/notadir exists and is not a directory

[file exists] is the wrong test and [file isdirectory] is the right one — the same distinction as Chapter 14.7's open on a directory, where file exists passed and the failure arrived at read. A name that exists is not a name you can use for what you wanted.

And the error message says what is wrong — exists and is not a directory — rather than passing through can't create directory, which describes the attempt rather than the cause.

5. What idempotency is worth, and what it is not

Being honest, because "make everything idempotent" is not free.

It makes a re-run a diagnostic. If running the setup twice prints unchanged throughout, the session is in the state the script describes. If it prints created, something had been lost — which is information you did not have before.

It does not make a flow resumable. A helper being safe to run twice says nothing about whether the flow can be restarted from the middle. Re-running create_clock_if_missing is harmless; re-running a synthesis step on a netlist that was already optimised is a different design. Step-level resumability needs recorded state — which stage completed, with what inputs — and is a much larger thing than helper-level idempotency.

It cannot protect state the helper does not own. create_clock_if_missing checks ::flow::CLOCKS because that is the mock's own record. In a real tool the clock lives in the tool's database, and the check has to ask the tool — get_clocks, or whatever that tool calls it. Checking your own bookkeeping instead of the tool's is how a helper comes to believe a clock exists that was lost when the design was reloaded.

Not everything should be. A step that deliberately regenerates a report should regenerate it; making it skip when the file exists produces stale reports that look current, which is worse than the cost of rewriting one. The question is whether the operation's purpose is to reach a state (idempotent) or to do work (not).

The failure mode of over-applied idempotency is a stale artefact that looks fresh.

A write_netlist_if_missing saves a few seconds and produces, eventually, a netlist from two revisions ago sitting next to timing reports from today — with nothing in either file saying they disagree.

So the rule is narrower than "make helpers idempotent": make helpers that establish a state idempotent, and make helpers that produce an artefact overwrite. ensure_dir and create_clock_if_missing establish states. write_netlist produces an artefact and must always write.

When something expensive genuinely should be skipped, the condition must be freshness rather than existence — compare modification times, or record the inputs' revision in the output — which is Chapter 24.2 §6's provenance argument doing a second job.

6. Exercises

Exercise 1. Why is "what does this do if it runs twice" not a hypothetical question?

Exercise 2. The naive setup left clocks unchanged and library paths duplicated. Why the difference?

Exercise 3. Why are the accidentally-idempotent operations the dangerous ones?

Exercise 4. Why do the idempotent helpers return created / unchanged rather than nothing?

Exercise 5. What are the three outcomes of create_clock_if_missing, and what goes wrong if the third is treated as the second?

Exercise 6. Why $have != $period rather than ne?

Exercise 7. Is file mkdir idempotent? Answer precisely.

Exercise 8. Why is file isdirectory the right test and file exists the wrong one?

Exercise 9. Why does helper-level idempotency not make a flow resumable?

Exercise 10. Why can a helper not protect state it does not own?

Exercise 11. State the narrower rule about which helpers should be idempotent, and the failure mode of getting it wrong.

Answers.

1. Because a flow script is run hundreds of times for every time it is written — after a crash, from a partial state, at a prompt, and by a driver that retries.

2. dict set overwrites an existing key, so writing the same clock twice changes nothing. lappend appends, so the list went from two entries to four. The data structure decides, not the calling code.

3. Because they make a script look idempotent in testing. A setup that only sets variables and defines procedures survives a double run, so nobody checks — and then one lappend removes the property with no visible change.

4. Because otherwise "I just created it" and "it was already there" are indistinguishable, and that is exactly the distinction needed when a re-run was supposed to be a no-op and something changed.

5. Created, unchanged, and conflict. Treating a conflict as unchanged silently keeps the wrong period, so every later timing number is computed against a constraint nobody asked for.

6. Because periods are numbers, so 5.0 and 5.00 must compare equal. ne is a string comparison and would report a spurious conflict.

7. On an existing directory, yes — it succeeds with no error. On an existing file, no — it fails with "file already exists".

8. Because a name that exists is not necessarily a directory you can use. file exists passes for a file in the way, and the failure then arrives from mkdir with a message about the attempt rather than the cause.

9. Because a helper being safe to run twice says nothing about whether a step can be restarted. Re-running a clock definition is harmless; re-running synthesis on an already-optimised netlist produces a different design. Resumability needs recorded stage state.

10. Because it can only check its own bookkeeping. In a real tool the clock lives in the tool's database, so the check must ask the tool — otherwise the helper believes in a clock that was lost when the design was reloaded.

11. Make helpers that establish a state idempotent, and helpers that produce an artefact always overwrite. Getting it wrong gives a stale artefact that looks fresh — a netlist from two revisions ago beside today's timing reports, with nothing saying they disagree.

Remember.

A flow script is re-run far more than it is written, so a double run is the normal case.
Whether a helper is idempotent depends on its data structure, not its intent: dict set overwrites, lappend appends.
The accidentally-idempotent operations are the dangerous ones — they make a script look safe until one lappend is added.
Return created / unchanged, or a re-run that was supposed to change nothing cannot be distinguished from one that did.
Three outcomes, not two: created, unchanged, conflict. Treating a conflict as a no-op keeps the wrong value silently.
Compare numbers numerically — 5.0 and 5.00 are the same period.
file mkdir is idempotent on a directory and fails on a file, so test file isdirectory, not file exists.
Report the cause, not the attempt, in the error message.
Idempotent helpers do not make a flow resumable — that needs recorded stage state.
A helper can only check state it owns. Ask the tool about the tool's database.
Make state-establishing helpers idempotent and artefact-producing helpers overwrite. The alternative is a stale file that looks current.

Where this fits

Part of the Tcl curriculum.

1. The same setup, run twice

2. Making it deliberate

3. A conflict is not a no-op

4. file mkdir is already idempotent — but only on a directory

5. What idempotency is worth, and what it is not

6. Exercises

Where this fits

4. `file mkdir` is already idempotent — but only on a directory