Linux & Bash · Part II · Module 34 · Defensive Automation Around EDA Tools

Licence Contention, Retries and Backoff

Some failures go away if you wait and most do not, and a retry loop that cannot tell the difference turns a deterministic bug into an intermittent one. This chapter builds a licence stand-in that really does refuse and then succeed, measures a classifying retry against a blind one, and documents the version of the wrapper that reported a failing design check as a success.

Advanced25 min readLinuxPart IIEDARetriesLicences

Part II · Module 34 · Chapter 34.3 · Licence Contention, Retries and Backoff

Chapters 34.1 and 34.2 made a failure reportable. This chapter is about the subset where reporting it is the wrong response, because the same command, run later, would succeed.

And about the much larger subset where it is the right response, and retrying does real damage.

1. A Licence That Really Does Refuse

The fixture needs a tool that fails for a transient reason, so it is a real one rather than a printed message. licsim refuses its first K invocations and then runs the simulator:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
c=$root/lic/attempts
n=$(cat "$c" 2>/dev/null || echo 0); n=$((n+1)); printf '%s\n' "$n" > "$c"
if [ "$n" -le "${FAIL_FIRST:-0}" ]; then
  echo "ERROR: Unable to check out licence 'sim_full' (attempt $n)" >&2
  echo "       All 4 licences are in use. Retry later." >&2
  exit 9
fi
exec vvp "$root/logs/sim.vvp" "$@"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  -- with FAIL_FIRST=2, three attempts --
    attempt 1: rc=9   ERROR: Unable to check out licence 'sim_full' (attempt
    attempt 2: rc=9   ERROR: Unable to check out licence 'sim_full' (attempt
    attempt 3: rc=0   TB  test=smoke seed=1 cycles=40 verbosity=LOW

A real state change across invocations, which is what a licence pool is. The counter file is the licence server.

2. The Retry

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# retry OUTDIR -- run one simulation, retrying ONLY the failures worth retrying.
#
# The distinction is the whole point. A licence shortage, a filesystem blip and
# a scheduler eviction are transient: the same command later may succeed. A
# failing check, a missing test and a compile error are not: retrying them
# produces N identical failures, wastes N times the compute, and -- worst --
# makes a deterministic bug look intermittent if any attempt differs.
set -u
d=${1:-}
[ -n "$d" ] || { echo "retry: need an output directory" >&2; exit 2; }
root=$(cd "$(dirname "$0")/.." && pwd)
shift || true
max=${MAX_ATTEMPTS:-4}
mkdir -p "$d" || exit 2
: > "$d/retry.log"

# Is this failure transient? Decided from the tool's own words, not its status,
# because a status cannot carry the reason.
transient() {
  local log=$1
  /usr/bin/grep -qE 'Unable to check out licence|All [0-9]+ licences are in use|license server|Resource temporarily unavailable|Input/output error|No space left on device' "$log"
}

attempt=0
while :; do
  attempt=$((attempt+1))
  ( cd "$d" && exec "$root/bin/licsim" "$@" ) > "$d/run.log" 2>&1
  rc=$?
  # NOT `[ $rc -eq 0 ]`. Chapter 34.1: the simulator exits 0 after TEST FAILED,
  # so a status-only retry treats a design failure as a success. Ask 34.2's
  # checker, which decides from the log.
  "$root/bin/logcheck" "$d/run.log" >/dev/null 2>&1; lc=$?
  printf 'attempt %d: rc=%s logcheck=%s\n' "$attempt" "$rc" "$lc" >> "$d/retry.log"
  [ "$lc" -eq 0 ] && { printf 'retry: succeeded on attempt %d of at most %d\n' "$attempt" "$max" >&2
                       printf '%s\n' "$attempt" > "$d/attempts"; exit 0; }
  if ! transient "$d/run.log"; then
    printf 'retry: attempt %d failed for a reason that will not change -- not retrying\n' "$attempt" >&2
    /usr/bin/grep -m1 -E 'ERROR|FATAL|TEST FAILED' "$d/run.log" 2>/dev/null | sed 's/^/       /' >&2
    printf '%s\n' "$attempt" > "$d/attempts"; exit 1
  fi
  if [ "$attempt" -ge "$max" ]; then
    printf 'retry: %d transient failures in a row -- giving up\n' "$attempt" >&2
    printf '%s\n' "$attempt" > "$d/attempts"; exit 3
  fi
  # exponential backoff, so a contended licence pool is not hammered
  sleep_for=$(( 1 << (attempt - 1) ))
  printf 'retry: transient failure on attempt %d, waiting %ds\n' "$attempt" "$sleep_for" >&2
  printf 'backoff %ds\n' "$sleep_for" >> "$d/retry.log"
  sleep "$sleep_for"
done

Three cases, measured.

A transient failure that clears:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  retry: transient failure on attempt 1, waiting 1s
  retry: transient failure on attempt 2, waiting 2s
  retry: succeeded on attempt 3 of at most 4
  attempts used: 3
  retry.log:
    attempt 1: rc=9 logcheck=2
    backoff 1s
    attempt 2: rc=9 logcheck=2
    backoff 2s
    attempt 3: rc=0 logcheck=0

A real design failure:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  retry: attempt 1 failed for a reason that will not change -- not retrying
         TB  ERROR t=35 gnt=10 expected 01
  attempts used: 1

A pool that never frees up:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  retry: transient failure on attempt 1, waiting 1s
  retry: transient failure on attempt 2, waiting 2s
  retry: 3 transient failures in a row -- giving up
  attempts used: 3

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  transient then success -> 0
  a real failure         -> 1
  transient, exhausted   -> 3

Three statuses, because they are three different reports. 1 is a design question. 3 says the flow never got to ask one, which is the environment's problem — Chapter 32.7's division by who acts, applied one level down.

My first version reported a failing design check as a success

The retry loop I wrote first tested the obvious thing:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

rc=$?
[ "$rc" -eq 0 ] && { echo "succeeded on attempt $attempt"; exit 0; }

and on a genuinely failing design check it said:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== 2. a REAL design failure: must not be retried ===
  retry: succeeded on attempt 1 of at most 4
  attempts used: 1

It reported success for a run whose log says TEST FAILED. The reason is Chapter 34.1's entire subject: $finish exits 0, so prio0 returns 0 and a status-based retry sees a success on the first attempt.

Two chapters earlier I had measured that five of five non-passing runs exit zero, and I still wrote [ "$rc" -eq 0 ] — because that is what a retry loop looks like everywhere it has ever been written. The fix is to ask the thing that already knows:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

"$root/bin/logcheck" "$d/run.log" >/dev/null 2>&1; lc=$?
printf 'attempt %d: rc=%s logcheck=%s\n' "$attempt" "$rc" "$lc" >> "$d/retry.log"
[ "$lc" -eq 0 ] && { … succeeded … }

And recording both numbers turns out to be the most useful line in the script:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

    attempt 1: rc=9 logcheck=2
    attempt 3: rc=0 logcheck=0

rc=9 logcheck=2 is a licence refusal — the tool failed and wrote no verdict. rc=0 logcheck=0 is a clean run. A row reading rc=0 logcheck=1 would be a design failure the tool called success, which is the pair worth grepping a night's logs for.

The general lesson is uncomfortable and worth stating plainly: a defensive wrapper built on top of a correct verdict function will still be wrong if it re-derives the verdict itself. Every layer that asks "did this work?" must ask the same component, or the flow has two definitions of success and they will disagree.

Figure 1 — the same failing design check through two retry loops. The blind loop runs it four times and waits seven seconds between attempts to learn what the first attempt already reported; the classifying loop reads the tool's own words, stops after one, and names the error.

3. Classify First, Always

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

transient() {
  local log=$1
  /usr/bin/grep -qE 'Unable to check out licence|All [0-9]+ licences are in use|license server|Resource temporarily unavailable|Input/output error|No space left on device' "$log"
}

The decision comes from the tool's own words, because a status cannot carry a reason. exit 9 tells you nothing about whether waiting would help; the sentence next to it tells you everything.

What belongs on that list, and what emphatically does not:

retryable	why
a licence checkout refused	the pool frees up
`Resource temporarily unavailable`	a process or fd limit under load
`Input/output error` on a network filesystem	a transient mount problem
`No space left on device`	sometimes — see 34.5
a scheduler eviction	the node was reclaimed

never retryable	why
a failing check	the design does not change between attempts
a test that does not exist	nor does the test list
a compile error	nor does the source
a testbench that contradicts itself	nor does the testbench
a run that checked nothing	nor does that

The second table is the one that matters, and the reason is not the wasted compute.

Retrying a real failure is how a deterministic bug becomes intermittent

Suppose the blind retry runs a failing test four times and, on the third attempt, something in the environment differs — a different node, a warmer cache, a parallel job that finished — and the run passes.

The loop now reports success. The regression is green. A real, reproducible design bug has been converted into a run that "passed on retry", and the record of the three failures is in a log nobody will read because the overall result was a pass.

That is strictly worse than the wasted time, and it is the reason classification comes before the loop rather than inside it. A retry is a statement that the failure was not about the work, and only the failure's own words can support that statement.

The same argument rules out two tempting conveniences:

Do not retry "just once, to be sure". One retry of an unclassified failure has exactly the property above, at half the cost.

And do not let a retry count be invisible. If a run succeeded on attempt 3, the result must say so — which is why attempts is written into the run directory and why Chapter 32.4's collector should carry it as a column. A regression where 40% of passes needed three attempts is a regression with a licence problem, and nothing in a pass rate shows it.

4. What Classification Is Worth

The same real design failure, through both loops:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== a retry loop with NO classification, on a real design failure ===
  failed after 4 attempts
  wall time: 8s (4 simulations + 1s + 2s + 4s of backoff)

=== the classifying retry, same failure ===
  retry: attempt 1 failed for a reason that will not change -- not retrying
  wall time: 0s  attempts: 1

Four attempts and eight seconds against one attempt and zero. On this fixture that is trivial; the scaling is not. A two-hour simulation retried four times is eight hours of compute to learn what the first attempt already said, and on a nightly regression with two hundred genuine failures it is the difference between a report at 6 a.m. and no report at all.

And the backoff compounds:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  attempts   backoff sequence           total wait
  3          1s 2s                      3s
  4          1s 2s 4s                   7s
  5          1s 2s 4s 8s                15s
  6          1s 2s 4s 8s 16s            31s
  8          1s 2s 4s 8s 16s 32s 64s    127s

Doubling is the right shape — a contended licence pool should not be hammered, and each wait gives the thing you are waiting for twice as long to resolve. But the total is 2^(n-1) - 1, so the last attempt costs more than all the others combined, and an eight-attempt policy spends over two minutes waiting even if every attempt is instant.

Three practical consequences:

Bound the attempts, not the time. MAX_ATTEMPTS=4 is legible; "retry for ten minutes" is a loop whose iteration count depends on how fast the tool fails.

Cap the individual wait. sleep_for=$(( 1 << (attempt-1) )) is unbounded in principle; [ "$sleep_for" -gt 60 ] && sleep_for=60 keeps a long policy sane.

And add jitter when many jobs retry together. Eight parallel runs that all lose a licence at the same moment will all retry at the same moment, twice, and then four times — synchronised contention that the backoff was supposed to prevent. A random fraction of the interval breaks the lockstep, and Chapter 31.3's rule applies: if the jitter matters, record the seed.

5. Choosing the Attempt Limit

MAX_ATTEMPTS is the one number this wrapper asks you for, so it is worth measuring rather than guessing. The licence stand-in takes a refusal count, which is exactly the knob needed:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  refusals       MAX=2      MAX=4        MAX=6      wall at MAX=4
  0              ok(1)      ok(1)        ok(1)      0s
  1              ok(2)      ok(2)        ok(2)      1s
  2              gave up    ok(3)        ok(3)      3s
  3              gave up    ok(4)        ok(4)      7s
  4              gave up    gave up      ok(5)      7s
  5              gave up    gave up      ok(6)      7s

Three readings, and the third is the useful one.

MAX=N tolerates N-1 refusals. Obvious once stated and easy to get wrong by one: a policy of "retry three times" usually means four attempts, and whether your loop counts attempts or retries decides whether it survives three refusals or two. The wrapper here counts attempts, which is why the message says succeeded on attempt 3 of at most 4.

The wall time saturates. At MAX=4 the cost rises 0s → 1s → 3s → 7s and then stays at 7s however bad the contention gets, because the loop has stopped. A bounded retry has a bounded cost, and that cost is the full backoff sum — which is the number to put in a nightly's budget, multiplied by how many runs can plausibly hit it at once.

And the limit should come from the contention, not from taste. The question the table answers is how many simultaneous refusals does this pool actually produce? That is measurable in a way a round number is not:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# over a week of nightlies, how many attempts did passing runs need?
$ awk -F'\t' 'NR>1 && $4=="PASS" {a[$12]++} END {for (k in a) printf "%s attempt(s): %d runs\n", k, a[k]}' \
    regr/*.results.tsv | sort -n
1 attempt(s): 1842 runs
2 attempt(s): 71 runs
3 attempt(s): 4 runs

If nothing ever needed more than three, MAX_ATTEMPTS=4 is measured rather than chosen, and a run that exhausts it is genuinely unusual — which is what makes the 3 status worth reporting. That query needs the attempt count in the results file, which is the argument from Section 3 for recording it as a column.

Two limits, not one

The attempt count bounds how many times one run is tried. It does not bound the regression's exposure, and those are different budgets.

Two hundred runs against a contended pool, each allowed four attempts, can spend 200 × 7 s = 23 minutes purely in backoff — with every run eventually succeeding, so nothing in the report says anything happened. Chapter 32.4's seconds column would show it; a pass rate would not.

So a regression wants a second, aggregate guard:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# if a quarter of the runs so far have needed a retry, the pool is the problem
if [ "$done" -ge 40 ] && [ $(( retried * 4 )) -ge "$done" ]; then
  printf 'regress: %d of %d runs needed a retry -- reduce SLOTS rather than waiting\n' \
    "$retried" "$done" >&2
fi

A warning rather than a refusal, because the runs are succeeding and the results are good. What is wrong is the slot count, and Chapter 31.5 measured that licences are one of the four bounds on it. A retry rate is the signal that the bound was set from cores rather than from licences.

6. The Licence Case Specifically

Licences deserve a note because the obvious fix is the wrong one.

Do not retry your way out of a licence shortage. If a regression launches sixteen jobs against four licences, twelve of them will fail, retry, fail again and eventually exhaust their attempts — and the regression's wall time becomes a function of the backoff rather than of the work. Chapter 31.5 measured the slot-count decision; the licence count is one of the four bounds listed there, and the correct response to a licence limit is a smaller slot count, not a longer retry.

Retries are for the residual: the case where the slot count is right and a colleague's interactive session took the last licence for ten minutes.

And distinguish "no licence now" from "no licence ever". The two produce different messages, and only the first is transient:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  ERROR: Unable to check out licence 'sim_full' (attempt 1)
         All 4 licences are in use. Retry later.

against the thing a misconfigured environment produces, which Chapter 25.4 measured — a server that cannot be reached, or a feature that is not licensed at all. Retrying that four times with backoff delays a configuration error by seven seconds and then reports it as a transient failure, which is the wrong report.

The discriminating test is whether the message names a contention or an absence, and it is worth being explicit in the pattern rather than matching the word "licence":

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# contention -- retry
'Unable to check out|All [0-9]+ licences are in use|licence queue'
# absence or misconfiguration -- do not retry
'Cannot connect to license server|No such feature exists|Invalid license file'

Practice

Build a tool stand-in that fails its first K invocations for a transient reason and then succeeds.
Make the state real — a counter file, not a printed message.
Write a retry loop and run it on the transient failure.
Run the same loop on a genuinely failing test and see what it reports.
If it reported success, work out why, and name the chapter that explains it.
Replace the status test with a verdict check and re-run both cases.
Record both the status and the verdict per attempt, and say what the pair tells you.
Find the pair that means "a design failure the tool called success".
Write the transient classifier from the tool's own messages.
List what belongs on each side, and defend the hardest case.
Construct the scenario where retrying a real failure produces a green regression.
Explain why that is worse than the wasted time.
Argue against "just one retry, to be sure".
Record the attempt count in the run directory and add it as a results column.
Measure the wall time of a blind retry against a classifying one on the same failure.
Scale that to a two-hour simulation and a nightly regression.
Tabulate the backoff totals for three to eight attempts.
Work out why the last attempt costs more than all the others.
Cap the individual wait and bound the attempts rather than the time.
Add jitter and say when it matters.
Compute how many of sixteen jobs fail against four licences, and what the right fix is.
Find two licence messages — one contention, one absence — and classify them differently.

Check Your Understanding

What subset of failures is this chapter about?
Why is the licence stand-in a counter file rather than a printed message?
Give the three cases and their statuses.
Why are "a real failure" and "transient, exhausted" different statuses?
What did the first retry loop report for a failing design check, and why?
Which earlier chapter explains it?
What is the fix, and what does recording both numbers give you?
What does rc=0 logcheck=1 mean?
State the general lesson about layers and verdicts.
Why must the transient decision come from the tool's words?
Name three retryable failures and three that never are.
Why is "retrying a real failure" worse than wasted compute?
What is wrong with "just one retry, to be sure"?
Why must the attempt count be visible in the result?
What would a regression where 40% of passes needed three attempts tell you?
Give the measured wall times for both loops.
What is the backoff total for eight attempts, and why is the last one so expensive?
Why bound the attempts rather than the time?
Why cap the individual wait?
When does jitter matter, and what does Chapter 31.3 add?
What is the right response to a licence shortage, and why not retries?
What are retries for, in the licence case?
How do you distinguish "no licence now" from "no licence ever", and why does it matter?

Answers

The subset where reporting the failure is the wrong response, because the same command run later would succeed.
Because a real state change across invocations is what a licence pool is; a printed message would not test the loop.
Transient then success → 0; a real failure → 1; transient but exhausted → 3.
Because 1 is a design question and 3 says the flow never got to ask one — the environment's problem.
"Succeeded on attempt 1", because it tested [ "$rc" -eq 0 ] and $finish exits 0 after TEST FAILED.
Chapter 34.1 — five of five non-passing runs exited zero.
Ask 34.2's logcheck instead of the status. Recording both gives a pair per attempt: rc=9 logcheck=2 is a licence refusal, rc=0 logcheck=0 a clean run.
A design failure that the tool reported as a success — the pair worth grepping a night's logs for.
A defensive wrapper built on a correct verdict function will still be wrong if it re-derives the verdict itself. Every layer must ask the same component.
Because a status cannot carry a reason: exit 9 says nothing about whether waiting would help, and the sentence beside it says everything.
Retryable: a refused licence checkout, Resource temporarily unavailable, a transient network-filesystem I/O error. Never: a failing check, a non-existent test, a compile error.
Because if any attempt happens to pass, the loop reports success — converting a reproducible design bug into a "passed on retry", with the failures buried in a log nobody opens because the result was green.
It has exactly that property at half the cost.
So that a result which needed three attempts says so, rather than looking identical to a first-attempt pass.
That the regression has a licence problem — and nothing in a pass rate would show it.
Blind: 4 attempts, 8 seconds. Classifying: 1 attempt, 0 seconds.
127 seconds. The total is 2^(n-1) - 1, so the final wait exceeds the sum of all the earlier ones.
Because a time bound makes the iteration count depend on how fast the tool fails, which is not legible.
Because doubling is unbounded in principle, and a long policy otherwise spends minutes asleep.
When many jobs retry together — they lose a licence at the same moment and retry in lockstep. Chapter 31.3 adds that if the jitter matters, the seed must be recorded.
A smaller slot count. Retries make the wall time a function of the backoff rather than of the work, and the licence count is one of the four bounds on parallelism.
The residual: the slot count is right and somebody's interactive session took the last licence for ten minutes.
By whether the message names contention or absence — "all 4 licences are in use" against "cannot connect to license server". Retrying an absence delays a configuration error and then reports it as transient.

7. Remember

Retry only what the tool's own words say is transient. A status cannot carry a reason.
My first retry loop called a failing design check a success, because it tested [ $rc -eq 0 ] and $finish exits 0. Ask the verdict function, never the status.
A wrapper that re-derives the verdict will disagree with the one that defines it. Every layer asks the same component.
Record the status and the verdict per attempt. rc=0 logcheck=1 is a failure the tool called success.
Three statuses: succeeded after retrying, a real failure, and transient-but-exhausted.
Retrying a real failure is worse than wasting compute — if any attempt passes, a deterministic bug becomes a green regression.
No "one retry to be sure". Same defect, half the price.
Make the attempt count visible in the run and in the results file.
4 attempts and 8 seconds against 1 and 0 on the same failure — and that scales to hours.
Backoff totals 2^(n-1) - 1: the last attempt costs more than all the others. Bound attempts, cap the wait, add jitter when jobs retry together.
A licence shortage is a slot-count problem, not a retry problem. Retries are for the residual.
Distinguish contention from absence. Retrying a misconfiguration delays it and then mislabels it.

8. What Is Next

A retry needs a failure to react to. Chapter 34.4 handles the case where there is none — a tool that neither succeeds nor fails but simply does not finish — on a machine where the obvious tool for the job is not installed.

Where this fits

Part of the Linux & Bash curriculum.