Linux & Bash · Part II · Module 35 · Reusable Flow Tooling and Hand-Off

Configuration Instead of Hardcoding

A flow that names one design can serve one design. This chapter inventories what is actually hardcoded in a working flow, separates the flow's own structure from the design it happens to be pointed at, builds a loader with four precedence levels — and finds that the first version of that loader did not implement the precedence its own comment described.

Advanced26 min readLinuxPart IIConfigurationReuseFlow

Part II · Module 35 · Chapter 35.2 · Configuration Instead of Hardcoding

Chapter 35.1 removed the duplicated logic. The scripts that remain still contain the design they were written against, and that is what stops one flow serving two blocks.

The first step is to find out what is actually in there, because the usual guess is wrong.

1. The Inventory

Chapter 34.7's working flow, grepped:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  literal path fragments             9
    $root/bin/bounded
    $root/bin/compile
    $root/bin/flowlib.sh
    $root/bin/logcheck
    $root/bin/spacecheck
    $root/bin/spaceclass
    $root/cfg/files.f
    $root/rtl/arb.v
    $root/tb/tb_arb.v

  defaulted knobs (${X:-y})          5
    ${CYCLES:-200}  ${LIMIT_S:-30}  ${MAX_RUN_BLOCKS:-5242880}
    ${NEED_MB:-10}  ${SEED:-1}

  literal test/tool names            5
    arb.v files.f iverilog tb_arb.v vvp

Nineteen things, and they are not all the same kind of thing.

what	example	move it?
the flow's own tools	`$root/bin/compile`	no — that is the flow's structure
the design's sources	`$root/rtl/arb.v`	yes — this is the hardcoding
tool names	`iverilog`, `vvp`	yes, with a default
knobs with sane defaults	`${CYCLES:-200}`	yes, and they already are

Six of the nine paths are the flow pointing at itself, and Chapter 35.1 argued those must stay hardcoded — a configurable library path makes a script's behaviour depend on a variable set before it ran. The three that matter are cfg/files.f, rtl/arb.v and tb/tb_arb.v: the design.

That is the distinction worth taking from this section, because it is what keeps a configuration file small: configure what varies between users of the flow, not everything that is a string. A flow whose own tool paths are configurable has thirty settings and no reviewers.

2. A Loader With a Documented Precedence

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# lib/conf.sh -- read key=value configuration with a documented precedence.
#
# PRECEDENCE, lowest to highest:
#   1 cfg/defaults.conf        the flow's own defaults
#   2 cfg/<PROJECT>.conf       the block's configuration
#   3 the environment          CONF_<KEY>, for one run
#   4 command-line flags       handled by the caller, highest
#
# The files are PARSED, not sourced. A sourced config can run any command, so
# `. cfg/chip_a.conf` makes every config file a program -- and a flow that
# sources config cannot be reviewed by reading the flow.
# conf_snapshot_env -- remember which CONF_* came from the ENVIRONMENT, before
# any file is read. Without this, conf_load overwrites them and the documented
# precedence is a comment rather than a behaviour. Measured: CONF_CYCLES=9999
# in the environment lost to a config file's 400.
conf_snapshot_env() {
  CONF_FROM_ENV=$(set | /usr/bin/grep -E '^CONF_[A-Z_0-9]+=' | cut -d= -f1 | tr '\n' ' ')
  CONF_FROM_ENV=" $CONF_FROM_ENV"
}
conf_load() {
  local f=$1 k v name
  [ -r "$f" ] || return 0                      # a missing file is not an error
  while IFS= read -r line || [ -n "$line" ]; do
    line=${line%%#*}                           # strip comments
    case $line in *=*) ;; *) continue;; esac    # ignore anything without =
    k=${line%%=*}; v=${line#*=}
    k=$(printf '%s' "$k" | tr -d ' \t')
    v=$(printf '%s' "$v" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
    [ -n "$k" ] || continue
    case $k in *[!a-z_0-9]*) printf 'conf: ignoring key %s in %s\n' "$k" "$f" >&2; continue;; esac
    name=CONF_$(printf '%s' "$k" | tr 'a-z' 'A-Z')
    # the environment outranks every file, so never overwrite one of its keys
    case ${CONF_FROM_ENV:-} in *" $name "*) continue;; esac
    eval "$name=\$v"
  done < "$f"
}
# conf_get KEY DEFAULT -- the resolved value, with the environment winning.
conf_get() {
  local k u
  k=$1; u=$(printf '%s' "$k" | tr 'a-z' 'A-Z')
  eval "printf '%s\n' \"\${CONF_$u:-\$2}\""
}
# conf_dump -- every resolved value, so a run can record what it was given.
conf_dump() { set | /usr/bin/grep -E '^CONF_[A-Z_0-9]+=' | /usr/bin/grep -v '^CONF_FROM_ENV=' | sort; }

Four levels, and the measurement:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== precedence, lowest to highest ===
  1. defaults only
    cycles=200    limit_s=30    design_root=none           seed=1
  2. + the block's config (chip_a sets cycles=400)
    cycles=400    limit_s=30    design_root=proj/chip_a    seed=1
  3. + the environment (CONF_CYCLES=9999)
    cycles=9999   limit_s=30    design_root=proj/chip_a    seed=1
  4. + a flag, which the caller applies last
    cycles=50     limit_s=30    design_root=proj/chip_a    seed=1

Chapter 21.3 establishes the order — defaults, then configuration, then environment, then flags — and the reason it is that order is about scope: a default serves everybody, a file serves a block, the environment serves one run, and a flag serves one invocation. Each narrower scope wins.

My first loader did not implement the precedence its own comment described

The comment block at the top of conf.sh said the environment outranks the files. Measured:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  -- plus the environment --
    cycles=400  <-- the environment did NOT win

CONF_CYCLES=9999 was exported, and a config file's 400 overwrote it — because conf_load assigns unconditionally, and files are loaded after the environment exists. The documented precedence was a comment; the implemented precedence was "last file wins".

The fix needs a snapshot taken before any file is read, because once a file has assigned CONF_CYCLES there is no way to tell whether the environment had set it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

conf_snapshot_env() {
  CONF_FROM_ENV=$(set | /usr/bin/grep -E '^CONF_[A-Z_0-9]+=' | cut -d= -f1 | tr '\n' ' ')
  CONF_FROM_ENV=" $CONF_FROM_ENV"
}

and then one guard inside the loop:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

name=CONF_$(printf '%s' "$k" | tr 'a-z' 'A-Z')
case ${CONF_FROM_ENV:-} in *" $name "*) continue;; esac
eval "$name=\$v"

The leading and trailing spaces in CONF_FROM_ENV are load-bearing: without them *"$name"* matches a prefix, so CONF_SEED in the environment would also protect CONF_SEED_COUNT from being set by a file. That is Chapter 33.5's unanchored-match hazard arriving in a variable test rather than a grep.

The general lesson is the uncomfortable one this track keeps producing: a documented behaviour that is not tested is a documented intention. The comment was written first, the code was written to match it, and the two disagreed from the first line — and nothing in the script's own output would ever have said so. Chapter 35.1's contract tests are the answer, and the precedence table above is now four assertions rather than four sentences.

3. Parsed, Never Sourced

The shortest config loader in the world is one line:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

. "cfg/$proj.conf"          # DO NOT

and it makes every configuration file a program. Measured against a file containing one extra line:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ printf 'cycles = 7
rm -rf /tmp/should_not_happen
bad-key = 1
' > /tmp/evil.conf
$ conf_load /tmp/evil.conf
conf: ignoring key bad-key in /tmp/evil.conf
    cycles=7  and /tmp/should_not_happen still exists: yes

The rm -rf was ignored — it contains no =, so the parser skipped it — and the malformed key was rejected with a message naming the file.

Three reasons this matters beyond the obvious one.

A sourced config cannot be reviewed by reading the flow. The flow's behaviour is then the union of the flow and every config file anybody wrote, and Chapter 35.7 is about reviewing flows — which is impossible if a one-line config can redefine die.

A sourced config inherits the flow's shell state, so a config that happens to use t as a loop variable breaks the caller exactly as Chapter 34.7's library did. Parsing gives config files no variables of their own to leak.

And a parsed config can be validated. The key check is three characters of case:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

case $k in *[!a-z_0-9]*) printf 'conf: ignoring key %s in %s\n' "$k" "$f" >&2; continue;; esac

A sourced file cannot be validated at all, because by the time you could inspect it, it has run.

The cost is real and worth stating: a parsed format has no variables, no conditionals and no arithmetic. A config that needs cycles = $BASE * 2 cannot have it. That limitation is the feature — a configuration that can compute is a program, and it will eventually contain the flow's logic. When a value genuinely must be derived, derive it in the flow from configured inputs.

4. One Flow, Two Blocks

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# flow PROJECT TEST OUTDIR -- one flow, any configured block.
# Nothing about a design appears below this line: the block's name selects a
# config file, and every path comes from it.
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
FLOW_NAME=flow; . "$root/lib/flow.sh"; . "$root/lib/conf.sh"

proj=${1:-}; test=${2:-smoke}; out=${3:-}
[ -n "$proj" ] && [ -n "$out" ] || die "usage: flow PROJECT TEST OUTDIR" 2

conf_snapshot_env                              # the environment outranks files
conf_load "$root/cfg/defaults.conf"
[ -r "$root/cfg/$proj.conf" ] || die "no configuration for project '$proj'" 3
conf_load "$root/cfg/$proj.conf"

out=$(abspath "$out") || die "cannot use '$3' as an output directory" 3
droot=$root/$(conf_get design_root '')
[ -d "$droot" ] || die "design_root '$droot' is not a directory" 3
# the block names itself from inside, so a wrong-but-valid root is caught (33.4)
got=$(tr -d ' \n' < "$droot/PROJECT" 2>/dev/null || echo '')
[ "$got" = "$proj" ] || die "design_root holds project '$got', not '$proj'" 5

step 1 4 "configuration for $proj"
conf_dump > "$out/conf.used"                   # what this run was actually given
need "$(conf_get com_tool iverilog)" "$(conf_get sim_tool vvp)"
srcs=""; for f in $(conf_get sources ''); do srcs="$srcs $droot/$f"; done
want $srcs

step 2 4 "disk headroom"
"$root/bin/spacecheck" "$out" "$(conf_get need_mb 10)" >/dev/null 2>&1 \
  || die "not enough space under '$out'" 3

step 3 4 "compiling"
COMPILE_LOG=$out/compile.log "$root/bin/compile" "$out/sim.vvp" $srcs >/dev/null 2>&1 \
  || die "the compile produced no usable image (see $out/compile.log)" 4

step 4 4 "running $test"
( ulimit -f "$(conf_get max_blocks 5242880)"
  "$root/bin/bounded" "$(conf_get limit_s 30)" "$out/run" \
      "$(conf_get sim_tool vvp)" "$out/sim.vvp" \
      "+TESTNAME=$test" "+SEED=$(conf_get seed 1)" "+CYCLES=$(conf_get cycles 200)" )
"$root/bin/logcheck" "$out/run/run.log" || die "the run produced no usable result" 5
printf '%s: OK\n' "$FLOW_NAME" >&2

The whole claim of the chapter, measured:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the SAME flow, two blocks, no edits ===
  PROJECT            rc   last line                                cycles used
  chip_a/smoke       0    flow: OK                                 400
  chip_b/smoke       0    flow: OK                                 200
  chip_a/prio0       5    flow: the run produced no usable resul    400
  chip_b/prio0       0    flow: OK                                 200

Two blocks, two designs, two cycle counts, and opposite verdicts on the same test — prio0 fails on chip_a's arbiter and passes on chip_b's. The flow was not edited between the four rows, and the only thing that changed is the first argument.

chip_b gets 200 cycles because its config does not mention cycles, so the default applies; chip_a gets 400 because it says so. That is the precedence doing its job rather than a special case.

What the flow still asserts

Configuration makes a flow flexible, which is also what makes it dangerous — a flow that will point anywhere will point somewhere wrong. So three checks, all from earlier chapters:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ -r "$root/cfg/$proj.conf" ] || die "no configuration for project '$proj'" 3
…
[ -d "$droot" ] || die "design_root '$droot' is not a directory" 3
got=$(tr -d ' \n' < "$droot/PROJECT" 2>/dev/null || echo '')
[ "$got" = "$proj" ] || die "design_root holds project '$got', not '$proj'" 5

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  CASE                         rc   MESSAGE
  an unconfigured project      3    flow: no configuration for project 'chip_z'
  no arguments                 2    flow: usage: flow PROJECT TEST OUTDIR
  a mislabelled root           5    flow: design_root holds project 'chip_b', no…

The third is Chapter 33.4's identity check, and it is the one configuration specifically creates the need for. A design_root pointing at a real but wrong area produces a complete run with a real log and a real verdict about another design — the silent failure that chapter measured — and the only defence is a marker read from inside the area rather than inferred from the path.

Recording what the run was given, which configuration makes mandatory

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

conf_dump > "$out/conf.used"

One line, and it is not optional once a flow is configurable.

Before configuration, "what did this run do?" was answered by reading the flow, because the flow contained everything. After configuration the answer is the flow plus four layers of resolved values, three of which are outside the repository — and one of which, the environment, is gone the moment the shell exits.

This is Chapter 32.2's env.txt and Chapter 33.1's manifest applied to configuration: record the resolved values, not the files they came from. A conf.used listing CONF_CYCLES=9999 is actionable; a note saying "defaults.conf and chip_a.conf were loaded" is not, because both files have since been edited.

And conf_dump excludes its own bookkeeping:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

conf_dump() { set | /usr/bin/grep -E '^CONF_[A-Z_0-9]+=' | /usr/bin/grep -v '^CONF_FROM_ENV=' | sort; }

CONF_FROM_ENV is a list of variable names, not a setting, and leaving it in the record would make every run's conf.used differ from every other run's by a field nobody set — which breaks exactly the night-to-night comparison Chapter 32.5 depends on.

5. What To Configure, and What Never To

configure	because
the design root	the one thing that makes a flow block-specific
the source list	it changes with the design, not the flow
tool names	`vvp` against a vendor simulator
run knobs — cycles, seeds, limits	they differ per block and per purpose
resource bounds	a big block needs a bigger cap

never configure	because
the library's path	Chapter 35.1: unverifiable behaviour
the flow's own tool paths	they are the flow's structure, not its inputs
the verdict rules	a result that depends on configurable rules is not comparable
whether checks run	a flow with `SKIP_CHECKS=1` has two behaviours and one name

The last row is the one that gets added under pressure. Somebody needs a run now, the environment check is in the way, and SKIP_CHECKS=1 is a two-line change. What it creates is a flow with two behaviours, where every result has to be read alongside the question "was this the checked version?" — and the answer is not in the result unless conf.used happens to record it.

If a check genuinely needs to be optional, the honest form is a different command rather than a flag:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

bin/flow        chip_a smoke out/   # checks everything
bin/flow-unsafe chip_a smoke out/   # for debugging; says so in its name

which is at least visible in a shell history and a log.

Practice

Grep your own flow for literal paths, defaulted knobs and tool names, and count them.
Classify each one as the flow's structure or the design.
Explain why the flow's own tool paths stay.
Write a loader with four precedence levels and document them in the file.
Test each level and find out whether your implementation matches your comment.
If the environment loses, work out why, and fix it with a snapshot.
Use an unanchored match in the snapshot test and find the key it wrongly protects.
Source a config file containing rm -rf /tmp/x in a scratch area and watch what happens.
Parse it instead and confirm the line is ignored.
Add key validation and feed it a malformed key.
Give three reasons parsing beats sourcing beyond the obvious one.
State what a parsed format cannot do, and why that is a feature.
Configure two blocks whose designs differ.
Run the same test on both and compare the verdicts.
Confirm the flow was not edited between the runs.
Leave a key out of one block's config and check the default applies.
Add the three refusals and trigger each.
Point a design_root at a real but wrong area and see what the flow reports.
Record the resolved values in the run directory.
Explain why recording the file names is not enough.
Leave the loader's bookkeeping in the dump and diff two runs' records.
Classify your own settings into configure and never-configure.
Add a SKIP_CHECKS flag, then write the sentence a reader needs beside every result.
Replace it with a differently-named command.

Check Your Understanding

What stops one flow from serving two blocks?
How many hardcoded things were counted, and in how many kinds?
Which six of the nine paths stay, and why?
Which three are the actual hardcoding?
State the rule that keeps a config file small.
Give the four precedence levels and the principle behind the order.
What did the first loader do wrong?
Why can the fix not be applied inside the loop alone?
What are the spaces in CONF_FROM_ENV for?
State the general lesson about documented behaviour.
What happened to the rm -rf line in the parsed config?
Give three reasons parsing beats sourcing.
What can a parsed format not express, and why is that good?
What should you do when a value must be derived?
What did the same flow produce for prio0 on the two blocks?
Why does chip_b get 200 cycles?
Which three checks does the configured flow add, and which chapter motivates the third?
Why does configuration specifically create the need for an identity check?
Why is recording the resolved configuration mandatory once a flow is configurable?
Why record resolved values rather than the file names?
Why does conf_dump exclude CONF_FROM_ENV?
Name four things to configure and four never to.
What does a SKIP_CHECKS flag create?
What is the honest alternative?

Answers

The design it was written against, still named inside the scripts.
Nineteen, in four kinds: the flow's own tools, the design's sources, tool names, and knobs with defaults.
The six under $root/bin — they are the flow pointing at itself, and a configurable library path makes behaviour depend on a variable set before the script ran.
cfg/files.f, rtl/arb.v and tb/tb_arb.v — the design.
Configure what varies between users of the flow, not everything that is a string.
Defaults, then the block's config, then the environment, then flags — ordered by narrowing scope: everybody, a block, one run, one invocation.
It assigned unconditionally, and files load after the environment exists, so a file's 400 overwrote an exported 9999. The documented precedence was a comment.
Because once a file has assigned the variable there is no way to tell whether the environment had set it — the snapshot must be taken before any file is read.
To anchor the match. Without them *"$name"* matches a prefix, so CONF_SEED in the environment would also protect CONF_SEED_COUNT.
A documented behaviour that is not tested is a documented intention.
It was ignored — it contains no = — and the directory it named survived.
A sourced config cannot be reviewed by reading the flow; it inherits the flow's shell state and can clobber a caller's variables; and it cannot be validated, because by the time you could inspect it, it has run.
Variables, conditionals and arithmetic. Good because a configuration that can compute is a program, and it will eventually contain the flow's logic.
Derive it in the flow from configured inputs.
chip_a/prio0 failed with status 5 and chip_b/prio0 passed — opposite verdicts, same test, unedited flow.
Its config does not mention cycles, so the default applies. That is the precedence working, not a special case.
A configuration exists for the project; design_root is a directory; and the area names itself as that project. The third is Chapter 33.4's identity check.
Because a configurable root will eventually point at a real but wrong area, which produces a complete run with a real verdict about another design.
Because the answer to "what did this run do" is now the flow plus four layers, three of them outside the repository and one of them gone when the shell exits.
Because the files have since been edited; only the resolved values are actionable.
Because it is a list of variable names rather than a setting, and leaving it in would make every run's record differ by a field nobody set, breaking night-to-night comparison.
Configure: the design root, the source list, tool names, run knobs, resource bounds. Never: the library's path, the flow's own tool paths, the verdict rules, and whether checks run.
A flow with two behaviours and one name, so every result must be read alongside "was this the checked version?" — and the answer is not in the result.
A differently-named command, which is at least visible in a shell history and a log.

6. Remember

Inventory before refactoring. Nineteen hardcoded things in four kinds, and only three of the nine paths were the design.
The flow's own tool paths stay hardcoded; the design's sources move out.
Configure what varies between users of the flow, not everything that is a string.
Four levels, ordered by narrowing scope: defaults, the block's file, the environment, a flag.
My loader ignored its own documented precedence — files overwrote the environment. The fix is a snapshot taken before any file is read.
Anchor the snapshot test with spaces, or CONF_SEED protects CONF_SEED_COUNT.
A documented behaviour that is not tested is a documented intention.
Parse configuration; never source it. A parsed file ignored an embedded rm -rf; a sourced one is a program, cannot be reviewed from the flow, and leaks variables into it.
A parsed format cannot compute, and that is the feature. Derive values in the flow.
One flow ran two blocks with opposite verdicts on the same test, unedited.
Configuration needs an identity check, read from inside the area — a valid-but-wrong root is the silent failure.
Record the resolved values in the run directory, not the file names — and keep the loader's bookkeeping out of them.
Never make checks configurable. SKIP_CHECKS=1 gives a flow two behaviours and one name; a differently-named command at least shows up in a log.

7. What Is Next

The flow is now one library, one configuration and one entry point — and it is still a sequence of steps that re-runs everything every time. Chapter 35.3 asks whether that is the right shape at all, and measures what a dependency graph gives you that a script cannot.

Where this fits

Part of the Linux & Bash curriculum.