Cutover, Rollback and Decommissioning

Part II · Module 28 · Chapter 28.10 · Cutover, Rollback and Decommissioning

1. Cutover Is a Change of State

By the time a flow switches implementations, the risky work should already be done. Chapter 28.8 showed the port is byte-identical over a corpus that covers the inventory, and Chapter 28.9's review found nothing outstanding. Cutover is not a leap of faith on top of that. It is a small, ordinary change with three properties:

• It is one change, in one place, so it can be read, reverted and blamed precisely.
• Undoing it has been done before it is needed, not just written down (Chapter 28.9's manual question M4).
• The old implementation stays available for a period agreed in advance, and leaves at the end of it by a decision, not because someone tidied up.

The rest of this chapter carries out those three in a throwaway repository and measures each step. Everything runs from one script, cutover.sh, and each section below explains one numbered block of its output.

2. One Variable First, With No Behaviour Change

If the flow names the Perl script directly in a recipe, the switch means editing the recipe, and every later rollback means editing it back. So the first commit, before any cutover, moves the implementation behind one variable and changes nothing else:

# Makefile (SAMPLE) - the signoff step, with the implementation behind ONE variable.
# The cutover is a change to the next line, and nothing else.
STA_SUMMARY ?= perl ../tools/sta_summary.pl
RUN ?= run

signoff:
	cd $(RUN) && $(STA_SUMMARY) reports > sta_summary.txt
	@echo "timing clean: signoff may proceed"

?= sets the variable only if it is not already set, and anything given on the make command line overrides it for that run. Those two facts are the whole rollback design of this chapter, and section 4 measures both.

This commit is deliberately boring: the recipe runs exactly the command it ran before. Land it on its own, ahead of the cutover, so that when the switch comes, its diff is one line and says only one thing.

The repository also holds the consumer Chapter 28.2 found that reads the -v output, adapted to this layout. Notice its comment:

#!/usr/bin/perl
# weekly_digest.pl (SAMPLE) - the consumer from Chapter 28.2. It calls the
# Perl script DIRECTLY, not through the Makefile.
use strict;
use warnings;
my @lines = `perl tools/sta_summary.pl -v run/reports`;
my @endpoints = grep { /^    \S/ } @lines;
print "failing endpoints this week: ", scalar(@endpoints), "\n";

3. The Switch

Here is the whole run. The forwarding stub it uses in step 7 is shown in section 8, and the script and the port in section 11.

Report : timing (synthetic teaching format)
Design : chip_top
Corner : ss_0p72v_125c

Startpoint: u_core/r_reg[3]
Endpoint  : u_core/acc_reg[7]
  slack (VIOLATED)   -0.043

Startpoint: u_core/r_reg[1]
Endpoint  : u_core/acc_reg[2]
  slack (VIOLATED)   -0.011

Startpoint: u_core/cnt_reg[0]
Endpoint  : u_core/cnt_reg[5]
  slack (MET)         0.120

End of report

Report : timing (synthetic teaching format)
Design : chip_top
Corner : tt_0p80v_25c

Startpoint: u_core/r_reg[3]
Endpoint  : u_core/acc_reg[7]
  slack (MET)         0.087

Startpoint: u_core/cnt_reg[0]
Endpoint  : u_core/cnt_reg[5]
  slack (MET)         0.240

End of report

#!/bin/bash
# Cutover, rollback and decommissioning of sta_summary.pl, in a throwaway
# git repository holding the sample flow. Every step prints what it changed.
repo=$(mktemp -d)/flow
mkdir -p "$repo/run/reports"
cp -R co/. "$repo"/
cp reports/*.rpt "$repo/run/reports/"
cp sta_summary.pl sta_summary.py "$repo/tools/"                 # the script and its validated port
mv "$repo/tools/sta_summary_stub.pl" "$repo/../stub.pl"      # not in the repo yet
cd "$repo" || exit 1
git init -q 2>/dev/null
git config user.name sample; git config user.email sample@example.invalid
git config commit.gpgsign false
printf 'run/\n*.log\n' > .gitignore
git add -A && git commit -qm "flow: signoff behind STA_SUMMARY (no behaviour change)"

signoff() {     # one signoff run: make's status and a checksum of each output
    rm -f run/sta_summary.txt run/tmp.csv
    make -s signoff "$@" > /dev/null 2>&1
    local st=$?
    echo "  make exit $st, summary $(cksum < run/sta_summary.txt | cut -d' ' -f1), tmp.csv $(cksum < run/tmp.csv | cut -d' ' -f1)"
}
digest() { echo "  digest: $(perl bin/weekly_digest.pl 2>/dev/null)"; }

echo "=== 1. before the switch"
signoff
echo "=== 2. the switch is one line"
sed -i.bak 's|^STA_SUMMARY ?= perl ../tools/sta_summary.pl$|STA_SUMMARY ?= python3 ../tools/sta_summary.py|' Makefile
rm Makefile.bak
git commit -qam "signoff: sta_summary via the Python port (see MIGRATION summary)"
git show --format= --numstat HEAD | sed 's/^/  changed: /'
signoff
echo "=== 3. rollback for one run, no commit"
signoff STA_SUMMARY="perl ../tools/sta_summary.pl"
echo "=== 4. rollback for everyone, one commit"
git revert --no-edit HEAD > /dev/null
grep '^STA_SUMMARY' Makefile | sed 's/^/  /'
signoff
git revert --no-edit HEAD > /dev/null                         # and forward again
grep '^STA_SUMMARY' Makefile | sed 's/^/  /'
echo "=== 5. who still runs the Perl?"
grep -rn 'sta_summary\.pl' --exclude-dir=.git --exclude-dir=tools . | sort | sed 's/^/  /'
digest
echo "=== 6. deleting it now, without looking"
cp -R . ../too_early && ( cd ../too_early && git rm -q tools/sta_summary.pl && digest )
echo "=== 7. a forwarding stub for one period"
cp ../stub.pl tools/sta_summary.pl
git commit -qam "sta_summary.pl: retired, forwards to the Python port and logs callers"
signoff
digest
sed 's/^/  log: /' tools/RETIRED_CALLS.log
echo "=== 8. repoint the last caller, then delete on purpose"
sed -i.bak 's|perl tools/sta_summary.pl|python3 tools/sta_summary.py|' bin/weekly_digest.pl && rm bin/weekly_digest.pl.bak
git commit -qam "weekly_digest: call the Python port directly"
rm tools/RETIRED_CALLS.log
signoff
digest
echo "  calls logged since the repoint: $( [ -e tools/RETIRED_CALLS.log ] && wc -l < tools/RETIRED_CALLS.log | tr -d ' ' || echo 0)"
git rm -q tools/sta_summary.pl
git commit -qm "sta_summary.pl: deleted after one clean period with no logged callers"
git log --format='  %s' | cat

=== 1. before the switch
  make exit 2, summary 34909085, tmp.csv 839603169
=== 2. the switch is one line
  changed: 1	1	Makefile
  make exit 2, summary 34909085, tmp.csv 839603169
=== 3. rollback for one run, no commit
  make exit 2, summary 34909085, tmp.csv 839603169
=== 4. rollback for everyone, one commit
  STA_SUMMARY ?= perl ../tools/sta_summary.pl
  make exit 2, summary 34909085, tmp.csv 839603169
  STA_SUMMARY ?= python3 ../tools/sta_summary.py
=== 5. who still runs the Perl?
  ./bin/weekly_digest.pl:6:my @lines = `perl tools/sta_summary.pl -v run/reports`;
  digest: failing endpoints this week: 2
=== 6. deleting it now, without looking
  digest: failing endpoints this week: 0
=== 7. a forwarding stub for one period
  make exit 2, summary 34909085, tmp.csv 839603169
  digest: failing endpoints this week: 2
  log: called by: perl bin/weekly_digest.pl  args: -v run/reports
=== 8. repoint the last caller, then delete on purpose
  make exit 2, summary 34909085, tmp.csv 839603169
  digest: failing endpoints this week: 2
  calls logged since the repoint: 0
  sta_summary.pl: deleted after one clean period with no logged callers
  weekly_digest: call the Python port directly
  sta_summary.pl: retired, forwards to the Python port and logs callers
  Reapply "signoff: sta_summary via the Python port (see MIGRATION summary)"
  Revert "signoff: sta_summary via the Python port (see MIGRATION summary)"
  signoff: sta_summary via the Python port (see MIGRATION summary)
  flow: signoff behind STA_SUMMARY (no behaviour change)

Read block 1 and block 2 together. Before the switch, make signoff exits 2. That is not a failure of anything in this chapter: the sample reports contain a violating corner, so the script exits non-zero and Make stops the signoff, which is the gate doing its job (Chapter 26.1). After the switch the exit status is the same, and so are the checksums of both outputs, the summary that Make saves and the tmp.csv that the dashboard reads. The switch commit itself is 1 1 Makefile: one line added, one removed, one file.

Comparing checksums rather than printing the outputs is deliberate. A cutover check should answer one question, "did anything a consumer reads change?", and a matching checksum answers it for every byte at once.

4. Two Rollbacks, Both Run Before They Are Needed

A migration needs two ways back, because there are two situations.

For one run, with no commit (block 3). Something looks wrong tonight, and the engineer on call wants the old behaviour now, for this run, without touching the repository. Because the Makefile uses ?=, a value on the command line wins:

make signoff STA_SUMMARY="perl ../tools/sta_summary.pl"

The output is identical again. That is expected, since the two implementations were validated as identical. The point of running it is to prove the override works: the variable name is right, the path resolves from inside the run directory, and Make honours it.

For everyone, with one commit (block 4). If the port has to come out of the flow for days, revert the switch commit. Block 4 shows the Makefile line back on the Perl script after git revert, the outputs still identical, and then the revert reverted, to go forward again. Because the switch was a single line in its own commit, the revert touches exactly that line and nothing else.

Both rollbacks ran here before anyone needed them. That is the habit to keep. A rollback command that was only written down can fail on the day it matters, through a misspelled variable, a wrong relative path, or a revert that conflicts with a later commit, and you find out in the middle of an incident.

5. The Defined Period

The canonical instruction for this step is: keep the Perl implementation available for a defined period. Both words matter.

Available means the old script is still in the repository and still works, so the per-run override in section 4 still has something to run. It does not mean it is still used.

Defined means the period is written down before the switch, as a condition someone can check, not a feeling that "it has been a while". The registry's own example is a good default: two regression cycles. A cycle is whatever the flow's real rhythm is, such as a nightly run, a weekly signoff or a milestone. During the period, watch for three things: any use of the rollback override, any consumer complaint, and any difference in the nightly outputs that the port's validation would not explain.

At the end of the period, someone named (Chapter 28.9's owner) decides to retire the old script. The next three sections are why that decision needs more than a calendar.

6. The Switch Did Not Move Every Caller

Block 5 searches the repository for anything that still names sta_summary.pl, outside tools/ where the script itself lives. It finds one line:

./bin/weekly_digest.pl:6:my @lines = `perl tools/sta_summary.pl -v run/reports`;

The weekly digest calls the Perl script directly, not through the Makefile. So the switch moved the signoff step and left the digest on the old implementation. Nothing is broken today, and the digest still reports 2 failing endpoints, because the Perl script is still there.

This is the most common way a cutover is incomplete. A switch changes the callers it was designed for. The callers that bypass it (a cron job, a helper script, a colleague's alias) keep running the old code, and nobody notices because nothing fails. Chapter 28.2's consumer search found the digest, and this search is the same search, repeated at the moment it matters.

7. Deleting It Now, Without Looking

Block 6 does in a copy of the repository what a tidy-minded engineer might do at the end of the period: git rm tools/sta_summary.pl. Then it runs the digest.

failing endpoints this week: 0.

The digest exits 0 and prints a perfectly plausible number. Perl's message, Can't open perl script "tools/sta_summary.pl", goes to STDERR, which the digest's backticks do not capture and which a weekly cron mail usually discards. A command that fails inside backticks gives an empty result, and the digest counts zero endpoints in it (Chapter 15.10 covers what backticks return). The mail goes out saying the block has no failing endpoints. The real number is 2, and it stays wrong every week until someone looks at the timing reports directly.

That is the failure "remove it deliberately rather than by neglect" is guarding against. The deletion itself was one command. Everything wrong with it was in what was not checked first.

8. A Forwarding Stub That Finds Callers

The safe way to retire a script that might still have unknown callers is to replace it, for one more period, with a forwarding stub: a tiny program with the same name that does two things.

#!/usr/bin/perl
# sta_summary.pl - RETIRED. The implementation is now tools/sta_summary.py.
# This forwarding stub stays for one defined period so that any caller nobody
# listed keeps working AND is found: each call is appended to a log file
# (never to STDERR, which is one of the script's observable channels).
use strict;
use warnings;
use File::Basename qw(dirname);
my $dir = dirname(__FILE__);
if (open my $log, '>>', "$dir/RETIRED_CALLS.log") {
    my $ppid = getppid();
    my $caller = `ps -o command= -p $ppid`;
    chomp $caller;
    print {$log} "called by: $caller  args: @ARGV\n";
    close $log;
}
exec 'python3', "$dir/sta_summary.py", @ARGV or die "cannot exec python3: $!\n";

It records who called it: its parent process's command line (ps -o command= -p $ppid) and its arguments, appended to RETIRED_CALLS.log. It writes to a log file and never to STDERR, because STDERR is one of the script's four observable channels (Chapter 28.2), and a stub that printed a warning there would itself be a behaviour change.

Then it forwards to the Python port with exec, passing the arguments unchanged. exec replaces the stub's process with the port's, so the exit status, STDOUT and STDERR the caller sees are the port's own (Chapter 26.7's exit-status rules apply unchanged). And because the port was validated as identical, every caller keeps getting exactly the output it always got.

Block 7 installs the stub as tools/sta_summary.pl and runs the signoff and the digest. The outputs are unchanged (identical checksums, 2 failing endpoints), and the log has one line:

called by: perl bin/weekly_digest.pl  args: -v run/reports

The stub has found the caller by name, with the exact arguments it uses. In a real flow, leave the stub in place for the defined period and read the log at the end of it. Every line is a caller to repoint, and an empty log after a full cycle is the evidence the deletion needs.

9. Repoint, Wait, Delete on Purpose

Block 8 finishes the job in three commits, each of which says what it did:

1. Repoint the caller the stub found: weekly_digest.pl now calls python3 tools/sta_summary.py. This is its own commit, because it is a change to a different consumer and should be revertible on its own.
2. Wait a period with the log empty. Here the period is one full cycle, a signoff and a digest, after clearing the log, and the count of logged calls is 0. The outputs are unchanged.
3. Delete the stub, with a commit message that records why: deleted after one clean period with no logged callers.

The commit history at the end of the output is the record of the whole migration, oldest at the bottom:

• the variable introduced with no behaviour change;
• the switch;
• the rehearsed rollback and its reapplication (Reapply "..." is how this version of git names a revert of a revert; older versions write Revert "Revert ...");
• the stub;
• the repointed caller;
• the deletion, with its reason.

Anyone reading git log next year can see what happened, in what order, and why the Perl is gone. "Deliberately rather than by neglect" ends up meaning exactly this: the removal has a commit, a reason and evidence.

10. After the Deletion

Retiring the script is the end of the migration, not the end of the work. Three things remain, and each was deferred on purpose by an earlier chapter.

Remove the migration scaffolding. The PARSE_IMPL switch and shadow mode of Chapter 28.5, a wrapper like Chapter 28.4's if it is no longer needed, and the stub's log file. Scaffolding left in place becomes a second code path nobody tests.

Keep the validation corpus. Chapter 28.8's harness and corpus are now the test suite of the Python implementation. Point the harness at the previous and the next version of the port, and every future change gets the same byte-for-byte check.

Fix the defects, one decision at a time. The port still reproduces B4, B9, B10, B11 and B12, the inventory's accidental behaviours. Now each can be fixed in its own commit with its own decision: line (Chapter 28.9), and the harness shows exactly which cases each fix changes. Chapter 28.3's rewrite tried to make all of these improvements at once, on day one. This is the same set of improvements, made one at a time, each with a person who agreed to it.

11. The Files, Repeated

The case-study script and its validated port, exactly as in Chapters 28.1 and 28.8, so that cutover.sh runs on this page:

#!/usr/bin/perl
# sta_summary.pl - summarise per-corner timing reports
# usage: sta_summary.pl [-v] [report_dir]
#   prints one line per corner: paths, violations, WNS and TNS
#
# (A SAMPLE legacy script written for this module, in the style of
#  scripts that have run unchanged in a flow for a decade.)

use Getopt::Long;

$verbose = 0;
GetOptions("v" => \$verbose);

$dir = shift @ARGV;
$dir = "." unless $dir;

opendir(D, $dir) || die "cannot open $dir: $!\n";
@files = grep(/\.rpt$/, readdir(D));
closedir(D);

if (!@files) {
    print "No reports found in $dir\n";
    exit 0;
}

foreach $f (@files) {
    ($corner = $f) =~ s/\.rpt$//;
    open(F, "$dir/$f") || next;
    while (<F>) {
        if (/^Endpoint\s*:\s*(\S+)/) {
            $ep = $1;
        }
        if (/slack \((\w+)\)\s+(\S+)/) {
            $paths{$corner}++;
            if ($1 eq "VIOLATED") {
                $viol{$corner}++;
                $tns{$corner} += $2;
                push(@{$eps{$corner}}, $ep);
            }
            $wns{$corner} = $2 if $2 < $wns{$corner};
        }
    }
    close(F);
}

open(CSV, ">tmp.csv");
$bad = 0;
foreach $c (sort keys %paths) {
    printf("%-16s paths=%-3d viol=%-3d wns=%.3f tns=%.3f\n",
           $c, $paths{$c}, $viol{$c}, $wns{$c}, $tns{$c});
    print CSV "$c,$viol{$c},$wns{$c}\n";
    if ($verbose) {
        foreach $e (@{$eps{$c}}) { print "    $e\n"; }
    }
    $bad++ if $viol{$c};
}
close(CSV);

exit $bad;

#!/usr/bin/env python3
"""sta_summary.py - a FAITHFUL port of sta_summary.pl (Chapter 28.1).

It reproduces the twelve behaviours of the Chapter 28.2 inventory, defects
included, so that it can be validated byte for byte against the original.
Fixes come AFTER validation, one at a time, as recorded decisions.
"""
import os
import re
import sys

ENDPOINT = re.compile(r"^Endpoint\s*:\s*(\S+)")
SLACK = re.compile(r"slack \((\w+)\)\s+(\S+)")
NUMBER = re.compile(r"\s*([+-]?(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?)")


def perl_true(v):
    """Perl's truth rule: undef, '', '0' and 0 are false."""
    return v is not None and v != "" and v != "0" and v != 0


def perl_num(v):
    """Numify like Perl: undef is 0, text uses its leading number, else 0."""
    if v is None:
        return 0.0
    m = NUMBER.match(v)
    return float(m.group(1)) if m else 0.0


def get_options(argv):
    """The parts of Getopt::Long's behaviour this script relies on (B11):
    options anywhere, names case-insensitive, '--' ends them, a lone '-'
    is an argument, and an unknown option is reported and then ignored."""
    verbose, rest, i = False, [], 0
    while i < len(argv):
        arg = argv[i]
        i += 1
        if arg == "--":
            rest.extend(argv[i:])
            break
        if arg.startswith("-") and arg != "-":
            name, eq, _ = arg.lstrip("-").partition("=")
            if name.lower() == "v":
                if eq:
                    sys.stderr.write("Option v does not take an argument\n")
                else:
                    verbose = True
            else:
                sys.stderr.write(f"Unknown option: {name.lower()}\n")
            continue
        rest.append(arg)
    return verbose, rest


def main(argv):
    verbose, args = get_options(argv)
    d = args[0] if args else None
    if not perl_true(d):                     # B9: "0" becomes "."
        d = "."
    try:
        names = os.listdir(d)
    except OSError as e:                     # B8: STDERR, exit = errno
        sys.stderr.write(f"cannot open {d}: {os.strerror(e.errno)}\n")
        return e.errno
    files = [n for n in names if n.endswith(".rpt")]
    if not files:                            # B7: STDOUT, exit 0, tmp.csv untouched
        print(f"No reports found in {d}")
        return 0

    paths, viol, tns, wns, eps = {}, {}, {}, {}, {}
    ep = None                                # global in the original: never reset
    for f in files:
        corner = f[:-len(".rpt")]
        try:
            fh = open(os.path.join(d, f))
        except OSError:                      # B10: skipped silently
            continue
        with fh:
            for line in fh:
                m = ENDPOINT.search(line)
                if m:
                    ep = m.group(1)
                m = SLACK.search(line)
                if m:
                    status, text = m.group(1), m.group(2)
                    paths[corner] = paths.get(corner, 0) + 1
                    if status == "VIOLATED":
                        viol[corner] = viol.get(corner, 0) + 1
                        tns[corner] = tns.get(corner, 0.0) + perl_num(text)
                        eps.setdefault(corner, []).append(ep)
                    if perl_num(text) < perl_num(wns.get(corner)):   # B4
                        wns[corner] = text                           # B6: the text

    try:
        csv = open("tmp.csv", "w")           # B5: the CURRENT directory
    except OSError:
        csv = None                           # the original never checked
    bad = 0
    for c in sorted(paths):                  # B1: string sort of corner names
        print("%-16s paths=%-3d viol=%-3d wns=%.3f tns=%.3f"
              % (c, paths[c], viol.get(c, 0), perl_num(wns.get(c)), tns.get(c, 0.0)))
        if csv:
            v = viol.get(c)
            csv.write(f"{c},{'' if v is None else v},{wns.get(c, '')}\n")   # B6: undef is empty
        if verbose:                          # B2
            for e in eps.get(c, []):
                print(f"    {'' if e is None else e}")
        if perl_true(viol.get(c)):
            bad += 1
    if csv:
        csv.close()
    return bad                               # B12: a count; 256 wraps to 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

12. Mistakes at Cutover

Switching by editing the recipe. Put the implementation behind one variable first, in a commit of its own.

Bundling the switch with anything else. Then the revert undoes more than the switch.

Writing the rollback instead of running it. Both rollbacks ran in this chapter before they were needed.

Assuming the switch moved every caller. Measured: the digest bypassed it.

Deleting at the end of the period without searching and logging. Measured: the digest reported 0 and exited 0.

A stub that warns on STDERR. It changes the script's behaviour on its last day. Log to a file.

Leaving the scaffolding. Shadow switches and stubs are code paths too.

13. Exercises

1. Override precedence. Change the Makefile's ?= to a plain = and run block 3 again. Does the command-line override still win? Then set STA_SUMMARY in the environment instead of on the command line, with both ?= and =. Write down which combinations give the on-call engineer a working per-run rollback.

2. A caller the search cannot find. Add a caller that builds the script's path at run time (for example, "$tools/sta_" . "summary.pl"). Show that the search in block 5 misses it and the stub's log does not.

3. Make the stub fail loudly. Some teams prefer a stub that refuses to run: it exits with a distinctive status after logging. Write that version, and decide which callers in this flow would notice it and which would silently print a wrong number. When is a refusing stub the better choice?

4. Plan your own cutover. For a script in your flow, write the six commits of section 9 as commit messages before writing any code: variable, switch, stub, each repoint, deletion. Name the period and the owner. Which commit are you least sure you could revert cleanly?

5. The first fix. Starting from the end state of cutover.sh, fix B12 in the Python port (exit 1 for "some corner violated" instead of a count). Run Chapter 28.8's harness against the previous port, list the cases that change, and write the decision: line and the commit message.

14. What To Remember

• Cutover is a small, ordinary change with preconditions: a passed gate (28.8) and a passed review (28.9).
• Put the implementation behind one variable first, in its own commit, with no behaviour change.
• The switch is one line in one file. Measured: identical outputs and exit status before and after.
• Two rollbacks: a command-line override for one run, and a revert for everyone. Run both before they are needed.
• Keep the old implementation for a defined period, written down before the switch; two regression cycles is a sound default.
• A switch moves only the callers routed through it. Measured: the weekly digest called the Perl directly.
• Deleting without looking is silent. Measured: the digest reported 0 failing endpoints and exited 0.
• A forwarding stub logs callers to a file and execs the port, so nothing changes for callers and every one of them is found.
• Repoint each caller in its own commit, wait with an empty log, then delete with a reason in the commit message.
• Afterwards: remove the scaffolding, keep the corpus as the test suite, and fix the inherited defects one decision at a time.

That completes the migration of one script, from the first hour of reading in Chapter 28.1 to a deletion with a reason in its commit message. Module 29 moves from migrating tools to building them. Each of its projects is a complete flow utility with defined inputs, a result another program can read, an honest exit code and tests that can fail.