Where a Perl-to-Python Translation Goes Wrong

Part II · Module 28 · Chapter 28.6 · Where a Perl-to-Python Translation Goes Wrong

1. Why "No Error" Is the Problem

Chapter 28.5's first Python twin printed exactly the same summary as the Perl code and disagreed with it on four records out of four. No exception, no warning, nothing on STDERR. That is the normal way a Perl-to-Python translation goes wrong, and it is why this chapter exists.

The two languages look close enough that a line-by-line translation feels safe: for becomes for, a hash becomes a dict, =~ becomes re.search. Most lines really do translate directly. The trouble is a handful of Perl behaviours that are implicit. They do not appear in the text of the line at all, so a translator working from the text cannot see them. Perl decides what a function returns from the call site. It turns a loop variable into an alias. It creates hash keys when you read them. It has its own rule for what is false. And it reorders hashes on every run.

Each section below is a short Perl program from a flow-script setting, its line-by-line translation, and the output of both. Each translation is the one a careful engineer would plausibly write, not a deliberately bad one.

2. Context-Dependent Returns

In Perl, the caller decides what a function's return expression means. The same return gives a list in list context and something else in scalar context, and what else depends on the expression (Chapters 2.9 and 10.6):

use strict;
use warnings;

my @results = ([alu_smoke => 'PASS'], [fifo_full => 'FAIL'], [uart_parity => 'FAIL']);
sub failing     { return map { $_->[0] } grep { $_->[1] ne 'PASS' } @results }
sub corner_wns_tns { my ($wns, $tns) = (-0.043, -0.054); return ($wns, $tns) }
sub sorted_fails   { return sort map { $_->[0] } grep { $_->[1] ne 'PASS' } @results }

my $n = failing();             # scalar context: the COUNT of a returned list value
my ($first) = failing();       # list context: the first element
my $wns = corner_wns_tns();    # scalar context on a LIST LITERAL: the LAST element
my $s = sorted_fails();        # sort in scalar context: undefined behaviour
print "failing tests: $n\n";
print "first failing: $first\n";
print "wns: $wns\n";
print "sorted, scalar: ", defined $s ? $s : 'undef', "\n";

results = [("alu_smoke", "PASS"), ("fifo_full", "FAIL"), ("uart_parity", "FAIL")]

def failing():
    return [name for name, v in results if v != "PASS"]

def corner_wns_tns():
    wns, tns = -0.043, -0.054
    return (wns, tns)

n = failing()                   # a line-by-line translation of the four calls
first = failing()
wns = corner_wns_tns()
s = sorted(failing())
print(f"failing tests: {n}")
print(f"first failing: {first}")
print(f"wns: {wns}")
print(f"sorted, scalar: {s}")

#!/bin/bash
echo "--- perl pit/context.pl"
perl pit/context.pl
echo "--- python3 pit/context.py"
python3 pit/context.py

--- perl pit/context.pl
failing tests: 2
first failing: fifo_full
wns: -0.054
sorted, scalar: undef
--- python3 pit/context.py
failing tests: ['fifo_full', 'uart_parity']
first failing: ['fifo_full', 'uart_parity']
wns: (-0.043, -0.054)
sorted, scalar: ['fifo_full', 'uart_parity']

Four calls, four different rules, all measured here on Perl 5.34.1:

• my $n = failing() gives 2. The return expression is a map, and map (like grep, and like an array) in scalar context gives its count (Chapter 5.7).
• my ($first) = failing() gives fifo_full. The parentheses make it list assignment, so the list arrives and its first element is kept.
• my $wns = corner_wns_tns() gives -0.054, which is the TNS. A list literal in scalar context is the comma operator, which evaluates to its last element. The caller asked for one value, named it $wns, and silently got the other one. That is a bug in the Perl, and it has been feeding a TNS into a WNS variable for as long as the script has run.
• my $s = sorted_fails() gives undef. sort in scalar context is documented as undefined behaviour.

The Python translation gives the whole list (or tuple) in all four places. Nothing fails at these lines. The f-strings happily print ['fifo_full', 'uart_parity'] tests failed, and the first comparison like if n > 0 raises a TypeError somewhere else, far from the cause. Or, worse, if n: is simply true and the code carries on.

3. Aliasing: the Loop Variable Is the Element

In a Perl foreach, the loop variable is not a copy. It is another name for the array element, so changing it changes the array (Chapter 9.2). @_ works the same way: its elements are aliases for the caller's arguments (Chapter 10.4).

use strict;
use warnings;

# endpoint names as they come out of a report column: padded
my @eps = ("u_core/acc_reg[7]   ", "u_core/cnt_reg[5]\t");
s/\s+$// for @eps;                       # foreach ALIASES: cleans @eps itself
sub upcase_all { $_ = uc for @_ }        # @_ ALIASES the caller's variables
my $corner = "ss_0p72v_125c";
upcase_all($corner);
my %waived = ("u_core/acc_reg[7]" => 1);
print "waived: ", scalar(grep { $waived{$_} } @eps), " of ", scalar(@eps), "\n";
print "corner: $corner\n";

eps = ["u_core/acc_reg[7]   ", "u_core/cnt_reg[5]\t"]
for e in eps:                     # translated: for (@eps) { s/\s+$// }
    e = e.rstrip()

def upcase_all(*args):            # translated: sub upcase_all { $_ = uc for @_ }
    for a in args:
        a = a.upper()

corner = "ss_0p72v_125c"
upcase_all(corner)
waived = {"u_core/acc_reg[7]"}
print(f"waived: {sum(1 for e in eps if e in waived)} of {len(eps)}")
print(f"corner: {corner}")

#!/bin/bash
echo "--- perl pit/alias.pl"
perl pit/alias.pl
echo "--- python3 pit/alias.py"
python3 pit/alias.py

--- perl pit/alias.pl
waived: 1 of 2
corner: SS_0P72V_125C
--- python3 pit/alias.py
waived: 0 of 2
corner: ss_0p72v_125c

s/\s+$// for @eps is a common Perl idiom for cleaning a list in place, and it works: the waiver lookup finds 1 of 2 endpoints. The Python loop rebinds the name e to a new string and throws it away, so eps still has its trailing spaces and the lookup finds 0 of 2. In a timing flow, that means a waived path is reported as a violation, or (with the logic the other way round) a real violation matches nothing and is dropped.

The @_ case is the same mistake through a function call. upcase_all($corner) changed the caller's variable in Perl. The Python function changed a local name and returned nothing.

Neither translation is wrong Python. Python simply has no way to make a loop variable or a parameter an alias for a string, and the translation needs a different shape: build a new list, or return the new value and assign it.

4. Autovivification: Reading Creates Keys

Reading $slack{$corner}{$ep} when $slack{$corner} does not exist creates $slack{$corner} as an empty hash, so that the second lookup has something to look in (Chapter 7.9). The read succeeds, the value is undefined, and the hash now has one more key than before.

use strict;
use warnings;

# slack per corner per endpoint; only one corner has been loaded so far
my %slack = (ss_0p72v_125c => { 'u_core/acc_reg[7]' => -0.043 });
for my $corner (qw(ss_0p72v_125c tt_0p80v_25c ff_0p88v_m40c)) {
    print "$corner: violated\n" if ($slack{$corner}{'u_core/acc_reg[7]'} // 0) < 0;
}
# the READS above created the two missing corners as empty hashes
printf "corners in the table: %d (%s)\n", scalar(keys %slack), join(' ', sort keys %slack);

from collections import defaultdict

loaded = {"ss_0p72v_125c": {"u_core/acc_reg[7]": -0.043}}
corners = ["ss_0p72v_125c", "tt_0p80v_25c", "ff_0p88v_m40c"]

# translation A: .get() -- the common, careful-looking choice
slack = {k: dict(v) for k, v in loaded.items()}
for c in corners:
    if slack.get(c, {}).get("u_core/acc_reg[7]", 0) < 0:
        print(f"{c}: violated")
print(f"A .get():       corners in the table: {len(slack)}")

# translation B: defaultdict -- reproduces Perl, including the side effect
slack = defaultdict(dict, {k: dict(v) for k, v in loaded.items()})
for c in corners:
    if slack[c].get("u_core/acc_reg[7]", 0) < 0:
        pass
print(f"B defaultdict:  corners in the table: {len(slack)}")

# translation C: plain indexing -- fails loudly
slack = {k: dict(v) for k, v in loaded.items()}
try:
    for c in corners:
        slack[c]["u_core/acc_reg[7]"]
except KeyError as e:
    print(f"C plain index:  KeyError {e}")

#!/bin/bash
echo "--- perl pit/autoviv.pl"
perl pit/autoviv.pl
echo "--- python3 pit/autoviv.py"
python3 pit/autoviv.py

--- perl pit/autoviv.pl
ss_0p72v_125c: violated
corners in the table: 3 (ff_0p88v_m40c ss_0p72v_125c tt_0p80v_25c)
--- python3 pit/autoviv.py
ss_0p72v_125c: violated
A .get():       corners in the table: 1
B defaultdict:  corners in the table: 3
C plain index:  KeyError 'tt_0p80v_25c'

All three corners were checked; only one was ever loaded. Perl now reports 3 corners in the table, because two of them were created by the check. Three equally plausible Python translations give three different behaviours:

• A, .get(c, {}): no side effect. 1 corner. The output line "corners in the table" changes from 3 to 1.
• B, defaultdict(dict): reproduces Perl exactly, including the side effect. 3 corners.
• C, plain indexing: raises KeyError. Loud, which is the best outcome of the three if nobody depends on the side effect.

Which one is "correct" depends on whether anything later counts the keys, iterates over them, or writes them out. If a report prints "3 corners analysed", the Perl script has been printing that for corners it never loaded, and translation A silently fixes it. Like every fix, that is a change in behaviour, and it needs an inventory row and a decision (Chapter 28.2).

5. The Truth of "0"

Perl has exactly four false values that matter here: undef, the empty string, the number 0, and the string "0" (Chapter 2.8). Python's rule for strings is simpler: only the empty string is false. So every string that Perl treats as false because it is "0" becomes true in a translation.

That matters most when the string came from a regular expression, because a capture is always a string:

use strict;
use warnings;

# the summary line a simulator prints at the end of a run
for my $line ("UVM_ERROR :    0", "UVM_ERROR :    3") {
    my ($errors) = $line =~ /UVM_ERROR\s*:\s*(\d+)/;     # captured TEXT: "0" or "3"
    print "$line -> ", ($errors ? "FAIL" : "PASS"), "\n";
}
for my $text ("0.000", "0", "0.0", "00", "") {
    printf "%-7s is %s in Perl\n", "'$text'", $text ? "TRUE" : "false";
}

import re

for line in ["UVM_ERROR :    0", "UVM_ERROR :    3"]:
    errors = re.search(r"UVM_ERROR\s*:\s*(\d+)", line).group(1)   # a str: "0" or "3"
    print(f"{line} -> {'FAIL' if errors else 'PASS'}")
for text in ["0.000", "0", "0.0", "00", ""]:
    print(f"{repr(text):<7} is {'TRUE' if text else 'false'} in Python")

#!/bin/bash
echo "--- perl pit/truth.pl"
perl pit/truth.pl
echo "--- python3 pit/truth.py"
python3 pit/truth.py

--- perl pit/truth.pl
UVM_ERROR :    0 -> PASS
UVM_ERROR :    3 -> FAIL
'0.000' is TRUE in Perl
'0'     is false in Perl
'0.0'   is TRUE in Perl
'00'    is TRUE in Perl
''      is false in Perl
--- python3 pit/truth.py
UVM_ERROR :    0 -> FAIL
UVM_ERROR :    3 -> FAIL
'0.000' is TRUE in Python
'0'     is TRUE in Python
'0.0'   is TRUE in Python
'00'    is TRUE in Python
''      is false in Python

A simulation with zero UVM errors passes in Perl and fails in the translation. ($errors ? "FAIL" : "PASS") is idiomatic Perl, and it works because the capture is the string "0", which is false. The translated 'FAIL' if errors else 'PASS' tests a non-empty string, which is true. Every passing test in the regression would be reported as failing. That one is at least noisy, and someone will notice by morning. The reverse case is quieter: a check written as "skip this corner if $count" that skips nothing in Python.

The table half of the output shows the boundary exactly. "0.000", "0.0" and "00" are true in both languages, so a Perl check like if ($slack) on a captured "0.000" was already true and the translation agrees. The single string where the two languages disagree is "0", which is also the string a count of zero errors produces.

6. Hash Order

A Perl hash has no defined order, and since Perl 5.18 it is deliberately different from run to run, as a defence against attacks that exploit predictable hashing (Chapter 6.6). A Python dict keeps insertion order. So the translation of an unsorted keys %wns prints a stable order where the original printed a different one each time.

use strict;
use warnings;

my %wns = (ss_0p72v_125c => -0.043, tt_0p80v_25c => 0.087, ff_0p88v_m40c => 0.150,
           ss_0p72v_m40c => -0.012, ff_0p99v_125c => 0.201);
print join(' ', keys %wns), "\n";          # no sort: the order is not defined

wns = {"ss_0p72v_125c": -0.043, "tt_0p80v_25c": 0.087, "ff_0p88v_m40c": 0.150,
       "ss_0p72v_m40c": -0.012, "ff_0p99v_125c": 0.201}
print(" ".join(wns.keys()))                 # dicts keep insertion order

#!/bin/bash
# Run each version 20 times and count the distinct orders it printed.
distinct() { for i in $(seq 20); do "$@"; done | sort -u | wc -l | tr -d ' '; }
verdict() { [ "$1" -eq 1 ] && echo "1 (stable)" || echo "more than 1 (changes between runs)"; }
echo "perl  keys %wns           : $(verdict $(distinct perl pit/order.pl))"
echo "python dict.keys()        : $(verdict $(distinct python3 pit/order.py))"
echo "perl, PERL_HASH_SEED=0 and PERL_PERTURB_KEYS=0:"
export PERL_HASH_SEED=0 PERL_PERTURB_KEYS=0
echo "                            $(verdict $(distinct perl pit/order.pl))"

perl  keys %wns           : more than 1 (changes between runs)
python dict.keys()        : 1 (stable)
perl, PERL_HASH_SEED=0 and PERL_PERTURB_KEYS=0:
                            1 (stable)

This pitfall is unlike the other four: the translation is better behaved than the original. The cost lands on the migration, because the Perl output cannot be used as a byte-for-byte reference. Run it twice and it disagrees with itself, so a side-by-side diff (Chapter 28.8) reports a difference that neither version caused.

There are two ways to handle it, and the measurement shows the second. One: decide that order is not part of the contract for that output, and compare it sorted on both sides. Two: when you need the Perl output to be reproducible for a comparison run, set PERL_HASH_SEED=0 and PERL_PERTURB_KEYS=0. In 20 runs that gave one order. It is an order, not insertion order, so it fixes reproducibility and nothing else. Never ship a flow that depends on it; use it only to make a reference run repeatable.

7. Six More, One Line Each

The five above are the ones that most often hide inside working scripts. These six are smaller, and each is a single expression that a translator writes without thinking:

#!/bin/bash
# Six more one-line differences: the Perl expression, then its literal Python.
pair() {
    printf '%-26s perl: %-14s python: %s\n' "$1" "$(perl -e "$2")" "$(python3 -c "$3")"
}
pair 'split, trailing empties'  'my @f = split /,/, "a,b,,"; print scalar @f' \
                                'print(len("a,b,,".split(",")))'
pair 'sort of 10 9 100'         'print join " ", sort 10, 9, 100' \
                                'print(" ".join(str(x) for x in sorted([10, 9, 100])))'
pair 'print 3.0'                'print 3.0' \
                                'print(3.0)'
pair 'print 0.1 + 0.2'          'print 0.1 + 0.2' \
                                'print(0.1 + 0.2)'
pair 'integer part of -7/2'     'print int(-7 / 2)' \
                                'print(-7 // 2)'
pair '"10" == "10.0"'           'print "10" == "10.0" ? "true" : "false"' \
                                'print("10" == "10.0")'

split, trailing empties    perl: 2              python: 4
sort of 10 9 100           perl: 10 100 9       python: 9 10 100
print 3.0                  perl: 3              python: 3.0
print 0.1 + 0.2            perl: 0.3            python: 0.30000000000000004
integer part of -7/2       perl: -3             python: -4
"10" == "10.0"             perl: true           python: False

• split drops trailing empty fields in Perl (2 fields from a,b,,); Python's str.split keeps them (4). A CSV row with empty last columns changes length (Chapter 5.8).
• Perl's default sort compares strings (Chapter 5.9), so 10 100 9; Python sorts integers numerically. A translation that "fixes" a sort changes the row order of every report.
• Perl prints 3.0 as 3, and 0.1 + 0.2 as 0.3, because it prints 15 significant digits; Python prints the shortest exact round-trip form. This is Chapter 28.5's tns disagreement in its simplest form.
• int() truncates toward zero; // floors. For negative numbers they differ by one.
• == compares numbers in Perl, so "10" == "10.0" is true; in Python, == on two strings compares the text.

8. Faithful Translations

A faithful translation does what the Perl does, decided case by case, not what the Perl line looks like:

"""The same four programs, translated for what the Perl DOES."""
import re
from collections import defaultdict


def perl_true(value):
    """Perl's truth rule for a string or number: false is undef, '', '0', 0."""
    return value is not None and value != "" and value != "0" and value != 0


# 1. context: decide, at every call site, which context the Perl caller used
results = [("alu_smoke", "PASS"), ("fifo_full", "FAIL"), ("uart_parity", "FAIL")]
failing = [name for name, v in results if v != "PASS"]
n = len(failing)                       # my $n = failing();      -> a count
first = failing[0]                     # my ($first) = failing(); -> first element
wns = (-0.043, -0.054)[-1]             # my $wns = (LIST);       -> LAST element
print(f"1 failing tests: {n}; first: {first}; 'wns': {wns}  <- the Perl bug, kept")

# 2. aliasing: the Perl loop rewrote the array, so build the new list
eps = ["u_core/acc_reg[7]   ", "u_core/cnt_reg[5]\t"]
eps = [e.rstrip() for e in eps]
corner = "ss_0p72v_125c".upper()       # upcase_all($corner) changed the caller's value
print(f"2 waived: {sum(1 for e in eps if e in {'u_core/acc_reg[7]'})} of {len(eps)}; corner: {corner}")

# 3. autovivification: if anything later counts the keys, reproduce the side effect
slack = defaultdict(dict, {"ss_0p72v_125c": {"u_core/acc_reg[7]": -0.043}})
for c in ["ss_0p72v_125c", "tt_0p80v_25c", "ff_0p88v_m40c"]:
    slack[c].get("u_core/acc_reg[7]")
print(f"3 corners in the table: {len(slack)}")

# 4. truth: test the captured text with Perl's rule, not Python's
for line in ["UVM_ERROR :    0", "UVM_ERROR :    3"]:
    errors = re.search(r"UVM_ERROR\s*:\s*(\d+)", line).group(1)
    print(f"4 {line} -> {'FAIL' if perl_true(errors) else 'PASS'}")

#!/bin/bash
python3 pit/faithful.py

1 failing tests: 2; first: fifo_full; 'wns': -0.054  <- the Perl bug, kept
2 waived: 1 of 2; corner: SS_0P72V_125C
3 corners in the table: 3
4 UVM_ERROR :    0 -> PASS
4 UVM_ERROR :    3 -> FAIL

Every line now matches the Perl output above, including the Perl bug in case 1. The faithful translation hands the TNS to the variable called wns because that is what the script has always done, and it says so in a comment. Fixing it is the right thing to do, but not inside a translation. It is a separate decision with its own inventory row, made after the migration is validated, when the only difference between the two versions is the one being fixed.

perl_true is worth keeping as a small library function during any migration. Use it wherever the Perl code tested a value that might be a captured string, and replace each use with an explicit comparison (int(errors) > 0) once the behaviour is validated and the intent is clear.

9. A Translation Checklist

Before translating any line that contains one of these, stop and decide explicitly:

The same table is the review list for someone else's translation. Most of these lines are a few characters long, which is exactly why they are missed.

10. Mistakes

Trusting a clean run. None of the five pitfalls raises an error. The only evidence is a different value, so the evidence has to come from a comparison (Chapters 28.5 and 28.8).

Remembering the context rule wrong. "Last element" is right for a list literal and wrong for an array. Look at the expression.

Fixing the Perl bug during translation. Then the validation shows a difference and nobody knows whether it is the fix or a mistake.

Treating every capture as a number. It is a string, and "0" is the one string the languages disagree about.

Using the Perl output as a reference without controlling hash order. Unsorted output disagrees with itself between runs.

11. Exercises

1. Find wantarray. Write a Perl function that uses wantarray to return a list in list context and a hash reference in scalar context. Translate it faithfully as two Python functions, and explain why one Python function cannot reproduce both call sites.

2. Aliasing through map. my @m = map { s/_reg$//; $_ } @eps; changes both @m and @eps. Measure it. Then write the faithful translation, and the translation that does what the author probably intended. Which one belongs in a migration?

3. A truth audit. Search a Perl script from your flow for every if (, unless ( and ? : whose condition is a single variable. For each, decide whether the variable can hold the string "0". How many of your conditions would a literal translation invert?

4. Order as a contract. In Chapter 28.1's sta_summary.pl, every output is sorted. Find a script in your flow that prints keys unsorted. Run it five times with and without PERL_HASH_SEED=0 PERL_PERTURB_KEYS=0, and decide whether its consumers could depend on the order.

5. The seventh pitfall. Find one more Perl behaviour that a line-by-line Python translation changes silently (hints: chomp versus strip(), substr past the end of a string, local $/, x on a list). Write the measured pair and add a row to the table in section 9.

12. What To Remember

• The dangerous translation runs cleanly and prints a different answer. None of these five raises an error.
• Context: an array, grep or map in scalar context gives the count; a list literal gives the last element; sort gives undefined behaviour. Measured: a scalar call silently handed the TNS to $wns.
• Aliasing: a Perl loop variable and the elements of @_ are aliases. A translated loop edits a copy; measured, the waiver match went from 1 of 2 to 0 of 2.
• Autovivification: reading a nested hash creates keys. Three plausible translations gave 1, 3, and a KeyError.
• The string "0" is false in Perl and true in Python. Measured: UVM_ERROR : 0 passed in Perl and failed in the translation.
• "0.000", "0.0" and "00" are true in both; "0" is the only string where they disagree.
• Perl hash order changes every run; Python's does not. Compare sorted, or use PERL_HASH_SEED=0 PERL_PERTURB_KEYS=0 only to make a reference run repeatable.
• Six more: split drops trailing empties, sort compares strings, numbers print with 15 digits, int truncates, // floors, == is numeric.
• Translate what the Perl does, bugs included, and fix bugs afterwards as separate, recorded decisions.

Chapter 28.7 does the same for the regular-expression constructs that need real work to port: /e substitutions, tr///, \G scanning, named captures and the implicit $_. Chapter 16.8 has already established that the regex engines mostly agree; the porting problems are in the operators around them.