Bitwise Operators: & | ^ ~ << >>

Part I · Module 3 · Chapter 3.11 · Bitwise Operators: & | ^ ~ << >>

1. The Operators

use strict;
use warnings;

printf "5 & 3   = %d\n", 5 & 3;
printf "5 | 3   = %d\n", 5 | 3;
printf "5 ^ 3   = %d\n", 5 ^ 3;
printf "1 << 4  = %d\n", 1 << 4;
printf "64 >> 3 = %d\n", 64 >> 3;

5 & 3   = 1
5 | 3   = 7
5 ^ 3   = 6
1 << 4  = 16
64 >> 3 = 8

So far, no surprises: 5 & 3 is 0b101 & 0b011, which is 0b001. The rest of the chapter is about the two places Perl does not behave the way a hardware background expects.

2. There Is No Declared Width

use strict;
use warnings;

print "~0 = ", ~0, "\n";
print "~5 = ", ~5, "\n";

~0 = 18446744073709551615
~5 = 18446744073709551610

~0 is not -1. It is not 0xFF, or 0xFFFF, or anything width-limited.

In Verilog or VHDL, a complement happens inside a declared width: ~8'h00 is 8'hFF because the signal is eight bits wide and the language knows it. Perl has no width. The operand is promoted to the native unsigned integer — 64 bits on this machine, which perl -V:ivsize reports as ivsize='8', meaning eight bytes — and complemented there. 18446744073709551615 is 2**64 - 1.

So if you are porting a hardware model or a C utility, every ~ needs a mask:

use strict;
use warnings;

printf "~0 masked to 8 bits  = 0x%02X\n", ~0 & 0xFF;
printf "~5 masked to 8 bits  = 0x%02X\n", ~5 & 0xFF;
printf "~0 masked to 16 bits = 0x%04X\n", ~0 & 0xFFFF;

~0 masked to 8 bits  = 0xFF
~5 masked to 8 bits  = 0xFA
~0 masked to 16 bits = 0xFFFF

Masking after complementing is the discipline. Write the mask even where the current data makes it unnecessary, because the mask is where the width is documented — Perl will not record it for you.

3. Shifts Have No Width Either

use strict;
use warnings;

print "1 << 4  = ", 1 << 4,  "\n";
print "1 << 63 = ", 1 << 63, "\n";
print "1 << 64 = ", 1 << 64, "\n";
print "1 << -1 = ", 1 << -1, "\n";

1 << 4  = 16
1 << 63 = 9223372036854775808
1 << 64 = 0
1 << -1 = 0

Two of those deserve attention, and note the empty STDERR — neither warns:

• 1 << 64 is 0. Shifting by the full register width loses the bit entirely. There is no error and no wrap.
• 1 << -1 is 0. A negative shift count does not shift the other way and does not complain. If a shift amount is computed, a negative result silently gives zero.

A bit position derived from parsed data — a field offset, a lane index — is therefore worth checking before it reaches a shift. Zero is a plausible-looking result for a mask, which puts this in the same category as everything else in this module: the wrong answer does not look wrong.

4. The Same Operators Do Something Else on Strings

This is the part to take seriously.

use strict;
use warnings;

print "'A' | ' '     = [", ("A" | " "),     "]\n";
print "'abc' & 'ABC' = [", ("abc" & "ABC"), "]\n";

'A' | ' '     = [a]
'abc' & 'ABC' = [ABC]

Those are not numeric results. Given two operands that are strings, &, |, ^ and ~ operate character by character on the character codes.

"A" | " " is a because A is 0x41, a space is 0x20, and 0x41 | 0x20 is 0x61, which is a. That is the old trick for lowercasing ASCII, and it is why you may find it in inherited code.

The coincidence that hides the behaviour

Here is why this is dangerous rather than merely surprising. The obvious way to test it does not work:

use strict;
use warnings;

print "'5' & '3' = [", ("5" & "3"), "]\n";

'5' & '3' = [1]

1. Which is exactly what numeric 5 & 3 gives.

And it is a coincidence. As a string operation, 5 is 0x35, 3 is 0x33, and 0x35 & 0x33 is 0x31 — the character 1. As a numeric operation, 5 & 3 is 1. Both answers are 1, by accident, so this test tells you nothing about which one ran.

To tell them apart you need operands where the two disagree:

use strict;
use warnings;

my $as_text_a = "12";
my $as_text_b = "10";

my $both_strings = $as_text_a & $as_text_b;

my $as_num_a = "12";
my $as_num_b = 10;

my $one_numeric = $as_num_a & $as_num_b;

print "'12' & '10'  (both strings) = [$both_strings]\n";
print "'12' & 10    (one numeric)  = [$one_numeric]\n";

'12' & '10'  (both strings) = [10]
'12' & 10    (one numeric)  = [8]

"10" and 8. Two different answers from the same operator and the same digits.

The string result is per character: 1 & 1 on 0x31 & 0x31 gives 0x31, and 2 & 0 on 0x32 & 0x30 gives 0x30 — so "10". The numeric result is 12 & 10, which is 8.

And it can be changed by an unrelated earlier line

use strict;
use warnings;

my $a = "12";
my $b = "10";

my $discarded = $a + 0;

my $result = $a & $b;

print "result = [$result]\n";

result = [8]

Compare that with the previous script, where the same $a & $b gave "10".

The only difference is line 7, which adds zero to $a and throws the result away. That read $a as a number, Perl cached the numeric value on the scalar — the caching you met in Chapter 3.6 §5 — and from then on & saw a value that had a number in it and did numeric arithmetic.

5. Why This Matters in Engineering Work

Register and field manipulation is the main use. Extracting a field from a register value read out of a log, building a mask for a set of lanes, checking a status bit. All of it is &, |, >> and <<, and all of it starts from text that a script parsed.

That is precisely the overlap of the two hazards in this chapter. The value came from a file, so it is a string, so the bitwise operators may take their string path. And there is no declared width, so a complement gives a 64-bit result that no hardware register has.

Both have the same one-line discipline:

• Numify explicitly at the point you parse. my $reg = 0 + $field; once, at the top, settles the question for everything downstream — but pick the right conversion. 0 + "0x3C" is 0 with an isn't numeric warning, because the 0 + idiom reads a decimal number and 0x3C does not start with one. Hexadecimal text needs hex($field), and octal or binary text needs oct($field). Chapter 2.13 covers which conversion reads which notation; the point here is that "numify it" is a decision with more than one answer.
• Mask every complement and every shift result to the width the hardware actually has. The mask documents the width; nothing else in the Perl does.

And the negative or oversized shift count. A bit position computed from a parsed index can come out negative or too large, and both give 0 with no warning. For a mask, 0 means "no bits", which reads as a legitimate answer: no lanes selected, no flags set, nothing to report.

6. Common Mistakes

Expecting ~0 to be -1 or a width-limited value. It is 2**64 - 1 on a 64-bit Perl. Mask it.

Expecting a declared width to exist. It does not. Perl has no [7:0].

Using & on parsed strings and assuming numeric arithmetic. Two strings get per-character bitwise. Numify first.

Testing the string/numeric question with "5" & "3". Both paths give 1, so the test cannot distinguish them. Use "12" & "10".

Deleting an apparently useless numeric expression. If it numified an operand of a later bitwise operator, removing it changes that operator's behaviour.

Assuming an out-of-range shift warns. 1 << 64 and 1 << -1 are both 0, silently.

Reaching for use integer to fix width. It changes ~ and also other arithmetic in its scope. An explicit mask is narrower and clearer.

7. Exercise

Exercise 1 — the complement. Print ~0 and ~5. Then print both masked to eight bits with printf "%02X". State what the unmasked values are in terms of a power of two, and check perl -V:ivsize to confirm the width.

Exercise 2 — the discriminating test. Run the §4 discriminating.pl script and record both answers. Then explain the "10" result by writing out the two character codes and the AND of each pair.

Exercise 3 — action at a distance. Reproduce action-at-a-distance.pl, confirm it prints 8, then delete the my $discarded = $a + 0; line and run it again. Record both results and write one sentence on what that means for deleting code that appears to have no effect.

Exercise 4 — make it deterministic. Rewrite the same script so that the result is 8 whether or not the discarded line is present, using the forcing idiom from Chapter 2.10. Confirm by running it both ways.

Exercise 5 — extract a field, and watch the conversion. Given my $reg = "0x3C"; as text, first try 0 + $reg and record both the value and the warning. Then convert it correctly with hex($reg), shift right by 2, mask off three bits, and report the field. State what the mask is and why, and say in one sentence why the 0 + idiom was the wrong tool here.

Exercise 6 — the silent shift. Compute 1 << $n for $n of 4, 64 and -1, capturing STDERR. Report which produced warnings, and say what you would add to a script that computes a shift amount from parsed data.

8. Summary

• Perl has & | ^ ~ << >> and the basic numeric results are unsurprising.
• There is no declared width. ~0 is 18446744073709551615 — 2**64 - 1 on a 64-bit Perl — not -1 and not 0xFF. Mask every complement.
• use integer makes ~0 give -1, but it also changes other arithmetic in its scope; an explicit mask is the narrower tool.
• Out-of-range shifts give 0 silently. 1 << 64 and 1 << -1 are both 0, with no warning.
• On two strings, these operators work per character on character codes. "A" | " " is "a".
• "5" & "3" is 1 either way, so the obvious test cannot tell the two modes apart. "12" & "10" gives "10" as strings and 8 as numbers.
• A discarded numeric expression earlier in the program can change a later bitwise result, because the numeric reading is cached on the scalar.
• Numify both operands explicitly before any bitwise arithmetic on parsed data, and the ambiguity disappears.

Chapter 3.12 closes the module with the rule that has been quietly behind several of these chapters: which operator Perl applies first, and why writing the parentheses anyway is the right habit.