Inspecting a String: length, index, substr

Part I · Module 4 · Chapter 4.5 · Inspecting a String: length, index, substr

1. Positions Start at Zero

use strict;
use warnings;

my $test = "alu_smoke_01";

print "length        : ", length($test),        "\n";
print "index of '_'  : ", index($test, "_"),    "\n";
print "rindex of '_' : ", rindex($test, "_"),   "\n";
print "substr(0,3)   : ", substr($test, 0, 3),  "\n";
print "substr(-2)    : ", substr($test, -2),    "\n";

length        : 12
index of '_'  : 3
rindex of '_' : 9
substr(0,3)   : alu
substr(-2)    : 01

Before anything else, the counting. "alu_smoke_01" has 12 characters at positions 0 through 11:

So:

• length counts characters: 12. The last valid position is length - 1, which is 11.
• index returns the position of the first match: the first _ is at 3.
• rindex returns the position of the last match: the second _ is at 9.
• substr($s, 0, 3) takes 3 characters starting at position 0: alu.
• substr($s, -2) with a negative offset counts from the end, and with no length takes everything from there: 01.

That is the whole model. The rest of the chapter is what happens at the edges of it.

2. The index Return Value Breaks the Obvious Test

index reports a position, not a yes-or-no. That has a consequence people trip over constantly.

use strict;
use warnings;

my $haystack = "alu_top";

for my $needle ("alu", "Z", "top") {
    my $i = index($haystack, $needle);
    printf "%-4s index=%2d   if(index) says %-7s   (index >= 0) says %s\n",
        $needle, $i,
        ($i      ? "FOUND" : "absent"),
        ($i >= 0 ? "FOUND" : "absent");
}

alu  index= 0   if(index) says absent    (index >= 0) says FOUND
Z    index=-1   if(index) says FOUND     (index >= 0) says absent
top  index= 4   if(index) says FOUND     (index >= 0) says FOUND

Read the first two rows carefully.

• alu is present, at position 0. if (index(...)) reports it absent, because 0 is false — Chapter 2.8.
• Z is absent, giving -1. if (index(...)) reports it FOUND, because -1 is true.

So the truthiness test is wrong in both interesting cases and right only for top, which is found somewhere in the middle. It is not "usually right with an edge case" — it inverts the answer exactly when the answer matters.

Searching from a position

index takes an optional third argument, the position to start from, which is how you find a second occurrence:

use strict;
use warnings;

my $test = "alu_smoke_01";

my $first  = index($test, "_");
my $second = index($test, "_", $first + 1);

print "first  _ at $first\n";
print "second _ at $second\n";

first  _ at 3
second _ at 9

rindex takes the same optional argument, searching backwards from it.

3. substr Takes a Span

Three arguments: the string, the start position, and how many characters.

use strict;
use warnings;

my $test = "alu_smoke_01";

print "substr(0, 3)  : ", substr($test, 0, 3),  "\n";
print "substr(4, 5)  : ", substr($test, 4, 5),  "\n";
print "substr(4)     : ", substr($test, 4),     "\n";
print "substr(-2)    : ", substr($test, -2),    "\n";
print "substr(-5, 2) : ", substr($test, -5, 2), "\n";

substr(0, 3)  : alu
substr(4, 5)  : smoke
substr(4)     : smoke_01
substr(-2)    : 01
substr(-5, 2) : ke

• Omit the length and you get everything from the start position to the end.
• A negative start counts from the end: -2 is the second-from-last character.
• A negative length is also allowed, and means "stop this many characters from the end" — a form worth knowing exists but rarely the clearest way to express an intent.

Combining index and substr extracts a field without pattern matching:

use strict;
use warnings;

my $test = "alu_smoke_01";

my $sep    = index($test, "_");
my $block  = substr($test, 0, $sep);
my $rest   = substr($test, $sep + 1);

print "block: $block\n";
print "rest : $rest\n";

block: alu
rest : smoke_01

For splitting on a separator this is the long way round — Chapter 4.7 does it in one call — but it is the right tool when the positions are fixed rather than delimited, which is what a fixed-width report gives you.

4. substr Can Also Write

Two forms replace part of a string in place.

The four-argument form replaces the span and returns what was there:

use strict;
use warnings;

my $test = "alu_smoke_01";

my $old = substr($test, 0, 3, "ALU");

print "returned: $old\n";
print "now     : $test\n";

returned: alu
now     : ALU_smoke_01

The assignment form does the same without handing back the old text:

use strict;
use warnings;

my $test = "alu_smoke_01";

substr($test, 0, 3) = "cpu";

print "now: $test\n";

now: cpu_smoke_01

A function call on the left of an assignment is unusual and worth recognising. Both forms modify the original variable, which is a real exception to the rule from Chapter 3.1 §5 that operators do not change their operands — these functions are explicitly designed to.

Note the replacement need not be the same length as the span it replaces; the string grows or shrinks to fit.

5. The Boundary Cases Differ From Each Other

This is worth measuring because an off-by-one lands on the quiet one.

use strict;
use warnings;

my $s = "abc";

my $at_end   = substr($s, 3, 2);
print "substr(3,2) = [", (defined $at_end ? $at_end : "undef"), "] length ", length($at_end), "\n";

my $past_end = substr($s, 10, 2);
print "substr(10,2) = [", (defined $past_end ? $past_end : "undef"), "]\n";

substr(3,2) = [] length 0
substr(10,2) = [undef]

substr outside of string at boundaries.pl line 9.

Two different behaviours:

• Starting exactly at length — position 3 in a 3-character string — is legal. You get the empty string, no warning. This is deliberate: it is the natural result of asking for what comes after the last character.
• Starting beyond length returns undef and warns substr outside of string.

So a loop whose index runs one too far gets an empty string silently; one that runs well past gets undef and a warning. The quiet case is the more likely bug.

length(undef) is undef, not zero

use strict;
use warnings;

my $missing;

my $len = length($missing);

print "length(undef) = [", (defined $len ? $len : "undef"), "]\n";

length(undef) = [undef]

No warning, and the answer is undef rather than 0. That is arguably the right answer — an absent string has no length, as distinct from having length zero — but it means these two tests are not the same thing:

if (length($field)) { ... }           # false for undef AND for ""
if (defined($field) && length($field)) { ... }   # says which

If the difference between "the field was missing" and "the field was present and empty" matters to your report, Chapter 2.7's defined is the test that distinguishes them.

6. The Engineering Case: Fixed-Width Columns

Tool reports are often fixed-width rather than delimited, and that is exactly what substr is for.

use strict;
use warnings;

#          0         1         2         3
#          0123456789012345678901234567890123
my $row = "alu_smoke_01    PASS       184  7";

my $name   = substr($row,  0, 16);
my $status = substr($row, 16, 11);
my $total  = substr($row, 27,  5);
my $fails  = substr($row, 32,  2);

for my $pair (["name", $name], ["status", $status], ["total", $total], ["fails", $fails]) {
    printf "%-7s [%s]\n", $pair->[0], $pair->[1];
}

name    [alu_smoke_01    ]
status  [PASS       ]
total   [184  ]
fails   [7]

Every field came out, and three of the four carry padding. That is the characteristic result of fixed-width extraction, and it is why the next chapter exists: [PASS ] is not eq to "PASS".

Note the last field, fails, came out as [7] with no padding at all — the row is 33 characters and substr($row, 32, 2) asked for two characters starting at the last one, so it got the one character that exists. That is §5's "starting at length" rule applying partway through a request: substr returns what it can and does not warn. A trailing column is therefore the one most likely to be shorter than its declared width.

Chapter 4.6 covers the trimming, and Chapter 3.5 explains why the numeric field survives a numeric comparison while the text field does not survive a textual one.

The comment row above the data is a habit worth copying: a ruler in the source makes the offsets checkable by eye instead of by counting.

7. Why This Matters in Engineering Work

Fixed-width reports are common and substr is the honest tool for them. Column positions from a tool's documentation map directly onto substr calls. Reaching for a regular expression to pull a column out of a fixed-width row is more code and more ways to be wrong.

Prefix and suffix checks. "Does this line start with ERROR" is index($line, "ERROR") == 0, and "does this filename end in .log" is a substr of the last four characters — or rindex($name, ".log") == length($name) - 4. Both avoid pattern matching entirely, and the == 0 form is the one §2 warns you to write deliberately.

The -1 bug costs real debugging time because of its shape: a prefix check that fails on exactly the lines it should match, and succeeds on lines containing nothing relevant. That reads as a data problem, which is where people look first.

Padding is the thing to expect. Anything taken out of a fixed-width row needs trimming before it is compared as text. A status column that is eq to nothing you test for, despite printing as the right word, is this bug.

8. Common Mistakes

if (index(...)). Wrong when the match is at position 0 and wrong when there is no match. Test >= 0.

Expecting index to return undef when absent. It returns -1, which is defined and true.

Counting positions from 1. The first character is at 0 and the last is at length - 1.

Forgetting that substr with four arguments modifies the original. That is its purpose, but it surprises a reader expecting a function to return a value and change nothing.

Assuming the two out-of-range cases behave alike. Starting at length gives "" silently; starting past it gives undef and warns.

Treating length(undef) as 0. It is undef, with no warning.

Comparing an untrimmed fixed-width field as text. "PASS " is not eq to "PASS".

9. Exercise

Exercise 1 — build the position table. For my $s = "fifo_wr_02"; print its length, and the positions of the first and last _. Then write out the position-to-character table as in §1 and confirm your numbers against it.

Exercise 2 — the double failure. Run the §2 script. Then write one sentence for each of the first two rows explaining why the truthiness test gave the answer it did, naming the chapter that established the relevant property of 0 and -1.

Exercise 3 — a prefix check done right. Write a loop over the three lines "ERROR: bad", "WARN: odd" and "an ERROR happened" that prints which ones start with ERROR. Get exactly one match. Then replace your test with if (index($line, "ERROR")) and record how many you now get and which.

Exercise 4 — slice a name apart. Using only index and substr, split "alu_smoke_01" into its three underscore-separated pieces and print each on its own line. Then say in one clause why Chapter 4.7 will be the better tool for this shape of problem.

Exercise 5 — replace in place. Starting from my $t = "alu_smoke_01";, use the four-argument substr to change the block name to cpu, printing both the returned value and the new string. Then achieve the same result with the assignment form and say which you find clearer.

Exercise 6 — the two boundaries. With my $s = "abcde"; call substr($s, 5, 2) and substr($s, 9, 2), capturing STDOUT and STDERR separately. Report which returned the empty string, which returned undef, and which warned.

Exercise 7 — the padded column. Take the §6 row and extract the status field. Confirm with eq that it does not equal "PASS", print its length, and then say what has to happen before the comparison can work.

10. Summary

• Positions start at 0; the last character is at length - 1.
• length counts characters, and length(undef) is undef — not 0, and with no warning.
• index returns a position, rindex the last one, and both return -1 when absent.
• if (index(...)) is wrong in both directions — 0 is false and -1 is true. Always test >= 0.
• substr($s, start, len) takes a span; omit the length for "to the end", and a negative start counts from the end.
• Four-argument and assignment substr modify the original string and the replacement need not match the span's length.
• Starting at length gives "" silently; starting past it gives undef and warns.
• Fixed-width extraction always brings padding with it, which must be removed before any textual comparison.

Chapter 4.6 deals with exactly that: removing the surrounding whitespace, and normalising case so that two spellings of the same word stop being two different values.