Filtering and Transforming a Result List

Part I · Module 12 · Chapter 12.3 · Filtering and Transforming a Result List

1. Two Jobs

Given a list of results, there are two useful things to do to it:

• Select — keep some items and drop the rest. Fewer items come out; the ones that do are unchanged.
• Project — turn each item into something else. The same number come out; each is a different shape.

A comprehension can do either, or both in one line. Everything in this chapter is one of those three cases.

The working data:

results = [
    {"test": "axi_smoke", "status": "PASS", "runtime_s": 120},
    {"test": "axi_burst", "status": "FAIL", "runtime_s": 900},
    {"test": "reset_seq", "status": "FAIL", "runtime_s": 45},
    {"test": "cdc_check", "status": "PASS", "runtime_s": 700},
]

print(len(results))

4

2. Select: Keep the Runs That Matter

The output expression is r — the whole record, unchanged — and the if does all the work:

results = [
    {"test": "axi_smoke", "status": "PASS", "runtime_s": 120},
    {"test": "axi_burst", "status": "FAIL", "runtime_s": 900},
    {"test": "reset_seq", "status": "FAIL", "runtime_s": 45},
    {"test": "cdc_check", "status": "PASS", "runtime_s": 700},
]

slow = [r for r in results if r["runtime_s"] > 600]

print(len(slow))
for r in slow:
    print(r["test"], r["runtime_s"])

2
axi_burst 900
cdc_check 700

Two of four kept, each one a complete record with every field it had. Nothing was reshaped.

The filter is an ordinary condition, so everything from Module 9 applies — comparisons, and/or/not, membership, truthiness:

results = [
    {"test": "axi_smoke", "status": "PASS", "runtime_s": 120},
    {"test": "axi_burst", "status": "FAIL", "runtime_s": 900},
    {"test": "reset_seq", "status": "FAIL", "runtime_s": 45},
]

slow_failures = [r for r in results if r["status"] == "FAIL" and r["runtime_s"] > 600]
axi_only = [r for r in results if r["test"].startswith("axi")]

print([r["test"] for r in slow_failures])
print([r["test"] for r in axi_only])

['axi_burst']
['axi_smoke', 'axi_burst']

3. Project: Reshape Every Item

Now the if goes away and the output expression does the work:

results = [
    {"test": "axi_smoke", "status": "PASS", "runtime_s": 120},
    {"test": "axi_burst", "status": "FAIL", "runtime_s": 900},
]

names = [r["test"].upper() for r in results]
minutes = [r["runtime_s"] / 60 for r in results]
labels = [r["test"] + ": " + r["status"] for r in results]

print(names)
print(minutes)
print(labels)

['AXI_SMOKE', 'AXI_BURST']
[2.0, 15.0]
['axi_smoke: PASS', 'axi_burst: FAIL']

Two in, two out, every time. The output expression can be a field lookup, a method call, arithmetic, a concatenation — anything that produces a value, because Chapter 3.1's definition of an expression is the only requirement.

A projection can also narrow a record to the fields a report needs:

results = [
    {"test": "axi_smoke", "status": "PASS", "runtime_s": 120, "seed": 4210, "sim": "vcs"},
    {"test": "axi_burst", "status": "FAIL", "runtime_s": 900, "seed": 4211, "sim": "vcs"},
]

summary = [{"test": r["test"], "status": r["status"]} for r in results]

print(summary)

[{'test': 'axi_smoke', 'status': 'PASS'}, {'test': 'axi_burst', 'status': 'FAIL'}]

The output expression is a dictionary literal — still one value per item, so this is still a list comprehension. Chapter 12.2's rule holds: the brackets are decided by what the whole comprehension builds, not by what the items are.

4. Both at Once

The common case in real scripts:

results = [
    {"test": "axi_smoke", "status": "PASS", "runtime_s": 120},
    {"test": "axi_burst", "status": "FAIL", "runtime_s": 900},
    {"test": "reset_seq", "status": "FAIL", "runtime_s": 45},
]

slow_failure_names = [r["test"] for r in results if r["status"] == "FAIL" and r["runtime_s"] > 600]

print(slow_failure_names)

['axi_burst']

Read it in written order: the test name — for each result — if it failed and took over ten minutes. Select first in meaning, project second; written the other way round, which Chapter 12.1 Section 7 flagged as the one genuinely unusual thing about the syntax.

5. The Count Tells You Which Job

A reliable diagnostic for a comprehension you did not write:

results = [
    {"test": "a", "status": "PASS"},
    {"test": "b", "status": "FAIL"},
    {"test": "c", "status": "FAIL"},
]

selected = [r for r in results if r["status"] == "FAIL"]
projected = [r["test"] for r in results]
both = [r["test"] for r in results if r["status"] == "FAIL"]

print("source   :", len(results))
print("selected :", len(selected))
print("projected:", len(projected))
print("both     :", len(both))

source   : 3
selected : 2
projected: 3
both     : 2

A filter is the only thing that can change the count. If a comprehension has no if and its result is shorter than its source, something else is wrong — and Chapter 12.2 Section 6's duplicate-key drop is the usual culprit.

6. A Filter Does Not Protect the Output

Here is the way this actually breaks in a script, and it is not a syntax problem:

results = [
    {"test": "axi_smoke", "status": "PASS", "runtime_s": 120},
    {"test": "axi_burst", "status": "FAIL"},
    {"test": "reset_seq", "status": "FAIL", "runtime_s": 45},
]

slow = [r["test"] for r in results if r["runtime_s"] > 600]

print(slow)

KeyError: 'runtime_s'

The comprehension is valid. The data is not uniform — the middle record has no runtime_s, perhaps because that run was killed before it finished — and the filter asks for the field on every item before deciding anything.

A filter runs on all of them; it protects nothing. Chapter 8.3's safe lookup is the fix:

results = [
    {"test": "axi_smoke", "status": "PASS", "runtime_s": 120},
    {"test": "axi_burst", "status": "FAIL"},
    {"test": "reset_seq", "status": "FAIL", "runtime_s": 45},
]

slow = [r["test"] for r in results if r.get("runtime_s", 0) > 600]
missing = [r["test"] for r in results if "runtime_s" not in r]

print("slow   :", slow)
print("missing:", missing)

slow   : []
missing: ['axi_burst']

.get("runtime_s", 0) supplies a default so every record can be tested. The second line is worth writing too — a record missing a field is usually something you want to know about rather than silently treat as zero.

7. Two Comprehensions or One?

A long filter can be split:

results = [
    {"test": "axi_smoke", "status": "PASS", "runtime_s": 120},
    {"test": "axi_burst", "status": "FAIL", "runtime_s": 900},
    {"test": "reset_seq", "status": "FAIL", "runtime_s": 45},
    {"test": "cdc_check", "status": "FAIL", "runtime_s": 700},
]

failures = [r for r in results if r["status"] == "FAIL"]
slow_failures = [r for r in failures if r["runtime_s"] > 600]

print([r["test"] for r in slow_failures])

['axi_burst', 'cdc_check']

results = [
    {"test": "axi_smoke", "status": "PASS", "runtime_s": 120},
    {"test": "axi_burst", "status": "FAIL", "runtime_s": 900},
    {"test": "reset_seq", "status": "FAIL", "runtime_s": 45},
    {"test": "cdc_check", "status": "FAIL", "runtime_s": 700},
]

slow_failures = [r for r in results if r["status"] == "FAIL" and r["runtime_s"] > 600]

print([r["test"] for r in slow_failures])

['axi_burst', 'cdc_check']

Same answer. The two-step version costs one extra name and one extra pass, and it buys two things: each step is readable on its own, and failures is now available if anything else needs it — a count for the summary, say.

Prefer the split when the intermediate has a name worth saying. failures is such a name. If the intermediate would be called temp or filtered1, the single line is better.

8. Predict, Run, Observe

rows = [
    {"n": "a", "v": 1},
    {"n": "b", "v": 5},
    {"n": "c", "v": 9},
]

# A
print("A:", [r["n"] for r in rows])

# B
print("B:", [r for r in rows if r["v"] > 4] == rows[1:])

# C
print("C:", [r["v"] * 2 for r in rows if r["v"] > 4])

# D
print("D:", len([r for r in rows]), len([r["n"] for r in rows if r["v"] > 100]))

The verified output:

A: ['a', 'b', 'c']
B: True
C: [10, 18]
D: 3 0

A — a pure projection: three in, three out.

B — a pure selection keeps the records themselves, so the result really is the last two rows, identical objects.

C — both jobs: filter to two, then double each.

D — no filter means the count is unchanged; a filter nothing matches gives an empty list, not an error.

9. Exercises

Exercise 1 — name the job. For each, say select, project or both: (a) [r for r in results if r["seed"] > 4210] (b) [r["seed"] for r in results] (c) [r["test"] for r in results if r["sim"] == "vcs"] (d) [len(r["test"]) for r in results].

Exercise 2 — select. From the Section 1 data, keep every result that failed, then every result that took under a minute. Print the count each time.

Exercise 3 — project. From the same data, build (a) the list of test names (b) the list of runtimes in minutes (c) a list of "name (status)" strings.

Exercise 4 — both. Build the names of the tests that passed and took over ten minutes. Say which clause did which job.

Exercise 5 — the count check. Write a comprehension with no if whose result is shorter than its source. Explain what happened, and name the chapter that covers it.

Exercise 6 — the missing field. Build a results list where one record lacks "seed", then write a comprehension that reads it. Record the error, then fix it two ways — with a default, and by filtering on presence first.

Exercise 7 — split or not. Take a comprehension with a three-part filter and split it into named steps. Say whether you would keep the split, and why, in two sentences.

Exercise 8 — narrow a record. From results with five fields each, build a list of dictionaries carrying only test and status. Say why this is still a list comprehension despite the braces in it.

10. Summary

• A comprehension does one of two jobs, or both: select (keep some items, unchanged) and project (reshape every item).
• The if clause selects; the output expression projects.
• A filter is an ordinary condition — everything from Module 9 applies, including and binding tighter than or.
• The output expression is an ordinary expression — a lookup, a method call, arithmetic, or a dictionary literal.
• A projection into dictionaries is still a list comprehension. The outer brackets decide the result type.
• The count is the diagnostic: a filter changes len(), a transformation does not, and only a filter should.
• A filter does not protect the output. It runs on every item, so a field read in the filter must exist on all of them.
• Non-uniform data is the realistic failure, and .get(key, default) or an in check is the fix.
• Split a long filter when the intermediate deserves a name; keep it one line when it would be called temp.

The next chapter takes the one piece of comprehension syntax that is genuinely confusing, and separates it from the filter.