Sets or Lists, and Removing Duplicates

Part I · Module 7 · Chapter 7.6 · Sets or Lists, and Removing Duplicates

1. The Same Names, Two Collections

suite = ["smoke", "reset", "smoke"]
failed = {"reset", "timeout"}
print(len(suite))
print(len(failed))

3
2

Both hold test names. They are not interchangeable, and the difference is not that one is tidier.

A list is a sequence: there is a first element, the order means something, and "smoke" appearing twice is two facts rather than one. A set is a group of distinct values: there is no first element and no way to appear twice.

So the question is never "which type is better". It is what does this data mean, and what will I ask of it?

2. The One Question

Apply it to the two collections above.

The suite is the order tests will run in. That order is the whole point, and if "smoke" is listed twice it runs twice. Both properties are real information, so it is a list.

The failures are which tests failed. A test that failed three times still just failed, and there is no meaningful "first failure" in a group of names. Neither property is information, so it is a set — and writing it as one says that to the next reader.

3. What Each One Can Do

The capabilities follow from the meanings rather than being arbitrary.

A list has positions, so you can ask where something is:

suite = ["smoke", "reset", "smoke"]
print(suite[0])
print(suite[-1])
print(suite.count("smoke"))

smoke
smoke
2

Position 0, the last position, and how many times a value occurs. A set can do none of those:

s = {"a", "b"}
print(s.index("a"))

AttributeError: 'set' object has no attribute 'index'

s = {"a", "b"}
print(s.count("a"))

AttributeError: 'set' object has no attribute 'count'

Neither is a missing feature. index asks where, and a set has no positions; count asks how many times, and a set stores each value once. The questions have no meaning, so the methods do not exist.

What a set has instead is Chapters 7.4 and 7.5 — relationships between whole groups:

a = {"x", "y"}
b = {"y", "z"}
print(sorted(a - b))
print(sorted(a & b))
print(a.isdisjoint(b))

['x']
['y']
False

Doing that with lists means writing the comparison yourself, and getting duplicates and order tangled up in it.

Both are changeable, which is worth noting since Chapter 6 contrasted a mutable list with an immutable tuple. The list/set choice is not about mutability at all.

4. Membership on Both

Both collections answer the membership question, and they agree:

l = ["a", "b", "a"]
s = {"a", "b"}
print("a" in l, "a" in s)

True True

So membership alone does not decide anything. What decides it is whether you also need the other properties — and whether membership is the dominant question. Chapter 7.2 explained the mechanism: a list checks its elements one after another, while a set goes more or less straight to the answer. For a handful of names that is irrelevant; for a large collection checked repeatedly it is the reason the set exists.

5. Removing Duplicates, and What It Costs

Here is the idiom everybody learns:

names = ["smoke", "reset", "smoke"]
unique = list(set(names))
print(len(names), len(unique))

3 2

Three in, two out. The duplicate is gone, and the original list is untouched:

names = ["smoke", "reset", "smoke"]
unique = list(set(names))
print(names)

['smoke', 'reset', 'smoke']

That line is taught constantly as "remove duplicates from a list", as though the result were the same list with one item deleted. It is not, and the difference matters.

Two things were discarded, not one.

The obvious loss is the duplicate itself — but notice that the count went with it:

names = ["smoke", "reset", "smoke"]
print(names.count("smoke"))
print(len(set(names)))

2
2

The list knows "smoke" appeared twice. After the conversion that fact is unrecoverable: the set holds two distinct names, and nothing in it records that one of them arrived twice. If "how many times did this test fail?" is a question you will ever ask, the conversion has already destroyed the answer.

The less obvious loss is order — and this is not a theoretical caveat. The expression list(set(names)) on four names was run in twelve separate Python processes:

Eight different orderings of the same four names, from identical code. Chapter 7.2 explained why: a set has no order to preserve, so converting back to a list produces an order with no relationship to the original and no stability between runs.

The honest one-liner

If you want deduplication and a reproducible result, sort it:

names = ["smoke", "reset", "smoke"]
print(sorted(set(names)))

['reset', 'smoke']

That is deterministic, it is one line, and it uses only sorted() from Chapter 5.5 and set() from Chapter 7.1. Read it inside-out: set(names) discards duplicates, sorted(...) imposes a defined order and hands back a list.

Be clear about what it does not do. It gives alphabetical order, not the original order. "reset" comes first in the result even though "smoke" came first in the data. If you genuinely need the order the items were first seen, neither form gives it to you, and the tools that do come from later modules — this is worth knowing you cannot yet do rather than reaching for one of these and assuming it worked.

When it is exactly right

raw = ["crc", "timeout", "crc", "reset", "timeout"]
failed_once_or_more = set(raw)
print(len(failed_once_or_more))
print("crc" in failed_once_or_more)
print(sorted(failed_once_or_more))

3
True
['crc', 'reset', 'timeout']

Here the question is which tests failed at all. A test failing twice is not two failures to report — it is one broken test. The order the failures were observed in is an artefact of scheduling, not information about the design. Both discarded properties were noise, so converting is not a loss; it is the collection finally matching the question.

6. Choosing, Worked Through

Five cases, with the question each one turns on:

That last one is worth pausing on. Runtimes are numbers and numbers repeat — storing them in a set would silently merge two tests that took exactly 12.4 seconds into one entry. Whenever identical values are genuinely distinct observations, a set is the wrong collection, regardless of how you plan to query it.

7. Predict, Run, Observe

Predict every line before reading on.

items = ["a", "b", "a", "c"]
s = set(items)
print(len(items))
print(len(s))
print(items.count("a"))
print("a" in items, "a" in s)
print(sorted(s))
print(items == ["a", "b", "a", "c"])
print(s == {"c", "b", "a"})

The verified output:

4
3
2
True True
['a', 'b', 'c']
True
True

Four items, three distinct values. items.count("a") is 2 — a question only the list can answer.

Membership agrees on both. sorted(s) is reproducible and alphabetical.

The list is unchanged by the conversion, and the set equals {"c", "b", "a"} because set equality ignores the order you wrote it in — Chapter 7.1.

Now the question the chapter is really testing:

print(list(set(items)))

What order will those three names be in? The question has no answer — eight different orders came out of twelve runs of the equivalent expression. If you predicted ['a', 'b', 'c'], you predicted sorted(). If you predicted the original order minus the duplicate, you predicted something no collection here promises.

8. Exercises

Exercise 1 — choose and justify. For each, say list or set and give the one-sentence reason: (a) the lines of a report, in order (b) which error codes appeared (c) the queue of jobs to run (d) which files a script touched (e) the measured runtime of each test (f) the set of supported file extensions

Exercise 2 — predict. For items = ["x", "y", "x"], give each result and verify: (a) len(items) (b) len(set(items)) (c) items.count("x") (d) "x" in set(items) (e) sorted(set(items)) (f) set(items) == {"y", "x"}

Exercise 3 — what was lost. After u = set(["crc", "timeout", "crc"]), name the two pieces of information that no longer exist, and give the expression on the original list that would have answered each.

Exercise 4 — the order claim. A colleague writes unique = list(set(names)) and says it "removes duplicates but otherwise leaves the list alone". Write three sentences correcting them, citing what happens across runs, and give the line you would submit instead if the result is printed.

Exercise 5 — the right trade. Describe a situation where converting a list of names to a set loses nothing at all, and one where it silently destroys the answer to a question somebody will ask later.

Exercise 6 — the method that does not exist. Predict what {"a", "b"}.count("a") does, run it, write down the exact error, and explain in two sentences why this is a consequence of what a set is rather than a gap in the type.

Exercise 7 — the numbers trap. Explain in two sentences why storing a run's measured runtimes in a set is a bug, even though you only ever want to ask which values occurred.

Exercise 8 — deterministic dedup. Write the one line that removes duplicates from a list of names and produces a reproducible result. Say what order it gives you, and state plainly what it does not give you.

9. Summary

• The choice is about what the data means, not about which type has more methods.
• If order matters, or a repeat is real information, use a list. If neither is true, use a set.
• A set has no index and no count — AttributeError for both — because where and how many times are not questions it can answer.
• A list has no difference, intersection, union or subset tests. Those are the questions a set is for.
• Both answer membership, so membership alone decides nothing. A set is better when membership is the dominant question on a large collection.
• Both are mutable — the list/set choice has nothing to do with mutability.
• list(set(items)) discards two things: the duplicate counts, and the order. Twelve runs of the same four names produced eight different orders.
• Never use list(set(...)) where the output order is observed.
• sorted(set(items)) deduplicates reproducibly — but gives alphabetical order, not the original order. Preserving first-seen order needs tools from later modules.
• Deduplication is a change of abstraction, correct when the duplicates and order were never information and a silent data loss when they were.
• Identical values that are genuinely distinct observations — runtimes, measurements — must not go in a set.

That completes Module 7 — Sets.

You now have a third collection, and it is not a variation on the first two. A set stores each distinct value once, has no positional order at all, and is built around a single question: is this value present? From that one description everything else in the module follows — why there is no indexing, why == ignores arrangement, why elements must be hashable, and why {} could not be the empty set.

The most valuable habit here is the one about order. Python gives you no positional-order contract for a set, and the measurements in Chapters 7.2 and 7.6 show what that means in practice: the same code, in twelve runs, producing five and eight different orderings. Whenever you need to see or write what is in a set, route it through sorted(). That single discipline prevents an entire category of bug that passes every test you write by hand.

The four operations are what sets are actually for. Two groups of names and one line each: what broke, what got fixed, what failed in both, what failed in either — plus the relationship tests that answer coverage questions as a plain yes or no.

Module 8 — Dictionaries introduces the last of the core collections, and it answers a question none of these can: not is this value present? but what value is filed under this name? A set remembers that "crc" failed; a dictionary remembers that "crc" failed with seed 4210, after 18.2 seconds. It is the structure most engineering results eventually live in. And Chapter 8.9 finally puts all five types — string, list, tuple, dictionary and set — into one decision.