Encapsulation and the Underscore Convention

Part I · Module 16 · Chapter 16.7 · Encapsulation and the Underscore Convention

1. There Is No Privacy

Start with the fact, executed:

class Parser:
    def __init__(self):
        self._buffer = []
        self.count = 0

    def feed(self, line):
        self._buffer.append(line)
        self.count += 1


p = Parser()
p.feed("UVM_ERROR drv.sv(88)")

print("reading the internal name  :", p._buffer)
p._buffer.append("inserted from outside")
print("writing it                 :", p._buffer)
p._buffer = "not even a list any more"
print("replacing it entirely      :", p._buffer)
print("the object is now broken   :", p.count)

reading the internal name  : ['UVM_ERROR drv.sv(88)']
writing it                 : ['UVM_ERROR drv.sv(88)', 'inserted from outside']
replacing it entirely      : not even a list any more
the object is now broken   : 1

Nothing raised at any point. Outside code read the internal buffer, appended to it, and then replaced it with a string — leaving an object whose feed method will now fail and whose count no longer matches its contents.

That is Python working as designed. There is no private, no protected, and no way to make an attribute unreachable.

2. The Convention

What there is instead:

class Parser:
    def __init__(self):
        self._buffer = []     # internal - by agreement, not enforcement

A single leading underscore marks a name as internal. It changes nothing about what the language permits. It is a message to the next person reading the code — who is usually you.

class Parser:
    def __init__(self):
        self._buffer = []
        self.errors = 0

    def feed(self, line):
        self._buffer.append(line)
        if "UVM_ERROR" in line:
            self.errors += 1

    def summary(self):
        return f"{len(self._buffer)} lines, {self.errors} errors"


p = Parser()
for line in ["UVM_INFO start", "UVM_ERROR drv.sv(88)", "UVM_INFO done"]:
    p.feed(line)

print("public  :", p.summary(), "| errors =", p.errors)
print("internal:", p._buffer)

public  : 3 lines, 1 errors | errors = 1
internal: ['UVM_INFO start', 'UVM_ERROR drv.sv(88)', 'UVM_INFO done']

The public surface is feed, summary and errors. _buffer is how the class currently happens to work.

3. What the Underscore Actually Promises

Not secrecy — stability:

A public name is a promise: I will not break this. An underscore name is a warning: I may change or delete this without telling you.

That is the whole content of the convention, and it is a genuinely useful thing to be able to say. A library author who marks _buffer internal is free to replace it next week; one who leaves it public is not.

Which makes the reader's obligation clear: reaching past an underscore is not forbidden, it is uninsured. Sometimes that is the right call — debugging, a workaround for a library bug, a test that needs to inspect internals. You are simply on your own when it changes.

4. The Double Underscore Is Not "More Private"

Readers find __name and reasonably assume it is the stronger form. It is a different mechanism entirely:

class Parser:
    def __init__(self):
        self._buffer = []
        self.__secret = "hidden"


p = Parser()
print("stored under:", sorted(p.__dict__))
print("single underscore:", p._buffer)
print(p.__secret)

stored under: ['_Parser__secret', '_buffer']
single underscore: []
Traceback (most recent call last):
  File "mangle.py", line 10, in <module>
    print(p.__secret)
          ^^^^^^^^^^
AttributeError: 'Parser' object has no attribute '__secret'

The attribute was renamed. __secret inside the class body became _Parser__secret in storage — this is called name mangling, and the AttributeError is not a privacy check, it is the simple consequence of looking up a name that genuinely does not exist.

The mangled name is perfectly reachable:

class Parser:
    def __init__(self):
        self.__secret = "hidden"


p = Parser()
print("mangled access:", p._Parser__secret)
print("in dir():", "_Parser__secret" in dir(p))

mangled access: hidden
in dir(): True

So it is not private, it is inconvenient — and inconvenience was never the goal. Its actual purpose is to stop a subclass accidentally reusing a parent's attribute name, which Chapter 16.9 makes possible. For ordinary internal state, one underscore is the right choice; two is a cargo-cult import from languages that have real access control.

5. The Public Surface Is What You Can Change Later

The practical payoff. Here is a class whose internals change completely, with its public surface untouched:

class ErrorLog:
    """Version 1: stores every line."""

    def __init__(self):
        self._lines = []

    def record(self, line):
        self._lines.append(line)

    def count(self):
        return len(self._lines)


log = ErrorLog()
for line in ["a", "b", "a"]:
    log.record(line)

print("public  count():", log.count())
print("internal _lines :", log._lines)

public  count(): 3
internal _lines : ['a', 'b', 'a']

Now version 2 keeps counts instead of lines, because the logs got large:

class ErrorLog:
    """Version 2: stores counts per distinct message."""

    def __init__(self):
        self._counts = {}

    def record(self, line):
        self._counts[line] = self._counts.get(line, 0) + 1

    def count(self):
        total = 0
        for n in self._counts.values():
            total += n
        return total


log = ErrorLog()
for line in ["a", "b", "a"]:
    log.record(line)

print("public  count():", log.count())
print("internal now    :", log._counts)
print("_lines still there?", hasattr(log, "_lines"))

public  count(): 3
internal now    : {'a': 2, 'b': 1}
_lines still there? False

count() gives the same answer; _lines no longer exists. Any caller using the public method is unaffected. Any caller that reached into log._lines is now broken — and was warned.

That is encapsulation's real benefit, and note it does not require enforcement to work. It requires only that the boundary be stated.

6. Keep the Surface Small

A useful discipline when writing a class: decide what the public names are first, and underscore everything else.

LEGAL = {"PENDING", "PASS", "FAIL"}


class TestRun:
    def __init__(self, name):
        self.name = name              # public: callers read it
        self.status = "PENDING"       # public: callers read it
        self._transitions = []        # internal: how we track history

    def set_status(self, status):
        if status not in LEGAL:
            raise ValueError(f"not a legal status: {status!r}")
        self._transitions.append((self.status, status))
        self.status = status

    def history(self):
        return list(self._transitions)


run = TestRun("smoke")
run.set_status("FAIL")
run.set_status("PASS")

print("name    :", run.name)
print("status  :", run.status)
print("history :", run.history())
print("internal:", run._transitions)
print("history is a copy:", run.history() is run._transitions)

name    : smoke
status  : PASS
history : [('PENDING', 'FAIL'), ('FAIL', 'PASS')]
internal: [('PENDING', 'FAIL'), ('FAIL', 'PASS')]
history is a copy: False

Two public attributes, two public methods, one internal list. The status ends as PASS because set_status was called twice and the history records both transitions — the object's current state and its record of how it got there are separate things, which is what the internal list is for.

Note history() returns list(self._transitions) — a copy, confirmed by the last line: history is a copy: False means the returned list is not the same object as the internal one. A caller mutating what they were given cannot corrupt the object's own record. That is Chapter 5.8's copy problem, handled at a class boundary.

7. Predict, Run, Observe

class Box:
    def __init__(self):
        self._items = []
        self.label = "box"

    def add(self, x):
        self._items.append(x)

    def items(self):
        return list(self._items)


b = Box()
b.add("a")

returned = b.items()
returned.append("b")

print("A:", b.items())
print("B:", returned)
print("C:", b._items)
b._items.append("c")
print("D:", b.items())
print("E:", [n for n in dir(b) if not n.startswith("__")])

A: ['a']
B: ['a', 'b']
C: ['a']
D: ['a', 'c']
E: ['_items', 'add', 'items', 'label']

A and B — appending to the returned list did not reach the object, because items() handed back a copy.

D — appending to _items directly did, because nothing stops it. The underscore is advisory and the object has no defence.

E — dir() lists the internal name alongside the public ones. There is no filtering, no hiding, and no section of the output marked "private". Everything this chapter describes lives in the names alone.

8. Exercises

Exercise 1 — no privacy. Write a class with an underscore attribute, then read it, mutate it and replace it from outside. Confirm nothing raises.

Exercise 2 — break an object. Replace an internal list with a string from outside, then call a method that uses it. Record the error.

Exercise 3 — the promise. Write one sentence each describing what a public name and an underscore name promise a caller.

Exercise 4 — mangling. Define a __name attribute. Show the mangled name in __dict__, access it from outside, and state what it actually protects against.

Exercise 5 — the refactor. Reproduce Section 5's two versions. Write a caller that survives the change and one that does not.

Exercise 6 — the copy. Write an accessor returning an internal list directly, show a caller corrupting the object, then fix it with a copy.

Exercise 7 — surface design. For a class representing a regression job, list the public names you would expose and the internals you would underscore. Justify one of each.

Exercise 8 — uninsured. Describe a situation where reaching past an underscore is the right decision, and say what you would do to limit the risk.

9. Summary

• Python has no private attributes. Every attribute is readable and writable from anywhere.
• A single leading underscore marks a name as internal and changes nothing the language permits.
• The convention is a promise about change, not secrecy: public means "I will not break this", underscore means "I may".
• Reaching past an underscore is uninsured, not forbidden.
• A double underscore triggers name mangling — __x becomes _Class__x — which is collision avoidance for subclasses, not privacy.
• The mangled name is reachable, so two underscores buy inconvenience only.
• Use one underscore; reserve two for the base-class collision case.
• The public surface is what you cannot change later, so keep it small.
• An accessor should return a copy of an internal collection, or callers can corrupt the object through it.
• dir() shows everything — there is no hidden section.

The next chapter is about the methods Python calls for you, starting with the one that decides how your object prints.