Python ·
Python dict assignment doesn't copy - it aliases the dict
config = defaults binds a second name to the same dict, so an update() writes into your defaults. How to spot aliasing and copy a dict properly.
Test yourself before reading - the write-up below gives it away.
# Build a per-environment config on top of the defaults.
def with_overrides(base, overrides):
merged = base
merged.update(overrides)
return merged
DEFAULTS = {"retries": 3, "debug": False}prod = with_overrides(DEFAULTS, {"debug": True})print(DEFAULTS["debug"])
# expected: False
# actual: True (!)
The debug flag that turned itself on everywhere
One deploy later, every environment was logging at debug level - including the two that never asked for it.
# Build a per-environment config on top of the defaults.
def with_overrides(base, overrides):
merged = base
merged.update(overrides)
return merged
DEFAULTS = {"retries": 3, "debug": False}
prod = with_overrides(DEFAULTS, {"debug": True})
print(DEFAULTS["debug"])
# expected: False
# actual: True (!)
Why does changing the "copy" change the original dict?
Because there is no copy. merged = base binds a second name to the same dict
object - assignment in Python never duplicates anything. merged.update(overrides)
therefore writes straight into DEFAULTS, and every config built afterwards
inherits debug: True. The fix: make the copy explicit - merged = dict(base) -
before mutating.
One dict, two names
Python variables are names bound to objects, never boxes holding values - Ned
Batchelder's Facts and myths about Python names and values
is the canonical walkthrough. After line 3 both names refer to one object, which
id() will confirm:
>>> merged = base
>>> id(merged) == id(base)
True
Line 4, the update, is the line everyone stares at, and it would be correct on a
private copy - the defect is one line up, where the private copy was supposed to
be made and a mere rebinding happened instead. The
copy module docs open with this
exact distinction: "Assignment statements in Python do not copy objects, they
create bindings between a target and an object."
What makes it nasty is that the first call returns the right answer. prod has
debug: True, as requested. The damage lands on whoever reads DEFAULTS after
this call - so the symptom shows up far from the cause, in code that did nothing
wrong. A test that calls with_overrides once and checks the return value passes.
A suite that happens to run the defaults-consuming test first also passes. Reorder
the tests and it fails - the classic signature of shared mutable state, and worth
remembering next time a suite goes red only under pytest -p no:randomly.
The staging outage that was really a dict
A team runs build_config("staging") in a request handler. Under the hood, each
call layers overrides onto module-level defaults exactly like the snippet. The
first staging request after boot works. From then on the defaults carry
staging's database URL, so the next environment to boot from the same module -
a worker, a cron job - quietly connects to staging's database. The postmortem
takes a day because everyone reads build_config and sees a function that
"returns a new config". It returns the old one, renamed.
The fix - and how deep it needs to go
def with_overrides(base, overrides):
merged = dict(base)
merged.update(overrides)
return merged
dict(base), base.copy(), or on Python 3.9+ the merge operator, which builds
the new dict in one step (PEP 584):
return base | overrides
The half-fix to watch for: a shallow copy duplicates only the top level. If your
defaults contain a nested dict - {"db": {"host": ..., "pool": 5}} - then
dict(base) gives you a new outer dict whose "db" key still points at the one
shared inner dict, and merged["db"]["pool"] = 50 is this bug all over again, one
level down. For nested config, reach for copy.deepcopy(base) and pay the copy
cost once, at startup.
How to find dict aliasing in your codebase
- Grep for functions that mutate their parameters:
grep -rnE '\b(def [a-z_]+\([^)]*\))' -A 5 --include='*.py' . | grep -E '\.(update|pop|setdefault)\('is crude but surfaces candidates fast. - The debugging tool is
id(): when a value changes "by itself", printid()of the two dicts you believed were independent. - Module-level mutable constants (
DEFAULTS = {...}) deserve a comment or aMappingProxyTypewrapper -types.MappingProxyType(DEFAULTS)makes accidental writes raise aTypeErrorinstead of corrupting state.
Related bugs
Aliasing has a whole family here: a list shared through a class attribute, a list shared through a mutable default argument, and in Go, two slices sharing one backing array. Hub: bug write-ups.
with_overrides was never a bad name. It just needed one honest dict() call to
live up to it.