Testing with pytest

"Testing shows the presence, not the absence of bugs." — Edsger W. Dijkstra, Software Engineering Techniques (NATO Conference Report, 1970)

Dijkstra meant that as a limitation, and it is one. It is also the most useful thing anyone has said about what a test suite is for. A suite does not prove your code correct. It records the specific ways you have thought about it being wrong, and it tells you when a change reintroduces one of them. That is a smaller claim than "our code is tested" and a far more actionable one.

Python's testing story is unusual in that the standard library ships one framework and almost everybody uses a different one. unittest came from JUnit and carries its shape: test classes, setUp, and a family of assertEqual-style methods. pytest threw that out and bet that a test could be a plain function, an assertion could be the assert statement, and setup could be requested by naming it as a parameter. The bet paid off, and the interesting part is how — because pytest's core features are all small mechanisms you can build yourself in a few lines.

That is what this chapter does. pytest cannot be installed in the lesson sandbox, so instead of describing it we will implement the parts that matter: collection, fixtures, scope, parametrisation. You will finish knowing not just how to use the framework but what it is doing, which is the difference between a suite you can debug and one you cannot.

A test framework is a naming convention and a loop

Collection is the whole of it: find callables whose names say they are tests, call them, and record which raised.

Forty lines and you have the shape of every test framework: a collection rule, a call, and an exception meaning failure. pytest's real collector adds file discovery (test_*.py), classes (Test*), and conftest.py — the rules from the last chapter — but the loop is this.

Look at what the failure printed: got 100, because the test supplied a message. test_deliberately_broken without that message would have printed <no message>, which is what a bare assert gives you. That is the problem pytest's most distinctive feature exists to solve.

The assert that explains itself

An assert statement raises AssertionError with whatever you put after the comma — and nothing if you put nothing.

Three levels of helpfulness from the same comparison. The bare assert tells you only that something was false. A hand-written message tells you the values, at the cost of writing it every time. unittest.assertEqual produces a real diff — which is why that API exists, and why it has forty-odd methods: each one knows how to describe its own kind of failure.

pytest's answer was to keep the bare assert and make it informative anyway. It rewrites the bytecode of your test modules at import time, replacing each assert with code that captures the sub-expressions and reports them on failure. So assert expected == actual prints a full dict diff with the differing key marked, and assert response.status_code == 200 prints the actual status — with no message written and no special method learned.

That is worth understanding rather than just enjoying, because it explains two real behaviours: assertions in helper modules are not rewritten unless you register them, and an assert with side effects is a bad idea, since the rewriter may evaluate parts of your expression more than once for reporting.

Fixtures are dependency injection by parameter name

The second distinctive idea: a test asks for what it needs by naming it, and the framework supplies it.

Two mechanisms, both visible. inspect.signature(fn).parameters reads the parameter names, and each name is looked up as a fixture — so def test_cart_total(cart) is a request, not an argument you have to pass. And cart itself requests rates, resolved recursively, so fixtures compose into a graph rather than a flat list.

The {} passed as the cache is the important detail. A fresh cache per test means every test gets its own rates dict, so one test mutating it cannot affect another. Shared mutable state between tests is the single most common cause of a suite that passes alone and fails together, and fixture scope is the dial that controls it.

Scope: what gets rebuilt, and when

Rebuilding everything per test is safe and sometimes far too slow. Scope trades one for the other.

db_schema is expensive and read-only, so it is built once for the whole run. order is mutable and each test writes to it, so it is rebuilt per test — and test_second proves the isolation held. pytest offers function, class, module, package and session, and the rule for choosing is exactly the one demonstrated: widen the scope only for things nobody mutates.

Real fixtures also need teardown, and pytest spells it with yield: everything before the yield is setup, the yielded value is what the test receives, and everything after runs when the scope ends. That is Part IV's context manager, reused — which is why @contextlib.contextmanager and a pytest fixture look almost identical.

Parametrize: one test, many cases

The third mechanism: generate several tests from one function and a table of inputs.

Each case reports separately, with an identifier naming its inputs. That is the whole benefit over a for loop inside one test: a loop stops at the first failure and reports one result, so you learn that something is broken. Parametrisation runs all five, reports test_rate_for[standard-30.0] by name, and tells you exactly which input class regressed.

The habit worth forming is putting the boundaries in the table — zero, one, the value just below a threshold and the value just above it. Four of the five cases above were chosen that way, and the surcharge boundary is the one a hand-written test would most likely have skipped.

Test doubles, and when not to use one

unittest.mock gives you an object that records calls and returns whatever you tell it.

The last two blocks are the point. A bare Mock invents any attribute you touch, so a test that mocks a collaborator keeps passing after you rename the real method — the test verifies a call that no longer exists. create_autospec builds the double from the real class, so the typo raises, and so does a call with the wrong number of arguments.

The deeper guidance is to reach for a double less often than instinct suggests. notify above did not need mock at all: a five-line class RecordingGateway with a send that appends to a list would test the same behaviour, be readable without knowing the mock API, and fail loudly if the interface changed. Mock at the boundary — network, clock, filesystem, payment provider — and use real objects inside it. A test suite that mocks its own domain objects is testing that your code calls the functions you wrote it to call, which is a tautology that still breaks on every refactor.

Testing the things that will not sit still

Time, randomness and external services make tests flaky. The fix is nearly always the same: pass the dependency in.

is_expired with an injected day is a pure function — same inputs, same output, no clock — so the boundary cases that matter (exactly 30, exactly 31) are trivially testable, and there is no way for the test to pass in the morning and fail at midnight. The version that reads the clock itself can only be tested by patching time.time, which is more machinery for a worse test.

The pattern generalises: push the impure part outward until it is a thin shell that a test can substitute, and keep the logic underneath pure. That is the same argument the last chapter made about entry points, and the same one Part V made about __post_init__ versus a property — moving the decision to where the caller can see it.

Coverage measures what ran, not what you checked

Coverage instruments the interpreter — sys.settrace is the same mechanism coverage.py uses — and reports which lines ran. That is genuinely useful for the negative case: a line at 0% is a line no test touches, which is a fact worth knowing. What it cannot tell you is whether anything checked the result. A test that calls every function and asserts nothing reports 100%.

So treat coverage as a floor, not a target. Below about 60% you are probably not testing; above about 90% you are usually adding tests to satisfy the number rather than to encode a belief. And prefer branch coverage over line coverage, because if x: do() on one line is fully covered by a call that never takes the branch.

Common Mistakes

1. Testing implementation instead of behaviour. Asserting that a helper was called, rather than that the output is right, produces a suite that breaks on every refactor and catches no bugs.
2. Mocking your own domain objects. Doubles belong at the boundary — network, clock, filesystem, payment provider. Inside it, a real object is simpler, more readable and actually verifies something.
3. Using a bare Mock where create_autospec belongs. A plain Mock invents any attribute, so a renamed method leaves the test passing while asserting a call that no longer exists.
4. Sharing mutable state across tests. A module-level list or a session-scoped fixture that tests mutate is the classic "passes alone, fails in the suite" bug. Widen fixture scope only for things nobody mutates.
5. Reading the clock or the random module inside the logic. Inject the day, inject the roll. The result is a pure function whose boundary cases are trivially testable and which cannot fail at midnight.
6. A for loop over cases inside one test. It stops at the first failure and reports one result. Parametrise, and each case reports separately with its inputs in the name.
7. Treating coverage as a target. It measures what ran, not what was checked — a suite that asserts nothing can reach 100%. Use it to find untested code, not to score tested code.
8. Skipping the boundaries. Zero, one, empty, the value at the threshold and the value just past it are where the bugs are. The middle of a range rarely surprises anyone.
9. Assertions with no message and no rewriting. Outside pytest, a bare assert x == y raises with no information. Supply a message or use a framework that rewrites.
10. Tests that depend on each other's order. If test_b needs test_a to have run, you have one test written in two halves. Run with pytest-randomly occasionally to find out.

Interview Corner

Testing questions sort quickly, because "how do you test X?" has an answer for every X and the differences are all about where the seam goes. The tell is whether a candidate reaches for mock.patch first or asks what the function actually depends on.

Summary

• A test framework is a naming rule, a call, and an exception. Collection finds test_* callables, running them is a loop, and failure is a raised AssertionError — everything else is convenience.
• A bare assert carries no information. pytest solves it by rewriting test-module bytecode at import time, which is also why assertions in un-registered helper modules produce worse output.
• Fixtures are dependency injection by parameter name. inspect.signature reads the names; each is resolved recursively, so fixtures compose into a graph.
• Scope is the isolation dial. A fresh cache per test is what stops one test's mutation reaching another. Widen scope only for things nobody mutates.
• Parametrise instead of looping. A loop stops at the first failure and reports one result; parametrisation reports every case separately, named by its inputs.
• Put the boundaries in the table. Zero, one, empty, the threshold and one past it — the middle of a range rarely surprises anyone.
• Mock at the boundary, use real objects inside it. Doubling your own domain objects tests that your code calls the functions you wrote, which breaks on refactor and catches nothing.
• create_autospec over a bare Mock. A plain Mock answers to any attribute, so a renamed method leaves the test green while it asserts a call that no longer exists.
• Inject the clock and the randomness. A function that takes the day as a parameter is pure, its boundary cases are trivial, and it cannot fail at midnight.
• Coverage is a floor, not a target. It reports what executed, not what was checked; a suite that asserts nothing can reach 100%. The real test of a suite is introducing a bug and seeing whether it goes red.

In the next chapter, The GIL, Threads & Processes, we take on the thing Python is most often criticised for and most often misunderstood about: what the global interpreter lock actually locks, why threads still help with I/O and not with computation, what a race condition looks like at the bytecode level, and how to choose between a thread, a process and an event loop.