Lesson 1 of 5

Testing Django & FastAPI

"I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence." — Kent Beck, answering "How deep are your unit tests?" (2008)

Volume 14 covered pytest as a tool. This chapter is about what changes when the thing under test is a web application, and the change is larger than it first appears — because almost everything interesting about a web application happens between its components rather than inside them.

Consider what a single endpoint actually does. It matches a URL, runs middleware, resolves dependencies or authentication, validates a payload, hits a database through an ORM, applies permissions, serialises a response, and sets a status code. A unit test of the handler function, with all of that mocked away, verifies the one part that was never in doubt. Meanwhile the bugs live in the seams: the permission that was not applied, the serializer that dropped a field, the query that returned another tenant's row, the migration that had not run.

So the pyramid tilts. In web code, the tests with the best ratio of confidence to cost are usually the ones that go through the framework's test client — routing, middleware and serialization included — against a real database. They are not "unit tests" by any strict definition and it does not matter. What does matter is the thing that makes them expensive: database setup dominates the runtime of a web test suite, and understanding that one fact is worth more than any argument about test taxonomy.

The big question

If a test that mocks the database, the middleware and the serializer runs in a millisecond and catches almost none of your real bugs, and a test that exercises all three catches most of them — what stops you from writing only the second kind, and what does it cost when you do?

Where the confidence actually comes from

Start by pricing the levels honestly, in the terms that decide a real suite: how long it takes, and what it catches.

Python — Try it Yourself

Loading editor…

Output

Click Run to execute the code.

The middle row is the one worth arguing about. A view test with the ORM, the serializer and the permission class all mocked verifies that the function calls the things you told it to call. It cannot catch a missing permission, a dropped field or a wrong status, because those all live in the parts that were replaced.

The endpoint test through the framework's client catches all of it, at roughly forty times the cost. That ratio is the whole design problem of a web test suite, and it is why the next section is about the database rather than about testing philosophy.

What a test client does and does not exercise

Both frameworks ship a client that dispatches into the application without a network. Knowing exactly where it enters and exits tells you what a green test proves.

Python — Try it Yourself

Loading editor…

Output

Click Run to execute the code.

The exercised layers are the reason a test-client test is worth its cost: routing, middleware, auth, validation, the ORM and serialization all run for real, so the seams where bugs live are covered.

The misses are equally instructive, and they share a shape. Worker timeouts, blocking calls, proxy behaviour, static file serving, connection pool exhaustion — all properties of the deployment rather than the application. No amount of test-client coverage finds them, which is why a small number of smoke tests against a real environment is a different tool rather than a redundant one.

Everyday analogy

Testing a restaurant. You can taste a sauce in isolation — that is a unit test, fast and precise, and it tells you nothing about whether the order for table nine arrives hot with the right sides. You can send a fake order through the kitchen from the pass and inspect what comes out — that is the test client: the full sequence of ticket, prep, plating and check, minus the customer, the waiter and the room. It catches the things that go wrong between stations, which is where most of them go wrong. What it cannot catch is what happens on a Friday at eight: the kitchen printer jams, two stations are waiting on the same pan, service backs up. Those only appear with real load in the real room, and no amount of tasting sauces or firing test tickets substitutes. But notice the ratio — you fire test tickets constantly and run a full Friday service occasionally, because one is cheap and repeatable and the other is expensive and cannot be run on demand. That is the shape of a good web test suite too.

The database is your runtime

Almost all the cost of a web test suite is creating and destroying data. The strategy you choose changes the number by an order of magnitude.

Python — Try it Yourself

Loading editor…

Output

Click Run to execute the code.

Transaction rollback is the default for a good reason: it is roughly twenty times faster than truncating and it scales to suites that would otherwise be unusable. Each test opens a transaction, does its work, and rolls back, so the database is never actually modified.

The exceptions matter because they are silent. Code that uses transaction.on_commit() — sending an email after a successful save, enqueuing a job — has its hook never fire inside a rollback-based test, so the test passes while asserting nothing about the behaviour you care about. That is a false green rather than a slow test, and it is the reason to know which class you are inheriting from.

A little history

Django shipped a test framework in its first public release in 2005, built on unittest, with a test client and a database created fresh per run — the assumption being that a test suite was small enough for that to be fine. It was not fine for long. Transaction-based rollback arrived in Django 1.1 (2009) as TestCase, with the older truncating behaviour preserved as TransactionTestCase, and that split still confuses people because the names suggest the opposite of what they do: TestCase is the one that uses a transaction to avoid committing, and TransactionTestCase is the one that lets transactions actually happen. Meanwhile pytest (2004, out of the PyPy project) grew a plugin ecosystem that eventually displaced unittest for most Python teams, and pytest-django added the piece that mattered most operationally: --reuse-db, which skips schema creation between runs and turns a thirty-second startup into nothing. On the other side, FastAPI inherited Starlette's TestClient, itself built on requests and later httpx, which made "call your ASGI app like an HTTP client, in-process" the normal way to test — and app.dependency_overrides, from Part IV, gave it the substitution seam that Django achieves with mock.patch and settings overrides. The convergence is notable: two ecosystems with very different philosophies both concluded that the right default unit of a web test is a request.

Tests that pass alone and fail together

The other tax on a web suite is flakiness, and almost all of it comes from state that outlives a test.

Python — Try it Yourself

Loading editor…

Output

Click Run to execute the code.

The failure has a signature worth recognising: a test that fails without having been changed, because another test ran first. In web applications the three reliable sources are module-level caches, an app.dependency_overrides entry that was never cleared, and Django's global settings or cache.

Randomising the order is the diagnostic. A suite that passes only in file order has ordering dependencies whether or not you have noticed them yet, and the failure will eventually arrive when someone adds a test in the middle.

The engineer's notebook

Practices with the best return in a web suite. Default to endpoint tests through the framework's client, and reserve mocked view tests for logic that genuinely has no I/O — the mocked ones cost little and prove little. Use transaction rollback, and know when you cannot: transaction.on_commit hooks never fire inside one, so a test asserting that an email was queued passes vacuously unless it uses TransactionTestCase or captureOnCommitCallbacks. Turn on --reuse-db, then parallelise with one database per worker; past a few hundred tests you are waiting on CREATE TABLE, not on your code. Build objects with factories, and build only what you assert on — a factory that creates a customer, an address, three orders and a payment for a test about one field is where suites silently get slow. Clear app.dependency_overrides in teardown, every time, because it is global to the app object. Freeze time and seed randomness rather than asserting on approximations. Stub outbound HTTP at the transport — responses, respx — and fail loudly on an unstubbed call, so a test that would have hit a real API in CI breaks instead of flaking. Assert query counts on list endpoints with assertNumQueries, which is the cheapest N+1 regression test there is. And run in random order, so ordering dependencies surface as a red test rather than a mystery.

Stubbing the outside, and the drift it causes

Any test that would make a real network call is not a test — it is a flake with an occasional useful result. Stubbing is mandatory, and it introduces its own failure mode.

Python — Try it Yourself

Loading editor…

Output

Click Run to execute the code.

The unstubbed-call assertion is the part people skip. A stub library configured to pass through to the real network turns a missing stub into a test that occasionally works, occasionally fails, and quietly charges you money in CI. Failing loudly is the correct default.

Contract drift is the deeper issue. A stub encodes your belief about somebody else's API at the moment you wrote it, and there is nothing in a green suite that revalidates that belief. The cheapest countermeasure is to check the recorded shape against the provider's published schema in CI; the most reliable is a scheduled contract test that makes exactly one real call and compares the shape rather than the values.

In the real world

The suite that becomes an obstacle rather than an asset usually gets there the same way. It starts fast. Somebody writes a factory that creates a fully populated customer — address, payment method, three historical orders — because it makes writing tests convenient, and it does. A year later that factory is used by four hundred tests, most of which assert on one field, and every one of them inserts a dozen rows. The suite takes eleven minutes. Nobody runs it locally any more, so it only fails in CI, which means the feedback loop is now twenty minutes instead of two. Then it starts flaking, because with four hundred tests something eventually depends on ordering, and the response is to retry failed jobs automatically — which hides the flake and also hides real failures. At that point the suite has stopped being a safety net and become a toll: it costs twenty minutes and its verdict is not fully trusted. What is striking is that no individual decision was wrong. The convenient factory was convenient, the retry was pragmatic. The compounding was invisible because nobody measured where the time went — and measuring it is remarkably easy, since both pytest's --durations and a per-test query count will point at the same handful of causes within an afternoon.

Common Mistakes

Unit-testing views with everything mocked. The permission class, the ORM and the serializer are exactly where the bugs are, so replacing them leaves you asserting that the function calls what you told it to call.
Not knowing which database strategy you are on. Rollback is roughly twenty times faster than truncation and seven hundred times faster than recreating, and the choice is often made by which base class somebody copied.
Testing on_commit behaviour under rollback. The hook never fires, so the test passes without exercising anything — a false green rather than a slow test.
Factories that build more than the test needs. A fully populated object graph per test is where a suite quietly becomes an eleven-minute suite.
Leaving app.dependency_overrides set. It is global to the app object, so one test's override changes another test's behaviour and the failure lands somewhere unrelated.
Running tests only in file order. Ordering dependencies exist whether or not you have found them; randomising is what turns them into a red test instead of a mystery.
Asserting on real time or unseeded randomness. Freeze the clock and seed the generator; otherwise the suite fails at midnight, in another timezone, or one run in fifty.
Stubs that fall through to the network. A missing stub then becomes a flaky test that also makes real calls from CI. Fail loudly on anything unstubbed.
Trusting a stub to stay accurate. It is a copy of somebody else's contract taken at one moment, and a green suite says nothing about whether that contract has changed.
Retrying failed CI jobs by default. It hides flakes and real failures equally, and it converts a suite people trust into a suite people wait for.

Interview Corner

Interview Corner — the same idea at three levels. Answer yours; recognise the one above it.

Junior

How would you test an API endpoint?

Model answer — I would use the framework's test client — Django's Client or DRF's APIClient, or FastAPI's TestClient — which sends a request through the application in-process without opening a socket. That means routing, middleware, authentication, validation, the database query and serialization all run for real, so the test covers the parts where bugs usually are rather than just the handler body. For a single endpoint I would write a few tests: the happy path asserting the status code and the response shape, validation failures asserting a 4xx and which field was rejected, and the authorization cases — including the denials, so an unauthenticated request and a request for somebody else's record both get refused. The denials are the ones that matter most, because they are what breaks when a permission check gets refactored. I would use a real test database rather than mocking the ORM, since mocking it means the query is never actually run. And I would avoid asserting things the framework guarantees, like that a filter filters — those tests cost time and never fail for a reason you care about.

Mid

Your test suite takes fifteen minutes and is occasionally flaky. Where do you look?

Model answer — For the time, the database, because that is where almost all of it goes in a web suite. First, which isolation strategy is in use: transaction rollback is about twenty times faster than truncating tables, so a suite using Django's TransactionTestCase everywhere — often copied without knowing what it does — is paying a large multiple for nothing. Second, whether the schema is recreated every run, which --reuse-db removes. Third, the factories: a factory that builds a customer with an address and three orders for a test that asserts on one field inserts a dozen rows every time, and pytest's --durations will point at that within minutes. Then parallelise, which needs a database per worker. For the flakiness, I would run the suite in a random order first, because most flakes are ordering dependencies — state that outlives a test. In a web application that is usually a module-level cache, Django settings or the cache backend, or app.dependency_overrides left set, which is global to the app object so one test changes another's behaviour and the failure appears somewhere unrelated. After that I would look for real time and unseeded randomness, and for outbound HTTP calls that are not stubbed, since a stub library that falls through to the network gives you a test that works most of the time. What I would not do is enable automatic retries on failed jobs, because that hides real failures alongside the flakes and costs you the suite's credibility.

Senior

How do you decide what to test, and how much?

Model answer — By what a failure would cost and what a test would cost to keep. I think in terms of confidence per unit of maintenance rather than coverage percentage, because coverage rewards testing the code that is easiest to test, which is rarely the code that is riskiest. In a web application the highest-value tests are usually endpoint tests through the test client, because the interesting behaviour is in the seams — permissions, serialization, the query, the status code — and a mocked view test replaces all of those with the thing you already believed. Below that, pure logic worth testing exhaustively: policy functions, money arithmetic, anything with branches and no I/O. Above it, a handful of smoke tests against a real deployment, because worker timeouts, proxy behaviour, static files and connection pools are properties of the deployment that no in-process client can see. I would deliberately not test framework behaviour, and I would deliberately over-test authorization denials, since those are the assertions that regress silently. On the maintenance side, the thing I watch is whether a test fails when behaviour changes or when implementation changes — a test coupled to implementation gets deleted the first time it is inconvenient, so it was never really protecting anything. And I would treat the suite's runtime and flake rate as product metrics, because a suite that takes twenty minutes and is retried on failure has stopped being a safety net; people route around it, and at that point the tests exist without doing their job.

Company lens — how the same topic gets asked

micro1Practical: testing an endpoint through the framework's own client.

GoogleDepth: transaction rollback versus truncation, and what each breaks.

MetaDesign: what to test at each level, and what not to test at all.

AmazonReality: a suite retried on failure until nobody trusted its verdict.

Hands-on

Build the runner. run_suite must execute tests in the order given, resetting shared state before each one and recording pass, fail or error — an exception is not a failure, and conflating them hides the difference between a wrong answer and a broken test. shuffled must produce a deterministic reordering so an ordering dependency shows up as a red test rather than a mystery. stub_get must refuse an unstubbed URL rather than falling through. report must summarise. The checks that matter are the ones about isolation: the same tests must pass in any order once the reset is in place, and fail without it. The checklist reports each requirement as PASS, FAIL or ERR; run it before writing anything and watch the lines flip.

Python — Try it Yourself

Loading editor…

Output

Click Run to execute the code.

Summary

Web bugs live in the seams, so a view test with the ORM, the serializer and the permission class mocked verifies the one part that was never in doubt.
The framework's test client is the best value in a web suite. It exercises routing, middleware, auth, validation, the ORM and serialization in-process, at roughly forty times the cost of a unit test.
What it cannot see are properties of the deployment — worker timeouts, blocking calls, proxies, static files, connection pools — which is what a handful of smoke tests against a real environment is for.
Database setup is the runtime. Transaction rollback is about twenty times faster than truncation and seven hundred times faster than recreating the schema.
Rollback silently breaks on_commit tests. The hook never fires, so the assertion passes without exercising anything — a false green, not a slow test.
--reuse-db plus one database per parallel worker is what stops you waiting on CREATE TABLE rather than on your tests.
Build only what the test asserts on. A convenient factory reused four hundred times is how a suite becomes an eleven-minute suite without any single bad decision.
Run in random order. Ordering dependencies exist whether or not you have found them, and the usual sources are module-level caches, Django settings, and dependency_overrides left set.
Fail loudly on unstubbed outbound calls. A stub that falls through to the network is a flaky test that also spends money in CI.
A stub is a copy of somebody else's contract, and a green suite says nothing about whether that contract has changed — schema-check the recording, or run one real call on a schedule.

Testing tells you the application is correct; the next question is whether it is fast enough, and the first tool everyone reaches for is also the one most likely to introduce subtle wrongness. The next chapter, Caching & Invalidation, covers where a cache can live, what each layer can and cannot know, why the hard part is never storing the value but deciding when it stopped being true, and the specific failure modes — stampedes, stale writes and keys that outlive their schema — that turn a performance fix into an incident.

Editorial reflection

The received wisdom about the test pyramid was formed largely by people testing systems where the units genuinely were the interesting part — libraries, compilers, algorithms. Web applications are not shaped like that. A Django view is mostly glue: it composes a permission, a queryset and a serializer, and the composition is the behaviour. Testing the glue with everything it glues replaced by mocks is a coherent activity that answers a question nobody asked. I think the honest version of the advice is Kent Beck's, in the epigraph: test as little as possible to reach a given level of confidence — with the emphasis on a given level, because that level is a business decision and it differs by endpoint. The authorization check on a billing endpoint deserves exhaustive denial tests; a preferences form does not. What I find worth watching is the way a test suite's credibility behaves, because it is more fragile than its coverage. A suite that is slow gets skipped locally; a suite that flakes gets retried; a suite that is retried stops being evidence. Each of those is a small operational decision made under deadline, and together they can hollow out a suite entirely while every metric still looks fine — coverage unchanged, tests still green, and nobody actually believing them. That is a failure mode worth monitoring directly, and the two numbers that reveal it are runtime and flake rate, neither of which appears on a coverage report.

Enrol to track progress

Q&A · 0

Enrol to ask questions and join the discussion.

No questions yet — be the first to ask.