Testing: Unit, Widget and Golden

"No matter how many instances of white swans we may have observed, this does not justify the conclusion that all swans are white." — Karl Popper, The Logic of Scientific Discovery (1959)

A passing test suite is a pile of white swans. It tells you that the cases you thought of behave as you expected, which is worth a great deal and is not the same as correctness. Holding that distinction is what separates a suite that catches regressions from one that exists to produce a number.

Flutter is unusually well served here, and the reason is architectural rather than cultural. Because the framework draws its own pixels and owns its own event loop, a test can run the entire UI — layout, painting, gestures, animation — inside a plain Dart process with no device, no emulator and no browser. Most toolkits cannot do that, which is why their equivalent tests are slow enough that people skip them.

This chapter measures the three tiers, because the numbers decide how you should use them.

Three tiers, and what each costs

Unit tests exercise plain Dart. No widgets, no framework.

Widget tests mount a real widget tree and drive it. Real layout, real gestures, real animation — but in a headless process on a fake clock.

Golden tests are widget tests that additionally compare the rendered pixels against a stored image.

The gap between the first two is the number that should shape your suite. Measured in one run:

200 unit assertions : 3,056us        (~15us each)
200 widget pumps    : 1,264ms        (~6,322us each)

A widget test costs roughly 420 times what a unit assertion costs. That is still extraordinarily cheap in absolute terms — six milliseconds to build, lay out and paint a screen — but it is not free, and a suite of five thousand widget tests is half a minute of pure framework overhead before any of your logic runs.

The practical consequence is the usual pyramid, arrived at by measurement rather than doctrine: push logic out of widgets so it can be tested at 15 microseconds, and spend widget tests on the thing only they can check — that the wiring produces the right screen.

What a widget test actually is

The thing worth internalising is that a widget test is not a simulation. It mounts the real framework. The same RenderFlex from Part IV lays out the same constraints, the same setState from Part V marks the same elements dirty. What is fake is the environment — and knowing exactly which parts are fake is what stops tests from lying to you.

Measured, the default environment:

surface           2400x1800 physical px
devicePixelRatio  3.0
logical size      800x600

Every widget test runs on an 800×600 logical surface at device pixel ratio 3 unless you say otherwise. That is a phone-ish aspect at a high density, and a layout that only breaks on a narrow screen will not break here — which is why the overflow chapter's advice to test at several sizes matters.

The clock is fake too, and that is the good part.

Running the real thing agrees with the model. A 300-millisecond animation:

finite: settled after 4 frames (9ms of real time)

Four frames, and nine milliseconds of wall-clock time for a three-hundred-millisecond animation. The test did not wait; it advanced a fake clock in steps. This is why widget tests can afford to be numerous, and why a test that calls Future.delayed with a real duration is doing something quite different from one that pumps.

And the repeating case, measured:

repeating -> FlutterError: pumpAndSettle timed out

This is the most common way a Flutter test hangs. pumpAndSettle pumps until no frame is scheduled, and a looping animation — a progress spinner, a pulsing button, a shimmer placeholder — schedules one forever. The fix is never to pumpAndSettle a screen containing an indeterminate indicator; pump a fixed number of frames instead.

Finding things

A widget test locates widgets through Finders, and the choice between them is a small performance decision that adds up. Measured, two thousand lookups of each:

2000 find.text    : 2,367us   (~1.2us each)
2000 find.byType  : 7,859us   (~3.9us each)

find.text is about three times faster than find.byType, because matching a string against text widgets is cheaper than a type check across every element in the tree. Neither is slow, and in a suite of thousands of assertions the difference is real but rarely decisive — the useful rule is to pick the finder that expresses what you mean and reach for find.byKey when a test needs to be unambiguous.

The finders worth knowing:

• find.text('Save') — matches the rendered string. The most readable, and it breaks when copy changes, which is usually correct.
• find.byType(ElevatedButton) — matches a widget class. Good for structure, fragile when a design system swaps components.
• find.byKey(const Key('submit')) — matches a key you placed deliberately. The most stable, and the right choice when a test must survive redesigns.
• find.byWidgetPredicate(...) — an arbitrary test. Use sparingly; it is where unreadable tests come from.

The matchers on the other side of a finder are equally small, and modelling them makes their failure messages predictable:

Golden tests, and the thing nobody warns you about

A golden test renders a widget and compares the pixels to a stored PNG. It is the only tier that catches "this looks wrong" rather than "this behaves wrong", and it is the only tier with a failure mode that surprises everybody the first time.

Here is what a golden of a card containing the text Hello actually looks like:

Five solid black rectangles.

Measured, and the reason is exact:

"Hello" at 20px measures 100.0 x 20.0

Five characters, each exactly twenty pixels wide and twenty tall — a perfect square per glyph. The test environment ships no real fonts, so text is laid out and painted with a placeholder in which every character is a filled em square. Your golden is dimensionally faithful and visually unreadable.

This matters in two directions. It means goldens do catch layout regressions — the boxes move when the layout moves — and it means they cannot catch anything about type: a wrong font, a wrong weight, a broken ligature, an unreadable colour contrast on text. If you need real glyphs you must load real fonts into the test binding, which is what packages like golden_toolkit exist to do.

The second surprise is in the same image: a MaterialApp paints its debug banner, so the corner of every golden carries a red diagonal stripe unless you set debugShowCheckedModeBanner: false.

When a golden fails, the output is genuinely helpful. Measured, after changing one colour:

Golden "goldens/card.png": Pixel test failed, 3.34%, 144240px diff detected.

and four images are written to test/failures/:

card_masterImage.png     what was stored
card_testImage.png       what this run produced
card_isolatedDiff.png    only the differing pixels
card_maskedDiff.png      the diff over the original

That percentage is worth noting: the comparison is exact by default, so any pixel difference fails. A one-pixel shift from a framework upgrade fails the test, which is both the point and the maintenance cost.

Driving the tree

Beyond finding, a widget test interacts: tap, enterText, drag, fling. Each one is followed by a pump, because nothing in Flutter updates until a frame is built — which is the previous Part's setState rule reappearing in a testing context.

The shape of nearly every widget test is the same three steps, and naming them makes the test readable:

// arrange
await tester.pumpWidget(const MaterialApp(home: CounterPage()));

// act
await tester.tap(find.byKey(const Key('increment')));
await tester.pump();            // let the setState-scheduled frame happen

// assert
expect(find.text('1'), findsOneWidget);

The pump() after the tap is the step people forget. Without it the tap has been delivered and the rebuild has been scheduled, but no frame has been built, so the tree still shows the old value and the assertion fails in a way that looks like the handler never ran.

Common Mistakes

1. Forgetting to pump after an interaction. The tap is delivered and the rebuild scheduled, but no frame is built until you pump, so the assertion sees the old tree.

2. Calling pumpAndSettle on a screen with a spinner. A repeating animation schedules frames forever — measured, this produces pumpAndSettle timed out, and in CI it looks like a hang.

3. Treating widget tests as free. Measured at ~6.3ms each against ~15µs for a unit assertion, they are about 420× the cost. Cheap, not free.

4. Testing pure logic through the UI. It works and diagnoses badly: a failure implicates the whole chain rather than naming the defect.

5. Assuming the default surface matches a phone. Every test runs at 800×600 logical, device pixel ratio 3. A layout that only breaks on a narrow screen passes here.

6. Expecting goldens to catch typography. The test environment has no fonts — measured, every glyph is a filled square exactly the font size wide. Goldens check layout and colour, not type.

7. Capturing goldens with the debug banner on. MaterialApp paints it, so it lands in the image. Set debugShowCheckedModeBanner: false.

8. Running --update-goldens without reading the diff. Failure writes four images including an isolated diff; regenerating blind accepts whatever else changed.

9. Leaning on find.byType for everything. It is the most fragile finder under a design-system change, and measured, the slowest of the common three. Use find.byKey where stability matters.

10. Believing a green suite means correct. It means the cases you imagined behaved as expected — Popper's swans. The bugs live in the cases nobody wrote.

Interview Corner

Hands-on

Summary

• Three tiers: unit tests exercise plain Dart, widget tests mount the real framework headlessly, golden tests compare rendered pixels.
• A widget test costs ~420× a unit assertion — measured, ~6.3ms against ~15µs. Cheap in absolute terms, not free in aggregate.
• Widget tests run the real framework on a fake clock. A 300ms animation settles in 4 frames and 9ms of wall-clock time.
• pumpAndSettle never returns for a repeating animation — measured, it raises pumpAndSettle timed out, which in CI looks like a hang.
• The default surface is 800×600 logical at device pixel ratio 3, so narrow-screen regressions do not appear unless you resize.
• Pump after every interaction. A tap schedules a rebuild; nothing changes until a frame is built.
• find.text is about 3× faster than find.byType, and find.byKey is the most stable under redesign.
• Goldens are blind to typography. With no fonts loaded, every glyph paints as a filled square exactly the font size wide — measured, Hello at 20px is 100×20.
• Disable the debug banner before capturing, or it appears in every image.
• Golden comparison is exact; a failure reports a percentage and writes master, test, isolated-diff and masked-diff images.

Next: Performance and the Frame Budget — the 16 milliseconds a frame actually has, where they go, and how to find what spent them.