Testing When the Compiler Helped

"test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out" — cargo test, on a crate that has no tests

A green test run is not evidence that a program works. It is evidence that the tests you wrote did not fail, and the epigraph is what that looks like when you wrote none. Every test suite is somewhere on the line between those two readings, and the useful question is never "do we have tests" but "which mistakes would these have caught".

In Rust that question has an unusual preamble, because the compiler has already caught some of them. There is no test for a null dereference, because there are no nulls. No test for use-after-free, for a forgotten break in a switch, for an integer silently becoming a string, or — as Part V established — for a data race. Those are not tests you pass; they are tests you cannot write, because the programs that would fail them do not compile.

That changes what a test suite is for. In a language where the type system is weak, a large fraction of tests exist to re-check things the compiler could have checked, and their value is real because nothing else was doing it. Port that habit to Rust and you write a lot of tests that can only pass, which costs time to write, time to run, and attention in review, while catching nothing.

What remains is more interesting and harder: logic, boundaries, and decisions. Whether the discount is applied before or after tax. Whether 254 characters is allowed and 255 is not. Whether a retry waits between attempts. Whether an endpoint returns 404 or 500 when a row is missing. The compiler has no opinion about any of those, and never will, because they are not properties of the types — they are claims about what the program is supposed to do.

Three kinds of test

What cargo test actually runs, and it is more than you may think.

//! The three kinds of test cargo runs, and what each one is for.

/// Doubles a number.
///
/// ```
/// assert_eq!(demo::double(21), 42);
/// ```
pub fn double(n: i32) -> i32 { n * 2 }

pub fn classify(n: i32) -> &'static str {
    if n < 0 { "negative" } else if n == 0 { "zero" } else { "positive" }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn classifies_sign() {
        assert_eq!(classify(-1), "negative");
        assert_eq!(classify(0), "zero");
        assert_eq!(classify(1), "positive");
    }

    #[test]
    #[should_panic(expected = "divide by zero")]
    fn panics_on_zero() {
        panic!("divide by zero");
    }
}

running 2 tests
test tests::classifies_sign ... ok
test tests::panics_on_zero - should panic ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in Xs


running 1 test
test src/lib.rs - double (line 6) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in Xs

Two sections of output from one command. The first is the unit tests: functions marked #[test] inside a #[cfg(test)] mod tests, which means the module is compiled only when testing and costs nothing in the shipped binary. Because it is an inner module, use super::* gives it access to everything in the file including private items — so Rust's unit tests can test internals without you weakening the visibility of the thing being tested.

#[should_panic(expected = "...")] asserts a failure path, and the expected string matters: without it the test passes if the function panics for any reason, including a reason you did not intend, which is how a should_panic test ends up passing because of a typo.

The second section is the one people miss. Doc tests. The example in the /// comment above double was compiled and executed — test src/lib.rs - double (line 6) ... ok. It is a real test, it runs in CI, and it fails the build when it stops being true.

That is a better deal than it first appears. The hardest thing about documentation is that it rots silently: the code changes, the example in the comment does not, and nobody notices until a reader follows it and it does not work. In Rust, that example is a test. An out-of-date example is a build failure, which means the documentation on docs.rs for any published crate is known to compile against the version you are reading.

One caveat worth knowing now: doc tests run only for library targets. A binary-only crate — one with just src/main.rs — silently runs none of them, which is a surprising amount of documentation quietly not being checked.

There is a third kind not shown here. Files under a tests/ directory at the crate root are compiled as separate crates, each linking your library the way an outside user would, which means they can only reach your public API. That constraint is the whole value: a unit test sitting inside the module can quietly lean on a private helper and tell you nothing about whether the public surface is usable. If an integration test is awkward to write, that is a finding about your API rather than about testing, and it is the cheapest design review available — which is also the practical argument for a binary crate keeping almost everything in lib.rs and leaving main.rs as a thin shell, since a binary has no public API to test against at all.

The tests you no longer write

The compiler as a member of the test suite.

#[derive(Debug)]
enum Payment { Card, Transfer, Cash }

fn fee(p: &Payment) -> u32 {
    match p {
        Payment::Card => 30,
        Payment::Transfer => 10,
    }
}

fn main() { println!("{}", fee(&Payment::Cash)); }

That does not compile:

error[E0004]: non-exhaustive patterns: &Payment::Cash not covered

Consider what that replaces. In a language with a non-exhaustive switch, adding a payment method is a defect waiting to be found: the new variant falls through to a default, the fee is silently zero, and the way you learn about it is a test somebody remembered to write, or a customer. The usual defensive practice is a test per enum variant asserting that each is handled — a test that exists purely because the compiler is not checking.

Here it is a build error, at every match in the codebase, the moment you add the variant. Adding Cash tells you everywhere that now needs a decision. That is not merely a test you do not have to write; it is better than the test, because a test only covers the call sites somebody thought of.

The same logic removes a family of others. You do not test that a String is not null. You do not test that a function returning Result has its error path considered, because ignoring it is a warning and using the value requires unwrapping. You do not test, as Part II showed, that a freed value is not used afterwards.

The practical version of this principle is push checks into types, then delete the tests they replace. A function taking Email instead of String does not need a validity test at each call site — it needs one test of Email::parse, and every other use is checked by the signature. That is the next block.

The honest caveat: this works for properties expressible in the type system, and most interesting properties are not. The compiler will never know that a 10% discount means pence / 10.

What is left: boundaries and decisions

The tests that earn their place.

//! What is left to test once the types have taken their share.
use std::error::Error;
use std::fmt;

/// A validated email. The ONLY way to make one is `parse`, so every
/// `Email` in the program is one that passed this check exactly once.
#[derive(Debug, PartialEq)]
pub struct Email(String);

#[derive(Debug, PartialEq)]
pub enum EmailError { Empty, NoAt, TooLong }

impl fmt::Display for EmailError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            EmailError::Empty => write!(f, "empty"),
            EmailError::NoAt => write!(f, "missing @"),
            EmailError::TooLong => write!(f, "too long"),
        }
    }
}
impl Error for EmailError {}

impl Email {
    pub fn parse(raw: &str) -> Result<Self, EmailError> {
        let raw = raw.trim();
        if raw.is_empty() { return Err(EmailError::Empty); }
        if raw.len() > 254 { return Err(EmailError::TooLong); }
        if !raw.contains('@') { return Err(EmailError::NoAt); }
        Ok(Email(raw.to_string()))
    }
    pub fn as_str(&self) -> &str { &self.0 }
}

#[cfg(test)]
mod tests {
    use super::*;

    // Table-driven: the cases are data, so adding one is one line.
    #[test]
    fn rejects_bad_input() {
        let cases = [
            ("", EmailError::Empty),
            ("   ", EmailError::Empty),
            ("ada.example.com", EmailError::NoAt),
        ];
        for (input, want) in cases {
            assert_eq!(Email::parse(input), Err(want), "input was {input:?}");
        }
    }

    // The boundary, both sides of it. 254 is allowed, 255 is not.
    #[test]
    fn length_boundary() {
        let at_limit = format!("{}@x.com", "a".repeat(254 - 6));
        assert_eq!(at_limit.len(), 254);
        assert!(Email::parse(&at_limit).is_ok());

        let over = format!("{}@x.com", "a".repeat(255 - 6));
        assert_eq!(over.len(), 255);
        assert_eq!(Email::parse(&over), Err(EmailError::TooLong));
    }

    // A test can return Result, so `?` works and a failure is a failure.
    #[test]
    fn trims_before_validating() -> Result<(), Box<dyn Error>> {
        let e = Email::parse("  ada@example.com  ")?;
        assert_eq!(e.as_str(), "ada@example.com");
        Ok(())
    }
}

running 3 tests
test tests::length_boundary ... ok
test tests::rejects_bad_input ... ok
test tests::trims_before_validating ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in Xs


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in Xs

Email has a private field and one constructor, so an invalid Email cannot exist. That is the "parse, don't validate" idea from Part III applied to testing: the validity check happens once, at the boundary, and the rest of the program is checked by the signature. A function taking Email needs no test for what it does with a malformed address, because it cannot be given one.

So the tests concentrate on the constructor, and on the three things the compiler cannot know.

The cases are data. The table in rejects_bad_input makes adding a case one line rather than one function, and the "input was {input:?}" message is what turns a failure from assertion failed: left != right into something that names the offending input. Without it, a table-driven test tells you that one of five cases failed and not which.

The boundary is tested on both sides. Not 100 characters and 1000, but 254 and 255 — the two values where the behaviour changes. Off-by-one errors live exactly there and nowhere else, and a test at 100 and 1000 passes whether the comparison is > or >=.

A test can return Result. -> Result<(), Box<dyn Error>> lets the body use ?, so a test that merely needs a value to be Ok does not need .unwrap() everywhere — and an Err fails the test with the error rather than a panic message about unwrapping.

The second output section is running 0 tests: that is the doc-test pass finding no examples in this file. It is a standing, mildly uncomfortable reminder of documentation not written.

Testing time without spending it

Async, and the clock as an input.

//! Testing code that awaits, and code that depends on a clock.
use std::time::Duration;

/// Retries a fallible operation with a fixed delay, up to `attempts` times.
pub async fn retry<F, Fut, T, E>(attempts: u32, delay: Duration, mut op: F) -> Result<T, E>
where
    F: FnMut(u32) -> Fut,
    Fut: std::future::Future<Output = Result<T, E>>,
{
    let mut last_err = None;
    for attempt in 1..=attempts {
        match op(attempt).await {
            Ok(v) => return Ok(v),
            Err(e) => {
                last_err = Some(e);
                if attempt < attempts {
                    tokio::time::sleep(delay).await;
                }
            }
        }
    }
    Err(last_err.expect("attempts must be at least 1"))
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::sync::atomic::{AtomicU32, Ordering};
    use std::sync::Arc;

    // #[tokio::test] builds a runtime for this one test.
    #[tokio::test]
    async fn succeeds_without_retrying() {
        let calls = Arc::new(AtomicU32::new(0));
        let c = Arc::clone(&calls);
        let out: Result<&str, ()> = retry(3, Duration::from_millis(1), |_| {
            let c = Arc::clone(&c);
            async move { c.fetch_add(1, Ordering::SeqCst); Ok("done") }
        }).await;
        assert_eq!(out, Ok("done"));
        assert_eq!(calls.load(Ordering::SeqCst), 1, "should not retry on success");
    }

    #[tokio::test]
    async fn retries_then_gives_up() {
        let calls = Arc::new(AtomicU32::new(0));
        let c = Arc::clone(&calls);
        let out: Result<(), &str> = retry(3, Duration::from_millis(1), |_| {
            let c = Arc::clone(&c);
            async move { c.fetch_add(1, Ordering::SeqCst); Err("nope") }
        }).await;
        assert_eq!(out, Err("nope"));
        assert_eq!(calls.load(Ordering::SeqCst), 3);
    }

    // start_paused makes the clock a test input: this asserts a two-second
    // wait happened without waiting two seconds.
    #[tokio::test(start_paused = true)]
    async fn waits_between_attempts_without_real_delay() {
        let began = tokio::time::Instant::now();
        let out: Result<(), &str> = retry(3, Duration::from_secs(1), |_| async { Err("nope") }).await;
        assert_eq!(out, Err("nope"));
        // Two sleeps between three attempts.
        assert_eq!(began.elapsed(), Duration::from_secs(2));
    }
}

running 3 tests
test tests::retries_then_gives_up ... ok
test tests::succeeds_without_retrying ... ok
test tests::waits_between_attempts_without_real_delay ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in Xs


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in Xs

#[tokio::test] replaces #[test] and builds a runtime for that test, so the body can be async and .await. It is the testing counterpart of #[tokio::main], and it is per-test: each one gets its own runtime, so tests do not share scheduler state.

The third test is the one worth stealing. #[tokio::test(start_paused = true)] pauses the clock, and tokio::time::sleep then advances it instantly when there is nothing else to run. So the test asserts that the retry logic waited two seconds — began.elapsed() is exactly Duration::from_secs(2) — and the whole suite finished in a fraction of a second.

That is a genuinely important technique, because the usual alternatives are both bad. Testing a timeout with a real sleep makes the suite slow, and slow suites get run less. Testing it by reaching in and shortening the delay to a millisecond tests different code than you ship, and is the reason "it worked in the test" is so often said about timing bugs. A virtual clock lets the production constants stay in the test.

The counter in the first two tests is the other half: asserting how many times the operation was called. That succeeds_without_retrying checks calls == 1 is the whole point of the test — a retry wrapper that retried a success would still return Ok("done") and pass any test that only looked at the return value.

A web handler with no web

Testing the whole stack from Part VI, without a port.

//! Testing a web handler without a port, a client, or a running server.
use axum::extract::{Path, State};
use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::{Json, Router};
use std::collections::HashMap;
use std::sync::{Arc, RwLock};

#[derive(Clone)]
pub struct AppState { pub users: Arc<RwLock<HashMap<i64, String>>> }

pub struct NotFound(i64);
impl IntoResponse for NotFound {
    fn into_response(self) -> Response {
        (StatusCode::NOT_FOUND, Json(serde_json::json!({ "error": format!("no user {}", self.0) }))).into_response()
    }
}

pub async fn show(State(st): State<AppState>, Path(id): Path<i64>) -> Result<String, NotFound> {
    st.users.read().unwrap().get(&id).cloned().ok_or(NotFound(id))
}

/// The router is a value, so a test can build its own.
pub fn app(state: AppState) -> Router {
    Router::new().route("/users/{id}", get(show)).with_state(state)
}

#[cfg(test)]
mod tests {
    use super::*;
    use axum::body::Body;
    use axum::http::Request;
    use tower::ServiceExt; // brings `oneshot` into scope

    fn state_with(pairs: &[(i64, &str)]) -> AppState {
        let m: HashMap<i64, String> =
            pairs.iter().map(|(k, v)| (*k, v.to_string())).collect();
        AppState { users: Arc::new(RwLock::new(m)) }
    }

    async fn get(app: Router, uri: &str) -> (StatusCode, String) {
        let res = app
            .oneshot(Request::builder().uri(uri).body(Body::empty()).unwrap())
            .await
            .unwrap();
        let status = res.status();
        let bytes = axum::body::to_bytes(res.into_body(), 64 * 1024).await.unwrap();
        (status, String::from_utf8(bytes.to_vec()).unwrap())
    }

    #[tokio::test]
    async fn returns_the_user() {
        let (status, body) = get(app(state_with(&[(1, "ada")])), "/users/1").await;
        assert_eq!(status, StatusCode::OK);
        assert_eq!(body, "ada");
    }

    #[tokio::test]
    async fn missing_user_is_404_in_our_shape() {
        let (status, body) = get(app(state_with(&[])), "/users/7").await;
        assert_eq!(status, StatusCode::NOT_FOUND);
        assert_eq!(body, r#"{"error":"no user 7"}"#);
    }

    // The rejection path too — this never reaches `show`.
    #[tokio::test]
    async fn unparseable_id_is_400() {
        let (status, _) = get(app(state_with(&[])), "/users/abc").await;
        assert_eq!(status, StatusCode::BAD_REQUEST);
    }

    #[tokio::test]
    async fn unknown_route_is_404() {
        let (status, _) = get(app(state_with(&[])), "/nope").await;
        assert_eq!(status, StatusCode::NOT_FOUND);
    }
}

running 4 tests
test tests::missing_user_is_404_in_our_shape ... ok
test tests::returns_the_user ... ok
test tests::unknown_route_is_404 ... ok
test tests::unparseable_id_is_400 ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in Xs


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in Xs

Four tests through the entire request path — routing, extraction, state, handler, error type, response serialisation — with no socket, no port, and no server task. ServiceExt::oneshot takes a Request and returns the Response, because a Router is just a tower::Service and calling it is an ordinary function call.

The design detail that makes it possible is fn app(state) -> Router. Because the router is a value returned by a function rather than something assembled inside main, a test constructs one with whatever state it likes. That is the same benefit chapter three claimed for State as an extractor, arriving where it pays off: state_with(&[(1, "ada")]) is a fixture, and there is no global to reset between tests.

Two of these are worth noticing because they test code you did not write. unparseable_id_is_400 never reaches show — the Path<i64> rejection happens first — and unknown_route_is_404 never reaches a handler at all. Those are exactly the responses the previous chapter warned are part of your contract and easy to forget, and here they are pinned by tests that will fail if a careless refactor changes them.

What this does not test is the socket layer: axum::serve, TLS, timeouts, and whatever your proxy does. Those need a real server, and the right amount of that is a small number of smoke tests, because they are slow and flaky in proportion to how much of the world they involve.

Common Mistakes

1. Porting defensive tests from weaker languages. Tests for nulls, type confusion or unhandled enum variants can only pass in Rust; they cost time and catch nothing.

2. Writing #[should_panic] without expected. The test then passes if the code panics for any reason at all, including a reason you did not intend.

3. Testing at round numbers. Off-by-one errors live at the boundary, so test 254 and 255, not 100 and 1000.

4. Omitting the message in a table-driven test. Without "input was {input:?}" a failure tells you one case broke and not which.

5. Forgetting doc tests are tests. They run in CI and fail the build, which is the mechanism that keeps examples from rotting — and they do not run at all for a binary-only crate.

6. Using .unwrap() throughout a test. Return Result<(), Box<dyn Error>> and use ?, so a failure reports the error instead of a panic about unwrapping.

7. Sleeping in tests. #[tokio::test(start_paused = true)] advances a virtual clock, so you can assert a two-second wait without waiting.

8. Shortening production timeouts to make tests fast. You are then testing different code than you ship, which is why timing bugs survive green suites.

9. Asserting only on return values. A retry wrapper that retries a success still returns the right value; counting calls is what catches it.

10. Spinning up a server to test a handler. A Router is a tower::Service, so oneshot exercises the whole request path as a function call.

Interview Corner

//! A discount calculator with thin tests. Your job is to thicken them.

#[derive(Debug, PartialEq)]
pub enum Tier { Standard, Member, Staff }

/// Returns the price in pence after the tier discount, never below zero.
pub fn price_after_discount(pence: u32, tier: &Tier) -> u32 {
    let off = match tier {
        Tier::Standard => 0,
        Tier::Member => pence / 10,
        Tier::Staff => pence / 2,
    };
    pence.saturating_sub(off)
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn applies_each_tier() {
        assert_eq!(price_after_discount(1000, &Tier::Standard), 1000);
        assert_eq!(price_after_discount(1000, &Tier::Member), 900);
        assert_eq!(price_after_discount(1000, &Tier::Staff), 500);
    }

    // TODO 1: add a doc example on price_after_discount and watch it appear
    //         as a third test. Why is a doc test worth more than the same
    //         assertion written here?

    // TODO 2: this passes for 1000 pence. Find the inputs where integer
    //         division makes the result surprising, and write the test that
    //         pins down the behaviour you decide is correct.

    // TODO 3: add a Tier::Trial variant and do NOT touch this module.
    //         Which of your tests fails, and which does the compiler catch
    //         first? What does that tell you about where to spend effort?

    // TODO 4: rewrite applies_each_tier as a table-driven test over
    //         (pence, tier, expected). Say what that buys and what it costs
    //         when one case fails.
}

running 1 test
test tests::applies_each_tier ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in Xs


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in Xs

Summary

• A green suite is evidence the tests did not fail, nothing more — the epigraph is what that looks like with no tests at all.
• cargo test runs three kinds: unit tests in #[cfg(test)] modules with access to private items, integration tests in tests/, and doc tests from /// examples.
• Doc tests make documentation rot a build failure, which is why published examples compile — but they do not run for a binary-only crate.
• The compiler is part of the suite. A non-exhaustive match is E0004, so adding an enum variant reports every site needing a decision, better than a test per variant.
• Push checks into types, then delete the tests they replace: a newtype with a private field and one constructor needs one test, not one per call site.
• Test boundaries, not round numbers — 254 and 255, where the behaviour changes and where off-by-one errors live.
• Table-driven tests need a failure message naming the case, or a failure tells you only that one of them broke.
• Tests can return Result, so ? replaces .unwrap() and failures report the error.
• A paused clock makes time an input: #[tokio::test(start_paused = true)] asserts a two-second wait in milliseconds, with production constants intact.
• A Router is a tower::Service, so oneshot tests routing, extraction, state, handler and serialisation with no socket.

In the next chapter, Builds, Features and Reproducibility, we turn to the build itself: what Cargo.lock does and does not promise, how features change what compiles, and why "it builds on my machine" is a statement about your machine.