Lesson 1 of 5
Architecting an AI Feature
"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." — Antoine de Saint-Exupéry, Terre des Hommes (1939)
Six parts of this book have been about capabilities. This one is about assembly, and the first thing to say is that assembling an AI feature well means using the model for considerably less than you expected to.
That is not a stylistic preference. It follows from everything measured so far. A model call is the slowest step in your pipeline, the only one that costs money per invocation, the only one that returns something different each time, the only one that cannot be unit tested in the ordinary way, and the only one whose behaviour can change because a provider shipped something on a Tuesday. Every one of those properties is a good reason to route work around it. So the design question is not "how do I build this with AI" but which specific parts of this genuinely require judgement, and the answer is almost always fewer parts than the feature description implies.
Teams get this backwards for an understandable reason: the model is the interesting component, and the interesting component attracts the design. A ticket-routing feature gets built as "an AI that reads tickets and routes them", when what it actually needs is a classifier for one ambiguous field and a lookup table for everything else. The AI version demos better, costs forty times more, fails in ways nobody can reproduce, and routes tickets slightly worse.
The second theme of this chapter is that you design the failure modes first. A model that is wrong, slow, overloaded or refusing is not an exception path — it is a regular Tuesday, at a rate you can measure. What the user sees in each of those states is a product decision, and if you make it after the happy path is built, you will make it badly and under time pressure.
Given a feature described as "use AI to do X", the instinct is to ask which model and which prompt. Those are the last two questions, not the first. What has to be decided before them — and why does answering in the wrong order reliably produce a system that is slower, costlier and less reliable than the one you could have built?
Decompose first: how much of this actually needs a model?
Break the feature into steps and classify each one. The result is usually surprising.
Run to execute the tests…
The fourth test gives the working test for the boundary. A step needs a model when its input is unstructured natural language and no enumerable rule covers the mapping. If either condition fails — structured input, or a rule you could write down — it is code, and code is faster, cheaper, testable and unchanging.
Think about hiring a skilled translator for a business trip. You would not have them read the flight number off your ticket, convert currency, or tell you the time — those have exact answers and using a person for them is slow and error-prone. You use them for the one thing nobody else can do: understanding what the person across the table actually meant. A well-designed AI feature has the same shape. The expensive, slow, occasionally-mistaken component is reserved for the part that genuinely requires interpretation, and everything with a right answer is done by the thing that gets right answers. The failure mode is equally recognisable: the traveller who has the translator handle everything, pays for hours of it, and gets the exchange rate slightly wrong.
Choose the simplest shape that works
Four shapes, in increasing order of cost and failure surface. Start at the top and only descend when the one above genuinely cannot do it.
Run to execute the tests…
The fifth test prices the most common over-engineering. An agent where a pipeline would do costs roughly seventeen times more and more than doubles the failure surface — and the pipeline is testable, reproducible, and debuggable with a stack trace.
Design the failure modes before the happy path
The model will be wrong, slow, overloaded and refusing, at rates you can look up. What the user sees in each state is a design decision.
Run to execute the tests…
The first test reframes the whole section. Fourteen percent of requests land in a degraded state — roughly one interaction in seven. Those are not edge cases; they are a substantial fraction of the product, and designing them last means designing them under incident pressure.
The third names the asymmetry that catches teams: an outage is loud and easy to handle, while a wrong answer is quiet and needs validation, visible provenance and a cheap correction path. The expensive one to design is the one usually left out.
The feature contract
An AI feature is a component with a contract, and writing it down is what makes the rest of Part VII possible.
Run to execute the tests…
The second test lists the three fields teams reliably skip: latency budget, fallback, and eval gate. All three get decided eventually. Decided at design time they are constraints; decided later they are whatever happened to be true.
The third turns the latency budget into arithmetic. Once you have written down that the model call is 1,800ms of a 3,000ms budget, a proposal to add a second call becomes a conversation about the remaining 840ms rather than a matter of taste.
The instinct to route work around an expensive, unreliable component is older than any of this. Early web architecture had the same shape: the database was slow and precious, so a generation of engineers learned to cache aggressively, denormalise deliberately, and push work to the edges. Search engines learned it too — the expensive ranking model runs over a shortlist that cheap retrieval produced, which is exactly the two-stage pattern of Part IV Chapter 4. What is unusual about the current moment is that the expensive component is also the interesting one, and interest distorts architecture in a way cost alone does not. Nobody ever built a feature around the database because databases were exciting. The closest historical parallel is probably the expert-systems era of the 1980s, where the same enthusiasm produced systems that encoded in an inference engine a great many rules that were, on inspection, ordinary conditionals — and the retrospective consensus was not that the technology failed but that it was applied to problems that did not need it.
The design exercise I run before writing any code is to describe the feature as a sequence of steps on a whiteboard and mark every step that genuinely needs judgement. It takes fifteen minutes and it has never once come back with more than a quarter of the steps marked. The conversation that follows is the valuable part, because someone always defends a step that could be a lookup table on the grounds that the model handles edge cases better — and sometimes that is true and worth paying for, and mostly the edge cases turn out to be three known ones that a rule handles exactly. The other habit is writing the fallback before the prompt. If I cannot describe what the user sees when the model is unavailable, I do not understand the feature well enough to build it, and I have noticed that features whose fallback is genuinely hard to specify are usually features where the model is doing something it should not be trusted with.
Rolling it out
The last architectural decision is how it reaches users, and the answer is gradually, with each step gated on evidence.
Run to execute the tests…
The third test is the one that turns a canary into a measurement. Compare against a concurrent control, not against nothing — a 0.14 regeneration rate reads as fine until you see the control sitting at 0.11.
A Nigerian logistics company scoped an AI feature to automatically handle delivery-status enquiries end to end: read the message, find the shipment, work out what to say, and reply. Built as an agent with four tools, it worked, took nine seconds, cost roughly forty times the estimate, and got the shipment wrong about one time in twelve because the model was matching references loosely where an exact lookup existed. The rewrite decomposed it: a regex plus a database lookup found the shipment reference (exact, instant, free), one structured model call classified the intent and pulled any secondary details, a lookup table mapped intent to a response template, and the template rendered with real data. One model call instead of a loop. Latency went from nine seconds to just over one, cost fell by roughly 95%, and the wrong-shipment rate went to zero because the lookup was exact. The engineer who did it made the observation that stuck: nothing in the original design was incompetent — the agent genuinely did the job — it was just that four of the five things it was doing had exact answers, and the model had been asked to approximate them.
Take an AI feature you are planning and write down every step it performs, then strike out each one where the input is structured or an enumerable rule covers the mapping. What remains is the part that needs a model. Now ask two questions about the remainder. Could those steps be one call instead of several — and if so, why are they separate? And what does the feature do when that one call fails, is slow, or returns something the validator rejects? If the answer to the second question is "we would have to think about that", you have found the part of the design that is missing, and it is the part that determines what one interaction in seven looks like. The uncomfortable follow-up: having struck out the deterministic steps, is what remains still worth building an AI feature for — or was the interesting part the bit you just crossed out?
Common Mistakes
Starting from the model. Which model and which prompt are the last two decisions. Starting there produces a feature shaped like the tool.
Not decomposing. A feature described as "AI does X" is typically two model steps out of eight; the rest are queries, templates and lookups.
Moving a rule into the prompt. Routing tables are readable, testable and editable by operations. In a prompt they become non-deterministic and owned by one person.
Separate calls for adjacent judgement steps. Classification and extraction over the same text are one structured call — two is double the latency and double the failure surface.
Descending the shape ladder too far. An agent where a pipeline would do is roughly seventeen times the cost and more than twice the failure surface.
Forgetting the top rung. "No model" is a shape. Many AI features are a search box, a rules engine or a better form.
Designing the happy path first. Around one interaction in seven lands in a degraded state; deciding those last means deciding them under incident pressure.
Only handling outages. Unavailability is loud and easy. A wrong answer is quiet and needs validation, provenance and a cheap correction path.
A single degraded mode. Build the ladder: flagged answer, retrieved documents, keyword search, honest message with escalation.
Silent degradation. A fallback that looks identical to the real thing teaches users to trust output produced differently than they believe.
Leaving human placement to default. Review-everything, review-flagged, escalation-only and nobody are all legitimate. Arriving at one by accident is not.
No latency budget. Written down, it makes "add another call" a conversation about milliseconds instead of taste.
No cost ceiling. At design time it is a constraint; in an invoice it is an incident.
No named eval gate. The feature then ships on whatever its author thought looked good.
Returning no confidence signal while planning to review low-confidence outputs. The contract has to carry what the process depends on.
Skipping shadow mode. Real traffic, logged and never shown, is the cheapest way to discover what your eval set did not contain.
A canary with no control. Without a concurrent comparison you have a number and nothing to judge it against.
Deleting the old path at partial rollout. It is both your fallback and your rollback.
Interview Corner
Take a feature you are about to build and decompose it on paper: every step, each marked code or judgement, using the two-condition test. Combine adjacent judgement steps into one structured call and count what is left — if it is more than a quarter of the steps, look again. Then choose the shape per step from the ladder and justify any descent below single-call. Write the nine-field contract before any implementation, and make the latency budget add up. Design the degradation ladder down to a non-AI floor with every rung labelled. Finally, plan the rollout as stages with evidence gates, keep the old path live, and run shadow mode first — real traffic, logged, never shown. It will find something your eval set did not contain.
Run to execute the tests…
Summary
- Model and prompt are the last two decisions. Start from what the user needs, decompose into steps, and classify each.
- A step needs a model only when its input is unstructured and no enumerable rule covers the mapping. Typically under a quarter of steps qualify.
- Combine adjacent judgement steps into one structured call. Two calls over the same text is double the latency and double the failure surface.
- Choose the simplest shape: no model → single call → pipeline → RAG → agent, descending only when the rung above genuinely cannot do it. An agent where a pipeline fits is roughly seventeen times the cost.
- Design the degraded states first. Wrong, slow, unavailable and refusing account for around one interaction in seven. Build a ladder down to a non-AI floor, and label every rung — silent degradation is worse than an error.
- Wrong is the expensive one to design: quiet, needs validation, visible provenance and a cheap correction path.
- Write the nine-field contract before implementing: typed input and output, pinned prompt and model versions, latency budget, cost ceiling, fallback, eval gate, human placement. The three teams skip are latency, fallback and gate.
- Make the latency budget arithmetic, so adding a call is a conversation about milliseconds.
- Roll out in stages with evidence gates: shadow → internal → canary → partial → full, comparing the canary against a concurrent control.
- Keep the old path live — it is both the fallback and the rollback — and treat a provider model change as a new rollout.
In the next chapter, Cost & Latency Control, we take the budgets this chapter wrote down and make them hold: where the money and the milliseconds actually go, which optimisations pay and which are folklore, and how to find out before the invoice does.
The advice in this chapter is almost entirely subtractive, and I notice that makes it harder to act on than advice that adds things. "Use the model for less" does not produce a diagram anyone is excited to present, and the version of this feature with four tools and an agent loop genuinely demos better than the version with one call and a lookup table — right up until it is in production, where the lookup table is the part that never fails. Saint-Exupéry was writing about aircraft design and the same asymmetry applied: adding a component is visible work with an obvious author, while removing one leaves nothing to point at. What makes this particularly sharp for AI features is that the expensive component is also the one everyone finds interesting, so the incentive to route work around it is fighting both the aesthetics and the org chart. The best architectural review question I know is still the plainest: for each step here, what would break if we did it in code — and if the honest answer is "nothing", that is not a small finding.
Q&A · 0
Enrol to ask questions and join the discussion.
No questions yet — be the first to ask.