Lesson 1 of 5
How AI Shows Up in Interviews
"Beware of bugs in the above code; I have only proved it correct, not tried it." — Donald Knuth, note to a correspondent (1977)
Knuth's line is the perfect epigraph for an interview chapter because it is the exact distinction interviewers are trying to detect. Anyone can describe RAG. The question is whether you have run one against real traffic and watched it fail in the specific ways it fails. Proved-correct and tried-it produce different answers to the same question, and the difference is audible within about ninety seconds.
Seven parts of this book have been about the work. This part is about being assessed on it, and the first thing to say is that AI engineering interviews are unusually badly calibrated across the industry right now. Some companies run genuinely good processes that probe judgement. Others ask you to recite prompt-engineering tips from 2023, or grill you on transformer internals you would never touch, or — most commonly — have no idea what to assess and default to a general backend interview with the word "AI" in the title.
That variance means two things for you. It is worth being able to recognise which kind of process you are in, because the right answer differs. And it is worth being good rather than prepared, because a rehearsed answer collapses on the first follow-up and a real one does not.
The good news is that the assessment target is narrower than the field. Interviewers who know what they are doing are looking for four things: whether you know what the technology genuinely cannot do, whether you reach for measurement without being asked, whether you design for failure as a normal state, and whether you can say "I don't know" precisely. Everything in Parts I through VII feeds one of those four, and none of them is about knowing a trick.
Two candidates give the same architecture for the same question. One has shipped an AI feature and one has read about them extensively. What, specifically, do they say differently — and can you produce those differences without having done the work, or is the only preparation actually doing it?
What is actually being assessed
Not vocabulary. Four capabilities, and each has a tell.
Run to execute the tests…
The second test is the distinction worth internalising. Both statements about RAG are true; only one could not have been produced by reading about it. The ceiling that retrieval recall places on answer accuracy becomes salient after you have spent a week improving prompts against 0.6 recall — and interviewers who have done that hear it immediately.
Think about hiring a driver. You can ask them to describe a roundabout, and anyone who has read the highway code will describe it correctly. What actually separates candidates is what they say without being asked — that they check the mirror before the indicator, that they would not attempt a particular junction in the rain, that the car pulls slightly left under braking so they leave extra distance. None of that is advanced knowledge. It is the residue of having driven, and it is nearly impossible to fake because you do not know which details to invent. Interviewers for this work are listening for the same thing: not whether you can describe retrieval, but whether you mention that the archived copy of a document tends to outrank the current one because it repeats key terms more often.
The formats, and what each is really testing
Five shapes cover almost everything, and each rewards a different preparation.
Run to execute the tests…
The third test names the highest-value three minutes available in a live build. Write five test cases with expected properties before writing the prompt. It costs almost nothing, it is the clearest signal in the entire format, and almost nobody does it.
The tells
What experienced interviewers are listening for, in both directions.
Run to execute the tests…
The last test describes the most common transcript in the industry right now: a strong generalist engineer with one good instinct and three tells that say read, not run. It is not a knowledge gap. It is fixable by building one small thing end to end, which is exactly what the hands-on section proposes.
The same question at three levels
Level calibration is mostly about what you volunteer, not what you know.
Run to execute the tests…
The fifth test is the finding that catches people preparing badly. All three levels give the same architecture. The differentiator is what you volunteer around it, so memorising architectures prepares you for the part nobody is scoring.
The last is the corresponding warning: interviewers probe whatever you raise. Mentioning cost per successful outcome and then being unable to define it is worse than never mentioning it.
Every technology wave produces a period where hiring runs ahead of any shared idea of competence. The mid-2000s had it with "web 2.0" roles, mobile had it around 2010, and data science had a particularly long one — for several years the interview consisted of machine-learning theory questions while the job was overwhelmingly SQL, data cleaning and stakeholder management, and the mismatch took most of a decade to correct. AI engineering is in that phase now, with an additional complication: the underlying technology changes fast enough that specific knowledge decays in months while the durable skills — decomposition, measurement, failure design — are exactly the ones that are hardest to assess in an hour. The historical pattern suggests the process converges, and it converges on whatever correlates with people succeeding in the role. If the data-science parallel holds, the questions that survive will be less about the model and more about everything around it.
When I interview for this work, the question I get the most from is deliberately open: "tell me about a time an AI feature behaved in a way that surprised you." Candidates who have shipped one answer immediately and specifically — the archived document that outranked the current one, the agent that hit its iteration cap having found the answer at step six, the cache that served another tenant's summary. Candidates who have not tend to describe a hallucination in general terms, because it is the only failure mode that is famous. Neither answer is disqualifying and the difference in information is enormous. The other thing I have learned is to press once on anything the candidate volunteers. Not adversarially — one level down, to see whether it is theirs. Someone who raises evaluation and can then describe what they put in the corpus, who labelled it, and what they discovered has told me more in ninety seconds than an hour of architecture discussion would.
A Lagos startup interviewing for its first AI engineer ran a take-home: build a small assistant over a provided document set. Most submissions were architecturally similar and several were more sophisticated than the one that got the offer. The winning submission was notably plain — a single model call over retrieved chunks, no agent, no framework — and it included a twenty-case eval set with expected properties, a README section titled "what this gets wrong", and a paragraph explaining that two of the requested behaviours had been implemented as lookups rather than model calls because they had exact answers. The hiring manager's note was that it was the only submission that could tell them how well it worked, and the only one that had clearly been run against the awkward cases rather than the pleasant ones. The candidate later said the eval set had taken about forty minutes and had been the reason they found the chunking bug they mentioned in the README.
Imagine being asked, right now, "tell me about an AI feature you built and what surprised you." Draft the answer. If the surprise you reach for is hallucination in general, you are describing the famous failure rather than one you experienced — and that is worth knowing before an interview rather than during one. Now go further: what would you say if pressed one level down on every claim in that answer? How was it measured, who decided the correct answers, what did you do when it regressed, what does it cost per successful outcome? The questions you cannot answer are not gaps in your preparation. They are the parts of the work you have not done yet, and the honest response is to go and do a small version of them — which takes an afternoon and is the only preparation that survives a follow-up.
Common Mistakes
Preparing by memorising architectures. All three levels answer with the same shape; the differentiator is what you volunteer around it.
Drawing before asking. Volume, cost of a wrong answer, and whether a non-AI path exists all change the architecture.
Reaching for an agent. The most reliable negative tell. Establish that the control flow is genuinely unenumerable first.
Proposing fine-tuning to add facts. It cannot cite, cannot update without retraining and cannot filter per user.
Describing prompt injection as fixable with a better prompt. There is no prompt-layer fix, and believing otherwise shapes every system you build.
Treating the prompt as the whole solution. It is one versioned component among several.
Being unable to say how you would know it worked. The fastest way to reveal that a design was never operated.
Quoting benchmarks instead of describing a measurement. Interviewers want the harness you would build, not a leaderboard.
No degraded path anywhere. A design with only a happy path says the candidate has never been paged.
Starting a debugging round at the prompt. Degraded mode, retrieval, tools and truncation come first and are right more often.
Building before writing test cases in a live exercise. Five cases with expected properties costs three minutes and is the clearest signal available.
"I'd make it clearer" in a prompt critique. Name specifics: unverifiable constraints, no unanswerable-case instruction, free text where an enum belongs.
A shipping story with no failure in it. Shipping one of these without a surprise does not happen, so the story reads as unlived.
Volunteering something you cannot defend one level down. It converts an unknown into a demonstrated gap.
Hedging instead of calibrating. "I haven't built that in production, so I'd be guessing past the basics" reads as senior; "not sure, maybe fine-tuning?" does not.
Not noticing which format you are in. Answering a design question with a debugging structure happens constantly under pressure.
Interview Corner
Build one small feature end to end this week — a classifier or a retrieval assistant over documents you have — and do it in the order that demonstrates everything: five eval cases with expected properties first, then the simplest call that could work, then run them. Keep a note of every surprise; that note is your answer to the unfakeable question. Then write the README section titled "what this gets wrong", because being able to state your own system's failure modes precisely is the single most senior-sounding thing available to you. Finally, rehearse one system-design answer end to end — questions first, simplest shape, the metric, the fallback, what you would need to test — and have someone press you one level down on every claim in it. Whatever you cannot answer is not a preparation gap; it is work you have not done, and it takes an afternoon.
Run to execute the tests…
Summary
- Four things are assessed: knowing the limits, reaching for measurement unprompted, designing for failure as a normal state, and saying "I don't know" precisely. None is vocabulary.
- The strongest signal is a specific failure mode named unprompted — the kind that only becomes salient after you have watched one.
- Five formats: system design, debugging, live build, prompt/eval critique, and the experience question. The cue is usually in the interviewer's first sentence.
- Design is a questions-first format. Volume, cost of a wrong answer, and whether a non-AI path exists reorder the whole architecture.
- Debugging is narrowing, not guessing. Degraded mode, retrieval, tools, truncation — generation last.
- In a live build, write five eval cases before the prompt. Three minutes, clearest signal available, almost nobody does it.
- All three levels give the same architecture. Seniority shows up in what you volunteer: measurement at mid, and at senior, what you would not use a model for and what would make you not build it.
- Never volunteer what you cannot defend one level down — it converts an unknown into a demonstrated gap.
- Calibration reads as senior; hedging does not. "I haven't built that in production" beats "not sure, maybe fine-tuning?"
- The only preparation that survives a follow-up is building one small thing end to end — and keeping a note of every surprise.
In the next chapter, Designing an AI System, we take the most common interview format and work it properly: how to open, how to sequence, what to say out loud while thinking, and how the same brief is answered under three different sets of constraints.
Writing an interview chapter for a field this young is uncomfortable, because the honest content is partly about the interviewers. A meaningful share of AI engineering processes in 2026 are not assessing the work — they are assessing familiarity with a vocabulary, sometimes because the people running them have not built one of these either. I have gone back and forth on how directly to say that, and settled on saying it plainly, because a candidate who assumes every rejection reflects a real gap will draw the wrong lesson from a process that was measuring something else. The corresponding responsibility, though, is not to let that become an excuse. The four capabilities this chapter names are genuinely what the work requires, they are assessable, and the ones you lack are visible to you before any interviewer sees them. Knuth's line is about the gap between proving and trying, and the uncomfortable part of it is that you always know which side of that gap you are on. The interview only discovers what you already knew.
Q&A · 0
Enrol to ask questions and join the discussion.
No questions yet — be the first to ask.