The short answer
Interview an AI engineer for production judgement, not trivia. Ask them to walk through an AI feature or model they shipped, how they measured quality, a failure they caught in production and the cost or latency trade-offs they made. Then run a short work sample, ideally one that allows AI tools, and score every candidate against the same written rubric, agreed before the first interview.
Key takeaways
- Decide what you're testing before you write a single question: shipped work, quality measurement, failure handling and trade-offs.
- Structured interviews, where every candidate gets the same questions and the same scoring, are among the strongest predictors of job performance in the research.
- Most developers now use AI coding tools, so a work sample that allows them tests the job as it's really done.
- Use different question banks for LLM, ML and MLOps engineers. Their evidence is different.
- Score with behaviourally anchored levels and write the evidence down, before the debrief.
- If AI tools score or rank candidates in your process, keep a person accountable for every decision.
On this page8 sections
What you're actually testing
An AI engineering interview should test four things: shipped work, how quality was measured, how failure was handled, and how trade-offs were made. These are the signals that separate an engineer who has run AI in production from one who has built a demo. They're also the signals we look for when we shortlist candidates for AI roles.
| Signal | What good looks like | What to probe |
|---|---|---|
| Shipped work | An AI feature or model that real users relied on, and what changed after launch | Who used it, how much, and what the candidate personally owned |
| Quality measurement | A defined evaluation: test sets, metrics, human review, regression checks | "How did you know it was good enough to ship?" |
| Failure handling | A production failure they found, contained and learned from | "What broke after launch, and how did you find out?" |
| Trade-offs | Explicit choices between quality, latency, cost and safety | "What did you give up, and why?" |
If you're not yet sure which AI role you're hiring for, settle that first. Our comparison of LLM engineer vs ML engineer vs AI engineer covers the differences, and our guide to hiring AI engineers covers the whole process from brief to offer.
Why structure matters
Structured interviews predict job performance better than most other selection methods. A 2022 re-analysis of decades of personnel-selection research by Sackett, Zhang, Berry and Lievens put structured interviews at the top of its ranking, with an estimated validity of .42, ahead of job knowledge tests (.40), work sample tests (.33) and general cognitive ability tests (.31) [2], [3].
Structure means three things:
- The same core questions for every candidate for the same role.
- A scoring guide written before the interviews, describing what a weak, solid and strong answer looks like.
- Independent scoring by each interviewer before the group discusses the candidate.
Structure is also fairer. It reduces the room for first impressions, similarity bias and "culture fit" judgements that aren't tied to the job.
The interview loop
A loop of four stages covers most AI engineering roles without wearing candidates out:
| Stage | What it tests | Format | Red flags |
|---|---|---|---|
| 1. Deep dive on past work | Ownership, depth, honesty about failure | 45–60 min conversation about one system they built | Can't say what they personally did; no numbers or examples; nothing ever went wrong |
| 2. Work sample | Practical judgement on a realistic problem | 2–3 hours at most, or 60–90 min live; AI tools allowed if your team uses them | Polished output with no reasoning about quality or edge cases |
| 3. System design for an AI feature | Architecture, evaluation design, cost and latency | 60 min discussion, e.g. "design support search over a large document base" | Jumps to a model choice before understanding the problem; no evaluation plan |
| 4. Collaboration | Working with product, data and non-AI engineers | 45 min with a future peer or product manager | Can't explain trade-offs to a non-specialist; dismisses user feedback |
Keep the whole loop short and schedule it tightly. Strong AI engineers are usually in more than one process, and a slow loop loses them.
Question bank by role
Use questions that ask for a specific story, then follow up for evidence. Each question below has a follow-up that separates real experience from a rehearsed answer.
LLM engineer (applied AI engineer)
- "Walk me through an LLM feature you shipped. Who used it, and what changed after launch?" Follow-up: "What did you change in the first month, and why?"
- "How did you measure quality before and after launch?" Follow-up: "How was your evaluation set built, and how did you keep it current?"
- "Tell me about a time retrieval gave the model the wrong context." Follow-up: "How did you detect it, and what did you change: chunking, embeddings, re-ranking, or the prompt?"
- "How did you handle a model provider upgrade or switch?" Follow-up: "What broke, and how did your regression checks catch it?"
- "What did a request cost, and how fast was it?" Follow-up: "What did you do to bring either one down?"
- "Where did you stop the model from acting on its own?" Follow-up: "What guardrails or human review did you add?"
Machine learning engineer
- "Tell me about a model you took from experiment to production." Follow-up: "How did it perform once it was live compared with offline?"
- "Which metric did you optimise, and why that one?" Follow-up: "When did the metric disagree with what the business wanted?"
- "Describe error analysis that changed your data or your model." Follow-up: "What did you find in the errors that the aggregate metric hid?"
- "How did you detect drift?" Follow-up: "What triggered retraining, and who decided?"
- "When did you decide not to use a more complex model?"
MLOps or AI platform engineer
- "Describe a platform or pipeline other engineers actually adopted." Follow-up: "What did it replace, and why did people switch?"
- "Walk me through the path from a commit to a production model." Follow-up: "How do you roll back?"
- "Tell me about a cost or GPU capacity problem you diagnosed." Follow-up: "What did you measure first?"
- "What did you monitor, and how fast did you know something was wrong?"
For deeper role profiles, see what an LLM engineer does, what an ML engineer does and what an MLOps engineer does.
Designing a work sample that allows AI tools
If your engineers use AI coding assistants at work, let candidates use them in the work sample, and test what the tools can't do for them. In Stack Overflow's 2025 survey, 84% of developers said they use or plan to use AI tools in their development process, and 51% of professional developers use them daily. The same survey found more developers distrust the accuracy of AI tools (46%) than trust it (33%), and the most common frustration was AI answers that are "almost right, but not quite" (66%) [1].
That last number is the point. The valuable skill is spotting "almost right". A good AI-allowed work sample:
- Is small and realistic. For an LLM engineer: a small retrieval or extraction task over a handful of documents, with a few deliberately tricky cases. For an ML engineer: a small dataset with a leakage or labelling problem hidden in it. For MLOps: a broken deployment or pipeline config to diagnose.
- Asks for reasoning, not just output. "How would you measure whether this works? What would you ship first? What would you not trust?"
- Includes a review step. Hand them a plausible but flawed AI-generated solution and ask them to review it like a pull request.
- Is time-boxed. Two to three hours at most for a take-home, or a 60–90 minute live session. Long unpaid take-homes filter out people with less free time, not weaker engineers.
- Is discussed afterwards. A 20-minute follow-up on their own submission shows what they understand.
A scoring rubric that measures judgement
Score each signal on a written scale with examples of what each level looks like. This is a behaviourally anchored rubric, and it stops "I liked them" from deciding the outcome.
| Signal | 1: Weak | 2: Mixed | 3: Solid | 4: Strong |
|---|---|---|---|---|
| Shipped work | Only demos or coursework | Contributed to a shipped feature, unclear ownership | Owned a shipped feature with real users | Owned it, iterated after launch, can show what changed |
| Quality measurement | "It looked right" | Some manual testing | A defined evaluation set and metric | Evaluation in CI, updated from production failures |
| Failure handling | Can't name a failure | Named one, vague on cause | Clear cause, fix and detection | Built monitoring or tests so it couldn't recur silently |
| Trade-offs | No trade-offs considered | Named one dimension | Weighed quality, latency and cost | Quantified the trade-offs and explained the decision to others |
| Communication | Hard to follow | Clear with prompting | Clear and structured | Adapts the explanation to the audience |
Agree the bar before the loop starts, for example "no 1s, and at least 3s on shipped work and quality measurement". Each interviewer scores and writes evidence alone, before the debrief.
Bias, consistency and AI in your own process
Consistency is the main defence against bias. Ask every candidate the same core questions, score them on the same rubric and keep written notes. Avoid questions about age, family, nationality or anything unrelated to the job.
If you use AI to screen applications, summarise interviews or score work samples, remember that the tool is part of your hiring decision. In the EU, AI used to evaluate candidates is high-risk under the AI Act, and Article 14 requires that people overseeing it can understand its limits, avoid over-relying on it and override it [4]. Those obligations apply from 2 December 2027. Our article on the EU AI Act and hiring covers the details.
How Tailored Minds helps
The four signals in this article are the ones our team checks before we shortlist a candidate for an AI role. When you brief a role on Tailored Minds, we help you define it first, then our team searches the talent network while approved independent recruiters can work the role at the same time. People review every candidate we shortlist, so your interview loop starts with candidates who have already shown evidence of shipped work. You see profiles and intro videos in the company portal and give feedback there, and your feedback steers the next candidates. AI assists the search; people decide. Hire AI engineers with Tailored Minds.
Frequently asked questions
How do I interview an LLM engineer?
What should an AI engineer take-home test?
Should candidates be allowed to use AI tools in interviews?
How many interview rounds should an AI engineer loop have?
How do I make AI engineer interviews fairer?
Sources
- 2025 Developer Survey: AI, Stack Overflow
- Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range (Sackett, Zhang, Berry and Lievens, 2022), Journal of Applied Psychology
- Is cognitive ability the best predictor of job performance? New research says it's time to think again, Society for Industrial and Organizational Psychology
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 14 and Annex III, EUR-Lex
About the author

Tercio Lima
Growth Lead at Tailored Minds
Tercio Lima is the Growth Lead at Tailored Minds. He runs the brand, content and search work that brings companies to the platform, and the sourcing work that finds engineers for their roles, so what he writes about hiring comes from live searches rather than desk research.
Share this article
