Skip to content
TailoredMinds

AI rolesGuideUpdated 7 min read

How to interview AI engineers: a practical evaluation framework

A loop, a question bank and a scoring rubric for LLM, ML and MLOps engineers, built to test production judgement rather than trivia.

Written by

Tercio Lima, Growth Lead

Reviewed by

Lucas Salvaia, Co-Founder & COO

The short answer

Interview an AI engineer for production judgement, not trivia. Ask them to walk through an AI feature or model they shipped, how they measured quality, a failure they caught in production and the cost or latency trade-offs they made. Then run a short work sample, ideally one that allows AI tools, and score every candidate against the same written rubric, agreed before the first interview.

Key takeaways

  1. Decide what you're testing before you write a single question: shipped work, quality measurement, failure handling and trade-offs.
  2. Structured interviews, where every candidate gets the same questions and the same scoring, are among the strongest predictors of job performance in the research.
  3. Most developers now use AI coding tools, so a work sample that allows them tests the job as it's really done.
  4. Use different question banks for LLM, ML and MLOps engineers. Their evidence is different.
  5. Score with behaviourally anchored levels and write the evidence down, before the debrief.
  6. If AI tools score or rank candidates in your process, keep a person accountable for every decision.

What you're actually testing

An AI engineering interview should test four things: shipped work, how quality was measured, how failure was handled, and how trade-offs were made. These are the signals that separate an engineer who has run AI in production from one who has built a demo. They're also the signals we look for when we shortlist candidates for AI roles.

SignalWhat good looks likeWhat to probe
Shipped workAn AI feature or model that real users relied on, and what changed after launchWho used it, how much, and what the candidate personally owned
Quality measurementA defined evaluation: test sets, metrics, human review, regression checks"How did you know it was good enough to ship?"
Failure handlingA production failure they found, contained and learned from"What broke after launch, and how did you find out?"
Trade-offsExplicit choices between quality, latency, cost and safety"What did you give up, and why?"

If you're not yet sure which AI role you're hiring for, settle that first. Our comparison of LLM engineer vs ML engineer vs AI engineer covers the differences, and our guide to hiring AI engineers covers the whole process from brief to offer.

Why structure matters

Structured interviews predict job performance better than most other selection methods. A 2022 re-analysis of decades of personnel-selection research by Sackett, Zhang, Berry and Lievens put structured interviews at the top of its ranking, with an estimated validity of .42, ahead of job knowledge tests (.40), work sample tests (.33) and general cognitive ability tests (.31) [2], [3].

Structure means three things:

  1. The same core questions for every candidate for the same role.
  2. A scoring guide written before the interviews, describing what a weak, solid and strong answer looks like.
  3. Independent scoring by each interviewer before the group discusses the candidate.

Structure is also fairer. It reduces the room for first impressions, similarity bias and "culture fit" judgements that aren't tied to the job.

The interview loop

A loop of four stages covers most AI engineering roles without wearing candidates out:

StageWhat it testsFormatRed flags
1. Deep dive on past workOwnership, depth, honesty about failure45–60 min conversation about one system they builtCan't say what they personally did; no numbers or examples; nothing ever went wrong
2. Work samplePractical judgement on a realistic problem2–3 hours at most, or 60–90 min live; AI tools allowed if your team uses themPolished output with no reasoning about quality or edge cases
3. System design for an AI featureArchitecture, evaluation design, cost and latency60 min discussion, e.g. "design support search over a large document base"Jumps to a model choice before understanding the problem; no evaluation plan
4. CollaborationWorking with product, data and non-AI engineers45 min with a future peer or product managerCan't explain trade-offs to a non-specialist; dismisses user feedback

Keep the whole loop short and schedule it tightly. Strong AI engineers are usually in more than one process, and a slow loop loses them.

Question bank by role

Use questions that ask for a specific story, then follow up for evidence. Each question below has a follow-up that separates real experience from a rehearsed answer.

LLM engineer (applied AI engineer)

  • "Walk me through an LLM feature you shipped. Who used it, and what changed after launch?" Follow-up: "What did you change in the first month, and why?"
  • "How did you measure quality before and after launch?" Follow-up: "How was your evaluation set built, and how did you keep it current?"
  • "Tell me about a time retrieval gave the model the wrong context." Follow-up: "How did you detect it, and what did you change: chunking, embeddings, re-ranking, or the prompt?"
  • "How did you handle a model provider upgrade or switch?" Follow-up: "What broke, and how did your regression checks catch it?"
  • "What did a request cost, and how fast was it?" Follow-up: "What did you do to bring either one down?"
  • "Where did you stop the model from acting on its own?" Follow-up: "What guardrails or human review did you add?"

Machine learning engineer

  • "Tell me about a model you took from experiment to production." Follow-up: "How did it perform once it was live compared with offline?"
  • "Which metric did you optimise, and why that one?" Follow-up: "When did the metric disagree with what the business wanted?"
  • "Describe error analysis that changed your data or your model." Follow-up: "What did you find in the errors that the aggregate metric hid?"
  • "How did you detect drift?" Follow-up: "What triggered retraining, and who decided?"
  • "When did you decide not to use a more complex model?"

MLOps or AI platform engineer

  • "Describe a platform or pipeline other engineers actually adopted." Follow-up: "What did it replace, and why did people switch?"
  • "Walk me through the path from a commit to a production model." Follow-up: "How do you roll back?"
  • "Tell me about a cost or GPU capacity problem you diagnosed." Follow-up: "What did you measure first?"
  • "What did you monitor, and how fast did you know something was wrong?"

For deeper role profiles, see what an LLM engineer does, what an ML engineer does and what an MLOps engineer does.

Designing a work sample that allows AI tools

If your engineers use AI coding assistants at work, let candidates use them in the work sample, and test what the tools can't do for them. In Stack Overflow's 2025 survey, 84% of developers said they use or plan to use AI tools in their development process, and 51% of professional developers use them daily. The same survey found more developers distrust the accuracy of AI tools (46%) than trust it (33%), and the most common frustration was AI answers that are "almost right, but not quite" (66%) [1].

That last number is the point. The valuable skill is spotting "almost right". A good AI-allowed work sample:

  1. Is small and realistic. For an LLM engineer: a small retrieval or extraction task over a handful of documents, with a few deliberately tricky cases. For an ML engineer: a small dataset with a leakage or labelling problem hidden in it. For MLOps: a broken deployment or pipeline config to diagnose.
  2. Asks for reasoning, not just output. "How would you measure whether this works? What would you ship first? What would you not trust?"
  3. Includes a review step. Hand them a plausible but flawed AI-generated solution and ask them to review it like a pull request.
  4. Is time-boxed. Two to three hours at most for a take-home, or a 60–90 minute live session. Long unpaid take-homes filter out people with less free time, not weaker engineers.
  5. Is discussed afterwards. A 20-minute follow-up on their own submission shows what they understand.

A scoring rubric that measures judgement

Score each signal on a written scale with examples of what each level looks like. This is a behaviourally anchored rubric, and it stops "I liked them" from deciding the outcome.

Signal1: Weak2: Mixed3: Solid4: Strong
Shipped workOnly demos or courseworkContributed to a shipped feature, unclear ownershipOwned a shipped feature with real usersOwned it, iterated after launch, can show what changed
Quality measurement"It looked right"Some manual testingA defined evaluation set and metricEvaluation in CI, updated from production failures
Failure handlingCan't name a failureNamed one, vague on causeClear cause, fix and detectionBuilt monitoring or tests so it couldn't recur silently
Trade-offsNo trade-offs consideredNamed one dimensionWeighed quality, latency and costQuantified the trade-offs and explained the decision to others
CommunicationHard to followClear with promptingClear and structuredAdapts the explanation to the audience

Agree the bar before the loop starts, for example "no 1s, and at least 3s on shipped work and quality measurement". Each interviewer scores and writes evidence alone, before the debrief.

Bias, consistency and AI in your own process

Consistency is the main defence against bias. Ask every candidate the same core questions, score them on the same rubric and keep written notes. Avoid questions about age, family, nationality or anything unrelated to the job.

If you use AI to screen applications, summarise interviews or score work samples, remember that the tool is part of your hiring decision. In the EU, AI used to evaluate candidates is high-risk under the AI Act, and Article 14 requires that people overseeing it can understand its limits, avoid over-relying on it and override it [4]. Those obligations apply from 2 December 2027. Our article on the EU AI Act and hiring covers the details.

How Tailored Minds helps

The four signals in this article are the ones our team checks before we shortlist a candidate for an AI role. When you brief a role on Tailored Minds, we help you define it first, then our team searches the talent network while approved independent recruiters can work the role at the same time. People review every candidate we shortlist, so your interview loop starts with candidates who have already shown evidence of shipped work. You see profiles and intro videos in the company portal and give feedback there, and your feedback steers the next candidates. AI assists the search; people decide. Hire AI engineers with Tailored Minds.

Frequently asked questions

How do I interview an LLM engineer?
Start with a deep dive on one LLM feature they shipped: who used it, how quality was measured, what broke after launch and what it cost to run. Follow with a short work sample, such as a retrieval or extraction task with a few tricky cases, and a system design discussion for an AI feature. Score every candidate on the same written rubric covering shipped work, quality measurement, failure handling and trade-offs.
What should an AI engineer take-home test?
Judgement on a small, realistic problem, not recall or endurance. Keep it to two or three hours at most, include a few deliberately tricky cases, and ask the candidate to explain how they would measure quality and what they would ship first. A useful addition is a flawed AI-generated solution for them to review. Discuss the submission with them afterwards.
Should candidates be allowed to use AI tools in interviews?
If your engineers use AI tools at work, yes, for the work sample. Most developers now use them, so a ban tests an unrealistic version of the job. Tell candidates in advance, then ask how they used the tools and how they checked the output. Keep at least one conversation without tools, such as the deep dive on past work, to hear their own reasoning.
How many interview rounds should an AI engineer loop have?
Four stages cover most roles: a deep dive on past work, a work sample, a system design discussion and a collaboration conversation. More rounds rarely add signal and do add time, and strong AI engineers are often in several processes at once. Schedule the loop tightly and agree a fixed window for written feedback after each stage.
How do I make AI engineer interviews fairer?
Use a structured interview: the same core questions for every candidate, a scoring guide written in advance and independent scoring before the debrief. Keep work samples short so people with less free time aren't filtered out, and avoid questions unrelated to the job. If AI tools help screen or score candidates, keep a trained person responsible for every decision.

Sources

  1. 2025 Developer Survey: AI, Stack Overflow
  2. Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range (Sackett, Zhang, Berry and Lievens, 2022), Journal of Applied Psychology
  3. Is cognitive ability the best predictor of job performance? New research says it's time to think again, Society for Industrial and Organizational Psychology
  4. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 14 and Annex III, EUR-Lex

About the author

Portrait of Tercio Lima

Tercio Lima

Growth Lead at Tailored Minds

Tercio Lima is the Growth Lead at Tailored Minds. He runs the brand, content and search work that brings companies to the platform, and the sourcing work that finds engineers for their roles, so what he writes about hiring comes from live searches rather than desk research.

More from TercioLinkedIn

Share this article

Put a whole network on your next hire.

Tell us who you're looking for. Our talent network and recruiters take it from there.

Talent? Join freeRecruiter? Join the workforce