How to Prepare for a Machine Learning Engineer Interview

“How would you build a model to predict customer churn?”

What they're really asking: Can you frame an ambiguous problem properly, or do you jump straight to an algorithm?

How to answer: Spend the first minutes defining churn precisely — what event counts, over what window, and what action the prediction triggers. Then describe the training data construction with a temporal split, name the leakage risks such as features generated by the cancellation process itself, and propose a simple baseline. Only then choose a model, and close with how the score becomes a decision through threshold selection tied to intervention cost.

“Your model scores well offline but performs poorly in production. How do you investigate?”

What they're really asking: Whether you have actually operated a model, because this is the defining experience of the role.

How to answer: Work through the likely causes in order of frequency: training–serving skew in feature computation, leakage that inflated the offline number, distribution shift between the training window and live traffic, and a threshold or business process that differs from what you assumed. Say how you would test each — compare feature distributions between training and live logs first, since that catches the most common cause quickly.

“How do you choose an evaluation metric for an imbalanced classification problem?”

What they're really asking: Whether you connect the metric to the cost of errors rather than defaulting to accuracy or AUC.

How to answer: Start from the asymmetry of the two error types in business terms, then explain why accuracy is uninformative at extreme imbalance and why ROC AUC can look healthy while precision at the operating threshold is poor. Recommend precision-recall analysis at the threshold you would actually deploy, and mention calibration if the score feeds a downstream decision rule.

“How do you make sure features are computed the same way in training and serving?”

What they're really asking: Whether you understand the single most common production failure in applied ML.

How to answer: Describe sharing one implementation rather than two — a single feature computation library or a feature store used by both paths — and point-in-time correctness so training features only use data that would have been available at inference. Add the practical safeguards: logging live feature values, comparing their distributions to training, and running a shadow deployment before switching traffic.

“When would you not use machine learning?”

What they're really asking: Whether you have judgement, or whether you will build a model for a problem a rule solves better.

How to answer: Give the concrete conditions: no reliable labels, the rule is already well understood, decision volume too low to justify maintenance, or a hard requirement for exact explainability. Follow with an example where a heuristic or a simple ranking rule was the right answer, and note that a deployed heuristic also gives you the logging you need to build a model later.

“How do you decide when to retrain a model?”

What they're really asking: Whether you monitor deliberately or retrain on a calendar because it feels safe.

How to answer: Separate triggers from cadence. Describe monitoring input drift, prediction distribution and, where labels arrive, rolling performance. Explain that you retrain on a trigger where possible and on a schedule where labels are too delayed to detect degradation. Mention validating each candidate against the incumbent before promotion, and the feedback loop risk where the model shapes the data it is later trained on.