Data science interviews still trip on the experiment

The model is often the least interesting part of a data science interview. You receive a messy table, start discussing random forests and then discover the target is missing for a chunk of users. The train and test sets also contain rows from the same accounts. Your choice of algorithm now sits behind 2 larger problems.

A good practice conversation keeps pulling you back to those decisions. The data science interview track covers probability, statistics, EDA, classical ML, evaluation and experiments, with deep learning for roles that need it. Use it to test whether your reasoning survives questions, especially after the easy answer has left the room.

The track begins with probability, descriptive statistics, inference and hypothesis tests. A cleaning and EDA session leads into 2 classical ML conversations for linear models, SVMs, kNN, naive Bayes, trees, ensembles and clustering. Separate sessions cover model evaluation, validation, hyperparameters, A/B testing, experimental design and deep learning. Use the deep learning session when the posting asks for it.

EDA should change your plan

Exploratory analysis should question the rows before any model is chosen. You are trying to learn whether they mean what you think they mean, whether the target can be trusted and whether the prediction would be useful at the moment it is made. Start by naming the unit of observation and how the data was collected.

Missing values need a story. Income may be missing because a user skipped the field, because the field wasn’t available in an older product flow or because a system failed. Mean imputation treats those causes as interchangeable and can erase a useful signal. Explain which missingness you would inspect, what indicator you might add and how you would test the effect.

Outliers can be data errors or the business event you care about. A huge transaction may be fraud, a currency conversion bug or an enterprise customer. Deleting it because a box plot looks untidy is an aesthetic choice disguised as analysis. Trace a sample back to the source and decide with domain context.

Leakage deserves its own pause. A feature recorded after the outcome, a status code that embeds the target or rows from one customer split across train and test can make evaluation look wonderful. Time, account and geography may need grouped or temporal splits. State what information exists at prediction time. The Data Cleaning, Feature Engineering and EDA session keeps this question attached to every feature you propose.

Choose a model you can defend

Linear and logistic models remain useful because they are quick to fit, easier to inspect and often good baselines. Their assumptions give the interviewer something concrete to discuss. You should be able to explain what a coefficient means, where collinearity hurts and why regularization changes the fit.

Trees handle nonlinear relationships and interactions without much feature scaling. Ensembles can improve predictive performance, though they cost interpretation and operational simplicity. SVMs, kNN and naive Bayes each have settings where their assumptions fit. Naming all of them in one breath doesn’t answer the selection question.

Choose with the dataset, error cost, latency and audience in mind. A fraud model may care about recall at a review capacity. A churn score used by account managers may need calibrated probabilities and a reason they can discuss with a customer. If a simple model performs close enough and is easier to monitor, shipping it can be the grown-up choice.

Clustering needs the same discipline. Decide what distance means, how features are scaled and whether the clusters lead to an action. A colorful 2D plot can make random segments look like a strategy. Ask what someone will do differently for cluster 4 on Monday morning.

Evaluation begins with the decision

Accuracy can hide the error that matters. Precision, recall, ranking metrics and calibration answer different questions. Pick the metric after naming the action and the cost of false positives or false negatives. Then set a threshold that matches capacity or risk.

Cross-validation helps estimate variation when observations are exchangeable. Time-dependent data often needs a forward split because the future cannot leak into the past. Grouped data may need every row from an account, patient or device to stay on one side. Explain the split before talking about hyperparameters.

Hyperparameter search should have a budget and a reason. On a small dataset, an enormous grid can overfit the validation process while burning time. Start with a baseline, choose a few parameters that change model behavior and inspect whether gains repeat across folds or time windows. Keep a final holdout untouched until the selection is done.

Calibration matters when people act on the probability itself. If a model assigns 0.8 to 100 cases, roughly 80 should occur over time for that score to mean what users think it means. A ranked list can still be useful without perfect calibration, so tie the requirement to the product. The Model Evaluation session lets the decision determine which of these measures deserves attention.

Experiments get messy after launch

An A/B test starts with the unit of randomization. Randomizing page views while measuring user retention can place the same person in both variants. Randomizing users may still fail if households, teams or sellers influence each other. Choose the unit that limits contamination and matches the decision.

Define the primary outcome before looking at results. Add guardrails for harms you are unwilling to trade away, such as refunds or latency. Then specify the exposure event, analysis population and test duration. Changing those choices after seeing a favorable graph turns analysis into negotiation with yourself.

Sample ratio mismatch is a warning that assignment or logging may be broken. Peeking creates another problem because repeated checks increase the chance of stopping on noise unless the method accounts for sequential decisions. Novelty and day-of-week effects can also fool a rushed launch. An interviewer may ask what you do when a PM has already rolled the variant to everyone. Be practical. Preserve logs, find any remaining comparison group and explain the uncertainty honestly.

CUPED or other variance reduction may appear in a deeper experiment round. Explain the intuition before the acronym. A pre-experiment measure correlated with the outcome can remove some noise if it was measured before treatment and used correctly. The Experimental Design and A/B Testing session still comes back to randomization because the method cannot repair a bad assignment.

Deep learning depends on the role

The track includes a deep learning session for data scientists. Use it when the posting mentions neural networks, ranking, image or language models, or a workload that calls for them. A product data science role centered on SQL and experiments may reward deeper preparation elsewhere.

If the role does need deep learning, be ready to discuss training data, objective functions, overfitting, evaluation and the operational cost of inference. Drawing a network is the opening. The interview usually gets more useful once the model fails on a slice of users and you have to decide what to inspect.

Bring one messy dataset story, one experiment you would design differently now and one model you can explain to a skeptical product manager. Write down the unit of observation, split, metric and decision before you practice. The data science interview track is where those choices can be challenged aloud, which is usually where the weak assumption finally shows itself.