Supervised learning interview questions

You have a tabular dataset and a classification task. What do you try first?

A trivial baseline, then logistic regression for an interpretable reference, then gradient boosted trees, which are the strongest general option on tabular data. A neural network only with evidence that it beats boosting on this dataset.

Why do gradient boosted trees still beat neural networks on tabular data?

Tabular features are heterogeneous and many are uninformative, and trees select among them rather than having to learn to ignore. Trees are also invariant to feature scaling, handle missing values natively, and work on the smaller datasets typical of tabular problems.

What is the difference between bagging and boosting?

Bagging, as in random forests, trains many independent models on resampled data and averages them, which reduces variance. Boosting trains models sequentially with each correcting the previous one's errors, which reduces bias and usually performs better with tuning.

How does a decision tree choose where to split?

It evaluates every feature and threshold and picks the split that most reduces impurity in the child nodes, measured by Gini impurity or entropy for classification and variance reduction for regression.

Why does gradient boosting work?

Each tree is fitted to the gradient of the loss with respect to the current ensemble's predictions, so adding it moves the ensemble towards lower loss. It is gradient descent over functions, which is why a small learning rate with more trees generally wins.

Can you trust linear regression coefficients as feature importance?

Not when features are correlated. The coefficients become unstable and can flip sign between runs while predictions barely change, so they stop meaning what they appear to.

Walk me through how you would approach a new tabular classification problem.

Look at the data and target balance, hold out a test set, get a dummy baseline, build a preprocessing pipeline so cross-validation is leak-free, try a regularised linear model then gradient boosting, choose on cross-validation, evaluate once on the test set, then analyse errors by slice and check calibration if probabilities are used.

What gets marked up. Strong answers mention a baseline, a pipeline to avoid leakage, a single use of the test set, and slice-level error analysis.

Explain the bias-variance trade-off.

Error from a model has a part from wrong assumptions, bias, and a part from sensitivity to the particular training sample, variance. Simple models have high bias and low variance; flexible ones the reverse. The goal is the complexity that minimises total error on new data.

Your dataset has 20% missing values in an important column. What do you do?

First find out why it is missing, since missingness can itself be informative. Options are imputation with a missing indicator, models that handle missing values natively such as gradient boosting, or dropping the column if it is missing for a reason that makes it unusable. Impute inside the cross-validation pipeline.

When would you choose logistic regression over gradient boosting?

When interpretability or calibrated, stable probabilities matter, when data is small, when features are already well engineered and roughly linear in effect, or when the model must be simple to deploy and audit. Otherwise boosting usually wins on accuracy for tabular data.

What is the curse of dimensionality?

As the number of features grows, data becomes sparse in the space and distances between points become similar, so methods that rely on neighbourhoods, like k-nearest neighbours, degrade. You need far more data to cover the space, which is why feature selection, regularisation and dimensionality reduction matter.

What does regularisation do?

It adds a cost for model complexity, such as large weights, so the model prefers simpler explanations that generalise better. L2 shrinks weights smoothly; L1 drives many to zero. Early stopping, dropout and augmentation play the same role in neural networks.