PyTorch interview questions

What does model.eval() do?

Switches layers that behave differently at inference, notably dropout and batch normalisation, into evaluation mode. Forgetting it makes validation numbers noisy and pessimistic, and forgetting to switch back disables dropout for the rest of training.

Why wrap validation in torch.no_grad()?

It stops the framework recording operations for gradient computation, which is unnecessary during evaluation. It saves memory and time, and without it a large validation pass can run out of memory.

Your GPU utilisation is 20 percent during training. What is wrong?

Almost certainly data loading, not the model. Raise num_workers, enable pin_memory, and check for expensive per-item preprocessing that should be precomputed or moved onto the GPU.

Why might a tensor's gradient be None after backward?

The operation that produced it was not recorded in the autograd graph, usually because it was detached, created from a Python number, or computed inside no_grad. Autograd can only backpropagate through what it recorded.

How would you speed up a training run?

Profile first, because the bottleneck is usually data loading or synchronisation rather than arithmetic. Then mixed precision, more data loader workers, removing per-step .item() calls, and torch.compile.

A regression model's loss decreases slowly but predictions are nonsense. What would you check first?

Shapes. A prediction of shape [N, 1] subtracted from targets of shape [N] broadcasts to [N, N] and computes a meaningless loss without error. Assert shapes where the loss is computed, and squeeze or reshape explicitly.

What gets marked up. Marked up for naming the silent broadcast bug specifically. It is one of the most common real training bugs.

Walk me through a minimal GPT-style model.

Token ids go through token and position embeddings and are added. A stack of blocks follows, each with layer norm, causal multi-head self-attention added back to the residual, then layer norm and an MLP added back. A final norm and a linear layer produce logits over the vocabulary. Training minimises cross-entropy between logits and the next token at every position.

What gets marked up. Strong candidates can write this from memory and explain residual adds, pre-norm and the mask. Many applicants for research roles are asked to implement part of it live.

Your model's training loss is flat from the first step. How do you debug it?

Check the initial loss against the expected value for uniform guessing, try to overfit one small batch, decode and inspect a batch with its labels, sweep the learning rate by factors of ten, and log gradient norms to see whether gradients reach the parameters at all.

What gets marked up. Marked up for 'overfit one batch' as an early step and for a systematic order rather than random changes.

What do you need to save to resume a training run exactly?

Model weights, optimiser state, scheduler state, the step or epoch counter, random number generator states, and the position in the data. Missing the optimiser state is the common mistake: Adam restarts with empty moment estimates and training behaves differently.