Pretraining interview questions
9 questions across 4 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Where capability comes from
Read the section that teaches thisIf this comes up in an interview
How does a language model learn?
By predicting the next token across an enormous corpus. General capability is a side effect: predicting well requires modelling whatever produced the text, so reasoning, world knowledge and style all emerge from that one objective.
What are scaling laws?
Loss falls predictably as a power law in compute, data and parameters, across many orders of magnitude. They let a lab decide how to spend a compute budget before committing, and the compute-optimal finding showed size and data should grow together.
Why do models produce confident wrong answers?
The objective rewards plausible continuations, not calibrated uncertainty. Fluency and correctness were never separated during pretraining, so a wrong answer is produced with the same confidence as a right one.
Roughly what would it cost to pretrain a 7B model?
Compute is about six FLOPs per parameter per token, so seven billion parameters on a compute-optimal 140 billion tokens is about 5.9e21 FLOPs. An H100 does 989 TFLOP/s of bf16 peak and you realise perhaps forty per cent of it, which is roughly 4,100 GPU-hours, so about twelve thousand dollars at three dollars an hour, or under three days on sixty-four cards. A production model trained on ten times the data costs ten times that.
Then they push further
Where does the six come from?
Two FLOPs per parameter per token for the forward pass, since each weight is one multiply and one add, and about twice that again for the backward pass, which computes gradients with respect to both activations and weights.
Why is forty per cent the number you used rather than peak?
Model FLOPs utilisation. Real runs lose throughput to communication, memory movement, pipeline bubbles and kernels that are not at peak. Thirty to fifty per cent is the usual band for a well-tuned large run, and quoting peak FLOPs gives an estimate that is two to three times optimistic.
What gets marked up. Strong answers compute rather than recall, state the utilisation assumption explicitly, and note that the compute-optimal figure is not what a model you can call was actually trained on.
Given a fixed compute budget, how do you choose model size and token count?
C equals 6ND fixes the trade, so choosing a size chooses the token count. The compute-optimal point is around twenty tokens per parameter, which for a 1e22 FLOP budget lands near nine billion parameters. Then adjust for serving: if the model will be called heavily, go smaller and train longer, because training is paid once and inference is paid every request.
Data is the model
Read the section that teaches thisIf this comes up in an interview
What matters most for pretraining quality?
Increasingly the data: deduplication, quality filtering and the mixture of sources. Architectures have largely converged, and much of the remaining gap between models comes from what they were trained on.
Why deduplicate a pretraining corpus?
Repeated text is memorised rather than generalised from, and it wastes compute on seeing the same thing many times. Near-duplicates matter as much as exact copies.
The pretraining pipeline, end to end
Read the section that teaches thisIf this comes up in an interview
Walk me through how a pretraining dataset is built.
Start from web crawls, extract main text, identify language, apply heuristic filters, use a quality classifier, deduplicate exactly and approximately, remove personal and unsafe content, then mix in code, books, papers and maths in tuned proportions, and tokenize.
Then they push further
Why does deduplication matter so much?
Duplicates waste compute on repeated content, overweight whatever happens to be copied often, increase memorisation of specific texts, and can leak benchmark items. Removing them improves quality per token.
How would you decide the data mixture?
Train small proxy models on different mixtures and compare their downstream evaluations, then scale up the best. Mixtures are chosen empirically, not by principle.
What gets marked up. Marked up for quality filtering and deduplication as central, and for empirical mixture selection with small models.
Reading a model card and technical report
Read the section that teaches thisIf this comes up in an interview
A new open model claims to beat a larger one on several benchmarks. How do you evaluate the claim?
Check whether the comparison used the same prompts, settings and number of shots, whether contamination was addressed, whether the benchmarks match my use case, and then run my own evaluation on my tasks. Also check the licence and the real usable context length.
These are the questions for one topic. The whole question bank covers every topic on the site.