Scaling laws interview questions
6 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Predicting capability from compute
Read the section that teaches thisIf this comes up in an interview
What are scaling laws and why do they matter?
Loss falls as a power law in compute, data and parameters, consistently across many orders of magnitude. They let a lab predict the loss a compute budget will reach before committing, which turns a very expensive training run into a planned decision.
What is compute-optimal training?
Allocating a fixed compute budget between model size and training tokens to minimise loss. The key finding was that they should grow together, and that earlier large models were undertrained for their size.
Why do deployed models often exceed compute-optimal training?
Compute-optimal minimises training cost. If inference volume is large, inference dominates total cost, so a smaller model trained well past that point is cheaper overall despite costing more to train.
Are emergent abilities real?
Disputed. Sudden capability jumps are widely reported, and a serious counter-argument holds that they are partly an artefact of discontinuous metrics, where smooth improvement looks like a jump under an all-or-nothing measure. The honest answer names both.
Using scaling laws: the arithmetic
Read the section that teaches thisIf this comes up in an interview
You have a fixed compute budget. How do you choose model size and data size?
Start from compute-optimal scaling, roughly 20 tokens per parameter, which gives the lowest loss for that training budget. Then adjust for inference: if the model will serve heavy traffic, a smaller model trained on more data is cheaper overall. Validate with small-scale runs and a fitted scaling curve before committing.
Then they push further
Estimate the training compute for a 1B parameter model on 100B tokens.
About 6 times 1e9 times 1e11, which is 6e20 FLOPs.
What if you do not have enough unique data?
Repeating data for a few epochs costs relatively little, but returns fall with more repetition. Beyond that, better filtering, synthetic data, or a smaller model are the options.
What gets marked up. Marked up for the 6ND estimate, the compute-optimal ratio, and the inference cost argument for overtraining.
What do scaling laws not tell you?
When specific abilities appear, how a model performs on a particular downstream task, and what happens with a different architecture or data mix than the one the law was fitted on.
These are the questions for one topic. The whole question bank covers every topic on the site.