Asked about Where capability comes from
How does a language model learn?
An answer that works
By predicting the next token across an enormous corpus. General capability is a side effect: predicting well requires modelling whatever produced the text, so reasoning, world knowledge and style all emerge from that one objective.
Reading an answer is not being able to give one. The section that teaches this is Where capability comes from, 5 hours of Pretraining.
Learn this properlyAlso asked about Pretraining
What are scaling laws?Why do models produce confident wrong answers?Roughly what would it cost to pretrain a 7B model?Given a fixed compute budget, how do you choose model size and token count?What matters most for pretraining quality?Why deduplicate a pretraining corpus?Walk me through how a pretraining dataset is built.A new open model claims to beat a larger one on several benchmarks. How do you evaluate the claim?