Asked about Build and train a small transformer from scratch
Walk me through a minimal GPT-style model.
An answer that works
Token ids go through token and position embeddings and are added. A stack of blocks follows, each with layer norm, causal multi-head self-attention added back to the residual, then layer norm and an MLP added back. A final norm and a linear layer produce logits over the vocabulary. Training minimises cross-entropy between logits and the next token at every position.
Then they push further
Why is the loss at initialisation a useful check?
An untrained model should predict roughly uniformly, so its loss should be close to the log of the vocabulary size. Much higher means a bad initialisation or a bug, such as targets not shifted correctly.
Why is the causal mask needed during training?
Training predicts every position at once. Without the mask, each position could attend to the tokens it is supposed to predict, and the model would learn to copy rather than to predict.
What gets marked up
Strong candidates can write this from memory and explain residual adds, pre-norm and the mask. Many applicants for research roles are asked to implement part of it live.
Reading an answer is not being able to give one. The section that teaches this is Build and train a small transformer from scratch, 8 hours of PyTorch.
Learn this properlyAlso asked about PyTorch