Transformers & attention interview questions

Explain how attention works.

Each position produces a query, a key and a value. The query is compared against every key to produce weights, and the output is a weighted mixture of the values. It lets any position draw directly on any other regardless of distance, and unlike recurrence it parallelises across the sequence.

What gets marked up. Strong answers explain queries, keys and values in plain terms, the scaling, and the mask, and connect them to the cost of long context. Reciting the formula without explaining it is marked down.

Why did transformers replace recurrent networks?

Recurrence processes sequentially, so distant information degrades through a carried summary and nothing parallelises across the sequence. Attention connects any two positions directly and computes all positions at once, which is what made training at scale practical.

Why is long context expensive?

Attention compares every position with every other, so cost grows with the square of sequence length. Doubling the input quadruples the work, which is why context limits exist and why efficient attention is an active research area.

What do multiple attention heads do?

Each head learns a different kind of relationship, such as grammatical agreement in one and topical relevance in another, and their outputs are combined. One set of weights can only represent one notion of relevance.

What does causal masking do?

It prevents each position attending to later positions, so the model learns left-to-right prediction. It also lets the full sequence be trained in parallel, since every position predicts its next token at once.

What is the difference between attention and the feed-forward layer?

Attention moves information between positions. The feed-forward layer transforms each position independently. A block alternates the two, with residual connections around each.

Why does a transformer need position information at all?

Attention compares every position with every other and blends the results, and none of those operations knows where anything is. With no position signal the model is exactly permutation-equivariant: shuffle the input tokens and the outputs come back in the same shuffled order, unchanged. Position has to be injected from outside, either as a vector added per slot or, in current models, by rotating queries and keys by an angle proportional to position so the comparison depends on the gap between two tokens.

What are residual connections and layer norm for?

Two different jobs. The residual makes each block add to a running vector rather than replace it, so information survives dozens of layers and gradients have a short path back; without it the output of a deep stack is uncorrelated with its own input. Normalisation keeps the scale of that running vector predictable, since every block adds to it and the norm grows with depth. Current models use RMSNorm, which divides by the root mean square and applies a learned scale, and it is a rounding error in the parameter count.

What is the KV cache?

Stored keys and values for tokens already processed, so each generation step computes only the new position rather than recomputing the whole prefix. It makes generation practical and consumes significant memory on long contexts.

What is grouped-query attention and why does it matter?

Several query heads share one key and value head instead of each having its own. The KV cache shrinks by exactly that ratio, so a model with 32 query heads and 8 key and value heads has a quarter of the cache. Since the cache rather than the weights usually caps concurrency, that is four times the requests or four times the context for the same memory. Quality drops slightly, and more so for multi-query attention with a single shared pair, which is why eight is the usual compromise.

What gets marked up. Strong answers give the ratio and connect it to concurrency rather than to speed. Saying it makes the model faster without mentioning the cache misses the point.

Why did rotary position embeddings replace learned absolute positions?

A learned table has a fixed number of rows, so there is nothing for a position beyond the training length, and it encodes absolute position when what attention needs is relative distance. RoPE rotates queries and keys by an angle proportional to position, so their dot product depends only on the gap between them. Extending context means rescaling those frequencies, which works to a point and costs some retrieval quality.

A model card says 120 billion total parameters and 5 billion active. What does that tell you about serving it?

It is a mixture of experts: a router picks a couple of experts per token, so compute per token is that of a 5 billion parameter model while all 120 billion must sit in memory. It will be cheap per token and expensive to host. Sizing the GPU from the active figure is the standard mistake.

Is FlashAttention an approximation?

No. It computes exactly the same attention, tiling the work so blocks fit in on-chip memory and using a running softmax so the full scores matrix is never written to GPU memory. It does slightly more arithmetic and far less memory traffic, which is the whole speedup, and it makes attention memory linear in sequence length.

Implement causal self-attention in NumPy.

Project the input three times to get q, k and v of shape (T, d_k). Compute scores as q @ k.T divided by the square root of d_k, giving (T, T). Set the strictly upper triangle to negative infinity, softmax each row, then multiply by v to get (T, d_k). For multiple heads, reshape the feature axis into (n_heads, d_head), transpose heads in front of positions, do the same computation batched, transpose back and reshape, then apply the output projection.

What gets marked up. Strong candidates state the shape of each intermediate as they write it and test their own code unprompted. Getting the mask right before the softmax, and scaling by d_head, separate people who have written this from people who have read it.

Walk through the shapes of multi-head attention for a batch of 2, length 6, d_model 16 and 4 heads.

Input (2, 6, 16). Each projection keeps (2, 6, 16). Reshape to (2, 6, 4, 4) and transpose to (2, 4, 6, 4), so heads sit in front of positions. Scores are (2, 4, 6, 6), one square matrix per batch item per head, and the mask (6, 6) broadcasts over the first two axes. Weights times values gives (2, 4, 6, 4); transpose back to (2, 6, 4, 4), reshape to (2, 6, 16), then the output projection leaves it there.

Why does the softmax need the maximum subtracted?

Purely for numerical stability. Exponentiating a large score overflows to infinity in floating point and the division then gives NaN. Subtracting the row maximum is algebraically a no-op because the constant factors out of numerator and denominator, and it guarantees the largest exponent is exp(0), so nothing overflows and the smallest underflows harmlessly to zero.

Your attention weights sum to 0.6 on the first row and 1.0 on the last. What is wrong?

The mask is being applied after the softmax rather than before. The first position can see one entry out of many, so once the remaining entries are zeroed most of its normalised weight has been thrown away; the last position can see everything, so nothing is lost. Move the mask to the scores and use negative infinity.