Sequence models before transformers interview questions
3 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Sequence models before transformers
Read the section that teaches thisIf this comes up in an interview
What problems did recurrent networks have?
Long-range dependencies degraded, because information had to survive repeated compression into a fixed-size hidden state and gradients vanished across many steps. And they could not be parallelised across a sequence, since each step depended on the previous one, which capped training scale.
What did LSTM add?
Gates that let information pass through the cell state largely unchanged, which helped gradients survive across many steps. It addressed long-range dependencies and did nothing about the lack of parallelism.
Where sequence models still matter
Read the section that teaches thisIf this comes up in an interview
Why might you choose a recurrent or state space model over a transformer?
When inputs stream indefinitely and memory or compute per step must stay fixed, such as on-device audio or sensors. Their cost per step does not grow with history, while a transformer's attention cost does.
Then they push further
What do you give up?
Exact access to any past token. The fixed state must compress the history, so tasks that need to look up a specific earlier detail tend to favour attention.
What gets marked up. Marked up for framing the choice as fixed state versus growing context, and naming the recall trade-off.
These are the questions for one topic. The whole question bank covers every topic on the site.