Asked about Where sequence models still matter
Why might you choose a recurrent or state space model over a transformer?
An answer that works
When inputs stream indefinitely and memory or compute per step must stay fixed, such as on-device audio or sensors. Their cost per step does not grow with history, while a transformer's attention cost does.
Then they push further
What do you give up?
Exact access to any past token. The fixed state must compress the history, so tasks that need to look up a specific earlier detail tend to favour attention.
What gets marked up
Marked up for framing the choice as fixed state versus growing context, and naming the recall trade-off.
Reading an answer is not being able to give one. The section that teaches this is Where sequence models still matter, 3 hours of Sequence models before transformers.
Learn this properlyAlso asked about Sequence models before transformers