Sequence models before transformers

Before transformers, sequences were handled by reading them one step at a time and carrying a hidden state forward — a recurrent network. The architecture is worth understanding not as history but because the two problems that killed it are exactly what attention was invented to solve, and an interviewer asking why transformers won is asking whether you know what they replaced. This section shows the mechanism, measures the failure, and says how much of it is worth learning now.

Recurrent networks#

The idea is a loop. Start with a hidden state, and for each element of the sequence combine that state with the current input to produce a new state. The same weights are used at every step, so the network can handle sequences of any length, and the final state is a summary of everything it has read.

import numpy as np

def rnn(inputs, W_x, W_h, b):
    """The whole idea. h carries everything the model remembers."""
    h = np.zeros(W_h.shape[0])
    for x in inputs:                       # strictly one step at a time
        h = np.tanh(W_x @ x + W_h @ h + b)
    return h

rng = np.random.default_rng(0)
H, D, T = 4, 3, 10
h = rnn(rng.normal(size=(T, D)), rng.normal(size=(H, D)) * 0.3,
        rng.normal(size=(H, H)) * 0.3, np.zeros(H))
print(np.round(h, 3))

# [ 0.079 -0.337  0.24   0.065]

The two problems#

The first problem is that loop. A transformer sees every position at once and computes them in parallel, so a GPU is fully occupied; a recurrent network must walk the sequence in order, and the hardware sits idle waiting for the previous step. That is a hard architectural ceiling on training speed, and it is why scaling to the sizes modern models reach was never realistic.

The second problem is that the memory does not actually reach back very far. Learning that step 100 depends on step 1 requires the gradient to travel back through a hundred multiplications by the same matrix, and repeated multiplication by anything other than exactly one goes to zero or to infinity.

multiplier   after 50 steps   after 200 steps
   0.9         5.15e-03          7.06e-10      vanishes
   0.99        6.05e-01          1.34e-01      survives, barely
   1.0         1.0               1.0           the knife edge
   1.1         1.17e+02          1.90e+08      explodes

That is the intuition; here is the measurement. Push a 60-step sequence through each architecture, take a loss that depends only on the final step, and look at how much gradient reaches each input.

import torch, torch.nn as nn
torch.manual_seed(0)

T, H = 60, 32
x = torch.randn(1, T, H)

def grad_at_each_step(cell):
    """How much does a loss at the END of the sequence depend on each input?
    That dependence is what the model is able to learn from."""
    xx = x.clone().requires_grad_(True)
    out, _ = cell(xx)
    out[:, -1].sum().backward()            # the loss uses only the last step
    return xx.grad[0].norm(dim=1)          # gradient magnitude per timestep

for name, cell in [("RNN (tanh)", nn.RNN(H, H, batch_first=True)),
                   ("LSTM", nn.LSTM(H, H, batch_first=True)),
                   ("GRU", nn.GRU(H, H, batch_first=True))]:
    g = grad_at_each_step(cell)
    print(f"{name:11} last step {g[-1]:.3e}   50 steps back {g[-50]:.3e}")

# RNN (tanh)  last step 2.084e+00   50 steps back 5.540e-19
# LSTM        last step 6.649e-01   50 steps back 7.073e-12
# GRU         last step 1.309e+00   50 steps back 6.198e-11

LSTM and GRU#

These are the fix that made recurrent networks work for a decade. Instead of overwriting the hidden state at every step, an LSTM keeps a separate cell state that is modified additively, and learns gates deciding what to forget, what to write and what to read out. The additive path is the point: information can travel many steps without being repeatedly multiplied, so the gradient has a route that does not decay. A GRU is the same idea with two gates instead of three, and it is cheaper and usually as good.

And the measurement above says how well that works, which is the honest version rarely given. The LSTM and GRU gradients fifty steps back are around 1e-11 or 1e-12 — roughly ten million times larger than the plain RNN's, and still vanishingly small. They pushed the usable range from tens of steps to hundreds. They did not remove the limit, which is why a document of several thousand tokens was out of reach until attention replaced the recurrence entirely.

How much of this to learn#

Enough to answer that question and no more, unless you are working on something that needs it. Concretely: be able to describe the recurrence in a sentence, know that the hidden state is a fixed-size summary and that this is both its strength and its limit, know what the gates in an LSTM are for, and know the two reasons transformers replaced them. Implementing an LSTM cell from scratch is a reasonable afternoon if you enjoy it and is not something interviews ask for.

What is worth more of your time is the property that made recurrence attractive in the first place, because it is coming back. A recurrent model processes a token in constant time and constant memory regardless of how much came before, where attention costs grow with the context. That trade is exactly what the state space models in the next section revisit, so the ideas here are not only historical.