Vanilla RNNs forget. Gradients vanish over 20 time steps, so the network can't remember what happened at the start of a sentence by the time it reaches the end. LSTMs (Hochreiter & Schmidhuber, 1997) fixed this with a gated "cell state" — a gradient highway that runs through time. Google Translate ran on LSTMs from 2016 to 2018. Then transformers came along and the highway became a freeway.
Learning Objectives
After this lesson, you will be able to:
Explain why vanilla RNNs fail on long sequences — gradients shrink or explode exponentially with depth in time — and why this killed sequence models for two decades
Walk through the LSTM cell: how the forget, input, and output gates use a separate cell state c_t as a 'gradient highway' so information can flow across hundreds of time steps without dying
Compare LSTM and GRU and pick the right one — GRU has fewer parameters and trains faster, LSTM has more capacity for long-range dependencies; benchmarks show they are usually within 1-2% of each other
Build a sequence-to-sequence encoder-decoder model with teacher forcing for training, and recognize the exposure-bias problem that makes inference brittle
Trace how Bahdanau attention (2014) — letting the decoder look back at every encoder hidden state instead of one fixed context vector — quietly invented the architecture that would become the transformer three years later
Don't worry if the gating math looks intimidating — every gate is just a sigmoid that outputs a number between 0 and 1, and you multiply it with a vector to control how much "flows through." Once you see the cell-state highway in action, the rest of the math falls into place.
LSTM (Long Short-Term Memory) introduced two innovations:
A separate cell state c_t that flows through time with only multiplicative gating (no transformation) — gradients can travel along it without vanishing.
Three gates (forget, input, output) — each a sigmoid that learns when to read, write, and erase.
The forget gate f_t looks at the previous hidden state and current input and outputs a vector of values in (0, 1) -- one per cell-state dimension. Multiplying f_t element-wise with c_ decides which dimensions of the old cell state to keep (1) or erase (0).
The input gate i_t controls how much new information to write. The candidate cell value g_t (sometimes called \tilde_t) is the actual content to write — a tanh of a learned linear transform.
Finally, the output gate o_t decides which parts of the (now updated) cell state to expose as the next hidden state h_t. The cell state goes through a tanh first to bring it back into a bounded range.
LSTM Gates in Action — Watch Information Flow Through the Cell StateInteractive
Loading visualization...
Try this: Step through the LSTM cell one time step at a time. Watch what each gate does as inputs come in -- the forget gate dimming old memory, the input gate writing new content, the output gate revealing only relevant parts to the next layer. Notice how the cell state can preserve a value across many time steps when forget=1 and input=0.
Try it! Open the Python REPL and type these lines yourself. PyTorch's LSTM is a one-liner: import torch; lstm = torch.nn.LSTM(input_size=10, hidden_size=20, num_layers=1); x = torch.randn(5, 3, 10); h, (h_n, c_n) = lstm(x); print(h.shape, h_n.shape, c_n.shape) — five time steps, batch of 3 sequences, 10-dim input, 20-dim hidden. PyTorch handles all the gates internally.
Empirical rule of thumb (Greff et al. 2017 "LSTM: A Search Space Odyssey"): on most NLP and time-series benchmarks, LSTM and GRU performance differs by less than the noise from random initialization. Pick whichever fits your compute budget; tune carefully on either.
What Do You Think?
You have a small dataset (~10K sequences) and limited GPU time. Your task is character-level text generation. Should you reach for LSTM or GRU first?
The right answer is GRU. With a small dataset, fewer parameters reduce overfitting risk. The faster training also lets you do more hyperparameter sweeps in the same wall-clock budget. Greff's empirical comparison showed GRU is the better default for resource-constrained settings.
#Sequence to Sequence: The Encoder-Decoder Pattern
The architecture, in pictures:
Source: "Hello world"
ENCODER (LSTM)
H → e → l → l → o → ' ' → w → o → r → l → d
\
c (context vector)
/
DECODER (LSTM)
<SOS> → "B" → "o" → "n" → "j" → "o" → "u" → "r" → <EOS>
Target: "Bonjour"
The decoder is autoregressive: at each step it takes the previously emitted token and the previous decoder hidden state, and predicts the next token's distribution.
Naively training the decoder by feeding it its own previous predictions creates a chicken-and-egg problem -- early in training the predictions are random, so the decoder learns to condition on garbage. Teacher forcing (Williams & Zipser 1989, applied to seq2seq by Sutskever) sidesteps this: during training, always feed the decoder the GROUND-TRUTH previous token, not its own prediction. Loss is still computed against the true next token.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
Tests · Verify both LSTM and GRU losses drop below 0.5 within 200 epochs. Confirm the LSTM cell's forget/input/output gates are computed correctly via chunk(4).
The vanilla seq2seq model has a fatal bottleneck: the entire source sentence has to fit into ONE fixed-size context vector. Translating a 50-word sentence? Compressed into the same 512-dim vector as a 5-word sentence. Information loss is catastrophic for long sentences.
Bahdanau, Cho, and Bengio (2014) introduced attention to fix this: instead of one context vector, let the decoder LOOK BACK at every encoder hidden state, with learned weights that say "for the word I am generating right now, which source words should I attend to?"
The win was immediate: Bahdanau-attention seq2seq outperformed vanilla seq2seq by 5+ BLEU on long sentences and matched the state-of-the-art statistical MT systems on short ones. Three years later, Vaswani et al. asked: "what if attention is all we need?" -- and showed that you can drop recurrence entirely. The transformer was born from this insight.
Vanishing gradients through time killed vanilla RNNs. The gradient is a product of T derivatives, and tanh derivatives < 1 collapse the product to zero before signals from 30+ steps ago can influence learning
LSTM's cell state is a gradient highway. The additive update c_t = f_t ⊙ c_ + i_t ⊙ g_t has no nonlinearity on the main path, so gradients flow back unchanged when forget≈1 and input≈0; this is what lets LSTMs learn 100+ step dependencies
GRU is a simpler, often-equivalent alternative. Merges forget and input into one update gate, drops the cell state, has ~25% fewer parameters; on most benchmarks matches LSTM within noise (Greff 2017)
Seq2Seq turned a stack of phrase rules into a single network. Encoder LSTM compresses input, decoder LSTM generates output, trained end-to-end with teacher forcing; powered Google Translate from 2016 to the transformer era
Bahdanau attention was the seed of the transformer. Letting the decoder look back at every encoder state with learned weights solved the fixed-context-vector bottleneck and inspired "Attention Is All You Need" three years later
Why does the LSTM cell state c_t allow gradients to flow back through long sequences without vanishing?
Gating gave us memory across hundreds of steps; attention gave us memory across the entire sequence at once. Up next: Embeddings — how words and categories become vectors that LSTMs and transformers can actually crunch.