Networks That Remember
Recurrent Neural Networks (RNNs) are designed for sequential data — text, time series, speech, video. Unlike feedforward networks, RNNs have loops that allow information to persist. They process inputs one step at a time, maintaining a hidden state that captures information from previous steps.
How RNNs Work
┌──────────────────────────────────────────────┐
│ RNN UNFOLDED │
│ │
│ x₁ x₂ x₃ x₄ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌───┐ ┌───┐ ┌───┐ ┌───┐ │
│ │ h₁│──►│ h₂│──►│ h₃│──►│ h₄│──► output │
│ └───┘ └───┘ └───┘ └───┘ │
│ │
│ Each step: hₜ = f(Wₕₕ·hₜ₋₁ + Wₓₕ·xₜ) │
│ │
│ The hidden state h carries memory forward │
└──────────────────────────────────────────────┘
At each time step, the network takes the current input and the previous hidden state, and produces a new hidden state. This allows information to flow across time steps — the network can "remember" earlier parts of the sequence.
The Vanishing Gradient Problem
Vanilla RNNs struggle with long sequences. During backpropagation, gradients get multiplied repeatedly and either shrink to zero (vanish) or explode. This means the network can't learn long-range dependencies — it might remember the last few words but forget the beginning of a long paragraph.
LSTM: Long Short-Term Memory
LSTMs solve the vanishing gradient problem with a more sophisticated cell structure that includes gates — mechanisms that control what information to keep, what to forget, and what to output.
┌──────────────────────────────────────────┐
│ LSTM CELL │
│ │
│ ┌────────┐ ┌────────┐ ┌────────┐ │
│ │Forget │ │Input │ │Output │ │
│ │Gate │ │Gate │ │Gate │ │
│ │ │ │ │ │ │ │
│ │What to │ │What new│ │What to │ │
│ │discard │ │info to │ │output │ │
│ │from │ │store │ │from │ │
│ │cell │ │in cell │ │cell │ │
│ └────────┘ └────────┘ └────────┘ │
│ │
│ Cell state: the "conveyor belt" that │
│ carries information across time steps │
└──────────────────────────────────────────┘
Applications
- Natural Language Processing — Machine translation, text generation, sentiment analysis
- Speech Recognition — Converting audio to text (voice assistants)
- Time Series — Stock prediction, weather forecasting, anomaly detection
- Music Generation — Composing melodies based on patterns
The Transformer Era
While RNNs and LSTMs were dominant for sequential data, Transformers (introduced in 2017 with "Attention Is All You Need") have largely replaced them for most NLP tasks. Transformers process entire sequences in parallel using self-attention, making them faster to train and better at capturing long-range dependencies. We'll cover Transformers in the NLP section.