Recurrent networks and LSTM

In short: Recurrent networks (RNN) read sequences step by step and carry a “state” along as memory. LSTM and GRU improve this with gates. They were largely replaced by transformers for text and speech.

In more detail: An RNN processes one word after another and updates a hidden state h as it goes. One problem: the state has to compress everything in the past and gradients vanish over long distances.

In detail

• RNN: hₜ = f(W·hₜ₋₁ + U·xₜ). Shares weights across all time steps.
• LSTM (Long Short-Term Memory, 1997): a cell-state line with gates (forget, input, output) decides what is remembered.
• GRU: simplified variant with fewer gates.
• Disadvantages: sequential, hard to parallelise, forgets with long texts.
• Why transformers won: attention sees all positions directly and can be computed in parallel on GPUs, see Self-attention in detail.
• Comeback: state-space models (Mamba, S4) and RWKV bring recurrent ideas back with linear cost.
• Use today: time series, small devices, audio.

See also: Self-attention in detail, Transformer architecture