Optimisers: SGD, momentum and Adam

In short: The optimiser decides how the weights are adjusted using the gradient. SGD takes simple steps, Adam adapts the step size per weight and is the standard for neural networks today.

In more detail: The gradient shows in which direction the loss rises. The optimiser goes in the opposite direction, the learning rate determines the step size.

In detail

• SGD (stochastic gradient descent): w ← w − lr · g, the gradient is estimated from a small part of the data (mini-batch). Noisy but fast.
• Momentum: averages past gradients, rolls like a ball through dips and dampens zig-zag.
• RMSprop/Adagrad: scale steps by the size of previous gradients.
• Adam: combines momentum and scaling (first and second moment). Defaults β1=0.9, β2=0.999. AdamW separates weight decay from it and is used for language models.
• Learning rate: the most important setting. Too large: training diverges. Too small: takes forever.
• Schedule: warm-up at the start, then decay (cosine or linear).
• Memory: Adam keeps two extra values per parameter, so training needs about four times the memory of the model weights, see VRAM and model size.
• Newer: Lion, Adafactor, Muon.

See also: Gradient descent and learning from data, Batch, epoch and learning rate