Gradient descent and learning from data

In short: In machine learning a model adjusts its parameters step by step so that an error measure (loss) gets smaller. Gradient descent follows the direction of steepest descent.

In more detail: A model makes predictions, the loss measures the deviation from correct values. The gradient shows how the loss changes for small changes of each parameter. You shift the parameters a small step against the gradient (learning rate). You repeat that thousands of times.

In Depth

• Update rule: w ← w − η · ∂L/∂w (η = learning rate).
• Learning rate: Too large: jumping/divergence. Too small: slow.
• Stochastic/mini-batch: Gradient from small chunks of data instead of all data, faster and noisy.
• Optimisers: Momentum, Adam (adaptive learning rates), AdamW.
• Local minima and saddle points: Mostly unproblematic in high dimensions.
• Loss examples: Mean squared error (regression), cross-entropy (classification).
• Normalisation and good initialisation stabilise training.

See also: Neural networks and backpropagation, Overfitting and the bias-variance trade-off