Batch, epoch and learning rate

In short: A batch is a stack of examples used for one training step. An epoch is one pass through all the training data. The learning rate determines the step size.

In more detail: Instead of using one example after another or all at once you take mini-batches. That uses graphics cards well and smooths the gradient.

In detail

• Batch size: typically 32 to 512 for images, for language models millions of tokens per step (across many GPUs).
• Step (iteration): one weight update per batch. Steps per epoch = data size / batch size.
• Epoch: small models train for many epochs. Large language models often see the data only one to four times.
• Gradient accumulation: combine several small batches one after another to simulate a large one when memory is tight.
• Learning rate: usual 1e-3 (small networks) to 1e-5 (fine-tuning large models). Set it together with the batch size.
• Loss curve: you watch training and validation loss. If the curve jumps, the learning rate is too high.
• Checkpoint: save intermediate states to continue after a crash.

See also: Optimisers: SGD, momentum and Adam, Training, validation and test