Neural networks and backpropagation

In short: A neural network consists of layers of simple computing units (neurons) that form weighted sums and pass them through a nonlinear function. Backpropagation efficiently computes how each weight affects the error.

In more detail: A layer computes y = f(W·x + b). Several layers in sequence can learn complex relationships. In training the computation runs forwards (prediction, loss), then the error is propagated backwards through all layers with the chain rule to obtain the gradients.

In Depth

• Activation functions: ReLU, GELU, sigmoid, softmax (probabilities). Without nonlinearity the network would stay linear.
• Universal approximation: Large enough networks can approximate almost any function.
• Architectures: Dense networks, convolutional networks (images), recurrent networks, transformers (text, speech).
• Vanishing/exploding gradients: Gradients become too small/large in deep networks. Remedies: ReLU, residual connections, normalisation.
• Regularisation: Dropout, weight decay, data augmentation.
• Hardware: Matrix multiplication on GPUs/TPUs.
• Embeddings: Words/objects as vectors.

See also: Gradient descent and learning from data, Transformer architecture