Neurons, layers and activation functions

In short: An artificial neuron multiplies its inputs by weights, adds them with a bias and passes the result through an activation function. Many neurons in layers form a neural network.

In more detail: The activation function brings non-linearity into play. Without it a deep network would be just one linear mapping and could not learn complex patterns.

In detail

• Neuron: a = f(w₁x₁ + w₂x₂ + … + b). Weights w and bias b are learned.
• Layer: many neurons side by side. As matrix calculation: a = f(W·x + b). This is exactly what graphics cards accelerate.
• Input, hidden, output layer: “deep” means many hidden layers.
• ReLU: max(0, x). Simple, fast, standard in many networks. Problem: neurons can “die”.
• GELU, SiLU/Swish: smoother variants, common in transformers. SwiGLU is used in modern LLMs.
• Sigmoid, tanh: common in the past, today mainly at the output or in gates (LSTM).
• Softmax: at the end for probabilities, see Softmax and probabilities.
• Universality: even a network with one wide hidden layer can approximate any continuous function, deep networks do so more efficiently.
• Counting parameters: a layer from 1000 to 1000 neurons has 1 million weights plus 1000 biases, see Parameter (AI model).

See also: Neural networks and backpropagation, Initialisation, normalisation and dropout