Maths basics for AI: vectors, matrices, tensors

In short: AI computes almost exclusively with lists of numbers. A vector is a list, a matrix a table, a tensor a multidimensional table. The most important operation is matrix multiplication.

In more detail: A word, an image or a sound is first turned into numbers. All layers of a network are then matrix multiplications with weights, followed by simple functions.

In detail

• Vector: [0.2, −1.5, 3.1]. An embedding is a vector with hundreds to thousands of numbers.
• Dot product: sum of pairwise products. It measures similarity (same direction = large). Basis of attention.
• Matrix: rows × columns. A layer with 4096 inputs and outputs is a 4096×4096 matrix (16.8 million numbers).
• Matrix multiplication: C = A·B, each cell is a dot product of a row and a column. Compute grows with size, GPUs are built for it.
• Tensor: multidimensional arrangement, for example an image batch [batch, height, width, colour channels] or [batch, tokens, dimension].
• Norm and distance: length of a vector, distance between two vectors, cosine similarity (angle).
• Derivative and gradient: the gradient is the vector of all partial derivatives and points in the direction of steepest ascent, see Gradient descent and learning from data.
• Probability: distributions, expected value, entropy, see Probability, entropy and KL divergence.
• Libraries: NumPy, PyTorch, JAX, see AI frameworks: PyTorch, TensorFlow and JAX.

See also: Neurons, layers and activation functions, Embedding (vector representation)