Principal component analysis (PCA)

In short: PCA finds the directions in the data with the greatest spread and describes the data with a few of them. Many features are thus condensed into a few without losing much information.

In more detail: The data is shifted around the mean, then the eigenvectors of the covariance matrix are computed. The eigenvectors with the largest eigenvalues are the principal components.

In detail

• Goal: fewer dimensions for visualisation (2D/3D), faster models, noise removal.
• Explained variance: you keep components until, for example, 95% of the spread is explained.
• Procedure: centre → covariance matrix → eigen decomposition (or SVD) → project data.
• Limits: linear relationships only. For non-linear structures t-SNE and UMAP suit visualisation.
• Making embeddings visible: high-dimensional vectors (see Embedding (vector representation)) are brought to 2D with PCA, t-SNE or UMAP.
• Standardise: bring features to the same scale beforehand.

See also: k-nearest neighbours, k-means and clustering, Data, features and labels