Sparse autoencoders: making features visible

In short: A sparse autoencoder (SAE) breaks the inner activations of a language model into many single, more understandable features. This shows which concepts the model is “thinking” of right now.

The problem

A single neuron in the network rarely stands for exactly one concept. It responds to a mixture (polysemanticity) because the network stores many more concepts than it has neurons and overlays them (superposition). That makes interpretability hard.

The solution

An SAE is a small network next to the model:

1. It takes the activation of a layer (a vector).
2. It maps it onto a much larger vector (for example 16 times), in which almost all entries are zero (“sparse”).
3. From that it reconstructs the original activation.

During training every non-zero entry is penalised in addition to the reconstruction error (see Autoencoder and VAE). What remains are few active features per input, and many of them have a clear meaning, such as “Golden Gate Bridge”, “code bug” or “deception”.

What you do with it

• Look at and name features.
• Amplify or dampen a feature artificially to steer behaviour.
• Search for features of unwanted behaviour.

Limits: not every feature is interpretable, the reconstruction is not perfect and the method is expensive. It is an active research field.

See also: Safety, alignment and jailbreaks