Interpretability: what happens inside the network?

In short: Neural networks are “black boxes”: you see the numbers but do not immediately understand why they lead to an answer. Interpretability research tries to make the inside understandable.

In more detail: There are two directions: explanations for individual predictions (why this decision?) and understanding of internal mechanisms (what does the network compute?).

In detail

• Feature importance: SHAP and LIME show which inputs influenced a prediction.
• Saliency/heat maps: for images they mark regions the model attends to (Grad-CAM).
• Attention maps: show what heads attend to but are not a full explanation.
• Mechanistic interpretability: looks for features and circuits in language models. With “sparse autoencoders” one finds directions that correspond to concepts like “Golden Gate Bridge” or “deception”.
• Probing: small classifiers on internal vectors show which information is there.
• Use: debugging, safety (detecting deception), trust in sensitive areas (medicine, credit).
• Limits: explanations can be plausible but wrong. The reason the model tells is not always the real computation path either.
• Simple models (linear, trees) are explainable by themselves, see Decision trees and random forests.

See also: Safety, alignment and jailbreaks, Limits of AI: what models cannot do