Explainable AI: SHAP and LIME

In short: Explainable AI (XAI) wants to show why a model reached a prediction. SHAP and LIME are two common methods that state for single predictions which features influenced them and how strongly.

LIME

LIME (Local Interpretable Model-agnostic Explanations) changes the input slightly, watches how the prediction changes and fits a simple model to it that holds near this input. The simple model is readable (“high because of A, low because of B”). It is fast, but the randomness of the variations can make results fluctuate a little.

SHAP

SHAP comes from game theory (Shapley values): each feature is a “player”, the prediction the “payout”. A feature’s value is its average contribution over all possible orders. The sum of all contributions plus a base value gives exactly the prediction. For tree models there is a fast, exact variant.

Example

Credit model: base 30% default risk. Income −8, debt +12, age −1 → prediction 33%. You see that mainly the debt raised the risk.

Warnings

• An explanation shows what the model uses, not what is causal in the world (causation instead of correlation).
• Strongly linked features share the contribution in a confusing way.
• For language models other ways are needed, see Interpretability.

See also: Bias, fairness and data quality