Anomaly detection

In short: Anomaly detection finds data that does not fit the normal: fraud, defects, intrusions in a network. There are usually hardly any examples of the deviating, so you learn the normal and look for the unusual.

Why not just classification?

Anomalies are rare (often under 1%) and constantly new. A classifier needs many labelled examples of each class. With anomalies you do not have them.

Methods

1. Statistics: values far from the mean (for example more than 3 standard deviations) or outside quartile limits count as unusual. Simple, but only for single quantities.
2. Isolation forest: random trees split the data. Unusual points are isolated after a few cuts, normal ones need many. Short paths = suspicious.
3. Distance and density: points far from their neighbours (k-nearest neighbours, LOF), see k-nearest neighbours, k-means and clustering.
4. Reconstruction: an autoencoder is trained only on normal data. For anomalies the reconstruction error is high.
5. Time series: deviation from the forecast, see Time series and forecasts with AI.

The evaluation matters

Because of the rarity accuracy deceives: 99% “all normal” is 99% right and useless. Precision and recall are better (metrics). And every alarm costs attention, so the threshold is a business decision.

See also: Data, features and labels