Naive Bayes and the spam filter

In short: Naive Bayes is a simple, fast classification method based on Bayes’ theorem. It became known through spam filters: it estimates how likely a mail is spam from the words in it.

Bayes’ theorem

P(spam | words) ∝ P(spam) · P(word1 | spam) · P(word2 | spam) · …

• P(spam): how common spam is at all (prior knowledge).
• P(word | spam): how often a word appears in spam mails, learned from examples.

You compute the same for “not spam” and take the larger number.

Why “naive”?

The method acts as if words were independent of each other, which is never true (“viagra” and “cheap” occur together). It still often works surprisingly well and needs very little data and compute time.

Practical tricks

• Smoothing (Laplace): a never-seen word would otherwise set the whole probability to 0. So you add 1 everywhere.
• Logarithms: many small probabilities are multiplied, giving numbers near 0. With logarithms you add instead.

Today

For large text tasks neural networks are stronger. Naive Bayes remains a good baseline: if a complex model is not clearly better, the effort is not worth it.

See also: Probability, entropy and KL divergence, Metrics: accuracy, precision, recall and F1