Support vector machines (SVM)

In short: A support vector machine separates two classes with a line (or plane) so that the distance to the nearest points of both classes is as large as possible. With the kernel trick it also separates data that is not linearly separable.

The basic idea

Many lines separate two point clouds. The SVM picks the one with the largest margin. Only the points near the boundary, the support vectors, determine the line. All other points do not matter.

Soft margin

Real data overlaps. The parameter C controls how many errors are allowed: small C = wide margin, more errors; large C = every point should be right, risk of overfitting (training, validation and test).

Kernel trick

If data cannot be separated by a line (for example a ring around a circle), you imagine mapping it into a higher space where a plane is enough. The kernel (for example RBF) gives the similarities directly without ever computing the high space.

When to use it?

• Good for small to medium data sets with many features (texts, genes).
• With millions of rows training gets slow; then decision trees and random forests or neural networks are usually better.
• Scaling the features is important.

See also: Classification and regression