Linear and logistic regression

In short: Linear regression lays a line (or plane) through data points: output = weight × input + intercept. Logistic regression turns this into a probability for two classes with a sigmoid function.

In more detail: Both are the simplest learning models and are a building block of neural networks: a single neuron is at its core a linear regression with an activation function.

In detail

• Linear regression: y = w1·x1 + w2·x2 + … + b. Weights w and b are chosen so that the mean squared error is minimal. There is a closed-form solution (normal equation) and gradient descent.
• Logistic regression: p = σ(w·x + b) with σ(z) = 1 / (1 + e^(-z)). Output between 0 and 1, interpretable as probability. Loss: cross-entropy.
• Interpretation: the sign and size of w show how a feature acts, hence well explainable.
• Limits: only linear relationships (except with feature tricks like squaring).
• Regularisation: L1 (lasso) sets weights exactly to zero, L2 (ridge) makes them small.
• Example: price ≈ 3000·area + 20000·rooms + 15000.

See also: Classification and regression, Neurons, layers and activation functions