Decision trees and random forests

In short: A decision tree asks yes-no questions about features one after another until it lands on an answer. A random forest combines many such trees that vote.

In more detail: At each split trees choose the question that separates the data best (Gini impurity or entropy). A single deep tree overfits easily, many different trees balance this out.

In detail

• Structure: root → question (“age > 30?”) → branches → leaves with prediction.
• Advantage: readable, needs hardly any preprocessing, handles numbers and categories.
• Random forest: hundreds of trees, each on a random sample of the data and with randomly chosen features (bagging). Result: majority or average.
• Gradient boosting: trees are built one after another, each correcting the errors of the previous ones (XGBoost, LightGBM, CatBoost). Often stronger than neural networks on tabular data.
• Feature importance: shows which features contribute a lot.
• Weakness: extrapolation outside the value range, images and text.

See also: Classification and regression, Overfitting and the bias–variance trade-off