Decision trees and random forest: learning rules from data

In short: A decision tree splits data into groups by yes/no questions; a random forest averages many random trees and is more robust and accurate than a single one.

In more detail: A decision tree learns from examples a sequence of questions (“Is age > 30?”), at whose end stands a class (classification) or a value (regression). It is easy to read and is often the first model tried for tabular data.

In Depth

How a tree learns

At each node the feature and threshold that separate the data best are chosen. Measure: Gini impurity 1 − Σ pᵢ² or entropy −Σ pᵢ log₂ pᵢ; the information gain is the reduction of impurity. Example: 10 mails, 5 spam/5 ham. A split “contains ‘prize’” gives left 4 spam/0 ham (pure), right 1 spam/5 ham: clearly purer → good split. The tree grows recursively until the leaves are pure or a depth/minimum size is reached.

Strengths and weaknesses

• + Comprehensible, little preprocessing (no normalisation), mixes numbers and categories, captures non-linear relationships and interactions.
• − A deep tree overfits (memorises noise, overfitting), small data changes change the tree (high variance), diagonal boundaries are hard, no extrapolation.
• Remedies: pruning, max_depth, min_samples_leaf, cross-validation.

Random forest

Idea bagging: many trees, each on a random sample with replacement (bootstrap) of the data and with a random subset of features at each split. Classification: majority vote; regression: mean. This lowers the variance while keeping bias small. Out-of-bag error gives an error estimate without extra test data. Feature importance shows which variables count (caution with correlated features).

from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=300, max_depth=None, random_state=42)
model.fit(X_train, y_train)
print(model.score(X_test, y_test))

Going further and practice

• Gradient boosting (XGBoost, LightGBM, CatBoost): trees one after another, each corrects errors of the previous; on tabular data often top (Kaggle).
• Extra trees, isolation forest (anomalies).
• When? Tabular data (customers, prices, sensors), baselines; not for raw images/text/audio (neural networks there).
• Explainability: visualise the tree (plot_tree), SHAP.
• Practice: class imbalance (class_weight), hyperparameter search, avoid data leakage, train/test split.
• Exam: compute Gini/entropy for a simple example, draw a tree by hand.

See also: Ensemble methods, Overfitting and underfitting, Classification and regression, Explainable AI