Ensemble methods: bagging, boosting and stacking

In short: Ensembles combine many simple models into a strong one. That evens out errors: many slightly different opinions together are often better than a single one.

Bagging

You train many models independently, each on a random sample of the data (with replacement), and average the answers. This lowers the variance. The random forest (decision trees and random forest) is bagging with trees.

Boosting

Models are trained one after another. Each new model concentrates on the cases the previous ones got wrong. This lowers the bias. Known implementations: AdaBoost, gradient boosting, XGBoost, LightGBM, CatBoost. For tabular data they are often the best classical method.

Stacking

Several different models (tree, SVM, network) give predictions and one more model learns how to combine them best.

Rules of thumb

1. Bagging: many deep, independent models, good against overfitting.
2. Boosting: many shallow models learning one after another, very accurate but more sensitive to noise.
3. Ensembles are harder to explain, see Explainable AI: SHAP and LIME.

See also: Loss functions, Classification and regression