Ensemble Learning

Overview

  • Ensemble Methods in Machine Learning - Reviews voting, bagging, boosting, and Bayesian averaging; explains how statistical, computational, and representational differences between learners can make their combination useful. Dietterich 2000.

Bagging and randomized trees

  • Bagging Predictors - Fits predictors on bootstrap samples and aggregates their outputs. Particularly useful for unstable learners whose predictions change substantially when the training data change. Breiman 1996.
  • Random Subspaces - Builds decision trees on randomly selected feature subsets and combines their predictions. Feature sampling occurs per tree; Breiman’s random forest also samples candidate features at individual splits. Tin Kam Ho 1998.
  • Random Forests - Combines randomized trees; the widely used construction couples bootstrap sampling with random feature subsets at each split. Individual tree strength and error correlation both affect ensemble performance. Breiman 2001.
    • Bias in Random Forest Variable Importance - Shows selection and importance biases with mixed predictor types or category counts; investigates conditional inference trees and subsampling as remedies. Strobl et al. 2007.
    • Variable Selection Using Random Forests - Uses importance rankings and nested models to select variables; distinguishes identifying relevant variables from choosing a smaller set for prediction. Genuer et al. 2010.
    • Random Forest Methodology and Practical Guidance - Reviews tuning, implementations, applications, and pitfalls in bioinformatics, including biases in variable-importance measures. Boulesteix et al. 2012.
  • Extremely Randomized Trees - Randomizes candidate features and split thresholds, then selects among those candidates. The original Extra-Trees construction uses the full training sample rather than bootstrap replicates. Geurts et al. 2006.

Boosting foundations

  • The Strength of Weak Learnability - Proves equivalence of weak and strong learnability in the distribution-free PAC model. The result assumes a weak learner for the required distributions, not merely one classifier above chance on one dataset. Schapire 1990.
  • AdaBoost - Sequentially reweights training examples and combines weak classifiers with error-dependent weights. Misclassified examples gain weight, directing later learners toward current errors. Freund and Schapire 1997.
  • Arcing the Edge - Formulates adaptive reweighting as optimization of functions of the edge; develops an alternative perspective on the behavior of voting classifiers. Breiman 1997.
  • Boosting the Margin - Relates voting-classifier generalization bounds to the margin distribution, helping explain continued improvement after training error reaches zero. Schapire et al. 1998.
  • A Short Introduction to Boosting - Compact overview of AdaBoost, training-error bounds, margins, and connections with support-vector machines. Freund and Schapire 1999.

Gradient boosting

  • Boosting Algorithms as Gradient Descent - Treats boosting as functional gradient descent on prediction or margin losses; introduces the DOOM II algorithm. Mason et al. 1999.
  • Gradient Boosting Machine - Builds an additive model by fitting base learners to negative loss gradients. The loss determines the target: residuals for squared error, with other objectives supporting classification and robust regression. Friedman 2001.
    • Stochastic Gradient Boosting - Fits each boosting update on a random subsample drawn without replacement. Adds row sampling to gradient boosting; distinct from stochastic gradient descent on model parameters. Friedman 2002.
  • XGBoost - Combines regularized tree objectives, first- and second-order loss information, sparsity-aware split finding, and weighted quantile sketches with systems optimizations for large datasets. Chen and Guestrin 2016.
    • Out-of-Core GPU Gradient Boosting - Combines external-memory data access with gradient-based sampling in XGBoost to train on data exceeding GPU memory capacity while limiting the working set. Ou 2020.
  • LightGBM - Introduces gradient-based one-side sampling, retaining large-gradient rows while sampling others, and exclusive feature bundling, combining sparse features that rarely take nonzero values together. Ke et al. 2017.
  • CatBoost - Uses ordered boosting and permutation-based target statistics for categorical features to reduce prediction shift caused by reusing training targets. Prokhorenkova et al. 2018.

Learning how to combine predictors

  • Voting and Averaging - Hard voting combines class labels; soft voting averages predicted probabilities. Regression ensembles commonly average numerical predictions. Weights can be fixed or estimated on separate validation data. docs.
  • Stacked Generalization - Trains a second-level model on base-model predictions for observations excluded from their fitting data. Learns how to combine predictors with different errors and biases. Wolpert 1992.
  • Ensemble Selection - Greedily selects models from a fitted library to optimize ensemble performance on validation data. Selection with replacement permits unequal model weights. Caruana et al. 2004.
  • Super Learner - Uses cross-validated risk to learn a combination of candidate prediction algorithms; theoretical guarantees depend on the loss and regularity conditions. van der Laan et al. 2007.
  • Stacking predictive distributions - Chooses mixture weights using held-out predictive scores, often via leave-one-out inference. Yao et al. target prediction when all candidate models may be wrong; weights are not posterior probabilities that models are true. Yao et al. 2018.

Distributional prediction and statistical estimation

  • Quantile Regression Forests - Uses forest-derived observation weights to estimate the conditional response distribution and its quantiles, rather than only the conditional mean. Meinshausen 2006.
  • Generalized Random Forests - Learns adaptive neighborhoods for local moment equations, supporting quantiles and heterogeneous effects. Its inference guarantees require the paper’s sampling and regularity assumptions. Athey et al. 2019.
  • SWAG - Fits a Gaussian approximation to SGD iterates using their mean and a low-rank-plus-diagonal covariance; sampled weights yield averaged predictions and approximate Bayesian uncertainty. Maddox et al. 2019.
  • NGBoost - Boosts parameters of a predictive distribution using natural gradients and a proper scoring rule. Produces distributional predictions instead of interpreting variation among ordinary trees as an interval. Duan et al. 2020.
  • Jackknife+ - Combines leave-one-out residuals with test-point predictions to form intervals. Under exchangeability and symmetric fitting, the general coverage bound is 1−2α for nominal 1−α; near-nominal results need further stability assumptions. Barber et al. 2021.

Neural ensembles

  • Deep Ensembles - Averages predictive distributions from separately initialized and trained networks. The method uses proper scoring rules and studies predictive uncertainty in classification and regression. Lakshminarayanan et al. 2017.
  • Snapshot Ensembles - Uses cyclic learning rates to collect several model checkpoints along one training trajectory, then averages their predictions. Multiple checkpoints still require evaluation at inference time. Huang et al. 2017.
  • Fast Geometric Ensembling - Uses short learning-rate cycles late in training to collect nearby predictors for an ensemble, motivated by low-loss paths connecting neural-network solutions. Garipov et al. 2018.
  • BatchEnsemble - Shares weight matrices across ensemble members and modulates them with member-specific rank-one factors. Reduces parameter storage while retaining separate predictions; members are not independently parameterized networks. Wen et al. 2020.
    • TabM - Adapts BatchEnsemble to tabular multilayer perceptrons, sharing most weights across predictors and averaging their outputs. Gorishniy et al. 2025; code.

Model merging and compression

  • Knowledge Distillation - Trains a student to match temperature-scaled teacher probabilities, including those from an ensemble. Retains information about relative class probabilities while reducing the models needed at inference. Hinton et al. 2014.
  • Model Soups - Averages weights of compatible models fine-tuned from a shared pretrained initialization. Produces one model for inference; weight averaging is a different operation from averaging predictions. Wortsman et al. 2022.
  • TIES-Merging - Merges fine-tuned models by trimming small parameter changes, resolving conflicting update signs and combining sign-aligned changes; targets interference between task-specific updates. Yadav et al. 2023.

Evaluation

  • Out-of-Bag Evaluation - Estimates performance using predictions from bootstrap members that excluded each observation. Reusing these scores for extensive model selection still calls for a separate final evaluation. Breiman 2001; docs.
  • Uncertainty Under Dataset Shift - Compares uncertainty methods, including ensembles, across shifted classification datasets. Calibration on the original distribution does not establish calibration after a shift. Ovadia et al. 2019.
  • Stacking Leakage - Train the combiner on held-out or out-of-fold base predictions. The documented prefit mode risks overfitting when base models and the combiner use the same training observations. docs.

Software

  • scikit-learn - Bagging, random forests, Extra-Trees, AdaBoost, gradient boosting, voting, and stacking with estimator APIs. Its guide distinguishes histogram-based and conventional gradient boosting. docs.
  • XGBoost - Tree boosting for classification, regression, and ranking, with CPU/GPU and external-memory workflows. code, docs.
  • LightGBM - Histogram-based tree boosting with leaf-wise growth, categorical splits, and distributed training. Algorithm and device settings determine which optimizations are used. docs.
  • CatBoost - Gradient boosting with categorical-feature processing, CPU/GPU training, and prediction tools. docs, docs.
  • ranger - C++ random forests with an R interface, including classification, regression, probability, and survival forests. code.
  • randomForest - R classification and regression forests. Classification and Regression by randomForest explains fitting, out-of-bag estimates, and importance measures. Liaw and Wiener 2002; code.
  • grf - R/C++ generalized forests for regression, quantiles, causal effects, and other statistical targets. code, code.
  • SuperLearner - R implementation of cross-validated ensemble prediction, with wrappers for candidate learners and methods for estimating their combination. code.