Ensembles

AdaBoost Classifier (Adaptive Boosting).

Fits a sequence of weak classifiers (decision stumps by default) on re-weighted versions of the data, then combines them via weighted majority vote.

Algorithm: SAMME (Stagewise Additive Modeling using a Multi-class Exponential loss). The decision tree does not take sample weights, so each round trains the tree on a bootstrap sample drawn with probability proportional to the current weights, while the weighted error and the weight update use the full training set.

Fitting stops early when a round has zero weighted error (that tree gets weight 1) or when a tree is no better than random guessing. Class labels must be integers.

AdaBoost Regressor (Adaptive Boosting for Regression).

Uses the AdaBoost.R2 algorithm with decision tree regressors as base estimators and predicts the weighted median of the trees' predictions. As in the classifier, each tree is trained on a bootstrap sample drawn with probability proportional to the current sample weights.

Bagging Classifier (Bootstrap Aggregating).

Trains decision tree classifiers on random subsets of the rows and features and averages their predicted class probabilities. predict returns the class with the largest averaged probability, as scikit-learn's BaggingClassifier does.

Class labels must be integers. Fitting again discards the previous ensemble.

Bagging Regressor (Bootstrap Aggregating for Regression).

Trains decision tree regressors on random subsets of the rows and features and averages their predictions. Fitting again discards the previous ensemble.

Gradient Boosting Classifier.

Uses gradient boosting with shallow regression trees for classification and the binomial deviance (log loss). The leaf values are Newton steps (Friedman's TreeBoost), computed on the rows each tree was trained on.

- Two classes: one boosted model on the log-odds, started from the log of the class ratio. - More classes: One-vs-Rest. One binary model is boosted per class, and probabilities are the per-class sigmoids normalized to sum to one. This differs from scikit-learn, which boosts all classes jointly with a multinomial deviance and a softmax.

Class labels must be integers. With subsample < 1 each stage trains on a random subset drawn without replacement.

Gradient Boosting Regressor.

Builds an additive model in a forward stage-wise fashion using regression trees as weak learners. Each tree is fitted to the negative gradient of the loss; for the "lad", "huber" and "quantile" losses the leaf values are then replaced by the line-search optimum for that loss (leaf median, Huber-corrected median or leaf quantile of the residuals), as in Friedman's TreeBoost and scikit-learn.

Losses: "ls" squared error (initial value: the mean of y), "lad" absolute error (initial value: the median), "huber" (initial value: the median; the transition point is the alpha quantile of the absolute residuals) and "quantile" (initial value: the alpha quantile of y).

With subsample < 1 each stage trains on a random subset drawn without replacement. With nIterNoChange set, validationFraction of the rows are held out at random and boosting stops after nIterNoChange stages without a validation-loss improvement.

Stacking Classifier.

Combines multiple classifiers using a meta-learner (final estimator) that learns to combine the base estimators' outputs.

The meta-learner is trained on out-of-fold predictions: with cv folds (default 5, rows assigned to folds in turn, stratified by class), each base estimator is cloned and fitted on cv - 1 folds and predicts the remaining one. The base estimators passed to the constructor are then fitted on all the data (in place) and used at prediction time. Training the meta-learner on in-sample predictions instead would let it trust base estimators that memorized the training set. Set cv: 1 to do that anyway.

Two cases fall back to in-sample predictions for the affected rows: a fold whose training part has fewer than two classes, and datasets with fewer than two rows.

By default (stackMethod: "auto") the final estimator receives the base estimators' class probabilities: one column per estimator for two classes (the probability of the larger label), K columns per estimator otherwise.

Class labels must be integers.

Stacking Regressor.

Combines multiple regressors using a meta-learner (final estimator) that learns to combine the base estimators' predictions.

The meta-learner is trained on out-of-fold predictions: with cv folds (default 5, rows assigned to folds in turn), each base estimator is cloned and fitted on cv - 1 folds and predicts the remaining one. The base estimators passed to the constructor are then fitted on all the data (in place) and used at prediction time. Set cv: 1 to train the final estimator on in-sample predictions instead, which lets it trust base estimators that memorized the training set.

Voting Classifier.

Combines multiple classifiers via weighted majority voting (hard) or weighted averaged probability (soft) voting. Hard voting resolves ties in favor of the smallest class label, like scikit-learn.

The estimators passed to the constructor are fitted in place (they are not cloned). Call VotingClassifier.clone to get an unfitted copy.

Voting Regressor.

Combines multiple regressors by averaging their predictions, optionally with per-estimator weights.

The estimators passed to the constructor are fitted in place (they are not cloned). Call VotingRegressor.clone to get an unfitted copy.