Concept¶

Due to their high variance, decision trees often fail to reach a level of precision comparable to other predictive algorithms. In the previous chapter, we introduced several ways to minimize the variance of a single decision tree, such as through pruning or direct size regulation. This chapter discusses another approach: ensemble methods. Ensemble methods combine the output of multiple simple models, often called “learners”, in order to create a final model with lower variance.

We will introduce ensemble methods in the context of tree-based learners, though ensemble methods can be applied to a wide range of learning algorithms. That said, the structure of decision trees makes ensemble methods particularly valuable. Here we discuss three tree-based ensemble methods: bagging, random forests, and boosting.

An example demonstrating the power of ensemble methods is given below. Using the tips dataset from scikit-learn, we build several tree-based learners. Then for $$b = 1, 2, \dots, 30$$, we use bagging to average the results of the $$b$$ learners. For each bagged model, we also calculate the out-of-sample $$RSS$$. The blue line below shows the $$RSS$$ for each bagged model—which clearly decreases with $$b$$—and the red line shows the $$RSS$$ for a single decision tree. By averaging many trees, rather than relying on a single one, we are able to improve the precision of our model.