Engineering Sandbox pilot ยท easy-first ML
When does a vote beat a single tree?
Hold tree accuracy at 60% and increase the number of voters in the opening model.
Compare the majority's accuracy with one tree, then trace the road-sign predictions.
The voting model assumes independent errors; real trees can make correlated mistakes.
Tune vote size and tree accuracy
Watch bagging create diversity
Trace a majority prediction
How to use this page
Scroll until both Condorcet sliders are available. Increase the tree count without
changing accuracy, then lower accuracy toward 50%. Notice how the voting benefit
shrinks: adding voters cannot rescue chance-level predictions in this model.
Replica Hub
Warm-up: can a vote beat one tree?
The short answer is yes, if the trees are allowed to disagree for useful reasons.
Ensemble Learning
Ensemble learning creates a stronger model by aggregating the predictions of multiple weak models, such as decision trees. The companion Decision Trees explainer covers why a single tree can be unstable. Condorcet's Jury Theorem suggests that majority-vote aggregation can outperform the individual models, although other aggregation methods exist too.
Random Forest is an example of ensemble learning where each model is a decision tree. In the next section, we will build a random forest model to classify if a road sign is a pedestrian crossing sign or not. These signs come in many variations, and we will use four simple features: Size, number of sides, number of colors used, and whether the sign has text or symbol.
We will start with a sampling method called the Bagging Method to create multiple samples from the training data to build each tree.
Why disagreement helps
Compare a column in the prediction grid: do the trees miss the same sign? Bagging and
feature subsampling can reduce correlation, so aggregation can reduce prediction variance.
Shared errors survive a vote; the illustrated forest is not guaranteed to beat every tree.
Variance in Composition
We previously discussed how decision tree models suffer from high variance. However, this variance among trees is employed in the random forest as a feature, not a bug. The inventor of the random forest model Leo Breiman says in his paper "[o]ur results indicate that better (lower generalization error) random forests have lower correlation between classifiers and higher strength." [Random Forest Article]
The high variance of the decision tree model can help keep the correlation among trees low. The Bagging Method as well as the Feature Selection are the key innovations to keep correlation low.
We've just shown how to construct random forests for a given dataset, but how different are our trees from one another in reality? To find out, we've trained a nine-tree random forest on our sign dataset and plotted it below.
The outer enclosing circle represents the random forest. Each inner circle represents a unique decision tree in the forest. Hovering over a tree will highlight the tree's classification accuracy, and its feature importances* for four features.
Observe that each tree has a fairly unique combination of feature importances and accuracy and there is no obvious pattern. Some trees only use two out of four features, and the tree with similar feature importances as the random forest model still performs much worse than the forest. As discussed in Condorcet's theorem, the key takeaway is the power these models employ when aggregated together in a smart way, as shown by the random forest model having the highest accuracy.
As expected, the random forest model (the red dots on the right) performs better than any individual tree. Notice the wide range of the feature importance scores across our trees, which can contribute to low correlation. In the very first random forest visualization tool developed by Breiman, he also attempted to show the range in feature importances.
Variance in Predictions
If each tree produces the same prediction, then the accuracy cannot improve. Below, each circle represents a prediction from a model. Circles in the same row share the same model, while circles in the same column share the same test data. Blue means "Yes" and pink means "No". Solid color means that the prediction is correct, while the stripe means that the prediction is incorrect. Not all test data points are shown.
The irregular pattern in the grid shows how the trees are different in where they make mistakes. As expected, the random forest model performs the best overall even if there are trees with very low accuracy. Note that for the first data point (first column), there are still three trees with the correct prediction while the majority is incorrect. This inspired people to consider other methods, such as Boosting, which is very popular today.
Conclusion
Random Forest models are a popular model for a large number of tasks. In short, it's a method to produce aggregated predictions using the predictions from several decision trees. The old theorem of Condorcet suggests that the majority vote from several weak models with more than 50% accuracy may do the trick. Later, Breiman came up with Bagging and Feature Selection that help keep correlation among the trees low, which is a key to success.
There has been theory built around the bias and variance of random forest models in relation to the trees they contain. The sections above focus on the intuition behind that behavior.
Thanks for reading. The public route keeps the original bagging, voting, barcode, and prediction walkthroughs while leaving deeper provenance to the docs layer. The references below remain part of the lesson for anyone who wants to keep digging.