Ensemble Learning: An Indispensable Ingredient for Enhancing Regression Tree Performance

It is a common knowledge that ensemble learning can be used to improve performance of predictive models, but it may not work well under certain situations, such as with stable learners. Thus, ensemble approach may come across to some practitioners as only an option when searching for an accurate model. In this paper, we highlight a situation where ensemble learning is not an option. Particularly with regression trees, we argue that ensemble learning is indispensable in enhancing the estimation performance. Regression trees, while popular, are known to produce rough regression surfaces, resulting in imprecise estimations when applied as a single individual tree model. This paper shows that ensemble learning addresses this issue by combining predictions from a diverse set of regression trees, effectively smoothing the regression surface. Experimental results across five real-world datasets show that ensemble learning consistently reduces predictive errors of regression trees, and the performance of single regression trees are consistently inferior. This underscores the efficacy of ensemble learning in smoothing prediction surfaces and reducing errors. For stable learners that do not suffer from the problem of rough prediction surfaces, such as Linear Regression and Support Vector Regression, ensemble learning offers limited to no improvements over single models. These findings emphasize the necessity of applying ensemble learning whenever regression trees are employed.