Bootstrap aggregation for regression problems via generalized least squares

Bootstrap aggregation, commonly known as bagging, is a foundational technique in ensemble learning aimed at improving the predictive performance of models. The effectiveness of bagging largely depends on how correlations among the aggregated models are managed. For example, random forests, a popular ensemble method, mitigate this issue by randomly selecting features to reduce the correlation between individual tree models. In this study, we introduce a bootstrap aggregation method for regression tasks based on the concept of generalized least squares to enhance the predictive accuracy of bagging models. Furthermore, we propose a two-stage method to balance computational accuracy and complexity. We provide theoretical analysis establishing unbiasedness and optimality, and empirical experiments demonstrate the effectiveness of the proposed approach while maintaining low computational cost.