Aggregation of Regression Models for Variance Minimization

In this work, a novel method of weighting regression models is presented. In order to minimize the variance of residual between predictions and true values of a given parameter, optimization problem is posed. Its solution invokes the use of covariance matrix, which can be reliably estimated from data. The suggested approach is tested on open source dataset, containing information on concentration of several chemical elements in various spatial locations. The performance of algorithm under study is compared to that of other algorithms, including Bootstrap aggregation (bagging), which is often considered a standard one. It is shown that from theoretical point of view the novel approach outperforms bagging, while in practice it gives better results only in some settings, which is attributed to numerical difficulties in matrix inversion.