Sign Diversity: A Method for Measuring Diversity in Base Learner Selection for Ensemble Regression

Data-driven modeling is essential for predicting the behavior of complex industrial systems, but such systems often challenge machine learning models due to their intricacies. Ensemble learning enhances predictive performance by combining multiple base learners (BLs), where diversity among BLs is a critical factor. Existing diversity measures typically rely on general statistical metrics not tailored for ensemble learning and do not address the challenge of determining an appropriate ensemble size. This study introduces the Weighted Sign Disagreement Metric (WSDM) and its extension, which are methods specifically designed for measuring diversity among BLs in stacking ensemble regression tasks. Both methods incorporate the concept of sign diversity by considering the signs of BL predictions relative to target values. These methods introduce an adjustable hyperparameter to balance the trade-off between accuracy and diversity among BLs. The extension of WSDM specifically addresses the challenge of determining an appropriate ensemble size, enhancing the flexibility and effectiveness of ensemble design. Experimental results on six synthetic datasets demonstrate that WSDM and its extension significantly improve predictive accuracy and generalization compared to traditional diversity measures. By addressing the deficiencies of existing methods and exploring the trade-off between accuracy and diversity, these methods offer a robust framework for heterogeneous stacking ensemble regression.