SyREC: A Symbolic-Regression-Based Ensemble Combiner

Symbolic Regression (SR) is the task of finding a concise white-box mathematical expression that fulfills a given machine learning (ML) objective. In this work, we introduce the first-of-its-kind SR-based ensemble combiner (SyREC) for ML which utilizes the quasi-arithmetic mean (parameterized by a function <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$f$</tex>) as an ensemble combiner and discovers <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$f$</tex> via SR. Our Sy REC demonstrates advantages over existing ensemble combiners, i.e., averaging methods and meta-learners. Compared to averaging methods, Sy REC allows for increased combiner complexity via exploring a large function class and provides a learning process to tune the ensemble combiner to the specific dataset. Compared to meta-learners, SyREC preserves important ensemble properties, like idempotency, monotonicity and boundedness, which produce predictions more consistent with the base models. We note SyREC produces a white-box ensemble combiner that lies in the intersection of both categories of existing combiners, addressing the weaknesses while capturing the strengths of each category. To evaluate Sy REC, we present experiments on 16 open-source PMLB datasets and commonly used base models, including a large variety of experiments and analyses across different settings: (i) regression and classification tasks, (ii) equally weighted combiners and custom-weighted combiners, (iii) homogeneous and heterogeneous ensembles, (iv) sequential and parallel ensembles. We find that in all these settings, SyREC shows consistent improvement over existing ensembling approaches. This indicates that SyREC is a robust ensemble combiner, which produces explainable and consistent ensemble predictions.