Optimizing Automated Machine Learning for Ensemble Performance and Overfitting Mitigation

Automated Machine Learning (AutoML) has revolutionized model development, but its impact on ensemble diversity and overfitting reduction remains underexplored. This Systematic Literature Review (SLR) analyzes 107 studies published between 2020 and 2024 to explore how AutoML enhances ensemble diversity, mitigates overfitting, and the challenges hindering its integration. Unlike previous reviews focusing on AutoML or ensemble methods independently, this study synthesizes their intersection and identifies key research trends. The findings reveal that AutoML improves ensemble robustness through automated hyperparameter tuning, meta-learning, and algorithmic blending while facing trade-offs in computational cost and interpretability. Four main themes emerge, integration mechanisms (19.6%), overfitting mitigation (26.2%), performance trade-offs (28.6%), and integration barriers (26.2%). Empirical results indicate that AutoML ensembles outperform traditional models by 22–41% in accuracy but require approximately 3.2 times higher computational resources. Hybrid AutoML and Explainable AI frameworks are recommended to balance accuracy and transparency. Theoretically, this study advances understanding of the synergy between AutoML and ensemble learning, while practically providing guidance for deploying reliable AI systems in sectors like healthcare, finance, and digital business. Policy implications align with the EU AI Act and the US Executive Order on trustworthy AI, supporting Sustainable Development Goals 9 and 8.