Enhanced code smell detection using random forest optimized with particle swarm variants

• A hybrid feature selection approach combining Naïve Bayes (NB) filtering with Correlation-based Feature Selection (CFS). • Eleven Particle Swarm Optimization (PSO) variants are evaluated for Random Forest (RF) hyperparameter tuning in code smell detection. • The integration of optimal feature selection and PSO-based optimization significantly improves RF performance compared to baseline methods. • Experimental results show that the proposed approach NBCFS and CLPSO achieves higher accuracy, precision, recall, and F1-score with reduced feature redundancy In the software industry, the rapid growth of large-scale systems and frequent release cycles have intensified the challenge of maintaining long-term software quality. Poor design decisions often manifest as code smells, which accumulate as technical debt and hinder software maintainability and evolution. Although Machine Learning (ML) techniques, particularly Random Forest (RF), have been widely adopted for automated code smell detection, their effectiveness is strongly influenced by hyperparameter configuration, which is commonly determined through manual tuning or grid search, leading to suboptimal performance and high computational cost. This research aims to investigate whether Particle Swarm Optimization (PSO) and its variants can effectively optimize RF hyperparameters to improve the accuracy and robustness of code smell detection. The research systematically evaluates 11 PSO variants, including standard, adaptive, and hybrid approaches, by integrating them with RF and assessing their performance on the Fontana dataset for Long Method detection. Model performance is measured using accuracy, precision, recall, F1-score, and execution time. The results reveal that PSO-based optimization consistently outperforms the baseline RF model, with Comprehensive Learning PSO (CLPSO) achieving the best balance between classification performance and convergence speed. While hybrid PSO approaches further improve detection accuracy, they introduce higher computational overhead. These findings are significant as they demonstrate that swarm intelligence-based hyperparameter optimization can provide an effective and scalable alternative to manual tuning, enabling more reliable ML-based code smell detection in large and evolving software systems.