Enhancing Software Quality with AI: A Transformer-Based Approach for Code Smell Detection

Software quality assurance is a critical aspect of software engineering, directly impacting maintainability, extensibility, and overall system performance. Traditional machine-learning techniques, such as gradient boosting and support vector machines (SVM), have demonstrated effectiveness in code smell detection but require extensive feature engineering and struggle to capture intricate semantic dependencies in software structures. In this study, we introduce Relation-Aware BERT (RABERT), a novel transformer-based model that integrates relational embeddings to enhance automated code smell detection. By modeling interdependencies among software complexity metrics, RABERT surpasses classical machine-learning methods, achieving an accuracy of 90.0% and a precision of 91.0%. However, challenges such as low recall (53.0%) and computational overhead indicate the need for further optimization. We present a comprehensive comparative analysis between classical machine-learning models and transformer-based architectures, evaluating their computational efficiency and predictive capabilities. Our findings contribute to the advancement of AI-driven software quality assurance, offering insights into optimizing transformer-based models for practical deployment in software development workflows. Future research will focus on lightweight transformer variants, cost-sensitive learning techniques, and cross-language generalizability to enhance real-world applicability.