Attentionsmelling: Using Large Language Models to Identify Code Smells

Large Language Models (LLMs) are becoming essential tools in software engineering, automating tasks like code generation, unit testing, and code review. However, their potential to identify code smells, indicators of poor code quality, remains underexplored. This study evaluates GPT-4’s effectiveness in identifying three common code smells (Long Method, God Class, and Feature Envy) across four experimental setups ranging from using only source code and code smell definitions to leveraging additional context, metrics, and hyperparameter optimization. Our analysis revealed remarkable improvements across all metrics when improving the prompt with more information, with overall performance increasing by 64% in ROC Curve and 56% in F1-score. These results emphasize the impact of incorporating metrics and hyperparameter tuning into LLM prompts, enabling significant advancements in automated software quality assessment. This may lead to better coding practices, particularly related to identifying code smells.