Do LLMs Suggest Consistent Identifiers? An Empirical Study on GitHub Pull Requests

The appropriate naming of identifiers is crucial in the source code, as names can represent up to 70% of all characters and play a key role in helping developers understand the intent behind the code. Renaming is also among the most common refactoring operations performed by developers. Although static analysis tools can detect violations of naming conventions, there remains a gap in tools capable of suggesting names that are semantically aligned with the purpose of the code. In this study, we collected 152 Java and 168 Python instances of GitHub pull requests (PRs) that improved the identifier names. We investigated whether Large Language Models (LLMs) are capable of suggesting names that are semantically consistent with those chosen by developers. We also compared three temperature settings, ranging from deterministic to creative, to assess how this parameter influences naming suggestions. Our results show that ChatGPT slightly outperforms the other models for Python identifiers, exactly matching the developers’ chosen names in 16.2% of the cases and achieving high semantic similarity in 65.1%. Overall, the three LLMs performed better in Python than in Java. Regarding the temperature setting, the moderate temperature yielded slightly better results in Python, while the creative setting performed slightly better in Java. These findings suggest that tasks such as naming identifiers may benefit from different parameter configurations than those typically used in more deterministic tasks such as code refinement.