Towards Unmasking LGTM Smells in Code Reviews: A Comparative Study of Comment-Free and Commented Reviews

Code review is a crucial component of the software development life cycle and is adopted as a best practice in the industry. However, like any process, counterproductive practices and pitfalls exist within code review, such as the occurrence of the “Looks Good to Me” (LGTM) smell. LGTM smell occurs when a superficial review is conducted. LGTM review smells can potentially result in the accidental inclusion of low-quality changesets in the codebase, leading to severe bugs, possibly many reopens of the associated issues, and additional time wasted on changes to the changeset. In this study, we aim to explore LGTM smells and examine their potential impacts on the related repository. Given the inherent challenges of automatically detecting LGTM smells in code reviews, this study introduces an alternative approach by categorizing code reviews into two distinct types: comment-free and commented reviews. The primary hypothesis is that comment-free reviews are more prone to LGTM smells due to their lack of detailed examination and discourse. To test this hypothesis, we conduct an empirical analysis on a subset of pull requests (PRs) comprising code reviews from five large-scale software projects. We further investigate the impact of comment-free and commented reviews on key development metrics, specifically focusing on the number of reopens and late commits in PRs. According to the results, 64.7% of the PRs in these projects exhibited comment-free reviews. Our manual analysis reveals that comment-free reviews exhibit the LGTM smell 3.5 times more frequently than the commented reviews. We also observed a statistically significant difference, indicating that comment-free review PRs tend to include more late commits (i.e., commits made after the reviewer's approval) than the commented PRs. However, no statistically significant difference was observed in the reopening ratio of associated issues between comment-free reviews and commented reviews. Our approach provides a novel method for detecting and exploring the impacts of LGTM smell, emphasizing the significance of comprehensive code reviews and setting the stage for future research aimed at automatically identifying LGTM smell occurrences.