Insights into the Generalizability and Robustness of AI-Driven Code Comment Generation: A Replication Study

In the software development life cycle (SDLC), clear and comprehensive code comments play a critical role in improving code readability, maintainability, and collaboration among developers. As software complexity increases, automated code comment generation is one of the solutions to reduce the burden of documentation while ensuring code understandability. With the rise of generative AI, large language models (LLMs) have demonstrated significant potential in generating high-quality context-sensitive code comments. However, validating the reliability and generalizability of these models requires rigorous replication studies. In this paper, we replicate the results of LAMNER: Code Comment Generation Using Character Language Model and Named Entity Recognition [17]. We replicate the main experiment using the original research setup and extend the evaluation by testing on an alternative code summarization dataset and employing an additional evaluation method that is Chatbox Arena methodology to assess generalizability. Our study provides insights into the robustness of the LAMNER model, compares replicated results with the original reported findings, and highlights challenges in replicating AI-driven code documentation techniques. We observed results that differ from those reported in the original article, with performance varying depending on the choice of data set.