Hydra-Reviewer: A Holistic Multi-Agent System for Automatic Code Review Comment Generation

Review comment generation is a crucial task in code review, and significant progress has been made in automating. Previous research has generated review comments by fine-tuning pre-trained models or Large Language Models (LLMs). However, these studies have overlooked the necessity of conducting code reviews from multiple perspectives, resulting in the omission of potential issues in code changes. Additionally, the complexity of review comments often hinders the accurate quantitative evaluation of automated tools’ effectiveness. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">In this paper, we first conduct an empirical study to propose a comprehensive taxonomy of code review dimensions. We also identify three major limitations of existing automated code review (ACR) methods: lack of comprehensiveness, incorrectness, and vagueness. Building on the insights from our empirical study, we introduce H<small>ydra</small>-R<small>eviewer</small>, a collaborative multi-agent framework powered by large language models, designed to automatically generate high-quality code reviews. We utilize the CodeReview and CodeReviewNew benchmark datasets, along with a newly constructed review comment generation dataset. We compare H<small>ydra</small>-R<small>eviewer</small> with several baselines, including CodeReviewer, LLaMA-Reviewer, ChatGPT, Comprehensive-ChatGPT, and DeepSeek-V3. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">The experimental results show that H<small>ydra</small>-R<small>eviewer</small> achieves a BLEU score of 8.20, outperforming the state-of-the-art baseline, DeepSeek-V3, which scores 7.85. In qualitative evaluation, H<small>ydra</small>-R<small>eviewer</small>’s generated comments span an average of 7.8 review dimensions, addressing the limitations of existing ACR methods effectively. Additionally, H<small>ydra</small>-R<small>eviewer</small> demonstrates strong generalization capabilities on unseen dataset. We further validate the contributions of each component of H<small>ydra</small>-R<small>eviewer</small> through an ablation study and confirm the helpfulness and readability of the generated comments via a User Study. Finally, a cost analysis reveals that H<small>ydra</small>-R<small>eviewer</small> generates review comments at an average cost of 0.018 dollars and 62.63 seconds per code change.