Bug Whisperer: Unleashing LLMs to Craft Context-Aware Tests Guided by Path Constraints and Bug Clues

Automated unit test generation using Large Language Models (LLMs) has garnered significant attention due to its potential to streamline software testing. Despite its promise, existing approaches often fall short in effectively detecting bugs. There are two key limitations in the bug detection capabilities of tests generated by LLMs: (1) It is difficult for them to identify and focus on bug-related information within large and complex codebases, particularly when bugs are embedded in intricate program dependencies and nested conditions. (2) Existing LLM-based methods are less effective in generating correct test cases when the focal method is provided without more context. To address these challenges, we propose BugWhisper, a novel approach comprising three core modules: bug detection & localization module, path constraint construction module, and test generation module. In the bug detection & localization module, we leverage custom-designed prompts to extract and highlight buggy information from LLMs. The path constraint construction module introduces a greedy-based path extraction algorithm to prioritize bug-triggering paths that cover the maximum number of buggy statements. Constraints from these paths are then used to generate path condition constraint prompts. Finally, the test generation module constructs context-aware prompts by integrating path constraints with multiple types of context information, including import context, class type context, focal class method context, and buggy statement information. These enriched prompts guide LLMs to generate more effective and precise test cases. Our evaluation on the Defects4J benchmark demonstrates that BugWhisper enhances the bug detection capabilities of LLMs. Specifically, compared with existing state-of-the-art baselines, it achieves a 78.02% improvement in compilation passing rate, and a 133.33% boost in the bug successful detection rate. Besides, BugWhisper improves line and branch coverage by 8.36% and 15.89%, respectively. Ablation studies further validate the contribution of each prompt component, highlighting BugWhisper's potential as a systematic and efficient solution for bug detection in complex software systems.