Exploring JVM Garbage Collector Testing with Event-Coverage

Garbage Collection (GC) in the Java Virtual Machine (JVM) serves as an automatic memory management mechanism, efficiently reclaiming unused memory space in different production scenarios. To optimize JVM performance, developers typically fine-tune the garbage collector by identifying an optimal set of GC configurations for specific scenarios. Despite the sophisticated design of garbage collectors, they still have the potential for bugs in different settings, and these bugs can result in more severe consequences. Hence, comprehensive testing of these garbage collectors is imperative before their release. Code coverage criteria are typically employed to assess the comprehensiveness of a test suite. However, traditional code coverage metrics, such as branch coverage, are hardly applicable for GC testing due to their inherent concurrency. Additionally, existing JVM testing techniques do not adequately consider the characteristics of garbage collectors, making it difficult to test these garbage collectors sufficiently. In this article, we make the first effort to design coverage criteria against garbage collectors based on the events of GC called Event-Coverage . Its key insight is to measure the diversity of GC executions for testing purposes by assessing the range of GC events these executions cover. Furthermore, we design a new testing method for maximizing Event-Coverage called GCFuzz . GCFuzz conducts an exhaustive investigation of the memory state space of GC and thoroughly explores the memory state under various GC configurations. To enhance GCFuzz’s efficiency in achieving higher Event-Coverage, we have further designed a coverage-driven strategy for preserving candidate seed programs and selecting GC configurations. Extensive evaluations demonstrate a positive correlation between Event-Coverage and the bug-revealing efficiency. Moreover, GCFuzz outperforms state-of-the-art techniques in detecting unique GC-related inconsistencies and achieving higher Event-Coverage. Remarkably, GCFuzz has identified 20 previously undetected GC bugs, with 15 of them already confirmed or fixed by developers.