MHFuzz: Advancing Heap Vulnerability Detection through Phased Fuzzing Strategies

With the rapid advancement of information technology, the issue of software security is increasingly gaining attention. Heap memory, as a key component of memory management in open-source software, presents particularly prominent security concerns. Heap-related vulnerabilities such as heap memory overflow, use after free, and memory leaks can pose serious security threats. To effectively detect and prevent such vulnerabilities, in this paper we introduce a new multi-phase and heap-related operations guided fuzzer called MHFuzz. This method focuses on heap-related operations, and through static instrumentation and dynamic runtime feedback, it prioritizes the mutation of seeds that trigger more heap-related operations and allocates more energy to them, thereby enhancing the capability and efficiency of vulnerability mining.The innovation of MHFuzz lies in its phased fuzzing strategy: the first phase focuses on extensively mining heap-related vulnerabilities, while the second phase, to avoid the situation of local optima, concentrates on testing low-frequency paths to improve testing efficiency. Furthermore, MHFuzz employs non-dominated sorting and the simulated annealing algorithm to optimize the processes of seed selection and energy distribution, ensuring that superior seeds have more opportunities for mutation. Experimental assessments have shown that MHFuzz outperforms four advanced fuzzing tools in tests across eight real-world programs, not only increasing code coverage but also successfully discovering five unknown heap memory vulnerabilities, which three of them were subsequently assigned CVE numbers.