<scp>Vital</scp> : Vulnerability-Oriented Symbolic Execution via Type-Unsafe Pointer-Guided Monte Carlo Tree Search

How do we find new memory safety bugs effectively when navigating a symbolic execution tree that suffers from the well-known path explosion challenge? Existing solutions either adopt path search heuristics to maximize coverage rate or chopped symbolic execution to skip uninteresting code (i.e., manually labeled as vulnerability-unrelated) during path exploration. However, most existing search heuristics are not vulnerability-oriented, and manual labeling of irrelevant code-to-be-skipped relies heavily on prior expert knowledge, making it hard to detect vulnerabilities effectively in practice. This paper proposes Vital , a new vulnerability-oriented path exploration for symbolic execution with two innovations. First, a new indicator (i.e., type-unsafe pointers) is suggested to approximate vulnerable paths. A pointer that is type-unsafe cannot be statically proven to be safely dereferenced without memory corruption. Our key hypothesis is that a path with more type-unsafe pointers is more likely to be vulnerable. Second, a new type-unsafe pointer-guided Monte Carlo Tree Search algorithm is implemented to guide the path exploration towards the areas that contain more unsafe pointers, aiming to increase the likelihood of detecting vulnerabilities. We built Vital on top of KLEE and compared it with existing path searching strategies and chopped symbolic execution. In the former, the results demonstrate that Vital could cover up to 90.03% more unsafe pointers and detect up to 57.14% more unique memory errors. In the latter, the results show that Vital could achieve a speedup of up to 30x execution time and a reduction of up to 20x memory consumption to detect known vulnerabilities without prior expert knowledge automatically. In practice, Vital also detected one previously unknown vulnerability (a new CVE ID is assigned), which has been fixed by developers.