Systematic Study of Compilers and Vulnerability Scanners Using the Example of Integer Bugs

Blockchain programs, also known as smart contracts, are particularly prone to attacks when they are freely accessible on public block chains and manage valuable assets. Therefore, compilers and vulnerability scanners aim at the auto-mated prevention and detection of weaknesses in such programs. To assess the effectiveness of these countermeasures, benchmark sets of real-world samples are the norm. While this approach is useful, it has shortcomings like selection bias, small size, and the controversial assessment of properties. In this paper, we advocate the use of benchmark sets that are systematically generated from templates by varying parameters along several dimensions. This way, we obtain large benchmark sets covering the weakness space in a fine-grained manner. By running tools like compilers and vulnerability scanners on such sets, we obtain profiles that allow us to assess individual properties of a tool as well as differences between tools. To showcase our approach, we take integer bugs in Solidity and the Ethereum virtual machine as an exemplary vulnerability. We collect variants of the bug from academic literature and generate two benchmark sets with 1411 and 174768 sample programs, respectively. We use the first set to compare the profiles of ten versions of the Solidity compiler and the second one to analyze the behavior of two vulnerability scanners, Mythril and Osiris. Our analysis reveals various discrepancies and peculiarities of the tools. Moreover, our systematic tests help explain why tools that address the same vulnerability tend to disagree.