Efficient Code Causes Inefficiency in Compiler Optimizations

Optimizations, a critical element within the compiler pipeline, is responsible for improving the efficiency of generated code. In this paper, we take a different angle to evaluate compiler optimizations than all existing works in compiler testing literature. In particular, we consider a specific scenario in software development, that is, when developers manually optimize a program to improve its performance, do compilers actually generate more efficient code with the help of developers' optimizations? To answer this question, we develop a simple approach which consists of (1) transforming a program into the same or less efficient version of the original program, and then (2) comparing the performance of the generated code from each version of the source code. Consequently, the original program can be treated as an optimized version of the transformed program. We use the approach to evaluate GCC and LLVM, two industry compilers, and observe that the extra optimizations presented in the original programs can be not only unhelpful, but more seriously counterproductive to both GCC and LLVM, resulting in the less efficient code generated overall. Out of our evaluation results, we summarized and reported 53 LLVM and 18 GCC defects, out of which 37 and 17 have been confirmed or fixed.