Measuring the performance overhead of obfuscating code transformations

Background. Code obfuscation is the practice of intentionally making a computer program, whether in native code, bytecode, or interpreted language, harder to understand. Obfuscation can ideally be performed in an automated fashion by a tool called an obfuscator. These obfuscators apply different code transformations to the source program to make it more difficult for an adversary to analyze. One example of such a transformation is virtualization in which some part of a program, like a function, gets lifted into a custom and uniquely generated virtual instruction set and replaced by an interpreter for this instruction set. The quality of these transformations is defined as a combination of the transformation's potency, resilience, and cost, where cost implies the extra space/execution time a transformation adds to an obfuscated program. Objectives. The objectives of this thesis are to measure and analyze the cost (i.e. performance overhead) of obfuscating code transformations on programs written in the C programming language. Methods. Using quantitative experiments on both single-threaded and multi-threaded pre-established benchmarks, the software measurement tool GNU Time was used to capture three metrics for programs with various transformations—or no transformation—applied: mean user time, mean elapsed time, and mean maximum resident set size. Results. Through the quantitative experiments on the various benchmarks some transformations, such as those based on virtualization, were shown to have a significant impact on execution time, especially in performance-critical functions. Others, resulted in more moderate overhead, or sometimes no significant overhead at all in certain contexts. The choice of transformation was demonstrated to have little importance on variation in maximum resident set size. Conclusions. Virtualization-based obfuscation was shown to introduce a significant overhead when applied to performance-critical functions, but not when applied to non-performance-critical functions. This demonstrates that even obfuscating transformations with a high performance cost can be applied in certain ways without significantly degrading the overall performance of the application. The transformations that showed no large overhead made it difficult to accurately draw any major conclusions about their performance cost as some statistical randomness still contributed to a slight variation in the metrics. To draw accurate conclusions about their performance impact more specifically detailed studies on the individual transformations in isolation are required.