Initial Investigation of Behavioral Changes of Obfuscated Programs Caused by Code Optimization

Code obfuscation is a software protection technique against malicious analysis and tampering attacks. Since code obfuscation modifies the representation and structure of code in complex ways, many obfuscating transformations cause performance overhead, such as a decrease in execution speed. To improve execution speed while obfuscating code, code optimization is sometimes combined with code obfuscation. However, it is reported that applying optimization to obfuscated code may break the original program behavior. In this research, the behavioral changes of obfuscated programs by code optimization is investigated by analyzing the logic and coverage information before and after optimization. The results show that most of the obfuscated programs preserved the same behavior before and after optimization, while some obfuscated programs were broken by optimization. Also, a comparison of coverage information before and after optimization revealed that optimization removes some of redundant code fragments introduced by obfuscation.