Exposing Vulnerabilities in PDF Malware Detectors to Evasion Attacks

We introduce a methodology to evaluate the effectiveness of PDF malware detectors by automatically generating diverse variants of malicious PDF documents capable of evading detection. Our technique employs a series of semantics-preserving transformations that adjust the characteristics of a PDF exploit while maintaining the integrity of the embedded code and the PDF document’s structure. We introduce a comprehensive set of transformations, as well as a time-efficient Monte Carlo Tree Search (MCTS) algorithm that determines the most effective combination of these transformations for black-box evasion of PDF malware detection systems.We demonstrate that our system can generate malicious documents for various classes of PDF malware that can evade both signature-based and machine-learning-based detectors. In targeted evasion scenarios against specific antivirus (AV) and machine-learning detectors, we achieved a 100% evasion rate for select malware classes. For untargeted evasion, our method improved the average evasion rate across 76 AVs on VirusTotal by 26.9% across 19 classes of PDF exploits.