Generic Malware Unpacking: Existing Solutions, Requirements, New Approach for Windows Malware

Malware continues to be a substantial threat to cybersecurity, amplified by the widespread use of malicious executable packing, so-called packers. These packers inflate the number of unique samples in the wild by introducing polymorphism and hinder and delay in-depth malware analysis, making unpacking an essential first step. <br> Security researchers are developing countermeasures against these packers, called unpackers. Early unpackers targeted specific packer types, but the variety of different packers prompted the development of generic malware unpackers. Such tools aim to unpack the original binaries without prior knowledge of used packer's properties and capabilities, relying instead on broadly applicable assumptions about packer behavior. The more generic the assumptions are made, the more generic the unpacker. However, the lack of empirical understanding of packer capabilities has forced researchers to rely on subjective experience in practical malware analysis when defining these assumptions. <br> To remedy this, this dissertation defines scientifically sound prerequisites for generic malware unpackers and demonstrates their application in a proof-of-concept tool. This is accomplished by conducting studies on packer capabilities, deriving unpacker requirements from those insights, and using these requirements as the basis to develop a generic malware unpacker. <br> Since Windows malware running on x86 and x86_64 processors is the most common type of malware, it is the main focus in this work. Two studies have been conducted to explore the unpacking behavior of Windows malware: one in a singular process and the second across multiple processes. The results of these studies have been used to formulate requirements for a Windows-focused generic malware unpacker. These requirements were then applied to evaluate the genericity of previously proposed solutions. No previously proposed unpacker meets all the requirements. The final steps demonstrate how a generic malware unpacker can be implemented based on the previously identified requirements. A new generic malware unpacker called GeMU is proposed and implemented into a proof of concept. Evaluation on three Windows malware data sets used in the unpacking behavior studies confirms that GeMU achieves high coverage across diverse samples.