Comprehensive Analysis of Supervised Machine Learning Algorithms in Obfuscated Malware Detection

ABSTRACT Obfuscated malware, characterized by intricate code concealment techniques, presents a formidable challenge in modern cybersecurity. This research paper delves into the nuanced landscape of obfuscated malware detection, utilizing the CIC‐MalMem‐2022 dataset comprising both benign and malicious attacks. The malicious samples in this dataset are further categorized into Spyware, Ransomware, and Trojan horse, each with multiple families, reflecting the diverse and sophisticated nature of contemporary cyber threats. The significance of this research lies in its systematic exploration of obfuscated malware detection methodologies. As cyber adversaries continually refine their tactics to evade traditional detection mechanisms, understanding and mitigating the risks posed by obfuscated malware become paramount. The proposed approach employs machine‐learning algorithms—specifically, k‐nearest neighbors, decision tree, random forest, XGBoost, and artificial neural networks—for binary and multi‐class classifications on memory dumps. In the binary classification scenario, the XGBoost algorithm stands out with a remarkable 100% accuracy, showcasing its effectiveness in distinguishing between benign and obfuscated malicious attacks. The study extends its focus to address the inherent dataset imbalance, where nearly 50% of samples are benign, and the remaining are distributed across the three malware categories. The Synthetic Minority Over‐Sampling Technique (SMOTE) is applied to balance the dataset for multi‐class classification. This research contributes to the field of cybersecurity by providing a comprehensive analysis of obfuscated malware detection, showcasing the strengths of various machine learning algorithms in binary and multi‐class scenarios. The findings underscore the significance of algorithm selection and dataset balancing techniques in enhancing the accuracy of malware detection systems, thereby advancing the state‐of‐the‐art in cybersecurity.