Improving Obfuscated Malware Classification: A Machine Learning Approach with Feature Selection and Data Balancing

Obfuscated malware refers to malicious software that has been deliberately modified to hide its malicious purpose and avoid detection by conventional security mechanisms. Classifying obfuscated malware poses a significant challenge due to its deceptive nature and the dynamic techniques used by attackers to obfuscate code. Existing approaches in malware analysis often struggle to accurately classify obfuscated malware, leading to potential security vulnerabilities and data breaches. In this research, we investigate the application of machine learning algorithms for malware analysis, specifically focusing on multi-class classification of obfuscated malware using the CIC-MalMem-2022, a latest dataset. We apply data preprocessing techniques, including Synthetic Minority Over-sampling Technique (SMOTE) and normalization, to enhance the dataset's balance and consistency. Subsequently, we train and evaluate various machine learning algorithms, including Random Forest (RF), XGBoost, K-Nearest Neighbors (KNN), LightGBM, and Decision Trees (DT), on different datasets comprising SMOTE, normalized, and SMOTE+normalized instances. Our study also explores the effect of feature selection process on the performance of the model. We identify the top 35 features for optimal accuracy, addressing the dimensionality challenge inherent in large-scale datasets. Our findings reveal that the Random Forest (RF) model achieves the highest accuracy of 96.23% on the dataset that is balanced by SMOTE with top 35 features.