Leveraging Feature Selection and Deep Learning for Accurate Malware and Ransomware Detection in PE Files

AI -driven malware detection, particularly using machine learning (ML) and deep learning (DL), has shown promise in the field of mal ware detection. While many studies focus on portable executable (PE) file features, fewer explore ransom ware detection using PE headers with deep learning. Additionally, limited research examines how feature selection impacts machine learning performance, which is crucial for optimizing detection accuracy. This paper investigates feature selection strategies to improve malware detection while minimizing feature count. We analyzed different dataset segments' impact on ML algorithms, refining a strategy to determine the optimal dataset proportion for training. We applied Principal Component Analysis, Mutual Information, and Chi-square feature selection techniques on two datasets: (1) 2,157 ransomware PE-header samples with 1,028 features and (2) 29,807 Windows malware samples with 54 features. Seven ML models were tested alongside deep learning models. For ransomware detection, the LSTM model achieved an accuracy of 99.7%. In the Malware dataset, an accuracy of 99.9% was obtained across all evaluation metrics using only 10 features selected with the Chi-square method when applied with NB, LR, ET, and SVM. Comparable results were achieved using mutual information in conjunction with RF and LR. For deep learning, the LSTM model attained an accuracy of 98.9%. In the Ransomware dataset, an accuracy of 99.9% was achieved using RF and ET with Chi-square on 500 selected features out of 1,027. PCA combined with LR resulted in an accuracy of 99.4%. These results emphasize the effectiveness of feature selection in enhancing both the accuracy and efficiency of mal ware and ransom ware detection.