MalRGBDet: Windows Malware Detection Method Based on RGB Image Representation and Heterogeneous Neural Network

As the world’s most widely used operating system, Windows has long been a primary target for malware attacks, causing severe economic losses and threats to data security for users and enterprises. Existing detection methods often struggle with low accuracy when dealing with complex malware, suffering from high false-negative and false-positive rates. Additionally, malware detection in Windows faces challenges such as limited datasets, a lack of benign sample contrast and insufficient original feature information. To address these issues, we propose a malware detection method based on RGB image representation and heterogeneous neural network (MalRGBDet). First, we collected malware samples from the GitHub and VirusShare platforms, along with benign software from Windows systems, to build a dataset named MalDet. This data set contains unprocessed malicious and benign samples, providing original feature information and addressing the lack of benign samples in existing data sets. Next, we extracted three key features from the malware samples: code sections, data sections and API call sequences. These features closely relate to the behavior of malware and accurately describe its operations. We then transformed these features into uniformly sized RGB images, which helped reveal hidden patterns. Finally, we employ a heterogeneous neural network that integrates ResNet and AlexNet for classification. ResNet, with its deep architecture and residual learning mechanism, significantly enhances the model’s representation capability and classification performance, thereby improving detection accuracy. Meanwhile, AlexNet’s Dropout regularization strategy effectively boosts the model’s generalization ability. In our data set of 1952 Windows software samples, MalRGBDet achieved more than 95% in accuracy, precision, recall and F1-score, improving these metrics by up to 4% compared to the latest methods. Furthermore, false-negative and false-positive rates were kept below 5%.