Enhanced Image-Based Malware Classification Using Transformer-Based Convolutional Neural Networks (CNNs)

As malware samples grow in complexity and employ advanced evasion techniques, traditional detection methods are insufficient for accurately classifying large volumes of sophisticated malware variants. To address this issue, image-based malware classification techniques leveraging machine learning algorithms have been developed as a more optimal solution to this challenge. However, accurately classifying content distribution-based features with unique pixel intensities from grayscale images remains a challenge. This paper proposes an enhanced image-based malware classification system using convolutional neural networks (CNNs) using ResNet-152 and vision transformer (ViT). The two architectures are then compared to determine their classification abilities. A total of 6137 benign files and 9861 malicious executables are converted from text files to unsigned integers and then to images. The ViT examined unsigned integers as pixel values, while ResNet-152 converted the pixel values into floating points for classification. The result of the experiments demonstrates a high-performance accuracy of 99.62% with effective hyperparameters of 10-fold cross-validation. The findings indicate that the proposed model is capable of being implemented in dynamic and complex malware environments, achieving a practical computational efficiency of 47.2 s for the identification and classification of new malware samples.