CNN-AutoMIC: Combining convolutional neural network and autoencoder to learn non-linear features for KNN-based malware image classification

Malware refers to malicious software or a component of software intended for malicious purposes. The manual analysis and detection of malicious software is challenging due to its complexity. Thus, several automated solutions have become popular for real-time malware detection. A spread-out approach consists of generating images from the samples bytecode and giving them to convolutional neural networks (CNNs), which are used either as classifiers or feature extractors for further classification algorithms. These systems perform extremely well when trained and tested on partitions of the same dataset. However, cross-dataset tests and malware detection verification on emerging real-world samples are required in the real-world context. This is a crucial challenge when probing the robustness of the systems and models. This paper proposes CNN-AutoMIC,a robust automated approach to extract features from malware images. CNN-AutoMIC employs a specific CNN architecture to extract features, followed by an autoencoder-based compressor that reduces features to two fundamental components. The two-dimensional projection of these components is the basis of the predictions performed by the K-nearest neighbors (K-NN) algorithm. Moreover, the observable placement of new samples on the obtained scatter plot makes it possible to explain why the AI-based system produced a certain prediction. It was benchmarked against several CNN-based models and a Vision Transformer. They were trained on the Malevis dataset and cross-dataset evaluated on four different real-world datasets. CNN-AutoMIC outperformed the competitors for each classification performance metric, while requiring a reasonable training and prediction time. In addition, it achieves a promising Akaike information criterion (AIC) score, indicating its efficiency in terms of model complexity. • This paper introduces CNN-AutoMIC, an innovative image-based malware detection system that includes a convolutional neural network, an autoencoder, and the K-nearest neighbors algorithm. • The convolutional neural network that extracts features from malware images is implemented with a simple but effective architecture. • The autoencoder has a deep architecture and is only employed in the encoding part. Given the extracted features, the autoencoder allows them to be compressed into pairs of values used as spatial coordinates. • The K-nearest neighbors allows a new point, placed on a scatter plot, to be predicted by considering points in the training dataset as neighbors. • The entire system was tested with a cross-dataset approach and on data obtained from real-world contexts.