malDetect: Malware Classification Using API Sequence and Comparison with Transformer Encoder

With each advancement in internet technology, new security challenges arise. The prevalence of malicious programs continues to increase, which makes it crucial to detect and address them effectively. Many researchers focus on solving different datasets by using deep learning methods and make significant progress. However, these strategies must be continuously improved to adapt to the latest data. In this paper, an improved model based on CNN-LSTM is proposed to detect and classify malware programs, named malDetect I. At the same time, the Transformer Encoder module is also modified based on model Bert to adapt to the classification task. Lastly, two models are compared with prediction results on evaluation indicators. The data used in this paper is the Windows API sequence extracted after dynamic operation. The text processing methods are also suitable for processing sequence data. The experiment uses Word2Vec and two different learning rate strategies, and the improved model accuracy is 9.83% higher than the original CNN-LSTM model. The model integrated the BiLSTM model with the Self-Attention mechanism, named malDetect II, is 11.46% higher than the basic model CNN-LSTM and 2.82% higher than the Transformer Encoder classification model.