CNN and LSTM-Based In-Memory Malware Detection: Performance Improvement with Transfer Learning

Malware detection has become increasingly complex in recent years. Traditional antivirus software and security tools struggle to keep pace with the rapid evolution of threats, often falling behind as new malicious techniques emerge. As a result, machine learning and AI-based approaches have emerged as valuable alternatives for detecting and classifying malware.Detecting malware directly from memory using conventional methods has proven to be a particularly difficult challenge. Consequently, deep learning and machine learning techniques have become more prominent in identifying hidden malware. This study demonstrates that combining Convolutional Neural Networks (CNN) with Long Short-Term Memory (LSTM) models yields an effective approach for malware detection. Additionally, utilizing transfer learning to fine-tune pre-trained models has proven to be a robust strategy for enhancing detection performance.In this research, the CIC-MalMem-2022 dataset was utilized. This dataset not only facilitates malware detection but also provides comprehensive insights into identifying specific malware types, offering a more granular understanding of malicious behaviors.