Transformer-Based API Call Sequence Modeling for Dynamic Malware Detection

Dynamic malware detection is based on analyzing behavioral patterns captured in execution logs. In this study, we propose a hybrid deep learning approach that integrates neural networks for feature extraction with BERT for API call sequence classification to detect and classify malware. We utilize execution logs generated from the Speakeasy emulator, which provides behavioral reports for 93,533 32-bit Portable Executables (PEs). Our methodology begins with data preprocessing and filtering, followed by feature extraction driven by neural networks to capture meaningful representations of malware behavior. The extracted features are then fed into a BERT-based sequence classifier trained on ordered API call sequences to identify malware types. Experimental results demonstrate an overall accuracy of 85% on the test dataset. The model achieves improvements in F1-score for clean files (0.91) and significant enhancements for various malware types, including Trojan (0.83), Backdoor (0.90), and Ransomware (0.85). These findings highlight the effectiveness of combining neural feature extraction with transformer-based sequence learning for dynamic malware detection.