Early Detection of Windows Crypto‐Ransomware Based on API Sequences and Deep Learning

ABSTRACT Ransomware attacks continue to cause substantial losses to individuals and organizations. Dynamic detection executes samples in a controlled environment and monitors run time behavior, making it suitable for identifying ransomware before destructive encryption begins. This paper proposes API‐RED, a Windows ransomware detection model based on API call sequences. API‐RED uses a domain‐specific BERT‐style encoder trained from scratch on Windows API sequences to generate contextual API embeddings and derive attention‐based API groups. These representations are then processed by a CNN‐BiLSTM feature extractor to capture local API‐group patterns and long‐range sequence dependencies. We evaluate API‐RED using 200 benign samples, 478 ransomware samples, and a public dataset. The proposed model achieves an accuracy of 98.82%, a precision of 98.59%, a recall of 97.02%, and an F1‐score of 97.80%, outperforming the compared baseline methods. Ablation experiments further confirm the contribution of the API‐domain encoder, API grouping, CNN, and BiLSTM components.