REAPER: Real-Time Detection of Malicious Traffic via Deep Time-Series Embedding Analysis

Existing malicious traffic detection methods face challenges in achieving real-time detection in high-bandwidth networks, suffer from information loss due to conventional discrete statistical feature extraction, and remain vulnerable to evasion by short-flow-based malicious traffic resembling benign behavior. To this end, we present REAPER, a high-performance system for real-time detection of malicious traffic at the network flow level, built on Intel DPDK. REAPER leverages an IP-trie to dynamically aggregate short flows in real time, exposing hidden sending patterns of malicious traffic. It leverages a deep time-series embedding analysis (DTEA) method, which uses the RNN algorithm to directly convert network flow data into embeddings, followed by outlier analysis on the embeddings using an unsupervised variational autoencoder (VAE) algorithm, capable of detecting zero-day attacks. Experiments show REAPER outperforms the baseline, especially for encrypted malware traffic, achieves an average AUC of 0.9429 and EER of 0.0348 under evolving cyber attack patterns, and supports up to 5 Gbps traffic throughput, with scalability to higher rates.